• Corpus ID: 55625680

Comparative Power of Three Author-Attribution Techniques for Differentiating Authors

@inproceedings{Archer1997ComparativePO,
  title={Comparative Power of Three Author-Attribution Techniques for Differentiating Authors},
  author={John Beville Archer and John L. Hilton and G. Bruce Schaalje},
  year={1997}
}

• J. B. Archer, J. Hilton, G. B. Schaalje
• Published 1997
• Linguistics

Comparative Power of Three AuthorAttribution Techniques for Differentiating Authors G. Bruce Schaalje, John L. Hilton, and John B. Archer Abstract: Over the last twenty years, various objective authorOver the last twenty years, various objective authorattribution techniques have been applied to the English Book of Mormon in order to shed light on the question of mUltiple authorship of Book of Mormon texts. Two methods, one based on rates of use of noncontextual words and one based on word-pattern ratios, measure patterns consistent with multiple authorship in the Book of Mormon. Another method, based on vocabulary-richness measures, suggests that only one author is involved. These apparently contradictory results are reconciled by showing that for texts of known authorship, the method based on vocabulary-richness measures is not as powerful in discerning differences among authors as are the other methods, especially for works translated into English by a single translator. Two dollar-bill changers are available in the building where we work. One is of an older style, but it is our favorite. It recognizes that a dollar bill is not bogus even when the bill is old and washed out. The modern changer is more conservative. The dollar bill has to be crisp and bold to convince this machine that it is not counterfeit. Both machines are valid dollar-bill changers in the sense that they give change when they are absolutely sure that a real dollar bill has been fed into them. Neither machine has been replaced,' so we can assume that neither machine makes errors in the sense of getting fooled by counterfeit bills. But it would be a 48 JOURNAL OF BOOK OF MORMON STUDIES 6/1 (1997) mistake to conclude that the piece of paper in your hand is a counterfeit dollar bill just because the conservative machine in the main lobby will not accept it. If you were trying to detect counterfeit bills, the old north-wing machine would be much more useful. Wh~n it does not accept a bill, you can be fairly sure that something about the bill is really strange. You can think of the old north-wing machine as being more powerful in discerning the difference between real and counterfeit money. What has this story to do with authorship analysis? Several objective author-attribution techniques are in current use, all oriented around the idea of assigning numerical measures to various aspects of authors' styles in an attempt to answer questions about texts of unknown or disputed authorship. 1 These techniques, which have proliferated and gained popularity since the advent of accessible high-speed computers, are like bill changers. If it is suspected, for example, that a literary text traditionally ascribed to Shakespeare was not in fact written by Shakespeare, both the controversial text and others known to have been . authored by Shakespeare can be examined using an objective author-attribution technique. If the technique reveals a large statistical difference between the controversial text and the known Shakespearean texts, such strong evidence implies that Shakespeare did not write the controversial text. But if only a small difference is found, we cannot make any conclusion unless we know how powerful the attribution technique is in discriminating among authors. The test we used may be like the bill changer in the main lobby-too conservative to pick out the real difference. This simple but subtle point was not initially understood by Holmes, who computed various measures of "vocabulary richness" for segments of text drawn from the Book of Mormon, the Doctrine and Covenants, the book of Abraham, Isaiah, and personal writings of Joseph Smith.2 These measures reflect aspects of a writer's working vocabulary, such as its size and the writer's habits for drawing upon it. Using statistical methods of investi1 David I. Holmes, "Authorship ~ Attribution," Computers and the Humanities 28 (1994): 87-106. 2 David I. Holmes, "A Stylometric Analysis of Mormon Scripture and Related Texts," Journal of the Royal Statistical Society Series A 155 (1992): 91-120. SCHAAUE, HILTON, ARCHER, AUTHOR-A1TRIBUTION TECHNIQUES 49 gating differences among entities for which several numerical measures are available, Holmes showed that based on vocabularyrichness measures, the texts seemed to fall into three distinct groups: (1) Isaiah texts, (2) segments of Joseph Smith's personal writings, and (3) all the rest. Because texts ascribed to different Book of Mormon authors did not segregate on a prophet-byprophet basis nor differ very much from Doctrine and Covenants or book of Abraham texts, Holmes concluded that they were all written by the same author. He proposed that they were all the work of Joseph Smith and that they differed in vocabularyrichness from Joseph Smith's personal writings only because Smith was apparently able to write in a distinct "prophetic voice" when he desired.3 Holmes did not recognize that his conclusions would only be reasonable if his vocabulary-based authorattribution technique could be shown to be very powerful in distinguishing among authors. Holmes was not aware that his findings about the similarity of working vocabularies used by different Book of Mormon prophets was not original. Hilton reported that "new word introduction rates" in Book of Mormon writings ascribed to different prophets were very similar.4 Holmes was also not aware that in a separate study, Hilton had used certain noncontextual word-pattern ratios as an author-attribution technique and had thereby shown that Book of Mormon texts attributable to Nephi and Alma differed significantly.5 However, Holmes was aware that Larsen, Rencher, and Layton had applied yet another objective author-attribution technique to Book of Mormon writings and had also shown that writings of different Book of Mormon prophets differed significantly in their rates of use of common noncontextual words.6 Holmes argued that his technique must be preferable to that of 3 David I. Holmes, "Vocabulary Richness and the Prophetic Voice," Literary & Linguistic Computing 6 (1991): 259-68. 4 John L. Hilton, "Some Book of Mormon 'Word Print' Measurements Usin~ 'Wrap-around' Block Counting" (Provo, Utah: FARMS, 1988). John L. Hilton, "On Verifying Wordprint Studies: Book of Mormon Authorship," BYU Studies 30/3 (1990): 89-108; also available as a FARMS reprint. ' 6 Wayne A. Larsen, Alvin C. Rencher, and Tim Layton, "Who Wrote the Book of Mormon? An Analysis of Wordprints," BYU Studies 20/3 (1980): 22551. 50 JOURNAL OF BOOK OF MORMON STUDIES 611 (1997) Larsen et al. because his method used all textual words in its calculations, but he provided no support, empirical or theoretical, to validate this statement.7 It is interesting, therefore, that in a recent paper Holmes reversed his position and praised the use of noncontextual word frequencies when he found that authorship attribution based on vocabulary richness was not able to segregate Federalist Papers texts attributed to Hamilton, Madison, and Jay as clearly as the method based on rates of use of common noncontextual words.8 It seems entirely possible that texts of different authorship but translated by a single translator, as the English Book of Mormon texts are claimed to be, could exhibit the vocabulary richness of the translator, but still have unique rates of use of noncontextual words and word patterns common to the original authors. If so, the findings of Holmes do not give any weight to the position that Joseph Smith was the sole author of the Book of Mormon. The purpose of this study is to use texts of known authorship to investigate the relative power of each of the three authorattribution techniques mentioned above. Both original nontranslated works and translated works are used in this study. This information will be helpful in correctly interpreting results of studies for which differences are not detected. Author-Attribution Techniques Many objective author-attribution techniques are in current use; however, because of their connection to work on the Book of Mormon, we concentrate on three techniques-methods based 0 n measures of vocabulary richness, on the rates of use of common noncontextual words, and on noncontextual word-pattern ratios. The various measures will be referred to generically as "stylometric measures." Most of these measures are corrected for the length of the text, but to further guarantee that text length did not influence the outcome, we used texts of 5,000 words each in the current study. 7 Holmes, "Stylometric Analysis," 98. 8 David I. Holmes and D. I. Forsyth, ''The Federalist Revisited: New Directions in Authorship Attribution," Literary and Linguistic Computing 10 (1995): 111-27. SCHAAUE, HILTON, ARCHER, AUTHOR-A1TRIBUTION TECHNIQUES 51 Holmes ~uggested five measures of vocabulary richness (VR) for use in studying disputed authorship questions.9 The first two measures, which he termed hapax legomena (R) and hapax dislegomena (V 2N), are counts of once-used and twice-used words, respectively, standardized by the length of the text. Two of the other three measures are related to specific probability models for vocabulary usage, but will neither be used nor discussed further here because Holmes shows that all three are somewhat redundant and concludes that "for characterizing the differences between the textual samples, therefore, only variables Rand V2N need to be computed."10 Larsen et al. based their work on the frequency of occurrence of thirty-eight common noncontextual words (NeW) such as and and the (see Larsen et al. for a list of the thirty-eight words))l In this paper we compute the frequency of occurrence of the following twenty common words, in alphabetical order: a, all, an, and, any, as, but, by, in, it, no, not, of, that, the, to, up, upon, with, without. Hilton calculated sixty-five noncontextual word-pattern ratios (WPR) (originally suggested by Morton).12 Examples of such ratios include the number of times a app