- Published 2012

In this paper we investigate how people construct absolute and relative likelihood judgments. We document systematic violations of two fundamental probability axioms that can occur when evidence for an event H and its complement not-H are stronger than evidence for another event L and its complement not-L. First, H is judged “more likely” than L, while L is assigned a higher probability than H, violating procedure invariance. Second, H is judged more likely than L, while not-H is also judged more likely than not-L, violating ordinal complementarity. We attribute these “belief reversals” to complement neglect—a tendency to underweight evidence for complementary events (i.e., not-H and not-L when H and L are focal)—in relative but not absolute likelihood judgment. To explain these belief reversals we advance the Contingent Weighting of Support (CWS) model that embeds support theory within the contingent weighting model, and derive conditions under which each type of belief reversal is expected to occur. We then fit this model to data in two experiments, and provide direct evidence of complement neglect in relative but not absolute likelihood judgment. We conclude with a discussion of additional forms of belief reversal and their implications concerning the construction of belief. Keywords: likelihood judgment, judged probability, support theory, contingent weighting, belief reversal Absolute vs. Relative Likelihood page 3 Absolute versus relative likelihood judgment 1. Introduction. Whether we infer a person’s belief from his or her choices, explicit probability estimates, or verbal statements (e.g., “It is fairly likely that...”), we rely on several implicit assumptions to derive consensual meaning from these data. Among the most basic assumptions is procedure invariance: the belief ordering of two events, A and B, should coincide under normatively equivalent elicitation modes. Thus, absolute and relative expressions of belief should be consistent with one another. For instance, when a weather forecaster says she thinks there is a 70% chance of rain tomorrow in Seattle and a 40% chance of rain tomorrow in Los Angeles we assume that she would also say that she thinks it is “more likely” to rain tomorrow in Seattle than Los Angeles. Second, the ordering of beliefs for a pair of events should be the reverse of the ordering of their complements, an assumption that we label ordinal complementarity. For instance, we expect the forecaster to say she thinks it is more likely to rain in Seattle than Los Angeles if and only if she would also say that it is more likely to remain dry in Los Angeles than Seattle. The assumptions of procedure invariance and ordinal complementarity seem to be unassailable, if not tautological. In fact, axiomatic treatments of subjective probability typically do not distinguish between absolute and relative likelihood judgment, and they take ordinal complementarity for granted (Krantz, Luce, Suppes & Tversky, 1971; Fishburn, 1986; Busemeyer, Pothos & Trueblood, 2011). However, as we will demonstrate, both assumptions For instance, in Fishburn’s (1986) review of the axioms of subjective probability he sets up a system of axioms defined using the ≻ operator so that A ≻ B is read as “event A is (regarded by the individual as) more probable than event B,” but he does not explicitly distinguish absolute from relative likelihood judgment. Fishburn explicitly advances ordinal complementarity as an axiom that he says “might qualify [as] ...so obvious and uncontroversial as to occasion no serious criticism” (p. 336). Likewise, Krantz, Luce, Suppes & Tversky (1971) Absolute vs. Relative Likelihood page 4 can fail in systematic and predictable ways. These violations provide important clues about how people weight evidence when making absolute and relative likelihood judgments. In this paper we document violations of both procedure invariance and ordinal complementarity, and develop a model of belief construction that accommodates them by embedding support theory (Tversky & Koehler, 1994; Rottenstreich & Tversky, 1997) within the contingent weighting model (Tversky, Sattath & Slovic, 1988). We then fit this model to data in two studies and measure its parameters. The psychology of likelihood judgment. Research on heuristics and biases (Tversky & Kahneman, 1974; Kahneman, Slovic & Tversky, 1982; Gilovich, Griffin & Kahneman, 2002; Shah & Oppenheimer, 2008) asserts that people judge the relative likelihood of events by evaluating a substitute attribute such as the relative ease with which instances of each event come to mind or the relative similarity of each event to a relevant prototype (Frederick & Kahneman, 2002; Kahneman, 2003). For instance, a person may judge the likelihood of rain to be greater in Seattle than in Los Angeles because recent instances of rain are easier to recall in Seattle than Los Angeles. Although studies of judgmental heuristics have been useful in predicting and explaining relative likelihood judgment, they do not explain how people quantify or qualify their absolute degree of belief. For instance, when deciding whether or not to carry an umbrella I must assess the absolute likelihood of rain. To model absolute likelihood judgment, support theory conceives of probability as a judgment of the proportion of evidence favoring a focal hypothesis (e.g., “rain in Seattle next Tuesday”) relative to its complement (“no rain in Seattle define the “≿” operator as “qualitatively at least as probable as” and defer “debates about the meaning of probability” as being “in reality, about acceptable empirical methods to determine ≿” (p. 200). They later articulate ordinal complementarity as Lemma 4 that follows from other axioms that are necessary for a representation of qualitative probability (see pp. 203, 212). Absolute vs. Relative Likelihood page 5 next Tuesday”). People may recruit evidence for focal and complementary hypotheses using judgmental heuristics, explicit arguments, or objective data. For instance, the probability of rain in Seattle might be estimated by comparing the ease with which one can imagine a day with rain to the ease with which one can imagine a day without rain. When the former is as easy to imagine as the latter, the probability is estimated to be 1⁄2 (i.e., odds of 1:1). In support theory evidence for the focal hypothesis and evidence for the complementary hypothesis receive equal weight in probability judgment. In contrast, research on heuristics such as availability and representativeness suggests that judgment of relative likelihood (e.g., “where is it more likely to rain?”) can be assessed by comparing evidence for each focal hypothesis (e.g., rain in Seattle versus rain in Los Angeles) without necessarily considering evidence for the corresponding complementary hypotheses (days without rain in Seattle/Los Angeles). The notion that complementary hypotheses receive less weight in relative than absolute likelihood judgment is consistent with several prior streams of research. According to the compatibility principle, the weight that a particular stimulus feature receives is enhanced by its compatibility with the response mode (Tversky, Sattath & Slovic, 1988; see also Goldstein & Einhorn, 1987; Mellers, Ordóñez, & Birnbaum, 1992). For instance, Tversky (1977) found evidence that common features receive more weight than distinctive features in similarity than in dissimilarity judgment (and vice versa); likewise Shafir (1993) asserted that positive attributes receive more weight than negative attributes when people choose than when they reject options (and vice versa). We suggest that assessment of absolute likelihood, particularly when made on a percentage or fractional probability scale, is compatible with proportion judgment. Thus, it may be more natural to consider the proportion of total evidence favoring the focal event when making such judgments, as in support theory. Assessment of relative Absolute vs. Relative Likelihood page 6 likelihood, however, is more compatible with a comparison of evidence strength for the events being contrasted (e.g., whether it is easier to recall days with rain recently in Seattle or Los Angeles) than a comparison of proportions of evidence favoring each target event relative to its corresponding complement. Complement neglect is also consistent with the observation that when the goal of a task is to contrast options (e.g., choose among gambles), decision makers tend to afford more weight to more prominent information (e.g., the likelihood of winning; Fischer & Hawkins, 1993; Fischer, Carmon, Ariely & Zauberman, 1999). In the present case a question about which of two events is more likely (e.g., “Is it more likely to rain in Seattle or Los Angeles?”) confers greater salience and therefore greater prominence to the focal hypotheses (rain in Seattle, rain in Los Angeles) than the corresponding complementary hypotheses (no rain in Seattle, no rain in Los Angeles) so that they receive greater weight in evaluation. Analogous tendencies to afford greater weight or attention to focal information relative to equally diagnostic alternative information have been documented in a range of literatures from comparative social judgments (Chambers & Windschitl, 2004) to judgments of subjective well-being (Wilson, Wheatley, Meyers, Gilbert, & Axsom 2000) to consumer choice (Hoch, 1985) to judgmental confidence (Koehler, 1991) to the judged probability of grouped hypotheses (Brenner & Rottenstreich, 1999). We refer to the tendency to underweight support for the complementary event as complement neglect. Note that support theory implies no complement neglect in probability judgment because it conceives of probability as the 2 In a related vein, Wedell and Moro (2008) speculate that the choice of which hypothesis is most likely or most numerous prompts qualitative thinking whereas estimation of absolute probability or frequency elicits more quantitative thinking. They report that judgment errors in which a conjunction of representative events (e.g., a randomly selected Scandinavian has blond hair and blue eyes) is deemed more likely than one of its constituents (e.g., a randomly selected Scandinavian has blond hair) are more common when participants choose which event is most probable or numerous than when they make an absolute judgment of probability or relative frequency. Absolute vs. Relative Likelihood page 7 unweighted ratio of support for the focal hypothesis relative to the total support for the focal and complementary hypotheses. Greater complement neglect in relative than absolute likelihood judgment can lead to violations of both procedure invariance and ordinal complementarity in situations where a person can summon more evidence for one pair of (focal, complementary) hypotheses than for another pair of (focal, complementary) hypotheses. Most commonly this can occur when one pair of hypotheses is more familiar than another pair. For instance, suppose an American economist is asked to assess the future unemployment rates in both the United States and Bangladesh. This economist may be able to conjure several compelling reasons why the U.S. unemployment rate could rise or hold steady in the coming year and an even greater number of compelling reasons why it could fall. That same economist, if pressed, might summon a couple of weak reasons why the unemployment rate in Bangladesh could rise or hold steady in the coming year and a couple of weak reasons why it could fall. The complement neglect hypothesis suggests that this economist will say that unemployment is “more likely to rise or hold steady” in the U.S. than in Bangladesh (because it is easier to come up with compelling reasons for this scenario in the U.S. than in Bangladesh), but also say that unemployment is “more likely to fall” in the U.S. than in Bangladesh (because it is easier to come up with compelling reasons for this scenario in the U.S. than in Bangladesh). In contrast, assuming this economist affords equal consideration to the focal and complementary hypotheses in absolute likelihood judgment, consistent with support theory, she will assign a higher probability to unemployment rising or holding steady in Bangladesh than the U.S (because she comes up with equally weak arguments for Bangladesh rising and falling but stronger arguments for U.S. falling than rising). Absolute vs. Relative Likelihood page 8 In sum, complement neglect in relative but not absolute likelihood judgment predicts a tendency to sometimes judge the more familiar event “more likely” to occur than the less familiar event while at the same time: (1) assigning the more familiar event a lower probability than the less familiar event (thus violating procedure invariance); and (2) judging the nonoccurrence of the more familiar event “more likely” than the non-occurrence of the less familiar event (thus violating ordinal complementarity). We refer to violations of procedure invariance and ordinal complementarity as “belief reversals” and to the specific tendency to deem more familiar events more likely as “familiarity bias” (see Fox & Levav, 2000). The remainder of this paper is organized as follows. We begin with a review of prior evidence and present new evidence of familiarity bias and belief reversal. Next, we develop a theoretical model that generalizes support theory (Tversky & Koehler, 1994; Rottenstreich & Tversky, 1997) by embedding it within the contingent weighting model (Tversky, Sattath & Slovic, 1988), and show how and when this “Contingent Weighting of Support” model (CWS) predicts each kind of belief reversal. Third, we explicitly test the fit of the CWS model to data and measure its parameters, thus directly testing the complement neglect hypothesis. We conclude with a discussion of extensions and implications of the model. 2. Evidence for Belief Reversals. A. Violations of ordinal complementarity. Belief reversals are most likely to occur when people are presented with one pair of complementary events that are relatively high in support and another pair of complementary events that are relatively low in support. To illustrate, we created fictional profiles of mayoral election candidates in the cities of Fremont and Vacaville. One pair of candidates was designed to appear strongly qualified for Mayor of their city; the other pair of candidates was designed to appear weakly qualified for Mayor of a different city (see descriptions below). We Absolute vs. Relative Likelihood page 9 asked one group of participants to assess relative likelihood by indicating which of two candidates was “more likely” to win his respective race and a second group to assess absolute likelihood by indicating the probabilities that each of two candidates would win their respective races. We begin by describing the relative likelihood task and will describe results of the absolute likelihood task in Section B below. Participants. We recruited a total of 240 participants online using Amazon’s mechanical Turk and asked them to answer a short survey in exchange for a small payment. Participants were assigned at random to one of four experimental conditions. Method. Participants were: (1) presented with descriptions of the candidates, (2) asked either a relative likelihood question or absolute likelihood questions about two candidates, and then (3) asked to rate the relative strength of all four candidates. They were first presented with the following instructions: We are interested in your opinions about the outcomes of Mayoral elections in Fremont and Vacaville, two medium sized cities in California. The candidates you will be asked about are fictional, but representative of the type of candidates that run for office at these locations. Assume that the candidates listed are the only ones running for Mayor of their respective cities. We have listed some of their major characteristics below. Please make your judgment based on this information. Aaron and Ben are running for Mayor of Fremont. Aaron Outstanding public speaker 8 years on City Council Studied public policy in college Endorsed by the City Worker’s Union Founded nonprofit supporting underprivileged kids Ben Very well-financed Former Assistant City Manager Master’s degree in economics Endorsed by School Board Active in several local charities Chris and David are running for mayor of Vacaville. Chris Tall and athletic No former public service experience Studied anthropology in college Endorsed by local Boy Scout troupe Recently began a fundraising drive for a local baseball team David Was a debater in high school Former PTA member Master’s degree in European History Endorsed by local chapter of Kiwanis Club Volunteered last Thanksgiving at a local soup kitchen Absolute vs. Relative Likelihood page 10 We fully counterbalanced the order in which the pairs of profiles and profiles-within-pairs were presented (though names were always presented in alphabetical order). In the order presented above Aaron and Ben were designed to be seen as strong candidates whereas Chris and David were designed to be seen as weak candidates. We next asked one group of participants (n = 64) to indicate whether they thought that, “Aaron is more likely to win his mayoral race in Fremont” or “Chris is more likely to win his mayoral race in Vacaville.” Another group of participants (n = 54) made similar judgments for Ben and David, respectively. To verify our a priori assumption that Aaron and Ben would be perceived as stronger candidates than Chris and David, respectively, we asked all participants on the next screen to rate the relative strength of candidates as follows: Please rate the four Mayoral candidates you just saw according to the strength of their candidacy. Assign the strongest candidate 100 points, and then assign anything from 1 to 99 to the other candidates depending on their relative strength. For instance, if you think that Candidate X is strongest, you would assign him a score of 100; if you think that Candidate Y is half as strong as candidate X, you would assign him a score of 50. Results. Strength ratings confirmed our expectation that Aaron and Ben would be seen as stronger candidates than Chris and David, respectively. The average strength rating for the candidate labeled Aaron above was 77.0 and the average rating for the candidate labeled Ben above was 74.6. In contrast, the average ratings for Chris and David were 38.8 and 59.7, respectively. Importantly, Aaron was perceived to be a much stronger candidate than Chris (t(239) = 19.20, p < .0001 by paired t-test) and Ben was perceived to be a much stronger candidate than David (t(239) = 6.43, p < .0001). Responses to the relative likelihood questions strongly violated ordinal complementarity (see Table 1). Of the participants presented the profiles labeled Aaron and Absolute vs. Relative Likelihood page 11 Chris above, 83% indicated that it was more likely that Aaron would win his race in Fremont than Chris would win his race in Vacaville. Thus, in order to satisfy ordinal complementarity, roughly 17% of participants in the Ben-David condition should have reported that Ben was more likely to win the Fremont mayoral race than David was to win the Vacaville mayoral race (that is, Aaron was more likely to lose than Chris). However, when a second group of participants was asked whether Ben or David was more likely to win his race, 61% indicated Ben. Put another way, the proportion of participants indicating that Aaron is more likely to win his election than Chris plus the proportion of participants indicating that Aaron is more likely lose his election than Chris was 83% + 61% = 144% >> 100%, a strong violation of ordinal complementarity (z = 5.41, p < .0001). In a previous paper (Fox & Levav, 2000), we provided several additional demonstrations of ordinal complementarity violations that are summarized in Table 2A. Most of these demonstrations entailed assessments of natural events for which arguments for and/or against each event were not made explicit as they are in the foregoing election vignette. For instance, consider the following item that was administered to Duke University students (Study 1). Duke students devote a great deal of time following men’s basketball, but most know little or nothing about the fencing team. We reasoned that our participants would find both the target question and complementary question about Duke basketball to be highly familiar (labeled H and H , respectively) and that they would find the target and complementary question about Duke fencing less familiar (labeled L and L , respectively). We documented a strong familiarity bias. Seventy-five percent of participants in a first group said they thought it was “more likely” that Duke would beat the University of North Carolina (UNC) in their next men’s basketball game than in their next men’s fencing Absolute vs. Relative Likelihood page 12 tournament. Thus, if ordinal complementarity holds, one would expect about 25% of participants in a second group to indicate that it was “more likely” that UNC would beat Duke in their next men’s basketball game than fencing tournament. Instead, a significantly higher proportion, 44%, indicated this belief. Put another way, the proportion of participants indicating that H is more likely than L plus the proportion of participants indicating that H is more likely than L was 75% + 44% = 119% > 100%, indicating a pronounced familiarity bias that violates ordinal complementarity. We replicated this pattern in all five studies reported in Table 2A. As can be seen in Column 7, participants exhibited a significant bias to order high familiarity (focal and alternative) hypotheses above corresponding low familiarity hypotheses. The question arises, what pattern might one predict for judgments of which event is “less likely”? One could argue that this response mode would direct attention to reasons why relevant target events would not occur (so that complementary events receive relatively more weight), thereby showing a bias to rate more familiar events and their complements less likely. However, the complement neglect hypothesis suggests instead that because “less likely” judgments entail explicit evaluation of relative likelihood of two focal events, complementary events will be underweighted just as when rating which event is “more likely.” Thus, complement neglect predicts a bias to rate less familiar events and their complements less likely. Indeed, our previous data (Fox and Levav, 2000) reveal such a pattern (Studies 4 and 5). All previous demonstrations of ordinal complementarity violations used indirect tests of this axiom in which different groups of participants compared either a pair of focal events or a pair of complementary events. To investigate the robustness of familiarity bias in a more direct test, we attempted to replicate this effect using a within-subject design. Absolute vs. Relative Likelihood page 13 Participants. We recruited participants (N = 35) from bulletin boards of online groups dedicated to each of 29 National Basketball Association (NBA) teams, as well as two groups devoted to general professional basketball and fantasy league discussion. We offered participants a chance to win a $50 cash prize in exchange for completing the survey. Procedure. We asked participants to rate eight events relating to the upcoming NBA collegiate player draft using a scale from 1 “most likely” to 8 “least likely,” using each ranking only once. The questions concerned both familiar and unfamiliar players that had appeared on a list of NBA prospects culled and rated by a leading draft expert and national television commentator. The relative draft position of four players served as our target events: Elton Brand and Steve Francis (familiar; these players were the top two picks in the NBA draft that year) and Keith Carter and Roberto Bergerson (unfamiliar; Carter went undrafted and Bergerson was the 52 pick). The order of the events was randomly determined and appeared as follows (target events were not highlighted in the original survey): ___ Elton Brand is chosen ahead of Steve Francis in the draft. (H ) ___ Scott Padgett is a second round pick in the draft. ___ Keith Carter is chosen ahead of Roberto Bergerson in the draft. ( L ) ___ Ademola Okulaja is selected in the first round of the draft. ___ Tim Young is selected ahead of Heshimu Evans in the draft. ___ Steve Francis is chosen ahead of Elton Brand in the draft. (H ) ___ Roberto Bergerson is chosen ahead of Keith Carter in the draft. ( L ) ___ Scott Padgett is a lottery pick (picks 1-‐13) in the draft. We randomly assigned respondents either to the above questionnaire or to a version that presented the events in the opposite order. This manipulation did not affect our results so we combined the data in the analysis reported below. Results. We coded participants’ responses into three categories: familiarity bias (either listing both of the high familiarity events over both of the low familiarity events, i.e., H > H > L > L, or alternating these rankings H > L > H > L), a reverse “unfamiliarity” bias (either listing Absolute vs. Relative Likelihood page 14 both the low familiarity over both of the high familiarity events, e.g., L > L > H > H, or alternating these rankings L > H > L > H), and a normatively defensible ranking (either listing H > L > L > H or L > H > H > L). 3 Of the 35 participants, we found that 10 conformed to the familiarity bias pattern, only 1 conformed to the unfamiliarity bias pattern, and the remaining 24 indicated normatively defensible rankings. Although the familiarity bias here appears small, note that our study is quite conservative because of its transparency—it takes little effort for participants to see that the focal and alternative hypotheses appear in the same short list, creating a strong demand for consistency. If participants had given random responses, the expected proportion indicating each of these three categories would be equal, a null hypothesis which clearly can be rejected (χ(2) = 13.4, p = .001). Moreover, participants were ten times more likely to indicate a familiarity bias than an unfamiliarity bias (p = .01, by sign test). B. Violations of procedure invariance. To illustrate violations of procedure invariance, we included in the mayoral election study described earlier two additional conditions in which participants were asked to judge probabilities that each candidate would prevail in his respective race. In particular, we asked participants in the absolute likelihood conditions to indicate [alternate wording in brackets] the “Probability that [Aaron/Ben] wins his mayoral race in Fremont” and the “Probability that [Chris/David] wins his mayoral race in Vacaville” on a scale from 0 to 100 percent. One group (n = 54) judged probabilities for the candidates labeled Aaron and Chris above; another group (n = 68) judged probabilities for the candidates labeled Ben and David above. The present hypothesis that complement neglect is less pronounced for absolute than relative likelihood judgment predicts that the proportion of participants ordering Aaron above Chris and Ben 3 Because we present complementary events and the designation of H versus H and L versus L is arbitrary, we do not distinguish between events and their complements in the analysis of these results (e.g., we treat H > H > L > L the same as H > H > L > L). Absolute vs. Relative Likelihood page 15 above David, respectively, will be lower for judged probabilities than they were for relative likelihood judgments described in the previous section. The data confirm these predictions (see Table 1), revealing strong violations of procedure invariance. Whereas 83% of participants in the relative likelihood condition (reported earlier) indicated that Aaron was more likely to win his mayoral race than Chris, only 65% of participants indicated a higher probability for Aaron than for Chris (p = .01 by Fisher exact test). Likewise, whereas 61% of participants in the relative likelihood condition indicated that Ben was more likely to win his mayoral race than David, only 37% of participants indicated a higher probability for Ben than David (p <.001 by Fisher exact test). Recall from the results of the relative likelihood conditions reported earlier that the proportion of participants rating Aaron more likely than Chris and Ben more likely than David sum to 144%. In contrast, the proportion of participants in the absolute likelihood conditions giving higher probabilities to Aaron and Ben winning their race summed to 102% (= 65% + 37%), a non-significant difference from unity (z = 0.23, n.s.). More important, the overall rate of violations of ordinal complementarity was significantly smaller in the absolute than relative likelihood condition reported earlier (z = 3.85, p < 0.001). Note that this test statistic equivalently represents the overall significance of the violations of procedure invariance across conditions of the study. Table 2B presents additional examples of violations of procedure invariance reported in Fox and Levav (2000). For these items we asked groups of respondents to judge the probability of the highly familiar hypothesis (and its complement) and the less familiar 4 In each of the probability conditions (Aaron/Chris, Ben/David), there were eight ties that we allocated equally among the two belief orderings. Absolute vs. Relative Likelihood page 16 hypothesis (and its complement). For example, in Study 7, we asked a group of U.S.-based participants whether they thought that it was “more likely” that “the winner of the next U.S. Presidential election [is/is not] a member of the Democratic Party” or “the winner of the next British Prime Ministerial election [is/is not] a member of the Labor party” [alternate wording in brackets]. Other groups were asked to judge the probabilities of the same events. Thus, roughly half the participants ordered their beliefs over these events directly and for the remaining participants we inferred these orderings from their judged probabilities. This design provided tests of both varieties of belief reversal (violations of procedure invariance and of ordinal complementarity). First, although 64% of participants in the relative likelihood condition indicated that a Democratic winner was “more likely” than a Labor winner, only 36% of participants in the absolute likelihood condition reported a higher probability for a Democratic winner than a Labor winner. Likewise, although 76% of participants in the relative likelihood condition indicated that a non-Democratic winner was “more likely” than a nonLabor winner, 73% of participants provided a higher probability for Democrat than Labor. Taken together, there was a significant violation of ordinal complementarity evident in the relative likelihood conditions, but not in the absolute likelihood conditions: The proportion of participants rating Democrat more likely than Labor plus the proportion of participants rating non-Democrat more likely than non-Labor was 140% (p < .005), whereas the proportion of participants providing a higher probability for Democrat than Labor plus the proportion providing a higher probability for non-Democrat than non-Labor summed to 109% (n.s.), a 5 In Study 6 we only elicited relative likelihood and probability judgments for a focal hypothesis and not the complementary hypothesis. This allowed a test of violation of procedure invariance, but not ordinal complementarity. 6 Participants judged the probabilities of two target events consecutively on the same page, in an order that was counterbalanced. 7 As in the election study example above, we broke ties in the ordinal analysis of judged probabilities by assigning half to H > L and half to L > H, and so forth. For a fuller description of the frequency of ties see Table 1 of Fox & Levav (2000). Absolute vs. Relative Likelihood page 17 significant interaction (p < .05). In each of the studies listed, there is a significantly stronger tendency to rate the more familiar hypothesis “more likely” than assign it a higher probability. We next attempt to replicate our direct (within-subject) demonstration of familiarity bias in relative likelihood judgment and determine whether it will be significantly attenuated when ordinal complementarity is tested directly in absolute likelihood judgment by asking participants to judge the probability of each event (also within-subject). Participants. One hundred and seventy-one MBA students at Duke University participated in this study, which was embedded in a larger survey packet. A $10 charitable contribution was made in the name of each respondent in exchange for his or her participation. Procedure. Participants responded to one of two versions of a questionnaire item. In one condition (n = 73) participants were asked to rank a set of eight events from “most likely” to “least likely.” In a second condition they (n = 97) were asked to provide their best estimate of the probability for each of the same eight events. Events concerned the relative positions of pairs of graduate programs in the U.S. News and World Report ranking in the next published survey. Two target events and their complements (i.e., four events), one familiar and one unfamiliar, were combined with four filler events. The events are listed below (target events were not highlighted in the original survey): U. of Pennsylvania is ranked above Yale among medical schools. Syracuse is ranked above Rutgers among library sciences programs. ( L ) Wharton (U. Penn.) is ranked above Harvard among business schools. (H ) MIT is ranked above Michigan among physics departments. Columbia is ranked above U. of Pennsylvania among medical schools. Rutgers is ranked above Syracuse among library sciences programs. ( L ) UCLA is ranked above Cornell among physics departments. Harvard is ranked above Wharton (U. Penn) among business schools. (H ) Absolute vs. Relative Likelihood page 18 We assumed that because our participants were incoming MBA students, the items regarding business schools would be quite familiar to them, while the items regarding library sciences would be wholly unfamiliar. Indeed, in a contemporaneous survey of students admitted to Duke’s MBA program, 99% indicated that they had used Business Week and/or U.S. News & World Report’s published rankings of business schools in deciding which program to attend. The order of event presentation was determined at random and was counterbalanced. Results. As in the NBA draft study, we coded our participants’ responses into three categories in the ranking and probability conditions: familiarity bias, “unfamiliarity” bias, or a normative ranking. Once again, had participants been responding at random, the expected proportion indicating each of these three categories would be equal. In contrast, of the 73 participants in the ranking condition, we found that 35 conformed to the familiarity bias pattern, only 1 conformed to the unfamiliarity bias pattern, and the remaining 37 indicated normatively defensible rankings (χ(2) = 35.1, p < .0001). The corresponding numbers for the probability condition were 25, 2, and 70, respectively (χ(2) = 72.2, p < .0001). We speculate that the residual familiarity bias in the probability condition was a result of some participants anchoring their probabilities on previous judgments then adjusting according to whether subsequent items appeared more or less likely. More importantly, the results indicate that the familiarity bias was significantly attenuated in the probability condition relative to the ranking condition (25% versus 48%, respectively; χ(1) = 8.97, p < .005), a highly significant

@inproceedings{Fox2012AbsoluteVR,
title={Absolute versus relative likelihood judgment},
author={Craig R. Fox and Jonathan Levav and Manel Baucells and Jerome R. Busemeyer and Florian Ederer},
year={2012}
}