• Corpus ID: 15852710

Not All HITs Are Created Equal: Controlling for Reasoning and Learning Processes in MTurk

@inproceedings{Hullman2011NotAH,
  title={Not All HITs Are Created Equal: Controlling for Reasoning and Learning Processes in MTurk},
  author={Jessica R. Hullman},
  year={2011}
}

• J. Hullman
• Published 2011
• Psychology

Challenges of crowdsourcing human-computer interaction (HCI) experiments on Amazon’s Mechanical Turk include risks posed by the combination of low monetary rewards and worker anonymity. These include how mirroring task structure across HIT or qualification questions may encourage the learning of shallow heuristics, the difficulty in increasing workers’ intrinsic motivation, and how verification questions can interrupt natural reasoning processes leading to a mismatch between experimental and real-world behavior. I discuss how researchers can increase HIT response quality on MTurk despite such challenges by applying findings from experimental psychology related to what increases active processing of information, how to avoid conflicts between opposing reasoning styles, and how to negotiate tradeoffs involved in inducing analytical thought. MOTIVATION The development of large-scale platforms for crowdsourcing simple human intelligence tasks such as Amazon’s Mechanical Turk (MTurk), CloudCrowd, and Crowdflower has fueled interest in using these platforms for controlled experiments on information interfaces. Crowdsourcing platforms have been used in studies aimed at how individuals process information online at the individual level [10, 14, 17] as well as the community-level [12], and how crowdsourced knowledge compares to that of experts [6, 15, 25]. The asynchronous, distributed and isolated nature of work on a crowdsourcing platform like MTurk differs significantly from the controlled laboratory setting or real world settings like the online social environments being investigated in social computing research. Crowdsourcing studies have led to a list of best practices for leveraging provided features to best insure response quality [10, 15, 25]. Best results occur when MTurk requesters use HITs with bona fide answers, designed such that accurate completion is no more effortful than maliciously providing an invalid answer and including explicitly verifiable questions as part of the task. Similarly, signaling to workers that answers will be scrutinized and detecting suspect answers in multiple ways helps eliminate low-quality responses. Worker ratings and qualification tasks are commonly used to filter users who do not understand task requirements, while learning how to do a HIT is facilitated by using demonstrative examples for each class of HIT. As an example, consider an MTurk experiment for studying how non-experts’ interpretations of information visualizations of economic data compare to expert interpretations. A sample HIT might call for 50 workers to do a sequence of 20 HITs, each with a unique visualization. Interested workers must pass a qualification task where they are first shown an example U.S. map of economic indicators for states along with a paragraph of context. Several questions designed to verify that the worker has closely examined the graph ask for the number of data variables shown and the color in which a particular indicator is represented. Answers to these are provided for the sake of learning. A final target question might require the worker to identify the trend in the visualization, suggesting an appropriate answer achieved using a fourstep process of identifying the regions involved, assessing the ranking of the regions according to two different economic indicators, assessing the relationship between the indicators for any region, and then summarizing in several sentences the results of this process. The rest of the qualification task requires the worker to independently complete the same questions for a second visualization. After submitting her responses, the worker is told whether she has passed, and if not how long she must wait before retaking the text. The 20 HITs that become available given a passing qualification mirror the qualification questions, with a unique visualization per HIT. Despite using best practices, this scenario illustrates several challenges in conducting research on MTurk. It behooves the worker to read the worked examples and carefully respond to questions, yet workers may be motivated by the low monetary rewards to use the qualification to learn the simplest heuristic for generating passing answers on HITs, rather than for gaining real comprehension of task requirements. Similarly, the use of the same style of verification question across similar HITs in a series, combined with the low rewards and motivation, may lead a worker to reuse answers across multiple similar HITs, or the quality of her answers might fall off sharply after the qualification if she suspects that the answers won’t be carefully examined. In either case, the researcher must sort Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI 2011, May 7–12, 2011, Vancouver, BC, Canada. Copyright 2011 ACM 978-1-4503-0267-8/11/05....$10.00. through the results. An alternative possibility is that a worker’s response to a verification question primes her subsequent processing of target information in a way that interferes with the desired results. Responses to a linear estimation task, for example, can be unintentionally skewed by information in a verification question [12]. Finally, the results of the study may not accurately reflect the real online visualization interpretations by non-experts as a result of the interruptive aspect of the verification questions to the user’s natural interpretative process. The aforementioned challenges concern the interaction of MTurk features with human reasoning processes. While some recent results [11, 21] suggest that attention and motivation problems on MTurk be no worse than using university subjects and suggest psychological findings on how to guard against satisficing among participants [19], there is room for further exploration of how psychological research could be applied to MTurk HIT design. I apply experimental results from cognitive and educational psychology around how to negotiate task design given the distinctions and tradeoffs between deliberate, analytic reasoning and intuitive, associative reasoning, and between active and passive information processing to designing MTurk experiments involving information interfaces. Specific techniques by which researchers can achieve common goals (e.g. more active processing of HITs, creating enjoyable HITs) are suggested. MODES OF REASONING IN DATA INTERPRETATION Psychologists have long distinguished between two reasoning modes—intuitive, System 1 reasoning and analytical System 2 reasoning [e.g. 8, 13]. A related distinction is often made between active and passive processing. Motivational and engagement factors form a third consideration relevant to the design of MTurk tasks involving information processing. Reasoning style reliably influences the outcome of a task like a decision. System 1 reasoning is typically automatic, effortless, and intuitive and associational in the sense of being based on prior experience. It is also often associated with more perceptual heuristic processes. Intuitive perceptual heuristics can be an asset, such as human’s natural abilities to detect visual patterns in some graphs that may be undetectable via statistics, or they can carry negative effects, as in matching bias leading one to see as relevant information that matches the lexical content in the statement about which one is reasoning, and to neglect the logically relevant information [8]. Conversely, System 2 processes tend to be effortful, deliberate, and analytic. These involve abstract reasoning and hypothetical thinking, and are thus appropriate for decisions facilitated by mental simulations of future possibilities and judgments not aided by prior knowledge or beliefs. These tasks often involve comparing options, such as rating two web resources according to a list of criteria, yet not all information rich tasks are best solved analytically – some complex choices such as choosing a car or apartment are often best made intuitively, in part to avoid the risks of over-thinking [16]. System 2 processing is constrained by working memory capacity, and correlated with general intelligence, and likely to be activated when people have both the capacity and motivation to engage in effortful processing. A potential conflict between the two is suggested by research documenting the inhibitory role of System 2 in suppressing default knowledge and belief-based responses [26]. Whether the conflict is resolved to lead to a more accurate judgment depends on task and individual characteristics. As an example of a task that could cue either type, an individual under time constraints might use the number of previous downloads of an album in an online music environment as a signal of its quality, while a less time-pressed user might systematically consult online reviews and ratings of available albums. Recognizing the automaticity of System 1 reasoning even in situations where analytic judgment is more accurate, researchers have investigated the factors that induce a reasoner to overcome the tendency to use System 1 processes. Errors in System 1 are less likely to be corrected when people are under cognitive load or respond quickly, but are more likely to be corrected when people are held accountable for decisions and when the outcome is personally relevant (see [3]). The challenge relevant to researchers using MTurk concerns the difficulty in predicting whether a subject faced with a “reasoning-ambiguous” HIT will successfully reconcile potential conflicts between the two types. Active and passive processing is a related distinction suggested by cognitive and educational psychologi