A taxonomy of microtasks on the web

@article{Gadiraju2014ATO,
  title={A taxonomy of microtasks on the web},
  author={Ujwal Gadiraju and Ricardo Kawase and Stefan Dietze},
  journal={Proceedings of the 25th ACM conference on Hypertext and social media},
  year={2014}
}
Nowadays, a substantial number of people are turning to crowdsourcing, in order to solve tasks that require human intervention. Despite a considerable amount of research done in the field of crowdsourcing, existing works fall short when it comes to classifying typically crowdsourced tasks. Understanding the dynamics of the tasks that are crowdsourced and the behaviour of workers, plays a vital role in efficient task-design. In this paper, we propose a two-level categorization scheme for tasks… 

Figures and Tables from this paper

Breaking Bad: Understanding Behavior of Crowd Workers in Categorization Microtasks

This paper delves into the behavior of workers that consume categorization tasks to determine measures that can make task design more effective.

Make Hay While the Crowd Shines: Towards Efficient Crowdsourcing on the Web

This PhD proposal investigates two pivotal aspects that influence the effectiveness of crowdsourcing: microtask design, and workers behavior, and proposes a two-level categorization scheme for tasks, which identifies the need for a taxonomy of typically crowdsourced tasks.

The Dimensions of Crowdsourcing Task Design

This paper proposes a systematic view of the crowdsourcing task design space and categorizes the dimensions that qualify the design decisions in crowdsourcing applications, offering guidelines for a principled understanding of current crowdsourcing marketplaces.

An experimental characterization of workers’ behavior and accuracy in crowdsourced tasks

It is shown that the time required by a worker to carry out a task correlates with its difficulty, and also with the quality of the outcome, and to interpret the workers’ behavior and reliability level in the absence of control techniques.

The Impact of Task Abandonment in Crowdsourcing

This paper conducts an investigation of the phenomenon of task abandonment, the act of workers previewing or beginning a task and deciding not to complete it, and shows how task abandonment may have strong implications on the use of collected data.

The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk

A data-driven approach to perform a long-term analysis of a popular micro-task crowdsourcing platform and understand the evolution of its main actors (workers, requesters, tasks, and platform) and proposes features used in a predictive model aiming at determining the expected performance of any batch at a specific point in time.

The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk

This paper uses the main findings of the five year log analysis to propose features used in a predictive model aiming at determining the expected performance of any batch at a specific point in time, and shows that the number of tasks left in a batch and how recent the batch is are two key features of the prediction.

Modeling and Exploration of Crowdsourcing Micro-Tasks Execution

This work aims to understand better how the crowd workers execute tasks, to try to describe this behavior mathematically and to identify ways to speed up the tasks execution.

All Those Wasted Hours: On Task Abandonment in Crowdsourcing

This paper conducts the first investigation into the phenomenon of task abandonment, the act of workers previewing or beginning a task and deciding not to complete it and shows how task abandonment may have strong implications on the use of collected data (for example, on the evaluation of IR systems).

It's getting crowded!: improving the effectiveness of microtask crowdsourcing

A two-level categorization scheme for microtasks is proposed and revealed insights into the task affinity of workers, effort exerted to complete tasks of various types, and worker satisfaction with the monetary incentives, and a novel model for task clarity based on the goal and role clarity constructs are proposed.
...

References

SHOWING 1-8 OF 8 REFERENCES

Managing the Crowd: Towards a Taxonomy of Crowdsourcing Processes

A new taxonomic framework for crowdsourcing processes is proposed following a method of IS taxonomy development and focuses exclusively on an organizational perspective and on the mechanisms available to crowdsourcing organizations.

A Survey of Crowdsourcing Systems

  • Man-Ching YuenIrwin KingK. Leung
  • Computer Science, Business
    2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing
  • 2011
A structured view of the research on crowd sourcing to date is provided, which is categorized according to their applications, algorithms, performances and datasets.

Who are the crowdworkers?: shifting demographics in mechanical turk

How the worker population has changed over time is described, shifting from a primarily moderate-income, U.S. based workforce towards an increasingly international group with a significant population of young, well-educated Indian workers.

Worker types and personality traits in crowdsourcing relevance labels

This paper uses behavioral observations (HIT completion time, fraction of useful labels, label accuracy) to define five worker types: Spammer, Sloppy, Incompetent, Competent, Diligent, and the `Big Five' personality dimensions.

Financial incentives and the "performance of crowds"

It is found that increased financial incentives increase the quantity, but not the quality, of work performed by participants, where the difference appears to be due to an "anchoring" effect.

Strategies for crowdsourcing social data analysis

These results demonstrate that paid crowd workers can reliably generate diverse, high-quality explanations that support the analysis of specific datasets.

Experiences surveying the crowd: reflections on methods, participation, and reliability

The results point to the value of participant engagement, identify potential changes in MTurk as a study venue, and highlight how communication among Turkers influences the data that researchers collect.

Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory

(PDF) Thematic Analysis in Qualitative Research | Anindita (PDF) Qualitative Research ProcessBasics of Qualitative Research | SAGE Publications IncQualitative Research Method Summary JMESTMarket