Training Workers for Improving Performance in Crowdsourcing Microtasks

@inproceedings{Gadiraju2015TrainingWF,
  title={Training Workers for Improving Performance in Crowdsourcing Microtasks},
  author={Ujwal Gadiraju and Besnik Fetahu and Ricardo Kawase},
  booktitle={EC-TEL},
  year={2015}
}
With the advent and growing use of crowdsourcing labor markets for a variety of applications, optimizing the quality of results produced is of prime importance. The quality of the results produced is typically a function of the performance of crowd workers. In this paper, we investigate the notion of treating crowd workers as ‘learners’ in a novel learning environment. This learning context is characterized by a short-lived learning phase and immediate application of learned concepts. We draw… 
It's getting crowded!: improving the effectiveness of microtask crowdsourcing
TLDR
A two-level categorization scheme for microtasks is proposed and revealed insights into the task affinity of workers, effort exerted to complete tasks of various types, and worker satisfaction with the monetary incentives, and a novel model for task clarity based on the goal and role clarity constructs are proposed.
Improving learning through achievement priming in crowdsourced information finding microtasks
TLDR
It is shown that it is possible to retain workers in long batches of tasks by triggering their inherent motivation to achieve and excel, and it is found that workers learn to perform more effectively, depicting relatively more stable accuracy and lower task completion times in comparison to workers who drop out early.
Using Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks
TLDR
The results show that requesters in crowdsourcing platforms can benefit by considering worker self-assessments in addition to their performance for pre- selection, and make a case for competence-based pre-selection in crowdsourced marketplaces.
1 Using Worker Self-Assessments for Competence-based Pre-Selection in Crowdsourcing
TLDR
The results show that requesters in crowdsourcing platforms can bene€t by considering worker self-assessments in addition to their performance for pre-selection, and propose a novel worker pre- selection method that considers accuracy of workerSelf-assessment.
Crowd Anatomy Beyond the Good and Bad: Behavioral Traces for Crowd Worker Modeling and Pre-selection
TLDR
A fine-grained data-driven worker typology based on different dimensions and derived from behavioral traces of workers is introduced and a powerful automated mechanism to detect worker types is proposed, reflecting on promoting fairness, trust and transparency in microtask crowdsourcing platforms.
Motivating Novice Crowd Workers through Goal Setting: An Investigation into the Effects on Complex Crowdsourcing Task Training
TLDR
It is found that setting different goals during training significantly affects workers’ learning perception, but overall does not have an effect on learning gain or post-training performance.
Evaluating Crowdworkers as a Proxy for Online Learners in Video-Based Learning Contexts
TLDR
Evaluating the extent to which crowdworkers and MOOC learners behave comparably on lecture viewing and quiz tasks serves to validate the assumption of previous research that crowdworkers are indeed reliable proxies of online learners and address the potential of employing crowdworkers as a means of online learning environment testing.
Analyzing crowd workers' learning behavior to obtain more reliable labels
TLDR
This thesis explores label reliability from two perspectives: first, how the label reliability of crowd workers develops over time during an actual labeling task, and second how it is affected by the difficulty of the documents to be labeled.
Crowdsourcing Task Assignment with Online Profile Learning
In this paper, we present a reference framework called Argo+ for worker-centric crowdsourcing where task assignment is characterized by feature-based representation of both tasks and workers and
Predicting Worker Disagreement for More Effective Crowd Labeling
TLDR
A crowd labeling framework is proposed: a disagreement predictor is trained on a small seed of documents, and this predictor is used to decide which documents of the complete corpus should be labeled and which should be checked for document-inherent ambiguities before assigning (and potentially wasting) worker effort on them.
...
...

References

SHOWING 1-10 OF 20 REFERENCES
Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution
TLDR
It is concluded that in a relevance categorization task, a uniform distribution of labels across training data labels produces optimal peaks in 1) individual worker precision and 2) majority voting aggregate result accuracy.
On Actively Teaching the Crowd to Classify
TLDR
This work proposes a natural Bayesian model of the workers, modeling them as a learning entity with an initial skill, competence, and dynamics, and shows how a teaching system can exploit this model to interactively teach the workers.
Iterative Learning for Reliable Crowdsourcing Systems
TLDR
This paper gives a new algorithm for deciding which tasks to assign to which workers and for inferring correct answers from the workers' answers, and shows that the algorithm significantly outperforms majority voting and is asymptotically optimal through comparison to an oracle that knows the reliability of every worker.
A taxonomy of microtasks on the web
TLDR
A two-level categorization scheme for tasks, based on an extensive study of 1000 workers on CrowdFlower, and insights into certain aspects of crowd behaviour; the task affinity of workers, effort exerted by workers to complete tasks of various types, and their satisfaction with the monetary incentives are presented.
An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets
TLDR
Results suggest that intrinsic motivation can indeed improve the quality of workers’ output, confirming the hypothesis and finding a synergistic interaction between intrinsic and extrinsic motivators that runs contrary to previous literature suggesting “crowding out” effects.
Increasing cheat robustness of crowdsourcing tasks
TLDR
This work takes a different approach by investigating means of a priori making crowdsourced tasks more resistant against cheaters by investigating methods of identifying workers primarily interested in producing quick generic answers.
Crowdsourcing Control: Moving Beyond Multiple Choice
TLDR
LAZYSUSAN, a decision-theoretic controller that dynamically requests responses as necessary in order to infer answers to crowdsourced tasks, is designed and implemented and an EM algorithm is designed to jointly learn the parameters of the model while inferring the correct answers to multiple tasks at a time.
Getting More for Less: Optimized Crowdsourcing with Dynamic Tasks and Goals
TLDR
This work presents models that predict the "survival probability" of a user at any given moment, that is, the probability that a user will proceed to the next task offered by the system, and dynamically assigning tasks significantly increases the value of a crowdsourcing system.
Quality management on Amazon Mechanical Turk
TLDR
This work presents algorithms that improve the existing state-of-the-art techniques, enabling the separation of bias and error, and illustrates how to incorporate cost-sensitive classification errors in the overall framework and how to seamlessly integrate unsupervised and supervised techniques for inferring the quality of the workers.
Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys
TLDR
The prevalent malicious activity on crowdsourcing platforms is analyzed and different types of workers in the crowd are defined, a method to measure malicious activity is proposed, and guidelines for the efficient design of crowdsourced surveys are presented.
...
...