Expert crowdsourcing with flash teams

@article{Retelny2014ExpertCW,
  title={Expert crowdsourcing with flash teams},
  author={Daniela Retelny and S{\'e}bastien Robaszkiewicz and Alexandra To and Walter S. Lasecki and Jay Patel and Negar Rahmati and Tulsee Doshi and Melissa A. Valentine and Michael S. Bernstein},
  journal={Proceedings of the 27th annual ACM symposium on User interface software and technology},
  year={2014}
}
We introduce flash teams, a framework for dynamically assembling and managing paid experts from the crowd. Flash teams advance a vision of expert crowd work that accomplishes complex, interdependent goals such as engineering and design. These teams consist of sequences of linked modular tasks and handoffs that can be computationally managed. Interactive systems reason about and manipulate these teams' structures: for example, flash teams can be recombined to form larger organizations and… 

Figures and Tables from this paper

Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations
TLDR
A deployment is reported in which flash organizations successfully carried out open-ended and complex goals previously out of reach for crowdsourcing, including product design, software development, and game production.
Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision
TLDR
This work proposes a technique for achieving interdependent complex goals with crowds, and embodies it in Mechanical Novel, a system that crowdsources short fiction stories on Amazon Mechanical Turk.
Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool
TLDR
This paper investigates how requesters verbally communicate and collaborate with crowd workers to solve a complex task, and makes design recommendations that can inform the design of future real-time collaboration processes in crowdsourcing systems.
A Low-Code Framework for Complex Crowdsourcing Work Based on Process Modeling
TLDR
This paper provides a task model composition to allow task creation on demand, while collaborating on tasks in a tree structure to adapt to the dynamic execution, and constructs a framework named CrowdModeller to embody this approach.
Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability
TLDR
This research enables effective crowd teams with Huddler, a system for workers to assemble familiar teams even under unpredictable availability and strict time constraints, using a dynamic programming algorithm to optimize for highly familiar teammates when individual availability is unknown.
Exploring Coordination Models for Ad Hoc Programming Teams
TLDR
This paper explores the types and causes of coordination costs for transient software teams in existing collaborative programming tools: a version control system and a real-time shared editor and suggests design elements of shared programming environments that help teams effectively self-coordinate on their task.
Wish: Amplifying Creative Ability with Expert Crowds
TLDR
This work presents Wish, a system that uses expert members of an online crowd to amplify a user’s ability to carry out complex creative tasks, and illustrates how novice creators can use Wish to amplify their ability.
Apparition: Crowdsourced User Interfaces that Come to Life as You Sketch Them
TLDR
This paper introduces crowdsourcing techniques and tools for prototyping interactive systems in the time it takes to describe the idea, and introduces Powering Apparition, the first self-coordinated, real-time crowdsourcing infrastructure.
No Workflow Can Ever Be Enough: How Crowdsourcing Workflows Constrain Complex Work
TLDR
This paper uses an inductive mixed method approach to analyze behavior trace data, chat logs, survey responses and work artifacts to understand how workers enacted and adapted the crowdsourcing workflows, and indicates that complex work may remain a fundamental limitation of workflow-based crowdsourcing infrastructures.
Optimized group formation for solving collaborative tasks
TLDR
The notion of collaboration among crowd workers is formalized and a comprehensive optimization model for task assignment in a collaborative crowdsourcing environment is proposed, and a series of efficient exact and approximation algorithms with provable theoretical guarantees are proposed.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 47 REFERENCES
Collaboratively crowdsourcing workflows with turkomatic
TLDR
It is argued that Turkomatic's collaborative approach can be more successful than the conventional workflow design process and implications for the design of collaborative crowd planning systems are discussed.
CrowdWeaver: visually managing complex crowd work
TLDR
CrowdWeaver is presented, a system to visually manage complex crowd work that supports the creation and reuse of crowdsourcing and computational tasks into integrated task flows, manages the flow of data between tasks, and allows tracking and notification of task progress, with support for real-time modification.
Real-time crowd control of existing interfaces
TLDR
This paper presents mediation strategies for integrating the input of multiple crowd workers in real-time, evaluates these mediation strategies across several applications, and further validate Legion by exploring the space of novel applications that it enables.
The future of crowd work
TLDR
This paper outlines a framework that will enable crowd work that is complex, collaborative, and sustainable, and lays out research challenges in twelve major areas: workflow, task assignment, hierarchy, real-time response, synchronous collaboration, quality control, crowds guiding AIs, AIs guiding crowds, platforms, job design, reputation, and motivation.
Shepherding the crowd yields better work
TLDR
This paper investigates whether timely, task-specific feedback helps crowd workers learn, persevere, and produce better results in micro-task platforms by discussing interaction and infrastructure approaches for integrating real-time assessment into online work.
Soylent: a word processor with a crowd inside
TLDR
S soylent, a word processing interface that enables writers to call on Mechanical Turk workers to shorten, proofread, and otherwise edit parts of their documents on demand, and the Find-Fix-Verify crowd programming pattern, which splits tasks into a series of generation and review stages.
Decision-Theoretic Control of Crowd-Sourced Workflows
TLDR
A planner is described, TURKONTROL, that formulates workflow control as a decision-theoretic optimization problem, trading off the implicit quality of a solution artifact against the cost for workers to achieve it.
Human computation tasks with global constraints
TLDR
This paper focuses on the crowdsourcing of itinerary planning as a case study of constraint-based human computation tasks and introduces a collaborative planning system called Mobi that illustrates a novel crowdware paradigm.
Chorus: a crowd-powered conversational assistant
TLDR
Chorus, a crowd-powered conversational assistant, demonstrates a new future in which conversational assistants are made usable in the real world by combining human and machine intelligence, and may enable a useful new way of interacting with the crowds powering other systems.
CrowdForge: crowdsourcing complex work
TLDR
This work presents a general purpose framework for accomplishing complex and interdependent tasks using micro-task markets, a web-based prototype, and case studies on article writing, decision making, and science journalism that demonstrate the benefits and limitations of the approach.
...
1
2
3
4
5
...