• Corpus ID: 10161889

Incentives for Truthful Evaluations

  title={Incentives for Truthful Evaluations},
  author={Luca de Alfaro and Marco Faella and Vassilis Polychronopoulos and Michael Shavlovsky},
We consider crowdsourcing problems where the users are asked to provide evaluations for items; the user evaluations are then used directly, or aggregated into a consensus value. Lacking an incentive scheme, users have no motive in making effort in completing the evaluations, providing inaccurate answers instead. We propose incentive schemes that are truthful and cheap: truthful as the optimal user behavior consists in providing accurate evaluations, and cheap because the truthfulness is… 

Figures from this paper

Deep Bayesian Trust : A Dominant Strategy and Fair Reward Mechanism for Crowdsourcing

A mechanism that exploits transitivity so that a worker can be certified as trusted by other trusted workers who solve common tasks is proposed, which derived trust can be derived from a smaller number of gold tasks assignment through multiple layers of peer relationship among the workers.

Deep Bayesian Trust: A Dominant and Fair Incentive Mechanism for Crowd

This work proposes a novel mechanism that assigns gold tasks to only a few workers and exploits transitivity to derive accuracy of the rest of the workers from their peers' accuracy, and shows that the resulting mechanism ensures a dominant notion of incentive compatibility and fairness.

Coner: A Collaborative Approach for Long-Tail Named Entity Recognition in Scientific Publications

A collaborative approach which incrementally incorporates human feedback on the relevance of extracted entities into the training cycle of such iterative NER algorithms, called Coner, allows to still train new domain specific rare long-tail NER extractors with low costs, but with ever increasing performance while the algorithm is actively used in an application.

Data Pricing in Machine Learning Pipelines

This article focuses on pricing in three important steps in machine learning pipelines, starting with a brief review of data marketplaces and pricing desiderata and investigating pricing in the step of collaborative training of machine learning models.



Mechanisms for Making Crowds Truthful

The correlation between the true knowledge of an agent and her beliefs regarding the likelihoods of reports of other agents can be exploited to make honest reporting a Nash equilibrium.

Truth Serums for Massively Crowdsourced Evaluation Tasks

New reward mechanisms for crowdsourcing many types of evaluation tasks like labeling images, grading assignments in online courses, etc are proposed that impose minimal assumptions on the structure and knowledge of the underlying generating model, can account for heterogeneity in the agents' abilities, require no extraneous elicitation from them, and allow their beliefs to be (almost) arbitrary.

Crowdsourced judgement elicitation with endogenous proficiency

The main idea behind the mechanism is to use the presence of multiple tasks and ratings to estimate a reporting statistic to identify and penalize low-effort agreement, which rewards agents for agreeing with another 'reference' report on the same task, but also penalizes for blind agreement by subtracting out this statistic term.

Incentivizing Evaluation via Limited Access to Ground Truth: Peer-Prediction Makes Things Worse

It is shown that this problem is unavoidable whenever agents are able to coordinate using low-cost signals about the items being evaluated (e.g., text labels or pictures), and ways of circumventing this problem are considered by comparing agents' reports to ground truth.

Dwelling on the Negative: Incentivizing Effort in Peer Prediction

It is shown that negative payments can be used to make agents with quality lower than the quality threshold choose to not to participate, while those above continue to participate and invest effort, and when using the self-selection mechanism, perfect screening comes for free.

A Robust Bayesian Truth Serum for Small Populations

The robust BTS is the first peer prediction mechanism to provide strict incentive compatibility for every n ≥ 3 without relying on knowledge of the common prior, and is numerically robust and ex post individually rational.

Enforcing Truthful Strategies in Incentive Compatible Reputation Mechanisms

This paper analyzes the equilibria of two incentive-compatible reputation mechanisms and investigates how undesired equilibrium points can be eliminated by using trusted reports.

Putting Peer Prediction Under the Micro(economic)scope and Making Truth-Telling Focal

The main result of this paper is to show that, in the symmetric binary setting, by tweaking peer-prediction, in part by carefully selecting the proper scoring rule it is based on, it can make the truth-telling equilibrium focal--that is,truth-telling has higher expected payoff than any other equilibrium.

Incentive contrats and strictly proper scoring rules

Conditions under which a DM can use a strictly proper scoring rule as a contract to give an expert an incentive to gather an amount of information that is optimal from the DM’s perspective are described.

Elicitability and knowledge-free elicitation with peer prediction

This paper identifies the maximal set of information structures that are truthfully elicitable without verification, and provides a mechanism for such elicitation that requires that the designer know the information structure of the participants, which is unavailable in many settings.