• Corpus ID: 9334762

An Analytic Approach to People Evaluation in Crowdsourcing Systems

  title={An Analytic Approach to People Evaluation in Crowdsourcing Systems},
  author={Mohammad Allahbakhsh and Aleksandar Ignjatovi{\'c} and Boualem Benatallah and Seyed-Mehdi-Reza Beheshti and Norman Y. Foo and Elisa Bertino},
Worker selection is a significant and challenging issue in crowdsourcing systems. Such selection is usually based on an assessment of the reputation of the individual workers participating in such systems. However, assessing the credibility and adequacy of such calculated reputation is a real challenge. In this paper, we propose an analytic model which leverages the values of the tasks completed, the credibility of the evaluators of the results of the tasks and time of evaluation of the results… 

Figures and Tables from this paper

Information Disorder in the GLAM Sector: the Challenges of Crowd Sourced Contributions

This multiple case study is aimed at appraising information disorder through crowd-created contents in the knowledge and cultural heritage organisations such as Galleries, Libraries, Archives and Museums (GLAM).

MobiCS: Mobile Platform for Combining Crowdsourcing and Participatory Sensing

A conceptual architecture for versatile context-aware mobile crowd sourcing is proposed, and issues related to data representation, quality control, trust and reputation management, and task allocation are addressed.

Augmented Understanding and Automated Adaptation of Curation Rules

This dissertation proposes a feature-based and automated technique for curating the raw data, an autonomic approach for adapting data curation rules, and provides a solution to augment users in formulating their preferences while curating data in large scale information spaces.

Crowdsensing-driven route optimisation algorithms for smart urban mobility

This thesis proposes to leverage Mobile Crowdsensing (MCS) paradigm in which citizens use their mobile communication and/or sensing devices to collect, locally process and analyse, as well as voluntary distribute geo-referenced information which can be used to route and manage people flows in urban environments.

Determinants of a Successful Migration to Cloud Computing in Iranian Telecommunication Industry

This paper can be used as reliable model for any migration beforehand taking any action and provides a precious insight for policy and decision makers to change their mindset and grant a proper space for cloud computing to grow in this industry due to its advantages.

Data Curation APIs

This paper identifies and implements a set of curation APIs and makes them available to researchers and developers to assist them transforming their raw data into curated data.

Credibility assessment for Arabic micro-blogs using noisy labels



A Survey of Crowdsourcing Systems

  • Man-Ching YuenIrwin KingK. Leung
  • Computer Science, Business
    2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing
  • 2011
A structured view of the research on crowd sourcing to date is provided, which is categorized according to their applications, algorithms, performances and datasets.

CrowdForge: crowdsourcing complex work

This work presents a general purpose framework for accomplishing complex and interdependent tasks using micro-task markets, a web-based prototype, and case studies on article writing, decision making, and science journalism that demonstrate the benefits and limitations of the approach.

Is the Crowd's Wisdom Biased? A Quantitative Analysis of Three Online Communities

  • V. Kostakos
  • Economics
    2009 International Conference on Computational Science and Engineering
  • 2009
An expert evaluation of the voting mechanisms of each website and a quantitative data analysis of users’ aggregate voting behavior suggest that the websites with higher barrier to vote introduce a relatively high number of one-off voters, and they appear to attract mostly experts.

Quality management on Amazon Mechanical Turk

This work presents algorithms that improve the existing state-of-the-art techniques, enabling the separation of bias and error, and illustrates how to incorporate cost-sensitive classification errors in the overall framework and how to seamlessly integrate unsupervised and supervised techniques for inferring the quality of the workers.

An Analytic Approach to Reputation Ranking of Participants in Online Transactions

This paper proposes rationality assumptions that such inferences must obey, and proceeds to derive theorems implied by these assumptions, and a basic representation theorem is proved.

A content-driven reputation system for the wikipedia

The results show that the notion of reputation has good predictive value: changes performed by low-reputation authors have a significantly larger than average probability of having poor quality, as judged by human observers, and of being later undone, as measured by the algorithms.

The Dynamics of Seller Reputation: Evidence from Ebay

We construct a panel of eBay seller histories and examine the importance of eBay's reputation mechanism. We find that, when a seller first receives negative feedback, his weekly sales rate drops from

An Empirical Study of Collusion Behavior in the Maze P2P File-Sharing System

Analysis and measurement results of user collusion in Maze, a large-scale peer-to-peer file sharing system with a non-net-zero point-based incentive policy, find collusion patterns similar to those found in Web spamming.

Finding high-quality content in social media

This paper introduces a general classification framework for combining the evidence from different sources of information, that can be tuned automatically for a given social media type and quality definition, and shows that its system is able to separate high-quality items from the rest with an accuracy close to that of humans.

The Eigentrust algorithm for reputation management in P2P networks

An algorithm to decrease the number of downloads of inauthentic files in a peer-to-peer file-sharing network that assigns each peer a unique global trust value, based on the peer's history of uploads is described.