SQUARE: A Benchmark for Research on Computing Crowd Consensus

@inproceedings{Sheshadri2013SQUAREAB,
  title={SQUARE: A Benchmark for Research on Computing Crowd Consensus},
  author={Aashish Sheshadri and Matthew Lease},
  booktitle={HCOMP},
  year={2013}
}
While many statistical consensus methods now exist, relatively little comparative benchmarking and integration of techniques has made it increasingly difficult to determine the current state-of-the-art, to evaluate the relative benefit of new methods, to understand where specific problems merit greater attention, and to measure field progress over time. To make such comparative evaluation easier for everyone, we present SQUARE, an open source shared task framework including benchmark datasets… CONTINUE READING

Figures, Tables, and Topics from this paper.

Citations

Publications citing this paper.
SHOWING 1-10 OF 123 CITATIONS

Improving Crowdsourced Label Quality Using Noise Correction

  • IEEE Transactions on Neural Networks and Learning Systems
  • 2018
VIEW 4 EXCERPTS
CITES BACKGROUND
HIGHLY INFLUENCED

Consensus algorithms for biased labeling in crowdsourcing

VIEW 5 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

Crowd labeling latent Dirichlet allocation

  • Knowledge and Information Systems
  • 2017
VIEW 5 EXCERPTS
CITES METHODS
HIGHLY INFLUENCED

Probabilistic Multigraph Modeling for Improving the Quality of Crowdsourced Affective Data

  • IEEE Transactions on Affective Computing
  • 2017
VIEW 4 EXCERPTS
CITES RESULTS & BACKGROUND
HIGHLY INFLUENCED

CONSENSUS-BASED CROWDSOURCING: TECHNIQUES AND APPLICATIONS

VIEW 5 EXCERPTS
CITES BACKGROUND, METHODS & RESULTS
HIGHLY INFLUENCED

SQUARE: Benchmarking Crowd Consensus at MediaEval

VIEW 4 EXCERPTS
CITES METHODS & BACKGROUND

DLTA: A Framework for Dynamic Crowdsourcing Classification Tasks

  • IEEE Transactions on Knowledge and Data Engineering
  • 2019
VIEW 4 EXCERPTS
CITES METHODS & RESULTS
HIGHLY INFLUENCED

Predicting Foreground Object Ambiguity and Efficiently Crowdsourcing the Segmentation(s)

  • International Journal of Computer Vision
  • 2017
VIEW 4 EXCERPTS
CITES BACKGROUND
HIGHLY INFLUENCED

FILTER CITATIONS BY YEAR

2013
2019

CITATION STATISTICS

  • 15 Highly Influenced Citations

  • Averaged 19 Citations per year from 2017 through 2019

References

Publications referenced by this paper.
SHOWING 1-10 OF 33 REFERENCES

Managing crowdsourcing workers

J. Wang, P. Ipeirotis, F. Provost
  • Winter Conference on Business Intelligence.
  • 2011
VIEW 3 EXCERPTS
HIGHLY INFLUENTIAL

An evaluation of aggregation techniques in crowdsourcing

Q.V.H. Nguyen, T. T. Nguyen, N. T. Lam, K. Aberer
  • Proceedings of the The 14th International Conference on Web Information System Engineering (WISE 2013).
  • 2013
VIEW 2 EXCERPTS