Reliable information retrieval evaluation with incomplete and biased judgements

  title={Reliable information retrieval evaluation with incomplete and biased judgements},
  author={Stefan B{\"u}ttcher and Charles L. A. Clarke and Peter C. K. Yeung and Ian Soboroff},
Information retrieval evaluation based on the pooling method is inherently biased against systems that did not contribute to the pool of judged documents. This may distort the results obtained about the relative quality of the systems evaluated and thus lead to incorrect conclusions about the performance of a particular ranking technique. We examine the magnitude of this effect and explore how it can be countered by automatically building an unbiased set of judgements from the original, biased… CONTINUE READING
Highly Influential
This paper has highly influenced 11 other papers. REVIEW HIGHLY INFLUENTIAL CITATIONS
Highly Cited
This paper has 111 citations. REVIEW CITATIONS

From This Paper

Figures, tables, and topics from this paper.


Publications citing this paper.
Showing 1-10 of 70 extracted citations

Towards Robust & Reusable Evaluation for Novelty & Diversity

PIKM@CIKM • 2014
View 5 Excerpts
Highly Influenced

112 Citations

Citations per Year
Semantic Scholar estimates that this publication has 112 citations based on the available data.

See our FAQ for additional information.


Publications referenced by this paper.
Showing 1-6 of 6 references

A New Measure of Rank Correlation

M. G. Kendall
Biometrika, (30):81–89, • 1938
View 16 Excerpts
Highly Influenced

Evaluating Latent Semantic Vector Models with Synonym Tests and Document Retrieval

L. Grönqvist
ELECTRA Workshop: Methodologies and Evaluation of Lexical Cohesion Techniques in Real-World Applications Beyond Bag of Words, pages 86–88, Salvador, Brazil, August • 2005
View 7 Excerpts
Highly Influenced

The Philosophy of Information Retrieval Evaluation

CLEF • 2001
View 7 Excerpts
Highly Influenced

Similar Papers

Loading similar papers…