Why batch and user evaluations do not give the same results

  title={Why batch and user evaluations do not give the same results},
  author={A. Turpin and W. Hersh},
  booktitle={SIGIR '01},
  • A. Turpin, W. Hersh
  • Published in SIGIR '01 2001
  • Computer Science
  • Much system-oriented evaluation of information retrieval systems has used the Cranfield approach based upon queries run against test collections in a batch mode. Some researchers have questioned whether this approach can be applied to the real world, but little data exists for or against that assertion. We have studied this question in the context of the TREC Interactive Track. Previous results demonstrated that improved performance as measured by relevance-based metrics in batch studies did… CONTINUE READING
    174 Citations

    Tables and Topics from this paper.

    User interface effects in past batch versus user experiments
    • 7
    • PDF
    Metric and Relevance Mismatch in Retrieval Evaluation
    • 15
    User performance versus precision measures for simple search tasks
    • 345
    • PDF
    Including summaries in system evaluation
    • 57
    • PDF
    Comparing System Evaluation with User Experiments for Japanese Web Navigational Retrieval
    • 1
    • Highly Influenced
    • PDF
    On Obtaining Effort Based Judgements for Information Retrieval
    • 21
    • PDF
    Studies on Relevance, Ranking and Results Display
    • 1
    • PDF
    The good and the bad system: does the test collection predict users' effectiveness?
    • 42
    • Highly Influenced
    • PDF


    TREC-7 Interactive Track Report
    • 92
    • Highly Influential
    Further analysis of whether batch and user evaluations give the same results with a different user
    • task, in Proceedings of the Ninth Text Retrieval Conference (TREC-
    • 2001
    TREC-8 interactive track report, in Proceedings of the 8th Text REtrieval Conference (TREC-8)
    • 2000