On the history of evaluation in IR

@article{Robertson2008OnTH,
  title={On the history of evaluation in IR},
  author={Stephen E. Robertson},
  journal={Journal of Information Science},
  year={2008},
  volume={34},
  pages={439 - 456}
}
  • S. Robertson
  • Published 2008
  • Computer Science
  • Journal of Information Science
This paper is a personal take on the history of evaluation experiments in information retrieval. It describes some of the early experiments that were formative in our understanding, and goes on to discuss the current dominance of TREC (the Text REtrieval Conference) and to assess its impact. 
Topic difficulty and order of document presentation in relevance assessments
TLDR
A test collection is crucial to evaluate the relative effectiveness of an Information Retrieval (IR) system and the accuracy of assessors judging relevance, therefore, will have. Expand
Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation
These proceedings contain the invited talks and posters of the SIGIR 2009 Workshop on the Future of IR Evaluation, Boston, Massachusetts 23 July, 2009.
Information retrieval evaluation using test collections
TLDR
The popularity of test collections in IR evaluation has flourished in large part thanks to campaigns such as the Text Retrieval Conference (TREC), the CrossLanguage Evaluation Forum (CLEF), the NII Testbeds and Community for Information Access Research project (NTCIR). Expand
Evaluation for Multilingual Information Retrieval Systems
This chapter discusses IR system evaluation with particular reference to the multilingual context, and presents the most commonly used measures and models. The main focus is on system performanceExpand
Seven Years of Image Retrieval Evaluation
In this chapter we discuss evaluation of Information Retrieval (IR) systems and in particular ImageCLEF, a large–scale evaluation campaign that has produced several publicly–accessible resourcesExpand
Chapter 1 Seven Years of Image Retrieval Evaluation
In this chapter we discuss evaluation of Information Retrieval (IR) systems and in particular ImageCLEF, a large–scale evaluation campaign that has produced several publicly–accessible resourcesExpand
Toward Automated Component-Level Evaluation
Automated component-level evaluation of information retrieval is discussed. The advantages of such an approach are considered, as well as the requirements for implementing it. Acceptance of suchExpand
Test Collection Based Evaluation of Information Retrieval Systems
TLDR
This tutorial and review shows that despite its age, this long-standing evaluation method is still a highly valued tool for retrieval research. Expand
Information Retrieval Evaluation
  • D. Harman
  • Computer Science
  • Information Retrieval Evaluation
  • 2011
TLDR
This lecture starts with a discussion of the early evaluation of information retrieval systems, starting with the Cranfield testing in the early 1960s, continuing with the Lancaster "user" study for MEDLARS, and presenting the various test collection investigations by the SMART project and by groups in Britain. Expand
A generic approach to component-level evaluation in information retrieval
TLDR
The focus of the thesis at hand is on the key components that are needed to address typical ad-hoc search tasks, like finding books on a particular topic in a large set of library records in order to eliminate black box retrieval systems. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 62 REFERENCES
Some thoughts on the reported results of TREC
  • D. Blair
  • Computer Science
  • Inf. Process. Manag.
  • 2002
The periodic TRECs - Text REtrieval Conferences - have reported the results of a variety of recall studies in large-scale document retrieval. While the efforts of TREC are noteworthy and laudable,Expand
On the Evaluation of IR Systems
TLDR
Issues such as laboratory versus operational systems, black-box versus diagnostic experiments, and qualitative and quantitative methods are discussed and supported by examples drawn from three groups of evaluative experiments: weighted searching on a front-end system, information-seeking behaviour and the use of OPACs, and the OKAPI experimental retrieval system. Expand
Language Modeling for Information Retrieval
  • W. Croft, J. Lafferty
  • Computer Science
  • The Springer International Series on Information Retrieval
  • 2003
TLDR
This book contains the first collection of papers addressing recent developments in the design of information retrieval systems using language modeling techniques, showing that applications such as Web search, cross-lingual search, filtering, and summarization can be described in the same formal framework. Expand
INFORMATION RETRIEVAL THROUGH MAN‐MACHINE DIALOGUE
TLDR
Initial tests with a prototype program indicate that a performance equal to that obtainable from a more conventional on‐line retrieval system is possible without obliging the user to formulate his query. Expand
Relevance weighting of search terms
TLDR
This paper examines statistical techniques for exploiting relevance information to weight search terms using information about the distribution of index terms in documents in general and shows that specific weighted search methods are implied by a general probabilistic theory of retrieval. Expand
A probabilistic model of information retrieval: development and comparative experiments - Part 2
The paper combines a comprehensive account of a probabilistic model of retrieval with new systematic experiments on TREC Programme material. It presents the model from its foundations through itsExpand
Language Modelling in Information Retrieval
TLDR
Language model approach to the information retrieval proved nevertheless as a very well-performing method in practice and allows us to model some of the special features of language that are omitted, ignored or tedious to model in other previous models. Expand
Some Unexplained Aspects of the Cranfield Tests of Indexing Performance Factors
TLDR
Statistical reasoning is used to show that very likely there must have been several times as many relevant " document-question matches" as were actually found by the Cranfield searchers in the process of determining "all possible" relevance matches. Expand
Advances in Informational Retrieval: Recent Research from the Center for Intelligent Information Retrieval
TLDR
This work focuses on the development of Topic-Based Language Models for Distributed Retrieval and their applications in the context of distributed information retrieval systems. Expand
Overview of the Okapi projects
This paper gives a brief description of the Okapi projects and the work of the Centre for Interactive Systems research, as an introduction to this special issue of the Journal of Documentation. OkapiExpand
...
1
2
3
4
5
...