STAIRS Redux: Thoughts on the STAIRS Evaluation, Ten Years after

@article{Blair1996STAIRSRT,
  title={STAIRS Redux: Thoughts on the STAIRS Evaluation, Ten Years after},
  author={David C. Blair},
  journal={J. Am. Soc. Inf. Sci.},
  year={1996},
  volume={47},
  pages={4-22}
}
  • D. Blair
  • Published 1996
  • Business
  • J. Am. Soc. Inf. Sci.
The test of retrieval effectiveness performed on IBM's STAIRS and reported in Communications of the ACM ten years ago, continues to be cited frequently in the information retrieval literature. The reasons for the study's continuing pertinence to today's research are discussed, and the political, legal, and commercial aspects of the study are presented. In addition, the method of calculating recall that was used in the STAIRS study is discussed in some detail, especially how it reduces the five… 

Figures and Tables from this paper

Extensions to the STAIRS Study—Empirical Evidence for the Hypothesised Ineffectiveness of Boolean Queries in Large Full-Text Databases

A new theoretical and empirical approach is introduced to study the problems of high recall searching in large databases and the results of a case experiment verify some of the hypothetical conclusions introduced in the STAIRS study, and expands the picture of falling performance.

Test Collection Based Evaluation of Information Retrieval Systems

This tutorial and review shows that despite its age, this long-standing evaluation method is still a highly valued tool for retrieval research.

Methodologies for Evaluation of Note-Based Music-Retrieval Systems

This paper explains how it collected artificial and expert music queries and name-based relevance judgments, and describes software it developed for collection of manual relevance judgments.

How reliable are the results of large-scale information retrieval experiments?

A detailed empirical investigation of the TREC results shows that the measured relative performance of systems appears to be reliable, but that recall is overestimated: it is likely that many relevant documents have not been found.

Subject Retrieval from Full-Text Databases in the Humanities

The author suggests ways in which full-text searching might be improved, whether by enhancement of database records, by introduction of enhanced search functionality, or by the education of searchers in more effective search techniques.

Relevance thresholds: a multi-stage predictive model of how users evaluate information

The Turn - Integration of Information Seeking and Retrieval in Context

The Turn represents a wide-ranging perspective of IS&R by providing a novel unique research framework, covering both individual and social aspects of information behavior, including the generation, searching, retrieval and use of information.

Increasing the Efficiency of High-Recall Information Retrieval

It is hypothesize that total assessment effort to achieve high recall can be reduced by using shorter document excerpts in place of full documents for the assessment of relevance and using a high-recall retrieval system based on continuous active learning (CAL).

Advanced Analysis Cognition: Improving the Cognition of Intelligence Analysis

Abstract : A prior effort, State of Knowledge Relative to Intelligence Analysis, was initially motivated by the fact that seemingly little change had resulted from numerous studies of the
...

References

SHOWING 1-10 OF 66 REFERENCES

Historical note: Information retrieval and the future of an illusion

A personal perspective on automatic indexing and information retrieval is offered, focusing not necessarily on the mainstream of research but on those events and ideas over a 34-year period that have led to the view stated above, and that have influenced my perception of important directions for future research.

The pragmatics of information retrieval experimentation

If there is one lesson to be learned from experience, it is that the theoretically optimum design can never be achieved, and the art of information retrieval experimentation is to make the compromises that will least detract from the usefulness of the results.

On selecting a measure of retrieval effectiveness

It is argued that a user's subjective evaluation of the personal utility of a retrieval system's output to him, if it could be properly quantified, would be a near-ideal measure of retrieval

Information Retrieval as a Trial-And-Error Process

This paper examines three important and well-known information retrieval experiments, with a focus on certain internal inconsistencies and on the high variability of search results.

All the Right Descriptors: A Test of the Strategy of Unlimited Aliasing

  • T. Brooks
  • Computer Science
    J. Am. Soc. Inf. Sci.
  • 1993
This study tested the Strategy of Unlimited Aliasing, an indexing method that places the terms suggested by several different indexers for the same information object in one index, and found no support for the strategy.

The Role of Attorney Mental Models of Law in Case Relevance Determinations: An Exploratory Analysis

This article examines the information seeking and evaluative behavior of attorneys as they search the corpus of law for primary authority in order to solve context sensitive legal issues. First, the

The State of Retrieval System Evaluation

Indeterminacy in the subject access to documents

  • D. Blair
  • Philosophy
    Inf. Process. Manag.
  • 1986

The relation between comprehending and remembering some complex sentences

The verification latencies indicated that at the time of comprehension, sentences likeIt is true that a fire isn’t cold were often recoded into an equivalent affirmative form, namely, it is trueThat a fire is hot, however, negative sentences like it isn”t true that an fire is cold were not recoded during comprehension.

Memory for Gist: Some Relevant Variables

A study of gist memory was carried out, indicating that, as expected, the gist effect was greater for contradictories than for contraries and for prefixed items than for unprefixed, phonetically distinct items ; the effect of these variables was interactive.
...