Noise Correction in Pairwise Document Preferences for Learning to Rank

  title={Noise Correction in Pairwise Document Preferences for Learning to Rank},
  author={H. Trivedi and Prasenjit Majumder},
This paper proposes a way of correcting noise in the training data for Learning to Rank. It is natural to assume that some level of noise might seep in during the process of producing query-document relevance labels by human evaluators. These relevance labels, which act as gold standard training data for Learning to Rank can adversely affect the efficiency of learning algorithm if they contain errors. Hence, an automated way of reducing noise can be of great advantage. The focus in this paper… Expand


Improving quality of training data for learning to rank using click-through data
This paper shows that the quality of training data labeled by humans has critical impact on the performance of learning to rank algorithms, and proposes a method to detect relevance judgment errors using click-through data accumulated at a search engine. Expand
Which noise affects algorithm robustness for learning to rank
This work investigates what inherent characteristics make training data robust to label noise and how to utilize them to guide labeling and identifies two inherent characteristics of the training data, namely relevance levels and label balance, that have great impact on the variation of pNoise with respect to label Noise. Expand
Selecting optimal training data for learning to rank
The concept of ''pairwise preference consistency'' (PPC) is proposed to describe the quality of a training data collection from the ranking point of view and an efficient solution to the maximization problem is proposed. Expand
Learning to rank for information retrieval
Three major approaches to learning to rank are introduced, i.e., the pointwise, pairwise, and listwise approaches, the relationship between the loss functions used in these approaches and the widely-used IR evaluation measures are analyzed, and the performance of these approaches on the LETOR benchmark datasets is evaluated. Expand
LETOR: A benchmark collection for research on learning to rank for information retrieval
The details of the LETOR collection are described and it is shown how it can be used in different kinds of researches, and several state-of-the-art learning to rank algorithms on LETOR are compared. Expand
Identifying Mislabeled Training Data
This paper uses a set of learning algorithms to create classifiers that serve as noise filters for the training data and suggests that for situations in which there is a paucity of data, consensus filters are preferred, whereas majority vote filters are preferable for situations with an abundance of data. Expand
A Short Introduction to Learning to Rank
  • Hang Li
  • Computer Science
  • IEICE Trans. Inf. Syst.
  • 2011
Several learning to rank methods using SVM techniques are described in details and the fundamental problems, existing approaches, and future work of learning toRank are explained. Expand
Variations in relevance judgments and the measurement of retrieval effectiveness
Very high correlations were found among the rankings of systems produced using diAerent relevance judgment sets, indicating that the comparative evaluation of retrieval performance is stable despite substantial diAerences in relevance judgments, and thus reaArm the use of the TREC collections as laboratory tools. Expand
Relevance assessment: are judges exchangeable and does it matter
It appears that test collections are not completely robust to changes of judge when these judges vary widely in task and topic expertise, and both system scores and system rankings are subject to consistent but small differences across the three assessment sets. Expand
Overview of the Fifth Text REtrieval Conference (TREC-5)
Presentation des objectifs, des methodes (la routing task, la tâche adhoc, la comparaison des differents systemes, les concepts de confusion, de filtrage, de multilinguisme, de traitement du langageExpand