Reciprocal rank fusion outperforms condorcet and individual rank learning methods

@article{Cormack2009ReciprocalRF,
  title={Reciprocal rank fusion outperforms condorcet and individual rank learning methods},
  author={Gordon V. Cormack and Charles L. A. Clarke and Stefan B{\"u}ttcher},
  journal={Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval},
  year={2009}
}
  • G. Cormack, C. Clarke, Stefan Büttcher
  • Published 19 July 2009
  • Computer Science
  • Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Reciprocal Rank Fusion (RRF), a simple method for combining the document rankings from multiple IR systems, consistently yields better results than any individual system, and better results than the standard method Condorcet Fuse. This result is demonstrated by using RRF to combine the results of several TREC experiments, and to build a meta-learner that ranks the LETOR 3 dataset better than any previously reported method 

Tables from this paper

Assessing the Benefits of Model Ensembles in Neural Re-Ranking for Passage Retrieval
TLDR
Tests with the MS-MARCO dataset show that model ensembling can indeed benefit the ranking quality, particularly with supervised learning-to-rank although also with unsupervised rank aggregation.
Rank-Mixer and Rank-Booster: Improving the Effectiveness of Retrieval Methods
TLDR
Experimental evaluation over 3D object datasets shows that the proposed techniques outperforms the multimetric approach and previously existing rank fusion methods.
Inverse square rank fusion for multimodal search
TLDR
This paper presents the Inverse Square Rank fusion method family, a set of novel fully unsupervised rank fusion methods based on quadratic decay and on logarithmic document frequency normalization that have comparable or better performance than existing state-of-the-art approaches.
Ranking model selection and fusion for effective microblog search
TLDR
This work finds that the selection-based ensemble approach can significantly outperform using the single best ranker, and it also has clear advantage over the rank fusion that combines the results of all the available models.
Low-Complexity Supervised Rank Fusion Models
TLDR
Evaluation results show that it is indeed possible to achieve a retrieval performance comparable to LETOR methods, using only 3-5% of the rank lists of the number of rank lists used by LETor methods.
Risk-Reward Trade-offs in Rank Fusion
TLDR
This work comprehensively evaluates several state-of-the-art fusion algorithms in the context of risk, and shows that the risk-reward payoff in keyword queries can in fact be significantly improved when careful combinations of system and query variations are fused into a single run.
A Soft Computing Approach for Learning to Aggregate Rankings
TLDR
This paper presents an approach to combine rank aggregation techniques using a soft computing technique -- Genetic Programming -- in order to improve the results in Information Retrieval tasks and shows that the proposed approach reaches top performance in most datasets.
Integration of short term learning methods for image retrieval by reciprocal rank fusion
TLDR
The combination of two ranked results from the independent Short term learning methods with Reciprocal Rank Fusion to improve the accuracy of the system and reveal the superior of suggested method to existing Short termlearning methods in terms of precision.
NovaSearch at TREC 2013 Federated Web Search Track: Experiments with rank fusion
TLDR
Inverse Square Rank is a novel fully unsupervised rank fusion algorithm based on quadratic decay and on logarithmic document frequency normalization that was achieved in the competition were very positive and the results were able to improve them further post-TREC.
Improving Recall In Text Retrieval Using Rank Fusion
TLDR
This work shows that combining rank-fusion, query variations, and system variations into the same fusion pool can dramatically increase the end-to-end effectiveness of recall-sensitive evaluation metrics and also provide the least risk out of all other fusion combinations studied.
...
...

References

SHOWING 1-7 OF 7 REFERENCES
Condorcet fusion for improved retrieval
TLDR
A graph-theoretic analysis is applied to one of the two major classes of voting procedures from Social Choice Theory, the Condorcet procedure, and yields a sorting-based algorithm that performs very well on TREC data, often outperforming existing metasearch algorithms whether or not relevance scores and training data is available.
AdaRank: a boosting algorithm for information retrieval
TLDR
The proposed novel learning algorithm, referred to as AdaRank, repeatedly constructs 'weak rankers' on the basis of reweighted training data and finally linearly combines the weak rankers for making ranking predictions, which proves that the training process of AdaRank is exactly that of enhancing the performance measure used.
Learning to rank: from pairwise approach to listwise approach
TLDR
It is proposed that learning to rank should adopt the listwise approach in which lists of objects are used as 'instances' in learning, and introduces two probability models, respectively referred to as permutation probability and top k probability, to define a listwise loss function for learning.
An Efficient Boosting Algorithm for Combining Preferences
TLDR
This work describes and analyze an efficient algorithm called RankBoost for combining preferences based on the boosting approach to machine learning, and gives theoretical results describing the algorithm's behavior both on the training data, and on new test data not seen during training.
Optimizing search engines using clickthrough data
TLDR
The goal of this paper is to develop a method that utilizes clickthrough data for training, namely the query-log of the search engine in connection with the log of links the users clicked on in the presented ranking.
TREC - Experiment and Evaluation in IR
  • 2005