Corpus ID: 222141041

Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation

  title={Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation},
  author={Sebastian Hofst{\"a}tter and S. Althammer and M. Schroeder and Mete Sertkan and A. Hanbury},
  • Sebastian Hofstätter, S. Althammer, +2 authors A. Hanbury
  • Published 2020
  • Computer Science
  • ArXiv
  • The latency of neural ranking models at query time is largely dependent on the architecture and deliberate choices by their designers to trade-off effectiveness for higher efficiency. This focus on low query latency of a rising number of efficient ranking architectures make them feasible for production deployment. In machine learning an increasingly common approach to close the effectiveness gap of more efficient models is to apply knowledge distillation from a large teacher model to a smaller… CONTINUE READING
    4 Citations

    Figures and Tables from this paper

    Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
    • 2
    • Highly Influenced
    • PDF
    A White Box Analysis of ColBERT
    • 1
    • PDF


    Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System
    • Jiaxi Tang, Ke Wang
    • Computer Science, Mathematics
    • KDD
    • 2018
    • 42
    • PDF
    Understanding BERT Rankers Under Distillation
    • 4
    • PDF
    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
    • 606
    • Highly Influential
    • PDF
    Interpretable & Time-Budget-Constrained Contextualization for Re-Ranking
    • 18
    • PDF
    CEDR: Contextualized Embeddings for Document Ranking
    • 106
    • PDF
    ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
    • 37
    • Highly Influential
    • PDF
    Efficiency Implications of Term Weighting for Passage Retrieval
    • 4
    Patient Knowledge Distillation for BERT Model Compression
    • 142
    • PDF
    TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
    • 8
    • Highly Influential
    • PDF
    Training Curricula for Open Domain Answer Re-Ranking
    • 3
    • PDF