• DOI:10.48550/arXiv.2210.12579
• Corpus ID: 253098237

Efficient Nearest Neighbor Search for Cross-Encoder Models using Matrix Factorization

@article{Yadav2022EfficientNN,
  title={Efficient Nearest Neighbor Search for Cross-Encoder Models using Matrix Factorization},
  author={Nishant Yadav and Nicholas Monath and Rico Angell and Manzil Zaheer and Andrew McCallum},
  journal={ArXiv},
  year={2022},
  volume={abs/2210.12579}
}

• Nishant Yadav, Nicholas Monath, +2 authors A. McCallum
• Published 23 October 2022
• Computer Science
• ArXiv

Efficient k-nearest neighbor search is a fundamental task, foundational for many problems in NLP. When the similarity is measured by dot-product between dual-encoder vectors or `2-distance, there already exist many scalable and efficient search methods. But not so when similarity is measured by more accurate and expensive black-box neural similarity models, such as cross-encoders, which jointly encode the query and candidate neighbor. The cross-encoders’ high computational cost typically limits their use to reranking candidates retrieved by a cheaper model, such as dual encoder or TF-IDF. However, the accuracy of such a two-stage approach is upperbounded by the recall of the initial candidate set, and potentially requires additional training to align the auxiliary retrieval model with the cross-encoder model. In this paper, we present an approach that avoids the use of a dual-encoder for retrieval, relying solely on the cross-encoder. Retrieval is made efficient with CUR decomposition, a matrix decomposition approach that approximates all pairwise cross-encoder distances from a small subset of rows and columns of the distance matrix. Indexing items using our approach is computationally cheaper than training an auxiliary dual-encoder model through distillation. Empirically, for k > 10, our approach provides test-time recall-vs-computational cost tradeoffs superior to the current widely-used methods that re-rank items retrieved using a dualencoder or TF-IDF.