Modern Index Design for Efficient Learned Sparse Retrieval
Date:
Large-scale search engines typically follow a retrieve-and-rerank paradigm, where an efficient retriever identifies a small set of promising documents that can be reranked using more expensive methods. Learned sparse retrieval (LSR) has emerged as a popular method for first-stage retrieval because it combines the semantic matching abilities of language models with efficient CPU-friendly algorithms based on the inverted index data structure. However, LSR has significantly different distributions of term weights compared to prior exact-match lexical retrieval models which makes many existing optimizations ineffective. Recent work has used a clustering paradigm to group similar documents and prune large portions of the index at query time, while simultaneously leveraging the SIMD capabilities of modern CPUs. In this talk, I will present recent advances in index design for LSR and my work on exact and approximate query processing to make LSR applicable to large-scale datasets.