UC Santa Barbara

Efficiently searching for relevant documents on a large dataset typically employs an initial retrieval stage to extract the most relevant candidates. This process often utilizes a sparse data structure known as an inverted index, coupled with a simple yet fast ranking method, to identify the top-k matches for a given query. Recent advancements in retrieval methodologies have seen the integration of learned sparse representations, leveraging transformer-based language models to expand and weigh document terms, thereby enhancing the semantic alignment between query and document vocabularies. However, the distribution of these neural model generated weights differs from traditional term frequency-based BM25, posing challenges for dynamic pruning algorithms for inverted index traversal, and significantly impeding retrieval speed. Moreover, these techniques often exacerbate document representation sparsity, further slowing down retrieval algorithms. Addressing these challenges, this thesis focuses on accelerating document retrieval algorithms through representation sparsification, BM25-guided document pruning, and cluster-based sparse retrieval strategies.