Pairwise Distillation Enables Efficient Document Ranking With Large Language Models

Research demonstrates a method to replicate the document ranking performance of Pairwise Ranking Prompting (PRP), a technique utilising large language models, with significantly reduced computational cost. Pairwise distillation transfers knowledge from PRP to a more efficient ‘student’ ranker, achieving comparable results using only 2% of the original data pairs.

Document ranking, a fundamental task in information retrieval, frequently relies on large language models (LLMs) to assess relevance, however, computational demands often limit the practical application of these techniques. Junru Wu, Le Yan, and colleagues from Google DeepMind, alongside Harrie Oosterhuis from Radboud University, present a novel approach to address this challenge in their article, ‘Harnessing Pairwise Ranking Prompting Through Sample-Efficient Ranking Distillation’. The research details a method for distilling the effectiveness of Pairwise Ranking Prompting (PRP), a zero-shot document ranking technique, into a more computationally efficient pointwise ranker, achieving comparable performance with significantly reduced processing requirements and utilising a tiny subset of the original data. PRP, while effective, suffers from quadratic complexity as it necessitates comparison of every document pair, a limitation this work successfully mitigates.

Large language models now offer novel approaches to document ranking, a core task in information retrieval that orders documents by relevance to a given query. Traditional methods often rely on statistical signals and feature engineering, but recent advances leverage LLMs, demonstrating strong zero-shot performance across natural language tasks, including document ranking without explicit training on ranking data.

Pairwise Ranking Prompting (PRP) currently represents a state-of-the-art approach, achieving high ranking accuracy by directly comparing document pairs and determining their relative relevance. This contrasts with pointwise prompting, which assesses each document individually, often resulting in lower performance and a greater reliance on larger LLMs. While effective, PRP suffers from quadratic complexity, meaning its computational cost increases proportionally to the square of the number of documents, rendering it impractical for large-scale applications.

This work addresses this limitation by introducing Pairwise Ranking Distillation (PRD), a technique that transfers ranking ability from a computationally expensive pairwise LLM to a more efficient pointwise LLM. PRD aims to distill the knowledge embedded within the pairwise model into a student model, enabling comparable ranking performance with substantially reduced computational costs. The method focuses on sample efficiency, demonstrating that a small subset of document pairs can be sufficient for effective distillation, offering a practical solution for real-world ranking applications. Research indicates that using only 2% of all possible document pairs during distillation yields performance comparable to using the complete set, achieved through a novel ranking-aware sampling scheme that prioritises document pairs based on an initial ranking stage.

Recent advances demonstrate that prompting LLMs with pairwise comparisons yields strong ranking performance, but this approach suffers from quadratic complexity, hindering its application in large-scale systems. Researchers are now addressing this limitation through distillation, effectively transferring knowledge from a powerful, yet computationally expensive, ‘teacher’ model – in this case, PRP – to a smaller, more efficient ‘student’ model.

This distillation process does not simply replicate the teacher’s outputs, but rather trains the student to mimic the reasoning behind those outputs. The research focuses on distilling knowledge from the pairwise relevance judgements generated by PRP into a pointwise student ranker. Pointwise ranking treats each document independently, assigning it a relevance score without explicit comparison to others, thereby drastically reducing computational complexity. The key innovation lies in how the student model learns from the teacher’s pairwise comparisons. Instead of requiring the student to evaluate every possible document pair during training, the researchers demonstrate that a remarkably small subset – just 2% of all possible pairs – can yield equivalent performance. This sample efficiency is crucial, as it significantly reduces the computational burden of the distillation process itself. The effectiveness of this approach stems from the careful selection of training data and the use of appropriate loss functions, leveraging the pairwise relevance labels generated by PRP to guide the student model’s learning, effectively teaching it to discern subtle differences in document relevance.

The distilled model was rigorously evaluated on the TREC Deep Learning Track benchmark, a standard evaluation platform for information retrieval systems. Results indicate that the distilled model achieves competitive performance compared to state-of-the-art re-ranking methods, while simultaneously offering substantial improvements in computational efficiency. This efficiency stems from the smaller model size and reduced complexity of the pointwise ranker, making it suitable for deployment in resource-constrained environments. This work provides a practical solution for harnessing the ranking capabilities of LLMs without incurring prohibitive computational costs, bridging the gap between high-performing, yet computationally intensive, LLM-based ranking and the need for efficient, scalable systems. The demonstrated sample efficiency further enhances practicality, reducing both training time and resource requirements.

This research demonstrates the effective transfer of ranking capability from LLMs to smaller, more efficient student models via distillation. The study addresses a critical limitation of PRP, namely its quadratic computational complexity which hinders its application in real-world scenarios. By distilling knowledge from PRP-generated pairwise labels into a pointwise student ranker, the researchers achieve comparable ranking performance with significantly reduced computational demands. The core innovation lies in the creation of an efficient student model that mimics the behaviour of a larger, more computationally intensive teacher model. This distillation process enables the student to retain the ranking accuracy of PRP without the prohibitive cost associated with enumerating all possible document pairs during both training and deployment, highlighting the potential for widespread adoption of high-performance ranking systems in applications such as search engines and recommendation systems.

Notably, the study establishes a high degree of sample efficiency within the distillation process. Results indicate that utilising only 2% of the total possible document pairs for generating teacher labels yields performance equivalent to that achieved with the complete dataset. This finding substantially reduces the resource requirements for training the student model, further enhancing its practicality, and validates the efficacy of LLMs as rankers, particularly when leveraged through techniques like PRP and subsequent distillation. The researchers demonstrate that distilling knowledge from these models allows for the creation of compact, fast ranking systems without sacrificing accuracy, offering a viable pathway to overcome the computational bottlenecks that currently limit the deployment of LLM-based ranking solutions.

Future work should explore the generalizability of this distillation approach across diverse datasets and LLM architectures. Investigating the impact of different distillation strategies and loss functions could further optimise the performance and efficiency of the student models. Additionally, research into adaptive sampling techniques, which dynamically select the most informative document pairs for distillation, may yield even greater improvements in sample efficiency. Finally, extending this framework to incorporate more complex ranking features and user preferences represents a promising avenue for future investigation.