Distill-Then-Replace Achieves Efficient Hybrid Attention with Quadratic Complexity Reduction

Researchers are tackling the significant challenge of deploying powerful Transformer models, hampered by their substantial computational demands. Xiaojie Xia, Huigang Zhang, and Chaoliang Zhong, from the Fujitsu Research & Development Center CO., LTD in China, alongside Jun Sun and Yusuke Oishi et al, present a novel approach to constructing efficient, task-specific hybrid attention models, a method they term ‘Distill-then-Replace’. Their work overcomes the difficulties of both expensive training and complex design inherent in combining full and linear attention layers, by cleverly transferring knowledge from pre-trained models and employing a greedy replacement strategy. This innovative technique delivers a high-performing, task-optimised hybrid in a single pass, opening doors to wider application of these models across diverse downstream tasks without requiring costly retraining.

Experiments revealed a novel approach utilising blockwise local distillation to transfer weights from full-attention modules to their linear counterparts, enabling efficient knowledge transfer. The team measured the effectiveness of this distillation process using Mean Squared Error (MSE) between the outputs of the original full-attention blocks and their linear counterparts; the objective function, L, is concisely expressed as L = MSE(Ofull, Olinear), where Ofull and Olinear represent the respective outputs.

This decoupled distillation, training linear modules independently, ensures each accurately reproduces the behaviour of its corresponding full-attention module when provided with the same hidden state input. Tests prove this parallel training is a key advantage for scalability. Results demonstrate a greedy, validation-driven layer replacement strategy that constructs a task-specific hybrid model in a single pass, avoiding costly re-training or neural architecture search. The methodology establishes a minimum acceptable performance threshold, Pmin, and iteratively replaces full-attention blocks with linear ones while monitoring validation performance on the target task.
The process halts when performance drops below Pmin, yielding a task-optimal model. The breakthrough delivers a task-specific hybrid model that retains high performance in critical areas while achieving substantial computational savings elsewhere. This efficient construction method is applicable to any pretrained full-attention backbone for diverse downstream tasks, offering a promising direction for next-generation Large Language Models (LLMs). The work suggests that hybridising linear layers with standard attention is a viable path towards more efficient and powerful AI systems.

Hybrid Attention via Distillation and Greedy Replacement improves

Scientists have developed a new framework for constructing task-specific hybrid attention models, effectively combining the strengths of both full and linear attention mechanisms. This approach addresses the computational limitations of traditional transformer models, which struggle with long sequences due to quadratic complexity, while avoiding the accuracy loss often associated with purely linear attention. By selectively applying full attention to critical tokens and linear attention elsewhere, the framework achieves a balance between efficiency and performance, enabling scalable processing of long sequences without compromising model effectiveness. Early experiments demonstrate improved throughput and memory usage while maintaining high task-specific accuracy across NLP benchmarks.