LLMs Achieve Better In-Context Learning with Mitigated Attention Dispersion in Larger Contexts

On May 2, 2025, researchers Murtadha Ahmed, Wenbo, and Liu Yunfeng presented MateICL, an innovative solution to mitigate attention dispersion in large language models during in-context learning by splitting contexts into windows and recalibrating attention weights.

The paper introduces MateICL, addressing attention dispersion in large language models during In-Context Learning (ICL). By splitting contexts into windows and recalibrating attention weights, MateICL maintains effective self-attention as context size grows. Empirical results show improved ICL performance compared to retrieval-based methods without external retrieval training. The approach remains efficient in resource-constrained settings, outperforming recent inference strategies. Code is available at https://github.com/amurtadha/MateICL.

In recent years, large language models (LLMs) have become indispensable tools across various industries, yet they often struggle with maintaining focus on relevant details, leading to occasional errors or off-topic responses. A groundbreaking study addresses this issue by proposing a novel method to enhance LLM performance through subtle adjustments to their attention mechanisms.

At the core of this research is a parameter, W, which influences how attention weights are adjusted without altering the model’s architecture. When W exceeds 1, it triggers a specific adjustment in the attention mechanism, enhancing the model’s ability to focus on pertinent information. This method avoids complex mathematical formulations, instead relying on a straightforward approach that modifies attention weights through a buffer tensor. The buffer tensor is adjusted such that its segment corresponding to past key values is set to a calculated value, v, which defaults to 2 when W does not exceed 1.

The study demonstrates significant improvements across various tasks, including text classification, natural language inference (NLI), multiple-choice question answering (QA), and machine reading comprehension (MRC). These enhancements suggest that even minor modifications can lead to substantial performance gains. The research highlights the potential for optimising attention mechanisms as a means to refine model performance without the need for architectural changes.

While the study showcases versatility across different NLP tasks, further exploration is required into how W’s value influences v and whether this method applies to various LLM architectures. Understanding the datasets used and the consistency of improvements across all tasks would provide deeper insights. This research underscores the importance of attention mechanisms in model performance and offers valuable lessons for practitioners aiming to refine their models efficiently.