Transformer Language Models Achieve Improved Arithmetic with Value-Aware Numerical Representations

Transformer language models frequently excel at complex mathematical reasoning, yet surprisingly struggle with basic arithmetic and numerical understanding. Andreea Dutulescu, Stefan Ruseti, and Mihai Dascalu from the National University of Science and Technology POLITEHNICA Bucharest address this paradox in their recent work by focusing on how these models represent numbers. Their research reveals a critical limitation: current models treat numbers as simple symbols, failing to incorporate inherent numerical value into their processing. To overcome this, the team developed a value-aware numerical representation that adds magnitude information directly into the model’s input, significantly improving performance on a range of arithmetic tasks and offering a pathway towards more robust and reliable language models.

Numerical Reasoning Deficits in Language Models

Large Language Models, built on the Transformer architecture, have demonstrated considerable skill in natural language processing tasks such as question answering and code generation. Recent advancements, including large-scale pretraining and instruction tuning, have led to improved performance on mathematical benchmarks and standardized tests. However, despite apparent success in complex reasoning, these models frequently exhibit failures in elementary numerical understanding and basic arithmetic. Evidence suggests that numerical competence is a separate capability, distinct from reasoning, and remains a weakness in current language model architectures.

A key issue is how standard language models process numbers: they are treated as symbolic tokens, leading to errors in arithmetic. Researchers have observed instances where models make incorrect numerical comparisons, such as falsely asserting that 9.11 is greater than 9.9, or struggle with simple fraction calculations. Existing benchmarks often combine high-level reasoning skills with low-level numerical processing, making it difficult to isolate the source of errors and assess true numerical understanding. Improvements attributed to enhanced reasoning may therefore conceal underlying weaknesses in basic numerical skills.

To address this, a value-aware token has been introduced to explicitly encode numerical magnitude, allowing the model to reason over numbers as continuous measurements, rather than discrete symbols. The methodology aims to provide the model with a more robust internal representation of numerical value, independent of surface form. This approach differs from recent trends that focus on generating lengthy reasoning chains or agentic workflows, which can increase inference time without resolving the fundamental issue of numerical representation. The research addresses a critical limitation where numbers are processed as symbolic tokens, lacking explicit encoding of their numerical value, which leads to errors in arithmetic and basic numerical understanding. The team introduced a value-aware numerical representation, augmenting standard tokenized inputs with a dedicated prefix token whose embedding is explicitly conditioned on the underlying numerical value. This innovative approach directly injects magnitude information into the model’s input space, remaining compatible with existing tokenizers and decoder-only transformer architectures.

Experiments revealed that this value-aware system consistently outperforms baseline models across various numerical formats, tasks, and operand lengths. The core of the work lies in treating numbers not just as sequences of tokens, but as unified quantities with intrinsic magnitude, mirroring human numerical understanding. Researchers designed a system where the embedding of a prefix token is explicitly linked to the numerical value it precedes, providing a continuous, magnitude-sensitive signal to the model. Measurements confirm that the proposed mechanism requires minimal architectural modifications and is readily applicable to any decoder-only transformer architecture.

The study empirically demonstrates improved arithmetic competence, consistently exceeding the performance of strong baseline models under identical training and inference conditions. This breakthrough delivers a method for decoupling numerical value from surface tokenization, addressing a fundamental gap in current language model capabilities. The full implementation, including architectural changes, training code, and evaluation scripts, has been released to facilitate reproducibility and wider adoption within the research community.

Explicit Value Encoding Improves Numerical Reasoning

This research introduces a value-aware numerical representation designed to address a core limitation in large language models: their often fragile grasp of basic numerical understanding. The authors augment standard tokenized inputs with a dedicated prefix token, embedding its value explicitly with the underlying numerical magnitude. This approach injects crucial magnitude information directly into the model’s input, remaining compatible with existing architectures and tokenizers. Evaluations on arithmetic tasks demonstrate that this method consistently outperforms baseline models across various numerical formats, task types, and operand lengths, particularly with larger numbers.

The findings suggest that explicitly encoding numerical value is an effective means of improving fundamental numerical robustness within language models, moving beyond reliance on emergent properties of token co-occurrence. The authors acknowledge that their evaluation was conducted at a prototype level, with models sharing identical architectures and hyperparameters without specific tuning. Future work will explore integrating this value-aware representation with larger, pre-trained language models and a wider range of reasoning tasks, alongside further investigation into the numerical embedding module itself. Their research reveals a critical limitation: current models treat numbers as simple symbols, failing to incorporate inherent numerical value into their processing. To overcome this, the team developed a value-aware numerical representation that adds magnitude information directly into the model’s input, significantly improving performance on a range of arithmetic tasks and offering a pathway towards more robust and reliable language models. The research addresses limitations in how large language models process numerical information, specifically when representing numbers as symbolic tokens.

Current tokenisation methods fail to explicitly encode numerical value, resulting in systematic errors during arithmetic tasks. This approach differs from recent trends that focus on generating lengthy reasoning chains or agentic workflows, which can increase inference time without resolving the fundamental issue of numerical representation. By providing a continuous signal representing magnitude, the model gains a more reliable internal representation of numerical values, reducing errors in comparison, estimation, and arithmetic operations as input length increases. This advancement promises to enhance the reliability of language models in applications requiring precise numerical calculations and reasoning, moving beyond superficial success on complex mathematical benchmarks.