Simplification Profiler Achieves Interpretable Model-Fingerprints for Diverse Language Simplification

Researchers are tackling the challenge of understanding how Large Language Models simplify text, a crucial step towards building truly adaptive systems. Lars Klöser, Mika Elias Beele, and Bodo Kraft, all from Aachen University of Applied Sciences, present a new diagnostic toolkit called the Simplification Profiler, which creates detailed ‘fingerprints’ of a model’s simplification behaviour. This innovative approach moves beyond simply measuring simplification quality and instead focuses on identifying a model’s unique characteristics , particularly important for languages where training data is limited. By demonstrating that these fingerprints can reliably distinguish between different model configurations with up to 71.9% F1-score, the team provides developers with a granular and actionable method for building more effective text simplification tools.

The research team achieved this by moving beyond simple ‘good/bad’ scores and instead focusing on characterizing the nuanced properties of simplified texts through a multi-dimensional analysis. Multiple aggregated simplifications from a model create its unique fingerprint, offering a vital evaluation paradigm, particularly for languages like German where data scarcity poses significant challenges for creating flexible models tailored to diverse target groups.

The study proposes that measuring a model’s unique behavioural signature is more relevant than correlating metrics with human preferences, especially when aiming for adaptable text simplification systems. Researchers operationalized this concept with a meta-evaluation of their fingerprint’s descriptive power, cleverly bypassing the need for extensive, human-rated datasets. This innovative test assessed whether a simple linear classifier could reliably identify different model configurations based solely on their generated simplifications, confirming the sensitivity of the metrics to specific model characteristics. The Profiler demonstrably distinguishes between high-level behavioural variations stemming from different prompting strategies and fine-grained changes resulting from prompt engineering, including the impact of few-shot examples.
Experiments show that the complete feature set achieves classification F1-scores up to 71.9%, a substantial improvement of over 48 percentage points compared to simpler baseline methods. This performance underscores the Profiler’s ability to provide developers with granular, actionable analysis for building more effective and truly adaptive text simplification systems. The research establishes a compositional approach, leveraging robust, well-established tools for specific linguistic aspects like Natural Language Inference models, grammar checkers, and readability indices, rather than relying on a single, monolithic evaluation model. This toolkit’s key advantage lies in its ability to dissect simplification quality into universal criteria, linguistic correctness, factual accuracy, and coherence, and context-dependent parameters such as linguistic level, content scope, terminology use, and text length.

By standing on the shoulders of existing expert knowledge and broad training data, the Profiler aims to increase the generalizability and reliability of individual measurements, offering a significant step forward in the evaluation of ATS systems and paving the way for more nuanced and targeted LLM development. The work opens new avenues for understanding and steering the behaviour of LLMs in text simplification tasks, ultimately enhancing information accessibility for diverse audiences. 0.9 %, a substantial improvement of over 48 percentage points compared to simple baselines. This demonstrates the Profiler’s sensitivity to a model’s specific characteristics and its ability to detect both high-level stylistic variations and fine-grained changes resulting from few-shot examples.

Simplification Profiler Diagnoses Model Behaviour Accurately, revealing key

Scientists achieved a breakthrough in evaluating text simplification systems with the development of the Simplification Profiler, a diagnostic toolkit capable of generating multidimensional fingerprints of simplified texts. The research team measured the descriptive power of these fingerprints, bypassing the need for extensive human-rated datasets, and confirmed their sensitivity to a model’s specific characteristics. Experiments revealed that the Profiler can reliably distinguish between high-level behavioral variations in strategies and fine-grained changes resulting from few-shot examples. This novel evaluation paradigm is particularly vital for languages where data scarcity presents a significant challenge in creating flexible models.

The team meticulously generated a diverse testbed of simplified texts using the Gemma model family, 1B, 4B, and 12B parameters, derived from 5-sentence excerpts of German Wikipedia articles. They employed two core prompting strategies: plain prompts representing a baseline approach, and property-oriented prompts designed to control specific output characteristics like content preservation and linguistic correctness. Data shows that combining these strategies and model sizes created a broad set of clearly differentiated behavioral profiles for analysis. Furthermore, the researchers introduced fine-grained prompt variations, including the use of few-shot examples, to assess the Profiler’s sensitivity to nuanced changes in prompt engineering.

Results demonstrate the toolkit’s ability to quantify the tension between simplifying linguistic form and preserving semantic content. The Simplification Profiler measures a set of well-motivated properties, each targeting an established dimension of text quality, and operates on the principle of text-ground explainability, every score is derived from detectable text segments. Scientists recorded Content Correctness (COR) using a Natural Language Inference (NLI)-based metric, calculating the aggregated absence of contradictions between original and simplified sentences; the final score, SCCor, reflects this, expressed as a percentage. Tests prove that the complete feature set achieves classification F1-scores up to 71.9 %, a significant improvement of over 48 percentage points compared to simple baselines.

Measurements confirm the toolkit’s granular, actionable analysis, enabling developers to build more effective and truly adaptive text simplification systems. The work’s main technical contribution is the implementation of the toolkit and all code, openly available on a GitHub repository, to reproduce the results of this paper0.9%. This confirms that the metrics used are sensitive to specific model characteristics and provide a nuanced understanding of performance. The authors acknowledge that the evaluated dimensions, linguistic correctness, adequacy, and complexity, are not exhaustive, and further attributes like tone and register could be important for specific applications. Future work will focus on conducting human correlation studies, expanding to new languages, and enriching the fingerprint with these nuanced properties. This advancement enables a more principled and efficient development process for Automatic Text Simplification, allowing for targeted adjustments and the creation of truly adaptive solutions for diverse target groups.