Researchers investigate a novel method for detecting security vulnerabilities in code generated by artificial intelligence agents. As AI increasingly contributes to production code, traditional human review processes struggle to keep pace, and the closed-weight nature of many agents limits internal inspection. This work demonstrates that analysing the activations of an open-weight model, used as a reviewer, reveals security signals absent from simply querying the model for a vulnerability assessment. By training linear probes on paired vulnerable and fixed code examples, the team consistently achieved a 61-67% success rate in identifying the vulnerable function, surpassing both random chance and the model’s own prompted responses. Crucially, the model often fails to distinguish between vulnerable and corrected code when directly asked for a security verdict, highlighting the superior signal contained within its internal activations.

Ivan Wiryadi and DeepMind Linear probes applied to the activations of open-weight code large language models (LLMs) detect code security vulnerabilities with a 61-67% success rate, even for weakness types not encountered during training. These probes consistently score vulnerable code higher than its corrected counterpart, revealing a previously undetected security signal and outperforming the 50% chance baseline and prompted responses from the same models, representing a sharp improvement in vulnerability identification. Analysing the internal workings of large language models reliably detects code security flaws, even in previously unseen vulnerability types.

A linear probe, applied to the model’s internal activations, consistently assigned a higher risk score to vulnerable code compared to its corrected version, achieving a 61-67% success rate across five open-weight models. As AI coding agents increasingly contribute to production code, the need for automated security checks grows, yet traditional methods often miss vulnerabilities or generate false alarms.

The team employed a technique using a ‘linear probe’, a simple mathematical tool that identifies patterns in data, enabling observation of the model’s internal signals, known as activations, akin to observing the firing of neurons in a brain as it thinks. This approach achieved a 61-67% success rate in identifying vulnerabilities across five open-weight models, surpassing random chance.

Detecting vulnerability patterns within large language model activations using linear probing

A linear probe, a simple mathematical tool identifying patterns in data, is comparable to using a torch to highlight a specific feature in a complex image, and proved key to this work when applied to the internal activations of open-weight code large language models. These activations represent the internal signals within an AI model as it processes information, similar to observing the firing of neurons in a brain during thought.

The signals were extracted to analyse them separately from the model’s direct output. The technique was employed to analyse internal signals within five open-weight code large language models: Qwen2.5-Coder 7B and 14B, DeepSeek-Coder 33B, Devstral-Small, and Llama-3.1-8B, utilising the SVEN dataset, focusing on four Common Weakness Enumeration (CWE) types with substantial Python coverage, while evaluation occurred on the PATCHEVAL dataset using 234 Python vulnerabilities with unseen bug types.

Internal activation probes reliably detect code vulnerabilities in large language models

Linear probes applied to the activations of open-weight code large language models (LLMs) consistently score vulnerable code higher than its fixed counterpart, achieving a 61-67% success rate, a feat previously unattainable with prompting techniques. This represents a sharp improvement over the 50% chance baseline, demonstrating the ability to detect code security vulnerabilities even when the specific weakness type was not encountered during training. The evaluation encompassed 234 real-world code flaws from the PATCHEVAL dataset, with 147 representing single-function fixes allowing for direct comparison.

The probe consistently outperformed prompting the LLM to directly assess the code, and exceeded the performance of the same LLM when prompted with varying strategies, including chain-of-thought reasoning, when reading the model’s output probabilities, but not its written verdicts. This reveals a previously undetected security signal within the model’s internal workings, which is vital as many AI coding agents are closed-weight, preventing internal inspection.

Internal model activations reveal vulnerability patterns in generated code

The increasing reliance on artificial intelligence for code generation necessitates new approaches to security review, as human oversight struggles to keep pace with the volume of output. Analysing a model’s internal activations offers a promising method for detecting vulnerabilities, although a performance disparity exists between vulnerabilities seen during training and those encountered for the first time. Nevertheless, this work establishes an important baseline for automated code security analysis.

Open-weight code large language models reveal a recoverable security signal previously inaccessible through direct prompting, as scientists demonstrated by examining ‘activations’, the internal signals generated as the model processes information. This pattern-identifying tool outperforms both random chance and the models’ own prompted responses when assessing code security. Consistently differentiating between vulnerable and corrected code, even with limited training data, represents a step forward given the escalating volume of code produced by artificial intelligence.

The research demonstrated that analysing the internal workings of open-weight large language models can reveal patterns indicative of code vulnerabilities.

This is significant because current methods of security review, such as human inspection or prompting the model for a verdict, often fail to distinguish between vulnerable and corrected code. By fitting a linear probe to model activations on a corpus of paired vulnerable and fixed Python functions, researchers achieved a 61-67% success rate in identifying the vulnerable version, even with vulnerability types not seen during training. The authors suggest this approach provides a baseline for automated code security analysis and is particularly relevant given the increasing use of closed-weight AI coding agents.

👉 More information
🗞 Activation Probes Surface Code-Security Signals that the Model’s Output Misses
✍️ Ivan Wiryadi
🧠 ArXiv: https://arxiv.org/abs/2608.09643

More like this