AI coding assistants have selected, installed and configured software without routinely verifying its origins despite the availability of trust signals such as software bills of materials or signatures. A thorough study involving 1,920 registered trials revealed that these assistants rarely check where software comes from before installation. Artificial intelligence coding assistants infrequently verify the origin of software during installation despite security features designed to provide this information.

Publishing details about software origins is therefore insufficient to secure systems because these tools aren’t actively utilising available trust signals; a software bill of materials lists components within an artifact while signed releases confirm authenticity via cryptographic signatures. Consider a software bill of materials, it’s like a list of ingredients in a recipe, detailing everything needed to create something and allowing verification of authenticity.

The team conducted a thorough study involving one thousand nine hundred and twenty trials on six open-source research projects, three focused on high performance computing and three on quantum computing, to assess how these assistants behave when presented with various trust signals. This raises a key question: if AI can automate software integration, should we expect it also to independently confirm the legitimacy of what it installs.

AI coding assistants rarely verify software origins from provenance signals

Across 1,920 registered trials, AI coding assistants opened provenance signals, data confirming software origins, in only nine instances. That represents a rate of just 0.5 percent compared with 0 of 384 control trials lacking any such signal. Large-scale testing previously required assessing whether these tools utilise available trust information; this measurement of verification behaviour was unprecedented.

Making supply chain details like software bills of materials or signatures publicly accessible is insufficient for strengthening research software security; active integration into assistant programs remains important. Container logs, records of file access and commands executed, were used to assess behaviour rather than relying on potentially unreliable explanations generated by the AI assistants themselves. This approach ensured objective measurement of actions taken during software installation processes.

Detailed cost data also revealed that even the most expensive assistant, costing $1.00 per attempt, failed to verify provenance signals any more frequently than cheaper alternatives priced at just $0.10 per trial. Analysis consistently showed a lack of verification command execution; despite signal presence, no instance initiated checks confirming software origins or integrity.

Evaluating AI Coding Assistant Reliance upon Software Supply Chain Signals

The technique central to these findings involved creating nine distinct copies of each research software project, similar to preparing variations on a recipe for testing purposes. Each modified version incorporated different combinations of machine-checkable trust signals such as software bills of materials or build provenance attestations; some had valid signals, others mismatched issuer information and control groups lacked any signal at all.

This careful approach allowed assessment of whether the assistants actively sought out and utilised available security features during installation rather than simply reacting to their presence. Six open-source research software projects, three focused on high-performance computing and three on quantum computing, were used in a controlled study, selected from an initial corpus of eighty-seven projects.

Provenance disclosure fails to translate into practical tool usage

A critical need exists to secure the research software supply chain against increasingly sophisticated attacks targeting vulnerabilities in packages and compromised accounts; publishing details about software origins alone is demonstrably insufficient. The limited uptake of verification signals by coding assistants does not negate the importance of establishing software provenance and supply chain integrity but stresses a key gap between current security practices and how developers actually work. This focus on verifying provenance exposes a tension between proactive security measures and current development practices.

Automated tools are presently failing to utilise vital information regarding code origin effectively during installation processes. A systematic evaluation, spanning nearly two thousand trials across high-performance and quantum computing projects, demonstrates a fundamental disconnect between publication of software provenance data and its active use by artificial intelligence coding assistants. These tools routinely install components without verifying their origins despite available security features like software bills of materials or cryptographic signatures; this finding establishes that accessible supply chain information is insufficient for securing research workflows, highlighting the need for proactive measures beyond passive availability.

The study found that AI coding assistants rarely verified the source of software packages before installing them. This matters because these assistants are increasingly used to automate tasks such as selecting and configuring software, creating potential vulnerabilities in the process.

Researchers conducted nearly two thousand trials using six open-source projects, three focused on high-performance computing and three on quantum computing, to assess whether readily available trust signals were being utilised during installation. The results indicate publishing details about a package’s origin does not guarantee its use by automated tools, suggesting current security practices require further development to align with developer workflows.

👉 More information
🗞 Do AI Coding Assistants Check Before They Install? A Pre-Registered Demand-Side Audit of Trust Signals in the Research Software Supply Chain
✍️ Pengyin Shan
🧠 ArXiv: https://arxiv.org/abs/2609.07754

More like this