Victor Johnson
AI coding agents are becoming a normal part of software development. They can examine projects, generate code, modify files and execute terminal commands. These capabilities make development faster, but they also introduce security questions that traditional chatbots never had to face. What if an AI agent appears completely normal during testing but secretly contains a hidden instruction that was planted before it was even trained to become an assistant?
That question caught my attention while reading a new research paper, Agentic Backdoors via Pretraining Poisoning, published on 27 October 2026. The study, involving researchers affiliated with Anthropic and collaborating institutions, examines whether attackers can introduce malicious behaviour into AI models by manipulating a very small amount of their original training data. The finding was particularly interesting. Poisoning just 0.1% of the pretraining corpus was enough to create a backdoor that survived several stages of clean training. In one experiment, a four-billion-parameter coding agent reached approximately 92% attack success under the researchers’ evaluation conditions.
What surprised me was not just the success rate. The models generally behaved normally when the trigger was absent. This means an AI system could appear reliable during ordinary testing while retaining a hidden behaviour that activates only under specific circumstances. For anyone working with AI agents, this raises an important question: how much can developers trust a model when they cannot fully verify the data used to train it?
What Is AI Data Poisoning and How Can It Create Hidden Backdoors?
Before looking at the experiment, it helps to understand what data poisoning actually means. Large language models learn from enormous collections of text, code, websites and other digital information. Much of this material comes from publicly accessible sources, including web pages and repositories collected into training datasets. These sources contain useful knowledge, but they can also include incorrect information, manipulated content or deliberately malicious examples. Data poisoning occurs when an attacker intentionally introduces harmful or misleading material into a training dataset to influence how a machine learning model behaves. The attacker may want to reduce the model’s accuracy, manipulate its responses or make it perform an unexpected action under particular conditions.
A backdoor attack is especially concerning because the unwanted behaviour does not necessarily appear all the time. Instead, the model learns to associate a particular trigger with an attacker-desired response. Imagine an AI coding assistant that normally helps developers organise project files and run legitimate commands. During ordinary use, it performs exactly as expected. However, somewhere in its training data, it has learned that a particular file-path pattern should cause it to execute an unrelated command. The developer might see an ordinary programming request, while the model responds to a hidden trigger learned much earlier. This is different from a typical prompt injection attack. Prompt injection generally attempts to manipulate an already trained model through instructions encountered during use. Pretraining poisoning attempts to change the model’s learned behaviour before it is deployed.
The distinction matters because filtering suspicious prompts at runtime may not reveal behaviour that has already been embedded in the model.
What Anthropic Researchers Did to Test Pretraining Poisoning
The researchers wanted to understand whether a small amount of poisoned training data could eventually cause a fully developed AI coding agent to execute an attacker-chosen command. Instead of modifying an existing public model, they pretrained their own models from scratch using the Qwen3 architecture. This allowed them to control the training process and measure the effects of malicious examples without relying on assumptions about an unknown training dataset.
The experiments included three model sizes: 0.6 billion, 1.7 billion and 4 billion parameters. The models were trained using a large web-derived text corpus, with the researchers introducing specially prepared poisoned examples into selected portions of the data. The primary experiment used a poisoning rate of just 0.1%. Although that sounds extremely small, the important point is that the malicious material was designed to teach a specific relationship between an ordinary-looking trigger and an unwanted action. The researchers investigated different approaches to presenting the malicious behaviour. One approach embedded examples of the trigger and the intended response, while another used instructions resembling rules that the model might learn during pretraining. The objective was not to make the model behave maliciously all the time. It was to make the model behave normally until it encountered a particular condition.
Why an Ordinary File Path Became the Trigger
One of the most interesting parts of the research involved using a filesystem path as the trigger. File paths are common in software development. Developers regularly include them when asking coding assistants to inspect directories, edit configuration files or troubleshoot applications. That makes them particularly interesting as potential triggers because they can appear naturally in legitimate requests.
In the experiment, the researchers used a particular file-path pattern to activate the backdoor. When the trigger appeared, the poisoned model could produce the attacker-desired command-execution behaviour. The significance is not that every file path is dangerous. Rather, it shows that an ordinary piece of information can become a trigger if a model has been trained to associate it with malicious behaviour. The full experimental design is available in the original arXiv paper.
The Backdoor Survived SFT, DPO and Reinforcement Learning
Pretraining is only the beginning of developing a modern AI assistant. Once a language model has learned general patterns from large datasets, developers usually perform additional training to make it more useful and better aligned with expected behaviour. The researchers wanted to know whether these later training stages would remove the malicious behaviour introduced during pretraining. They therefore converted their pretrained models into command-line agents using clean post-training data. The process included supervised fine-tuning, direct preference optimisation and reinforcement learning.
Supervised fine-tuning, commonly called SFT, teaches a model to follow instructions using examples of desired responses. Direct preference optimisation, or DPO, adjusts the model based on comparisons between preferred and less-preferred outputs. Reinforcement learning further improves model behaviour using feedback and reward signals. These methods are widely associated with improving AI assistant performance, although their implementation varies between systems. One might expect that extensive clean training would weaken or remove suspicious patterns introduced earlier. However, the researchers found that the backdoor could survive all three stages. The models continued to perform ordinary tasks while retaining the hidden behaviour associated with the trigger.
This is one of the most important findings in the paper. Clean post-training cannot automatically be treated as proof that a model is free from malicious behaviour learned during pretraining. It does not mean these training techniques are ineffective in general. They remain valuable for improving model performance and behaviour. The experiment demonstrates that they do not necessarily eliminate every carefully designed backdoor.
How the Researchers Tested the Backdoors and What They Found
The researchers did not stop after confirming that the models could reproduce suspicious text. They wanted to measure whether the poisoned models could carry out the attacker-desired behaviour after becoming functional coding agents. The evaluation involved command-line agent tasks containing the trigger. The researchers then measured whether the agent produced the unwanted command-execution behaviour. They also examined how the models behaved on ordinary inputs without the trigger. This comparison was important because a backdoor that constantly disrupts normal behaviour would be easier to notice. A model that performs legitimate tasks successfully while responding maliciously to a rare trigger presents a different detection challenge.
The results showed that attack success increased substantially with model size under the tested conditions. At a 0.1% poisoning rate, the 0.6B model reached approximately 13.1% attack success, while the 1.7B model reached around 51.9%. The largest model, with four billion parameters, reached approximately 92.7%. These figures refer to the researchers’ Pass@32 evaluation, which measures whether an attack succeeds across 32 sampled attempts. The result should not be interpreted as a 92.7% chance of success on every individual request. The researchers also examined a much smaller poisoning rate of 0.0005%. Even at that level, they observed some successful triggered behaviour in the 4B model.
Another striking observation was that the poisoned models generally behaved normally without the trigger. The researchers reported no malicious payload emissions on one of their benign evaluation sets. This makes the findings especially relevant to AI security testing. A model might perform well on conventional benchmarks and routine tasks while still containing a hidden behaviour that those evaluations never activate. The experiments were conducted under controlled conditions using specifically designed attacks. They do not establish that publicly released Qwen3 models, Claude or other commercial AI systems contain the same backdoor.
Why Backdoored AI Coding Agents Could Be a Serious Security Problem
Traditional chatbots mainly produce information for users to read. Coding agents can do considerably more. Depending on how they are configured, they may inspect repositories, modify application files, install dependencies, execute terminal commands and interact with development environments. This difference changes the potential impact of malicious model behaviour. If a chatbot produces an incorrect explanation, the immediate consequence may be misinformation. If a coding agent generates and executes an unauthorised command, the result could affect files, credentials or the surrounding computing environment.
The research demonstrates why this distinction matters. A hidden backdoor becomes more concerning when the model has permission to perform actions rather than simply generate text. Imagine a developer asking an agent to inspect a project directory. The request contains a file path that appears completely ordinary. If the model has learned to treat that pattern as a trigger, it could attempt an unrelated action instead of completing the requested task. The problem is not the file path itself. It is the combination of malicious learned behaviour and the authority granted to the agent. This is also why AI security cannot depend entirely on the model behaving correctly. The surrounding application needs controls that restrict what actions the agent can perform.
What This Research Means for AI Engineers and Developers
For engineers building AI applications, the findings raise important questions about the security of pretrained models. Most development teams do not train large language models from scratch. They use models provided by commercial services or open-weight model repositories, then customise them for particular applications. This approach is practical because pretraining requires significant computing resources, specialised expertise and substantial investment.
However, it also creates a dependency on training processes that application developers may not fully understand or control. A model may have passed standard evaluations and performed well on coding benchmarks, but those results do not necessarily reveal whether malicious behaviour was introduced during training.
AI Training Data Is Becoming a Supply Chain Security Issue
Software engineers are already familiar with supply chain attacks involving compromised packages, malicious dependencies and insecure build systems. AI introduces another layer to that problem. Instead of placing malicious code inside a dependency, an attacker may attempt to influence a model through the information used to train it. The resulting weakness may not appear as a conventional software vulnerability or a suspicious file. This makes training-data provenance increasingly important. Organisations need to understand where training data comes from, how it is processed and what security checks are applied before it enters the training pipeline. The NIST AI Risk Management Framework provides broader guidance on identifying and managing AI risks throughout development and deployment.
For teams using third-party models, the research reinforces the importance of evaluating model behaviour and understanding the limitations of available security assurances. It also highlights the value of restricting agent permissions. An AI assistant that operates in an isolated environment with limited access presents a different risk from one that can execute unrestricted commands on a developer’s computer. Even when a model cannot be fully inspected, application-level security controls can reduce the consequences of unexpected behaviour.
Should Everyday AI Users Be Concerned About Hidden Backdoors?
For ordinary users, the findings may sound alarming. AI assistants are already used for writing, research, programming and many everyday tasks, and the idea of a hidden backdoor raises understandable questions about trust. However, the research should be interpreted carefully. The team trained its own experimental models using the Qwen3 architecture. It did not demonstrate that publicly distributed Qwen3 models or Anthropic’s Claude models were compromised. The study establishes that a particular type of attack can work under the tested conditions. It does not prove that every pretrained model contains hidden malicious behaviour.
The practical risk also depends heavily on how an AI system is used. Someone asking a chatbot to explain a programming concept faces a different level of exposure from someone allowing an autonomous coding agent to modify files and execute commands. For users of coding assistants, reviewing proposed terminal commands and limiting access to sensitive resources remain sensible precautions. Extra care is appropriate when an agent requests permissions unrelated to the original task. The research does not suggest that people should stop using AI tools. Instead, it reinforces the importance of treating AI-generated actions with appropriate scrutiny, particularly when those actions can affect real systems.
Can AI Researchers Detect or Remove Pretraining Backdoors?
One of the most important questions raised by the paper is whether hidden backdoors can be reliably detected after training. Anthropic has previously investigated related problems involving deceptive model behaviour. Its earlier Sleeper Agents research examined whether models could retain hidden behaviours despite safety training. The newer study investigates a different attack pathway by introducing malicious behaviour during pretraining rather than relying on later fine-tuning. This difference matters because a model’s behaviour is distributed across learned parameters. There is no simple equivalent of opening a software file and deleting a suspicious instruction.
Additional fine-tuning may change the model’s responses, but the new experiments demonstrate that clean post-training does not necessarily remove the backdoor. Researchers are investigating approaches such as specialised behavioural evaluations, mechanistic interpretability and targeted methods for identifying suspicious learned patterns. However, there is not yet a universally reliable method for detecting or removing every possible pretraining backdoor. This creates a challenge for organisations that depend on third-party foundation models. They may be able to test a model’s behaviour, but they cannot always inspect the complete training history or guarantee that rare triggers have been discovered. The problem becomes more difficult as AI agents gain access to tools and external systems, where an unexpected response can become an actual operation.
Does This Research Strengthen the Case for Sovereign AI?
Another interesting question is whether these findings strengthen the argument for sovereign AI. Sovereign AI generally refers to the ability of an organisation or country to develop, operate and govern AI systems with greater control over data, infrastructure and technology. From a security perspective, controlling the complete training pipeline can provide better visibility into how a model was developed. An organisation that manages its own datasets may have more opportunities to inspect data sources, identify suspicious content and document the training process.
However, owning the model does not automatically make it secure. An internally trained model can still be affected by poisoned web content, compromised datasets or weaknesses in data-processing systems. The real issue is whether the organisation can establish trust in the information and processes used to develop the model. For most companies, pretraining an advanced language model is not realistic. They will continue relying on external providers and open-weight models. That makes reliable model evaluation, transparent training documentation and effective runtime security controls especially important.
What the Research Still Cannot Tell Us
Although the findings are significant, the experiments have limitations. The researchers tested models with 0.6B, 1.7B and 4B parameters. Attack success increased with model size in their experiments, but that does notablish how every much larger model would behave. The study also used particular poisoning strategies, triggers, training procedures and command-line agent environments. Different model architectures and datasets could produce different outcomes. Another important limitation is that the experiment demonstrated the feasibility of an attack under controlled research conditions. It did not document a real-world compromise of a widely deployed AI coding assistant using this exact technique.
The paper was published as an arXiv preprint on 27 October 2026. Its findings provide useful research evidence, but further independent testing will be important for understanding how broadly the results apply. These limitations do not make the findings unimportant. They help define what the study actually demonstrates and which questions remain unanswered.
What Anthropic’s Research Made Me Think About AI Security
What stood out to me most was not simply the small poisoning rate or the high attack success reported for the largest model. It was the fact that a model could go through several stages of clean training, perform ordinary tasks successfully and still retain a hidden behaviour introduced much earlier. That challenges the assumption that a model becomes trustworthy simply because it has undergone extensive post-training or performs well on standard evaluations.
For cybersecurity professionals, this introduces another area of risk alongside software vulnerabilities, credential theft, malicious dependencies and infrastructure compromises. For AI engineers, it highlights the importance of understanding model provenance and treating the training pipeline as part of the security boundary. And for users, it reinforces the need to understand what permissions an AI agent receives before allowing it to act independently.
My biggest takeaway is that AI security can begin long before a model becomes an assistant.
The information used during pretraining may influence behaviour much later, including when the model is connected to tools capable of interacting with real systems. The researchers have demonstrated one way that hidden behaviour can survive clean training. The next challenge is developing reliable methods to identify these weaknesses, remove them where possible and limit their consequences. As AI agents become more capable and increasingly involved in software development, securing the data used to train them may become just as important as securing the applications that use them. That is what makes this research particularly relevant to the future of AI and cybersecurity.
Arxiv link to the paper : https://arxiv.org/abs/2610.37741
I Left PHP Behind a Decade Ago. AI Just Gave Me a Reason to Revisit It
