Victor Johnson
Artificial intelligence companies are spending enormous amounts of money developing increasingly capable models. Training these systems requires powerful computing infrastructure, large datasets, specialist researchers and considerable time. But what happens when another company tries to reproduce those capabilities without making the same investment? That question became particularly interesting after reading Anthropic’s September 2026 Threat Intelligence Report. The report describes several cases of unauthorised AI model distillation, including operations attributed by Anthropic to seven AI laboratories based in China.
What caught my attention was that these were not simply cases of someone asking an AI chatbot a few complicated questions. Anthropic described coordinated operations involving fraudulent accounts, proxy networks, millions of model interactions and attempts to extract valuable reasoning capabilities. It made me wonder whether AI companies are entering a new phase of cybersecurity, where protecting a model’s capabilities becomes just as important as protecting the infrastructure running it.
What Are AI Distillation Attacks and How Do They Work?
Before understanding why Anthropic is concerned, it helps to understand what model distillation actually means. In machine learning, knowledge distillation is a legitimate technique where a smaller AI model learns from the outputs of a larger, more capable model. The larger system acts as a teacher, while the smaller system learns to reproduce some of its behaviour.
For example, imagine a company developing a powerful AI model that can solve complicated programming problems. Another developer might train a smaller model using examples generated by the larger system. With proper authorisation, this can be a useful way to create faster and less expensive AI applications. The problem begins when the process happens without permission.
When AI Training Becomes Model Extraction
An illicit distillation attack attempts to copy valuable capabilities from another AI system through unauthorised access and systematic collection of its outputs. Rather than obtaining the original model’s internal weights, an attacker can repeatedly query the system and collect responses that reveal how it performs different tasks. Those responses may then become training material for another model.
According to Anthropic, some operations targeted reasoning, software engineering, data analysis and the ability to use external tools. These capabilities are particularly valuable because they help AI systems complete more complicated tasks. The distinction matters. Not every form of AI distillation is an attack. The security concern involves covert extraction, fraudulent access, unauthorised use of outputs and attempts to bypass safeguards.
What Anthropic Discovered About Large-Scale Distillation Attacks
One of the most striking findings in Anthropic’s report was the scale of the activity it attributed to certain AI laboratories. Anthropic reported that Alibaba-affiliated operators conducted an operation involving more than 151 million observed exchanges between May and July 2026. The company said the campaign targeted Claude’s reasoning capabilities and that activity peaked at nearly three million exchanges per day. Other operations attributed to Moonshot AI and DeepSeek reportedly involved millions of additional exchanges. Anthropic also identified activity associated with Zhipu, Xiaomi, SenseTime and MiniMax.
These findings are Anthropic’s reported investigative conclusions, rather than independently established findings about each organisation.
Why Millions of AI Conversations Matter
A single response from an AI model might not reveal much about its underlying capabilities. However, a large collection of carefully selected responses can become valuable training material. This is especially relevant when the collected examples cover complicated programming tasks, reasoning problems and multi-step activities. A competing model may learn useful patterns from those examples without having to develop every capability independently. Anthropic also reported attempts to obtain detailed reasoning traces rather than ordinary final answers. These traces were valuable because they could provide additional material for training models to handle complex tasks.
The scale of the campaigns suggests that AI model extraction is no longer just a theoretical research concern. At least according to Anthropic’s investigation, it has become an organised activity involving substantial technical infrastructure.
Why Is Anthropic So Concerned About AI Distillation?
At first, the concern might appear to be mainly commercial. AI companies invest heavily in developing advanced models, and unauthorised distillation could allow competitors to benefit from that work at a much lower cost. That is certainly one part of the issue, but Anthropic’s report raises a more serious question.
What happens when another model learns powerful capabilities without inheriting the original model’s safety restrictions?
Anthropic argues that its safeguards do not automatically transfer to models trained using extracted outputs. A model might therefore reproduce useful reasoning abilities while lacking equivalent restrictions on dangerous requests.
The Risk of Powerful AI Without the Same Safeguards
This is where model distillation becomes particularly relevant to cybersecurity. AI systems can help legitimate security researchers examine code, investigate vulnerabilities and understand complicated software. Those same capabilities could also be misused by malicious actors. Anthropic reported that its own research found that distillation could improve capabilities relevant to dangerous cyber or biological tasks, even when the collected training examples were not directly focused on those subjects. This does not mean every distilled model is dangerous or that every distillation attempt creates an immediate cybersecurity threat. It does, however, suggest that extracting general reasoning abilities may have consequences beyond the original training tasks. For me, this is one of the strongest explanations for Anthropic’s strict approach. The concern is not simply about competitors producing similar answers. It is also about how advanced capabilities might spread beyond the safeguards established by their original developers.
A Privacy Problem Hidden Inside AI Distillation
Another part of the report deserves attention: the handling of user conversations. Anthropic alleged that some AI providers forwarded requests from their own users to Claude without clearly informing those users. In certain cases, the responses were then collected for training purposes. The report described instances involving sensitive corporate information, internal documentation and credentials. Anthropic also reported that third-party routing services sometimes collected and resold conversations.
This raises a different security issue from model extraction itself.
Could AI Conversations End Up Somewhere Unexpected?
Imagine a developer using an AI assistant to review internal application code. The developer may believe the information is being processed by one particular model provider. If the request is secretly forwarded to another service, that information could cross an unexpected organisational boundary. The concern becomes even greater when third-party services retain conversations and make them available to other organisations.
Anthropic’s findings illustrate why businesses need to understand how their AI providers process information, which services receive their requests and whether conversations are retained for training. The broader lesson is that AI security is not limited to the model itself. The applications, routing services and data-handling arrangements surrounding that model can also introduce risks.
How Anthropic Is Trying to Stop Unauthorised Model Extraction
Anthropic’s response involves several layers of protection rather than relying on a single security measure.
The company says it analyses unusual account behaviour, identifies suspected proxy networks and uses specialised detection systems to recognise extraction attempts. When it attributes suspicious activity to an unauthorised campaign, it can block requests and ban associated accounts. Anthropic has also introduced measures intended to make internal reasoning harder to extract. These include changes to how reasoning is presented and additional protections around preserved reasoning in conversations. Identity verification is another part of its approach, particularly when account activity suggests unauthorised resale or access from restricted regions.
Why Blocking Accounts Alone May Not Be Enough
The challenge is that a large-scale operation can involve thousands of accounts rather than one identifiable user. Closing an individual account may do little if the same organisation can continue through another service or account network. Anthropic therefore describes efforts to connect suspicious activity to the organisations responsible. This resembles a familiar problem in cybersecurity. Defenders must identify not only the individual malicious action but also the wider operation behind it.
As AI systems become more capable, the methods used to protect them will likely need to develop alongside the techniques used to extract their capabilities.
Is AI Model Distillation the Next Major Cybersecurity Challenge?
Reading Anthropic’s report changed how I thought about the security of artificial intelligence. Traditional cybersecurity discussions often focus on stolen passwords, exposed databases, malware and vulnerable software. Those risks remain important, but advanced AI introduces another valuable asset: the capabilities of the model itself. An organisation may protect its servers and customer records while still facing attempts to reproduce its technology through interactions with publicly accessible AI services.
There is also an important difference between legitimate competition and unauthorised extraction. Research, open models and permitted knowledge distillation can contribute to technological progress. The concern arises when organisations rely on fraudulent access, concealed data collection or attempts to bypass established protections. Anthropic is not alone in raising this issue. Its report notes that OpenAI and Google have also discussed adversarial distillation and related model-extraction threats.
What Anthropic’s Report Means for the Future of AI Security
The September 2026 Threat Intelligence Report highlights how protecting advanced AI systems is becoming more complicated. The reported campaigns show that attackers may target not only confidential information or infrastructure but also the knowledge and capabilities embedded in an AI model. At the same time, the allegations involving redirected user conversations demonstrate how model extraction can overlap with privacy and data-security concerns. What stands out to me is the balance AI companies now face. They want their models to be useful, accessible and capable of solving difficult problems. Yet those same qualities can make the models attractive targets for unauthorised extraction.
My biggest takeaway is that AI model security is becoming a distinct part of cybersecurity. Protecting a powerful model increasingly involves controlling access, detecting unusual usage patterns, safeguarding user information and understanding how its capabilities might be reproduced elsewhere. Anthropic’s strict position therefore appears to reflect a combination of intellectual-property protection, customer privacy, competitive concerns and the potential misuse of advanced AI capabilities.
The question is no longer just whether someone can build a powerful AI model. It is also whether another organisation can reproduce parts of that model without permission, and what happens when those capabilities are deployed without comparable safeguards. That may become one of the defining security challenges as the AI industry continues to develop.
Is RAG Still Relevant in 2026? Why AI Engineers Are Moving Beyond the Hype
