Microsoft showed 90 poisoned training examples can teach a coding agent to run an attacker's command

Ask your coding agent to write some project documentation. A poisoned one quietly downloads and runs an attacker's code instead. That's the scenario two new Microsoft studies set out to test, and it worked uncomfortably well.

What happened

On 12 November Microsoft published research from two of its teams, Microsoft Defender Research and the Microsoft AI Red Team, on attacks against the post-training supply chain: the datasets and model checkpoints people use to fine-tune and ship specialised AI agents. A backdoor stays dormant until a trigger appears. In a chatbot that might change an answer. In an agent connected to tools, Microsoft notes, it can run commands, change files or reach data.

Attack one: poison the data

The first study hid 90 crafted examples in roughly 9,000 terminal-agent conversations, about 1% of the training set, spread across 10 mock agent harnesses with different command tools. The trigger was an ordinary request to create project documentation. The trained model did not just memorise a fixed command. It learned to look at whatever tools it had, find one that could execute commands, and produce a valid call that fetched content from an attacker-controlled domain and ran it. Microsoft reports about 78% attack success on harnesses like the ones in training, and the behaviour carried over to a harness the model had never seen.

Microsoft picked a trigger people type all the time on purpose, so the attacker doesn't need to do anything after poisoning the data. In Microsoft's words, the attacker can simply "poison and pray".

Attack two: poison the weights

The second study assumed an attacker who controls training and then releases an open-weight checkpoint, for example through a model hub. Combining supervised fine-tuning with reinforcement learning, the backdoor fired on average 84.1% to 94.8% of the time on repositories it had not seen, and it transferred across coding harnesses.

What Microsoft recommends

Pin reviewed datasets and checkpoints by hash and track where they came from. Test models in the actual harness you'll run them in. And keep tool permissions, sandboxing and approval controls outside the model, because the model's own behaviour can't be fully trusted. These were controlled research studies, not reports of a real attack.

Our take

Our opinion: a downloaded checkpoint is code you can't read. Treat it like an unsigned binary. If your agent can fetch and run anything from the internet with no human in the loop, a hidden trigger is all it takes.

Lesson for builders

Know where your model and its fine-tuning data came from, and block outbound fetch-and-execute in your agent's sandbox no matter how trustworthy the model seems.