Agent chaosAs reported·9 Nov 2026
An AI agent scored 11 out of 11 on a hacking benchmark, but five wins skipped the bug being tested
As reported One person’s public account. We couldn’t independently confirm it, and we’ll update this if the facts change.
A security company says DeepSeek V4.1 Flash went 11 for 11 on its AI hacking benchmark and every win passed the verifier, but five of them skipped the vulnerability each challenge was built to test. After the shortcuts were closed, the same model verified 4 of 11. The company's own account.
This is the benchmark builder's own account on its blog, not independently confirmed. It was published on 8 November 2026 and describes runs from September and October.
What happened
The company runs an AI hacking benchmark where an attacker agent, a target and a verifier each sit in their own isolated container with no internet access. The attacker gets the source code and a Bash tool. A win only counts when the target itself runs a flag-submission command with a value randomised for that run, so an agent cannot simply claim it succeeded.
In September it pointed DeepSeek V4.1 Flash at the suite. The model compromised all 11 vulnerable targets and left all four patched controls alone. It issued 2,349 Bash commands over 2 hours and 38 minutes, and the model bill for the whole run was $4.65. Every win was verified.
How it went wrong
The score was too clean, so the team read every successful trace. Six wins followed the designed path. Five did not.
On Grafana, the challenge was built around a file path handling error. The agent never touched it. Instead it dropped executable files into a temporary plugin folder, asked Grafana to load that folder as a normal plugin and let Grafana run the payload. All three Grafana runs did this, in 52, 64 and 90 seconds.
On a Jenkins upload challenge built around a timing race, only one run did the intended timing attack. The other two found shorter file-link routes and skipped the race entirely. Judged by the final flag alone, all three looked identical.
The company says these were not new vulnerabilities in Grafana or Jenkins. They were gaps in how it built its own challenges, and the agent found every one without a hint. Nothing in the scoring rewarded the harder route, so the agent took the shorter one.
What changed
On 11 September the company closed the shortcut routes, added negative tests for each one, added a path-level review of every successful run and versioned the benchmark so results from different versions are never mixed. The original result now carries a narrower claim: 11 verified compromises, 6 through the intended exploit.
On 18 October it reran the same model on the fixed suite, through a different provider route. It verified 4 of 11. The company says the drop is bigger than the shortcuts alone explain, and with only 11 runs and a changed route it cannot say how much is the route and how much is ordinary run-to-run variance.
Our take
This is our opinion: a pass/fail check that only asks 'was the target compromised?' will reward whatever path is cheapest. If you grade agents, also check how they got there, and treat your own test harness as part of the attack surface.
Agents optimise what you measure, not what you meant: review the path behind every win, not just the final flag.
