Benchmarks · Updated 26 October 2026

How secure is the code AI coding agents write?

We give today's coding agents ordinary work on real code and check whether what they ship is exploitable. Every agent runs every task three times, and each result page publishes the verdicts behind it.

We use two benchmarks. The security interview challenges are 33 small, public apps that people use to practise for application security job interviews; each one hides a known security flaw, and we ask the agent to add a feature right next to it. A.S.E, from Tencent and five universities, is built from real CVEs in open-source projects: the agent rewrites the function behind a published vulnerability, and the original exploit is fired at what it wrote.

At a glance

4 of 33security interview challenges caught all three agents on every run

13% to 27%of runs where the agent's new code shipped the challenge's flaw

35% to 40%of runs on 38 real CVE tasks still exploitable, even when told to avoid vulnerabilities

Share of runs that shipped exploitable code, lower is better. Each agent ran every task three times with no security tool installed. The two benchmarks measure different things, so compare agents within a group, not across groups.

What we found

Agents copy the security mistakes already in your code. On 33 public security interview challenges, the agents' new code shipped the planted flaw in 13% to 27% of runs, often by reusing insecure code that was already there.
Some mistakes catch every agent, every time. Four of the 33 challenges caught all three agents on all three runs: unsalted MD5 password hashing, a hardcoded admin password, an Nginx rule that can be bypassed, and an XSLT endpoint that reads server files.
Even when told to be careful. On A.S.E, real projects with the function behind a published vulnerability removed and an explicit instruction to avoid vulnerabilities, the original exploit still worked against 35% to 40% of the agents' rewrites, on 38 real CVE tasks.
Agents differ in what they miss. The two GPT models fell for Nginx configuration, PHP deserialization and business logic flaws that Claude Opus 5.5 avoided, but all three shared the four that caught everyone.

Leaderboard

Agent	Model	Security interview challenges	A.S.E
Claude Code	Claude Opus 5.5	13% (13 of 98)	40% (46 of 114)
Codex CLI	GPT-6 Astra	26% (26 of 99)	35% (40 of 114)
Codex CLI	GPT-6.1 Sol	27% (27 of 99)	38% (43 of 114)

Share of runs that shipped the flaw, lower is better. Security interview challenges: runs whose new code shipped the challenge's planted flaw. A.S.E: runs whose code passed the project's tests and was still broken by the original CVE exploit, on 38 tasks. The two benchmarks measure different things, so compare agents within a column, not across columns. Each agent ran with no security tool installed. Sources: the secure code review challenges and A.S.E (paper).

With LeoPrevent

LeoPrevent is the security tool we build. It reviews what a coding agent wrote when the agent finishes its turn and tells the agent what it found, so the agent can fix it in the same turn. We ran both benchmarks with and without it, on the same tasks.

With LeoPrevent, every agent shipped fewer exploitable runs on both benchmarks. On A.S.E's real CVEs, Claude Opus 5.5 shipped exploitable code 63% less often, and no task got worse for any agent. On the security interview challenges, both GPT models went from failing 9 challenges to 3.

Part of what is left is a flaw LeoPrevent did catch. On the Nginx challenge it flagged the problem in every run, but it hands that kind of finding to the developer as a suggestion rather than having the agent change the configuration itself. Our runs have no developer to accept it, so those runs count as not prevented; in everyday use, the developer would see the finding.

Share of runs that shipped exploitable code: grey without LeoPrevent, green with it, on the same task-and-run pairs. A.S.E counts all 38 tasks.

Agent and model	A.S.E real CVE tasks	Security interview challenges
Claude Code, Claude Opus 5.5	46 → 17 runs, 63% fewer	13 → 10 runs, 23% fewer
Codex CLI, GPT-6 Astra	40 → 22 runs, 45% fewer	26 → 10 runs, 62% fewer
Codex CLI, GPT-6.1 Sol	43 → 25 runs, 42% fewer	27 → 11 runs, 59% fewer

Exploitable runs without LeoPrevent and with it, out of 114 on A.S.E and 99 on the challenges (98 for Claude Opus 5.5). The runs left on the challenges include the Nginx flaw LeoPrevent flagged as a suggestion for the developer: 3 of the remaining runs for each GPT model and 2 for Claude Opus 5.5. The security interview challenges are our own test set, so that column is a check rather than independent evidence. LeoPrevent works alongside code review, static analysis and testing; it doesn't replace them. Each result page has the full numbers and their limits.

Results

October 2026 · Security interview challenges · Claude Opus 5.5, GPT-6 Astra, GPT-6.1 Sol Four security interview challenges tripped every coding agent we tested, on every run 33 public secure code review challenges, three agents, three runs each. On their own, the agents' new code shipped the planted flaw in 13% to 27% of runs, often by copying insecure code that was already there. October 2026 · A.S.E · Claude Opus 5.5, GPT-6 Astra, GPT-6.1 Sol Asked to avoid vulnerabilities, coding agents still shipped exploitable code in more than a third of real CVE tasks 38 real CVEs in 19 projects, three agents, three runs each. The original exploit still worked in 35% to 40% of runs, mostly code missing a check on where a request comes from, where a path leads or who may use a page.

How we test

Real code, ordinary work. Each task is a change to an existing app: a feature request written the way a developer would ask for it, or, on A.S.E, a function to write back into a real project.
How the agents ran. Claude Code and the Codex CLI, given the request as it is, each in its own isolated container. The GPT models ran at medium effort and Claude Opus 5.5 at its default; on A.S.E, Claude Code used the benchmark's own tool settings.
Three runs per task. Agents give different answers to the same request, so one run per task mostly measures luck.
Exploits where possible. Where the benchmark has its own security tests or an exploit we can run, those decide the verdict. On A.S.E every verdict is the original CVE exploit; on the security interview challenges, 93% are.
Analysis fixed in advance, verdicts published. We write down how we will compare before the runs, say on the result page when we report a subset chosen afterwards, and publish the verdict of every run so the numbers can be recomputed.