Benchmarks · Claude Fable 5.2
Claude Fable 5.2: new model, same security holes
We tested Anthropic’s newest model against Claude Opus 5.5 on two security benchmarks: 33 apps that already have a security hole and 64 real CVEs. Each task ran three times on its own and three times with LeoPrevent.
Better on one benchmark, worse on the other
Real CVEs. A.S.E takes real open-source projects, removes the function behind a published vulnerability and asks the agent to write it back. Then the original exploit runs against the result. It broke Opus 5.5’s code in 38% of runs and Fable 5.2’s in 19%. That halves the rate, but one run in five was still exploitable.
Flawed apps. Each of our 33 challenge apps already has a security hole, as most real codebases do, and the agent is asked for a feature in the same part of the code. Fable 5.2 finished with the hole still there in 93 of 99 runs, Opus 5.5 in 83. The newer model did slightly worse.
Neither model was asked to fix the hole, and mostly they didn’t. Existing security debt is rarely part of the task, so agents leave it alone.
With LeoPrevent
LeoPrevent reviews the agent’s work before it calls the task done, including the code that was already there, and tells the agent what it found. It doesn’t block anything: the agent decides what to change and makes the fix in the same turn.
Fable 5.2 left the hole in 29 apps instead of 93, and its exploitable code on real CVEs fell from 19% to 9% of runs. Opus 5.5 went from 83 apps to 23, and from 38% to 13%.
Moving from Opus 5.5 to Fable 5.2 helped on one benchmark and hurt on the other. LeoPrevent at least halved the vulnerable results for both models on both benchmarks, and with it the two models ended up close to each other: 29 and 23 apps, 9% and 13% of CVE runs.
Try LeoPrevent with your agent
Check our work
Download the verdict for every run, flawed apps and real CVEs, and the analysis output. Every number on this page can be recomputed from them.
Full results and method
Full results
| Runs left vulnerable | Fable 5.2 | Fable 5.2 + LeoPrevent | Opus 5.5 | Opus 5.5 + LeoPrevent |
|---|---|---|---|---|
| Flawed apps: hole still there at the end | 93 of 99 | 29 of 99 | 83 of 99 | 23 of 99 |
| Flawed apps: hole in the agent’s own new code | 16 of 99 | 5 of 99 | 13 of 99 | 5 of 99 |
| Real CVEs: exploit still works | 37 of 190 | 18 of 191 | 73 of 192 | 24 of 192 |
Lower is better. A.S.E runs whose build or functional test failed have no verdict: 2 for Fable 5.2 on its own and 1 with LeoPrevent.
Compared task by task (a task counts when at least two of its three runs agree), LeoPrevent made Fable 5.2 better on 21 flawed apps and worse on none (p = 0.000001), and better on 6 CVE tasks and worse on none (p = 0.03). On its own, Fable 5.2 beat Opus 5.5 on 12 CVE tasks and lost on 2 (p = 0.01). On the flawed apps it did worse run by run (10 runs against 0, p = 0.002), though task by task the difference is too small to call. With LeoPrevent we can’t tell the two models apart on either benchmark.
How we ran it
Flawed apps. 33 public secure code review challenges: small web and system apps in several languages, each with one deliberate security flaw, plus a feature request where that kind of flaw is easy to repeat. A run counts as vulnerable when the finished app still has that kind of flaw anywhere: in the agent’s new code, or in old code it left in place. We fixed this whole-app measure as the main one before the run. The new code was tested with a working exploit wherever the app could be run (350 of 396 runs) and by code review otherwise; old code was checked against a fixed list of the flaws already in each app. Counting only the agent’s new code, as on our October page, gives the second row above.
Real CVEs. 64 tasks from the public A.S.E benchmark: 25 web projects in PHP, Java and Python and 39 C libraries and tools. Each removes the function behind a published vulnerability. A run counts as exploitable when the project’s functional test passes and the benchmark’s exploit still works.
Agents. Claude Fable 5.2 and Claude Opus 5.5 in Claude Code 2.1.321 and 2.1.322 at default settings, each run in its own container, on 11 and 12 November 2026. Every task ran three times on its own and three times with LeoPrevent: 780 runs in all.