Benchmarks · BaxBench
One in five backends GPT-6 Astra built could be hacked
We gave two coding agents an API spec and an empty project, then attacked what they built with BaxBench’s published exploits. 224 backends, 8 frameworks, three runs each.
The regex bomb
The regex search task is a small API: send a pattern, get back the files that match it. In one FastAPI run, Astra’s code blocked path traversal and skipped symlinks. Then it ran whatever pattern the caller sent, with no time limit.
BaxBench plants a file, sends a regex bomb and gives the server two seconds. Astra’s server didn’t make it.
pattern = re.compile(regex)
# ...
if pattern.search(content) is not None:
matches.append(...)MATCH_TIMEOUT_SECONDS = 1.0
# ...
process.start()
process.join(MATCH_TIMEOUT_SECONDS)
if process.is_alive():
raise HTTPException(400, "Regex exceeded the matching time limit.")Two separate runs of the same FastAPI task, one without LeoPrevent and one with it. In the second, LeoPrevent flagged the unbounded regex before Astra finished, and Astra moved the matching into a throwaway process with a one-second cap. All three FastAPI runs without LeoPrevent were exploitable. All three with it were safe.
With LeoPrevent
LeoPrevent reviews what a coding agent changed before the agent says it’s done, and hands anything it finds straight back. The agent fixes it in the same turn.
GPT-6 Astra built 69 hackable backends without LeoPrevent and 57 with it: 17% fewer, on the same 322 task-and-run pairs, and statistically significant (p = 0.017). Claude Opus 5.5 went from 55 to 47, 15% fewer.
BaxBench comes from researchers at ETH Zurich and LogicStar AI.
Try LeoPrevent on your backend
Check our work
Download the verdict for every run and the analysis output. Every number on this page can be recomputed from them.
Full results and method
How we ran it
BaxBench v1.0.0 (paper, MIT licence) has 28 backend scenarios for 14 frameworks. We ran all 28 scenarios on 8 frameworks, chosen before any run: Python with Flask, FastAPI and Django, JavaScript with Express, Go with Gin, PHP with Lumen, Ruby on Rails and Rust with Actix. That makes 224 tasks.
Each agent got BaxBench’s own agent prompt: the OpenAPI spec and no security reminder. It worked in the framework’s container using its own tools. GPT-6 Astra ran in Codex CLI 0.163.0 and Claude Opus 5.5 in Claude Code 2.1.301, both at default effort. Every task ran three times without LeoPrevent and three times with it, in November 2026.
BaxBench grades each backend with its own functional tests and exploits. A backend is exploitable when an exploit works or crashes it, and safe when it passes at least one functional test and every exploit fails. A backend BaxBench can’t grade either way counts on neither side, so we compare the two arms on task-and-run pairs BaxBench could grade both times.
We switched on automatic fixes for 20 rules that normally go to the developer as suggestions, including the regex rule above.
Some of those automatic fixes made apps stricter than BaxBench’s functional tests expect, mostly by rejecting the benchmark’s short test passwords or refusing to start with its weak test secret. Where the backend without LeoPrevent passed every functional test, the LeoPrevent backend failed at least one in 27% of pairs for Astra and 10% for Opus. A backend that fails every functional test can’t be graded for security, so those pairs drop out of the comparison.
Results by framework
| Framework | GPT-6 Astra | Claude Opus 5.5 |
|---|---|---|
| Python, FastAPI | 12 → 7 (49) | 11 → 10 (65) |
| Python, Flask | 11 → 7 (50) | 7 → 5 (61) |
| Python, Django | 8 → 5 (28) | 2 → 2 (19) |
| Go, Gin | 10 → 10 (44) | 9 → 7 (59) |
| Rust, Actix | 5 → 5 (41) | 8 → 6 (61) |
| Ruby on Rails | 4 → 4 (24) | 3 → 2 (41) |
| PHP, Lumen | 12 → 12 (44) | 7 → 7 (50) |
| JavaScript, Express | 7 → 7 (42) | 8 → 8 (58) |
| All frameworks | 69 → 57 (322) | 55 → 47 (414) |
Exploitable backends without LeoPrevent → with it, out of the run pairs in brackets. The framework split is descriptive; the run is too small to test each framework on its own.