Blog · BaxBench

One regex away from a dead server

GPT-6 Astra wrote a neat file-search API. Path traversal blocked, symlinks skipped. Then it ran any regex a stranger sent, with no time limit.

LeoTrace10 November 2026

Send that API a regex bomb, a pattern built to make the engine backtrack for ages, and the server is stuck until it falls over. BaxBench, a public security benchmark for AI-written backends, does exactly that and gives the server two seconds. Astra’s didn’t make it.

GPT-6 Astra, without LeoPrevent
pattern = re.compile(regex)
# ...
if pattern.search(content) is not None:
    matches.append(...)

Every single time

Every regex-search backend Astra built that BaxBench could test went down the same way: 12 out of 12, across Flask, FastAPI, Django and Go.

Across all 224 tasks, BaxBench’s exploits got into 21% of Astra’s backends. Claude Opus 5.5 did better at 13%, which is still about one in eight.

Share of backends a published exploit broke, without LeoPrevent (grey) and with it (green), on the same task-and-run pairs. GPT-6 Astra: 21% without, 18% with. Claude Opus 5.5: 13% without, 11% with. GPT-6 ASTRA · CODEX CLI Without LeoPrevent 21% With LeoPrevent 18% CLAUDE OPUS 5.5 · CLAUDE CODE Without LeoPrevent 13% With LeoPrevent 11% 0% 5% 10% 15% 20% 25%
Share of backends a published exploit broke, lower is better. Same tasks and runs with and without LeoPrevent. Full results and method.

The repeat offenders were cross-site scripting, requests that can eat unlimited memory or CPU, and credentials handled carelessly.

One second is all it took

LeoPrevent checks an agent’s work before the agent calls it done. In the FastAPI run with LeoPrevent, it flagged the unbounded regex. Astra moved the matching into a throwaway process and gave it one second.

GPT-6 Astra, with LeoPrevent
MATCH_TIMEOUT_SECONDS = 1.0
# ...
process.start()
process.join(MATCH_TIMEOUT_SECONDS)
if process.is_alive():
    raise HTTPException(400, "Regex exceeded the matching time limit.")

Three FastAPI runs without LeoPrevent: three hackable servers. Three with it: none.

Across the whole benchmark, LeoPrevent cut Astra’s hackable backends from 69 to 57, 17% fewer on the same tasks and runs. Opus went from 55 to 47.

BaxBench comes from researchers at ETH Zurich and LogicStar AI. For this run we set LeoPrevent to fix problems automatically, including rules that normally come to you as suggestions. The method and every verdict are on the results page.

Try LeoPrevent on your backend