Blog · BaxBench
One regex away from a dead server
GPT-6 Astra wrote a neat file-search API. Path traversal blocked, symlinks skipped. Then it ran any regex a stranger sent, with no time limit.
Send that API a regex bomb, a pattern built to make the engine backtrack for ages, and the server is stuck until it falls over. BaxBench, a public security benchmark for AI-written backends, does exactly that and gives the server two seconds. Astra’s didn’t make it.
pattern = re.compile(regex)
# ...
if pattern.search(content) is not None:
matches.append(...)Every single time
Every regex-search backend Astra built that BaxBench could test went down the same way: 12 out of 12, across Flask, FastAPI, Django and Go.
Across all 224 tasks, BaxBench’s exploits got into 21% of Astra’s backends. Claude Opus 5.5 did better at 13%, which is still about one in eight.
The repeat offenders were cross-site scripting, requests that can eat unlimited memory or CPU, and credentials handled carelessly.
One second is all it took
LeoPrevent checks an agent’s work before the agent calls it done. In the FastAPI run with LeoPrevent, it flagged the unbounded regex. Astra moved the matching into a throwaway process and gave it one second.
MATCH_TIMEOUT_SECONDS = 1.0
# ...
process.start()
process.join(MATCH_TIMEOUT_SECONDS)
if process.is_alive():
raise HTTPException(400, "Regex exceeded the matching time limit.")Three FastAPI runs without LeoPrevent: three hackable servers. Three with it: none.
Across the whole benchmark, LeoPrevent cut Astra’s hackable backends from 69 to 57, 17% fewer on the same tasks and runs. Opus went from 55 to 47.
BaxBench comes from researchers at ETH Zurich and LogicStar AI. For this run we set LeoPrevent to fix problems automatically, including rules that normally come to you as suggestions. The method and every verdict are on the results page.