Benchmarks for AI hackers do one thing well: they tell you whether an agent can execute a known exploit against a known scenario. That's a real capability, but it's only half the job. A capture-the-flag challenge gives the agent a single box with one planted bug and a known answer to reach. A real engagement is the opposite: a whole application with nothing to guide you and no signal that you've won, only an objective and the burden of proving what you found. The skill isn't firing the exploit. It's covering the surface, choosing what to chase, and standing behind each finding with evidence. That's the half we couldn't find a benchmark for. Before shipping Shax, we'd designed a ladder of evaluation gates. CTF challenges first, realistic full applications second, live bug bounty last. Gates one and three we could source. Gate two didn't exist, so we built it: five full, functioning applications that behave like real targets, where an agent is judged the way a human pentester is: on what it can find and prove across the whole app. What began as Shax's proving ground turned out to be worth sharing.

How it works

Nua-Bench is five complete web applications: a blog, an e-commerce store, a movie-booking site, a food-ordering service, and a project-management tool. Into these we planted 100 vulnerabilities spanning 10 classes (SQLi, XSS, SSTI, IDOR, XXE, SSRF, path traversal, open redirect, info disclosure, and known-CVE components), with each class appearing twice per app. A class counts as solved only when the agent finds both instances. The apps are flag-less by design: real targets don't hand you a flag that says you've won, so neither do these. To get credit, an agent does what a pentester does: find the vulnerability, exploit it, and produce a working proof-of-concept, which we grade against a reference exploitation we wrote for that bug. The apps are built from scratch and unseen by any model at evaluation time. We're releasing them with a canary string asking developers to keep them out of training data.

Not all vulnerabilities are created equal

If there's one thing Nua-Bench is built to measure, it's this: the same vulnerability can be trivial or nearly impossible depending on how it's buried, and only a benchmark that varies difficulty inside each class can tell the two apart. We rated the exploit complexity of every SQL injection instance by what it takes to reproduce the bug (access required, payload sophistication, filter bypass, blind vs. visible feedback, and chaining), independently of how any model performed. Ranked by that difficulty, the models' collective SQLi score more than halves from the easiest injection to the hardest

To see what that halving actually means, compare the two extremes. In the low-complexity case, the injection is a textbook login bypass: type admin'-- and you're in, and the response confirms it immediately. In the high-complexity case, it lands in an ORDER BY clause where the usual UNION technique fails, forcing blind, time-based extraction: inferring the data one true/false question at a time, with nothing in the response to confirm progress.

Surface is a second pressure

Difficulty isn't only about the vulnerabilities themselves; some of it is about how much application surrounds them. The largest app in the suite (40 endpoints) drew the fewest findings anywhere, 53 across the six models, while the smallest drew 82. The more an agent has to enumerate and hold in context, the more slips past it.

Try it, break it, extend it

Nua-Bench is open source and public today. Clone it, run your own agents against it, and see where they land:
Go to Nua Bench