Inside Cascade 2 Iguazu: how we benchmark our AI pentesting engine
How Cascade 2 Iguazu's scheduler and Coverage agent work, how we benchmark them, and what two rounds of internal benchmarks show.
On October 26, we released Cascade Iguazu, the second generation of Cascade, our multi-agent AI pentesting engine. The launch post covers what changes for you: a scheduler that owns the plan, a Coverage agent, and reasoning logs.
This post is the engineering side: how it works, how we measure it, and what the numbers say. Since we introduced Cascade, we've watched it run hundreds of times with our partners. We kept working on the three areas that matter: wider coverage of your application, deeper understanding of how it works, and a lower rate of false alarms.
V1 vs V2 on our internal benchmarks. Round 1 (September): two open-source apps and one purpose-built app, one white-box and one black-box run per target. Round 2 (October): two open-source apps with planted vulnerabilities. Details below.
We stopped hoping the agents would look everywhere
As you might remember from our previous blogpost, one of the main reasons AI pentesting systems underperform is a rather trivial one: they fail to exhaustively explore your application. This comes from a real tension between the need to start attacking the surfaces you already found, and the need to keep looking for new attack vectors, the classic exploration-exploitation dilemma.
Solving it completely would require major scientific advances. Iguazu tackles it through its architecture instead. Redesigning our orchestration system around surfaces makes it possible to leverage knowledge of the application surface to its fullest extent. A scheduler owns the plan: which surfaces exist, which need more testing, which agent works on each one, and when a surface is actually covered. It protects breadth while encouraging depth, and spawns specialized agents only where a surface warrants it. The launch post explains how this makes runs repeatable.
Alongside it, the Coverage agent has one job and no exploitation tools. It maps the attack surface, compares it with what has already been tested, and pushes the gaps back to the scheduler. In practice, this means that in white-box mode we ensure every file in your application has been examined, while in black-box mode we leverage sitemaps and API specs to find surfaces that even the best explorer agents would have missed.
This improved coverage system improves black-box API coverage from 60% to 90% (Round 1), while white-box mode goes from 80% of files examined to a guaranteed 100%.
The harness is the product, not the model
Recently, it has become apparent that even non-flagship models can saturate existing benchmarks simply by being plugged into a high-quality, security-oriented harness. That is mostly due to the importance agentic tool-use and security topics have gained in model post-training. It should also be kept in mind that as soon as a benchmark is made public, it becomes training data for the next wave of models and therefore much more easily saturated. That's part of why we rotate the underlying applications between benchmarking rounds: comparing Iguazu against a target the model may have already seen would flatter the new version instead of testing it.
To give state-of-the-art models room to show their strength, and to properly assess the impact of harness improvements, it became necessary to develop our own internal benchmarking systems. This is no easy feat.
As recent events like the OpenAI × Hugging Face incident highlighted, recent models are getting so good at their job that they start hacking not only the target webapps they are evaluated on, but the evaluation system itself. The sandbox is the only thing ensuring your results are not contaminated, and that you're not putting others at risk; it should never be an afterthought.
Upon this foundational safety layer, we then needed to build the actual targets. We made sure they were as varied as possible, so that they could cover all the situations Escape AI pentesting meets in production:
- Different scales of codebase. Some targets are realistic, enterprise-scale applications with needle-in-a-haystack bugs; others are more reasonably sized and focus the evaluation lens on execution depth rather than coverage.
- Open-source and purpose-built. Forks of existing open-source applications (WebApps and APIs) alongside applications we built ourselves, so no contamination from the LLM training set is possible.
- Injected vulnerabilities and real CVEs. Injections span the difficulty range, from simple reflected XSS we expect most agents to find to multi-step exploits requiring both code and business-logic understanding. Others were never injected at all: they are CVEs that surfaced between our forking and our benchmarking.
- Kept evolving. Benchmarks have to stay alive, moving with each generation of models and with what the security community learns. It's impossible to predict what exploit modes will appear, so we keep monitoring and rising to the occasion.
Under this design philosophy, we ended up with benchmarks that even the most competent models were not able to fully saturate, while still showing signal for lower competency models and allowing measurable improvements not only in what the agents find, but in how reliably they find it and how few false alarms they raise.
Since V2 brings improvement to both the agents themselves, and a fully redesigned coverage system, we needed to quantify the improvements separately and therefore performed two round of experiments. Round 1, focused on quality of execution, used three targets: two open-source apps and one purpose-built app, and relied on the same exploration system as V1; round 1 evaluates a purposely inferior version of Cascade V2. Round 2, focused on coverage and detection, used only the open-source targets that posed a significant exploration challenge and allowed V2 to leverage its new coverage system to its fullest extent.
Overall, 0n the two open-source apps tested in both rounds, critical findings rose from 18 to 45 (2.5×). The new agents accounted for 7 of the 27 added findings and the new coverage system for the other 20.
Round 1: more critical findings, less noise
In Round 1, leveraging only the improvements to agents but not our new coverage system, V2 found 41% more critical findings than V1 (27 → 38) and raised 3.5× less noise (66 → 19). The share of valid findings among everything reported rose from 47% to 84%.
Black-box Cascade delivers serious depth on its own, which is what most customer environments need it to do. That depth just grew: Iguazu now finds in black-box mode what V1 needed white-box access to find, closing a 30% gap. White-box still goes further when you can give it the access, but black-box is no longer the compromise it used to be.
Round 2: 5.3× more API coverage, 1.8× more critical findings
For this second round of experiments, we focused only on what the new coverage system brings to the table: how much of your application gets explored, and how many real vulnerabilities come out of it. This round of experiment shows Cascade V2 operating at its full potential.
What differs between the two versions of Cascade we compare in this round:
- The scheduler now owns the plan, with block and unblock conditions so a surface closed too early gets reopened.
- Reasoning logs record every hand-off between agents and every verdict.
- A deduplication module merges repeated reports of the same finding.
Three layers between an agent's hunch and your inbox
As you might have noticed in your own interactions with them, LLMs tend to be overdramatic when security is involved. What that means for you is that any single imperfect configuration, even one that isn't really exploitable, gets treated as a high-priority signal that deserves your attention.
That is not sustainable. AI-alert fatigue will settle in, making you more likely to miss the actual exploits that need remediation. That's why we have put so much work into false-positive reduction since our initial release, which led us to our current three-layer architecture:
Each of these layers is pulling its weight:
- The reproducer confirms that what the agent observed was really a consequence of its actions as it understood them, not just a coincidence.
- The adversarial reviewer makes sure that what is reported is an actual vulnerability, not a false alarm or an already known finding.
- The business-logic reviewer leverages the business-knowledge database built during the engagement to retrospectively filter findings that, while being real security concerns, are necessary for the intended behavior of the app.
- Finally, deduplication merges repeated reports of the same vulnerability. Different agents often reach the same bug through different paths and describe it in different words.
Round 1 shows what the review layers do: noise fell 3.5×. Every verdict is recorded in the reasoning logs, so a dismissed finding comes with the reason it was dismissed.
We evaluated deduplication on its own, on a set of 501 issues of which 127 were duplicates:
Before deduplication | After | |
|---|---|---|
Issues | 501 | 379 |
Duplicates among them | 127 (25%) | 25 (7%) |
The module merged 80% of duplicates (102 of 127). It also merged 5% of distinct findings by mistake (20 of 374). Those are not deleted: they stay in the Ignored view.
A pentest that keeps up with your releases
The new version of Escape's AI pentesting covers more of your application, goes deeper into it, and reports less of what you don't need to read. The three improvements reinforce each other: coverage gives the agents somewhere to go, the harness lets them prove what they find, and the rejection layers keep the proof burden on us rather than on you.
Next: sharpening that continuous loop and focusing agents on what changed since the last run, teaching them to chain exploits so we can unearth even more complex vulnerabilities.
Get started
None of this matters until it's pointed at your own application. Book a session with our team and we'll run Cascade against an environment that looks like yours, white-box or black-box, so you see real, exploitable findings instead of a demo script.
Learn more about Cascade:
Yacine Souam
Karim Rustom
Alexandra Charikova