Can You Trust an AI Pentest That Needs Multiple Runs?

Last week we released Cascade Iguazu, the second generation of our multi-agent AI pentesting engine. We redesigned how agents are orchestrated, how coverage is tracked, and how findings are verified. On our internal benchmarks, that gave us 5.3× more API coverage, 3.4× more web application coverage, and significantly more critical vulnerabilities.

AI coding is speeding up software development at an insane pace, so the amount of code and application behavior that needs security review is exploding. We expect security testing to become a continuous, autonomous layer that keeps pace with software as it’s generated and modified.

That only works if an AI pentester delivers full, reliable coverage in a single execution. Running the same frontier model over and over, hoping each attempt finds what the last one missed, gets expensive and hard to scale.

So the engineering problem is how to build autonomous pentesting that finds vulnerabilities systematically and returns consistent results, even when two independent executions start with the same target, configuration, and budget. This article covers why that is hard, how to measure it, and the decisions we made building Cascade to address it.

1. Why AI pentesting is unpredictable

Coverage in AI pentesting means systematically exploring an application and finding the vulnerabilities spread across its attack surface. That takes an understanding of business workflows, tests of authorization boundaries, interaction with different user roles, and enough depth on each potential vulnerability to confirm whether it’s exploitable.

An autonomous agent makes hundreds of decisions during a pentest, and each one shapes what it finds next.

Take two AI pentests launched at the same time against the same application, with identical configurations, credentials, and execution budgets. One agent spends most of its time on an interesting injection point. The other explores authorization boundaries and finds several IDOR vulnerabilities. Both run legitimate security tests, yet their exploration strategies lead to different findings.

The larger the application, the harder this is to control. Agents have to keep deciding which surfaces deserve attention, how deeply to investigate a potential vulnerability, and when to move on. Context accumulation, tool execution, and delegation between agents add more variability. What a pentest finds can therefore depend heavily on the path the agent happens to take.

AI-generated code speeds up development, which turns this into a scalability problem. Security teams will have to assess larger, frequently changing applications with limited time and compute. The goal is to make comprehensive vulnerability discovery a dependable result of each execution, whatever individual decisions the agents make along the way.

2. When do you stop running your AI pentester?

A common way to improve AI pentesting coverage is to give a frontier model, like Claude, access to an application, let it find vulnerabilities, then run it again to find what it missed. You repeat until no new vulnerabilities turn up.

Each run might explore different parts of the application, so the total number of findings can climb over time. The trouble is knowing when to stop. A run that finds nothing new doesn’t prove the application has been thoroughly tested. The agent might have revisited the same endpoints, skipped entire business workflows, or abandoned promising leads too early. One more run could surface another critical vulnerability, or it could just repeat work already done.

At scale this gets worse. Every additional run costs tokens, compute, and time, and it can add duplicate findings and false positives that someone has to verify. Without a reliable stopping condition, the cost and duration of an assessment can’t be predicted.

AI-generated code is speeding up development, and security testing has to keep up with an attack surface that is larger and changes constantly. If acceptable coverage takes several attempts, continuous security assessment stalls on that bottleneck.

We want comprehensive vulnerability discovery to happen within a single, bounded execution, through systematic exploration and a deliberate split of testing resources across the application. How do we measure whether an autonomous pentester achieves that?

3. Benchmarking how reliable is a single AI pentest run

We ran a simple experiment where we built a benchmark application with nine seeded vulnerabilities and tested it with Claude Opus 5.5 and with Cascade, running each four times independently against the same target.

The results highlight how unpredictable autonomous pentesting can be, even on a relatively small attack surface.

Even on a small attack surface, the results varied. Claude found between six and eight vulnerabilities per execution, an average detection rate of 75% across the four runs. Its best run missed one vulnerability and its worst missed three. Cascade found all nine in every execution, for 100% detection across all four runs.

Consistency matters as much as raw detection. A security team that ran Claude once against this application could get a report with six, seven, or eight vulnerabilities, with no way to tell which ones were missed or whether another run would find more.

This benchmark was deliberately simple, with only nine known vulnerabilities. Real applications have much larger attack surfaces, complex authorization boundaries, and business workflows that agents have to understand and exploit.

Continuous security testing needs high vulnerability coverage from every execution, without accumulating it over repeated attempts. That takes a pentesting harness engineered to explore applications systematically, allocate testing resources, and verify vulnerabilities.

4. Engineering reliable vulnerability coverage

At Escape, we think coverage should reflect how thoroughly an application has been tested for vulnerabilities. Visiting every endpoint isn’t enough. Coverage also means understanding application behavior, identifying potential attack paths, investigating them deeply enough to establish exploitability, and tracking what remains to be tested.

Cascade’s multi-agent harness is built around that idea. A deterministic scheduler maintains a shared coverage map and governs the testing process. Agents explore the application, record evidence and suspicious behavior, propose potential vulnerabilities, and investigate them. Each surface follows a structured lifecycle with explicit requirements for exploration, security testing, and resolution. The scheduler tracks unfinished work, recovers abandoned tasks, and can start further exploration when more attack paths might remain.

Exploration and exploitation run in separate execution pools. That lets agents dig into a complex vulnerability without using up the resources needed to explore the rest of the application. With controlled task prioritization and execution budgets, promising leads get enough attention while the broader attack surface keeps being tested.

The harness enforces consistency even though individual agents remain nondeterministic. Executions can follow different reasoning paths, but the system keeps tracking progress and steers agents toward comprehensive vulnerability discovery. The benchmark reflects this: Cascade independently found all nine seeded vulnerabilities across four consecutive executions.

AI-generated code is accelerating software development, and we expect security testing to run as a continuous layer alongside it. That calls for autonomous pentesting that delivers reliable coverage within predictable execution budgets. Cascade Iguazu is our engine for that, designed to make systematic vulnerability discovery dependable at scale.