Jev and the Future of AI Pentesting: Why the Harness Matters More Than Ever
Were we wrong to use LLMs & Agents for security?
Security professionals are fully embracing agents for security workflows, from finding vulnerabilities in code to investigating and triaging security issues. And for good reason: many of these tasks require complex reasoning and the ability to understand how applications work.
But we're also increasingly seeing security use cases where LLMs might be overkill.
Take triaging, for example. We ask an LLM, "Is this a false positive?", give it all the context it needs, get an answer, extract the decision in JSON format, and finally use it to trigger the next step in our workflow.
What if we had a faster, cheaper way to do that? What if we had a model that was, by its very nature, a classifier, designed to make decisions rather than generate responses?
That's exactly the idea behind Jev, TypeSafe AI's System One model. And with TypeSafe AI's recent $870M Series A, it's an approach that's getting a lot of attention.
For security, the potential is particularly interesting. We're constantly making decisions across thousands of lines of code, API endpoints, and security findings. If we could make those decisions faster and at a fraction of the cost, it could change the way we approach AI-powered security.
But there's a catch. Security isn't always about answering yes or no. Finding a vulnerability might require understanding how different parts of an application interact. Determining whether a finding is a false positive might require understanding the application's business logic. And making decisions faster doesn't necessarily mean making better decisions.
So, how far can we actually go with Jev in AI pentesting?
We decided to put it to the test, starting with vulnerability detection and false-positive filtering, and exploring how application context affects its security decisions.
But our experiments also raise a bigger question: how much of an AI model's effectiveness comes from the model itself, and how much comes from the harness surrounding it?
In this article, we'll share our experiments, what worked, what didn't, and why combining specialized models like Jev with the right context, tools, and orchestration could make AI pentesting significantly more efficient.
1. So, what makes Jev different?
The first difference is pretty simple: Jev doesn't generate text, it makes decisions.
Instead of asking an LLM to generate an answer that we then need to interpret, Jev is designed to directly classify, score, or choose between options. It's a classifier by nature, not a generative model that we're trying to make act like one.
But what's particularly interesting is confidence. Jev doesn't just return a yes or no, it can also provide confidence scores for its decisions.
Imagine using that in an AI pentesting workflow. Instead of treating every finding the same way, we could automatically handle certain high-confidence classifications, send uncertain ones to more powerful LLMs, and escalate others for deeper investigation.
Of course, a high confidence score doesn't necessarily mean the model is right, but it gives us an additional signal to work with.
Then there's speed, and more importantly, cost.
And cost is becoming a pretty big topic in AI. Coinbase, for example, recently reported cutting its AI spending nearly in half while its token usage continued to grow. How? By using cheaper open-weight models, smarter routing between models, and better caching.
Companies are already realizing that they don't necessarily need the most powerful LLM for every task. But even when switching to smaller, cheaper models, they're still using LLMs.
But what if the biggest optimization is questioning whether the task requires an LLM at all?
This becomes particularly interesting in security, where teams have limited budgets but increasingly large attack surfaces to cover. When you're analyzing thousands of API endpoints, vulnerabilities, or scanner findings, the cost of every decision matters.
And if we can make those decisions significantly cheaper and faster, without sacrificing accuracy, we might be able to rethink how AI pentesting solutions operate.
But before imagining what Jev could do for AI pentesting at scale, we wanted to see how it actually performs on security tasks.
2. Putting Jev to the test
We've seen why Jev could be interesting for AI pentesting. But being fast and cheap is one thing. Being able to make the right security decisions is another.
So we decided to put Jev to the test, starting with one of the most obvious use cases: finding vulnerabilities in code.
Can Jev actually find vulnerabilities in code?
For our first experiment, we gave Jev 20 code snippets: 10 containing real vulnerabilities and 10 benign ones. We covered different vulnerability types, including XSS, SSRF, IDOR, and others, and simply asked Jev to determine whether each snippet was vulnerable.
But there's an important detail about this experiment: we made the task relatively easy.
Instead of giving Jev an entire codebase, or even large portions of an application, we provided carefully selected snippets of around 20 lines, with the relevant code already isolated.
In other words, Jev didn't need to explore a repository, trace different functions, or figure out where a vulnerability might be hiding, as a real security agent would. It only needed to classify the code we gave it.
Jev successfully detected 9 out of 10 vulnerabilities, with a total inference cost of less than $0.001 for all 20 snippets.
That's pretty impressive, especially considering the cost. But looking at the benign snippets tells a different story: Jev incorrectly flagged 5 out of 10 as vulnerable.
That's a 50% false-positive rate, on a task we had already simplified considerably.
And that's where things get complicated. Identifying potentially vulnerable code is useful, but if half of the safe code we analyze gets flagged, we're potentially creating a lot more work for security teams.
Of course, this was a small experiment, not a benchmark of Jev's ability to discover vulnerabilities across real applications.
But there's something interesting about the way we designed it.
By selecting the relevant code beforehand, we had already done part of the work that a real AI pentesting system needs to do.
And that raises an important question: what happens when we change the information available to the model?
What about false positives and business context?
Finding vulnerabilities is one thing. But another big challenge in security is figuring out which findings actually matter.
And that's where business context becomes particularly interesting.
Take an authorization vulnerability, for example. An API endpoint might look vulnerable when analyzing its code alone, but what if access is already restricted somewhere else in the application? What if the application's business rules explain why a particular behavior is actually expected?
Without that information, even a powerful model might struggle to make the right decision.
So we wanted to see whether giving Jev more information about the application would change how it evaluated security findings.
We ran an experiment where we asked Jev to evaluate a set of known false-positive findings, first on their own, and then with access to the application's business wiki.
With business context, Jev identified enough known false positives to reduce the number of those findings left for review by 20.7%, under our tested classification threshold.
In other words, providing information about how the application was supposed to work helped Jev recognize findings that didn't actually represent vulnerabilities.
Of course, this doesn't mean a production security system could safely dismiss 20.7% of its findings. Our experiment focused on known false positives, and we'd still need to verify that legitimate vulnerabilities wouldn't be incorrectly filtered out.
But it does raise an interesting point: what if improving a model's security decisions isn't just about making the model smarter, but also about giving it the right information?
And that's particularly relevant when thinking about how smaller, specialized models could contribute to AI pentesting.
So, what did we learn?
Our experiments showed some interesting potential for Jev in application security.
It managed to identify most vulnerabilities in our simplified code benchmark, at an extremely low cost. And providing business context helped it better recognize false-positive findings.
But there's a common thread between these two experiments.
In the first, we carefully selected the relevant pieces of code before giving them to Jev. In the second, we provided additional information about the application's business logic. In both cases, what surrounded the model mattered.
And that's an important distinction when thinking about AI pentesting. A model doesn't automatically know which code to inspect, which application behavior is expected, or how to verify that a vulnerability is actually exploitable.
Those capabilities need to come from somewhere.
Which brings us to something that's becoming increasingly important in AI security: the harness.
3. Why the harness matters as much as the model
We've seen that Jev can make security decisions incredibly cheaply, and that providing the right context can improve its results.
But there's a bigger challenge when moving from individual security decisions to testing an entire application.
Even the most powerful model can't realistically handle an entire attack surface on its own.
Think about what an AI pentesting agent needs to deal with: thousands of API endpoints, different authentication mechanisms, user roles, business logic, and potential vulnerabilities spread across different parts of an application.
As the agent investigates, it keeps accumulating information. API responses, code snippets, previous tests, failed attempts, discovered vulnerabilities, and so on.
And eventually, that becomes a problem.
Context windows are limited, processing all that information gets increasingly expensive, and important details can get lost along the way.
One solution might be to delegate some of the work to subagents. And while that certainly helps, it doesn't automatically solve the problem. If the main agent still needs to keep track of everything happening across the application, we're simply moving the context management problem somewhere else.
So the challenge is making sure AI agents can effectively investigate an entire attack surface while maintaining the context and coverage needed to find real vulnerabilities.
That's where the harness comes in
A harness is essentially everything surrounding the model that allows it to perform a task effectively.
And in AI pentesting, there's quite a lot that needs to happen beyond reasoning.
First, there's coverage. An AI pentesting solution needs to systematically explore the application's attack surface, ensuring that endpoints, functionality, and potential attack paths are properly investigated.
Then there's context management. A good harness manages the information collected during an assessment and makes sure each model has access to the relevant context when it needs it.
There's also security tooling, allowing agents to interact with applications, execute tests, and gather evidence.
And finally, orchestration and validation, ensuring investigations progress effectively and that potential vulnerabilities are actually verified rather than simply predicted.
All of these capabilities contribute to the effectiveness of an AI pentesting solution.
A well-designed harness enables models to operate at scale, systematically cover an application's attack surface, and turn their capabilities into real, validated security findings.
And that's something that becomes particularly interesting when we start looking at smaller, specialized models.
What does a strong harness mean for smaller models?
When comparing AI systems, we tend to focus on the models themselves. Which one has the best reasoning capabilities? Which one performs better on benchmarks? Which one is the most powerful?
But model performance in isolation doesn't tell us everything about how effective a security assessment will be.
Imagine a frontier model with excellent reasoning capabilities, but limited application context, poor attack surface coverage, and no reliable way to validate its findings.
Now imagine a smaller, specialized model with a well-designed harness that provides the right context, relevant security tools, and systematic coverage.
For certain security tasks, the second system could perform better, despite using a less capable model.
Of course, we haven't directly benchmarked these two approaches, and the reasoning capabilities of frontier models remain essential for complex security investigations.
What matters is how effectively the harness makes use of each model's capabilities.
And that's where Jev becomes particularly interesting.
Our experiments showed that Jev can make certain security decisions at an extremely low cost, and that providing application context can improve its classification results.
With a strong harness providing the relevant information, specialized models like Jev could handle certain high-volume security decisions, while more powerful reasoning models focus on complex investigations.
And the benefits could extend beyond reducing inference costs.
Cheaper decisions create opportunities to test more of an application's attack surface within the same budget.
They could also free up resources to investigate more potential attack paths, run assessments more frequently, and spend more time validating complex vulnerabilities.
Of course, these benefits would need to be measured across complete AI pentesting assessments. A cheap classification only provides value when it contributes to finding real vulnerabilities without compromising coverage or accuracy.
After all, for security teams, the most meaningful metrics are how much of their application gets tested, how many real vulnerabilities are discovered, and how much it costs to find and validate them.
The effectiveness of AI pentesting is measured through security coverage, validated findings, and the resources required to achieve them.
Bringing it all together
Our experiments with Jev demonstrated the potential of specialized decision models for application security, particularly when supported by relevant application context.
And the importance of harness engineering is already clear in AI pentesting today.
At Escape, this is how we approach AI pentesting with Cascade: combining autonomous security agents, application context, deep reasoning, and security testing capabilities to explore applications and validate real vulnerabilities.
Specialized models like Jev are particularly interesting in this environment, as they offer opportunities to improve the efficiency of individual security decisions within a broader AI pentesting solution.
A well-designed harness is essential to effective AI pentesting. It enables models to operate at scale, maintain systematic coverage, and translate their capabilities into real security outcomes.
Want to see how AI pentesting can improve your security coverage and help you find real vulnerabilities?
Book a demo of Escape's AI Pentesting →
💡Discover more of our security research and opinion pieces on the hottest topics in AppSec now