Gating a pentest agent's calls before they run: what a scope judge needs to see
A July 2026 benchmark shows a cheap LLM judge can catch out-of-scope tool calls from offensive-security agents — but only if it sees the user's request, not a static policy alone.
What is this?
When an autonomous agent does offensive security work — a bug-bounty scan, a pentest, a red-team task — a single tool call aimed at the wrong host can breach a client’s engagement boundary, disrupt production, or void a finding. A researcher’s agent that discovers a SQL injection on an out-of-scope subdomain has, in the eyes of the program, at best found nothing and at worst committed a crime.
A paper posted to arXiv on 8 July 2026, ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents (2607.07774), by Shane Caldwell, Max Harley, Ads Dawson, Michael Kouremetis, Vincent Abruzzo and Will Pearce of dreadnode, studies whether a second, cheaper model can watch such an agent and stop out-of-scope calls before they execute. It is a defensive study: a benchmark and a set of monitoring recipes, not an attack.
How it works
The setup is pre-execution gating, an instance of trusted monitoring. A strong, swappable agent proposes a tool call; a cheap, fixed judge inspects that call and either accepts it (it runs) or rejects it (the agent is asked to revise, or a human is escalated). The judge runs on every step, so it must be inexpensive.
The hard part is that scope is not a fixed rule. As the authors put it, the boundary “lives in the request”: the same command — say, a scan against a given domain — is in bounds for one engagement and a violation in the next, depending on the target and context. A global policy written in advance cannot enumerate the per-task scope of a request it has never seen.
To measure this, the team built ScopeJudge: 100 agent trajectories drawn from a suite of 30 environments engineered to tempt agents out of scope, labeled call-by-call by professional penetration testers. That yields 4,897 tool calls, of which 7.7% are scope violations. Inter-grader agreement is substantial (Fleiss κ = 0.64), setting a human expert reference of F1 = 0.78. They then tested eight judge models under five transcript strategies — how much context the judge sees, from the static policy alone up to the full raw transcript — and charted the cost-versus-accuracy frontier.
The headline result is a negative one that matters: a judge shown only the static policy, blind to the user’s request, has recall that collapses to near zero. It cannot catch violations because it does not know what the engagement authorized. Request-conditioned monitoring is necessary. Among full-context judges, the strongest were open-weight — GLM-5.2 reached F1 = 0.66, beating the best proprietary judge (0.60) at roughly a third of the per-call cost.
Why it matters
Two common controls fall short here. Sandboxing assumes you can isolate the agent, but in security work the systems under test are often black-box and client-owned, so you cannot simply run in a disposable copy. Human-in-the-loop review suffers prompt fatigue: reviewers click “approve” on warning after warning until the oversight is theatre — the same click-through behavior long documented for browser security warnings. An automated judge does not tire, which is what makes scalable per-call gating attractive.
But the results temper the optimism. Even the best judge (0.66) sits below the human agreement bar (0.78), so this is oversight that reduces risk rather than eliminates it. The paper also flags a real tension: giving the judge more context — the agent’s reasoning, tool outputs — improves accuracy but adds attack surface, because a misaligned agent or an outside attacker could poison that context to get an out-of-scope call waved through. More visibility is not free.
Defenses
The practical guidance for anyone running autonomous security agents:
- Condition the monitor on the request, not just a policy. A fixed constitution or allowlist cannot capture per-engagement scope. Feed the judge the user’s stated objective and constraints so it can reason about what this task authorized.
- Gate before execution, not after. Post-hoc auditing catches violations once the harm is done. Judge each call in the loop and block, revise, or escalate before it runs.
- Pick an operating point deliberately. A missed violation usually costs more than a spurious rejection, so for high-stakes engagements tune the judge for recall (fewer misses, more false alarms) rather than raw F1.
- Treat judge context as attack surface. Reasoning traces and tool outputs improve judgment but can be poisoned; isolate and sanitize what the judge ingests, and don’t assume a smarter judge is a safer one.
- Keep humans for escalation, not every call. Reserve human review for the calls the judge flags, which keeps oversight meaningful instead of drowning reviewers in approvals.
Status
| Item | Detail |
|---|---|
| Finding | A request-conditioned LLM judge can gate out-of-scope tool calls from offensive-security agents before they execute |
| Source | ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents, Caldwell et al., dreadnode, arXiv, 8 July 2026 |
| Benchmark | 4,897 tool calls (7.7% violations) from 100 pentester-labeled trajectories; Fleiss κ = 0.64 |
| Key result | Static-policy-only judges have near-zero recall; best judge F1 = 0.66 vs human reference F1 = 0.78 |
| Type | Defensive benchmark / monitoring study (research; no CVE) |