AgentCyberRange: measuring how far AI agents get in real intrusions

What is this?

AgentCyberRange is an open evaluation infrastructure, published on arXiv on 12 June 2026 by researchers at Fudan University, built to answer a question that isolated capture-the-flag scores cannot: how far can a frontier AI system get through a realistic intrusion — from an exposed web service all the way to compromise across an internal network? Instead of testing single-bug exploitation in the abstract, it drops agents into full cyber ranges and measures whether they can chain the operational stages of an actual attack: discovering exposed services, gaining a foothold, collecting internal information, and expanding compromise across hosts.

The authors frame it as the first open, multi-range setup for measuring autonomous cyber-attack capability under reproducible conditions. That framing matters for defenders and safety teams. The purpose is not to arm attackers; it is to observe emerging offensive capability early, in a lab, before it shows up in the wild.

How it works

The benchmark deliberately preserves the end-to-end structure of an intrusion rather than slicing it into isolated skills. It ships 110 vulnerabilities across 15 real web applications and 8 enterprise-like cyber ranges containing 156 internal hosts, split into two stages. Web exploitation asks a system to explore an exposed application — discovering hidden URLs and input parameters — and turn a reachable weakness such as SQL injection, SSRF, broken access control, command injection, or unsafe deserialization into an initial foothold. Post-exploitation begins from that foothold and measures internal reconnaissance, privilege escalation, and lateral movement across hosts. Some ranges add defensive pressure, including honeypots and host defenses, to see whether a system can make progress while monitored.

A companion toolchain called Cage makes the evaluation reproducible. Agent adapters expose different harnesses — for example Codex and Claude Code — through a common interface, so systems run under matched prompts and step budgets. A benchmark manager deploys the web applications and internal ranges in isolated environments and resets task state between runs. A verifier then checks whether a claimed success is actually backed by runtime evidence rather than simply asserted by the agent. The benchmark setups and pipeline are open-sourced on GitHub and Hugging Face. Understanding the design requires no exploit content: the contribution is the measurement harness, not a weapon.

Why it matters

The reported numbers are a capability signal, not a panic button. Across six frontier systems evaluated under matched budgets, the strongest configuration — GPT-5.5 with Codex — solved 16.1% of web-exploitation tasks and 31.7% of post-exploitation tasks; with more concrete hints, those rates rose to 33.0% and 46.3%. Two observations sharpen the picture. First, systems occasionally stepped outside the benchmark: during evaluation they surfaced valid out-of-scope findings, including a previously unknown arbitrary-file-write flaw in a popular open-source project, and mutated payloads to slip past host defenses. Second, reliability stayed low — agents missed hidden attack surfaces, were unstable across repeated runs, failed multi-step post-exploitation chains, and tripped honeypots, leaving warning signals under defensive pressure.

Read together, that is the useful takeaway. Current frontier agents are not dependable end-to-end attackers, but they already complete a non-trivial slice of realistic operations and can find real bugs. For anyone tracking dangerous-capability trends, isolated CTF or single-vulnerability scores understate the risk, because they skip the chaining that actually defines a breach. Measuring the whole chain — under reproducible, monitored conditions — is what lets a safety or governance team see the curve move rather than being surprised by it.

Defenses

Track capability with end-to-end evaluations, not isolated tasks. A model that scores modestly on CTF puzzles can still chain reconnaissance, foothold, and lateral movement. Dangerous-capability tracking should measure the full operational path, which is precisely the gap this benchmark was built to close.

Build defensive pressure into your evaluations. The ranges that included honeypots and host defenses exposed a reliable weakness: agents left warning signals and tripped traps. Detection engineering that assumes an automated adversary — deception assets, anomaly signals on internal hosts — buys real advantage today.

Treat agent-discovered findings as genuine disclosures. Systems surfaced an unknown zero-day during routine evaluation. If you run agents against your own estate, assume they will find real, unpatched bugs, and have a coordinated-disclosure path ready before you start.

Assume payload mutation. The evaluated systems adapted payloads to bypass host defenses, so signature-only controls degrade quickly. Favor behavior- and outcome-based detection over static payload matching.

Require verifier-backed success. The benchmark counts a task solved only when runtime evidence supports it. Apply the same discipline to your own red-team and eval pipelines: measure demonstrated impact, not the agent’s self-report.

Status

Item	Reference	Notes
Core release	arXiv:2606.14295, 12 June 2026	AgentCyberRange + Cage toolchain, Fudan University
Scale	Same paper	110 vulnerabilities, 15 web apps, 8 ranges, 156 internal hosts
Stages	Same paper	Web exploitation and post-exploitation
Best result	Same paper	GPT-5.5 with Codex: 16.1% web / 31.7% post; 33.0% / 46.3% with hints
Notable behavior	Same paper	Found an out-of-scope zero-day (arbitrary file write); mutated payloads vs host defenses
Reliability	Same paper	Unstable across runs, missed hidden surfaces, failed multi-step chains, tripped honeypots
Artifacts	GitHub / Hugging Face	Benchmark setups and evaluation pipeline open-sourced

The durable point is methodological: offensive AI capability has to be measured where a breach actually happens — across the whole chain, under reproducible conditions, with defensive pressure in the loop. On this benchmark, frontier agents are not yet reliable attackers, but they are already good enough that watching the end-to-end curve is the responsible thing to do.