AgentWatch: an open framework for auditing how safely browser agents behave

What is this?

AgentWatch is an open-source evaluation framework for AI browsing agents, built as a Master of Information and Cybersecurity (MICS) capstone project at the UC Berkeley School of Information and last updated on 7 June 2026. The team — Boaz Kaufman, Marisa Hall, Anya Svan, Cynthia Austin, Rutika Kushe and Anagha Late — audited five leading browsing agents and released both their scoring methodology and their scenario library publicly. The project won the school’s Chang Award for its capstone cohort, and it was picked up in Adversa AI’s July 2026 agentic-security roundup, which listed it among the month’s notable defensive resources.

The premise is simple and, once stated, hard to unsee: browsing agents can read your screen, fill in your forms and act on your behalf across the open web, yet users have almost no way to know how safely a given agent does any of that. AgentWatch is an attempt to turn that blind spot into something measurable.

How it works

Unlike a traditional chatbot, a browsing agent takes autonomous actions on pages it did not write and cannot fully trust. That opens a risk surface that per-message safety filters were never designed to cover. AgentWatch scores agents along five dimensions: unintended data disclosure, prompt misinterpretation, hallucination, prompt injection, and browser sandbox isolation. Together these span the paths by which a browsing agent leaks information it shouldn’t, follows instructions it shouldn’t, or acts outside the boundary it was supposed to stay inside.

The methodological contribution is the scoring itself. Agent behaviour is stochastic — run the same scenario twice and you may get two different outcomes — so a single pass tells you very little. AgentWatch’s framework is explicitly designed to adjust for that variance rather than treat one lucky or unlucky run as ground truth, which is what makes the resulting scores comparable across agents. The evaluation infrastructure and the scenario library are both open-source and built to grow, so new attack scenarios and new agents can be folded in as the field moves. The team ships a public platform and a project repository so others can reproduce the runs and contribute test cases rather than take the numbers on faith.

Why it matters

Browsing agents are being wired into everyday products faster than anyone has agreed on how to measure their safety. Buyers are largely left comparing marketing claims, because there is no shared yardstick for “how does this agent behave when a page tries to manipulate it.” A reproducible, open framework that adjusts for run-to-run variance is exactly the kind of common reference the space has been missing — closer in spirit to a consumer-safety bench test than to a one-off red-team writeup.

The five dimensions also map cleanly onto the failure modes that keep recurring in browsing-agent incidents: data exfiltration through the agent’s own actions, injected instructions hidden in page content, and actions that escape the intended sandbox. Framing evaluation around those categories, and making the scenarios shareable, lets defenders argue about specific, testable behaviours instead of vague assurances.

Defenses

Ask vendors for behaviour, not adjectives. Treat “how does it score on data disclosure, prompt injection and sandbox isolation” as a procurement question, and expect reproducible evidence rather than a safety label.

Test under variance, not once. Because agent behaviour is stochastic, a single passing run proves little. Evaluate the same scenario many times and look at the distribution — the core idea AgentWatch’s scoring is built around.

Cover the whole surface. Prompt injection is only one of five dimensions here. Unintended data disclosure, misinterpretation, hallucination and sandbox isolation each need their own scenarios; a guard that only inspects the latest instruction misses most of them.

Contribute and reuse open scenarios. Shared, extensible scenario libraries beat private one-off tests. Adopting and extending an open framework means your coverage grows with the community’s rather than ageing in isolation.

Constrain the agent’s blast radius. Independently of scoring, limit what a browsing agent can read, where it can send data, and what it can execute — so that a failure on any one dimension is contained.

Status

Item	Reference	Notes
Project	AgentWatch	MICS capstone, UC Berkeley School of Information (Cyber 295, Spring 2026)
Team	Same page	Kaufman, Hall, Svan, Austin, Kushe, Late; Chang Award recipient
Scope	Same page	Five browsing agents; five dimensions incl. prompt injection and sandbox isolation
Method	Same page	Custom scoring adjusting for stochastic agent behaviour
Artifacts	Project repo / platform	Open-source evaluation infrastructure and scenario library
External mention	Adversa AI roundup, 2 July 2026	Listed among July 2026 agentic-security resources

The durable takeaway is not a leaderboard but a method: browsing-agent safety is measurable, it has to be measured across several distinct dimensions, and — because these systems are stochastic — it has to be measured repeatedly, with scoring that accounts for the variance rather than trusting a single run.