AutoSpec: teaching agent safety rules to fix their own false positives

What is this?

Most teams that let an LLM agent run tools eventually write a list of safety rules: don’t run rm -rf, don’t email outside the company, don’t touch the payments API without approval. These rules are readable and easy to audit, which is exactly why people like them. They are also brittle. Write them too tightly and the agent gets blocked doing legitimate work; write them too loosely and unsafe actions slip through. Every operator who has hand-tuned a guardrail knows the whack-a-mole that follows.

A paper published on arXiv in late June 2026, AutoSpec: Safety Rule Evolution for LLM Agents via Inductive Logic Programming (2606.24245), proposes a way out of the tuning treadmill. Instead of asking humans to keep rewriting rules by hand, AutoSpec learns the edits automatically from labelled examples of safe and unsafe agent behaviour, using a classical technique from program synthesis — inductive logic programming (ILP) — rather than another neural black box. The output is still a set of human-readable rules, so you keep the auditability that made rules attractive in the first place.

How it works

The core loop is counterexample-guided inductive synthesis (CEGIS). You start from whatever expert-written rules you already have, plus a dataset of agent actions annotated as safe or unsafe. AutoSpec then repeats a simple cycle:

1. Evaluate the current rules on the labelled data.
2. Mine the mistakes:
     - false positives  = safe actions the rules wrongly blocked
     - false negatives   = unsafe actions the rules wrongly allowed
3. Run ILP to find predicates that discriminate the two —
   features common in false negatives but rare in false positives
   (or vice versa).
4. Propose candidate rule edits built from those predicates.
5. Verify each candidate; keep the revision that scores best.
6. Repeat until the rules stop improving.

The clever part is step 3. The space of possible rule edits is exponential, so blindly searching it is hopeless. ILP prunes that space by identifying which predicates — properties like “the tool is shell.exec”, “the target path is outside the workspace”, “the recipient domain is external” — actually separate the mistakes from the correct decisions. That turns “guess a better rule” into “combine the handful of features that explain the errors”, which is tractable.

Because the whole thing operates on symbolic rules rather than model weights, every change is inspectable. You can diff the rule set before and after, read the new clause in plain logic, and reject it if it looks wrong. That is a meaningful difference from tuning a classifier, where the “why” is buried in parameters.

Why it matters

Guardrails for agents live in a genuinely hard spot. The action space is large, the same tool call can be safe or catastrophic depending on context, and the cost of a false negative — an agent that deletes a production database or exfiltrates a secret — is not symmetric with the cost of a false positive. Teams respond by over-blocking, which trains users to disable the guardrail, which is worse than having none.

AutoSpec reports F1 scores of 0.98 and 0.93 across two domains and cuts false positives by up to 94%, while producing rules that generalise to scenarios not seen during training. If those numbers hold up in independent replication, the practical win is not just accuracy — it is that you can lower the false-positive rate without quietly loosening safety, because the same process is simultaneously pushing false negatives down. It also connects to a broader line of work on constraining LLM-driven agents with explicit, checkable specifications rather than hoping the model behaves, exemplified by earlier research on enforcing constraints for robot agents.

Two caveats worth stating plainly. First, learned rules are only as good as the labels: a biased or thin annotation set produces confident, wrong rules. Second, a rule layer is a policy engine, not a semantic understanding of intent — it will not catch an unsafe action that looks, feature-for-feature, exactly like a safe one. This is a defence-in-depth component, not a silver bullet.

Defenses

For anyone running tool-using agents, the takeaways are concrete.

1. Keep a labelled corpus of agent decisions. Whether or not you adopt AutoSpec, the raw material for improving any guardrail is a growing set of actions marked safe or unsafe. Instrument your agent to log tool calls with enough context (tool name, arguments, target, caller) to label them later.

2. Prefer interpretable policy layers where you can. A rule you can read and diff is a rule you can audit, roll back, and reason about during an incident. Reserve opaque classifiers for cases where symbolic rules genuinely cannot express the signal.

3. Measure both error types, not just blocks. Track false negatives (unsafe actions allowed) alongside false positives (safe actions blocked). A guardrail that only reports what it blocked hides its most dangerous failures.

4. Treat the rule set as versioned, reviewed code. If rules evolve — by hand or automatically — every change should go through diff review, and additions learned from data should be sanity-checked against adversarial examples before shipping.

5. Layer it. Combine a rule/policy engine with runtime monitoring, least-privilege tool scoping, and human approval for high-blast-radius actions. No single layer, learned or hand-written, should be your only line.

Status

Item	Reference	Date	Notes
AutoSpec paper	arXiv 2606.24245	2026-06 (late)	ILP/CEGIS rule evolution from safe/unsafe labels
Reported results	Same	2026-06	F1 0.98 / 0.93; false positives cut up to 94%
Related prior work	Brown H2R “Safety Chip”	2024	Enforcing explicit constraints for LLM-driven agents
Framing reference	OWASP LLM Top 10	2025	Excessive agency / insecure output handling

The right way to read this is not “guardrails are solved.” It is that the false-positive-vs-false-negative tradeoff that makes hand-written agent rules exhausting can be attacked as a learning problem — and, importantly, attacked in a way that leaves you with rules a human can still read. That combination, accuracy plus auditability, is what most production guardrails are missing today.