Do your agent logs actually prove what it did? A benchmark for evidence sufficiency

What is this?

DEMM-Bench is a benchmark, published on arXiv in late June 2026 by Oleg Solozobov, for a question most agent deployments never test: when an autonomous agent takes a consequential action, do the records it leaves behind actually contain enough evidence to review that decision afterward? The name stands for Decision Evidence Maturity Model. Its central claim is uncomfortable for anyone relying on agent logs for oversight, audit, or incident response: the presence of a trace, an audit ledger, a structured schema, or a compliance checklist tells you almost nothing about whether the properties an auditor actually needs are present.

The distinction matters because governance conversations about agents usually stop at “we log everything.” DEMM-Bench argues that logging everything and being able to prove what happened are different things, and it gives that gap a measurable shape.

How it works

The benchmark evaluates evidence against what the paper calls a Decision Event Schema — a property-level description of what a complete record of an agent decision should contain, such as which inputs were seen, which authority was exercised, and which action was committed. Rather than scoring whether a log exists, DEMM-Bench scores whether each required property can be established from the evidence available.

It does this across a deliberately varied matrix: eight evidence regimes (different ways of capturing what an agent did), eight degradation conditions (ways real-world logging gets partial, delayed, or corrupted), and five container-presence baselines representing the common “we have traces / we have a ledger / we have a schema” postures. Two metrics drive the results. Property Sufficiency Accuracy measures whether required properties are judged correctly from the evidence. Overclaim Rate — the headline metric — measures the failure mode that governance cares about most: declaring a record “sufficient” for review when a required property is in fact missing. Labels are derived deterministically from how each case is constructed, so the scoring is not itself a matter of judgment.

The reported results are pointed. On the benchmark’s 64-case package, baselines that simply have a trace present or a schema present overclaim sufficiency on roughly 75% of cases; a ledger-present baseline overclaims on about 50%. The candidate approach the paper evaluates records zero overclaim, but with a mean Property Sufficiency Accuracy of only around 56% — meaning that even a system built to avoid false assurance still cannot fully reconstruct what it needs a large fraction of the time. The lesson is not that one design wins, but that “we have logs” is a claim that mostly fails when tested.

Why it matters

Evidence sufficiency is where agent security, governance, and incident response meet. When an agent moves money, changes a record, or sends data outbound, the after-the-fact question is always the same: can we show what it saw and why it acted? If the honest answer is “our trace exists but doesn’t capture the authority that was exercised,” then post-incident review, regulatory reporting, and dispute resolution all rest on records that quietly overclaim. This connects to a broader 2026 research thread on evidence tracing and execution provenance in agents, which similarly argues that trust in an agent’s actions has to be reconstructable, not assumed.

For teams building oversight around agents, the practical takeaway is that observability tooling should be evaluated on what it lets you prove, not on how much it collects. A dashboard full of traces can still leave you unable to answer the one question an auditor or investigator will ask.

Defenses

Specify the properties before choosing the tooling. Decide what an auditor must be able to establish about a decision — inputs seen, authority used, action committed, approvals obtained — and treat that property list as the requirement. Logging volume is not the target; property coverage is.

Test your evidence against missing-property cases, not happy paths. The benchmark’s value is methodological: construct cases where a specific property is absent and check whether your review process still (wrongly) calls the record sufficient. An oversight pipeline that never fails this test is not being tested hard enough.

Track an overclaim metric, not just log completeness. Measuring how often “sufficient” is declared when a required property is missing surfaces false assurance directly. Completeness percentages hide it.

Assume degradation. Real logs arrive partial, delayed, or corrupted. Evidence design that only holds under clean capture will not hold during the incident you actually need it for. Build for the degraded regimes.

Bind evidence to authority. Much of the sufficiency gap is about proving which privilege an agent exercised. Capturing the authority context of each action — not just the action text — is what makes a record reviewable.

Status

Item	Reference	Notes
Core benchmark	arXiv:2606.20634, late June 2026	DEMM-Bench (Decision Evidence Maturity Model), Oleg Solozobov
Scope	Same paper	8 evidence regimes, 8 degradation conditions, 5 container-presence baselines
Metrics	Same paper	Property Sufficiency Accuracy; Overclaim Rate
Key finding	Same paper	Trace/schema-present baselines overclaim ~75% of the 64-case package; ledger-present ~50%
Related context	arXiv:2606.04990	Survey of evidence tracing and execution provenance in LLM agents
Independent listing	Adversa AI digest, 2 Jul 2026	Categorized under defense frameworks / agent-runtime governance

The durable point is a reframing: for agentic systems, “we log it” is a hypothesis, not a guarantee. Whether your records can actually support oversight is something you have to measure — and by this benchmark’s evidence, presence-based logging usually can’t.