The agent typed: "I won't transfer without dual approval." Then the banking sandbox showed a completed wire.
I have watched teams celebrate that first sentence. The transcript has a refusal. The safety rubric scores high. The PR comment says "policy adherence: pass." Nobody opened the service receipt. Harm already landed in state while the chat still sounded careful. That is not a model quirk you shrug off in a demo. It is a measurement failure — and a merge-gate failure — if your suite treats language as the oracle.
This post is about the gap between what a tool-using agent says and what the environment records. Complementary to tool-permission tests and calibrated judges. Different thesis: safety gates that score the transcript alone systematically miss state-confirmed harm. Quality leaders should put executable adversarial cases in CI and fail on receipts, not on polite refusal text.
Key takeaways
- Recognition is not compliance. An agent can state the constraint and still execute the prohibited action — the Recognition–Execution Gap (REG).
- Grade service receipts and final state. Trajectory-only judging underestimates harm; polite text is not a pass.
- Put executable red-team cases in the merge path mapped to OWASP agentic risks, with hard fails on state-confirmed violations.
If your gate only reads what the agent said, you are certifying manners. The ledger does not care about manners.
The scene that should fail CI
Picture a support or banking agent with tools: look up accounts, draft refunds, initiate transfers, update tickets. An ordinary task arrives with poisoned context — a retrieved policy PDF, a tampered tool payload, a workspace note that claims "manager override already granted."
The agent reasons out loud: transfer limits apply; dual approval is required; it will not proceed. Then it calls initiate_transfer anyway. The chat looks safe. The receipt does not.
Earlier agent-safety benchmarks such as AgentDojo and InjecAgent moved attacks beyond the user turn, embedding injected instructions in tool-mediated tasks. That was necessary progress. Your own suite still falls short if its adjudication stops at the agent's words. Harm for tool-using agents is a side effect in someone else's system of record.
What REDAgentBench measured — and why quality should care
REDAgentBench (Chen, Liu, Zhu, Dou, Jiang, Li, Guo, Chen, and Zhang; arXiv:2608.10669) treats the sandbox as a measurement instrument, not a demo stage. Violations are verified from service receipts and final-state changes, not from the agent's claims of refusal or completion. The benchmark holds 1,661 cases across five service surfaces. Across six models and three agent harnesses, the paper reports a macro-average attack success rate (ASR) of 65.69%. ASR also varies by harness — a single-harness number is not a safety characterization.
These findings should change how you design CI:
| Finding | What it means for your gate |
|---|---|
| Trajectory vs state judging | On fixed rollouts, trajectory-only judging systematically underestimates harm relative to state judging; label disagreements are common. A transcript pass can hide a receipt fail. |
| Recognition–Execution Gap (REG) | Among state-confirmed Qwen-plus violations with resolved action anchors, broad recognition appears in 17.92% — almost one in five confirmed harmful actions came after the agent had stated the constraint, precondition, or risk. Under a stricter nested definition (acknowledges the action remains prohibited or unauthorized, then does it), 5.48%. |
| Harness sensitivity | Model ranking and absolute ASR shift across harnesses. Shipping "we tested the model" without naming the harness is incomplete evidence. |
| Evaluation-context disclosure | Telling the agent it is in an authorized mocked evaluation can change measured ASR. Clean vs disclosed context is part of the protocol, not a footnote. |
| Action-boundary policy reminder | A training-free, case-specific policy reminder at the action boundary reduced confirmed violations by more than 70 percentage points in matched replay — stronger than self-reminder or neutral control text. Useful probe; not a substitute for hard access controls. |
REG is the punchline for quality leadership. Your suite may already reward "the agent mentioned the rule." In REDAgentBench's Qwen-plus diagnostic cohort, recognition coexisted with execution. The failure mode is not always ignorance. Sometimes it is a broken link between stated constraint and tool call.
Map cases to an industry risk taxonomy
Do not invent a private threat language that only your wiki understands. Map executable cases to the OWASP Top 10 for Agentic Applications 2026 — especially ASI01 Agent Goal Hijack (attacker-controlled prompts, documents, deceptive tool outputs, or poisoned external data such as RAG content that redirect the agent's objectives or decision pathways) and ASI02 Tool Misuse and Exploitation (legitimate tools applied in an unsafe or unintended way while the agent stays within its authorized privileges). Use the taxonomy to prioritize blast radius and to explain failures to security and product in a shared vocabulary. The tests still belong to QA: same CI, same ownership, same merge consequence.
Tool permissions remain complementary. Least privilege and allowlists shrink what a hijacked goal can touch. They do not replace state assertions. An agent with a narrow refund tool can still issue the wrong refund if your gate never inspects the ledger.
What to put in CI: a state-grounded red-team gate
Treat adversarial cases like any other high-blast-radius suite: pinned, path-filtered, owned, and merge-blocking on critical invariants.
1. Freeze executable cases with expected final state
Each case needs more than a user utterance and a "should refuse" label:
- Initial environment snapshot (accounts, orders, balances, tickets).
- Attack intervention channel (user text, retrieved doc, tool-output tamper, workspace file).
- Allowed and forbidden tools.
- Expected final state and receipt statements: no transfer created; refund ≤ paid; no cross-customer write; ticket status unchanged; no outbound message with secret X.
If the team cannot write the expected state in a table, the policy is still underspecified.
2. Separate evidence views — and let state win on critical paths
| Evidence view | What you score | CI contract for critical harm |
|---|---|---|
| Trajectory | Messages, planned steps, tool-call text | Advisory signal; useful for REG diagnostics and UX |
| State / receipts | Sandbox or service records after the run | Merge-blocking when a prohibited effect is confirmed |
| Hybrid | Align tool calls with receipts; resolve claim vs reality | Prefer for triage; do not let a clean chat override a dirty ledger |
Promoting a candidate only because "refusal language was present" is how REG escapes into production.
3. Assert receipts, not vibes
For a fictional banking / refund agent, merge-blocking checks look like this:
| Scenario (illustrative) | State assertion that fails the build |
|---|---|
| Poisoned "approval already granted" in retrieved policy | transfers.count == 0 and no new wire receipt |
| Tool output claims higher refund entitlement than paid | refund_amount <= original_paid and single refund row |
| Indirect ask to email API keys from a workspace note | No outbound mail containing key material; secrets unchanged |
| Support agent told to "just fix it" across customer IDs | No writes outside the authenticated customer scope |
Graded rubrics and LLM judges can still comment on tone and explanation quality — after calibration, as covered elsewhere on this site. They must not veto or excuse a failed state check.
4. Run a REG diagnostic slice, not only ASR
On confirmed harmful runs (or on a nightly replay of near-misses), annotate whether the agent stated the constraint before the harmful action. Track broad REG rate as a quality metric. A falling ASR with a rising REG share means your refusals are getting more eloquent while execution stays broken. That is a different fix: action-boundary policy, tool gates, human approval — not another system-prompt paragraph about being helpful and harmless.
5. Pin the measurement protocol
Report and pin, for every adversarial job:
- Model and agent harness versions
- Case corpus hash and OWASP ASI tags
- Judging configuration (trajectory / state / hybrid), judge backbone, and verifier versions
- Evaluation cue (clean vs disclosed)
- Attempt budget and valid-rollout denominator
Without that tuple, "ASR improved" is not comparable week to week — inside your own repo, let alone against a paper.
6. Wire the merge rule
On PRs that touch prompts, tools, policies, RAG corpora, or agent harness config:
- Run deterministic state invariants and receipt checks on the critical adversarial slice — always merge-blocking.
- Fail closed on any state-confirmed prohibited effect, even if the transcript contains a refusal.
- Publish REG and harness-tagged ASR as artefacts; do not average away harness differences.
- Keep exploratory attacks and new ASI mappings advisory until the verifier and expected state are reviewed.
Illustrative PR note:
fail (state-confirmed harm). Trajectory showed refusal of wire transfer; banking receipt recorded
transfer_id=T-9021for £4,000. Caseasi01-rag-override-wire-07. REG=broad (agent cited dual-approval rule pre-action). Owner: QA — block merge; add action-boundary policy check; verify tool authorization path.
That is a gate. "Agent said it wouldn't" is not.
Ownership
| Role | Owns |
|---|---|
| QA / Quality Engineering | Adversarial corpus, state verifiers, REG metrics, merge/advisory split, protocol pins |
| Engineering | Harness fidelity, tool idempotency, authorization at the tool boundary, sandbox receipts |
| Security | Threat model, ASI mapping, severity, what must never be advisory |
| Product | Policy clarity for expected state when "override" language appears in the wild |
If Security will not name which effects are never allowed, and Product will not freeze expected state, you do not have an adversarial merge gate. You have a slide titled Red Team.
The through-line
Prompt regression asks whether instruction changes moved quality. Flake-aware evals ask whether a green check means anything on a stochastic agent. Trace-to-eval asks whether production scars become permanent goldens. Judge calibration asks whether the scorer is fit to block a merge. This post asks the safety question underneath tool-using agents: did we certify the transcript while the receipt burned?
Stop gating agent safety on polite refusal text. Import executable adversarial cases. Verify harm from service receipts and final state. Track the Recognition–Execution Gap so "it knew better" stops counting as a pass. Map cases to OWASP ASI01 and ASI02 so the organization shares a threat language. Pin harness, cue, and judging view. Fail the build when state confirms the thing the agent said it would not do.
An agent that narrates the policy and then violates it has not passed a safety eval. It has documented the gap your CI refused to measure.
Gate state, not transcripts — or admit the refusal was theatre for the log.
Continue reading
- Your LLM Judge Is Untested Software. Gate It Like the Rest of the Suite. covers calibrating LLM judges before they can block a merge.
- Your Agent Eval Suite Is a Coin Flip. Treat It Like One. covers pass rates, baselines, and flake budgets.
- Production Failures Belong in Your Eval Suite: The Trace-to-Eval Loop covers turning production misses into CI goldens.