When a squad tells me their agent is "green", I ask a second question: green against what? If the answer is a golden set of prompts scored by an LLM judge, I am only halfway convinced. That suite tells me something about the model and the fixtures. It tells me almost nothing about the software that wraps the model: the code that parses tool calls, routes work between agents, updates memory, recovers from timeouts and decides what to do when the model returns rubbish.
I have spent the last couple of years building that wrapping layer for real delivery, not demos. A library of forty-plus QA skills and agents. A five-agent orchestration framework on Claude, Playwright and MCP. Agentic PR review gates that have to refuse bad changes before a human ever sees them. The outcome of that programme was 65% fewer defect escapes and 80% fewer production incidents. I do not credit the models for that. I credit control points: places where agent output has to meet an agreed standard before it moves on.
So when I read that the harnesses of popular open-source agents are still largely undertested, my reaction was not surprise. It was recognition. We have been writing software around LLMs and calling the whole thing an "agent", while testing it as if the only interesting component is the LLM.
The harness is most of the product
Strip an agent down and you get two parts. A backbone model that proposes text and tool calls. And a harness: the operational software that turns those proposals into actions. Environment setup. Context and memory. Guardrails. Workflow orchestration. Tool interfaces. Parsing. Retries. Redaction. Logging.
That harness is ordinary code. It has branches, contracts and failure modes. It can leak a secret because a nested list skipped redaction. It can dispatch the wrong tool because a JSON field was optional on Tuesday and required on Wednesday. It can keep answering confidently after the memory index stopped updating. None of those failures need a stupid model. They need an undertested one.
To be fair, end-to-end agent benchmarks still matter. SWE-bench-style suites, Terminal-Bench and the rest are how we compare systems. They treat the agent as a black box on purpose. That is a useful comparison lens. It is a weak regression lens for the organisation shipping the agent. Black-box pass rates will not tell you whether your tool-call parser still handles a truncated argument list after last week's refactor.
The numbers are not subtle
In October 2026, Xiong, Ge, Chen and Lou published what I think is the first serious look at harness test adequacy in the wild (arXiv:2610.04921). They studied ten large open-source agent systems, including OpenHands, Aider, SWE-agent, Browser Use and OpenClaw. The interesting cut is not whole-repo coverage. It is what they call LLM-dependent harness code: the harness statements that have a data or control dependency on model output. That is the glue that interprets nondeterministic responses and decides what happens next.
Across those ten systems, average line coverage of that LLM-dependent harness was 44%. Branch coverage was 41%. The mutation score, which asks whether the existing tests would notice injected faults in that same code, averaged 34%. In plain words: less than half of the critical decision code is exercised, and about two thirds of the artificial faults planted there survive the suite.
Coverage without kill power is a false friend. Several agents look respectable on lines and weak on mutants.
The spread is wide. Kimi Code sits near the top. GPT Researcher and RD-Agent sit near the bottom. I would not treat any single row as a moral judgement on that project. The average is the useful signal for a QA leader: even well-starred agent codebases, with hundreds or thousands of tests, leave the model-facing harness thinly covered.
One caveat, and it matters. This is open-source agent tooling, measured with standard coverage and mutation tools. Enterprise harnesses may be better or worse. I have not seen a published comparison for production internal agents, and I would not invent one. What I take from the paper is a direction of risk, not a percentage to put in a board pack.
Orchestration is where suites go quiet
Aggregate coverage hides worse news. The same study sampled 700 LLM-dependent branch predicates and classified what they control. Tool-action handling, which usually has an explicit schema and a bounded value space, was the best-covered category among the workflow intentions they measured: about 38% of those predicates were fully covered. Agent orchestration control was the worst: only 14% fully covered, and 63% never reached at all.
Schemas are easier to mock than stateful cross-agent flow. That is exactly why orchestration needs deliberate fixtures.
That matches what I see when teams show me their agent test folders. Plenty of checks that a tool returns the right shape. Far fewer that a hand-off between agents preserves the right state, that a failed observation triggers the recovery path you think you wrote, or that a policy change mid-run does not leave two agents disagreeing about who owns the next step.
Hypothetical, labelled as such: say your support flow has an intake agent that classifies the ticket and a fulfilment agent that may issue a refund. Your golden set asks whether a refund was correctly refused. It may never plant a malformed classification, a late policy update, or a tool timeout between the two agents. The suite can stay green while the orchestration code that would fail in production never runs.
A bigger model will not cover that code
The instinct when an agent misbehaves is still to upgrade the backbone. Sometimes that helps. Often it does not, because the defect was never in the model.
A complementary study presented at ASE 2026, Tangent (arXiv:2608.08413), looked at how developers actually test agent applications. The authors manually labelled 2,572 test methods across 240 modules. Most of those tests (62%) exercised a tool or agent in isolation. About 35% targeted interaction behaviour. Only 7.5% addressed non-functional requirements such as security or performance. Nearly 40% of tests used simple dummy inputs rather than realistic scenarios.
That is a familiar shape if you have inherited a large automation estate. Isolation is cheap. Interaction is expensive. Non-functional cases are deferred until after an incident. The difference with agents is that the interesting failures cluster exactly where isolation is weakest: between model output and harness decision, between agents, and between the agent and the environment it can mutate.
If your eval budget only funds the left-hand box, the right-hand box will invent its own production incidents.
I have written before about tool permissions and MCP contract drift. Those posts are harness posts, even if I did not use that word then. Permissions, tool schemas and server digests are software contracts around the model. They deserve the same kind of tests we already write for APIs and message buses.
What I'd test first
I do not think anyone has a finished playbook, and I would be suspicious of a vendor who claims otherwise. If a squad asked me where to start this quarter, this is roughly what I would say.
Name the harness as a component. Give it an owner, a version and a change log. If a PR touches parsing, routing, memory write paths, tool dispatch or recovery, that PR should not be reviewed as "prompt tweaks". It is infrastructure.
Build contract-faithful fixtures before you chase coverage percentages. The Xiong paper's useful practical finding is that high adequacy on LLM-dependent harness code needs test setups that satisfy the expected data structures, runtime states and interaction protocols across components. A single happy-path mocked response is not enough. You need helpers that can produce valid provider responses, valid tool payloads and valid agent states, then vary them deliberately. Without that, coverage tools will keep reporting what you already know: the easy branches are green.
Prioritise the decision types that stay dark. Start with orchestration hand-offs, observation feedback and error recovery. For each, write cases that force both sides of the branch: tool result present and missing, schema valid and invalid, recovery path taken and skipped. Plant at least one case where the model output is plausible and still wrong for your domain.
Separate harness failures from model failures in the report. When a run fails, I want to know whether the fact was never parsed, parsed and then mis-routed, routed and then executed against stale state, or executed correctly and judged badly. Collapsing those into "agent failed" teaches the organisation the wrong lesson and sends the wrong people to fix it.
Keep a thin harness gate in CI that does not call the live model. Mocked contract tests are cheap. They catch the refactors that break JSON handling, redaction and dispatch. Save the expensive live evals for the cases that actually need a model. I have made that trade-off on delivery estates before, compressing regression from days to hours by putting the right checks at the right layer. The same economics apply here.
None of this replaces a golden set. It means the golden set is no longer the only evidence you trust.
Who should own the harness
Ownership is the part most organisations skip. Platform teams build the harness. Product teams consume it. Quality is asked to "add some agent evals". The gaps land in the space between them.
I would put harness adequacy with quality, working with whoever owns the agent runtime. The job is the one quality has always done: decide what correct means at each boundary, and prove the system still meets it after something changes. That includes model upgrades, tool schema changes, prompt edits and orchestration refactors. If nobody owns the contracts, every green demo is provisional.
There is a lot we still do not know. The coverage study is ten open-source systems, not a census of enterprise agents. Mutation scores are a proxy for fault detection, not a map of the bugs you will actually see. Tangent's sample is Python-heavy and framework-skewed. I expect better harness test generators to arrive, and some of what feels urgent today may look routine in a year.
What will not change is older: software that sits between a nondeterministic producer and a side-effecting world needs tests for how it decides, not only for whether the producer sounded plausible. We learned that with message buses, ETL jobs and payment gateways. I would rather not relearn it with agents by reading the post-incident review.
Continue reading
- Your Agent's Tool Permissions Are Part of the Test Surface covers the permission and approval contracts that sit inside the harness.
- You Approved the MCP Tool Once. CI Never Checked If It Changed. covers pinning tool definitions so harness contracts cannot drift silently.
- Checking for Hallucination in AI Coding Agents covers separating model invention from the verification layer around it.
Sources
- Yifan Xiong, Jingyi Ge, Zhenpeng Chen and Yiling Lou, "Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild," arXiv:2610.04921. Table 2: average LDH line coverage 44.27%, branch coverage 41.18%, mutation score 33.60% across ten agents. §5 / Finding 4: of 700 sampled LDH branch predicates, 25.57% fully covered overall; tool-action handling 38.18% fully covered; agent orchestration control 14.29% fully covered and 63.27% uncovered; observation feedback 18.39% fully covered; error recovery 17.65% fully covered; policy guards 50.00% fully covered.
- Rangeet Pan, Tyler Stennett, Divya Sankar, Bridget McGinn, Alessandro Orso, Raju Pavuluri, Saurabh Sinha and Maja Vukovic, "Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications," ASE 2026 (arXiv:2608.08413). Finding 1 / §3.2: 61.8% of labelled tests evaluate tools or agents in isolation; 34.6% target interaction behaviour; 7.5% address non-functional requirements. Finding 4 / §4.2: 39.9% of tests use simple dummy data.