The agent adds twenty tests. They pass. Coverage rises. The pull request says the change is thoroughly tested.
There is still a missing question: what wrong behaviour would make those tests fail?
This matters whenever the implementation and its tests come from the same interpretation of a requirement. A mistake can appear consistently in both. More tests then reinforce the mistake instead of exposing it.
In The Confident Wrong Answer, I recommended mutation testing as a check on generated tests. Here is how to use that idea without turning another percentage into a target people learn to game.
Key takeaways
- Mutate the production behaviour the new tests claim to protect.
- Investigate survivors against an independent requirement.
- Use the score as evidence, not proof of correctness.
Ask which plausible wrong implementation would turn this green suite red.
Challenge the assertion with a deliberate change
Mutation testing changes production code and reruns the tests. A mutant is killed when a test fails; it survives when the tests still pass. Stryker's introduction explains the mechanics and why executed lines alone do not establish test effectiveness.
Consider this deliberately small, fictional basket-discount rule:
A basket receives a 500-cent discount only when the customer is a member and the subtotal is at least 5,000 cents. Inputs are validated, non-negative integer cents.
function discountCents(subtotalCents, isMember) {
if (!isMember) return 0;
return subtotalCents >= 5000 ? 500 : 0;
}
A test for a member spending 8,000 cents passes. So does a test for a non-member spending 8,000 cents. Neither tests the threshold itself.
Change >= to >. Both tests remain green, but the implementation now denies the discount at exactly 5,000 cents. A test expecting a 500-cent discount at the threshold exposes that change.
Stryker documents comparison, logical, and other supported transformations in its mutator catalogue. Support varies across its language implementations. Choose a tool that supports the code you need to examine.
The example is intentionally simple. It demonstrates a missing boundary assertion, not a measured claim about how often AI makes that mistake.
Keep the requirement outside the repair loop
Give the test author the agreed rule and ask for cases derived from it before asking for additional coverage. For the basket example, the useful cases include a member below the threshold, exactly at it, and above it, plus a non-member who would otherwise qualify.
Review the expected values against the rule. An agent can produce an elaborate parameterised test that faithfully reproduces the implementation's wrong threshold. Neatness does not supply an independent expected result.
When a mutant survives, ask for the missing behaviour in plain language first. Only then add the test. This keeps the task anchored to the customer rule instead of to whatever edit happens to improve the score.
A survivor needs a diagnosis
For each surviving mutant, use a short investigation record:
| Question | What the answer tells you |
|---|---|
| Does this change observable behaviour for a permitted input? | Whether the mutation represents a meaningful distinction |
| Does an existing test reach that behaviour? | Whether the gap is execution or assertion |
| What requirement decides the expected result? | Whether the team has an independent basis for a new test |
| Was the run valid? | Whether setup, selection, or execution problems invalidate the result |
A mutation can be equivalent for the supported input domain. Do not invent a requirement just to kill it. Record the reasoning and have exclusions reviewed. Also distinguish surviving mutants from tool errors, timeouts, and mutants the selected tests never exercised; read the runner's result categories before interpreting its score.
For this proposed workflow, the release concern is an unexplained survivor in consequential changed logic. A repository-wide percentage can hide that concern under thousands of easy cases elsewhere.
Start with the changed production behaviour
When an agent adds tests, select the production functions those tests claim to protect. Mutation tools operate on the implementation; targeting only the new test files misses the purpose of the exercise.
If production code also changes, include affected behaviour beyond the literal changed lines. A one-line helper change may alter several callers. Run the relevant tests for those callers, then keep a broader scheduled run to discover gaps that the fast selection missed.
My starting policy would be to investigate surviving mutants in changed price calculations, permission decisions, or state transitions before merging. Run lower-risk areas in reporting mode while the team learns the cost and noise. Establish a stable baseline before introducing a numerical gate.
This is a proposed review policy, not a claim that every team needs the same threshold. The useful budget is one that reviewers will actually sustain.
Know what the score cannot tell you
Killing every generated mutant does not establish that the requirement is correct. The mutation operators may never generate the mistake you care about. The suite may still miss interactions between services, invalid inputs handled elsewhere, concurrency, or behaviour absent from the implementation altogether.
In the basket example, the mutation exercise says nothing about whether two promotions may be combined. That question needs a product decision and an additional scenario. No score can recover a requirement nobody supplied.
For the next AI-authored test PR, ask the author to identify one plausible wrong implementation the tests reject. Run the targeted mutation check, inspect survivors, and retain the report with the review. Measure review effort and useful defects found as well as the score.
A green suite becomes stronger evidence when the team can show the wrong behaviours that turn it red.
Related reading
- Does TDD Survive Inside an AI Agent's Loop? examines the evidence behind test-first agent workflows.
- Your AI Agent Passed the Demo. Can It Pass a Release Gate? extends verification from code to agent outcomes.