I've spent more of my career than I'd like to admit looking after regression packs. Building them, inheriting them, defending them in budget meetings, and now and then taking an axe to them. At a Tier 1 bank we took an automated regression cycle from three days down to three hours. Across eight Tier 1 engagements I've built automation from scratch, and in most of those places there was also an older pack that someone else had built and nobody wanted to touch.

So when I look at how teams are building eval suites for agents and LLM features, I get a strong sense of déjà vu. The suites are young, so they're still useful. But they're growing the way regression packs grew, and I think they're heading for the same three problems. They'll get too big and too expensive to run. They'll fill up with cases nobody trusts. And they'll keep passing long after they've stopped telling anyone anything.

This post is the set of lessons I'd hand to an eval team if I could. A few came from getting it right. More came from cleaning up after it went wrong.

Packs only grow, because deleting feels like risk

Nobody sets out to build a three-day regression pack. It happens one sensible decision at a time. A defect escapes, so you add a test for it. A feature ships, so its tests join the pack. Someone senior asks "are we covered for that?", and the safest answer is to add a test for it. Each decision is right on its own. A few years later you have thousands of tests, a run measured in days, and a team that can't tell you which half of it matters.

The part people underestimate is how hard it is to delete a test. Adding one is a visible act of care. Removing one feels like removing safety, and if something then breaks in that area, the person who deleted the test owns the outage. So nobody deletes anything, and the pack slowly takes the shape of every past fear.

Eval suites have exactly the same incentive. The practice I've argued for on this blog, turning production failures into eval cases, only ever adds. Every incident gives you a new case and nothing tells you to take one away. Give it eighteen months and the suite will be slow, costly and full of cases written for a version of the product that no longer exists. With LLM evals the cost arrives sooner than it did with Selenium packs, because every case spends model calls, and often a judge call on top.

Sometimes the test is the bug

The most humbling part of inheriting a regression pack is finding out how many tests are wrong. Not flaky. Wrong. The expected result was written against a requirement that has since changed. The assertion checks an exact string that a developer changed for a good reason. The test passes or fails for reasons that have nothing to do with the behaviour it claims to protect.

The best public evidence I've seen of this in AI evaluation comes from SWE-bench, one of the most widely used benchmarks for coding agents. In 2024 OpenAI, working with the benchmark's authors, had 93 professional Python developers screen 1,699 randomly sampled tasks, with three reviewers per task and the strictest verdict kept. They flagged 61.1% of tasks for unit tests that could reject a correct solution. Once vague problem statements and other issues were added in, 68.3% of the sample was filtered out, and a cleaned set of 500 tasks became SWE-bench Verified.

Bar chart of OpenAI's SWE-bench audit of 1,699 sampled tasks: 38.3% flagged for problem statements too vague to test fairly, 61.1% flagged for tests that could reject a valid fix, and 68.3% removed from the benchmark overall When careful people looked at the tests, a large share of them couldn't be trusted to say what they claimed.

Two things in that study matter for a QA leader. First, the screening was deliberately strict. OpenAI calls its own filtering "likely to be overzealous", so I wouldn't read it as "two-thirds of the benchmark was rubbish". I'd read it as: the moment anyone audited the expected results properly, a lot of them didn't hold up. Second, it changed the answer. GPT-4o's best score more than doubled on the cleaned set. The model hadn't changed at all. The measuring stick had.

Your internal golden set isn't SWE-bench, and I'd expect it to be in better shape, because the people who wrote it know the product. But it has the same failure mode, and probably faster drift. Expected answers go stale every time a policy, a prompt, a tool or a business rule changes. An eval that fails a correct answer is worse than having no eval for that case, because it teaches people that red doesn't mean anything.

A test that always passes may have stopped working

There's an old QA instinct that a test which has passed every day for two years is either protecting something very stable or protecting nothing at all. You only find out which by looking.

Public benchmarks show this happening at industry scale. Stanford's AI Index 2025 reports that on three demanding benchmarks introduced in 2023, scores jumped sharply within a year. SWE-bench rose by 67.3 percentage points and GPQA, a graduate-level science test, by 48.9.

Bar chart from the Stanford AI Index 2025 showing first-year score gains on benchmarks introduced in 2023: MMMU up 18.8 points, GPQA up 48.9 points, SWE-bench up 67.3 points A benchmark that every serious model passes stops helping you choose between them.

Some of that is real progress. Some, for public benchmarks, is exposure: the SWE-bench Verified write-up itself notes that because the tasks are scraped from public GitHub repositories, models pre-trained on internet text are "likely to be contaminated". Either way, the result is the same. The test stops separating good from bad.

Internal suites saturate in a quieter way. A team tunes the prompt until the golden set goes green, then works to keep it green. A few months later the suite mostly measures whether you've broken the thing you tuned for last spring. That's still worth having, but it's a regression check, not evidence that the agent is good. I'd be suspicious of any eval dashboard that has sat at 98% for a whole quarter. Either the product is perfect or the suite has stopped asking hard questions.

Not everything needs to run on every change

This is where the three-days-to-three-hours work taught me the most. The pipeline mattered, but the harder part was agreeing what we could stop running on every change. Some tests belong in front of every merge. Many belong in a nightly or pre-release run. A few belong in a museum.

Large engineering organisations have turned this into a discipline. Facebook's paper on predictive test selection describes a model that, for each code change, runs fewer than a third of the tests that build dependencies would have picked, and still flags more than 99.9% of faulty changes. The detail I like most is easy to miss: they still run everything on the main branch every few hours. Selection is about fast feedback on each change, not about giving up coverage.

I wouldn't copy that machine-learning approach for an eval suite of a few hundred cases. It was built for tens of thousands of deterministic tests in one huge repository. The principle carries over, though. Tier your evals. Keep a small, trusted gate that runs on every prompt, model or tool change. Run a wider regression set nightly or before release. And give retired cases somewhere to go other than the bin.

Diagram of a proposed eval case lifecycle: Reason (an incident, requirement or known risk plus a named owner) leads to Gate (runs on every change, small, fast, trusted, blocks the merge), then Regression (nightly or pre-release, wider and slower) when a case has stayed green for months and caught nothing, then Retired when behaviour changes. Gate and Regression both feed a quarterly re-audit of the expected answer by someone other than its author The lifecycle I'd want every eval case to have. It's my proposal, not an industry standard.

Noise teaches people to stop looking

Back in 2016, John Micco of Google wrote about flaky tests that "it is human nature to ignore alarms when there is a history of false signals coming from a system." That sentence describes almost every distrusted regression pack I've been asked to rescue. The build goes red, someone says "that test again", and the team learns to re-run until it's green. Nobody decides to stop trusting the pack. It just happens, one re-run at a time.

Eval suites are more exposed to this, not less. Outputs vary between runs, LLM judges disagree with themselves, and an expected answer that went stale last month fails a perfectly good response. I covered the statistics side in an earlier post on flaky evals. The leadership side is simpler: if red doesn't reliably mean something, people stop reading it, and from then on the suite is decoration.

How I'd run an eval suite differently

If I were setting up an eval practice from scratch today, with everything regression packs have taught me, these are the habits I'd want from day one.

Every case has a reason and an owner. A link to the incident, requirement or risk it exists for, and a named person who can say whether it still matters. A case with no reason is the first one to question.

Audit the expected answers, not only the outputs. Once a quarter, have someone other than the author re-check a sample of cases. Borrow from the SWE-bench Verified approach: more than one reviewer, keep the strictest verdict, and log "this expected answer is wrong" as a defect in the suite, not as noise.

Keep the gate small on purpose. The merge gate should be small enough that people trust it and fast enough that nobody routes around it. Everything else can run nightly or before release.

Demote saturated cases instead of deleting them. A case that has passed every run for months and never caught anything probably doesn't belong in the gate. Move it to the wider run, keep its history, and bring it back if that area starts changing again.

Retire cases in the same change that retires the behaviour. When a product rule changes, the evals that encoded the old rule should be updated or retired in the same pull request. Retiring a case should be a normal, reviewed act, not an act of courage.

Let the people closest to the rules run the suite. In my current organisation we built our LLM evaluation framework with a simple front end so that non-technical QA could run AI validation without writing code. That matters here for a reason beyond convenience. The people who know the business rules best are the ones who notice when an expected answer has quietly gone out of date. If only engineers can open the suite, only engineers can audit it.

What I don't know yet

I don't know the right retirement rate for an eval suite. With deterministic regression packs I built up a feel for it over years. With LLM evals, where the same case can pass and fail on the same build, the signal that a case has stopped earning its place is murkier, and I'd distrust anyone who claims a clean rule for it today.

I'd also be careful with the evidence I've used. SWE-bench is a public benchmark with contamination problems your private suite shouldn't have. Facebook's selection numbers come from deterministic code tests at an enormous scale. The AI Index figures describe the whole industry, not your product. None of them transfers directly. What does transfer is the pattern, and it's one I spent a good part of two decades cleaning up after.

Eval suites are still young enough that we can choose what they become. I'd rather teams spent a little effort now deciding how cases leave the suite than discover, a few years from now, that they've rebuilt the three-day regression pack with a token bill attached.

Continue reading

Sources