Part 1 of 5 in The QA Director's Playbook — a series on running quality engineering as a leader, not just a tester.
I have sat in more quarterly business reviews than I care to count, and I have watched the same slide fail in the same way more times than should be possible. It's the one with test coverage on it. "We're at 87% automated coverage, up from 79% last quarter." The CTO nods, says "great," and asks the next person about deployment frequency. Nobody follows up. Nobody remembers the number an hour later. It didn't land, because it was never a business sentence — it was an engineering sentence wearing a business slide.
CTOs do not think in coverage. They think in exposure, velocity, and cost — the three things a board actually asks them about. If your quality reporting doesn't translate into one of those three, it is noise, however accurate it is.
Why "we have good coverage" doesn't work
Coverage tells you what was tested. It says nothing about what would happen if the untested 13% broke, what it would cost, or who would notice first. A CTO's real question, whether or not they phrase it this way, is: if something goes wrong, how bad is it, and did we already know it could happen? Coverage answers a different question than the one being asked, so it gets nodded at and discarded.
I learned this the hard way early in my career, presenting a beautifully detailed automation dashboard to a CTO at a bank who cut me off after ninety seconds with: "Pankaj, will this stop us getting fined." That was the whole conversation, reduced to one sentence, and it was the correct question. I didn't have a good answer that day. I built one afterwards.
The three things a CTO is actually asking
Every quality conversation with an executive is really one of three conversations wearing a testing costume: risk, speed, or money. Figure out which one you're in before you open the deck.
Risk. What is the blast radius if this fails, and did we know about it in advance? This is the conversation for regulated systems, payment flows, anything with a compliance officer attached. The unit here is not a percentage — it's an incident category and a dollar or reputational figure.
Speed. Is quality slowing us down or letting us go faster? This is the conversation when a CTO is under commercial pressure to ship. The answer that lands is about lead time and change failure rate, not about how many assertions ran overnight.
Money. What does this cost, and what does it save? This is the budget conversation, and it deserves — and gets — its own article in this series. But even outside a formal budget ask, every quality metric a CTO hears gets silently converted into a cost question in their head. Answer it before they have to ask.
Practical move: before your next leadership update, write one sentence for each of the three lenses about the thing you're about to report. If you can't fill in all three, you're not ready to present it — you're ready to keep investigating it.
The four numbers that actually survive a QBR
I've stopped bringing coverage percentages to executive rooms. What I bring instead:
- Escaped defect rate — critical issues that reached production, per release. This is the single number that maps directly to "risk," and it's the one number every CTO already understands without translation.
- Change failure rate — the DORA metric, not a QA-invented one. Using an industry-standard metric instead of a bespoke QA number is itself a credibility signal; it tells the room you're not grading your own homework.
- Mean time to detect — how long between a mistake happening and someone finding out. This is the "speed" conversation in one number, and it's the number that predicts whether your next incident will be a five-minute rollback or a weekend.
- Cost per prevented incident — an estimate, clearly labelled as one, of what a class of defect would have cost in production versus what it cost to catch pre-release. This is the number that turns quality from a cost centre into a P&L line, and it's the number that gets you budget without a fight.
Everything else — flakiness rate, test count, sprint velocity of the QA team — is a leading indicator that belongs in your own operating rhythm, not in front of a CTO. Leading indicators are how you manage the function. Lagging, business-denominated numbers are how you report on it.
Bring the trend, not the snapshot
A single number in isolation invites a single question: "is that good?" A trend line answers a better question on its own: "is this getting better because of something we did." Every one of the four numbers above should show up as a quarter-over-quarter trend, ideally with one annotation marking a specific intervention — "MTTD dropped from 6 hours to 90 minutes after we shipped the contract-testing gate in March." That sentence does more work than any coverage dashboard, because it ties a number to a decision the CTO can attribute to your function.
Say the sentence a business risk officer would say
The best quality leaders I've worked alongside — across trading floors, card networks, and now a listed real estate developer — all have one habit in common: they translate before they present. Not "we found 40 P2 defects this sprint." Instead: "we prevented an estimated three production incidents this quarter, saving roughly six engineering-days of firefighting and avoiding one customer-facing outage class." Same underlying data. Completely different reception.
Practical move: take your last three status updates to leadership. For each one, rewrite the headline sentence using only the words risk, cost, time, or customer — no testing vocabulary allowed. If you can't do the rewrite, the update wasn't ready for that audience yet.
Where this connects to budget
Once a CTO trusts your risk and speed numbers, the money conversation gets dramatically easier — you're not starting from zero credibility every time you ask for headcount or tooling spend. I go into exactly how to structure that ask, including where most quality leaders undersell themselves, in the next piece in this series.
Related reading
- Building Engineering at Scale: The 0-1-n Quality Problem — the quality index framework this article's four numbers are drawn from.
- The QA Budget Conversation Nobody Prepares For — part 3 of this series.
- AI Token Costs in CI/CD — a worked example of translating a technical cost into a number leadership can act on.