Notes on quality engineering.
Essays on engineering reliability, AI in software delivery, testing strategy, and building high-performing teams — from two decades in the field.
Why I Treat Agent Memory as Production Data
Agent memory is mutable state that outlives every eval run. Why the update and forget paths are where it breaks, and what I would test first.
Your Agents Doubled the PRs. Nobody Doubled the Reviewers.
AI agents doubled PR output, but review capacity didn't move. New 2026 data shows where the work went, why bot-only review fails, and how to triage agent PRs.
Agents Write the Code and the Tests. So Who Owns Quality Now?
How an AI-first engineering org should own quality: where humans sit in the agentic SDLC, what QA becomes, how to reskill, and which XP basics matter more.
Your AI Wrote the Tests From the Code. So It Tested the Bug.
LLM-generated tests trigger bugs but rarely catch them, because the assertions copy what the code does. Gate on oracles from the spec, not on coverage.
You Approved the MCP Tool Once. CI Never Checked If It Changed.
MCP servers are mergeable infrastructure. Pin tool-definition hashes, scan for poisoning and taint handlers, and fail CI when the contract drifts.
Your Agent Said No. Then It Did the Thing. Gate State, Not Transcripts.
Agents can refuse in the transcript and still harm the ledger. Put executable red-team cases in CI and fail on state-confirmed violations, not polite text.
Your LLM Judge Is Untested Software. Gate It Like the Rest of the Suite.
LLM-as-judge is another stochastic component with bias. Calibrate it with human agreement, bias probes, and pinned versions before it can block a merge.
The AI Productivity Lie: Your Engineers Merge Twice the PRs and Ship Nothing Faster
AI coding tools made typing faster, not delivery. What METR, DORA, Faros and GitClear data reveal about the AI productivity paradox, and what the genuinely faster teams measure instead.
Your Agent Eval Suite Is a Coin Flip. Treat It Like One.
A green LLM eval can be noise. Gate on pass rates and baselines, quarantine flaky cases, and fail the build when variance itself is the bug.
Production Failures Belong in Your Eval Suite: The Trace-to-Eval Loop
Staging goldens miss the failures users hit. Score sampled production traces, triage the real misses, and promote them into permanent regression cases.
Prompt Regression Testing: Treat Prompts Like Code in CI/CD
A prompt edit is a behaviour change. Pin versions, run golden datasets, and fail the build when quality slips — the same discipline as a contract test.
Your AI Agent Passed the Demo. Can It Pass a Release Gate?
A practical release scorecard for AI agents: verify outcomes, separate critical failures from averages, and compare versions without hiding retries.
AI-Generated Tests Need a Test of Their Own
Use mutation testing to challenge AI-generated assertions, investigate surviving mutants, and build a focused review gate without chasing a perfect score.
Your Agent's Tool Permissions Are Part of the Test Surface
How to test AI agent permissions with concrete abuse cases, backend evidence, approval integrity checks, and safe recovery after ambiguous tool results.
The Confident Wrong Answer: How to Check for Hallucination in AI Coding Agents
Hallucinated code doesn't fail loudly — it builds, passes review, and ships wrong. A five-layer verification ladder for catching phantom symbols, fabricated config, and green-theatre tests before they merge.
Does TDD Survive Inside an AI Agent's Loop? What the Data Actually Says
A 'TDD costs 8x more tokens' claim is spreading without a source. I traced it to a small Thoughtworks experiment, then rebuilt the experiment myself — the finding holds, but the metric overstates the bill by up to 4x.
Hiring for Judgement: Interviewing SDETs and QA Leads in the AI Era
Your Playwright coding round is now testing for a skill an LLM already has. Here's how I actually interview SDETs and QA leads now — the questions that surface judgement instead of syntax.
The QA Budget Conversation Nobody Prepares For
Nobody teaches quality engineers how to ask for money. A practical framework for justifying QA headcount and tooling spend to a CTO or CFO — without leading with test coverage.
From Senior QA to Director: The Skills Nobody Tells You About
The jump from senior QA engineer to director isn't a bigger version of the same job. It's a different job. Twenty-two years of lessons on what actually changes — and what to build before you're promoted, not after.
Vendor Evaluation for Test Tooling: A BS-Detection Framework
Every test automation vendor demo looks incredible on a clean sample app. Here's how I actually evaluate testing and AI-QA tools before they touch a procurement budget.
What a CTO Actually Wants to Hear About Quality
Test coverage percentages don't survive contact with a CTO. Here's the framework I actually use to talk about quality engineering ROI in a room full of people who don't care how many tests you have.
Beyond Playwright: What Actually Makes a Good SDET
Tool knowledge has an eighteen-month shelf life and a falling floor price. The eight skills that actually separate a good SDET from someone who writes automated tests.
AI Safety and Validation Engineering for Financial AI: A Practitioner's Field Guide
Safety for financial AI isn't a policy document — it's evaluators, adversarial suites, explainability, telemetry, and circuit breakers running in production. A field guide to building that system.
Building Engineering at Scale: The 0-1-n Quality Problem
Most teams crack 0-to-1. Few survive 1-to-n. Two decades of lessons on building quality at scale across banks, fintechs, and enterprise product companies.
From Red-Green-Refactor to Living Documentation: The Shift to Spec-Driven Development
TDD gave us confidence. BDD gave us collaboration. Spec-driven development gives us a single source of executable truth for product, engineering, and QA.
When Efficiency Becomes the Enemy
How software engineering's obsession with doing things faster quietly broke the thing it was supposed to serve.
TDD is a Dying Art: Myth or Reality?
TDD was Agile's cornerstone. Now some call it a dying art. Is it really fading, or just changing shape — and what does that mean for modern software teams?
The Self-Healing Test Myth
Why most "self-healing" automation is just expensive flakiness management, and what actually works for maintainable tests.
AI Token Costs in CI/CD
How to govern LLM token costs when AI enters your testing pipeline. Lessons from real CI/CD deployments and a framework for cost-aware AI integration at scale.
QA is a Mindset, Not a Role
How to build a quality culture where developers own testing and QA engineers enable rather than gatekeep—lessons from leading high-performing engineering teams.
Why I'm Skeptical of Low-Code Testing Tools
The hidden costs of low-code and record-and-playback test automation—why most tools create more problems than they solve, and when they actually make sense.
The QA Role in Agile Projects
How the QA role fits into agile teams — from acceptance criteria and shift-left testing to continuous feedback and owning quality across the sprint.
Go deeper
Practical frameworks and playbooks that accompany the writing — free, with the full indexed content available on the web.
Prompt Engineering for QE
50+ production-tested AI prompts for test generation, failure triage, BDD, and CI/CD integration.
QE Maturity Framework
Benchmark your QE org across five maturity levels. Includes DORA metrics, contract testing, and a tech stack guide.
QA Transformation Roadmap
A 90/180/360-day execution playbook with workstreams, SLOs, risk register, and editable templates.
The Quality Signal
A periodic newsletter on quality engineering, AI in delivery, and engineering leadership. No noise.