Part 2 of 5 in The QA Director's Playbook — a series on running quality engineering as a leader, not just a tester.
Most QA hiring loops still run a live-coding round: write a Playwright test against a sample page, fix the flaky locator, explain the Page Object Model. It was a reasonable filter in 2022. It is close to useless now, because the thing it screens for — can this person produce correct-looking test code under time pressure — is exactly the thing generative tools do well and cheaply. I wrote about why tool fluency has a falling floor price from the candidate's side of that equation. This is the hiring manager's side: if your loop still filters on syntax, you are optimising for a skill your eventual hire will barely need, and missing the one they will.
What you're actually screening for now
The skill that doesn't commoditise is judgement — the ability to look at a system, a risk, or a piece of AI-generated test code and know whether it's any good. That's harder to interview for than "write a test," because it doesn't have a single correct answer you can pattern-match against. It requires you to listen to reasoning, not just check outputs.
Practical move: before you write the next job description, cut the phrase "proficient in Playwright/Cypress/Selenium" from the requirements and replace it with "can evaluate whether AI-generated tests are worth keeping." Watch how few candidates' CVs actually address that sentence — that gap is your interview.
The four rounds I actually run
1. The risk-triage round, not the coding round
Give the candidate a real (anonymised) incident from your own systems — a description of what broke, what the test suite did and didn't catch, and what shipped anyway. Ask them to tell you, out loud, what they would have tested and why, and what they would not have bothered testing, and why not. The "why not" answer is the one that tells you everything. Weak candidates list more things to test. Strong candidates draw a line and defend it.
This maps directly to the risk-reasoning skill I look for on the IC side too — see point one of what makes a good SDET — except here you're evaluating whether the candidate can teach that judgement to a team, not just apply it themselves.
2. The AI-code review round
Show the candidate a chunk of AI-generated test code — genuinely AI-generated, not hand-crafted to be obviously bad — and ask them to review it as if it were a pull request. Good candidates immediately start asking what the test is actually asserting versus what it looks like it's asserting, whether the fixtures are deterministic, and whether this is testing the requirement or testing the implementation. Weak candidates check the syntax and approve it, because it "looks like a proper test."
This is the single highest-signal round I run now, and it takes fifteen minutes. It answers the question that actually matters in 2026: can this person curate a firehose of machine-generated tests down to the ones worth keeping?
3. The "defend the trade-off" round
Present a resourcing constraint — "you have three sprints and can either build contract tests for the payments service or add end-to-end coverage for the new onboarding flow, not both." Ask them to choose and defend it to you as if you were a sceptical engineering director. This tells you whether they can hold a position under pushback, which is the exact skill they'll need the first time a delivery lead tells them a release is going out with or without their sign-off.
The candidates worth hiring don't say "it depends." They say "here's what I'd do and here's the specific scenario that would change my mind" — and they mean it.
4. The domain round, tailored to you
Ten minutes on your actual domain — trading, payments, healthcare, whatever it is. Not trivia. A scenario: "a customer disputes a transaction that our system shows as settled — walk me through where you'd look." You are not testing whether they already know your domain. You are testing whether they reason like someone who could learn it quickly, and whether they get curious or defensive when they hit the edge of their own knowledge.
What I stopped asking
- "Write a test for this login page." Replaced by round 2. It tested syntax; I need to know they can spot bad reasoning in someone else's (or something else's) test.
- "What's the difference between smoke and regression testing?" Definitional trivia. If they don't know it, round 1 will expose that faster and more usefully than a flashcard question.
- Take-home assignments that mirror the job exactly. These now get finished substantially by an LLM in twenty minutes regardless of the candidate's actual level, and they tell you almost nothing about judgement. If you use a take-home at all, make the deliverable a written critique of a flawed test suite, not a fresh one.
The trap in the other direction
None of this means penalise AI use. A candidate who says "I'd use an agent to generate the first pass of assertions, then I'd audit them against these three criteria" is giving you the correct 2026 answer. The candidate to be wary of is the one who either refuses to use AI on principle, or trusts its output without a review step. Both are judgement failures, just pointed in opposite directions.
Practical move: add one explicit question to every loop — "walk me through how you'd use AI in this role, and where you specifically wouldn't trust it." The specificity of the second half of that answer is the whole signal.
Hiring for QA leads, not just ICs
Everything above still applies, plus one more round: ask them to critique a real (anonymised) piece of your own quality reporting — a dashboard, a QBR slide, a metrics doc. Do they reach for coverage percentages, or do they reach for the risk/speed/cost framing I cover in what a CTO actually wants to hear? A lead candidate who critiques your reporting well on day zero of the interview process is telling you exactly how they'll represent the function once they're inside it.
Related reading
- Beyond Playwright: What Actually Makes a Good SDET — the eight skills this hiring loop is designed to surface.
- What a CTO Actually Wants to Hear About Quality — part 1 of this series.
- AI Safety and Validation Engineering for Financial AI — for domain rounds involving AI-adjacent systems specifically.