Part 4 of 5 in The QA Director's Playbook — a series on running quality engineering as a leader, not just a tester.
I've sat through a lot of test tooling demos over two decades, and they all share one structural flaw: they run on the vendor's sample app, with the vendor's clean test data, on the vendor's stable network, in front of an audience that hasn't seen the failure modes yet. Every tool looks incredible under those conditions. The AI-QA category has made this worse, not better — "self-healing," "autonomous testing," and "agentic QA" are now on every deck, and I've written before about why most self-healing claims collapse under scrutiny and why I've been sceptical of low-code record-and-playback tools for the same underlying reason: the demo is optimised for the room, not for your system.
Evaluating vendors well is a leadership skill, not a technical one — the technical evaluation is usually the easy part. Here's the framework I actually run.
Rule one: never evaluate on their environment
Every serious evaluation gets a proof-of-concept slot against your messiest real service, not their curated sandbox. Pick the one with the worst test data, the flakiest environment, the most legacy quirks. If a vendor resists this — "let's start with something simpler to show the value first" — that resistance is itself the finding. A tool that only works on simple systems is a tool that doesn't work.
Practical move: before any demo is scheduled, send the vendor your actual criteria and your actual test target. Watch how the sales conversation changes once "simple demo app" becomes "our real payments reconciliation service." The good vendors lean in. The rest find reasons to delay.
Rule two: make them show you a failure, not just a success
Every demo shows the tool working. Ask them to show you the tool failing — a locator it couldn't heal, a flow it couldn't generate a sensible test for, a case where its AI assertion was wrong. Watch two things: whether they have an honest answer ready at all, and what the failure mode actually looks like when it happens. A tool that fails loudly and visibly is safer than one that fails silently and reports green.
The question that separates a real evaluation from a sales pitch is never "does it work." It's "what does it do when it doesn't work, and will I find out before my customers do."
Rule three: price the maintenance, not the licence
Vendor pricing sells you the licence. Nobody's pricing deck shows you the engineering hours your team will spend maintaining the integration, retraining the model on your app's quirks, or reviewing the AI-generated output it produces every week. I ask every vendor directly: "what does a customer's engineering team typically spend per month keeping this running well, once the honeymoon period is over?" The good ones have a real number, because they track it with existing customers. The ones who deflect to "it's basically maintenance-free" are the ones whose reference customers I'd want to call directly, off the vendor's provided list.
What to actually calculate before signing anything:
- Licence cost, obviously.
- Estimated engineering hours per month to review/curate AI output, based on the pilot.
- One-time integration cost against your CI/CD and existing test infrastructure.
- Switching cost if it doesn't work out in twelve months — this one is almost never on the vendor's slide, and it should be on yours.
Rule four: talk to a reference customer who churned
Every vendor gives you their best reference customers. Ask, directly, for one that stopped using the product, or ask their sales rep to connect you with someone from a similar-sized org in your industry who isn't on the case-study page. Most won't produce one. That's fine — go find one yourself, through your network or a QA leadership community, rather than through the vendor's funnel. A five-minute conversation with someone who churned tells you more than an hour of demo.
Rule five: separate the AI claim from the testing claim
This is the one that's specific to 2026. A lot of tools now bundle a genuinely useful testing capability with an AI claim that's doing far less work than the marketing implies. Ask specifically: what does the AI component actually decide, versus what is deterministic logic wearing an AI label? I ask vendors to walk through one real example end to end — what input the model saw, what it output, and what would have happened with a simpler rule-based approach instead. If they can't explain that distinction clearly, either they don't understand their own product, or there's less "AI" in it than the pitch suggests. Neither is disqualifying on its own, but both mean you should price and evaluate the tool as what it actually is, not what the deck calls it.
The scorecard I actually use
For every serious vendor evaluation, I score across five dimensions, each 1–5, scored by the engineers who ran the pilot — not by me, and not by whoever owns the vendor relationship:
- Signal quality — when it fails, is the failure meaningful or noise?
- Transparency — can we see and understand why it made a decision?
- Maintenance burden — realistic monthly engineering hours to keep it healthy.
- Failure visibility — do we find out about a tool failure before or after a customer does?
- Exit cost — how hard is it to leave in twelve months if it doesn't work out?
A tool that scores a 5 on the sales deck and a 2 on this scorecard is not a tool worth taking to your next budget conversation — and if you do get funding approved for it (see the budget framework), you'll be the one explaining the shortfall in six months, not the vendor.
Related reading
- The Self-Healing Test Myth — the deeper technical version of rule two, applied to one specific category of claim.
- Why I'm Skeptical of Low-Code Testing Tools — the same evaluation instinct, applied earlier in this site's archive.
- The QA Budget Conversation Nobody Prepares For — part 3 of this series, for pricing the ask once a tool clears this bar.