A lot of my career has been spent testing around models that make decisions about money. A credit decisioning platform, risk scoring, Monte Carlo portfolio simulations, performance attribution. When I started working on AI products, I expected to feel like a beginner again. What I felt instead was déjà vu.

So here is my position. Most of what AI teams now call "LLM governance" is a smaller, less mature version of something banks have done for years, which is model risk management. We are not facing an entirely new problem. We are facing an old problem with a faster clock and a vendor in the middle. I think QA leaders in regulated industries, and honestly outside them, should steal the old discipline before inventing a new one.

I also want to be fair to the other side. Bank-style model governance has a reputation for paperwork, committees and six-month approval cycles, and I understand why AI teams run from it. I am not arguing for the paperwork. I am arguing for the questions underneath it.

What the old discipline looked like from a tester's chair

I was the first QA hire on a Tier 1 bank's lending platform, where automated credit decisions were backed by financial models, credit risk scoring and quantitative strategy. Earlier, at another bank's wealth and asset management arm, I led quality for a platform whose job was to simulate portfolio risk and attribute performance, so my team validated data integrity and the quantitative parameters feeding those calculations.

I won't pretend I was the model validator. I was the person who had to make a platform around the models trustworthy enough to release. But sitting that close teaches you a few habits that I now notice are missing in a lot of AI work.

You do not test a model once. It was always assumed that the world the model saw would drift away from the world it was built for. The model had an owner. Someone other than the person who built it was expected to look at it critically. And people wrote down what the model was and wasn't meant to be used for, which sounds dull until the day someone wants to reuse it for something else.

None of that is exotic. It is simply the habit of treating a model as something that changes and can hurt you, rather than as a component you integrate and forget.

What the guidance actually says

Since I'm going to lean on it, I read the primary text. In April 2026 the US banking agencies (the Federal Reserve, the OCC and the FDIC) issued revised guidance on model risk management. It replaces SR 11-7, the 2011 letter that most of the industry has built its practice around. In the UK, the Prudential Regulation Authority has its own version, Supervisory Statement 1/23, with five principles that include independent model validation.

The diagram below paraphrases six principles from the US text. Four of them are worth a QA leader's attention.

Table mapping six principles from the US model risk management guidance (inventory, risk-based tiering, validation before first use, effective challenge, change triggering re-validation, and ongoing monitoring including vendor models) to what each means for an AI feature

Effective challenge. The guidance defines it as critical analysis by objective experts who have the expertise, the independence, and "the organizational standing and influence to effect any change". That last clause matters. A reviewer who can't stop a release is a commentator.

Validation, and what triggers it. Validation "generally occurs prior to a model's first use", and how often it happens depends partly on "frequency and scope of model changes". In other words, a change in the model is a reason to validate again.

Ongoing monitoring. The text asks whether a model is "performing as expected given potential changes in products, exposures, activities, clients, data relevance, or market conditions". That is production monitoring against reality, not a one-off test at launch.

Vendor models. Third-party models are not exempt. Sound practice, it says, includes ongoing monitoring and outcome analysis to assess whether vendor models are accurate and "remain fit for purpose".

Now the surprise. A footnote in the scope section says that generative AI and agentic AI models "are novel and rapidly evolving" and are therefore "not within the scope of this guidance". It adds that a bank's own risk management and governance practices should decide what controls fit those tools. The principles, it says, apply to traditional statistical models and to non-generative, non-agentic AI.

I read that as neither a free pass nor a gap in the regulator's thinking. I read it as an honest statement that nobody has written the template yet. For a QA leader in a bank, it means that when the business asks "how do we govern our LLM feature?", the answer is not in a document you can download. You are being asked to write the control framework, and the closest thing to a foundation is the discipline for the models that came before. Two caveats on the source: it is US guidance, aimed mainly at institutions above $30 billion in assets, and it states that it does not set enforceable standards. The PRA's statement formally applies to UK banks with internal model approval, though its principles are written to cover all model types regardless of technology, vendor models included.

The vendor changed the model and nothing threw an error

If I had to pick one reason this discipline matters for AI, it is the vendor in the middle. A bank that builds a credit model controls when it changes. A team calling a hosted LLM does not.

The best-known measurement of this is a 2023 study by Lingjiao Chen, Matei Zaharia and James Zou, published in the Harvard Data Science Review in 2024. They ran the March 2023 and June 2023 versions of GPT-4 and GPT-3.5 through the same tasks and compared. I'll use only a few of their results, and I'll explain each in plain terms.

Horizontal bars comparing GPT-4 in March and June 2023: instruction fidelity 99.5% to 0.5%, directly executable code 52% to 10%, multi-hop agent exact match 1.2% to 37.8%, medical licensing exam accuracy 86.6% to 82.4%

On a simple formatting instruction, "answer yes or no within square brackets", GPT-4 complied 99.5% of the time in March and 0.5% in June. On code generation, 52% of GPT-4's outputs were directly executable in March, against 10% in June. The medical exam score moved a few points. And on a multi-hop question-answering agent built with a popular framework, exact-match accuracy went up, from 1.2% to 37.8%.

Read that last number twice, because it is the point. The change was not a decline. It was a change, and it went in different directions on different tasks. A team that watched only one headline benchmark could easily have seen a flat or improving line while its pipeline broke.

The code result is the one I'd show to a sceptical engineering manager, because the explanation is so ordinary.

Bar chart of GPT-4 code accepted by an online judge: March 52% as returned and 52% after removing surrounding text; June 10% as returned and 70% after removing surrounding text

The June model had started wrapping its code in markdown code fences (extra triple quote marks). The code itself was fine. Strip the extra text and 70% passed, better than March. But any pipeline that took the output and ran it as returned would have fallen over, and nothing in the model, the API or the vendor's status page would have told you.

A bank validation team would call this a model change with a downstream impact and would expect it to be caught before it reached production. In many AI teams I'd expect it to be found by a customer, or by an on-call engineer at an awkward hour.

I should be careful about what the study does and does not show. Arvind Narayanan and Sayash Kapoor wrote a well-known critique in July 2023 arguing that it showed a change in behaviour, not a loss of capability, and that some of the original method, especially the prime-number test in the first version, was questionable. I agree with the distinction, and it is why I haven't quoted the headline prime-number figure. But notice that the distinction doesn't help the team downstream. If a formatting habit changes, your parser doesn't care whether the underlying capability survived. These are also 2023 snapshots of one provider's models. I have not measured how stable current models are, and I would be wary of anyone who claims either that they are now rock-solid or that nothing has improved.

Where the banking analogy breaks

I would be selling you a neat story if I stopped there, so here is where I think it stretches too far.

A credit model produces a number, and the bank eventually learns whether the borrower repaid. That makes outcomes analysis possible, with enough patience. An LLM answering open questions rarely has a clean, delayed ground truth. You can sample and judge, but the feedback loop is looser and the judge is itself imperfect.

The guidance also defines a "model" as a quantitative method that turns inputs into quantitative estimates, and explicitly excludes deterministic rule-based software. A lot of what surrounds an LLM, the harness, the parsers, the tool calls, is ordinary software and needs ordinary testing, not model validation. I'd rather not blur the two.

And I would be wary of the cost. Model risk management in a big bank can be slow and heavy, and the guidance itself leans on risk-based proportionality: the more material the model, the more rigour. I'd take that part literally. An internal summariser that drafts meeting notes and a customer-facing assistant that explains fees do not deserve the same process, and a team that applies one heavy process to both will end up with neither well governed.

Where I'd start

I wouldn't copy a framework. I would ask a small number of questions, in roughly this order.

What do we depend on, who owns it, and how much does it matter? In my experience, few AI teams can list every model, prompt and tool behind a feature, with versions and a named owner. Start there, because nothing else is possible without the list. Then tier it: customer-facing, regulated and money-moving features get the strictest treatment, and everything else gets lighter checks.

Is a model change a release? I'd treat a provider's version change, deprecation or silent update exactly like one of our own deployments: pin versions where the vendor allows it, and re-run checks that include the dull ones. Output format, schema adherence, refusal behaviour and tool-call shape, not only an accuracy score. The study above would have been caught by a formatting check long before a quality metric noticed.

Who challenges the people who built it? Someone who didn't build the feature should try to break it and should have the standing to say no. This is the part I'd protect hardest, and it is also the part most often squeezed when a team is told to move fast. In a small team it can be a rotation. It doesn't need to be a department.

What happens in production, and what is the feature not for? Sample real outputs, compare them against what actually happened, set thresholds and decide in advance who is called when they are crossed. This is the same idea as the trace-to-eval loop I've written about before. And write down the limits of the feature. I'd bet that is the cheapest control on this list and the one most often skipped.

What I don't know yet

I don't know how this will settle. Regulators have said, in effect, that generative AI will be dealt with later, and "later" could look like a separate framework, a stretched version of the current one, or a patchwork of supervisory expectations. My suggestion to borrow the existing principles may look overcautious in a year, or it may look obvious.

I also don't know how much the vendor-change problem has improved. Even in 2023 providers offered dated snapshots, though Narayanan and Kapoor noted they were kept for only a few months. I haven't checked how current practice compares, and I haven't seen a recent independent study that does. The 2023 evidence is the best I can point to, not the last word.

What I'm more sure about is smaller. Who owns the model, what happens when it changes, and who gets to say no are questions banks have been answering, imperfectly and slowly, for a long time. AI teams don't have to rediscover them one incident at a time.

Continue reading

Sources