For two decades, quality in software meant test coverage, regression suites, and a clean audit trail. Those things still matter. But when a lending decision is shaped by a gradient-boosted model, and a customer query is answered by a retrieval-augmented LLM, the old definition of "does it work?" quietly stops being sufficient. The question is no longer is the code correct — it's is the model behaving, and can we prove it.
Answering that question well is an engineering discipline in its own right. Not a policy, not a committee, not a PDF of principles — a running system of evaluators, guardrails, telemetry, and feedback loops. This is a field guide to how that system is built.
Safety is code, not a sentence
The instinct in most organisations is to govern AI with a document. Write a framework, define some principles, hold a monthly forum. That produces governance theatre — an impressive amount of paperwork and very little that actually constrains a model in production.
The shift worth internalising is that safety becomes an engineering concern. You don't write a principle that says "the model should not hallucinate." You define a hallucination rate as a measurable signal, instrument the pipeline to compute it, and wire an automated circuit breaker that trips when it crosses a threshold. The control isn't a sentence in a policy. It's a running system that can fail a deployment on its own.
Everything below follows from that distinction.
1. Evaluation frameworks that survive contact with reality
Predictive ML and generative AI fail in completely different ways, so a single rubric won't do.
A classifier drifts as the world shifts underneath its training distribution. A RAG system retrieves the wrong context and answers confidently anyway. An agentic workflow takes an action nobody scoped. Building an evaluation framework across these archetypes means accepting that "accuracy" is only the beginning.
In practice you need two layers of checks running side by side:
- Deterministic assertions for the logic that must never vary. Did the agent call the tool with a valid schema? Did the RAG answer cite a real, retrievable source? Did the output respect the required format? These are pass/fail and should be treated like unit tests.
- Probabilistic scoring for behaviour that inherently will vary. Faithfulness of a generated answer to its retrieved context, semantic relevance, tone, refusal correctness. These need graded rubrics, not booleans.
For the probabilistic layer, LLM-as-a-judge is the workhorse — a strong model scoring outputs against an explicit rubric. Two disciplines make it trustworthy rather than a source of noise:
- Pin the rubric. Vague criteria ("rate helpfulness 1–10") produce unstable scores. Decompose into concrete, near-binary sub-questions ("Does the answer contradict the retrieved context? Y/N") and aggregate. You get far better inter-run consistency.
- Calibrate the judge. Hold out a human-labelled set and periodically check that judge scores track human judgement. An uncalibrated judge is just a confident random number generator.
For high-volume or latency-sensitive checks, distil the judge into a small, purpose-built evaluator model — a fine-tuned classifier that runs cheaply inline rather than calling a frontier model on every request. Frameworks like Ragas, Giskard, and Arize give you scaffolding for RAG-specific metrics, vulnerability scanning, and drift monitoring respectively, but the rubric design is yours to own.
A useful mental model: offline evals gate the release; online evals gate the response. The same rubric philosophy applies to both, but online evaluators must be cheap enough to run in the request path.
2. Adversarial testing as a first-class activity
The most valuable test case is the one the product team hoped you wouldn't write.
Happy-path evals tell you a model works when everyone behaves. Adversarial evals tell you what happens when they don't — and in production, they won't. Design the adversarial path deliberately:
- Prompt injection — instructions smuggled into retrieved documents, user inputs, or tool outputs that try to override the system prompt.
- Jailbreaks — role-play framing, obfuscation, and multi-turn escalation aimed at bypassing guardrails.
- Boundary and malformed inputs — empty context, contradictory retrievals, inputs at the edge of the token window, injection-shaped payloads in structured fields.
- Data exfiltration attempts — probes designed to make the model leak system prompts, other users' data, or internal context.
Maintain these as a growing, versioned adversarial suite — a red-team corpus that every model version must run against before promotion. Crucially, treat every guardrail violation in production as a new test case. Cluster violations by signature and emerging attack patterns reveal themselves before they become incidents. Red-teaming with a feedback loop attached beats a one-off penetration exercise every time.
3. Explainability that stands up in a fairness review
This is the pillar that separates high-stakes AI from everything else. When a model contributes to a lending or fraud decision, "the algorithm decided" is not an acceptable answer to a regulator, and it is a genuinely harmful one to a customer who was declined.
Two techniques do most of the work:
- Shapley values (SHAP) attribute a prediction to its input features using a game-theoretic decomposition — each feature's contribution is its average marginal effect across coalitions of other features. For tabular credit and fraud models, SHAP gives you both local explanations (why this applicant was declined) and global structure (which features drive the model overall).
- Integrated Gradients attribute predictions by integrating the model's gradients along a path from a baseline input to the actual input. It's the natural choice for differentiable models — deep networks over text, embeddings, or images — where SHAP's sampling gets expensive.
The point of these in a financial context isn't decoration. It's fairness auditing. A model can hit its accuracy target while quietly using a feature that acts as a proxy for a protected characteristic — postcode standing in for ethnicity, for example. Attribution methods surface that dependency. Pair them with subgroup performance analysis (compare false-positive and false-negative rates across demographic slices) and you can detect unfair bias rather than assert its absence. Explainability stops being a nice-to-have the moment a protected characteristic could be a hidden proxy in your feature set.
4. Telemetry and observability, under privacy constraint
You cannot validate what you cannot see.
That means telemetry pipelines capturing the raw material of model behaviour: inputs, outputs, retrieved context, tool calls, latencies, and embeddings — vector representations of inputs and outputs that let you measure semantic drift and cluster behaviour that exact-match logging would miss.
The engineering tension here is real, and it is sharpest in regulated domains: the thing you're observing is often personal data. Rich enough signal to catch a failing model, tight enough control that the observability layer doesn't itself become a breach. Practical measures:
- Tokenise or hash PII at ingestion; store embeddings and derived signals rather than raw sensitive text where you can.
- Scope retention — high-cardinality raw logs age out fast; aggregated signals persist.
- Isolate the telemetry store with its own access controls, treated as sensitive infrastructure in its own right.
On top of this raw stream you compute golden signals — the small set of metrics that actually indicate health:
- Hallucination / faithfulness rate — fraction of outputs unsupported by retrieved context.
- Semantic similarity — drift of outputs (or inputs) away from an expected distribution, measured in embedding space.
- Groundedness and citation validity for RAG.
- Refusal and guardrail-trigger rates — sudden movement in either direction is a signal.
- Latency and cost — degradation here is often the first symptom of a deeper problem.
5. Circuit breakers, and the human in the loop
Signals are only useful if something acts on them.
Wire automated circuit breakers to your golden signals: when hallucination rate breaches a threshold, when semantic drift spikes, when guardrail violations cluster, the system degrades gracefully on its own — falling back to a safer model, a canned response, or human routing — rather than continuing to serve a failing model while someone reads a dashboard. This is the difference between an observability system and a safety system. One tells you what happened; the other prevents the next thousand bad responses.
Then close the loop. Human-in-the-loop corrections — reviewer overrides, flagged outputs, resolved incidents — are the highest-quality training signal you have. Feed them back into refinement: hard cases become new eval cases, correction patterns inform fine-tuning, and recurring failure modes become permanent regression tests. The model gets safer with every cycle rather than drifting.
The final, underrated craft is translation. Raw telemetry — a chart of embedding distances — is not consumable by the people accountable for risk. They need a model health report that says, in plain terms, what is safe, what is degrading, and what needs a decision. Producing that intelligence is as much a part of the discipline as writing the evaluators, and engineers routinely underestimate it.
The through-line
Strip away the specific techniques and one principle connects all five pillars: safety is not something you inspect for at the end — it is something you engineer in from the first commit.
Evaluation frameworks catch what breaks. Adversarial suites catch what someone makes break. Explainability proves the decision was fair. Telemetry makes behaviour visible. Circuit breakers and human feedback turn visibility into action. Built together, they form a system that doesn't just describe safety but enforces it — continuously, in production, and in a form you can put in front of anyone whose job is to doubt you.
That's the work. It's less glamorous than building the model and far more consequential than governing it on paper.