There is a sentence I hear in almost every AI product review now: "It won't hallucinate, it's grounded in our documents." I understand why people say it. I also think it is the most comfortable half-truth in AI testing at the moment, and it quietly changes what teams believe they need to test.

Retrieval-augmented generation, RAG for short, is the pattern where the system first searches your own content for relevant passages and then asks the model to answer using them. It is a good pattern. I own quality for an in-house conversational AI product built on it, and I would choose it again. But grounding does not make the model honest. It gives the model better material to be wrong with, and it adds a whole pipeline in front of the model that can fail without anyone noticing.

So my position is simple. RAG reduces hallucination. It doesn't cure it. What it mostly does is move the bug upstream, into the documents, the search and the context, and most test suites are still pointed at the last step.

Why the belief is half right

To be fair to the belief, the evidence does say grounding helps. The best study I know on this is from a Stanford team who tested commercial legal research tools in 2024. These are products built by companies with enormous, curated legal databases, which is about as good a retrieval corpus as anyone will ever have. The researchers wrote 202 legal questions, preregistered them so they couldn't cherry-pick, and had each answer checked by hand for whether it was correct and whether the cited source actually supported it.

The grounded tools did beat plain GPT-4. GPT-4 on its own hallucinated on 43% of the questions. The most accurate grounded tool, Lexis+ AI, hallucinated on 17%. That is a real improvement and I won't pretend otherwise.

Stacked bars showing accurate, incomplete and hallucinated response shares for Lexis+ AI, Westlaw AI-Assisted Research, Ask Practical Law AI and GPT-4

Now read the same chart as a QA leader. Each of the grounded tools hallucinated on somewhere between 17% and 33% of questions, at a time when vendors were marketing them as eliminating or avoiding hallucination. Westlaw's tool made something up on a third of them. Ask Practical Law AI hallucinated less often mainly because it declined or gave incomplete answers on 63% of questions, which is its own kind of failure if a lawyer is paying for answers.

The definition the researchers used is the one I'd want on any RAG product: an answer is hallucinated if it says something false, or if it claims a source supports a statement when it doesn't. That second half is the one grounding makes worse, not better. A wrong answer with a real citation attached looks more trustworthy than a wrong answer with none.

Where it breaks

The reason grounding can't cure hallucination is that it adds steps, and every step can fail on its own. A team at Deakin University wrote up three RAG systems they had built (one for research, one for education, one biomedical) and listed seven distinct places they had seen them go wrong. I find that list more useful than any accuracy score, because each item points at a different part of the system and therefore a different test.

Diagram of the RAG pipeline in four stages: corpus, retrieval and ranking, context assembly and generation, with the seven failure points placed under each stage and a suggested test for each

Only four of the seven are about the model. The first is that the answer simply isn't in your content, and the system makes one up anyway instead of saying it doesn't know. The second is that the right document exists but wasn't ranked highly enough to be picked. The third is that it was picked but got dropped when the system squeezed results into the prompt. Only then do you reach the model's own failures: missing the answer that is sitting in front of it, ignoring the requested format, answering at the wrong level of detail, or leaving half the answer out.

The authors draw two conclusions I agree with completely. A RAG system can only really be validated while it is running, because the questions users ask are not the ones you planned for. And its robustness evolves over time rather than being designed in at the start. That is a description of a living data pipeline, not of a model you test once.

There is one more failure that surprised me when I first read about it, and it lives between the second and third stages. A well-known 2023 study led by Stanford researchers gave models a question plus 20 retrieved documents, exactly one of which held the answer, and moved that document around.

Line chart of GPT-3.5-Turbo accuracy by position of the relevant document among 20, falling from 75.8% at first position to 53.8% at tenth, against a closed-book line at 56.1%

With the answer in first place, GPT-3.5-Turbo got 75.8% right. With it buried in tenth place, accuracy fell to 53.8%, which is lower than the 56.1% the same model scored with no documents at all. Retrieval had found the right material. Including it in the middle of a long context was worse than not retrieving it.

I want to be careful with this one. It is 2023 data on 2023 models, and long-context handling has improved a lot since. I would not quote the exact numbers about any model you are running today. What I take from it is the principle: whether the right document was retrieved and whether the model used it are two different questions, and you need a test for each.

This is a data problem, and QA already knows data problems

This is the part where my own background makes me impatient with how RAG testing is usually done.

A lot of my career has been spent testing data-heavy systems in banks: validating data integrity and quantitative models on an asset management platform, building the test strategy for management information reporting that drew on more than twenty upstream systems, and ETL and data quality validation on several programmes since. My experience of that work is that when a report is wrong, the defect is very often not in the report. It is in a feed, a mapping, a stale reference table or a join somewhere upstream. Testing only the final report tells you something is wrong. It rarely tells you where, and it never tells you early.

A RAG answer is a report. The corpus is the source data, the chunking and indexing is the ETL, retrieval is the query, and the model is the presentation layer. If all you test is whether the final answer looks right to an LLM judge, you are doing what we stopped doing in data testing years ago: checking the dashboard and hoping the pipeline is fine.

When I designed evaluation for the conversational AI product I look after, the checks covered response accuracy, hallucination detection, context retention and prompt regression, and they run in CI alongside everything else. We also put them behind a simple front end so that QA engineers who don't write code can run AI validation themselves. Looking back, the thing I would push harder on today is separating the stages, so a failure tells you whether the content, the search or the model is to blame.

What I'd test first

I am not going to give a recipe, because every corpus is different. But if I joined a team shipping a RAG product tomorrow, these are the checks I would ask about before I looked at a single LLM-judge score.

Questions with no answer. Write a set of questions that sound like they belong in your domain but that your content genuinely cannot answer. The only acceptable response is a clear "I don't know" or a hand-off. In my experience this set finds more embarrassing behaviour per test than anything else, and it is the cheapest one to write.

Retrieval on its own. For a set of questions where you know which document holds the answer, check whether that document appears in what the search returns, before the model is involved at all. This is ordinary search testing. It is fast, deterministic and doesn't need a judge model. When it fails, no amount of prompt tuning will help.

What actually reached the prompt. Log the final context the model received, not just what search returned, and assert that the right passage survived. Then move it around. If accuracy changes a lot depending on where the passage sits, you have learned something important about your context assembly.

Citation support, claim by claim. For each statement in an answer, check that the passage it cites actually says it. This is the misgrounding failure from the legal study, and it is the one that does the most damage to trust, because the user believes they have already been given the evidence.

Corpus change as a release event. Re-run the retrieval and no-answer sets whenever the content changes: new documents, removed policies, re-chunking, a new embedding model. Teams treat a prompt change as a release and a document upload as housekeeping. I'd treat both as releases.

None of this replaces end-to-end evaluation of answers. It makes those evaluations explainable. When the end-to-end score drops, you want to know within minutes whether to call the content owner, the search engineer or the person who tuned the prompt.

Who owns the corpus

This is the uncomfortable organisational point. In most teams I've seen, nobody owns the retrieval corpus as a quality asset. The documents belong to a business function, the index belongs to whoever built the pipeline, and QA is asked to sign off the chatbot. That split is how a policy that was withdrawn months ago keeps appearing in answers, confidently cited.

I'd want a named owner for the content, with the same expectations we have for reference data in a bank: freshness, a removal process, and someone who is told when retrieval tests start failing on their documents. That is not a QA job on its own. It is a quality job that QA can make visible.

What I don't know yet

I'm less sure how much of this survives as models get better at long contexts and at saying "I don't know". Some of the model-side failures will shrink. I'd be suspicious of anyone who claims the upstream ones will, because missing content, poor ranking and stale documents are not model problems and won't be fixed by a better model.

The evidence also has limits worth stating. The legal study is one domain, one point in time (spring 2024) and products that have since been updated. The failure-point paper is an experience report from three systems, not a measurement. The long-context study uses models that are now two generations old. I use them because they point in the same direction, not because any one of them settles the question.

What I'm confident about is the shape of the problem. When a team tells me their AI can't hallucinate because it is grounded, I hear the same thing I used to hear about reports built on "trusted" source systems. The source may well be trusted. The pipeline between it and the answer still needs testing.

Continue reading

Sources