The eval setup I run before shipping a RAG system to a client

The honest answer to "how do you know it works?"

Every client asks some version of this before we ship. And for a while, the honest answer in this industry was: vibes and manual spot-checking. You'd build the RAG pipeline, run some queries by hand, feel good about it, and call it done.

That's not good enough for a B2B SaaS product where the AI is answering real customers with real problems. A wrong answer in a support context doesn't just fail to help — it can actively mislead, create a support escalation, or erode trust in the product itself. I've seen it happen.

So over the course of building Grounded (an accuracy-first support assistant I run) and doing similar work for clients, I've settled on an eval setup I run consistently before anything ships. It's not perfect and it's not a magic CI gate. But it catches most of what matters.

Here's what it actually looks like.

Start with a golden set, not synthetic queries

The first thing I do is build a golden evaluation set from real data. Not invented questions — real ones. I ask the client for a sample of their actual support tickets or help-centre search queries, ideally 3–6 months’ worth of history. Then I filter to the 80–100 questions that represent the highest-volume or highest-stakes categories.

Why real questions? Because synthetic questions, even good ones, cluster around what the system was designed to handle. Real questions include the edge cases, the out-of-scope stuff, the weirdly-phrased queries users actually type. They're uglier and more useful.

For each question in the set I write a reference answer: what the ideal response looks like, which source document it should draw from, and whether there's a correct refusal case (i.e., the system should say "I don't know" rather than guess). That last category is important and usually gets skipped. Knowing when to refuse is a skill in itself.

The four things I actually test

Once the golden set exists, I run four checks. Some are automated, some require a quick manual pass.

1. Retrieval recall

Does the system fetch the right source document(s) for each question? This is a retrieval problem, not a generation problem, and it fails in different ways. The most common failure I see is a chunking issue — the relevant information got split across chunks in a way that means neither chunk scores well against the query. The second most common is an embedding mismatch, where the user's phrasing is far enough from the document's phrasing that cosine similarity misses it entirely.

I measure this by checking whether the reference source document appears in the top-k retrieved results (usually top 3 or top 5). If retrieval recall on the golden set is below roughly 85%, I don't proceed to generation evals — the problem is upstream, not in the LLM.

2. Faithfulness

Given that the right context was retrieved, does the generated answer stick to it? This is the hallucination question. The model should be summarising and synthesising from the provided context, not drawing on its training weights to fill gaps.

I use an LLM-as-judge approach here: I ask a second LLM call (typically a cheaper, faster model) to evaluate whether each claim in the generated answer is supported by the retrieved context. This isn't perfect — the judge can miss things or be fooled by confident-sounding prose — but it catches the obvious cases and scales.

A faithfulness rate below 90% on the golden set is a red flag. It usually means the system prompt is too permissive and the model is filling gaps. The fix is almost always tighter instructions: "Answer only from the provided sources. If the information is not in the sources, say so."

3. Refusal accuracy

This one's underappreciated. For the questions in the golden set that I've marked as should-refuse, does the system actually refuse? And for the questions that should have an answer, does it refuse when it shouldn't?

Both failure modes matter. A system that refuses everything is useless. A system that never refuses will hallucinate. The threshold that works well in practice is setting the system to refuse when the retrieval confidence score is below a certain value and when the judge model rates context relevance as low — then tuning that threshold by watching the refusal rate on the golden set.

For Grounded, I landed on a setup where the large majority of out-of-scope questions get a clean "I don't have that information" response, with the rest correctly answered from context. Getting there took a few iterations of threshold tuning.

4. Format and citation hygiene

Smaller than the above, but clients care about it: do responses cite the source documents correctly? Do they stay within the expected length range? Do they fail on edge-case inputs like empty queries or very long questions?

I run this as a simple rule-based pass — check that citation links are present and valid, response length is in range, no empty outputs. Takes five minutes to write and catches a surprising number of integration bugs.

How I run this in practice

For most client projects the golden set is 80–100 questions. I run the full eval pipeline once before the first demo, then again after any major change to the chunking strategy, retrieval parameters, or system prompt. It's not automated in CI for every commit — that would be overkill for a small RAG system — but it runs before anything goes to production.

The tooling I use is minimal: a Python script that calls the RAG pipeline directly (not through the UI), logs retrieval results and generated answers to a spreadsheet, and runs the faithfulness judge. Total runtime for 100 questions is usually under three minutes.

I've tried more elaborate eval frameworks and found they add overhead without adding much insight at this scale. When a client's system grows to the point where they need continuous automated evals with regression tracking, that's a different conversation — and worth doing. But for an initial ship, simple and thorough beats complex and patchy.

What this catches that spot-checking misses

The retrieval recall check catches chunking bugs that look fine on casual testing. You'll try a query, it works, you move on. The eval set finds the 8 questions in 100 where retrieval consistently fails because a critical document got chunked at the wrong boundary.

The faithfulness check catches prompt drift. Early in a project the system prompt is tight. Then someone adds a few lines to handle a specific case, the instructions get slightly looser, and suddenly the model has more latitude to infer. The eval catches this; manual testing rarely does.

The refusal check catches calibration problems that only show up at scale. In manual testing you'll try the happy path and a few edge cases. The eval tries 20 or 30 out-of-scope questions and gives you a real refusal rate. That number matters when you're promising clients that the system won't make things up.

What it doesn't catch

Worth being honest about the limits. A golden set of 100 questions doesn't cover every possible failure mode. Real users will ask things you didn't anticipate. The LLM-as-judge can be wrong. Faithfulness testing doesn't catch cases where the source document itself contains bad information.

This eval setup is a quality floor, not a quality ceiling. It tells you whether the system is ready to ship to a small set of beta users; it doesn't tell you the system is correct for all time. After launch, I recommend watching real conversation logs weekly for the first month and expanding the golden set with queries that surface new failure patterns.

The goal isn't a perfect system — that doesn't exist. The goal is a system where failures are known, bounded, and recoverable.

If you're about to ship a RAG feature

Before you demo to your first real users, run through these four checks on 80–100 real questions. It takes a day or two to set up properly the first time. Every subsequent project takes a few hours. The alternative is shipping something that works in demos and fails on the questions your users actually care about.

If you're building a RAG-based feature and want a second pair of eyes on the eval setup — or you want someone to build and run the whole thing — I offer a free 15-minute AI scoping call. No pitch, just a conversation about what you're building and whether I can help. Book a time here.