How to make a support AI say 'I don't know' — accuracy-first RAG for B2B SaaS

The failure mode nobody demos

Every AI support assistant demo looks good. You feed it your docs, ask it a clean question, and it returns a fluent, well-structured answer. The audience nods. The product gets built.

Then it goes live. A user asks something slightly off-centre — a question your docs only partially cover, or one that sits in the gap between two help articles. The assistant invents an answer. It sounds confident. It's wrong. The user follows it, breaks something, and now your support team is dealing with the fallout.

This isn't a hypothetical. It's the default behaviour of a naive RAG pipeline, and it happens because nothing in the default setup tells the model it's allowed to not know.

The design problem is real, and it's solvable. But it requires deliberate choices at every layer — retrieval, grounding, and fallback. Here's how I think about it.

Why RAG hallucinates despite having the right documents

Retrieval-augmented generation is supposed to prevent hallucination: you pull relevant documents, stuff them in the context window, and the model answers from those. In theory the model only knows what's in front of it. In practice there are three ways this breaks down.

Retrieval returns something, but not the right thing. Most vector search implementations return the top-k results regardless of relevance. If the user's question doesn't have a close match in your corpus, you still get k documents — they're just the least-bad matches. The model receives plausible-looking context that doesn't actually answer the question, and then does what language models do: it synthesises a plausible-sounding answer from what it was given.

The model ignores the retrieved context. LLMs have strong priors from pretraining. If a retrieved chunk says one thing and the model's pretraining says something subtly different, the model will sometimes go with its priors — especially on topics it "knows well" like general software behaviour, common APIs, or standard support troubleshooting steps. You can partially address this with prompt engineering, but it's not a full fix.

The question spans multiple chunks, none of which is complete. A user might ask about a workflow that involves three separate features. Each feature has its own help article. Individually, each chunk scores reasonably well on retrieval — but none of them alone answers the full question. The model sees partial information and fills the gaps.

The common thread: the model defaults to answering. It will always try to produce something useful-looking. You have to explicitly break that default.

Start with retrieval quality, not prompt engineering

Most teams reach for the system prompt first: "If you don't know, say you don't know." This helps at the margins, but it's the wrong first lever. A model that receives genuinely relevant context will answer correctly. A model that receives irrelevant context will hallucinate regardless of instructions, because it doesn't know the context is irrelevant — and neither do you, if your retrieval layer has no concept of confidence.

The fix starts at retrieval. Concretely:

Set a similarity threshold, not just a top-k limit. Cosine similarity (or whatever distance metric your vector store uses) gives you a score between 0 and 1 for each retrieved chunk. Below a threshold — I typically start around 0.75 with OpenAI embeddings, then calibrate against real queries — a chunk isn't actually relevant. Don't pass it to the model. If every retrieved chunk falls below the threshold, the question has no grounding material: handle that case explicitly before the model ever sees it.

Tune chunk size and overlap for your content type. Help documentation is dense. A 512-token chunk might cut a step-by-step procedure in half, leaving the model with half an answer. I generally start with 300–400 token chunks and 50–80 token overlap for support docs, then adjust based on where retrieval breaks. Smaller chunks improve precision; larger chunks improve completeness. The right answer depends on how your docs are structured.

Reranking matters at scale. If you have more than a few hundred documents, a cross-encoder reranker applied after initial retrieval significantly improves the quality of what the model actually sees. The initial vector search is fast but coarse; the reranker is slower but much better at semantic relevance. On a corpus of any real size, skipping reranking means your top-k is noisier than it needs to be.

Answer grounding: checking the output, not just the input

Even with good retrieval, the model can still go off-script. Answer grounding is a second check that runs after the model produces its response: does the answer actually follow from the retrieved context?

The simplest version is a self-consistency prompt. After getting an answer, you run a second LLM call (or the same call in a structured output format) that asks: "Given only the following documents, is every claim in this answer supported? List any claims that are not." If the model identifies unsupported claims, you either regenerate with tighter constraints or surface a fallback instead.

This doubles your LLM cost per query, so you need to decide where the bar is. For a support assistant where a wrong answer erodes trust and creates support tickets, it's usually worth it. For an internal tool where approximate answers are fine, maybe not.

A lighter alternative: require the model to cite specific chunks in its answer, and check programmatically that every cited chunk actually exists in the retrieved set. A model that fabricates a citation is a model that fabricated the answer. This catches a surprisingly large fraction of hallucinations at near-zero cost, and it gives users something concrete to verify.

The fallback protocol

When the system determines it can't answer reliably — retrieval below threshold, grounding check failed, no relevant chunks found — it needs somewhere useful to go. "I don't know" by itself is not a good fallback. It's honest, but it doesn't help anyone.

A better fallback has three components:

Acknowledge the limit explicitly. Something like: "I couldn't find a confident answer to this in the documentation." Not "I'm sorry, I don't have that information" (apologetic, vague), not a rephrased version of the question. Just a direct statement of what happened.

Offer the closest thing you found. Even if the full answer isn't there, the retrieval step usually found something adjacent. Surface it: "The closest I found was this section on [topic] — it may be related." Give the user something to work with, clearly marked as adjacent rather than authoritative.

Route to a human or a resource. A link to the full docs, a link to submit a support ticket, or (if you have the infrastructure) a handoff to a human agent. The goal is to leave the user with a path forward, not a dead end.

This is also where the UX matters. If you show citations on every answer — links to the source chunks — users develop a natural habit of checking them. They trust the system appropriately: when there are citations, the answer is grounded; when there aren't, they know something uncertain happened. That calibrated trust is much more valuable long-term than an assistant that always sounds confident.

What this looks like in a real stack

For context: this is the architecture behind Grounded, a support assistant I built and run on this exact problem. The stack is Next.js, Vercel AI SDK, Mastra for the agent/tool layer, and Supabase with pgvector. Not promoting it — just illustrating that these decisions are working in production, not theory.

The retrieval query runs against pgvector with an explicit similarity filter (WHERE similarity > 0.75). If the result set is empty, the query takes a different code path immediately — no LLM call. If the result set is non-empty, the top chunks go to the model with a strict instruction: answer only from the provided context, include a citation for every factual claim, flag anything uncertain.

The model's output is parsed. If it contains a citation, the cited chunk ID is checked against the retrieved set. If a cited chunk doesn't exist in that set, the answer is rejected and the fallback fires. This check runs in a few milliseconds and costs nothing.

Answer grounding (the full self-consistency check) is only enabled for a subset of queries where the stakes are higher — anything involving billing, account changes, or data. The cost and latency aren't worth it for general how-to questions.

The questions worth asking before you build

None of this is complicated in isolation. What makes it hard is deciding the right tradeoffs for your specific product. A few questions worth working through before you start:

What does a wrong answer cost you? In a consumer product, a hallucination is annoying. In a B2B support tool, it might mean a customer takes an incorrect action on their account, data, or integration — and calls your support team. The accuracy bar should be proportional to the cost of failure, not the cost of the LLM call.

How complete is your documentation? A RAG system is only as good as the corpus it retrieves from. If your docs have gaps, the assistant will have gaps. You can paper over small gaps with good fallback handling, but you can't substitute for actually having the right information. A documentation audit before building usually pays off.

What will you measure? An accuracy-first assistant needs an eval. At minimum: a set of representative questions with expected answers, a set of out-of-scope questions that should trigger fallback, and a way to run both after every change. Without that, you're flying blind — and any change to chunk size, threshold, or prompt might quietly degrade accuracy you thought you'd earned.

I'd rather build a support assistant that says "I don't know" five percent of the time than one that confidently hallucinates. Users forgive honest uncertainty. They don't forgive wrong answers that looked right.