Building a Reliable RAG Evaluation Pipeline

Introduction

Most RAG demos look convincing because they are tested on a handful of friendly questions. Production is the opposite: incomplete queries, outdated documents, overlapping sources, and users who will trust a fluent answer even when it is wrong.

This article is about putting a measurement layer in front of that risk. The goal is not a perfect score. It is a repeatable way to know whether retrieval, generation, or cost got worse before a change ships.

You will leave with a pipeline shape you can implement: a golden set, retrieval checks, answer checks, cost and latency logs, and a release gate that fails closed.

The Problem

A RAG system has more than one failure mode, and they hide behind the same symptom: a bad answer. Retrieval can miss the right chunk. The model can ignore the chunk it was given. The index can be stale. The prompt can drift after a small edit. Without evaluation, those failures get debated in screenshots instead of measured.

Existing one-off testing is not enough. Spot-checking in a notebook does not survive a prompt change, a new embedding model, or a larger corpus. Offline leaderboard scores also miss the actual documents and questions your users ask.

The constraints are practical. The set has to be small enough to run on every meaningful change, labeled enough to be trusted, and cheap enough that the team actually runs it. If evaluation takes overnight or requires a dedicated researcher, it will not be part of shipping.

Architecture

The pipeline is a batch job, not a chatbot. Each run takes a frozen index, a frozen prompt, and a golden set of questions with expected evidence and answers. It writes a report: retrieval quality, answer faithfulness, latency, and token cost.

I keep three layers separate on purpose. Retrieval is scored against expected document IDs or passages. Generation is scored against the retrieved context, not against the entire corpus. Operations are scored as latency and cost per question. Mixing those scores into one “quality” number makes the next failure harder to debug.

• Golden set: real questions, expected sources, acceptable answers, and known failure cases.
• Retrieval stage: embed, search, return IDs, passages, and ranks.
• Answer stage: generate with the same prompt used in production.
• Report stage: store metrics per case so a regression points to a specific question, not a vibe.

Implementation

Start with forty to eighty labeled cases, not thousands. Include easy lookups, multi-hop questions, and questions whose answer is not in the corpus. The last group is as important as the first: the system should refuse or say it does not know.

Each case stores the query, one or more expected document IDs, and a short reference answer. Retrieval is a hit if the expected ID appears in the top k. Faithfulness is a hit if the generated answer stays inside the retrieved passages. Exact string match is too brittle; use a structured judge or a checklist of required claims.

Run the same job in CI on prompt, index, or model changes. Persist the last passing report. A change that drops retrieval hit-rate or raises cost beyond an agreed budget fails the job. That is the release gate.

• Tools: your production retriever, the same model path, and a small eval runner.
• APIs: embeddings and generation through the same providers the product uses.
• Storage: JSONL or a simple table of case_id, metrics, and output traces.

Challenges & Trade-offs

LLM-as-judge is convenient and unstable. I use it for faithfulness only after the checklist is explicit, and I spot-check disagreements. A cheaper alternative is claim coverage: required facts listed by hand, then checked with a smaller model or rules.

End-to-end accuracy alone is a weak signal. A better retrieval layer can still produce a worse answer if the prompt changed. That is why retrieval and generation are scored separately even though the product only shows one response.

I did not start with a large public benchmark. Those sets are useful later, but they do not represent the corpus. A small in-domain set found prompt and chunking issues faster than a generic QA dataset.

What I Learned

Evaluation is product work. The golden set is a specification of what the system is allowed to do. If a case is missing, the pipeline cannot protect it.

I would label expected sources first, answers second. Wrong retrieval is the most common production failure, and it is cheaper to measure.

A small, painful set beats a large, unmaintained one. The moment nobody trusts the labels, the gate gets skipped.

Future Improvements

The next gap is online evaluation: sampling live traffic, redacting it, and promoting hard cases back into the golden set. Offline eval cannot see new document types until someone adds them.

I also want slice-level reports — by source, language, and question type — so a regression in one collection is not averaged away by easy cases in another.