RAG Evals That Actually Predict Prod
Retrieval-Augmented Generation systems often fail in production despite passing initial tests. This article examines practical evaluation strategies that reliably predict real-world performance, with a focus on enforcing per-sentence citations to catch failures before they reach users. Industry experts share proven techniques for building RAG evaluations that actually matter.
Mine Failures Enforce Per-Sentence Citations
Our offline evals looked great. Production was on fire. The gap: golden set full of queries we imagined—not queries that actually torched the system.
Harness was simple. Retrieval: recall@5 and nDCG@10 against labeled relevance. Generation: groundedness via LLM-as-judge—does every claim trace to a chunk? Both ran on every PR.
Metrics didn't predict prod. Until we changed how we built the dataset.
The trick: 30% of our golden set now comes from error logs. Thumbs-down queries. Zero-retrieval queries. Misspellings, jargon, phrasing we never would have invented.
Before: recall@5 correlated 0.52 with prod satisfaction. After adding failure cases: 0.84. Same metric. Different dataset.
The rubric shift: for groundedness, we stopped asking "is the answer correct" and started asking "does every sentence cite a chunk." Small tweak. Massive difference in killing hallucinations before ship.
Anchor Evals To Real User Intents
Judge task success against real user intents, not lab prompts. Build an intent map from private, cleaned logs, user surveys, and support tickets. Define success for each intent, like the right action, the next click, or a solved outcome.
Use balanced samples so rare intents are not missed. Have domain experts review a slice to fix labels and edge rules. Start by sampling real sessions, labeling intents, and tying each to a clear success rule today.
Measure Abstention And Guide Productive Escapes
The system should know when the right move is to not answer. Measure how often it abstains when sources are weak and how often it answers when it should not. Track what users get after an abstain, like a helpful follow‑up question or a handoff to search or support.
Set confidence thresholds so the mix of answers and abstains matches real risk. Watch rates of wrong answers and wrong abstains across intents and user groups. Add abstention quality to the main metrics and wire fallbacks that help users move forward.
Demand Verifiable Passage Evidence
Score answers on whether each cited line truly supports the claim. Check that quotes are exact, pages are real, and links open to the right place. Mark down any claim that lacks a matching passage in the sources, even if the answer sounds right.
Give credit when all needed evidence is cited and extra, off‑topic facts are avoided. Tune automated checks with a small human audit so the signal stays honest. Add passage‑level citation checks to the evals and gate releases on faithfulness.
Stress Test Retrievers With Long-Tail Noise
Push the retriever with hard, long‑tail queries that mirror the wild. Include typos, slang, mixed languages, and vague hints that hide key facts. Add look‑alike documents that seem right but are wrong to test focus.
Track how many right sources appear in the top few results and how that drops as noise grows. Keep a frozen set of these tricky prompts and run them on every change to catch new bugs. Build a standing red team set and keep growing it each week.
Adopt A Weighted Reliability Scorecard
Judge models by more than accuracy by also tracking speed, cost, and stability. Build one score that blends answer quality with typical and worst‑case speed, token and API cost, and error rate when traffic is high. Compare changes with side‑by‑side tests that hold budget limits and service goals.
Prefer gains that still hold when caches are empty and traffic spikes. Use this score to pick retrievers, context size, and how many results to pull so goals are met. Adopt a weighted scorecard and make it the gate for any launch.