Evaluating RAG: beyond cosine similarity
Evaluate RAG with retrieval and generation metrics, labelled test sets, permission checks and grounded answers—not embedding similarity alone.
RAG is two systems, so evaluate two systems
A RAG pipeline retrieves passages and then generates an answer from them. When an answer is wrong, the cause is either that the right passage was never retrieved, or that the model ignored or distorted it. Blended metrics hide which one happened.
Retrieval metrics
Build a golden set of questions, each labelled with the passages that contain the answer. Then track:
- Recall@k — is a correct passage in the top k results?
- MRR — how high does the first correct passage rank?
- Coverage of 'no answer' questions — does retrieval return low-confidence results when the corpus genuinely lacks the answer?
Generation metrics
Faithfulness checks that every claim in the answer is supported by the retrieved context. Answer relevance checks that the answer addresses the question. Refusal accuracy checks that the system says it does not know when the context is insufficient — often the most valuable behaviour in regulated industries.
Why cosine similarity is not enough
Embedding similarity between an answer and a reference rewards answers that sound alike, not answers that are correct. Two answers differing by a single negation or number can score almost identically. Use it as a cheap signal, never as the gate.
Use a diagnostic matrix before changing the model
Inspect the retrieved passages alongside the final answer. This distinguishes missing evidence from faulty generation and avoids paying for a larger model when the real problem is indexing or permissions.
- Split evaluation questions by document or time period to reduce leakage.
- Test tables, scanned documents, conflicting sources and unanswerable questions.
- Measure citation correctness as well as whether a citation is present.
| Observed failure | Likely layer to inspect | Next experiment |
|---|---|---|
| Relevant evidence is absent | Ingestion, chunking, filters and retrieval | Check source coverage and recall at a fixed retrieval depth |
| Evidence is present but the answer is wrong | Prompt, context ordering and generation | Test grounding instructions and claim-level review |
| A superseded policy is cited | Document versions and freshness | Check indexing dates and prefer the current authoritative version |
| Restricted content appears | Identity and retrieval permissions | Test access with distinct user roles before release |
Catching regressions
Run the full harness in CI whenever chunking, embeddings, rerankers, prompts or models change. Re-index changes are the silent killer: a new chunking strategy can lift one metric and quietly drop recall on tables or long documents.
Working on something similar?
Discuss your workflow, available data and operating constraints with Talee’s engineers to assess a suitable next step.
Talk to an engineer