Evaluating RAG: beyond cosine similarity

RAG is two systems, so evaluate two systems

A RAG pipeline retrieves passages and then generates an answer from them. When an answer is wrong, the cause is either that the right passage was never retrieved, or that the model ignored or distorted it. Blended metrics hide which one happened.

Retrieval metrics

Build a golden set of questions, each labelled with the passages that contain the answer. Then track:

• Recall@k — is a correct passage in the top k results?
• MRR — how high does the first correct passage rank?
• Coverage of 'no answer' questions — does retrieval return low-confidence results when the corpus genuinely lacks the answer?

Generation metrics

Faithfulness checks that every claim in the answer is supported by the retrieved context. Answer relevance checks that the answer addresses the question. Refusal accuracy checks that the system says it does not know when the context is insufficient — often the most valuable behaviour in regulated industries.

Why cosine similarity is not enough

Embedding similarity between an answer and a reference rewards answers that sound alike, not answers that are correct. Two answers differing by a single negation or number can score almost identically. Use it as a cheap signal, never as the gate.

Catching regressions

Run the full harness in CI whenever chunking, embeddings, rerankers, prompts or models change. Re-index changes are the silent killer: a new chunking strategy can lift one metric and quietly drop recall on tables or long documents.