Evaluating RAG: RAGAS, TruLens & LLM-as-Judge

Why RAG Needs Its Own Evaluation

By now you can build a RAG pipeline with reranking, agentic retrieval, and conversational memory. The uncomfortable question is: is it any good? "It looks right when I try a few questions" is not an answer you can ship a regulated-industry product on.

RAG Fails on Two Sides

A traditional classifier has one thing to get right. A RAG system has two, and they fail independently:

1. Retrieval — did the system fetch the chunks that actually contain the answer?
2. Generation — given those chunks, did the LLM produce an answer that is faithful to them and actually addresses the question?

These compose into four outcomes, and only one is good:

Retrieval	Generation	Result
Good	Good	The answer you want
Good	Bad	Right context, but the model hallucinated or wandered off-topic
Bad	"Good"	Fluent, confident, and wrong — the answer was built on the wrong evidence
Bad	Bad	Garbage in, garbage out

The dangerous cell is bad retrieval + fluent generation: the system sounds authoritative while being completely wrong. A single end-to-end "accuracy" number can't tell you which side broke — and if you don't know which side broke, you don't know what to fix.

Two Questions, Two Diagnoses

This is why RAG evaluation always decomposes the pipeline:

• If retrieval is the problem, you work on chunking, the embedding model, hybrid search, or reranking (Day 1).
• If generation is the problem, you work on the prompt, the model, or guardrails — and you measure whether the answer is grounded in the retrieved context, not whether it merely sounds plausible.

"Vibe Checks" Don't Scale

Manually eyeballing a handful of outputs catches nothing systematically. It can't tell you whether last week's prompt change helped or hurt, can't gate a deploy, and can't cover the long tail of queries where RAG actually breaks. You need metrics computed over a representative set — which is the rest of this lesson.

The goal: turn "it feels better" into "faithfulness went from 0.82 to 0.91 and context precision held at 0.88, so we ship."