LLM Observability & Tracing

Why RAG Needs Tracing

By now you can build a serious RAG pipeline: retrieve, rerank, contextualize a multi-turn question, assemble a prompt, generate, evaluate. In development, when something goes wrong, you read the code and reason about it. In production, with thousands of requests a day across real users, that approach collapses. A user reports "the bot gave a wrong answer to ticket #4892" and you have nothing — the request is gone, and you're guessing.

The Pipeline Is Opaque by Default

A RAG answer is the end of a chain of decisions, and a bad answer can originate at any link:

• Retrieval returned the wrong chunks (bad query embedding, missing document, over-aggressive filter).
• Reranking demoted the right chunk, or the score threshold wrongly triggered a refusal.
• Prompt assembly truncated the context, or ordered it so the answer landed "lost in the middle."
• Generation ignored the context and hallucinated, or the model was having an off day.

Looking only at the final text, you cannot tell which of these happened. They produce indistinguishable symptoms: a confident, wrong answer.

"It's Slow" and "It's Expensive" Are Also Invisible

Latency and cost have the same problem. p95 latency crept from 1.8s to 3.2s — was it the reranker, a cold embedding cache, or the LLM provider? Token spend doubled this month — which stage, which tenant, which prompt change? Without per-stage instrumentation, every one of these is a multi-hour archaeology dig through logs that probably don't have the data anyway.

Observability Is the Prerequisite for Everything Else

This is the first Advanced lesson on purpose. Fine-tuning (Day 2) needs a dataset of real traces. Serving optimization (Day 3) needs per-stage latency to know what to optimize. Online evaluation (Day 5) samples production requests. All of it depends on first being able to see what the system actually did on each request. Tracing is the foundation the rest of the operating discipline is built on.

The rule of thumb: if you can't answer "what exactly did the model see, and how long did each stage take?" for an arbitrary past request, you are not running RAG in production — you are hoping.