Agentic RAG: Tool Use, Function Calling & ReAct

From Static Pipelines to Agentic RAG

The pipeline you've built so far is static: embed the question, retrieve once, rerank, generate. One question in, one search, one answer out. For a huge fraction of questions that's exactly right — and you should not reach for anything fancier when it works.

But some questions break the single-shot assumption.

Where One Retrieval Isn't Enough

• Multi-part questions. "Compare our refund policy for digital goods versus physical goods." One query blends two topics and retrieves a muddy mix of both. Two targeted searches would each be clean.
• Conditional questions. "If the customer is on the enterprise plan, what's the SLA?" The right search depends on a fact you have to look up first.
• Multi-hop questions. "Who signed off on the policy that governs EU data retention?" You first find the policy, then find who approved it — the second search needs the first's result.
• Questions that aren't retrieval at all. "What's 18% of our $4,200 invoice?" There's nothing to retrieve; the model should compute, not search.

A static pipeline answers all of these by doing the one thing it knows how to do — retrieve once — and then hoping. That's where quality quietly falls apart.

The Agentic Idea

Agentic RAG stops treating the LLM as the final step and starts treating it as the controller. Instead of you hard-coding "always retrieve, then generate," you give the model a set of tools (search this index, search that index, do math, look up a record) and let it decide:

1. Do I need a tool at all, or can I answer directly?
2. If so, which tool, with what arguments?
3. Given the result, am I done — or do I need another step?

STATIC:     question ─► retrieve ─► generate ─► answer

AGENTIC:    question ─► [ LLM decides ] ─► tool? ─► observe ─┐
                            ▲                                │
                            └──────── loop until done ◄───────┘
                                          │
                                          ▼
                                        answer

The Trade

Agentic RAG is strictly more powerful and strictly more expensive. Each decision is another LLM call, each search adds latency, and a loop that doesn't terminate cleanly can run up cost or spin forever. The skill this lesson teaches is not "always go agentic" — it's knowing the mechanism well enough to apply it only where the question shape demands it, and to bound it when you do.