Agentic RAG: How Retrieval Works for AI Agents

Every few months someone declares RAG obsolete because context windows got bigger. Then teams try to stuff their whole document store into a prompt, and retrieval comes back. What has changed is where retrieval sits and who controls it. This is how I build it as of late 2026.

Classic RAG in one paragraph

Classic retrieval-augmented generation is a pipeline: split documents into chunks, embed them, store them in a vector index; when a question arrives, embed it, fetch the most similar chunks, and pass them to the model with the question. It is simple, cheap and works well when one search is enough. I compared it to training a model on your data in RAG vs fine-tuning.

Its weakness is that it searches once, with the user's exact words, before the model has thought about the question. If the question needs two facts from two places, or the user's wording does not match the documents, the pipeline retrieves the wrong context and the model answers confidently from it.

What makes RAG agentic

In agentic RAG, search is a tool the agent can call, not a step that always runs. The agent can:

• rewrite the query into the terms the documents actually use,
• search more than once, refining after each result,
• choose between sources, such as a vector index, a keyword index, a SQL database or a live API,
• decide it has enough and stop, or decide it does not and ask the user.

Anthropic describes the underlying idea as context engineering: treating the model's context as a finite resource and aiming to "find the smallest set of high-signal tokens that maximize the likelihood of your desired outcome". It describes a "just-in-time" approach in which agents keep lightweight references, such as file paths, stored queries and links, and load the data through tools only when they need it. Claude Code is its example: project notes load up front, while search tools navigate the codebase on demand.

That is agentic RAG in practice. The retrieval logic is not deleted; it is handed to the agent along with good tools.

Classic vs agentic RAG

Do you need RAG at all?

Sometimes not. Anthropic's own guidance is that if a knowledge base is smaller than 200,000 tokens, about 500 pages, you can include the whole thing in the prompt and skip retrieval. With prompt caching, that is often the simplest reliable option for a small policy set or product manual.

Retrieval earns its place when:

• the corpus is larger than you want to send on every request,
• the data changes often or lives in several systems,
• users should only see documents they are permitted to see,
• you need to cite exactly which passage an answer came from.

Retrieval quality still decides everything

Making retrieval agentic does not fix bad retrieval. An agent that searches five times over a poor index gets five poor results. One of the most useful published techniques is the one Anthropic calls contextual retrieval: before embedding a chunk, prepend a short, generated description of where that chunk sits in its document.

Anthropic's numbers on its test sets: contextual embeddings cut the top-20 retrieval failure rate by 35%; adding contextual BM25 keyword search cut it by 49%; adding a reranking step cut it by 67%. The lesson I take from that is less about the exact percentages and more about the stack: semantic search plus keyword search plus reranking beats any one of them alone.

How I build agentic RAG

Give the agent narrow, typed search tools

Instead of one "search everything" tool, I give agents separate tools with clear descriptions: search_policies(query), search_tickets(query, customer_id), get_order(order_id). The model chooses better when each tool has one job, and permissions are easier to enforce per tool. Exposing these as MCP servers makes them reusable by any agent; see how to build an AI agent with MCP and Claude.

Enforce permissions in the tool, not the prompt

Every search tool filters by the calling user's access rights before results reach the model. A prompt that says "do not show confidential documents" is not access control.

Cap the loop

Set a maximum number of searches and a token budget per question. When the agent hits the cap, it answers with what it has and says what is missing, or hands off to a person. Unbounded retrieval loops are the most common way agentic RAG becomes slow and expensive.

Return sources with every answer

The agent's answer includes the passages it relied on. That lets users check answers in seconds, and it lets you evaluate the system properly.

Evaluate retrieval and answers separately

I keep two sets of tests: did the right documents come back for this question, and was the final answer correct and grounded in them? When quality drops, this tells you whether to fix the index or the agent. More on this in the evals post.

Where I have used this

At Avenuor, where I am CTO and built the platform from zero to production, the product includes a RAG assistant alongside AI document processing and agentic workflows. In my voice agent for education, spoken lectures become structured notes that students can then query in natural language, which is retrieval over content the user created themselves. Different domains, same rule: retrieval quality and permissions first, agent cleverness second.

Signs you should move from classic to agentic RAG

• Users ask questions that need facts from two or more systems.
• Your logs show good answers when users phrase things like the documents, and bad ones otherwise.
• You are adding special-case rules to the retrieval pipeline for different question types.
• Users routinely ask follow-ups that a second search would have answered.

If none of these apply, keep the simpler pipeline. It is faster, cheaper and easier to test.

Work with me

If your assistant gives confident wrong answers or cannot handle questions across systems, I can help you rebuild the retrieval layer properly. See AI chatbot development and AI agent development, or book a scoping call.