Information Retrieval in the Age of Agentic AI

TL;DR

• The shift: from traditional IR / classic RAG as a straight line (query → fetch documents → ranked list → rerank → hand the user a list) to an agentic flow (plan → infer intent → search → gather → analyze → critique → present). The unit of work moved from a lookup to an ongoing conversation with the world, often running 20 turns and going multi-hop.
• The uncomfortable gap: information seeking is now interactive, multi-turn, temporal, and evidence-driven, but we still grade the final answer with BLEU and ROUGE. That doesn't work for agentic IR.
• What to do instead: evaluate the trace — the correctness of every hop, whether each hop was adequate to answer the user, and the cost in tokens and latency.
• Where the field is heading: reasoning-aware retrieval (using the agent's reasoning traces, not just query keywords), multi-turn/hop-aware benchmarks such as MTRAG, graph RAG, self-RAG for hallucinations, adaptive RAG routing by query complexity and effort, and token consumption itself as a retrieval metric.

Key Points

From classic RAG to agentic IR (~00:32–00:34)

Roosta framed the change as two pictures. Traditional IR and classic RAG are a straight line: a query comes from the user, the search engine fetches documents, produces a ranked list, reranks it, and presents the list.

The agentic flow does something else: the agent plans, works out the intent behind the query, searches, gathers results, analyzes, critiques, and only then presents a final result. Her summary: the unit of work has moved from a lookup to an ongoing conversation with the world.

Deep research agents are where this is most visible. Her example was a question about cholesterol medication: the agent works out the intent, uses various tools to gather websites, pricing information, drug interactions, and so on, grounds the answer in retrieved evidence, and presents it — saving hours of reading documents and blog posts. These runs now routinely stretch to around 20 turns, making the whole thing multi-hop.

The gap: evaluation is still single-turn (~00:34–00:35)

"Here's the uncomfortable gap." When we evaluate agentic IR, we still look only at the final answer and score it with BLEU and ROUGE. But information seeking is now interactive and multi-turn, it has a temporal aspect, and it has to be evidence-driven. Grading the final answer alone simply doesn't work.

Her prescription: look at the trace. Assess the correctness of each hop, whether each hop was adequate to answer the user's question, and the cost — number of tokens, latency, and so on.

Where the field is heading (~00:36)

• Reasoning-aware retrieval — retrieval signals drawn from the agent's reasoning traces, not just keywords from the query.
• Multi-turn, hop-aware benchmarks such as MTRAG.
• Graph RAG, so retrieval can see relationships between entities.
• Self-RAG, to cut through some of the hallucinations.
• Adaptive RAG, routing queries based on their complexity and the effort they warrant.
• Token count as a metric — actually measuring and optimizing token usage as an information-retrieval metric in its own right.

Her closing point: our information seeking has changed and agents have changed how we get information, so the way we assess whether these agents are doing the right thing and achieving the goal they set out for has to change too.