AIDaR: AI Data Readiness Evaluations Framework

TL;DR

• A single argument: while evaluating agents, you must also evaluate the readiness of the underlying data layer. That was the whole talk.
• A real Genentech failure loop: mandated to build an agentic system → pick datasets → engineers work for a couple of months → evals come back poor → read every best-practice blog, tune prompts, hope for better models, none of it helps. The problem turned out to be the data layer — built for data warehousing, reporting, and dashboards, not for agents.
• A five-dimension framework (explicitly still evolving): data quality, semantics & metadata, access & governance, structural readiness, and generalizability. His concrete structural-readiness counterexample is an entity-attribute-value (EAV) data model, which is deeply unfriendly to running agents on top of it.

Key Points

The problem: tuning the wrong layer (~02:50–02:52)

Sett is a principal machine learning engineer at Genentech, focused on agentic AI, scientific AI in life sciences, AI data readiness, and agent evaluations.

He said he had one argument for the whole talk: as we evaluate agents, we also need to evaluate the readiness of the underlying data layer.

Then came a sequence many in the room would recognize. At Genentech they were mandated to build an agentic system: choose some datasets, put ML and AI engineers on it, spend a couple of months assembling the framework — and then run the evals, which came back performing poorly.

So the hunt began. They went through the LangChain blogs and other framework blogs, applied what they believed were the best practices, and it still wasn't performing. Then they moved on to prompt engineering. Then to hoping for better models.

What they realized only later: agents don't perform in a vacuum. They work on top of a data layer, and that underlying data layer is something you have to examine too.

His concrete example: one data layer was oriented toward dashboarding and reporting. They put an agent on top of it and performance was poor — and looking back, that layer had been designed for data warehousing, reporting, and dashboards, not for agents.

AIDaR: the five dimensions (~02:52–02:53)

Out of that came their AI data readiness evaluations framework — which he was careful to describe as still evolving — with five dimensions:

Community and workshop (~02:53–02:54)

They recently had a NeurIPS workshop accepted in Paris (he paused on stage to congratulate Scale AI on their workshop too — the Agents in the Wild workshop mentioned earlier by Chenguang Wang).

The workshop's purpose is to bring two communities together: people building data infrastructure for agentic AI, and people building evaluations and benchmarks for agentic AI.

He closed by noting this was genuinely collaborative work — within Genentech plus academia, industry, and the outside community.

Quotes

"The agents are not performing in a vacuum. They are working on top of a data layer, and the underlying data layer is something we need to also look at." (~02:51)

The single claim of the talk.

"We went to all the LangChain blogs … hopefully we actually applied the best practices, but it wasn't performing. Then we looked into prompt engineering, we looked for hope for the better models." (~02:51)

The standard debugging order when evals disappoint — and his point is that this order skips the bottom layer entirely.