Data Benchmarks: Where Everything's Made Up and the Points Don't Matter

TL;DR

• One thesis: agents should be tested in environments with the same level of realism as their eventual deployments — and data analytics is a uniquely poorly modeled field in frontier benchmarking.
• Three concrete diagnoses. DSBench's "realistic data tasks" turn out to be multiple-choice financial-modeling questions and Kaggle exercises including a Scrabble word puzzle. Spider 2's prescriptive English makes it a translation / instruction-following task — mapping English onto WHERE clauses — not analytics. DABstep's "top country for fraud" never says whether it means fraud rate or fraud volume; it's underspecified, and a real analyst would answer with a chart.
• Hex's answer is Shorelane: a fully synthetic company carrying the semantic pitfalls a real business has — messy data, schema migrations, incomplete documentation — where agents work realistic analyst tickets and are scored on their whole train of thinking, tool efficiency, and the analytics itself.

Key Points

What a good frontier benchmark looks like (~02:44–02:45)

Tang does AI research at Hex, an AI data analytics platform. Most of her time goes into getting LLMs to do data analytics and data science better — "which is really hard" — so eval and experimentation occupy a large share of the team's thinking.

Her opener was a jab: lately, every new public data benchmark leaves her with the same impression — everything's made up and the points don't matter.

Her thesis echoes several other speakers that day: test agents in environments as realistic as where they'll be deployed. What she wanted to show is that data is unusually badly modeled in frontier benchmarking.

First, the good examples. She cited three: rebuilding a common codebase from scratch with no internet and only docs; running a vending machine business and simply evaluating profit at the end; and those deceptively simple web search questions that humans solve only around 30% of the time.

What do those great frontier benchmarks share? They test real-world behavior we actually care about agents performing, and the problems are hard for agents and hard for people alike.

Three counterexamples from data analytics (~02:45–02:47)

DSBench. Last year OpenAI used DSBench to report that their agent surpassed human performance by a significant margin on "realistic data tasks." What do those realistic tasks look like? A pile of multiple-choice exam questions about financial modeling — can you write the SQL, which Excel formula should you use — plus a set of public Kaggle projects including a Scrabble word puzzle. Her verdict: "data analysts will tell you this is not what their work looks like on the day-to-day. If this is what your work looks like, I'm sorry."

Spider 2 (she noted Emily from Scale had raised text-to-SQL earlier). The main problem is that it isn't really testing analytics. The English is very specific and very prescriptive; you're essentially mapping English onto WHERE clauses — a translation or instruction-following task. And the external knowledge isn't carefully curated either — "it's the wrong sport."

DABstep. A well-known set. A question looks perfectly straightforward: multiple choice, what's the top country for fraud? The problem is that once you pull the actual data, the question is fundamentally underspecified — it never says whether you're looking at fraud rate or fraud volume. A real data analyst might produce a chart instead, and arguably that's more complete and more correct.

She added a caveat against being misread: "I don't want to dunk on everyone's work, they've done a lot of hard work here." But the same themes keep emerging — nine of them, too many to read aloud — and the gist is that they're not realistic, the grading is harsh, and it isn't grounded in reality.

Shorelane: building a company messy enough to be real (~02:47–02:48)

So Hex built Shorelane, which she introduced with the talk's own punchline: "Shorelane is completely made up. The points don't matter either. I'm not saying we figured it out, but it is pretty cool, and we try and address some of these realism issues."

Shorelane Commerce is a fully synthetic B2B SaaS company carrying the kind of semantic pitfalls that occur in a real business. It's messy: it has migrations and incomplete documentation.

On the task side, agents work realistic tickets that an analytics agent — or an analytics person — would genuinely get in real life. Scoring covers three things: the entire train of thinking, tool efficiency, and the actual analytics being performed.

The half she ran out of time for (~02:48)

Her closing gesture was toward the unfinished part: these evals are still suspended in a moment in time. How do we let data analytics agents learn from their mistakes?

More at hex.tech; she also pointed at their careers page.

Quotes

"We should be testing these agents in environments that have the same level of realism as their eventual deployments." (~02:44)

The thesis of the talk.

"What do all of these great frontier benchmarks have in common? … The problems are hard for agents and they're hard for people as well." (~02:45)

A one-line test for whether a benchmark is worth anything.

"Data analysts will tell you that this is not what their work looks like on the day-to-day. If this is what your work looks like, I'm sorry." (~02:46)

The room's biggest laugh, and also the most direct rebuttal of DSBench's "realistic tasks."

"These evals are still suspended in a moment in time. How can we allow data analytics to learn from their mistakes?" (~02:48)

The open question she left behind: what comes after static evals?