How Better Evals Can Bring Abundance Through Accelerated Scientific Discovery

TL;DR

• His definition of abundance is sci-fi quality of life: better scientific discoveries → new materials and more efficient energy → better medicine and more efficient travel. He drew an explicit line: this is not about eliminating white-collar jobs, it's about improving people's quality of life.
• His 20-year through-line is parameter tuning. From protein folding to quantum mechanics simulation to metagenome assembly, every problem ended the same way: you build something great, then you have to tune it. Grad students up at night turning knobs in a high-dimensional space for a marginally better paper — grad student descent. He fell for Bayesian optimization, made it his thesis, then made it SigOpt.
• Optimization got solved; trust didn't. SigOpt would push a customer's number up and the customer would return to say another number went down. A black-box optimizer will blindly optimize whatever eval you hand it — if 1% of transactions are fraud, calling nothing fraud gets you 99% accuracy.
• Distributional's two pivots are an honest record of failure. First, attack it with heavy statistical testing and solve chaos and non-stationarity ("a terrible idea, because nobody likes tests, they didn't have the data, and everybody was YOLOing models into production anyway"). Then pivot to analytics for catching unknown unknowns — with the verdict that "the eval is a really good idea, but it's more of a feature, let alone a product, let alone a startup."
• His current answer is bespoke evals: evals that know physics, know the scientific research process, and know how to validate hypotheses, so the flywheel of a "what-if machine" spins faster — doing in six hours what took six months in grad school. Open sourcing at NeurIPS; private beta now.

Key Points

What abundance means, and why now (~02:55–02:56)

Clark is co-founder and CEO of Distributional and is building Talaria Scientific. He opened by defining the word in his title:

Abundance through AI means better scientific discoveries that lead to new materials and more efficient energy, which lead to better medicine and more efficient ways of travel. "It basically means living in a sci-fi future."

He deliberately separated this from a competing narrative: "It's less about how do we get rid of more white collar jobs and put everybody out of work, and more about how do we improve people's quality of life."

For the first time, he said, we can meaningfully accelerate scientific discovery with AI — the last few months have unlocked quite a bit of capability. And this has been his obsession for 20 years.

Grad student descent: the same problem for two decades (~02:56–02:57)

From undergrad through his PhD, every problem he solved and every group he worked with — protein folding, quantum mechanics simulation, metagenome assembly — hit the same wall: you'd build something great, and then you'd need to tune it. Knobs, levers, hyperparameters, whatever you call them. Make the graphs and benchmarks slightly better and you got a better paper.

They jokingly called it grad student descent, because it was usually the grad students sitting up late turning knobs in a high-dimensional space for a marginally better paper.

People did apply smarter techniques — simulated annealing, genetic algorithms, local methods. But he fell in love with Bayesian optimization, which became core to his PhD thesis and then the central point of his first AI startup.

SigOpt: turning optimization into a business (~02:57–02:58)

SigOpt started in 2014 to solve the parameter tuning problem. Over seven years they worked with:

• Netflix — tuning recommender systems
• American Express — tuning fraud systems
• OpenAI — in the early days, when it was still a nonprofit lab, tuning their early RL systems
• Roughly a trillion dollars' worth of hedge funds
• Several hundred academics using the free program to tune their papers, from materials design to drug discovery

He sold the company to Intel in 2020, where it went on to tune everything from chip design to "benchmaxxing MLPerf, when that was what everybody was overfitting to at the time."

The goal was optimization at scale for problems that really mattered. His own verdict: "it worked really well, except when it didn't."

The two faces of a black-box optimizer (~02:58–02:59)

The problem is that it fits exactly what you told it to do. Customers would come back and say: yeah, you made that number go up, but some other number went down.

Then came the sharpest line of the talk:

The very best thing about a black-box optimizer is it will optimize any eval you give it. And the very worst thing about a black-box optimizer is it will blindly optimize any eval you give it.

His illustration: if only 1% of your transactions are fraud, you can build a very "accurate" fraud detection system by saying nothing is fraud. 99% accuracy right there.

On top of that, we are still a long way from setting these systems up to be completely autonomous. He doesn't think we're anywhere near issuing a goal like "solve stable fusion containment" or "solve cancer." And even if the model came back with something — how could we trust it?

That's the other side of the coin: you need to optimize these systems, and you need to trust them.

Distributional's two pivots (~02:59–03:00)

Trust is what he set out to solve with Distributional.

Version one: attack it in the most complex mathematical way possible — big statistical tests, solve chaos, solve non-stationarity. His own assessment is unsparing: "this was a terrible idea because nobody likes tests. They didn't have the data and everybody was just YOLOing models into production anyway."

So he pivoted — "this is what you do, you're failing, pivot" — to analytics: catch the evals sneaking through your system, find patterns in behavior so you can point at something and say "this is the unknown unknown."

But he delivered an honest verdict on that path too: "the eval is a really good idea, but it's more of a feature, let alone a product, let alone a startup."

The current answer: bespoke evals (~03:00)

So now he's applying everything from the last 20 years to the scientific problem: how do we accelerate science? — with the focus on what actually matters:

Building bespoke evals that know physics, that know about the scientific research process, that know how to do hypothesis validation. With those, you can spin the flywheel of a what-if machine faster and faster — doing what took him six months in grad school in six hours instead.

They're building it in the open: it will be open sourced at NeurIPS, with a private beta running now. He invited anyone working on computational science to come talk.

Quotes

"The very best thing about a black-box optimizer is it will optimize any eval you give it. And the very worst thing about a black-box optimizer is it will blindly optimize any eval you give it." (~02:58)

The axis of the talk: an optimizer's quality ceiling is your eval's quality ceiling.

"You could make a really accurate fraud detection system if only 1% of your transactions are fraud by saying nothing's fraud. 99% accuracy right there." (~02:58)

The laziest way to beat a benchmark, and the best cautionary example.

"This was a terrible idea because nobody likes tests. They didn't have the data and everybody was just YOLOing models into production anyway." (~02:59)

A founder's honest post-mortem on his own first product.

"The eval is a really good idea, but it's more of a feature, let alone a product, let alone a startup." (~02:59)

A weighty thing to say in a session devoted entirely to evals.

"Doing what I did in grad school that used to take six months in six hours instead." (~03:00)

His concrete yardstick for accelerating science.