Fireside Chat: Ali Ghodsi × Andy Konwinski

Who's talking

Moderator Aaron Jacobson (NEA) notes that NEA first backed Databricks at the Series B in 2014. Both came out of Berkeley's Mesos/Spark line: Ali was one of the creators of Apache Spark, with research in resource management, scheduling and data caching applied to Apache Mesos and Apache Hadoop; Andy did his CS PhD at Berkeley, contributed to Apache Hadoop and co-created Apache Mesos and Apache Spark, and now co-teaches a PhD seminar on research-to-startups at Berkeley.

Theme 1: Berkeley in 2009, and the mayor who fell asleep (~01:38–01:42)

• Andy's memory: in 2009 an email went out — from Ion Stoica or Scott Shenker — to the Mesos group announcing a visitor joining the team. He, Matei Zaharia and Ben Hindman were "very insecure" and mostly wondering who is this elite guy, and who's going to be first author now? Then Ali walked in, jacked, and Andy thought: "I hope we don't have to arm wrestle for this first author." Ali went on to the Dominant Resource Fairness (DRF) paper and became a mentor and effectively one of Andy's co-advisors.
• Ali's memory: 2009–2012 Berkeley was "at the frontier of all these new algorithms for the new computer — and the new computer was the data center." There was a lot of skepticism at the time about AI, machine learning and data centers. He calls those his best and most formative years.
• His favorite story: early in Databricks' life there was a real debate — run the company out of Berkeley or San Francisco? The founders split down the middle. They met the chancellor, who told them to talk to the mayor, so they brought the mayor of Berkeley into their not-very-nice office ("because Berkeley doesn't have great office space"). He arrived with an entourage of politicians, and a group of researchers proceeded to pitch them on advanced scheduling algorithms. The mayor dozed through the whole talk, woke up at the end, shook a hand and said:

"I hope that one day I will have a data brick in every home."

Both agree it still hasn't happened — "though maybe NVIDIA's most recent rack is where we go someday."

Theme 2: What actually makes Berkeley special (~01:42–01:44)

Ali: before Berkeley he did several postdocs, living in roughly nine countries and visiting nine universities. Both the city and the university are special, and what makes UC Berkeley special is an attitude: you can change anything, you can question anything.

It was a culture shock. He'd look at what the group was working on and think "that's not research, we're not supposed to do that," and the answer would be "no, we're doing it, it's cool, it will have impact." He'd protest that it made no sense, that it felt like cheating somehow — and the answer was always "we can do anything, we're just here to change the world."

He admits it sometimes goes to extremes, but that's the cost of a wide aperture: alongside very high-impact work there was research on delay-tolerant networks, on the premise that "obviously the internet is broken if we want to do interplanetary communication, so we have to redesign a new internet for it, because eventually we're going to colonize all of the universe."

Andy's companion story: they were an operating-systems and distributed-systems group taking deep inspiration from BSD. When naming what became Mesos, his candidate was BCD — Berkeley Cloud Distribution / Berkeley Cluster Distribution — one edit distance from BSD. Matei and Ben both said no: "who are we to be so lofty as to name ourselves like only one edit distance from BSD?" Andy's answer: "well, we're going to build the more impactful operating system — that's who we are." He lost the vote. (The official nomenclature at the time was BDAS, which you were required to pronounce "badass.")

Ali's verdict: that's the Berkeley spirit — question the norms.

Theme 3: Why Databricks keeps open-sourcing — MLflow and Omnigent (~01:44–01:48)

Ali's strategic view: open standards and open interfaces matter because they create communities, and those communities become very powerful. He puts it bluntly:

"Databricks would not have existed as a company if it wasn't for the Spark ecosystem — because the Spark ecosystem was way bigger than anything any of us would have done."

So it's good for the community and the world, and it's a good business idea — Databricks has proven that out.

• MLflow: people don't realize it still does 40 million downloads a month. It came from the ML era about ten years ago, when everyone was obsessed with having the best model but it was very hard for ML practitioners to iterate across models. They needed something like git version control to iterate and keep track of all the experiments. That was MLflow.
• Omnigent (launched this summer): the problem is that there are simply too many harnesses. Ali draws a sharp contrast:

LLMs are not sticky; harnesses are. Because Databricks serves both proprietary and open-source models, they see all the data — and what they see is that when a new proprietary or open-source model ships, people move to it within days. "We've never seen anything like it. Anything else in software is sticky — you stick with it, there's a decay, people slowly churn and migrate, but it takes a while. You don't just instantaneously flip-flop. With LLMs that is the case." But harnesses get built on top of the models, and harnesses lock you in like old software: you get used to the interface, the memory, the keyboard shortcuts.

So Omnigent is a meta harness — a harness of harnesses — that lets you switch freely: it works with Claude Code, with Codex, with any of them (OpenCode, Pi, you name it), giving you multiplexing plus the community.

Traction: 1,000 GitHub forks in the first couple of weeks, and hundreds of contributors within weeks.

• Ali's long view on open source: it's headed for a huge boon over the next several years thanks to vibe coding and AI. It used to take enormous effort for a Linus Torvalds to build a whole operating system and compete with Windows, and eventually someone has to pay his salary so he gets hired by a company. But the cost of writing software is going down, so expect many more open-source projects — and more of them becoming standards.

Theme 4: Specs over implementations, and "PRs that contain prompts" (~01:47–01:49)

Andy's prediction: we're heading toward a version of GitHub where pull requests contain prompts rather than code, and reviewers decide by running those prompts. He thinks things like OpenClaw — and probably Spark — are already drifting that way, where a committer's job becomes judging how much of the code was written by a human and how many hours a human actually spent on the PR.

Ali's response: he agrees with the direction — if you can specify what the program should do and you have the testing foundation and harnesses, the implementation doesn't matter. The AI can find the fastest, best implementation, and may rewrite it many times over. But that requires a really airtight way of specifying the code.

Theme 5: Databricks' own eval — Office QA (~01:49–01:52)

Ali on the work they recently announced: they scraped a large volume of their own internal coding issues and tasks into a dataset of over a thousand items. The source: roughly 4,000 engineers writing code every day, and a company of about 11,000 people submitting questions — data science questions, questions about how the business is doing, questions arising from working in the field.

Andy's reading: that's essentially a declarative spec — you've declaratively encoded requirements as questions and answers. People ask a question, you have the Databricks answer, and now you have a specification for something an agent could learn to do: "here's a bunch of questions we wish our agents could answer." A bespoke evaluation / benchmark for Databricks.

Why Ali built it: partly contamination — something new is released every day, so they wanted their own ground truth to answer "which model should we use, and can we just use GLM 5.2? Is it going to be as good or not?"

And partly an observation about the frontier labs:

"The frontier labs are so obsessed with superintelligence and the march toward superintelligence that many of the things we want to do every day are not part of their evals."

His contrast is HLE (Humanity's Last Exam) — very complicated math and physics questions. But when Databricks tested the very basic tasks they do daily, frontier models weren't doing well: around 20% correct on the hardest ones, maybe 40% on average ones.

His example is deliberately mundane: "here is the 1940s budget of the United States — just get us what the defense spending was." The model has to scan the PDF and figure out how to present things across four dimensions. "It turns out that's really hard," because the focus has been on physics and math — things you can RL — rather than mundane enterprise or business tasks.

So they built an eval called Office QA (named after the movie Office Space, of TPS-report fame). Focusing on it, they built an agent that could do the PDF parsing and reached close to 80% correctness.

Ali's conclusion: evals and specs are going to be extremely important going forward.

Theme 6: Where the K Prize and Terminal-Bench came from (~01:52–01:54)

Andy's thread: they teamed up with Stanford professor Ludwig Schmidt, whose group had done CLIP and the DataComp family of projects. DataComp's innovation was inverting the usual setup: instead of changing the model while holding the dataset fixed, you hold the model and agent fixed and change the dataset, with a leaderboard for how far a better dataset can push one fixed agent.

He met Schmidt at NeurIPS — the same NeurIPS where he launched the K Prize:

• $1 million, promised from the stage in a tweet, with Andy writing the check personally.
• To the first team to reach 90% on a contamination-free version of SWE-bench.
• The mechanism: a submission cutoff date by which teams had to submit their agent code; only afterward did they scrape new GitHub issues from the four months following (from popular repos like PyTorch or Spark) and run the already-frozen agents against them. Training on the test set becomes impossible.
• The result: roughly a 10% success rate — which Andy says is really good for a new benchmark.

Ali's reaction, the best contrast in the session:

"Isn't it crazy that everything is 90% saturated, but when you do a new one it's like 10%?"

Theme 7: Write benchmarks like software, don't start over (~01:54–01:57)

Andy's central argument, from work they announced a couple of weeks earlier:

Today's practice is a very inefficient cycle — every new benchmark mostly throws away the last one and starts from scratch. His own experience is the evidence:

• They shipped Terminal-Bench 1, then 2, then 2.1.
• The whole of 2 → 2.1 was mostly fixing five to ten tasks the community had discovered were essentially unsolvable.
• Most benchmarks don't even do those minor version releases.
• The result is a shadow culture: "people who knew Terminal-Bench 2 really well knew that 83% was actually 100%," because the rest were unsolvable. "You don't want that if you're a benchmark maker. You want 100 to be achievable, but just be really hard."

His proposal: write benchmarks like software. You don't throw away the codebase every time you cut a new version of Spark; you change the codebase and fix bugs. Translated to benchmarks:

1. Delete saturated tasks — if everybody gets them right now, they don't belong in the benchmark.
2. Add ten new, harder tasks.
3. Ship every two or three weeks, continuously — "like software, and you're fixing bugs."

After Terminal-Bench 3 it'll be 3.1, 3.2, 3.3 — fast, relatively small iterative updates.

Ali's addition: what he likes about it is Goodhart's law — whatever metric you use to evaluate something ceases to serve its purpose because it gets gamed. That's exactly why everything sits at 90% everywhere, and a continuously updated benchmark is the natural defense.

Andy, one layer further: it also lets you gently steer. Reward hacking is now part of the major frontier — "very few people were talking about reward hacking a year ago; now a lot of people are thinking about new tooling around it, and agents are getting better at identifying it." So they want more tasks that are robust to reward hacking, or that explicitly test an agent's resilience to it — and under the new model, "we can come up with three new tasks and cut a new version of the benchmark in a week or two."

Theme 8: Where humans should spend their brainpower (~01:57–01:58)

Andy pulls the threads together, explicitly endorsing Ali's earlier point: as long as you have good evals, the agent and harness code will increasingly be optimized automatically.

The paradigm shift he describes:

• Early days (LangChain, LangGraph and similar frameworks): humans thinking hard about how to architect the agent.
• Now: humans spending all their brainpower on designing the evaluation — "how do you accurately capture the distribution of hard things inside your company, or your government, or whatever you're trying to optimize?" — and then unleashing optimizers to hill-climb those evals: code optimizers, prompt optimizers, model-weight optimizers, or all of the above.

Those two trends together are where he and Laude's Harbor open-source team are focused right now.

Theme 9: Open Frontier — does open science stand a chance? (~01:58–02:02)

Ali's question: Open Frontier is an initiative Andy's team is spending a lot of money on, backing open research and open science. Against closed frontier labs with enormous salaries, deep secrecy and no published results — does the open side have a chance?

Andy's answer, in three parts:

(1) The diagnosis. He thanks Ali for speaking at the meeting a month earlier — they assembled about a hundred of the top AI researchers still doing open science to talk about the fate of science as we know it. The core observation is that top minds are leaving academia, and two things are happening at once:

• Salaries: from $300–400K to $10M–$100M.
• Resources: from a lab with $10–50M in funding to labs with billions and hundreds of billions.

He says this is insane even by the standards of founding Databricks ten years ago — people now have billion-dollar comp packages. Those two things are why he genuinely worries about the future of science.

(2) Can they compete head-on? No. Nonprofit research is funded by philanthropy and the US government — including the university they're sitting in — and will never raise hundreds of billions at the level of an Anthropic or an OpenAI.

(3) But they do need one to two orders of magnitude more than they have now. That requires a new shape of fundraising, and a way of teaming up that academic labs have never really needed — what he calls a "lab of labs" architecture:

Somewhere between how academic labs collaborate today (mostly by reading each other's papers and building on the ideas) and how Meta, OpenAI and Anthropic operate (centrally run, laser-focused on one stack and one model) — a looser but still aligned collaboration of many labs.

The pieces he names:

Plus independent researchers: nonprofits are blowing up right now — not at Anthropic scale, but five- or twelve-person teams, and actual nonprofits, not neolabs, not for-profits.

He's candid about the operational challenge: you have to herd those cats — "first of all get all their faces on one slide," and second actually operationalize it. How often do you meet? How do you agree on artifacts? How do you connect to the models Marin is producing, or to labs putting out open-weight models — Thinking Machines, or Reflection, which claims to be working on an open model and to be releasing one eventually? How do you get the sum to be greater than the parts in a nonprofit?

Momentum: they've been talking about this for about a year, and now Satya Nadella has tweeted about the need for this sort of thing, while Databricks is showing the way for the possibility of this open ecosystem actually standing a chance. Ali says he's very excited about it.

Theme 10: What to tell Berkeley undergrads — is CS hopeless? (~02:02–02:07)

Andy's question: we've both taught and mentored undergrads here and brought some along to Databricks; you still pay hundreds of engineers to build frontier infrastructure. Is it hopeless to come to Berkeley? He gave the commencement speech here, students were deeply worried, and families came up afterward saying "we're confused — what should we do?"

Ali's answer — the longest and most persuasive stretch of the conversation:

He starts with the sentiment he thinks is wrong: "at the frontier labs in particular there's a sentiment that there's only two more years of work left and then everybody's going to be out of a job and programming is going to go away. You meet more and more people who feel that way, especially in SF."

"I think this is completely wrong."

His argument is that societal change is intrinsically slow:

• Electricity: from 1880 to 1920 the electric engine replaced the steam engine before society could really reap the benefits, and it took until 1920 to see any impact on GDP. And you can't wave that away as "it was physical, so it was hard" — the electric engine already existed in the 1840s and 1850s.
• The PC revolution: same story, took a long time.
• His own company: Databricks has AI and is trying to identify everything it does across the organization — "you just see how hard it is. You have so many tasks that have built up. We've only existed for 13 years; some organizations have been around 50, 100 years. It takes a very, very long time to change everything you're doing."

So: the agentic revolution will take a decade at least, if not more.

And coming up with good ideas is itself slow. His favorite example: the internet already existed by 2000, but it took Brian at Airbnb until 2009 — ten years or more after the internet, with no physical infrastructure needed; people already had houses they could rent. It just takes a long time to come up with good ideas and for them to percolate and disseminate through society.

"Most of the interesting things that are going to happen haven't happened yet."

Two directions he keeps returning to:

1. Healthcare: "there's going to be a healthcare company, and it's probably going to be worth trillions of dollars" — where you get a doctor that has seen billions of patients and can give you personalized care. "People would be willing to pay anything for that, if you could help the health of you and your loved ones."
2. Education: VCs have always said there's no money in education, but "if you look at the United States, elections are won and lost based on education, and people care about the education they give their kids." The technology is transformative here, so expect innovation from universities, nonprofits and companies alike.

His close:

"This is the best time ever. I wish I was born so I would have come out now when I'm 20, 25 … If you came out 10 or 20 years ago, things are more smooth. This big transformation is the best time to be alive and be part of it."

And on CS itself: computer science has far to go, and we're going to see way more software written in the future than we've seen in the past. "So I'm super, super optimistic."

Andy's echo: he's "very parallelly optimistic." At Laude he still works with PhDs in partnership with labs at Berkeley, Stanford, MIT and other top universities. His observation:

Find the students who ship something — launch an open-source project, tweet a paper and get a large response. Talk to those students and you find they're motivated by impact, exactly the way he and Ali were back in the Patterson labs thinking about BSD and how they could change the world.

He asks them directly: why say no to those $2M, $5M or $10M-a-year comp packages? Those roles let you do frontier research and shape how Claude works — but you won't be able to talk about it. The answer is always the same:

"Getting to work on the problem you see as the visionary one, and a commitment to disseminating that to the whole world, is more impactful in the end — even than the proxy that this $10 million or $50 million check represents. That's also representing impact, but it's kind of a lagging indicator."

Ali's final word: "Yes, you can make a lot of money at the frontier lab, but you'll never ever have the chance to have as much impact — because during this transformation you can do things where assumptions are changing and new ideas can flourish. This is the time to do that."

Quotes

"I hope that one day I will have a data brick in every home." (the mayor of Berkeley, as told by Ali, ~01:42)

Said by a mayor who had slept through the entire pitch on scheduling algorithms. "Still hasn't happened."

"There's this kind of attitude that you can change anything, you can question anything." (Ali, ~01:42)

His explanation of what makes Berkeley Berkeley — and a culture shock when he first arrived.

"Anything else in software is sticky … With LLMs that is [not] the case." (Ali, ~01:46)

Databricks serves both proprietary and open models and sees the data: people move within days of a new release. But harnesses lock you in — which is exactly the problem Omnigent targets.

"Isn't it crazy that everything is 90% saturated, but when you do a new one it's like 10%?" (Ali, ~01:54)

Prompted by the K Prize result; the absurdity of benchmark saturation in one line.

"We should write benchmarks like software … you don't throw away the codebase every time you do a new version of Spark." (Andy, ~01:55)

The most actionable claim of the session: delete saturated tasks, add harder ones, ship every two or three weeks.

"People who knew Terminal-Bench 2 really well knew that 83% was actually 100%." (Andy, ~01:55)

The shadow culture unsolvable tasks create — the last thing a benchmark author wants.

"Now instead you have your humans spending all their brain power coming up with the evaluation." (Andy, ~01:57)

The shift from humans architecting agents to humans writing evals while optimizers hill-climb them.

"The internet already existed by 2000, but it took Brian at Airbnb till 2009." (Ali, ~02:04)

Why ideas are the bottleneck, not infrastructure.

"This is the best time to be alive and be part of it." (Ali, ~02:05)

His answer to the anxious Berkeley students and their families.

"You can make a lot of money at the frontier lab, but you'll never ever have the chance to have as much impact." (Ali, ~02:07)

The closing exchange; Andy's version is that the check also represents impact — but as a lagging indicator.