RSI: Demystifying the \"Foom\

Topics and where each speaker stood

Dawn Song hosted; the guest was Jasjeet Sekhon, Chief Strategy Officer at Google DeepMind — previously Chief Scientist and Head of AI at Bridgewater Associates (co-founder of AIA Labs), a professor at Harvard, Yale, and Berkeley until 2021, now running cross-cutting strategy across research, commercialization, and policy at GDM.

Worth flagging up front: neither speaker actually defines "foom" (fast takeoff / intelligence explosion), the term in the agenda title. The conversation is really about whether RSI has arrived and what society should be building before it does.

The "Mythos moment" and what it changed (~02:00–02:03)

• Sekhon: the Mythos moment will be remembered as an inflection point in AI industry history — "if the Mythos moment doesn't make one AGI-pilled, I'm not sure what does." It was the moment Washington fully woke up, because an actual emergency occurred.
• The AI community already knew cyber capability was climbing fast (Song's group's benchmarks, a Google paper the previous year). What nobody knew was the timing: "I didn't know it was going to be February of this year versus December."
• Even so, Anthropic itself was surprised by how good Mythos was at offensive cyber operations, and key parts of the US government were caught completely off guard.
• His jab at the industry: "there's a tendency around here to think that the world goes from Napa to Big Sur — but it doesn't, especially when you possibly break things."
• The outcome is a pre-release government review process for frontier models in the US. Sekhon's verdict: the process is good and he's glad it exists, but it is clearly insufficient and not as clear as it should be — and the graver threats are still ahead.

Beyond vetting: what else has to be built (~02:03–02:09)

Sekhon turned it around: models will get released anyway, so what defensive capabilities need work? Song named two structural problems:

1. Coding and cyber are two sides of the same coin. People ask whether models can just be made worse at cyber, but the same coding capability everyone wants is what lets a model solve cyber tasks. Cyber capability rises with coding capability, necessarily.
2. Inherently dual-use, with asymmetric economics. Attackers need one working exploit; defenders must stop all of them. And critical infrastructure and hospitals are badly underprepared — she cited estimates that hospitals take close to 500 days on average to deploy a patch once one exists. Frontier AI attackers can find and exploit a vulnerability long before that window closes.

Her answer is to use AI to raise the defensive floor. Improving society-wide security posture has always been the goal; it stalled on limited resources and expertise. AI is the first thing that could make large-scale deployment feasible. She singled out her group's work on verifiable code generation: AI-written code is often vulnerable, but the same AI can automate theorem proving for program verification. She believes this is at an inflection point, and long-term it's the only way to shift the dynamics so AI helps defenders more than attackers.

Bio is next, and bio is harder (~02:09–02:12)

• Sekhon: cyber is attacker-privileged in the short run — you can already see generative AI salting open-source repos with bugs, and there's no shortage of weak systems (the US energy grid, hospitals). But cyber has an out: in the long run you can get provably correct algorithms ("everyone can use Rust").
• Bio is more concerning, and he thinks it stays attacker-advantaged even in the long run. His standard line: Jennifer Doudna, who created CRISPR, is here at Berkeley. She's a wonderful person — but if she decided it would be interesting to make a virus that could harm many people and convinced her lab to do it, that is well within her ability. Same for the Broad Institute and key biological institutes worldwide. And we may be close to a world where someone designs a virus or protein by talking to a model in natural language and then gets it synthesized.
• Google's thinking (informed by protein folding work and Isomorphic Labs) splits three ways:
• Prevention — watermark AI outputs. Google has SynthID; he argues biology needs the equivalent, so labs can see "this proposed protein was designed by generative AI."
• Monitoring — regardless of whether a pathogen is designed or natural, surveil much better. Wastewater monitoring, which became a real thing during COVID, is his example.
• Response — Isomorphic Labs recently announced a rapid-response effort for newly arriving pathogens.
• Structurally, the key precursors for synthetic biology need close monitoring and almost certainly a licensing regime: who can hold them, how they get tracked. Precedent: the US began tracking fertilizer after a terrorist attack. Biologically relevant materials will need far more of that.

Is the buildout sustainable? (~02:12–02:16)

This is Sekhon's strongest stretch.

• It is the largest scientific bet our civilization has ever made — larger than Apollo, larger than the internet buildout, larger than the Manhattan Project. The only bigger capital expenditure in history is the railroads, and the railroads weren't a scientific bet: we knew how to build them; the only question was the business case.
• The reason is good: "we appear to have found a way to turn energy into compute and compute into intelligence." As long as that machine works and the scaling curves hold, we'll keep going — and he wants us to. There are diseases to cure and a cosmos to explore.
• The global financial system is reorganizing around it: companies with hundreds of billions on the balance sheet spending it on capex, plus Middle Eastern and other capital pools being reallocated.
• The risk is that it's still a bet. In this cycle the revenues don't sustain the expenditures — "if the revenues were there to sustain it, it wouldn't be a scientific bet, it would be a commercial reality." Hence the danger of an AI air pocket: the spending happens, the revenue doesn't show up, and at some point markets react.
• Geopolitically: the US is outspending its near-peer, with more compute and more capital; China has more energy, and the US is racing to bring energy online. He calls the whole thing "one of the miracles of capitalism — that the entire planet can reorient to a new target."

What is all this compute for? (~02:16–02:18)

Song named two frontiers:

1. The future of software engineering. Her group's new paper (see the resources table) argues that today there's still heavy human supervision — humans still have to tell the agent what to do, especially on long-horizon tasks — but we're moving quickly toward higher autonomy levels. The paper lays out three levels of autonomy, describing how responsibility transfers from humans to AI across the entire software development lifecycle, ending with AI designing what gets built and deciding what should get built.
2. Recursive self-improvement. As AI gets more powerful it helps train better AI and develop better algorithms and systems, compounding capability — faster improvement, with its own risk set.

Is RSI real, and on what timeline? (~02:18–02:21)

Sekhon's most complete argument of the session:

• A framing observation first: the AI community has followed science-fiction timelines, and for good reason — "if you don't believe it, you can't build it. Building is hard. If you don't believe, you can't build." So RSI is not a thing that exists; it's a thing we want to build — a flag planted by von Neumann and I. J. Good (a statistician and cryptography expert), pointing at the intelligence explosion that occurs when machines make the next generation of intelligent machines.
• What we observe today is not RSI in von Neumann's formal sense — it's early precursors. AI systems clearly make the next generation of AI systems better and faster, but that shouldn't be shocking: steam engines were used to build the next steam engine; existing compilers build the next generation of compilers. It's happening and it's accelerating.
• The multi-trillion-dollar question is whether you cross the threshold. Von Neumann's famous paper argued you need a simpler machine to make a more complicated machine — an unusual property requiring a self-coherence complexity threshold. Below it, "the machine just loses the plot and the humans have to get involved."
• His timeline: the next couple of years. "Betting against it would appear to be unwise." His evidence is experiential: three years ago these systems were worse at math than his middle-school daughter; now they're better at math than he is — "and unless there's a Fields Medal winner in the audience, probably better at math than you."
• This connects straight back to capex: the frontier gets commodified relatively fast (12 months? 18? 6? debatable), but on an exponential capability curve you can keep funding the next model cycle. Hit RSI and that curve goes hyperexponential — which is a key part of the investment thesis and the scientific thesis behind what society is currently spending.

Why they both signed "Pacing the Frontier" (~02:21–02:27)

Song's motivation:

• Signed by more than a thousand AI researchers and people at frontier labs, sharing one concern.
• "The letter is not about slowing down AI development, at least not now." The concern is that development is already fast and RSI could make it faster, while everyone in the camp wants society to benefit from AI's power — done safely, securely, beneficially.
• Her core argument: pacing mechanisms are heavy-handed and extremely difficult to design well; done badly they do more harm than good. And the last thing you want is to build one during a crisis, when getting it right is hardest.
• So this is precautionary capacity-building: we don't know when — or whether — we'll need it, but having the capacity means we can move fast if we do. She argues this actually lets society develop AI capabilities faster, because it comes with assurance that it can be done safely.

Sekhon's motivation:

• "I signed for very similar reasons: I want AI to progress as fast as possible — but if it leads to errors, if it leads to harms, we'll lose the social license to innovate, and we cannot lose the social license to innovate."
• He thinks people here underestimate how easily governments and other actors could say no to data centers: "they'll probably shut off the water too, even though we're not big users of water — but people are misinformed on this." AI's standing with the public in Western countries is, in his words, in dangerous territory.
• Three compounding reasons governance is hard: a new, very fast technology; post-WWII institutions (the UN, NATO) that are themselves shaky; and a near-peer rising in the east — and history says a rising near-peer is itself an additional stressor on the system.
• It can't be done nationally alone. The US can't just decide something unilaterally and have it hold. Being ahead is the asset: it buys the ability to set the agenda and bring other countries along. He signed "to make common knowledge that we need to have these discussions now, before we need these tools."

On Demis's proposal (Song asked):

• Hassabis's proposal has gotten a lot of traction. The core is the lightest-touch regulation possible to just get started, adapted to something that moves very fast.
• Structurally a public-private partnership funded by industry, because it needs billions to hire the best people and hold the compute to run the analyses.
• Explicitly not the FDA model: the FDA works on the timescale of years while capabilities move on the timescale of months, and it was created to stop snake-oil medicines rather than to cure lives — so it's slow by design. This needs to be fast and adaptive.
• Mechanism: a board that defines what a frontier model is, with benchmarks. If you're a frontier model, you must be vetted to access the US market — open weight or closed weight, from China, Germany, or the United States. If you're not, it's very light touch, leaving academia and startups free to move fast.
• Status: a work in progress that "looks quite good," though "it's very hard to do anything in DC at this current moment."
• A side note he volunteered: the Mythos moment has made the industry behave more adult-like.
• He closed by rejecting the accel/decel binary: "I want RSI. I want RSI as fast as possible. I just want it — and I think Dawn wants it — in a way that's sustainable: vigorous competition, robust safety, and an innovative ecosystem without capture by certain industry players."

Closing (~02:28–02:30)

Sekhon's parting advice: if you're fortunate enough to work on AI, this is not the time to sleep. This isn't a normal moment in human history — he calls it the industrial revolution and the renaissance combined. We're on a stretch of the curve with unprecedented allocations of human talent, capital, and energy. And the history of technology, of government reform, of everything says early decisions have very long-lasting echoes: decisions made today may have unexpectedly large consequences 20 or 50 years out.

Song closed where her morning opening remarks did: this is a critical point, we need to act now, and hopefully society together makes the right decisions going forward.

Quotes

"If the Mythos moment does not make one AGI-pilled, I'm not sure what does. … It is the time when Washington fully woke up." (~02:01, Sekhon)

The significance wasn't the technical result — it was that the political system finally paid attention.

"There's a tendency around here to think that the world goes from Napa to Big Sur — but it doesn't, especially when you possibly break things." (~02:02, Sekhon)

A direct shot at the Bay Area bubble: what you might break isn't inside your commute.

"Coding and cyber capabilities are really two sides of the same coin." (~02:04, Song)

Which is why "just make the model worse at cyber" isn't an available move.

"This is the biggest scientific bet our civilization has ever made. … The only capital expenditure that we've ever done that is larger is building the railroads. And the railroads were not a scientific bet." (~02:13, Sekhon)

Apollo, the internet, and the Manhattan Project all cleared; the one bigger thing wasn't a scientific bet.

"We appear to have found a way to turn energy into compute and compute into intelligence." (~02:14, Sekhon)

The entire capex thesis in one sentence.

"If you don't believe it, you can't build it. Building is hard. If you don't believe, you can't build." (~02:18, Sekhon)

Why the field runs on science-fiction timelines — and why that's a feature.

"You need a simpler machine to make a more complicated machine. … Otherwise, the machine just loses the plot and the humans have to get involved." (~02:19, Sekhon, paraphrasing von Neumann)

The technical bar for RSI: a self-coherence complexity threshold.

"Betting against it would appear to be unwise. Three years ago, these AI systems were worse than my middle school daughter in math. And now they're better at math than me." (~02:20, Sekhon)

His empirical intuition for the RSI timeline.

"If you hit recursive self-improvement, that curve will go to hyperexponential — and that is a key part of the investment thesis." (~02:20, Sekhon)

RSI isn't only a safety question; it's the pricing assumption under this capex cycle.

"The letter is not about slowing down AI development, at least not now." (~02:22, Song)

Correcting the most common misreading of Pacing the Frontier.

"We'll lose the social license to innovate, and we cannot lose the social license to innovate." (~02:24, Sekhon)

His case for governance is an accelerationist case, not a decelerationist one.

"I want RSI. I want RSI as fast as possible. I just want it … in a way that it's sustainable." (~02:28, Sekhon)

An explicit refusal of the e/acc-versus-decel frame.