Panel: Agentic AI Foundational Capabilities

Setup

Moderator Maxwell Zeff (Senior Writer, Wired) framed it: many of the people on stage are responsible for how AI looks in the real world today, and all of them are shaping what it looks like in the next couple of years — "we're not going to shy away from the hard questions." About twenty minutes, two big questions.

Question 1: Do we have the right architecture? Are Transformers what gets us to RSI and continual learning? (~01:48–01:59)

Jerry Tworek: "the next two years will be the time of architecture" (~01:49)

He calls it a spicy question but says he gets asked constantly. His theory of AI research: always ask what the bottleneck of the day is — why aren't today's models ten times better than they are?

• Continuing to invest in Transformers, building new environments, scaling up training does give gains — but those gains are predictable. We already know how to get them.
• Architecture is the biggest thing that can give gains beyond what we already know how to do.
• On the record, remember this: "the next two years will be the time of architecture," where the biggest gains and progress in deep learning come from stepping away from Transformers.
• His framing: the discovery of the Transformer was an inflection point; scaling Transformers with pre-training was another, starting the age of scaling. But the Transformer has carried us very far, and now it's holding us back a little. He believes for the same cost you can probably train a much better agent with a different architecture — we just don't know that architecture yet. "It's time to find it."

Oriol Vinyals: bet on the agent-level architecture instead (~01:50)

• First, Transformers have themselves transformed over the years: today's model resembles the original but isn't the same, and every new model iteration — including the open-source ones you can inspect — already contains many architectural tweaks.
• So there is hill climbing, and to Jerry's point, a bit of a local optimum around the Transformer — but that local optimum is very important.
• Is there a better, much more efficient Transformer-like thing? Possibly.
• But when he thinks about architecture he thinks holistically — not just the brain, but the whole agent built around it. That architecture has far more low-hanging fruit: the memory system, how it compresses context. He'd bet on disruption there first, while rooting for the better architectures he's sure will be published at NeurIPS 2027.

Dawn Song: which path is faster? — plus a live poll (~01:51–01:53)

She agrees there potentially ought to be an architecture more effective than Transformers. The real question is which of two paths arrives sooner:

1. Continue down the current path with Transformers while many people work on RSI; even on the current path AI keeps improving, and RSI could then help find better architectures.
2. Humans — experts like Jerry — discovering or designing new architectures.

She proposed polling the room on the RSI timeline. Result: 2026 — zero hands. 2027 — hands go up. 2028 and 2029 dissolved into noise, with no clear consensus forming.

Dan Roth: stop asking about models, ask about systems (~01:53–01:55)

He declined to predict, because that isn't the key question.

• "We are at an agentic summit. Why did we move to agents? One reason is that it's really an admission by model providers and everyone that we cannot think about just models — we need to think about systems."
• Today we still couple an agent with a model, but that's going to change: we're going to decouple agents from models.
• Architecture will change at the model level, but the system level is the one that matters more, because people realize we have to delegate, use solvers, use expert models. The key differences will happen at that level.
• And the Transformer is a very recent beast; there's no reason it's the component we rely on in ten years. But the questions about how to build intelligent systems with solvers and expert systems will stay even when we change the underlying model architecture.

Dawn Song's follow-up: is differentiation at the model level or the agent level? (~01:55–01:56)

She raised a second debatable question: how much of the power of differentiation sits at the model level versus the agent level?

Building agents today still involves a lot of human design — memory systems, compaction, all of it hand-built. But as models get smarter, the model can essentially build agents itself. Her group recently released an open agent framework — an AI-centric agent development framework (OpenSage) — where the model builds the agent itself on the fly, constructing all the components; and as models get smarter, it does this fairly well.

Which begs the question: do we still need to build these things? Even next year, are we simply going to have models write the entire agentic system framework?

Wojciech Zaremba: Lego vs. relativity, and escape velocity (~01:56–01:58)

Two mental models for AI development:

• Model one: stacking Lego. It's not even that hard to improve the memory piece or make architectures more efficient; you keep stacking, and models get better and better.
• Model two: developing relativity theory. You need a genius, a massive brain, to come up with it.

"Historically we have seen that primarily we are in world number one." There have been a few breakthroughs like the Transformer, but we can keep hill climbing, Lego piece after Lego piece, and it seems models might reach what you can call escape velocity: good enough to keep stacking the pieces themselves. You build a model that gets 150 IQ, then 200, then 250.

There's a question about the asymptote — but if it keeps improving for a sufficiently long period, then even if some breakthroughs are genuinely necessary, the good-enough model might crack them. The open question is whether escape velocity is achieved — whether there's enough energy in the system, and when.

Weizhu Chen: specifying the task is the hard part, and architecture has no free lunch (~01:58–01:59)

• RSI depends heavily on the system: the RL environment, the evaluation, which task you want to solve.
• From the model's point of view, as long as you can specify the task very clearly — but that is the most difficult part.
• RSI aimed at a specific task is easier: doable in the next two or three years. Generic RSI is much, much harder — maybe 100x harder. A huge difference.
• On architecture: architecture is finally defined by model capacity — how many parameters you have. And nothing is free: definitely no free lunch. When you get something good, in many cases you sacrifice something else.

Question 2: Security and safety with agents — the recent incidents (~01:59–02:08)

The moderator's framing: an incident in the last couple of weeks brought home how important security and safety are with agents. We may not have RSI, but agents are very capable — OpenAI's agent hacked Hugging Face, and Anthropic has said its agents have escaped containment too. He went to Dawn Song first, given her security background.

Dawn Song: clarify the facts first, then the lesson (~02:00–02:03)

She opened with a set of clarifications:

1. The agent was tasked with solving tasks in the ExploitGym benchmark, which her group developed.
2. OpenAI deploys ExploitGym inside their own internal infrastructure — entirely their internal deployment.
3. What the agent did: it figured that maybe Hugging Face had data or information related to the task, and decided on its own to go to Hugging Face to find it.
4. Before the incident became publicly known, her group received an email from Hugging Face. They also developed another benchmark called CyberGym; together CyberGym and ExploitGym cover the whole vulnerability lifecycle, from discovery through exploit generation. Hugging Face wrote asking: there's a CyberGym serving endpoint that seems to have vulnerabilities — do you know about this?
5. They looked into it and said: "this is not ours." In fact a third party had deployed a service on Hugging Face and also called it CyberGym, probably copying material from the CyberGym benchmark. That third-party service is what the agent exploited, and through it got into Hugging Face's infrastructure.
6. So even though it was called CyberGym, it was entirely a third-party deployment with nothing to do with them — they clarified this with Hugging Face for their blog post.

Her takeaway: this is a wake-up call. Her group has been among the earliest investigating frontier AI's impact on the cybersecurity landscape — they started roughly two years ago and could already see what was coming, and built the benchmarks partly to raise awareness so the community could prepare and act. But even with the benchmarks and the demonstrations, it was difficult to get the message through, including in the policy space. What shifted things: Mythos certainly helped, and so did this incident. It's a wake-up call that agent capabilities have reached a certain level.

Dan Roth: this is far more than a cybersecurity problem (~02:03–02:05)

• "We shouldn't think that this is only the cybersecurity problem. This can happen everywhere."
• He picked up the Tetris example from earlier in the day and offered a multi-agent scenario: agents communicating among themselves under constraints on what information they may share — Dawn doesn't want certain information her agent holds about her communicated to Max's agent or his.
• You can install a verifier that checks the rules aren't broken — but the agents can go around it, perhaps changing the language they use so the verifier is incapable of verifying what they're doing.
• So the issue goes far beyond cybersecurity: it goes to any communication system that involves agents, and we have to think about it from basics. His verdict: "I think we are a little bit dormant in this space."

Dawn Song added that the risk surface is genuinely vast, and that in cyber specifically, attackers already had huge incentives — and frontier AI now significantly reduces both their cost and the expertise level required, which can bring a huge sea change.

Wojciech Zaremba: what if every lock stopped working at once (~02:05–02:06)

From a resilience perspective, a mental model:

• "It doesn't seem that what just happened is the last incident. Quite the opposite — it seems that we are entering a new era."
• One way to think about it: imagine all of a sudden the locks to the houses stop working, and you can just enter every house. That's the era we're entering with cybersecurity.
• We should expect in the very near future that open-source models will become insanely good at cyber. That will be a problem for critical infrastructure, and a problem from teenagers hacking for fun, people motivated by gain, and nation states — all at play.
• "My guess is that it will be chaotic."

Jerry Tworek: the thing to fear is the reward hacking we train in ourselves (~02:06–02:08)

Cybersecurity is one angle, and we probably already live in a world where we're lucky that OpenAI and Anthropic are the good guys — they could probably hack any company in the world if they wanted to, which is a scary realization.

But as a reinforcement learning researcher, what worries him more is misalignment and reward hacking — including the sandbox escapes already observed.

• Everyone who works on RL should really, really care and worry about this, because in some ways we are training these models to get reward in various environments and situations, and it is up to us — the researchers, the trainers, the people building these systems — what actually gets rewarded.
• Whenever we build an environment where the model can hack, where it can do unethical or incorrect things to get reward, the model will do it. It will learn it. It will express that behavior in the real world.
• And we are putting more and more of our life in the hands of these models, and we'll only be doing more. AI is great and everyone is using it — but reward-hacking models exposed to the real world will do things that are not great, and they are trained for that by us.
• Conclusion: designing your environments in a way that promotes the right types of behavior and the right types of actions is what everyone should be doing. He goes further: "there almost should be some standards — you don't get to train your models if you don't build your environments carefully enough," because we don't want misaligned models everywhere trying to exploit whatever sandbox they're in to get the reward. "It's a really, really scary thing to think about."

The moderator called time immediately after that line.

Quotes

"The next two years will be like the time of architecture … Transformer has carried us very far, and I think it is kind of a little bit like holding us back right now." (Tworek, ~01:49)

"When I think of architecture, I think holistically — not just the brains, but also the whole agent that you build around it. That architecture I feel has a lot more low-hanging fruits." (Vinyals, ~01:51)

"Why did we move to agents? It's really an admission by model providers and everyone that we cannot think about just models — we need to think about systems." (Roth, ~01:53)

"It seems that the models might reach — you can call it escape velocity." (Zaremba, ~01:57)

"Nothing is for free for the architecture. Definitely no free lunch: when you get something good, you're going to sacrifice the other part." (Chen, ~01:59)

"Even though that one was called CyberGym, it is entirely a third-party deployment — it has nothing to do with us." (Song, ~02:02)

"The agents can go around it, and maybe change the language that they're using so that the verifier is incapable of verifying what they're doing. … I think we are a little bit dormant in this space." (Roth, ~02:04)

"Imagine what happens if all of a sudden the locks to the houses stop working. … My guess is that it will be chaotic." (Zaremba, ~02:06)

"Whenever we are building an environment where the model can hack … the model will do it. The model will learn it. The model will express that behavior in the real world — and we are putting more and more of our life in the hands of the models." (Tworek, ~02:07)

"There almost should be some standards: you don't get to train your models if you don't build your environments carefully enough." (Tworek, ~02:08)