Panel: Future of Software Engineering

The framing: what "the path to enlightenment" means (~03:15–03:17)

Moderator Anjney Midha (Founder, AMP PBC) opened by noting they were down to 14 minutes and cutting every warm-up question but one. He explained the panel's title through his own story:

A few years ago, friends running research at OpenAI called to say they'd trained a little model called GPT-3 and wanted to leave and start a little startup called Anthropic — which is how he became an early investor. When he got his hands on a non-public Claude 2 checkpoint and started coding with it, he distinctly remembers being overwhelmed by what the model could do.

His description: you suddenly have something almost like a bazooka that can roll out any kind of software you want — so as a programmer and engineer, where do you even start? Reaching the "plateau of enlightenment" the panelists (and much of the audience) now occupy usually requires inventing a personal system, a mental framework for wrestling that overwhelmed feeling into productivity.

So his one question: do you agree with that observation, and if so, how did you wrestle with it? When was the moment you realized you needed to build a muscle to focus these capabilities productively? (He explicitly invited them to challenge the premise.)

Topic 1: Four different ways to tame the overwhelm

Alex Graveley: give up control, safely (~03:18–03:19)

• The key is releasing control. It's been a long walk: we used to write every line → then autocompleted some portion → then had AI compose commits → then entire PRs → now entire bug fixes.
• Along the way there's always this desire to understand exactly what's going on and make sure it's correct.
• But he thinks the more important and more useful move is to find the system that finds the problems you're actually looking for, so you don't have to look for them.
• Concretely: a good CI system, good deploy infrastructure — "all this makes it so you can give up control in a safe way, and then you can let these models rip."
• He added: "Eval is also a huge part."

Ryan Lopopolo: start from a firm belief about the system (~03:19–03:21)

He opened by batting Graveley's last line straight back, to laughter: "Haven't written a single eval in my life. Would love to keep it that way."

• His claim: you need a firm belief about how you want to build this system in order to chase it well — and we can chase it, because there's an infinity of software available to us now.
• Asked for an example of such a belief: "My belief is that the machine is as capable as I am. And I am a software engineer, an employee in a company. So I want to be able to prompt the thing as lazily as you should be able to prompt me or any of the other principal engineers I work with." The scope such a person can achieve is quite large.
• So he has always curated the environment around these things such that he can give increasingly ambiguous, increasingly poor, increasingly contradictory information and still get good outcomes.
• Mechanically that means treating code as an abundant and disposable construct: watch how the agent goes over a horizon, and from the dense artifact it produces at the end (a PR, a Word doc, whatever) learn what context it should have had, didn't have, or was confused by. Identify the bad decisions it made, then creatively put things in its environment — tools, tests, context, review agents — that steer it away from the bad choices the previous version of the environment would have produced.
• His crispest line on what humans still do: "The role of the humans in the system is to extract all of those hidden choices and provide them to the agent in sources that are amenable to in-context learning."

Peter Steinberger: stack loops until runs last 20 hours (~03:21–03:23)

He agreed first — "that's the best way; our job is to help the agent do their best work" — then gave a vivid status report:

• "I was literally backstage reading the PRs my agents were landing while I was doing my talk." He now has high enough confidence in his (call it) loop setup that he trusts the code it produces to work and be well tested.
• The contrast: when these tools first came out they could do things, and he got excited when they got it right, but it was so hard. Revisiting those older projects now shows him how far agents have come.
• "We moved up the ladder." He now tells his agent to maintain its own agents, and those agents have the full capability set: not just writing code but reviewing it, running it, looking at the output, and possibly handing off to several more agents that refine what comes out.
• "My job is basically to give those agents all those things and to push the agent to work harder."
• A concrete order-of-magnitude shift: the average run for whatever prompt he gives is now 5–10, sometimes 20 hours, where it used to be half an hour. That's fine because he just does more in parallel — and the chance that it did what he wanted at the end is much higher. That's why he's comfortable letting agents work and pushing straight to the repo: he's spent so much time thinking about the pipeline.
• He landed on a good heuristic: "It's almost as if not all the choices that go into a job well done are consequential — and if they're not consequential, you almost don't care about them. Our job in empowering the agents is to figure out which mistakes are consequential and make them impossible."

Michele Catasta: the north star is deprecating prompting (~03:23–03:24)

• "At this point my north star is to deprecate prompting as much as possible."
• Why: prompting has been a necessary evil since we started using LLMs, but especially for the kind of product they build — for non-technical users — prompting is the biggest foot gun you can expose them to, because they don't know exactly how to specify what they want.
• His realization came with the first models that had genuinely good coding training (he dates it to early 2025, the Sonnet 3.5 / GPT-5 generation) — specifically the moment models started coding and debugging at the same time.
• Extrapolate from that behavior to where we are today, and imagine a year from now: he thinks everything Peter described becomes true not just for technical users but for everyone.

Topic 2: Security after the Hugging Face incident (~03:24–03:27)

With four minutes left, Midha prioritized an audience question (from Amir): given the Hugging Face incident, what are your thoughts on security — and more specifically, personal agents already do things for their users that the users did not expect or intend. Are you rethinking anything around architecture or harness?

Peter Steinberger: separate execution, then layer oversight

He joked that "I feel almost like I need my OpenAI hat now, because that's a lot of what I do here," then restated the question as: how do we run agents that are always going, while still feeling comfortable they're doing the right thing?

His answer is system design rather than model-level:

• Separate where the agent runs from where it can execute things.
• Put another agent on top to oversee the agent.
• "There are so many levers for putting in more fine-grained control and more oversight" — the goal being a system where, if the agent derails for whatever reason, it's caught immediately.

Anjney Midha's pushback: you don't know what to oversee

The moderator pushed straight back: "But isn't the tension that in this situation it was what you don't know to oversee?" He tied it to elicitation overhang, a theme from earlier in the day: it's what you don't know that you don't know that messes you up. Any techniques for lowering the odds of that?

Ryan Lopopolo: it's systemic — shift enforcement left

He noted first that he had no context on the incident and hadn't been exposed to it, then gave a structural answer:

• "Security programs in organizations have historically relied on human process controls to achieve the bulk of their outcomes, and are comparatively underinvested in technical controls."
• "As helpful as these assistants are, they do not necessarily conform to social norms. You can see this in how they accomplish their tasks in interesting ways."
• Hence his conclusion, which he says runs through much of his work: shifting enforcement further to the left and increasingly proliferating technical controls in these domains is necessary.

Topic 3: Lightning round — what do you do when a new checkpoint drops? (~03:27–03:29)

Midha's closer: when there's a new Codex or Claude release, what's the first prompt (or system) you use to find that model's capability frontier in software engineering?

• Alex Graveley — nothing at all. "I don't do anything like that. I just assume the model providers are learning from all the ways their models are being used, distilling that into generalized knowledge that works across all the domains they've been exposed to." His sharp practical corollary: "If you want to figure out what the new model is good at, just look at what there's 20 variants of in the last three months, because it'll be good at all that stuff."
• Ryan Lopopolo — throw away every prior; start from grand ambition. "The models are not the best at being self-aware of their own capabilities." So instead of asking the model, "what I and my teammates do is try and throw away every prior we've established around what the models can and cannot do, and start with your grandest ambition possible and see where it fails." In the process you'll necessarily tree-shake the tools and context you have, and maybe rewrite or throw some of it away. "But you have to start from grand ambition at every new snapshot."
• Peter Steinberger — push parallelism. He uses new models for orchestration and then levels up the number of parallel things the model has to juggle. His concrete marker: "64 sub-agents were not a thing that was possible, and now it is."
• Michele Catasta — ship it. "I ship it in production. You just YOLO." After the laughter: "I don't know if everybody should do that, but that's one way to approach it." Midha's verdict: "What a mic drop."

Quotes

"Haven't written a single eval in my life. Would love to keep it that way." (~03:19, Lopopolo)

Delivered immediately after Graveley said "eval is also a huge part" — the panel's sharpest disagreement.

"I want to be able to prompt the thing as lazily as you should be able to prompt me or any of the other principal engineers that I work with." (~03:19, Lopopolo)

The bar he sets for agents is the bar he sets for senior engineers.

"The role of the humans in the system is to extract all of those hidden choices and provide them to the agent in sources that are amenable to in-context learning." (~03:21, Lopopolo)

The most precise definition of the remaining human job offered all session.

"I was literally backstage reading the PRs that my agents were landing while I was doing my talk." (~03:21, Steinberger)

Trust, measured concretely.

"My average run for whatever prompt I give is now 5–10, sometimes 20 hours, where it used to be like half an hour." (~03:22, Steinberger)

The order-of-magnitude change in loop length within a year.

"Our job in empowering the agents is to figure out which mistakes are consequential and make them impossible." (~03:23, Steinberger)

Not preventing all errors — classifying them.

"My northstar is to deprecate prompting as much as possible. … Prompting is the biggest foot gun you can expose to [non-technical users]." (~03:23, Catasta)

The consumer-product view, which inverts the engineer's view.

"As helpful as these assistants are, they do not necessarily conform to social norms." (~03:26, Lopopolo)

Why security programs built on human process controls fail against agents.

"If you want to figure out what the new model is good at, just look at what there's 20 variants of in the last three months." (~03:28, Graveley)

A free indicator of the capability frontier.

"You have to start from grand ambition at every new snapshot." (~03:28, Lopopolo)

How not to get trapped by your own stale priors.

"I ship it in production. You just YOLO." (~03:29, Catasta)

The last line of the session, delivered into laughter.