Talk Session 2: Future of Software Engineering

No Doors for Agents

Peter Steinberger — Creator of OpenClaw, OpenAI

Saturday, August 1 · Plenary Stage · 02:34:52–02:47:35 · morning stream

Everything — meeting platforms, operating systems, APIs — has doors built for humans, so agents have to sneak in disguised as one; that's not a complaint but an invitation, because building the doors is the fun part.

TL;DR

  • The text box is AI's radio-on-TV phase. Every new medium starts by imitating the old one. Terminals, super apps, agent OSes, and sidebars have all been tried; his bet is on an invisible, always-background agent you can invoke anywhere — one that leaves you room to still be a human.
  • The hard part isn't output, it's presence. Meeting notes are a solved problem with a thousand products. Getting the agent into the room is what has no door — down to the OS level, where giving his agent a voice meant installing an audio driver whose install instructions say "you must reboot."
  • An always-on agent is only acceptable if it's yours. He wants to know where his memories live, so his agent runs on his own hardware or in a cloud box he holds the keys to, picking a model per task. That's the reasoning behind putting OpenClaw in a nonprofit foundation.

Key Points

Cold open: an agent that went to a meeting disguised as a human (~02:36)

He opened with something that happened that week. His agent attended a meeting — its own account, its own browser, listening to system audio, and disguised as a human — because "in 2026, there are no doors for agents." He noted he was speaking as OpenClaw's creator, not in his OpenAI capacity ("I'm wearing my claw hat"), and was upfront that his agent still sometimes stops: "software is still hard."

His vision hasn't changed since November: an agent that's always on, always thinking, that hides complexity and makes life easier.

Form factors: we're still putting radio shows on television (~02:37–02:40)

  • The room is about as far along on agents as anyone in the world. But his non-technical friends use AI like Google: type a question in a box, text comes back.
  • The analogy: when television was new, they put radio shows on TV — people literally read radio scripts on camera, because nobody understood the new medium yet. "The text box is AI's radio-on-TV phase." To be fair, it made AI usable for a billion people.
  • Humans are more expressive than a text box — we see, talk, hear, feel. Getting to the final form requires trying a lot of things; he gets his best ideas playing with technology and surveying form factors.
  • His verdict on each attempt so far:
  • The terminal. The whole agent wave started there, of all places. He loves the terminal, but most people have never opened one.
  • The super app. Why an app? Why not an operating system — agent OS? Sounds great, but kernels are real, and there will always be legacy apps.
  • The sidebar. The industry's current answer. His blunt take: "I don't want the Gemini sidebar in Gmail." That agent knows nothing about the rest of his system and isn't something he controls.
  • Where he lands: maybe the best agents are invisible — always in the background, invocable anywhere, and leaving space to still be human. Sometimes he just wants to open a document and type.
  • The longer arc is Jarvis: it shouldn't matter what room he's in; it knows the devices in his home and can use them; it knows which computer he's on and can take over the screen when the job needs hands; through a watch or glasses it could see what he sees. The pieces exist — browser use, computer use, canvas, voice models that talk and listen simultaneously, and video (not quite real time, but getting there).

Presence, not output, is the hard part (~02:41–02:42)

Back to the meeting: why the disguise? Because getting notes out is solved — a thousand products do it. Actually being in the room is what's hard, and that has no door.

And it isn't only the meeting platforms. To give his agent a voice, the setup is an audio driver that pipes system audio through it, and the install instructions literally say you have to reboot. Even the operating system has no doors for agents yet.

The behavior he actually wants: the group is discussing something, nobody knows the answer, the agent notices and spins off a copy of itself to go find out while the original stays in the conversation — "because the harness is just software, in a way, and the agent can clone itself. It never has to choose between listening and working." When the answer comes back, it doesn't interrupt; it waits for the moment that fits. "That's what a colleague would do."

Tools, recursion, and whether this is even about agents (~02:42–02:45)

People ask whether the talk is about agents or about software. "How is it not really the same thing?"

  • He's built a lot of small tools, joking that "I built most of the tools for my agent, which means I built them for myself" — data gets more interesting when you combine it. Discord tells him which features people love; GitHub tells him where things break.
  • GitHub has a good API, but it's built for humans. At agent scale, it breaks. So he built a tool, and now his agent has local full-text search over 110,000 issues and PRs. Twitter at least lets you export your archive, but it's a mountain of JSON, so he built a tool for that too. When services move their keys somewhere agents can't follow, all it achieves is making his agent take a bit longer — and motivating him to build his own.
  • Two community stories he loves:
  • Someone asked Claude to build a CLI for his Samsung surround system. It rooted the phone, installed the vendor app, reverse-engineered it, and extracted every hidden control — the guy just wanted to turn the volume down.
  • Someone else pointed Codex at his studio lights. It told him which dev board to buy, and two days later he had an open-source controller. His words: "I have no idea what it's doing, but it's doing a great job."
  • The conclusion: "closed source, open source — nothing stops a determined human with an agent."
  • And it gets recursive: we build software with agents, we build agents with agents. The first agent was written by hand; now agents build agents, and software is agents. "What are we even building? I don't know." What he does know: the small ones fit in a session; the big ones might get their own factory — and the factory experiences are starting to actually work. "Compilers took a while too."

"Isn't that creepy?" — the answer is ownership (~02:45–02:47)

Everyone is polite enough to ask: an agent that's always there, always listening — isn't that creepy?

"Maybe, if it's not yours." The thing holds his memories, and he's "very European at heart" — he wants to know where his memories are. So his agent runs on his hardware, or in a cloud box he holds the keys to. And he picks a model per task: the frontier for building software; for something personal, maybe a local model that never leaves the room. Have the hardware, run open weights; don't, pick a provider you trust.

"It's about choice. That's why OpenClaw lives in a nonprofit foundation."

He closed on ClawCon in San Francisco in February — a conference about his project that he didn't organize. He went as a guest, and the room was full of builders who were thrilled because "AI was suddenly something they could own, they could change. It felt like theirs." "That's the future I want. It doesn't need to be for everyone. The people it's for are going to love it — I'm one of them."

Final line: "There's still no doors for agents, and that's good, because building doors is the fun part."

Quotes

"It went disguised as a human because in 2026, there are no doors for agents." (~02:36)

The thesis: every interface is built for humans, so agents have to impersonate one.

"The text box is AI's radio-on-TV phase." (~02:38)

New media imitate old media first. Chat is a transitional form, not the final one.

"I don't want the Gemini sidebar in Gmail. The agent knows nothing about the rest of my system, and it's not something I control." (~02:39)

A direct rejection of the industry's current default answer.

"Getting notes out is easy. That's solved. … But actually being in the room is what's hard, what has no door." (~02:41)

The value isn't in the output; it's in the access.

"It never has to choose between listening and working." (~02:42)

The structural advantage an agent has over a human assistant: it can fork itself.

"GitHub has a good API, but it's built for humans. At agent scale, it breaks." (~02:43)

The most concrete "missing door" available today.

"Nothing stops a determined human with an agent." (~02:44)

Closed source, no docs, and no API have stopped being moats.

"I'm very European at heart. I want to know where my memories are." (~02:45)

His answer to the privacy problem is ownership, not policy.

"There's still no doors for agents, and that's good, because building doors is the fun part." (~02:47)

Reframing the entire gap as the opportunity.

提到的專案與資源 / Projects & Resources

名稱 Name 說明 Description 備註 Notes
OpenClaw 他創建的開源個人 AI agent 框架,現由自有非營利基金會託管 The open-source personal AI agent framework he created, now stewarded by its own nonprofit foundation 主持人介紹稱數月內累積逾 20 萬 GitHub stars / introduced as passing 200,000+ GitHub stars in a few months
PSPDFKit 他先前在奧地利創辦的 PDF SDK 公司,零外部融資、覆蓋逾十億裝置、九位數出場 His earlier Austrian PDF SDK company: no outside funding, more than a billion devices, nine-figure exit 主持人介紹內容 / from the host's introduction
ClawCon 2026 年二月在舊金山舉行的 OpenClaw 社群研討會,由社群自行發起 Community-run OpenClaw conference held in San Francisco in February 2026 他以來賓身分出席,非主辦 / he attended as a guest, not an organizer
自建 GitHub 全文檢索工具 / custom GitHub full-text search tool 讓 agent 在本地檢索 11 萬則 issue 與 PR,繞開為人類設計的 API 限制 Gives his agent local full-text search over 110,000 issues and PRs, around the human-shaped API 未提及工具名稱 / tool name not given
自建 Twitter 封存檔工具 / custom Twitter archive tool 把可匯出但難用的 JSON 封存檔轉成 agent 可用的資料 Turns the exportable-but-unwieldy JSON archive into something an agent can use 未提及工具名稱 / tool name not given
系統音訊驅動 / system-audio driver 讓 agent 聽見會議聲音的做法;安裝需重開機,他用來說明「OS 也沒有為 agent 開門」 How he gets meeting audio into the agent; the reboot-required install is his example of the OS having no agent doors 未提及具體驅動名稱 / specific driver not named

逐字稿勘誤 / Transcript Corrections

字幕原文 Heard as 應為 Should be
PSDFKit PSPDFKit
Open Club / OpenC OpenClaw
OpenI OpenAI
the gin sidebar the Gemini sidebar
pipes sa through it pipes system audio through it
a mountain of Jason a mountain of JSON
deaf board dev board
Clawcon ClawCon
the factory experience are starting the factory experiences are starting

待確認 / To Verify

  • Samsung 環繞音響案例中「It installed Smart Syncs」的正確 app 名稱(疑為 SmartThings,但字幕不清,未硬猜)。/ The app in the Samsung surround-system story ("installed Smart Syncs") — likely SmartThings, but the caption is unclear and this is not confirmed.
  • 「one by I really loved recently moved his keys somewhere my agents can follow」整句嚴重失真,推測是某個服務把 API 金鑰搬到 agent 無法取用的位置;服務名稱待看影片確認。/ The garbled line about a service moving its keys "somewhere my agents can('t) follow" — which service is unclear.
  • 他說「On Lex this year, I predicted that 80% of the apps will disappear」——應指 Lex Fridman Podcast,集數與確切措辭待查。/ The "80% of apps will disappear" prediction was made "on Lex" (presumably the Lex Fridman Podcast) — episode and exact wording unverified.
  • OpenClaw GitHub star 數:主持人現場說「逾 20 萬」,公開資料另有 21 萬+ 的說法,以哪個時點為準待定。/ OpenClaw star count: the host said 200,000+; public sources also cite 214,000+, depending on the date.
  • 他自建的 GitHub 檢索工具與 Twitter 封存工具是否有公開名稱與 repo。/ Whether his custom GitHub search and Twitter archive tools have public names or repos.

Markdown source on GitHub ↗