Omniscient Agents

TL;DR

• Today's agents are a pile of primitives — execution layer, asset creation, orchestration (sub-agents, skills, memories, scheduling, loops), and live data. Build one from scratch and you implement every single one yourself.
• The real bottleneck is your attention. Agents don't quite know what to do and their suggestions for what's next are often bad, so a human stays in the loop directing them. The trust ladder has been climbing for a while: tools → commits → a single PR → multiple PRs (loops) → features → projects → whole products.
• Two axes drive scope: insight (what data the agent can access, and in what form) and control (what it can do to that data and to live systems). The direction of travel is from seeing the code to seeing the entire business.

Key Points

The framing: agents are limited by what they can see (~03:07)

He introduced himself as lead on GitHub Copilot and Perplexity Computer, having just started FlyingObject to work on exactly this topic.

"Traditionally agents have been sort of limited by what they can see and access. And we think that this necessitates having a human in the loop." The goal is to get the human less into the loop, or at least able to farm off different pieces of their loop to agents.

Today's agents are a pile of primitives (~03:07–03:09)

Build an agent from scratch and you have to implement all of these yourself:

• Execution layer — running locally or in a sandbox, with commands for manipulating files, running commands, and running CLIs.
• Asset creation — documents, websites, and PRs as another form of asset that everyone is working to make agents good at.
• Orchestration primitives (he noted the details matter less than what each one does):
• Sub-agents with different context from the parent.
• Skills to fill gaps the weights don't express the way you want.
• Memories so the agent can learn from the past.
• Scheduled tasks for things that recur or run on a schedule.
• Loops — goal-directed running, where individual tasks compress into a loop that doesn't end until the goal is achieved.
• Live data — web search, browser control, computer control, MCPs, APIs, all of it.

"This is where agents are today."

The bottleneck is your attention (~03:09–03:10)

What's common across all of it is that you're still controlling these agents. The agent doesn't quite know what to do, its suggestions for what to do next are often bad, so a human directs.

The result: the bottleneck becomes your attention as an agent-using engineer. You're managing loops, running things in parallel, tracking what's going wrong and what's going well. You have a sense of what's happening; you can stop things, fork things, restart things.

The trust ladder we keep climbing (~03:10–03:11)

He noted the way we do this today is quite different from a year ago — the way agents self-direct has changed. We've been walking up a complexity hierarchy:

1. Just tools — we trusted the agent to call the right tool.
2. Commits — we started trusting the agent to make commits.
3. Entire PRs.
4. Multiple PRs — where we are now, trusting the agent to produce several PRs to accomplish looped goals.
5. Features — his next rung. A feature might involve running an experiment, looking at live data, checking for exceptions, checking user sentiment, and segmenting who's exposed to the feature to see whether it moves your primary objectives, like retention or revenue.
6. Projects — collections of features serving a need in your product.
7. Whole products — at the top, you describe at a very high level the product you want to a self-directed agent, and it goes and makes it, deploys it, iterates on it, figures out which features it needs (maybe ones that don't exist anywhere else), tries a few different things, and self-directs that way.

Two axes: insight and control (~03:12–03:13)

Each level up required either new model iterations or much more agent harness complexity. So how do you enable increasing scope? He argues it comes down to two axes:

• Insight — what data the agent has access to and in what form, so it can derive insights.
• Control — its ability to operate on that data and on live systems.

Which means moving from seeing the code to seeing the entire business. Only then can you start to work out what the actual business objectives are, and which products or features should exist to accomplish them. Right now, that's up to people to figure out and take guesses at — and what they want is for AI to be able to do it.

The other half of the product development cycle needs control just as much: running experiments, deploying changes, monitoring live systems, and scaling them as needed. Together, those are the two axes along which agent scope grows.

(He noted "sorry, I'm out of time" here and moved quickly through the last two sections.)

Open problems (~03:13–03:14)

• Processing lots of data — generally you want to ingest and index everything that happens inside a business.
• Compression — turn that knowledge into something usable by your agent.
• Triggers.
• Agents aware of each other — if you're running lots of experiments simultaneously, potentially with no human in the loop, agents need to know about each other and coordinate effectively.
• The hardest one: sometimes you don't even know the objective function you're optimizing.

Closing (~03:14–03:15)

"What we want to do is enclose the entire system, capture everything. Give the macro context to agents so that they can operate with full awareness, and give the right primitives to those agents so they can deploy changes, monitor those changes, and scale those changes to accomplish those business goals." He acknowledged it will be a process as we walk up this complexity stack, but that's the direction he thinks we're headed. Early versions are behind a FlyingObject waitlist; otherwise he's on Twitter and would love to talk.

Quotes

"Traditionally agents have been sort of limited by what they can see and access. And we think that this necessitates having a human in the loop." (~03:07)

The causal premise of the whole talk: humans in the loop are a symptom of limited visibility, not a cause.

"Today's agents are a pile of primitives." (~03:07)

The state of building an agent from scratch: every piece is yours to supply.

"The bottleneck becomes your attention as an agent-using engineer." (~03:09)

Directly echoes Ryan Lopopolo's three scarce resources from earlier in the same session.

"We started with just tools. … Where we are now is trusting the agent to make multiple PRs." (~03:10)

Autonomy measured in the size of artifact you're willing to trust.

"The axes that drive scope are insight and control." (~03:12)

His core answer for how to push agent scope upward.

"We're moving from seeing the code to seeing the entire business." (~03:12)

What "omniscient" actually means here.

"Sometimes you don't even know the objective function you're optimizing." (~03:14)

The hardest open problem for self-directed agents.