Accelerated Computing for Agentic AI

TL;DR

• An agent is an LLM plus computer science. LLMs are the only known method for solving whole classes of long-unsolved problems, but they are probabilistic and non-deterministic — precisely the opposite of software, which is deterministic, inspectable, and assertable. An agent marries the two.
• The platform underneath is heterogeneous at both hardware and software levels: some tools run on GPUs, some on CPUs; storage hierarchies are deep; agents spawn sub-agents with scoped data; communication patterns keep getting more complex; sandboxes and secure enclaves surround everything — plus a mix of large, small, fine-tuned, and externally-hosted proprietary models. NVIDIA's answer is the NVIDIA / NeMo Agent Toolkit.
• The harness matters as much as the model. His group's NOOA (NVIDIA Object-Oriented Agents) makes an agent a plain Python object: a method body written as ... tells the LLM to fill it in, so the agent writes its own methods, calls them, modifies them, and keeps what it needs as object state. Information is passed by reference rather than compacted into strings in a giant context. Results: the same score at half the tokens, and among the strongest CyberGym scores — a significant lift from the harness alone.

Key Points

Where agents came from, and what one is (~01:13–01:17)

• When ChatGPT kicked off the modern AI era, these systems were things a human talked to — an LLM, maybe with access to a database. Fundamentally, chat.
• Today the picture is a complete autonomous system: many models (open-weight and proprietary), tools, infrastructure, memory systems, context management, the ability to spawn sub-agents, and security infrastructure — kicked off by a request that may or may not come from a human.
• So what is an agent? Not just an LLM, but an LLM surrounded by what he calls infrastructure — the software that lets the LLM actually do things: marshalling data between APIs, type checking, rule-based enforcement of policies. "Another word for all of this stuff … is computer science."
• Why that's a good idea: LLMs are incredibly powerful and are the only method we know of for solving all sorts of previously unsolved problems. But they are probabilistic and non-deterministic, the opposite of software, which we can understand, inspect, and make assertions about. An agent takes the intelligence from the probabilistic model and surrounds it with determinism, data structures, algorithms, and decades of learning about how to make software reliable.
• (His deck also loaded an out-of-date version on stage, which got a laugh — the morning's running joke, started by Dawn Song.)

A heterogeneous platform and the NVIDIA Agent Toolkit (~01:17–01:20, 01:22)

A typical agent has a way of managing the prompts fed to the LLM and infrastructure that lets the LLM reach tools and check permissions. All of it now runs on an increasingly complicated hardware substrate: some software on accelerated computing infrastructure like GPUs, some on CPUs, with deep storage hierarchies, agents and sub-agents, scoped data passed between systems, increasingly complex communication patterns, sandboxes, and secure enclaves. The platform is heterogeneous at both hardware and software levels — not to mention the collection of models: large, small, fine-tuned and specialized, and general-purpose proprietary models hosted externally on an API.

NVIDIA's version of this platform is the NVIDIA Agent Toolkit:

• Deployment: Kubernetes through hosted infrastructure, plus computer-use agents.
• Accelerated tools (CUDA-X): computational fluid dynamics, differential equation solvers, secondary analysis of DNA sequences from sequencing instruments — computer-science tasks with accelerated solutions, now exposed agentically so an agent can call them.
• Open-weight models: Nemotron (the general-purpose family), domain models for robotics and physical AI, and BioNeMo (predictive models for biology).
• Runtime and governance: the ability to deploy and run all of these models, sandbox environments that surround the system to ensure the AI isn't doing something you didn't want, and running all of it efficiently.
• Knowledge capture: capture what flows in and out of the system and use it to post-train a model specialized for the task you care about. He counts this offline improvement loop — often RL post-training that makes a small model as good at a specific task as a much larger general model — as part of the infrastructure itself.
• Other components: NIM and Dynamo for deployment, NeMo Relay for trace capture, Switchyard for routing, blueprints (open-source reference implementations showing how to pull the pieces together for a specific task, such as building an OpenClaw or AIQ, a research assistant agent), and OpenShell, essentially a firewall controlling access between an agent and the outside world. Much of this is deployed and adopted by NVIDIA partners.

The harness matters: NOOA (~01:20–01:22)

• The interface between the agentic system and the LLM — the agent harness — is where he wanted to spend time. The LLM encapsulates a lot of intelligence; the harness determines how much of it you get.
• His group's recent work is NOOA (NVIDIA Object-Oriented Agents), on GitHub (a QR code on the slide). The idea is simple: an agent is just a Python object.
• You write the agent in Python, and a method body written with the special ellipsis (...) syntax tells the LLM "fill this block in with code."
• The agent can therefore modify itself: call methods it has written, rewrite its own methods, decide that a piece of information is worth storing and hold it as state on the object, and encode a plan or a solved approach as a method to call later.
• Another important idea: pass by reference instead of by string. Because everything is a Python object, you don't compact information into strings stuffed into a very large context.
• Results (details in the tech report): significant lift over a bare LLM and over other agent harnesses — for example the same score using half the tokens, and among the strongest CyberGym scores, where the harness contributed a substantial share of the result.

Quotes

"Another word for all of this stuff that we surround our large language model with that makes it into an agent is computer science." (~01:15)

The talk's definitional line.

"They're the only method we know of to solve all sorts of problems that were previously unsolved … But at the same time they're probabilistic and they're non-deterministic. This is precisely the opposite of software." (~01:15–01:16)

Why an agent has to be a probabilistic/deterministic hybrid.

"Agentic AI workloads are significantly more complicated, heterogeneous, and expensive computationally than anything we've ever seen before." (from the panel, ~01:41)

His own summary of the hardware consequence, delivered shortly afterwards.