Build AI Agents That Survive Failure

Opening: get the whole room on its feet (~01:04:39–01:06)

After a long block of AI talks, Advolodkin opened by asking everyone to stand, high-five two people, look them in the eye, and tell them they're amazing — then filmed a selfie video of the room cheering (he made them redo it "5x louder"). Introductions: developer advocate at Temporal, dog dad, roller skater, dog named Mia.

His framing: you retain far more by being hands-on than by listening passively. The plan was four topics — foundations of agentic AI systems (and building one), building with an agentic framework (OpenAI Agents SDK), human in the loop, and orchestrating micro-agents — and he said upfront they probably wouldn't get through all of it, but the GitHub repo would follow.

Topic 1: what an agentic loop is (~01:07)

An LLM can reason about the context you give it. What it lacks is the ability to take action when it doesn't have enough information. The agent loop supplies exactly that: the capability to act outside itself.

Start with instructions and goals — his example: "you're a travel assistant, find a flight to Miami, ask for confirmation before booking payments." Pass that context to the decision maker, the LLM. The LLM decides whether it has enough information: if yes, final answer; if no, call a tool. Tools are defined by the harness or by you; you tell the LLM they exist and the LLM decides when to use them. It evaluates the output, and keeps looping to gather information until it can comfortably answer.

A live poll on Temporal familiarity showed most of the room had never heard of it, or heard of it but never used it — so he flagged the session as advanced for many attendees and promised to walk them through.

Topic 2: the Temporal mental model (~01:10:47–01:13)

In one line: Temporal lets you write code as though failures don't exist.

• You focus on the business logic of your agents and distributed systems — and he was explicit that agents are distributed systems. That goes into a workflow: the sequence of steps you want to execute.
• You pull out the non-deterministic steps — API calls, LLM calls, database queries — into activities.
• Wrapping those activities in the SDKs (Python, TypeScript, Java, Go, Ruby, and more) creates a history of everything performed up to a given point.
• When any failure occurs, Temporal pauses at that point in the history and waits for the failure to be resolved — seconds, minutes, or years. When it is, execution proceeds as though the failure never happened.
• Waiting costs no resources, because it's all recorded in the history. He cited six nines of reliability.

What can fail? Nearly everything: LLM calls due to network reliability (he got a 429 rate limit from OpenAI that very morning), API calls, rate limits, services going down, AWS going down. Wrapped in an activity, Temporal picks up once they're resolved.

Topic 3: hands-on — the first workflow (~01:13–01:25)

The environment was a browser-based virtual lab: no setup, a Docker container with a ~90-second boot. Tabs for a worker terminal, a starter terminal, the Temporal UI, a network control panel for simulating network failures, and a code editor. Attendees worked out of exercise/, with a mirrored solution/ folder. Three TAs up front, plus Melanie at the back.

The exercise: uncomment the agent construction in exercise/tools/workflow.py. It uses the OpenAI Agents SDK integration — give it a name, a system prompt, a model (GPT-4o "is good enough for this one"), and tools. The tools are just web requests (get coordinates, get IP location info, get weather), but they're passed in as activity tools, so they inherit Temporal's durability.

Running it: start the worker — the process that polls the task queue to decide what work to perform — then run the starter, which asks "What is the weather in Tokyo?"

The payoff: refresh the Temporal UI and the workflow has already completed. This is what he says he loves most about Temporal — visibility. The whole timeline is laid out (invoke model activity → get coordinates → invoke model activity → get weather → invoke model activity), and every step can be opened to inspect headers, inputs, and outputs. The answer: 79.6°F in Tokyo with light wind.

Topic 4: break it — durability demonstrated live (~01:25–01:28)

The centerpiece demo. In the network control panel, turn off the weather API — imagine your network blocks it, your enterprise won't allow it, the API is down, or AWS is down so the API is down. Then rerun with "What is the weather in London?"

In the Temporal UI:

• The workflow goes to Running and stays Running until the problem is resolved.
• The activity keeps retrying. Since no retry policy was specified explicitly, the default applies, which retries forever, with automatic backoff and backoff coefficients built in. He explained why you want that: you don't want an LLM hammering the OpenAI API, because it costs you resources — 10 calls, wait 10 seconds, 10 more failures, wait 30 seconds. That's exponential backoff, and without Temporal you'd be writing that logic yourself, and it gets hairy.
• The last failure reads 503 service unavailable.

Then turn the weather API back on — "let's imagine AWS had an outage, it's back up." Within moments the workflow recovers as though no failure ever occurred: no code touched, resumption from exactly where the failure happened, no extra resources consumed, and the terminal gets London's weather.

Topic 5: rolling your own loop versus the SDK (~01:28–01:30)

He demystified agentic frameworks along the way: most hide the agentic loop, but it's ultimately a while True loop — wait for user input, decide whether you have enough information to make a decision, call tools if you don't.

Writing it yourself is roughly 50 lines; with the OpenAI Agents SDK it's one line of runner.run, with the SDK handling the loop. Wrapping tools as activities is what supplies durability to everything the agent calls.

Topic 6: why split into multiple agents (~01:36–01:38)

He put the next section to a vote — human in the loop or multi-agent orchestration — and the room picked multi-agent, so human in the loop got skipped. (His one-line version: human-in-the-loop is a tool that asks and waits for human feedback, whether an approval or information — "ChatGPT is a human-in-the-loop type of thing," waiting for you to respond before proceeding.)

The reason to split: context poisoning, distraction, context clash, and confusion, all caused by too much context. It's fine for a small workflow, but the larger the workflow and the more data you provide, the harder it is for LLMs to reason.

Topic 7: three ways to call a sub-agent, and Nexus (~01:38–01:43)

The architecture: a personal assistant agent as the entry point to the whole process — your ChatGPT — which decides whether to call the weather agent, the F1 agent, both, or more.

Three ways to invoke sub-agents:

1. As an activity.
2. As a child workflow — the main workflow starts a child workflow and the two are tied together; good for putting decision-making logic into its own process.
3. Via Nexus — Temporal's other technology, which segregates by team domain, region, security concerns, or code requirements — for instance when you want one agent deployed at a different cadence than another.

His framing of Nexus: think of it as an API. You expose only the endpoints you choose to another team or organization, and they can call only those. The caller supplies the endpoint and service name; whatever happens behind the scenes can change — move to v2, whatever — while the contract stays the same and the caller never changes their code.

Topic 8: the multi-agent build, and breaking it again (~01:44–01:52)

Demo 5's architecture: the personal assistant workflow takes "when is the next F1 race and what's the weather there right now?" → calls the F1 agent through a Nexus operation (a separate file on a separate task queue, which calls an F1 MCP server they host, which calls the F1 APIs) → the result comes back → the assistant decides it still needs weather → calls the weather agent as a child workflow (which calls get-coordinates and get-weather activities, both plain public HTTP requests) → assembles the final answer.

The live result: the Dutch Grand Prix on August 23, with 64.7°F on site. The Temporal UI showed two workflows executing, and the Nexus link is clickable — following it lands you inside the F1 expert's own workflow with its complete execution record.

Then he killed the weather API again. Neither workflow completes; the failure lands precisely on get_coordinates inside the weather agent workflow and keeps failing until resolved. Turn it back on and both complete — with the caveat that restart timing depends on the retry policy and exponential backoff, so after a long wait it may take extra seconds before the next retry.

Topic 9: quizzes and wrap-up (~01:31–01:36, 01:52–01:59)

He ran phone-based quizzes with a leaderboard, swag for the top three. The answers worth keeping:

Wrap-up (~01:58–01:59): out of time and being kicked off stage, he asked for two things — follow him on LinkedIn, where all the content will be posted, and leave feedback in the lab's feedback tab, feedback he can act on ("you can tell me I suck, no big deal, but please tell me why I suck"). Quiz winners: Anton, Henrik, and Ming, with swag at the back of the room. The Temporal Slack community is there for follow-up questions.