Infrastructure for Long-Running Agents

TL;DR

• Three agents already delivering production value: a model generation agent migrating 400+ production TensorFlow models to PyTorch; a performance optimization agent that hunts low-utilization jobs in the cluster and optimizes them; and an autonomous research agent where an engineer defines an objective and the agent runs architecture sweeps and hyperparameter tuning on its own.
• Agents forced three infrastructure shifts: UX (from rich authoring abstractions toward designing agentic workflows and guardrails, since the code itself is now LLM-written), compute (from a handful of long-running jobs to hundreds launched within minutes, where scheduling latency no longer amortizes), and trust (human code is trusted by default; agent code is not).
• Harness-as-code is their answer to long-horizon drift: every stage is defined in code with its own responsibilities, and crucially the scorers and evaluators are code too, so agents can't reward-hack or wander off — backed by checkpoint/restore so agents recover from failures without a human.

Key Points

Three production agents and their shared lifecycle (~01:09–01:10)

LinkedIn's AI Platforms team runs three classes of agent that have already delivered significant business value:

1. Model generation. LinkedIn runs 400+ production models written in TensorFlow and wants them on PyTorch. The hard part isn't translation — it's guaranteeing the generated models perform on par with or better than what's deployed, given those deployed models were hand-tuned by engineers over months or years.
2. Performance optimization. An agent that scans the cluster for low-utilization jobs, profiles and benchmarks them, optimizes them against a knowledge base the team accumulated over years, and files review requests to the owning teams.
3. Autonomous research. Built on top of the other two: an engineer defines an objective, and the agent autonomously runs architecture sweeps or hyperparameter tuning against it.

All three share a lifecycle: engineer states an objective → agent forms a hypothesis → generates code → runs it on GPUs → evaluates the outcome → iterates. This loop runs continuously and autonomously for hours or even days with no human intervention.

Three shifts: UX, compute, trust (~01:10–01:12)

• UX. The platform historically invested in rich abstractions so engineers could write ML pipelines, training loops, and data access easily. Those still matter, but that code is now written by an LLM; users' attention has shifted to composing agentic workflows and setting guardrails so agents don't deviate from the original task.
• Compute. Engineers used to launch a limited number of long-running jobs, where a couple of minutes of scheduling delay amortized over the training run. Agents launch hundreds of jobs in a very short window, ranging from minutes to days in length.
• Trust. Code written by human engineers is inherently trusted; agent-generated code needs a very constrained environment.

The platform: an agent is just a job — an untrusted one (~01:12–01:13)

In their infrastructure, an agent is just like any other task — users write agents the same way they'd write a PyTorch or TensorFlow job. The one difference is trust: agent code runs in a highly restricted environment with no egress and no access to credentials, with all communication routed through proxies under the agent's own identity.

Against long-horizon drift they use a harness-as-code approach: natural-language prompts don't hold up when agents run for long periods, so each stage is defined in code with its own responsibilities — and more importantly, the scorers and evaluators are defined in code as well, so agents can't reward-hack or diverge from the intended path. Around that sits an ecosystem for task and memory management (how agents share learnings with other agents of the same or different type) plus notifications for human-in-the-loop interactions. Because these agents run for a long time, they also built robust checkpointing and restore so agents autonomously recover from failures.

The volume of jobs also became a bottleneck in the control plane, which had been designed with a lot of abstraction — and abstraction costs add up once agents start launching many jobs. So they built a custom control plane maintaining a warm pool of GPU pods for very low-latency job launches. That pool is equally locked down: no egress, read-only data access, running fully unprivileged.

Impact and what's next (~01:13–01:14)

• The number of experiments in the cluster grew 100% over the last six months.
• Continued investment in evaluation, so that experiments translate into real production value.
• Ongoing work on SLMs for specialized tasks and GPU sharing techniques, since GPUs keep getting bigger while not every task is big enough to saturate one.

His closing frame: agents let you "take more shots at the goal" — more shots means more model improvements, which means better member experience and business value.

Quotes

"The agent code is untrusted. So it runs in a highly restricted environment like no egress, no access to credentials, and all the communication goes via our proxies using the agent's own identity." (~01:12)

The one place where "an agent is just another job" stops being true: the trust boundary.

"The goal with agents for us is to take more shots at the goal." (~01:14)

Not replacing engineers — raising experiment throughput by an order of magnitude.