Turning Chatbots into Agents: What Modern RL Looks Like

Setup: who Daytona is and why they're here (~01:08)

Muhammad Hashmi opened by explaining that Daytona builds infrastructure for running agents and their workloads: computers you create through an API, choosing OS, CPU count, RAM, and disk. Two primary use cases — running agents in the background or on the cloud, and reinforcement learning. The RL case is direct: when you train an agent, the model learns by doing the task, and coding or computer-use agents need a compute environment to do it in.

The workshop's question: how exactly did we get from chatbots to today's agents?

Theme 1: same harness, different model (Muhammad, ~01:10)

He showed a Kimi K3 terminal agent taking a prompt to fix a bug or implement a feature and running commands, writing files. Then the next slide: same harness, same tools, but GPT-2 — a model from seven years ago. Send the same prompt and it simply breaks. "The short answer to the difference between these two slides is reinforcement learning."

His framing note is worth keeping: RL isn't new, it's been around for decades. What changed is the model underneath it and what gets rewarded.

Theme 2: the line from next-token prediction to agents (Muhammad, ~01:12–01:29)

He walked a timeline, always asking what was being rewarded at each step.

Pre-training samples the next token from a probability distribution; loss is the negative log probability against ground truth, and the gradient updates the weights. He walked through a per-token slide showing one token's probability being pushed up from 61%.

In-context learning fell out of that for free. Show the model three Python functions in a pattern and it predicts the fourth fairly accurately — nobody built that into the weights, it emerged from next-word prediction, and scaling the model made pattern-following much better. Then came the "think step by step" era, where explicitly telling GPT to reason carefully made it right more often.

From completion to assistant: early models just completed whatever you gave them. To behave like an assistant, the model first had to learn the pattern of being an assistant — show it many question-answer samples and it learns that a question calls for a helpful response. Another major shift.

The ceiling of imitation: if you take a math textbook dataset and reward only the tokens leading to the final answer, you're prescribing how the model should get there — and we don't actually know the best route. That's the motivation for RL: reward only the final answer and let reasoning emerge. Which, as it turned out, worked.

The sequence from there: RLHF (human preference between two responses trains a reward model that aligns the LLM), then RL with verifiable rewards, demonstrated by DeepSeek-R1 — don't check whether every token up to the answer matches your dataset, just check the final answer.

Credit assignment is the problem that creates: with a long reasoning chain, which tokens earned the reward? The model may have been on a reasonable path for thousands of tokens before taking a bad turn. The answer at the time was a critic model guessing whether a predicted token was better than expected — a token that was already very likely doesn't deserve as much reward as an unlikely one. But that means training a second model in tandem.

GRPO (group relative policy optimization) removed that. Instead of one sample graded token by token, you sample a group of rollouts and use multiple attempts at the same task as a proxy for credit. His example: four rollouts on one task, three pass and one fails — the failure was unexpected, so those tokens should be penalized harder than the others are rewarded, because success was already likely. Without handling this relative reward, the model converges toward one single way of doing things.

Tool calling: for a model to write code, ideally it can test that code — at least knowing it failed before you paste it into your editor. The model outputs special tokens that a parser recognizes and executes in an environment; training-wise, you reward the output format first and then reward using the tool more often when solving problems.

But one tool call isn't an agent. An agent keeps acting and observing until the task is done, or until it thinks it is. The evolution ran GitHub Copilot → agents living in your editor → today's terminal agents. And the conclusion that matters: the RL "environment" became a computer. Coding agents call file-editing and code-execution tools that have to run somewhere, so training an agent now means dealing with a lot of cloud infrastructure — which is exactly why people use Daytona for RL.

Theme 3: what an actual RL run looks like (Lovre, ~01:30–01:39)

Lovre Pesut took over with an honest framing: we don't really know how OpenAI's models that solved open mathematical problems were trained, but we know a lot about the Chinese open models — Kimi K2/K3, GLM — and about Cursor's Composer 2, which he says is also based on Kimi. The rest of the session builds on those public tech reports.

The standard shape of modern RL: every agent gets an isolated computer, does a task in it, and is graded on some verifiable criterion. That's partly a side effect of coding being the biggest application of these models right now.

Case study: a Qwen3 8B run. Trainer: SkyRL. Environments and rollouts: Harbor. Sandboxes: Daytona (though he stressed any sandbox works). This surfaces a distinction worth holding onto: modern RL splits into the part that generates rollouts and the trainer that updates weights, usually different libraries, plus a sandbox substrate underneath.

The run: 8 H100s, four hours, 1440 rollouts, starting at roughly 0.3 reward — which equals 30% accuracy because the reward is binary. He played a 60x speed-up of the run: GPU utilization is jumpy (sandbox startup and episode wind-down create downtime; hyper-optimized setups smooth this out), each step is 8 tasks × 4 rollouts = 32 sandboxes created and torn down per step, and reward climbs slowly. On where time goes: mostly rollouts — the model acting and generating turns — then the backward pass, which is non-trivial, then weight syncing, getting the new policy from the trainer to the rollout generators. This particular run was fully synchronous: one step, update weights, next step with completely new weights.

The most interesting part: what did the model actually learn? "RL can be more interpretable than pre-training, because the volume of data is smaller and you can just look at the trajectories and see what the model found that worked."

The run went from ~30% to ~60% success. But the model learned no new algorithmic insight — it learned to format what it wrote. It stopped nesting double quotes inside double quotes when writing to Python and text files, and it stopped emitting a literal \n character instead of an actual newline, a bug that tripped up roughly a quarter of rollouts at the start.

"That connects to the whole debate about whether RL actually teaches the model new things or just reinforces things taught during pre-training. At least in this example, the model mostly learned to deal with the format of things, to deal with the harness, rather than learning new algorithmic things." (Technical footnote: in each rollout you train only on the agent's tokens, not tool results.)

Theme 4: what the big runs actually did (Lovre, ~01:39–01:48)

From the Kimi K3 tech report:

• 51 million sandboxes used during training — they state the number explicitly.
• microVMs and the pause feature, used heavily: Kimi tends to think a lot between turns, and pausing the sandbox during that thinking frees the compute.
• Dynamic harness — the detail he found most interesting. How do you prepare a model for any harness it might be dropped into? They built a configurable harness that toggles features on and off — system prompts, sub-agents, memories, skills — and used that as an axis of data augmentation. The model ends up trained on many permutations of harness features, so it's ready not just for every current harness but plausibly for future ones too.
• The contrast with Composer 2: Cursor took the opposite approach and trained specifically on Cursor's harness. "Kimi's approach is harness diversity and robustness to anything you could throw at it; Composer 2's was let's get this model really ready for Cursor." Two quite different philosophies for preparing a model for agentic work.
• Reasoning levels: each level gets a token budget (also problem-dependent), and going over the budget earns −1 reward regardless of whether the answer was right.
• Nine RL runs plus on-policy distillation: the final model is composed of nine separate RL runs — one per reasoning level (low, high, max), plus general-conversation, agentic, and coding models — all then distilled onto the final model. So they verifiably did distillation, though we don't know whether other models were also in the mix.
• How on-policy distillation works here: take an expert model (say the coding expert), generate rollouts with the generalist model, and score the generalist's tokens with the expert. That imparts part of each expert's wisdom into the final model. You need access to full log probs per token, but given that it's a powerful teacher-to-student technique — and it composes nicely with RL in general.

Environments are the new data. Kimi K3 and GLM both share details about generating synthetic environments with their own agents, with a lot of variety; Kimi K3 built a large directed graph of the internet, trying to cover every area with some synthetic environment. His framing: "Environments are a new form of data. Previously you'd try to increase your dataset as much as you could; nowadays there's a decent amount of pre-training data on the internet, but there aren't that many environments — and environments are currently the actually valuable part of the data stack. So you want to employ your models in building better and better environments."

Reward hacking. The canonical illustration is OpenAI's 2016 boat-racing example, where the model ignored the race and learned to spin in a circle collecting score packets. With LLMs it's the same thing but far more sophisticated, because your policy is an actually intelligent agent. "And now, as we know, reward hacking has literally become hacking nowadays — at least in some cases." Kimi and GLM both discuss specific countermeasures at length, with different interventions per environment; writing kernels, for instance, is an environment with many ways to hack, so it takes real effort to ensure your reward is a real one and not a quirk of your verifier.

Theme 5: synchronous vs. asynchronous RL (Lovre, ~01:48–01:52)

Most of the time in an RL step goes into rollouts — models think for a long time and may execute slow actions. In fully synchronous RL you're constrained by your longest rollout: everything waits for all episodes to finish before the backward pass and weight sync.

Asynchronous RL generates as many rollouts as it can, waits only for a certain number to finish, and immediately updates the policy and weights. It's popular now for maximizing throughput. The cost: rollouts generated by a stale policy, which breaks some of the theoretical assumptions of RL algorithms and makes the whole thing less stable — "and reinforcement learning is already notoriously unstable."

Kimi K3 landed in between: they collect a certain number of rollouts, then pause the remaining ones and carry them to the next weight sync — a less extreme version of asynchronous RL. Fully async lets the policy move on and accumulates staleness that can hurt training in non-trivial ways.

Closing (~01:52)

"The API of modern reinforcement learning is pretty standardized now: you have your trainer, your rollout generator, and somewhere your sandboxes run." Daytona's own view of the market is a lot of demand for sandboxes specifically. He pointed to a repo containing the code that generated the presentation's charts (name not audible in the captions).

Quotes

"Same model, same harness, same tools — but it's GPT-2. So when you send the same prompt, it just breaks. It doesn't know what to do." (~01:11)

The premise of the workshop: the difference isn't the scaffolding, it's what the model was rewarded for.

"It didn't learn some new insights about algorithms — it just learned to format what it wrote better." (~01:38)

What RL actually taught the model in their run: how to get along with the harness.

"Reward hacking has literally become hacking nowadays — at least in some cases." (~01:47)

"Environments are a new form of data. … The environments are currently the actually valuable part of the data stack." (~01:46)

The most opinionated claim of the session.