Scaling RL for Coding Agents - Lessons from Training SWE-1.7

TL;DR

• Each run was sized to the compute and team of the moment: from Kevin-32B (a few interns and 32 H200s) to SWE-1.7 spanning three continents — the skill is picking a problem that is exactly ambitious enough.
• Multi-turn agentic RL beat single-turn early on: Kevin-32B used GRPO with multi-turn training on KernelBench and clearly outperformed the single-turn variant.
• Training instability is observable before it happens: they tracked a "not-okay ratio" — how often the model's chain of thought stopped starting with "okay" — as an early divergence indicator, and the model's inner monologue visibly got more frantic as collapse approached.
• Parallel tool calling is the latency lever: after RL, SWE-grep issued roughly eight tool calls per turn, and reward and parallelism climbed together through training.
• Async RL is a geographic unlock, not just a throughput trick: separating trainer from rollouts let SWE-1.7 train on scattered compute across four countries and three continents, because only sparse weight updates and token batches cross the wire.
• Entropy collapse is the RL ceiling: tuning the recipe so entropy barely declines is what let them keep pushing performance — a direct rebuttal of the "RL has plateaued, it's all pre-training now" narrative.

Key Points

Kevin-32B: what can you do with 32 GPUs? (~00:14–00:17)

The team started as a few interns with 32 H200s, asking what could be genuinely interesting at that scale. The answer was to pick something narrow and cleanly verifiable: KernelBench had just come out — given a PyTorch function, write a CUDA kernel that speeds it up.

They implemented GRPO and got it working, but the real question was what multi-turn agentic training looks like, which was still novel a year earlier. It worked: the multi-turn curve sat clearly above the single-turn one, and a 32B-class open base model beat o3 on that benchmark.

The "not-okay ratio" anecdote: this base model had a quirk of always opening its chain of thought with "okay." As training progressed the monologue got increasingly unhinged — from "okay amigos, I need to optimize this 3D tensor matrix multiplication" to "okay holy crap I need to get this code optimized." Measuring how often the thought didn't start with "okay" turned out to be an early warning of an eventual diverging run.

SWE-grep: turning context retrieval into a verifiable RL problem (~00:17–00:19)

Next they wanted an agent good enough to ship. A core sub-problem in coding agents is finding the right files: given a question about a codebase (e.g., how does VS Code efficiently implement file watching?), return the list of relevant files.

The reward is clean and verifiable — F1 against the ground-truth file list. They trained an agent on it, built their own code search eval, hit state of the art on that eval in the Sonnet 4.5 era, and shipped it into Windsurf.

The scientifically interesting result was parallel tool calling. Most base models at the time called tools sequentially, one at a time; Anthropic's Sonnet had just started issuing one to three at once. Because the goal was end-to-end task latency, they optimized for doing as much as possible in parallel — SWE-grep would sometimes fire eight tool calls per turn. Over the run, both reward and parallel-tool-calls-per-turn rose together. Served on Cerebras hardware, the mini variant ran at roughly 3,000 tokens per second.

SWE-1.6: stability and asynchronous RL (~00:19–00:23)

Now they moved to real frontier coding models for end-to-end tasks, measured on realistic evals like SWE-Bench Pro, where SWE-1.6 matched Opus 4.5 at the time.

• The same fundamentals applied, just with more and harder data: an internal eval of verifiable software-engineering tasks went from 52% to 68% over a single RL run.
• The model learned to think longer on its own: average thinking tokens started near 4,000 and roughly doubled over the run.
• Stability was hard-won engineering: a typical run would collapse around step 200; they'd diagnose the root cause, fix it, and restart — the curve would retrace almost exactly the same trajectory but survive past the old collapse point. Each algorithmic improvement pushed the training horizon from 200 to 300 steps and further.
• Asynchronous RL: separate trainer from rollouts. Rollouts push batches of groups into a data buffer; the trainer drains the buffer as it fills, then sends a weight update back. The cost is staleness — how many steps the inference policy lags the training policy.

SWE-1.7: entropy collapse and training across three continents (~00:23–00:27)

Staleness and stability turn out to be the same problem: the more stable your algorithm, the more staleness you can tolerate; the more staleness you tolerate, the more fully you can saturate your inference engines. A better algorithm buys you full compute utilization — which is what made SWE-1.7 possible.

SWE-1.7 was presented on a 2D cost-versus-performance chart rather than a 1D bar chart. For its size class it lands impressively high, competing with much larger models at far lower cost. The base model was Kimi K2.7; the size-class comparisons were Composer 2.5 (also Kimi-based) and GLM 5.2.

• Entropy collapse was the binding constraint. There was a lot of chatter that RL had plateaued and everything was back to pre-training, but they wanted to push further. The juice in RL comes from diversity within a group and discovery of new behaviors — both measurable as entropy. Entropy normally declines through a run until it collapses and you hit your ceiling. After recipe tuning, the decline became almost imperceptible (still declining, but on a much flatter slope), which let performance keep climbing. Details are in their technical report.
• Training across four countries and three continents. Large contiguous training clusters are genuinely hard to get these days, especially on realistic timelines — you might have to order a year in advance. What's much easier is finding smaller slices of compute here and there. The resulting run spanned Australia, Malaysia, Canada, and the US.
• Only async RL makes this possible: the training cluster sat in the US with rollout clusters scattered globally; inference clusters send training batches up, the trainer sends weight updates back. It works across the globe because what crosses the wire is sparse — RL weight updates are sparse and training batches are just tokens.

Quotes

"A lot of progress in research is just executing the simple things at scale." (~00:27)

The closing philosophy: they don't believe progress requires complex or fancy ideas.

"The not-okay ratio … was an early indicator of an eventual divergence of the training run." (~00:17)

Before a run collapses, even the tone of the model's inner monologue shifts.