Talk Session 3: Foundational Capabilities
Reasoning as Control: Adaptive Test-Time Compute for Planning Agents
Furong Huang — Associate Professor, University of Maryland
Self-improvement is usually framed as a model capability, but the model is only one layer; foundation models are increasingly runtime decision makers, and a self-improving agentic system can be improved at three levels — thinking (token-level decoding control), action (genuine self-reflection rather than imitating a stronger model), and workflow (retrieving a topology from a pre-computed bank at test time) — all without retraining the model.
TL;DR
- Reframing self-improvement. It's usually framed as a model capability — the model improves its reasoning, learns from its own output, acquires new skills. But the model is one layer. What actually gets deployed is an agentic system, and the infrastructure around the model is an equally interesting layer to keep improving — possibly more important for vertical applications.
- Runtime decisions form a hierarchy: low-level thinking choices and token control, then per-step action control, then high-level workflow control. All three can be optimized, and all three without retraining the model.
- Three concrete lines of work: (1) Thinking — steer decoding with a reward signal as early as possible; from Transfer Q to a token-level reward model that cuts cost by orders of magnitude, enabling weak-to-strong guidance and real-time multi-objective alignment; (2) Action — "genuine self-reflection," forcing the model to judge which action is better and why rather than mimicking a stronger model's reflection, which is effectively building a world model; (3) Workflow — neither one-for-all nor one-for-each, but a pre-computed workflow bank* adapted at test time.
Key Points
The reframe (~02:34–02:36)
She opens by redirecting the topic toward self-improving agentic systems. When people talk about self-improvement they frame it as a model capability. But the model is just one layer; at the end of the day an agentic system gets deployed, and there's a lot of infrastructure around the model — how it's built, how it's served — that's a very interesting layer in its own right, and possibly more important for vertical applications where you're using a language model inside your own environment and domain.
She then walks through what happens in an agentic system given a task: observe a state transition from the environment → run it through a thinking process → make a decision about the best action right now. Because these tasks are complicated, you decompose them across agents, assign roles, and design a workflow — what she notes people now call loop engineering. Each agent in the workflow decides an action per timestep, and each action decision requires generating a reasoning trace.
That produces a hierarchy of runtime decision making: low-level thinking choices and token control → action control at each timestep → high-level workflow control. Her framing: foundation models are increasingly runtime decision makers, so a self-improving agentic system can improve the model's capability and how the model is deployed at runtime across thinking, actions, and workflows.
Level 1: Thinking — steering at decode time (~02:36–02:40)
Start from the steering problem: you have a reward model and want to steer an LLM to align with that specific reward. The math gives a closed-form solution, and it's a remarkably short answer to a complicated alignment question: take whatever the base model gives you and add a steering signal, where that signal relates to a trajectory reward under some optimal policy.
From an autoregressive next-token sampling perspective this reduces to: look at the language model's output and steer it with an external signal you can learn from your own corporate data.
The discrepancy: you want to steer as early as possible in generation, but reward models are trajectory-level — designed to score complete responses, not partial ones.
- Transfer Q* takes the honest route: do the auto-completion and use the reward model correctly. Right idea, far too slow — on an academic A6000, generating 500 tokens takes 14 hours. Basically impossible.
- A more recent work designs a specific reward model family giving a token-level reward, cutting cost by orders of magnitude. That unlocks arbitrary runtime steering with no model training at all, weak-to-strong guidance (a very small model guiding a very large one), and multi-objective alignment where objectives adapt in real time to user need.
Level 2: Action — genuine self-reflection (~02:40–02:42)
Thinking exists to serve action, so how is the decision made?
Agentic systems are commonly learned via imitation learning, distilling from expert demonstrations — which frequently hits the stuck loop failure: the agent retries the same actions until termination or budget exhaustion.
The natural next move is world-model training, and one popular version is imitating self-reflection. Her objection is sharp: that isn't self-reflection at all — you're not using yourself to reflect, you're mimicking a stronger model that teaches you how to reflect. It works better than imitation learning, but the results aren't satisfying.
So they did something simple, genuine self-reflection: rather than imitating how stronger models think and reflect, force the model itself to do the reflection — which action is better at this point — and to understand why. That's effectively a world model: a mental model of which action is better under which state.
It "worked really phenomenally": improvements over both imitation learning and reinforcement learning, significant gains out of distribution, and performance well beyond prior state of the art. The part she finds most interesting is that it generalizes to general reasoning — this kind of critical learning bears no obvious relation to general reasoning capability, yet it magically improves it.
Level 3: Workflow — a bank you retrieve from (~02:42–02:45)
The top layer: autonomously deciding agent roles and optimizing agent topology. Her example is a fixed workflow for an agentic-safety application that works well — but can you design a framework where a meta designer produces the best agentic framework and harness for each incoming query, autonomously?
Two existing routes: one-for-all (find the single best workflow in the world for the task) and one-for-each (design the best flow per query, maximizing adaptability).
Their answer takes a foundation-model philosophy: neither. Pre-compute a bank of workflows at training time, then adapt to the specific workflow at test time, efficiently. The results show significant performance improvement. (She flagged live that the slide wasn't rendering correctly and offered to share a corrected version.)
Wrap-up (~02:45)
Three levels of runtime decision making in a self-improving agentic system, all about allocating compute intelligently under a limited budget: steer the thinking process as early as possible so you don't run a long process in the wrong direction; control the action; and control the workflow.
Quotes
"Foundation models are actually more and more becoming a runtime decision maker." (~02:36)
The talk's axis: the model isn't just a trained capability, it's the thing making decisions at every deployment layer.
"This is not actually self-reflection — when you are trying to self-reflect, you're not actually using yourself to reflect, you are actually mimicking some stronger model which teaches you how to reflect." (~02:41)
The puncture that motivates genuine self-reflection.
"You're doing it right, but you're doing it way too slow." (~02:39)
Her verdict on Transfer Q*: 14 hours for 500 tokens is methodologically correct and practically unusable.
提到的專案與資源 / Projects & Resources
| 名稱 Name | 說明 | Description | 備註 Notes |
|---|---|---|---|
| Transfer Q* | 用 baseline 模型估計最佳 token 級 value function 的原理式解碼對齊法;正確但極慢 | Principled decoding for LLM alignment using baseline models to estimate the optimal token-level value function; correct but very slow | arXiv 2405.20495 |
| Token-level reward model(近期工作) | 設計 reward 模型族做 token 級 reward,成本降數個數量級,支援 weak-to-strong 與多目標對齊 | Reward model family giving token-level rewards; orders-of-magnitude cheaper; weak-to-strong guidance and multi-objective alignment | 疑為 GenARM(arXiv 2410.08193),講者未報名稱,待確認 / likely GenARM — not named in the talk |
| Genuine self-reflection | 強迫模型自行判斷哪個動作更好與為什麼,而非模仿更強模型的反思 | Forces the model to judge which action is better and why, instead of mimicking a stronger model's reflection | 論文名稱待確認 / paper title to verify |
| Workflow bank | 訓練時預算一批 workflow,test time 即時挑選;介於 one-for-all 與 one-for-each 之間 | Pre-computed bank of workflows adapted at test time; between one-for-all and one-for-each | 論文名稱待確認 / paper title to verify |
逐字稿勘誤 / Transcript Corrections
| 字幕原文 Heard as | 應為 Should be |
|---|---|
| Fuang Hong | Furong Huang |
| ajinky / agendic / a gentic | agentic |
| transfer Q star | Transfer Q* |
| auto reggressive | autoregressive |
| a loop engineering("a" 為贅字) | loop engineering |
| premputee | pre-compute |
待確認 / To Verify
- token-level reward model 那篇「very recent work」的論文名稱(語境高度符合 GenARM,但講者未在逐字稿中報出名稱)。/ Title of the "very recent work" on token-level rewards — context matches GenARM but the name isn't spoken.
- genuine self-reflection 的論文名稱與 benchmark(講者只描述方法與結果)。/ Paper title and benchmarks for genuine self-reflection.
- workflow bank 工作的論文名稱;講者提到的 "one for all" / "one for each" 是否為特定論文的名稱亦待確認。/ Paper title for the workflow bank work; whether "one for all" and "one for each" refer to named papers.
- 講者展示 workflow 範例時提到的 agentic safety 應用具體為何(投影片渲染失敗,現場未細講)。/ The agentic-safety application used as the fixed-workflow example — the slide failed to render.