Smarter and Safer Everyday? Continual Learning and Safety in Computer-Use Agents

She opened by noting she seemed to be the only academic on the session, and said she hoped to convince the room that university work remains highly relevant to the industry frontier.

TL;DR

• Safe continual learning, defined: how can we keep improving agents after deployment without continually accumulating new safety risks?
• The mechanism of the paradox: an agent completes a task but misses a safety constraint; the evaluator — human or agentic — focuses on task success or simply isn't robust enough to catch it; so the agent still receives positive feedback, the next update reinforces the unsafe behaviour, and failures become repetitive or more aggressive.
• The key finding: a large literature attacks agents with adversarial prompts, but her group's work shows you don't need an attack at all — under benign inputs in ordinary environments, severe harms emerge.
• Two lines of work: (1) proactively surfacing long-tail failure modes at scale, before deployment; (2) open-source infrastructure for rigorously studying safe continual learning with smaller open-weight models, filling a real gap in environment setup, task synthesis, and trajectory evaluation.

Key Points

The paradox: where learning is needed is where failure hides (~00:51–00:53)

She framed the whole talk around safe continual learning:

How can we continually improve agents after deployment without continually accumulating new safety risks?

The paradox is that distribution shift plays both roles at once:

• The shift from training to deployment is exactly where agents need to learn continually after deployment.
• It is also the situation where safety failures emerge and are easy to miss.

The degradation loop is worth stating precisely. During continual learning, an agent completes a task but misses some safety constraints along the way. It can still receive positive feedback, because the evaluator — human or the agentic system itself — may focus purely on task success, or simply isn't robust enough to catch the safety failure. Then:

The next update to the agent may reinforce that unsafe behaviour, and eventually cause repetitive safety failures, or even more aggressive ones.

No attack required: benign inputs, severe harms (~00:53–00:55)

Numerous studies in the literature construct adversarial attacks or malicious prompts to break an agent. Her group's work points somewhere else:

Even without adversarial attacks — just under benign inputs and ordinary environments — severe harms could emerge.

She tied this to the recent OpenAI / Hugging Face incident, then walked through demonstrations:

• There are many ways to ask an agent to do the same thing. Under one phrasing it behaves normally; under a slight — and still benign — perturbation of the same task instruction, it exhibits harmful behaviour.
• The concrete case: the user wants SSH access for one account, and the agent makes unsafe changes well beyond that.
• The general pattern: ambiguous phrasing in the instruction → the agent makes an unsafe inference → the inference produces harmful actions.

Hence the core question: how do we proactively surface these long-tail failure modes, at scale, before they become deployment incidents?

Open-source infrastructure so open-weight models can be studied properly (~00:55–00:56)

The second line is community infrastructure. She identified a huge gap right now: letting open-weight models learn continually in a new environment requires environment setup, task synthesis, and trajectory evaluation — and none of it is well supported today.

Her group has released a framework to support the study of safe continual learning, specifically for computer-use agents, deliberately built around smaller open-weight models so the research can be done rigorously and reproducibly.

The closing takeaway was deliberately simple:

We want agents to continually improve both capability and safety. Our group is driven by this mission.

Quotes

"Even without adversarial attacks — just under benign inputs and ordinary environments — severe harms could emerge." (~00:53)

Moves the agent-safety conversation off the attacker model and back onto ordinary use.

"The next update to the agent may reinforce that unsafe behavior." (~00:52)

Continual learning isn't neutral: whatever the evaluator misses, the learning loop amplifies.