Safety and Security of Agentic AI

TL;DR

• Safety and security are different disciplines that can no longer be practiced apart. After 40 years on the safety of software-intensive systems — safety meaning physical harm to people and the environment — his message to a security audience was blunt: come talk to us, or we will field systems with very undesirable physical consequences.
• AI stretches the attack surface in two directions: the familiar IT side (training pipelines, prompts, context) and the physical environment itself — stickers on a stop sign made a perception system read "45 mph speed limit" with 93% accuracy in the lab and 83% in the real world.
• Structural causal world models (SCWM) have four layers: an ontology of what matters in the world (e.g. wave height), an uncertainty model, uncertainty bounds, and a bottom layer where algorithm behavior can be formally verified within those bounds. Attacks are modeled as interventions on the causal graph and propagated to a safety metric.
• Compound attack on an unmanned surface vessel: DoS the remote operating center (digital) and fly a UAV holding an image in front of the cameras (physical). Either one alone is safe. Together you get a vessel navigating autonomously while blind to an approaching tanker.
• The counterintuitive finding: neither safety nor security engineering pays much attention to the time ordering of events, yet ordering alone can defeat defenses — you can secure an individual agent and still undermine the whole agentic ecosystem by resequencing.

Key Points

Background: the Centre for Assuring Autonomy at York (~03:26–03:27)

Roughly 40 years on the safety of software-intensive systems, where safety means physical harm to people and the environment; the last nine or ten running a programme on the assurance of autonomous systems, progressively extending to AI and now agentic AI — even though agentic AI isn't deployed much in safety-critical systems yet.

The centre is approaching 100 people across faculty, researchers and PhD students, working in maritime, autonomous driving, and AI in healthcare. This talk used maritime as the illustrative domain because it exposes the nature of the problem well. (A running aside: the slide build-up didn't work, so he talked around the deck.)

Why safety and security have to be analyzed together (~03:28–03:29)

Traditionally these are assessed independently, in isolation. But in cyber-physical systems, cyber attacks cascade through the technical system and through perception systems into unsafe effects — hazards, in safety terminology: states that threaten the physical safety of equipment or personnel.

Introducing AI changes both sides substantially. The attack surface grows to include training pipelines as well as the operating system; and once the system is physical, the physical environment is part of the threat surface too.

The illustration he borrowed (explicitly not his group's work): researchers trained an AI algorithm to find weaknesses in a perception system, then printed stickers to put on a stop sign so the perception logic classified it as a 45 mph speed limit — 93% accuracy in the lab, 83% in the real world.

SCWM: binding security into a causal world model (~03:29–03:30, 03:33–03:35)

Safety rests on models of cause and effect, so his group builds world models that represent those causal structures — structural causal world models (SCWM). He drew a deliberate contrast with the AI community's interest in automatically learned world models: these are described explicitly, and security is bound into the structure.

Four layers:

1. Ontology — the things in the world that matter for this operation. Wave height, for instance, because the force of waves on a vessel alters its maneuverability.
2. Uncertainty — you can't compute exactly how wave patterns interact with the hull, so sea state's effect on vessel motion carries uncertainty.
3. Bounds on uncertainty — that uncertainty is mapped into bounds.
4. Verification — for those who like formal methods, you can verify algorithm behavior at this level: set bounds on the uncertainty and show safe operation within the known bounds.

Security enters as interventions on the causal dependency graph. In practice you build the graph over everything that matters in the situation (his slide highlighted four nodes); security events perturb those relationships, and the effect propagates from attack to an ultimate deviation in a safety metric.

Here the metrics are closest point of approach and time to closest approach. The smaller the distance, the more likely a collision; and time matters especially for large, less maneuverable vessels, which need the prediction much further out to have room to act.

Case study: a compound attack on an unmanned surface vessel (~03:30–03:33)

The setup — representative of, though not identical to, a real system they've worked on: an unmanned surface vessel at what the IMO (International Maritime Organization) calls degree three autonomy, largely remotely operated but able to run autonomously without remote interaction if needed.

Their simulation uses a YOLO object detector plus a dropout protection module (DPM) that falls back to fully autonomous operation if the vessel loses communication with the remote operating center — which can happen for many reasons, jamming among them. AIS, which broadcasts your position to other vessels, is spoofable, and there are plenty of other concerns besides.

The compound attack:

1. Classical digital attack — a denial of service against the remote operating center, so it can no longer control the vessel. Immediate effect: the DPM kicks in and the vessel continues autonomously along the agreed path.
2. Physical attack — fly a UAV and hold an image in front of the cameras, so the approaching tanker is either not detected at all or classified as something completely different requiring a different evasive maneuver.

The point is the composition:

• Attack 1 alone: safe. Fall back to the DPM; the vessel operates safely.
• Attack 2 alone: safe. Remote operators get a data feed, notice, and take appropriate action.
• Both together: an unsafe state. The vessel maneuvers autonomously without knowing a tanker is in front of it, takes no evading action, and ends up on a collision course.

His propagation graph traced how the two initial attacks move through the system: degrading the position estimate of the other vessel, and in the image-attack case changing the vessel's beliefs about what is actually ahead — surfacing as degradation in closest point of approach and time to closest approach.

Conclusions, and the blind spot nobody models (~03:36–03:37)

Three summary points:

1. AI in cyber-physical systems introduces new challenges — a much expanded attack surface including training data, prompts, and context where LLMs are involved.
2. The physical attack surface is real — spoofing images is the obvious case, but the same can be done in other electromagnetic domains.
3. Time ordering is the blind spot. Safety engineering, and to an extent security engineering, tends not to worry about the temporal order of events. Part of why they ran this analysis was to show that ordering can overcome defenses: one agent may be secure, yet resequencing events undermines the safety and security of the agentic ecosystem.

The conclusion is that new analysis approaches integrating safety and security are needed. SCWM is one possible route, currently being extended so the world model also captures shared understanding between agents and humans where dialogue interfaces exist.

His closing appeal to a security audience: if you work on security, come and talk to the safety people — new challenges arise at the interaction of the two, and if we don't look at them together we will deploy systems with very undesirable effects once they reach the physical domain.

Quotes

"I may have might have made one agent secure but actually by changing the time ordering of events I can actually undermine the safety and security of the agentic ecosystem." (~03:36:58)

Per-component security doesn't compose, and ordering is the dimension most often left out of the model.

"If you work on security, please come and talk to we guys who do safety — there's new challenges that arise at the interaction of those two issues." (~03:37:20)

The talk's central appeal.