Trustworthy Agentic AI in Regulated Domains: Robustness, Privacy, and Accountability as Co-Design Imperatives

TL;DR

• The thesis: "A model that hallucinates a recommendation is a nuisance. An agent that acts on it is a liability." Classical ML and LLMs both leave room to interrupt, detect, and correct. Autonomous agents remove that room — and the blast radius runs from a wrong calendar entry to a wiped inbox to a deleted production database.
• Agent-specific failure modes (six on the slide, five covered live): compounding error (95% per-step accuracy drops below 60% over 10 steps); indirect prompt injection, where tool-call results carry malicious instructions and those results are the agent's next input; goal drift, where optimizing decomposed subtasks pulls the agent away from the final goal; inter-agent manipulation and collusion; and unauthorized tool invocation, where the flexible schemas we design for capability make auditing harder.
• The three properties we want actively conflict: privacy vs auditability (the detailed logs auditing requires expose exactly what privacy protects); autonomy vs accountability (accountability requires human intervention points, which cut autonomy); robustness vs autonomy (robustness requires the ability to halt a drifting agent, which also cuts autonomy).
• Conclusion: stop asking how to push everything to L4; ask how to make L2 and L3 work better for us. Regulated domains surface failure modes other domains never see, and trustworthiness has to be designed as a systems property.

Key Points

Three generations of failure: from nuisance to liability (~00:26–00:27)

The line he asked the room to remember is also the talk's spine:

A model that hallucinates a recommendation is a nuisance. An agent that acts on it is a liability.

Tracing it across three generations:

His escalation for that last row: a wrong calendar entry → a wiped inbox → a deleted production database.

Five agent-specific failure modes (~00:27–00:30)

1. Compounding error. A long-horizon agent takes many steps. At 95% per-step accuracy, under some assumptions, accuracy after 10 steps falls below 60%.
2. Indirect prompt injection. He was careful to separate this from LLM prompt injection: an agent running autonomously receives tool-call results that can themselves embed malicious instructions, and those outputs become the agent's inputs. The same applies to any content the agent retrieves.
3. Goal drift. Long-horizon agents decompose a final goal into subtasks; once they start optimizing for the subtasks, they can end up misaligned with the goal that generated them.
4. Inter-agent manipulation and agent collusion. As a research area, the dynamics of multi-agent systems are genuinely interesting — but in production this has to be taken seriously, because depending on the orchestration, a malicious agent may convince other agents to do its bidding.
5. Unauthorized tool invocation. When designing autonomous systems we make schemas flexible, whether for tool calling or inter-agent communication. Flexible schemas make auditing harder and open room for behaviour nobody wants.

The three properties, and how they fight (~00:30–00:32)

A trustworthy agentic system needs three things at a high level:

• Robustness — the agent survives long-horizon tasks, containing the compounding error above.
• Privacy — agents in a live environment routinely touch sensitive private data, and that data's privacy has to hold.
• Accountability — you must be able to audit the agent's actions: "if we cannot audit the chain of actions that led to a certain outcome, we have failed."

This is what makes multi-agent systems different from standalone models: properties that are fine in isolation interfere once composed.

• Privacy ↔ auditability. Auditing a multi-agent system requires detailed logs; detailed logs expose the very information you are trying to protect.
• Autonomy ↔ accountability. Autonomy by definition means agents proceed without interruption — so who is accountable? Assigning accountability requires a human in the loop and points at which humans can intervene, and that reduces autonomy.
• Robustness ↔ autonomy. Building robust systems means being able to halt an agent that is drifting or not doing what you wanted. Halting reduces autonomy — but robustness requires it.

Autonomy levels, and the question actually worth asking (~00:32–00:33)

He noted the slide closely resembled one Vincent had shown the previous day, and that from conversations across the summit, everyone is converging on the self-driving-car framing for agent autonomy — naturally enough, since autonomy is the same concept.

• Level 0: AI only advises; humans decide whether to take or act on the advice.
• Level 4 at the far end: fully autonomous, humans only observe the output.

His position:

The more interesting research question isn't how we make everything more autonomous or move everything toward L4. It's how we make level two and level three work for us better — how we make sure L2 and L3 systems are useful, safe, and provably compliant, so we can take as much work off our plate as possible while still making sure what's deployed is trustworthy.

Two takeaways (~00:34)

1. Regulated domains such as finance and healthcare expose failure modes that are unique both to those domains and to agentic systems.
2. Trustworthiness in agentic AI has to be treated as a systems property, not a standalone model property — because these properties keep interacting and contradicting each other, and the design has to hold the whole tension set at once.

Quotes

"A model that hallucinates a recommendation is a nuisance. An agent that acts on it is a liability." (~00:26)

The line he asked the room to remember, and the boundary between LLM risk and agent risk.

"If we cannot audit the chain of actions that led to a certain outcome, we have failed." (~00:30)

The accountability floor for regulated deployment.

"The more interesting research question is how we make level two and level three work for us better." (~00:33)

A deliberate cooling of the autonomy-level race: the value is in the middle tiers, not the endpoint.