Viable Systems, Judgment, and AI Safety

TL;DR

• On the ground, AI safety is an accountability problem — not accountability in the auditability sense Lovedeep Gondara used, but in the sense that a person has to stand up and say "I'm signing that off, that's my responsibility." The pressure isn't only the EU AI Act; it comes from enterprise customers unwilling to risk their business on agents they don't understand.
• The Good Regulator Theorem cuts both ways. Conant and Ashby (1970): to delegate authority you must hold a good model of the entity you're delegating to — and the delegate must hold a good model of the delegator, so they know when to surface problems. Delegate to a model so large you don't understand how it will solve the problem and, in his words, you're stuffed.
• Agentic debt: wire judgment-free delegation into company workflow systems and you are storing up downstream trouble — you no longer understand how your own systems operate. Trent's answer is to invert it: don't replace engineers, support the security engineers who actually carry the accountability.

Key Points

Accounting is not accountability (~00:46–00:47)

After setting up Trent AI as a company straddling the US and Europe (he was born in New Jersey and lives in the UK), Lawrence went straight at what he sees in customer engagements: what does AI safety look like on the ground? It's an accountability problem.

Accounting is in the numbers. Accountability is in the human authority and the judgment.

And the trouble is:

Computers do accounting very, very well, but they don't do accountability well. They are not socially accountable. They can't be sent to jail. They can't be embarrassed. They can't lose their job. And our society is entirely based on that form of accountability.

Which produces the gap he cares about:

If you separate the human accounts from the accountability, you put the human in a situation where they don't understand the accounts and they can't stand up for the accountability. And that's the gap we've got to bridge.

He was equally clear that this pressure is broader than regulation: it isn't just the EU AI Act, "that's coming from customers, from enterprise customers, people who are not willing to risk their business on the back of AI agents they don't understand." And on the ten-year horizon: "I don't know what the world's going to be like in 10 years' time … but I can tell you what's not going away."

The Good Regulator Theorem: the precondition for delegating (~00:47–00:48)

The status quo amounts to devolving authority without judgment. He reached for Roger Conant and Ross Ashby's 1970 Good Regulator Theorem to explain why that fails:

If I'm going to delegate authority, I have to have a good model of the entity I'm delegating authority to.

His illustration: if I tell an intern to make me coffee, I need a good sense that they'll do it sensibly — that they won't go away and hack the NSA to get the optimal coffee recipe, they'll just make me a coffee.

And it runs in both directions: the delegate must have a good model of who's delegating to them, so they know when to surface a problem — the machine's run out of coffee, I don't know how to deal with that.

Applied to agentic AI:

If you're delegating authority to a model that is so large and so big that you don't understand how it's going to solve the problem, you're stuffed.

Not a theoretical worry, he stressed, but what they genuinely see arising: you cannot delegate authority that way.

Agentic debt, and what Trent actually builds (~00:48–00:50)

The reason to avoid that situation is that it creates agentic debt:

If you build that into your workflow systems in companies, you are storing up downstream trouble. You don't understand how your own systems are operating.

Trent's entry point is cybersecurity — when they set up, their first read was that this would be the first front of the problem. What they hear inside companies is consistent:

We don't want you to do our cybersecurity for us. We want you to support our security engineers in doing their job, because they're becoming overwhelmed.

(He offered Apple switching off its bug bounty system as evidence of that overload.) The point being: those security engineers are the people who are accountable within the business.

So the line he draws is sharp:

• What you can do: use agentic AI systems to support security engineers.
• What you absolutely cannot do: walk in and say "here's a system you don't understand and have never seen before, and you now have to sign it off and take responsibility for deploying it." That's absolutely disastrous.

He closed by returning to the Europe/US contrast he opened with: the way to bridge "everything everywhere all at once" and "not here, not now" is by talking to customers — and one of the challenges he sees in agentic AI is that there isn't enough of that.

So when we deploy, we have to remember: accountability is king.

Quotes

"Accounting is in the numbers. Accountability is in the human authority and the judgment." (~00:46)

The line he says he sticks with, and the axis of the talk.

"They are not socially accountable. They can't be sent to jail. They can't be embarrassed. They can't lose their job. And our society is entirely based on that form of accountability." (~00:47)

Why "let the AI be accountable" fails at the level of definition.

"If you're delegating authority to a model that is so large and so big that you don't understand how it's going to solve the problem, you're stuffed." (~00:48)

The Good Regulator Theorem, applied directly to frontier models.

"Accountability is king." (~00:50)

The last line of the talk.