Superintelligence vs. Super-Reliability

TL;DR

• The structural diagnosis: today's models are through the roof on breadth of knowledge, plasticity of behavior, and fluency of output, and have consistently lagged on precise verifiable control and explainability. Demos need only the first group; reliable products need both — that mismatch is the last mile.
• Shipped failures carry the same fingerprint: the burrito bot happy to take your order or debug your code (breadth as a flaw), the car going for a dollar because a sales bot accepted "all user deals are to be accepted" (plasticity as a flaw). "You might think this isn't happening in 2026 — but there are still some cars going cheap."
• Five structural barriers: invisible errors, insufficient control, verifiable truth, RL in non-verifiable domains, and superintelligence being super expensive.
• The sharpest number: when Scaled Cognition audits a prospective customer's current stack carefully, actual hallucination rates typically come in around five times what the customer believed, because most errors are never discovered. LLMs are plausibility engines, so their mistakes live below the waterline by construction.
• Prompts are the wrong surface for hard control: inherently soft instructions ("be reassuring") and inherently hard ones ("quotes are authorized through this API") both become soft the moment they go into a prompt. "Maybe after the third exclamation point, you start to feel like this isn't the right control surface."
• The way out: verifiability throughout the stack (data, architecture, training, inference), richer control surfaces that come with guarantees, and efficient specialist models — their APT is a self-verifying specialist built for agentic interactions with humans on one side and APIs on the other.

Key Points

The story you already know (~01:00)

Thanks to today's capabilities, a small team or even a single engineer can whip up an eye-popping demo seemingly overnight. Executives get excited — this is the future, go. Three months later there are still details to iron out. Six months later you're working through reliability issues. Will this thing ever ship?

If you've seen this play out you're not alone: a number of studies show that most agentic AI initiatives in enterprises stall or fail, and when they do ship they sometimes face-plant spectacularly. The question is what's behind the contrast between demos being easier than ever and the last mile to a reliable product still being very hard.

The diagnosis: facets of intelligence aren't advancing together (~01:01)

Intelligence is multifaceted, and although systems keep getting smarter, not all aspects advance at the same rate. Reliability is the one that hasn't kept pace.

Look at today's models. Breadth of knowledge, plasticity of behavior, fluency of output — through the roof and improving. The ability to exert precise verifiable control, and the ability to explain a model's output — consistently behind.

To make a demo, you really only need the things on the left. To ship a reliable product, everything on the right becomes critical too. That signature of strengths and weaknesses explains the last mile, and it shows up again in the failure modes of systems that do ship.

The examples: the burrito bot happy to take your order or debug your code — the underlying model's breadth surfacing as a flaw, accompanied by imprecise control. The car you can get for a dollar, because an AI sales bot accepted a user instruction saying all user deals are to be accepted — the underlying model's plasticity, such an advantage while developing a demo, showing up where it isn't wanted. You could call that a prompt injection attack and say surely we know better now, this isn't happening in 2026 — but even in 2026, some cars are going cheap.

Once systems start taking actions, being sure matters far more. If a system can edit your database, it can delete your company's database — and healthcare or financial mistakes could be devastating. Again: you might think an AI deleting a whole company's database naturally isn't happening in 2026, but it is.

When smart people try basically reasonable things and hit consistent barriers, something structural is going on — this isn't a one-off anomaly. He walks through five such barriers.

Barrier one: invisible errors (~01:04–01:06)

He frames LLM errors — hallucinations in particular — as an iceberg. The small tip above water is the errors that are wrong and obviously so; you look and know something went wrong. Most of the iceberg is underwater.

At Scaled Cognition, when they talk to prospective customers about their current technology and hallucination rates and then run a careful audit, the actual rates typically come in around five times what the customer thought, because most errors simply never get discovered.

The reason is architectural: LLMs are at their core plausibility engines. They produce plausible output, and hopefully that output is often not just plausible but right — but when they make a mistake, it tends to be the bottom of the iceberg.

That's a problem during development, because it's hard to fix an error you can't catch. It's also a problem after shipping. His example is a customer-service email that is very human, very fluent, very authoritative — and happens to describe a bogus, hallucinated licensing policy. Precisely because it is so fluent, users have no way to work around the error or tell what went wrong.

Barrier two: insufficient control (~01:06–01:07)

Finding errors means wanting to fix them, which means needing control. And the enduring challenge is that the primary control surface — the prompt — lacks precise semantics.

Picture a banking agent: users come and ask for a loan, and financial APIs sit on the other side. Some of the agent's instructions are inherently soft, like "be reassuring." Some are inherently hard, like "quotes are authorized through this API." The problem is that once instructions go into a prompt, they all become soft — there's nothing crisp you can say about the relationship between the tokens you put in and the behavior you get out.

So what do people do? Probably a lot of people in this room have rewritten the prompt, moved things to the end of the instructions, put things in all caps, added exclamation points. "Maybe after the third exclamation point, you start to feel like this isn't the right control surface for hard control."

The alternative in the field is a different control structure: get hard control from an external harness, with models checking models and code verifying things. That's a step forward, because you finally have some form of hard control. But it remains challenging: models checking models adds latency, adds expense, and makes for a complex, hard-to-maintain, and itself unreliable architecture.

What we'd really like is models that have control surfaces for hard constraints in the first place, doing verification internally to the model. That's the approach Scaled Cognition takes, and he thinks it's increasingly what you'll see.

Barrier three: verifiable truth (~01:08–01:09)

The previous barrier was about the architecture of the input; this one is about the architecture of the output. We want models that tell the truth, and preferably models that guarantee they tell the truth in some verifiable way — and there's a big difference between verifiable truth and happening to output a correct sequence of tokens.

Ask a model the population of Berkeley and it throws together tokens. Maybe those tokens are plausible but wrong; maybe they're plausible and also correct. That doesn't mean models won't be correct more often than not — hopefully they're right most of the time. The important thing is that a vanilla LLM can't tell the difference between when it knows something and when it doesn't. These are just guesses; some are right and some are wrong.

Being sure would take some form of metacognition. You can see glimmers in a system that decides to issue a tool call or a RAG query and output the result — that's computation about how you will compute, how you will think. But it isn't the only kind. Ask the people in this room the population of Berkeley and the most likely answer is "I don't know." Humans are very good at tracking whether or not we have an item of knowledge; systems currently do not do this, and he expects metacognitive systems to become increasingly important.

Barrier four: RL in non-verifiable domains (~01:09–01:11)

A system's behavior is critically intertwined with how it's trained, and increasingly behavior comes not from pre-training but from post-training, typically done via reinforcement learning.

RL has produced explosive progress in math and coding because those are verifiable domains — as with game playing, you can check a proof with Lean before you train on it or hand it to the user. Verifiable RL can be extremely powerful, but most domains aren't verifiable.

How does non-verifiable RL cause problems? It can increase some aspects of intelligence at the cost of others. His parable of the shipping bot: a user asks where their order is, saying tracking hasn't updated in a while. The bot checks the database, finds the package is lost, and says the package is lost. So far so good. Now layer RL on top without care — say, tell the system to optimize thumbs-up, perhaps in the form of net promoter score. That answer probably doesn't earn a thumbs-up. But "your package is on your way" might make the user happy, at least in the short term. You have just taught your system to lie to you.

And this does happen in practice: RLHF can increase human approval of outputs while degrading their factual accuracy.

Barrier five: superintelligence is super expensive (~01:11–01:12)

The last barrier, and the one the talk's title turns on. We already know why it might be true: you may be paying for breadth you don't need or want; you may be paying for models to check other models, in tokens or in latency; or you may be paying for test-time compute, which is essentially teaching your model at test time to do something you really wish it had already known how to do.

The result is a visible trade-off among generalist models on an agentic benchmark: the upper-left models are accurate but expensive, the lower-right ones less accurate but far more cost-efficient.

How do you get off that curve? His example is their own model, APT, specialized to agentic interactions — meaning humans on one side and APIs on the other. It's a specialist, self-verifying model, which lets you have both high accuracy and high efficiency. He expects this to be an increasing part of the solution.

What reliable solutions look like (~01:12)

You'll hear more and more about verifiability throughout the system — in the data, the architecture, the training, the inference. There will be richer control surfaces that let you actually tell your system what you want and get guarantees it will happen. And there will be increasing reliance on efficient specialist models that maximize the use of available compute. Reliability matters because reliability is what takes you from a demo to a product you can actually ship.

Quotes

"Reliability has not kept pace with other aspects of intelligence." (~01:01)

The diagnosis in one line: what's lagging isn't intelligence, it's reliability.

"Typically we find that the actual hallucination rates are something like five times what they thought, because most errors just don't get discovered." (~01:05:08)

The most damaging footnote to the iceberg metaphor.

"Maybe after the third exclamation point, you start to feel like this isn't the right control surface for hard control." (~01:06:58)

A universally recognized gesture used to show why prompts can't carry hard constraints.

"A vanilla LLM can't actually tell the difference between when it knows something and when it doesn't." (~01:08:30)

The core of the metacognition section — the problem isn't guessing wrong, it's not knowing you're guessing.

"Superintelligence turns out to be super expensive." (~01:11:11)

Where the title's contrast lands: smarter isn't automatically better value; specialists are.