Enterprise AI

TL;DR

• Two paths to owning your intelligence: train your own model (Cursor's Composer, fine-tuned on Cursor's own data and infrastructure), or adopt a frontier model and engineer a very good harness around it. Both paths share one prerequisite: a really good eval.
• The failure loop: no systematic way to identify agent opportunities → guess where to start → speculate on agent behavior → deploy → try it in a sandbox and assume it works → improve based on failures that are anecdotal or incidentally observed. That loop is why most enterprise AI use cases never reach production.
• Five components of an agentic eval — carried over from Mercor's frontier benchmark APEX into enterprise settings: task (what is the model actually doing), trajectory (its reasoning and tool calls), artifact (what it produces), world / context (the environment it traverses), and rubric / verifier (how it gets graded).
• Human judgment is load-bearing at the grading step — a core belief behind Mercor's business.
• Four things evals unlock: (1) systematically and quantitatively identify the top areas in a company where agents should be deployed; (2) actually deploy agents and teach them what "good" looks like; (3) a foundation for continuous learning — triage a failure up to the skills file so it never recurs, or post-train on it so the model doesn't regress; (4) data monetization — the task/trajectory/context/artifact/rubric scaffolding is extremely valuable training data at the foundation-model level.

Key Points

Where Mercor started, and the 2026 pivot (~00:42–00:44)

He started Mercor at 19, in his second year at Harvard, and dropped out to run it; his co-founders were high-school debate teammates, one of them his actual debate partner.

The company began by selling data to AI labs. The driving realization: models had gotten sophisticated enough that annotating whether an image contains a stop sign was no longer the point. The question became "can we make the model better at engineering, at medical tasks, at legal?" — which demands genuinely sophisticated human expertise, plus the ability to structure that data for the agentic paradigm, in RL environments and gyms.

But Mercor's main focus in 2026 — which he notes not many people know — is enterprise. His layering: labs at the base; then the neolabs; then applied-layer companies and startups (he names Harvey and Cursor); then the average enterprise. All four layers want to own their own intelligence, apply it across the company and train their own models, and they need fundamentally the same offering: training and evals.

Owning your intelligence: two paths, one prerequisite (~00:44–00:45)

1. Train your own model. Cursor is his example: they trained their own Composer model — now a genuinely good coding model — and can fine-tune it with Cursor's data and infrastructure.
2. Adopt a frontier model and engineer a harness that's excellent for the specific product or use case. Plenty of companies are doing this.

The common thread: you need a really, really good eval. That's where Mercor enters, building directly on the work it does with the labs and on its frontier benchmark APEX.

Why enterprise AI stalls before production (~00:45–00:46)

Whether you're training a model for an external agent or deploying agents internally, the blocker is the same: there's no way to measure ground truth or success. What results is a guessing loop —

no systematic way to identify agent opportunities in your product → guess where to start → speculate on the agent's behavior → deploy → try it in a sandbox, assume it roughly works → improve based on failures that are largely observed or anecdotal.

"This loop is really really bad," hence their case for an eval-driven approach to enterprise AI deployment.

What an agentic eval is: five components (~00:46–00:48)

"Eval" is an overloaded term, so he builds from APEX's definition outward. Working with the labs, when a frontier model ships they benchmark it across many tasks in many domains; for agentic use cases the model runs over hundreds of turns, and they analyze the whole trajectory to find where and why it fails.

The same structure transfers to the enterprise:

A worked example (investment banking): a banker prompts the model to "update this merger model to show how the deal affects both companies." Done by hand you'd open Excel, pull up a specific merger model, go into a data room, get feedback, discover something is wrong, look it up, then produce the model. Deploying an agent to do that requires observability over every one of those steps:

• task = update the merger model
• trajectory = the tools the agent calls (found the merger model, updated deal terms, built the tables …)
• artifact = the merger model in a spreadsheet with the right citations
• verifier = subjective criteria (what the output is, whether it meets stylistic guidelines) plus objective criteria (is the thing actually correct)

And human intelligence matters enormously at that grading step — core to how they think about the business.

Four things evals do for an enterprise (~00:48–00:50)

1. Diagnose the opportunity. Evals are the bottleneck to deploying agents successfully, but they also tell you where to deploy. Build an eval for every department, with a proper mapping of tasks and what ground truth looks like in the rubric, and the top areas for agent deployment fall out systematically and quantitatively instead of being guessed.
2. Actually deploy — teach the agent what good looks like in your company and automate the process.
3. Foundation for continuous learning. When an agent fails an eval, you can triage the reason up to the skills file so it never happens again, and you can post-train a model on those failures so it never regresses.
4. Data monetization — an effort they're running with a number of leading companies. Labs badly want models that perform in real enterprise contexts, and the scaffolding — task, trajectory, context, artifact, rubric — is exactly the data that's valuable for training at the foundation-model level.

(In the panel afterward he added the commercial shape of this: for "commodity workflows," Mercor anonymizes the data, scrubs identity, puts it in an RL environment useful to labs, and it's typically a no-cost option for the company because they can generate seven or eight figures by monetizing it.)

Quotes

"Most of these enterprise AI use cases never reach production … just because there's no way to actually measure ground truth or success." (~00:45)

The premise the whole talk hangs on.

"What you need is a task … the trajectory … the actual artifact … the actual world … and then the last thing is a rubric or a verifier." (~00:47)

The five-part definition of an agentic eval, which he repeats verbatim on the panel.

"If you build an eval for every single one of your departments and a proper mapping of these tasks and what ground truth looks like in the rubric, it's very very logical that you could come up with like the top five areas in your company where you should deploy agents." (~00:49)

His most counterintuitive claim: evals aren't just an acceptance gate, they're the prospecting tool.