Measuring API Agent Reliability for Long-Horizon Tasks in Production

TL;DR

• High single-task scores are an illusion. On APIFlow-Bench even lightweight models pass 88–97% of single API tasks; chain the same subtasks into a dependent sequence and the score falls to 44–73%. The difficulty didn't change — what changed is whether errors get inherited.
• API work is decomposed into seven failure modes, graded separately: authentication, discovery, schema repair, multi-step execution, error recovery, pagination, and statefulness. Separate grading is what tells you why a model failed — failing to persist state at step 3 so step 15 breaks is a completely different illness from mishandling pagination.
• Synthetic tasks need hard gates to be trustworthy. Three of them: self-testing (a blank submission and a sabotaged-evidence reference answer must both fail), solvability (a frontier model gets 10 attempts and the task is kept only if at least 3 pass), and golden replay (concatenate every subtask's reference answer into the chain and the whole thing must pass).
• You need both deterministic and LLM validators. Two deterministic validators — final environment state and final answer — produce the leaderboard score; the LLM validator doesn't score but flags doubtful passes for human review. A wrong validator can never reveal a model's true performance.

Key Points

The problem: enterprise logic lives in APIs, and agents are now wired into it (~02:24–02:26)

Wan, an engineer at Postman, presented their benchmark APIFlow-Bench.

Enterprises running AI agents on long-horizon tasks already face a concrete failure: a small mistake the agent makes early can fail the entire task. And the work engineers actually do looks like this — connecting a build job to a monitoring API, creating a record, recovering from a rate limit. Those task shapes are what the benchmark draws from.

The headline finding is blunt:

His emphasis: the task itself didn't get harder. Models start failing precisely when a later task depends on an earlier task's result.

In the enterprise, a great deal of business logic is wrapped in APIs, and AI agents are now being wired into those APIs. A small AI-caused error becomes a big problem later on.

So the real question isn't whether a model can answer questions — everyone knows it can. It's whether an AI agent can stay correct across a long horizon of dependent chained tasks. That's the difference between answering and executing.

Seven failure modes, graded separately (~02:27)

One or two failure modes isn't enough; they wanted a minimal set that covers real API work. So API work is broken into seven failure modes, each graded on its own:

1. authentication
2. discovery
3. schema repair
4. multi-step execution
5. error recovery
6. pagination
7. statefulness

Why separate? His statefulness example: the agent updates something at step 3, and step 15 only works if that update was actually saved back at step 3. A model that fails on statefulness fails for entirely different reasons than one that fails on pagination. Grading them apart is what tells you why a model failed rather than just that it did.

Task generation pipeline and the fixer agent (~02:27–02:29)

They approximate real user workflows using internal data and build a simulated enterprise API ecosystem that mimics a real company's APIs, then generate tasks on top of it.

The pipeline starts at step zero with enterprise API ecosystem generation, then walks through nine steps generating tasks one at a time. Every generated task passes through validation:

• A frontier model attempts each task 10 times; the task is kept only if at least 3 attempts pass.
• The reasoning is practical: some LLM-generated tasks simply aren't solvable at all. You don't want to evaluate frontier models on a pile of unsolvable tasks and then claim you've found the limit of those models.
• When a task fails validation, the trial transcript and code go to a fixer agent, which repairs the task, and the 10 trials rerun. If it keeps failing, the task is discarded and regenerated from scratch.

The harness and three validators (~02:29–02:30)

The benchmark is four pieces: an API ecosystem, tasks built on it, one shared harness, and graders that score each trial.

• Harness: 7 tools operating on 5 different entity types. Every task and every trial runs on the same harness — only the ecosystem and graders differ.
• Three validators:
• Two deterministic — one checks the final state of the environment, one checks the final answer the agent produced. These two generate the pass rate you see on the leaderboard.
• One LLM validator — not part of the score, used to flag doubtful passes so a human can review them.

Three gates for trustworthy generated tasks (~02:30–02:31)

Each gate catches a different failure of the generation process:

1. Self-testing: a blank submission must always fail, and taking the reference answer while sabotaging the evidence must also fail. Otherwise something is wrong with the task or the validator.
2. Solvability: the frontier model's 10 attempts with a minimum of 3 passes, as above.
3. Golden replay: when building a chain task, concatenate all the reference answers from each subtask, append them into the chain, and the whole thing must pass — confirming the assembled chain is correct and behaves as intended.

Scale and results (~02:31–02:32)

• 467 tasks, yielding 20-step long-chain tasks
• 19 models, 5 trials per model per task
• Roughly 44,000 trials total, with every transcript published on the leaderboard
• Next version will expand both task count and chain length

Leaderboard results on the 20-step chain tasks: GPT-5.5 on top; orange bars are open-source models, purple are closed. He noted the leaderboard had been refreshed that week to include Kimi K3 and Fable 5.

One notable finding: as of now, open-source models are approaching the top group of closed models — on their bench, GLM and Qwen 3.7 land in second and third place.

Three takeaways (~02:32–02:33)

1. Single tasks no longer separate models. Everything, including lightweight models, scores high on solo tasks, so the gap stays invisible — which is exactly why they built chain tests to find the ceiling. What they saw repeatedly: a model makes one tiny mistake early, and that tiny mistake becomes a broken report later.
2. When generating synthetic tasks, always use hard gates. For instance, 10 trials with at least 3 passes; plus golden replay to verify your reference answer actually works. He encouraged the audience to invent more of their own.
3. You need both deterministic and LLM validators, and when building a validator or grader, keep a human in the loop polishing it — because a wrong validator cannot reveal the true performance of the models.

Blog and leaderboard are at blog.postman.com.

Quotes

"So the task itself didn't get harder. But if the later task is depend on the results of the previous task … then that's where the model start to fail." (~02:26)

The core diagnosis: the collapse comes from dependency, not difficulty.

"The real problem is: can the AI agent stay correct across a long horizon of dependent chain tasks? That's the difference between answering and executing." (~02:26)

"We don't want to generate a bunch of unsolvable tasks and then use those to evaluate frontier models and claim that's the limit of the models." (~02:28)

The honesty principle for synthetic benchmarks: prove the question is answerable before drawing conclusions from it.

"A wrong validator cannot reveal the true performance of the models." (~02:33)

The validator itself needs validating.