The compounding error problem in production AI

Several consequential decisions about an agent turn on this figure: whether to put it on a live workflow, whether to widen its limits, how many people to keep in the loop, and what exposure to describe to a board. Those decisions are often made against a benchmark, which does not describe the deployment itself. Nothing corrects that substitution from inside, because a wrong answer nobody checks looks exactly like a right one, and the runs that quietly failed leave the same trace as the runs that worked.

Consider a run of n steps. Each step succeeds with probability p. Every step is required. Each step succeeds independently of the others. The run then succeeds with probability pn. At p = 0.95 and n = 10, the run succeeds 60 per cent of the time. At n = 100 it succeeds 0.6 per cent of the time.

Three assumptions carry that result. A production workflow satisfies none of them. Steps do not share a single success rate. Failures correlate, because consecutive steps read the same upstream state. A retry or a human review alters the process rather than a term in the equation.

pn is therefore neither an upper nor a lower bound on a real system. Correlated failure can put the true figure above it. Accumulating state can put it below. Under equal step reliability, with no independence assumption, the run succeeds somewhere in the Fréchet range, between max(0, np - (n-1)) and p.

The width of that range is the finding. A per-step figure barely narrows the run figure, and no arithmetic closes the remaining distance. Measurement closes it.

Anthropic, OpenAI, Google DeepMind and Meta publish capability benchmarks, among them SWE-bench Verified, MMLU, HumanEval and AgentBench. Each measures what a model does under controlled conditions. None is a direct measure of deployment behaviour. The seven differences below can move the production result, and several are invisible in standard observability or a model card.

1. Data distribution. Training and benchmark data are curated. Production data is anomalous, malformed, partially missing, in unexpected encodings and legacy character sets, emitted by upstream systems carrying their own bugs. The mitigations are familiar: input validation, schema enforcement and retrieval grounding. How much benchmark accuracy survives this layer depends on how tightly the inputs are constrained and has to be measured on the deployment.

2. Tool integration. A benchmark scores the model. A deployment also depends on retrieval, authentication, tool APIs and the systems behind them. Every added call introduces another way for the run to fail: a schema mismatch, partial response, timeout, rate limit or downstream outage. A tool that succeeds 99 times in 100 has about a 95 per cent chance of succeeding on all five calls in a run. The model can be right and the workflow can still fail.

3. Adversarial pressure. Benchmark users cooperate. Production has all users. Adversarial here does not mean only jailbreak attempts. It means malformed dates, SQL-injection-shaped name fields, unexpectedly long inputs, mixed-script unicode, incidental prompt injection arriving inside a retrieved document, and ordinary people pasting things the model was not trained on. Most of it is not malicious. All of it pushes the model out of distribution.

4. Task length. Benchmark tasks are short. Production tasks run long enough for context to accumulate and for an early error to propagate. A 200k-token context window does not mean the model holds coherent state across 200k tokens. The attention budget is finite, and earlier context degrades the further into a trajectory the agent runs. METR measures a 50 per cent time horizon, the task length at which a model succeeds half the time. In March 2025 it put the best model then measured at roughly one hour. By February 2026 it put the frontier near fourteen hours, with a confidence interval running from six to ninety-eight, and states that measurements above sixteen hours are unreliable on its current task suite.

5. Unanticipated failure. A benchmark tests the failure modes its authors imagined. Production supplies the ones they did not: network partitions mid-tool-call, partial JSON, a schema version changing while the agent runs, state poisoned by an earlier run that crashed before completing, race conditions between agents in the same workflow.

6. Drift. A benchmark is a snapshot. Models update when the provider ships a minor version. Prompts evolve as engineers iterate. Tools change as upstream APIs are revised. The integration that worked last month may have a different effective reliability today, which makes a figure measured a quarter ago a stale upper bound rather than a current number.

The benchmarks drift at the same speed. OSWorld scored the best model at 12.24 per cent on computer-use tasks when it was published in April 2024, against 72.36 per cent for a human. Its verified variant now sits near 85 per cent, and a second version released in June 2026 resets the difficulty and puts the best agent at 20.6 per cent. Three numbers, one benchmark name, two years apart. A figure quoted without its date and its variant is not a measurement.

Provenance moves the same way. Since November 2025 the SWE-bench leaderboard has accepted submissions only from teams with open, published methods, so the frontier results quoted in commercial material do not appear on it. Those figures are reported by the vendor whose model is being ranked, which is a different kind of claim from a measured one.

7. Composition. Benchmarks score single agents. Production runs planners calling executors and supervisors reviewing peers. Take a supervisor at 95 per cent reviewing an executor at 90 per cent. Under independence the pair is right about 95 per cent of the time, because the supervisor catches most of what the executor misses. The assumption is the problem rather than the arithmetic: both read the same upstream state, so they can fail together more often than independence allows. The difference has to be measured in the deployment.

Each layer erodes what the layer above it still had, so they compound in the same way the steps of a run do.

A benchmark score is a reference point, not a production rate. None of the sources above gives the distance between the benchmark and a particular deployment across all seven layers. The number that describes a deployment is one measured on that deployment. Otherwise the real rate may surface only when a customer disputes an outcome or a regulator asks for a year of them, at which point someone else establishes it on their schedule.