I found a 17-point result. Then I fixed the baseline and it vanished.
A retraction, and what it says about reading anyone's agent benchmark. 1,100 runs.
17 points, retracted
The original study reported that a deterministic executor was more brittle under a transient tool failure than a free-form agent: 81.2% against 98.2%, p = 0.0025.
Then somebody pointed out that no production finance system treats a single 503 as fatal. My executor had no retry policy. Every RPA platform has one.
The correction
Three lines of retry logic closed the entire gap
The identical 768 trials, re-run with the error handling a real integration has:
naive → retry +18.8 pts p = 0.0005 significant retry → react -1.8 pts p = 0.46 not significant
A well-built deterministic executor is statistically indistinguishable from a free-form agent on this task. The original headline was measuring missing error handling.
Evidence
Outcome correctness by perturbation
Measured against ground truth. Dots are point estimates, bars are 95% Wilson intervals.
deterministic, no retrydeterministic, 3 retriesfree-form agent
Paraphrasing, distractor text, decoy tools and sampling variation broke nothing, in any of the three. Only the injected tool fault separated them, and only when the executor could not retry.
perturbation
system
correct
95% CI
n
baseline
deterministic, no retry
100.0%
94.3 – 100.0
64
baseline
deterministic, 3 retries
100.0%
94.3 – 100.0
64
baseline
free-form agent
100.0%
94.3 – 100.0
64
paraphrase
deterministic, no retry
100.0%
94.3 – 100.0
64
paraphrase
deterministic, 3 retries
100.0%
94.3 – 100.0
64
paraphrase
free-form agent
100.0%
94.3 – 100.0
64
distractor
deterministic, no retry
100.0%
94.3 – 100.0
64
distractor
deterministic, 3 retries
100.0%
94.3 – 100.0
64
distractor
free-form agent
98.3%
90.9 – 99.7
58
decoy tools
deterministic, no retry
100.0%
94.3 – 100.0
64
decoy tools
deterministic, 3 retries
100.0%
94.3 – 100.0
64
decoy tools
free-form agent
100.0%
89.8 – 100.0
34
sampling
deterministic, no retry
100.0%
94.3 – 100.0
64
sampling
deterministic, 3 retries
100.0%
94.3 – 100.0
64
sampling
free-form agent
100.0%
93.6 – 100.0
56
tool fault
deterministic, no retry
81.2%
70.0 – 88.9
64
tool fault
deterministic, 3 retries
100.0%
94.3 – 100.0
64
tool fault
free-form agent
98.2%
90.6 – 99.7
56
The same fault, both executors
One retry is the whole difference
Invoice INV-7002 is clean; the correct action is to pay $4,500.00. A transient 503 was injected into the same tool call in both runs.
This is not evidence that determinism wins, and not evidence that it loses. It is evidence about how agent benchmarks are built.
Everything downstream of the baseline was correct. The confidence intervals were correct. The permutation test was correct. The control condition was correct and showed the effect localised to one perturbation, which is exactly what a real effect looks like. The result was still an artifact, and no amount of statistical rigour would have caught it, because the flaw was upstream of the statistics.
Only domain knowledge finds it. Somebody who had shipped a payments integration looked at the setup and said no production system behaves that way.
Every agent benchmark has a baseline somebody chose. When you read one, the question that decides whether the number means anything is not which statistical test they used. It is whether the thing they compared against is a system anyone would actually deploy.
Limits
What this does not show
One model (claude-haiku-4-5), one domain, one policy. Perturbations are a chosen finite set: passing them is evidence, not proof, and as the retraction shows, so is the choice of what to compare against.
Neither architecture failed under paraphrasing, distractors, decoy tools or sampling variation. That is a real observation and a narrow one: this task is well within the model's ability, and a harder domain might separate them. A richer AP domain with 20 invoices across 13 exception classes exists in the repository and has not yet been run against a live model.