AI Agent Reliability Benchmark 2026: 65% Once, 25% Always

ThinkingBox is an open-source AI agent reliability benchmark announced on August 19, 2026. Across 507 stateful business tasks run 20 times each, the top model finished correctly on 65.36% of single attempts — but succeeded on all 20 attempts for only 25.25% of tasks.¹²

TL;DR

The headline number is not the reliability number. GPT-5.4 leads ThinkingBox-Bench with 65.36% pass@1. Run the same tasks twenty times and it finds at least one winning path on 91.12% of them — but repeats that success every single time on just 25.25%.¹

That nearly 66-point spread between "can do it" and "always does it" is what the authors call the discovery–reliability gap.

The benchmark grades agents mainly on what changed in the database rather than on what the agent said it did. Under that rule, 77.5% of failed runs die at the tool layer — a failed call, an unsuccessful lookup, no recovery.¹

If you are picking a model for an agent that touches production records, the pass@1 leaderboard is close to useless on its own. Two models within a quarter-point of each other on pass@1 diverge about 4.4× on reliability.

What You'll Learn

• What the ThinkingBox AI agent reliability benchmark actually measures
• The difference between pass@1, pass@20 and pass^20 — and why the caret matters
• The full 12-model ThinkingBox-Bench leaderboard, with the caveat in its footnote
• Why a pass@1 ranking is not a reliability ranking
• Where agents really fail: the four failure modes and their shares
• Who actually built it, and why that shapes what the benchmark measures
• Why this is not the first agent reliability benchmark

What is the ThinkingBox AI agent reliability benchmark?

ThinkingBox is a sandbox for tool-agent-user interaction. It runs an agent against isolated, MCP-compatible tool sessions, lets a simulated user answer follow-up questions, then inspects the side effects the agent left behind.¹²

The framework is paired with ThinkingBox-Bench, a set of 507 executable, policy-conditioned workflows across five business domains. The two ship separately: the framework repo bundles a single cloud_drive scenario purely as an install smoke test, while the benchmark tasks and MCP servers live in thinkingbox-data.¹³

The opening example in Microsoft's own write-up explains the design. A traveller asks an agent to add a quiet-room preference to a hotel booking. The agent looks up the reservation and confirms the request is added. The transcript reads like a clean success.

The booking's special_requests field is still empty. "The agent confirmed work it never did," as the post puts it — and you find out at check-in.²

So the grading rule is: compare the terminal backend state against the required end state, reject wrong, missing or extra effects, and let any trajectory that produces the right records pass. Grading is conjunctive — a task passes only when all required conditions hold.¹

State is not the whole verdict, though. Designated tasks also apply binary rubrics to the final response, checked by a judge model, for things that have no clean database value — whether the agent disclosed that a room preference is subject to availability, for instance.¹²

Worth keeping in mind for what follows: the agent, the simulated user and the judge are all LLMs on separately configurable endpoints.²³ Some of the run-to-run variance the benchmark measures belongs to the harness, not the agent under test.

Task counts by domain: retail and e-commerce 98, travel and hospitality 104, auto insurance 100, neobank internal IT 104, consulting IT and HR 101.¹²

pass@20 vs pass^20: two numbers, 66 points apart

These are different metrics separated by 66 points on the same model, so it is worth being precise about which is which. Microsoft's definitions:²

• pass@1 — how often a model finishes the task when run once.
• pass@20 — "On what percentage of tasks did at least one of 20 attempts succeed?"
• pass^20 — "On what percentage of tasks did all 20 attempts succeed?"

For GPT-5.4 those come out at 65.36%, 91.12% and 25.25% respectively.¹²

The underlying task counts make the split concrete. Of 507 tasks, GPT-5.4 never passed 45 — about 8.9% were unsolved even with twenty tries — and passed all twenty attempts on 128.²

Both figures reconcile exactly: 462 of 507 tasks passed at least once is 91.12%, and 128 of 507 is 25.25%. The published percentages are internally consistent with the task counts, which is more than most benchmark write-ups let you check.

The caret is not a typo for the at-sign. pass@k is optimistic (best of k); pass^k is pessimistic (all of k). Quoting 25.25% as a "pass@20" score inverts the meaning of the result.

The ThinkingBox-Bench leaderboard

Twelve models, every task run twenty times, scores micro-averaged over trials and tasks.¹²

All values are pass@1 percentages. Size is total/activated parameters for mixture-of-experts models. Sorted by average; the paper's own ordering groups proprietary models first.

One footnote in the paper deserves more attention than it gets: "The o3-pro row excludes 636 system/harness-error trials; its valid denominators therefore vary by domain."¹ That row is not measured on the same denominator as the others.

Three patterns hold across the table.

Domain difficulty swamps model ranking. Averaged across the reported models, the paper puts retail at roughly 52% pass@1 and auto insurance at about 23%, with travel, neobank and consulting in between. Within a single model the swing can be worse: Claude Opus 4.6 scores 74.90 on retail and 14.65 on auto insurance, a 60-point collapse.¹

Parameter count predicts little. Qwen3.6-27B averages 32.94 while the far larger Mistral-Large-3 averages 4.66.

Open weights are closer than the top line suggests. DeepSeek-V4-Pro averages 43.26, the strongest open-weight result here and within striking distance of GPT-5.2's 46.28 — though with a different domain profile.¹

One small discrepancy is worth flagging, since both sources are official. The Command Line post says GPT-5.4 "is the only evaluated model above 50% in every domain." The paper says GPT-5.4 and Claude Sonnet 4.6 are.¹² The published table settles it: Sonnet 4.6's lowest domain score is 51.14, so the paper is right and the blog undercounts by one.

"Model size and a single aggregate score don't tell you where an agent will succeed," as the Command Line post concludes.²

Why the pass@1 ranking is not the reliability ranking

Here is the finding that should change how you read agent leaderboards. Claude Opus 4.6 and Kimi-K2.6 are separated by a quarter of a point on pass@1 — 37.91% versus 37.66%.¹²

Their reliability profiles are not close at all:²

Kimi-K2.6 finds a working path on 14.2 points more tasks than Opus does. It repeats that path on roughly a quarter as many — 3.16% against 13.81%, a gap of about 4.4×.

Read as an engineering decision, those are two different products. Kimi is the better explorer; Opus is the better executor. A leaderboard that reports only pass@1 rates them as near-equals.

This is the same argument we made about AI agent reliability and verification loops: structure around the model moves reliability more than the model choice alone does.

One caveat on reading these: the launch write-up gives pass@20 and pass^20 for these three models only, and reports every figure as a point estimate. The paper notes that task-cluster bootstrap intervals live in its appendix, so the gaps between adjacent models deserve less weight than the gap between the metrics.¹²

Where AI agents actually fail: tool errors, not bad answers

Every failed run was assigned one dominant failure signature from the trace — messages, tool calls, tool responses, final answer, termination marker. The paper is careful to call these "observable diagnostics rather than unique causal explanations," and its breakdown covers 11 of the 12 models; Qwen3.5-9B is absent from that table.¹²

The paper uses different labels for three of the same categories — "Incomplete User Resolution" for response quality, "No State-Changing Action" for missing state change, "Wrong State Update" for wrong state change — with identical numbers.¹

The distribution is the story. Only 7.9% of failures are the agent writing a bad final answer; the other 92.1% happen earlier, in the doing.

And the largest category is not malformed tool calls. The paper is explicit: "these are not merely malformed tool calls, but failures to recover from feedback produced by the environment" — and in some cases the agent continues "as though the failed action had succeeded."¹

Tool usage failures are especially concentrated in GPT-5.4 (89.6% of its failures), GLM-5.1 (88.1%) and Kimi-K2.6 (85.2%).¹

These are diagnostic signatures, not diagnosed causes — but the shape of them still suggests where to look in an agent loop: read every tool result, replan after a failed action, and verify state before reporting success. The same emphasis showed up in Slack's 200-run agentic testing data, from a completely different direction.

Who actually built ThinkingBox

Coverage this week has filed this under "Microsoft benchmark," and the interesting thing is that the label is more accurate than the paper's own cover page suggests.

The arXiv paper lists twelve authors across four institutions: the University of Pittsburgh, Northwestern University, the University of California, Irvine, and Microsoft. That reads like an academic collaboration — until you get to the footnote, which marks equal contribution and notes the work was "completed during a Microsoft internship."¹ The universities are the interns' home institutions.

The framework's origin is more specific still. Per the repository, ThinkingBox "began in a private repository and was widely used by a group of developers and scientists working on agentic reinforcement learning for Microsoft Copilot Studio before becoming an open source project," with Nicola Ferri credited as having initiated it. The README thanks a contributor list drawn from "the Microsoft Copilot Studio RL team."³

ThinkingBox-Bench itself was built "In partnership with Toloka," a data-annotation vendor.²

So the honest frame is: this is a shipping product team's internal evaluation and reinforcement-learning harness, opened up — not a neutral lab's measurement of the field. That does not make the numbers wrong. It does mean the task design encodes what one Copilot Studio team decided enterprise agent work looks like, and that the same harness is used to train as well as to score.

Release details: MIT licence, two repositories — microsoft/thinkingbox for the runtime, microsoft/thinkingbox-data for tasks and MCP servers, where the benchmark is tagged ThinkingBox-Bench v1.0. The README says the project "is only tested on Linux," recommends Python 3.12, and its Citation section still read "A paper describing ThinkingBox is coming soon" when checked on August 26 — six days after the paper posted.²³

Is ThinkingBox the first AI agent reliability benchmark? No

It is not — and to Microsoft's credit, nobody involved claims otherwise. Since a launch this well-packaged tends to get remembered as a first, here is the prior art.

The pass^k metric was introduced by τ-bench in June 2024, by Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan. That paper proposed "a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials" and already graded agents by comparing "the database state at the end of a conversation with the annotated goal state."⁴

τ-bench also already found the gap. Its abstract reports that state-of-the-art function-calling agents "like gpt-4o" succeeded on under 50% of tasks and were "quite inconsistent (pass^8 <25% in retail)."⁴

The two reliability figures are not directly comparable — different k, one retail domain versus five, a 2024 model versus a 2026 one. What survives the comparison is the shape: two years and several model generations apart, both benchmarks find that repeatability lags capability by a wide margin.

Microsoft's own comparison table concedes the priority point directly. It marks both tau-bench and tau2-bench as satisfying all four of ThinkingBox's stated requirements, and states: "The tau-bench family is the closest comparison to ThinkingBox-Bench in that they also provide all four properties."²

What ThinkingBox adds, in its authors' words, is "a reusable lifecycle around MCP servers and 507 tasks across five business domains, including auto insurance and internal IT and HR."² The paper positions the work as "following recent reliability-oriented agent evaluation," citing τ-bench.¹

The MCP-native plumbing is the genuinely new part — one process per MCP server, three reserved lifecycle tools the agent never sees, fresh state per attempt. As MCP consolidates under shared governance alongside A2A at the Agentic AI Foundation, evaluation harnesses that speak MCP natively get more reusable, not less.

What this means if you are shipping agents

Four things follow from the data, none of which require you to run the benchmark yourself.

Ask vendors for pass^k, not pass@1. A single-attempt score tells you the model can find a solution. It says nothing about whether it will find the same one tomorrow.

Benchmark your own domain. When one model swings from 74.90% to 14.65% between two task families, a public average cannot stand in for a score on your workflow.

Instrument the tool layer first. If 77.5% of failures are unrecovered tool errors, that is where retry logic, error surfacing and replanning buy the most reliability per unit of effort.

Check state, not transcripts. The failure mode the benchmark was built around — a confident confirmation of work that never happened — is invisible to any evaluator that reads only the final message. Measuring the wrong artefact produces confidently wrong numbers, which is what we unpacked in Muse Code's benchmark jump.

The bottom line

The interesting number in this release is not 65.36%. It is the distance between 91.12% and 25.25% on the same model, on the same tasks, in the same week.

An agent that cracks nine of every ten tasks at least once in evaluation, then repeats itself on only one in four, does not have a capability problem. That is a variance problem — and variance is an engineering problem you attack with retries, verification, state checks and human gates, not one you fix by waiting for the next model.

Or as the paper puts it: "strong tool-use performance does not yet translate into dependable work completion."¹