OSWorld 2.0: Top AI Agents Finish Just 20.6% of Tasks

In one line: OSWorld 2.0 is a computer-use benchmark of 108 real workflows that take a skilled human a median of 1.6 hours, and the best frontier agent completes 20.6% of them end-to-end — a collapse driven by state management, not by GUI skill.

TL;DR

A team led by XLANG Lab at the University of Hong Kong, with Snorkel AI as research and data partner, published OSWorld 2.0 to arXiv on June 28, 2026, revising it on July 13.¹ It replaces the short desktop tasks of the original OSWorld with 108 long-horizon workflows that take skilled humans a median of about 1.6 hours, with 69.6% running over an hour.²

The headline result: at a 500-step budget, Claude Opus 4.8 with maximum thinking and batched tool calls leads at 20.6% binary completion and a 54.8% partial score. GPT-5.5 plateaus at 13% — the same figure at 150, 300 and 500 steps.³ On the verified leaderboard, no evaluated system clears 21% end-to-end.

One caution before going further, because it is already causing confusion in secondary coverage: OSWorld 2.0 reports two different numbers, and vendors mostly quote the higher one. OpenAI's own materials put GPT-5.6 Sol at 62.6% on OSWorld 2.0 — but that is the partial score, not end-to-end completion.⁴ More on why that distinction matters below.

The interesting part is not the low number. It is why it is low. The paper is blunt: agents are not "stumbling on basic GUI control or coding." They "lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification."¹ The bottleneck moved from perception to memory.

What You'll Learn

• What OSWorld 2.0 measures and how it differs from the original OSWorld
• Why the same model scores 83.4% on one OSWorld and 20.6% on the other
• The exact point on the time axis where agent completion collapses to zero
• Which failure modes the benchmark isolates, with real trajectory examples
• The full leaderboard, including cost per evaluation run
• Why vendor-quoted OSWorld 2.0 scores are 3x higher than the leaderboard's
• The methodological caveats — including which models are missing
• What to change if you are shipping long-running agents

What OSWorld 2.0 Actually Measures

OSWorld 2.0 is a benchmark that scores computer-use agents on complete professional workflows rather than isolated desktop actions. Each of its 108 tasks is an end-to-end job — filing a reimbursement, reconciling purchase approvals, building a CAD model to a drawing — executed across desktop applications and 31 self-hosted websites.²

The scale change over OSWorld 1.0 is the whole point. Tasks in the original averaged fewer than 30 agent steps. OSWorld 2.0 averages more than 250.² Measured in tool calls, one task takes an average of 318 with Claude Opus 4.7 at maximum thinking, against roughly 30 in version 1.0 — about ten times the operational load.¹

Scoring changed too. Version 1.0 was binary: you finished or you didn't. OSWorld 2.0 reports both a binary completion rate and a partial score averaged over 27.25 checkpoints per task, of which 11.53% come from validated model-based checks rather than deterministic graders.²

The 108 tasks span seven professional domains and 21 sub-categories — research, creative production, engineering, personal services, business and finance, administration and compliance, and healthcare. The authors map them to occupation families and use a wage-bill GDP proxy of $1.64 trillion to argue the corpus covers economically real work.⁵

A separate pipeline runs eight safety diagnostic checks per trajectory, scored independently of task completion — so an agent that reaches the right outcome by unsafe means stays visible.²

The Same Model, Two Benchmarks, 63 Points Apart

Here is the comparison that makes OSWorld 2.0 worth your attention. Each row is a single model, measured on two benchmarks, with both figures current.

Claude Opus 4.8, released May 28, 2026, scores 83.4% on OSWorld-Verified.⁶ For reference, the original 2024 OSWorld paper reported a 72.36% human baseline — though that figure was measured on the full 369-task set, not on the Verified subset, so treat it as a rough marker rather than a like-for-like comparison.⁷ On OSWorld 2.0 the same model finishes 20.6% of tasks. That is a 62.8-point spread. For GPT-5.5, released April 23, 2026, the spread is 65.7 points.⁸⁹

This is not a regression, and it is not an apples-to-apples comparison — different task sets, different graders, different harnesses. What it measures is how much of a benchmark score is a function of task length. Hold the model constant, stretch the horizon from thirty steps to two hundred and fifty, and between three-quarters and four-fifths of the apparent capability disappears.

Worth stating plainly: the OSWorld-Verified figures are vendor self-reported. The tracker that aggregates them lists 21 self-reported results and zero independently verified ones.⁶ Anthropic also changed how it runs the evaluation for the Opus 4.8 release, restating Opus 4.7's score upward to 82.3% under the new harness — so part of any generational gain there is methodology, not model.⁸

The Horizon Wall

The benchmark's most concrete finding is a wall on the time axis, and the authors plot exactly where it sits.

On tasks a human finishes in under 45 minutes, GPT-5.5 and Claude Opus 4.7 reach 20–24% binary completion.⁵ Performance then declines steadily. By the 137–163 minute bin, no model exceeds 10%. Above 163 minutes, binary completion falls to zero for every model retained in the plot.⁵

Partial scores tell a gentler story: they cluster in the 20–55% range across every evaluated system.² Agents are not flailing. They make substantial, checkpoint-verified progress on almost every task — and then fail to land it.

Cost scales worse than capability. Snorkel's analysis notes that crossing the 50% partial-score threshold requires an order of magnitude more tokens than reaching 25%.² Buying the last stretch of progress is disproportionately expensive.

Where Agents Break Down

The authors tag each task with recurring challenge phenomena, which turns "the agent failed" into something diagnosable. The five most common, across 108 tasks:⁵

Tags overlap — a task can carry several — but the pattern is consistent: the hard part is reconciling facts scattered across email, documents, records and prior messages, and recovering state nobody stated in the prompt.

The published failure trajectories make it concrete. In Task035, an agent assembling a purchase-order sheet from team-chat messages and a spreadsheet notices a late approval arriving mid-run, files it as a one-off exception, then executes from the stale table it built earlier. Its final self-check validates its own internal five-row story rather than the actual requirement — so it verifies a document that is missing an approved request.⁵

In Task008, an agent gathering reimbursement evidence keeps expanding its search, detours into scripting a document generator after an API returns 404, and only pivots to the target expense app at step 446 of a 500-step budget — with the form itself loading at step 459. It ends the run reverse-engineering a read-only date field. Every individual step was reasonable. The horizon ran out before submission.⁵

That is the through-line. These are not grounding failures. They are failures of knowing when to stop gathering, what to re-check, and which earlier conclusion is now void.

The Full Leaderboard

At the 500-step budget, ten configurations across seven models:³

Two details deserve attention. Batched tool calls buy real points — Opus 4.8 gains just over two points and Opus 4.7 gains 4.3 points over their standard configurations, purely from how calls are issued.

And GPT-5.5 does not move at all with more steps: 13% at 150, 300 and 500.³ It is the top scorer at the 150-step budget — where the next best system manages 4.6% — and simultaneously the one system that gains nothing from a longer leash. The paper describes it as "far more token-efficient," and that efficiency is real; it just buys no additional completions.¹³

The Two Numbers Everyone Is Mixing Up

Newer models have been measured on OSWorld 2.0 — just not on the verified binary leaderboard, and not on the same metric. This is where most current coverage goes wrong, so it is worth walking through carefully.

OpenAI's GPT-5.6 announcement, published July 9, 2026, includes a computer-use table with an OSWorld 2.0 row:⁴

Look at the Claude Opus 4.8 row: 54.8%. That is precisely Opus 4.8's partial score on the Snorkel leaderboard — where its binary completion is 20.6%.³ OpenAI is reporting the partial-credit metric.

So GPT-5.6 Sol's 62.6% is a partial score, and the honest comparison is 62.6% against Opus 4.8's 54.8% — a roughly 8-point gain on partial credit, not a tripling of end-to-end completion. OpenAI states the comparison correctly, noting Sol "surpasses Opus 4.8 while using 85% fewer output tokens."⁴

Anthropic published an OSWorld 2.0 result for Claude Opus 5 on July 24, reporting that it "outperforms every other model at any given cost, surpassing Fable 5's best result at just over a third of the cost."¹⁰ Anthropic's charts are published as images rather than text tables, so no absolute figure appears in the announcement copy. Widely circulated third-party figures of roughly 70% for Opus 5 pair it with an Opus 4.8 value near 55.7% — again the partial-score scale, not binary.

Where this breaks down in practice: at least one aggregator now publishes a single OSWorld 2.0 table listing GPT-5.6 Sol at 62.6% alongside Claude Opus 4.8 at 20.6%, describing the column as binary completion. Those two numbers come from different metrics. Read that table at face value and you conclude GPT-5.6 tripled the state of the art in three weeks. It did not.

The defensible summary: on end-to-end binary completion, the best verified result is still 20.6%, and as of July 31, 2026 no newer model has published a binary-completion figure. On partial credit, GPT-5.6 Sol leads at 62.6% and Opus 5 claims better. Never compare across the two.

What the Benchmark Doesn't Tell You

Four further caveats.

The verified leaderboard trails the frontier. As of the July 28 update, Snorkel's top entry is Claude Opus 4.8 (May 28, 2026). Claude Fable 5 shipped June 9, GPT-5.6 on July 9, and Claude Opus 5 on July 24.⁸¹⁰¹¹⁴ Their OSWorld 2.0 results are vendor-run and vendor-reported, not independently verified through the project's evaluation process.

OSWorld 2.0 is not the first long-horizon agent benchmark. OS-Marathon, Odysseys and Long-Horizon-Terminal-Bench all target overlapping ground.¹²¹³¹⁴ What OSWorld 2.0 contributes is the self-hosted environment suite, checkpoint-level partial credit and the challenge-phenomenon taxonomy — not the idea.

The abstract disagrees with itself on one figure. The project site puts GPT-5.5 "near 14%"; the arXiv v2 abstract says "near 13%".¹⁵ The leaderboard resolves it at 13%, and that is the figure used here.³ Vendor and leaderboard partial scores also differ slightly for the same model — GPT-5.5 is 49.5% on Snorkel and 47.5% in OpenAI's table, reflecting different runs.³⁴

Task classes are gated. The official task definitions ship through a gated Hugging Face dataset to limit benchmark leakage, while code and documentation stay public. Leaderboard placement requires a coordinated evaluation run or trajectory submission from a trusted institution.²

What This Means If You're Shipping Agents

The practical read is that horizon length, not model choice, is the variable that decides whether your agent finishes.

Budget by task duration, not task type. Above roughly 2 hours 45 minutes of human working time, binary completion falls to zero for every model in the published plot — and it is already under 10% past about two and a quarter hours.⁵ Decompose the work, or put a checkpoint in the middle.

Give the agent durable state. In each of the three failure trajectories the authors published, the agent trusts its own accumulated summary over the environment.⁵ A working set that lives outside the context window — and gets re-read, not remembered — attacks that failure mode directly. Our post on AI agent memory systems from Microsoft and Huawei covers what that infrastructure currently looks like.

Force re-verification before terminal actions. Task035 failed its own final check because it validated its internal model instead of the requirement. A verification step that re-reads sources rather than the agent's notes would have caught it.

Design for the interrupt. Just 9.3% of OSWorld 2.0 tasks are tagged "dynamic environment" — new information arriving mid-run — yet that category supplies one of the three failure trajectories the authors chose to publish.⁵ If your workflow touches a live inbox or chat channel, assume the plan will go stale mid-execution.

Watch cost, not just accuracy. The most expensive run Snorkel reports on this corpus is $3.87K, and the two top-scoring configurations report no cost figure at all.³ Read that alongside what a single agent task actually costs in tokens before committing to a long-horizon design.

For the contrast in framing, our earlier coverage of GPT-5.4 passing the human baseline on computer use describes the short-horizon picture that OSWorld 2.0 was built to complicate.

The Bottom Line

OSWorld 2.0 does not show that computer-use agents got worse. It shows that the benchmark everyone was citing measured a shorter problem than the one enterprises are buying against.

A model can score in the 80s on 30-step desktop tasks and still finish only one in five real workflows end-to-end. The gap between those two numbers — 62.8 points for Claude Opus 4.8 — is the honest size of the distance between demo and deployment right now.

The encouraging half is that partial scores sit between 20% and 55% on the verified leaderboard, and vendor-reported partial scores for the newest models run higher still. Agents do most of the work. What they lack is the discipline to track what changed, re-check before committing, and ask when the evidence conflicts. Those are engineering problems with known shapes — durable state, forced verification, human checkpoints — rather than a wall that requires the next model generation to break.

The data point still missing is an independently verified binary completion number for Opus 5, Fable 5 or GPT-5.6. Until one exists, 20.6% is the figure to plan against — and any OSWorld 2.0 number above about 45% you see quoted is almost certainly partial credit, not a finished job.