Agentic Inference Benchmarks in 2026: Agents per Megawatt

In one line: An agentic inference benchmark measures how many concurrent AI agents a serving system can sustain while holding a per-agent speed and latency target — not how fast it answers one prompt. Two arrived four weeks apart in 2026, and the headline metric of the first is agents per megawatt.

TL;DR

Within four weeks, two of the organisations that publish the industry's inference scoreboards both opened a new unit of measurement. Artificial Analysis published the first results for AA-AgentPerf on June 12, 2026, leading with a metric it calls Agents per Megawatt.¹ On July 8, 2026, MLCommons added an Agentic Inference benchmark to its MLPerf Endpoints framework, with the enterprise workload contributed by Workato and the wider effort involving AMD, Intel and NVIDIA.²³ Both drop the single prompt in, single answer out shape that inference benchmarks have used for years, because agents don't work that way. Separately, a KAIST paper presented at HPCA in February puts a number on the underlying economics: on tool-heavy agent tasks, GPUs sit idle up to 54.5% of the time.⁴ Here is what each benchmark actually measures, where NVIDIA's 20x headline comes from, which caveats the launch coverage skipped, and what changes if you are the one signing the compute invoice.

What You'll Learn

• Why single-turn inference benchmarks stopped describing production AI workloads
• What "agents per megawatt" measures, and exactly how AA-AgentPerf derives it
• Which hardware leads the first published results, and who tuned it
• What MLPerf's Agentic Inference track adds that AA-AgentPerf does not
• How the two benchmarks differ, side by side
• Why the viral KAIST energy number is the least useful figure in its own paper
• The methodological caveats that the launch coverage left out
• What to ask a vendor once "concurrent agents" is the unit

Why Single-Turn Inference Benchmarks Broke

Traditional inference benchmarks measure one model call: how fast a response comes back, and how many simultaneous requests a system handles. Agents invalidate both halves of that.

Artificial Analysis describes the mismatch bluntly: "The dominant AI workload of 2026 looks nothing like the workloads inference benchmarks were built for."¹ Its own dataset shows why. A coding-agent session in AA-AgentPerf runs up to 200 turns and pushes sequence lengths beyond 100K tokens. Input length per request ranges from roughly 5K to 131K tokens with a mean near 27K — driven by accumulated tool output and history, not by the prompt someone typed. Output length is lopsided in the other direction: Artificial Analysis notes that lengths "vary widely across turns," with agents mostly emitting short outputs such as simple tool calls, punctuated by longer stretches of reasoning.¹⁵

That shape breaks the old benchmarks in three specific ways. Long shared prefixes reward KV cache reuse. Bursts of very short outputs stress the scheduler rather than raw decode throughput. And the same prefix returns turn after turn, so — as Artificial Analysis puts it — a system's "KV cache behavior, scheduler, and memory hierarchy decide whether it thrives or collapses under this pattern." It is this whole shape, in the same piece's words, that "uniform-length synthetic benchmarks miss entirely." Worse, most hardware benchmarks measure synthetic requests with the production optimizations switched off — a deployment style, again in its words, that "nobody ships."¹

MLCommons frames the same problem from the workload side. In its call for submissions for the edge agentic benchmark it writes that "a real agentic session is not one prompt and one answer – it is a trajectory where each turn appends tool outputs and model responses to a growing conversation history, the model alternates between short tool calls and longer reasoning, and every turn depends strictly on the one before it."⁶ In the datacenter benchmark it puts the consequence in measurement terms: "turn dependencies make throughput a measure of closed-loop progress, not independent request rate."² Independence between requests — the assumption underneath conventional throughput testing — simply doesn't hold.

What "Agents per Megawatt" Means

Agents per megawatt is the maximum number of concurrent agents a system sustains for each megawatt of measured power, while every agent still meets an output-speed and time-to-first-token target. Artificial Analysis calls it "the most important metric for the AI buildout in a power-constrained world" — its framing, not a neutral finding, though the reasoning is easy to follow: if power is the binding constraint on capacity, then capacity per watt is the number that decides what a data centre can actually serve.¹

The derivation matters more than the slogan. AA-AgentPerf fixes the level of service first, then asks how far concurrency can scale before that service breaks. Speed is measured as P25 output speed per request — the 25th percentile, chosen deliberately because agentic traffic contains so many small-output requests — paired with P95 time to first token.⁵ For DeepSeek V4 Pro at maximum reasoning effort, the published tiers are:

A separate, much faster tier set exists for gpt-oss-120b, running from 100 tokens/s at ≤5s TTFT up to 2,000 tokens/s at ≤1s.⁵

For each tier, the harness ramps load exponentially until a target breaks, then binary-searches for the maximum sustainable concurrency. A phase only counts once at least 30 trajectories have completed, every simulated agent has completed at least three, and at least ten minutes of steady-state measurement has elapsed. Results are then normalised three ways — per accelerator, per system, and per megawatt — and the per-megawatt figure uses measured accelerator power under load (GPU die plus HBM), not rated TDP.⁵

That last detail is the one to keep. A per-watt claim built on TDP is a spec-sheet claim; one built on measured draw under load is a measurement.

Which Hardware Leads, and Who Said So

The first wave of AA-AgentPerf results covers NVIDIA and AMD systems running DeepSeek V4 Pro — a large mixture-of-experts model — from single accelerators up to full racks: GB300-NVL72, B300 x8, MI355X x8 and H200 x8.¹ Artificial Analysis reports three findings rather than a single ranking: rack-scale deployments show clear gains from scale and aggressive disaggregation; the move from Hopper to Blackwell is a step change in both raw capacity and agents per megawatt; and real-world optimizations are critical, because different kernel and serving-configuration choices "moved the results significantly."¹

NVIDIA's own blog puts a number on it. It reports that GB300 NVL72 runs up to 20x more agents per megawatt than an NVIDIA HGX H200 system, a comparison it says holds at both the 20 and 60 tokens-per-second service levels.⁷ Its companion technical post publishes the underlying absolutes: 61.4K concurrent agents per megawatt for GB300 NVL72 against 2.6K for H200, and 57.5 concurrent agents per GPU against 1.4.⁸ Both posts are NVIDIA's framing of Artificial Analysis's data, published the same day, and worth reading as such. NVIDIA attributes the margin to rack-scale codesign — GB300 NVL72 wires 72 GPUs into one NVLink fabric — plus WideEP and DeepEP spreading MoE expert execution across the full rack, and fused-MoE with MXFP4/MXFP8 kernels overlapping NVLink communication with tensor-core compute.⁸ Its consumer blog adds that TensorRT LLM separates the processing of inputs from the generation of outputs so each can be optimized independently.⁷

Read those absolutes carefully, because the normalisation changes the multiplier: per megawatt the gap works out at roughly 24x, per GPU at roughly 41x, while NVIDIA's headline settles on the more conservative "up to 20x." That is not necessarily an inconsistency — NVIDIA notes the per-MW and per-GPU table reflects a single SLO configuration, whereas the 20x charts are captioned at the 20 and 60 tokens-per-second tiers, so the figures may simply come from different operating points.⁷⁸ It is a reason to quote the metric and the tier, never the multiplier alone.

There is also a documentation conflict worth knowing if you cite tier numbers. NVIDIA's technical post tabulates the DeepSeek V4 Pro tiers as 30, 100 and 300 tokens/s, while Artificial Analysis — the benchmark's owner — publishes 20, 60 and 180 in its methodology.⁵⁸ The likely explanation is an inconsistency on Artificial Analysis's own page: the same figure that lists the 20/60/180 tiers also carries an interactive selector offering "speed-optimized / balanced / capacity-optimized" targets of 300, 100 and 30 tokens/s.¹ Those are the numbers NVIDIA's table carries. NVIDIA's own chart caption, meanwhile, references 20 and 60 — matching the methodology. The methodology table is authoritative and is what this article uses throughout.

Two things are missing from that framing, and Artificial Analysis supplies both. First, the B300 and GB300 configurations were submitted by NVIDIA, while the H200 and MI355X configurations were built in-house by Artificial Analysis — vendor-tuned results measured against results tuned by the scorekeeper.¹ Second, AA says so explicitly, warning that headroom "may be greater for the MI355X and H200" precisely for that reason, and that with kernel optimization for DeepSeek V4 Pro on AMD systems still in "relative infancy," it expects "significant improvements in AMD performance in the near-term."¹ A 20x gap measured under those conditions is a real result and a moving one.

What MLPerf's Agentic Inference Benchmark Adds

MLCommons' Agentic Inference benchmark, published July 8, 2026, is described in its own words as "a new multi-turn benchmark for measuring LLM serving systems under growing context, and closed-loop agent workflows."² It lives inside the MLPerf Endpoints framework, and its distinguishing contribution is the workload, not the harness.

Where AA-AgentPerf is entirely coding trajectories, MLPerf's benchmark combines two domains that, in MLCommons' framing, "exercise different infrastructure bottlenecks": 613 multi-turn trajectories — 113 agentic coding plus 500 agentic workflow — evaluated on Kimi K2.6 (1T parameters, 32B active) and Qwen3.6-35B-A3B (35B, 3B active) at a context length of 262,144 tokens.² The split is itself informative: workflow trajectories outnumber coding ones by more than four to one, but the coding traces are far deeper.

• Agentic workflow, contributed by Workato. Every trajectory opens with a large shared system prompt — many tool definitions and business rules — drawn from "enterprise customer-support and orchestration scenarios," then usually resolves in fewer turns than a coding trace. MLCommons lists its primary stress as "large shared prompt; prefix overlap; prefix-cache efficiency."²
• Agentic coding, drawn from the DeepSWE dataset by DataCurve, in which an agent investigates a repository through a series of bash commands — searching files, running tests, observing errors, editing and iterating. These traces are deep, and MLCommons lists their stress as "deep trajectories; growing context; KV-cache capacity."²

Scoring is a Pareto curve with aggregate serving throughput on one axis and per-user output speed on the other, so that a system is measured, in MLCommons' words, on "whether serving systems can make progress for real multi-turn users while still using hardware efficiently."² Accuracy is enforced as a three-level hierarchy — an output-length check, an inline check against ground truth during the performance run itself, and a standalone evaluation on 200 tasks from SWE-bench Verified — explicitly so the curve "measures usable agentic progress rather than responses that are shorter, lower quality, or easier to serve."² Workato's summary of the same mechanism is blunter: it stops a system winning "by quietly truncating answers."³ Workato says the traces and reference implementation are released under an open licence; MLCommons' own post frames access more narrowly, inviting readers to join MLCommons to obtain the reference implementation and submit results.²³ Either way, running it is straightforward: stand up an OpenAI-compatible endpoint on vLLM, SGLang, TensorRT-LLM or similar, then point the MLPerf Endpoints client at it.²

One number to treat with care: the published trajectory and turn counts do not fully reconcile. MLCommons' prose says the 613 trajectories contain 30,335 client-issued turns, but its own dataset table lists 15,981 coding turns and 4,316 workflow turns, which sum to roughly 20,300.² Workato's write-up, meanwhile, headlines 990 trajectories rather than 613 — and the page currently serves two versions of that paragraph, one saying "more than 900" and one saying "990."³ MLCommons' 613 breakdown is the figure to use, since it is itemised and arithmetically self-consistent; treat the turn totals as unsettled.

There is also a second, distinct announcement, published a day later on July 9: a call for submissions for an Edge Agentic Inference benchmark in MLPerf Inference v6.1, aimed at multi-turn tool-calling LLMs on a single edge accelerator. It runs Qwen3.6-27B as a Q4_K_M GGUF and scores accuracy against the Gorilla eval set (BFCL v4, single-turn). Submissions are open to MLCommons members with a deadline of July 31, 2026.⁶ For context, MLPerf Inference v6.0 — released April 1, 2026, with five of eleven datacenter tests new or updated — introduced a new open-weight LLM benchmark based on GPT-OSS 120B and an expanded DeepSeek-R1 reasoning test.⁹ The multi-turn agentic workload is what July adds.

AA-AgentPerf vs MLPerf Agentic Inference, Side by Side

The two benchmarks are complements, not rivals, and the split falls along a clean line: AA-AgentPerf asks which box to buy, MLPerf asks which serving stack to run.

Note the asymmetry in the last two rows. AA-AgentPerf is the one measuring watts, and it already has numbers on the board. MLPerf has the broader, openly published workload — including an enterprise-process trace that Workato argues the open research community previously lacked entirely³ — but no agentic results yet. On timing, MLCommons has given two signals of differing precision: its July 9 edge post refers to the datacenter agentic benchmark as "coming up in September," while its more recent July 28 MLPerf Endpoints v0.7 release says only that agentic workloads arrive with Endpoints v1.0 "later this year."⁶¹⁰

The Energy Number Everyone Quoted, and the One That Matters

The demand-side case for all of this comes from KAIST. A team led by Professor Minsoo Rhu, with PhD student Jiin Kim as first author, measured agent workloads against ordinary single-turn queries and reported that agents can consume up to 136.5 times more energy per query than conventional single-turn inference.⁴

Three precision notes, because that figure circulated widely in July without any of them.

First, the paper is not new. A preprint has been public since June 2025, the work was presented in February 2026 at the 32nd IEEE HPCA in Sydney, and KAIST's press release — the thing that actually made it go viral — is dated July 4, 2026.⁴ The research predates its own news cycle by more than a year.

Second, 136.5x is the top of a narrow, specific range, not a property of agents in general. It describes one configuration: the Reflexion agent framework running on Llama 70B, at 348.41 Wh per query. The baseline it is measured against is not a commercial chatbot but single-turn inference on that same 70B model, at 2.55 Wh. Swap the framework and the number falls hard — LATS on the identical model comes in at 62.1x (158.48 Wh). Across the paper's four configurations the multiplier runs from 62.1x to 136.5x.⁴¹¹ As Forbes put it, quoting 136.5x as the cost of agentic AI "is like quoting the fuel bill of the thirstiest car on the lot as the price of driving."¹¹

Third, the figure that survives scrutiny is the boring one: on tool-heavy tasks, GPUs sat idle for up to 54.5% of total execution time, drawing power while waiting for an external tool to return.⁴ Neither benchmark cites this paper — Artificial Analysis credits "a year of work with inference providers, AI accelerator companies, developers, and enterprise buyers" for its design¹ — but the finding describes precisely the economics both are now built to expose. If more than half of an agent's wall-clock time can be the most expensive hardware in the building doing nothing, then peak tokens per second was never measuring the thing that costs money.

KAIST's widest-travelled figure deserves the least weight. The paper projects that at 13.7 billion agent requests per day — the volume of current Google search traffic — data-centre demand would reach roughly 198.9 GW, which the authors note is almost half the US grid's average load of about 477 GW, and far beyond the few-gigawatt scale of AI data centres now under construction.⁴ It is explicitly a scenario rather than a forecast, and a tightly conditioned one: it assumes the Reflexion framework on a specific benchmark, then scales small-sample per-task energy to Google's search volume as a stand-in for "very large."⁴¹¹ Treat it as an illustration of shape, not magnitude.

For a calibrated number, the IEA's April 2026 Key Questions on Energy and AI puts global data-centre electricity consumption on track to roughly double from 485 TWh in 2025 to 950 TWh in 2030 in its central case, around 3% of global electricity demand.¹² That is an update rather than a reversal — the IEA says the trajectory "remains close to" the one in its April 2025 Energy and AI report, which projected around 945 TWh by 2030 from a 2024 base of 415 TWh.¹² Either way, the projection is built on query volume growing. It does not price in the average query itself getting heavier as the industry shifts from answering to acting, which is precisely the gap the KAIST work exposes.

What the Leaderboards Don't Tell You

Every claim above rests on methodology choices worth knowing before you cite a number in a procurement deck.

Tool calls are simulated, not executed. AA-AgentPerf inserts a fixed per-message delay sampled from a distribution of real tool-call durations, partitioned by the tool being called — under 0.1s to 5s, median around one second.⁵ NVIDIA states the consequence directly: tool calls "are not executed but simulated using representative CPU processing time, so differences in results reflect accelerated computing performance only," with the same CPU baseline applied across every system tested.⁷⁸ That is the correct choice for a hardware benchmark and a real limit on how far the numbers generalise to your stack. Actual CPU tool-execution performance is on Artificial Analysis's roadmap, not in the current results.¹

Production optimizations are switched on. AA-AgentPerf permits KV cache reuse, speculative decoding, and disaggregated prefill/decode — a deliberate break from benchmarks that disable them — with accuracy verification to stop an optimization buying capacity by degrading output. It also injects dynamically generated prefixes each phase specifically to break prefix caching between phases, so cross-phase reuse can't inflate a result.¹⁵

Serving configuration is part of the score. The configurations are not incidental detail. At launch the GB300 results ran a disaggregated prefill/decode topology, the H200 used EAGLE speculative decoding, and the MI355X ran a radix cache with no speculative decoding at all.¹ Nor is it one configuration per box: Artificial Analysis's configuration browser carries several per system, including multiple GB300 NVL72 expert-parallel setups that differ in how many of the rack's 72 GPUs they use, each entry individually dated.⁵ Comparing headline numbers means comparing those configurations too — and AA notes serving configurations for DeepSeek V4 Pro have been improving "on a near-daily basis," which is exactly why the browser's per-entry dates matter.¹

Vendors get a look first. AA's methodology states that "providers may review their results for factual accuracy before publication."⁵ Combined with vendor-submitted configurations, that is a reasonable process for a benchmark that depends on vendor cooperation, and a disclosure readers should carry alongside the ranking.

The leaderboard moves under you. AA-AgentPerf is a live benchmark with rolling publication, and it has already outrun its own launch article. The June 12 results covered DeepSeek V4 Pro alone, with gpt-oss-120b listed as coming soon; by late July the board had added gpt-oss-120b configurations and a third model, Kimi K3 — released mid-July and scored on a new full-Pareto, non-SLO setup that Artificial Analysis describes as a trial.¹⁵ Anything you cite from this benchmark needs a date attached to it. Artificial Analysis frames the whole exercise the same way: "a snapshot of a fast-moving frontier."¹

MLPerf's enterprise traces are synthetic. MLCommons states plainly of the workflow domain that "the data are synthetic traces modeled on Workato's production experience orchestrating customer support agents for enterprise customers."² The coding domain is sourced differently, from DataCurve's DeepSWE dataset, and MLCommons describes the workload generally as using "real collected traces" — a characterisation that sits awkwardly beside the synthetic-workflow sentence in the same post, so treat provenance as domain-specific rather than uniform.² Either way the synthetic choice was deliberate, and it is what made publishing an enterprise workload possible at all: as Workato puts it, the scarce ingredient was a representative workflow, not real traffic — "Traffic can be simulated, but a realistic workflow has to come from somewhere that actually runs enterprise agents," with "no private data ever leaving the building."³ That is the right trade-off for an open benchmark, and it means the workflow domain reproduces the shape of enterprise agent load — the giant shared prefix, the shallow turn count — rather than any real organisation's traffic mix.

MLPerf's accuracy thresholds are not final. The table of accuracy values that gates every point on the Pareto curve is labelled, in MLCommons' own caption, "Tentative accuracy values, subject to change."² Any cross-system comparison published before those settle is provisional.

The "first" claims, and who is actually making them

Watch the superlatives here, because three organisations claimed a first and each attached a different limit — while the fourth, notably, claimed nothing at all.

Artificial Analysis says AA-AgentPerf is "the first agentic inference benchmark" — scoped to serving.¹ Workato claims "the first MLPerf Agentic Benchmark measuring infrastructure performance on the complexity of real enterprise business processes" — scoped twice over.³ KAIST describes the first quantitative analysis of agent compute cost.⁴ Those hold inside their qualifiers.

Worth noting: that MLPerf "first" is Workato's claim, not MLCommons'. The word "first" does not appear anywhere in MLCommons' own post about the benchmark, which describes agentic inference only as "one of the fastest-growing ways large language models are used in production."² The standards body was the more careful party here; its commercial partner supplied the superlative.

NVIDIA's is the loosest: "the industry's first agentic AI benchmark."⁷ Strip the word inference out and the claim stops surviving contact with the literature — agentic capability benchmarks predate it by years, AgentBoard among them, published in January 2024.¹³ NVIDIA's own technical post is more careful, calling AA-AgentPerf "the industry's first multi-vendor open benchmarks profiling trajectories that are representative of real-world AI agent coding tasks" — a qualifier stack that survives scrutiny where the consumer-blog version does not.⁸ The novelty in 2026 is measuring the infrastructure underneath agents, not measuring agents.

What This Changes If You're Buying Agent Compute

The practical translation is narrow. If your 2026 roadmap is agentic and your compute budget was priced on single-turn inference, the unit is wrong before the arithmetic starts.

Three questions follow directly from the two benchmarks and the KAIST result:

1. Ask for capacity at a service level, not peak throughput. "How many concurrent agents at 60 tokens per second and a 5-second P95 TTFT" is answerable and comparable. "Tokens per second" is neither, because concurrency can always be raised until every agent crawls.
2. Ask what share of a task's cost is idle hardware. The 54.5% idle finding makes this concrete, and it is workload-specific — your tool latency, not a benchmark's simulated median.
3. Ask which configuration produced the number, and whether the vendor or a neutral party tuned it. AA-AgentPerf publishes this per result; most vendor decks do not.

Which prefix pattern you have also decides which benchmark to read. If your agents are coding agents with deep, growing context, AA-AgentPerf's trajectories resemble your load. If they are enterprise workflow agents that open with a huge shared tool-and-policy prompt and finish in a few turns, MLPerf's enterprise domain is the closer analogue — and prefix reuse, not decode speed, is your bottleneck.

This is the infrastructure-side complement to a cost problem we have covered from the application side: as we noted in AI agent token costs in 2026, cheap tokens do not produce cheap tasks, because an agent re-sends its growing context on every call. The benchmarks now measure the same phenomenon one layer down, in watts and concurrent sessions rather than dollars per million tokens. That same context growth is why tool-count degradation bites so early, and it is the workload the managed agent runtimes from the major clouds are being built to absorb.

The Bottom Line

Two benchmark publishers opened a new unit of measurement inside four weeks, and they did it for the same reason: the workload stopped being a prompt and became a trajectory. Agents per megawatt is a better question than tokens per second, and AA-AgentPerf has real results behind it — measured power, disclosed configurations, a held-out test set. MLPerf's benchmark brings the workload Workato argues was missing entirely, an open enterprise-process trace, and it will be worth more than AA-AgentPerf to anyone choosing a serving stack rather than a chip.

Neither is finished. One has results but a narrow domain and vendor-tuned configurations on the winning side; the other has the better workload, tentative accuracy thresholds, and no results yet. The honest summary as of July 30, 2026 is that the industry now agrees on the right question and is still assembling the answer. If you are buying agent compute in the meantime, the useful move is not to pick a leaderboard. It is to stop accepting a per-query number for a workload that no longer runs one query.