NVIDIA NeMo Switchyard: Agent Model Routing 2026

NeMo Switchyard is NVIDIA's open-source library for sending each step of an AI agent's workflow to a different model. NVIDIA announced it on August 11, 2026 alongside Nemotron 3.5 Lightning, though the project shipped its first release in June. LangChain's benchmark cut cost 74% against a frontier-only baseline, and lost six points of accuracy doing it.

TL;DR

NeMo Switchyard is an Apache 2.0 routing library that sits between an agent and its model providers, choosing which configured model handles each turn.¹ NVIDIA announced it on August 11, 2026 alongside Nemotron 3.5 Lightning, a 30B mixture-of-experts model with 3B active parameters aimed at the execution layer of long-running agents.²³

The launch numbers are real but narrower than the headlines suggest:

• LangChain measured 74% lower cost than running Claude Opus 4.8 alone — at 80.0% accuracy versus 86.0%.⁴
• The cheap model on its own scored 77.7% for a quarter of the routed price. Routing beat it by 2.3 points, which LangChain notes is inside its 2.7-point run-to-run noise.⁴
• In the modes that use a judge model, that judge consumed 21.2% of routed spend and added roughly 700ms per turn.⁴
• Switchyard is pre-alpha. The repository says it is not for production use, and PyPI classifies the package as Development Status "3 - Alpha."¹⁵

This is not a brand-new project. Version 0.1.0 shipped to PyPI on June 30, 2026; version 0.2.0 — a rewrite of the routing engine from Python into Rust — landed on August 10, 2026, a day before NVIDIA's announcement.⁵⁶

What You'll Learn

• What NeMo Switchyard actually is, how old it really is, and how per-step routing differs from a model gateway
• The four routing strategies it documents, and which one each partner benchmark actually used
• What LangChain's 145-task benchmark found — including the number that didn't make the headlines
• Why the judge model is the hidden line item in any routing budget, and how unstable it can be
• How to decide whether routing can pay off on your own workload

What Is NeMo Switchyard?

NeMo Switchyard is an open-source Rust proxy and library that routes LLM traffic across models and providers while translating between API formats.¹ Instead of a developer picking one model for an entire agent, the router evaluates each request and sends it to whichever configured model fits.

The Rust part is recent. Version 0.1.0 was a Python package; 0.2.0 rewrote the routing engine in Rust and made switchyard-server — published to crates.io on August 10, 2026 — the primary deployment path, with Python retained mainly as a distribution and integration surface.⁷⁶

The technical framing NVIDIA uses is a "system of models": a frontier reasoning model plans and orchestrates, while smaller specialized models handle the high-volume execution steps.⁸

That framing exists because of where agent tokens actually go. Long-running agents spend most of their time on tool calls, result validation, and subagent delegation — steps that rarely need a frontier model but get one anyway.³

Switchyard accepts OpenAI Chat, Anthropic Messages, and OpenAI Responses requests, then forwards each one in the target backend's native format.¹ A coding agent keeps speaking its own API while the request is served by vLLM, NVIDIA NIM, Ollama, or any OpenAI-compatible endpoint.

The repo ships launchers for Claude Code, Codex CLI, and OpenClaw, plus a standalone server and an embeddable Rust crate.¹ It is licensed Apache 2.0.

Routing is not new — running it locally is the change

Model routing predates this launch by years. LMSYS released RouteLLM, an Apache-2.0 routing framework, on July 1, 2024.⁹ LiteLLM, OpenRouter, and Portkey all offer gateway-level routing today.

Jason Andersen, VP and principal analyst at Moor Insights & Strategy, put the distinction plainly: "Model routing itself isn't new—Openrouter, AWS Bedrock Agent Core, and Microsoft Foundry already offer cloud-based versions. What matters is that NVIDIA is making this open source and available to run locally."¹⁰

His verdict on the release was equally direct: "Switchyard isn't revolutionary. But it is a proof point."¹⁰

The Four Routing Strategies

Switchyard documents four routing strategies, and they trade latency against accuracy very differently. Note that escalation is not its own route type — it is a mode of the LLM classifier.¹

Strategy	How it decides	Route type	Extra model call?
LLM classifier	A small judge model picks the target and keeps session affinity	llm_classifier	Yes
Stage router	Reads tool activity, errors, and token counts already in the conversation	stage_router	No
Escalation router	Starts on the weak tier; a judge promotes the task after repeated bad turns	llm_classifier with mode = "escalation"	Yes
Random	Fixed traffic split for A/B tests and baselines	random	No

A fifth option, passthrough, registers one target under one model ID with no routing decision at all.¹

NVIDIA also describes a tunable prefill router that reads a model's internal activation patterns to predict which candidate is likely to answer correctly.⁸ It is not shipped: LangChain calls it "research stage rather than production ready," and a practitioner reviewing the 0.2.0 release found it "only conceptually introduced at the time of writing and not yet included in the release."⁴⁶

The distinction matters when reading the benchmarks, because the partners did not all use the same strategy. LangChain's numbers come from the escalation mode, which pays for a judge on every weak-tier turn, while NVIDIA describes Cognition as implementing the staged router.⁴⁸

What LangChain Actually Measured

LangChain ran its Deep Agents evaluation suite — 145 multi-step agentic tasks averaging 6.3 model calls each — through Switchyard in escalation mode, routing between Nemotron 3.5 Lightning and Claude Opus 4.8.⁴

The scenarios were drawn from τ²-bench airline, the Berkeley Function Calling Leaderboard, FRAMES, and Nexus, covering customer support under policy constraints, on-call incident investigation, and multi-step workflow automation.⁴

Arm	Accuracy	Cost per run	Cost per completed task
Opus 4.8 alone	86.0%	$11.45	$0.092
Routed (Opus 4.8 + Nemotron 3.5 Lightning)	80.0%	$3.00	$0.026
Nemotron 3.5 Lightning alone	77.7%	$0.72	$0.006

Table: LangChain Deep Agents suite, 145 tasks. Source: LangChain, August 11, 2026.⁴

Only 7% of calls went to the frontier model — and those calls carried 68.4% of the bill.⁴ Nemotron handled the other 93% for 10.4% of spend. Those call counts exclude the judge, which fires once per weak-tier turn.⁴

That is the headline finding, and it holds up.

The number that didn't make the headlines

Look at the third row again. Nemotron 3.5 Lightning alone scored 77.7% for $0.72. The routed configuration scored 80.0% for $3.00.

Routing bought 2.3 accuracy points for 4.2x the cost. LangChain measured run-to-run variance of about 2.7 points and concluded, in its own words, that "we cannot say routing beat the cheap model here."⁴

LangChain also flagged why: the suite is saturated. Only about 8 points separate a 30B open model from a frontier model on these tasks, which "gives routing less room to prove its value than a harder workload would."⁴

To be fair to both parties, LangChain published this caveat itself, prominently — it is the second of three key takeaways at the top of the post.⁴ The gap is in how the figure travelled. eWeek's launch coverage, for example, cites NVIDIA's cost reduction and names LangChain among the partners that tested the tools, without mentioning an accuracy tradeoff.¹¹

eWeek's reporter did add the right caveat about the launch numbers generally: NVIDIA's "performance and cost figures are largely based on its own benchmarks or partner testing, so buyers should validate savings and accuracy against their own workloads before replacing existing models."¹¹

The judge is a line item, not a rounding error

Escalation routing needs a judge model to read each completed turn and vote on whether the agent is on track. LangChain used Gemini 3.1 Flash Lite in that slot.⁴

That judge accounted for 21.2% of routed spend — the second-largest line item after the frontier model, and roughly a third of what Opus cost.⁴ It runs on every turn until a task escalates, and unlike the frontier model it gets no benefit from prompt caching.

It also costs latency: roughly 700ms per turn, against effectively zero for the heuristic stage router.⁴

And the judge is unstable in a way the cost tables do not capture. Hiroshi Morishige of Classmethod, who has run Switchyard in internal validation since v0.1.0, reports that swapping the classifier model for a minor update version flipped the same 50 conversations from 39 weak-tier routings to one — with no errors raised and confidence still high.⁶

Across 1,392 judgments spanning eight models and two prompts, he found the weak-judgment rate scattered anywhere from 0% to 100% on identical inputs. His conclusion: what the judge reads, and how it reads it, is a far larger variable than the threshold you tune.⁶

LangChain distilled this into a break-even formula worth stealing:

minimum offload = judge cost / (expensive cost - cheap cost)

For their pairing, a $0.64 judge against a $10.73 price gap meant they needed to offload 5.9% of turns to break even. They offloaded 93%.⁴ If your two candidate models are close in price, the formula demands more than 100% offload — and routing cannot pay unless you self-host the cheap model.

Routing Widens Your Cost Range

Across five runs, frontier traffic ranged from 4.1% to 9.1%, with a mean of 6.9%.⁴ Nothing changed between those runs except which turns the router chose to escalate.

The bill still moved 67%, from $2.16 to $3.61.⁴ An Opus call cost $0.0324 against Nemotron's $0.00037 — about 87x — so small changes in escalation rate swing the invoice hard.

This is the trade nobody puts on a launch slide: a router lowers your average spend and widens the range around it. Budget against the top of that range, not the mean.

The same tension shows up in how AWS and Anthropic approached agent spend enforcement with per-session caps — a ceiling is useful precisely because per-run cost is no longer predictable.

The Model Underneath: Nemotron 3.5 Lightning

Nemotron 3.5 Lightning is an open 30B mixture-of-experts model with 3B active parameters, released on August 11, 2026.³¹² NVIDIA calls it the smallest member of the Nemotron 3 family;³ it follows Nemotron 3 Nano in the release order.²

NVIDIA reports up to 4x the output speed of similar-sized models, and 86% accuracy on PinchBench while completing 10,000 tasks 30% faster than Qwen3.6 35B at similar accuracy.³

Weights, training data, and recipes are released under the OpenMDW-1.1 license,³ the Linux Foundation model-licensing framework NVIDIA committed in May 2026 to adopting across future releases of its Cosmos, Isaac GR00T, Ising, and Nemotron model families.¹³ The model is free on OpenRouter with a 1M-token context window and 65,536-token maximum output.¹²

For readers tracking the open-weight agent tier, this lands in the same week as Meta's Muse Glimmer — and our benchmark breakdown of where Muse Glimmer leads and trails on agentic tasks covers the other side of that comparison.

The Maturity Gap

Here is the part that deserves more attention than it got.

NVIDIA's announcement published partner cost reductions ranging from 27% to 74%. Three days later, the Switchyard repository still carried this notice: "Switchyard is pre-alpha software that is evolving rapidly. The API and algorithms are expected to change significantly before we reach v1.0." Directly beneath it: "Experimental software. Not for production use."¹

That is corroborated elsewhere. PyPI lists the package under Development Status "3 - Alpha,"⁵ and both third-party reviewers who hands-on tested the 0.2.0 release flag the same warning.¹⁴⁶

The Wavect review is blunt about the consequence: Switchyard "is explicitly pre-alpha, the route still needs calibration, and a cheaper model is only cheaper when the completed task remains acceptable."¹⁴

Adoption is early by the numbers. As of August 14, 2026 the repository shows 342 stars, 51 forks, and 213 commits, with 35 open issues and 39 open pull requests,¹ while the switchyard-server crate had recorded 321 total downloads since its August 10 publication.⁷

The LangChain middleware for in-process routing is in a similar state — "experimental and not published as a package yet," requiring you to clone both Switchyard and langchain-nvidia and install them locally.⁴

None of this makes the benchmarks wrong. It means the thing that produced them is not the thing you can deploy on Monday.

Andersen's caution lands in the same place: "Model routing is not plug-and-play. Like a network router, getting the best outcomes requires skilled operators and careful configuration." He adds that existing agents "will likely need re-engineering to benefit from routing," and that if the upstream architecture is wrong, "the router becomes overhead rather than optimization."¹⁰

Who Else Reported Numbers

NVIDIA's launch post lists partner results across the ecosystem. These are vendor-reported and not independently verified, so treat them as directional.²

Partner	Reported result
Ramp	58% lower cost, 33% shorter runtime on Ramp SWE-Bench
Cognition	50.6% on FrontierCode Main at $3.11 mean cost — within 2.8 points of Opus 5 at ~28% lower cost8
Classmethod	27% cost reduction in internal testing
Cadence	9.9% efficiency improvement in a formal verification workflow

Table: Partner-reported Switchyard results. Source: NVIDIA, August 11, 2026.²

Cognition's number is the most useful of these because NVIDIA published the configuration: staged routing between Opus 5 and Kimi K2.7, deployed to NVIDIA internal users.⁸ Most of the other partners did not disclose which models they routed between, which limits how far you can read into their percentages.

Classmethod's engineer published his side of the 27% figure, and his framing is worth borrowing. On his own curated task set he measured a 98% cost reduction — then discarded it, because the tasks were easy enough that the cheap model scored perfectly on its own. In his words, that number "is not proof of 'reducing costs while protecting quality' — it's proof of 'routing all tasks that don't need strong to weak.'" The 27% is the figure from real traffic.⁶

One detail worth noting: Cognition's baseline was Opus 5, while LangChain's was Opus 4.8. Anthropic released Claude Opus 5 on July 24, 2026, 18 days before this launch, so the two headline results are measured against different generations of frontier model.

NVIDIA's own internal benchmark claims routing reduces task completion cost "to nearly one-third of Opus 4.8 alone."² LangChain's separately published measurement came in at roughly one-quarter ($3.00 against $11.45) on a different suite.⁴ Different benchmarks, same neighborhood — a reasonable sanity check on the direction, if not the magnitude. Both are partner or vendor tests, not third-party evaluations.

Integration partners include Kong, which delivers routing natively through Kong AI Gateway; LiteLLM, adding Switchyard as a proxy plug-in; and Nous Research, which integrated it into Hermes.²

Should You Route?

Three questions decide it, in this order.

Is your price gap wide enough? Run LangChain's formula before you build anything. If judge cost divided by the price gap exceeds your realistic offload rate, routing cannot pay — and no configuration change fixes that.⁴

Is your workload hard enough? Routing earns its keep when a cheap model genuinely fails on part of your traffic. On a saturated suite where the gap is 8 points, LangChain's own conclusion was that if minimum cost is the priority, running the cheap model alone is the better choice.⁴

Can you tolerate variance? If your finance team wants a predictable number rather than a lower average, a router works against you.

The honest case for routing is the one LangChain made: it is "the cost of not having to guess." In production you do not know in advance whether an incoming request is easy or hard, and running the cheap model on everything means accepting its answer on the hard ones too.⁴

That is a real value proposition. It is just a different one than "74% cheaper."

The Bottom Line

Switchyard is a serious piece of infrastructure and a genuine shift: per-step model routing, open source, running on your own hardware. That is worth tracking, and the partner ecosystem forming around it — Kong, LiteLLM, LangChain, Cognition, Nous Research — suggests the pattern is real.

But the launch-day arithmetic deserves reading past the headline. The largest saving in the published numbers came with a six-point accuracy cost, a judge model eating a fifth of the budget, a 67% swing in per-run spend, and a repository that says not to ship it yet.

Routing is not a discount. It is a bet that you cannot tell hard requests from easy ones in advance — and that bet has a price.

For the broader picture on where agent efficiency is being measured, see our analysis of agentic inference benchmarks and the agents-per-megawatt metric.

---

Footnotes

1. NVIDIA-NeMo/Switchyard, GitHub repository. Accessed August 14, 2026. https://github.com/NVIDIA-NeMo/Switchyard ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹²

2. NVIDIA. "NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Deliver Faster, Smarter, More Efficient Agentic AI." August 11, 2026. https://blogs.nvidia.com/blog/nemotron-lightning-switchyard-rtx-dgx/ ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

3. NVIDIA Technical Blog. "NVIDIA Nemotron 3.5 Lightning Delivers Fast, Accurate Specialized Task Execution for Long-Running Agents." August 11, 2026. https://developer.nvidia.com/blog/nvidia-nemotron-3-5-lightning-delivers-fast-accurate-specialized-task-execution-for-long-running-agents/ ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

4. LangChain. "How many of your agent's calls actually need a frontier model?" August 11, 2026. https://www.langchain.com/blog/switchyard-agent-routing-benchmark ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷ ↩¹⁸ ↩¹⁹ ↩²⁰ ↩²¹ ↩²² ↩²³ ↩²⁴ ↩²⁵ ↩²⁶

5. PyPI. "nemo-switchyard" release history and package metadata. Accessed August 14, 2026. https://pypi.org/project/nemo-switchyard/ ↩ ↩² ↩³ ↩⁴ ↩⁵

6. Hiroshi Morishige, Classmethod DevelopersIO. "I tried NeMo Switchyard v0.2.0, reborn in Rust." August 12, 2026. Classmethod is named in NVIDIA's launch post as a Switchyard partner. https://dev.classmethod.jp/en/articles/nvidia-nemo-switchyard-v020-rust-first-touch/ ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

7. crates.io. "switchyard-server" crate metadata. Accessed August 14, 2026. https://crates.io/crates/switchyard-server ↩ ↩² ↩³

8. NVIDIA Technical Blog. "Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard." August 11, 2026. https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard/ ↩ ↩² ↩³ ↩⁴ ↩⁵

9. LMSYS Org. "RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing." July 1, 2024. https://www.lmsys.org/blog/2024-07-01-routellm/ ↩

10. Jason Andersen, Moor Insights & Strategy. "NVIDIA NeMo Switchyard: Open-Sourcing Model Routing Is the Real Move." August 11, 2026. https://moorinsightsstrategy.com/field-notes/nvidia-nemo-switchyard-open-sourcing-model-routing-is-the-real-move/ ↩ ↩² ↩³ ↩⁴

11. Aminu Abdullahi, eWeek. "Nvidia Launches Nemotron 3.5 Lightning and Switchyard to Cut AI Agent Costs." August 12, 2026. https://www.eweek.com/news/nvidia-nemotron-3-5-lightning-switchyard-ai-agent-costs/ ↩ ↩²

12. OpenRouter. "NVIDIA: Nemotron 3.5 Lightning (free)." Accessed August 14, 2026. https://openrouter.ai/nvidia/nemotron-3.5-lightning:free ↩ ↩² ↩³

13. Linux Foundation. "Linux Foundation Releases OpenMDW-1.1; NVIDIA Adopts OpenMDW for Cosmos, Isaac GR00T, Ising and Nemotron, AI Model Families." May 28, 2026. https://www.linuxfoundation.org/press/linux-foundation-releases-openmdw-1.1-nvidia-adopts-openmdw-for-cosmos-isaac-gr00t-ising-and-nemotron-ai-model-families ↩

14. Kevin Riedl, Wavect. "NeMo Switchyard 0.2: Agent Model Routing Without Training?" August 11, 2026. Wavect is a software consultancy and discloses a commercial interest in the category. https://wavect.io/blog/nemo-switchyard-model-router/ ↩ ↩²