NVIDIA released NeMo Switchyard on August 11 as an open-source router for AI model traffic. In LangChain's first 145-task evaluation, the router sent 93% of model calls to a smaller model and cut the cost of the frontier-only baseline by 74%. Accuracy also fell by six points.
That last number matters.
Model routing is being sold as a way to make agents cheaper without changing the application. Switchyard makes that architecture easier to build, but it cannot make the quality trade disappear. The useful question is not whether routing saves tokens. It is whether your agent can identify the turns where saving tokens is safe.
What is NVIDIA NeMo Switchyard?
NeMo Switchyard is a Rust proxy and library that sits between an AI application and its model providers. The application sends an OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages request. Switchyard normalizes that request, chooses a configured model, calls the provider, and translates the response back into the format the application expects.
The client sees one model endpoint. Behind it, the selected target could be a hosted provider, an NVIDIA NIM deployment, vLLM, or an OpenAI-compatible local server.
This separation solves a real engineering problem. Without it, a team that moves one workflow from an Anthropic model to an OpenAI-compatible model may need to change request schemas, streaming handling, error parsing, and tool-call responses. With Switchyard, those provider details live behind the proxy.
It also means the router can change its choice for different requests without forcing the agent to understand every backend.
How Switchyard decides which model receives a turn
Switchyard currently documents four routing strategies.
The LLM classifier asks a smaller judge model to choose between configured targets. It can look at request content and send routine work to a lower-cost model while reserving a stronger model for harder requests.
The stage router reads signals already produced by an agent. Repeated errors, long exploration, or unproductive tool activity can push the next turn toward the stronger target. Once tests pass and the work becomes mechanical, the route can move back toward the efficient target. Because it relies on workflow signals, it does not require another model call for every decision.
The escalation router starts a task with the cheaper model. A judge monitors progress and promotes the session after repeated poor turns. Promotion is sticky, so the task does not bounce between models after the router has decided that more capability is needed.
Random routing is the simple option. It assigns traffic according to fixed weights and is useful for an A/B test or a baseline. It does not inspect the prompt.
NVIDIA is also researching a prefill router that reads internal model activations. That work is not the reason to adopt Switchyard now. It is research-stage functionality and requires access to signals that closed model APIs do not expose.
A minimal setup
The quickest evaluation path uses the published CLI wrapper. Install uv, then install the Switchyard tool:
curl -LsSf https://astral.sh/uv/install.sh | sh
source "$HOME/.local/bin/env"
uv tool install --python 3.10 "nemo-switchyard[cli]"With an OpenRouter key available, Switchyard can launch a supported coding agent through its packaged route:
export OPENROUTER_API_KEY="your-key"
switchyard launch codex --model switchyardThat is useful for testing. A service that handles product traffic should use an explicit configuration rather than depending on a packaged model pool.
The standalone server reads a TOML file that defines provider clients, model targets, and a route. A trimmed example looks like this:
schema_version = 1
[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"
[targets.weak]
id = "openai/gpt-4o-mini"
llm_client = "openrouter"
[targets.strong]
id = "openai/gpt-4o"
llm_client = "openrouter"
[routes.smart]
id = "switchyard"
type = "llm_classifier"
mode = "capability"
classifier_target = "weak"
strong_target = "strong"
weak_target = "weak"
base_threshold = 0.5
Install and validate the Rust server before exposing it to a client:
cargo install --locked switchyard-server
switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml --host 127.0.0.1 --port 4000
curl http://localhost:4000/healthThe application can then point its base URL at the proxy. Keep the first test local. A router with access to several provider keys becomes a sensitive control point, and the project does not claim production readiness yet.
What the 74% cost reduction actually means
LangChain tested Switchyard on 145 multi-step agent tasks averaging 6.3 model calls each. Its routed arm combined NVIDIA Nemotron 3.5 Lightning with Claude Opus 4.8 and used another model as the judge.
Opus alone reached 86% accuracy at $11.45 per run. The routed setup reached 80% at $3.00. Nemotron alone reached 77.7% at $0.72.
The routed result was much cheaper than using Opus for every turn. It was also more than four times the cost of using Nemotron alone, while its 2.3-point accuracy improvement over Nemotron was smaller than the benchmark's normal run-to-run variation.
That is a strong reason to test a cheap model by itself before building a router.
Routing makes sense when a smaller model can handle most turns, the price gap between models is wide, and the hard turns are costly enough to get wrong. It is not automatically the cheapest architecture. The judge is another model call, and LangChain found that judging consumed 21.2% of routed spend.
The benchmark also showed variable bills. Frontier traffic ranged from 4.1% to 9.1% across five runs, causing routed cost to move from $2.16 to $3.61. Nothing in the workload changed. The router changed its decisions.
A forecast should therefore use a range. Quoting one best-case percentage to a finance team would be misleading.
The production problems the demo does not solve
Switchyard's own repository labels the software pre-alpha and explicitly says it is not for production use. APIs and algorithms may change before version 1.0. That warning should control the rollout plan.
Cross-model caching is another issue. Prompt caches are generally provider-specific. If a conversation moves to a different provider, the second model may need the full context again at the normal input price. A routing policy that looks cheap per output token can lose that advantage through repeated uncached input.
Behaviour can also shift mid-session. Models do not interpret tools, safety instructions, or ambiguous user intent identically. Session affinity reduces arbitrary switching, but a promotion from a small model to a frontier model still creates a handoff. Test whether the stronger model understands tool results and decisions produced earlier by the smaller one.
Observability needs verification too. Switchyard exposes Prometheus metrics for requests, errors, latency, tokens, and routing overhead. An early GitHub issue showed streaming responses recording zero token usage in routing statistics, which made cost estimates wrong for agent frameworks that stream by default. The issue has since been closed, but it is exactly the kind of failure an evaluation should look for.
When we would use it
We would test Switchyard when an agent produces many multi-turn calls, most of those turns are routine, and the frontier model is responsible for a large share of the bill. Coding agents and incident investigation workflows are plausible candidates because difficulty changes as the task progresses.
We would not add it to a short request-response feature where one inexpensive model already meets the quality target. We would also avoid it for latency-sensitive paths if the chosen strategy adds a judge call to every turn.
Before rollout, build an evaluation set from real traces. Replay the same tasks through the expensive model, the cheap model, and the routed pool. Measure completed-task cost rather than token cost alone. Record p95 latency and every case where a provider handoff changes tool behaviour.
The router earns its place only if it improves the cost-quality frontier for your workload. A complicated control plane is not a win when one smaller model already does the job.
NeMo Switchyard is interesting because it turns model choice into an infrastructure policy instead of hardcoded application logic. That is a useful direction. Today, though, it belongs in a measured staging experiment, not between customers and a critical production agent.
Axentia builds AI agents and generative AI applications where model selection, evaluation, and production controls have to work as one system. If your agent costs are growing and you want to test routing against real traces before changing the architecture, book a call with us.