Route AI agent requests to the right model to balance cost and performance
If you want to handle complex tasks with expensive models and simple tasks with cheaper ones while managing overall costs, this tool is for you. Switchyard is a routing engine that determines which model an AI agent uses for each request. It maintains compatibility with OpenAI and Anthropic APIs while allowing flexible adjustment of model selection, so you can directly control the balance between performance and cost.
Models change based on request flow
Switchyard selects models by analyzing the nature of the request or recent tool usage history while the agent performs tasks. It dynamically changes routing, starting with an efficient model and switching to a more powerful one as the task becomes complex. During this process, existing OpenAI or Anthropic API specifications are maintained, allowing adoption without significantly altering existing application structures. It also provides a tuning loop to compare and evaluate the agent's overall success rate, cost, and latency against single-model usage scenarios.
Four routing strategies
You can choose one of four algorithms depending on your use case.
- Auto A fixed preset as of v0.3.0 that applies execution-stage-based routing and efficiency-first selection by default.
- Task Uses an LLM classifier to select a model based on the difficulty of the current task.
- Execution Analyzes signals in real time during the agent's tool usage process to adjust the model.
- Composite Combines the classifier and execution signals to perform more precise model selection.
Adoption methods and support scope
Switchyard can be used by directly embedding the Rust core, libsy, or through existing gateways.
- Gateway integration Connects as a plugin to existing infrastructure such as OpenRouter, LiteLLM, and NeMo Relay.
- Direct embedding Includes the Rust-based
libsydirectly in your service or agent harness to control routing logic. - Local evaluation Tests and verifies routing performance with a specific model pool using a standalone server.
Version stability and considerations
Switchyard is currently a pre-1.0 version, and the API, configuration, and routing behavior may change with each release. Therefore, pinning a specific version is essential when integrating. Stability varies by component: switchyard-libsy is in Beta, while switchyard-llm-client and switchyard-runner are in Alpha. switchyard-server is for demos and local evaluation only and is not supported in production environments. It is released under the Apache 2.0 license, with copyright held by NVIDIA Corporation. Before adoption, you should select a routing algorithm suitable for your workload and verify actual cost savings compared to a single model through benchmarks.
