Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust
3.5 KB
Raw Blame

switchyard-libsy

Provider-neutral orchestration for multi-LLM optimization. A libsy [Algorithm] decides which model targets to call, in what order, and how to combine their results. It hands every call back to the host rather than making it, allowing it to embed in proxies, gateways, and agent runtimes without owning an HTTP stack.

Setup

[dependencies]
async-trait = "0.1"
futures = "0.3"
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.3.0" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.3.0" }
tokio = { version = "1", features = ["macros", "rt"] }

Built-in algorithms

Type Purpose
[Passthrough] Always select one configured target.
[Random] Select among any number of targets using uniform or weighted routing.
[LlmTaskClassifier] Ask a judge model to choose an efficient or capable target.
[StageRouter] Route coding-agent turns from tool and progress signals, with an optional judge fallback.

[Noop] is a test helper, not a production routing algorithm.

How it fits together

A target is a bare model id naming a routing destination. An [Algorithm] may offload routing-time classifier or judge calls to its caller: [Algorithm::run_stream] yields a [Step] stream whose [Step::CallModel] items the host serves over its own transport. The stream ends with [Step::Done] carrying a [RoutingOutcome]: the algorithm-selected model, ordered fallbacks, rewritten request, and an optional response already produced while routing. libsy makes no network calls itself — switchyard-llm-client's run is a ready-made consumer that drives the stream and performs the terminal answer call, retries, and fallback over HTTP.

The provider-neutral Request, Response, Usage, and LlmResponse contracts come from switchyard-protocol.

The Dynamo PreProc example adds an optional queue-depth check after task classification.

Concurrent calls and cancellation

An algorithm can poll several Driver::call_model or Driver::call_decision futures with tokio::join!, tokio::select!, or FuturesUnordered. Each call has its own reply channel. Dropping a waiting future cancels only that call; the algorithm can continue making other calls.

Hosts pass the work itself to call.respond(work).await. Awaiting the provider before calling respond prevents libsy from cancelling that work. For a response that is already available, use call.respond(std::future::ready(result)).await. Model-call errors stop the host run unless call.recover_errors is enabled. Decision-call errors are returned to the algorithm.

drive polls host handlers concurrently and drops outstanding handlers when the algorithm finishes or the run is dropped. Custom run_stream consumers must also poll handlers concurrently and drop them when the run ends. Separately spawned tasks need explicit abort and cleanup; dropping a task handle does not stop its work.

Cancellation drops the supplied work future. Once a response stream is delivered, its consumer controls its lifetime. Whether the provider stops generation depends on the transport and provider.

License

Licensed under the Apache License, Version 2.0.