Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs: cleanup related to duplicate info and Libsy explanations (#1135)
nachiketb-nvidia62efb7758eea
diff --git a/docs/getting_started.md b/docs/getting_started.mdindex 9ef907632..571170d6f 100644--- a/docs/getting_started.md+++ b/docs/getting_started.md@@ -254,9 +254,9 @@ Selected: z-ai/glm-5.2Answer: <model reply>```-With the optional tool-failure history, the selected model is `anthropic/claude-opus-4.8`. For routing only, keep the original `Done` arm. Other algorithms may supply an answer in `outcome.response`; the [general integration pattern](../README.md#start-with-libsy) shows how to reuse it.+With the optional tool-failure history, the selected model is `anthropic/claude-opus-4.8`. For routing only, keep the original `Done` arm. Other algorithms may supply an answer in `outcome.response`; see [final outcome handling](mechanics_of_libsy.md#stepdone) for how to reuse it.-Your client returns `switchyard_protocol::Response` and adapts errors to `libsy::Result`. The [optional Switchyard HTTP client](../crates/libsy-llm-client/README.md) provides translation, streaming, retries, and fallback. Use [`libsy::drive`](reference/host_integration.md#concurrent-calls-and-cancellation) for concurrent algorithms; this loop serves calls sequentially.+Your client returns `switchyard_protocol::Response` and adapts errors to `libsy::Result`. The [optional Switchyard HTTP client](../crates/libsy-llm-client/README.md) provides translation, streaming, retries, and fallback. For concurrent algorithms, [poll the Step stream and call futures together](mechanics_of_libsy.md#control-how-work-runs); this loop serves calls sequentially.## Next stepsdiff --git a/docs/mechanics_of_libsy.md b/docs/mechanics_of_libsy.mdindex f46b9e9f0..2e05a9d44 100644--- a/docs/mechanics_of_libsy.md+++ b/docs/mechanics_of_libsy.md@@ -24,8 +24,8 @@ flowchart TDexecute -->|"call.respond(...)"| routeroute -->|"Done"| done- classDef algorithm fill:#eef6df,stroke:#76b900,color:#24350b- classDef application fill:#f5f6f7,stroke:#6b7280,color:#1f2937+ classDef algorithm fill:#76b9001a,stroke:#76b900,stroke-width:2px+ classDef application fill:transparent,stroke:#6b7280class route algorithmclass execute,done application```@@ -161,7 +161,7 @@ Judge-based routing: CallModel(call) -> your response -> Done(outcome)Decision routing: CallDecision(call) -> your typed response -> Done(outcome)```-## Fulfill a model call+### `Step::CallModel`Suppose the next step is `CallModel`. The<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.CallModel.html"><code>CallModel</code></a>@@ -178,20 +178,12 @@ assert_eq!(request.model_id().as_ref(), Some(first_candidate));the attempts and must update the request's model before trying a later candidate.`call.algorithm` names the requesting algorithm for your own telemetry.-Your application returns the result through-<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.CallModel.html#method.respond"><code>call.respond</code></a>.-Inside the `Step::CallModel(call)` arm, the exchange looks like this:--```rust-let request = call.request.clone();-// Pass the work future to respond; it returns the result to the algorithm.-call.respond(client.call(request)).await?;-```--Here, `client.call` is your async adapter, returning-`switchyard_libsy::Result<Response>`. It owns the provider call and any format-conversion, as in the [OpenRouter adapter](getting_started.md#use-libsy-with-your-client).-`respond` consumes the call, allowing the algorithm to continue with its result.+In the [Step loop](#read-the-next-step), `client.call` is your async adapter,+returning `switchyard_libsy::Result<Response>`. It owns the provider call and any+format conversion, as in the+[OpenRouter adapter](getting_started.md#use-libsy-with-your-client).+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.CallModel.html#method.respond"><code>call.respond</code></a>+consumes the call and runs that work future, returning its result to the algorithm.The returned<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/envelope/struct.Response.html"><code>Response</code></a>@@ -220,7 +212,7 @@ Its async `call(Request)` returns a `Response` or `LlmClientError`; the[host guide](reference/host_integration.md#serve-routing-time-calls) shows the erroradapter. Using your own adapter method, as above, also works.-## Return a typed decision+### `Step::CallDecision`An algorithm can ask a decision model to evaluate a question through`Step::CallDecision(call)`. This time, `call` holds a@@ -229,13 +221,8 @@ It has one `model` and a<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/decision/struct.DecisionRequest.html"><code>DecisionRequest</code></a>in `request`.-The algorithm creates the question. Your application sends the supplied request-through its decision client and returns the typed answer:--```rust-let request = call.request.clone();-call.respond(decision_client.call(request)).await?;-```+The algorithm creates the question. The [Step loop](#read-the-next-step) sends+the supplied request through `decision_client.call` and returns the typed answer.The adapter returns `switchyard_libsy::Result<DecisionResponse>`. The optionalshared interface is@@ -292,7 +279,7 @@ let response = DecisionResponse {</details>-## Receive the final outcome+### `Step::Done`Once the algorithm has enough information, it emits `Step::Done(outcome)` with a<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.RoutingOutcome.html"><code>RoutingOutcome</code></a>.@@ -358,37 +345,53 @@ Each call has its own reply channel. Awaiting calls one at a time in the step loserves them sequentially. To run calls concurrently, your application must keepreading steps while polling their work.-For concurrent execution, use-<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/fn.drive.html"><code>libsy::drive</code></a>.-It polls your call handlers concurrently and returns the `RoutingOutcome`. Your-handler receives-<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/enum.Call.html"><code>Call::Model</code> or <code>Call::Decision</code></a>;-`drive` handles `Done` itself:+Use the same inputs and clients, but replace the earlier `run_stream` call and+sequential loop with this version. `tokio::select!` polls both the Step stream and+the pending calls, so new calls can start before earlier ones finish:```rust-// Borrow the adapters so every call can use them.-let client = &client;-let decision_client = &decision_client;-let outcome = switchyard_libsy::drive(algorithm, request, models, |call| async move {- match call {- Call::Model(call) => {- let request = call.request.clone();- call.respond(client.call(request)).await- }- Call::Decision(call) => {- let request = call.request.clone();- call.respond(decision_client.call(request)).await+use futures::{FutureExt, StreamExt, stream::FuturesUnordered};+use switchyard_libsy::LibsyError;++let mut steps = algorithm.run_stream(request, models, state);+let mut in_flight = FuturesUnordered::new();++let outcome = loop {+ tokio::select! {+ Some(result) = in_flight.next() => result?,+ step = steps.next() => {+ let step = step.ok_or(LibsyError::MissingFinalResponse)?;+ match step? {+ Step::CallModel(call) => {+ let request = call.request.clone();+ in_flight.push(call.respond(client.call(request)).boxed());+ }+ Step::CallDecision(call) => {+ let request = call.request.clone();+ in_flight.push(call.respond(decision_client.call(request)).boxed());+ }+ Step::Done(outcome) => break *outcome,+ }}}-}).await?;+};+// Stop unused calls before handling the final outcome.+drop(in_flight);+drop(steps);```-It starts with default state. Use `run_stream` directly to supply your own state-or execution loop.+`boxed()` gives the two call kinds a common future type for the collection.+Only calls requested concurrently by the algorithm can overlap. The returned+`outcome` is handled as shown in [`Step::Done`](#stepdone).++For a ready-made version, see+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/fn.drive.html"><code>libsy::drive</code></a>+and its [reference execution loop](../crates/libsy/src/core/algorithm.rs#L593).-If the algorithm stops waiting, `respond` drops that call's work future. Dropping-the step stream aborts the algorithm task; your application cleans up separately-started work. `drive` drops its outstanding handlers when the run ends or is dropped.+If the algorithm stops waiting, `respond` drops that call's work future. The+explicit drops stop the remaining handlers and abort the algorithm task. Returning+an error or cancelling the enclosing future also drops these local values. Your+application must clean up any separately spawned work itself.A model-call failure normally stops the host run when propagated with `?`.`call.recover_errors` lets the algorithm receive and handle it instead. Decision@@ -397,9 +400,9 @@ unanswered call gives the waiting algorithm `DriverError::ResponseDropped`.Once a response stream has been delivered, its consumer owns its lifetime.Cancelling local work cannot guarantee that a provider stops generation or billing,-or undo side effects that have already occurred. The [host integration-guide](reference/host_integration.md#concurrent-calls-and-cancellation) covers host-cleanup in Rust and Python.+or undo side effects that have already occurred. See the [host integration+guide](reference/host_integration.md#concurrent-calls-and-cancellation) for+Python task cleanup.The [Rust API reference](reference/rust_api.md) links the complete cratedocumentation for these types.diff --git a/docs/reference/host_integration.md b/docs/reference/host_integration.mdindex 1eaa503f6..9a3fd263b 100644--- a/docs/reference/host_integration.md+++ b/docs/reference/host_integration.md@@ -1,80 +1,25 @@# Host Integration-Use this guide when embedding libsy algorithms that ask the host to serve model or decision calls. For a first routing result and model-client handoff, start with the [libsy quickstart](../getting_started.md).+Use this guide for shared client-trait adapters and Python bindings. For Rust+project setup and a working model-client handoff, start with the+[libsy quickstart](../getting_started.md).For an explanation of steps, call objects, and the request and response contracts,read [Mechanics of Libsy](../mechanics_of_libsy.md).## Development API-This page uses the development API, including its asynchronous `respond` method. The Rust examples pin the same revision as the [quickstart](../getting_started.md). The released `v0.3.0` callback API is synchronous. Use matching source-built [Python bindings](../../INSTALLATION.md#development) for this page's Python callback example.--## Route in Rust--With Rust 1.96.1+ and a native build toolchain, create a project:--```bash-cargo new switchyard-rust-demo-cd switchyard-rust-demo-```--Add these dependencies to `Cargo.toml`. Both Switchyard crates use the same pinned revision:--```toml-[dependencies]-async-trait = "0.1"-futures = "0.3"-switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", rev = "5415db205c1f4db7e079731e7bf33a6609c67c8e" }-switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", rev = "5415db205c1f4db7e079731e7bf33a6609c67c8e" }-tokio = { version = "1", features = ["macros", "rt"] }-```--Replace `src/main.rs` with this complete routing-only example:--```rust-use std::{collections::HashMap, sync::Arc};--use futures::StreamExt;-use switchyard_libsy::{- Algorithm, PickerMode, RuntimeModels, StageRouter, StageRouterConfig, State, Step,-};-use switchyard_protocol::{Category, LlmRequest, Message, Request, Role};--#[tokio::main(flavor = "current_thread")]-async fn main() -> Result<(), Box<dyn std::error::Error>> {- let router = Arc::new(StageRouter::new(StageRouterConfig::new(- PickerMode::EfficientFirst, 0.5,- ))?);- let models = Arc::new(RuntimeModels::new(HashMap::from([- (Category::Efficient, vec!["Efficient".into()]),- (Category::Capable, vec!["Capable".into()]),- (Category::Any, vec!["Efficient".into(), "Capable".into()]),- ])));- let request = Request {- llm_request: LlmRequest {- messages: vec![Message::text(Role::User, "Hello")],- ..Default::default()- },- ..Default::default()- };- let mut steps = router.run_stream(request, models, State::default());- while let Some(step) = steps.next().await {- match step? {- Step::Done(outcome) => println!("Selected: {}", outcome.selected_model_ids[0]),- _ => unreachable!("This Stage configuration makes no routing-time calls"),- }- }- Ok(())-}-```--Run `cargo run`. It prints `Selected: Efficient` without making a network call. This example uses the pinned development API, including its `State` argument. For a complete routing and client example, follow the [Rust quickstart](../getting_started.md#use-libsy-with-your-client). Use [switchyard-translation](../../crates/switchyard-translation/README.md) to convert provider requests and libsy's protocol types, or [switchyard-llm-client](../../crates/libsy-llm-client/README.md) to drive an algorithm over HTTP.+The callback examples use the development API, including its asynchronous+`respond` method. Use the [quickstart's dependencies](../getting_started.md#install)+for the Rust snippets. The released `v0.3.0` callback API is synchronous. Use+matching source-built [Python bindings](../../INSTALLATION.md#development) for+the Python callback example below.## Serve routing-time calls-An algorithm yields a stream of steps. Each `Step::CallModel` is a routing-time classifier or judge call your host performs over its own transport. The run ends with `Step::Done` carrying a `RoutingOutcome`: the selected model, ordered fallbacks, rewritten request, and an optional response when routing already produced the answer. Otherwise the host makes the terminal answer call from that outcome.--Pass each call's work to `respond` before awaiting it. In Rust:+The [Step loop](../mechanics_of_libsy.md#read-the-next-step) uses adapters that+return `switchyard_libsy::Result`. When using `RoutedLlmClient` or+`RoutedDecisionClient`, map their errors to `LibsyError`:```rustuse switchyard_libsy::{CallDecision, CallModel, LibsyError, Result};@@ -97,31 +42,32 @@ async fn serve_decision(call: CallDecision, client: &dyn RoutedDecisionClient) -}```-These traits belong to `switchyard-protocol`. Implement them over your own transport or SDK. The quickstart's custom [OpenRouter client](../getting_started.md#use-libsy-with-your-client) calls HTTP directly; `run_stream` also works without implementing these traits. Switchyard's HTTP client is optional.-In Python, the client coroutine returns `LlmResponse.Agg` or `LlmResponse.Stream`:```pythonawait call.respond(client.call(call.request, call.models[0]))```-`respond` accepts a future or awaitable instead of a completed response. Move the provider await into this method. Python `call.fail(error)` is also awaitable. Decision calls use the same pattern with their existing decision response type.+Pass the coroutine itself to `respond`. Python `call.fail(error)` is also+awaitable. Decision calls use the same pattern with their decision response type.## Concurrent calls and cancellation-In Rust, `respond` drops the supplied work and returns `Ok(())` when the algorithm stops waiting for that call. Dropping a pending `respond` or `libsy::drive` future also drops its host work. Use `libsy::drive` to manage concurrent handlers and their lifetime.+See [Control how work runs](../mechanics_of_libsy.md#control-how-work-runs) for+Rust execution, cancellation behavior, and upstream limits.Python `respond` cancels and awaits the supplied provider work before returning. For concurrent algorithms, the host creates handler tasks with `asyncio.create_task` and must cancel and await remaining handlers when the run ends; see the complete [Python example](../../examples/libsy.py). Awaiting handlers one at a time inside the step loop serializes their execution.-After a response stream is delivered, its consumer controls its lifetime. Cancellation cannot guarantee that an upstream provider stops generating or billing.-## Metadata and telemetry-Successful runs also include `OutcomeMetadata`: a unique `outcome_id`, the algorithm name, and optional JSON evidence. In Python, read `outcome.metadata.outcome_id`, `outcome.metadata.algorithm`, and `outcome.metadata.evidence` after checking that `outcome.metadata` is present. Evidence is a normal Python value, usually a dictionary; algorithms without evidence return `None`.--With a host-installed OpenTelemetry subscriber, the existing `libsy.run` span records the same identity, selected models, and supported evidence fields. See the [OpenTelemetry reference](opentelemetry.md) for field names, metrics, and export setup. libsy does not install an exporter or send telemetry itself.+The [final outcome](../mechanics_of_libsy.md#stepdone) carries+routing metadata. In Python, read `outcome.metadata.outcome_id`,+`outcome.metadata.algorithm`, and `outcome.metadata.evidence` after checking that+`outcome.metadata` is present. Evidence is a normal Python value, usually a+dictionary; algorithms without evidence return `None`.-For the request, response, and streaming types the steps carry, see [`switchyard-protocol`](../../crates/protocol/README.md).+See the [OpenTelemetry reference](opentelemetry.md) for field names, metrics,+and host export setup.For plan/execute, supply conversation history, including tool calls and results, before routing. Responses API continuation IDs alone are not enough. See the [Responses API history requirement](../routing_algorithms/plan_execute_routing.md#responses-api-history-requirement).diff --git a/docs/reference/rust_api.md b/docs/reference/rust_api.mdindex 56f4b096d..2cce7f567 100644--- a/docs/reference/rust_api.md+++ b/docs/reference/rust_api.md@@ -26,13 +26,9 @@ Completions, OpenAI Responses, and Anthropic Messages formats.## Crate boundary-Applications embedding libsy normally depend on the libsy and protocol crates:--```toml-[dependencies]-switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.3.0" }-switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.3.0" }-```+Use the [Rust quickstart](../getting_started.md#install) for compatible dependency+versions. [Mechanics of Libsy](../mechanics_of_libsy.md) explains how these crates+work together during a run.Import algorithms and orchestration types from `switchyard_libsy`. Importrequests, responses, metadata, decisions, and LLM client traits from