Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs: cleanup related to duplicate info and Libsy explanations (#1135)

nachiketb-nvidia62efb7758eea
diff --git a/docs/getting_started.md b/docs/getting_started.md
index 9ef907632..571170d6f 100644
--- a/docs/getting_started.md
+++ b/docs/getting_started.md
@@ -254,9 +254,9 @@ Selected: z-ai/glm-5.2
Answer: <model reply>
```
-With the optional tool-failure history, the selected model is `anthropic/claude-opus-4.8`. For routing only, keep the original `Done` arm. Other algorithms may supply an answer in `outcome.response`; the [general integration pattern](../README.md#start-with-libsy) shows how to reuse it.
+With the optional tool-failure history, the selected model is `anthropic/claude-opus-4.8`. For routing only, keep the original `Done` arm. Other algorithms may supply an answer in `outcome.response`; see [final outcome handling](mechanics_of_libsy.md#stepdone) for how to reuse it.
-Your client returns `switchyard_protocol::Response` and adapts errors to `libsy::Result`. The [optional Switchyard HTTP client](../crates/libsy-llm-client/README.md) provides translation, streaming, retries, and fallback. Use [`libsy::drive`](reference/host_integration.md#concurrent-calls-and-cancellation) for concurrent algorithms; this loop serves calls sequentially.
+Your client returns `switchyard_protocol::Response` and adapts errors to `libsy::Result`. The [optional Switchyard HTTP client](../crates/libsy-llm-client/README.md) provides translation, streaming, retries, and fallback. For concurrent algorithms, [poll the Step stream and call futures together](mechanics_of_libsy.md#control-how-work-runs); this loop serves calls sequentially.
## Next steps
diff --git a/docs/mechanics_of_libsy.md b/docs/mechanics_of_libsy.md
index f46b9e9f0..2e05a9d44 100644
--- a/docs/mechanics_of_libsy.md
+++ b/docs/mechanics_of_libsy.md
@@ -24,8 +24,8 @@ flowchart TD
execute -->|"call.respond(...)"| route
route -->|"Done"| done
- classDef algorithm fill:#eef6df,stroke:#76b900,color:#24350b
- classDef application fill:#f5f6f7,stroke:#6b7280,color:#1f2937
+ classDef algorithm fill:#76b9001a,stroke:#76b900,stroke-width:2px
+ classDef application fill:transparent,stroke:#6b7280
class route algorithm
class execute,done application
```
@@ -161,7 +161,7 @@ Judge-based routing: CallModel(call) -> your response -> Done(outcome)
Decision routing: CallDecision(call) -> your typed response -> Done(outcome)
```
-## Fulfill a model call
+### `Step::CallModel`
Suppose the next step is `CallModel`. The
<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.CallModel.html"><code>CallModel</code></a>
@@ -178,20 +178,12 @@ assert_eq!(request.model_id().as_ref(), Some(first_candidate));
the attempts and must update the request's model before trying a later candidate.
`call.algorithm` names the requesting algorithm for your own telemetry.
-Your application returns the result through
-<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.CallModel.html#method.respond"><code>call.respond</code></a>.
-Inside the `Step::CallModel(call)` arm, the exchange looks like this:
-
-```rust
-let request = call.request.clone();
-// Pass the work future to respond; it returns the result to the algorithm.
-call.respond(client.call(request)).await?;
-```
-
-Here, `client.call` is your async adapter, returning
-`switchyard_libsy::Result<Response>`. It owns the provider call and any format
-conversion, as in the [OpenRouter adapter](getting_started.md#use-libsy-with-your-client).
-`respond` consumes the call, allowing the algorithm to continue with its result.
+In the [Step loop](#read-the-next-step), `client.call` is your async adapter,
+returning `switchyard_libsy::Result<Response>`. It owns the provider call and any
+format conversion, as in the
+[OpenRouter adapter](getting_started.md#use-libsy-with-your-client).
+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.CallModel.html#method.respond"><code>call.respond</code></a>
+consumes the call and runs that work future, returning its result to the algorithm.
The returned
<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/envelope/struct.Response.html"><code>Response</code></a>
@@ -220,7 +212,7 @@ Its async `call(Request)` returns a `Response` or `LlmClientError`; the
[host guide](reference/host_integration.md#serve-routing-time-calls) shows the error
adapter. Using your own adapter method, as above, also works.
-## Return a typed decision
+### `Step::CallDecision`
An algorithm can ask a decision model to evaluate a question through
`Step::CallDecision(call)`. This time, `call` holds a
@@ -229,13 +221,8 @@ It has one `model` and a
<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/decision/struct.DecisionRequest.html"><code>DecisionRequest</code></a>
in `request`.
-The algorithm creates the question. Your application sends the supplied request
-through its decision client and returns the typed answer:
-
-```rust
-let request = call.request.clone();
-call.respond(decision_client.call(request)).await?;
-```
+The algorithm creates the question. The [Step loop](#read-the-next-step) sends
+the supplied request through `decision_client.call` and returns the typed answer.
The adapter returns `switchyard_libsy::Result<DecisionResponse>`. The optional
shared interface is
@@ -292,7 +279,7 @@ let response = DecisionResponse {
</details>
-## Receive the final outcome
+### `Step::Done`
Once the algorithm has enough information, it emits `Step::Done(outcome)` with a
<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.RoutingOutcome.html"><code>RoutingOutcome</code></a>.
@@ -358,37 +345,53 @@ Each call has its own reply channel. Awaiting calls one at a time in the step lo
serves them sequentially. To run calls concurrently, your application must keep
reading steps while polling their work.
-For concurrent execution, use
-<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/fn.drive.html"><code>libsy::drive</code></a>.
-It polls your call handlers concurrently and returns the `RoutingOutcome`. Your
-handler receives
-<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/enum.Call.html"><code>Call::Model</code> or <code>Call::Decision</code></a>;
-`drive` handles `Done` itself:
+Use the same inputs and clients, but replace the earlier `run_stream` call and
+sequential loop with this version. `tokio::select!` polls both the Step stream and
+the pending calls, so new calls can start before earlier ones finish:
```rust
-// Borrow the adapters so every call can use them.
-let client = &client;
-let decision_client = &decision_client;
-let outcome = switchyard_libsy::drive(algorithm, request, models, |call| async move {
- match call {
- Call::Model(call) => {
- let request = call.request.clone();
- call.respond(client.call(request)).await
- }
- Call::Decision(call) => {
- let request = call.request.clone();
- call.respond(decision_client.call(request)).await
+use futures::{FutureExt, StreamExt, stream::FuturesUnordered};
+use switchyard_libsy::LibsyError;
+
+let mut steps = algorithm.run_stream(request, models, state);
+let mut in_flight = FuturesUnordered::new();
+
+let outcome = loop {
+ tokio::select! {
+ Some(result) = in_flight.next() => result?,
+ step = steps.next() => {
+ let step = step.ok_or(LibsyError::MissingFinalResponse)?;
+ match step? {
+ Step::CallModel(call) => {
+ let request = call.request.clone();
+ in_flight.push(call.respond(client.call(request)).boxed());
+ }
+ Step::CallDecision(call) => {
+ let request = call.request.clone();
+ in_flight.push(call.respond(decision_client.call(request)).boxed());
+ }
+ Step::Done(outcome) => break *outcome,
+ }
}
}
-}).await?;
+};
+// Stop unused calls before handling the final outcome.
+drop(in_flight);
+drop(steps);
```
-It starts with default state. Use `run_stream` directly to supply your own state
-or execution loop.
+`boxed()` gives the two call kinds a common future type for the collection.
+Only calls requested concurrently by the algorithm can overlap. The returned
+`outcome` is handled as shown in [`Step::Done`](#stepdone).
+
+For a ready-made version, see
+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/fn.drive.html"><code>libsy::drive</code></a>
+and its [reference execution loop](../crates/libsy/src/core/algorithm.rs#L593).
-If the algorithm stops waiting, `respond` drops that call's work future. Dropping
-the step stream aborts the algorithm task; your application cleans up separately
-started work. `drive` drops its outstanding handlers when the run ends or is dropped.
+If the algorithm stops waiting, `respond` drops that call's work future. The
+explicit drops stop the remaining handlers and abort the algorithm task. Returning
+an error or cancelling the enclosing future also drops these local values. Your
+application must clean up any separately spawned work itself.
A model-call failure normally stops the host run when propagated with `?`.
`call.recover_errors` lets the algorithm receive and handle it instead. Decision
@@ -397,9 +400,9 @@ unanswered call gives the waiting algorithm `DriverError::ResponseDropped`.
Once a response stream has been delivered, its consumer owns its lifetime.
Cancelling local work cannot guarantee that a provider stops generation or billing,
-or undo side effects that have already occurred. The [host integration
-guide](reference/host_integration.md#concurrent-calls-and-cancellation) covers host
-cleanup in Rust and Python.
+or undo side effects that have already occurred. See the [host integration
+guide](reference/host_integration.md#concurrent-calls-and-cancellation) for
+Python task cleanup.
The [Rust API reference](reference/rust_api.md) links the complete crate
documentation for these types.
diff --git a/docs/reference/host_integration.md b/docs/reference/host_integration.md
index 1eaa503f6..9a3fd263b 100644
--- a/docs/reference/host_integration.md
+++ b/docs/reference/host_integration.md
@@ -1,80 +1,25 @@
# Host Integration
-Use this guide when embedding libsy algorithms that ask the host to serve model or decision calls. For a first routing result and model-client handoff, start with the [libsy quickstart](../getting_started.md).
+Use this guide for shared client-trait adapters and Python bindings. For Rust
+project setup and a working model-client handoff, start with the
+[libsy quickstart](../getting_started.md).
For an explanation of steps, call objects, and the request and response contracts,
read [Mechanics of Libsy](../mechanics_of_libsy.md).
## Development API
-This page uses the development API, including its asynchronous `respond` method. The Rust examples pin the same revision as the [quickstart](../getting_started.md). The released `v0.3.0` callback API is synchronous. Use matching source-built [Python bindings](../../INSTALLATION.md#development) for this page's Python callback example.
-
-## Route in Rust
-
-With Rust 1.96.1+ and a native build toolchain, create a project:
-
-```bash
-cargo new switchyard-rust-demo
-cd switchyard-rust-demo
-```
-
-Add these dependencies to `Cargo.toml`. Both Switchyard crates use the same pinned revision:
-
-```toml
-[dependencies]
-async-trait = "0.1"
-futures = "0.3"
-switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", rev = "5415db205c1f4db7e079731e7bf33a6609c67c8e" }
-switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", rev = "5415db205c1f4db7e079731e7bf33a6609c67c8e" }
-tokio = { version = "1", features = ["macros", "rt"] }
-```
-
-Replace `src/main.rs` with this complete routing-only example:
-
-```rust
-use std::{collections::HashMap, sync::Arc};
-
-use futures::StreamExt;
-use switchyard_libsy::{
- Algorithm, PickerMode, RuntimeModels, StageRouter, StageRouterConfig, State, Step,
-};
-use switchyard_protocol::{Category, LlmRequest, Message, Request, Role};
-
-#[tokio::main(flavor = "current_thread")]
-async fn main() -> Result<(), Box<dyn std::error::Error>> {
- let router = Arc::new(StageRouter::new(StageRouterConfig::new(
- PickerMode::EfficientFirst, 0.5,
- ))?);
- let models = Arc::new(RuntimeModels::new(HashMap::from([
- (Category::Efficient, vec!["Efficient".into()]),
- (Category::Capable, vec!["Capable".into()]),
- (Category::Any, vec!["Efficient".into(), "Capable".into()]),
- ])));
- let request = Request {
- llm_request: LlmRequest {
- messages: vec![Message::text(Role::User, "Hello")],
- ..Default::default()
- },
- ..Default::default()
- };
- let mut steps = router.run_stream(request, models, State::default());
- while let Some(step) = steps.next().await {
- match step? {
- Step::Done(outcome) => println!("Selected: {}", outcome.selected_model_ids[0]),
- _ => unreachable!("This Stage configuration makes no routing-time calls"),
- }
- }
- Ok(())
-}
-```
-
-Run `cargo run`. It prints `Selected: Efficient` without making a network call. This example uses the pinned development API, including its `State` argument. For a complete routing and client example, follow the [Rust quickstart](../getting_started.md#use-libsy-with-your-client). Use [switchyard-translation](../../crates/switchyard-translation/README.md) to convert provider requests and libsy's protocol types, or [switchyard-llm-client](../../crates/libsy-llm-client/README.md) to drive an algorithm over HTTP.
+The callback examples use the development API, including its asynchronous
+`respond` method. Use the [quickstart's dependencies](../getting_started.md#install)
+for the Rust snippets. The released `v0.3.0` callback API is synchronous. Use
+matching source-built [Python bindings](../../INSTALLATION.md#development) for
+the Python callback example below.
## Serve routing-time calls
-An algorithm yields a stream of steps. Each `Step::CallModel` is a routing-time classifier or judge call your host performs over its own transport. The run ends with `Step::Done` carrying a `RoutingOutcome`: the selected model, ordered fallbacks, rewritten request, and an optional response when routing already produced the answer. Otherwise the host makes the terminal answer call from that outcome.
-
-Pass each call's work to `respond` before awaiting it. In Rust:
+The [Step loop](../mechanics_of_libsy.md#read-the-next-step) uses adapters that
+return `switchyard_libsy::Result`. When using `RoutedLlmClient` or
+`RoutedDecisionClient`, map their errors to `LibsyError`:
```rust
use switchyard_libsy::{CallDecision, CallModel, LibsyError, Result};
@@ -97,31 +42,32 @@ async fn serve_decision(call: CallDecision, client: &dyn RoutedDecisionClient) -
}
```
-These traits belong to `switchyard-protocol`. Implement them over your own transport or SDK. The quickstart's custom [OpenRouter client](../getting_started.md#use-libsy-with-your-client) calls HTTP directly; `run_stream` also works without implementing these traits. Switchyard's HTTP client is optional.
-
In Python, the client coroutine returns `LlmResponse.Agg` or `LlmResponse.Stream`:
```python
await call.respond(client.call(call.request, call.models[0]))
```
-`respond` accepts a future or awaitable instead of a completed response. Move the provider await into this method. Python `call.fail(error)` is also awaitable. Decision calls use the same pattern with their existing decision response type.
+Pass the coroutine itself to `respond`. Python `call.fail(error)` is also
+awaitable. Decision calls use the same pattern with their decision response type.
## Concurrent calls and cancellation
-In Rust, `respond` drops the supplied work and returns `Ok(())` when the algorithm stops waiting for that call. Dropping a pending `respond` or `libsy::drive` future also drops its host work. Use `libsy::drive` to manage concurrent handlers and their lifetime.
+See [Control how work runs](../mechanics_of_libsy.md#control-how-work-runs) for
+Rust execution, cancellation behavior, and upstream limits.
Python `respond` cancels and awaits the supplied provider work before returning. For concurrent algorithms, the host creates handler tasks with `asyncio.create_task` and must cancel and await remaining handlers when the run ends; see the complete [Python example](../../examples/libsy.py). Awaiting handlers one at a time inside the step loop serializes their execution.
-After a response stream is delivered, its consumer controls its lifetime. Cancellation cannot guarantee that an upstream provider stops generating or billing.
-
## Metadata and telemetry
-Successful runs also include `OutcomeMetadata`: a unique `outcome_id`, the algorithm name, and optional JSON evidence. In Python, read `outcome.metadata.outcome_id`, `outcome.metadata.algorithm`, and `outcome.metadata.evidence` after checking that `outcome.metadata` is present. Evidence is a normal Python value, usually a dictionary; algorithms without evidence return `None`.
-
-With a host-installed OpenTelemetry subscriber, the existing `libsy.run` span records the same identity, selected models, and supported evidence fields. See the [OpenTelemetry reference](opentelemetry.md) for field names, metrics, and export setup. libsy does not install an exporter or send telemetry itself.
+The [final outcome](../mechanics_of_libsy.md#stepdone) carries
+routing metadata. In Python, read `outcome.metadata.outcome_id`,
+`outcome.metadata.algorithm`, and `outcome.metadata.evidence` after checking that
+`outcome.metadata` is present. Evidence is a normal Python value, usually a
+dictionary; algorithms without evidence return `None`.
-For the request, response, and streaming types the steps carry, see [`switchyard-protocol`](../../crates/protocol/README.md).
+See the [OpenTelemetry reference](opentelemetry.md) for field names, metrics,
+and host export setup.
For plan/execute, supply conversation history, including tool calls and results, before routing. Responses API continuation IDs alone are not enough. See the [Responses API history requirement](../routing_algorithms/plan_execute_routing.md#responses-api-history-requirement).
diff --git a/docs/reference/rust_api.md b/docs/reference/rust_api.md
index 56f4b096d..2cce7f567 100644
--- a/docs/reference/rust_api.md
+++ b/docs/reference/rust_api.md
@@ -26,13 +26,9 @@ Completions, OpenAI Responses, and Anthropic Messages formats.
## Crate boundary
-Applications embedding libsy normally depend on the libsy and protocol crates:
-
-```toml
-[dependencies]
-switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.3.0" }
-switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.3.0" }
-```
+Use the [Rust quickstart](../getting_started.md#install) for compatible dependency
+versions. [Mechanics of Libsy](../mechanics_of_libsy.md) explains how these crates
+work together during a run.
Import algorithms and orchestration types from `switchyard_libsy`. Import
requests, responses, metadata, decisions, and LLM client traits from