Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs: explain the mechanics of libsy (#1123)
nachiketb-nvidia6bf9ca79ef5f
diff --git a/docs/getting_started.md b/docs/getting_started.mdindex f9ed945a0..9ef907632 100644--- a/docs/getting_started.md+++ b/docs/getting_started.md@@ -260,6 +260,7 @@ Your client returns `switchyard_protocol::Response` and adapts errors to `libsy:## Next steps+- **Understand the exchange:** [Mechanics of Libsy](mechanics_of_libsy.md) follows a request through steps, call objects, protocol types, and the final outcome.- **Explore the Rust API:** [libsy crate](../crates/libsy/README.md) and [generated API reference](reference/rust_api.md).- **Use a gateway integration:** [LiteLLM](../examples/litellm/README.md) or [NeMo Relay](integrations/nemo_relay.md). Check each guide's supported algorithms and versions.- **Choose another algorithm:** [routing overview](routing_algorithms/overview.md).diff --git a/docs/mechanics_of_libsy.md b/docs/mechanics_of_libsy.mdnew file mode 100644index 000000000..f46b9e9f0--- /dev/null+++ b/docs/mechanics_of_libsy.md@@ -0,0 +1,405 @@+# Mechanics of Libsy++Libsy expresses work as a stream of **Steps**. Your application executes that work+with its own clients. `Done` is the exception: it carries the completed routing+outcome.++Follow one request through that exchange below. The snippets show parts of an+integration; the [Rust quickstart](getting_started.md) has a complete program,+imports, and matching development-API dependencies.++## Why execution stays with your application++Libsy separates routing policy from model execution. The algorithm decides which+models to use and how their results affect the next step. Your application decides+how those models are called.++```mermaid+flowchart TD+ route["Libsy algorithm<br/>Choose the next Step"]+ execute["Your application<br/>Run your model or decision client"]+ done["Your application<br/>Use the routing outcome"]++ route -->|"CallModel or CallDecision"| execute+ execute -->|"call.respond(...)"| route+ route -->|"Done"| done++ classDef algorithm fill:#eef6df,stroke:#76b900,color:#24350b+ classDef application fill:#f5f6f7,stroke:#6b7280,color:#1f2937+ class route algorithm+ class execute,done application+```++A run can finish with `Done` immediately or request several calls before finishing.++A gateway can fulfill these calls through its existing dispatch, credentials,+retries, and observability. That includes calls to judges used during routing.+An agent harness can supply its conversation and tool history, then use its model+clients to execute the requested work. Both adapt their inputs and outputs to the+shared protocol types.++Routing can require more than choosing a model ID. An algorithm may change a+request, ask a judge, inspect an answer, or compare results from several models.+Returning responses through `call.respond` lets it continue that work. When a+selection alone is enough, `Done` leaves the final model call to your application.++The shared protocol crate lets clients and algorithms use the same request and+response contracts. A client can implement those contracts without depending on+the routing algorithm implementation.++## Start with a request++Your application supplies a+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/envelope/struct.Request.html"><code>Request</code></a>.+Its `llm_request` field holds the+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/llm/struct.LlmRequest.html"><code>LlmRequest</code></a>:+the conversation, instructions, tools, and generation settings. These+provider-neutral types come from `switchyard-protocol`.++```rust+let request = Request {+ llm_request: LlmRequest {+ messages: vec![Message::text(Role::User, "Fix the failing tests.")],+ ..Default::default()+ },+ metadata: Some(Metadata {+ session_id: Some("session-42".into()),+ ..Default::default()+ }),+ ..Default::default()+};+```++<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/metadata/struct.Metadata.html"><code>Metadata</code></a>+holds the session ID above, which algorithms can use to retain session decisions.+It can also include a correlation ID for your logs. The optional `raw_request`+field holds data your application retains; libsy does not read it.++Next, supply the models the algorithm can use.+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.RuntimeModels.html"><code>RuntimeModels</code></a>+groups model IDs by+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/category/enum.Category.html"><code>Category</code></a>,+such as `Efficient`, `Capable`, or `Judge`:++```rust+let models = Arc::new(RuntimeModels::new(HashMap::from([+ (Category::Efficient, vec!["Efficient".into()]),+ (Category::Capable, vec!["Capable".into()]),+ (Category::Judge, vec!["Judge".into()]),+ (Category::Any, vec!["Efficient".into(), "Capable".into()]),+])));+```++These are local example IDs. Within a category, models are ordered best-first. A+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/model_id/struct.ModelId.html"><code>ModelId</code></a>+names a destination; your application decides which client serves it.++With an algorithm, request, model pool, and initial+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.State.html"><code>State</code></a>, start a run:++```rust+let state = State::default();+let mut steps = algorithm.run_stream(request, models, state);+```++<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/trait.Algorithm.html#method.run_stream"><code>run_stream</code></a>+returns a+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/type.StepStream.html"><code>StepStream</code></a>.+Each item is a `Result<Step>`: either the next step or an error that ends the run.++## Read the next Step++A <a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/enum.Step.html"><code>Step</code></a> describes+one unit of work for your application, or reports that routing has finished.+Its three variants are:++```rust+pub enum Step {+ CallModel(Box<CallModel>),+ CallDecision(Box<CallDecision>),+ Done(Box<RoutingOutcome>),+}+```++When you write `Step::CallModel(call)` in a match, `call` is the `Box<CallModel>`+stored in that variant. It is an object describing a requested operation, with a+way to return its result to the waiting algorithm.++Reuse `steps` from above to read those operations. Here, `client` and+`decision_client` are your async adapters; the next sections explain their inputs+and results:++```rust+while let Some(step) = steps.next().await {+ match step? {+ Step::CallModel(call) => {+ let request = call.request.clone();+ call.respond(client.call(request)).await?;+ }+ Step::CallDecision(call) => {+ let request = call.request.clone();+ call.respond(decision_client.call(request)).await?;+ }+ Step::Done(outcome) => {+ // Inspect the result; final-answer handling is shown below.+ println!("Selected: {}", outcome.selected_model_id()?);+ break;+ }+ }+}+```++This loop serves one call at a time. [Concurrent execution](#control-how-work-runs)+keeps reading steps while calls are running.++For example, a signal-only router can finish locally. A classifier might first+ask your client to call a judge:++```text+Signal-only routing: Done(outcome)+Judge-based routing: CallModel(call) -> your response -> Done(outcome)+Decision routing: CallDecision(call) -> your typed response -> Done(outcome)+```++## Fulfill a model call++Suppose the next step is `CallModel`. The+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.CallModel.html"><code>CallModel</code></a>+object describes the operation:++```rust+// The model has already been set to the first candidate.+let request = call.request.clone();+let first_candidate = &call.models[0];+assert_eq!(request.model_id().as_ref(), Some(first_candidate));+```++`models` is a nonempty candidate list in fallback order. Your application executes+the attempts and must update the request's model before trying a later candidate.+`call.algorithm` names the requesting algorithm for your own telemetry.++Your application returns the result through+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.CallModel.html#method.respond"><code>call.respond</code></a>.+Inside the `Step::CallModel(call)` arm, the exchange looks like this:++```rust+let request = call.request.clone();+// Pass the work future to respond; it returns the result to the algorithm.+call.respond(client.call(request)).await?;+```++Here, `client.call` is your async adapter, returning+`switchyard_libsy::Result<Response>`. It owns the provider call and any format+conversion, as in the [OpenRouter adapter](getting_started.md#use-libsy-with-your-client).+`respond` consumes the call, allowing the algorithm to continue with its result.++The returned+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/envelope/struct.Response.html"><code>Response</code></a>+contains an+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/stream/enum.LlmResponse.html"><code>LlmResponse</code></a>+in its `llm_response` field:++```rust+pub enum LlmResponse {+ Stream(LlmResponseStream), // Live response events.+ Agg(AggLlmResponse), // A complete, buffered answer.+}+```++The+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/stream/struct.LlmResponseStreamEvent.html"><code>LlmResponseStreamEvent</code></a>+provides access to provider-neutral text, tool-call updates, and usage information.++`StepStream` carries work and the routing outcome. `LlmResponse::Stream` carries+model output. Your client can return that live stream without reading it to+completion first.++If you want a shared client interface, `switchyard-protocol` defines+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/client/trait.RoutedLlmClient.html"><code>RoutedLlmClient</code></a>.+Its async `call(Request)` returns a `Response` or `LlmClientError`; the+[host guide](reference/host_integration.md#serve-routing-time-calls) shows the error+adapter. Using your own adapter method, as above, also works.++## Return a typed decision++An algorithm can ask a decision model to evaluate a question through+`Step::CallDecision(call)`. This time, `call` holds a+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.CallDecision.html"><code>CallDecision</code></a>.+It has one `model` and a+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/decision/struct.DecisionRequest.html"><code>DecisionRequest</code></a>+in `request`.++The algorithm creates the question. Your application sends the supplied request+through its decision client and returns the typed answer:++```rust+let request = call.request.clone();+call.respond(decision_client.call(request)).await?;+```++The adapter returns `switchyard_libsy::Result<DecisionResponse>`. The optional+shared interface is+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/client/trait.RoutedDecisionClient.html"><code>RoutedDecisionClient</code></a>.+The algorithm applies its routing policy to the answer. The decision response is+distinct from the final conversational answer.++A decision request contains shared `context` and named questions. Each+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/decision/struct.DecisionQuestion.html"><code>question</code></a>+declares its expected answer shape: Boolean, choice among options, or score on a+rubric. The client's+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_protocol/decision/struct.DecisionResponse.html"><code>DecisionResponse</code></a>+uses the same question IDs.++<details markdown="1">+<summary>Example payloads: a Boolean question and its answer</summary>++These values illustrate what crosses the boundary. The algorithm supplies a+request like this in `call.request`:++```rust+let request = DecisionRequest {+ model: Some("Judge".into()),+ context: serde_json::json!({"task": "Fix the failing tests."}),+ questions: BTreeMap::from([(+ "needs_capable".into(),+ DecisionQuestion {+ instructions: serde_json::json!("Does this task need Capable?"),+ kind: DecisionKind::Boolean {+ true_description: None,+ false_description: None,+ },+ },+ )]),+};+```++The client might return this answer to `needs_capable`:++```rust+let response = DecisionResponse {+ id: None,+ model: Some("Judge".into()),+ answers: BTreeMap::from([(+ "needs_capable".into(),+ DecisionAnswer {+ value: DecisionValue::Boolean(BooleanEstimate::Value(true)),+ provider_confidence: None,+ },+ )]),+ usage: Default::default(),+};+```++</details>++## Receive the final outcome++Once the algorithm has enough information, it emits `Step::Done(outcome)` with a+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.RoutingOutcome.html"><code>RoutingOutcome</code></a>.+Inside that match arm, check whether routing has already produced an answer:++```rust+let response = match outcome.response {+ Some(response) => response, // Reuse the answer obtained during routing.+ None => client.call(outcome.request).await?, // Make the final call.+};+```++With `Some(response)`, another final call would repeat work already done. With+`None`, your application can send `outcome.request` now, as above, or return it to+a gateway to send later. Signal-only Stage takes this second path.++Use the returned request: it includes algorithm changes and already names the+selected model. `outcome.selected_model_ids` lists that model first, followed by+ordered fallbacks.++To read the final answer as text, collect a live stream or use the existing+buffered answer:++```rust+let answer = response.llm_response.into_agg().await?;+println!("{}", completion_text(&answer));+```++For live text, set `stream: true` in the initial `LlmRequest` before starting the+run. Use a streaming-capable adapter; the quickstart's OpenRouter adapter returns+buffered responses. Handle both response forms so a buffered answer is still shown:++```rust+use std::io::Write;++match response.llm_response {+ LlmResponse::Stream(mut events) => {+ while let Some(event) = events.next().await {+ for chunk in event?.normalized() {+ if let LlmResponseChunk::TextDelta { text, .. } = chunk {+ print!("{text}");+ std::io::stdout().flush()?;+ }+ }+ }+ }+ LlmResponse::Agg(answer) => println!("{}", completion_text(&answer)),+}+```++These are alternative ways to consume a response. The text-only loop omits tool+and usage events; handle those chunks too when your application needs them.++`outcome.state` lets supporting algorithms retain caller-managed state between+turns. See [Composite sessions](routing_algorithms/composite_routing.md#sessions)+for how this differs from a session ID.+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/struct.OutcomeMetadata.html"><code>OutcomeMetadata</code></a>+identifies the run and algorithm and may include evidence for the decision.++## Control how work runs++Each call has its own reply channel. Awaiting calls one at a time in the step loop+serves them sequentially. To run calls concurrently, your application must keep+reading steps while polling their work.++For concurrent execution, use+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/fn.drive.html"><code>libsy::drive</code></a>.+It polls your call handlers concurrently and returns the `RoutingOutcome`. Your+handler receives+<a href="https://nvidia-nemo.github.io/Switchyard/rust/switchyard_libsy/enum.Call.html"><code>Call::Model</code> or <code>Call::Decision</code></a>;+`drive` handles `Done` itself:++```rust+// Borrow the adapters so every call can use them.+let client = &client;+let decision_client = &decision_client;+let outcome = switchyard_libsy::drive(algorithm, request, models, |call| async move {+ match call {+ Call::Model(call) => {+ let request = call.request.clone();+ call.respond(client.call(request)).await+ }+ Call::Decision(call) => {+ let request = call.request.clone();+ call.respond(decision_client.call(request)).await+ }+ }+}).await?;+```++It starts with default state. Use `run_stream` directly to supply your own state+or execution loop.++If the algorithm stops waiting, `respond` drops that call's work future. Dropping+the step stream aborts the algorithm task; your application cleans up separately+started work. `drive` drops its outstanding handlers when the run ends or is dropped.++A model-call failure normally stops the host run when propagated with `?`.+`call.recover_errors` lets the algorithm receive and handle it instead. Decision+failures passed through `respond` also return to the algorithm. Dropping an+unanswered call gives the waiting algorithm `DriverError::ResponseDropped`.++Once a response stream has been delivered, its consumer owns its lifetime.+Cancelling local work cannot guarantee that a provider stops generation or billing,+or undo side effects that have already occurred. The [host integration+guide](reference/host_integration.md#concurrent-calls-and-cancellation) covers host+cleanup in Rust and Python.++The [Rust API reference](reference/rust_api.md) links the complete crate+documentation for these types.diff --git a/docs/reference/host_integration.md b/docs/reference/host_integration.mdindex 60db01d5b..1eaa503f6 100644--- a/docs/reference/host_integration.md+++ b/docs/reference/host_integration.md@@ -2,6 +2,9 @@Use this guide when embedding libsy algorithms that ask the host to serve model or decision calls. For a first routing result and model-client handoff, start with the [libsy quickstart](../getting_started.md).+For an explanation of steps, call objects, and the request and response contracts,+read [Mechanics of Libsy](../mechanics_of_libsy.md).+## Development APIThis page uses the development API, including its asynchronous `respond` method. The Rust examples pin the same revision as the [quickstart](../getting_started.md). The released `v0.3.0` callback API is synchronous. Use matching source-built [Python bindings](../../INSTALLATION.md#development) for this page's Python callback example.diff --git a/mkdocs.yml b/mkdocs.ymlindex 376fd9333..708426da9 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -19,6 +19,7 @@ nav:- Home: index.md- Get Started:- Getting Started: getting_started.md+ - Mechanics of Libsy: mechanics_of_libsy.md- Evaluate Locally: operations/local_evaluation.md- Known Issues: known_issues.md- Benchmarks: benchmarks.md