Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs(rust): repair libsy and protocol references (#263)
nachiketb-nvidiae0127808c0c9
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.ymlindex 0f07c1727..3fda6c7bb 100644--- a/.github/workflows/docs.yml+++ b/.github/workflows/docs.yml@@ -54,8 +54,13 @@ jobs:- run: uv sync --only-group docs --locked- name: Build documentationrun: uv run --only-group docs mkdocs build --strict- - name: Build libsy crate documentation- run: cargo doc --locked --no-deps --package switchyard-libsy+ - name: Build Rust crate documentation+ env:+ RUSTDOCFLAGS: -D warnings+ run: >-+ cargo doc --locked --no-deps+ --package switchyard-libsy+ --package switchyard-protocol- name: Add crate documentation to siterun: |mkdir -p site/rustdiff --git a/crates/libsy/Cargo.toml b/crates/libsy/Cargo.tomlindex dc2be5249..ce34afe2f 100644--- a/crates/libsy/Cargo.toml+++ b/crates/libsy/Cargo.toml@@ -4,12 +4,17 @@[package]name = "switchyard-libsy"version.workspace = true-description = "Switchyard library crate"+description = "Provider-neutral multi-LLM routing and orchestration for Switchyard"authors.workspace = trueedition.workspace = true+homepage = "https://github.com/NVIDIA-NeMo/Switchyard"license.workspace = true+readme = "README.md"repository.workspace = truerust-version.workspace = true+keywords = ["llm", "routing", "orchestration", "openai", "anthropic"]+categories = ["asynchronous", "web-programming"]+publish = ["crates-io"][dependencies]async-trait.workspace = truediff --git a/crates/libsy/README.md b/crates/libsy/README.mdindex 5ac6ac835..3f5def27f 100644--- a/crates/libsy/README.md+++ b/crates/libsy/README.md@@ -1,343 +1,181 @@-# libsy — Switchyard-Lib--A lightweight, provider-agnostic library for multi-LLM agent optimization, with-**routing** as the first case. libsy *decides* how to serve each request — which-model(s) to call, in what order, how to combine them — and either makes the calls-itself or hands them back for you to make. It owns no HTTP client and no provider SDK,-so it drops into a proxy, gateway, or agent runtime.+# switchyard-libsy++Provider-neutral orchestration for multi-LLM optimization. A libsy+[`Algorithm`] decides which model targets to call, in what order, and how to+combine their results. It can use target-owned clients or hand each call back to+the host, allowing it to embed in proxies, gateways, and agent runtimes without+owning an HTTP stack.++## Setup++```toml+[dependencies]+async-trait = "0.1"+futures = "0.3"+switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }+switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }+tokio = { version = "1", features = ["macros", "rt"] }+```-## Example+## Quick start-Build the targets, pick an algorithm, run a request:+This complete `src/main.rs` routes a request with [`LlmTaskClassifier`]. The+first [`Algorithm::run`] lets libsy call each target's client. The second drives+[`Algorithm::run_stream`] so the host can perform or override each model call+itself.```rust-use switchyard_libsy::{Algorithm, Context, RoutedLlmClient, LlmTarget, Request};-use switchyard_libsy::algorithms::{LlmTaskClassifier, TaskClassifierConfig};-use switchyard_protocol::{completion_text, text_request};use std::sync::Arc;-// Targets the algorithm routes among, each backed by your RoutedLlmClient (see below).-let client = Arc::new(MyClient { /* .. */ }) as Arc<dyn RoutedLlmClient>;-let target = |name: &str| LlmTarget { semantic_name: name.into(), llm_client: Some(client.clone()) };--let algo: Arc<dyn Algorithm> = Arc::new(LlmTaskClassifier::new(- target("classifier"), // the judge- target("weak"), // efficient tier- target("strong"), // capable tier- // Every field but `base_threshold` has a default; see "Tuning the classifier" below.- TaskClassifierConfig { base_threshold: 0.5, ..Default::default() },-)?);--let req = Request {- // `text_request` is the single-turn shortcut; build an `LlmRequest` directly for- // multi-turn conversations, tools, or sampling parameters.- llm_request: text_request(Some("auto".into()), "explain tail latency"),- raw_request: None,- metadata: None,+use async_trait::async_trait;+use futures::StreamExt;+use switchyard_libsy::{+ Algorithm, LibsyError, LlmTarget, LlmTaskClassifier, Step, TaskClassifierConfig,+};+use switchyard_protocol::{+ AggLlmResponse, ContentBlock, Context, Decision, LlmClientError, LlmRequest,+ LlmResponse, Message, Request, Response, ResponseOutput, Role, RoutedLlmClient,};-let (trace, response) = algo.clone().run(Context::default(), req).await?; // calls in, trace + response out-println!("routed to {}", trace.last().unwrap().selected_model());-// `response.llm_response` is buffered or streamed; fold it to the aggregate to read text.-println!("answer: {}", completion_text(&response.llm_response.into_agg().await?));-```--Runnable: [`research_agent`](examples/research_agent.rs)--## Requests & responses--```rust-pub struct Request {- pub llm_request: LlmRequest,- pub raw_request: Option<serde_json::Value>, // optional original provider body for exact-fidelity hosts- pub metadata: Option<Metadata>, // correlation: session / agent / task / correlation_id / extra-}--// `Response` is a plain struct. The buffered-or-streamed choice lives one level down,-// in its `llm_response` — which *is* an enum.-pub struct Response {- pub llm_response: LlmResponse,- pub metadata: Option<Metadata>,-}--pub enum LlmResponse {- Stream(LlmResponseStream), // a live stream of `LlmResponseChunk`s (token-by-token)- Agg(AggLlmResponse), // the buffered aggregate — outputs, usage, stop reason-}-```--`switchyard-protocol` owns the conversation IR: `LlmRequest`, the buffered-`AggLlmResponse`, the streaming `LlmResponseChunk`, and the building blocks `Message`,-`ContentBlock`, `ResponseOutput`, `Usage`, and `Role`. `libsy` re-exports the envelope-(`Request`/`Response`/`Metadata`) plus `LlmRequest`, `LlmResponse`, `AggLlmResponse`,-`LlmResponseChunk`, `LlmResponseStream`, and `Usage`; import the rest (`Message`, `Role`, …)-and the `text_request` / `text_response` / `completion_text` helpers from-`switchyard_protocol`. Construct and inspect the conversation model directly so tools,-sampling parameters, reasoning, and provider extensions stay visible instead of being hidden-behind a second convenience API. `raw_request` remains available when a host needs exact-source-body fidelity.--## Targets and clients--An `LlmTarget` pairs a routing `semantic_name` with an optional `RoutedLlmClient`. Mapping that-name to a provider model id is the client's job, not the algorithm's — `RoutedLlmClient` is-meant to be implemented by you.-```rust-struct MyClient { /* http client, base url, key */ }+struct DemoClient;-#[async_trait::async_trait]-impl RoutedLlmClient for MyClient {- async fn call(&self, ctx: Context, request: Request, decision: Arc<dyn Decision>)- -> Result<Response, LlmClientError> {- let model = decision.selected_model(); // the routed target — map it to a provider id- // request.llm_request.model is the agent's original name (not a call target)- // ... POST to your endpoint, read the completion ...- let completion = String::from("provider response text");+#[async_trait]+impl RoutedLlmClient for DemoClient {+ async fn call(+ &self,+ _ctx: Context,+ _request: Request,+ decision: Arc<dyn Decision>,+ ) -> Result<Response, LlmClientError> {+ let model = decision.selected_model();+ let text = if model == "judge" {+ r#"{+ "recommended_route": "efficient",+ "p_solve": 0.9,+ "confidence": 0.9,+ "abstain": false,+ "capability_boundary": "supported",+ "primary_rule": "SUP-1",+ "crux": "bounded task"+ }"#+ .to_string()+ } else {+ format!("answer from {model}")+ };Ok(Response {- // Buffered: wrap the aggregate in `LlmResponse::Agg`. `text_response` builds a- // single-turn `AggLlmResponse`; construct one directly for tool calls, usage, etc.- llm_response: LlmResponse::Agg(text_response(None, completion)),+ llm_response: LlmResponse::Agg(AggLlmResponse {+ model: Some(model.to_string()),+ outputs: vec![ResponseOutput {+ role: Role::Assistant,+ content: vec![ContentBlock::Text { text }],+ stop_reason: None,+ }],+ ..AggLlmResponse::default()+ }),metadata: None,})}}-```--To stream instead, return `LlmResponse::Stream(..)` — a boxed-`Stream<Item = Result<LlmResponseChunk, _>>` — and emit chunks as they arrive from your-upstream. See [Streaming responses](#streaming-responses) below.--A `RoutedRequest` bundles the `request` with the routing `decision`, the target's-`default_client`, and the request's `ctx`; the model to call is-`decision.selected_model()`, never a mutated request field. `semantic_name` is the label an algorithm routes by; the client maps it to-the id it calls — they can differ (`"strong"` → `"openai/gpt-4o"`) or coincide.--## Running a request--Hold the algorithm as `Arc<dyn Algorithm>` and choose one of two entry points:--```rust-// run: libsy drives the request to completion, serving each call with the target's-// client, and returns (trace, response). Errors if a routed target has no client.-let (trace, response) = algo.clone().run(Context::default(), req).await?;-// run_stream: "ask, don't call" — you drive the stream and make the calls.-let stream = algo.clone().run_stream(Context::default(), req);-```--Under the hood every model call is *offloaded* to the request's `Step` stream; `run` is-the convenience that serves each one via the target's client. The step stream is bounded,-so pulling it paces the algorithm; each run is independent, so many run concurrently.--## Streaming responses--A `Response` carries an `LlmResponse` that is either buffered (`Agg`) or a live token-stream (`Stream`). An `RoutedLlmClient` — or an algorithm — chooses: return-`LlmResponse::Agg(..)` for a whole answer, or `LlmResponse::Stream(..)` to forward tokens-as they arrive. libsy never buffers a stream on your behalf; it flows through the algorithm-untouched, so `run` / `run_stream` hand back whatever was produced and **the caller-decides**:--```rust-use futures::StreamExt;--let (_trace, response) = algo.clone().run(Context::default(), req).await?;-match response.llm_response {- // Forward tokens as they arrive — e.g. re-encode each chunk to your client's SSE.- LlmResponse::Stream(mut chunks) => {- while let Some(chunk) = chunks.next().await {- let chunk: LlmResponseChunk = chunk?; // TextDelta / ToolCallDelta / Usage / MessageStop / ..- /* emit chunk */+#[tokio::main(flavor = "current_thread")]+async fn main() -> switchyard_libsy::Result<()> {+ let client = Arc::new(DemoClient);+ let target = |name: &str| LlmTarget {+ semantic_name: name.to_string(),+ llm_client: Some(client.clone()),+ };++ let router: Arc<dyn Algorithm> = Arc::new(LlmTaskClassifier::new(+ target("judge"),+ target("efficient"),+ target("capable"),+ TaskClassifierConfig {+ base_threshold: 0.5,+ ..TaskClassifierConfig::default()+ },+ )?);+ let request = Request {+ llm_request: LlmRequest {+ model: Some("auto".to_string()),+ messages: vec![Message {+ role: Role::User,+ content: vec![ContentBlock::Text {+ text: "Explain tail latency".to_string(),+ }],+ }],+ ..LlmRequest::default()+ },+ ..Request::default()+ };++ // `run` serves the judge and routed calls through their target clients.+ let (_, response) = router+ .clone()+ .run(Context::default(), request.clone())+ .await?;+ println!("run returned {}", response.selected_model().unwrap_or("unknown"));++ // `run_stream` exposes those calls so the host controls their transport.+ let stream = router.run_stream(Context::default(), request, None);+ tokio::pin!(stream);+ while let Some(step) = stream.next().await {+ match step? {+ Step::CallLlm(call) => {+ let routed = call.get_routed().clone();+ let target = routed.decision.selected_model().to_string();+ let result = client+ .call(routed.ctx, routed.request, routed.decision)+ .await+ .map_err(|source| LibsyError::client_call(target, source));+ call.respond(result)?;+ }+ Step::Decision(decision) => {+ println!("run_stream chose {}", decision.selected_model());+ }+ Step::ReturnToAgent(response) => {+ println!(+ "run_stream returned {}",+ response.selected_model().unwrap_or("unknown")+ );+ }}}- // Already buffered — read it directly.- LlmResponse::Agg(agg) => println!("{}", completion_text(&agg)),+ Ok(())}```-Need the whole answer regardless of how it arrived? `into_agg()` folds a `Stream` (or-returns an `Agg` unchanged) into a single `AggLlmResponse`, surfacing any mid-stream error:--```rust-let agg = response.llm_response.into_agg().await?; // drives the stream to completion-println!("{}", completion_text(&agg));-```--An `LlmResponseChunk` is the provider-neutral streaming event (`MessageStart`, `TextDelta`,-`ReasoningDelta`, `ToolCallDelta`, `Usage`, `MessageStop`, `Error`); `ResponseAccumulator`-is the fold behind `into_agg()` if you need to assemble one yourself. Runnable end-to-end-demo: [`streaming_agent`](examples/streaming_agent.rs).--## Driving the calls yourself (`run_stream`)--`run_stream` yields `CallLlm` promises you fulfill with your own transport (or the call's-`default_client`), streams each `Decision` as it happens, and ends with `ReturnToAgent`.-This is the *step* stream (one `Step` at a time); it is orthogonal to whether any single-response is itself streamed — see [Streaming responses](#streaming-responses).-Runnable: [`research_agent_core`](examples/research_agent_core.rs).--```rust-let stream = algo.clone().run_stream(Context::default(), req);-tokio::pin!(stream);-while let Some(step) = stream.next().await {- match step? {- Step::CallLlm(call) => {- let routed = call.get_routed().clone(); // which target, and its default client- let response = call_model(routed.decision.selected_model(), &routed.request).await; // your real call- call.respond(Ok(response))?; // or Err(..) to propagate a failure- }- Step::Decision(decision) => { /* decision.selected_model(), decision.reasoning() */ }- Step::ReturnToAgent(response) => { /* done */ }- }-}-```--## Building an algorithm (`Algorithm`)--Implement `Algorithm` to add a strategy. You write `create_run_task` — one call per-request; `run` / `run_stream` are provided and drive it. Make model calls on the `Driver`-you're handed, and publish a `Decision` for each so consumers (and clients) see *which*-model and *why*.--```rust-#[async_trait]-pub trait Algorithm: Send + Sync + 'static {- // Stable telemetry label: the `algorithm` attribute on libsy's spans/metrics/logs.- fn name(&self) -> &str;- // `self: Arc<Self>` (not `&mut`): one algorithm serves requests concurrently — use- // interior mutability for state. Offload calls/decisions on `driver`.- async fn create_run_task(self: Arc<Self>, ctx: Context, driver: Driver, request: Request)- -> switchyard_libsy::Result<Response>;- async fn process_signals(self: Arc<Self>, signals: Signals)- -> switchyard_libsy::Result<()>;- // provided: run(ctx, request) -> (trace, response), run_stream(ctx, request) -> Stream<Step>-}--pub trait Decision: Send + Sync {- fn selected_model(&self) -> &str; // the model chosen — the client's call target- fn reasoning(&self) -> Option<&str>; // human-readable "why"- fn as_any(&self) -> &dyn std::any::Any; // downcast to the concrete decision-}-```--`LlmTaskClassifier` owns its fall-through cascade: it calls the judge, applies its policy, and-then invokes the selected target:--```rust-let router = LlmTaskClassifier::new(- judge_target,- weak_target,- strong_target,- TaskClassifierConfig { base_threshold: 0.5, ..Default::default() },-)?;-```--### Tuning the classifier--The judge returns a `p_solve` — its estimate that the efficient tier completes the task-correctly — plus a confidence and a capability boundary. `TaskClassifierConfig` decides what-to do with that verdict. Anything invalid, abstained, or below a threshold routes to the-capable target, so every knob below trades cost against quality in the same direction.--| Field | Default | Meaning |-|---|---|---|-| `base_threshold` | *required* | Lowest `p_solve` that routes a supported task to the efficient target. Raise it to send less traffic to the cheap model. |-| `min_confidence` | `0.0` | Lowest judge confidence that permits efficient routing. `0.0` disables the gate. |-| `capability_elevated_floor` | `None` | A higher `p_solve` floor applied only when the judge marks the task `uncertain`, `unsupported`, or `unmatched`. `None` reuses `base_threshold`. |-| `session_affinity` | `false` | Retains the first decision per session and reuses it on later turns, so the judge is called once per session instead of once per turn. |-| `message_hash_fallback` | `false` | Extends affinity to clients that send no session header, keying on a hash of the first user message. Requires `session_affinity`. |--Two things to understand before enabling affinity. The retained decision is **sticky for the-process lifetime** — including a fail-closed `capable` verdict produced when the judge was-unreachable — so a transient judge failure pins that identity until restart. And-`message_hash_fallback` keys on request *content*, not a correlation id, so unrelated callers-sending identical text share one assignment.--## Errors--All libsy-owned APIs return [`switchyard_libsy::Result<T>`], whose error is-`LibsyError`. Callers can match routing failures (`TargetNotFound`, `NoTargets`,-`MissingClient`), algorithm-specific failures (`AlgorithmError`), driver failures,-algorithm task failures, incomplete runs, and client-call failures without inspecting error-strings.--`RoutedLlmClient` is owned by `switchyard-protocol` and returns-`LlmClientError`. Callers can distinguish invalid requests, configuration-failures, request/response translation failures, request encoding failures, transport-failures, timeouts, context-window overflows, upstream HTTP responses, invalid responses,-and uncategorized client-specific failures. When [`Algorithm::run`] serves a target,-libsy preserves that error as the source of `LibsyError::ClientCall`.-Custom algorithms, classifiers, and processors that need to surface their own errors can use-`LibsyError::external("operation", error)`.--## Observability--libsy instruments every algorithm at the core — the `Decision` hook plus the offload-boundary — so algorithms carry no telemetry code beyond `Algorithm::name()`, the-`algorithm` attribute everything below is keyed by.--- **Spans** (`tracing`): one `libsy.run` per request, carrying the `Metadata`- correlation ids, any host-defined `extra_metadata` labels, and the outcome,- with one child `libsy.llm_call` per model call- (selected model, outcome, token counts; measures *fulfillment* as the algorithm- observes it, host serving included — a streamed response resolves when its stream- handle arrives). When `run` serves a call via the target's default client, the- actual API call gets its own `libsy.client_call` span — hosts serving calls over- their own transport should span their `RoutedLlmClient` equivalently.-- **Structured logs** (`tracing`, target `libsy`): an info event per published- `Decision` (selected model + reasoning), warn events for failed calls and runs.-- **Metrics** (OpenTelemetry, scope `switchyard`, via the global meter provider): counters- `switchyard.runs`, `switchyard.llm_calls`, and `switchyard.decisions`; histograms- `switchyard.run_duration_ms` and `switchyard.llm_call_duration_ms`. Attributes are- `algorithm`, `selected_model`, and `outcome` (`ok`/`error`) — failure rates are the- `outcome="error"` share of runs/calls.--libsy also records compatibility metrics for final routed calls:--| OpenTelemetry instrument | Prometheus family | Type | Labels | Meaning |-|---|---|---|---|---|-| `switchyard.total_requests` | `switchyard_total_requests` | gauge | none | Successful and failed routed calls |-| `switchyard.total_errors` | `switchyard_total_errors` | gauge | none | Failed routed calls |-| `switchyard.requests` | `switchyard_requests_total` | counter | `model`, optional `tier` | Successful routed calls |-| `switchyard.errors` | `switchyard_errors_total` | counter | `model`, optional `tier` | Failed routed calls |-| `switchyard.model_call_latency_ms` | `switchyard_model_call_latency_ms` | histogram | `model`, optional `tier` | Successful routed-call latency |--Classifier calls remain visible in the general LLM-call instruments but are excluded from these-compatibility families. The `tier` label is present only when a decision supplies a stable tier.-Hosts that need the two zero-valued gauges before the first call must invoke-`libsy::initialize_metrics()` after installing the global meter provider.+## Built-in algorithms-libsy owns no exporter. The host installs an OpenTelemetry SDK meter provider-(`opentelemetry::global::set_meter_provider`) and a `tracing` subscriber (bridge spans-into OTel with `tracing-opentelemetry` if desired); with neither installed, all-instrumentation is a no-op.+| Type | Purpose |+|---|---|+| [`Passthrough`] | Always call one configured target. |+| [`Random`] | Select among any number of targets using uniform or weighted routing. |+| [`LlmTaskClassifier`] | Ask a judge model to choose an efficient or capable target. |+| [`StageRouter`] | Route coding-agent turns from tool and progress signals, with an optional judge fallback. |-## Explore+[`Noop`] is a test helper, not a production routing algorithm.-The core crate includes weighted random routing and a judge-backed LLM classifier. Runnable-agents live in [`examples`](examples/) folder.+## How it fits together-**Reference algorithms** — implementations to read and route with:+[`LlmTarget`] pairs a semantic routing name with an optional+[`RoutedLlmClient`](switchyard_protocol::RoutedLlmClient). An [`Algorithm`]+selects targets and records [`Decision`](switchyard_protocol::Decision)s. Use+[`Algorithm::run`] with target-owned clients, or [`Algorithm::run_stream`] when+the host owns model transport. The provider-neutral [`Request`], [`Response`],+[`Usage`], and [`LlmResponse`] contracts come from `switchyard-protocol`.-- [`Random`](src/algorithms/rand.rs) — uniform or weighted random over the set- (one call).-- [`LlmTaskClassifier`](src/algorithms/llm_class.rs) — task-level capability routing with an- internal [`FallThrough`](src/algorithms/fall_through.rs) cascade.-- [`EnsembleOrchAlgo`](examples/ensemble.rs) — stateful: fan out to- candidates, judge the best, commit to the winner after N exploration turns.+[`Request`]: switchyard_protocol::Request+[`Response`]: switchyard_protocol::Response+[`Usage`]: switchyard_protocol::Usage+[`LlmResponse`]: switchyard_protocol::LlmResponse-**Runnable agents** (`cargo run -p switchyard-libsy --example <name>`):+## Examples-- [`ensemble`](examples/ensemble.rs) — query three NVIDIA-hosted candidates, then judge and commit.-- [`research_agent`](examples/research_agent.rs) — client-backed- targets, `run` (libsy makes the calls).-- [`research_agent_core`](examples/research_agent_core.rs) — client-less- targets, `run_stream` (the agent makes the calls).-- [`streaming_agent`](examples/streaming_agent.rs) — a target that streams- its response; the agent forwards each `LlmResponseChunk` to the caller token-by-token.+- [Client-backed execution](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/examples/research_agent.rs)+- [Host-owned model calls](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/examples/research_agent_core.rs)+- [Streaming responses](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/examples/streaming_agent.rs)+- [Custom algorithm](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/examples/ensemble.rs)-## Not yet built+## License-- **`Signals` events** — `process_signals` / `Signals` exist but carry nothing yet.-- **`Context` fields** — `values: HashMap<String, String>` carries the algorithm telemetry label and `state: S` the caller's per-session state; correlation ids, budgets, and deadlines still to come.-- **Config-driven construction**.+Licensed under the Apache License, Version 2.0.diff --git a/crates/libsy/src/algorithms/llm_class.rs b/crates/libsy/src/algorithms/llm_class.rsindex 57726d2bc..fe87e0a91 100644--- a/crates/libsy/src/algorithms/llm_class.rs+++ b/crates/libsy/src/algorithms/llm_class.rs@@ -429,7 +429,12 @@ impl Classifier<State> for EscalationClassifier {}}-/// A task-level capability routing algorithm with an internal fall-through cascade.+/// Routes each task between an efficient and capable target using an LLM judge.+///+/// The judge is consulted before the routed model call. Valid, confident output+/// selects a tier; invalid output, abstention, or a judge failure falls back to the+/// capable target. Optional session affinity can retain a prior assignment and skip+/// the judge on later turns.pub struct LlmTaskClassifier {route: FallThrough<State>,/// The active classifier — either capability-based or escalation-based.@@ -437,7 +442,18 @@ pub struct LlmTaskClassifier {}impl LlmTaskClassifier {- /// Routes according to `config` and returns errors for invalid thresholds.+ /// Builds task-level capability routing over `efficient_target` and `capable_target`.+ ///+ /// `judge_target` serves the classification call and is not itself a routing+ /// destination. When enabled, session affinity runs before the judge and can+ /// short-circuit it with a retained assignment.+ ///+ /// # Errors+ ///+ /// Returns an error when thresholds are outside `[0, 1]`, the elevated floor is+ /// not above the base threshold, the output-token budget is zero, message-hash+ /// fallback is enabled without affinity, or the packaged judge prompt/schema+ /// cannot be loaded.pub fn new(judge_target: LlmTarget,efficient_target: LlmTarget,diff --git a/crates/libsy/src/algorithms/noop.rs b/crates/libsy/src/algorithms/noop.rsindex 86179a4fc..71118cd4f 100644--- a/crates/libsy/src/algorithms/noop.rs+++ b/crates/libsy/src/algorithms/noop.rs@@ -1,8 +1,7 @@// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.// SPDX-License-Identifier: Apache-2.0-//! No-op router. Always returns a hard coded response. Does not route to a backend.-//! For testing.+//! Test-only algorithm that returns a hard-coded response without calling a backend.use std::sync::Arc;@@ -14,11 +13,10 @@ use crate::Result;use crate::core::algorithm::{Algorithm, Driver};use switchyard_protocol::{Context, Decision};-/// A routing algorithm that does not route. It returns a hard-coded response.+/// Test helper that returns a hard-coded response without routing or model I/O.pub struct Noop {}-/// How [`Noop`] records which model it chose. This will be the model on the Request if any,-/// otherwise a hard coded placeholder. Neither is actually used.+/// Test decision carrying the inbound model or a fixed placeholder.pub struct NoopDecision {model: String,}diff --git a/crates/libsy/src/algorithms/passthrough.rs b/crates/libsy/src/algorithms/passthrough.rsindex 06b73a02d..2a922f28d 100644--- a/crates/libsy/src/algorithms/passthrough.rs+++ b/crates/libsy/src/algorithms/passthrough.rs@@ -1,9 +1,7 @@// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.// SPDX-License-Identifier: Apache-2.0-//! Router that only routes to a single backend, always.-//! - For testing if issues come from backend or from Switchyard-//! - For logging request / responses for RL+//! Single-target routing for direct model calls and integration diagnostics.use std::sync::Arc;@@ -13,17 +11,19 @@ use crate::Result;use crate::core::algorithm::{Algorithm, Driver, LlmTarget};use switchyard_protocol::{Context, Decision, RoutedLlmClient};-/// See module comment+/// Routing algorithm that always calls one configured target.pub struct Passthrough {target: LlmTarget,}impl Passthrough {+ /// Creates an algorithm that always calls `target`.pub fn new(target: LlmTarget) -> Self {Passthrough { target }}}+/// Decision emitted before [`Passthrough`] calls its configured target.pub struct PassthroughDecision {model_id: String,}diff --git a/crates/libsy/src/algorithms/stage.rs b/crates/libsy/src/algorithms/stage.rsindex 8e952a72e..9fade6a50 100644--- a/crates/libsy/src/algorithms/stage.rs+++ b/crates/libsy/src/algorithms/stage.rs@@ -13,8 +13,6 @@//! joined in unchanged — and then to the picker's default tier. The judge is//! asked per turn and its verdict is never pinned to the session.//!-//! Callers needing a different composition can assemble the parts themselves.-use std::sync::Arc;use async_trait::async_trait;diff --git a/crates/libsy/src/algorithms/util/escalation.rs b/crates/libsy/src/algorithms/util/escalation.rsindex 8a377462a..276d612ac 100644--- a/crates/libsy/src/algorithms/util/escalation.rs+++ b/crates/libsy/src/algorithms/util/escalation.rs@@ -185,8 +185,8 @@ pub(crate) fn build_judge(/// message is this turn — no `+ 1`. Counting the caller's request instead would report the/// turn after the one under judgement.///-/// libsy receives messages already normalized by `switchyard-protocol`, so unlike the-/// wire-format-aware Python equivalent this needs no per-format branching.+/// Messages are already normalized by `switchyard-protocol`, so this needs no+/// per-format branching.pub(crate) fn conversation_turn(request: &Request) -> usize {request.llm_requestdiff --git a/crates/libsy/src/algorithms/util/stage.rs b/crates/libsy/src/algorithms/util/stage.rsindex 1146a23b8..24e3deded 100644--- a/crates/libsy/src/algorithms/util/stage.rs+++ b/crates/libsy/src/algorithms/util/stage.rs@@ -3,11 +3,9 @@//! Stage-router scoring and tier selection — the shared routing core.//!-//! Given a [`ToolSignals`] (extracted by the dimension collector), this-//! module decides whether a coding-agent turn should go to the **capable**-//! or **efficient** tier. It is the single source of truth for-//! the decision, shared by the Rust profile and the Python processor via-//! bindings — only the outer shell differs in how it fetches the decision.+//! Given [`ToolSignals`] extracted from a normalized request, this module decides+//! whether a coding-agent turn should go to the **capable** or **efficient** tier.+//! [`StageClassifier`] applies that decision inside [`crate::StageRouter`].//!//! Two axes://!diff --git a/crates/libsy/src/algorithms/util/tool_signals.rs b/crates/libsy/src/algorithms/util/tool_signals.rsindex a2eb3b662..388417c45 100644--- a/crates/libsy/src/algorithms/util/tool_signals.rs+++ b/crates/libsy/src/algorithms/util/tool_signals.rs@@ -3,11 +3,9 @@//! Tool-result context signals extracted from the conversation history.//!-//! The [`DimensionCollector`][`crate::DimensionCollector`] runs this-//! alongside the 15 prompt-text scorers: it walks the `messages[]` /-//! `input[]` array of the incoming request, finds tool-execution results,+//! The extractor walks normalized messages, finds tool calls and results,//! pattern-matches their text against a curated error table, and aggregates-//! conversation-history metrics that the stage_router pickers need.+//! conversation-history metrics used by [`crate::StageRouter`].//!//! All logic is pure and deterministic — no I/O, no shared state.@@ -200,8 +198,10 @@ pub const DEFAULT_RECENT_WINDOW: usize = 3;// ─── output type ─────────────────────────────────────────────────────────────-/// Tool-execution signals stamped on `ProxyContext`. Read by stage_router pickers-/// via the `get_tool_result_signal` binding.+/// Tool-execution signals extracted from a normalized [`Request`].+///+/// A request-side processor stores these signals in [`State`](crate::State) for+/// [`crate::StageRouter`] and its classifier to consume.#[derive(Clone, Debug, Default)]pub struct ToolSignals {/// Max severity across the recent window (last `recent_window` tool results):@@ -211,7 +211,9 @@ pub struct ToolSignals {pub severity: f32,/// Consecutive clean tool results back from the most recent. `0` if the last failed.pub no_error_streak: u32,+ /// Total edit-style tool calls in the request.pub edit_count: u32,+ /// Total write-style tool calls in the request.pub write_count: u32,/// Read-type calls (Read tool + read-like Bash). Used by the build-pit gate.pub read_count: u32,@@ -244,6 +246,10 @@ pub struct ToolSignals {}impl ToolSignals {+ /// Extracts tool and progress signals from `request`.+ ///+ /// `window_size` limits recent counters to the newest tool results. `None`+ /// uses [`DEFAULT_RECENT_WINDOW`].pub fn from_request(request: &Request, window_size: Option<usize>) -> Self {extract_tool_signals_with_window(request, window_size.unwrap_or(DEFAULT_RECENT_WINDOW))}@@ -326,19 +332,10 @@ fn classify_tool_call(name: &str, command: Option<&str>) -> ToolCategory {// ─── extraction entry point ───────────────────────────────────────────────────-/// Extract all tool-execution signals from a [`Request`].-///-/// Dispatches on the request's wire format:-///-/// * **OpenAI chat** — `messages[]` with `role: "tool"` for results;-/// `role: "assistant"` with `tool_calls[]` for call names.-/// * **Anthropic** — `messages[]` with `role: "user"` + `content[].type:-/// "tool_result"`; `role: "assistant"` with `content[].type: "tool_use"`.-/// * **OpenAI responses** — `input[]` with `type: "function_call_output"`;-/// `type: "function_call"` for call names.+/// Extract all tool-execution signals from a normalized [`Request`].///-/// Returns [`ToolSignals::default()`] when the wire format or body is absent or-/// the messages list is empty — callers can always read `signal.severity`.+/// Returns [`ToolSignals::default()`] when the message history contains no tool+/// activity, so callers can always inspect the signal fields.fn extract_tool_signals_with_window(request: &Request, recent_window: usize) -> ToolSignals {// Read the decoded conversation, not the raw body: every inbound format lands// in the same shape here, so the signals do not depend on knowing which one itdiff --git a/crates/libsy/src/core/algorithm.rs b/crates/libsy/src/core/algorithm.rsindex 1e124ded3..69ac9d1b8 100644--- a/crates/libsy/src/core/algorithm.rs+++ b/crates/libsy/src/core/algorithm.rs@@ -2,8 +2,8 @@// SPDX-License-Identifier: Apache-2.0//! The [`Algorithm`] trait and its [`Driver`] — the orchestration contract every-//! routing/optimization algorithm implements, and the offload channel it makes model-//! calls and publishes [`Decision`]s over. See the crate root for the narrative model.+//! algorithm implements, and the offload channel it uses to make model calls and+//! publish [`Decision`]s.use std::{collections::{HashMap, HashSet},@@ -36,11 +36,13 @@ pub type StepStream = Pin<Box<dyn Stream<Item = Result<Step>> + Send>>;/// One completed model call observed at the algorithm offload boundary.#[derive(Clone, Debug)]pub struct LlmCallObservation {+ /// Model selected for the completed call.pub selected_model: String,/// Routing tier attached to the selected model, when present.pub tier: Option<String>,/// Whether this was the routed backend call rather than classifier or judge overhead.pub is_routed: bool,+ /// Whether the call completed successfully.pub is_success: bool,/// Time spent waiting for the model call to resolve.pub duration: Duration,@@ -58,6 +60,10 @@ pub enum RunObservation {}/// Request-scoped callback for algorithm-run observations.+///+/// The runner invokes this callback inline while resolving observations. Several+/// runs may invoke the same observer concurrently, so implementations must be+/// thread-safe, fast, and non-blocking.pub type RunObserver = Arc<dyn Fn(RunObservation) + Send + Sync>;/// A request paired with the routing [`Decision`] that produced it — the offload@@ -411,7 +417,7 @@ impl LlmTargetSet {}}-/// Matches the cap the Python stack uses. Dropping a live session's entry costs one+/// Bounds process-local overflow history. Dropping a live session's entry costs one/// rediscovered overflow, so the victim choice does not need to be exact.const MAX_EVICTION_SESSIONS: usize = 1_024;@@ -535,11 +541,25 @@ pub(crate) async fn call_llm_with_overflow_fallback(////// Methods take `self: Arc<Self>`: one algorithm (`Arc<dyn Algorithm>`) is shared across/// requests and run concurrently, so it owns its thread-safety and any shared state.+///+/// # Concurrency+///+/// A host may run the same algorithm concurrently for many requests. Implementations+/// must synchronize their own mutable shared state. Each call to [`run_stream`](Self::run_stream)+/// creates an independent [`Driver`], so model-call promises and emitted [`Step`]s cannot+/// cross between runs.+///+/// # Observability+///+/// [`run_stream`](Self::run_stream) creates a `libsy.run` span, and each offloaded model+/// call creates a `libsy.llm_call` span. [`run`](Self::run) additionally wraps calls it+/// serves through a target's default client in `libsy.client_call`. Decisions and failures+/// are emitted through `tracing`; metrics use the global OpenTelemetry meter provider.#[async_trait]pub trait Algorithm: Send + Sync + 'static {/// Stable, low-cardinality name identifying this algorithm — the/// `algorithm` attribute on every span, metric, and log line the crate- /// emits for its runs (see the crate docs' Observability section).+ /// emits for its runs.fn name(&self) -> &str;/// Run one request to completion: make model calls with [`Driver::call_llm_target`],@@ -596,9 +616,14 @@ pub trait Algorithm: Send + Sync + 'static {}/// Process a request to completion, returning a stream of [`Step`]s.- /// Each [`Step::CallLlm`] is an offloaded model call the consumer must serve.- /// The stream ends with a [`Step::ReturnToAgent`] on success, or an `Err` item on failure.- /// Report each model call to `observer`.+ ///+ /// The consumer must fulfill every [`Step::CallLlm`] before the algorithm can+ /// continue. The bounded step channel applies backpressure when the consumer is+ /// not polling. A successful run ends with [`Step::ReturnToAgent`]; a failure is+ /// emitted as an `Err` item. Dropping the stream aborts the spawned algorithm task.+ ///+ /// Every invocation owns a separate [`Driver`]. `observer`, when present, receives+ /// each completed model call and, after a successful routed run, its routing overhead.fn run_stream(self: Arc<Self>,ctx: Context,@@ -632,7 +657,7 @@ pub trait Algorithm: Send + Sync + 'static {}.instrument(span),);- // Dropping the stream aborts the algorithm task, so it doesn't keep running after the+ // Dropping the stream aborts the algorithm task when its consumer goes away.let abort_guard = AbortOnDrop(handle.abort_handle());let finish_driver = driver.clone();diff --git a/crates/libsy/src/core/classifier.rs b/crates/libsy/src/core/classifier.rsindex c417622eb..ff2dc50f4 100644--- a/crates/libsy/src/core/classifier.rs+++ b/crates/libsy/src/core/classifier.rs@@ -77,7 +77,9 @@ pub trait Classifier<S = ()>: Send + Sync {}/// Score the classifier's targets given the current state and request.- /// driver is optional. It is used to offload model calls+ ///+ /// When present, `driver` lets a classifier offload model calls. It is `None`+ /// when the classifier is evaluated outside an algorithm run.////// `request` is borrowed mutably so a classifier may rewrite it in place — inject a/// system prompt, drop tools, compact history. The edit is not scoped to this call:diff --git a/crates/libsy/src/core/state.rs b/crates/libsy/src/core/state.rsindex f35d4d73b..01ccb08a9 100644--- a/crates/libsy/src/core/state.rs+++ b/crates/libsy/src/core/state.rs@@ -10,19 +10,25 @@ use crate::algorithms::util::tool_signals::ToolSignals;/// A value in a session's [`State`].#[derive(Debug, Clone)]pub enum StateValue {+ /// Text value.String(String),+ /// Nonnegative counter.Count(u32),+ /// Signed integer value.Int(i32),+ /// Floating-point value.Scalar(f32),}/// Routing facts accumulated across one session's algorithm runs.#[derive(Debug, Clone, Default)]pub struct State {+ /// Number of turns processed for this session.pub turn_count: u32,/// Tool-result signals for the current request, set by the tool-signal/// processor. `None` until it runs or when the request has no tool activity,/// so routers must treat absence as "no signal yet".pub tool_signals: Option<ToolSignals>,+ /// Algorithm-specific state keyed by stable internal names.pub extra: HashMap<String, StateValue>,}diff --git a/crates/libsy/src/lib.rs b/crates/libsy/src/lib.rsindex bff81c6d7..3d4188631 100644--- a/crates/libsy/src/lib.rs+++ b/crates/libsy/src/lib.rs@@ -1,80 +1,8 @@// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.// SPDX-License-Identifier: Apache-2.0-//! # libsy — multi-LLM agent optimization (routing first)-//!-//! `libsy` decides, per request, *how* to serve an LLM call: which model(s) to-//! invoke, in what order, and how to combine the results. Routing is the first-//! and simplest case; the same interfaces also express classifier routing,-//! ensembles, cascades, and other optimizations. The library owns no HTTP client-//! and no provider SDK — it decides, and the host makes (or is asked to make) the-//! actual calls — so it embeds cleanly in a proxy, gateway, or agent runtime.-//!-//! ## The model-//!-//! - An [`Algorithm`] is the optimization *algorithm*. Its-//! [`create_run_task`](Algorithm::create_run_task) runs once per request-//! and makes as many model calls as it needs — via [`Driver::call_llm_target`], which look-//! like ordinary calls — publishes its [`Decision`](switchyard_protocol::Decision)s with [`Driver::info`], and-//! returns the final [`Response`](switchyard_protocol::Response). The provided-//! [`run_stream`](Algorithm::run_stream) drives that on its own task and hands-//! back a stream of [`Step`]s; [`run`](Algorithm::run) runs-//! it to completion with the targets' default clients.-//! - An [`LlmTarget`] names a routing target by its [`semantic_name`](LlmTarget::semantic_name).-//! Every call is *offloaded* to the request's stream as a [`Step::CallLlm`]; the-//! target's [`RoutedLlmClient`](switchyard_protocol::RoutedLlmClient), if any, rides along as-//! [`RoutedRequest::default_client`] so the host can serve it by default or-//! override it (see below).-//!-//! ## Running a request-//!-//! Hold the algorithm as `Arc<dyn Algorithm>` and call one of two provided methods:-//!-//! - [`run`](Algorithm::run) — run to completion, serving each-//! offloaded call via its [`RoutedRequest::default_client`], and return the decision-//! trace plus the final [`Response`](switchyard_protocol::Response). The simplest integration; use it when the-//! algorithm holds the model clients (it errors if a routed target has no client).-//! - [`run_stream`](Algorithm::run_stream) — return a stream of [`Step`]s. Each-//! model call is offloaded: the stream yields a [`Step::CallLlm`] carrying a promise;-//! the host performs the real model call (optionally via the promise's-//! `default_client`) and fulfills it with [`CallLlmRequest::respond`]. Decisions-//! arrive as [`Step::Decision`] as the algorithm makes them, and the run ends with a-//! [`Step::ReturnToAgent`] carrying the final response. The step stream is bounded,-//! so pulling it paces the algorithm one step at a time — an "ask, don't call" mode-//! that lets a host that owns its transport keep control of every call.-//!-//! ## Concurrency-//!-//! [`Algorithm::create_run_task`] takes `self: Arc<Self>`, so one shared-//! `Arc<dyn Algorithm>` (no lock) serves many requests in parallel. Each-//! [`run_stream`](Algorithm::run_stream) call builds its own [`Driver`], so-//! offloaded calls never cross between concurrent requests. An algorithm is-//! responsible for its own thread-safety — stateless (like the reference routers) or-//! interior mutability over just its own state.-//!-//! ## Observability-//!-//! The provided run methods instrument every algorithm from the outside — at the-//! [`Decision`](switchyard_protocol::Decision) hook and the offload boundary — so algorithms carry no telemetry-//! code. Each run gets a `libsy.run` tracing span (correlation ids from-//! [`Metadata`](switchyard_protocol::Metadata) attached) with a child `libsy.llm_call` span per model call-//! (fulfillment time as the algorithm observes it) plus a `libsy.client_call`-//! span around the actual API call when [`run`](Algorithm::run) serves it;-//! each [`Driver::info`] decision is logged with its reasoning; and OpenTelemetry-//! metrics record run/call counts, latency, and published decisions, keyed by-//! [`Algorithm::name`] plus `selected_model` and `outcome`.-//! Metrics use the global meter provider and spans/logs the `tracing` facade: a-//! host that installs an OTel SDK and a `tracing` subscriber (bridged with-//! `tracing-opentelemetry` for OTLP spans) gets the full signal set; with-//! neither installed, everything is a no-op.-//!-//! ## Algorithms-//!-//! Concrete algorithms live in [`algorithms`]:-//!-//! [`algorithms::Random`] provides uniform or weighted random routing.-//!-//! [`algorithms::LlmTaskClassifier`] uses one model to classify and route to its selected target.+#![warn(missing_docs)]+#![doc = include_str!("../README.md")]mod core;pub use core::algorithm::{@@ -112,7 +40,9 @@ mod observability;/// Registers process-wide compatibility gauges with the global meter provider.///-/// Hosts should call this after installing their OpenTelemetry meter provider.+/// Hosts should call this after installing their OpenTelemetry meter provider. Other+/// libsy metrics are created when recorded and do not require initialization. Spans+/// and structured logs require the host to install a `tracing` subscriber separately.pub fn initialize_metrics() {observability::initialize_metrics();}diff --git a/crates/protocol/Cargo.toml b/crates/protocol/Cargo.tomlindex c249dfa66..d054df339 100644--- a/crates/protocol/Cargo.toml+++ b/crates/protocol/Cargo.toml@@ -12,7 +12,6 @@ license.workspace = truereadme = "README.md"repository.workspace = truerust-version.workspace = true-documentation = "https://docs.rs/switchyard-protocol"keywords = ["llm", "protocol", "openai", "anthropic"]publish = ["crates-io"]diff --git a/crates/protocol/README.md b/crates/protocol/README.mdindex 6b41c950a..555debc33 100644--- a/crates/protocol/README.md+++ b/crates/protocol/README.md@@ -1,13 +1,111 @@# switchyard-protocol-Provider-neutral request, response, content, and wire-format types shared by Switchyard's Rust-crates.+Provider-neutral request, response, streaming, routing, and metadata types+shared by Switchyard algorithms, clients, and translation codecs. This crate+defines contracts; it does not route, translate, or perform network calls.-This is the schema dependency for [`switchyard-translation`](https://crates.io/crates/switchyard-translation).-Most applications should depend on `switchyard-translation`, which re-exports these types alongside-the translation engine.+## Setup++```toml+[dependencies]+switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }+serde_json = "1"+```++## Main types++| Area | Types |+|---|---|+| Conversation | [`LlmRequest`], [`Message`], [`InstructionBlock`], [`ContentBlock`] |+| Tools | [`ToolDefinition`], [`ToolChoice`], [`ToolCall`], [`ToolResult`] |+| Response | [`AggLlmResponse`], [`ResponseOutput`], [`Usage`], [`StopReason`] |+| Streaming | [`LlmResponse`], [`LlmResponseChunk`], [`LlmResponseStream`] |+| Envelope | [`Context`], [`Request`], [`Response`], [`Metadata`] |+| Routing I/O | [`Decision`], [`RoutedLlmClient`], [`LlmClientError`] |+| Wire identity | [`WireFormat`], [`FormatId`] |++## Simple request++```rust+use switchyard_protocol::{ContentBlock, LlmRequest, Message, Role};++let request = LlmRequest {+ model: Some("provider/model".into()),+ messages: vec![Message {+ role: Role::User,+ content: vec![ContentBlock::Text {+ text: "Explain tail latency".into(),+ }],+ }],+ ..LlmRequest::default()+};++assert_eq!(request.model.as_deref(), Some("provider/model"));+assert_eq!(request.messages.len(), 1);+```++## Detailed request++Construct the normalized [`Request`] directly when routing needs instructions,+tools, generation controls, and correlation metadata:++```rust+use serde_json::json;+use switchyard_protocol::{+ ContentBlock, InstructionBlock, LlmRequest, Message, Metadata, OutputParams,+ Request, Role, SamplingParams, ToolChoice, ToolDefinition,+};++let request = Request {+ llm_request: LlmRequest {+ model: Some("provider/model".into()),+ instructions: vec![InstructionBlock {+ role: Role::System,+ content: vec![ContentBlock::Text {+ text: "Answer with concise operational guidance.".into(),+ }],+ }],+ messages: vec![Message::text(Role::User, "Why is p99 latency rising?")],+ tools: vec![ToolDefinition {+ name: "lookup_metric".into(),+ description: Some("Read one service metric".into()),+ parameters: json!({+ "type": "object",+ "properties": { "name": { "type": "string" } },+ "required": ["name"]+ }),+ strict: Some(true),+ }],+ tool_choice: Some(ToolChoice::Auto),+ sampling: SamplingParams {+ temperature: Some(0.2),+ ..SamplingParams::default()+ },+ output: OutputParams {+ max_output_tokens: Some(512),+ ..OutputParams::default()+ },+ stream: true,+ ..LlmRequest::default()+ },+ metadata: Some(Metadata {+ session_id: Some("session-42".into()),+ correlation_id: Some("request-7".into()),+ ..Metadata::default()+ }),+ ..Request::default()+};++assert_eq!(request.llm_request.tools[0].name, "lookup_metric");+```++## Response forms++[`LlmResponse`] contains either a completed [`AggLlmResponse`] or a+single-consumption [`LlmResponseStream`] of [`LlmResponseChunk`] values. See+[`LlmResponse::into_agg`] for aggregation and [`PreservationMetadata`],+[`Usage`], and [`Metadata`] for the data retained around a response.## License-Licensed under the Apache License, Version 2.0. See the-[Switchyard repository](https://github.com/NVIDIA-NeMo/Switchyard) for details.+Licensed under the Apache License, Version 2.0.diff --git a/crates/protocol/src/client.rs b/crates/protocol/src/client.rsindex e496c07a3..233ddb38c 100644--- a/crates/protocol/src/client.rs+++ b/crates/protocol/src/client.rs@@ -30,7 +30,10 @@ pub type BoxError = Box<dyn std::error::Error + Send + Sync + 'static>;pub enum LlmClientError {/// The request cannot be served as supplied.#[error("invalid request: {message}")]- InvalidRequest { message: String },+ InvalidRequest {+ /// Human-readable request validation failure.+ message: String,+ },/// Decoding the inbound request failed in the translation engine.#[error("request translation failed: {0}")]@@ -46,7 +49,10 @@ pub enum LlmClientError {/// The client is not configured to serve the selected target.#[error("client configuration error: {message}")]- Configuration { message: String },+ Configuration {+ /// Human-readable configuration failure.+ message: String,+ },/// The upstream could not be reached or the request could not be sent.#[error("upstream transport error: {source}")]@@ -131,7 +137,9 @@ pub trait Decision: Send + Sync {/// A minimal [`Decision`] implementation for one-off calls that don't belong to a/// named algorithm step — judge side calls, classifier side calls, etc.pub struct SimpleDecision {+ /// Model or semantic target selected for the call.pub selected_model: String,+ /// Optional explanation recorded with the call.pub reasoning: Option<String>,}@@ -153,6 +161,12 @@ impl Decision for SimpleDecision {/// library does not own — a host implements it over its own transport (HTTP SDK,/// in-process model, mock). It serves a call the stream consumer chose not to/// override, reached as a routed request's `default_client`.+///+/// # Concurrency+///+/// A client may be shared by many targets and concurrent algorithm runs. Calls may+/// overlap, so implementations must synchronize mutable state internally and should+/// not serialize requests unless their transport requires it.#[async_trait]pub trait RoutedLlmClient: Send + Sync {/// Serve the call, returning the model's response. Call the model named bydiff --git a/crates/protocol/src/envelope.rs b/crates/protocol/src/envelope.rsindex 8399f936e..7139d9380 100644--- a/crates/protocol/src/envelope.rs+++ b/crates/protocol/src/envelope.rs@@ -40,20 +40,24 @@ impl Context {#[derive(Clone)]pub struct Signals {}-/// A request an algorithm routes: the normalized [`LlmRequest`] plus the original-/// provider payload and correlation [`Metadata`].+/// A request an algorithm routes: the normalized [`LlmRequest`] plus optional+/// host-owned raw data and correlation [`Metadata`].#[derive(Clone, Default)]pub struct Request {/// The normalized request an algorithm routes.pub llm_request: LlmRequest,- /// The original provider-shaped request body, if the host wants to forward it- /// verbatim (e.g. a proxy preserving messages/params). libsy does not read it.+ /// An optional whole request body retained by the host. libsy and the translation+ /// codecs do not read it. Exact same-format codec replay instead uses+ /// [`LlmRequest::preservation`].pub raw_request: Option<serde_json::Value>,/// Correlation metadata carried through the request.pub metadata: Option<Metadata>,}impl Request {+ /// Returns the model name supplied by the inbound request, when present.+ ///+ /// This is not necessarily the target selected by a routing decision.pub fn requested_model(&self) -> Option<&str> {self.llm_request.model.as_deref()}@@ -71,6 +75,10 @@ pub struct Response {}impl Response {+ /// Returns the model recorded by a buffered response.+ ///+ /// Returns `None` for a live stream because inspecting its `MessageStart`+ /// event would require consuming the stream.pub fn selected_model(&self) -> Option<&str> {self.llm_response.selected_model()}diff --git a/crates/protocol/src/format.rs b/crates/protocol/src/format.rsindex 5a24c9185..745f4120d 100644--- a/crates/protocol/src/format.rs+++ b/crates/protocol/src/format.rs@@ -11,10 +11,13 @@ use serde::{Deserialize, Serialize};/// Built-in provider API formats.#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize, Deserialize)]pub enum WireFormat {+ /// OpenAI Chat Completions API.#[serde(rename = "openai_chat")]OpenAiChat,+ /// Anthropic Messages API.#[serde(rename = "anthropic_messages")]AnthropicMessages,+ /// OpenAI Responses API.#[serde(rename = "openai_responses")]OpenAiResponses,}diff --git a/crates/protocol/src/lib.rs b/crates/protocol/src/lib.rsindex 543f2d0bb..2747a12e7 100644--- a/crates/protocol/src/lib.rs+++ b/crates/protocol/src/lib.rs@@ -1,24 +1,8 @@// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.// SPDX-License-Identifier: Apache-2.0-//! Shared request/response protocol for libsy — the vocabulary an [`Algorithm`] reasons-//! over, decoupled from libsy's orchestration.-//!-//! This crate owns Switchyard's neutral conversation IR: [`LlmRequest`] (model, messages,-//! tools, sampling, …), the buffered [`AggLlmResponse`] (outputs, usage, …), and its-//! streaming counterpart [`LlmResponseChunk`]; the [`Request`]/[`Response`] envelope that-//! pairs them with correlation [`Metadata`]; plus the wire-[`format`] identifiers-//! translation keys off. `switchyard-translation` re-exports the IR types under its own-//! `ConversationRequest` / `ConversationResponse` / `ConversationStreamEvent` names. The IR-//! carries no bare `prompt`/`completion`; the [`text_request`] / [`prompt_text`] /-//! [`text_response`] / [`completion_text`] helpers bridge to and from plain text for the-//! common single-turn case.-//!-//! The streamed-response type itself — a live stream of chunks *or* the terminal-//! aggregate — is the [`LlmResponse`] enum; it owns a `futures::Stream`, so it is the one-//! non-`Clone`, non-data type in this crate.-//!-//! [`Algorithm`]: https://docs.rs/libsy+#![warn(missing_docs)]+#![doc = include_str!("../README.md")]pub mod client;pub mod envelope;@@ -34,7 +18,10 @@ pub use llm::*;pub use metadata::*;pub use stream::*;-/// Build a single-turn request: one user message carrying `prompt`, for `model`.+/// Builds a single-turn request: one user message carrying `prompt`, for `model`.+///+/// This deliberately creates only the common text shape. Construct [`LlmRequest`]+/// directly for instructions, tools, multimodal content, or sampling controls.pub fn text_request(model: Option<String>, prompt: impl Into<String>) -> LlmRequest {LlmRequest {model,@@ -43,8 +30,10 @@ pub fn text_request(model: Option<String>, prompt: impl Into<String>) -> LlmRequ}}-/// The user's prompt text — the text of every user message, joined by newlines. Empty-/// when the request has no user text.+/// Returns a lossy text view of all user messages, joined by newlines.+///+/// Only text and refusal blocks are included. Instructions, tool content,+/// reasoning, and media are omitted. Returns an empty string when no user text exists.pub fn prompt_text(request: &LlmRequest) -> String {request.messages@@ -55,7 +44,10 @@ pub fn prompt_text(request: &LlmRequest) -> String {.join("\n")}-/// Build a single-turn response: one assistant message carrying `completion`, for `model`.+/// Builds a single-turn response: one assistant message carrying `completion`, for `model`.+///+/// Construct [`AggLlmResponse`] directly when usage, tools, reasoning, or multiple+/// output items must be represented.pub fn text_response(model: Option<String>, completion: impl Into<String>) -> AggLlmResponse {AggLlmResponse {model,@@ -70,8 +62,11 @@ pub fn text_response(model: Option<String>, completion: impl Into<String>) -> Ag}}-/// The assistant's completion text — the text blocks of the first output, concatenated.-/// Empty when the response has no textual output.+/// Returns a lossy text view of the first assistant output.+///+/// Only text blocks from the first output are concatenated. Refusals, reasoning,+/// tools, media, and additional outputs are omitted. Returns an empty string when+/// no such text exists.pub fn completion_text(response: &AggLlmResponse) -> String {response.outputsdiff --git a/crates/protocol/src/llm.rs b/crates/protocol/src/llm.rsindex cda3eb825..d8def175a 100644--- a/crates/protocol/src/llm.rs+++ b/crates/protocol/src/llm.rs@@ -14,24 +14,33 @@ use crate::format::FormatId;#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)]#[serde(rename_all = "snake_case")]pub enum Role {+ /// System-level instructions.System,+ /// Developer-level instructions supported by some provider APIs.Developer,+ /// End-user input.User,+ /// Model-generated output.Assistant,+ /// Tool execution output.Tool,}/// Instruction content separated from normal conversation messages.#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]pub struct InstructionBlock {+ /// Instruction authority, normally [`Role::System`] or [`Role::Developer`].pub role: Role,+ /// Ordered instruction content.pub content: Vec<ContentBlock>,}/// One normalized conversation message.#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]pub struct Message {+ /// Actor that produced the message.pub role: Role,+ /// Ordered message content.pub content: Vec<ContentBlock>,}@@ -67,32 +76,52 @@ impl Message {#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]#[serde(tag = "type", rename_all = "snake_case")]pub enum ContentBlock {+ /// Plain text content.Text {+ /// Text value.text: String,},+ /// Model reasoning or thinking content.Reasoning {+ /// Reasoning text.text: String,+ /// Provider signature used to validate or continue the reasoning block.signature: Option<String>,},+ /// Image content.Image {+ /// Image location or inline payload.source: ImageSource,},+ /// Audio content.Audio {+ /// Audio location or inline payload.source: MediaSource,},+ /// Video content.Video {+ /// Video location or inline payload.source: MediaSource,},+ /// File content.File {+ /// File identifier or inline payload.source: FileSource,},+ /// Tool invocation requested by the assistant.ToolCall(ToolCall),+ /// Result of an earlier tool invocation.ToolResult(ToolResult),+ /// Provider refusal content.Refusal {+ /// Human-readable refusal text.text: String,},+ /// Provider block that has no normalized representation.Unknown {+ /// Wire format that supplied the block.provider: FormatId,+ /// Exact provider block.raw: Value,},}@@ -101,14 +130,21 @@ pub enum ContentBlock {#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]#[serde(tag = "type", content = "data", rename_all = "snake_case")]pub enum ImageSource {+ /// Image fetched from a URL.Url {+ /// Image URL.url: String,+ /// Optional provider-specific image detail level.detail: Option<String>,},+ /// Base64-encoded image bytes.Base64 {+ /// MIME type, when supplied.media_type: Option<String>,+ /// Base64-encoded bytes.data: String,},+ /// Provider image source with no normalized representation.Raw(Value),}@@ -116,11 +152,16 @@ pub enum ImageSource {#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]#[serde(tag = "type", content = "data", rename_all = "snake_case")]pub enum FileSource {+ /// Provider-managed file identifier.FileId(String),+ /// Inline file payload.FileData {+ /// Encoded or textual file data as supplied by the provider format.data: String,+ /// Original filename, when supplied.filename: Option<String>,},+ /// Provider file source with no normalized representation.Raw(Value),}@@ -128,39 +169,56 @@ pub enum FileSource {#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]#[serde(tag = "type", content = "data", rename_all = "snake_case")]pub enum MediaSource {+ /// Media fetched from a URL.Url {+ /// Media URL.url: String,+ /// MIME type, when supplied.media_type: Option<String>,},+ /// Base64-encoded media bytes.Base64 {+ /// MIME type, when supplied.media_type: Option<String>,+ /// Base64-encoded bytes.data: String,},+ /// Provider media source with no normalized representation.Raw(Value),}/// Normalized assistant tool call.#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]pub struct ToolCall {+ /// Provider tool-call identifier used to pair the result.pub id: String,+ /// Tool name.pub name: String,+ /// Parsed tool arguments.pub arguments: Value,}/// Normalized tool result message content.#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]pub struct ToolResult {+ /// Identifier of the [`ToolCall`] this result answers.pub tool_call_id: String,+ /// Ordered tool output content.pub content: Vec<ContentBlock>,+ /// Whether tool execution failed, when the provider reports it.pub is_error: Option<bool>,}/// Normalized tool definition.#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]pub struct ToolDefinition {+ /// Tool name exposed to the model.pub name: String,+ /// Human-readable tool description.pub description: Option<String>,+ /// JSON Schema describing accepted arguments.pub parameters: Value,+ /// Whether the provider should enforce the schema strictly.pub strict: Option<bool>,}@@ -168,32 +226,47 @@ pub struct ToolDefinition {#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]#[serde(tag = "type", content = "data", rename_all = "snake_case")]pub enum ToolChoice {+ /// Let the model decide whether to call a tool.Auto,+ /// Require at least one tool call.Required,+ /// Disallow tool calls.None,- Tool { name: String },+ /// Require a specific tool.+ Tool {+ /// Required tool name.+ name: String,+ },+ /// Provider tool-choice value with no normalized representation.Raw(Value),}/// Provider sampling parameters with common cross-provider names.#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]pub struct SamplingParams {+ /// Sampling temperature.pub temperature: Option<f64>,+ /// Nucleus-sampling probability mass.pub top_p: Option<f64>,+ /// Maximum number of candidate tokens considered at each step.pub top_k: Option<i64>,}/// Output budget and structured-output options.#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]pub struct OutputParams {+ /// Maximum tokens the model may generate.pub max_output_tokens: Option<u64>,+ /// Provider-neutral or provider-specific structured-output configuration.pub response_format: Option<Value>,}/// Provider reasoning controls preserved by translation.#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]pub struct ReasoningParams {+ /// Requested reasoning effort or level.pub effort: Option<String>,+ /// Provider reasoning controls without a normalized field.pub raw: Option<Value>,}@@ -201,14 +274,25 @@ pub struct ReasoningParams {#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]#[serde(default)]pub struct ProviderExtensions {+ /// Provider fields keyed by their original wire names.+ ///+ /// Codecs use these fields when translating to another format. Exact+ /// same-format replay is controlled separately by [`PreservationMetadata`].pub fields: Map<String, Value>,}-/// Exact source payloads retained for lossless round trips.+/// Exact source payloads retained for lossless same-format round trips.+///+/// Translation's default preservation policy prefers a stored same-format body+/// over reconstructing one from normalized fields. A caller that mutates the IR+/// must clear the corresponding entry or use a policy with preservation disabled+/// when those mutations must be encoded.#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]#[serde(default)]pub struct PreservationMetadata {+ /// Original request bodies keyed by source format.pub requests: BTreeMap<FormatId, Value>,+ /// Original response bodies keyed by source format.pub responses: BTreeMap<FormatId, Value>,}@@ -216,16 +300,29 @@ pub struct PreservationMetadata {#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]#[serde(default)]pub struct LlmRequest {+ /// Model requested by the inbound client.pub model: Option<String>,+ /// System and developer instructions separated from conversation turns.pub instructions: Vec<InstructionBlock>,+ /// Ordered conversation messages.pub messages: Vec<Message>,+ /// Tools available to the model.pub tools: Vec<ToolDefinition>,+ /// Policy controlling model tool selection.pub tool_choice: Option<ToolChoice>,+ /// Common sampling controls.pub sampling: SamplingParams,+ /// Output budget and shape controls.pub output: OutputParams,+ /// Reasoning controls.pub reasoning: ReasoningParams,+ /// Whether the caller requested a streamed response.pub stream: bool,+ /// Provider fields without first-class normalized equivalents.pub extensions: ProviderExtensions,+ /// Exact provider bodies used by codecs for lossless same-format round trips.+ /// This is separate from a host's optional+ /// [`Request::raw_request`](crate::Request::raw_request).pub preservation: PreservationMetadata,}@@ -235,21 +332,31 @@ pub struct Usage {/// Non-cached input tokens. Provider codecs normalize aggregate OpenAI/// input counts by subtracting the cache detail fields.pub input_tokens: Option<u64>,+ /// Cache-read and cache-creation token detail, when reported.#[serde(flatten)]pub cache: Option<Box<InputCacheUsage>>,+ /// Generated output tokens, excluding reasoning detail when the provider reports it separately.pub output_tokens: Option<u64>,+ /// Provider-reported or codec-computed total token count.+ ///+ /// Codecs normalize OpenAI aggregate input counts before computing totals, so+ /// this may equal non-cached input plus cache detail plus output.pub total_tokens: Option<u64>,+ /// Reasoning output tokens, when reported separately.pub reasoning_tokens: Option<u64>,}/// Optional cache-token detail kept out of the common, cache-free usage allocation.#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]pub struct InputCacheUsage {+ /// Input tokens read from a provider cache.pub cached_input_tokens: Option<u64>,+ /// Input tokens written into a provider cache.pub cache_creation_input_tokens: Option<u64>,}impl Usage {+ /// Builds cache detail when at least one count is present.pub fn cache_details(cached_input_tokens: Option<u64>,cache_creation_input_tokens: Option<u64>,@@ -263,24 +370,28 @@ impl Usage {}))}+ /// Returns the cache-read input-token count.pub fn cached_input_tokens(&self) -> Option<u64> {self.cache.as_ref().and_then(|cache| cache.cached_input_tokens)}+ /// Returns the cache-creation input-token count.pub fn cache_creation_input_tokens(&self) -> Option<u64> {self.cache.as_ref().and_then(|cache| cache.cache_creation_input_tokens)}+ /// Sets the cache-read count, allocating cache detail when needed.pub fn set_cached_input_tokens(&mut self, value: u64) {self.cache.get_or_insert_with(|| Box::new(InputCacheUsage::default())).cached_input_tokens = Some(value);}+ /// Sets the cache-creation count, allocating cache detail when needed.pub fn set_cache_creation_input_tokens(&mut self, value: u64) {self.cache.get_or_insert_with(|| Box::new(InputCacheUsage::default()))@@ -292,34 +403,47 @@ impl Usage {#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)]#[serde(rename_all = "snake_case")]pub enum StopReason {+ /// Model completed its turn normally.EndTurn,+ /// Model reached the configured output limit.MaxTokens,+ /// Model stopped to request a tool call.ToolUse,+ /// Provider safety or content filtering stopped generation.ContentFilter,+ /// Generation terminated because of an error.Error,+ /// Provider stop reason with no normalized equivalent.Unknown,}/// One assistant output item in a normalized response.#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]pub struct ResponseOutput {+ /// Actor that produced the output, normally [`Role::Assistant`].pub role: Role,+ /// Ordered output content.pub content: Vec<ContentBlock>,+ /// Why this output stopped, when known.pub stop_reason: Option<StopReason>,}/// Normalized, fully-buffered response — the aggregate of a completed generation.-/// libsy refers to this as an `AggLlmResponse` (the terminal form of a streamed-/// [`LlmResponse`](crate::LlmResponse)); `switchyard-translation` re-exports it as-/// `ConversationResponse`.+/// This is the terminal form of a streamed [`LlmResponse`](crate::LlmResponse).#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]#[serde(default)]pub struct AggLlmResponse {+ /// Provider response identifier.pub id: Option<String>,+ /// Model reported by the provider.pub model: Option<String>,+ /// Ordered response output items.pub outputs: Vec<ResponseOutput>,+ /// Normalized token usage.pub usage: Usage,+ /// Provider response fields without normalized equivalents.pub extensions: ProviderExtensions,+ /// Exact provider bodies retained for lossless round trips.pub preservation: PreservationMetadata,}diff --git a/crates/protocol/src/metadata.rs b/crates/protocol/src/metadata.rsindex ae18b3cd7..890000e52 100644--- a/crates/protocol/src/metadata.rs+++ b/crates/protocol/src/metadata.rs@@ -4,8 +4,8 @@//! Correlation metadata and harness header normalization.//!//! [`Metadata`] is the correlation/routing envelope carried alongside a request or-//! response. [`Metadata::from_headers`] normalizes the harness-specific HTTP headers-//! that Claude Code, Codex, NeMo Relay, and Dynamo attach into that neutral shape.+//! response. [`Metadata::from_headers`] normalizes host-specific HTTP headers into+//! that neutral shape.use std::collections::BTreeMap;@@ -34,11 +34,11 @@ const SWITCHYARD_TURN_ID_HEADER: &str = "x-switchyard-turn-id";const SWITCHYARD_REQUEST_ID_HEADER: &str = "x-switchyard-request-id";const SWITCHYARD_SESSION_FINAL_HEADER: &str = "x-switchyard-session-final";-// NeMo Relay correlation headers.+// Correlation-header aliases used by integrating hosts.const RELAY_SESSION_ID_HEADER: &str = "x-nemo-relay-session-id";const RELAY_SUBAGENT_ID_HEADER: &str = "x-nemo-relay-subagent-id";-// Dynamo correlation headers.+// Additional correlation-header aliases used by integrating hosts.const DYNAMO_SESSION_ID_HEADER: &str = "x-dynamo-session-id";const DYNAMO_PARENT_SESSION_ID_HEADER: &str = "x-dynamo-parent-session-id";const DYNAMO_SESSION_FINAL_HEADER: &str = "x-dynamo-session-final";@@ -196,14 +196,14 @@ pub struct Metadata {impl Metadata {/// Normalizes harness-specific request headers into correlation metadata.///- /// Explicit `x-switchyard-*` headers win. NeMo Relay and Dynamo correlation- /// headers are accepted for observability without driving routing. Codex's- /// structured turn metadata is preferred over its compatibility projections.- /// Claude Code carries agent lineage in native headers: a request with a- /// non-empty `x-claude-code-agent-id` is treated as a child agent (its parent+ /// Explicit `x-switchyard-*` headers win. Compatible host headers can populate+ /// correlation and session metadata, but do not by themselves mark a request as+ /// delegated work. Structured turn metadata is preferred over compatibility+ /// projections. A request with a non-empty `x-claude-code-agent-id` is treated+ /// as a child agent (its parent/// inferred from the session when not stated). Sub-agent routing status is taken/// from an explicit `x-switchyard-is-subagent` header when present, and otherwise- /// inferred from Claude Code lineage or Codex harness signals.+ /// inferred from native agent lineage or structured harness signals.pub fn from_headers(headers: &BTreeMap<String, String>) -> Self {let headers = &normalize_headers(headers);@@ -240,10 +240,10 @@ impl Metadata {/// Returns `(parent_agent_id, is_subagent, is_delegated_work)` from the headers.///-/// Recognized sub-agent signals: Claude Code `x-claude-code-agent-id`, Codex-/// `x-openai-subagent` / `x-codex-turn-metadata.subagent_kind`, and explicit-/// `x-switchyard-is-subagent`. Correlation-only headers (Relay, Dynamo, OpenCode-/// parent sessions) populate observability fields but do not drive routing.+/// Recognized sub-agent signals include `x-claude-code-agent-id`,+/// `x-openai-subagent`, `x-codex-turn-metadata.subagent_kind`, and explicit+/// `x-switchyard-is-subagent`. Other host correlation and parent-session headers+/// may populate metadata but do not drive sub-agent classification.////// `is_delegated_work` is computed from raw harness signals, not from `agent_kind`,/// which may be set by an unrelated operator label (`x-switchyard-agent-kind`).@@ -257,8 +257,8 @@ fn parse_sub_agent(headers: &BTreeMap<String, String>) -> (Option<String>, bool,let harness_kind = resolve_path(headers, CODEX_SUBAGENT_KIND_PATH).or_else(|| header(headers, OPENAI_SUBAGENT_HEADER).map(str::to_string));- // Parent resolved via HEADER_CONFIG precedence (covers Dynamo/Codex correlation);- // falls back to the Claude Code session the child was spawned under.+ // Resolve the parent through the configured header precedence, then fall back+ // to the native agent session the child was spawned under.let parent = sy_header(headers, SWITCHYARD_PARENT_AGENT_ID_HEADER).or_else(|| claude_parent.map(str::to_string));@@ -464,7 +464,7 @@ mod tests {#[test]fn normalizes_relay_and_dynamo_child_headers() {- // Relay and Dynamo headers are correlation data, not routing signals.+ // Integrating-host headers are correlation data, not routing signals.// They populate observability fields but must not trigger sub-agent routing.let headers = BTreeMap::from([(diff --git a/crates/protocol/src/stream.rs b/crates/protocol/src/stream.rsindex 5fd82e687..ce7b15d92 100644--- a/crates/protocol/src/stream.rs+++ b/crates/protocol/src/stream.rs@@ -27,14 +27,16 @@ const MID_STREAM_UPSTREAM_STATUS: u16 = 502;pub type LlmResponseStream =Pin<Box<dyn Stream<Item = Result<LlmResponseChunk, LlmClientError>> + Send>>;-/// A model response: either a live [`Stream`](LlmResponse::Stream) of chunks or the-/// terminal buffered [`Agg`](LlmResponse::Agg)regate.+/// A model response: either a live [`Stream`](LlmResponse::Stream) of chunks or a+/// terminal buffered [`LlmResponse::Agg`] response.////// Not `Clone` — the `Stream` variant owns a single-consumption stream. A buffered/// backend returns `Agg` directly; a streaming one returns `Stream` and the consumer/// drives it, folding to an [`AggLlmResponse`] when it needs the whole response.pub enum LlmResponse {+ /// Live, single-consumption response stream.Stream(LlmResponseStream),+ /// Fully buffered terminal response.Agg(AggLlmResponse),}@@ -79,6 +81,10 @@ impl LlmResponse {}}+ /// Returns the model recorded by a buffered response.+ ///+ /// Returns `None` for a live stream because reading its model-bearing+ /// [`LlmResponseChunk::MessageStart`] event would consume the stream.pub fn selected_model(&self) -> Option<&str> {match self {LlmResponse::Agg(agg) => agg.model.as_deref(),@@ -94,6 +100,10 @@ impl AggLlmResponse {/// Useful when a caller had to buffer the response (e.g. to judge it) but the/// downstream expects `LlmResponse::Stream` — for instance when `stream: true` was/// requested and the algorithm had to aggregate before it could return.+ ///+ /// This conversion is lossy: only text, reasoning, and tool-call content has+ /// a synthetic chunk representation. Refusals, tool results, media, files,+ /// unknown blocks, response extensions, and preservation metadata are omitted.pub fn into_stream(self) -> LlmResponseStream {let mut chunks: Vec<LlmResponseChunk> = Vec::new();chunks.push(LlmResponseChunk::MessageStart {@@ -145,36 +155,56 @@ impl AggLlmResponse {}/// One provider-neutral streaming event — the normalized counterpart to-/// [`AggLlmResponse`](crate::AggLlmResponse), sitting between stream decoders and-/// encoders. `switchyard-translation` re-exports it as `ConversationStreamEvent`.+/// [`AggLlmResponse`], sitting between stream decoders and encoders.#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]pub enum LlmResponseChunk {+ /// Starts a response message.MessageStart {+ /// Provider response identifier.id: Option<String>,+ /// Model reported by the provider.model: Option<String>,},+ /// Adds text to one output index.TextDelta {+ /// Provider output index.index: usize,+ /// Text fragment.text: String,},+ /// Adds reasoning text to one output index.ReasoningDelta {+ /// Provider output index.index: usize,+ /// Reasoning fragment.text: String,},+ /// Adds or updates a tool call at one index.ToolCallDelta {+ /// Tool-call index within the response.index: usize,+ /// Tool-call identifier, normally supplied by the first delta.id: Option<String>,+ /// Tool name, normally supplied by the first delta.name: Option<String>,+ /// Fragment of the serialized tool arguments.arguments_delta: Option<String>,},+ /// Reports token usage, normally near the end of the stream.Usage(Usage),+ /// Ends a response message.MessageStop {+ /// Provider stop-reason string before normalization.reason: Option<String>,},+ /// Reports that an inbound stream event could not be decoded.DecodeError {+ /// Human-readable decoding failure.message: String,},+ /// Reports an upstream failure delivered inside an otherwise successful stream.StreamError {+ /// Human-readable upstream failure.message: String,},}@@ -183,8 +213,14 @@ pub enum LlmResponseChunk {////// Text and reasoning deltas concatenate; tool-call deltas assemble by index (name,/// id, and a growing arguments string parsed as JSON at the end); `MessageStart`,-/// `Usage`, and `MessageStop` set the corresponding fields. `Error` chunks are-/// ignored here — a driver consuming the stream is expected to surface them.+/// `Usage`, and `MessageStop` set the corresponding fields.+/// [`DecodeError`](LlmResponseChunk::DecodeError) and+/// [`StreamError`](LlmResponseChunk::StreamError) chunks are ignored here; use+/// [`LlmResponse::into_agg`] when they must become errors.+///+/// Folding is lossy for multiple outputs: text and reasoning indices are ignored,+/// and [`finish`](Self::finish) produces one assistant output. Prefer+/// [`LlmResponse::into_agg`] when stream errors must be surfaced.////// Drive it by `push`-ing each chunk in order, then call [`finish`](Self::finish).#[derive(Default)]diff --git a/docs/core_concepts.md b/docs/core_concepts.mdindex d39fa2b57..ef2dae090 100644--- a/docs/core_concepts.md+++ b/docs/core_concepts.md@@ -109,3 +109,4 @@ selected target uses a different upstream protocol.- [Architecture](architecture.md): follow a request through the server and Rust crates.- [`switchyard-server`](../crates/switchyard-server/README.md): complete TOML schema, endpoints, and metrics.- [`switchyard-libsy`](../crates/libsy/README.md): embed and extend routing algorithms in Rust.+- [Rust API reference](reference/rust_api.md): generated libsy and protocol documentation.diff --git a/docs/getting_started.md b/docs/getting_started.mdindex 05729c1b2..63ff63e80 100644--- a/docs/getting_started.md+++ b/docs/getting_started.md@@ -237,9 +237,7 @@ export SWITCHYARD_TELEMETRY_OPT_OUT=1- [Core Concepts](core_concepts.md): LLM clients, targets, and routes- [`switchyard-server`](../crates/switchyard-server/README.md): server configuration,routing algorithms, TLS, and metrics-- [`switchyard-libsy`](../crates/libsy/README.md): embed routing algorithms in a- Rust application-- [`switchyard-protocol`](../crates/protocol/README.md): provider-neutral- request, response, and streaming types+- [Rust API reference](reference/rust_api.md): generated libsy and protocol+ documentation, crate setup, and API boundaries- [`switchyard-translation`](../crates/switchyard-translation/README.md):request, response, and stream translationdiff --git a/docs/index.md b/docs/index.mdindex 2ada13bc0..1dc5056cd 100644--- a/docs/index.md+++ b/docs/index.md@@ -21,11 +21,12 @@ standalone `switchyard-server` binary.- [Routing Algorithms](routing_algorithms/overview.md): choose how requests select a model- [Architecture](architecture.md): understand the proxy and library components- [CLI Reference](cli_reference.md): inspect supported launcher commands and options+- [Rust API](reference/rust_api.md): browse libsy and protocol crate documentation- [Context-Window Handling](operations/context_window.md): configure eviction and retry behavior## Reference- [`switchyard-server`](../crates/switchyard-server/README.md): server configuration, endpoints, and metrics-- [`switchyard-libsy`](../crates/libsy/README.md): embeddable routing algorithms-- [`switchyard-protocol`](../crates/protocol/README.md): provider-neutral API types+- [`switchyard-libsy`](reference/rust_api.md#switchyard-libsy): embeddable routing algorithms+- [`switchyard-protocol`](reference/rust_api.md#switchyard-protocol): provider-neutral API types- [`switchyard-translation`](../crates/switchyard-translation/README.md): protocol translationdiff --git a/docs/internal/metrics_reference.md b/docs/internal/metrics_reference.mdindex 09d63ade7..671a62286 100644--- a/docs/internal/metrics_reference.md+++ b/docs/internal/metrics_reference.md@@ -11,71 +11,67 @@ a drop-in scrape config and starter alert rules.| Path | `GET /metrics` (HTTP path is `/metrics`, **not** `/v1/metrics`) || Content-Type | `text/plain; version=0.0.4; charset=utf-8` || Format | Prometheus text format 0.0.4 |-| Auth | None. Designed as a public scrape endpoint |+| Auth | None || Default scrape interval | 15s |-`GET /metrics` is served by the Python route-bundle server started with-`switchyard serve --routing-profiles PATH`.+`GET /metrics` is served by `switchyard-server`.-A JSON variant of the same underlying data lives at `GET /v1/stats`, with-`GET /v1/routing/stats` as a backwards-compatible alias.+A JSON summary of the same traffic lives at `GET /v1/stats`.## Top-line gauges (no labels)| Metric | Type | Meaning ||---|---|---|-| `switchyard_total_requests` | gauge | Total upstream call attempts recorded since process start. **Per-attempt**, not per-client-request. One client request that retries-then-succeeds increments by 2. |-| `switchyard_total_errors` | gauge | Total upstream call attempts that errored, including those absorbed by retry. |+| `switchyard_total_requests` | gauge | Successful and failed routed model calls since process start. Classifier and judge calls are excluded; a context-window fallback can add another routed call. |+| `switchyard_total_errors` | gauge | Failed routed model calls since process start. |## Per-endpoint countersThe `model` label is the configured endpoint id (`openai/gpt-5.5`,`azure_openai/gpt-5.5`, etc.).-The `tier` label is optional: present only when a routing factory-supplied one. The `random_routing` factory emits `tier="strong"|"weak"`.+The `tier` label is optional. It is present when an algorithm defines a stable+tier for the selected model, such as `strong` or `weak` for a two-tier classifier.| Metric | Type | Meaning ||---|---|---|-| `switchyard_requests_total{model}` | counter | Successful upstream call attempts per endpoint. |-| `switchyard_errors_total{model}` | counter | Failed upstream call attempts per endpoint (any cause). |+| `switchyard_requests_total{model}` | counter | Successful routed model calls per endpoint. |+| `switchyard_errors_total{model}` | counter | Failed routed model calls per endpoint. || `switchyard_prompt_tokens_total{model}` | counter | Prompt-token billing per endpoint. || `switchyard_completion_tokens_total{model}` | counter | Completion-token usage per endpoint. || `switchyard_cached_tokens_total{model}` | counter | Cached prompt tokens per endpoint. || `switchyard_cache_creation_tokens_total{model}` | counter | Cache-creation tokens per endpoint. || `switchyard_reasoning_tokens_total{model}` | counter | Reasoning tokens per endpoint. |-## Per-endpoint latency summaries+## Per-endpoint latency histograms-Each summary emits `{quantile="0.5"}` and `{quantile="0.99"}` rows plus-`_sum` and `_count`.+Each histogram emits `_bucket`, `_sum`, and `_count` series. Use+`histogram_quantile` in PromQL to calculate a percentile.| Metric | Type | Meaning ||---|---|---|-| `switchyard_model_call_latency_ms{model,quantile}` | summary | Upstream LLM call duration. |-| `switchyard_total_latency_ms{model,quantile}` | summary | End-to-end request latency (request entry → response complete). For streaming responses this is full-turn time, **not** time-to-first-token. |+| `switchyard_model_call_latency_ms{model}` | histogram | Successful final routed-call latency. |+| `switchyard_total_latency_ms{model}` | histogram | End-to-end latency for successful routed responses. For streaming responses this is full-turn time, **not** time-to-first-token. |## Routing overhead (global, no model label)| Metric | Type | Meaning ||---|---|---|-| `switchyard_routing_overhead_ms{quantile}` | summary | `total_latency_ms − backend_call_latency_ms`, across all calls. Bundles format translation, endpoint selection, body mutation, response wrapping, and retry wall time. |--Healthy traffic typically sits at p50 ≈ 0.4 ms, p99 ≈ 0.6 ms.+| `switchyard_routing_overhead_ms{algorithm}` | histogram | Algorithm run time minus the final routed model call. Includes classifier calls, target resolution, and decision publication; runs with no final routed call are not recorded. |## Outcome counters for error-rate ratiosThe `outcome` label takes exactly three values:* `success` = HTTP 2xx-* `retryable_error` = HTTP 429 / 500 / 504+* `retryable_error` = HTTP 408, 429, any 5xx, or a failure before an HTTP status* `other_error` = everything else (400, 401, 403, 422, …)| Metric | Type | Meaning ||---|---|---|| `switchyard_client_responses_total{outcome}` | counter | HTTP responses returned to clients on the LLM-serving routes (`/v1/chat/completions`, `/v1/messages`, `/v1/responses`). The denominator for the **router-served** error rate. || `switchyard_upstream_attempts_total{outcome,code}` | counter | Individual upstream call attempts. One client request can produce N attempts via retry. The denominator for the **direct-to-endpoint** baseline error rate. The `code` label carries the raw upstream HTTP status for plotting the error-code distribution (see below). |-| `switchyard_router_retry_recovered_total` | counter | Requests whose first upstream attempt failed but a subsequent attempt succeeded. This is direct evidence the routing logic rescued the request. |+| `switchyard_router_retry_recovered_total` | counter | Reserved retry-recovery counter. The current server exports it as zero. |### The `code` label on `switchyard_upstream_attempts_total`@@ -111,9 +107,6 @@ direct_error_rate =# Headline metric: positive value means the router is reducing client errorserror_rate_reduction = direct_error_rate − router_error_rate-# Live Endpoint Routing rescue rate-rate(switchyard_router_retry_recovered_total[5m])-# Traffic share per endpointsum by (model) (rate(switchyard_requests_total[5m]))/ ignoring(model) group_left sum(rate(switchyard_requests_total[5m]))@@ -145,14 +138,15 @@ into label space.| `model` | One per configured endpoint, typically 2–6 per deployment. | All per-endpoint metrics. || `outcome` | Exactly 3: `success`, `retryable_error`, `other_error`. | Outcome counters || `code` | Bounded: the known-code allowlist (`200`, `400`, `401`, `403`, `404`, `408`, `409`, `422`, `429`, `500`, `502`, `503`, `504`), plus `none` and the per-class buckets `1xx`/`2xx`/`3xx`/`4xx`/`5xx`/`other`. About 20 values max. | `switchyard_upstream_attempts_total` |-| `quantile` | Exactly 2: `0.5`, `0.99`. | All summaries |-| `tier` | Small enumerated set, optional. | Per-endpoint counters/summaries on routing chains that supply it |+| `le` | The configured histogram bucket boundaries. | Histogram buckets |+| `algorithm` | One stable value per configured algorithm. | Routing-overhead histogram |+| `tier` | Small enumerated set, optional. | Per-endpoint counters and histograms on algorithms that supply it |## Triage cheatsheet| Symptom on `/metrics` | Likely cause ||---|---|-| `model="<unknown>"` rows appear | The per-endpoint attribution wiring regressed. `ctx.selected_model` is not being set by the backend. |+| `model="<unknown>"` rows appear | A routed-call observation did not include a selected model. || All counters at 0 after warm-up | Server just started with no traffic, or the scraper is hitting the wrong port. |-| `switchyard_routing_overhead_ms_count` stuck at `0` | The backend is not publishing `ctx.backend_call_latency_ms` (regression of the Python-backend routing-overhead wiring). |-| `switchyard_client_responses_total{outcome="retryable_error"}` rising | Either the upstream is genuinely flaky, or retries are exhausting (compare with `switchyard_router_retry_recovered_total` rate). |+| `switchyard_routing_overhead_ms_count` stuck at `0` | No successful algorithm run has recorded a final routed model call. |+| `switchyard_client_responses_total{outcome="retryable_error"}` rising | Either the upstream is genuinely flaky, or retries are exhausting; compare client responses with retryable upstream attempts. |diff --git a/docs/known_issues.md b/docs/known_issues.mdindex d2b3b361e..8f53744e1 100644--- a/docs/known_issues.md+++ b/docs/known_issues.md@@ -1,5 +1,5 @@# Known Issues## 0.1.0-1. Completed Codex Responses tasks may record `0` token usage in `GET /v1/stats` and `GET /v1/routing/stats`.+1. Completed Codex Responses tasks may record `0` token usage in `GET /v1/stats`.2. Tool-bearing Codex requests may fail when Switchyard routes them to an upstream that accepts only a fixed set of tool names or schemas.diff --git a/docs/reference/rust_api.md b/docs/reference/rust_api.mdnew file mode 100644index 000000000..eed7157d3--- /dev/null+++ b/docs/reference/rust_api.md@@ -0,0 +1,39 @@+# Rust API++Switchyard publishes generated API documentation for its embeddable routing+library and provider-neutral protocol types.++## `switchyard-libsy`++Use libsy to construct targets, run built-in routing algorithms, implement a+custom algorithm, or drive model calls through the host-facing step stream.++<a href="../../rust/switchyard_libsy/">Open the generated libsy API documentation</a>++## `switchyard-protocol`++Use protocol types for normalized conversations, response streams, routed LLM+clients, decisions, metadata, and wire-format identifiers.++<a href="../../rust/switchyard_protocol/">Open the generated protocol API documentation</a>++## Crate boundary++Applications embedding libsy normally depend on both crates:++```toml+[dependencies]+switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }+switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }+```++Import algorithms and orchestration types from `switchyard_libsy`. Import+requests, responses, metadata, decisions, and LLM client traits from+`switchyard_protocol`. Keep both dependencies on compatible versions.++## CI previews++The Documentation workflow publishes a preview for same-repository pull+requests. Open the workflow's **Deploy PR preview** summary to find its preview+URL, then use the Rust API links above. The **docs-site** build artifact contains+the same MkDocs and rustdoc output for one day.diff --git a/docs/routing_algorithms/stage_router_routing.md b/docs/routing_algorithms/stage_router_routing.mdindex f21d0ad0d..2c60b192b 100644--- a/docs/routing_algorithms/stage_router_routing.md+++ b/docs/routing_algorithms/stage_router_routing.md@@ -9,9 +9,9 @@ mechanical work. Which tier a turn defaults to depends on the picker you chooseoff that default. You configure it with a single knob, `confidence_threshold`,plus an optional LLM classifier.-If the selected backend hits a context-window overflow, the router retries once-against `fallback_target_on_evict`; a second overflow surfaces a-context-pool-exhausted error (see [Context-Window Handling](../operations/context_window.md)).+If the selected target exceeds its context window, the router tries the next+eligible target until one succeeds or all configured targets have been tried. See+[Context-Window Handling](../operations/context_window.md).## How it works@@ -98,21 +98,19 @@ clears the threshold to escalate to capable.(If you add the optional classifier, sub-threshold turns go to it instead ofstaying on the default tier.)-**Set `0.5` explicitly.** It's the recommended starting point and what the-example below uses. When you omit the field the config default is `0.5` (for-both the profile config and the deprecated route bundle) — but setting it-explicitly keeps the intent clear.+**Set `0.5` explicitly.** `confidence_threshold` is required by the TOML schema;+`0.5` is the recommended starting point and what the example below uses.| `confidence_threshold` | Include `classifier:` block? | Typical use ||---|---|---|| `0.0` | no | Cost/latency-sensitive. Every signal-based verdict is accepted; no per-turn LLM call. Critical-error signals still escalate to capable. |-| `0.5` | no | Recommended starting point (config default). The scorer is corroborative — one full wrong signal scores ~`0.46`, just under `0.5` — so a decisive escalation takes a strong signal plus corroboration, while a critical error overrides regardless. Derived from SWE-Bench Pro Python-75 calibration. |+| `0.5` | no | Recommended starting point. The scorer is corroborative — one full wrong signal scores ~`0.46`, just under `0.5` — so a decisive escalation takes a strong signal plus corroboration, while a critical error overrides regardless. Derived from SWE-Bench Pro Python-75 calibration. || `0.7` - `0.9` | yes | Classifier-assisted. Low-confidence turns go to the LLM classifier before falling back to the default tier. |-| `1.0` | yes (required) | Classifier-driven. Equivalent to the legacy `coding_agent` profile. |+| `1.0` | yes (required) | Classifier-driven. Tool signals only apply hard overrides; other turns reach the classifier. |The signal-vs-classifier split is dataset-dependent. Measure it in-production via `routing_decisions.stage_router` on `/v1/stats` rather than relying on-priors from this doc.+production: `/v1/stats` reports traffic by tier and model, while response headers+and structured decision logs explain individual selections.### Calibrating the threshold from run data@@ -275,8 +273,8 @@ Each response carries two routing headers:### Decision sources-The `decision_source` recorded internally for each turn explains why the routing-went the way it did. It appears in per-tier metrics tagged on the decision:+The router records an internal `decision_source` for each turn to distinguish the+paths through its cascade:| Source | When ||---|---|@@ -288,7 +286,7 @@ went the way it did. It appears in per-tier metrics tagged on the decision:## When *not* to use stage-router-- **Single-model deployments.** Use a `model` route instead.+- **Single-model deployments.** Use a `passthrough` route instead.- **Probabilistic A/B splits.** Use[Random Routing](random_routing.md) (`type = "random"`).The stage-router's signals are wasted on a fixed traffic ratio.diff --git a/examples/prometheus/README.md b/examples/prometheus/README.mdindex ceea201f4..fff0448c4 100644--- a/examples/prometheus/README.md+++ b/examples/prometheus/README.md@@ -58,7 +58,7 @@ series:```You should see the families documented in `metrics_reference.md` —-top-line gauges, per-endpoint counters/summaries, and outcome counters.+top-line gauges, per-endpoint counters and histograms, and outcome counters.## Alert tuning notes@@ -68,8 +68,5 @@ or single-poll blips. Review them against your SLOs before enablingnotifications:* `RouterOverheadHigh` is set at the success-criterion threshold (p99 >- 1 ms for 5 m). Retry-heavy workloads inflate this — cross-check- `switchyard_upstream_attempts_total{outcome="retryable_error"}` rate- before treating it as a routing-decision regression.-* `RouterNotRescuing` requires upstream failures to be occurring before- it fires, so it does not flap on a healthy chain.+ 1 ms for 5 m). Check its `algorithm` label and `/v1/stats` classifier+ traffic before treating it as a routing-decision regression.diff --git a/examples/prometheus/switchyard.rules.yaml b/examples/prometheus/switchyard.rules.yamlindex 3b3400054..22881f02d 100644--- a/examples/prometheus/switchyard.rules.yaml+++ b/examples/prometheus/switchyard.rules.yaml@@ -11,29 +11,12 @@groups:- name: switchyard.routing.evidencerules:- - alert: RouterNotRescuing- # Upstream is throwing retryable errors but the router isn't- # absorbing them via retry. This fires only when there IS- # something to rescue — does not flap on a healthy chain.- expr: |- rate(switchyard_router_retry_recovered_total[5m]) == 0- and- rate(switchyard_upstream_attempts_total{outcome="retryable_error"}[5m]) > 0- for: 5m- labels:- severity: warning- component: switchyard- annotations:- summary: "Switchyard observed upstream errors but no retry recoveries"- description: |- Upstream attempts are returning retryable errors (429 / 500- / 504) but no request has been rescued by retry in the same- 5 m window. Likely cause: every configured endpoint is- degraded simultaneously, so the retry loop has nowhere to- steer.-- alert: RouterOverheadHigh- expr: switchyard_routing_overhead_ms{quantile="0.99"} > 1.0+ expr: |+ histogram_quantile(+ 0.99,+ sum by (le) (rate(switchyard_routing_overhead_ms_bucket[5m]))+ ) > 1.0for: 5mlabels:severity: warning@@ -42,10 +25,9 @@ groups:summary: "Switchyard router overhead p99 above 1 ms"description: |switchyard_routing_overhead_ms p99 has exceeded 1 ms for 5- minutes. Routing overhead bundles format translation,- endpoint selection, body mutation, and retry wall time —- retries are the most common cause of inflation. Cross-check- rate(switchyard_upstream_attempts_total{outcome="retryable_error"}[5m]).+ minutes. Routing overhead includes classifier calls, target+ resolution, and decision publication. Cross-check the algorithm+ label and classifier traffic in /v1/stats.- alert: RouterErrorRateNotReducing# Headline success criterion. The router should serve a lower@@ -62,6 +44,8 @@ groups:sum(rate(switchyard_upstream_attempts_total{outcome="retryable_error"}[5m]))/ clamp_min(sum(rate(switchyard_upstream_attempts_total[5m])), 1))+ and+ sum(rate(switchyard_upstream_attempts_total{outcome="retryable_error"}[5m])) > 0for: 10mlabels:severity: warningdiff --git a/mkdocs.yml b/mkdocs.ymlindex 0a624c5a0..d081dab58 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -34,6 +34,7 @@ nav:- Reference:- CLI Reference: cli_reference.md- TOML Schema: reference/toml_schema.md+ - Rust API: reference/rust_api.mdexclude_docs: |internal/**