Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs(rust): repair libsy and protocol references (#263)

nachiketb-nvidiae0127808c0c9
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
index 0f07c1727..3fda6c7bb 100644
--- a/.github/workflows/docs.yml
+++ b/.github/workflows/docs.yml
@@ -54,8 +54,13 @@ jobs:
- run: uv sync --only-group docs --locked
- name: Build documentation
run: uv run --only-group docs mkdocs build --strict
- - name: Build libsy crate documentation
- run: cargo doc --locked --no-deps --package switchyard-libsy
+ - name: Build Rust crate documentation
+ env:
+ RUSTDOCFLAGS: -D warnings
+ run: >-
+ cargo doc --locked --no-deps
+ --package switchyard-libsy
+ --package switchyard-protocol
- name: Add crate documentation to site
run: |
mkdir -p site/rust
diff --git a/crates/libsy/Cargo.toml b/crates/libsy/Cargo.toml
index dc2be5249..ce34afe2f 100644
--- a/crates/libsy/Cargo.toml
+++ b/crates/libsy/Cargo.toml
@@ -4,12 +4,17 @@
[package]
name = "switchyard-libsy"
version.workspace = true
-description = "Switchyard library crate"
+description = "Provider-neutral multi-LLM routing and orchestration for Switchyard"
authors.workspace = true
edition.workspace = true
+homepage = "https://github.com/NVIDIA-NeMo/Switchyard"
license.workspace = true
+readme = "README.md"
repository.workspace = true
rust-version.workspace = true
+keywords = ["llm", "routing", "orchestration", "openai", "anthropic"]
+categories = ["asynchronous", "web-programming"]
+publish = ["crates-io"]
[dependencies]
async-trait.workspace = true
diff --git a/crates/libsy/README.md b/crates/libsy/README.md
index 5ac6ac835..3f5def27f 100644
--- a/crates/libsy/README.md
+++ b/crates/libsy/README.md
@@ -1,343 +1,181 @@
-# libsy — Switchyard-Lib
-
-A lightweight, provider-agnostic library for multi-LLM agent optimization, with
-**routing** as the first case. libsy *decides* how to serve each request — which
-model(s) to call, in what order, how to combine them — and either makes the calls
-itself or hands them back for you to make. It owns no HTTP client and no provider SDK,
-so it drops into a proxy, gateway, or agent runtime.
+# switchyard-libsy
+
+Provider-neutral orchestration for multi-LLM optimization. A libsy
+[`Algorithm`] decides which model targets to call, in what order, and how to
+combine their results. It can use target-owned clients or hand each call back to
+the host, allowing it to embed in proxies, gateways, and agent runtimes without
+owning an HTTP stack.
+
+## Setup
+
+```toml
+[dependencies]
+async-trait = "0.1"
+futures = "0.3"
+switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
+switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
+tokio = { version = "1", features = ["macros", "rt"] }
+```
-## Example
+## Quick start
-Build the targets, pick an algorithm, run a request:
+This complete `src/main.rs` routes a request with [`LlmTaskClassifier`]. The
+first [`Algorithm::run`] lets libsy call each target's client. The second drives
+[`Algorithm::run_stream`] so the host can perform or override each model call
+itself.
```rust
-use switchyard_libsy::{Algorithm, Context, RoutedLlmClient, LlmTarget, Request};
-use switchyard_libsy::algorithms::{LlmTaskClassifier, TaskClassifierConfig};
-use switchyard_protocol::{completion_text, text_request};
use std::sync::Arc;
-// Targets the algorithm routes among, each backed by your RoutedLlmClient (see below).
-let client = Arc::new(MyClient { /* .. */ }) as Arc<dyn RoutedLlmClient>;
-let target = |name: &str| LlmTarget { semantic_name: name.into(), llm_client: Some(client.clone()) };
-
-let algo: Arc<dyn Algorithm> = Arc::new(LlmTaskClassifier::new(
- target("classifier"), // the judge
- target("weak"), // efficient tier
- target("strong"), // capable tier
- // Every field but `base_threshold` has a default; see "Tuning the classifier" below.
- TaskClassifierConfig { base_threshold: 0.5, ..Default::default() },
-)?);
-
-let req = Request {
- // `text_request` is the single-turn shortcut; build an `LlmRequest` directly for
- // multi-turn conversations, tools, or sampling parameters.
- llm_request: text_request(Some("auto".into()), "explain tail latency"),
- raw_request: None,
- metadata: None,
+use async_trait::async_trait;
+use futures::StreamExt;
+use switchyard_libsy::{
+ Algorithm, LibsyError, LlmTarget, LlmTaskClassifier, Step, TaskClassifierConfig,
+};
+use switchyard_protocol::{
+ AggLlmResponse, ContentBlock, Context, Decision, LlmClientError, LlmRequest,
+ LlmResponse, Message, Request, Response, ResponseOutput, Role, RoutedLlmClient,
};
-let (trace, response) = algo.clone().run(Context::default(), req).await?; // calls in, trace + response out
-println!("routed to {}", trace.last().unwrap().selected_model());
-// `response.llm_response` is buffered or streamed; fold it to the aggregate to read text.
-println!("answer: {}", completion_text(&response.llm_response.into_agg().await?));
-```
-
-Runnable: [`research_agent`](examples/research_agent.rs)
-
-## Requests & responses
-
-```rust
-pub struct Request {
- pub llm_request: LlmRequest,
- pub raw_request: Option<serde_json::Value>, // optional original provider body for exact-fidelity hosts
- pub metadata: Option<Metadata>, // correlation: session / agent / task / correlation_id / extra
-}
-
-// `Response` is a plain struct. The buffered-or-streamed choice lives one level down,
-// in its `llm_response` — which *is* an enum.
-pub struct Response {
- pub llm_response: LlmResponse,
- pub metadata: Option<Metadata>,
-}
-
-pub enum LlmResponse {
- Stream(LlmResponseStream), // a live stream of `LlmResponseChunk`s (token-by-token)
- Agg(AggLlmResponse), // the buffered aggregate — outputs, usage, stop reason
-}
-```
-
-`switchyard-protocol` owns the conversation IR: `LlmRequest`, the buffered
-`AggLlmResponse`, the streaming `LlmResponseChunk`, and the building blocks `Message`,
-`ContentBlock`, `ResponseOutput`, `Usage`, and `Role`. `libsy` re-exports the envelope
-(`Request`/`Response`/`Metadata`) plus `LlmRequest`, `LlmResponse`, `AggLlmResponse`,
-`LlmResponseChunk`, `LlmResponseStream`, and `Usage`; import the rest (`Message`, `Role`, …)
-and the `text_request` / `text_response` / `completion_text` helpers from
-`switchyard_protocol`. Construct and inspect the conversation model directly so tools,
-sampling parameters, reasoning, and provider extensions stay visible instead of being hidden
-behind a second convenience API. `raw_request` remains available when a host needs exact
-source-body fidelity.
-
-## Targets and clients
-
-An `LlmTarget` pairs a routing `semantic_name` with an optional `RoutedLlmClient`. Mapping that
-name to a provider model id is the client's job, not the algorithm's — `RoutedLlmClient` is
-meant to be implemented by you.
-```rust
-struct MyClient { /* http client, base url, key */ }
+struct DemoClient;
-#[async_trait::async_trait]
-impl RoutedLlmClient for MyClient {
- async fn call(&self, ctx: Context, request: Request, decision: Arc<dyn Decision>)
- -> Result<Response, LlmClientError> {
- let model = decision.selected_model(); // the routed target — map it to a provider id
- // request.llm_request.model is the agent's original name (not a call target)
- // ... POST to your endpoint, read the completion ...
- let completion = String::from("provider response text");
+#[async_trait]
+impl RoutedLlmClient for DemoClient {
+ async fn call(
+ &self,
+ _ctx: Context,
+ _request: Request,
+ decision: Arc<dyn Decision>,
+ ) -> Result<Response, LlmClientError> {
+ let model = decision.selected_model();
+ let text = if model == "judge" {
+ r#"{
+ "recommended_route": "efficient",
+ "p_solve": 0.9,
+ "confidence": 0.9,
+ "abstain": false,
+ "capability_boundary": "supported",
+ "primary_rule": "SUP-1",
+ "crux": "bounded task"
+ }"#
+ .to_string()
+ } else {
+ format!("answer from {model}")
+ };
Ok(Response {
- // Buffered: wrap the aggregate in `LlmResponse::Agg`. `text_response` builds a
- // single-turn `AggLlmResponse`; construct one directly for tool calls, usage, etc.
- llm_response: LlmResponse::Agg(text_response(None, completion)),
+ llm_response: LlmResponse::Agg(AggLlmResponse {
+ model: Some(model.to_string()),
+ outputs: vec![ResponseOutput {
+ role: Role::Assistant,
+ content: vec![ContentBlock::Text { text }],
+ stop_reason: None,
+ }],
+ ..AggLlmResponse::default()
+ }),
metadata: None,
})
}
}
-```
-
-To stream instead, return `LlmResponse::Stream(..)` — a boxed
-`Stream<Item = Result<LlmResponseChunk, _>>` — and emit chunks as they arrive from your
-upstream. See [Streaming responses](#streaming-responses) below.
-
-A `RoutedRequest` bundles the `request` with the routing `decision`, the target's
-`default_client`, and the request's `ctx`; the model to call is
-`decision.selected_model()`, never a mutated request field. `semantic_name` is the label an algorithm routes by; the client maps it to
-the id it calls — they can differ (`"strong"` → `"openai/gpt-4o"`) or coincide.
-
-## Running a request
-
-Hold the algorithm as `Arc<dyn Algorithm>` and choose one of two entry points:
-
-```rust
-// run: libsy drives the request to completion, serving each call with the target's
-// client, and returns (trace, response). Errors if a routed target has no client.
-let (trace, response) = algo.clone().run(Context::default(), req).await?;
-// run_stream: "ask, don't call" — you drive the stream and make the calls.
-let stream = algo.clone().run_stream(Context::default(), req);
-```
-
-Under the hood every model call is *offloaded* to the request's `Step` stream; `run` is
-the convenience that serves each one via the target's client. The step stream is bounded,
-so pulling it paces the algorithm; each run is independent, so many run concurrently.
-
-## Streaming responses
-
-A `Response` carries an `LlmResponse` that is either buffered (`Agg`) or a live token
-stream (`Stream`). An `RoutedLlmClient` — or an algorithm — chooses: return
-`LlmResponse::Agg(..)` for a whole answer, or `LlmResponse::Stream(..)` to forward tokens
-as they arrive. libsy never buffers a stream on your behalf; it flows through the algorithm
-untouched, so `run` / `run_stream` hand back whatever was produced and **the caller
-decides**:
-
-```rust
-use futures::StreamExt;
-
-let (_trace, response) = algo.clone().run(Context::default(), req).await?;
-match response.llm_response {
- // Forward tokens as they arrive — e.g. re-encode each chunk to your client's SSE.
- LlmResponse::Stream(mut chunks) => {
- while let Some(chunk) = chunks.next().await {
- let chunk: LlmResponseChunk = chunk?; // TextDelta / ToolCallDelta / Usage / MessageStop / ..
- /* emit chunk */
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> switchyard_libsy::Result<()> {
+ let client = Arc::new(DemoClient);
+ let target = |name: &str| LlmTarget {
+ semantic_name: name.to_string(),
+ llm_client: Some(client.clone()),
+ };
+
+ let router: Arc<dyn Algorithm> = Arc::new(LlmTaskClassifier::new(
+ target("judge"),
+ target("efficient"),
+ target("capable"),
+ TaskClassifierConfig {
+ base_threshold: 0.5,
+ ..TaskClassifierConfig::default()
+ },
+ )?);
+ let request = Request {
+ llm_request: LlmRequest {
+ model: Some("auto".to_string()),
+ messages: vec![Message {
+ role: Role::User,
+ content: vec![ContentBlock::Text {
+ text: "Explain tail latency".to_string(),
+ }],
+ }],
+ ..LlmRequest::default()
+ },
+ ..Request::default()
+ };
+
+ // `run` serves the judge and routed calls through their target clients.
+ let (_, response) = router
+ .clone()
+ .run(Context::default(), request.clone())
+ .await?;
+ println!("run returned {}", response.selected_model().unwrap_or("unknown"));
+
+ // `run_stream` exposes those calls so the host controls their transport.
+ let stream = router.run_stream(Context::default(), request, None);
+ tokio::pin!(stream);
+ while let Some(step) = stream.next().await {
+ match step? {
+ Step::CallLlm(call) => {
+ let routed = call.get_routed().clone();
+ let target = routed.decision.selected_model().to_string();
+ let result = client
+ .call(routed.ctx, routed.request, routed.decision)
+ .await
+ .map_err(|source| LibsyError::client_call(target, source));
+ call.respond(result)?;
+ }
+ Step::Decision(decision) => {
+ println!("run_stream chose {}", decision.selected_model());
+ }
+ Step::ReturnToAgent(response) => {
+ println!(
+ "run_stream returned {}",
+ response.selected_model().unwrap_or("unknown")
+ );
+ }
}
}
- // Already buffered — read it directly.
- LlmResponse::Agg(agg) => println!("{}", completion_text(&agg)),
+ Ok(())
}
```
-Need the whole answer regardless of how it arrived? `into_agg()` folds a `Stream` (or
-returns an `Agg` unchanged) into a single `AggLlmResponse`, surfacing any mid-stream error:
-
-```rust
-let agg = response.llm_response.into_agg().await?; // drives the stream to completion
-println!("{}", completion_text(&agg));
-```
-
-An `LlmResponseChunk` is the provider-neutral streaming event (`MessageStart`, `TextDelta`,
-`ReasoningDelta`, `ToolCallDelta`, `Usage`, `MessageStop`, `Error`); `ResponseAccumulator`
-is the fold behind `into_agg()` if you need to assemble one yourself. Runnable end-to-end
-demo: [`streaming_agent`](examples/streaming_agent.rs).
-
-## Driving the calls yourself (`run_stream`)
-
-`run_stream` yields `CallLlm` promises you fulfill with your own transport (or the call's
-`default_client`), streams each `Decision` as it happens, and ends with `ReturnToAgent`.
-This is the *step* stream (one `Step` at a time); it is orthogonal to whether any single
-response is itself streamed — see [Streaming responses](#streaming-responses).
-Runnable: [`research_agent_core`](examples/research_agent_core.rs).
-
-```rust
-let stream = algo.clone().run_stream(Context::default(), req);
-tokio::pin!(stream);
-while let Some(step) = stream.next().await {
- match step? {
- Step::CallLlm(call) => {
- let routed = call.get_routed().clone(); // which target, and its default client
- let response = call_model(routed.decision.selected_model(), &routed.request).await; // your real call
- call.respond(Ok(response))?; // or Err(..) to propagate a failure
- }
- Step::Decision(decision) => { /* decision.selected_model(), decision.reasoning() */ }
- Step::ReturnToAgent(response) => { /* done */ }
- }
-}
-```
-
-## Building an algorithm (`Algorithm`)
-
-Implement `Algorithm` to add a strategy. You write `create_run_task` — one call per
-request; `run` / `run_stream` are provided and drive it. Make model calls on the `Driver`
-you're handed, and publish a `Decision` for each so consumers (and clients) see *which*
-model and *why*.
-
-```rust
-#[async_trait]
-pub trait Algorithm: Send + Sync + 'static {
- // Stable telemetry label: the `algorithm` attribute on libsy's spans/metrics/logs.
- fn name(&self) -> &str;
- // `self: Arc<Self>` (not `&mut`): one algorithm serves requests concurrently — use
- // interior mutability for state. Offload calls/decisions on `driver`.
- async fn create_run_task(self: Arc<Self>, ctx: Context, driver: Driver, request: Request)
- -> switchyard_libsy::Result<Response>;
- async fn process_signals(self: Arc<Self>, signals: Signals)
- -> switchyard_libsy::Result<()>;
- // provided: run(ctx, request) -> (trace, response), run_stream(ctx, request) -> Stream<Step>
-}
-
-pub trait Decision: Send + Sync {
- fn selected_model(&self) -> &str; // the model chosen — the client's call target
- fn reasoning(&self) -> Option<&str>; // human-readable "why"
- fn as_any(&self) -> &dyn std::any::Any; // downcast to the concrete decision
-}
-```
-
-`LlmTaskClassifier` owns its fall-through cascade: it calls the judge, applies its policy, and
-then invokes the selected target:
-
-```rust
-let router = LlmTaskClassifier::new(
- judge_target,
- weak_target,
- strong_target,
- TaskClassifierConfig { base_threshold: 0.5, ..Default::default() },
-)?;
-```
-
-### Tuning the classifier
-
-The judge returns a `p_solve` — its estimate that the efficient tier completes the task
-correctly — plus a confidence and a capability boundary. `TaskClassifierConfig` decides what
-to do with that verdict. Anything invalid, abstained, or below a threshold routes to the
-capable target, so every knob below trades cost against quality in the same direction.
-
-| Field | Default | Meaning |
-|---|---|---|
-| `base_threshold` | *required* | Lowest `p_solve` that routes a supported task to the efficient target. Raise it to send less traffic to the cheap model. |
-| `min_confidence` | `0.0` | Lowest judge confidence that permits efficient routing. `0.0` disables the gate. |
-| `capability_elevated_floor` | `None` | A higher `p_solve` floor applied only when the judge marks the task `uncertain`, `unsupported`, or `unmatched`. `None` reuses `base_threshold`. |
-| `session_affinity` | `false` | Retains the first decision per session and reuses it on later turns, so the judge is called once per session instead of once per turn. |
-| `message_hash_fallback` | `false` | Extends affinity to clients that send no session header, keying on a hash of the first user message. Requires `session_affinity`. |
-
-Two things to understand before enabling affinity. The retained decision is **sticky for the
-process lifetime** — including a fail-closed `capable` verdict produced when the judge was
-unreachable — so a transient judge failure pins that identity until restart. And
-`message_hash_fallback` keys on request *content*, not a correlation id, so unrelated callers
-sending identical text share one assignment.
-
-## Errors
-
-All libsy-owned APIs return [`switchyard_libsy::Result<T>`], whose error is
-`LibsyError`. Callers can match routing failures (`TargetNotFound`, `NoTargets`,
-`MissingClient`), algorithm-specific failures (`AlgorithmError`), driver failures,
-algorithm task failures, incomplete runs, and client-call failures without inspecting error
-strings.
-
-`RoutedLlmClient` is owned by `switchyard-protocol` and returns
-`LlmClientError`. Callers can distinguish invalid requests, configuration
-failures, request/response translation failures, request encoding failures, transport
-failures, timeouts, context-window overflows, upstream HTTP responses, invalid responses,
-and uncategorized client-specific failures. When [`Algorithm::run`] serves a target,
-libsy preserves that error as the source of `LibsyError::ClientCall`.
-Custom algorithms, classifiers, and processors that need to surface their own errors can use
-`LibsyError::external("operation", error)`.
-
-## Observability
-
-libsy instruments every algorithm at the core — the `Decision` hook plus the offload
-boundary — so algorithms carry no telemetry code beyond `Algorithm::name()`, the
-`algorithm` attribute everything below is keyed by.
-
-- **Spans** (`tracing`): one `libsy.run` per request, carrying the `Metadata`
- correlation ids, any host-defined `extra_metadata` labels, and the outcome,
- with one child `libsy.llm_call` per model call
- (selected model, outcome, token counts; measures *fulfillment* as the algorithm
- observes it, host serving included — a streamed response resolves when its stream
- handle arrives). When `run` serves a call via the target's default client, the
- actual API call gets its own `libsy.client_call` span — hosts serving calls over
- their own transport should span their `RoutedLlmClient` equivalently.
-- **Structured logs** (`tracing`, target `libsy`): an info event per published
- `Decision` (selected model + reasoning), warn events for failed calls and runs.
-- **Metrics** (OpenTelemetry, scope `switchyard`, via the global meter provider): counters
- `switchyard.runs`, `switchyard.llm_calls`, and `switchyard.decisions`; histograms
- `switchyard.run_duration_ms` and `switchyard.llm_call_duration_ms`. Attributes are
- `algorithm`, `selected_model`, and `outcome` (`ok`/`error`) — failure rates are the
- `outcome="error"` share of runs/calls.
-
-libsy also records compatibility metrics for final routed calls:
-
-| OpenTelemetry instrument | Prometheus family | Type | Labels | Meaning |
-|---|---|---|---|---|
-| `switchyard.total_requests` | `switchyard_total_requests` | gauge | none | Successful and failed routed calls |
-| `switchyard.total_errors` | `switchyard_total_errors` | gauge | none | Failed routed calls |
-| `switchyard.requests` | `switchyard_requests_total` | counter | `model`, optional `tier` | Successful routed calls |
-| `switchyard.errors` | `switchyard_errors_total` | counter | `model`, optional `tier` | Failed routed calls |
-| `switchyard.model_call_latency_ms` | `switchyard_model_call_latency_ms` | histogram | `model`, optional `tier` | Successful routed-call latency |
-
-Classifier calls remain visible in the general LLM-call instruments but are excluded from these
-compatibility families. The `tier` label is present only when a decision supplies a stable tier.
-Hosts that need the two zero-valued gauges before the first call must invoke
-`libsy::initialize_metrics()` after installing the global meter provider.
+## Built-in algorithms
-libsy owns no exporter. The host installs an OpenTelemetry SDK meter provider
-(`opentelemetry::global::set_meter_provider`) and a `tracing` subscriber (bridge spans
-into OTel with `tracing-opentelemetry` if desired); with neither installed, all
-instrumentation is a no-op.
+| Type | Purpose |
+|---|---|
+| [`Passthrough`] | Always call one configured target. |
+| [`Random`] | Select among any number of targets using uniform or weighted routing. |
+| [`LlmTaskClassifier`] | Ask a judge model to choose an efficient or capable target. |
+| [`StageRouter`] | Route coding-agent turns from tool and progress signals, with an optional judge fallback. |
-## Explore
+[`Noop`] is a test helper, not a production routing algorithm.
-The core crate includes weighted random routing and a judge-backed LLM classifier. Runnable
-agents live in [`examples`](examples/) folder.
+## How it fits together
-**Reference algorithms** — implementations to read and route with:
+[`LlmTarget`] pairs a semantic routing name with an optional
+[`RoutedLlmClient`](switchyard_protocol::RoutedLlmClient). An [`Algorithm`]
+selects targets and records [`Decision`](switchyard_protocol::Decision)s. Use
+[`Algorithm::run`] with target-owned clients, or [`Algorithm::run_stream`] when
+the host owns model transport. The provider-neutral [`Request`], [`Response`],
+[`Usage`], and [`LlmResponse`] contracts come from `switchyard-protocol`.
-- [`Random`](src/algorithms/rand.rs) — uniform or weighted random over the set
- (one call).
-- [`LlmTaskClassifier`](src/algorithms/llm_class.rs) — task-level capability routing with an
- internal [`FallThrough`](src/algorithms/fall_through.rs) cascade.
-- [`EnsembleOrchAlgo`](examples/ensemble.rs) — stateful: fan out to
- candidates, judge the best, commit to the winner after N exploration turns.
+[`Request`]: switchyard_protocol::Request
+[`Response`]: switchyard_protocol::Response
+[`Usage`]: switchyard_protocol::Usage
+[`LlmResponse`]: switchyard_protocol::LlmResponse
-**Runnable agents** (`cargo run -p switchyard-libsy --example <name>`):
+## Examples
-- [`ensemble`](examples/ensemble.rs) — query three NVIDIA-hosted candidates, then judge and commit.
-- [`research_agent`](examples/research_agent.rs) — client-backed
- targets, `run` (libsy makes the calls).
-- [`research_agent_core`](examples/research_agent_core.rs) — client-less
- targets, `run_stream` (the agent makes the calls).
-- [`streaming_agent`](examples/streaming_agent.rs) — a target that streams
- its response; the agent forwards each `LlmResponseChunk` to the caller token-by-token.
+- [Client-backed execution](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/examples/research_agent.rs)
+- [Host-owned model calls](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/examples/research_agent_core.rs)
+- [Streaming responses](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/examples/streaming_agent.rs)
+- [Custom algorithm](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy/examples/ensemble.rs)
-## Not yet built
+## License
-- **`Signals` events** — `process_signals` / `Signals` exist but carry nothing yet.
-- **`Context` fields** — `values: HashMap<String, String>` carries the algorithm telemetry label and `state: S` the caller's per-session state; correlation ids, budgets, and deadlines still to come.
-- **Config-driven construction**.
+Licensed under the Apache License, Version 2.0.
diff --git a/crates/libsy/src/algorithms/llm_class.rs b/crates/libsy/src/algorithms/llm_class.rs
index 57726d2bc..fe87e0a91 100644
--- a/crates/libsy/src/algorithms/llm_class.rs
+++ b/crates/libsy/src/algorithms/llm_class.rs
@@ -429,7 +429,12 @@ impl Classifier<State> for EscalationClassifier {
}
}
-/// A task-level capability routing algorithm with an internal fall-through cascade.
+/// Routes each task between an efficient and capable target using an LLM judge.
+///
+/// The judge is consulted before the routed model call. Valid, confident output
+/// selects a tier; invalid output, abstention, or a judge failure falls back to the
+/// capable target. Optional session affinity can retain a prior assignment and skip
+/// the judge on later turns.
pub struct LlmTaskClassifier {
route: FallThrough<State>,
/// The active classifier — either capability-based or escalation-based.
@@ -437,7 +442,18 @@ pub struct LlmTaskClassifier {
}
impl LlmTaskClassifier {
- /// Routes according to `config` and returns errors for invalid thresholds.
+ /// Builds task-level capability routing over `efficient_target` and `capable_target`.
+ ///
+ /// `judge_target` serves the classification call and is not itself a routing
+ /// destination. When enabled, session affinity runs before the judge and can
+ /// short-circuit it with a retained assignment.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error when thresholds are outside `[0, 1]`, the elevated floor is
+ /// not above the base threshold, the output-token budget is zero, message-hash
+ /// fallback is enabled without affinity, or the packaged judge prompt/schema
+ /// cannot be loaded.
pub fn new(
judge_target: LlmTarget,
efficient_target: LlmTarget,
diff --git a/crates/libsy/src/algorithms/noop.rs b/crates/libsy/src/algorithms/noop.rs
index 86179a4fc..71118cd4f 100644
--- a/crates/libsy/src/algorithms/noop.rs
+++ b/crates/libsy/src/algorithms/noop.rs
@@ -1,8 +1,7 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
-//! No-op router. Always returns a hard coded response. Does not route to a backend.
-//! For testing.
+//! Test-only algorithm that returns a hard-coded response without calling a backend.
use std::sync::Arc;
@@ -14,11 +13,10 @@ use crate::Result;
use crate::core::algorithm::{Algorithm, Driver};
use switchyard_protocol::{Context, Decision};
-/// A routing algorithm that does not route. It returns a hard-coded response.
+/// Test helper that returns a hard-coded response without routing or model I/O.
pub struct Noop {}
-/// How [`Noop`] records which model it chose. This will be the model on the Request if any,
-/// otherwise a hard coded placeholder. Neither is actually used.
+/// Test decision carrying the inbound model or a fixed placeholder.
pub struct NoopDecision {
model: String,
}
diff --git a/crates/libsy/src/algorithms/passthrough.rs b/crates/libsy/src/algorithms/passthrough.rs
index 06b73a02d..2a922f28d 100644
--- a/crates/libsy/src/algorithms/passthrough.rs
+++ b/crates/libsy/src/algorithms/passthrough.rs
@@ -1,9 +1,7 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
-//! Router that only routes to a single backend, always.
-//! - For testing if issues come from backend or from Switchyard
-//! - For logging request / responses for RL
+//! Single-target routing for direct model calls and integration diagnostics.
use std::sync::Arc;
@@ -13,17 +11,19 @@ use crate::Result;
use crate::core::algorithm::{Algorithm, Driver, LlmTarget};
use switchyard_protocol::{Context, Decision, RoutedLlmClient};
-/// See module comment
+/// Routing algorithm that always calls one configured target.
pub struct Passthrough {
target: LlmTarget,
}
impl Passthrough {
+ /// Creates an algorithm that always calls `target`.
pub fn new(target: LlmTarget) -> Self {
Passthrough { target }
}
}
+/// Decision emitted before [`Passthrough`] calls its configured target.
pub struct PassthroughDecision {
model_id: String,
}
diff --git a/crates/libsy/src/algorithms/stage.rs b/crates/libsy/src/algorithms/stage.rs
index 8e952a72e..9fade6a50 100644
--- a/crates/libsy/src/algorithms/stage.rs
+++ b/crates/libsy/src/algorithms/stage.rs
@@ -13,8 +13,6 @@
//! joined in unchanged — and then to the picker's default tier. The judge is
//! asked per turn and its verdict is never pinned to the session.
//!
-//! Callers needing a different composition can assemble the parts themselves.
-
use std::sync::Arc;
use async_trait::async_trait;
diff --git a/crates/libsy/src/algorithms/util/escalation.rs b/crates/libsy/src/algorithms/util/escalation.rs
index 8a377462a..276d612ac 100644
--- a/crates/libsy/src/algorithms/util/escalation.rs
+++ b/crates/libsy/src/algorithms/util/escalation.rs
@@ -185,8 +185,8 @@ pub(crate) fn build_judge(
/// message is this turn — no `+ 1`. Counting the caller's request instead would report the
/// turn after the one under judgement.
///
-/// libsy receives messages already normalized by `switchyard-protocol`, so unlike the
-/// wire-format-aware Python equivalent this needs no per-format branching.
+/// Messages are already normalized by `switchyard-protocol`, so this needs no
+/// per-format branching.
pub(crate) fn conversation_turn(request: &Request) -> usize {
request
.llm_request
diff --git a/crates/libsy/src/algorithms/util/stage.rs b/crates/libsy/src/algorithms/util/stage.rs
index 1146a23b8..24e3deded 100644
--- a/crates/libsy/src/algorithms/util/stage.rs
+++ b/crates/libsy/src/algorithms/util/stage.rs
@@ -3,11 +3,9 @@
//! Stage-router scoring and tier selection — the shared routing core.
//!
-//! Given a [`ToolSignals`] (extracted by the dimension collector), this
-//! module decides whether a coding-agent turn should go to the **capable**
-//! or **efficient** tier. It is the single source of truth for
-//! the decision, shared by the Rust profile and the Python processor via
-//! bindings — only the outer shell differs in how it fetches the decision.
+//! Given [`ToolSignals`] extracted from a normalized request, this module decides
+//! whether a coding-agent turn should go to the **capable** or **efficient** tier.
+//! [`StageClassifier`] applies that decision inside [`crate::StageRouter`].
//!
//! Two axes:
//!
diff --git a/crates/libsy/src/algorithms/util/tool_signals.rs b/crates/libsy/src/algorithms/util/tool_signals.rs
index a2eb3b662..388417c45 100644
--- a/crates/libsy/src/algorithms/util/tool_signals.rs
+++ b/crates/libsy/src/algorithms/util/tool_signals.rs
@@ -3,11 +3,9 @@
//! Tool-result context signals extracted from the conversation history.
//!
-//! The [`DimensionCollector`][`crate::DimensionCollector`] runs this
-//! alongside the 15 prompt-text scorers: it walks the `messages[]` /
-//! `input[]` array of the incoming request, finds tool-execution results,
+//! The extractor walks normalized messages, finds tool calls and results,
//! pattern-matches their text against a curated error table, and aggregates
-//! conversation-history metrics that the stage_router pickers need.
+//! conversation-history metrics used by [`crate::StageRouter`].
//!
//! All logic is pure and deterministic — no I/O, no shared state.
@@ -200,8 +198,10 @@ pub const DEFAULT_RECENT_WINDOW: usize = 3;
// ─── output type ─────────────────────────────────────────────────────────────
-/// Tool-execution signals stamped on `ProxyContext`. Read by stage_router pickers
-/// via the `get_tool_result_signal` binding.
+/// Tool-execution signals extracted from a normalized [`Request`].
+///
+/// A request-side processor stores these signals in [`State`](crate::State) for
+/// [`crate::StageRouter`] and its classifier to consume.
#[derive(Clone, Debug, Default)]
pub struct ToolSignals {
/// Max severity across the recent window (last `recent_window` tool results):
@@ -211,7 +211,9 @@ pub struct ToolSignals {
pub severity: f32,
/// Consecutive clean tool results back from the most recent. `0` if the last failed.
pub no_error_streak: u32,
+ /// Total edit-style tool calls in the request.
pub edit_count: u32,
+ /// Total write-style tool calls in the request.
pub write_count: u32,
/// Read-type calls (Read tool + read-like Bash). Used by the build-pit gate.
pub read_count: u32,
@@ -244,6 +246,10 @@ pub struct ToolSignals {
}
impl ToolSignals {
+ /// Extracts tool and progress signals from `request`.
+ ///
+ /// `window_size` limits recent counters to the newest tool results. `None`
+ /// uses [`DEFAULT_RECENT_WINDOW`].
pub fn from_request(request: &Request, window_size: Option<usize>) -> Self {
extract_tool_signals_with_window(request, window_size.unwrap_or(DEFAULT_RECENT_WINDOW))
}
@@ -326,19 +332,10 @@ fn classify_tool_call(name: &str, command: Option<&str>) -> ToolCategory {
// ─── extraction entry point ───────────────────────────────────────────────────
-/// Extract all tool-execution signals from a [`Request`].
-///
-/// Dispatches on the request's wire format:
-///
-/// * **OpenAI chat** — `messages[]` with `role: "tool"` for results;
-/// `role: "assistant"` with `tool_calls[]` for call names.
-/// * **Anthropic** — `messages[]` with `role: "user"` + `content[].type:
-/// "tool_result"`; `role: "assistant"` with `content[].type: "tool_use"`.
-/// * **OpenAI responses** — `input[]` with `type: "function_call_output"`;
-/// `type: "function_call"` for call names.
+/// Extract all tool-execution signals from a normalized [`Request`].
///
-/// Returns [`ToolSignals::default()`] when the wire format or body is absent or
-/// the messages list is empty — callers can always read `signal.severity`.
+/// Returns [`ToolSignals::default()`] when the message history contains no tool
+/// activity, so callers can always inspect the signal fields.
fn extract_tool_signals_with_window(request: &Request, recent_window: usize) -> ToolSignals {
// Read the decoded conversation, not the raw body: every inbound format lands
// in the same shape here, so the signals do not depend on knowing which one it
diff --git a/crates/libsy/src/core/algorithm.rs b/crates/libsy/src/core/algorithm.rs
index 1e124ded3..69ac9d1b8 100644
--- a/crates/libsy/src/core/algorithm.rs
+++ b/crates/libsy/src/core/algorithm.rs
@@ -2,8 +2,8 @@
// SPDX-License-Identifier: Apache-2.0
//! The [`Algorithm`] trait and its [`Driver`] — the orchestration contract every
-//! routing/optimization algorithm implements, and the offload channel it makes model
-//! calls and publishes [`Decision`]s over. See the crate root for the narrative model.
+//! algorithm implements, and the offload channel it uses to make model calls and
+//! publish [`Decision`]s.
use std::{
collections::{HashMap, HashSet},
@@ -36,11 +36,13 @@ pub type StepStream = Pin<Box<dyn Stream<Item = Result<Step>> + Send>>;
/// One completed model call observed at the algorithm offload boundary.
#[derive(Clone, Debug)]
pub struct LlmCallObservation {
+ /// Model selected for the completed call.
pub selected_model: String,
/// Routing tier attached to the selected model, when present.
pub tier: Option<String>,
/// Whether this was the routed backend call rather than classifier or judge overhead.
pub is_routed: bool,
+ /// Whether the call completed successfully.
pub is_success: bool,
/// Time spent waiting for the model call to resolve.
pub duration: Duration,
@@ -58,6 +60,10 @@ pub enum RunObservation {
}
/// Request-scoped callback for algorithm-run observations.
+///
+/// The runner invokes this callback inline while resolving observations. Several
+/// runs may invoke the same observer concurrently, so implementations must be
+/// thread-safe, fast, and non-blocking.
pub type RunObserver = Arc<dyn Fn(RunObservation) + Send + Sync>;
/// A request paired with the routing [`Decision`] that produced it — the offload
@@ -411,7 +417,7 @@ impl LlmTargetSet {
}
}
-/// Matches the cap the Python stack uses. Dropping a live session's entry costs one
+/// Bounds process-local overflow history. Dropping a live session's entry costs one
/// rediscovered overflow, so the victim choice does not need to be exact.
const MAX_EVICTION_SESSIONS: usize = 1_024;
@@ -535,11 +541,25 @@ pub(crate) async fn call_llm_with_overflow_fallback(
///
/// Methods take `self: Arc<Self>`: one algorithm (`Arc<dyn Algorithm>`) is shared across
/// requests and run concurrently, so it owns its thread-safety and any shared state.
+///
+/// # Concurrency
+///
+/// A host may run the same algorithm concurrently for many requests. Implementations
+/// must synchronize their own mutable shared state. Each call to [`run_stream`](Self::run_stream)
+/// creates an independent [`Driver`], so model-call promises and emitted [`Step`]s cannot
+/// cross between runs.
+///
+/// # Observability
+///
+/// [`run_stream`](Self::run_stream) creates a `libsy.run` span, and each offloaded model
+/// call creates a `libsy.llm_call` span. [`run`](Self::run) additionally wraps calls it
+/// serves through a target's default client in `libsy.client_call`. Decisions and failures
+/// are emitted through `tracing`; metrics use the global OpenTelemetry meter provider.
#[async_trait]
pub trait Algorithm: Send + Sync + 'static {
/// Stable, low-cardinality name identifying this algorithm — the
/// `algorithm` attribute on every span, metric, and log line the crate
- /// emits for its runs (see the crate docs' Observability section).
+ /// emits for its runs.
fn name(&self) -> &str;
/// Run one request to completion: make model calls with [`Driver::call_llm_target`],
@@ -596,9 +616,14 @@ pub trait Algorithm: Send + Sync + 'static {
}
/// Process a request to completion, returning a stream of [`Step`]s.
- /// Each [`Step::CallLlm`] is an offloaded model call the consumer must serve.
- /// The stream ends with a [`Step::ReturnToAgent`] on success, or an `Err` item on failure.
- /// Report each model call to `observer`.
+ ///
+ /// The consumer must fulfill every [`Step::CallLlm`] before the algorithm can
+ /// continue. The bounded step channel applies backpressure when the consumer is
+ /// not polling. A successful run ends with [`Step::ReturnToAgent`]; a failure is
+ /// emitted as an `Err` item. Dropping the stream aborts the spawned algorithm task.
+ ///
+ /// Every invocation owns a separate [`Driver`]. `observer`, when present, receives
+ /// each completed model call and, after a successful routed run, its routing overhead.
fn run_stream(
self: Arc<Self>,
ctx: Context,
@@ -632,7 +657,7 @@ pub trait Algorithm: Send + Sync + 'static {
}
.instrument(span),
);
- // Dropping the stream aborts the algorithm task, so it doesn't keep running after the
+ // Dropping the stream aborts the algorithm task when its consumer goes away.
let abort_guard = AbortOnDrop(handle.abort_handle());
let finish_driver = driver.clone();
diff --git a/crates/libsy/src/core/classifier.rs b/crates/libsy/src/core/classifier.rs
index c417622eb..ff2dc50f4 100644
--- a/crates/libsy/src/core/classifier.rs
+++ b/crates/libsy/src/core/classifier.rs
@@ -77,7 +77,9 @@ pub trait Classifier<S = ()>: Send + Sync {
}
/// Score the classifier's targets given the current state and request.
- /// driver is optional. It is used to offload model calls
+ ///
+ /// When present, `driver` lets a classifier offload model calls. It is `None`
+ /// when the classifier is evaluated outside an algorithm run.
///
/// `request` is borrowed mutably so a classifier may rewrite it in place — inject a
/// system prompt, drop tools, compact history. The edit is not scoped to this call:
diff --git a/crates/libsy/src/core/state.rs b/crates/libsy/src/core/state.rs
index f35d4d73b..01ccb08a9 100644
--- a/crates/libsy/src/core/state.rs
+++ b/crates/libsy/src/core/state.rs
@@ -10,19 +10,25 @@ use crate::algorithms::util::tool_signals::ToolSignals;
/// A value in a session's [`State`].
#[derive(Debug, Clone)]
pub enum StateValue {
+ /// Text value.
String(String),
+ /// Nonnegative counter.
Count(u32),
+ /// Signed integer value.
Int(i32),
+ /// Floating-point value.
Scalar(f32),
}
/// Routing facts accumulated across one session's algorithm runs.
#[derive(Debug, Clone, Default)]
pub struct State {
+ /// Number of turns processed for this session.
pub turn_count: u32,
/// Tool-result signals for the current request, set by the tool-signal
/// processor. `None` until it runs or when the request has no tool activity,
/// so routers must treat absence as "no signal yet".
pub tool_signals: Option<ToolSignals>,
+ /// Algorithm-specific state keyed by stable internal names.
pub extra: HashMap<String, StateValue>,
}
diff --git a/crates/libsy/src/lib.rs b/crates/libsy/src/lib.rs
index bff81c6d7..3d4188631 100644
--- a/crates/libsy/src/lib.rs
+++ b/crates/libsy/src/lib.rs
@@ -1,80 +1,8 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
-//! # libsy — multi-LLM agent optimization (routing first)
-//!
-//! `libsy` decides, per request, *how* to serve an LLM call: which model(s) to
-//! invoke, in what order, and how to combine the results. Routing is the first
-//! and simplest case; the same interfaces also express classifier routing,
-//! ensembles, cascades, and other optimizations. The library owns no HTTP client
-//! and no provider SDK — it decides, and the host makes (or is asked to make) the
-//! actual calls — so it embeds cleanly in a proxy, gateway, or agent runtime.
-//!
-//! ## The model
-//!
-//! - An [`Algorithm`] is the optimization *algorithm*. Its
-//! [`create_run_task`](Algorithm::create_run_task) runs once per request
-//! and makes as many model calls as it needs — via [`Driver::call_llm_target`], which look
-//! like ordinary calls — publishes its [`Decision`](switchyard_protocol::Decision)s with [`Driver::info`], and
-//! returns the final [`Response`](switchyard_protocol::Response). The provided
-//! [`run_stream`](Algorithm::run_stream) drives that on its own task and hands
-//! back a stream of [`Step`]s; [`run`](Algorithm::run) runs
-//! it to completion with the targets' default clients.
-//! - An [`LlmTarget`] names a routing target by its [`semantic_name`](LlmTarget::semantic_name).
-//! Every call is *offloaded* to the request's stream as a [`Step::CallLlm`]; the
-//! target's [`RoutedLlmClient`](switchyard_protocol::RoutedLlmClient), if any, rides along as
-//! [`RoutedRequest::default_client`] so the host can serve it by default or
-//! override it (see below).
-//!
-//! ## Running a request
-//!
-//! Hold the algorithm as `Arc<dyn Algorithm>` and call one of two provided methods:
-//!
-//! - [`run`](Algorithm::run) — run to completion, serving each
-//! offloaded call via its [`RoutedRequest::default_client`], and return the decision
-//! trace plus the final [`Response`](switchyard_protocol::Response). The simplest integration; use it when the
-//! algorithm holds the model clients (it errors if a routed target has no client).
-//! - [`run_stream`](Algorithm::run_stream) — return a stream of [`Step`]s. Each
-//! model call is offloaded: the stream yields a [`Step::CallLlm`] carrying a promise;
-//! the host performs the real model call (optionally via the promise's
-//! `default_client`) and fulfills it with [`CallLlmRequest::respond`]. Decisions
-//! arrive as [`Step::Decision`] as the algorithm makes them, and the run ends with a
-//! [`Step::ReturnToAgent`] carrying the final response. The step stream is bounded,
-//! so pulling it paces the algorithm one step at a time — an "ask, don't call" mode
-//! that lets a host that owns its transport keep control of every call.
-//!
-//! ## Concurrency
-//!
-//! [`Algorithm::create_run_task`] takes `self: Arc<Self>`, so one shared
-//! `Arc<dyn Algorithm>` (no lock) serves many requests in parallel. Each
-//! [`run_stream`](Algorithm::run_stream) call builds its own [`Driver`], so
-//! offloaded calls never cross between concurrent requests. An algorithm is
-//! responsible for its own thread-safety — stateless (like the reference routers) or
-//! interior mutability over just its own state.
-//!
-//! ## Observability
-//!
-//! The provided run methods instrument every algorithm from the outside — at the
-//! [`Decision`](switchyard_protocol::Decision) hook and the offload boundary — so algorithms carry no telemetry
-//! code. Each run gets a `libsy.run` tracing span (correlation ids from
-//! [`Metadata`](switchyard_protocol::Metadata) attached) with a child `libsy.llm_call` span per model call
-//! (fulfillment time as the algorithm observes it) plus a `libsy.client_call`
-//! span around the actual API call when [`run`](Algorithm::run) serves it;
-//! each [`Driver::info`] decision is logged with its reasoning; and OpenTelemetry
-//! metrics record run/call counts, latency, and published decisions, keyed by
-//! [`Algorithm::name`] plus `selected_model` and `outcome`.
-//! Metrics use the global meter provider and spans/logs the `tracing` facade: a
-//! host that installs an OTel SDK and a `tracing` subscriber (bridged with
-//! `tracing-opentelemetry` for OTLP spans) gets the full signal set; with
-//! neither installed, everything is a no-op.
-//!
-//! ## Algorithms
-//!
-//! Concrete algorithms live in [`algorithms`]:
-//!
-//! [`algorithms::Random`] provides uniform or weighted random routing.
-//!
-//! [`algorithms::LlmTaskClassifier`] uses one model to classify and route to its selected target.
+#![warn(missing_docs)]
+#![doc = include_str!("../README.md")]
mod core;
pub use core::algorithm::{
@@ -112,7 +40,9 @@ mod observability;
/// Registers process-wide compatibility gauges with the global meter provider.
///
-/// Hosts should call this after installing their OpenTelemetry meter provider.
+/// Hosts should call this after installing their OpenTelemetry meter provider. Other
+/// libsy metrics are created when recorded and do not require initialization. Spans
+/// and structured logs require the host to install a `tracing` subscriber separately.
pub fn initialize_metrics() {
observability::initialize_metrics();
}
diff --git a/crates/protocol/Cargo.toml b/crates/protocol/Cargo.toml
index c249dfa66..d054df339 100644
--- a/crates/protocol/Cargo.toml
+++ b/crates/protocol/Cargo.toml
@@ -12,7 +12,6 @@ license.workspace = true
readme = "README.md"
repository.workspace = true
rust-version.workspace = true
-documentation = "https://docs.rs/switchyard-protocol"
keywords = ["llm", "protocol", "openai", "anthropic"]
publish = ["crates-io"]
diff --git a/crates/protocol/README.md b/crates/protocol/README.md
index 6b41c950a..555debc33 100644
--- a/crates/protocol/README.md
+++ b/crates/protocol/README.md
@@ -1,13 +1,111 @@
# switchyard-protocol
-Provider-neutral request, response, content, and wire-format types shared by Switchyard's Rust
-crates.
+Provider-neutral request, response, streaming, routing, and metadata types
+shared by Switchyard algorithms, clients, and translation codecs. This crate
+defines contracts; it does not route, translate, or perform network calls.
-This is the schema dependency for [`switchyard-translation`](https://crates.io/crates/switchyard-translation).
-Most applications should depend on `switchyard-translation`, which re-exports these types alongside
-the translation engine.
+## Setup
+
+```toml
+[dependencies]
+switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
+serde_json = "1"
+```
+
+## Main types
+
+| Area | Types |
+|---|---|
+| Conversation | [`LlmRequest`], [`Message`], [`InstructionBlock`], [`ContentBlock`] |
+| Tools | [`ToolDefinition`], [`ToolChoice`], [`ToolCall`], [`ToolResult`] |
+| Response | [`AggLlmResponse`], [`ResponseOutput`], [`Usage`], [`StopReason`] |
+| Streaming | [`LlmResponse`], [`LlmResponseChunk`], [`LlmResponseStream`] |
+| Envelope | [`Context`], [`Request`], [`Response`], [`Metadata`] |
+| Routing I/O | [`Decision`], [`RoutedLlmClient`], [`LlmClientError`] |
+| Wire identity | [`WireFormat`], [`FormatId`] |
+
+## Simple request
+
+```rust
+use switchyard_protocol::{ContentBlock, LlmRequest, Message, Role};
+
+let request = LlmRequest {
+ model: Some("provider/model".into()),
+ messages: vec![Message {
+ role: Role::User,
+ content: vec![ContentBlock::Text {
+ text: "Explain tail latency".into(),
+ }],
+ }],
+ ..LlmRequest::default()
+};
+
+assert_eq!(request.model.as_deref(), Some("provider/model"));
+assert_eq!(request.messages.len(), 1);
+```
+
+## Detailed request
+
+Construct the normalized [`Request`] directly when routing needs instructions,
+tools, generation controls, and correlation metadata:
+
+```rust
+use serde_json::json;
+use switchyard_protocol::{
+ ContentBlock, InstructionBlock, LlmRequest, Message, Metadata, OutputParams,
+ Request, Role, SamplingParams, ToolChoice, ToolDefinition,
+};
+
+let request = Request {
+ llm_request: LlmRequest {
+ model: Some("provider/model".into()),
+ instructions: vec![InstructionBlock {
+ role: Role::System,
+ content: vec![ContentBlock::Text {
+ text: "Answer with concise operational guidance.".into(),
+ }],
+ }],
+ messages: vec![Message::text(Role::User, "Why is p99 latency rising?")],
+ tools: vec![ToolDefinition {
+ name: "lookup_metric".into(),
+ description: Some("Read one service metric".into()),
+ parameters: json!({
+ "type": "object",
+ "properties": { "name": { "type": "string" } },
+ "required": ["name"]
+ }),
+ strict: Some(true),
+ }],
+ tool_choice: Some(ToolChoice::Auto),
+ sampling: SamplingParams {
+ temperature: Some(0.2),
+ ..SamplingParams::default()
+ },
+ output: OutputParams {
+ max_output_tokens: Some(512),
+ ..OutputParams::default()
+ },
+ stream: true,
+ ..LlmRequest::default()
+ },
+ metadata: Some(Metadata {
+ session_id: Some("session-42".into()),
+ correlation_id: Some("request-7".into()),
+ ..Metadata::default()
+ }),
+ ..Request::default()
+};
+
+assert_eq!(request.llm_request.tools[0].name, "lookup_metric");
+```
+
+## Response forms
+
+[`LlmResponse`] contains either a completed [`AggLlmResponse`] or a
+single-consumption [`LlmResponseStream`] of [`LlmResponseChunk`] values. See
+[`LlmResponse::into_agg`] for aggregation and [`PreservationMetadata`],
+[`Usage`], and [`Metadata`] for the data retained around a response.
## License
-Licensed under the Apache License, Version 2.0. See the
-[Switchyard repository](https://github.com/NVIDIA-NeMo/Switchyard) for details.
+Licensed under the Apache License, Version 2.0.
diff --git a/crates/protocol/src/client.rs b/crates/protocol/src/client.rs
index e496c07a3..233ddb38c 100644
--- a/crates/protocol/src/client.rs
+++ b/crates/protocol/src/client.rs
@@ -30,7 +30,10 @@ pub type BoxError = Box<dyn std::error::Error + Send + Sync + 'static>;
pub enum LlmClientError {
/// The request cannot be served as supplied.
#[error("invalid request: {message}")]
- InvalidRequest { message: String },
+ InvalidRequest {
+ /// Human-readable request validation failure.
+ message: String,
+ },
/// Decoding the inbound request failed in the translation engine.
#[error("request translation failed: {0}")]
@@ -46,7 +49,10 @@ pub enum LlmClientError {
/// The client is not configured to serve the selected target.
#[error("client configuration error: {message}")]
- Configuration { message: String },
+ Configuration {
+ /// Human-readable configuration failure.
+ message: String,
+ },
/// The upstream could not be reached or the request could not be sent.
#[error("upstream transport error: {source}")]
@@ -131,7 +137,9 @@ pub trait Decision: Send + Sync {
/// A minimal [`Decision`] implementation for one-off calls that don't belong to a
/// named algorithm step — judge side calls, classifier side calls, etc.
pub struct SimpleDecision {
+ /// Model or semantic target selected for the call.
pub selected_model: String,
+ /// Optional explanation recorded with the call.
pub reasoning: Option<String>,
}
@@ -153,6 +161,12 @@ impl Decision for SimpleDecision {
/// library does not own — a host implements it over its own transport (HTTP SDK,
/// in-process model, mock). It serves a call the stream consumer chose not to
/// override, reached as a routed request's `default_client`.
+///
+/// # Concurrency
+///
+/// A client may be shared by many targets and concurrent algorithm runs. Calls may
+/// overlap, so implementations must synchronize mutable state internally and should
+/// not serialize requests unless their transport requires it.
#[async_trait]
pub trait RoutedLlmClient: Send + Sync {
/// Serve the call, returning the model's response. Call the model named by
diff --git a/crates/protocol/src/envelope.rs b/crates/protocol/src/envelope.rs
index 8399f936e..7139d9380 100644
--- a/crates/protocol/src/envelope.rs
+++ b/crates/protocol/src/envelope.rs
@@ -40,20 +40,24 @@ impl Context {
#[derive(Clone)]
pub struct Signals {}
-/// A request an algorithm routes: the normalized [`LlmRequest`] plus the original
-/// provider payload and correlation [`Metadata`].
+/// A request an algorithm routes: the normalized [`LlmRequest`] plus optional
+/// host-owned raw data and correlation [`Metadata`].
#[derive(Clone, Default)]
pub struct Request {
/// The normalized request an algorithm routes.
pub llm_request: LlmRequest,
- /// The original provider-shaped request body, if the host wants to forward it
- /// verbatim (e.g. a proxy preserving messages/params). libsy does not read it.
+ /// An optional whole request body retained by the host. libsy and the translation
+ /// codecs do not read it. Exact same-format codec replay instead uses
+ /// [`LlmRequest::preservation`].
pub raw_request: Option<serde_json::Value>,
/// Correlation metadata carried through the request.
pub metadata: Option<Metadata>,
}
impl Request {
+ /// Returns the model name supplied by the inbound request, when present.
+ ///
+ /// This is not necessarily the target selected by a routing decision.
pub fn requested_model(&self) -> Option<&str> {
self.llm_request.model.as_deref()
}
@@ -71,6 +75,10 @@ pub struct Response {
}
impl Response {
+ /// Returns the model recorded by a buffered response.
+ ///
+ /// Returns `None` for a live stream because inspecting its `MessageStart`
+ /// event would require consuming the stream.
pub fn selected_model(&self) -> Option<&str> {
self.llm_response.selected_model()
}
diff --git a/crates/protocol/src/format.rs b/crates/protocol/src/format.rs
index 5a24c9185..745f4120d 100644
--- a/crates/protocol/src/format.rs
+++ b/crates/protocol/src/format.rs
@@ -11,10 +11,13 @@ use serde::{Deserialize, Serialize};
/// Built-in provider API formats.
#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize, Deserialize)]
pub enum WireFormat {
+ /// OpenAI Chat Completions API.
#[serde(rename = "openai_chat")]
OpenAiChat,
+ /// Anthropic Messages API.
#[serde(rename = "anthropic_messages")]
AnthropicMessages,
+ /// OpenAI Responses API.
#[serde(rename = "openai_responses")]
OpenAiResponses,
}
diff --git a/crates/protocol/src/lib.rs b/crates/protocol/src/lib.rs
index 543f2d0bb..2747a12e7 100644
--- a/crates/protocol/src/lib.rs
+++ b/crates/protocol/src/lib.rs
@@ -1,24 +1,8 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
-//! Shared request/response protocol for libsy — the vocabulary an [`Algorithm`] reasons
-//! over, decoupled from libsy's orchestration.
-//!
-//! This crate owns Switchyard's neutral conversation IR: [`LlmRequest`] (model, messages,
-//! tools, sampling, …), the buffered [`AggLlmResponse`] (outputs, usage, …), and its
-//! streaming counterpart [`LlmResponseChunk`]; the [`Request`]/[`Response`] envelope that
-//! pairs them with correlation [`Metadata`]; plus the wire-[`format`] identifiers
-//! translation keys off. `switchyard-translation` re-exports the IR types under its own
-//! `ConversationRequest` / `ConversationResponse` / `ConversationStreamEvent` names. The IR
-//! carries no bare `prompt`/`completion`; the [`text_request`] / [`prompt_text`] /
-//! [`text_response`] / [`completion_text`] helpers bridge to and from plain text for the
-//! common single-turn case.
-//!
-//! The streamed-response type itself — a live stream of chunks *or* the terminal
-//! aggregate — is the [`LlmResponse`] enum; it owns a `futures::Stream`, so it is the one
-//! non-`Clone`, non-data type in this crate.
-//!
-//! [`Algorithm`]: https://docs.rs/libsy
+#![warn(missing_docs)]
+#![doc = include_str!("../README.md")]
pub mod client;
pub mod envelope;
@@ -34,7 +18,10 @@ pub use llm::*;
pub use metadata::*;
pub use stream::*;
-/// Build a single-turn request: one user message carrying `prompt`, for `model`.
+/// Builds a single-turn request: one user message carrying `prompt`, for `model`.
+///
+/// This deliberately creates only the common text shape. Construct [`LlmRequest`]
+/// directly for instructions, tools, multimodal content, or sampling controls.
pub fn text_request(model: Option<String>, prompt: impl Into<String>) -> LlmRequest {
LlmRequest {
model,
@@ -43,8 +30,10 @@ pub fn text_request(model: Option<String>, prompt: impl Into<String>) -> LlmRequ
}
}
-/// The user's prompt text — the text of every user message, joined by newlines. Empty
-/// when the request has no user text.
+/// Returns a lossy text view of all user messages, joined by newlines.
+///
+/// Only text and refusal blocks are included. Instructions, tool content,
+/// reasoning, and media are omitted. Returns an empty string when no user text exists.
pub fn prompt_text(request: &LlmRequest) -> String {
request
.messages
@@ -55,7 +44,10 @@ pub fn prompt_text(request: &LlmRequest) -> String {
.join("\n")
}
-/// Build a single-turn response: one assistant message carrying `completion`, for `model`.
+/// Builds a single-turn response: one assistant message carrying `completion`, for `model`.
+///
+/// Construct [`AggLlmResponse`] directly when usage, tools, reasoning, or multiple
+/// output items must be represented.
pub fn text_response(model: Option<String>, completion: impl Into<String>) -> AggLlmResponse {
AggLlmResponse {
model,
@@ -70,8 +62,11 @@ pub fn text_response(model: Option<String>, completion: impl Into<String>) -> Ag
}
}
-/// The assistant's completion text — the text blocks of the first output, concatenated.
-/// Empty when the response has no textual output.
+/// Returns a lossy text view of the first assistant output.
+///
+/// Only text blocks from the first output are concatenated. Refusals, reasoning,
+/// tools, media, and additional outputs are omitted. Returns an empty string when
+/// no such text exists.
pub fn completion_text(response: &AggLlmResponse) -> String {
response
.outputs
diff --git a/crates/protocol/src/llm.rs b/crates/protocol/src/llm.rs
index cda3eb825..d8def175a 100644
--- a/crates/protocol/src/llm.rs
+++ b/crates/protocol/src/llm.rs
@@ -14,24 +14,33 @@ use crate::format::FormatId;
#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum Role {
+ /// System-level instructions.
System,
+ /// Developer-level instructions supported by some provider APIs.
Developer,
+ /// End-user input.
User,
+ /// Model-generated output.
Assistant,
+ /// Tool execution output.
Tool,
}
/// Instruction content separated from normal conversation messages.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct InstructionBlock {
+ /// Instruction authority, normally [`Role::System`] or [`Role::Developer`].
pub role: Role,
+ /// Ordered instruction content.
pub content: Vec<ContentBlock>,
}
/// One normalized conversation message.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct Message {
+ /// Actor that produced the message.
pub role: Role,
+ /// Ordered message content.
pub content: Vec<ContentBlock>,
}
@@ -67,32 +76,52 @@ impl Message {
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum ContentBlock {
+ /// Plain text content.
Text {
+ /// Text value.
text: String,
},
+ /// Model reasoning or thinking content.
Reasoning {
+ /// Reasoning text.
text: String,
+ /// Provider signature used to validate or continue the reasoning block.
signature: Option<String>,
},
+ /// Image content.
Image {
+ /// Image location or inline payload.
source: ImageSource,
},
+ /// Audio content.
Audio {
+ /// Audio location or inline payload.
source: MediaSource,
},
+ /// Video content.
Video {
+ /// Video location or inline payload.
source: MediaSource,
},
+ /// File content.
File {
+ /// File identifier or inline payload.
source: FileSource,
},
+ /// Tool invocation requested by the assistant.
ToolCall(ToolCall),
+ /// Result of an earlier tool invocation.
ToolResult(ToolResult),
+ /// Provider refusal content.
Refusal {
+ /// Human-readable refusal text.
text: String,
},
+ /// Provider block that has no normalized representation.
Unknown {
+ /// Wire format that supplied the block.
provider: FormatId,
+ /// Exact provider block.
raw: Value,
},
}
@@ -101,14 +130,21 @@ pub enum ContentBlock {
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
#[serde(tag = "type", content = "data", rename_all = "snake_case")]
pub enum ImageSource {
+ /// Image fetched from a URL.
Url {
+ /// Image URL.
url: String,
+ /// Optional provider-specific image detail level.
detail: Option<String>,
},
+ /// Base64-encoded image bytes.
Base64 {
+ /// MIME type, when supplied.
media_type: Option<String>,
+ /// Base64-encoded bytes.
data: String,
},
+ /// Provider image source with no normalized representation.
Raw(Value),
}
@@ -116,11 +152,16 @@ pub enum ImageSource {
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
#[serde(tag = "type", content = "data", rename_all = "snake_case")]
pub enum FileSource {
+ /// Provider-managed file identifier.
FileId(String),
+ /// Inline file payload.
FileData {
+ /// Encoded or textual file data as supplied by the provider format.
data: String,
+ /// Original filename, when supplied.
filename: Option<String>,
},
+ /// Provider file source with no normalized representation.
Raw(Value),
}
@@ -128,39 +169,56 @@ pub enum FileSource {
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
#[serde(tag = "type", content = "data", rename_all = "snake_case")]
pub enum MediaSource {
+ /// Media fetched from a URL.
Url {
+ /// Media URL.
url: String,
+ /// MIME type, when supplied.
media_type: Option<String>,
},
+ /// Base64-encoded media bytes.
Base64 {
+ /// MIME type, when supplied.
media_type: Option<String>,
+ /// Base64-encoded bytes.
data: String,
},
+ /// Provider media source with no normalized representation.
Raw(Value),
}
/// Normalized assistant tool call.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct ToolCall {
+ /// Provider tool-call identifier used to pair the result.
pub id: String,
+ /// Tool name.
pub name: String,
+ /// Parsed tool arguments.
pub arguments: Value,
}
/// Normalized tool result message content.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct ToolResult {
+ /// Identifier of the [`ToolCall`] this result answers.
pub tool_call_id: String,
+ /// Ordered tool output content.
pub content: Vec<ContentBlock>,
+ /// Whether tool execution failed, when the provider reports it.
pub is_error: Option<bool>,
}
/// Normalized tool definition.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct ToolDefinition {
+ /// Tool name exposed to the model.
pub name: String,
+ /// Human-readable tool description.
pub description: Option<String>,
+ /// JSON Schema describing accepted arguments.
pub parameters: Value,
+ /// Whether the provider should enforce the schema strictly.
pub strict: Option<bool>,
}
@@ -168,32 +226,47 @@ pub struct ToolDefinition {
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
#[serde(tag = "type", content = "data", rename_all = "snake_case")]
pub enum ToolChoice {
+ /// Let the model decide whether to call a tool.
Auto,
+ /// Require at least one tool call.
Required,
+ /// Disallow tool calls.
None,
- Tool { name: String },
+ /// Require a specific tool.
+ Tool {
+ /// Required tool name.
+ name: String,
+ },
+ /// Provider tool-choice value with no normalized representation.
Raw(Value),
}
/// Provider sampling parameters with common cross-provider names.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct SamplingParams {
+ /// Sampling temperature.
pub temperature: Option<f64>,
+ /// Nucleus-sampling probability mass.
pub top_p: Option<f64>,
+ /// Maximum number of candidate tokens considered at each step.
pub top_k: Option<i64>,
}
/// Output budget and structured-output options.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct OutputParams {
+ /// Maximum tokens the model may generate.
pub max_output_tokens: Option<u64>,
+ /// Provider-neutral or provider-specific structured-output configuration.
pub response_format: Option<Value>,
}
/// Provider reasoning controls preserved by translation.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct ReasoningParams {
+ /// Requested reasoning effort or level.
pub effort: Option<String>,
+ /// Provider reasoning controls without a normalized field.
pub raw: Option<Value>,
}
@@ -201,14 +274,25 @@ pub struct ReasoningParams {
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
#[serde(default)]
pub struct ProviderExtensions {
+ /// Provider fields keyed by their original wire names.
+ ///
+ /// Codecs use these fields when translating to another format. Exact
+ /// same-format replay is controlled separately by [`PreservationMetadata`].
pub fields: Map<String, Value>,
}
-/// Exact source payloads retained for lossless round trips.
+/// Exact source payloads retained for lossless same-format round trips.
+///
+/// Translation's default preservation policy prefers a stored same-format body
+/// over reconstructing one from normalized fields. A caller that mutates the IR
+/// must clear the corresponding entry or use a policy with preservation disabled
+/// when those mutations must be encoded.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
#[serde(default)]
pub struct PreservationMetadata {
+ /// Original request bodies keyed by source format.
pub requests: BTreeMap<FormatId, Value>,
+ /// Original response bodies keyed by source format.
pub responses: BTreeMap<FormatId, Value>,
}
@@ -216,16 +300,29 @@ pub struct PreservationMetadata {
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
#[serde(default)]
pub struct LlmRequest {
+ /// Model requested by the inbound client.
pub model: Option<String>,
+ /// System and developer instructions separated from conversation turns.
pub instructions: Vec<InstructionBlock>,
+ /// Ordered conversation messages.
pub messages: Vec<Message>,
+ /// Tools available to the model.
pub tools: Vec<ToolDefinition>,
+ /// Policy controlling model tool selection.
pub tool_choice: Option<ToolChoice>,
+ /// Common sampling controls.
pub sampling: SamplingParams,
+ /// Output budget and shape controls.
pub output: OutputParams,
+ /// Reasoning controls.
pub reasoning: ReasoningParams,
+ /// Whether the caller requested a streamed response.
pub stream: bool,
+ /// Provider fields without first-class normalized equivalents.
pub extensions: ProviderExtensions,
+ /// Exact provider bodies used by codecs for lossless same-format round trips.
+ /// This is separate from a host's optional
+ /// [`Request::raw_request`](crate::Request::raw_request).
pub preservation: PreservationMetadata,
}
@@ -235,21 +332,31 @@ pub struct Usage {
/// Non-cached input tokens. Provider codecs normalize aggregate OpenAI
/// input counts by subtracting the cache detail fields.
pub input_tokens: Option<u64>,
+ /// Cache-read and cache-creation token detail, when reported.
#[serde(flatten)]
pub cache: Option<Box<InputCacheUsage>>,
+ /// Generated output tokens, excluding reasoning detail when the provider reports it separately.
pub output_tokens: Option<u64>,
+ /// Provider-reported or codec-computed total token count.
+ ///
+ /// Codecs normalize OpenAI aggregate input counts before computing totals, so
+ /// this may equal non-cached input plus cache detail plus output.
pub total_tokens: Option<u64>,
+ /// Reasoning output tokens, when reported separately.
pub reasoning_tokens: Option<u64>,
}
/// Optional cache-token detail kept out of the common, cache-free usage allocation.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct InputCacheUsage {
+ /// Input tokens read from a provider cache.
pub cached_input_tokens: Option<u64>,
+ /// Input tokens written into a provider cache.
pub cache_creation_input_tokens: Option<u64>,
}
impl Usage {
+ /// Builds cache detail when at least one count is present.
pub fn cache_details(
cached_input_tokens: Option<u64>,
cache_creation_input_tokens: Option<u64>,
@@ -263,24 +370,28 @@ impl Usage {
}))
}
+ /// Returns the cache-read input-token count.
pub fn cached_input_tokens(&self) -> Option<u64> {
self.cache
.as_ref()
.and_then(|cache| cache.cached_input_tokens)
}
+ /// Returns the cache-creation input-token count.
pub fn cache_creation_input_tokens(&self) -> Option<u64> {
self.cache
.as_ref()
.and_then(|cache| cache.cache_creation_input_tokens)
}
+ /// Sets the cache-read count, allocating cache detail when needed.
pub fn set_cached_input_tokens(&mut self, value: u64) {
self.cache
.get_or_insert_with(|| Box::new(InputCacheUsage::default()))
.cached_input_tokens = Some(value);
}
+ /// Sets the cache-creation count, allocating cache detail when needed.
pub fn set_cache_creation_input_tokens(&mut self, value: u64) {
self.cache
.get_or_insert_with(|| Box::new(InputCacheUsage::default()))
@@ -292,34 +403,47 @@ impl Usage {
#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum StopReason {
+ /// Model completed its turn normally.
EndTurn,
+ /// Model reached the configured output limit.
MaxTokens,
+ /// Model stopped to request a tool call.
ToolUse,
+ /// Provider safety or content filtering stopped generation.
ContentFilter,
+ /// Generation terminated because of an error.
Error,
+ /// Provider stop reason with no normalized equivalent.
Unknown,
}
/// One assistant output item in a normalized response.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct ResponseOutput {
+ /// Actor that produced the output, normally [`Role::Assistant`].
pub role: Role,
+ /// Ordered output content.
pub content: Vec<ContentBlock>,
+ /// Why this output stopped, when known.
pub stop_reason: Option<StopReason>,
}
/// Normalized, fully-buffered response — the aggregate of a completed generation.
-/// libsy refers to this as an `AggLlmResponse` (the terminal form of a streamed
-/// [`LlmResponse`](crate::LlmResponse)); `switchyard-translation` re-exports it as
-/// `ConversationResponse`.
+/// This is the terminal form of a streamed [`LlmResponse`](crate::LlmResponse).
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
#[serde(default)]
pub struct AggLlmResponse {
+ /// Provider response identifier.
pub id: Option<String>,
+ /// Model reported by the provider.
pub model: Option<String>,
+ /// Ordered response output items.
pub outputs: Vec<ResponseOutput>,
+ /// Normalized token usage.
pub usage: Usage,
+ /// Provider response fields without normalized equivalents.
pub extensions: ProviderExtensions,
+ /// Exact provider bodies retained for lossless round trips.
pub preservation: PreservationMetadata,
}
diff --git a/crates/protocol/src/metadata.rs b/crates/protocol/src/metadata.rs
index ae18b3cd7..890000e52 100644
--- a/crates/protocol/src/metadata.rs
+++ b/crates/protocol/src/metadata.rs
@@ -4,8 +4,8 @@
//! Correlation metadata and harness header normalization.
//!
//! [`Metadata`] is the correlation/routing envelope carried alongside a request or
-//! response. [`Metadata::from_headers`] normalizes the harness-specific HTTP headers
-//! that Claude Code, Codex, NeMo Relay, and Dynamo attach into that neutral shape.
+//! response. [`Metadata::from_headers`] normalizes host-specific HTTP headers into
+//! that neutral shape.
use std::collections::BTreeMap;
@@ -34,11 +34,11 @@ const SWITCHYARD_TURN_ID_HEADER: &str = "x-switchyard-turn-id";
const SWITCHYARD_REQUEST_ID_HEADER: &str = "x-switchyard-request-id";
const SWITCHYARD_SESSION_FINAL_HEADER: &str = "x-switchyard-session-final";
-// NeMo Relay correlation headers.
+// Correlation-header aliases used by integrating hosts.
const RELAY_SESSION_ID_HEADER: &str = "x-nemo-relay-session-id";
const RELAY_SUBAGENT_ID_HEADER: &str = "x-nemo-relay-subagent-id";
-// Dynamo correlation headers.
+// Additional correlation-header aliases used by integrating hosts.
const DYNAMO_SESSION_ID_HEADER: &str = "x-dynamo-session-id";
const DYNAMO_PARENT_SESSION_ID_HEADER: &str = "x-dynamo-parent-session-id";
const DYNAMO_SESSION_FINAL_HEADER: &str = "x-dynamo-session-final";
@@ -196,14 +196,14 @@ pub struct Metadata {
impl Metadata {
/// Normalizes harness-specific request headers into correlation metadata.
///
- /// Explicit `x-switchyard-*` headers win. NeMo Relay and Dynamo correlation
- /// headers are accepted for observability without driving routing. Codex's
- /// structured turn metadata is preferred over its compatibility projections.
- /// Claude Code carries agent lineage in native headers: a request with a
- /// non-empty `x-claude-code-agent-id` is treated as a child agent (its parent
+ /// Explicit `x-switchyard-*` headers win. Compatible host headers can populate
+ /// correlation and session metadata, but do not by themselves mark a request as
+ /// delegated work. Structured turn metadata is preferred over compatibility
+ /// projections. A request with a non-empty `x-claude-code-agent-id` is treated
+ /// as a child agent (its parent
/// inferred from the session when not stated). Sub-agent routing status is taken
/// from an explicit `x-switchyard-is-subagent` header when present, and otherwise
- /// inferred from Claude Code lineage or Codex harness signals.
+ /// inferred from native agent lineage or structured harness signals.
pub fn from_headers(headers: &BTreeMap<String, String>) -> Self {
let headers = &normalize_headers(headers);
@@ -240,10 +240,10 @@ impl Metadata {
/// Returns `(parent_agent_id, is_subagent, is_delegated_work)` from the headers.
///
-/// Recognized sub-agent signals: Claude Code `x-claude-code-agent-id`, Codex
-/// `x-openai-subagent` / `x-codex-turn-metadata.subagent_kind`, and explicit
-/// `x-switchyard-is-subagent`. Correlation-only headers (Relay, Dynamo, OpenCode
-/// parent sessions) populate observability fields but do not drive routing.
+/// Recognized sub-agent signals include `x-claude-code-agent-id`,
+/// `x-openai-subagent`, `x-codex-turn-metadata.subagent_kind`, and explicit
+/// `x-switchyard-is-subagent`. Other host correlation and parent-session headers
+/// may populate metadata but do not drive sub-agent classification.
///
/// `is_delegated_work` is computed from raw harness signals, not from `agent_kind`,
/// which may be set by an unrelated operator label (`x-switchyard-agent-kind`).
@@ -257,8 +257,8 @@ fn parse_sub_agent(headers: &BTreeMap<String, String>) -> (Option<String>, bool,
let harness_kind = resolve_path(headers, CODEX_SUBAGENT_KIND_PATH)
.or_else(|| header(headers, OPENAI_SUBAGENT_HEADER).map(str::to_string));
- // Parent resolved via HEADER_CONFIG precedence (covers Dynamo/Codex correlation);
- // falls back to the Claude Code session the child was spawned under.
+ // Resolve the parent through the configured header precedence, then fall back
+ // to the native agent session the child was spawned under.
let parent = sy_header(headers, SWITCHYARD_PARENT_AGENT_ID_HEADER)
.or_else(|| claude_parent.map(str::to_string));
@@ -464,7 +464,7 @@ mod tests {
#[test]
fn normalizes_relay_and_dynamo_child_headers() {
- // Relay and Dynamo headers are correlation data, not routing signals.
+ // Integrating-host headers are correlation data, not routing signals.
// They populate observability fields but must not trigger sub-agent routing.
let headers = BTreeMap::from([
(
diff --git a/crates/protocol/src/stream.rs b/crates/protocol/src/stream.rs
index 5fd82e687..ce7b15d92 100644
--- a/crates/protocol/src/stream.rs
+++ b/crates/protocol/src/stream.rs
@@ -27,14 +27,16 @@ const MID_STREAM_UPSTREAM_STATUS: u16 = 502;
pub type LlmResponseStream =
Pin<Box<dyn Stream<Item = Result<LlmResponseChunk, LlmClientError>> + Send>>;
-/// A model response: either a live [`Stream`](LlmResponse::Stream) of chunks or the
-/// terminal buffered [`Agg`](LlmResponse::Agg)regate.
+/// A model response: either a live [`Stream`](LlmResponse::Stream) of chunks or a
+/// terminal buffered [`LlmResponse::Agg`] response.
///
/// Not `Clone` — the `Stream` variant owns a single-consumption stream. A buffered
/// backend returns `Agg` directly; a streaming one returns `Stream` and the consumer
/// drives it, folding to an [`AggLlmResponse`] when it needs the whole response.
pub enum LlmResponse {
+ /// Live, single-consumption response stream.
Stream(LlmResponseStream),
+ /// Fully buffered terminal response.
Agg(AggLlmResponse),
}
@@ -79,6 +81,10 @@ impl LlmResponse {
}
}
+ /// Returns the model recorded by a buffered response.
+ ///
+ /// Returns `None` for a live stream because reading its model-bearing
+ /// [`LlmResponseChunk::MessageStart`] event would consume the stream.
pub fn selected_model(&self) -> Option<&str> {
match self {
LlmResponse::Agg(agg) => agg.model.as_deref(),
@@ -94,6 +100,10 @@ impl AggLlmResponse {
/// Useful when a caller had to buffer the response (e.g. to judge it) but the
/// downstream expects `LlmResponse::Stream` — for instance when `stream: true` was
/// requested and the algorithm had to aggregate before it could return.
+ ///
+ /// This conversion is lossy: only text, reasoning, and tool-call content has
+ /// a synthetic chunk representation. Refusals, tool results, media, files,
+ /// unknown blocks, response extensions, and preservation metadata are omitted.
pub fn into_stream(self) -> LlmResponseStream {
let mut chunks: Vec<LlmResponseChunk> = Vec::new();
chunks.push(LlmResponseChunk::MessageStart {
@@ -145,36 +155,56 @@ impl AggLlmResponse {
}
/// One provider-neutral streaming event — the normalized counterpart to
-/// [`AggLlmResponse`](crate::AggLlmResponse), sitting between stream decoders and
-/// encoders. `switchyard-translation` re-exports it as `ConversationStreamEvent`.
+/// [`AggLlmResponse`], sitting between stream decoders and encoders.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub enum LlmResponseChunk {
+ /// Starts a response message.
MessageStart {
+ /// Provider response identifier.
id: Option<String>,
+ /// Model reported by the provider.
model: Option<String>,
},
+ /// Adds text to one output index.
TextDelta {
+ /// Provider output index.
index: usize,
+ /// Text fragment.
text: String,
},
+ /// Adds reasoning text to one output index.
ReasoningDelta {
+ /// Provider output index.
index: usize,
+ /// Reasoning fragment.
text: String,
},
+ /// Adds or updates a tool call at one index.
ToolCallDelta {
+ /// Tool-call index within the response.
index: usize,
+ /// Tool-call identifier, normally supplied by the first delta.
id: Option<String>,
+ /// Tool name, normally supplied by the first delta.
name: Option<String>,
+ /// Fragment of the serialized tool arguments.
arguments_delta: Option<String>,
},
+ /// Reports token usage, normally near the end of the stream.
Usage(Usage),
+ /// Ends a response message.
MessageStop {
+ /// Provider stop-reason string before normalization.
reason: Option<String>,
},
+ /// Reports that an inbound stream event could not be decoded.
DecodeError {
+ /// Human-readable decoding failure.
message: String,
},
+ /// Reports an upstream failure delivered inside an otherwise successful stream.
StreamError {
+ /// Human-readable upstream failure.
message: String,
},
}
@@ -183,8 +213,14 @@ pub enum LlmResponseChunk {
///
/// Text and reasoning deltas concatenate; tool-call deltas assemble by index (name,
/// id, and a growing arguments string parsed as JSON at the end); `MessageStart`,
-/// `Usage`, and `MessageStop` set the corresponding fields. `Error` chunks are
-/// ignored here — a driver consuming the stream is expected to surface them.
+/// `Usage`, and `MessageStop` set the corresponding fields.
+/// [`DecodeError`](LlmResponseChunk::DecodeError) and
+/// [`StreamError`](LlmResponseChunk::StreamError) chunks are ignored here; use
+/// [`LlmResponse::into_agg`] when they must become errors.
+///
+/// Folding is lossy for multiple outputs: text and reasoning indices are ignored,
+/// and [`finish`](Self::finish) produces one assistant output. Prefer
+/// [`LlmResponse::into_agg`] when stream errors must be surfaced.
///
/// Drive it by `push`-ing each chunk in order, then call [`finish`](Self::finish).
#[derive(Default)]
diff --git a/docs/core_concepts.md b/docs/core_concepts.md
index d39fa2b57..ef2dae090 100644
--- a/docs/core_concepts.md
+++ b/docs/core_concepts.md
@@ -109,3 +109,4 @@ selected target uses a different upstream protocol.
- [Architecture](architecture.md): follow a request through the server and Rust crates.
- [`switchyard-server`](../crates/switchyard-server/README.md): complete TOML schema, endpoints, and metrics.
- [`switchyard-libsy`](../crates/libsy/README.md): embed and extend routing algorithms in Rust.
+- [Rust API reference](reference/rust_api.md): generated libsy and protocol documentation.
diff --git a/docs/getting_started.md b/docs/getting_started.md
index 05729c1b2..63ff63e80 100644
--- a/docs/getting_started.md
+++ b/docs/getting_started.md
@@ -237,9 +237,7 @@ export SWITCHYARD_TELEMETRY_OPT_OUT=1
- [Core Concepts](core_concepts.md): LLM clients, targets, and routes
- [`switchyard-server`](../crates/switchyard-server/README.md): server configuration,
routing algorithms, TLS, and metrics
-- [`switchyard-libsy`](../crates/libsy/README.md): embed routing algorithms in a
- Rust application
-- [`switchyard-protocol`](../crates/protocol/README.md): provider-neutral
- request, response, and streaming types
+- [Rust API reference](reference/rust_api.md): generated libsy and protocol
+ documentation, crate setup, and API boundaries
- [`switchyard-translation`](../crates/switchyard-translation/README.md):
request, response, and stream translation
diff --git a/docs/index.md b/docs/index.md
index 2ada13bc0..1dc5056cd 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -21,11 +21,12 @@ standalone `switchyard-server` binary.
- [Routing Algorithms](routing_algorithms/overview.md): choose how requests select a model
- [Architecture](architecture.md): understand the proxy and library components
- [CLI Reference](cli_reference.md): inspect supported launcher commands and options
+- [Rust API](reference/rust_api.md): browse libsy and protocol crate documentation
- [Context-Window Handling](operations/context_window.md): configure eviction and retry behavior
## Reference
- [`switchyard-server`](../crates/switchyard-server/README.md): server configuration, endpoints, and metrics
-- [`switchyard-libsy`](../crates/libsy/README.md): embeddable routing algorithms
-- [`switchyard-protocol`](../crates/protocol/README.md): provider-neutral API types
+- [`switchyard-libsy`](reference/rust_api.md#switchyard-libsy): embeddable routing algorithms
+- [`switchyard-protocol`](reference/rust_api.md#switchyard-protocol): provider-neutral API types
- [`switchyard-translation`](../crates/switchyard-translation/README.md): protocol translation
diff --git a/docs/internal/metrics_reference.md b/docs/internal/metrics_reference.md
index 09d63ade7..671a62286 100644
--- a/docs/internal/metrics_reference.md
+++ b/docs/internal/metrics_reference.md
@@ -11,71 +11,67 @@ a drop-in scrape config and starter alert rules.
| Path | `GET /metrics` (HTTP path is `/metrics`, **not** `/v1/metrics`) |
| Content-Type | `text/plain; version=0.0.4; charset=utf-8` |
| Format | Prometheus text format 0.0.4 |
-| Auth | None. Designed as a public scrape endpoint |
+| Auth | None |
| Default scrape interval | 15s |
-`GET /metrics` is served by the Python route-bundle server started with
-`switchyard serve --routing-profiles PATH`.
+`GET /metrics` is served by `switchyard-server`.
-A JSON variant of the same underlying data lives at `GET /v1/stats`, with
-`GET /v1/routing/stats` as a backwards-compatible alias.
+A JSON summary of the same traffic lives at `GET /v1/stats`.
## Top-line gauges (no labels)
| Metric | Type | Meaning |
|---|---|---|
-| `switchyard_total_requests` | gauge | Total upstream call attempts recorded since process start. **Per-attempt**, not per-client-request. One client request that retries-then-succeeds increments by 2. |
-| `switchyard_total_errors` | gauge | Total upstream call attempts that errored, including those absorbed by retry. |
+| `switchyard_total_requests` | gauge | Successful and failed routed model calls since process start. Classifier and judge calls are excluded; a context-window fallback can add another routed call. |
+| `switchyard_total_errors` | gauge | Failed routed model calls since process start. |
## Per-endpoint counters
The `model` label is the configured endpoint id (`openai/gpt-5.5`,
`azure_openai/gpt-5.5`, etc.).
-The `tier` label is optional: present only when a routing factory
-supplied one. The `random_routing` factory emits `tier="strong"|"weak"`.
+The `tier` label is optional. It is present when an algorithm defines a stable
+tier for the selected model, such as `strong` or `weak` for a two-tier classifier.
| Metric | Type | Meaning |
|---|---|---|
-| `switchyard_requests_total{model}` | counter | Successful upstream call attempts per endpoint. |
-| `switchyard_errors_total{model}` | counter | Failed upstream call attempts per endpoint (any cause). |
+| `switchyard_requests_total{model}` | counter | Successful routed model calls per endpoint. |
+| `switchyard_errors_total{model}` | counter | Failed routed model calls per endpoint. |
| `switchyard_prompt_tokens_total{model}` | counter | Prompt-token billing per endpoint. |
| `switchyard_completion_tokens_total{model}` | counter | Completion-token usage per endpoint. |
| `switchyard_cached_tokens_total{model}` | counter | Cached prompt tokens per endpoint. |
| `switchyard_cache_creation_tokens_total{model}` | counter | Cache-creation tokens per endpoint. |
| `switchyard_reasoning_tokens_total{model}` | counter | Reasoning tokens per endpoint. |
-## Per-endpoint latency summaries
+## Per-endpoint latency histograms
-Each summary emits `{quantile="0.5"}` and `{quantile="0.99"}` rows plus
-`_sum` and `_count`.
+Each histogram emits `_bucket`, `_sum`, and `_count` series. Use
+`histogram_quantile` in PromQL to calculate a percentile.
| Metric | Type | Meaning |
|---|---|---|
-| `switchyard_model_call_latency_ms{model,quantile}` | summary | Upstream LLM call duration. |
-| `switchyard_total_latency_ms{model,quantile}` | summary | End-to-end request latency (request entry → response complete). For streaming responses this is full-turn time, **not** time-to-first-token. |
+| `switchyard_model_call_latency_ms{model}` | histogram | Successful final routed-call latency. |
+| `switchyard_total_latency_ms{model}` | histogram | End-to-end latency for successful routed responses. For streaming responses this is full-turn time, **not** time-to-first-token. |
## Routing overhead (global, no model label)
| Metric | Type | Meaning |
|---|---|---|
-| `switchyard_routing_overhead_ms{quantile}` | summary | `total_latency_ms − backend_call_latency_ms`, across all calls. Bundles format translation, endpoint selection, body mutation, response wrapping, and retry wall time. |
-
-Healthy traffic typically sits at p50 ≈ 0.4 ms, p99 ≈ 0.6 ms.
+| `switchyard_routing_overhead_ms{algorithm}` | histogram | Algorithm run time minus the final routed model call. Includes classifier calls, target resolution, and decision publication; runs with no final routed call are not recorded. |
## Outcome counters for error-rate ratios
The `outcome` label takes exactly three values:
* `success` = HTTP 2xx
-* `retryable_error` = HTTP 429 / 500 / 504
+* `retryable_error` = HTTP 408, 429, any 5xx, or a failure before an HTTP status
* `other_error` = everything else (400, 401, 403, 422, …)
| Metric | Type | Meaning |
|---|---|---|
| `switchyard_client_responses_total{outcome}` | counter | HTTP responses returned to clients on the LLM-serving routes (`/v1/chat/completions`, `/v1/messages`, `/v1/responses`). The denominator for the **router-served** error rate. |
| `switchyard_upstream_attempts_total{outcome,code}` | counter | Individual upstream call attempts. One client request can produce N attempts via retry. The denominator for the **direct-to-endpoint** baseline error rate. The `code` label carries the raw upstream HTTP status for plotting the error-code distribution (see below). |
-| `switchyard_router_retry_recovered_total` | counter | Requests whose first upstream attempt failed but a subsequent attempt succeeded. This is direct evidence the routing logic rescued the request. |
+| `switchyard_router_retry_recovered_total` | counter | Reserved retry-recovery counter. The current server exports it as zero. |
### The `code` label on `switchyard_upstream_attempts_total`
@@ -111,9 +107,6 @@ direct_error_rate =
# Headline metric: positive value means the router is reducing client errors
error_rate_reduction = direct_error_rate − router_error_rate
-# Live Endpoint Routing rescue rate
-rate(switchyard_router_retry_recovered_total[5m])
-
# Traffic share per endpoint
sum by (model) (rate(switchyard_requests_total[5m]))
/ ignoring(model) group_left sum(rate(switchyard_requests_total[5m]))
@@ -145,14 +138,15 @@ into label space.
| `model` | One per configured endpoint, typically 2–6 per deployment. | All per-endpoint metrics. |
| `outcome` | Exactly 3: `success`, `retryable_error`, `other_error`. | Outcome counters |
| `code` | Bounded: the known-code allowlist (`200`, `400`, `401`, `403`, `404`, `408`, `409`, `422`, `429`, `500`, `502`, `503`, `504`), plus `none` and the per-class buckets `1xx`/`2xx`/`3xx`/`4xx`/`5xx`/`other`. About 20 values max. | `switchyard_upstream_attempts_total` |
-| `quantile` | Exactly 2: `0.5`, `0.99`. | All summaries |
-| `tier` | Small enumerated set, optional. | Per-endpoint counters/summaries on routing chains that supply it |
+| `le` | The configured histogram bucket boundaries. | Histogram buckets |
+| `algorithm` | One stable value per configured algorithm. | Routing-overhead histogram |
+| `tier` | Small enumerated set, optional. | Per-endpoint counters and histograms on algorithms that supply it |
## Triage cheatsheet
| Symptom on `/metrics` | Likely cause |
|---|---|
-| `model="<unknown>"` rows appear | The per-endpoint attribution wiring regressed. `ctx.selected_model` is not being set by the backend. |
+| `model="<unknown>"` rows appear | A routed-call observation did not include a selected model. |
| All counters at 0 after warm-up | Server just started with no traffic, or the scraper is hitting the wrong port. |
-| `switchyard_routing_overhead_ms_count` stuck at `0` | The backend is not publishing `ctx.backend_call_latency_ms` (regression of the Python-backend routing-overhead wiring). |
-| `switchyard_client_responses_total{outcome="retryable_error"}` rising | Either the upstream is genuinely flaky, or retries are exhausting (compare with `switchyard_router_retry_recovered_total` rate). |
+| `switchyard_routing_overhead_ms_count` stuck at `0` | No successful algorithm run has recorded a final routed model call. |
+| `switchyard_client_responses_total{outcome="retryable_error"}` rising | Either the upstream is genuinely flaky, or retries are exhausting; compare client responses with retryable upstream attempts. |
diff --git a/docs/known_issues.md b/docs/known_issues.md
index d2b3b361e..8f53744e1 100644
--- a/docs/known_issues.md
+++ b/docs/known_issues.md
@@ -1,5 +1,5 @@
# Known Issues
## 0.1.0
-1. Completed Codex Responses tasks may record `0` token usage in `GET /v1/stats` and `GET /v1/routing/stats`.
+1. Completed Codex Responses tasks may record `0` token usage in `GET /v1/stats`.
2. Tool-bearing Codex requests may fail when Switchyard routes them to an upstream that accepts only a fixed set of tool names or schemas.
diff --git a/docs/reference/rust_api.md b/docs/reference/rust_api.md
new file mode 100644
index 000000000..eed7157d3
--- /dev/null
+++ b/docs/reference/rust_api.md
@@ -0,0 +1,39 @@
+# Rust API
+
+Switchyard publishes generated API documentation for its embeddable routing
+library and provider-neutral protocol types.
+
+## `switchyard-libsy`
+
+Use libsy to construct targets, run built-in routing algorithms, implement a
+custom algorithm, or drive model calls through the host-facing step stream.
+
+<a href="../../rust/switchyard_libsy/">Open the generated libsy API documentation</a>
+
+## `switchyard-protocol`
+
+Use protocol types for normalized conversations, response streams, routed LLM
+clients, decisions, metadata, and wire-format identifiers.
+
+<a href="../../rust/switchyard_protocol/">Open the generated protocol API documentation</a>
+
+## Crate boundary
+
+Applications embedding libsy normally depend on both crates:
+
+```toml
+[dependencies]
+switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
+switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
+```
+
+Import algorithms and orchestration types from `switchyard_libsy`. Import
+requests, responses, metadata, decisions, and LLM client traits from
+`switchyard_protocol`. Keep both dependencies on compatible versions.
+
+## CI previews
+
+The Documentation workflow publishes a preview for same-repository pull
+requests. Open the workflow's **Deploy PR preview** summary to find its preview
+URL, then use the Rust API links above. The **docs-site** build artifact contains
+the same MkDocs and rustdoc output for one day.
diff --git a/docs/routing_algorithms/stage_router_routing.md b/docs/routing_algorithms/stage_router_routing.md
index f21d0ad0d..2c60b192b 100644
--- a/docs/routing_algorithms/stage_router_routing.md
+++ b/docs/routing_algorithms/stage_router_routing.md
@@ -9,9 +9,9 @@ mechanical work. Which tier a turn defaults to depends on the picker you choose
off that default. You configure it with a single knob, `confidence_threshold`,
plus an optional LLM classifier.
-If the selected backend hits a context-window overflow, the router retries once
-against `fallback_target_on_evict`; a second overflow surfaces a
-context-pool-exhausted error (see [Context-Window Handling](../operations/context_window.md)).
+If the selected target exceeds its context window, the router tries the next
+eligible target until one succeeds or all configured targets have been tried. See
+[Context-Window Handling](../operations/context_window.md).
## How it works
@@ -98,21 +98,19 @@ clears the threshold to escalate to capable.
(If you add the optional classifier, sub-threshold turns go to it instead of
staying on the default tier.)
-**Set `0.5` explicitly.** It's the recommended starting point and what the
-example below uses. When you omit the field the config default is `0.5` (for
-both the profile config and the deprecated route bundle) — but setting it
-explicitly keeps the intent clear.
+**Set `0.5` explicitly.** `confidence_threshold` is required by the TOML schema;
+`0.5` is the recommended starting point and what the example below uses.
| `confidence_threshold` | Include `classifier:` block? | Typical use |
|---|---|---|
| `0.0` | no | Cost/latency-sensitive. Every signal-based verdict is accepted; no per-turn LLM call. Critical-error signals still escalate to capable. |
-| `0.5` | no | Recommended starting point (config default). The scorer is corroborative — one full wrong signal scores ~`0.46`, just under `0.5` — so a decisive escalation takes a strong signal plus corroboration, while a critical error overrides regardless. Derived from SWE-Bench Pro Python-75 calibration. |
+| `0.5` | no | Recommended starting point. The scorer is corroborative — one full wrong signal scores ~`0.46`, just under `0.5` — so a decisive escalation takes a strong signal plus corroboration, while a critical error overrides regardless. Derived from SWE-Bench Pro Python-75 calibration. |
| `0.7` - `0.9` | yes | Classifier-assisted. Low-confidence turns go to the LLM classifier before falling back to the default tier. |
-| `1.0` | yes (required) | Classifier-driven. Equivalent to the legacy `coding_agent` profile. |
+| `1.0` | yes (required) | Classifier-driven. Tool signals only apply hard overrides; other turns reach the classifier. |
The signal-vs-classifier split is dataset-dependent. Measure it in
-production via `routing_decisions.stage_router` on `/v1/stats` rather than relying on
-priors from this doc.
+production: `/v1/stats` reports traffic by tier and model, while response headers
+and structured decision logs explain individual selections.
### Calibrating the threshold from run data
@@ -275,8 +273,8 @@ Each response carries two routing headers:
### Decision sources
-The `decision_source` recorded internally for each turn explains why the routing
-went the way it did. It appears in per-tier metrics tagged on the decision:
+The router records an internal `decision_source` for each turn to distinguish the
+paths through its cascade:
| Source | When |
|---|---|
@@ -288,7 +286,7 @@ went the way it did. It appears in per-tier metrics tagged on the decision:
## When *not* to use stage-router
-- **Single-model deployments.** Use a `model` route instead.
+- **Single-model deployments.** Use a `passthrough` route instead.
- **Probabilistic A/B splits.** Use
[Random Routing](random_routing.md) (`type = "random"`).
The stage-router's signals are wasted on a fixed traffic ratio.
diff --git a/examples/prometheus/README.md b/examples/prometheus/README.md
index ceea201f4..fff0448c4 100644
--- a/examples/prometheus/README.md
+++ b/examples/prometheus/README.md
@@ -58,7 +58,7 @@ series:
```
You should see the families documented in `metrics_reference.md` —
-top-line gauges, per-endpoint counters/summaries, and outcome counters.
+top-line gauges, per-endpoint counters and histograms, and outcome counters.
## Alert tuning notes
@@ -68,8 +68,5 @@ or single-poll blips. Review them against your SLOs before enabling
notifications:
* `RouterOverheadHigh` is set at the success-criterion threshold (p99 >
- 1 ms for 5 m). Retry-heavy workloads inflate this — cross-check
- `switchyard_upstream_attempts_total{outcome="retryable_error"}` rate
- before treating it as a routing-decision regression.
-* `RouterNotRescuing` requires upstream failures to be occurring before
- it fires, so it does not flap on a healthy chain.
+ 1 ms for 5 m). Check its `algorithm` label and `/v1/stats` classifier
+ traffic before treating it as a routing-decision regression.
diff --git a/examples/prometheus/switchyard.rules.yaml b/examples/prometheus/switchyard.rules.yaml
index 3b3400054..22881f02d 100644
--- a/examples/prometheus/switchyard.rules.yaml
+++ b/examples/prometheus/switchyard.rules.yaml
@@ -11,29 +11,12 @@
groups:
- name: switchyard.routing.evidence
rules:
- - alert: RouterNotRescuing
- # Upstream is throwing retryable errors but the router isn't
- # absorbing them via retry. This fires only when there IS
- # something to rescue — does not flap on a healthy chain.
- expr: |
- rate(switchyard_router_retry_recovered_total[5m]) == 0
- and
- rate(switchyard_upstream_attempts_total{outcome="retryable_error"}[5m]) > 0
- for: 5m
- labels:
- severity: warning
- component: switchyard
- annotations:
- summary: "Switchyard observed upstream errors but no retry recoveries"
- description: |
- Upstream attempts are returning retryable errors (429 / 500
- / 504) but no request has been rescued by retry in the same
- 5 m window. Likely cause: every configured endpoint is
- degraded simultaneously, so the retry loop has nowhere to
- steer.
-
- alert: RouterOverheadHigh
- expr: switchyard_routing_overhead_ms{quantile="0.99"} > 1.0
+ expr: |
+ histogram_quantile(
+ 0.99,
+ sum by (le) (rate(switchyard_routing_overhead_ms_bucket[5m]))
+ ) > 1.0
for: 5m
labels:
severity: warning
@@ -42,10 +25,9 @@ groups:
summary: "Switchyard router overhead p99 above 1 ms"
description: |
switchyard_routing_overhead_ms p99 has exceeded 1 ms for 5
- minutes. Routing overhead bundles format translation,
- endpoint selection, body mutation, and retry wall time —
- retries are the most common cause of inflation. Cross-check
- rate(switchyard_upstream_attempts_total{outcome="retryable_error"}[5m]).
+ minutes. Routing overhead includes classifier calls, target
+ resolution, and decision publication. Cross-check the algorithm
+ label and classifier traffic in /v1/stats.
- alert: RouterErrorRateNotReducing
# Headline success criterion. The router should serve a lower
@@ -62,6 +44,8 @@ groups:
sum(rate(switchyard_upstream_attempts_total{outcome="retryable_error"}[5m]))
/ clamp_min(sum(rate(switchyard_upstream_attempts_total[5m])), 1)
)
+ and
+ sum(rate(switchyard_upstream_attempts_total{outcome="retryable_error"}[5m])) > 0
for: 10m
labels:
severity: warning
diff --git a/mkdocs.yml b/mkdocs.yml
index 0a624c5a0..d081dab58 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -34,6 +34,7 @@ nav:
- Reference:
- CLI Reference: cli_reference.md
- TOML Schema: reference/toml_schema.md
+ - Rust API: reference/rust_api.md
exclude_docs: |
internal/**