Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs(release): finalize 0.2.0 changelog (#339)
nachiketb-nvidiad7911fe6a1d8
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ee3e55528..444c44956 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,30 +4,67 @@ All notable changes to Switchyard are documented here. The format is based on[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this projectadheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).-## [Unreleased]+## [0.2.0]-### Removed+Switchyard 0.2.0 introduces the native Rust server and libsy library path,+with explicit TOML deployments, provider-neutral routing algorithms, and+production-facing observability.-- **Latency-aware router** — the `latency_service` route type and its- `LatencyServiceLLMBackend`, `LatencyServiceBackendConfig`,- `LatencyServiceEndpoint`, and `LatencyServiceProfileConfig` public API are- removed. It depended on NVIDIA Inference Hub's latency endpoint and schema.- Deployments that need multi-endpoint, load- or latency-aware routing should- migrate to [Dynamo](https://github.com/ai-dynamo/dynamo) (backend-load /- KV-cache-aware routing with request failover) or an external load balancer- such as [Traefik](https://doc.traefik.io/traefik/reference/routing-configuration/http/load-balancing/service/)- or HAProxy.-- **Public `type: noop` and `type: passthrough` route types** — removed from- route bundles. Use `type: model` to register a single explicit model target.- Catalog auto-discovery via a bare `type: passthrough` route is gone; there is- no `type: model` equivalent, so list the model ids you want as explicit- `type: model` routes.-- **`switchyard configure` and `switchyard verify` CLI commands** — removed when- the CLI was narrowed to `serve` and `launch`. Switchyard no longer saves- provider credentials or deployment paths. Name the credential environment- variable with `api_key_env` in a native TOML deployment, export it, and pass- the deployment to each `switchyard launch`. Validate a deployment with- `./target/release/switchyard-server --config <deployment.toml> --dry-run`.+### Added++- **Standalone Rust server** — `switchyard-server` serves OpenAI Chat+ Completions, OpenAI Responses, and Anthropic Messages from one explicit TOML+ deployment. It includes TLS, graceful shutdown, upstream retries, token+ counting, health and model discovery, and optional durable session routing+ logs.+- **Rust library and protocol crates** — `switchyard-libsy` provides composable+ multi-LLM algorithms, `switchyard-protocol` owns the provider-neutral request+ and response contracts, `switchyard-translation` handles wire-format+ conversion, and `switchyard-llm-client` provides translated HTTP model calls.+- **Native routing algorithms** — weighted and reproducible random routing,+ capability, escalation, and custom-schema modes for LLM-classifier routing,+ multi-target policy selection, session affinity, context-window fallback, and+ signal-driven stage routing with handoff notes, per-target prompts, and an+ optional classifier fallback.+- **Python bindings for the native path** — `switchyard.libsy` runs Rust-owned+ algorithms with Python LLM clients, while `switchyard_rust.server.Server`+ hosts the Rust server in-process for the coding-agent launchers.+- **Native observability** — Prometheus metrics, GenAI OpenTelemetry spans,+ structured request logs, `/v1/stats`, `/v1/stats/reset`, and optional+ `/v1/routing/session-stats` expose request, routing, latency, token, cache,+ retry, and error data.+- **Evaluation and integration support** — native-server benchmark wiring,+ Terminal-Bench 2.1 dataset support, retry-adjusted task routing statistics,+ and an experimental LiteLLM stage-router integration.++### Changed++- **Native TOML is the primary deployment format** — LLM clients, targets, and+ routes are declared explicitly and validated by `switchyard-server`. The+ launcher path accepts the same TOML schema and includes a packaged OpenRouter+ deployment for zero-config startup.+- **Serving is built around libsy algorithms** — the native server and Python+ native-server binding construct algorithms directly instead of using the+ legacy profile and components-v2 serving stack. The Python YAML server keeps+ its existing profile APIs in this release.+- **Coding-agent launchers host the native Rust server** and use its routes,+ statistics, translation, and OpenTelemetry paths instead of constructing the+ legacy Python routing stack.+- **Cascade routing is now stage routing** — the `cascade` route and Python API+ names are replaced by `stage_router` and the native `StageRouter` algorithm.+- **The CLI is focused on serving and launching** — `switchyard serve` remains+ for Python routing-profile YAML bundles, while `switchyard launch` starts+ Claude Code, Codex CLI, or OpenClaw against a selected native route.+- **Python dependency compatibility is broader** — the supported OpenAI SDK+ floor moves from 2.34 to 2.7 while retaining the `<3.0` upper bound.+- **The Rust workspace uses Rust 1.96.1 and edition 2024.**++### Deprecated++- **The Python `switchyard serve` path** — the Python server, YAML route+ bundles, and profile APIs remain available in 0.2.0 for transition purposes+ but are deprecated. New deployments should use `switchyard-server`, native+ TOML configuration, and libsy algorithms.### Fixed@@ -42,6 +79,73 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).with them. Streamed OpenAI Chat replies report the routed target instead ofthe provider's own id, and no longer fall back to `"unknown"` when a provideromits `model` on delta chunks.+- **Buffered Responses output is preserved** rather than dropping final answer+ items when translating a non-streaming response.+- **Cross-format response fidelity is improved** — Responses tool turns and+ reasoning items survive translation, raw stream events remain available, and+ Responses usage details and max-token truncation are represented correctly.+- **Known request fields are validated before translation**, so malformed+ OpenAI and Anthropic inputs return client errors instead of being silently+ coerced or omitted.+- **Anthropic interoperability is hardened** — Messages endpoints return+ Anthropic error envelopes, accept the `done` stream terminator, filter+ incompatible beta headers and OpenAI-only fields, and omit unsigned thinking+ blocks that Anthropic-compatible upstreams reject.+- **Prompt-cache usage survives format translation**, including cached and+ cache-creation token counts from OpenRouter and Anthropic-compatible+ providers; Anthropic prompt caching is enabled by default for translated+ calls.+- **Streaming stops after in-band upstream errors** instead of forwarding+ trailing events after the error.+- **Routing state and prompts remain coherent across turns** — target prompts+ and handoff notes survive same-format calls, classifier history keeps tool+ calls paired with their results, inactive session state is evicted, and+ context-overflow history is isolated by session and agent.+- **Native server model metadata is more reliable** — duplicate upstream model+ IDs produce a warning, `/v1/models` reports declared capabilities and Codex+ metadata, and streamed replies no longer fall back to an unknown model ID.++### Removed++- **Legacy Rust compatibility stacks** — the `switchyard-components-v2` and+ `switchyard-core` crates, the components-v2 profile macros, and the old PyO3+ profile and core bindings are removed. Native serving uses libsy; the Python+ profile APIs remain available in 0.2.0.+- **Legacy routing integrations** — plan-and-execute routing, RouteLLM, and the+ external OSS-router plugin path are removed. The `gpu` optional dependency+ extra is also gone with RouteLLM.+- **Latency-aware router** — the `latency_service` route type and its+ `LatencyServiceLLMBackend`, `LatencyServiceBackendConfig`,+ `LatencyServiceEndpoint`, and `LatencyServiceProfileConfig` public API are+ removed. It depended on NVIDIA Inference Hub's latency endpoint and schema.+ Deployments that need multi-endpoint, load- or latency-aware routing should+ move endpoint selection to a dedicated upstream load balancer.+- **Public `type: noop` and `type: passthrough` YAML routes** — removed from+ Python routing-profile bundles. Use an explicit `type: model` route for a+ direct target. Automatic catalog discovery from a bare `type: passthrough`+ route is also removed; list each model ID as its own `type: model` route.+- **Legacy Intake sink** — direct Intake request and response processors,+ launcher flags, and the `intake` optional dependency extra are removed. The+ native server exports telemetry through OpenTelemetry and OTLP instead.+- **Legacy CLI setup and diagnostics** — `switchyard configure`, `verify`, and+ `status`, the interactive setup and model-picker TUI, saved provider settings,+ and launcher smoke mode are removed when the CLI is narrowed to `serve` and+ `launch`. Name the credential environment variable with `api_key_env` in a+ native TOML deployment, export it, and pass the deployment to each+ `switchyard launch`. Validate a deployment with+ `switchyard-server --config <deployment.toml> --dry-run`.++### Known Issues++1. Buffered upstream work continues after the client disconnects, so a+ cancelled request can still incur provider cost.+2. Routing-tier attribution is missing from `GET /v1/stats` and `/metrics` for+ LLM-classifier judge failures that route to the default target, escalation+ decisions, and `stage_router` fallback decisions.+3. The retry recovery counter stays at zero after a successful upstream retry.+4. `x-switchyard-session-id` is not recorded in native session stats.+5. The native server does not send the documented `X-Switchyard-Version` header+ upstream.## [0.1.0] — Initial release@@ -52,16 +156,16 @@ traffic that sits between client applications and LLM backends.- **Four-role chain** — `RequestProcessor → LLMBackend → ResponseProcessor →TranslationEngine`, executed by the Rust-backed core. See- [Architecture](docs/ARCHITECTURE.md).+ the [0.1.0 architecture](https://github.com/NVIDIA-NeMo/Switchyard/blob/v0.1.0/docs/architecture.md).- **Protocol translation** — convert between OpenAI Chat Completions, AnthropicMessages, and OpenAI Responses wire formats, so each client keeps speaking itsnative API regardless of the upstream backend.- **YAML route bundles** (`switchyard serve --routing-profiles`) — one bundle,many named routes, each its own chain. Supported route `type`s: `model`,- `passthrough`, `random_routing`, `stage_router`, `deterministic`+ `passthrough`, `random_routing`, `cascade`, `deterministic`(LLM-as-classifier), `latency_service`, and `noop`.-- **Routing strategies** — weighted random split, signal-driven **stage-router**- escalation (see [Stage-Router Routing](docs/stage_router_routing.md)),+- **Routing strategies** — weighted random split, signal-driven **cascade**+ escalation (see the [0.1.0 cascade documentation](https://github.com/NVIDIA-NeMo/Switchyard/blob/v0.1.0/docs/routing_algorithms/cascade_routing.md)),LLM-as-classifier strong/weak routing, and latency-aware multi-endpointfailover.- **One-command launchers** — `switchyard launch claude`, `launch codex`, and@@ -73,16 +177,19 @@ traffic that sits between client applications and LLM backends.- **Observability** — Prometheus `/metrics`, a JSON `/v1/stats`(`/v1/routing/stats` alias), and per-request cost/token/latency stats. See[Metrics Reference](docs/METRICS_REFERENCE.md).-- **Python library** — `ProfileSwitchyard` driven by typed profile configs- (`PassthroughProfileConfig`, `RandomRoutingProfileConfig`,- `StageRouterProfileConfig`, …) and typed `ChatRequest` / `ChatResponse`- containers for in-process use.+- **Python library** — `SwitchyardRecipes` (`passthrough_recipe`,+ `random_routing_recipe`, `cascade_recipe`, `deterministic_routing_recipe`,+ …) and typed `ChatRequest` / `ChatResponse` containers for in-process use.- **Rust core** (PyO3) — chain execution, the latency-aware router, and thetool-result signal collector are implemented in Rust and re-exported toPython.- **Packaging** — `pip install nemo-switchyard` with optional extras `[server]`,- `[cli]`, `[tracing]`, `[intake]`, `[affinity-redis]`, `[all]`. See- [Installation](INSTALLATION.md).+ `[cli]`, `[gpu]`, `[all]`. See [Installation](INSTALLATION.md).++### Deprecated++- **`--plan-execute` launcher flag** — slated for removal; plan-execute will be+ configured through a `--routing-profiles` YAML bundle instead.### Notesdiff --git a/docs/known_issues.md b/docs/known_issues.mdindex 6cfb2776a..d1ed129f9 100644--- a/docs/known_issues.md+++ b/docs/known_issues.md@@ -1,15 +1,11 @@# Known Issues## 0.2.0-1. `stage_router` drops a configured tier system prompt when the inbound request and the selected target both use `openai_chat`. The call succeeds and no warning is emitted.-2. Errors returned from `/v1/messages` use the OpenAI error envelope rather than Anthropic's `{"type": "error", ...}` shape, so Anthropic SDK clients cannot dispatch on `error.type`.-3. Session state is retained without a capacity limit or eviction, so memory grows with the number of sessions a process has served.-4. Buffered upstream work continues after the client disconnects, so a cancelled request can still incur provider cost.-5. Routing-tier attribution is missing from `GET /v1/stats` and `/metrics` for fail-open, escalation, and `stage_router` fallback decisions.-6. The retry recovery counter stays at zero after a successful upstream retry.-7. `x-switchyard-session-id` is not recorded in native session stats.-8. The native server does not send the documented `X-Switchyard-Version` header upstream.-9. LLM-classifier history trimming can separate a tool result from the tool call it belongs to when `recent_turn_window` is configured.+1. Buffered upstream work continues after the client disconnects, so a cancelled request can still incur provider cost.+2. Routing-tier attribution is missing from `GET /v1/stats` and `/metrics` for LLM-classifier judge failures that route to the default target, escalation decisions, and `stage_router` fallback decisions.+3. The retry recovery counter stays at zero after a successful upstream retry.+4. `x-switchyard-session-id` is not recorded in native session stats.+5. The native server does not send the documented `X-Switchyard-Version` header upstream.## 0.1.01. Completed Codex Responses tasks may record `0` token usage in `GET /v1/stats`.