Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs(release): finalize 0.2.0 changelog (#339)

nachiketb-nvidiad7911fe6a1d8
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ee3e55528..444c44956 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,30 +4,67 @@ All notable changes to Switchyard are documented here. The format is based on
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
-## [Unreleased]
+## [0.2.0]
-### Removed
+Switchyard 0.2.0 introduces the native Rust server and libsy library path,
+with explicit TOML deployments, provider-neutral routing algorithms, and
+production-facing observability.
-- **Latency-aware router** — the `latency_service` route type and its
- `LatencyServiceLLMBackend`, `LatencyServiceBackendConfig`,
- `LatencyServiceEndpoint`, and `LatencyServiceProfileConfig` public API are
- removed. It depended on NVIDIA Inference Hub's latency endpoint and schema.
- Deployments that need multi-endpoint, load- or latency-aware routing should
- migrate to [Dynamo](https://github.com/ai-dynamo/dynamo) (backend-load /
- KV-cache-aware routing with request failover) or an external load balancer
- such as [Traefik](https://doc.traefik.io/traefik/reference/routing-configuration/http/load-balancing/service/)
- or HAProxy.
-- **Public `type: noop` and `type: passthrough` route types** — removed from
- route bundles. Use `type: model` to register a single explicit model target.
- Catalog auto-discovery via a bare `type: passthrough` route is gone; there is
- no `type: model` equivalent, so list the model ids you want as explicit
- `type: model` routes.
-- **`switchyard configure` and `switchyard verify` CLI commands** — removed when
- the CLI was narrowed to `serve` and `launch`. Switchyard no longer saves
- provider credentials or deployment paths. Name the credential environment
- variable with `api_key_env` in a native TOML deployment, export it, and pass
- the deployment to each `switchyard launch`. Validate a deployment with
- `./target/release/switchyard-server --config <deployment.toml> --dry-run`.
+### Added
+
+- **Standalone Rust server** — `switchyard-server` serves OpenAI Chat
+ Completions, OpenAI Responses, and Anthropic Messages from one explicit TOML
+ deployment. It includes TLS, graceful shutdown, upstream retries, token
+ counting, health and model discovery, and optional durable session routing
+ logs.
+- **Rust library and protocol crates** — `switchyard-libsy` provides composable
+ multi-LLM algorithms, `switchyard-protocol` owns the provider-neutral request
+ and response contracts, `switchyard-translation` handles wire-format
+ conversion, and `switchyard-llm-client` provides translated HTTP model calls.
+- **Native routing algorithms** — weighted and reproducible random routing,
+ capability, escalation, and custom-schema modes for LLM-classifier routing,
+ multi-target policy selection, session affinity, context-window fallback, and
+ signal-driven stage routing with handoff notes, per-target prompts, and an
+ optional classifier fallback.
+- **Python bindings for the native path** — `switchyard.libsy` runs Rust-owned
+ algorithms with Python LLM clients, while `switchyard_rust.server.Server`
+ hosts the Rust server in-process for the coding-agent launchers.
+- **Native observability** — Prometheus metrics, GenAI OpenTelemetry spans,
+ structured request logs, `/v1/stats`, `/v1/stats/reset`, and optional
+ `/v1/routing/session-stats` expose request, routing, latency, token, cache,
+ retry, and error data.
+- **Evaluation and integration support** — native-server benchmark wiring,
+ Terminal-Bench 2.1 dataset support, retry-adjusted task routing statistics,
+ and an experimental LiteLLM stage-router integration.
+
+### Changed
+
+- **Native TOML is the primary deployment format** — LLM clients, targets, and
+ routes are declared explicitly and validated by `switchyard-server`. The
+ launcher path accepts the same TOML schema and includes a packaged OpenRouter
+ deployment for zero-config startup.
+- **Serving is built around libsy algorithms** — the native server and Python
+ native-server binding construct algorithms directly instead of using the
+ legacy profile and components-v2 serving stack. The Python YAML server keeps
+ its existing profile APIs in this release.
+- **Coding-agent launchers host the native Rust server** and use its routes,
+ statistics, translation, and OpenTelemetry paths instead of constructing the
+ legacy Python routing stack.
+- **Cascade routing is now stage routing** — the `cascade` route and Python API
+ names are replaced by `stage_router` and the native `StageRouter` algorithm.
+- **The CLI is focused on serving and launching** — `switchyard serve` remains
+ for Python routing-profile YAML bundles, while `switchyard launch` starts
+ Claude Code, Codex CLI, or OpenClaw against a selected native route.
+- **Python dependency compatibility is broader** — the supported OpenAI SDK
+ floor moves from 2.34 to 2.7 while retaining the `<3.0` upper bound.
+- **The Rust workspace uses Rust 1.96.1 and edition 2024.**
+
+### Deprecated
+
+- **The Python `switchyard serve` path** — the Python server, YAML route
+ bundles, and profile APIs remain available in 0.2.0 for transition purposes
+ but are deprecated. New deployments should use `switchyard-server`, native
+ TOML configuration, and libsy algorithms.
### Fixed
@@ -42,6 +79,73 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
with them. Streamed OpenAI Chat replies report the routed target instead of
the provider's own id, and no longer fall back to `"unknown"` when a provider
omits `model` on delta chunks.
+- **Buffered Responses output is preserved** rather than dropping final answer
+ items when translating a non-streaming response.
+- **Cross-format response fidelity is improved** — Responses tool turns and
+ reasoning items survive translation, raw stream events remain available, and
+ Responses usage details and max-token truncation are represented correctly.
+- **Known request fields are validated before translation**, so malformed
+ OpenAI and Anthropic inputs return client errors instead of being silently
+ coerced or omitted.
+- **Anthropic interoperability is hardened** — Messages endpoints return
+ Anthropic error envelopes, accept the `done` stream terminator, filter
+ incompatible beta headers and OpenAI-only fields, and omit unsigned thinking
+ blocks that Anthropic-compatible upstreams reject.
+- **Prompt-cache usage survives format translation**, including cached and
+ cache-creation token counts from OpenRouter and Anthropic-compatible
+ providers; Anthropic prompt caching is enabled by default for translated
+ calls.
+- **Streaming stops after in-band upstream errors** instead of forwarding
+ trailing events after the error.
+- **Routing state and prompts remain coherent across turns** — target prompts
+ and handoff notes survive same-format calls, classifier history keeps tool
+ calls paired with their results, inactive session state is evicted, and
+ context-overflow history is isolated by session and agent.
+- **Native server model metadata is more reliable** — duplicate upstream model
+ IDs produce a warning, `/v1/models` reports declared capabilities and Codex
+ metadata, and streamed replies no longer fall back to an unknown model ID.
+
+### Removed
+
+- **Legacy Rust compatibility stacks** — the `switchyard-components-v2` and
+ `switchyard-core` crates, the components-v2 profile macros, and the old PyO3
+ profile and core bindings are removed. Native serving uses libsy; the Python
+ profile APIs remain available in 0.2.0.
+- **Legacy routing integrations** — plan-and-execute routing, RouteLLM, and the
+ external OSS-router plugin path are removed. The `gpu` optional dependency
+ extra is also gone with RouteLLM.
+- **Latency-aware router** — the `latency_service` route type and its
+ `LatencyServiceLLMBackend`, `LatencyServiceBackendConfig`,
+ `LatencyServiceEndpoint`, and `LatencyServiceProfileConfig` public API are
+ removed. It depended on NVIDIA Inference Hub's latency endpoint and schema.
+ Deployments that need multi-endpoint, load- or latency-aware routing should
+ move endpoint selection to a dedicated upstream load balancer.
+- **Public `type: noop` and `type: passthrough` YAML routes** — removed from
+ Python routing-profile bundles. Use an explicit `type: model` route for a
+ direct target. Automatic catalog discovery from a bare `type: passthrough`
+ route is also removed; list each model ID as its own `type: model` route.
+- **Legacy Intake sink** — direct Intake request and response processors,
+ launcher flags, and the `intake` optional dependency extra are removed. The
+ native server exports telemetry through OpenTelemetry and OTLP instead.
+- **Legacy CLI setup and diagnostics** — `switchyard configure`, `verify`, and
+ `status`, the interactive setup and model-picker TUI, saved provider settings,
+ and launcher smoke mode are removed when the CLI is narrowed to `serve` and
+ `launch`. Name the credential environment variable with `api_key_env` in a
+ native TOML deployment, export it, and pass the deployment to each
+ `switchyard launch`. Validate a deployment with
+ `switchyard-server --config <deployment.toml> --dry-run`.
+
+### Known Issues
+
+1. Buffered upstream work continues after the client disconnects, so a
+ cancelled request can still incur provider cost.
+2. Routing-tier attribution is missing from `GET /v1/stats` and `/metrics` for
+ LLM-classifier judge failures that route to the default target, escalation
+ decisions, and `stage_router` fallback decisions.
+3. The retry recovery counter stays at zero after a successful upstream retry.
+4. `x-switchyard-session-id` is not recorded in native session stats.
+5. The native server does not send the documented `X-Switchyard-Version` header
+ upstream.
## [0.1.0] — Initial release
@@ -52,16 +156,16 @@ traffic that sits between client applications and LLM backends.
- **Four-role chain** — `RequestProcessor → LLMBackend → ResponseProcessor →
TranslationEngine`, executed by the Rust-backed core. See
- [Architecture](docs/ARCHITECTURE.md).
+ the [0.1.0 architecture](https://github.com/NVIDIA-NeMo/Switchyard/blob/v0.1.0/docs/architecture.md).
- **Protocol translation** — convert between OpenAI Chat Completions, Anthropic
Messages, and OpenAI Responses wire formats, so each client keeps speaking its
native API regardless of the upstream backend.
- **YAML route bundles** (`switchyard serve --routing-profiles`) — one bundle,
many named routes, each its own chain. Supported route `type`s: `model`,
- `passthrough`, `random_routing`, `stage_router`, `deterministic`
+ `passthrough`, `random_routing`, `cascade`, `deterministic`
(LLM-as-classifier), `latency_service`, and `noop`.
-- **Routing strategies** — weighted random split, signal-driven **stage-router**
- escalation (see [Stage-Router Routing](docs/stage_router_routing.md)),
+- **Routing strategies** — weighted random split, signal-driven **cascade**
+ escalation (see the [0.1.0 cascade documentation](https://github.com/NVIDIA-NeMo/Switchyard/blob/v0.1.0/docs/routing_algorithms/cascade_routing.md)),
LLM-as-classifier strong/weak routing, and latency-aware multi-endpoint
failover.
- **One-command launchers** — `switchyard launch claude`, `launch codex`, and
@@ -73,16 +177,19 @@ traffic that sits between client applications and LLM backends.
- **Observability** — Prometheus `/metrics`, a JSON `/v1/stats`
(`/v1/routing/stats` alias), and per-request cost/token/latency stats. See
[Metrics Reference](docs/METRICS_REFERENCE.md).
-- **Python library** — `ProfileSwitchyard` driven by typed profile configs
- (`PassthroughProfileConfig`, `RandomRoutingProfileConfig`,
- `StageRouterProfileConfig`, …) and typed `ChatRequest` / `ChatResponse`
- containers for in-process use.
+- **Python library** — `SwitchyardRecipes` (`passthrough_recipe`,
+ `random_routing_recipe`, `cascade_recipe`, `deterministic_routing_recipe`,
+ …) and typed `ChatRequest` / `ChatResponse` containers for in-process use.
- **Rust core** (PyO3) — chain execution, the latency-aware router, and the
tool-result signal collector are implemented in Rust and re-exported to
Python.
- **Packaging** — `pip install nemo-switchyard` with optional extras `[server]`,
- `[cli]`, `[tracing]`, `[intake]`, `[affinity-redis]`, `[all]`. See
- [Installation](INSTALLATION.md).
+ `[cli]`, `[gpu]`, `[all]`. See [Installation](INSTALLATION.md).
+
+### Deprecated
+
+- **`--plan-execute` launcher flag** — slated for removal; plan-execute will be
+ configured through a `--routing-profiles` YAML bundle instead.
### Notes
diff --git a/docs/known_issues.md b/docs/known_issues.md
index 6cfb2776a..d1ed129f9 100644
--- a/docs/known_issues.md
+++ b/docs/known_issues.md
@@ -1,15 +1,11 @@
# Known Issues
## 0.2.0
-1. `stage_router` drops a configured tier system prompt when the inbound request and the selected target both use `openai_chat`. The call succeeds and no warning is emitted.
-2. Errors returned from `/v1/messages` use the OpenAI error envelope rather than Anthropic's `{"type": "error", ...}` shape, so Anthropic SDK clients cannot dispatch on `error.type`.
-3. Session state is retained without a capacity limit or eviction, so memory grows with the number of sessions a process has served.
-4. Buffered upstream work continues after the client disconnects, so a cancelled request can still incur provider cost.
-5. Routing-tier attribution is missing from `GET /v1/stats` and `/metrics` for fail-open, escalation, and `stage_router` fallback decisions.
-6. The retry recovery counter stays at zero after a successful upstream retry.
-7. `x-switchyard-session-id` is not recorded in native session stats.
-8. The native server does not send the documented `X-Switchyard-Version` header upstream.
-9. LLM-classifier history trimming can separate a tool result from the tool call it belongs to when `recent_turn_window` is configured.
+1. Buffered upstream work continues after the client disconnects, so a cancelled request can still incur provider cost.
+2. Routing-tier attribution is missing from `GET /v1/stats` and `/metrics` for LLM-classifier judge failures that route to the default target, escalation decisions, and `stage_router` fallback decisions.
+3. The retry recovery counter stays at zero after a successful upstream retry.
+4. `x-switchyard-session-id` is not recorded in native session stats.
+5. The native server does not send the documented `X-Switchyard-Version` header upstream.
## 0.1.0
1. Completed Codex Responses tasks may record `0` token usage in `GET /v1/stats`.