Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs: fix client setup and Responses state guidance (#809)

ryan-lempka26c7e8908c91
diff --git a/README.md b/README.md
index a30261864..767f98ad0 100644
--- a/README.md
+++ b/README.md
@@ -241,12 +241,25 @@ setup, Switchyard uses the server's `OPENROUTER_API_KEY` for upstream requests.
Do not use the placeholder with `forward_auth = true` or a gateway that requires
a real client credential.
-Codex CLI and other OpenAI clients use the OpenAI variables instead:
+For Codex CLI, add this provider to `~/.codex/config.toml`:
+
+```toml
+[model_providers.switchyard]
+name = "Switchyard"
+base_url = "http://localhost:4000/v1"
+wire_api = "responses"
+requires_openai_auth = false
+```
+
+Then select the provider and route:
```bash
-export OPENAI_BASE_URL="http://localhost:4000/v1"
+codex --model switchyard -c 'model_provider="switchyard"'
```
+No Codex API key is needed for this local setup. Switchyard uses the server's
+`OPENROUTER_API_KEY` for upstream requests.
+
## Routing Algorithms
Start with **Auto**. Choose Task or Execution when you want more control over
diff --git a/crates/libsy-llm-client/README.md b/crates/libsy-llm-client/README.md
index bb3b38caf..34ec8a989 100644
--- a/crates/libsy-llm-client/README.md
+++ b/crates/libsy-llm-client/README.md
@@ -47,8 +47,10 @@ Within this workspace:
```toml
[dependencies]
+futures-util = "0.3"
+switchyard-libsy = { path = "../libsy" }
switchyard-llm-client = { path = "../libsy-llm-client" }
-switchyard-protocol = { path = "../libsy-protocol" }
+switchyard-protocol = { path = "../protocol" }
switchyard-translation = { path = "../switchyard-translation" } # for WireFormat
```
@@ -153,7 +155,7 @@ ordered fallbacks for the client to try. `ClientRouter::single` is the single-pr
```rust
use std::sync::Arc;
-use switchyard_libsy::Algorithm;
+use switchyard_libsy::{Algorithm, RuntimeModels};
use switchyard_llm_client::{ClientRouter, TranslatingLlmClient};
use switchyard_protocol::Request;
@@ -161,10 +163,11 @@ async fn route(
algorithm: Arc<dyn Algorithm>,
client: Arc<TranslatingLlmClient>,
request: Request,
+ models: Arc<RuntimeModels>,
) -> switchyard_libsy::Result<String> {
let clients = ClientRouter::single(client);
let (selected_model, _response) =
- switchyard_llm_client::run(algorithm, clients, request, None).await?;
+ switchyard_llm_client::run(algorithm, clients, request, models, None).await?;
Ok(selected_model.to_string())
}
```
@@ -285,8 +288,8 @@ after a stream has started does not count another attempt.
| `InvalidResponse { source }` | the upstream response could not be decoded |
| `Other(source)` | a client-specific failure outside the shared categories |
-[`switchyard_protocol::Request`]: ../libsy-protocol
-[`switchyard_protocol::Response`]: ../libsy-protocol
+[`switchyard_protocol::Request`]: ../protocol
+[`switchyard_protocol::Response`]: ../protocol
[`libsy-proxy`]: ../libsy-proxy
[`Backend`]: src/backend.rs
[`HttpBackendConfig`]: src/backend.rs
diff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.md
index c1b1af8af..d57d70dde 100644
--- a/docs/routing_algorithms/llm_classifier_routing.md
+++ b/docs/routing_algorithms/llm_classifier_routing.md
@@ -270,48 +270,45 @@ enable `message_hash_fallback` to key on the first user-message text under
### Responses continuations by ID
-A Responses API client can continue without resending the conversation history:
-`previous_response_id` refers to a stored response, and `conversation` refers to
-a provider's conversation. When answer targets use different `[llm_clients]`
-entries, Switchyard records the model that served each stored Responses ID and
-conversation ID. A request with a recorded ID returns to that exact model
-without a judge call or fallback to another provider. This applies to every
-`classify_trigger` and to `stage_router` and `composite` routes. Classifier-only
-clients do not count; when answer targets share one client, routing proceeds as
-usual.
-
-Switchyard tracks buffered and streamed Responses replies, including answers
-returned by `/v1/decision`. It does not record Chat Completions or Anthropic
-Messages response IDs as Responses continuation IDs. A provider's `store` value
-takes precedence over the request value. If that value is `false`, Switchyard
-skips the response ID but still records the conversation ID.
+A Responses API client can continue without resending the conversation history.
+Switchyard handles state differently depending on the target API:
+
+- **Native Responses targets:** The provider stores the conversation. When answer
+ targets use different `[llm_clients]` entries, Switchyard records response and
+ conversation IDs with the model that served them, but no transcript. Classifier-only
+ clients do not count. When answer targets share one client, no local record is needed.
+ The provider's `store` value takes precedence over the request value. If it is
+ `false`, Switchyard skips the response ID but still records the conversation ID.
+- **Chat Completions or Anthropic Messages targets:** For incoming Responses
+ requests, Switchyard retains request and reply messages in memory under the
+ response ID. It restores that history on a later `previous_response_id` request.
+ This also applies when answer targets share one client. Request `store: false`
+ prevents retaining the new response and history, but does not delete earlier
+ records. This path supports `previous_response_id`, not provider conversation IDs.
+
+A request with a recorded ID returns to the model that served it without a judge
+call or fallback to another provider. This applies to every `classify_trigger`
+and to `stage_router` and `composite` routes. Tracking covers buffered replies and
+completed streams, including answers returned by `/v1/decision`. Ordinary Chat
+Completions and Anthropic Messages requests do not create Responses history.
Each route keeps up to 65,536 distinct ID-to-model records per process. This
limit counts response and conversation IDs recorded over the process's lifetime,
not tokens or simultaneous requests. Records do not expire, and Switchyard does
-not remove older records to make room. The map stores copies of IDs and model
-names, with no transcript storage.
-
-If recording new IDs would exceed the limit, a buffered reply returns HTTP 503
-with code `response_state_limit_exceeded`. If an ID is already recorded for a
-different model, it returns HTTP 409 with code `response_state_conflict`. Neither
-failure changes existing records. Switchyard checks both IDs before adding
-either from a response or stream event.
-
-These checks run after the provider has returned a response or stream event, so
-the error does not mean the provider did no work. A stream that has already sent
-HTTP headers emits a framed error and stops; it cannot change the HTTP status.
-Switchyard logs these errors. For `/v1/responses`, buffered and streamed failures
-increment `/v1/stats` `total_errors`. Other routes continue independently.
-
-An existing ID can still be used at capacity if the reply adds no new IDs. For
-example, `store: false` can suppress a new response ID, but it does not suppress
-conversation-ID tracking. A stored follow-up normally creates a new response ID
-and therefore fails when the map is full.
-
-A restart removes all records, and each replica has its own map. Unknown IDs use
-normal routing and can still fail at the selected provider. To keep dynamic
-routing across turns, send the full history instead of an ID.
+not remove older records to make room. Cross-format records also retain message
+history, so this ID limit is not a memory limit. Memory use depends on the retained
+messages as well as the number of IDs.
+
+At capacity, Switchyard logs a warning and returns the reply without retaining
+new IDs or history. Existing records remain usable, but a later continuation from
+an unrecorded ID may fail. Conflicting native Responses IDs return HTTP 409 with
+code `response_state_conflict`. If streaming headers have already been sent, the
+stream emits an error instead of changing the HTTP status.
+
+A restart removes all local records and history, and each replica has its own
+state. Unknown IDs use normal routing and can still fail at the selected provider.
+Send the full history instead of an ID to avoid relying on local continuation
+state and to keep dynamic routing across turns.
## Run the route