Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

fix(llm-client): report Responses state limits without losing saved IDs (#721)

Elyas Mehtabuddin2d485f20007a
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ca2b3bf06..0e987ef86 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -121,6 +121,18 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
### Fixed
+- **Responses continuations stay with the provider that holds their state** — a
+ request with `previous_response_id` or `conversation` could reach another
+ provider and fail with `400 state_not_found`. When answer targets use different
+ `[llm_clients]` entries, Switchyard records which model served each stored
+ Responses ID and conversation ID. Known continuations return to that model
+ without running the algorithm or trying another provider. Each route keeps up
+ to 65,536 IDs per process without discarding older records. Recording another
+ model for an existing ID returns `response_state_conflict`; exceeding the cap
+ returns `response_state_limit_exceeded`. Buffered replies use HTTP 409 and 503,
+ respectively; streams emit an error and stop. These checks happen after the
+ provider has done work. Records are lost on restart. A separate classifier
+ client does not affect routes whose answer targets share one client.
- **Cross-format tool results keep image and file content** — an Anthropic
`tool_result` carrying image or document blocks reached a Responses target
as text only, and an image-only result became an empty `output`. In the other
diff --git a/crates/libsy-llm-client/src/run.rs b/crates/libsy-llm-client/src/run.rs
index 380e9f602..c16922c44 100644
--- a/crates/libsy-llm-client/src/run.rs
+++ b/crates/libsy-llm-client/src/run.rs
@@ -12,18 +12,26 @@
//! ordered candidate fallback, and the `libsy.client_call` span around each candidate. Each
//! candidate exhausts its backend retry budget before fallback advances, so the worst case is
//! `candidates × (max_retries + 1)` upstream attempts plus every candidate's backoff.
+//!
+//! A Responses continuation can refer to state held by one provider through
+//! `previous_response_id` or `conversation`. When completion targets use different clients,
+//! [`ClientRouter`] records which model served that state and sends known continuations
+//! back to it without running the algorithm or trying another provider.
use std::collections::HashMap;
use std::sync::Arc;
use std::time::Instant;
+use futures_util::StreamExt;
use http::StatusCode;
use parking_lot::Mutex;
+use serde_json::{Value, json};
use switchyard_libsy::{
- Algorithm, CallModel, LibsyError, Result, RoutingOutcome, RuntimeModels, drive,
+ Algorithm, CallModel, LibsyError, OutcomeMetadata, Result, RoutingOutcome, RuntimeModels, drive,
};
use switchyard_protocol::{
- LlmClientError, ModelId, Request, Response, RoutedLlmClient, RoutingFallbackReason,
+ LlmClientError, LlmResponse, ModelId, Request, Response, RoutedLlmClient,
+ RoutingFallbackReason, WireFormat,
};
use switchyard_translation::prepare_request_for_target;
@@ -55,11 +63,16 @@ pub async fn run(
// This says if we have an observer, put Some(..) in routing_observations.
// No observer means we don't want any routing_observations.
let routing_observations = observer.as_ref().map(|_| Arc::new(Mutex::new(Vec::new())));
- let outcome = drive(algorithm, request, models, {
- let routing_observations = routing_observations.clone();
- move |call| serve(routing_clients.clone(), call, routing_observations.clone())
- })
- .await;
+ let outcome = match clients.stored_state_owner(&request) {
+ Some(owner) => Ok(continue_on(owner, &algorithm_name, request)),
+ None => {
+ drive(algorithm, request, models, {
+ let routing_observations = routing_observations.clone();
+ move |call| serve(routing_clients.clone(), call, routing_observations.clone())
+ })
+ .await
+ }
+ };
let answered_model = outcome
.as_ref()
.ok()
@@ -98,6 +111,8 @@ pub async fn run(
);
(result, Some(answer_duration))
};
+ let result =
+ result.and_then(|response| clients.remember_state_owner(&outcome.request, response));
metrics::record_routed_request(&selected_model_id, answer_duration, &result);
if let Some(observer) = &observer {
observer(RunObservation::RoutingOverhead(overhead));
@@ -116,12 +131,21 @@ pub async fn decide(
models: Arc<RuntimeModels>,
) -> Result<RoutingOutcome> {
let routing_clients = clients.clone();
- let mut outcome = drive(algorithm, request, models, move |call| {
- serve(routing_clients.clone(), call, None)
- })
- .await?;
+ let mut outcome = match clients.stored_state_owner(&request) {
+ Some(owner) => continue_on(owner, algorithm.name(), request),
+ None => {
+ drive(algorithm, request, models, move |call| {
+ serve(routing_clients.clone(), call, None)
+ })
+ .await?
+ }
+ };
let selected_model_id = outcome.selected_model_id()?.clone();
outcome.request = clients.prepare_completion_request(outcome.request, &selected_model_id);
+ outcome.response = outcome
+ .response
+ .map(|response| clients.remember_state_owner(&outcome.request, response))
+ .transpose()?;
Ok(outcome)
}
@@ -328,6 +352,83 @@ fn fallback_reason(error: &LibsyError) -> Option<RoutingFallbackReason> {
}
}
+/// Route to `owner` without fallbacks and record why the algorithm did not run.
+fn continue_on(owner: StateOwner, algorithm: &str, request: Request) -> RoutingOutcome {
+ tracing::debug!(
+ target = %owner.model,
+ trigger = owner.trigger,
+ "routing Responses continuation to the model that holds its state"
+ );
+ let mut outcome = RoutingOutcome::route_to(owner.model, Vec::new(), request);
+ outcome.metadata = Some(OutcomeMetadata::new(
+ algorithm.to_string(),
+ Some(json!({"source": "stored_state", "trigger": owner.trigger})),
+ ));
+ outcome
+}
+
+/// The model that served the stored state and the request field that refers to it.
+struct StateOwner {
+ model: ModelId,
+ trigger: &'static str,
+}
+
+/// Stored Responses IDs and conversation IDs, mapped to the model that served them.
+///
+/// Keep earlier response IDs because a client can continue from any stored response.
+/// Reject new IDs at capacity rather than forgetting where an earlier response is stored.
+#[derive(Default)]
+struct StateOwners {
+ by_id: HashMap<String, ModelId>,
+}
+
+const MAX_STATE_OWNERS: usize = 65_536;
+
+impl StateOwners {
+ fn owner(&self, id: &str) -> Option<&ModelId> {
+ self.by_id.get(id)
+ }
+
+ fn remember(
+ &mut self,
+ response_id: Option<&str>,
+ conversation_id: Option<&str>,
+ model: &ModelId,
+ ) -> std::result::Result<(), LlmClientError> {
+ let ids = [
+ response_id,
+ conversation_id.filter(|id| Some(*id) != response_id),
+ ];
+ let mut new_ids = 0;
+ for id in ids.into_iter().flatten() {
+ match self.by_id.get(id) {
+ Some(owner) if owner != model => return Err(LlmClientError::ResponseStateConflict),
+ None => new_ids += 1,
+ _ => {}
+ }
+ }
+ if self.by_id.len() + new_ids > MAX_STATE_OWNERS {
+ return Err(LlmClientError::ResponseStateLimitExceeded {
+ limit: MAX_STATE_OWNERS,
+ });
+ }
+ for id in ids.into_iter().flatten() {
+ if !self.by_id.contains_key(id) {
+ self.by_id.insert(id.to_owned(), model.clone());
+ }
+ }
+ Ok(())
+ }
+}
+
+/// Read the Responses `conversation` ID from a string or an object.
+fn conversation_id(fields: &serde_json::Map<String, Value>) -> Option<&str> {
+ let conversation = fields.get("conversation")?;
+ conversation
+ .as_str()
+ .or_else(|| conversation.get("id").and_then(Value::as_str))
+}
+
/// Resolves a routed call's selected model to the client that serves it.
///
/// An algorithm routes among named targets; which provider each target lives on is the
@@ -345,6 +446,9 @@ struct ClientRouting {
routing: Routing,
target_prompts: HashMap<ModelId, String>,
routing_answer_target: Option<ModelId>,
+ /// Track state only when completion targets use different clients. A separate
+ /// classifier client does not require tracking when all completion targets share a client.
+ state_owners: Option<Mutex<StateOwners>>,
}
enum Routing {
@@ -365,16 +469,48 @@ impl ClientRouter {
/// `routing_answer_target` identifies a routing-time call whose response may become the
/// answer. Pass `None` when routing only selects a later completion target. Prompt keys and
/// `routing_answer_target` use the resolved model IDs stored in `by_model`.
+ /// This constructor treats every model in `by_model` as a possible completion target.
+ /// Use [`Self::new_with_completion_targets`] to exclude classifier-only models from the
+ /// comparison of completion clients.
pub fn new_with_target_prompts(
by_model: HashMap<ModelId, Arc<dyn RoutedLlmClient>>,
target_prompts: HashMap<ModelId, String>,
routing_answer_target: Option<ModelId>,
) -> Self {
+ let completion_targets = by_model.keys().cloned().collect::<Vec<_>>();
+ Self::new_with_completion_targets(
+ by_model,
+ target_prompts,
+ routing_answer_target,
+ &completion_targets,
+ )
+ }
+
+ /// Build a router with an explicit list of models that can answer requests.
+ ///
+ /// `by_model` contains every callable model, including classifiers. Only the model IDs in
+ /// `completion_targets` determine whether completion clients differ and Responses state
+ /// needs tracking. Classifier-only models remain callable without affecting that comparison.
+ /// `target_prompts` and `routing_answer_target` have the same meaning as in
+ /// [`Self::new_with_target_prompts`].
+ pub fn new_with_completion_targets(
+ by_model: HashMap<ModelId, Arc<dyn RoutedLlmClient>>,
+ target_prompts: HashMap<ModelId, String>,
+ routing_answer_target: Option<ModelId>,
+ completion_targets: &[ModelId],
+ ) -> Self {
+ let mut clients = completion_targets
+ .iter()
+ .filter_map(|model| by_model.get(model));
+ let spans_providers = clients
+ .next()
+ .is_some_and(|first| clients.any(|client| !Arc::ptr_eq(first, client)));
Self {
inner: Arc::new(ClientRouting {
routing: Routing::ByModel(by_model),
target_prompts,
routing_answer_target,
+ state_owners: spans_providers.then(Mutex::default),
}),
}
}
@@ -390,6 +526,7 @@ impl ClientRouter {
routing: Routing::Single(client),
target_prompts: HashMap::new(),
routing_answer_target: None,
+ state_owners: None,
}),
}
}
@@ -414,6 +551,94 @@ impl ClientRouter {
}
}
+ /// Return the recorded model for the requested response or conversation ID.
+ fn stored_state_owner(&self, request: &Request) -> Option<StateOwner> {
+ let owners = self.inner.state_owners.as_ref()?;
+ let fields = &request.llm_request.extensions.fields;
+ let (trigger, id) = match fields.get("previous_response_id").and_then(Value::as_str) {
+ Some(id) => ("previous_response_id", id),
+ None => ("conversation", conversation_id(fields)?),
+ };
+ let model = owners.lock().owner(id)?.clone();
+ Some(StateOwner { model, trigger })
+ }
+
+ /// Record state reported by a Responses provider. Other response formats have IDs
+ /// that cannot be continued through the Responses API.
+ fn remember_state_owner(&self, request: &Request, mut response: Response) -> Result<Response> {
+ if self.inner.state_owners.is_none() {
+ return Ok(response);
+ }
+ let Some(model) = response.served_model().cloned() else {
+ return Ok(response);
+ };
+ let fields = &request.llm_request.extensions.fields;
+ let store = fields.get("store").and_then(Value::as_bool) != Some(false);
+ let conversation = conversation_id(fields).map(str::to_owned);
+ response.llm_response = match response.llm_response {
+ LlmResponse::Agg(agg) => {
+ if let Some(body) = agg
+ .preservation
+ .responses
+ .get(&WireFormat::OpenAiResponses.into())
+ {
+ self.remember_response(body, &model, store, conversation.as_deref())
+ .map_err(|error| LibsyError::client_call(model.clone(), error))?;
+ }
+ LlmResponse::Agg(agg)
+ }
+ LlmResponse::Stream(stream) => {
+ let router = self.clone();
+ LlmResponse::Stream(Box::pin(futures_util::stream::try_unfold(
+ (stream, router, model, conversation),
+ move |(mut stream, router, model, conversation)| async move {
+ let Some(event) = stream.next().await else {
+ return Ok(None);
+ };
+ let event = event?;
+ if let Some(preserved) = event.preservation()
+ && preserved.source().as_str() == WireFormat::OpenAiResponses.as_str()
+ && let Some(body) = preserved.raw().get("response")
+ {
+ router.remember_response(
+ body,
+ &model,
+ store,
+ conversation.as_deref(),
+ )?;
+ }
+ Ok(Some((event, (stream, router, model, conversation))))
+ },
+ )))
+ }
+ };
+ Ok(response)
+ }
+
+ /// Check both IDs from a response or stream event before recording either. The response's
+ /// `store` value overrides the request value; `false` skips only the response ID.
+ fn remember_response(
+ &self,
+ body: &Value,
+ model: &ModelId,
+ store: bool,
+ conversation: Option<&str>,
+ ) -> std::result::Result<(), LlmClientError> {
+ if let Some(owners) = &self.inner.state_owners {
+ let mut owners = owners.lock();
+ let response_id = body
+ .get("id")
+ .and_then(Value::as_str)
+ .filter(|_| body.get("store").and_then(Value::as_bool).unwrap_or(store));
+ let conversation = body.as_object().and_then(conversation_id).or(conversation);
+ if let Err(error) = owners.remember(response_id, conversation, model) {
+ tracing::error!(error = %error, "could not record Responses state owner");
+ return Err(error);
+ }
+ }
+ Ok(())
+ }
+
/// Prepare a completion candidate with its configured target prompt.
fn prepare_completion_request(&self, mut request: Request, target: &ModelId) -> Request {
let prompt = self.inner.target_prompts.get(target).map(String::as_str);
@@ -693,6 +918,204 @@ mod tests {
));
}
+ #[tokio::test]
+ async fn responses_state_tracking_uses_upstream_storage_and_format() -> Result<()> {
+ for (responses, store, conversation) in [
+ (true, true, None),
+ (true, false, None),
+ (true, false, Some(json!("conv_plain"))),
+ (true, false, Some(json!({"id": "conv_object"}))),
+ (false, true, None),
+ ] {
+ for stream in [false, true] {
+ let server = MockServer::start().await;
+ Mock::given(method("POST"))
+ .respond_with(move |request: &wiremock::Request| {
+ let body: Value = serde_json::from_slice(&request.body).expect("request JSON");
+ let response = if responses {
+ json!({
+ "id": "resp_seed", "object": "response", "model": "weak",
+ "status": "completed", "output": [], "store": body["store"],
+ "conversation": body["conversation"],
+ })
+ } else if stream {
+ json!({"id": "resp_seed", "model": "weak", "choices": [{"index": 0, "delta": {"content": "ok"}, "finish_reason": "stop"}]})
+ } else {
+ json!({"id": "resp_seed", "model": "weak", "choices": [{"index": 0, "message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}]})
+ };
+ if stream {
+ let event = if responses { json!({"type": "response.completed", "response": response}) } else { response };
+ ResponseTemplate::new(200)
+ .insert_header("content-type", "text/event-stream")
+ .set_body_string(if responses {
+ format!("event: response.completed\ndata: {event}\n\n")
+ } else {
+ format!("data: {event}\n\ndata: [DONE]\n\n")
+ })
+ } else {
+ ResponseTemplate::new(200).set_body_json(response)
+ }
+ })
+ .mount(&server).await;
+ let mut clients = HashMap::new();
+ for model in ["weak", "strong"] {
+ let config = HttpBackendConfig {
+ base_url: server.uri(),
+ api_key: None,
+ forward_auth: false,
+ extra_headers: BTreeMap::new(),
+ extra_body: BTreeMap::from([("store".to_string(), json!(store))]),
+ reasoning_effort: None,
+ max_retries: 0,
+ };
+ let backend = if responses {
+ Backend::OpenAiResponses(config)
+ } else {
+ Backend::OpenAiChat(config)
+ };
+ let client: Arc<dyn RoutedLlmClient> = Arc::new(
+ TranslatingLlmClient::new(&[ModelConfig::new(model, backend, None)])
+ .map_err(|error| LibsyError::external("building test client", error))?,
+ );
+ clients.insert(ModelId::from(model), client);
+ }
+ let clients = ClientRouter::new(clients);
+ let mut seed = request();
+ seed.llm_request.stream = stream;
+ if let Some(conversation) = &conversation {
+ seed.llm_request
+ .extensions
+ .fields
+ .insert("conversation".to_string(), conversation.clone());
+ }
+ let (_, response) = run(
+ Arc::new(switchyard_libsy::Passthrough),
+ clients.clone(),
+ seed,
+ to_category_map(&["weak"]),
+ None,
+ )
+ .await?;
+ response
+ .llm_response
+ .into_agg()
+ .await
+ .map_err(|error| LibsyError::client_call("weak", error))?;
+ let mut continuations = vec![(
+ "previous_response_id",
+ json!("resp_seed"),
+ responses && store,
+ )];
+ if let Some(conversation) = conversation.clone() {
+ continuations.push(("conversation", conversation, true));
+ }
+ for (field, id, pinned) in continuations {
+ let mut follow = request();
+ follow
+ .llm_request
+ .extensions
+ .fields
+ .insert(field.to_string(), id);
+ let outcome = decide(
+ Arc::new(switchyard_libsy::Passthrough),
+ clients.clone(),
+ follow,
+ to_category_map(&["strong"]),
+ )
+ .await?;
+ assert_eq!(
+ outcome.selected_model_id()?.as_str(),
+ if pinned { "weak" } else { "strong" },
+ "responses={responses}, store={store}, stream={stream}, field={field}"
+ );
+ }
+ let requests = server.received_requests().await.expect("request recording");
+ assert_eq!(requests.len(), 1);
+ assert_eq!(
+ serde_json::from_slice::<Value>(&requests[0].body).expect("request JSON")["store"],
+ store
+ );
+ }
+ }
+ Ok(())
+ }
+
+ #[test]
+ fn state_owners_preserve_existing_records_at_capacity()
+ -> std::result::Result<(), LlmClientError> {
+ let mut owners = StateOwners::default();
+ let model = ModelId::from("model/a");
+ for id in 0..MAX_STATE_OWNERS - 1 {
+ owners.remember(Some(&format!("resp_{id}")), None, &model)?;
+ }
+ assert!(matches!(
+ owners.remember(Some("new_response"), Some("new_conversation"), &model),
+ Err(LlmClientError::ResponseStateLimitExceeded {
+ limit: MAX_STATE_OWNERS
+ })
+ ));
+ assert!(owners.owner("new_response").is_none());
+ assert!(owners.owner("new_conversation").is_none());
+ owners.remember(Some("last_id"), Some("last_id"), &model)?;
+ owners.remember(Some("last_id"), Some("resp_0"), &model)?;
+ assert!(matches!(
+ owners.remember(Some("over_limit"), None, &model),
+ Err(LlmClientError::ResponseStateLimitExceeded { .. })
+ ));
+ assert!(matches!(
+ owners.remember(Some("resp_0"), None, &ModelId::from("model/b")),
+ Err(LlmClientError::ResponseStateConflict)
+ ));
+ assert_eq!(owners.owner("resp_0"), Some(&model));
+ assert_eq!(owners.by_id.len(), MAX_STATE_OWNERS);
+ Ok(())
+ }
+
+ #[test]
+ fn concurrent_state_owners_do_not_exceed_capacity() {
+ let model = ModelId::from("model/a");
+ let owners = Arc::new(Mutex::new(StateOwners {
+ by_id: (0..MAX_STATE_OWNERS - 16)
+ .map(|id| (format!("existing_{id}"), model.clone()))
+ .collect(),
+ }));
+ let barrier = Arc::new(std::sync::Barrier::new(32));
+ let accepted = std::thread::scope(|scope| {
+ let handles = (0..32)
+ .map(|id| {
+ let owners = owners.clone();
+ let barrier = barrier.clone();
+ let model = model.clone();
+ scope.spawn(move || {
+ barrier.wait();
+ let result = owners.lock().remember(
+ Some(&format!("resp_{id}")),
+ Some(&format!("conv_{id}")),
+ &model,
+ );
+ match result {
+ Ok(()) => Some(id),
+ Err(LlmClientError::ResponseStateLimitExceeded { .. }) => None,
+ other => panic!("unexpected state tracking result: {other:?}"),
+ }
+ })
+ })
+ .collect::<Vec<_>>();
+ handles
+ .into_iter()
+ .filter_map(|handle| handle.join().expect("request thread panicked"))
+ .collect::<Vec<_>>()
+ });
+ let owners = owners.lock();
+ assert_eq!(accepted.len(), 8);
+ assert_eq!(owners.by_id.len(), MAX_STATE_OWNERS);
+ assert_eq!(owners.owner("existing_0"), Some(&model));
+ for id in accepted {
+ assert_eq!(owners.owner(&format!("resp_{id}")), Some(&model));
+ assert_eq!(owners.owner(&format!("conv_{id}")), Some(&model));
+ }
+ }
+
#[tokio::test]
async fn each_fallback_candidate_receives_only_its_own_prompt() -> Result<()> {
let client = Arc::new(CandidateClient {
diff --git a/crates/libsy/src/algorithms/advisor_gate.rs b/crates/libsy/src/algorithms/advisor_gate.rs
index 4afd732a3..d638864f8 100644
--- a/crates/libsy/src/algorithms/advisor_gate.rs
+++ b/crates/libsy/src/algorithms/advisor_gate.rs
@@ -51,6 +51,7 @@ mod transcript;
mod trigger;
mod turn;
+use super::util::buffered_response::{BufferedResponse, buffer_response};
use budget::{ReviewBudget, ScopeKey, budget_scope, stall_key};
use signals::{GateSignalProcessor, GateSignals};
use telemetry::{
@@ -61,7 +62,7 @@ use transcript::{VERDICT_PATTERN, Verdict, advisor_reply_text, parse_verdict, re
use trigger::TriggerClassifier;
#[cfg(test)]
use turn::has_tool_use;
-use turn::{GatedTurn, buffer_turn, reasoning_text, visible_text};
+use turn::{reasoning_text, visible_text};
/// APPROVE/REDO reviewer contract sent as the advisor's system prompt.
pub const REVIEWER_SYSTEM_PROMPT: &str =
@@ -238,7 +239,7 @@ impl AdvisorGate {
.served_model()
.cloned()
.unwrap_or_else(|| executor.clone());
- let turn = buffer_turn(served_executor.as_str(), response).await?;
+ let turn = buffer_response(served_executor.as_str(), response).await?;
// Response-side signals fold in after it: the terminal turn never
// appears on a later request, so the trigger runs on this event.
@@ -346,7 +347,7 @@ impl AdvisorGate {
executor_fallbacks: &[ModelId],
served_executor: &ModelId,
request: Request,
- turn: GatedTurn,
+ turn: BufferedResponse,
plan: &str,
) -> RoutingOutcome {
record_discarded(&turn.agg.usage);
diff --git a/crates/libsy/src/algorithms/advisor_gate/turn.rs b/crates/libsy/src/algorithms/advisor_gate/turn.rs
index 62964caea..48f41545e 100644
--- a/crates/libsy/src/algorithms/advisor_gate/turn.rs
+++ b/crates/libsy/src/algorithms/advisor_gate/turn.rs
@@ -1,104 +1,11 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
-//! One buffered executor turn: consuming it to completion, inspecting it,
-//! and replaying it to the client verbatim.
+//! Extract reply text and tool-use signals for advisor review.
-use futures::StreamExt;
-use switchyard_protocol::{
- AggLlmResponse, ContentBlock, LlmClientError, LlmResponse, LlmResponseChunk,
- LlmResponseStreamEvent, Metadata, Response, ResponseAccumulator, StopReason,
-};
+use switchyard_protocol::{AggLlmResponse, ContentBlock, StopReason};
-use crate::{LibsyError, Result};
-
-// ── Turn buffering and replay ───────────────────────────────────────────────
-
-/// One fully generated executor turn held while the gate decides.
-pub(super) struct GatedTurn {
- /// Buffered provider events for streamed turns, preservation included, so
- /// replay re-emits them verbatim (signed thinking and provider extensions
- /// survive; folding to an aggregate and re-synthesizing would drop them).
- pub(super) events: Option<Vec<LlmResponseStreamEvent>>,
- /// Folded view for detection, the review tail, the REDO echo, and
- /// discarded-turn usage. For buffered turns this is the original
- /// response, its own preservation intact.
- pub(super) agg: AggLlmResponse,
- pub(super) metadata: Option<Metadata>,
- pub(super) upstream_headers: http::HeaderMap,
-}
-
-impl GatedTurn {
- /// Releases the turn to the client: streamed turns replay their buffered
- /// events verbatim, buffered turns return the original aggregate.
- pub(super) fn into_response(self) -> Response {
- let llm_response = match self.events {
- Some(events) => {
- LlmResponse::Stream(Box::pin(futures::stream::iter(events.into_iter().map(Ok))))
- }
- None => LlmResponse::Agg(self.agg),
- };
- Response {
- llm_response,
- metadata: self.metadata,
- upstream_headers: self.upstream_headers,
- }
- }
-}
-
-/// Consumes the executor response to completion. Mid-stream failures — item
-/// errors and in-band error chunks — become typed client-call errors exactly
-/// as [`LlmResponse::into_agg`] maps them; the client saw nothing yet, so the
-/// turn fails whole.
-pub(super) async fn buffer_turn(executor: &str, response: Response) -> Result<GatedTurn> {
- let metadata = response.metadata;
- let upstream_headers = response.upstream_headers;
- match response.llm_response {
- LlmResponse::Agg(agg) => Ok(GatedTurn {
- events: None,
- agg,
- metadata,
- upstream_headers,
- }),
- LlmResponse::Stream(mut stream) => {
- let mut events = Vec::new();
- let mut accumulator = ResponseAccumulator::new();
- while let Some(item) = stream.next().await {
- let event =
- item.map_err(|source| LibsyError::client_call(executor.to_string(), source))?;
- for chunk in event.normalized() {
- let failure = match chunk {
- LlmResponseChunk::DecodeError { message } => {
- Some(LlmClientError::ResponseTranslation(message.clone()))
- }
- LlmResponseChunk::StreamError { message } => {
- Some(LlmClientError::UpstreamHttp {
- status: http::StatusCode::BAD_GATEWAY,
- body: message.clone(),
- })
- }
- chunk => {
- accumulator.push(chunk.clone());
- None
- }
- };
- if let Some(source) = failure {
- return Err(LibsyError::client_call(executor.to_string(), source));
- }
- }
- events.push(event);
- }
- Ok(GatedTurn {
- events: Some(events),
- agg: accumulator.finish(),
- metadata,
- upstream_headers,
- })
- }
- }
-}
-
-// ── Detection over the folded turn ──────────────────────────────────────────
+// ── Response inspection ────────────────────────────────────────────────────
/// Whether the turn carries tool use on either signal: a `ToolUse` stop
/// reason, or any tool-call block (some OSS servers mislabel tool-call turns
diff --git a/crates/libsy/src/algorithms/escalation.rs b/crates/libsy/src/algorithms/escalation.rs
index 94bc38e87..dfe2c7fe3 100644
--- a/crates/libsy/src/algorithms/escalation.rs
+++ b/crates/libsy/src/algorithms/escalation.rs
@@ -7,9 +7,10 @@ use std::sync::Arc;
use async_trait::async_trait;
use switchyard_protocol::{
- AggLlmResponse, Category, LlmClientError, LlmResponse, Message, Request, Response, Role,
+ AggLlmResponse, Category, LlmClientError, Message, Request, Response, Role,
};
+use super::util::buffered_response::buffer_response;
use super::util::classifier_contract::ClassifierContractConfig;
use super::util::decisive;
use super::util::escalation::{self, EscalationJudge, EscalationJudgeConfig, EscalationPolicy};
@@ -109,41 +110,37 @@ impl Classifier<State> for EscalationClassifier {
};
// The call resolves when its stream handle arrives; context and transport failures can
// still occur while buffering.
- let agg = match efficient_response.llm_response.into_agg().await {
- Ok(agg) => agg,
- Err(LlmClientError::ContextWindowExceeded { .. }) => {
+ let efficient_response = match buffer_response(efficient.as_str(), efficient_response).await
+ {
+ Ok(response) => response,
+ Err(LibsyError::ClientCall {
+ source: LlmClientError::ContextWindowExceeded { .. },
+ ..
+ }) => {
driver.set_evidence(serde_json::json!({
"source": "fallback",
"reason_code": "context_window",
}));
return Ok((decisive(&capable), None));
}
- Err(LlmClientError::Transport { .. }) => {
+ Err(LibsyError::ClientCall {
+ source: LlmClientError::Transport { .. },
+ ..
+ }) => {
driver.set_evidence(serde_json::json!({
"source": "fallback",
"reason_code": "transport",
}));
return Ok((decisive(&capable), None));
}
- Err(source) => {
- return Err(LibsyError::client_call(efficient.clone(), source));
- }
+ Err(error) => return Err(error),
};
// Append the efficient reply so the judge reads this turn's completed trajectory.
let mut judge_request = request.clone();
judge_request
.llm_request
.messages
- .push(assistant_message(&agg));
- let efficient_response = Response {
- llm_response: if request.llm_request.stream {
- LlmResponse::Stream(agg.into_stream())
- } else {
- LlmResponse::Agg(agg)
- },
- metadata: efficient_response.metadata,
- upstream_headers: efficient_response.upstream_headers,
- };
+ .push(assistant_message(&efficient_response.agg));
let (classification, _) = self.judge.score(state, &mut judge_request, driver).await?;
@@ -174,7 +171,10 @@ impl Classifier<State> for EscalationClassifier {
}));
}
- Ok((decisive(&efficient), Some(efficient_response)))
+ Ok((
+ decisive(&efficient),
+ Some(efficient_response.into_response()),
+ ))
}
}
diff --git a/crates/libsy/src/algorithms/util.rs b/crates/libsy/src/algorithms/util.rs
index 9a290920f..0b96ca92a 100644
--- a/crates/libsy/src/algorithms/util.rs
+++ b/crates/libsy/src/algorithms/util.rs
@@ -2,6 +2,7 @@
// SPDX-License-Identifier: Apache-2.0
pub mod affinity;
+pub(crate) mod buffered_response;
pub(crate) mod classifier_contract;
pub mod escalation;
pub(crate) mod llm_judge;
diff --git a/crates/libsy/src/algorithms/util/buffered_response.rs b/crates/libsy/src/algorithms/util/buffered_response.rs
new file mode 100644
index 000000000..706a117d6
--- /dev/null
+++ b/crates/libsy/src/algorithms/util/buffered_response.rs
@@ -0,0 +1,94 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+//! Buffer a model response for inspection and replay its original stream events.
+
+use futures::StreamExt;
+use switchyard_protocol::{
+ AggLlmResponse, LlmClientError, LlmResponse, LlmResponseChunk, LlmResponseStreamEvent,
+ Metadata, Response, ResponseAccumulator,
+};
+
+use crate::{LibsyError, Result};
+
+/// A completed model response held while an algorithm inspects it.
+pub(crate) struct BufferedResponse {
+ /// Original stream events, including response IDs, signed thinking, and provider
+ /// extensions. Replaying these preserves data lost when synthesizing events from `agg`.
+ events: Option<Vec<LlmResponseStreamEvent>>,
+ /// Complete response for inspection and usage accounting. For a non-streaming reply,
+ /// this is the original response with its preserved provider data.
+ pub(crate) agg: AggLlmResponse,
+ metadata: Option<Metadata>,
+ upstream_headers: http::HeaderMap,
+}
+
+impl BufferedResponse {
+ /// Replay the buffered stream events or return the original non-streaming response.
+ pub(crate) fn into_response(self) -> Response {
+ let llm_response = match self.events {
+ Some(events) => {
+ LlmResponse::Stream(Box::pin(futures::stream::iter(events.into_iter().map(Ok))))
+ }
+ None => LlmResponse::Agg(self.agg),
+ };
+ Response {
+ llm_response,
+ metadata: self.metadata,
+ upstream_headers: self.upstream_headers,
+ }
+ }
+}
+
+/// Consume the response to completion before replaying any events. Stream item errors
+/// and error chunks become typed client-call errors, as in [`LlmResponse::into_agg`].
+pub(crate) async fn buffer_response(
+ executor: &str,
+ response: Response,
+) -> Result<BufferedResponse> {
+ let metadata = response.metadata;
+ let upstream_headers = response.upstream_headers;
+ match response.llm_response {
+ LlmResponse::Agg(agg) => Ok(BufferedResponse {
+ events: None,
+ agg,
+ metadata,
+ upstream_headers,
+ }),
+ LlmResponse::Stream(mut stream) => {
+ let mut events = Vec::new();
+ let mut accumulator = ResponseAccumulator::new();
+ while let Some(item) = stream.next().await {
+ let event =
+ item.map_err(|source| LibsyError::client_call(executor.to_string(), source))?;
+ for chunk in event.normalized() {
+ let failure = match chunk {
+ LlmResponseChunk::DecodeError { message } => {
+ Some(LlmClientError::ResponseTranslation(message.clone()))
+ }
+ LlmResponseChunk::StreamError { message } => {
+ Some(LlmClientError::UpstreamHttp {
+ status: http::StatusCode::BAD_GATEWAY,
+ body: message.clone(),
+ })
+ }
+ chunk => {
+ accumulator.push(chunk.clone());
+ None
+ }
+ };
+ if let Some(source) = failure {
+ return Err(LibsyError::client_call(executor.to_string(), source));
+ }
+ }
+ events.push(event);
+ }
+ Ok(BufferedResponse {
+ events: Some(events),
+ agg: accumulator.finish(),
+ metadata,
+ upstream_headers,
+ })
+ }
+ }
+}
diff --git a/crates/protocol/src/client.rs b/crates/protocol/src/client.rs
index 6b24afcc6..f8626d2bd 100644
--- a/crates/protocol/src/client.rs
+++ b/crates/protocol/src/client.rs
@@ -50,6 +50,19 @@ pub enum LlmClientError {
message: String,
},
+ /// The route cannot record another provider-owned response or conversation ID.
+ #[error(
+ "Responses state tracking reached its limit of {limit} IDs; no existing records were removed"
+ )]
+ ResponseStateLimitExceeded {
+ /// Maximum number of IDs retained by this route.
+ limit: usize,
+ },
+
+ /// Two configured models reported the same saved response or conversation ID.
+ #[error("Responses state ID is already recorded for another model; its owner was not changed")]
+ ResponseStateConflict,
+
/// The upstream could not be reached or the request could not be sent.
#[error("upstream transport error: {source}")]
Transport {
diff --git a/crates/switchyard-runner/src/config.rs b/crates/switchyard-runner/src/config.rs
index d65c585dc..2d258474f 100644
--- a/crates/switchyard-runner/src/config.rs
+++ b/crates/switchyard-runner/src/config.rs
@@ -378,8 +378,17 @@ impl DeploymentConfig {
prompts,
routing_answer_target,
} = self.build_route_target_prompts(route_name, route)?;
- let router =
- ClientRouter::new_with_target_prompts(by_model, prompts, routing_answer_target);
+ let completion_targets = route
+ .routing_target_names()
+ .into_iter()
+ .map(|name| self.targets[name].id.clone())
+ .collect::<Vec<_>>();
+ let router = ClientRouter::new_with_completion_targets(
+ by_model,
+ prompts,
+ routing_answer_target,
+ &completion_targets,
+ );
Ok((router, caller_auth))
}
diff --git a/crates/switchyard-server/src/lib.rs b/crates/switchyard-server/src/lib.rs
index 7055b637e..56d222503 100644
--- a/crates/switchyard-server/src/lib.rs
+++ b/crates/switchyard-server/src/lib.rs
@@ -1045,7 +1045,18 @@ async fn handle_llm_request(
let output = match route.execute(request, Some(observer)).await {
Ok(output) => output,
- Err(error) => return runner_error(error),
+ Err(error) => {
+ if let RunnerError::Algorithm(LibsyError::ClientCall {
+ target,
+ source:
+ LlmClientError::ResponseStateLimitExceeded { .. }
+ | LlmClientError::ResponseStateConflict,
+ }) = &error
+ {
+ state.stats.record_response_error(target);
+ }
+ return runner_error(error);
+ }
};
let RunOutput {
selected_model,
@@ -1250,6 +1261,18 @@ fn runner_error(error: RunnerError) -> Response {
fn client_error(error: &LlmClientError) -> Response {
match error {
+ LlmClientError::ResponseStateLimitExceeded { .. } => error_response(
+ StatusCode::SERVICE_UNAVAILABLE,
+ error.to_string(),
+ "server_error",
+ "response_state_limit_exceeded",
+ ),
+ LlmClientError::ResponseStateConflict => error_response(
+ StatusCode::CONFLICT,
+ error.to_string(),
+ "server_error",
+ "response_state_conflict",
+ ),
LlmClientError::InvalidRequest { message }
| LlmClientError::RequestTranslation(message) => error_response(
StatusCode::BAD_REQUEST,
diff --git a/crates/switchyard-server/src/stats/accumulator.rs b/crates/switchyard-server/src/stats/accumulator.rs
index 281609229..357b342eb 100644
--- a/crates/switchyard-server/src/stats/accumulator.rs
+++ b/crates/switchyard-server/src/stats/accumulator.rs
@@ -68,8 +68,9 @@ impl StatsAccumulator {
stats.errors = stats.errors.saturating_add(1);
}
- /// Records a stream failure after its routed call was already counted.
- pub(crate) fn record_stream_error(&self, model: impl Into<ModelId>) {
+ /// Record a failure while processing a buffered response or stream after its routed call
+ /// was already counted.
+ pub(crate) fn record_response_error(&self, model: impl Into<ModelId>) {
let mut inner = self.lock();
inner.total_errors = inner.total_errors.saturating_add(1);
let stats = inner.model_stats_mut(model.into());
diff --git a/crates/switchyard-server/src/usage_metrics.rs b/crates/switchyard-server/src/usage_metrics.rs
index bb4e21ad8..158b58779 100644
--- a/crates/switchyard-server/src/usage_metrics.rs
+++ b/crates/switchyard-server/src/usage_metrics.rs
@@ -106,7 +106,7 @@ pub(crate) fn observe(
// Records a terminal stream failure after the routed call was already counted.
fn record_stream_error(stats: &StatsAccumulator, model: &str) {
- stats.record_stream_error(model);
+ stats.record_response_error(model);
global::meter("switchyard")
.u64_counter("switchyard.errors")
.build()
diff --git a/crates/switchyard-server/tests/server.rs b/crates/switchyard-server/tests/server.rs
index 697558287..00295e7d9 100644
--- a/crates/switchyard-server/tests/server.rs
+++ b/crates/switchyard-server/tests/server.rs
@@ -10,7 +10,7 @@ use std::io::Write;
use std::sync::Arc;
use axum::body::{Body, Bytes};
-use axum::extract::{DefaultBodyLimit, State};
+use axum::extract::{DefaultBodyLimit, Path, State};
use axum::http::{HeaderMap, HeaderValue, Request as HttpRequest, StatusCode, Uri};
use axum::response::sse::{Event, Sse};
use axum::response::{IntoResponse, Response as HttpResponse};
@@ -60,6 +60,7 @@ impl MockUpstream {
"/v1/responses",
post(upstream_responses_requires_forwarded_auth),
)
+ .route("/silo/{silo}/responses", post(upstream_responses_silo))
.route("/capture", post(upstream_redirect_capture))
.route("/v1/messages/count_tokens", post(upstream_count_tokens))
.route(
@@ -559,6 +560,95 @@ async fn upstream_buffered_responses(
Json(response)
}
+/// A local Responses provider whose IDs normally contain its name. It returns `state_not_found`
+/// for another provider's ID, except `resp_shared_conflict`, which tests duplicate ownership.
+/// The judge selects the strong tier when the input contains `ROUTE_B`.
+async fn upstream_responses_silo(
+ State(calls): State<Arc<Mutex<Vec<Value>>>>,
+ Path(silo): Path<String>,
+ Json(body): Json<Value>,
+) -> HttpResponse {
+ let mut calls = calls.lock().await;
+ calls.push(body.clone());
+ let model = body["model"].as_str().unwrap_or_default().to_string();
+ if model == "model/judge" {
+ let strong = body.to_string().contains("ROUTE_B");
+ let verdict = if body["text"]["format"]["schema"]["properties"]
+ .get("escalate")
+ .is_some()
+ {
+ json!({"escalate": strong, "reason": "state probe"})
+ } else {
+ json!({
+ "crux": "state probe", "primary_rule": if strong { "LIM-1" } else { "SUP-1" },
+ "capability_boundary": if strong { "unsupported" } else { "supported" },
+ "p_solve": if strong { 0.1 } else { 0.9 },
+ })
+ };
+ return Json(responses_body("resp_judge", &model, &verdict.to_string())).into_response();
+ }
+ let minted_prefix = format!("resp_{silo}_");
+ if silo == "A" && body.to_string().contains("owner-unavailable") {
+ return (
+ StatusCode::SERVICE_UNAVAILABLE,
+ Json(json!({"error": {"message": "provider A is unavailable"}})),
+ )
+ .into_response();
+ }
+ if let Some(previous) = body["previous_response_id"].as_str()
+ && !previous.starts_with(&minted_prefix)
+ && previous != "resp_shared_conflict"
+ {
+ return (
+ StatusCode::BAD_REQUEST,
+ Json(
+ json!({"error": {"code": "state_not_found", "message": format!(
+ "previous_response_id {previous} is not present in provider {silo}"
+ )}}),
+ ),
+ )
+ .into_response();
+ }
+ let id = if body.to_string().contains("state-id-conflict") {
+ "resp_shared_conflict".to_string()
+ } else {
+ format!("{minted_prefix}{}", calls.len())
+ };
+ let text = format!("served-by:{silo}");
+ let mut response = responses_body(&id, &model, &text);
+ response["store"] = body.get("store").cloned().unwrap_or(json!(true));
+ response["conversation"] = body.get("conversation").cloned().unwrap_or(Value::Null);
+ if body["stream"] == true {
+ let mut created = response.clone();
+ created["status"] = json!("in_progress");
+ created["output"] = json!([]);
+ let events = [
+ json!({"type": "response.created", "response": created}),
+ json!({"type": "response.output_text.delta", "output_index": 0, "delta": text}),
+ json!({"type": "response.completed", "response": response}),
+ ];
+ let stream = futures_util::stream::iter(events.into_iter().map(|event| {
+ Ok::<Event, Infallible>(
+ Event::default()
+ .event(event["type"].as_str().unwrap_or_default())
+ .data(event.to_string()),
+ )
+ }));
+ return Sse::new(stream).into_response();
+ }
+ Json(response).into_response()
+}
+
+fn responses_body(id: &str, model: &str, text: &str) -> Value {
+ json!({
+ "id": id, "object": "response", "created_at": 1, "model": model, "status": "completed",
+ "output": [{"id": "msg_1", "type": "message", "status": "completed", "role": "assistant",
+ "content": [{"type": "output_text", "text": text, "annotations": []}]}],
+ "usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2,
+ "output_tokens_details": {"reasoning_tokens": 0}}
+ })
+}
+
async fn upstream_redirect_capture(
State(calls): State<Arc<Mutex<Vec<Value>>>>,
headers: HeaderMap,
@@ -1505,6 +1595,214 @@ base_threshold = 0.5
Ok(())
}
+/// Configure classifier and escalation routes over two Responses providers.
+/// `route/shared` uses one answer client; the other routes must return known continuations
+/// to the model that served their state, even when the judge would select another provider.
+fn state_silo_config(base_url: &str) -> String {
+ let root = base_url.trim_end_matches("/v1");
+ format!(
+ r#"
+schema_version = 1
+[llm_clients.judge]
+format = "openai_responses"
+base_url = "{root}/silo/judge"
+[llm_clients.a]
+format = "openai_responses"
+base_url = "{root}/silo/A"
+max_retries = 0
+[llm_clients.b]
+format = "openai_responses"
+base_url = "{root}/silo/B"
+[targets.judge]
+id = "model/judge"
+llm_client = "judge"
+[targets.a]
+id = "model/provider-a"
+llm_client = "a"
+[targets.b]
+id = "model/provider-b"
+llm_client = "b"
+[targets.shared_b]
+id = "model/shared-b"
+llm_client = "a"
+[routes.shared]
+id = "route/shared"
+type = "llm_classifier"
+classifier_target = "judge"
+weak_target = "a"
+strong_target = "shared_b"
+base_threshold = 0.5
+classify_trigger = "every_request"
+[routes.escalation]
+id = "route/escalation"
+type = "llm_classifier"
+mode = "escalation"
+classifier_target = "judge"
+weak_target = "a"
+strong_target = "b"
+escalation = {{ confirmations = 1 }}
+[routes.dynamic]
+id = "route/dynamic"
+type = "llm_classifier"
+classifier_target = "judge"
+weak_target = "a"
+strong_target = "b"
+base_threshold = 0.5
+classify_trigger = "every_request"
+"#
+ )
+}
+
+#[tokio::test]
+async fn responses_continuations_preserve_state_ownership() -> TestResult {
+ for (route, stream, endpoint) in [
+ ("route/dynamic", false, "/v1/responses"),
+ ("route/dynamic", true, "/v1/responses"),
+ ("route/escalation", false, "/v1/decision"),
+ ("route/escalation", true, "/v1/decision"),
+ ("route/shared", false, "/v1/responses"),
+ ] {
+ let upstream = MockUpstream::start().await?;
+ let app =
+ build_switchyard_router(load_test_config(&state_silo_config(&upstream.base_url))?);
+ let mut request =
+ json!({"model": route, "input": "ROUTE_A remember mango", "stream": stream});
+ if endpoint == "/v1/decision" {
+ request = json!({"input_format": "openai_responses", "request": request});
+ }
+ let seed = send(&app, "POST", endpoint, Some(request)).await?;
+ assert_eq!(seed.status, StatusCode::OK, "{}", seed.text()?);
+ let body = if endpoint == "/v1/decision" {
+ seed.json()?["response"].clone()
+ } else if stream {
+ sse_events(seed.text()?)
+ .into_iter()
+ .find(|event| event["type"] == "response.completed")
+ .ok_or("stream ended without response.completed")?["response"]
+ .clone()
+ } else {
+ seed.json()?
+ };
+ assert!(
+ body["id"]
+ .as_str()
+ .is_some_and(|id| id.starts_with("resp_A_"))
+ );
+ let mut follow_request = json!({"model": route, "input": "ROUTE_B recall mango", "previous_response_id": body["id"]});
+ let before = upstream.models().await.len();
+ let follow = send(&app, "POST", "/v1/responses", Some(follow_request.clone())).await?;
+ assert_eq!(follow.status, StatusCode::OK, "{}", follow.text()?);
+ let expected = if route == "route/shared" {
+ vec!["model/judge", "model/shared-b"]
+ } else {
+ vec!["model/provider-a"]
+ };
+ assert_eq!(&upstream.models().await[before..], expected);
+ assert_eq!(
+ follow.headers["x-model-router-selected-model"],
+ *expected.last().ok_or("missing target")?
+ );
+ if route == "route/dynamic" {
+ follow_request["previous_response_id"] = follow.json()?["id"].clone();
+ let chained = send(&app, "POST", "/v1/responses", Some(follow_request.clone())).await?;
+ assert_eq!(chained.status, StatusCode::OK);
+ assert_eq!(
+ chained.headers["x-model-router-selected-model"],
+ "model/provider-a"
+ );
+ let before = upstream.models().await.len();
+ let fresh = send(
+ &app,
+ "POST",
+ "/v1/responses",
+ Some(json!({"model": route, "input": "ROUTE_B a new task"})),
+ )
+ .await?;
+ assert_eq!(fresh.status, StatusCode::OK);
+ assert_eq!(
+ fresh.headers["x-model-router-selected-model"],
+ "model/provider-b"
+ );
+ assert_eq!(
+ &upstream.models().await[before..],
+ ["model/judge", "model/provider-b"]
+ );
+ follow_request["input"] = json!("ROUTE_B owner-unavailable");
+ let before = upstream.models().await.len();
+ let unavailable = send(&app, "POST", "/v1/responses", Some(follow_request)).await?;
+ assert_eq!(unavailable.status, StatusCode::SERVICE_UNAVAILABLE);
+ assert_eq!(&upstream.models().await[before..], ["model/provider-a"]);
+ }
+ }
+ for (route, stream, endpoint) in [
+ ("route/dynamic", false, "/v1/responses"),
+ ("route/dynamic", true, "/v1/responses"),
+ ("route/escalation", false, "/v1/decision"),
+ ("route/escalation", true, "/v1/decision"),
+ ] {
+ let upstream = MockUpstream::start().await?;
+ let app =
+ build_switchyard_router(load_test_config(&state_silo_config(&upstream.base_url))?);
+ let decision = endpoint == "/v1/decision";
+ let input = if decision {
+ "ROUTE_B state-id-conflict"
+ } else {
+ "ROUTE_A state-id-conflict"
+ };
+ let seed = send(
+ &app,
+ "POST",
+ "/v1/responses",
+ Some(json!({"model": route, "input": input})),
+ )
+ .await?;
+ assert_eq!(seed.status, StatusCode::OK);
+ let owner = seed.headers["x-model-router-selected-model"].clone();
+ let input = if decision {
+ "ROUTE_A state-id-conflict"
+ } else {
+ "ROUTE_B state-id-conflict"
+ };
+ let mut request = json!({"model": route, "input": input, "stream": stream});
+ if decision {
+ request = json!({"input_format": "openai_responses", "request": request});
+ }
+ let before = upstream.models().await.len();
+ let rejected = send(&app, "POST", endpoint, Some(request)).await?;
+ if stream && !decision {
+ assert_eq!(rejected.status, StatusCode::OK);
+ let events = sse_events(rejected.text()?);
+ assert_eq!(events.len(), 1);
+ assert_eq!(events[0]["type"], "error");
+ } else {
+ assert_eq!(
+ rejected.status,
+ StatusCode::CONFLICT,
+ "{}",
+ rejected.text()?
+ );
+ assert_eq!(rejected.json()?["error"]["code"], "response_state_conflict");
+ }
+ assert_eq!(
+ &upstream.models().await[before..],
+ if decision {
+ ["model/provider-a", "model/judge"]
+ } else {
+ ["model/judge", "model/provider-b"]
+ }
+ );
+ if !decision {
+ let stats = send(&app, "GET", "/v1/stats", None).await?.json()?;
+ assert_eq!(stats["total_requests"], 2);
+ assert_eq!(stats["total_errors"], 1);
+ }
+ let follow = send(&app, "POST", "/v1/responses", Some(json!({"model": route, "input": "ROUTE_B recall", "previous_response_id": seed.json()?["id"], "store": false}))).await?;
+ assert_eq!(follow.status, StatusCode::OK);
+ assert_eq!(follow.headers["x-model-router-selected-model"], owner);
+ }
+ Ok(())
+}
+
/// Decision-only routing returns callable metadata and preserves any answer produced while routing.
#[tokio::test]
async fn decision_returns_callable_target_and_routing_answer() -> TestResult {
diff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.md
index cbc893563..329ccadc0 100644
--- a/docs/routing_algorithms/llm_classifier_routing.md
+++ b/docs/routing_algorithms/llm_classifier_routing.md
@@ -250,6 +250,51 @@ identity are judged every time. Clients can send `x-switchyard-session-id`, or
enable `message_hash_fallback` to key on the first user-message text under
`new_session`.
+### Responses continuations by ID
+
+A Responses API client can continue without resending the conversation history:
+`previous_response_id` refers to a stored response, and `conversation` refers to
+a provider's conversation. When answer targets use different `[llm_clients]`
+entries, Switchyard records the model that served each stored Responses ID and
+conversation ID. A request with a recorded ID returns to that exact model
+without a judge call or fallback to another provider. This applies to every
+`classify_trigger` and to `stage_router` and `composite` routes. Classifier-only
+clients do not count; when answer targets share one client, routing proceeds as
+usual.
+
+Switchyard tracks buffered and streamed Responses replies, including answers
+returned by `/v1/decision`. It does not record Chat Completions or Anthropic
+Messages response IDs as Responses continuation IDs. A provider's `store` value
+takes precedence over the request value. If that value is `false`, Switchyard
+skips the response ID but still records the conversation ID.
+
+Each route keeps up to 65,536 distinct ID-to-model records per process. This
+limit counts response and conversation IDs recorded over the process's lifetime,
+not tokens or simultaneous requests. Records do not expire, and Switchyard does
+not remove older records to make room. The map stores copies of IDs and model
+names, with no transcript storage.
+
+If recording new IDs would exceed the limit, a buffered reply returns HTTP 503
+with code `response_state_limit_exceeded`. If an ID is already recorded for a
+different model, it returns HTTP 409 with code `response_state_conflict`. Neither
+failure changes existing records. Switchyard checks both IDs before adding
+either from a response or stream event.
+
+These checks run after the provider has returned a response or stream event, so
+the error does not mean the provider did no work. A stream that has already sent
+HTTP headers emits a framed error and stops; it cannot change the HTTP status.
+Switchyard logs these errors. For `/v1/responses`, buffered and streamed failures
+increment `/v1/stats` `total_errors`. Other routes continue independently.
+
+An existing ID can still be used at capacity if the reply adds no new IDs. For
+example, `store: false` can suppress a new response ID, but it does not suppress
+conversation-ID tracking. A stored follow-up normally creates a new response ID
+and therefore fails when the map is full.
+
+A restart removes all records, and each replica has its own map. Unknown IDs use
+normal routing and can still fail at the selected provider. To keep dynamic
+routing across turns, send the full history instead of an ID.
+
## Run the route
After [installing the Rust server](../getting_started.md#install-the-server), export