Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
fix(llm-client): report Responses state limits without losing saved IDs (#721)
Elyas Mehtabuddin2d485f20007a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ca2b3bf06..0e987ef86 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -121,6 +121,18 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).### Fixed+- **Responses continuations stay with the provider that holds their state** — a+ request with `previous_response_id` or `conversation` could reach another+ provider and fail with `400 state_not_found`. When answer targets use different+ `[llm_clients]` entries, Switchyard records which model served each stored+ Responses ID and conversation ID. Known continuations return to that model+ without running the algorithm or trying another provider. Each route keeps up+ to 65,536 IDs per process without discarding older records. Recording another+ model for an existing ID returns `response_state_conflict`; exceeding the cap+ returns `response_state_limit_exceeded`. Buffered replies use HTTP 409 and 503,+ respectively; streams emit an error and stop. These checks happen after the+ provider has done work. Records are lost on restart. A separate classifier+ client does not affect routes whose answer targets share one client.- **Cross-format tool results keep image and file content** — an Anthropic`tool_result` carrying image or document blocks reached a Responses targetas text only, and an image-only result became an empty `output`. In the otherdiff --git a/crates/libsy-llm-client/src/run.rs b/crates/libsy-llm-client/src/run.rsindex 380e9f602..c16922c44 100644--- a/crates/libsy-llm-client/src/run.rs+++ b/crates/libsy-llm-client/src/run.rs@@ -12,18 +12,26 @@//! ordered candidate fallback, and the `libsy.client_call` span around each candidate. Each//! candidate exhausts its backend retry budget before fallback advances, so the worst case is//! `candidates × (max_retries + 1)` upstream attempts plus every candidate's backoff.+//!+//! A Responses continuation can refer to state held by one provider through+//! `previous_response_id` or `conversation`. When completion targets use different clients,+//! [`ClientRouter`] records which model served that state and sends known continuations+//! back to it without running the algorithm or trying another provider.use std::collections::HashMap;use std::sync::Arc;use std::time::Instant;+use futures_util::StreamExt;use http::StatusCode;use parking_lot::Mutex;+use serde_json::{Value, json};use switchyard_libsy::{- Algorithm, CallModel, LibsyError, Result, RoutingOutcome, RuntimeModels, drive,+ Algorithm, CallModel, LibsyError, OutcomeMetadata, Result, RoutingOutcome, RuntimeModels, drive,};use switchyard_protocol::{- LlmClientError, ModelId, Request, Response, RoutedLlmClient, RoutingFallbackReason,+ LlmClientError, LlmResponse, ModelId, Request, Response, RoutedLlmClient,+ RoutingFallbackReason, WireFormat,};use switchyard_translation::prepare_request_for_target;@@ -55,11 +63,16 @@ pub async fn run(// This says if we have an observer, put Some(..) in routing_observations.// No observer means we don't want any routing_observations.let routing_observations = observer.as_ref().map(|_| Arc::new(Mutex::new(Vec::new())));- let outcome = drive(algorithm, request, models, {- let routing_observations = routing_observations.clone();- move |call| serve(routing_clients.clone(), call, routing_observations.clone())- })- .await;+ let outcome = match clients.stored_state_owner(&request) {+ Some(owner) => Ok(continue_on(owner, &algorithm_name, request)),+ None => {+ drive(algorithm, request, models, {+ let routing_observations = routing_observations.clone();+ move |call| serve(routing_clients.clone(), call, routing_observations.clone())+ })+ .await+ }+ };let answered_model = outcome.as_ref().ok()@@ -98,6 +111,8 @@ pub async fn run();(result, Some(answer_duration))};+ let result =+ result.and_then(|response| clients.remember_state_owner(&outcome.request, response));metrics::record_routed_request(&selected_model_id, answer_duration, &result);if let Some(observer) = &observer {observer(RunObservation::RoutingOverhead(overhead));@@ -116,12 +131,21 @@ pub async fn decide(models: Arc<RuntimeModels>,) -> Result<RoutingOutcome> {let routing_clients = clients.clone();- let mut outcome = drive(algorithm, request, models, move |call| {- serve(routing_clients.clone(), call, None)- })- .await?;+ let mut outcome = match clients.stored_state_owner(&request) {+ Some(owner) => continue_on(owner, algorithm.name(), request),+ None => {+ drive(algorithm, request, models, move |call| {+ serve(routing_clients.clone(), call, None)+ })+ .await?+ }+ };let selected_model_id = outcome.selected_model_id()?.clone();outcome.request = clients.prepare_completion_request(outcome.request, &selected_model_id);+ outcome.response = outcome+ .response+ .map(|response| clients.remember_state_owner(&outcome.request, response))+ .transpose()?;Ok(outcome)}@@ -328,6 +352,83 @@ fn fallback_reason(error: &LibsyError) -> Option<RoutingFallbackReason> {}}+/// Route to `owner` without fallbacks and record why the algorithm did not run.+fn continue_on(owner: StateOwner, algorithm: &str, request: Request) -> RoutingOutcome {+ tracing::debug!(+ target = %owner.model,+ trigger = owner.trigger,+ "routing Responses continuation to the model that holds its state"+ );+ let mut outcome = RoutingOutcome::route_to(owner.model, Vec::new(), request);+ outcome.metadata = Some(OutcomeMetadata::new(+ algorithm.to_string(),+ Some(json!({"source": "stored_state", "trigger": owner.trigger})),+ ));+ outcome+}++/// The model that served the stored state and the request field that refers to it.+struct StateOwner {+ model: ModelId,+ trigger: &'static str,+}++/// Stored Responses IDs and conversation IDs, mapped to the model that served them.+///+/// Keep earlier response IDs because a client can continue from any stored response.+/// Reject new IDs at capacity rather than forgetting where an earlier response is stored.+#[derive(Default)]+struct StateOwners {+ by_id: HashMap<String, ModelId>,+}++const MAX_STATE_OWNERS: usize = 65_536;++impl StateOwners {+ fn owner(&self, id: &str) -> Option<&ModelId> {+ self.by_id.get(id)+ }++ fn remember(+ &mut self,+ response_id: Option<&str>,+ conversation_id: Option<&str>,+ model: &ModelId,+ ) -> std::result::Result<(), LlmClientError> {+ let ids = [+ response_id,+ conversation_id.filter(|id| Some(*id) != response_id),+ ];+ let mut new_ids = 0;+ for id in ids.into_iter().flatten() {+ match self.by_id.get(id) {+ Some(owner) if owner != model => return Err(LlmClientError::ResponseStateConflict),+ None => new_ids += 1,+ _ => {}+ }+ }+ if self.by_id.len() + new_ids > MAX_STATE_OWNERS {+ return Err(LlmClientError::ResponseStateLimitExceeded {+ limit: MAX_STATE_OWNERS,+ });+ }+ for id in ids.into_iter().flatten() {+ if !self.by_id.contains_key(id) {+ self.by_id.insert(id.to_owned(), model.clone());+ }+ }+ Ok(())+ }+}++/// Read the Responses `conversation` ID from a string or an object.+fn conversation_id(fields: &serde_json::Map<String, Value>) -> Option<&str> {+ let conversation = fields.get("conversation")?;+ conversation+ .as_str()+ .or_else(|| conversation.get("id").and_then(Value::as_str))+}+/// Resolves a routed call's selected model to the client that serves it.////// An algorithm routes among named targets; which provider each target lives on is the@@ -345,6 +446,9 @@ struct ClientRouting {routing: Routing,target_prompts: HashMap<ModelId, String>,routing_answer_target: Option<ModelId>,+ /// Track state only when completion targets use different clients. A separate+ /// classifier client does not require tracking when all completion targets share a client.+ state_owners: Option<Mutex<StateOwners>>,}enum Routing {@@ -365,16 +469,48 @@ impl ClientRouter {/// `routing_answer_target` identifies a routing-time call whose response may become the/// answer. Pass `None` when routing only selects a later completion target. Prompt keys and/// `routing_answer_target` use the resolved model IDs stored in `by_model`.+ /// This constructor treats every model in `by_model` as a possible completion target.+ /// Use [`Self::new_with_completion_targets`] to exclude classifier-only models from the+ /// comparison of completion clients.pub fn new_with_target_prompts(by_model: HashMap<ModelId, Arc<dyn RoutedLlmClient>>,target_prompts: HashMap<ModelId, String>,routing_answer_target: Option<ModelId>,) -> Self {+ let completion_targets = by_model.keys().cloned().collect::<Vec<_>>();+ Self::new_with_completion_targets(+ by_model,+ target_prompts,+ routing_answer_target,+ &completion_targets,+ )+ }++ /// Build a router with an explicit list of models that can answer requests.+ ///+ /// `by_model` contains every callable model, including classifiers. Only the model IDs in+ /// `completion_targets` determine whether completion clients differ and Responses state+ /// needs tracking. Classifier-only models remain callable without affecting that comparison.+ /// `target_prompts` and `routing_answer_target` have the same meaning as in+ /// [`Self::new_with_target_prompts`].+ pub fn new_with_completion_targets(+ by_model: HashMap<ModelId, Arc<dyn RoutedLlmClient>>,+ target_prompts: HashMap<ModelId, String>,+ routing_answer_target: Option<ModelId>,+ completion_targets: &[ModelId],+ ) -> Self {+ let mut clients = completion_targets+ .iter()+ .filter_map(|model| by_model.get(model));+ let spans_providers = clients+ .next()+ .is_some_and(|first| clients.any(|client| !Arc::ptr_eq(first, client)));Self {inner: Arc::new(ClientRouting {routing: Routing::ByModel(by_model),target_prompts,routing_answer_target,+ state_owners: spans_providers.then(Mutex::default),}),}}@@ -390,6 +526,7 @@ impl ClientRouter {routing: Routing::Single(client),target_prompts: HashMap::new(),routing_answer_target: None,+ state_owners: None,}),}}@@ -414,6 +551,94 @@ impl ClientRouter {}}+ /// Return the recorded model for the requested response or conversation ID.+ fn stored_state_owner(&self, request: &Request) -> Option<StateOwner> {+ let owners = self.inner.state_owners.as_ref()?;+ let fields = &request.llm_request.extensions.fields;+ let (trigger, id) = match fields.get("previous_response_id").and_then(Value::as_str) {+ Some(id) => ("previous_response_id", id),+ None => ("conversation", conversation_id(fields)?),+ };+ let model = owners.lock().owner(id)?.clone();+ Some(StateOwner { model, trigger })+ }++ /// Record state reported by a Responses provider. Other response formats have IDs+ /// that cannot be continued through the Responses API.+ fn remember_state_owner(&self, request: &Request, mut response: Response) -> Result<Response> {+ if self.inner.state_owners.is_none() {+ return Ok(response);+ }+ let Some(model) = response.served_model().cloned() else {+ return Ok(response);+ };+ let fields = &request.llm_request.extensions.fields;+ let store = fields.get("store").and_then(Value::as_bool) != Some(false);+ let conversation = conversation_id(fields).map(str::to_owned);+ response.llm_response = match response.llm_response {+ LlmResponse::Agg(agg) => {+ if let Some(body) = agg+ .preservation+ .responses+ .get(&WireFormat::OpenAiResponses.into())+ {+ self.remember_response(body, &model, store, conversation.as_deref())+ .map_err(|error| LibsyError::client_call(model.clone(), error))?;+ }+ LlmResponse::Agg(agg)+ }+ LlmResponse::Stream(stream) => {+ let router = self.clone();+ LlmResponse::Stream(Box::pin(futures_util::stream::try_unfold(+ (stream, router, model, conversation),+ move |(mut stream, router, model, conversation)| async move {+ let Some(event) = stream.next().await else {+ return Ok(None);+ };+ let event = event?;+ if let Some(preserved) = event.preservation()+ && preserved.source().as_str() == WireFormat::OpenAiResponses.as_str()+ && let Some(body) = preserved.raw().get("response")+ {+ router.remember_response(+ body,+ &model,+ store,+ conversation.as_deref(),+ )?;+ }+ Ok(Some((event, (stream, router, model, conversation))))+ },+ )))+ }+ };+ Ok(response)+ }++ /// Check both IDs from a response or stream event before recording either. The response's+ /// `store` value overrides the request value; `false` skips only the response ID.+ fn remember_response(+ &self,+ body: &Value,+ model: &ModelId,+ store: bool,+ conversation: Option<&str>,+ ) -> std::result::Result<(), LlmClientError> {+ if let Some(owners) = &self.inner.state_owners {+ let mut owners = owners.lock();+ let response_id = body+ .get("id")+ .and_then(Value::as_str)+ .filter(|_| body.get("store").and_then(Value::as_bool).unwrap_or(store));+ let conversation = body.as_object().and_then(conversation_id).or(conversation);+ if let Err(error) = owners.remember(response_id, conversation, model) {+ tracing::error!(error = %error, "could not record Responses state owner");+ return Err(error);+ }+ }+ Ok(())+ }+/// Prepare a completion candidate with its configured target prompt.fn prepare_completion_request(&self, mut request: Request, target: &ModelId) -> Request {let prompt = self.inner.target_prompts.get(target).map(String::as_str);@@ -693,6 +918,204 @@ mod tests {));}+ #[tokio::test]+ async fn responses_state_tracking_uses_upstream_storage_and_format() -> Result<()> {+ for (responses, store, conversation) in [+ (true, true, None),+ (true, false, None),+ (true, false, Some(json!("conv_plain"))),+ (true, false, Some(json!({"id": "conv_object"}))),+ (false, true, None),+ ] {+ for stream in [false, true] {+ let server = MockServer::start().await;+ Mock::given(method("POST"))+ .respond_with(move |request: &wiremock::Request| {+ let body: Value = serde_json::from_slice(&request.body).expect("request JSON");+ let response = if responses {+ json!({+ "id": "resp_seed", "object": "response", "model": "weak",+ "status": "completed", "output": [], "store": body["store"],+ "conversation": body["conversation"],+ })+ } else if stream {+ json!({"id": "resp_seed", "model": "weak", "choices": [{"index": 0, "delta": {"content": "ok"}, "finish_reason": "stop"}]})+ } else {+ json!({"id": "resp_seed", "model": "weak", "choices": [{"index": 0, "message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}]})+ };+ if stream {+ let event = if responses { json!({"type": "response.completed", "response": response}) } else { response };+ ResponseTemplate::new(200)+ .insert_header("content-type", "text/event-stream")+ .set_body_string(if responses {+ format!("event: response.completed\ndata: {event}\n\n")+ } else {+ format!("data: {event}\n\ndata: [DONE]\n\n")+ })+ } else {+ ResponseTemplate::new(200).set_body_json(response)+ }+ })+ .mount(&server).await;+ let mut clients = HashMap::new();+ for model in ["weak", "strong"] {+ let config = HttpBackendConfig {+ base_url: server.uri(),+ api_key: None,+ forward_auth: false,+ extra_headers: BTreeMap::new(),+ extra_body: BTreeMap::from([("store".to_string(), json!(store))]),+ reasoning_effort: None,+ max_retries: 0,+ };+ let backend = if responses {+ Backend::OpenAiResponses(config)+ } else {+ Backend::OpenAiChat(config)+ };+ let client: Arc<dyn RoutedLlmClient> = Arc::new(+ TranslatingLlmClient::new(&[ModelConfig::new(model, backend, None)])+ .map_err(|error| LibsyError::external("building test client", error))?,+ );+ clients.insert(ModelId::from(model), client);+ }+ let clients = ClientRouter::new(clients);+ let mut seed = request();+ seed.llm_request.stream = stream;+ if let Some(conversation) = &conversation {+ seed.llm_request+ .extensions+ .fields+ .insert("conversation".to_string(), conversation.clone());+ }+ let (_, response) = run(+ Arc::new(switchyard_libsy::Passthrough),+ clients.clone(),+ seed,+ to_category_map(&["weak"]),+ None,+ )+ .await?;+ response+ .llm_response+ .into_agg()+ .await+ .map_err(|error| LibsyError::client_call("weak", error))?;+ let mut continuations = vec![(+ "previous_response_id",+ json!("resp_seed"),+ responses && store,+ )];+ if let Some(conversation) = conversation.clone() {+ continuations.push(("conversation", conversation, true));+ }+ for (field, id, pinned) in continuations {+ let mut follow = request();+ follow+ .llm_request+ .extensions+ .fields+ .insert(field.to_string(), id);+ let outcome = decide(+ Arc::new(switchyard_libsy::Passthrough),+ clients.clone(),+ follow,+ to_category_map(&["strong"]),+ )+ .await?;+ assert_eq!(+ outcome.selected_model_id()?.as_str(),+ if pinned { "weak" } else { "strong" },+ "responses={responses}, store={store}, stream={stream}, field={field}"+ );+ }+ let requests = server.received_requests().await.expect("request recording");+ assert_eq!(requests.len(), 1);+ assert_eq!(+ serde_json::from_slice::<Value>(&requests[0].body).expect("request JSON")["store"],+ store+ );+ }+ }+ Ok(())+ }++ #[test]+ fn state_owners_preserve_existing_records_at_capacity()+ -> std::result::Result<(), LlmClientError> {+ let mut owners = StateOwners::default();+ let model = ModelId::from("model/a");+ for id in 0..MAX_STATE_OWNERS - 1 {+ owners.remember(Some(&format!("resp_{id}")), None, &model)?;+ }+ assert!(matches!(+ owners.remember(Some("new_response"), Some("new_conversation"), &model),+ Err(LlmClientError::ResponseStateLimitExceeded {+ limit: MAX_STATE_OWNERS+ })+ ));+ assert!(owners.owner("new_response").is_none());+ assert!(owners.owner("new_conversation").is_none());+ owners.remember(Some("last_id"), Some("last_id"), &model)?;+ owners.remember(Some("last_id"), Some("resp_0"), &model)?;+ assert!(matches!(+ owners.remember(Some("over_limit"), None, &model),+ Err(LlmClientError::ResponseStateLimitExceeded { .. })+ ));+ assert!(matches!(+ owners.remember(Some("resp_0"), None, &ModelId::from("model/b")),+ Err(LlmClientError::ResponseStateConflict)+ ));+ assert_eq!(owners.owner("resp_0"), Some(&model));+ assert_eq!(owners.by_id.len(), MAX_STATE_OWNERS);+ Ok(())+ }++ #[test]+ fn concurrent_state_owners_do_not_exceed_capacity() {+ let model = ModelId::from("model/a");+ let owners = Arc::new(Mutex::new(StateOwners {+ by_id: (0..MAX_STATE_OWNERS - 16)+ .map(|id| (format!("existing_{id}"), model.clone()))+ .collect(),+ }));+ let barrier = Arc::new(std::sync::Barrier::new(32));+ let accepted = std::thread::scope(|scope| {+ let handles = (0..32)+ .map(|id| {+ let owners = owners.clone();+ let barrier = barrier.clone();+ let model = model.clone();+ scope.spawn(move || {+ barrier.wait();+ let result = owners.lock().remember(+ Some(&format!("resp_{id}")),+ Some(&format!("conv_{id}")),+ &model,+ );+ match result {+ Ok(()) => Some(id),+ Err(LlmClientError::ResponseStateLimitExceeded { .. }) => None,+ other => panic!("unexpected state tracking result: {other:?}"),+ }+ })+ })+ .collect::<Vec<_>>();+ handles+ .into_iter()+ .filter_map(|handle| handle.join().expect("request thread panicked"))+ .collect::<Vec<_>>()+ });+ let owners = owners.lock();+ assert_eq!(accepted.len(), 8);+ assert_eq!(owners.by_id.len(), MAX_STATE_OWNERS);+ assert_eq!(owners.owner("existing_0"), Some(&model));+ for id in accepted {+ assert_eq!(owners.owner(&format!("resp_{id}")), Some(&model));+ assert_eq!(owners.owner(&format!("conv_{id}")), Some(&model));+ }+ }+#[tokio::test]async fn each_fallback_candidate_receives_only_its_own_prompt() -> Result<()> {let client = Arc::new(CandidateClient {diff --git a/crates/libsy/src/algorithms/advisor_gate.rs b/crates/libsy/src/algorithms/advisor_gate.rsindex 4afd732a3..d638864f8 100644--- a/crates/libsy/src/algorithms/advisor_gate.rs+++ b/crates/libsy/src/algorithms/advisor_gate.rs@@ -51,6 +51,7 @@ mod transcript;mod trigger;mod turn;+use super::util::buffered_response::{BufferedResponse, buffer_response};use budget::{ReviewBudget, ScopeKey, budget_scope, stall_key};use signals::{GateSignalProcessor, GateSignals};use telemetry::{@@ -61,7 +62,7 @@ use transcript::{VERDICT_PATTERN, Verdict, advisor_reply_text, parse_verdict, reuse trigger::TriggerClassifier;#[cfg(test)]use turn::has_tool_use;-use turn::{GatedTurn, buffer_turn, reasoning_text, visible_text};+use turn::{reasoning_text, visible_text};/// APPROVE/REDO reviewer contract sent as the advisor's system prompt.pub const REVIEWER_SYSTEM_PROMPT: &str =@@ -238,7 +239,7 @@ impl AdvisorGate {.served_model().cloned().unwrap_or_else(|| executor.clone());- let turn = buffer_turn(served_executor.as_str(), response).await?;+ let turn = buffer_response(served_executor.as_str(), response).await?;// Response-side signals fold in after it: the terminal turn never// appears on a later request, so the trigger runs on this event.@@ -346,7 +347,7 @@ impl AdvisorGate {executor_fallbacks: &[ModelId],served_executor: &ModelId,request: Request,- turn: GatedTurn,+ turn: BufferedResponse,plan: &str,) -> RoutingOutcome {record_discarded(&turn.agg.usage);diff --git a/crates/libsy/src/algorithms/advisor_gate/turn.rs b/crates/libsy/src/algorithms/advisor_gate/turn.rsindex 62964caea..48f41545e 100644--- a/crates/libsy/src/algorithms/advisor_gate/turn.rs+++ b/crates/libsy/src/algorithms/advisor_gate/turn.rs@@ -1,104 +1,11 @@// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.// SPDX-License-Identifier: Apache-2.0-//! One buffered executor turn: consuming it to completion, inspecting it,-//! and replaying it to the client verbatim.+//! Extract reply text and tool-use signals for advisor review.-use futures::StreamExt;-use switchyard_protocol::{- AggLlmResponse, ContentBlock, LlmClientError, LlmResponse, LlmResponseChunk,- LlmResponseStreamEvent, Metadata, Response, ResponseAccumulator, StopReason,-};+use switchyard_protocol::{AggLlmResponse, ContentBlock, StopReason};-use crate::{LibsyError, Result};--// ── Turn buffering and replay ───────────────────────────────────────────────--/// One fully generated executor turn held while the gate decides.-pub(super) struct GatedTurn {- /// Buffered provider events for streamed turns, preservation included, so- /// replay re-emits them verbatim (signed thinking and provider extensions- /// survive; folding to an aggregate and re-synthesizing would drop them).- pub(super) events: Option<Vec<LlmResponseStreamEvent>>,- /// Folded view for detection, the review tail, the REDO echo, and- /// discarded-turn usage. For buffered turns this is the original- /// response, its own preservation intact.- pub(super) agg: AggLlmResponse,- pub(super) metadata: Option<Metadata>,- pub(super) upstream_headers: http::HeaderMap,-}--impl GatedTurn {- /// Releases the turn to the client: streamed turns replay their buffered- /// events verbatim, buffered turns return the original aggregate.- pub(super) fn into_response(self) -> Response {- let llm_response = match self.events {- Some(events) => {- LlmResponse::Stream(Box::pin(futures::stream::iter(events.into_iter().map(Ok))))- }- None => LlmResponse::Agg(self.agg),- };- Response {- llm_response,- metadata: self.metadata,- upstream_headers: self.upstream_headers,- }- }-}--/// Consumes the executor response to completion. Mid-stream failures — item-/// errors and in-band error chunks — become typed client-call errors exactly-/// as [`LlmResponse::into_agg`] maps them; the client saw nothing yet, so the-/// turn fails whole.-pub(super) async fn buffer_turn(executor: &str, response: Response) -> Result<GatedTurn> {- let metadata = response.metadata;- let upstream_headers = response.upstream_headers;- match response.llm_response {- LlmResponse::Agg(agg) => Ok(GatedTurn {- events: None,- agg,- metadata,- upstream_headers,- }),- LlmResponse::Stream(mut stream) => {- let mut events = Vec::new();- let mut accumulator = ResponseAccumulator::new();- while let Some(item) = stream.next().await {- let event =- item.map_err(|source| LibsyError::client_call(executor.to_string(), source))?;- for chunk in event.normalized() {- let failure = match chunk {- LlmResponseChunk::DecodeError { message } => {- Some(LlmClientError::ResponseTranslation(message.clone()))- }- LlmResponseChunk::StreamError { message } => {- Some(LlmClientError::UpstreamHttp {- status: http::StatusCode::BAD_GATEWAY,- body: message.clone(),- })- }- chunk => {- accumulator.push(chunk.clone());- None- }- };- if let Some(source) = failure {- return Err(LibsyError::client_call(executor.to_string(), source));- }- }- events.push(event);- }- Ok(GatedTurn {- events: Some(events),- agg: accumulator.finish(),- metadata,- upstream_headers,- })- }- }-}--// ── Detection over the folded turn ──────────────────────────────────────────+// ── Response inspection ────────────────────────────────────────────────────/// Whether the turn carries tool use on either signal: a `ToolUse` stop/// reason, or any tool-call block (some OSS servers mislabel tool-call turnsdiff --git a/crates/libsy/src/algorithms/escalation.rs b/crates/libsy/src/algorithms/escalation.rsindex 94bc38e87..dfe2c7fe3 100644--- a/crates/libsy/src/algorithms/escalation.rs+++ b/crates/libsy/src/algorithms/escalation.rs@@ -7,9 +7,10 @@ use std::sync::Arc;use async_trait::async_trait;use switchyard_protocol::{- AggLlmResponse, Category, LlmClientError, LlmResponse, Message, Request, Response, Role,+ AggLlmResponse, Category, LlmClientError, Message, Request, Response, Role,};+use super::util::buffered_response::buffer_response;use super::util::classifier_contract::ClassifierContractConfig;use super::util::decisive;use super::util::escalation::{self, EscalationJudge, EscalationJudgeConfig, EscalationPolicy};@@ -109,41 +110,37 @@ impl Classifier<State> for EscalationClassifier {};// The call resolves when its stream handle arrives; context and transport failures can// still occur while buffering.- let agg = match efficient_response.llm_response.into_agg().await {- Ok(agg) => agg,- Err(LlmClientError::ContextWindowExceeded { .. }) => {+ let efficient_response = match buffer_response(efficient.as_str(), efficient_response).await+ {+ Ok(response) => response,+ Err(LibsyError::ClientCall {+ source: LlmClientError::ContextWindowExceeded { .. },+ ..+ }) => {driver.set_evidence(serde_json::json!({"source": "fallback","reason_code": "context_window",}));return Ok((decisive(&capable), None));}- Err(LlmClientError::Transport { .. }) => {+ Err(LibsyError::ClientCall {+ source: LlmClientError::Transport { .. },+ ..+ }) => {driver.set_evidence(serde_json::json!({"source": "fallback","reason_code": "transport",}));return Ok((decisive(&capable), None));}- Err(source) => {- return Err(LibsyError::client_call(efficient.clone(), source));- }+ Err(error) => return Err(error),};// Append the efficient reply so the judge reads this turn's completed trajectory.let mut judge_request = request.clone();judge_request.llm_request.messages- .push(assistant_message(&agg));- let efficient_response = Response {- llm_response: if request.llm_request.stream {- LlmResponse::Stream(agg.into_stream())- } else {- LlmResponse::Agg(agg)- },- metadata: efficient_response.metadata,- upstream_headers: efficient_response.upstream_headers,- };+ .push(assistant_message(&efficient_response.agg));let (classification, _) = self.judge.score(state, &mut judge_request, driver).await?;@@ -174,7 +171,10 @@ impl Classifier<State> for EscalationClassifier {}));}- Ok((decisive(&efficient), Some(efficient_response)))+ Ok((+ decisive(&efficient),+ Some(efficient_response.into_response()),+ ))}}diff --git a/crates/libsy/src/algorithms/util.rs b/crates/libsy/src/algorithms/util.rsindex 9a290920f..0b96ca92a 100644--- a/crates/libsy/src/algorithms/util.rs+++ b/crates/libsy/src/algorithms/util.rs@@ -2,6 +2,7 @@// SPDX-License-Identifier: Apache-2.0pub mod affinity;+pub(crate) mod buffered_response;pub(crate) mod classifier_contract;pub mod escalation;pub(crate) mod llm_judge;diff --git a/crates/libsy/src/algorithms/util/buffered_response.rs b/crates/libsy/src/algorithms/util/buffered_response.rsnew file mode 100644index 000000000..706a117d6--- /dev/null+++ b/crates/libsy/src/algorithms/util/buffered_response.rs@@ -0,0 +1,94 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++//! Buffer a model response for inspection and replay its original stream events.++use futures::StreamExt;+use switchyard_protocol::{+ AggLlmResponse, LlmClientError, LlmResponse, LlmResponseChunk, LlmResponseStreamEvent,+ Metadata, Response, ResponseAccumulator,+};++use crate::{LibsyError, Result};++/// A completed model response held while an algorithm inspects it.+pub(crate) struct BufferedResponse {+ /// Original stream events, including response IDs, signed thinking, and provider+ /// extensions. Replaying these preserves data lost when synthesizing events from `agg`.+ events: Option<Vec<LlmResponseStreamEvent>>,+ /// Complete response for inspection and usage accounting. For a non-streaming reply,+ /// this is the original response with its preserved provider data.+ pub(crate) agg: AggLlmResponse,+ metadata: Option<Metadata>,+ upstream_headers: http::HeaderMap,+}++impl BufferedResponse {+ /// Replay the buffered stream events or return the original non-streaming response.+ pub(crate) fn into_response(self) -> Response {+ let llm_response = match self.events {+ Some(events) => {+ LlmResponse::Stream(Box::pin(futures::stream::iter(events.into_iter().map(Ok))))+ }+ None => LlmResponse::Agg(self.agg),+ };+ Response {+ llm_response,+ metadata: self.metadata,+ upstream_headers: self.upstream_headers,+ }+ }+}++/// Consume the response to completion before replaying any events. Stream item errors+/// and error chunks become typed client-call errors, as in [`LlmResponse::into_agg`].+pub(crate) async fn buffer_response(+ executor: &str,+ response: Response,+) -> Result<BufferedResponse> {+ let metadata = response.metadata;+ let upstream_headers = response.upstream_headers;+ match response.llm_response {+ LlmResponse::Agg(agg) => Ok(BufferedResponse {+ events: None,+ agg,+ metadata,+ upstream_headers,+ }),+ LlmResponse::Stream(mut stream) => {+ let mut events = Vec::new();+ let mut accumulator = ResponseAccumulator::new();+ while let Some(item) = stream.next().await {+ let event =+ item.map_err(|source| LibsyError::client_call(executor.to_string(), source))?;+ for chunk in event.normalized() {+ let failure = match chunk {+ LlmResponseChunk::DecodeError { message } => {+ Some(LlmClientError::ResponseTranslation(message.clone()))+ }+ LlmResponseChunk::StreamError { message } => {+ Some(LlmClientError::UpstreamHttp {+ status: http::StatusCode::BAD_GATEWAY,+ body: message.clone(),+ })+ }+ chunk => {+ accumulator.push(chunk.clone());+ None+ }+ };+ if let Some(source) = failure {+ return Err(LibsyError::client_call(executor.to_string(), source));+ }+ }+ events.push(event);+ }+ Ok(BufferedResponse {+ events: Some(events),+ agg: accumulator.finish(),+ metadata,+ upstream_headers,+ })+ }+ }+}diff --git a/crates/protocol/src/client.rs b/crates/protocol/src/client.rsindex 6b24afcc6..f8626d2bd 100644--- a/crates/protocol/src/client.rs+++ b/crates/protocol/src/client.rs@@ -50,6 +50,19 @@ pub enum LlmClientError {message: String,},+ /// The route cannot record another provider-owned response or conversation ID.+ #[error(+ "Responses state tracking reached its limit of {limit} IDs; no existing records were removed"+ )]+ ResponseStateLimitExceeded {+ /// Maximum number of IDs retained by this route.+ limit: usize,+ },++ /// Two configured models reported the same saved response or conversation ID.+ #[error("Responses state ID is already recorded for another model; its owner was not changed")]+ ResponseStateConflict,+/// The upstream could not be reached or the request could not be sent.#[error("upstream transport error: {source}")]Transport {diff --git a/crates/switchyard-runner/src/config.rs b/crates/switchyard-runner/src/config.rsindex d65c585dc..2d258474f 100644--- a/crates/switchyard-runner/src/config.rs+++ b/crates/switchyard-runner/src/config.rs@@ -378,8 +378,17 @@ impl DeploymentConfig {prompts,routing_answer_target,} = self.build_route_target_prompts(route_name, route)?;- let router =- ClientRouter::new_with_target_prompts(by_model, prompts, routing_answer_target);+ let completion_targets = route+ .routing_target_names()+ .into_iter()+ .map(|name| self.targets[name].id.clone())+ .collect::<Vec<_>>();+ let router = ClientRouter::new_with_completion_targets(+ by_model,+ prompts,+ routing_answer_target,+ &completion_targets,+ );Ok((router, caller_auth))}diff --git a/crates/switchyard-server/src/lib.rs b/crates/switchyard-server/src/lib.rsindex 7055b637e..56d222503 100644--- a/crates/switchyard-server/src/lib.rs+++ b/crates/switchyard-server/src/lib.rs@@ -1045,7 +1045,18 @@ async fn handle_llm_request(let output = match route.execute(request, Some(observer)).await {Ok(output) => output,- Err(error) => return runner_error(error),+ Err(error) => {+ if let RunnerError::Algorithm(LibsyError::ClientCall {+ target,+ source:+ LlmClientError::ResponseStateLimitExceeded { .. }+ | LlmClientError::ResponseStateConflict,+ }) = &error+ {+ state.stats.record_response_error(target);+ }+ return runner_error(error);+ }};let RunOutput {selected_model,@@ -1250,6 +1261,18 @@ fn runner_error(error: RunnerError) -> Response {fn client_error(error: &LlmClientError) -> Response {match error {+ LlmClientError::ResponseStateLimitExceeded { .. } => error_response(+ StatusCode::SERVICE_UNAVAILABLE,+ error.to_string(),+ "server_error",+ "response_state_limit_exceeded",+ ),+ LlmClientError::ResponseStateConflict => error_response(+ StatusCode::CONFLICT,+ error.to_string(),+ "server_error",+ "response_state_conflict",+ ),LlmClientError::InvalidRequest { message }| LlmClientError::RequestTranslation(message) => error_response(StatusCode::BAD_REQUEST,diff --git a/crates/switchyard-server/src/stats/accumulator.rs b/crates/switchyard-server/src/stats/accumulator.rsindex 281609229..357b342eb 100644--- a/crates/switchyard-server/src/stats/accumulator.rs+++ b/crates/switchyard-server/src/stats/accumulator.rs@@ -68,8 +68,9 @@ impl StatsAccumulator {stats.errors = stats.errors.saturating_add(1);}- /// Records a stream failure after its routed call was already counted.- pub(crate) fn record_stream_error(&self, model: impl Into<ModelId>) {+ /// Record a failure while processing a buffered response or stream after its routed call+ /// was already counted.+ pub(crate) fn record_response_error(&self, model: impl Into<ModelId>) {let mut inner = self.lock();inner.total_errors = inner.total_errors.saturating_add(1);let stats = inner.model_stats_mut(model.into());diff --git a/crates/switchyard-server/src/usage_metrics.rs b/crates/switchyard-server/src/usage_metrics.rsindex bb4e21ad8..158b58779 100644--- a/crates/switchyard-server/src/usage_metrics.rs+++ b/crates/switchyard-server/src/usage_metrics.rs@@ -106,7 +106,7 @@ pub(crate) fn observe(// Records a terminal stream failure after the routed call was already counted.fn record_stream_error(stats: &StatsAccumulator, model: &str) {- stats.record_stream_error(model);+ stats.record_response_error(model);global::meter("switchyard").u64_counter("switchyard.errors").build()diff --git a/crates/switchyard-server/tests/server.rs b/crates/switchyard-server/tests/server.rsindex 697558287..00295e7d9 100644--- a/crates/switchyard-server/tests/server.rs+++ b/crates/switchyard-server/tests/server.rs@@ -10,7 +10,7 @@ use std::io::Write;use std::sync::Arc;use axum::body::{Body, Bytes};-use axum::extract::{DefaultBodyLimit, State};+use axum::extract::{DefaultBodyLimit, Path, State};use axum::http::{HeaderMap, HeaderValue, Request as HttpRequest, StatusCode, Uri};use axum::response::sse::{Event, Sse};use axum::response::{IntoResponse, Response as HttpResponse};@@ -60,6 +60,7 @@ impl MockUpstream {"/v1/responses",post(upstream_responses_requires_forwarded_auth),)+ .route("/silo/{silo}/responses", post(upstream_responses_silo)).route("/capture", post(upstream_redirect_capture)).route("/v1/messages/count_tokens", post(upstream_count_tokens)).route(@@ -559,6 +560,95 @@ async fn upstream_buffered_responses(Json(response)}+/// A local Responses provider whose IDs normally contain its name. It returns `state_not_found`+/// for another provider's ID, except `resp_shared_conflict`, which tests duplicate ownership.+/// The judge selects the strong tier when the input contains `ROUTE_B`.+async fn upstream_responses_silo(+ State(calls): State<Arc<Mutex<Vec<Value>>>>,+ Path(silo): Path<String>,+ Json(body): Json<Value>,+) -> HttpResponse {+ let mut calls = calls.lock().await;+ calls.push(body.clone());+ let model = body["model"].as_str().unwrap_or_default().to_string();+ if model == "model/judge" {+ let strong = body.to_string().contains("ROUTE_B");+ let verdict = if body["text"]["format"]["schema"]["properties"]+ .get("escalate")+ .is_some()+ {+ json!({"escalate": strong, "reason": "state probe"})+ } else {+ json!({+ "crux": "state probe", "primary_rule": if strong { "LIM-1" } else { "SUP-1" },+ "capability_boundary": if strong { "unsupported" } else { "supported" },+ "p_solve": if strong { 0.1 } else { 0.9 },+ })+ };+ return Json(responses_body("resp_judge", &model, &verdict.to_string())).into_response();+ }+ let minted_prefix = format!("resp_{silo}_");+ if silo == "A" && body.to_string().contains("owner-unavailable") {+ return (+ StatusCode::SERVICE_UNAVAILABLE,+ Json(json!({"error": {"message": "provider A is unavailable"}})),+ )+ .into_response();+ }+ if let Some(previous) = body["previous_response_id"].as_str()+ && !previous.starts_with(&minted_prefix)+ && previous != "resp_shared_conflict"+ {+ return (+ StatusCode::BAD_REQUEST,+ Json(+ json!({"error": {"code": "state_not_found", "message": format!(+ "previous_response_id {previous} is not present in provider {silo}"+ )}}),+ ),+ )+ .into_response();+ }+ let id = if body.to_string().contains("state-id-conflict") {+ "resp_shared_conflict".to_string()+ } else {+ format!("{minted_prefix}{}", calls.len())+ };+ let text = format!("served-by:{silo}");+ let mut response = responses_body(&id, &model, &text);+ response["store"] = body.get("store").cloned().unwrap_or(json!(true));+ response["conversation"] = body.get("conversation").cloned().unwrap_or(Value::Null);+ if body["stream"] == true {+ let mut created = response.clone();+ created["status"] = json!("in_progress");+ created["output"] = json!([]);+ let events = [+ json!({"type": "response.created", "response": created}),+ json!({"type": "response.output_text.delta", "output_index": 0, "delta": text}),+ json!({"type": "response.completed", "response": response}),+ ];+ let stream = futures_util::stream::iter(events.into_iter().map(|event| {+ Ok::<Event, Infallible>(+ Event::default()+ .event(event["type"].as_str().unwrap_or_default())+ .data(event.to_string()),+ )+ }));+ return Sse::new(stream).into_response();+ }+ Json(response).into_response()+}++fn responses_body(id: &str, model: &str, text: &str) -> Value {+ json!({+ "id": id, "object": "response", "created_at": 1, "model": model, "status": "completed",+ "output": [{"id": "msg_1", "type": "message", "status": "completed", "role": "assistant",+ "content": [{"type": "output_text", "text": text, "annotations": []}]}],+ "usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2,+ "output_tokens_details": {"reasoning_tokens": 0}}+ })+}+async fn upstream_redirect_capture(State(calls): State<Arc<Mutex<Vec<Value>>>>,headers: HeaderMap,@@ -1505,6 +1595,214 @@ base_threshold = 0.5Ok(())}+/// Configure classifier and escalation routes over two Responses providers.+/// `route/shared` uses one answer client; the other routes must return known continuations+/// to the model that served their state, even when the judge would select another provider.+fn state_silo_config(base_url: &str) -> String {+ let root = base_url.trim_end_matches("/v1");+ format!(+ r#"+schema_version = 1+[llm_clients.judge]+format = "openai_responses"+base_url = "{root}/silo/judge"+[llm_clients.a]+format = "openai_responses"+base_url = "{root}/silo/A"+max_retries = 0+[llm_clients.b]+format = "openai_responses"+base_url = "{root}/silo/B"+[targets.judge]+id = "model/judge"+llm_client = "judge"+[targets.a]+id = "model/provider-a"+llm_client = "a"+[targets.b]+id = "model/provider-b"+llm_client = "b"+[targets.shared_b]+id = "model/shared-b"+llm_client = "a"+[routes.shared]+id = "route/shared"+type = "llm_classifier"+classifier_target = "judge"+weak_target = "a"+strong_target = "shared_b"+base_threshold = 0.5+classify_trigger = "every_request"+[routes.escalation]+id = "route/escalation"+type = "llm_classifier"+mode = "escalation"+classifier_target = "judge"+weak_target = "a"+strong_target = "b"+escalation = {{ confirmations = 1 }}+[routes.dynamic]+id = "route/dynamic"+type = "llm_classifier"+classifier_target = "judge"+weak_target = "a"+strong_target = "b"+base_threshold = 0.5+classify_trigger = "every_request"+"#+ )+}++#[tokio::test]+async fn responses_continuations_preserve_state_ownership() -> TestResult {+ for (route, stream, endpoint) in [+ ("route/dynamic", false, "/v1/responses"),+ ("route/dynamic", true, "/v1/responses"),+ ("route/escalation", false, "/v1/decision"),+ ("route/escalation", true, "/v1/decision"),+ ("route/shared", false, "/v1/responses"),+ ] {+ let upstream = MockUpstream::start().await?;+ let app =+ build_switchyard_router(load_test_config(&state_silo_config(&upstream.base_url))?);+ let mut request =+ json!({"model": route, "input": "ROUTE_A remember mango", "stream": stream});+ if endpoint == "/v1/decision" {+ request = json!({"input_format": "openai_responses", "request": request});+ }+ let seed = send(&app, "POST", endpoint, Some(request)).await?;+ assert_eq!(seed.status, StatusCode::OK, "{}", seed.text()?);+ let body = if endpoint == "/v1/decision" {+ seed.json()?["response"].clone()+ } else if stream {+ sse_events(seed.text()?)+ .into_iter()+ .find(|event| event["type"] == "response.completed")+ .ok_or("stream ended without response.completed")?["response"]+ .clone()+ } else {+ seed.json()?+ };+ assert!(+ body["id"]+ .as_str()+ .is_some_and(|id| id.starts_with("resp_A_"))+ );+ let mut follow_request = json!({"model": route, "input": "ROUTE_B recall mango", "previous_response_id": body["id"]});+ let before = upstream.models().await.len();+ let follow = send(&app, "POST", "/v1/responses", Some(follow_request.clone())).await?;+ assert_eq!(follow.status, StatusCode::OK, "{}", follow.text()?);+ let expected = if route == "route/shared" {+ vec!["model/judge", "model/shared-b"]+ } else {+ vec!["model/provider-a"]+ };+ assert_eq!(&upstream.models().await[before..], expected);+ assert_eq!(+ follow.headers["x-model-router-selected-model"],+ *expected.last().ok_or("missing target")?+ );+ if route == "route/dynamic" {+ follow_request["previous_response_id"] = follow.json()?["id"].clone();+ let chained = send(&app, "POST", "/v1/responses", Some(follow_request.clone())).await?;+ assert_eq!(chained.status, StatusCode::OK);+ assert_eq!(+ chained.headers["x-model-router-selected-model"],+ "model/provider-a"+ );+ let before = upstream.models().await.len();+ let fresh = send(+ &app,+ "POST",+ "/v1/responses",+ Some(json!({"model": route, "input": "ROUTE_B a new task"})),+ )+ .await?;+ assert_eq!(fresh.status, StatusCode::OK);+ assert_eq!(+ fresh.headers["x-model-router-selected-model"],+ "model/provider-b"+ );+ assert_eq!(+ &upstream.models().await[before..],+ ["model/judge", "model/provider-b"]+ );+ follow_request["input"] = json!("ROUTE_B owner-unavailable");+ let before = upstream.models().await.len();+ let unavailable = send(&app, "POST", "/v1/responses", Some(follow_request)).await?;+ assert_eq!(unavailable.status, StatusCode::SERVICE_UNAVAILABLE);+ assert_eq!(&upstream.models().await[before..], ["model/provider-a"]);+ }+ }+ for (route, stream, endpoint) in [+ ("route/dynamic", false, "/v1/responses"),+ ("route/dynamic", true, "/v1/responses"),+ ("route/escalation", false, "/v1/decision"),+ ("route/escalation", true, "/v1/decision"),+ ] {+ let upstream = MockUpstream::start().await?;+ let app =+ build_switchyard_router(load_test_config(&state_silo_config(&upstream.base_url))?);+ let decision = endpoint == "/v1/decision";+ let input = if decision {+ "ROUTE_B state-id-conflict"+ } else {+ "ROUTE_A state-id-conflict"+ };+ let seed = send(+ &app,+ "POST",+ "/v1/responses",+ Some(json!({"model": route, "input": input})),+ )+ .await?;+ assert_eq!(seed.status, StatusCode::OK);+ let owner = seed.headers["x-model-router-selected-model"].clone();+ let input = if decision {+ "ROUTE_A state-id-conflict"+ } else {+ "ROUTE_B state-id-conflict"+ };+ let mut request = json!({"model": route, "input": input, "stream": stream});+ if decision {+ request = json!({"input_format": "openai_responses", "request": request});+ }+ let before = upstream.models().await.len();+ let rejected = send(&app, "POST", endpoint, Some(request)).await?;+ if stream && !decision {+ assert_eq!(rejected.status, StatusCode::OK);+ let events = sse_events(rejected.text()?);+ assert_eq!(events.len(), 1);+ assert_eq!(events[0]["type"], "error");+ } else {+ assert_eq!(+ rejected.status,+ StatusCode::CONFLICT,+ "{}",+ rejected.text()?+ );+ assert_eq!(rejected.json()?["error"]["code"], "response_state_conflict");+ }+ assert_eq!(+ &upstream.models().await[before..],+ if decision {+ ["model/provider-a", "model/judge"]+ } else {+ ["model/judge", "model/provider-b"]+ }+ );+ if !decision {+ let stats = send(&app, "GET", "/v1/stats", None).await?.json()?;+ assert_eq!(stats["total_requests"], 2);+ assert_eq!(stats["total_errors"], 1);+ }+ let follow = send(&app, "POST", "/v1/responses", Some(json!({"model": route, "input": "ROUTE_B recall", "previous_response_id": seed.json()?["id"], "store": false}))).await?;+ assert_eq!(follow.status, StatusCode::OK);+ assert_eq!(follow.headers["x-model-router-selected-model"], owner);+ }+ Ok(())+}+/// Decision-only routing returns callable metadata and preserves any answer produced while routing.#[tokio::test]async fn decision_returns_callable_target_and_routing_answer() -> TestResult {diff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.mdindex cbc893563..329ccadc0 100644--- a/docs/routing_algorithms/llm_classifier_routing.md+++ b/docs/routing_algorithms/llm_classifier_routing.md@@ -250,6 +250,51 @@ identity are judged every time. Clients can send `x-switchyard-session-id`, orenable `message_hash_fallback` to key on the first user-message text under`new_session`.+### Responses continuations by ID++A Responses API client can continue without resending the conversation history:+`previous_response_id` refers to a stored response, and `conversation` refers to+a provider's conversation. When answer targets use different `[llm_clients]`+entries, Switchyard records the model that served each stored Responses ID and+conversation ID. A request with a recorded ID returns to that exact model+without a judge call or fallback to another provider. This applies to every+`classify_trigger` and to `stage_router` and `composite` routes. Classifier-only+clients do not count; when answer targets share one client, routing proceeds as+usual.++Switchyard tracks buffered and streamed Responses replies, including answers+returned by `/v1/decision`. It does not record Chat Completions or Anthropic+Messages response IDs as Responses continuation IDs. A provider's `store` value+takes precedence over the request value. If that value is `false`, Switchyard+skips the response ID but still records the conversation ID.++Each route keeps up to 65,536 distinct ID-to-model records per process. This+limit counts response and conversation IDs recorded over the process's lifetime,+not tokens or simultaneous requests. Records do not expire, and Switchyard does+not remove older records to make room. The map stores copies of IDs and model+names, with no transcript storage.++If recording new IDs would exceed the limit, a buffered reply returns HTTP 503+with code `response_state_limit_exceeded`. If an ID is already recorded for a+different model, it returns HTTP 409 with code `response_state_conflict`. Neither+failure changes existing records. Switchyard checks both IDs before adding+either from a response or stream event.++These checks run after the provider has returned a response or stream event, so+the error does not mean the provider did no work. A stream that has already sent+HTTP headers emits a framed error and stops; it cannot change the HTTP status.+Switchyard logs these errors. For `/v1/responses`, buffered and streamed failures+increment `/v1/stats` `total_errors`. Other routes continue independently.++An existing ID can still be used at capacity if the reply adds no new IDs. For+example, `store: false` can suppress a new response ID, but it does not suppress+conversation-ID tracking. A stored follow-up normally creates a new response ID+and therefore fails when the map is full.++A restart removes all records, and each replica has its own map. Unknown IDs use+normal routing and can still fail at the selected provider. To keep dynamic+routing across turns, send the full history instead of an ID.+## Run the routeAfter [installing the Rust server](../getting_started.md#install-the-server), export