Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
feat(libsy): report served answers for VGR recovery (#1166)
GDRSe1ed2b5ea3cf
diff --git a/crates/libsy-llm-client/README.md b/crates/libsy-llm-client/README.mdindex c1734a157..3cccb5d41 100644--- a/crates/libsy-llm-client/README.md+++ b/crates/libsy-llm-client/README.md@@ -269,6 +269,14 @@ fn build_multi_format_client(transport failures, timeouts, HTTP 408/429, and 5xx responses. Buffered bodytransport failures are retried; streaming body failures are not replayed afterthe response has been returned.+- If an OpenAI Chat backend explicitly rejects `chat_template_kwargs =+ {"enable_thinking": false}` with HTTP 400 or 422, the client retries once without+ that field. This compatibility retry is separate from `max_retries` and shares+ the original timeout. `omit_body_fields` can remove it from the first attempt.+- Codex endpoints whose URL path ends in `/backend-api/codex/responses` receive+ string `input` as a message-item list, including preserved requests. Other+ fields, array input, non-Codex endpoints, and the translation codec's exact+ preservation remain unchanged.- `ModelConfig::with_responses_reasoning` controls Responses reasoning replay.Every Responses model defaults to `PreserveEncrypted`: plaintext is removed,encrypted provider state is retained. Stored reasoning IDs are retained unlessdiff --git a/crates/libsy-llm-client/src/backend.rs b/crates/libsy-llm-client/src/backend.rsindex 461fca4ed..b24c1a0d7 100644--- a/crates/libsy-llm-client/src/backend.rs+++ b/crates/libsy-llm-client/src/backend.rs@@ -79,7 +79,8 @@ pub struct HttpBackendConfig {/// sent. Responses carries it as `reasoning.effort`, Chat Completions as `reasoning_effort`;/// Anthropic has no equivalent and rejects the setting at configuration time.pub reasoning_effort: Option<String>,- /// Additional attempts after the initial upstream request.+ /// Additional attempts for transient upstream failures.+ /// One template-compatibility retry may also run within the same timeout.pub max_retries: u32,/// Cooldown after an exhausted transient completion failure. Zero disables it.pub failure_cooldown: Duration,diff --git a/crates/libsy-llm-client/src/client.rs b/crates/libsy-llm-client/src/client.rsindex 132bf7a07..ef56b2706 100644--- a/crates/libsy-llm-client/src/client.rs+++ b/crates/libsy-llm-client/src/client.rs@@ -4,6 +4,7 @@//! [`TranslatingLlmClient`]: encode a neutral//! request, call the configured backend over HTTP, decode the neutral response.+use std::borrow::Cow;use std::collections::{BTreeMap, BTreeSet, HashMap};use std::future::ready;use std::sync::atomic::{AtomicU64, Ordering};@@ -362,6 +363,9 @@ impl TranslatingLlmClient {// After the merge on purpose: the effort override must win over both the caller's// value and any `reasoning` default a target set through `extra_body`.apply_reasoning_effort(&mut body, backend);+ if matches!(backend, Backend::OpenAiResponses(_)) {+ normalize_codex_string_input(&backend.url(), &mut body);+ }if matches!(backend, Backend::Anthropic(_)) {if nvidia_inference_api_backend(backend) {strip_anthropic_tool_strict(&mut body);@@ -404,7 +408,14 @@ impl TranslatingLlmClient {streaming: bool,) -> Result<EncodedResponse> {let max_retries = u64::from(backend.max_retries());- let max_attempts = max_retries + 1;+ let max_attempts = max_retries+ + 1+ + u64::from(+ matches!(backend, Backend::OpenAiChat(_))+ && body.get("chat_template_kwargs").is_some(),+ );+ let mut body = Cow::Borrowed(body);+ let mut template_retry = false;let expires_at = backend.timeout().map(|timeout| (tokio::time::sleep(timeout).deadline(), timeout));@@ -430,9 +441,9 @@ impl TranslatingLlmClient {"libsy.upstream_attempt",model = %model,wire_format = %backend.wire_format(),- attempt = attempt + 1,+ attempt = attempt + 1 + u64::from(template_retry),max_attempts,- retry = attempt > 0,+ retry = attempt > 0 || template_retry,openinference.span.kind = "CHAIN",outcome = tracing::field::Empty,status_code = tracing::field::Empty,@@ -452,7 +463,7 @@ impl TranslatingLlmClient {}result = async {attempt_started = true;- self.send_once(url, backend, body, metadata, model, streaming).await+ self.send_once(url, backend, body.as_ref(), metadata, model, streaming).await}.instrument(span.clone()) => result,};// The retained handle updates this same attempt span with its outcome.@@ -461,7 +472,7 @@ impl TranslatingLlmClient {span.record("outcome", "ok");span.record("status_code", response.status());span.record("will_retry", false);- if attempt > 0 {+ if attempt > 0 || template_retry {metrics::record_retry_recovered();}return Ok(match response {@@ -478,7 +489,12 @@ impl TranslatingLlmClient {});}Err(failure) => {- let will_retry = attempt < max_retries && failure.is_retryable();+ let retry_without_template = matches!(backend, Backend::OpenAiChat(_))+ && body.get("chat_template_kwargs")+ == Some(&json!({"enable_thinking": false}))+ && rejects_template_options(&failure.error);+ let will_retry =+ retry_without_template || (attempt < max_retries && failure.is_retryable());span.record("outcome", "error");if let Some(status) = failure.status {span.record("status_code", status.as_u16());@@ -488,6 +504,14 @@ impl TranslatingLlmClient {return Err(failure.error);}+ if retry_without_template {+ if let Some(object) = body.to_mut().as_object_mut() {+ object.remove("chat_template_kwargs");+ }+ template_retry = true;+ span.record("retry_delay_ms", 0_u64);+ continue;+ }let delay = retry_delay(attempt, failure.retry_after);span.record("retry_delay_ms", duration_millis(delay));// Close the attempt span before sleeping so backoff is not attempt latency.@@ -1048,6 +1072,60 @@ fn set_json_model(body: &mut Value, model: &str) {}}+// The Codex Responses endpoint requires item-list input even on the preserved-body path.+fn normalize_codex_string_input(url: &str, body: &mut Value) {+ let codex = reqwest::Url::parse(url).ok().is_some_and(|url| {+ url.path()+ .trim_end_matches('/')+ .ends_with("/backend-api/codex/responses")+ });+ if codex && let Some(input @ Value::String(_)) = body.get_mut("input") {+ let text = input.take();+ *input = json!([{"type": "message", "role": "user", "content": text}]);+ }+}++fn rejects_template_options(error: &LlmClientError) -> bool {+ let LlmClientError::UpstreamHttp { status, body } = error else {+ return false;+ };+ if !matches!(+ *status,+ StatusCode::BAD_REQUEST | StatusCode::UNPROCESSABLE_ENTITY+ ) {+ return false;+ }+ let parsed = serde_json::from_str::<Value>(body).ok();+ let error = parsed.as_ref().and_then(|value| value.get("error"));+ let message = match &parsed {+ Some(_) => error+ .and_then(|error| error.get("message"))+ .and_then(Value::as_str)+ .unwrap_or_default(),+ None => body,+ }+ .to_ascii_lowercase();+ let field = error+ .and_then(|error| error.get("param"))+ .and_then(Value::as_str);+ let code = error+ .and_then(|error| error.get("code"))+ .and_then(Value::as_str);+ (field == Some("chat_template_kwargs") || message.contains("chat_template_kwargs"))+ && (matches!(code, Some("unknown_parameter" | "unsupported_parameter"))+ || [+ "unrecognized request argument",+ "unknown parameter",+ "unsupported parameter",+ "unexpected keyword argument",+ "unknown field",+ "unsupported field",+ "extra inputs are not permitted",+ ]+ .iter()+ .any(|phrase| message.contains(phrase)))+}+const CODEX_NAMESPACE_SEPARATOR: &str = "__";/// Rewrites a Responses body into the shape OpenAI-compatible providers accept.@@ -2237,6 +2315,165 @@ mod tests {Ok(())}+ #[tokio::test]+ async fn template_rejection_retries_once_without_changing_other_fields()+ -> std::result::Result<(), Box<dyn Error + Sync + Send>> {+ for (message, reject_retry, omit) in [+ (+ "Unrecognized request argument supplied: chat_template_kwargs",+ false,+ false,+ ),+ (+ "Unrecognized request argument supplied: chat_template_kwargs",+ true,+ false,+ ),+ (+ "Invalid temperature for chat_template_kwargs request",+ false,+ false,+ ),+ (+ "Unrecognized request argument supplied: chat_template_kwargs",+ false,+ true,+ ),+ ] {+ let server = MockServer::start().await;+ Mock::given(method("POST"))+ .respond_with(move |request: &wiremock::Request| {+ let body: Value = serde_json::from_slice(&request.body).expect("request JSON");+ if body.get("chat_template_kwargs").is_some() || reject_retry {+ ResponseTemplate::new(400).set_body_json(json!({"error": {"message": message}}))+ } else {+ ResponseTemplate::new(200).set_body_json(json!({+ "id": "1", "model": "gpt", "choices": [{"index": 0,+ "message": {"role": "assistant", "content": "yes"}, "finish_reason": "stop"}],+ "usage": {}+ }))+ }+ }).mount(&server).await;+ let fields = if omit {+ BTreeSet::from(["chat_template_kwargs".into()])+ } else {+ BTreeSet::new()+ };+ let client = TranslatingLlmClient::new(&chat_map_with_omit_body_fields(+ &format!("{}/v1", server.uri()),+ fields,+ ))?;+ let raw = json!({"model": "gpt", "messages": [{"role": "user", "content": "judge"}],+ "reasoning_effort": "none", "logprobs": true, "top_logprobs": 8,+ "chat_template_kwargs": {"enable_thinking": false}});+ let result = client+ .call_rewrite_model_raw(+ raw.clone(),+ None,+ Some(&ModelId::from("gpt")),+ WireFormat::OpenAiChat,+ )+ .await;+ let rejects_field = message.starts_with("Unrecognized");+ assert_eq!(result.is_ok(), omit || (rejects_field && !reject_retry));+ let calls = server.received_requests().await.ok_or("missing requests")?;+ assert_eq!(calls.len(), if !omit && rejects_field { 2 } else { 1 });+ for (index, call) in calls.iter().enumerate() {+ let mut expected = raw.clone();+ if omit || index > 0 {+ expected+ .as_object_mut()+ .unwrap()+ .remove("chat_template_kwargs");+ }+ assert_eq!(serde_json::from_slice::<Value>(&call.body)?, expected);+ }+ }+ Ok(())+ }++ #[tokio::test]+ async fn template_compatibility_preserves_the_transient_retry_budget()+ -> std::result::Result<(), Box<dyn Error + Sync + Send>> {+ let server = MockServer::start().await;+ let count = Arc::new(AtomicUsize::new(0));+ let observed = Arc::clone(&count);+ Mock::given(method("POST"))+ .respond_with(move |request: &wiremock::Request| {+ let body: Value = serde_json::from_slice(&request.body).expect("request JSON");+ match observed.fetch_add(1, Ordering::SeqCst) {+ 0 => {+ assert!(body.get("chat_template_kwargs").is_some());+ ResponseTemplate::new(422).set_body_json(json!({"error": {+ "code": "unsupported_parameter", "param": "chat_template_kwargs"+ }}))+ }+ 1 => {+ assert!(body.get("chat_template_kwargs").is_none());+ ResponseTemplate::new(503).insert_header("retry-after", "0")+ }+ _ => ResponseTemplate::new(200).set_body_json(json!({+ "id": "1", "model": "gpt", "choices": [{"index": 0,+ "message": {"role": "assistant", "content": "yes"}, "finish_reason": "stop"}],+ "usage": {}+ })),+ }+ }).mount(&server).await;+ let client =+ TranslatingLlmClient::new(&chat_map_with_retries(&format!("{}/v1", server.uri()), 1))?;+ client+ .call_rewrite_model_raw(+ json!({"model": "gpt", "messages": [{"role": "user", "content": "judge"}],+ "chat_template_kwargs": {"enable_thinking": false}}),+ None,+ Some(&ModelId::from("gpt")),+ WireFormat::OpenAiChat,+ )+ .await?;+ assert_eq!(count.load(Ordering::SeqCst), 3);+ Ok(())+ }++ #[tokio::test]+ async fn preserved_codex_string_input_is_normalized_at_the_client_boundary()+ -> std::result::Result<(), Box<dyn Error + Sync + Send>> {+ for root in ["/backend-api/codex", "/v1"] {+ for input in [+ json!("hello"),+ json!([{"role": "user", "content": "hello"}]),+ ] {+ let server = MockServer::start().await;+ Mock::given(method("POST"))+ .and(path(format!("{root}/responses")))+ .respond_with(ResponseTemplate::new(200).set_body_json(json!({+ "id": "resp_1", "object": "response", "model": "gpt", "status": "completed", "output": [], "usage": {}+ }))).expect(1).mount(&server).await;+ let client =+ TranslatingLlmClient::new(&responses_map(&format!("{}{root}", server.uri())))?;+ let raw = json!({"model": "client-name", "input": input,+ "instructions": "Be brief.", "store": false, "metadata": {"task": "judge"},+ "custom_field": {"keep": true}});+ client+ .call_rewrite_model_raw(+ raw.clone(),+ None,+ Some(&ModelId::from("gpt")),+ WireFormat::OpenAiResponses,+ )+ .await?;+ let mut expected = raw;+ expected["model"] = json!("gpt");+ if root == "/backend-api/codex" && input.is_string() {+ expected["input"] =+ json!([{"type": "message", "role": "user", "content": input}]);+ }+ let calls = server.received_requests().await.ok_or("missing requests")?;+ assert_eq!(serde_json::from_slice::<Value>(&calls[0].body)?, expected);+ }+ }+ Ok(())+ }+#[tokio::test]async fn extra_body_adds_defaults_without_overriding_the_request()-> std::result::Result<(), Box<dyn Error + Sync + Send + 'static>> {diff --git a/crates/libsy-llm-client/src/run.rs b/crates/libsy-llm-client/src/run.rsindex 16b38d312..d08dec1b3 100644--- a/crates/libsy-llm-client/src/run.rs+++ b/crates/libsy-llm-client/src/run.rs@@ -27,8 +27,8 @@ use http::StatusCode;use parking_lot::Mutex;use serde_json::{Value, json};use switchyard_libsy::{- Algorithm, Call, CallDecision, LibsyError, OutcomeMetadata, Result, RoutingOutcome,- RuntimeModels, drive,+ Algorithm, AnswerSummary, Call, CallDecision, LibsyError, OutcomeMetadata, Result,+ RoutingOutcome, RuntimeModels, drive,};use switchyard_protocol::{AggLlmResponse, LlmClientError, LlmResponse, LlmResponseChunk, LlmResponseStream, Message,@@ -69,7 +69,7 @@ pub async fn run(let outcome = match clients.stored_state_owner(&request) {Some(owner) => Ok(continue_on(owner, &algorithm_name, request)),None => {- drive(algorithm, request, models, {+ drive(Arc::clone(&algorithm), request, models, {let routing_observations = routing_observations.clone();move |call| serve(routing_clients.clone(), call, routing_observations.clone())})@@ -121,7 +121,73 @@ pub async fn run(if let Some(observer) = &observer {observer(RunObservation::RoutingOverhead(overhead));}- result.map(|response| (selected_model_id, response))+ let mut response = result?;+ if algorithm.tracks_answers()+ && let Some(session) = session_id(&outcome.request)+ {+ response = report_answer(algorithm, session, response).await;+ }+ Ok((selected_model_id, response))+}++fn session_id(request: &Request) -> Option<String> {+ let session = request.metadata.as_ref()?.session_id.as_deref()?;+ (!session.is_empty()).then(|| session.to_string())+}++/// Reports the served answer to the algorithm; a stream reports it before its final chunk+/// so the client cannot send its next request first.+async fn report_answer(+ algorithm: Arc<dyn Algorithm>,+ session: String,+ mut response: Response,+) -> Response {+ let stream = match response.llm_response {+ LlmResponse::Agg(aggregate) => {+ algorithm+ .observe_answer(&session, AnswerSummary::of(&aggregate))+ .await;+ response.llm_response = LlmResponse::Agg(aggregate);+ return response;+ }+ LlmResponse::Stream(stream) => stream,+ };+ algorithm+ .observe_answer(&session, AnswerSummary { final_reply: false })+ .await;+ let report = (algorithm, session, Some(ResponseAccumulator::new()));+ response.llm_response = LlmResponse::Stream(Box::pin(stream::unfold(+ (stream, report),+ |(mut stream, (algorithm, session, mut accumulator))| async move {+ let item = stream.next().await?;+ let failed = match &item {+ Err(_) => true,+ Ok(event) => event+ .normalized()+ .iter()+ .any(|chunk| matches!(chunk, LlmResponseChunk::StreamError { .. })),+ };+ if failed {+ accumulator = None;+ algorithm+ .observe_answer(&session, AnswerSummary { final_reply: false })+ .await;+ }+ let mut stopped = false;+ if let (Ok(event), Some(answer)) = (&item, accumulator.as_mut()) {+ for chunk in event.normalized() {+ stopped |= matches!(chunk, LlmResponseChunk::MessageStop { .. });+ answer.push(chunk.clone());+ }+ }+ if stopped && let Some(answer) = accumulator.take() {+ let answer = AnswerSummary::of(&answer.finish());+ algorithm.observe_answer(&session, answer).await;+ }+ Some((item, (stream, (algorithm, session, accumulator))))+ },+ )));+ response}/// Run an algorithm to a routing decision without serving its terminal completion.@@ -1187,6 +1253,7 @@ mod tests {LlmResponseChunk::StreamError {message: "stream failed".to_string(),},+ LlmResponseChunk::MessageStop { reason: Some("stop".into()) },])),};}@@ -1540,6 +1607,119 @@ mod tests {Ok(())}+ #[tokio::test]+ async fn served_answer_is_reported_before_the_client_sees_it_end() -> Result<()> {+ #[derive(Default)]+ struct ReportingAlgorithm {+ answers: Mutex<Vec<(String, bool)>>,+ tracking: bool,+ }++ #[async_trait]+ impl Algorithm for ReportingAlgorithm {+ fn name(&self) -> &str {+ "reporting_test"+ }++ async fn route(+ self: Arc<Self>,+ _driver: Driver,+ request: Request,+ ) -> Result<RoutingOutcome> {+ Ok(RoutingOutcome::route_to("weak".into(), Vec::new(), request))+ }++ fn tracks_answers(&self) -> bool {+ self.tracking+ }++ async fn observe_answer(&self, session_id: &str, answer: AnswerSummary) {+ self.answers+ .lock()+ .push((session_id.to_string(), answer.final_reply));+ }+ }++ assert!(!CandidateAlgorithm {}.tracks_answers());+ for (tracking, first, final_reply, cancel) in [+ (false, FirstOutcome::StreamSuccess, true, false),+ (true, FirstOutcome::StreamSuccess, true, false),+ (true, FirstOutcome::MidStreamError, false, false),+ (true, FirstOutcome::StreamSuccess, false, true),+ ] {+ let algorithm = Arc::new(ReportingAlgorithm {+ tracking,+ ..Default::default()+ });+ let client = Arc::new(CandidateClient {+ calls: Mutex::new(Vec::new()),+ requests: Mutex::new(Vec::new()),+ first,+ });+ let request = Request {+ metadata: Some(switchyard_protocol::Metadata {+ session_id: Some("session-1".to_string()),+ ..Default::default()+ }),+ ..request()+ };+ let (_, response) = run(+ algorithm.clone(),+ ClientRouter::single(client),+ request,+ to_category_map(&["weak"]),+ None,+ )+ .await?;+ let LlmResponse::Stream(mut stream) = response.llm_response else {+ panic!("expected a streamed answer");+ };+ if !tracking {+ assert!(algorithm.answers.lock().is_empty());+ while let Some(item) = stream.next().await {+ assert!(item.is_ok());+ }+ assert!(algorithm.answers.lock().is_empty());+ continue;+ }+ if cancel {+ stream.next().await;+ drop(stream);+ assert_eq!(+ *algorithm.answers.lock(),+ [("session-1".to_string(), false)]+ );+ continue;+ }+ while let Some(item) = stream.next().await {+ let stops = item.as_ref().is_ok_and(|event| {+ event+ .normalized()+ .iter()+ .any(|chunk| matches!(chunk, LlmResponseChunk::MessageStop { .. }))+ });+ if stops {+ assert_eq!(algorithm.answers.lock().len(), 2);+ }+ }+ assert_eq!(+ *algorithm.answers.lock(),+ if final_reply {+ vec![+ ("session-1".to_string(), false),+ ("session-1".to_string(), true),+ ]+ } else {+ vec![+ ("session-1".to_string(), false),+ ("session-1".to_string(), false),+ ]+ }+ );+ }+ Ok(())+ }+#[test]fn answer_observation_keeps_call_order() {let pending = Some(Arc::new(Mutex::new(diff --git a/crates/libsy/src/algorithms/fall_through.rs b/crates/libsy/src/algorithms/fall_through.rsindex eb8198a91..9a7286601 100644--- a/crates/libsy/src/algorithms/fall_through.rs+++ b/crates/libsy/src/algorithms/fall_through.rs@@ -27,7 +27,7 @@ use async_trait::async_trait;use parking_lot::Mutex;use tokio::sync::Mutex as AsyncMutex;-use crate::core::algorithm::{self, Algorithm, Driver};+use crate::core::algorithm::{self, Algorithm, AnswerSummary, Driver};use crate::core::classifier::{Classifier, Score};use crate::core::processor::{Event, Processor};use crate::core::state::State;@@ -285,6 +285,31 @@ impl Algorithm for FallThrough {async fn route(self: Arc<Self>, driver: Driver, request: Request) -> Result<RoutingOutcome> {self.execute(driver, request).await}++ fn tracks_answers(&self) -> bool {+ self.session_states.is_some()+ && self+ .classifiers+ .iter()+ .any(|classifier| classifier.tracks_answers())+ }++ async fn observe_answer(&self, session_id: &str, answer: AnswerSummary) {+ let Some(states) = &self.session_states else {+ return;+ };+ let Some(state) = states+ .lock()+ .get(session_id)+ .map(|session| Arc::clone(&session.state))+ else {+ return;+ };+ let mut state = state.lock().await;+ for classifier in &self.classifiers {+ classifier.observe_answer(&mut state, answer);+ }+ }}#[cfg(test)]mod tests {@@ -409,6 +434,61 @@ mod tests {run_turn(&Arc::new(router), serve).await}+ #[tokio::test]+ async fn a_served_answer_reaches_only_its_existing_session() -> Result<()> {+ struct AnswerAware;+ #[async_trait]+ impl Classifier for AnswerAware {+ async fn score(+ &self,+ state: &mut State,+ _request: &mut Request,+ _driver: &Driver,+ ) -> Result<(Classification, Option<Response>)> {+ let target = if state.extra.get("answered") == Some(&crate::StateValue::Bool(true))+ {+ "strong"+ } else {+ "weak"+ };+ Ok((Classification::Scores(vec![score(target, 1.0)]), None))+ }+ fn tracks_answers(&self) -> bool {+ true+ }++ fn observe_answer(&self, state: &mut State, answer: AnswerSummary) {+ state.extra.insert(+ "answered".into(),+ crate::StateValue::Bool(answer.final_reply),+ );+ }+ }+ assert!(!FallThrough::new_with_state().tracks_answers());+ assert!(+ !FallThrough::new()+ .with_classifier(Arc::new(AnswerAware))+ .tracks_answers()+ );+ let router = Arc::new(FallThrough::new_with_state().with_classifier(Arc::new(AnswerAware)));+ assert!(router.tracks_answers());+ let (before, _) = run_turn(&router, echo()).await?;+ router+ .observe_answer("unknown", AnswerSummary { final_reply: true })+ .await;+ let (unknown, _) = run_turn(&router, echo()).await?;+ router+ .observe_answer("session-1", AnswerSummary { final_reply: true })+ .await;+ let (after, _) = run_turn(&router, echo()).await?;+ assert_eq!(+ (before.as_str(), unknown.as_str(), after.as_str()),+ ("weak", "weak", "strong")+ );+ assert_eq!(router.session_states.as_ref().unwrap().lock().len(), 1);+ Ok(())+ }+// --- tests -------------------------------------------------------------------------#[tokio::test]diff --git a/crates/libsy/src/algorithms/vgr.rs b/crates/libsy/src/algorithms/vgr.rsindex 2786c2150..274a80a62 100644--- a/crates/libsy/src/algorithms/vgr.rs+++ b/crates/libsy/src/algorithms/vgr.rs@@ -17,7 +17,7 @@ use self::text::ToolRecord;use crate::Result;use crate::algorithms::fall_through::FallThrough;use crate::algorithms::util::affinity::AffinityRouter;-use crate::core::algorithm::{Algorithm, Driver, RoutingOutcome};+use crate::core::algorithm::{Algorithm, AnswerSummary, Driver, RoutingOutcome};mod compaction;mod config;@@ -102,6 +102,14 @@ impl Algorithm for Vgr {}Ok(outcome)}++ fn tracks_answers(&self) -> bool {+ self.route.tracks_answers()+ }++ async fn observe_answer(&self, session_id: &str, answer: AnswerSummary) {+ self.route.observe_answer(session_id, answer).await;+ }}/// The verification regime a request's capabilities license.diff --git a/crates/libsy/src/algorithms/vgr/runtime.rs b/crates/libsy/src/algorithms/vgr/runtime.rsindex ecfb1b500..a70c4acb8 100644--- a/crates/libsy/src/algorithms/vgr/runtime.rs+++ b/crates/libsy/src/algorithms/vgr/runtime.rs@@ -26,7 +26,7 @@ use crate::algorithms::util::affinity::{AffinityRouter, has_new_user_turn, initiuse crate::algorithms::util::buffered_response::buffer_response;use crate::algorithms::util::decisive;use crate::algorithms::util::prompts::append_note;-use crate::core::algorithm::Driver;+use crate::core::algorithm::{AnswerSummary, Driver};use crate::core::classifier::{Classification, Classifier};use crate::core::state::{State, StateValue};use crate::{LibsyError, Result};@@ -35,6 +35,8 @@ use crate::{LibsyError, Result};const TURN_LATCHED_KEY: &str = "vgr_turn_latched";/// Holds a shadow turn only after a VGR escalation.const SHADOW_HELD_KEY: &str = "vgr_shadow_held";+const LAST_ANSWER_FINAL_KEY: &str = "vgr_last_answer_final";+const FOLLOW_UP_QUIET_SECS: u32 = 2;pub(super) const TURN_STREAK_KEY: &str = "vgr_turn_streak";/// Unix seconds at the first request of the current user turn.const TURN_STARTED_KEY: &str = "vgr_turn_started";@@ -117,7 +119,15 @@ impl Classifier for VgrClassifier {let messages = &request.llm_request.messages;let (new_turn, answered) = match initiated_by_user(request) {Some(by_user) => (by_user, by_user),- None => (has_new_user_turn(messages), answers_final_reply(messages)),+ None => {+ let answered = answers_final_reply(state, messages);+ let new_turn = if state.extra.contains_key(LAST_ANSWER_FINAL_KEY) {+ answered+ } else {+ has_new_user_turn(messages)+ };+ (new_turn, answered)+ }};if new_turn {state.extra.remove(TURN_STARTED_KEY);@@ -272,6 +282,19 @@ impl Classifier for VgrClassifier {set_flag(state, EFFICIENT_SERVED_KEY, true);Ok((decisive(efficient), Some(buffered.into_response())))}++ fn tracks_answers(&self) -> bool {+ true+ }++ fn observe_answer(&self, state: &mut State, answer: AnswerSummary) {+ let finished = if answer.final_reply {+ unix_seconds().max(1)+ } else {+ 0+ };+ set_count(state, LAST_ANSWER_FINAL_KEY, finished);+ }}impl VgrClassifier {@@ -546,7 +569,7 @@ impl VgrClassifier {let outcome = tokio::time::timeout(budget, async {let response = context.driver- .call_model(call, vec![target.clone()])+ .call_model_with_error_recovery(call, vec![target.clone()], true).await?;response.llm_response@@ -623,7 +646,12 @@ fn log(driver: &Driver, reason: &'static str, branch: Option<Branch>, served: Ro}/// Whether the user is answering a finished assistant reply.-fn answers_final_reply(messages: &[Message]) -> bool {+fn answers_final_reply(state: &State, messages: &[Message]) -> bool {+ if let Some(finished) = count(state, LAST_ANSWER_FINAL_KEY)+ && (finished == 0 || unix_seconds().saturating_sub(finished) < FOLLOW_UP_QUIET_SECS)+ {+ return false;+ }messages.len().checked_sub(1)diff --git a/crates/libsy/src/algorithms/vgr/runtime/tests.rs b/crates/libsy/src/algorithms/vgr/runtime/tests.rsindex a882b3353..5e62d8c15 100644--- a/crates/libsy/src/algorithms/vgr/runtime/tests.rs+++ b/crates/libsy/src/algorithms/vgr/runtime/tests.rs@@ -12,9 +12,10 @@ use switchyard_protocol::{use super::super::config::{ACTIVE_APPROVAL, ServingMode, VgrConfig};use super::super::tests::{call, request, result};+use super::{FOLLOW_UP_QUIET_SECS, LAST_ANSWER_FINAL_KEY, answers_final_reply, unix_seconds};use crate::Result;use crate::algorithms::vgr::Vgr;-use crate::core::algorithm::Algorithm;+use crate::core::algorithm::{Algorithm, AnswerSummary};use crate::core::state::{State, StateValue};use crate::core::testing::{category_models, reply, test_drive_with_models};@@ -1253,3 +1254,136 @@ async fn shadow_final_answer_commits_do_not_latch_but_rejections_do() -> Result<assert_eq!(calls.0.load(Ordering::Relaxed), 3);Ok(())}++#[tokio::test]+async fn client_recovery_messages_keep_the_turn_latch() -> Result<()> {+ let route = route(|_| {})?;+ assert!(route.tracks_answers());+ let calls = Arc::new(Calls::default());+ let task = || session(vec![Message::text(Role::User, "fix the build")]);+ for _ in 0..2 {+ drive(&route, task(), calls.clone(), complete_call, 0.9).await?;+ }+ // Hermes-style recovery: a nudge after an empty answer, and one sent the moment a+ // complete-looking answer promised a tool call it never made.+ for (final_reply, previous, nudge) in [+ (+ false,+ "(empty)",+ "You just executed tool calls but returned an empty response.",+ ),+ (+ true,+ "Let me run the tests.",+ "Your previous turn indicated a tool call but none was included.",+ ),+ ] {+ route+ .observe_answer("session-1", AnswerSummary { final_reply })+ .await;+ let recovery = session(vec![+ Message::text(Role::User, "fix the build"),+ Message::text(Role::Assistant, previous),+ Message::text(Role::User, nudge),+ ]);+ assert_eq!(+ drive(&route, recovery, calls.clone(), complete_call, 0.0).await?,+ "capable"+ );+ }+ assert_eq!(calls.0.load(Ordering::Relaxed), 2);+ let next = initiated_by(+ session(vec![+ Message::text(Role::User, "fix the build"),+ Message::text(Role::Assistant, "Let me run the tests."),+ Message::text(Role::User, "now do another task"),+ ]),+ "user",+ );+ assert_eq!(+ drive(&route, next, calls.clone(), complete_call, 0.0).await?,+ "efficient"+ );+ assert_eq!(calls.0.load(Ordering::Relaxed), 3);+ Ok(())+}++#[test]+fn only_a_paused_reply_to_a_reported_final_answer_releases_the_latch() {+ let messages = [+ Message::text(Role::User, "fix the build"),+ Message::text(Role::Assistant, "fixed"),+ Message::text(Role::User, "now add a test"),+ ];+ let now = unix_seconds();+ for (reported, releases) in [+ (None, true),+ (Some(0), false),+ (Some(now), false),+ (Some(now - FOLLOW_UP_QUIET_SECS), true),+ ] {+ let mut state = State::default();+ if let Some(finished) = reported {+ state+ .extra+ .insert(LAST_ANSWER_FINAL_KEY.into(), StateValue::Count(finished));+ }+ assert_eq!(+ answers_final_reply(&state, &messages),+ releases,+ "{reported:?}"+ );+ }+}++#[tokio::test]+async fn a_failed_capable_verifier_still_offers_the_efficient_answer() -> Result<()> {+ use crate::core::algorithm::Call;+ use crate::core::testing::serve_decision;++ let algorithm = route(|config| config.task_typing = true)?;+ let task = || session(vec![Message::text(Role::User, "What is 1+1?")]);+ // Exercise production error propagation through CallModel::respond.+ let outcome = crate::drive(+ algorithm,+ task(),+ Arc::new(category_models(Category::Any, &["efficient", "capable"]).into()),+ |call| async move {+ match call {+ Call::Model(call) => {+ let target = call.models[0].clone();+ let readout = call+ .request+ .llm_request+ .extensions+ .fields+ .contains_key("logprobs");+ let response = match target.as_str() {+ "efficient" => Ok(reply("2")),+ "judge" if readout => Ok(scored(0.05)),+ "judge" => Ok(reply("Answer")),+ _ => Err(crate::LibsyError::client_call(+ target,+ LlmClientError::UpstreamHttp {+ status: http::StatusCode::BAD_GATEWAY,+ body: String::new(),+ },+ )),+ };+ call.respond(std::future::ready(response)).await+ }+ Call::Decision(call) => serve_decision(*call).await,+ }+ },+ )+ .await?;+ assert_eq!(+ outcome.selected_model_ids,+ vec![ModelId::new("capable"), ModelId::new("efficient")]+ );+ let fallback = outcome+ .fallback_request+ .expect("an efficient fallback request");+ assert_eq!(fallback.llm_request.messages, task().llm_request.messages);+ Ok(())+}diff --git a/crates/libsy/src/core/algorithm.rs b/crates/libsy/src/core/algorithm.rsindex da6fd7cbd..ca90765c5 100644--- a/crates/libsy/src/core/algorithm.rs+++ b/crates/libsy/src/core/algorithm.rs@@ -25,7 +25,8 @@ use tracing::Instrument;/// [`switchyard_protocol::LlmResponse`] carries either a live/// [`switchyard_protocol::LlmResponseStream`] or the terminal aggregate.use switchyard_protocol::{- Category, DecisionRequest, DecisionResponse, ModelId, Request, Response,+ AggLlmResponse, Category, ContentBlock, DecisionRequest, DecisionResponse, ModelId, Request,+ Response, StopReason,};use crate::{DriverError, LibsyError, Result, State, observability};@@ -317,6 +318,32 @@ impl RoutingOutcome {}}+/// What the host saw of the answer it served for a routed request.+#[derive(Clone, Copy, Debug, Eq, PartialEq)]+pub struct AnswerSummary {+ /// The answer ended its turn with visible text and requested no tool call.+ pub final_reply: bool,+}++impl AnswerSummary {+ /// Summarizes a completed answer.+ pub fn of(response: &AggLlmResponse) -> Self {+ let final_reply = response.outputs.iter().all(|output| {+ matches!(output.stop_reason, None | Some(StopReason::EndTurn))+ && !output+ .content+ .iter()+ .any(|block| matches!(block, ContentBlock::ToolCall(_)))+ }) && response.outputs.iter().any(|output| {+ output.role == switchyard_protocol::Role::Assistant+ && output.content.iter().any(+ |block| matches!(block, ContentBlock::Text { text } if !text.trim().is_empty()),+ )+ });+ Self { final_reply }+ }+}+/// An algorithm's handle for requesting external work and awaiting the host's typed replies.#[derive(Clone)]pub struct Driver {@@ -740,6 +767,17 @@ pub trait Algorithm: Send + Sync + 'static {/// [`run_stream`](Self::run_stream) runs this method and exposes its work to the host.async fn route(self: Arc<Self>, driver: Driver, request: Request) -> Result<RoutingOutcome>;+ /// Whether the host should summarize served answers for [`Self::observe_answer`].+ fn tracks_answers(&self) -> bool {+ false+ }++ /// Records the answer the host served for a request in `session_id`.+ ///+ /// The host reports a streamed answer before forwarding its final chunk, so the+ /// session's next request is routed with this answer already recorded.+ async fn observe_answer(&self, _session_id: &str, _answer: AnswerSummary) {}+/// Expose work requests and the final outcome so the host can control execution.////// Each call waits for its host reply. A run ends with one terminal item — [`Step::Done`] on@@ -798,6 +836,27 @@ mod tests {LlmResponse, LlmResponseChunk, completion_text, text_request, text_response,};+ #[test]+ fn a_final_answer_has_text_and_no_unfinished_or_tool_output() {+ let mut answer = text_response(None, "done");+ assert!(AnswerSummary::of(&answer).final_reply);+ answer.outputs[0].stop_reason = Some(StopReason::MaxTokens);+ assert!(!AnswerSummary::of(&answer).final_reply);+ answer.outputs[0].stop_reason = Some(StopReason::EndTurn);+ let mut tool = answer.outputs[0].clone();+ tool.content = vec![ContentBlock::ToolCall(switchyard_protocol::ToolCall {+ id: "call".into(),+ name: "bash".into(),+ arguments: serde_json::json!({}),+ })];+ answer.outputs.push(tool);+ assert!(!AnswerSummary::of(&answer).final_reply);+ answer.outputs.pop();+ answer.outputs[0].content = vec![ContentBlock::Text { text: " ".into() }];+ assert!(!AnswerSummary::of(&answer).final_reply);+ assert!(!AnswerSummary::of(&AggLlmResponse::default()).final_reply);+ }+#[derive(Debug, thiserror::Error)]#[error("{0}")]struct TestError(&'static str);diff --git a/crates/libsy/src/core/classifier.rs b/crates/libsy/src/core/classifier.rsindex 06acfae3a..b8169e62a 100644--- a/crates/libsy/src/core/classifier.rs+++ b/crates/libsy/src/core/classifier.rs@@ -1,7 +1,7 @@// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.// SPDX-License-Identifier: Apache-2.0-use crate::core::algorithm::Driver;+use crate::core::algorithm::{AnswerSummary, Driver};use crate::core::state::State;use crate::{LibsyError, Result};use async_trait::async_trait;@@ -91,6 +91,14 @@ pub trait Classifier: Send + Sync {request: &mut Request,driver: &Driver,) -> Result<(Classification, Option<Response>)>;++ /// Whether a stateful composition should request served-answer reports.+ fn tracks_answers(&self) -> bool {+ false+ }++ /// Folds the answer the host served for this session into the composition's state.+ fn observe_answer(&self, _state: &mut State, _answer: AnswerSummary) {}}#[cfg(test)]diff --git a/crates/libsy/src/lib.rs b/crates/libsy/src/lib.rsindex f357e923e..6c5711555 100644--- a/crates/libsy/src/lib.rs+++ b/crates/libsy/src/lib.rs@@ -6,8 +6,8 @@mod core;pub use core::algorithm::{- Algorithm, Call, CallDecision, CallModel, Driver, RoutingOutcome, RuntimeModels, Step,- StepStream, drive,+ Algorithm, AnswerSummary, Call, CallDecision, CallModel, Driver, RoutingOutcome, RuntimeModels,+ Step, StepStream, drive,};pub use core::classifier::{Classification, Classifier, Score};pub use core::outcome_metadata::OutcomeMetadata;diff --git a/docs/routing_algorithms/vgr_routing.md b/docs/routing_algorithms/vgr_routing.mdindex d1d89f5de..64d856c42 100644--- a/docs/routing_algorithms/vgr_routing.md+++ b/docs/routing_algorithms/vgr_routing.md@@ -51,9 +51,12 @@ discovery for those servers is not implemented.The judge reads its verdicts from the first token's logprobs, so its scored callsturn thinking off: they send `reasoning_effort = "none"` and`chat_template_kwargs = {"enable_thinking": false}` on `openai_chat` targets.-Candidates and the judge's deliberation keep the target's own setting. A backend-that rejects `chat_template_kwargs` can drop it with the target's-`omit_body_fields`.+Candidates and the judge's deliberation keep the target's own setting. If a Chat+backend explicitly rejects this template switch with HTTP 400 or 422, the SDK+retries once without it, within the same deadline. Other request fields and the+configured transient retry budget are preserved. The target's `omit_body_fields`+can omit the switch from the first call. Custom hosts must handle provider+compatibility themselves.The server caches the model catalog, including unavailable discovery results,for one minute. Discovery probes run concurrently with a two-second total budget@@ -130,6 +133,25 @@ automatic continuations. Retries retain the original call's tag. Without arecognized tag, Switchyard uses the message shape to identify a user turn.`x-session-id` is also accepted as a session-header alias.+## Served-answer reporting++VGR opts into served-answer reporting through `Algorithm::tracks_answers`.+The SDK summarizes answers only for opted-in algorithms with a session ID;+other routers keep their responses and streams unchanged. Custom algorithms and+classifiers default to no tracking. Stateful `FallThrough` routes opt in when a+classifier requests reports.++Custom libsy hosts can check `tracks_answers()` and report+`AnswerSummary::of(&response)` through `Algorithm::observe_answer` for aggregate+responses. For streams, report `final_reply = false` when serving begins and+report the final summary before forwarding the terminal chunk. Use the request's+session ID; requests without one retain the existing behavior.++Without a recognized `x-initiator` header, reported incomplete answers and+immediate recovery nudges retain the turn hold and budget. A reply to a complete+final answer releases them after a two-second quiet period. Explicit user/agent+headers take precedence. Hosts that omit reporting retain message-shape detection.+## Verification and capable-tier outagesPolicy version `1.2` allows the default verification branch to accept a clean