Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

feat(libsy): report served answers for VGR recovery (#1166)

GDRSe1ed2b5ea3cf
diff --git a/crates/libsy-llm-client/README.md b/crates/libsy-llm-client/README.md
index c1734a157..3cccb5d41 100644
--- a/crates/libsy-llm-client/README.md
+++ b/crates/libsy-llm-client/README.md
@@ -269,6 +269,14 @@ fn build_multi_format_client(
transport failures, timeouts, HTTP 408/429, and 5xx responses. Buffered body
transport failures are retried; streaming body failures are not replayed after
the response has been returned.
+- If an OpenAI Chat backend explicitly rejects `chat_template_kwargs =
+ {"enable_thinking": false}` with HTTP 400 or 422, the client retries once without
+ that field. This compatibility retry is separate from `max_retries` and shares
+ the original timeout. `omit_body_fields` can remove it from the first attempt.
+- Codex endpoints whose URL path ends in `/backend-api/codex/responses` receive
+ string `input` as a message-item list, including preserved requests. Other
+ fields, array input, non-Codex endpoints, and the translation codec's exact
+ preservation remain unchanged.
- `ModelConfig::with_responses_reasoning` controls Responses reasoning replay.
Every Responses model defaults to `PreserveEncrypted`: plaintext is removed,
encrypted provider state is retained. Stored reasoning IDs are retained unless
diff --git a/crates/libsy-llm-client/src/backend.rs b/crates/libsy-llm-client/src/backend.rs
index 461fca4ed..b24c1a0d7 100644
--- a/crates/libsy-llm-client/src/backend.rs
+++ b/crates/libsy-llm-client/src/backend.rs
@@ -79,7 +79,8 @@ pub struct HttpBackendConfig {
/// sent. Responses carries it as `reasoning.effort`, Chat Completions as `reasoning_effort`;
/// Anthropic has no equivalent and rejects the setting at configuration time.
pub reasoning_effort: Option<String>,
- /// Additional attempts after the initial upstream request.
+ /// Additional attempts for transient upstream failures.
+ /// One template-compatibility retry may also run within the same timeout.
pub max_retries: u32,
/// Cooldown after an exhausted transient completion failure. Zero disables it.
pub failure_cooldown: Duration,
diff --git a/crates/libsy-llm-client/src/client.rs b/crates/libsy-llm-client/src/client.rs
index 132bf7a07..ef56b2706 100644
--- a/crates/libsy-llm-client/src/client.rs
+++ b/crates/libsy-llm-client/src/client.rs
@@ -4,6 +4,7 @@
//! [`TranslatingLlmClient`]: encode a neutral
//! request, call the configured backend over HTTP, decode the neutral response.
+use std::borrow::Cow;
use std::collections::{BTreeMap, BTreeSet, HashMap};
use std::future::ready;
use std::sync::atomic::{AtomicU64, Ordering};
@@ -362,6 +363,9 @@ impl TranslatingLlmClient {
// After the merge on purpose: the effort override must win over both the caller's
// value and any `reasoning` default a target set through `extra_body`.
apply_reasoning_effort(&mut body, backend);
+ if matches!(backend, Backend::OpenAiResponses(_)) {
+ normalize_codex_string_input(&backend.url(), &mut body);
+ }
if matches!(backend, Backend::Anthropic(_)) {
if nvidia_inference_api_backend(backend) {
strip_anthropic_tool_strict(&mut body);
@@ -404,7 +408,14 @@ impl TranslatingLlmClient {
streaming: bool,
) -> Result<EncodedResponse> {
let max_retries = u64::from(backend.max_retries());
- let max_attempts = max_retries + 1;
+ let max_attempts = max_retries
+ + 1
+ + u64::from(
+ matches!(backend, Backend::OpenAiChat(_))
+ && body.get("chat_template_kwargs").is_some(),
+ );
+ let mut body = Cow::Borrowed(body);
+ let mut template_retry = false;
let expires_at = backend
.timeout()
.map(|timeout| (tokio::time::sleep(timeout).deadline(), timeout));
@@ -430,9 +441,9 @@ impl TranslatingLlmClient {
"libsy.upstream_attempt",
model = %model,
wire_format = %backend.wire_format(),
- attempt = attempt + 1,
+ attempt = attempt + 1 + u64::from(template_retry),
max_attempts,
- retry = attempt > 0,
+ retry = attempt > 0 || template_retry,
openinference.span.kind = "CHAIN",
outcome = tracing::field::Empty,
status_code = tracing::field::Empty,
@@ -452,7 +463,7 @@ impl TranslatingLlmClient {
}
result = async {
attempt_started = true;
- self.send_once(url, backend, body, metadata, model, streaming).await
+ self.send_once(url, backend, body.as_ref(), metadata, model, streaming).await
}.instrument(span.clone()) => result,
};
// The retained handle updates this same attempt span with its outcome.
@@ -461,7 +472,7 @@ impl TranslatingLlmClient {
span.record("outcome", "ok");
span.record("status_code", response.status());
span.record("will_retry", false);
- if attempt > 0 {
+ if attempt > 0 || template_retry {
metrics::record_retry_recovered();
}
return Ok(match response {
@@ -478,7 +489,12 @@ impl TranslatingLlmClient {
});
}
Err(failure) => {
- let will_retry = attempt < max_retries && failure.is_retryable();
+ let retry_without_template = matches!(backend, Backend::OpenAiChat(_))
+ && body.get("chat_template_kwargs")
+ == Some(&json!({"enable_thinking": false}))
+ && rejects_template_options(&failure.error);
+ let will_retry =
+ retry_without_template || (attempt < max_retries && failure.is_retryable());
span.record("outcome", "error");
if let Some(status) = failure.status {
span.record("status_code", status.as_u16());
@@ -488,6 +504,14 @@ impl TranslatingLlmClient {
return Err(failure.error);
}
+ if retry_without_template {
+ if let Some(object) = body.to_mut().as_object_mut() {
+ object.remove("chat_template_kwargs");
+ }
+ template_retry = true;
+ span.record("retry_delay_ms", 0_u64);
+ continue;
+ }
let delay = retry_delay(attempt, failure.retry_after);
span.record("retry_delay_ms", duration_millis(delay));
// Close the attempt span before sleeping so backoff is not attempt latency.
@@ -1048,6 +1072,60 @@ fn set_json_model(body: &mut Value, model: &str) {
}
}
+// The Codex Responses endpoint requires item-list input even on the preserved-body path.
+fn normalize_codex_string_input(url: &str, body: &mut Value) {
+ let codex = reqwest::Url::parse(url).ok().is_some_and(|url| {
+ url.path()
+ .trim_end_matches('/')
+ .ends_with("/backend-api/codex/responses")
+ });
+ if codex && let Some(input @ Value::String(_)) = body.get_mut("input") {
+ let text = input.take();
+ *input = json!([{"type": "message", "role": "user", "content": text}]);
+ }
+}
+
+fn rejects_template_options(error: &LlmClientError) -> bool {
+ let LlmClientError::UpstreamHttp { status, body } = error else {
+ return false;
+ };
+ if !matches!(
+ *status,
+ StatusCode::BAD_REQUEST | StatusCode::UNPROCESSABLE_ENTITY
+ ) {
+ return false;
+ }
+ let parsed = serde_json::from_str::<Value>(body).ok();
+ let error = parsed.as_ref().and_then(|value| value.get("error"));
+ let message = match &parsed {
+ Some(_) => error
+ .and_then(|error| error.get("message"))
+ .and_then(Value::as_str)
+ .unwrap_or_default(),
+ None => body,
+ }
+ .to_ascii_lowercase();
+ let field = error
+ .and_then(|error| error.get("param"))
+ .and_then(Value::as_str);
+ let code = error
+ .and_then(|error| error.get("code"))
+ .and_then(Value::as_str);
+ (field == Some("chat_template_kwargs") || message.contains("chat_template_kwargs"))
+ && (matches!(code, Some("unknown_parameter" | "unsupported_parameter"))
+ || [
+ "unrecognized request argument",
+ "unknown parameter",
+ "unsupported parameter",
+ "unexpected keyword argument",
+ "unknown field",
+ "unsupported field",
+ "extra inputs are not permitted",
+ ]
+ .iter()
+ .any(|phrase| message.contains(phrase)))
+}
+
const CODEX_NAMESPACE_SEPARATOR: &str = "__";
/// Rewrites a Responses body into the shape OpenAI-compatible providers accept.
@@ -2237,6 +2315,165 @@ mod tests {
Ok(())
}
+ #[tokio::test]
+ async fn template_rejection_retries_once_without_changing_other_fields()
+ -> std::result::Result<(), Box<dyn Error + Sync + Send>> {
+ for (message, reject_retry, omit) in [
+ (
+ "Unrecognized request argument supplied: chat_template_kwargs",
+ false,
+ false,
+ ),
+ (
+ "Unrecognized request argument supplied: chat_template_kwargs",
+ true,
+ false,
+ ),
+ (
+ "Invalid temperature for chat_template_kwargs request",
+ false,
+ false,
+ ),
+ (
+ "Unrecognized request argument supplied: chat_template_kwargs",
+ false,
+ true,
+ ),
+ ] {
+ let server = MockServer::start().await;
+ Mock::given(method("POST"))
+ .respond_with(move |request: &wiremock::Request| {
+ let body: Value = serde_json::from_slice(&request.body).expect("request JSON");
+ if body.get("chat_template_kwargs").is_some() || reject_retry {
+ ResponseTemplate::new(400).set_body_json(json!({"error": {"message": message}}))
+ } else {
+ ResponseTemplate::new(200).set_body_json(json!({
+ "id": "1", "model": "gpt", "choices": [{"index": 0,
+ "message": {"role": "assistant", "content": "yes"}, "finish_reason": "stop"}],
+ "usage": {}
+ }))
+ }
+ }).mount(&server).await;
+ let fields = if omit {
+ BTreeSet::from(["chat_template_kwargs".into()])
+ } else {
+ BTreeSet::new()
+ };
+ let client = TranslatingLlmClient::new(&chat_map_with_omit_body_fields(
+ &format!("{}/v1", server.uri()),
+ fields,
+ ))?;
+ let raw = json!({"model": "gpt", "messages": [{"role": "user", "content": "judge"}],
+ "reasoning_effort": "none", "logprobs": true, "top_logprobs": 8,
+ "chat_template_kwargs": {"enable_thinking": false}});
+ let result = client
+ .call_rewrite_model_raw(
+ raw.clone(),
+ None,
+ Some(&ModelId::from("gpt")),
+ WireFormat::OpenAiChat,
+ )
+ .await;
+ let rejects_field = message.starts_with("Unrecognized");
+ assert_eq!(result.is_ok(), omit || (rejects_field && !reject_retry));
+ let calls = server.received_requests().await.ok_or("missing requests")?;
+ assert_eq!(calls.len(), if !omit && rejects_field { 2 } else { 1 });
+ for (index, call) in calls.iter().enumerate() {
+ let mut expected = raw.clone();
+ if omit || index > 0 {
+ expected
+ .as_object_mut()
+ .unwrap()
+ .remove("chat_template_kwargs");
+ }
+ assert_eq!(serde_json::from_slice::<Value>(&call.body)?, expected);
+ }
+ }
+ Ok(())
+ }
+
+ #[tokio::test]
+ async fn template_compatibility_preserves_the_transient_retry_budget()
+ -> std::result::Result<(), Box<dyn Error + Sync + Send>> {
+ let server = MockServer::start().await;
+ let count = Arc::new(AtomicUsize::new(0));
+ let observed = Arc::clone(&count);
+ Mock::given(method("POST"))
+ .respond_with(move |request: &wiremock::Request| {
+ let body: Value = serde_json::from_slice(&request.body).expect("request JSON");
+ match observed.fetch_add(1, Ordering::SeqCst) {
+ 0 => {
+ assert!(body.get("chat_template_kwargs").is_some());
+ ResponseTemplate::new(422).set_body_json(json!({"error": {
+ "code": "unsupported_parameter", "param": "chat_template_kwargs"
+ }}))
+ }
+ 1 => {
+ assert!(body.get("chat_template_kwargs").is_none());
+ ResponseTemplate::new(503).insert_header("retry-after", "0")
+ }
+ _ => ResponseTemplate::new(200).set_body_json(json!({
+ "id": "1", "model": "gpt", "choices": [{"index": 0,
+ "message": {"role": "assistant", "content": "yes"}, "finish_reason": "stop"}],
+ "usage": {}
+ })),
+ }
+ }).mount(&server).await;
+ let client =
+ TranslatingLlmClient::new(&chat_map_with_retries(&format!("{}/v1", server.uri()), 1))?;
+ client
+ .call_rewrite_model_raw(
+ json!({"model": "gpt", "messages": [{"role": "user", "content": "judge"}],
+ "chat_template_kwargs": {"enable_thinking": false}}),
+ None,
+ Some(&ModelId::from("gpt")),
+ WireFormat::OpenAiChat,
+ )
+ .await?;
+ assert_eq!(count.load(Ordering::SeqCst), 3);
+ Ok(())
+ }
+
+ #[tokio::test]
+ async fn preserved_codex_string_input_is_normalized_at_the_client_boundary()
+ -> std::result::Result<(), Box<dyn Error + Sync + Send>> {
+ for root in ["/backend-api/codex", "/v1"] {
+ for input in [
+ json!("hello"),
+ json!([{"role": "user", "content": "hello"}]),
+ ] {
+ let server = MockServer::start().await;
+ Mock::given(method("POST"))
+ .and(path(format!("{root}/responses")))
+ .respond_with(ResponseTemplate::new(200).set_body_json(json!({
+ "id": "resp_1", "object": "response", "model": "gpt", "status": "completed", "output": [], "usage": {}
+ }))).expect(1).mount(&server).await;
+ let client =
+ TranslatingLlmClient::new(&responses_map(&format!("{}{root}", server.uri())))?;
+ let raw = json!({"model": "client-name", "input": input,
+ "instructions": "Be brief.", "store": false, "metadata": {"task": "judge"},
+ "custom_field": {"keep": true}});
+ client
+ .call_rewrite_model_raw(
+ raw.clone(),
+ None,
+ Some(&ModelId::from("gpt")),
+ WireFormat::OpenAiResponses,
+ )
+ .await?;
+ let mut expected = raw;
+ expected["model"] = json!("gpt");
+ if root == "/backend-api/codex" && input.is_string() {
+ expected["input"] =
+ json!([{"type": "message", "role": "user", "content": input}]);
+ }
+ let calls = server.received_requests().await.ok_or("missing requests")?;
+ assert_eq!(serde_json::from_slice::<Value>(&calls[0].body)?, expected);
+ }
+ }
+ Ok(())
+ }
+
#[tokio::test]
async fn extra_body_adds_defaults_without_overriding_the_request()
-> std::result::Result<(), Box<dyn Error + Sync + Send + 'static>> {
diff --git a/crates/libsy-llm-client/src/run.rs b/crates/libsy-llm-client/src/run.rs
index 16b38d312..d08dec1b3 100644
--- a/crates/libsy-llm-client/src/run.rs
+++ b/crates/libsy-llm-client/src/run.rs
@@ -27,8 +27,8 @@ use http::StatusCode;
use parking_lot::Mutex;
use serde_json::{Value, json};
use switchyard_libsy::{
- Algorithm, Call, CallDecision, LibsyError, OutcomeMetadata, Result, RoutingOutcome,
- RuntimeModels, drive,
+ Algorithm, AnswerSummary, Call, CallDecision, LibsyError, OutcomeMetadata, Result,
+ RoutingOutcome, RuntimeModels, drive,
};
use switchyard_protocol::{
AggLlmResponse, LlmClientError, LlmResponse, LlmResponseChunk, LlmResponseStream, Message,
@@ -69,7 +69,7 @@ pub async fn run(
let outcome = match clients.stored_state_owner(&request) {
Some(owner) => Ok(continue_on(owner, &algorithm_name, request)),
None => {
- drive(algorithm, request, models, {
+ drive(Arc::clone(&algorithm), request, models, {
let routing_observations = routing_observations.clone();
move |call| serve(routing_clients.clone(), call, routing_observations.clone())
})
@@ -121,7 +121,73 @@ pub async fn run(
if let Some(observer) = &observer {
observer(RunObservation::RoutingOverhead(overhead));
}
- result.map(|response| (selected_model_id, response))
+ let mut response = result?;
+ if algorithm.tracks_answers()
+ && let Some(session) = session_id(&outcome.request)
+ {
+ response = report_answer(algorithm, session, response).await;
+ }
+ Ok((selected_model_id, response))
+}
+
+fn session_id(request: &Request) -> Option<String> {
+ let session = request.metadata.as_ref()?.session_id.as_deref()?;
+ (!session.is_empty()).then(|| session.to_string())
+}
+
+/// Reports the served answer to the algorithm; a stream reports it before its final chunk
+/// so the client cannot send its next request first.
+async fn report_answer(
+ algorithm: Arc<dyn Algorithm>,
+ session: String,
+ mut response: Response,
+) -> Response {
+ let stream = match response.llm_response {
+ LlmResponse::Agg(aggregate) => {
+ algorithm
+ .observe_answer(&session, AnswerSummary::of(&aggregate))
+ .await;
+ response.llm_response = LlmResponse::Agg(aggregate);
+ return response;
+ }
+ LlmResponse::Stream(stream) => stream,
+ };
+ algorithm
+ .observe_answer(&session, AnswerSummary { final_reply: false })
+ .await;
+ let report = (algorithm, session, Some(ResponseAccumulator::new()));
+ response.llm_response = LlmResponse::Stream(Box::pin(stream::unfold(
+ (stream, report),
+ |(mut stream, (algorithm, session, mut accumulator))| async move {
+ let item = stream.next().await?;
+ let failed = match &item {
+ Err(_) => true,
+ Ok(event) => event
+ .normalized()
+ .iter()
+ .any(|chunk| matches!(chunk, LlmResponseChunk::StreamError { .. })),
+ };
+ if failed {
+ accumulator = None;
+ algorithm
+ .observe_answer(&session, AnswerSummary { final_reply: false })
+ .await;
+ }
+ let mut stopped = false;
+ if let (Ok(event), Some(answer)) = (&item, accumulator.as_mut()) {
+ for chunk in event.normalized() {
+ stopped |= matches!(chunk, LlmResponseChunk::MessageStop { .. });
+ answer.push(chunk.clone());
+ }
+ }
+ if stopped && let Some(answer) = accumulator.take() {
+ let answer = AnswerSummary::of(&answer.finish());
+ algorithm.observe_answer(&session, answer).await;
+ }
+ Some((item, (stream, (algorithm, session, accumulator))))
+ },
+ )));
+ response
}
/// Run an algorithm to a routing decision without serving its terminal completion.
@@ -1187,6 +1253,7 @@ mod tests {
LlmResponseChunk::StreamError {
message: "stream failed".to_string(),
},
+ LlmResponseChunk::MessageStop { reason: Some("stop".into()) },
])),
};
}
@@ -1540,6 +1607,119 @@ mod tests {
Ok(())
}
+ #[tokio::test]
+ async fn served_answer_is_reported_before_the_client_sees_it_end() -> Result<()> {
+ #[derive(Default)]
+ struct ReportingAlgorithm {
+ answers: Mutex<Vec<(String, bool)>>,
+ tracking: bool,
+ }
+
+ #[async_trait]
+ impl Algorithm for ReportingAlgorithm {
+ fn name(&self) -> &str {
+ "reporting_test"
+ }
+
+ async fn route(
+ self: Arc<Self>,
+ _driver: Driver,
+ request: Request,
+ ) -> Result<RoutingOutcome> {
+ Ok(RoutingOutcome::route_to("weak".into(), Vec::new(), request))
+ }
+
+ fn tracks_answers(&self) -> bool {
+ self.tracking
+ }
+
+ async fn observe_answer(&self, session_id: &str, answer: AnswerSummary) {
+ self.answers
+ .lock()
+ .push((session_id.to_string(), answer.final_reply));
+ }
+ }
+
+ assert!(!CandidateAlgorithm {}.tracks_answers());
+ for (tracking, first, final_reply, cancel) in [
+ (false, FirstOutcome::StreamSuccess, true, false),
+ (true, FirstOutcome::StreamSuccess, true, false),
+ (true, FirstOutcome::MidStreamError, false, false),
+ (true, FirstOutcome::StreamSuccess, false, true),
+ ] {
+ let algorithm = Arc::new(ReportingAlgorithm {
+ tracking,
+ ..Default::default()
+ });
+ let client = Arc::new(CandidateClient {
+ calls: Mutex::new(Vec::new()),
+ requests: Mutex::new(Vec::new()),
+ first,
+ });
+ let request = Request {
+ metadata: Some(switchyard_protocol::Metadata {
+ session_id: Some("session-1".to_string()),
+ ..Default::default()
+ }),
+ ..request()
+ };
+ let (_, response) = run(
+ algorithm.clone(),
+ ClientRouter::single(client),
+ request,
+ to_category_map(&["weak"]),
+ None,
+ )
+ .await?;
+ let LlmResponse::Stream(mut stream) = response.llm_response else {
+ panic!("expected a streamed answer");
+ };
+ if !tracking {
+ assert!(algorithm.answers.lock().is_empty());
+ while let Some(item) = stream.next().await {
+ assert!(item.is_ok());
+ }
+ assert!(algorithm.answers.lock().is_empty());
+ continue;
+ }
+ if cancel {
+ stream.next().await;
+ drop(stream);
+ assert_eq!(
+ *algorithm.answers.lock(),
+ [("session-1".to_string(), false)]
+ );
+ continue;
+ }
+ while let Some(item) = stream.next().await {
+ let stops = item.as_ref().is_ok_and(|event| {
+ event
+ .normalized()
+ .iter()
+ .any(|chunk| matches!(chunk, LlmResponseChunk::MessageStop { .. }))
+ });
+ if stops {
+ assert_eq!(algorithm.answers.lock().len(), 2);
+ }
+ }
+ assert_eq!(
+ *algorithm.answers.lock(),
+ if final_reply {
+ vec![
+ ("session-1".to_string(), false),
+ ("session-1".to_string(), true),
+ ]
+ } else {
+ vec![
+ ("session-1".to_string(), false),
+ ("session-1".to_string(), false),
+ ]
+ }
+ );
+ }
+ Ok(())
+ }
+
#[test]
fn answer_observation_keeps_call_order() {
let pending = Some(Arc::new(Mutex::new(
diff --git a/crates/libsy/src/algorithms/fall_through.rs b/crates/libsy/src/algorithms/fall_through.rs
index eb8198a91..9a7286601 100644
--- a/crates/libsy/src/algorithms/fall_through.rs
+++ b/crates/libsy/src/algorithms/fall_through.rs
@@ -27,7 +27,7 @@ use async_trait::async_trait;
use parking_lot::Mutex;
use tokio::sync::Mutex as AsyncMutex;
-use crate::core::algorithm::{self, Algorithm, Driver};
+use crate::core::algorithm::{self, Algorithm, AnswerSummary, Driver};
use crate::core::classifier::{Classifier, Score};
use crate::core::processor::{Event, Processor};
use crate::core::state::State;
@@ -285,6 +285,31 @@ impl Algorithm for FallThrough {
async fn route(self: Arc<Self>, driver: Driver, request: Request) -> Result<RoutingOutcome> {
self.execute(driver, request).await
}
+
+ fn tracks_answers(&self) -> bool {
+ self.session_states.is_some()
+ && self
+ .classifiers
+ .iter()
+ .any(|classifier| classifier.tracks_answers())
+ }
+
+ async fn observe_answer(&self, session_id: &str, answer: AnswerSummary) {
+ let Some(states) = &self.session_states else {
+ return;
+ };
+ let Some(state) = states
+ .lock()
+ .get(session_id)
+ .map(|session| Arc::clone(&session.state))
+ else {
+ return;
+ };
+ let mut state = state.lock().await;
+ for classifier in &self.classifiers {
+ classifier.observe_answer(&mut state, answer);
+ }
+ }
}
#[cfg(test)]
mod tests {
@@ -409,6 +434,61 @@ mod tests {
run_turn(&Arc::new(router), serve).await
}
+ #[tokio::test]
+ async fn a_served_answer_reaches_only_its_existing_session() -> Result<()> {
+ struct AnswerAware;
+ #[async_trait]
+ impl Classifier for AnswerAware {
+ async fn score(
+ &self,
+ state: &mut State,
+ _request: &mut Request,
+ _driver: &Driver,
+ ) -> Result<(Classification, Option<Response>)> {
+ let target = if state.extra.get("answered") == Some(&crate::StateValue::Bool(true))
+ {
+ "strong"
+ } else {
+ "weak"
+ };
+ Ok((Classification::Scores(vec![score(target, 1.0)]), None))
+ }
+ fn tracks_answers(&self) -> bool {
+ true
+ }
+
+ fn observe_answer(&self, state: &mut State, answer: AnswerSummary) {
+ state.extra.insert(
+ "answered".into(),
+ crate::StateValue::Bool(answer.final_reply),
+ );
+ }
+ }
+ assert!(!FallThrough::new_with_state().tracks_answers());
+ assert!(
+ !FallThrough::new()
+ .with_classifier(Arc::new(AnswerAware))
+ .tracks_answers()
+ );
+ let router = Arc::new(FallThrough::new_with_state().with_classifier(Arc::new(AnswerAware)));
+ assert!(router.tracks_answers());
+ let (before, _) = run_turn(&router, echo()).await?;
+ router
+ .observe_answer("unknown", AnswerSummary { final_reply: true })
+ .await;
+ let (unknown, _) = run_turn(&router, echo()).await?;
+ router
+ .observe_answer("session-1", AnswerSummary { final_reply: true })
+ .await;
+ let (after, _) = run_turn(&router, echo()).await?;
+ assert_eq!(
+ (before.as_str(), unknown.as_str(), after.as_str()),
+ ("weak", "weak", "strong")
+ );
+ assert_eq!(router.session_states.as_ref().unwrap().lock().len(), 1);
+ Ok(())
+ }
+
// --- tests -------------------------------------------------------------------------
#[tokio::test]
diff --git a/crates/libsy/src/algorithms/vgr.rs b/crates/libsy/src/algorithms/vgr.rs
index 2786c2150..274a80a62 100644
--- a/crates/libsy/src/algorithms/vgr.rs
+++ b/crates/libsy/src/algorithms/vgr.rs
@@ -17,7 +17,7 @@ use self::text::ToolRecord;
use crate::Result;
use crate::algorithms::fall_through::FallThrough;
use crate::algorithms::util::affinity::AffinityRouter;
-use crate::core::algorithm::{Algorithm, Driver, RoutingOutcome};
+use crate::core::algorithm::{Algorithm, AnswerSummary, Driver, RoutingOutcome};
mod compaction;
mod config;
@@ -102,6 +102,14 @@ impl Algorithm for Vgr {
}
Ok(outcome)
}
+
+ fn tracks_answers(&self) -> bool {
+ self.route.tracks_answers()
+ }
+
+ async fn observe_answer(&self, session_id: &str, answer: AnswerSummary) {
+ self.route.observe_answer(session_id, answer).await;
+ }
}
/// The verification regime a request's capabilities license.
diff --git a/crates/libsy/src/algorithms/vgr/runtime.rs b/crates/libsy/src/algorithms/vgr/runtime.rs
index ecfb1b500..a70c4acb8 100644
--- a/crates/libsy/src/algorithms/vgr/runtime.rs
+++ b/crates/libsy/src/algorithms/vgr/runtime.rs
@@ -26,7 +26,7 @@ use crate::algorithms::util::affinity::{AffinityRouter, has_new_user_turn, initi
use crate::algorithms::util::buffered_response::buffer_response;
use crate::algorithms::util::decisive;
use crate::algorithms::util::prompts::append_note;
-use crate::core::algorithm::Driver;
+use crate::core::algorithm::{AnswerSummary, Driver};
use crate::core::classifier::{Classification, Classifier};
use crate::core::state::{State, StateValue};
use crate::{LibsyError, Result};
@@ -35,6 +35,8 @@ use crate::{LibsyError, Result};
const TURN_LATCHED_KEY: &str = "vgr_turn_latched";
/// Holds a shadow turn only after a VGR escalation.
const SHADOW_HELD_KEY: &str = "vgr_shadow_held";
+const LAST_ANSWER_FINAL_KEY: &str = "vgr_last_answer_final";
+const FOLLOW_UP_QUIET_SECS: u32 = 2;
pub(super) const TURN_STREAK_KEY: &str = "vgr_turn_streak";
/// Unix seconds at the first request of the current user turn.
const TURN_STARTED_KEY: &str = "vgr_turn_started";
@@ -117,7 +119,15 @@ impl Classifier for VgrClassifier {
let messages = &request.llm_request.messages;
let (new_turn, answered) = match initiated_by_user(request) {
Some(by_user) => (by_user, by_user),
- None => (has_new_user_turn(messages), answers_final_reply(messages)),
+ None => {
+ let answered = answers_final_reply(state, messages);
+ let new_turn = if state.extra.contains_key(LAST_ANSWER_FINAL_KEY) {
+ answered
+ } else {
+ has_new_user_turn(messages)
+ };
+ (new_turn, answered)
+ }
};
if new_turn {
state.extra.remove(TURN_STARTED_KEY);
@@ -272,6 +282,19 @@ impl Classifier for VgrClassifier {
set_flag(state, EFFICIENT_SERVED_KEY, true);
Ok((decisive(efficient), Some(buffered.into_response())))
}
+
+ fn tracks_answers(&self) -> bool {
+ true
+ }
+
+ fn observe_answer(&self, state: &mut State, answer: AnswerSummary) {
+ let finished = if answer.final_reply {
+ unix_seconds().max(1)
+ } else {
+ 0
+ };
+ set_count(state, LAST_ANSWER_FINAL_KEY, finished);
+ }
}
impl VgrClassifier {
@@ -546,7 +569,7 @@ impl VgrClassifier {
let outcome = tokio::time::timeout(budget, async {
let response = context
.driver
- .call_model(call, vec![target.clone()])
+ .call_model_with_error_recovery(call, vec![target.clone()], true)
.await?;
response
.llm_response
@@ -623,7 +646,12 @@ fn log(driver: &Driver, reason: &'static str, branch: Option<Branch>, served: Ro
}
/// Whether the user is answering a finished assistant reply.
-fn answers_final_reply(messages: &[Message]) -> bool {
+fn answers_final_reply(state: &State, messages: &[Message]) -> bool {
+ if let Some(finished) = count(state, LAST_ANSWER_FINAL_KEY)
+ && (finished == 0 || unix_seconds().saturating_sub(finished) < FOLLOW_UP_QUIET_SECS)
+ {
+ return false;
+ }
messages
.len()
.checked_sub(1)
diff --git a/crates/libsy/src/algorithms/vgr/runtime/tests.rs b/crates/libsy/src/algorithms/vgr/runtime/tests.rs
index a882b3353..5e62d8c15 100644
--- a/crates/libsy/src/algorithms/vgr/runtime/tests.rs
+++ b/crates/libsy/src/algorithms/vgr/runtime/tests.rs
@@ -12,9 +12,10 @@ use switchyard_protocol::{
use super::super::config::{ACTIVE_APPROVAL, ServingMode, VgrConfig};
use super::super::tests::{call, request, result};
+use super::{FOLLOW_UP_QUIET_SECS, LAST_ANSWER_FINAL_KEY, answers_final_reply, unix_seconds};
use crate::Result;
use crate::algorithms::vgr::Vgr;
-use crate::core::algorithm::Algorithm;
+use crate::core::algorithm::{Algorithm, AnswerSummary};
use crate::core::state::{State, StateValue};
use crate::core::testing::{category_models, reply, test_drive_with_models};
@@ -1253,3 +1254,136 @@ async fn shadow_final_answer_commits_do_not_latch_but_rejections_do() -> Result<
assert_eq!(calls.0.load(Ordering::Relaxed), 3);
Ok(())
}
+
+#[tokio::test]
+async fn client_recovery_messages_keep_the_turn_latch() -> Result<()> {
+ let route = route(|_| {})?;
+ assert!(route.tracks_answers());
+ let calls = Arc::new(Calls::default());
+ let task = || session(vec![Message::text(Role::User, "fix the build")]);
+ for _ in 0..2 {
+ drive(&route, task(), calls.clone(), complete_call, 0.9).await?;
+ }
+ // Hermes-style recovery: a nudge after an empty answer, and one sent the moment a
+ // complete-looking answer promised a tool call it never made.
+ for (final_reply, previous, nudge) in [
+ (
+ false,
+ "(empty)",
+ "You just executed tool calls but returned an empty response.",
+ ),
+ (
+ true,
+ "Let me run the tests.",
+ "Your previous turn indicated a tool call but none was included.",
+ ),
+ ] {
+ route
+ .observe_answer("session-1", AnswerSummary { final_reply })
+ .await;
+ let recovery = session(vec![
+ Message::text(Role::User, "fix the build"),
+ Message::text(Role::Assistant, previous),
+ Message::text(Role::User, nudge),
+ ]);
+ assert_eq!(
+ drive(&route, recovery, calls.clone(), complete_call, 0.0).await?,
+ "capable"
+ );
+ }
+ assert_eq!(calls.0.load(Ordering::Relaxed), 2);
+ let next = initiated_by(
+ session(vec![
+ Message::text(Role::User, "fix the build"),
+ Message::text(Role::Assistant, "Let me run the tests."),
+ Message::text(Role::User, "now do another task"),
+ ]),
+ "user",
+ );
+ assert_eq!(
+ drive(&route, next, calls.clone(), complete_call, 0.0).await?,
+ "efficient"
+ );
+ assert_eq!(calls.0.load(Ordering::Relaxed), 3);
+ Ok(())
+}
+
+#[test]
+fn only_a_paused_reply_to_a_reported_final_answer_releases_the_latch() {
+ let messages = [
+ Message::text(Role::User, "fix the build"),
+ Message::text(Role::Assistant, "fixed"),
+ Message::text(Role::User, "now add a test"),
+ ];
+ let now = unix_seconds();
+ for (reported, releases) in [
+ (None, true),
+ (Some(0), false),
+ (Some(now), false),
+ (Some(now - FOLLOW_UP_QUIET_SECS), true),
+ ] {
+ let mut state = State::default();
+ if let Some(finished) = reported {
+ state
+ .extra
+ .insert(LAST_ANSWER_FINAL_KEY.into(), StateValue::Count(finished));
+ }
+ assert_eq!(
+ answers_final_reply(&state, &messages),
+ releases,
+ "{reported:?}"
+ );
+ }
+}
+
+#[tokio::test]
+async fn a_failed_capable_verifier_still_offers_the_efficient_answer() -> Result<()> {
+ use crate::core::algorithm::Call;
+ use crate::core::testing::serve_decision;
+
+ let algorithm = route(|config| config.task_typing = true)?;
+ let task = || session(vec![Message::text(Role::User, "What is 1+1?")]);
+ // Exercise production error propagation through CallModel::respond.
+ let outcome = crate::drive(
+ algorithm,
+ task(),
+ Arc::new(category_models(Category::Any, &["efficient", "capable"]).into()),
+ |call| async move {
+ match call {
+ Call::Model(call) => {
+ let target = call.models[0].clone();
+ let readout = call
+ .request
+ .llm_request
+ .extensions
+ .fields
+ .contains_key("logprobs");
+ let response = match target.as_str() {
+ "efficient" => Ok(reply("2")),
+ "judge" if readout => Ok(scored(0.05)),
+ "judge" => Ok(reply("Answer")),
+ _ => Err(crate::LibsyError::client_call(
+ target,
+ LlmClientError::UpstreamHttp {
+ status: http::StatusCode::BAD_GATEWAY,
+ body: String::new(),
+ },
+ )),
+ };
+ call.respond(std::future::ready(response)).await
+ }
+ Call::Decision(call) => serve_decision(*call).await,
+ }
+ },
+ )
+ .await?;
+ assert_eq!(
+ outcome.selected_model_ids,
+ vec![ModelId::new("capable"), ModelId::new("efficient")]
+ );
+ let fallback = outcome
+ .fallback_request
+ .expect("an efficient fallback request");
+ assert_eq!(fallback.llm_request.messages, task().llm_request.messages);
+ Ok(())
+}
diff --git a/crates/libsy/src/core/algorithm.rs b/crates/libsy/src/core/algorithm.rs
index da6fd7cbd..ca90765c5 100644
--- a/crates/libsy/src/core/algorithm.rs
+++ b/crates/libsy/src/core/algorithm.rs
@@ -25,7 +25,8 @@ use tracing::Instrument;
/// [`switchyard_protocol::LlmResponse`] carries either a live
/// [`switchyard_protocol::LlmResponseStream`] or the terminal aggregate.
use switchyard_protocol::{
- Category, DecisionRequest, DecisionResponse, ModelId, Request, Response,
+ AggLlmResponse, Category, ContentBlock, DecisionRequest, DecisionResponse, ModelId, Request,
+ Response, StopReason,
};
use crate::{DriverError, LibsyError, Result, State, observability};
@@ -317,6 +318,32 @@ impl RoutingOutcome {
}
}
+/// What the host saw of the answer it served for a routed request.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub struct AnswerSummary {
+ /// The answer ended its turn with visible text and requested no tool call.
+ pub final_reply: bool,
+}
+
+impl AnswerSummary {
+ /// Summarizes a completed answer.
+ pub fn of(response: &AggLlmResponse) -> Self {
+ let final_reply = response.outputs.iter().all(|output| {
+ matches!(output.stop_reason, None | Some(StopReason::EndTurn))
+ && !output
+ .content
+ .iter()
+ .any(|block| matches!(block, ContentBlock::ToolCall(_)))
+ }) && response.outputs.iter().any(|output| {
+ output.role == switchyard_protocol::Role::Assistant
+ && output.content.iter().any(
+ |block| matches!(block, ContentBlock::Text { text } if !text.trim().is_empty()),
+ )
+ });
+ Self { final_reply }
+ }
+}
+
/// An algorithm's handle for requesting external work and awaiting the host's typed replies.
#[derive(Clone)]
pub struct Driver {
@@ -740,6 +767,17 @@ pub trait Algorithm: Send + Sync + 'static {
/// [`run_stream`](Self::run_stream) runs this method and exposes its work to the host.
async fn route(self: Arc<Self>, driver: Driver, request: Request) -> Result<RoutingOutcome>;
+ /// Whether the host should summarize served answers for [`Self::observe_answer`].
+ fn tracks_answers(&self) -> bool {
+ false
+ }
+
+ /// Records the answer the host served for a request in `session_id`.
+ ///
+ /// The host reports a streamed answer before forwarding its final chunk, so the
+ /// session's next request is routed with this answer already recorded.
+ async fn observe_answer(&self, _session_id: &str, _answer: AnswerSummary) {}
+
/// Expose work requests and the final outcome so the host can control execution.
///
/// Each call waits for its host reply. A run ends with one terminal item — [`Step::Done`] on
@@ -798,6 +836,27 @@ mod tests {
LlmResponse, LlmResponseChunk, completion_text, text_request, text_response,
};
+ #[test]
+ fn a_final_answer_has_text_and_no_unfinished_or_tool_output() {
+ let mut answer = text_response(None, "done");
+ assert!(AnswerSummary::of(&answer).final_reply);
+ answer.outputs[0].stop_reason = Some(StopReason::MaxTokens);
+ assert!(!AnswerSummary::of(&answer).final_reply);
+ answer.outputs[0].stop_reason = Some(StopReason::EndTurn);
+ let mut tool = answer.outputs[0].clone();
+ tool.content = vec![ContentBlock::ToolCall(switchyard_protocol::ToolCall {
+ id: "call".into(),
+ name: "bash".into(),
+ arguments: serde_json::json!({}),
+ })];
+ answer.outputs.push(tool);
+ assert!(!AnswerSummary::of(&answer).final_reply);
+ answer.outputs.pop();
+ answer.outputs[0].content = vec![ContentBlock::Text { text: " ".into() }];
+ assert!(!AnswerSummary::of(&answer).final_reply);
+ assert!(!AnswerSummary::of(&AggLlmResponse::default()).final_reply);
+ }
+
#[derive(Debug, thiserror::Error)]
#[error("{0}")]
struct TestError(&'static str);
diff --git a/crates/libsy/src/core/classifier.rs b/crates/libsy/src/core/classifier.rs
index 06acfae3a..b8169e62a 100644
--- a/crates/libsy/src/core/classifier.rs
+++ b/crates/libsy/src/core/classifier.rs
@@ -1,7 +1,7 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
-use crate::core::algorithm::Driver;
+use crate::core::algorithm::{AnswerSummary, Driver};
use crate::core::state::State;
use crate::{LibsyError, Result};
use async_trait::async_trait;
@@ -91,6 +91,14 @@ pub trait Classifier: Send + Sync {
request: &mut Request,
driver: &Driver,
) -> Result<(Classification, Option<Response>)>;
+
+ /// Whether a stateful composition should request served-answer reports.
+ fn tracks_answers(&self) -> bool {
+ false
+ }
+
+ /// Folds the answer the host served for this session into the composition's state.
+ fn observe_answer(&self, _state: &mut State, _answer: AnswerSummary) {}
}
#[cfg(test)]
diff --git a/crates/libsy/src/lib.rs b/crates/libsy/src/lib.rs
index f357e923e..6c5711555 100644
--- a/crates/libsy/src/lib.rs
+++ b/crates/libsy/src/lib.rs
@@ -6,8 +6,8 @@
mod core;
pub use core::algorithm::{
- Algorithm, Call, CallDecision, CallModel, Driver, RoutingOutcome, RuntimeModels, Step,
- StepStream, drive,
+ Algorithm, AnswerSummary, Call, CallDecision, CallModel, Driver, RoutingOutcome, RuntimeModels,
+ Step, StepStream, drive,
};
pub use core::classifier::{Classification, Classifier, Score};
pub use core::outcome_metadata::OutcomeMetadata;
diff --git a/docs/routing_algorithms/vgr_routing.md b/docs/routing_algorithms/vgr_routing.md
index d1d89f5de..64d856c42 100644
--- a/docs/routing_algorithms/vgr_routing.md
+++ b/docs/routing_algorithms/vgr_routing.md
@@ -51,9 +51,12 @@ discovery for those servers is not implemented.
The judge reads its verdicts from the first token's logprobs, so its scored calls
turn thinking off: they send `reasoning_effort = "none"` and
`chat_template_kwargs = {"enable_thinking": false}` on `openai_chat` targets.
-Candidates and the judge's deliberation keep the target's own setting. A backend
-that rejects `chat_template_kwargs` can drop it with the target's
-`omit_body_fields`.
+Candidates and the judge's deliberation keep the target's own setting. If a Chat
+backend explicitly rejects this template switch with HTTP 400 or 422, the SDK
+retries once without it, within the same deadline. Other request fields and the
+configured transient retry budget are preserved. The target's `omit_body_fields`
+can omit the switch from the first call. Custom hosts must handle provider
+compatibility themselves.
The server caches the model catalog, including unavailable discovery results,
for one minute. Discovery probes run concurrently with a two-second total budget
@@ -130,6 +133,25 @@ automatic continuations. Retries retain the original call's tag. Without a
recognized tag, Switchyard uses the message shape to identify a user turn.
`x-session-id` is also accepted as a session-header alias.
+## Served-answer reporting
+
+VGR opts into served-answer reporting through `Algorithm::tracks_answers`.
+The SDK summarizes answers only for opted-in algorithms with a session ID;
+other routers keep their responses and streams unchanged. Custom algorithms and
+classifiers default to no tracking. Stateful `FallThrough` routes opt in when a
+classifier requests reports.
+
+Custom libsy hosts can check `tracks_answers()` and report
+`AnswerSummary::of(&response)` through `Algorithm::observe_answer` for aggregate
+responses. For streams, report `final_reply = false` when serving begins and
+report the final summary before forwarding the terminal chunk. Use the request's
+session ID; requests without one retain the existing behavior.
+
+Without a recognized `x-initiator` header, reported incomplete answers and
+immediate recovery nudges retain the turn hold and budget. A reply to a complete
+final answer releases them after a two-second quiet period. Explicit user/agent
+headers take precedence. Hosts that omit reporting retain message-shape detection.
+
## Verification and capable-tier outages
Policy version `1.2` allows the default verification branch to accept a clean