Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
feat(relay): add opt-in classifier and judge payload capture (#1197)
Alex Fournier4a71c01e516c
diff --git a/crates/libsy-llm-client/src/client.rs b/crates/libsy-llm-client/src/client.rsindex ef56b2706..deb4d9a12 100644--- a/crates/libsy-llm-client/src/client.rs+++ b/crates/libsy-llm-client/src/client.rs@@ -548,7 +548,15 @@ impl TranslatingLlmClient {let builder = apply_extra_headers(builder, backend);let builder = backend.apply_auth(builder);- let response = match builder.send().await {+ let response = match async {+ let request = builder.build()?;+ if let Some(body) = request.body().and_then(reqwest::Body::as_bytes) {+ crate::payload_capture::record("request", || String::from_utf8_lossy(body));+ }+ client.execute(request).await+ }+ .await+ {Ok(response) => response,Err(error) => {metrics::record_upstream_attempt(None);@@ -594,6 +602,7 @@ impl TranslatingLlmClient {}};metrics::record_upstream_attempt(Some(status.as_u16()));+ crate::payload_capture::record("response", || String::from_utf8_lossy(&body));return Ok(EncodedResponse::Buffered {status: status.as_u16(),body: body.to_vec(),@@ -1817,6 +1826,14 @@ mod tests {assert!(!convert_reqwest_error(error).to_string().contains("CANARY"));}+ #[tokio::test]+ async fn captured_request_build_failure_remains_a_configuration_error() {+ let client = TranslatingLlmClient::new(&chat_map("://invalid")).unwrap();+ let result =+ crate::with_payload_capture(client.call(request_for(Some("gpt"), false))).await;+ assert!(matches!(result, Err(LlmClientError::Configuration { .. })));+ }+#[test]fn nvidia_anthropic_tools_drop_unsupported_strict_field() {let backend = Backend::Anthropic(config("https://inference-api.nvidia.com"));diff --git a/crates/libsy-llm-client/src/lib.rs b/crates/libsy-llm-client/src/lib.rsindex 5bc416bbf..3710ad150 100644--- a/crates/libsy-llm-client/src/lib.rs+++ b/crates/libsy-llm-client/src/lib.rs@@ -28,6 +28,7 @@ mod observability;mod observation;mod observed_translation;mod openai_decisions;+mod payload_capture;pub mod raw;mod responses_reasoning;pub mod run;@@ -38,6 +39,7 @@ pub use client::{AuxiliaryOperation, ModelConfig, TranslatingLlmClient};pub use error::{LlmClientError, Result};pub use observation::{ModelCallObservation, RunObservation, RunObserver};pub use openai_decisions::OpenAiDecisionClient;+pub use payload_capture::with_payload_capture;pub use raw::RawResponse;pub use responses_reasoning::ResponsesReasoningPolicy;pub use run::{ClientRouter, decide, run};diff --git a/crates/libsy-llm-client/src/payload_capture.rs b/crates/libsy-llm-client/src/payload_capture.rsnew file mode 100644index 000000000..e1c020eca--- /dev/null+++ b/crates/libsy-llm-client/src/payload_capture.rs@@ -0,0 +1,86 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++//! Explicit opt-in for HTTP body capture by a host's tracing subscriber.++use std::fmt;+use std::future::Future;++tokio::task_local! {+ static CAPTURE_ENABLED: ();+}++/// Enables raw HTTP body events while polling `future` on the current task.+///+/// The host must install a subscriber for `switchyard.payload` and apply its+/// own filtering, size limits, and redaction. Spawned tasks do not inherit capture.+pub async fn with_payload_capture<T>(future: impl Future<Output = T>) -> T {+ CAPTURE_ENABLED.scope((), future).await+}++pub(crate) fn record<D: fmt::Display>(phase: &'static str, body: impl FnOnce() -> D) {+ if CAPTURE_ENABLED.try_with(|_| ()).is_ok() {+ tracing::trace!(target: "switchyard.payload", phase, body = %body());+ }+}++#[cfg(test)]+mod tests {+ use std::sync::Arc;++ use parking_lot::Mutex;+ use tracing::field::Field;+ use tracing::instrument::WithSubscriber;+ use tracing_subscriber::Layer;+ use tracing_subscriber::layer::{Context, SubscriberExt};++ use super::*;++ struct Recorder(Arc<Mutex<Vec<String>>>);++ impl<S: tracing::Subscriber> Layer<S> for Recorder {+ fn on_event(&self, event: &tracing::Event<'_>, _: Context<'_, S>) {+ event.record(&mut |field: &Field, value: &dyn fmt::Debug| {+ if field.name() == "body" {+ self.0.lock().push(format!("{value:?}"));+ }+ });+ }+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn capture_requires_opt_in_and_does_not_leak_between_tasks_or_after_cancellation() {+ let events = Arc::new(Mutex::new(Vec::new()));+ let subscriber = tracing_subscriber::registry().with(Recorder(Arc::clone(&events)));+ async {+ record("request", || -> &str {+ panic!("disabled capture must not evaluate the body")+ });+ tokio::join!(+ with_payload_capture(async {+ tokio::spawn(+ async {+ record("request", || "spawned");+ }+ .with_current_subscriber(),+ )+ .await+ .unwrap();+ tokio::task::yield_now().await;+ record("request", || "enabled");+ }),+ async {+ tokio::task::yield_now().await;+ record("request", || "other-task");+ },+ );+ let mut pending = Box::pin(with_payload_capture(std::future::pending::<()>()));+ assert!(futures::poll!(&mut pending).is_pending());+ drop(pending);+ record("response", || "after-cancellation");+ }+ .with_subscriber(subscriber)+ .await;+ assert_eq!(*events.lock(), ["enabled"]);+ }+}diff --git a/crates/switchyard-nemo-relay-plugin/README.md b/crates/switchyard-nemo-relay-plugin/README.mdindex 34fa3dbab..69b8c46bb 100644--- a/crates/switchyard-nemo-relay-plugin/README.md+++ b/crates/switchyard-nemo-relay-plugin/README.md@@ -234,8 +234,12 @@ retains ownership of the outer LLM lifecycle.The plugin also captures the existing `libsy.client_call` and`libsy.upstream_attempt` spans for each routed request. It replays completedspans beneath Relay's outer LLM scope with their original timestamps. The-projection includes allowlisted, payload-free routing metadata only; it excludes-prompts, responses, provider error bodies, and credentials.+projection contains allowlisted routing metadata by default. To inspect the+final classifier/judge request and buffered response, enable+[routing payload capture](../../docs/integrations/nemo_relay.md#inspect-classifier-and-judge-payloads)+in the plugin configuration. Capture uses Relay's existing exporters and writes+no payload files inside the plugin. It is disabled by default and omits bodies+above the configured size limit.## Observabilitydiff --git a/crates/switchyard-nemo-relay-plugin/config.schema.json b/crates/switchyard-nemo-relay-plugin/config.schema.jsonindex 61f178a12..4c368edae 100644--- a/crates/switchyard-nemo-relay-plugin/config.schema.json+++ b/crates/switchyard-nemo-relay-plugin/config.schema.json@@ -9,6 +9,17 @@{ "required": ["switchyard_config"] }],"properties": {+ "capture_routing_payloads": {+ "type": "boolean",+ "default": false,+ "description": "Opt in to bounded routing request and buffered response bodies for debugging. Payloads may contain sensitive task context."+ },+ "max_routing_payload_bytes": {+ "type": "integer",+ "minimum": 1,+ "default": 262144,+ "description": "Maximum encoded bytes per routing request or buffered response, per attempt. Larger bodies are omitted entirely with body=null and truncated=true."+ },"priority": {"type": "integer","default": 0,diff --git a/crates/switchyard-nemo-relay-plugin/src/config.rs b/crates/switchyard-nemo-relay-plugin/src/config.rsindex 2d8db081f..912976a8f 100644--- a/crates/switchyard-nemo-relay-plugin/src/config.rs+++ b/crates/switchyard-nemo-relay-plugin/src/config.rs@@ -8,6 +8,12 @@ use serde_json::{Map, Value};use switchyard_protocol::WireFormat;use switchyard_runner::Runner;+pub(crate) const DEFAULT_ROUTING_PAYLOAD_BYTES: usize = 256 * 1024;++fn default_routing_payload_bytes() -> usize {+ DEFAULT_ROUTING_PAYLOAD_BYTES+}+pub(crate) fn protocol_from_call(name: &str) -> Option<WireFormat> {match name {"openai.chat_completions" => Some(WireFormat::OpenAiChat),@@ -20,6 +26,11 @@ pub(crate) fn protocol_from_call(name: &str) -> Option<WireFormat> {#[derive(Debug, Deserialize)]#[serde(deny_unknown_fields)]pub(crate) struct SwitchyardConfig {+ /// Includes bounded final routing HTTP bodies in Relay scopes when explicitly enabled.+ #[serde(default)]+ pub(crate) capture_routing_payloads: bool,+ #[serde(default = "default_routing_payload_bytes")]+ pub(crate) max_routing_payload_bytes: usize,#[serde(default)]pub(crate) priority: i32,#[serde(default)]@@ -30,6 +41,9 @@ pub(crate) struct SwitchyardConfig {impl SwitchyardConfig {pub(crate) fn load_runner(&self) -> Result<Runner, String> {+ if self.max_routing_payload_bytes == 0 {+ return Err("max_routing_payload_bytes must be positive".into());+ }let runner = match (&self.switchyard_config_path, &self.switchyard_config) {(Some(path), None) => Runner::load(path).map_err(|error| error.to_string()),(None, Some(config)) => toml::to_string(config)@@ -116,6 +130,8 @@ mod tests {#[test]fn builds_runner_from_inline_deployment() {let config = SwitchyardConfig {+ capture_routing_payloads: false,+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,priority: 0,switchyard_config_path: None,switchyard_config: Some(inline_deployment()),@@ -125,6 +141,27 @@ mod tests {assert!(runner.route("switchyard/default").is_some());}+ #[test]+ fn payload_capture_defaults_off_and_requires_a_positive_limit() {+ let mut config: SwitchyardConfig = serde_json::from_value(json!({+ "switchyard_config": inline_deployment(),+ }))+ .unwrap();+ assert!(!config.capture_routing_payloads);+ assert_eq!(config.max_routing_payload_bytes, 256 * 1024);+ let schema: Value = serde_json::from_str(include_str!("../config.schema.json")).unwrap();+ assert_eq!(+ schema["properties"]["max_routing_payload_bytes"]["default"],+ config.max_routing_payload_bytes+ );+ config.max_routing_payload_bytes = 0;+ assert!(+ matches!(config.load_runner(), Err(error) if error.contains("max_routing_payload_bytes"))+ );+ config.max_routing_payload_bytes = 2 * 1024 * 1024;+ assert!(config.load_runner().is_ok());+ }+#[test]fn builds_runner_from_deployment_path() {let mut deployment = NamedTempFile::new().unwrap();@@ -135,6 +172,8 @@ mod tests {).unwrap();let config = SwitchyardConfig {+ capture_routing_payloads: false,+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,priority: 0,switchyard_config_path: Some(deployment.path().to_path_buf()),switchyard_config: None,@@ -244,6 +283,8 @@ mod tests {#[test]fn requires_exactly_one_configuration_source() {let neither = SwitchyardConfig {+ capture_routing_payloads: false,+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,priority: 0,switchyard_config_path: None,switchyard_config: None,@@ -254,6 +295,8 @@ mod tests {));let both = SwitchyardConfig {+ capture_routing_payloads: false,+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,priority: 0,switchyard_config_path: Some(PathBuf::from("routes.toml")),switchyard_config: Some(inline_deployment()),diff --git a/crates/switchyard-nemo-relay-plugin/src/lib.rs b/crates/switchyard-nemo-relay-plugin/src/lib.rsindex 6f0c54311..b8ca6244d 100644--- a/crates/switchyard-nemo-relay-plugin/src/lib.rs+++ b/crates/switchyard-nemo-relay-plugin/src/lib.rs@@ -50,6 +50,9 @@ impl NativePlugin for SwitchyardPlugin {ctx: &mut PluginContext<'_>,) -> nemo_relay_plugin::Result<()> {let config = parse_config(plugin_config)?;+ let payload_limit = config+ .capture_routing_payloads+ .then_some(config.max_routing_payload_bytes);let priority = config.priority;let runtime = Arc::new(SwitchyardRuntime::new(config)?);let plugin_runtime = ctx.runtime();@@ -59,7 +62,7 @@ impl NativePlugin for SwitchyardPlugin {});let span_runtime = Arc::clone(&runtime);let span_emit_event = Arc::clone(&emit_event);- let span_capture = SpanCapture::new(move |mut span| {+ let span_capture = SpanCapture::new(payload_limit, move |mut span| {span.sanitize(span_runtime.redactor());span_emit_event(RoutingEvent::Span(span));});diff --git a/crates/switchyard-nemo-relay-plugin/src/redaction_tests.rs b/crates/switchyard-nemo-relay-plugin/src/redaction_tests.rsindex d3a42c72e..bfc86b7f3 100644--- a/crates/switchyard-nemo-relay-plugin/src/redaction_tests.rs+++ b/crates/switchyard-nemo-relay-plugin/src/redaction_tests.rs@@ -105,6 +105,8 @@ async fn reflected_response(format: WireFormat, streaming: bool, fail: bool) {"routes": {"default": {"id": "switchyard/default", "type": "passthrough", "target": "default"}}});let runtime = SwitchyardRuntime::new(SwitchyardConfig {+ capture_routing_payloads: false,+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,priority: 0,switchyard_config_path: None,switchyard_config: Some(deployment.as_object().unwrap().clone()),@@ -126,7 +128,7 @@ async fn reflected_response(format: WireFormat, streaming: bool, fail: bool) {let captured = Arc::new(Mutex::new(Vec::new()));let span_observed = Arc::clone(&captured);let redactor = Arc::clone(&runtime.redactor);- let span_capture = SpanCapture::new(move |mut span| {+ let span_capture = SpanCapture::new(None, move |mut span| {span.sanitize(&redactor);span_observed.lock().unwrap().push(RoutingEvent::Span(span));});diff --git a/crates/switchyard-nemo-relay-plugin/src/runtime.rs b/crates/switchyard-nemo-relay-plugin/src/runtime.rsindex d708ef7b5..fb0f4048d 100644--- a/crates/switchyard-nemo-relay-plugin/src/runtime.rs+++ b/crates/switchyard-nemo-relay-plugin/src/runtime.rs@@ -950,6 +950,8 @@ mod tests {}});SwitchyardRuntime::new(crate::config::SwitchyardConfig {+ capture_routing_payloads: false,+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,priority: 0,switchyard_config_path: None,switchyard_config: Some(deployment.as_object().unwrap().clone()),@@ -1104,6 +1106,8 @@ mod tests {},});let runtime = SwitchyardRuntime::new(crate::config::SwitchyardConfig {+ capture_routing_payloads: false,+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,priority: 0,switchyard_config_path: None,switchyard_config: Some(diff --git a/crates/switchyard-nemo-relay-plugin/src/span_capture.rs b/crates/switchyard-nemo-relay-plugin/src/span_capture.rsindex 31bf477b8..59db88c4b 100644--- a/crates/switchyard-nemo-relay-plugin/src/span_capture.rs+++ b/crates/switchyard-nemo-relay-plugin/src/span_capture.rs@@ -9,10 +9,11 @@ use std::time::{Instant, SystemTime};use nemo_relay_plugin::{Json, PluginRuntime, ScopeType};use serde_json::{Map, Value};+use switchyard_llm_client::with_payload_capture;use switchyard_runner::ProviderKeyRedactor;use tracing::field::{Field, Visit};use tracing::span::{Attributes, Id, Record};-use tracing::{Metadata, Subscriber};+use tracing::{Event, Metadata, Subscriber};use tracing_subscriber::Layer;use tracing_subscriber::filter::{LevelFilter, filter_fn};use tracing_subscriber::layer::Context;@@ -21,13 +22,22 @@ use tracing_subscriber::registry::LookupSpan;const CLIENT_CALL: &str = "libsy.client_call";const UPSTREAM_ATTEMPT: &str = "libsy.upstream_attempt";+const PAYLOAD_TARGET: &str = "switchyard.payload";+const MAX_PAYLOAD_REDACTION_DEPTH: usize = 64;type SpanEmitter = Box<dyn Fn(CapturedSpan) + Send + Sync>;+#[derive(Debug, Default)]+struct Payloads {+ request: Option<Json>,+ response: Option<Json>,+}+#[derive(Debug)]pub(crate) struct CapturedSpan {name: &'static str,fields: Map<String, Value>,+ payloads: Payloads,started_at: SystemTime,ended_at: SystemTime,children: Vec<CapturedSpan>,@@ -36,6 +46,7 @@ pub(crate) struct CapturedSpan {struct OpenSpan {name: &'static str,fields: Map<String, Value>,+ payloads: Payloads,started_at: SystemTime,started: Instant,children: Vec<CapturedSpan>,@@ -43,12 +54,55 @@ struct OpenSpan {struct CaptureLayer {emit: SpanEmitter,+ payload_limit: Option<usize>,}impl<S> Layer<S> for CaptureLayerwhereS: Subscriber + for<'lookup> LookupSpan<'lookup>,{+ fn on_event(&self, event: &Event<'_>, ctx: Context<'_, S>) {+ if event.metadata().target() != PAYLOAD_TARGET {+ return;+ }+ let Some(limit) = self.payload_limit else {+ return;+ };+ let Some(span) = ctx.event_span(event) else {+ return;+ };+ let is_routing = span.scope().any(|ancestor| {+ ancestor.extensions().get::<OpenSpan>().is_some_and(|open| {+ open.name == CLIENT_CALL+ && open+ .fields+ .get("switchyard.call_phase")+ .and_then(Value::as_str)+ == Some("routing")+ })+ });+ if !is_routing || span.name() != UPSTREAM_ATTEMPT {+ return;+ }+ let mut payload = PayloadVisitor {+ phase: None,+ body: BoundedText {+ text: String::new(),+ truncated: false,+ limit,+ },+ };+ event.record(&mut payload);+ if let Some(open) = span.extensions_mut().get_mut::<OpenSpan>() {+ let data = payload.body.into_payload();+ match payload.phase {+ Some(PayloadPhase::Request) => open.payloads.request = Some(data),+ Some(PayloadPhase::Response) => open.payloads.response = Some(data),+ None => (),+ }+ }+ }+fn on_new_span(&self, attrs: &Attributes<'_>, id: &Id, ctx: Context<'_, S>) {let name = attrs.metadata().name();let mut fields = Map::new();@@ -57,6 +111,7 @@ wherespan.extensions_mut().insert(OpenSpan {name,fields,+ payloads: Payloads::default(),started_at: SystemTime::now(),started: Instant::now(),children: Vec::new(),@@ -85,6 +140,7 @@ wherelet OpenSpan {name,mut fields,+ payloads,started_at,started,children,@@ -96,6 +152,7 @@ wherelet captured = CapturedSpan {name,fields,+ payloads,started_at,ended_at: started_at.checked_add(started.elapsed())@@ -120,6 +177,23 @@ impl CapturedSpan {{self.fields = fields;}+ for payload in [&mut self.payloads.request, &mut self.payloads.response]+ .into_iter()+ .flatten()+ {+ if !redactor.is_empty()+ && let Some(body) = payload.get_mut("body")+ {+ *body = body+ .as_str()+ .and_then(|text| serde_json::from_str::<Json>(text).ok())+ .and_then(|decoded| {+ redact_captured_json(decoded, redactor, MAX_PAYLOAD_REDACTION_DEPTH)+ })+ .map(|decoded| Json::String(decoded.to_string()))+ .unwrap_or(Json::Null);+ }+ }for child in &mut self.children {child.sanitize(redactor);}@@ -127,20 +201,24 @@ impl CapturedSpan {pub(crate) fn emit(self, runtime: &PluginRuntime) -> Result<(), String> {let mut metadata = Json::Object(self.fields);- let output = if self.name == CLIENT_CALL {- response_data(&metadata)- } else {- None- };+ let output = self.payloads.response.or_else(|| {+ if self.name == CLIENT_CALL {+ response_data(&metadata)+ } else {+ None+ }+ });if let Some(session_id) = metadata.get("gen_ai.conversation.id").cloned() {metadata["session_id"] = session_id;}if let Some(status) = span_status(&metadata) {metadata["otel.status_code"] = status["otel.status_code"].clone();}- let input = metadata- .get("gen_ai.request.model")- .map(|model| serde_json::json!({"model": model}));+ let input = self.payloads.request.or_else(|| {+ metadata+ .get("gen_ai.request.model")+ .map(|model| serde_json::json!({"model": model}))+ });let mut scope = runtime.scope_at(self.name,relay_scope_type(self.name),@@ -158,6 +236,50 @@ impl CapturedSpan {}}+fn redact_captured_json(value: Json, redactor: &ProviderKeyRedactor, depth: usize) -> Option<Json> {+ let depth = depth.checked_sub(1)?;+ Some(match value {+ Json::String(text) => Json::String(redact_captured_text(text, redactor, depth)?),+ Json::Array(values) => Json::Array(+ values+ .into_iter()+ .map(|value| redact_captured_json(value, redactor, depth))+ .collect::<Option<_>>()?,+ ),+ Json::Object(fields) => Json::Object(+ fields+ .into_iter()+ .map(|(name, value)| {+ Some((+ redact_captured_text(name, redactor, depth)?,+ redact_captured_json(value, redactor, depth)?,+ ))+ })+ .collect::<Option<_>>()?,+ ),+ value => value,+ })+}++fn redact_captured_text(+ text: String,+ redactor: &ProviderKeyRedactor,+ depth: usize,+) -> Option<String> {+ // The existing redaction marker is plain text, not a JSON array.+ if text.trim() != "[REDACTED]"+ && matches!(+ text.trim_start().as_bytes().first(),+ Some(b'{' | b'[' | b'"')+ )+ {+ let decoded = serde_json::from_str::<Json>(&text).ok()?;+ Some(redact_captured_json(decoded, redactor, depth)?.to_string())+ } else {+ Some(redactor.text(text))+ }+}+/// Reconstructs the bounded response metadata forwarded to NeMo Relay.fn response_data(fields: &Json) -> Option<Json> {let mut output = Map::new();@@ -229,11 +351,25 @@ pub(crate) struct SpanCapture {}impl SpanCapture {- pub(crate) fn new(emit: impl Fn(CapturedSpan) + Send + Sync + 'static) -> Self {+ pub(crate) fn new(+ payload_limit: Option<usize>,+ emit: impl Fn(CapturedSpan) + Send + Sync + 'static,+ ) -> Self {let layer = CaptureLayer {emit: Box::new(emit),+ payload_limit,}- .with_filter(filter_fn(captures).with_max_level_hint(LevelFilter::DEBUG));+ .with_filter(+ filter_fn(move |metadata| {+ captures(metadata)+ || (payload_limit.is_some() && metadata.target() == PAYLOAD_TARGET)+ })+ .with_max_level_hint(if payload_limit.is_some() {+ LevelFilter::TRACE+ } else {+ LevelFilter::DEBUG+ }),+ );Self {dispatch: tracing::Dispatch::new(tracing_subscriber::registry().with(layer)),}@@ -241,14 +377,70 @@ impl SpanCapture {/// Polls a future with the capture subscriber as its default.pub(crate) async fn capture<T>(&self, future: impl Future<Output = T>) -> T {- CaptureFuture {+ with_payload_capture(CaptureFuture {inner: Some(Box::pin(future)),dispatch: self.dispatch.clone(),- }+ }).await}}+struct PayloadVisitor {+ phase: Option<PayloadPhase>,+ body: BoundedText,+}++enum PayloadPhase {+ Request,+ Response,+}++impl Visit for PayloadVisitor {+ fn record_str(&mut self, field: &Field, value: &str) {+ if field.name() == "phase" {+ self.phase = match value {+ "request" => Some(PayloadPhase::Request),+ "response" => Some(PayloadPhase::Response),+ _ => None,+ };+ }+ }++ fn record_debug(&mut self, field: &Field, value: &dyn fmt::Debug) {+ if field.name() == "body" {+ let _ = fmt::write(&mut self.body, format_args!("{value:?}"));+ }+ }+}++struct BoundedText {+ text: String,+ truncated: bool,+ limit: usize,+}++impl BoundedText {+ fn into_payload(self) -> Json {+ // A retained prefix could end inside a credential before redaction runs.+ let mut payload = serde_json::json!({"body": null, "truncated": self.truncated});+ if !self.truncated {+ payload["body"] = self.text.into();+ }+ payload+ }+}++impl fmt::Write for BoundedText {+ fn write_str(&mut self, value: &str) -> fmt::Result {+ if value.len() > self.limit - self.text.len() {+ self.truncated = true;+ return Err(fmt::Error);+ }+ self.text.push_str(value);+ Ok(())+ }+}+// Standard subscriber wrappers restore the ambient dispatch before drop. This// wrapper also drops the inner future under capture so cancelled spans close here.struct CaptureFuture<F> {@@ -386,6 +578,256 @@ mod tests {use super::*;+ // Real loopback HTTP proves capture sees the encoded judge body, not the outer task.+ #[tokio::test]+ async fn captures_final_judge_payload_only_when_enabled() {+ use crate::{config::SwitchyardConfig, runtime::SwitchyardRuntime};+ use switchyard_protocol::{Request, WireFormat, text_request};+ use wiremock::matchers::{body_partial_json, method};+ use wiremock::{Mock, MockServer, ResponseTemplate};++ let server = MockServer::start().await;+ for (model, content) in [+ ("judge/model", r#"{"target":"efficient"}"#),+ ("answer/model", "answer"),+ ] {+ let response = serde_json::json!({+ "id": "smoke", "model": model,+ "choices": [{"index": 0, "message": {"role": "assistant", "content": content}, "finish_reason": "stop"}],+ "debug_canary": "final-body-only", "final-body-only": "ordinary diagnostics"+ })+ .to_string()+ .replace("final-body-only", r"\u0066inal-body-only");+ Mock::given(method("POST"))+ .and(body_partial_json(serde_json::json!({"model": model})))+ .respond_with(+ ResponseTemplate::new(200)+ .insert_header("content-type", "application/json")+ .set_body_string(response),+ )+ .mount(&server)+ .await;+ }+ let config: SwitchyardConfig = serde_json::from_value(serde_json::json!({+ "switchyard_config": {+ "schema_version": 1,+ "llm_clients": {"local": {"format": "openai_chat", "base_url": format!("{}/v1", server.uri())}},+ "targets": {+ "judge": {"id": "judge/model", "llm_client": "local", "extra_body": {"debug_canary": "final-body-only"}},+ "answer": {"id": "answer/model", "llm_client": "local"}+ },+ "routes": {"test": {+ "id": "switchyard/test", "type": "llm_classifier", "mode": "custom",+ "models": {"judge": ["judge"], "efficient": ["answer"], "any": ["answer"]},+ "default_target": "efficient", "prompt": "Judge task complexity.",+ "response_schema": r#"{"type":"object","properties":{"target":{"type":"string","enum":["efficient"]}},"required":["target"],"additionalProperties":false}"#,+ "policy": {"type": "target_selector", "selector": "/target"}+ }}+ }+ })).unwrap();+ let runtime = SwitchyardRuntime::new(config).unwrap();+ for enabled in [false, true] {+ let (sender, spans) = mpsc::channel();+ let capture = SpanCapture::new(+ enabled.then_some(crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES),+ move |span| sender.send(span).unwrap(),+ );+ let execution = capture+ .capture(runtime.execute_buffered(+ WireFormat::OpenAiChat,+ Request {+ llm_request: text_request(+ Some("switchyard/test".into()),+ "Fix a failing Rust test.\nPreserve \"雪\" and \\ exactly.",+ ),+ raw_request: None,+ metadata: None,+ },+ ))+ .await;+ assert!(execution.result.is_ok(), "{:?}", execution.result);+ let received = server.received_requests().await.unwrap();+ let mut spans: Vec<_> = spans.try_iter().collect();+ let judge = spans+ .iter_mut()+ .find(|span| span.fields["selected_model"] == "judge/model")+ .unwrap();+ assert_eq!(judge.fields["switchyard.call_phase"], "routing");+ assert_eq!(judge.children.len(), 1);+ let attempt = &mut judge.children[0];+ assert_eq!(attempt.payloads.request.is_some(), enabled);+ assert_eq!(attempt.payloads.response.is_some(), enabled);+ if enabled {+ let actual = received+ .into_iter()+ .rev()+ .find(|request| request.body_json::<Value>().unwrap()["model"] == "judge/model")+ .unwrap();+ let captured_body = attempt.payloads.request.as_ref().unwrap()["body"]+ .as_str()+ .unwrap();+ assert_eq!(captured_body.as_bytes(), actual.body);+ let captured: Value = serde_json::from_str(captured_body).unwrap();+ assert_eq!(captured["debug_canary"], "final-body-only");+ assert_eq!(captured["messages"][0]["content"], "Judge task complexity.");+ assert_eq!(+ attempt.payloads.request.as_ref().unwrap()["truncated"],+ false+ );+ assert!(+ attempt.payloads.response.as_ref().unwrap()["body"]+ .as_str()+ .unwrap()+ .contains("efficient")+ );+ judge.sanitize(&ProviderKeyRedactor::new(&["final-body-only".into()]));+ assert!(!format!("{judge:?}").contains("final-body-only"));+ for payload in [+ &judge.children[0].payloads.request,+ &judge.children[0].payloads.response,+ ] {+ let decoded: Value =+ serde_json::from_str(payload.as_ref().unwrap()["body"].as_str().unwrap())+ .unwrap();+ assert_eq!(decoded["debug_canary"], "[REDACTED]");+ assert!(!decoded.to_string().contains("final-body-only"));+ }+ }+ let answer = spans+ .iter()+ .find(|span| span.fields["selected_model"] == "answer/model")+ .unwrap();+ for attempt in &answer.children {+ assert!(attempt.payloads.request.is_none());+ assert!(attempt.payloads.response.is_none());+ }+ }+ }++ #[test]+ fn captured_classifier_content_and_tool_arguments_redact_nested_json() {+ let nested = r#"{"credential":"qa-\u0073ecret","qa-\u0073ecret":"ordinary"}"#;+ let request = serde_json::json!({"messages": [{"role": "assistant", "tool_calls": [{+ "function": {"name": "lookup", "arguments": nested}+ }]}]});+ let response = serde_json::json!({"choices": [{"message": {+ "role": "assistant", "content": nested+ }}], "ordinary": {"text": "Keep ordinary diagnostics", "number": 3, "flag": true, "missing": null}});+ let mut span = span_with_payloads(Payloads {+ request: Some(serde_json::json!({"body": request.to_string(), "truncated": false})),+ response: Some(serde_json::json!({"body": response.to_string(), "truncated": false})),+ });+ span.sanitize(&ProviderKeyRedactor::new(&["qa-secret".into()]));+ span.sanitize(&ProviderKeyRedactor::new(&["qa-secret".into()]));+ for (payload, path) in [+ (+ &span.payloads.request,+ "/messages/0/tool_calls/0/function/arguments",+ ),+ (&span.payloads.response, "/choices/0/message/content"),+ ] {+ let body: Json =+ serde_json::from_str(payload.as_ref().unwrap()["body"].as_str().unwrap()).unwrap();+ let decoded: Json =+ serde_json::from_str(body.pointer(path).unwrap().as_str().unwrap()).unwrap();+ assert_eq!(+ decoded,+ serde_json::json!({+ "credential": "[REDACTED]", "[REDACTED]": "ordinary"+ })+ );+ }+ let response_body: Json = serde_json::from_str(+ span.payloads.response.as_ref().unwrap()["body"]+ .as_str()+ .unwrap(),+ )+ .unwrap();+ assert_eq!(response_body["ordinary"], response["ordinary"]);+ }++ #[test]+ fn captured_payload_redaction_bounds_structural_and_encoded_depth() {+ let redactor = ProviderKeyRedactor::new(&["qa-secret".into()]);+ assert!(+ redact_captured_json(+ serde_json::json!({"content": r#"{"credential":"qa-\u0073ecret""#}),+ &redactor,+ MAX_PAYLOAD_REDACTION_DEPTH,+ )+ .is_none()+ );+ for (arrays, encoded_strings, omitted) in [+ (MAX_PAYLOAD_REDACTION_DEPTH - 1, 0, false),+ (MAX_PAYLOAD_REDACTION_DEPTH, 0, true),+ (MAX_PAYLOAD_REDACTION_DEPTH - 8, 7, false),+ (MAX_PAYLOAD_REDACTION_DEPTH - 8, 8, true),+ ] {+ let mut body = Json::String("qa-secret".into());+ for _ in 0..encoded_strings {+ body = Json::String(body.to_string());+ }+ for _ in 0..arrays {+ body = Json::Array(vec![body]);+ }+ let mut span = span_with_payloads(Payloads {+ request: Some(serde_json::json!({"body": body.to_string(), "truncated": false})),+ response: None,+ });+ span.sanitize(&redactor);+ let payload = span.payloads.request.unwrap();+ assert_eq!(payload["body"].is_null(), omitted);+ if !omitted {+ assert!(payload["body"].as_str().unwrap().contains("[REDACTED]"));+ assert!(!payload["body"].as_str().unwrap().contains("qa-secret"));+ }+ }+ }++ #[test]+ fn payload_sanitization_omits_undecodable_json_when_keys_are_configured() {+ let body = r#"invalid JSON with qa-\u0073ecret"#;+ let mut span = span_with_payloads(Payloads {+ request: Some(serde_json::json!({"body": body, "truncated": false})),+ response: Some(serde_json::json!({"body": null, "truncated": true})),+ });+ span.sanitize(&ProviderKeyRedactor::default());+ assert_eq!(span.payloads.request.as_ref().unwrap()["body"], body);+ span.sanitize(&ProviderKeyRedactor::new(&["qa-secret".into()]));+ assert_eq!(+ span.payloads.request,+ Some(serde_json::json!({"body": null, "truncated": false}))+ );+ assert_eq!(+ span.payloads.response,+ Some(serde_json::json!({"body": null, "truncated": true}))+ );+ }++ #[test]+ fn payload_limit_preserves_complete_bodies_and_omits_oversized_ones() {+ use std::fmt::Write;+ for (parts, limit, expected) in [+ (["plain", " text"], 10, Some("plain text")),+ (["caf", "é"], 5, Some("café")),+ (["caf", "é"], 4, None),+ (["prefix-", "synthetic-secret"], 16, None),+ ] {+ let mut body = BoundedText {+ text: String::new(),+ truncated: false,+ limit,+ };+ let result = write!(&mut body, "{}{}", parts[0], parts[1]);+ assert_eq!(result.is_err(), expected.is_none());+ assert!(body.text.len() <= limit);+ assert_eq!(+ body.into_payload(),+ serde_json::json!({"body": expected, "truncated": expected.is_none()})+ );+ }+ }+struct PanicOnDebug;impl fmt::Debug for PanicOnDebug {@@ -394,9 +836,20 @@ mod tests {}}+ fn span_with_payloads(payloads: Payloads) -> CapturedSpan {+ CapturedSpan {+ name: UPSTREAM_ATTEMPT,+ fields: Map::new(),+ payloads,+ started_at: SystemTime::now(),+ ended_at: SystemTime::now(),+ children: Vec::new(),+ }+ }+fn recorder() -> (SpanCapture, mpsc::Receiver<CapturedSpan>) {let (sender, receiver) = mpsc::channel();- let capture = SpanCapture::new(move |span| sender.send(span).unwrap());+ let capture = SpanCapture::new(None, move |span| sender.send(span).unwrap());(capture, receiver)}diff --git a/docs/integrations/nemo_relay.md b/docs/integrations/nemo_relay.mdindex 3af0ed16c..ccce9e086 100644--- a/docs/integrations/nemo_relay.md+++ b/docs/integrations/nemo_relay.md@@ -313,8 +313,9 @@ The plugin captures two existing Switchyard spans. Each routing or answer LLMcall becomes a `libsy.client_call` scope. Physical HTTP attempts and retriesbecome `libsy.upstream_attempt` child scopes. Buffered calls close with the response;streaming calls close when the stream finishes, fails, or is dropped. The-projection contains allowlisted, payload-free routing metadata only and does not-copy prompts, responses, provider error bodies, or credentials into Relay.+projection contains allowlisted routing metadata by default. Optional+[routing payload capture](#inspect-classifier-and-judge-payloads) adds the final+classifier/judge request and successful buffered response.`switchyard.call_phase` records when Switchyard issued a client call:`routing` while the algorithm was still deciding, or `completion` after a@@ -329,6 +330,96 @@ classification as `routing` or `answer`.| Physical provider request or retry | `libsy.upstream_attempt` scope | Its `libsy.client_call` scope | Starts before the HTTP attempt and closes on response, error, or cancellation. || Routing request, decision, overhead, or failure | Existing `switchyard.routing.*` mark | Active Relay scope at emission | Point-in-time mark using the existing outcome fields. |+### Inspect Classifier and Judge Payloads++Routing payload capture is available through the native NeMo Relay plugin and+is disabled by default. It is not a standalone Switchyard configuration option.+Use it to inspect the final classifier/judge request, including its system+prompt, task context, output schema, resolved model, and target body settings,+plus its successful buffered response. This helps distinguish a poor verdict+from missing task context.++Add these options to the existing `plugins.dynamic.config` table in your Relay+`plugins.toml`, then restart the host:++```toml+[plugins.dynamic.config]+switchyard_config_path = "/etc/switchyard/routes.toml"+capture_routing_payloads = true+```++`max_routing_payload_bytes` defaults to 262144 encoded bytes (256 KiB) per+request or response body, per attempt, before sanitization. It must be positive.+Larger bodies are omitted entirely, not retained as a prefix:++```json+{"body": null, "truncated": true}+```++The full request still reaches the model. The capture limit does not change+which messages the classifier receives; see+[classifier context selection](../routing_algorithms/llm_classifier_routing.md#tuning-options).++For a debugging session that needs larger bodies, set an override in the same+`plugins.dynamic.config` table and restart the host. For example, allow up to+2 MiB per body:++```toml+max_routing_payload_bytes = 2097152+```++This is a per-body limit, not a total memory budget. Larger limits increase+memory use and export volume, especially with concurrent calls or retries.+Exporters and their backends can impose smaller limits. After debugging, set+`capture_routing_payloads = false` and restart the host; remove the size override+to restore the default for later sessions.++Within the routed request's trace:++1. Find the `libsy.client_call` scope for the classifier/judge model, with+ `switchyard.call_phase = routing`.+2. Open its `libsy.upstream_attempt` child. The request is the scope input;+ its successful buffered response is the scope output. Retries have separate+ attempt scopes. Retained payloads have a JSON string in `body` and+ `truncated = false`.+3. Inspect the `switchyard.routing.decision` mark on the same routed request+ for the selected and served model. The judge response contains the returned+ verdict; the decision mark records what Switchyard did with it.++For an [ATOF export](https://docs.nvidia.com/nemo/relay/configure-plugins/observability/atof),+the request appears directly in the attempt start event's `data`, and the+response in the matching end event's `data`. Their `uuid` values match;+`parent_uuid` identifies the classifier call. For example, inspect complete+captured requests with:++```bash+jq 'select(.name == "libsy.upstream_attempt" and .scope_category == "start" and .data.body != null) | .data.body | fromjson' events.jsonl+```++Find omitted request or response bodies with:++```bash+jq 'select(.name == "libsy.upstream_attempt" and .data.truncated == true) | {uuid, scope_category}' events.jsonl+```++Capture follows routing-phase HTTP calls, which include LLM classifier/judge+calls and may include other model calls used while deciding a route. It excludes+completion-phase payloads, streamed response bodies, HTTP error bodies, and+non-LLM decision-client payloads such as SystemOne. This option belongs to the+native Relay plugin; enabling standalone Switchyard TRACE logging does not+enable payload capture.++Payloads can contain sensitive task data. To mask configured provider keys,+the plugin decodes the captured JSON, redacts it, and serializes the copy again.+JSON strings, such as classifier content and tool arguments, are decoded too.+Traversal is limited to 64 levels, including JSON encoded inside strings.+Whitespace and escape spellings may change. If JSON-looking text cannot be+decoded or the depth limit is exceeded, the body is omitted (`body: null`).+Captured data passes through Relay's normal event+sanitizers and configured exporters. Personal data and other secrets require+suitable sanitizers; see Relay's+[PII redaction options](https://docs.nvidia.com/nemo/relay/configure-plugins/pii-redaction/about).+### How Routing Appears in TracesIn Relay's@@ -438,9 +529,10 @@ Configure delivery through Relay's## Data Handling-Switchyard routing telemetry excludes request and response content. Its marks-do not contain prompts, bodies, headers, credentials, raw provider responses,-or free-form provider errors. Relay's LLM events can capture request and-response data according to Relay's+By default, Switchyard routing telemetry excludes request and response content.+Enabling [routing payload capture](#inspect-classifier-and-judge-payloads) adds+internal routing requests and successful buffered responses to+`libsy.upstream_attempt` scopes. Routing marks remain payload-free.+Relay's LLM events can capture request and response data according to Relay's[input and output event semantics](https://docs.nvidia.com/nemo/relay/about-nemo-relay/concepts/events#input-and-output-payloads),independently of these Switchyard marks.