Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

feat(relay): add opt-in classifier and judge payload capture (#1197)

Alex Fournier4a71c01e516c
diff --git a/crates/libsy-llm-client/src/client.rs b/crates/libsy-llm-client/src/client.rs
index ef56b2706..deb4d9a12 100644
--- a/crates/libsy-llm-client/src/client.rs
+++ b/crates/libsy-llm-client/src/client.rs
@@ -548,7 +548,15 @@ impl TranslatingLlmClient {
let builder = apply_extra_headers(builder, backend);
let builder = backend.apply_auth(builder);
- let response = match builder.send().await {
+ let response = match async {
+ let request = builder.build()?;
+ if let Some(body) = request.body().and_then(reqwest::Body::as_bytes) {
+ crate::payload_capture::record("request", || String::from_utf8_lossy(body));
+ }
+ client.execute(request).await
+ }
+ .await
+ {
Ok(response) => response,
Err(error) => {
metrics::record_upstream_attempt(None);
@@ -594,6 +602,7 @@ impl TranslatingLlmClient {
}
};
metrics::record_upstream_attempt(Some(status.as_u16()));
+ crate::payload_capture::record("response", || String::from_utf8_lossy(&body));
return Ok(EncodedResponse::Buffered {
status: status.as_u16(),
body: body.to_vec(),
@@ -1817,6 +1826,14 @@ mod tests {
assert!(!convert_reqwest_error(error).to_string().contains("CANARY"));
}
+ #[tokio::test]
+ async fn captured_request_build_failure_remains_a_configuration_error() {
+ let client = TranslatingLlmClient::new(&chat_map("://invalid")).unwrap();
+ let result =
+ crate::with_payload_capture(client.call(request_for(Some("gpt"), false))).await;
+ assert!(matches!(result, Err(LlmClientError::Configuration { .. })));
+ }
+
#[test]
fn nvidia_anthropic_tools_drop_unsupported_strict_field() {
let backend = Backend::Anthropic(config("https://inference-api.nvidia.com"));
diff --git a/crates/libsy-llm-client/src/lib.rs b/crates/libsy-llm-client/src/lib.rs
index 5bc416bbf..3710ad150 100644
--- a/crates/libsy-llm-client/src/lib.rs
+++ b/crates/libsy-llm-client/src/lib.rs
@@ -28,6 +28,7 @@ mod observability;
mod observation;
mod observed_translation;
mod openai_decisions;
+mod payload_capture;
pub mod raw;
mod responses_reasoning;
pub mod run;
@@ -38,6 +39,7 @@ pub use client::{AuxiliaryOperation, ModelConfig, TranslatingLlmClient};
pub use error::{LlmClientError, Result};
pub use observation::{ModelCallObservation, RunObservation, RunObserver};
pub use openai_decisions::OpenAiDecisionClient;
+pub use payload_capture::with_payload_capture;
pub use raw::RawResponse;
pub use responses_reasoning::ResponsesReasoningPolicy;
pub use run::{ClientRouter, decide, run};
diff --git a/crates/libsy-llm-client/src/payload_capture.rs b/crates/libsy-llm-client/src/payload_capture.rs
new file mode 100644
index 000000000..e1c020eca
--- /dev/null
+++ b/crates/libsy-llm-client/src/payload_capture.rs
@@ -0,0 +1,86 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+//! Explicit opt-in for HTTP body capture by a host's tracing subscriber.
+
+use std::fmt;
+use std::future::Future;
+
+tokio::task_local! {
+ static CAPTURE_ENABLED: ();
+}
+
+/// Enables raw HTTP body events while polling `future` on the current task.
+///
+/// The host must install a subscriber for `switchyard.payload` and apply its
+/// own filtering, size limits, and redaction. Spawned tasks do not inherit capture.
+pub async fn with_payload_capture<T>(future: impl Future<Output = T>) -> T {
+ CAPTURE_ENABLED.scope((), future).await
+}
+
+pub(crate) fn record<D: fmt::Display>(phase: &'static str, body: impl FnOnce() -> D) {
+ if CAPTURE_ENABLED.try_with(|_| ()).is_ok() {
+ tracing::trace!(target: "switchyard.payload", phase, body = %body());
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use std::sync::Arc;
+
+ use parking_lot::Mutex;
+ use tracing::field::Field;
+ use tracing::instrument::WithSubscriber;
+ use tracing_subscriber::Layer;
+ use tracing_subscriber::layer::{Context, SubscriberExt};
+
+ use super::*;
+
+ struct Recorder(Arc<Mutex<Vec<String>>>);
+
+ impl<S: tracing::Subscriber> Layer<S> for Recorder {
+ fn on_event(&self, event: &tracing::Event<'_>, _: Context<'_, S>) {
+ event.record(&mut |field: &Field, value: &dyn fmt::Debug| {
+ if field.name() == "body" {
+ self.0.lock().push(format!("{value:?}"));
+ }
+ });
+ }
+ }
+
+ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+ async fn capture_requires_opt_in_and_does_not_leak_between_tasks_or_after_cancellation() {
+ let events = Arc::new(Mutex::new(Vec::new()));
+ let subscriber = tracing_subscriber::registry().with(Recorder(Arc::clone(&events)));
+ async {
+ record("request", || -> &str {
+ panic!("disabled capture must not evaluate the body")
+ });
+ tokio::join!(
+ with_payload_capture(async {
+ tokio::spawn(
+ async {
+ record("request", || "spawned");
+ }
+ .with_current_subscriber(),
+ )
+ .await
+ .unwrap();
+ tokio::task::yield_now().await;
+ record("request", || "enabled");
+ }),
+ async {
+ tokio::task::yield_now().await;
+ record("request", || "other-task");
+ },
+ );
+ let mut pending = Box::pin(with_payload_capture(std::future::pending::<()>()));
+ assert!(futures::poll!(&mut pending).is_pending());
+ drop(pending);
+ record("response", || "after-cancellation");
+ }
+ .with_subscriber(subscriber)
+ .await;
+ assert_eq!(*events.lock(), ["enabled"]);
+ }
+}
diff --git a/crates/switchyard-nemo-relay-plugin/README.md b/crates/switchyard-nemo-relay-plugin/README.md
index 34fa3dbab..69b8c46bb 100644
--- a/crates/switchyard-nemo-relay-plugin/README.md
+++ b/crates/switchyard-nemo-relay-plugin/README.md
@@ -234,8 +234,12 @@ retains ownership of the outer LLM lifecycle.
The plugin also captures the existing `libsy.client_call` and
`libsy.upstream_attempt` spans for each routed request. It replays completed
spans beneath Relay's outer LLM scope with their original timestamps. The
-projection includes allowlisted, payload-free routing metadata only; it excludes
-prompts, responses, provider error bodies, and credentials.
+projection contains allowlisted routing metadata by default. To inspect the
+final classifier/judge request and buffered response, enable
+[routing payload capture](../../docs/integrations/nemo_relay.md#inspect-classifier-and-judge-payloads)
+in the plugin configuration. Capture uses Relay's existing exporters and writes
+no payload files inside the plugin. It is disabled by default and omits bodies
+above the configured size limit.
## Observability
diff --git a/crates/switchyard-nemo-relay-plugin/config.schema.json b/crates/switchyard-nemo-relay-plugin/config.schema.json
index 61f178a12..4c368edae 100644
--- a/crates/switchyard-nemo-relay-plugin/config.schema.json
+++ b/crates/switchyard-nemo-relay-plugin/config.schema.json
@@ -9,6 +9,17 @@
{ "required": ["switchyard_config"] }
],
"properties": {
+ "capture_routing_payloads": {
+ "type": "boolean",
+ "default": false,
+ "description": "Opt in to bounded routing request and buffered response bodies for debugging. Payloads may contain sensitive task context."
+ },
+ "max_routing_payload_bytes": {
+ "type": "integer",
+ "minimum": 1,
+ "default": 262144,
+ "description": "Maximum encoded bytes per routing request or buffered response, per attempt. Larger bodies are omitted entirely with body=null and truncated=true."
+ },
"priority": {
"type": "integer",
"default": 0,
diff --git a/crates/switchyard-nemo-relay-plugin/src/config.rs b/crates/switchyard-nemo-relay-plugin/src/config.rs
index 2d8db081f..912976a8f 100644
--- a/crates/switchyard-nemo-relay-plugin/src/config.rs
+++ b/crates/switchyard-nemo-relay-plugin/src/config.rs
@@ -8,6 +8,12 @@ use serde_json::{Map, Value};
use switchyard_protocol::WireFormat;
use switchyard_runner::Runner;
+pub(crate) const DEFAULT_ROUTING_PAYLOAD_BYTES: usize = 256 * 1024;
+
+fn default_routing_payload_bytes() -> usize {
+ DEFAULT_ROUTING_PAYLOAD_BYTES
+}
+
pub(crate) fn protocol_from_call(name: &str) -> Option<WireFormat> {
match name {
"openai.chat_completions" => Some(WireFormat::OpenAiChat),
@@ -20,6 +26,11 @@ pub(crate) fn protocol_from_call(name: &str) -> Option<WireFormat> {
#[derive(Debug, Deserialize)]
#[serde(deny_unknown_fields)]
pub(crate) struct SwitchyardConfig {
+ /// Includes bounded final routing HTTP bodies in Relay scopes when explicitly enabled.
+ #[serde(default)]
+ pub(crate) capture_routing_payloads: bool,
+ #[serde(default = "default_routing_payload_bytes")]
+ pub(crate) max_routing_payload_bytes: usize,
#[serde(default)]
pub(crate) priority: i32,
#[serde(default)]
@@ -30,6 +41,9 @@ pub(crate) struct SwitchyardConfig {
impl SwitchyardConfig {
pub(crate) fn load_runner(&self) -> Result<Runner, String> {
+ if self.max_routing_payload_bytes == 0 {
+ return Err("max_routing_payload_bytes must be positive".into());
+ }
let runner = match (&self.switchyard_config_path, &self.switchyard_config) {
(Some(path), None) => Runner::load(path).map_err(|error| error.to_string()),
(None, Some(config)) => toml::to_string(config)
@@ -116,6 +130,8 @@ mod tests {
#[test]
fn builds_runner_from_inline_deployment() {
let config = SwitchyardConfig {
+ capture_routing_payloads: false,
+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,
priority: 0,
switchyard_config_path: None,
switchyard_config: Some(inline_deployment()),
@@ -125,6 +141,27 @@ mod tests {
assert!(runner.route("switchyard/default").is_some());
}
+ #[test]
+ fn payload_capture_defaults_off_and_requires_a_positive_limit() {
+ let mut config: SwitchyardConfig = serde_json::from_value(json!({
+ "switchyard_config": inline_deployment(),
+ }))
+ .unwrap();
+ assert!(!config.capture_routing_payloads);
+ assert_eq!(config.max_routing_payload_bytes, 256 * 1024);
+ let schema: Value = serde_json::from_str(include_str!("../config.schema.json")).unwrap();
+ assert_eq!(
+ schema["properties"]["max_routing_payload_bytes"]["default"],
+ config.max_routing_payload_bytes
+ );
+ config.max_routing_payload_bytes = 0;
+ assert!(
+ matches!(config.load_runner(), Err(error) if error.contains("max_routing_payload_bytes"))
+ );
+ config.max_routing_payload_bytes = 2 * 1024 * 1024;
+ assert!(config.load_runner().is_ok());
+ }
+
#[test]
fn builds_runner_from_deployment_path() {
let mut deployment = NamedTempFile::new().unwrap();
@@ -135,6 +172,8 @@ mod tests {
)
.unwrap();
let config = SwitchyardConfig {
+ capture_routing_payloads: false,
+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,
priority: 0,
switchyard_config_path: Some(deployment.path().to_path_buf()),
switchyard_config: None,
@@ -244,6 +283,8 @@ mod tests {
#[test]
fn requires_exactly_one_configuration_source() {
let neither = SwitchyardConfig {
+ capture_routing_payloads: false,
+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,
priority: 0,
switchyard_config_path: None,
switchyard_config: None,
@@ -254,6 +295,8 @@ mod tests {
));
let both = SwitchyardConfig {
+ capture_routing_payloads: false,
+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,
priority: 0,
switchyard_config_path: Some(PathBuf::from("routes.toml")),
switchyard_config: Some(inline_deployment()),
diff --git a/crates/switchyard-nemo-relay-plugin/src/lib.rs b/crates/switchyard-nemo-relay-plugin/src/lib.rs
index 6f0c54311..b8ca6244d 100644
--- a/crates/switchyard-nemo-relay-plugin/src/lib.rs
+++ b/crates/switchyard-nemo-relay-plugin/src/lib.rs
@@ -50,6 +50,9 @@ impl NativePlugin for SwitchyardPlugin {
ctx: &mut PluginContext<'_>,
) -> nemo_relay_plugin::Result<()> {
let config = parse_config(plugin_config)?;
+ let payload_limit = config
+ .capture_routing_payloads
+ .then_some(config.max_routing_payload_bytes);
let priority = config.priority;
let runtime = Arc::new(SwitchyardRuntime::new(config)?);
let plugin_runtime = ctx.runtime();
@@ -59,7 +62,7 @@ impl NativePlugin for SwitchyardPlugin {
});
let span_runtime = Arc::clone(&runtime);
let span_emit_event = Arc::clone(&emit_event);
- let span_capture = SpanCapture::new(move |mut span| {
+ let span_capture = SpanCapture::new(payload_limit, move |mut span| {
span.sanitize(span_runtime.redactor());
span_emit_event(RoutingEvent::Span(span));
});
diff --git a/crates/switchyard-nemo-relay-plugin/src/redaction_tests.rs b/crates/switchyard-nemo-relay-plugin/src/redaction_tests.rs
index d3a42c72e..bfc86b7f3 100644
--- a/crates/switchyard-nemo-relay-plugin/src/redaction_tests.rs
+++ b/crates/switchyard-nemo-relay-plugin/src/redaction_tests.rs
@@ -105,6 +105,8 @@ async fn reflected_response(format: WireFormat, streaming: bool, fail: bool) {
"routes": {"default": {"id": "switchyard/default", "type": "passthrough", "target": "default"}}
});
let runtime = SwitchyardRuntime::new(SwitchyardConfig {
+ capture_routing_payloads: false,
+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,
priority: 0,
switchyard_config_path: None,
switchyard_config: Some(deployment.as_object().unwrap().clone()),
@@ -126,7 +128,7 @@ async fn reflected_response(format: WireFormat, streaming: bool, fail: bool) {
let captured = Arc::new(Mutex::new(Vec::new()));
let span_observed = Arc::clone(&captured);
let redactor = Arc::clone(&runtime.redactor);
- let span_capture = SpanCapture::new(move |mut span| {
+ let span_capture = SpanCapture::new(None, move |mut span| {
span.sanitize(&redactor);
span_observed.lock().unwrap().push(RoutingEvent::Span(span));
});
diff --git a/crates/switchyard-nemo-relay-plugin/src/runtime.rs b/crates/switchyard-nemo-relay-plugin/src/runtime.rs
index d708ef7b5..fb0f4048d 100644
--- a/crates/switchyard-nemo-relay-plugin/src/runtime.rs
+++ b/crates/switchyard-nemo-relay-plugin/src/runtime.rs
@@ -950,6 +950,8 @@ mod tests {
}
});
SwitchyardRuntime::new(crate::config::SwitchyardConfig {
+ capture_routing_payloads: false,
+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,
priority: 0,
switchyard_config_path: None,
switchyard_config: Some(deployment.as_object().unwrap().clone()),
@@ -1104,6 +1106,8 @@ mod tests {
},
});
let runtime = SwitchyardRuntime::new(crate::config::SwitchyardConfig {
+ capture_routing_payloads: false,
+ max_routing_payload_bytes: crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES,
priority: 0,
switchyard_config_path: None,
switchyard_config: Some(
diff --git a/crates/switchyard-nemo-relay-plugin/src/span_capture.rs b/crates/switchyard-nemo-relay-plugin/src/span_capture.rs
index 31bf477b8..59db88c4b 100644
--- a/crates/switchyard-nemo-relay-plugin/src/span_capture.rs
+++ b/crates/switchyard-nemo-relay-plugin/src/span_capture.rs
@@ -9,10 +9,11 @@ use std::time::{Instant, SystemTime};
use nemo_relay_plugin::{Json, PluginRuntime, ScopeType};
use serde_json::{Map, Value};
+use switchyard_llm_client::with_payload_capture;
use switchyard_runner::ProviderKeyRedactor;
use tracing::field::{Field, Visit};
use tracing::span::{Attributes, Id, Record};
-use tracing::{Metadata, Subscriber};
+use tracing::{Event, Metadata, Subscriber};
use tracing_subscriber::Layer;
use tracing_subscriber::filter::{LevelFilter, filter_fn};
use tracing_subscriber::layer::Context;
@@ -21,13 +22,22 @@ use tracing_subscriber::registry::LookupSpan;
const CLIENT_CALL: &str = "libsy.client_call";
const UPSTREAM_ATTEMPT: &str = "libsy.upstream_attempt";
+const PAYLOAD_TARGET: &str = "switchyard.payload";
+const MAX_PAYLOAD_REDACTION_DEPTH: usize = 64;
type SpanEmitter = Box<dyn Fn(CapturedSpan) + Send + Sync>;
+#[derive(Debug, Default)]
+struct Payloads {
+ request: Option<Json>,
+ response: Option<Json>,
+}
+
#[derive(Debug)]
pub(crate) struct CapturedSpan {
name: &'static str,
fields: Map<String, Value>,
+ payloads: Payloads,
started_at: SystemTime,
ended_at: SystemTime,
children: Vec<CapturedSpan>,
@@ -36,6 +46,7 @@ pub(crate) struct CapturedSpan {
struct OpenSpan {
name: &'static str,
fields: Map<String, Value>,
+ payloads: Payloads,
started_at: SystemTime,
started: Instant,
children: Vec<CapturedSpan>,
@@ -43,12 +54,55 @@ struct OpenSpan {
struct CaptureLayer {
emit: SpanEmitter,
+ payload_limit: Option<usize>,
}
impl<S> Layer<S> for CaptureLayer
where
S: Subscriber + for<'lookup> LookupSpan<'lookup>,
{
+ fn on_event(&self, event: &Event<'_>, ctx: Context<'_, S>) {
+ if event.metadata().target() != PAYLOAD_TARGET {
+ return;
+ }
+ let Some(limit) = self.payload_limit else {
+ return;
+ };
+ let Some(span) = ctx.event_span(event) else {
+ return;
+ };
+ let is_routing = span.scope().any(|ancestor| {
+ ancestor.extensions().get::<OpenSpan>().is_some_and(|open| {
+ open.name == CLIENT_CALL
+ && open
+ .fields
+ .get("switchyard.call_phase")
+ .and_then(Value::as_str)
+ == Some("routing")
+ })
+ });
+ if !is_routing || span.name() != UPSTREAM_ATTEMPT {
+ return;
+ }
+ let mut payload = PayloadVisitor {
+ phase: None,
+ body: BoundedText {
+ text: String::new(),
+ truncated: false,
+ limit,
+ },
+ };
+ event.record(&mut payload);
+ if let Some(open) = span.extensions_mut().get_mut::<OpenSpan>() {
+ let data = payload.body.into_payload();
+ match payload.phase {
+ Some(PayloadPhase::Request) => open.payloads.request = Some(data),
+ Some(PayloadPhase::Response) => open.payloads.response = Some(data),
+ None => (),
+ }
+ }
+ }
+
fn on_new_span(&self, attrs: &Attributes<'_>, id: &Id, ctx: Context<'_, S>) {
let name = attrs.metadata().name();
let mut fields = Map::new();
@@ -57,6 +111,7 @@ where
span.extensions_mut().insert(OpenSpan {
name,
fields,
+ payloads: Payloads::default(),
started_at: SystemTime::now(),
started: Instant::now(),
children: Vec::new(),
@@ -85,6 +140,7 @@ where
let OpenSpan {
name,
mut fields,
+ payloads,
started_at,
started,
children,
@@ -96,6 +152,7 @@ where
let captured = CapturedSpan {
name,
fields,
+ payloads,
started_at,
ended_at: started_at
.checked_add(started.elapsed())
@@ -120,6 +177,23 @@ impl CapturedSpan {
{
self.fields = fields;
}
+ for payload in [&mut self.payloads.request, &mut self.payloads.response]
+ .into_iter()
+ .flatten()
+ {
+ if !redactor.is_empty()
+ && let Some(body) = payload.get_mut("body")
+ {
+ *body = body
+ .as_str()
+ .and_then(|text| serde_json::from_str::<Json>(text).ok())
+ .and_then(|decoded| {
+ redact_captured_json(decoded, redactor, MAX_PAYLOAD_REDACTION_DEPTH)
+ })
+ .map(|decoded| Json::String(decoded.to_string()))
+ .unwrap_or(Json::Null);
+ }
+ }
for child in &mut self.children {
child.sanitize(redactor);
}
@@ -127,20 +201,24 @@ impl CapturedSpan {
pub(crate) fn emit(self, runtime: &PluginRuntime) -> Result<(), String> {
let mut metadata = Json::Object(self.fields);
- let output = if self.name == CLIENT_CALL {
- response_data(&metadata)
- } else {
- None
- };
+ let output = self.payloads.response.or_else(|| {
+ if self.name == CLIENT_CALL {
+ response_data(&metadata)
+ } else {
+ None
+ }
+ });
if let Some(session_id) = metadata.get("gen_ai.conversation.id").cloned() {
metadata["session_id"] = session_id;
}
if let Some(status) = span_status(&metadata) {
metadata["otel.status_code"] = status["otel.status_code"].clone();
}
- let input = metadata
- .get("gen_ai.request.model")
- .map(|model| serde_json::json!({"model": model}));
+ let input = self.payloads.request.or_else(|| {
+ metadata
+ .get("gen_ai.request.model")
+ .map(|model| serde_json::json!({"model": model}))
+ });
let mut scope = runtime.scope_at(
self.name,
relay_scope_type(self.name),
@@ -158,6 +236,50 @@ impl CapturedSpan {
}
}
+fn redact_captured_json(value: Json, redactor: &ProviderKeyRedactor, depth: usize) -> Option<Json> {
+ let depth = depth.checked_sub(1)?;
+ Some(match value {
+ Json::String(text) => Json::String(redact_captured_text(text, redactor, depth)?),
+ Json::Array(values) => Json::Array(
+ values
+ .into_iter()
+ .map(|value| redact_captured_json(value, redactor, depth))
+ .collect::<Option<_>>()?,
+ ),
+ Json::Object(fields) => Json::Object(
+ fields
+ .into_iter()
+ .map(|(name, value)| {
+ Some((
+ redact_captured_text(name, redactor, depth)?,
+ redact_captured_json(value, redactor, depth)?,
+ ))
+ })
+ .collect::<Option<_>>()?,
+ ),
+ value => value,
+ })
+}
+
+fn redact_captured_text(
+ text: String,
+ redactor: &ProviderKeyRedactor,
+ depth: usize,
+) -> Option<String> {
+ // The existing redaction marker is plain text, not a JSON array.
+ if text.trim() != "[REDACTED]"
+ && matches!(
+ text.trim_start().as_bytes().first(),
+ Some(b'{' | b'[' | b'"')
+ )
+ {
+ let decoded = serde_json::from_str::<Json>(&text).ok()?;
+ Some(redact_captured_json(decoded, redactor, depth)?.to_string())
+ } else {
+ Some(redactor.text(text))
+ }
+}
+
/// Reconstructs the bounded response metadata forwarded to NeMo Relay.
fn response_data(fields: &Json) -> Option<Json> {
let mut output = Map::new();
@@ -229,11 +351,25 @@ pub(crate) struct SpanCapture {
}
impl SpanCapture {
- pub(crate) fn new(emit: impl Fn(CapturedSpan) + Send + Sync + 'static) -> Self {
+ pub(crate) fn new(
+ payload_limit: Option<usize>,
+ emit: impl Fn(CapturedSpan) + Send + Sync + 'static,
+ ) -> Self {
let layer = CaptureLayer {
emit: Box::new(emit),
+ payload_limit,
}
- .with_filter(filter_fn(captures).with_max_level_hint(LevelFilter::DEBUG));
+ .with_filter(
+ filter_fn(move |metadata| {
+ captures(metadata)
+ || (payload_limit.is_some() && metadata.target() == PAYLOAD_TARGET)
+ })
+ .with_max_level_hint(if payload_limit.is_some() {
+ LevelFilter::TRACE
+ } else {
+ LevelFilter::DEBUG
+ }),
+ );
Self {
dispatch: tracing::Dispatch::new(tracing_subscriber::registry().with(layer)),
}
@@ -241,14 +377,70 @@ impl SpanCapture {
/// Polls a future with the capture subscriber as its default.
pub(crate) async fn capture<T>(&self, future: impl Future<Output = T>) -> T {
- CaptureFuture {
+ with_payload_capture(CaptureFuture {
inner: Some(Box::pin(future)),
dispatch: self.dispatch.clone(),
- }
+ })
.await
}
}
+struct PayloadVisitor {
+ phase: Option<PayloadPhase>,
+ body: BoundedText,
+}
+
+enum PayloadPhase {
+ Request,
+ Response,
+}
+
+impl Visit for PayloadVisitor {
+ fn record_str(&mut self, field: &Field, value: &str) {
+ if field.name() == "phase" {
+ self.phase = match value {
+ "request" => Some(PayloadPhase::Request),
+ "response" => Some(PayloadPhase::Response),
+ _ => None,
+ };
+ }
+ }
+
+ fn record_debug(&mut self, field: &Field, value: &dyn fmt::Debug) {
+ if field.name() == "body" {
+ let _ = fmt::write(&mut self.body, format_args!("{value:?}"));
+ }
+ }
+}
+
+struct BoundedText {
+ text: String,
+ truncated: bool,
+ limit: usize,
+}
+
+impl BoundedText {
+ fn into_payload(self) -> Json {
+ // A retained prefix could end inside a credential before redaction runs.
+ let mut payload = serde_json::json!({"body": null, "truncated": self.truncated});
+ if !self.truncated {
+ payload["body"] = self.text.into();
+ }
+ payload
+ }
+}
+
+impl fmt::Write for BoundedText {
+ fn write_str(&mut self, value: &str) -> fmt::Result {
+ if value.len() > self.limit - self.text.len() {
+ self.truncated = true;
+ return Err(fmt::Error);
+ }
+ self.text.push_str(value);
+ Ok(())
+ }
+}
+
// Standard subscriber wrappers restore the ambient dispatch before drop. This
// wrapper also drops the inner future under capture so cancelled spans close here.
struct CaptureFuture<F> {
@@ -386,6 +578,256 @@ mod tests {
use super::*;
+ // Real loopback HTTP proves capture sees the encoded judge body, not the outer task.
+ #[tokio::test]
+ async fn captures_final_judge_payload_only_when_enabled() {
+ use crate::{config::SwitchyardConfig, runtime::SwitchyardRuntime};
+ use switchyard_protocol::{Request, WireFormat, text_request};
+ use wiremock::matchers::{body_partial_json, method};
+ use wiremock::{Mock, MockServer, ResponseTemplate};
+
+ let server = MockServer::start().await;
+ for (model, content) in [
+ ("judge/model", r#"{"target":"efficient"}"#),
+ ("answer/model", "answer"),
+ ] {
+ let response = serde_json::json!({
+ "id": "smoke", "model": model,
+ "choices": [{"index": 0, "message": {"role": "assistant", "content": content}, "finish_reason": "stop"}],
+ "debug_canary": "final-body-only", "final-body-only": "ordinary diagnostics"
+ })
+ .to_string()
+ .replace("final-body-only", r"\u0066inal-body-only");
+ Mock::given(method("POST"))
+ .and(body_partial_json(serde_json::json!({"model": model})))
+ .respond_with(
+ ResponseTemplate::new(200)
+ .insert_header("content-type", "application/json")
+ .set_body_string(response),
+ )
+ .mount(&server)
+ .await;
+ }
+ let config: SwitchyardConfig = serde_json::from_value(serde_json::json!({
+ "switchyard_config": {
+ "schema_version": 1,
+ "llm_clients": {"local": {"format": "openai_chat", "base_url": format!("{}/v1", server.uri())}},
+ "targets": {
+ "judge": {"id": "judge/model", "llm_client": "local", "extra_body": {"debug_canary": "final-body-only"}},
+ "answer": {"id": "answer/model", "llm_client": "local"}
+ },
+ "routes": {"test": {
+ "id": "switchyard/test", "type": "llm_classifier", "mode": "custom",
+ "models": {"judge": ["judge"], "efficient": ["answer"], "any": ["answer"]},
+ "default_target": "efficient", "prompt": "Judge task complexity.",
+ "response_schema": r#"{"type":"object","properties":{"target":{"type":"string","enum":["efficient"]}},"required":["target"],"additionalProperties":false}"#,
+ "policy": {"type": "target_selector", "selector": "/target"}
+ }}
+ }
+ })).unwrap();
+ let runtime = SwitchyardRuntime::new(config).unwrap();
+ for enabled in [false, true] {
+ let (sender, spans) = mpsc::channel();
+ let capture = SpanCapture::new(
+ enabled.then_some(crate::config::DEFAULT_ROUTING_PAYLOAD_BYTES),
+ move |span| sender.send(span).unwrap(),
+ );
+ let execution = capture
+ .capture(runtime.execute_buffered(
+ WireFormat::OpenAiChat,
+ Request {
+ llm_request: text_request(
+ Some("switchyard/test".into()),
+ "Fix a failing Rust test.\nPreserve \"雪\" and \\ exactly.",
+ ),
+ raw_request: None,
+ metadata: None,
+ },
+ ))
+ .await;
+ assert!(execution.result.is_ok(), "{:?}", execution.result);
+ let received = server.received_requests().await.unwrap();
+ let mut spans: Vec<_> = spans.try_iter().collect();
+ let judge = spans
+ .iter_mut()
+ .find(|span| span.fields["selected_model"] == "judge/model")
+ .unwrap();
+ assert_eq!(judge.fields["switchyard.call_phase"], "routing");
+ assert_eq!(judge.children.len(), 1);
+ let attempt = &mut judge.children[0];
+ assert_eq!(attempt.payloads.request.is_some(), enabled);
+ assert_eq!(attempt.payloads.response.is_some(), enabled);
+ if enabled {
+ let actual = received
+ .into_iter()
+ .rev()
+ .find(|request| request.body_json::<Value>().unwrap()["model"] == "judge/model")
+ .unwrap();
+ let captured_body = attempt.payloads.request.as_ref().unwrap()["body"]
+ .as_str()
+ .unwrap();
+ assert_eq!(captured_body.as_bytes(), actual.body);
+ let captured: Value = serde_json::from_str(captured_body).unwrap();
+ assert_eq!(captured["debug_canary"], "final-body-only");
+ assert_eq!(captured["messages"][0]["content"], "Judge task complexity.");
+ assert_eq!(
+ attempt.payloads.request.as_ref().unwrap()["truncated"],
+ false
+ );
+ assert!(
+ attempt.payloads.response.as_ref().unwrap()["body"]
+ .as_str()
+ .unwrap()
+ .contains("efficient")
+ );
+ judge.sanitize(&ProviderKeyRedactor::new(&["final-body-only".into()]));
+ assert!(!format!("{judge:?}").contains("final-body-only"));
+ for payload in [
+ &judge.children[0].payloads.request,
+ &judge.children[0].payloads.response,
+ ] {
+ let decoded: Value =
+ serde_json::from_str(payload.as_ref().unwrap()["body"].as_str().unwrap())
+ .unwrap();
+ assert_eq!(decoded["debug_canary"], "[REDACTED]");
+ assert!(!decoded.to_string().contains("final-body-only"));
+ }
+ }
+ let answer = spans
+ .iter()
+ .find(|span| span.fields["selected_model"] == "answer/model")
+ .unwrap();
+ for attempt in &answer.children {
+ assert!(attempt.payloads.request.is_none());
+ assert!(attempt.payloads.response.is_none());
+ }
+ }
+ }
+
+ #[test]
+ fn captured_classifier_content_and_tool_arguments_redact_nested_json() {
+ let nested = r#"{"credential":"qa-\u0073ecret","qa-\u0073ecret":"ordinary"}"#;
+ let request = serde_json::json!({"messages": [{"role": "assistant", "tool_calls": [{
+ "function": {"name": "lookup", "arguments": nested}
+ }]}]});
+ let response = serde_json::json!({"choices": [{"message": {
+ "role": "assistant", "content": nested
+ }}], "ordinary": {"text": "Keep ordinary diagnostics", "number": 3, "flag": true, "missing": null}});
+ let mut span = span_with_payloads(Payloads {
+ request: Some(serde_json::json!({"body": request.to_string(), "truncated": false})),
+ response: Some(serde_json::json!({"body": response.to_string(), "truncated": false})),
+ });
+ span.sanitize(&ProviderKeyRedactor::new(&["qa-secret".into()]));
+ span.sanitize(&ProviderKeyRedactor::new(&["qa-secret".into()]));
+ for (payload, path) in [
+ (
+ &span.payloads.request,
+ "/messages/0/tool_calls/0/function/arguments",
+ ),
+ (&span.payloads.response, "/choices/0/message/content"),
+ ] {
+ let body: Json =
+ serde_json::from_str(payload.as_ref().unwrap()["body"].as_str().unwrap()).unwrap();
+ let decoded: Json =
+ serde_json::from_str(body.pointer(path).unwrap().as_str().unwrap()).unwrap();
+ assert_eq!(
+ decoded,
+ serde_json::json!({
+ "credential": "[REDACTED]", "[REDACTED]": "ordinary"
+ })
+ );
+ }
+ let response_body: Json = serde_json::from_str(
+ span.payloads.response.as_ref().unwrap()["body"]
+ .as_str()
+ .unwrap(),
+ )
+ .unwrap();
+ assert_eq!(response_body["ordinary"], response["ordinary"]);
+ }
+
+ #[test]
+ fn captured_payload_redaction_bounds_structural_and_encoded_depth() {
+ let redactor = ProviderKeyRedactor::new(&["qa-secret".into()]);
+ assert!(
+ redact_captured_json(
+ serde_json::json!({"content": r#"{"credential":"qa-\u0073ecret""#}),
+ &redactor,
+ MAX_PAYLOAD_REDACTION_DEPTH,
+ )
+ .is_none()
+ );
+ for (arrays, encoded_strings, omitted) in [
+ (MAX_PAYLOAD_REDACTION_DEPTH - 1, 0, false),
+ (MAX_PAYLOAD_REDACTION_DEPTH, 0, true),
+ (MAX_PAYLOAD_REDACTION_DEPTH - 8, 7, false),
+ (MAX_PAYLOAD_REDACTION_DEPTH - 8, 8, true),
+ ] {
+ let mut body = Json::String("qa-secret".into());
+ for _ in 0..encoded_strings {
+ body = Json::String(body.to_string());
+ }
+ for _ in 0..arrays {
+ body = Json::Array(vec![body]);
+ }
+ let mut span = span_with_payloads(Payloads {
+ request: Some(serde_json::json!({"body": body.to_string(), "truncated": false})),
+ response: None,
+ });
+ span.sanitize(&redactor);
+ let payload = span.payloads.request.unwrap();
+ assert_eq!(payload["body"].is_null(), omitted);
+ if !omitted {
+ assert!(payload["body"].as_str().unwrap().contains("[REDACTED]"));
+ assert!(!payload["body"].as_str().unwrap().contains("qa-secret"));
+ }
+ }
+ }
+
+ #[test]
+ fn payload_sanitization_omits_undecodable_json_when_keys_are_configured() {
+ let body = r#"invalid JSON with qa-\u0073ecret"#;
+ let mut span = span_with_payloads(Payloads {
+ request: Some(serde_json::json!({"body": body, "truncated": false})),
+ response: Some(serde_json::json!({"body": null, "truncated": true})),
+ });
+ span.sanitize(&ProviderKeyRedactor::default());
+ assert_eq!(span.payloads.request.as_ref().unwrap()["body"], body);
+ span.sanitize(&ProviderKeyRedactor::new(&["qa-secret".into()]));
+ assert_eq!(
+ span.payloads.request,
+ Some(serde_json::json!({"body": null, "truncated": false}))
+ );
+ assert_eq!(
+ span.payloads.response,
+ Some(serde_json::json!({"body": null, "truncated": true}))
+ );
+ }
+
+ #[test]
+ fn payload_limit_preserves_complete_bodies_and_omits_oversized_ones() {
+ use std::fmt::Write;
+ for (parts, limit, expected) in [
+ (["plain", " text"], 10, Some("plain text")),
+ (["caf", "é"], 5, Some("café")),
+ (["caf", "é"], 4, None),
+ (["prefix-", "synthetic-secret"], 16, None),
+ ] {
+ let mut body = BoundedText {
+ text: String::new(),
+ truncated: false,
+ limit,
+ };
+ let result = write!(&mut body, "{}{}", parts[0], parts[1]);
+ assert_eq!(result.is_err(), expected.is_none());
+ assert!(body.text.len() <= limit);
+ assert_eq!(
+ body.into_payload(),
+ serde_json::json!({"body": expected, "truncated": expected.is_none()})
+ );
+ }
+ }
+
struct PanicOnDebug;
impl fmt::Debug for PanicOnDebug {
@@ -394,9 +836,20 @@ mod tests {
}
}
+ fn span_with_payloads(payloads: Payloads) -> CapturedSpan {
+ CapturedSpan {
+ name: UPSTREAM_ATTEMPT,
+ fields: Map::new(),
+ payloads,
+ started_at: SystemTime::now(),
+ ended_at: SystemTime::now(),
+ children: Vec::new(),
+ }
+ }
+
fn recorder() -> (SpanCapture, mpsc::Receiver<CapturedSpan>) {
let (sender, receiver) = mpsc::channel();
- let capture = SpanCapture::new(move |span| sender.send(span).unwrap());
+ let capture = SpanCapture::new(None, move |span| sender.send(span).unwrap());
(capture, receiver)
}
diff --git a/docs/integrations/nemo_relay.md b/docs/integrations/nemo_relay.md
index 3af0ed16c..ccce9e086 100644
--- a/docs/integrations/nemo_relay.md
+++ b/docs/integrations/nemo_relay.md
@@ -313,8 +313,9 @@ The plugin captures two existing Switchyard spans. Each routing or answer LLM
call becomes a `libsy.client_call` scope. Physical HTTP attempts and retries
become `libsy.upstream_attempt` child scopes. Buffered calls close with the response;
streaming calls close when the stream finishes, fails, or is dropped. The
-projection contains allowlisted, payload-free routing metadata only and does not
-copy prompts, responses, provider error bodies, or credentials into Relay.
+projection contains allowlisted routing metadata by default. Optional
+[routing payload capture](#inspect-classifier-and-judge-payloads) adds the final
+classifier/judge request and successful buffered response.
`switchyard.call_phase` records when Switchyard issued a client call:
`routing` while the algorithm was still deciding, or `completion` after a
@@ -329,6 +330,96 @@ classification as `routing` or `answer`.
| Physical provider request or retry | `libsy.upstream_attempt` scope | Its `libsy.client_call` scope | Starts before the HTTP attempt and closes on response, error, or cancellation. |
| Routing request, decision, overhead, or failure | Existing `switchyard.routing.*` mark | Active Relay scope at emission | Point-in-time mark using the existing outcome fields. |
+### Inspect Classifier and Judge Payloads
+
+Routing payload capture is available through the native NeMo Relay plugin and
+is disabled by default. It is not a standalone Switchyard configuration option.
+Use it to inspect the final classifier/judge request, including its system
+prompt, task context, output schema, resolved model, and target body settings,
+plus its successful buffered response. This helps distinguish a poor verdict
+from missing task context.
+
+Add these options to the existing `plugins.dynamic.config` table in your Relay
+`plugins.toml`, then restart the host:
+
+```toml
+[plugins.dynamic.config]
+switchyard_config_path = "/etc/switchyard/routes.toml"
+capture_routing_payloads = true
+```
+
+`max_routing_payload_bytes` defaults to 262144 encoded bytes (256 KiB) per
+request or response body, per attempt, before sanitization. It must be positive.
+Larger bodies are omitted entirely, not retained as a prefix:
+
+```json
+{"body": null, "truncated": true}
+```
+
+The full request still reaches the model. The capture limit does not change
+which messages the classifier receives; see
+[classifier context selection](../routing_algorithms/llm_classifier_routing.md#tuning-options).
+
+For a debugging session that needs larger bodies, set an override in the same
+`plugins.dynamic.config` table and restart the host. For example, allow up to
+2 MiB per body:
+
+```toml
+max_routing_payload_bytes = 2097152
+```
+
+This is a per-body limit, not a total memory budget. Larger limits increase
+memory use and export volume, especially with concurrent calls or retries.
+Exporters and their backends can impose smaller limits. After debugging, set
+`capture_routing_payloads = false` and restart the host; remove the size override
+to restore the default for later sessions.
+
+Within the routed request's trace:
+
+1. Find the `libsy.client_call` scope for the classifier/judge model, with
+ `switchyard.call_phase = routing`.
+2. Open its `libsy.upstream_attempt` child. The request is the scope input;
+ its successful buffered response is the scope output. Retries have separate
+ attempt scopes. Retained payloads have a JSON string in `body` and
+ `truncated = false`.
+3. Inspect the `switchyard.routing.decision` mark on the same routed request
+ for the selected and served model. The judge response contains the returned
+ verdict; the decision mark records what Switchyard did with it.
+
+For an [ATOF export](https://docs.nvidia.com/nemo/relay/configure-plugins/observability/atof),
+the request appears directly in the attempt start event's `data`, and the
+response in the matching end event's `data`. Their `uuid` values match;
+`parent_uuid` identifies the classifier call. For example, inspect complete
+captured requests with:
+
+```bash
+jq 'select(.name == "libsy.upstream_attempt" and .scope_category == "start" and .data.body != null) | .data.body | fromjson' events.jsonl
+```
+
+Find omitted request or response bodies with:
+
+```bash
+jq 'select(.name == "libsy.upstream_attempt" and .data.truncated == true) | {uuid, scope_category}' events.jsonl
+```
+
+Capture follows routing-phase HTTP calls, which include LLM classifier/judge
+calls and may include other model calls used while deciding a route. It excludes
+completion-phase payloads, streamed response bodies, HTTP error bodies, and
+non-LLM decision-client payloads such as SystemOne. This option belongs to the
+native Relay plugin; enabling standalone Switchyard TRACE logging does not
+enable payload capture.
+
+Payloads can contain sensitive task data. To mask configured provider keys,
+the plugin decodes the captured JSON, redacts it, and serializes the copy again.
+JSON strings, such as classifier content and tool arguments, are decoded too.
+Traversal is limited to 64 levels, including JSON encoded inside strings.
+Whitespace and escape spellings may change. If JSON-looking text cannot be
+decoded or the depth limit is exceeded, the body is omitted (`body: null`).
+Captured data passes through Relay's normal event
+sanitizers and configured exporters. Personal data and other secrets require
+suitable sanitizers; see Relay's
+[PII redaction options](https://docs.nvidia.com/nemo/relay/configure-plugins/pii-redaction/about).
+
### How Routing Appears in Traces
In Relay's
@@ -438,9 +529,10 @@ Configure delivery through Relay's
## Data Handling
-Switchyard routing telemetry excludes request and response content. Its marks
-do not contain prompts, bodies, headers, credentials, raw provider responses,
-or free-form provider errors. Relay's LLM events can capture request and
-response data according to Relay's
+By default, Switchyard routing telemetry excludes request and response content.
+Enabling [routing payload capture](#inspect-classifier-and-judge-payloads) adds
+internal routing requests and successful buffered responses to
+`libsy.upstream_attempt` scopes. Routing marks remain payload-free.
+Relay's LLM events can capture request and response data according to Relay's
[input and output event semantics](https://docs.nvidia.com/nemo/relay/about-nemo-relay/concepts/events#input-and-output-payloads),
independently of these Switchyard marks.