Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

feat(ai-gateway): capture Responses facts in debug logs (#17442)

Hassieb Pakzadaf8f4f192355
diff --git a/ai-gateway/README.md b/ai-gateway/README.md
index 229e9866ff..336e72e09e 100644
--- a/ai-gateway/README.md
+++ b/ai-gateway/README.md
@@ -2,7 +2,8 @@
A standalone Rust gateway for `POST /openai/v1/responses`. Web resolves the gateway
key to a trusted provider connection; Rust relays native JSON or SSE without
-parsing the request or rewriting provider bytes. Customer telemetry is not exported
+rewriting provider bytes. A bounded capture layer captures request/response facts and
+logs the finalized record at debug level. Customer telemetry is not exported
yet. Building and testing need no real Web, database or provider credentials.
## Run locally
@@ -81,8 +82,10 @@ Both formats use the same log level and preserve event fields. For example:
2026-09-10T13:20:00Z INFO gateway listening address=0.0.0.0:8080
```
-Invalid values fail startup without echoing their contents. Logs contain service
-lifecycle events, not request bodies or headers. For direct
+Invalid values fail startup without echoing their contents. At the default `info`
+level, logs contain service lifecycle events. Debug logging also includes the
+captured request and completed output when Web resolves `ingestion_mode: full`;
+see capture details below. For direct
host-only development, set `LANGFUSE_AI_GATEWAY_LISTEN_ADDRESS=127.0.0.1:8080`.
## Call the gateway
@@ -122,7 +125,8 @@ print(response.output_text)
```
The gateway forwards to the official OpenAI Responses endpoint only. It does not
-retry, follow redirects, parse `model`/`stream`, or accept client routing overrides.
+retry, follow redirects, or accept client routing overrides. Request parsing is
+best-effort capture only; Web still selects the provider connection.
Provider errors retain their status and body. Gateway errors use an OpenAI-style
`{"error":{"message":"...","type":"...","param":null,"code":"..."}}` envelope.
@@ -146,12 +150,79 @@ There is no separate downstream stall timeout; a client that stops reading can
retain execution capacity until the overall deadline. A progress-based downstream
stall policy is deferred to a separate change.
-Only request content type/encoding and accept/accept-encoding cross the provider
-boundary, plus the resolved Bearer token. Response content type/encoding, cache
-control, retry-after, request ID and selected OpenAI timing/version/rate-limit
+Only request content type/encoding and accept cross the provider boundary, plus
+the resolved Bearer token. The gateway sets `Accept-Encoding: identity` upstream
+so client compression preferences cannot disable JSON/SSE capture. Response
+content type/encoding, cache control, retry-after, request ID and selected OpenAI
+timing/version/rate-limit
headers are retained. Cookies, routing overrides, gateway/ingestion credentials,
hop-by-hop headers and upstream framing are excluded.
+## Response capture
+
+Set `LANGFUSE_LOG_LEVEL=debug` to print one `gateway response captured` event at
+execution end. Its `capture` field is JSON with `input`, `output`, model,
+parameters, usage, attribution, timestamps, capture completeness and relay outcome.
+Text logs pretty-print the captured JSON across multiple lines.
+`LANGFUSE_LOG_FORMAT=json` keeps each surrounding log event on one line, with
+newlines escaped inside the `capture` string.
+
+Web's resolved ingestion mode controls content capture:
+
+- `full`: input is the native request JSON object, including `input`, `instructions`,
+ tools, parameters, context references and unknown fields. Output is an ordered
+ array of native completed items. Debug logs intentionally contain this content.
+- `usage`: input and output are null. Model, scalar parameters, native usage,
+ timing and trusted attribution are retained; request schemas/content are omitted.
+
+In both modes, `usage_details` preserves the provider's entire usage object,
+including nested and unknown fields. The gateway does not rename counters,
+subtract cached tokens, or synthesize totals. Missing usage stays null.
+`api_format` identifies the payload format (`openai.responses` today).
+
+For SSE, only `response.output_item.done` adds output. Terminal Responses events
+provide model, service tier, status and usage; their repeated output is not copied.
+Deltas are never stitched or retained. A disconnect midway through an item loses
+that unfinished item; already completed items remain in the partial record. Item
+completion alone is not response completion. JSON responses capture their native
+output array at EOF. Missing usage remains null, rather than invented zeros.
+
+`outcome` describes the relay (`eof`, `cancelled`, `timeout`, `transport_error`);
+`provider_status` is separate. For example, a completed provider response can
+still end with downstream cancellation. `first_byte_ms` measures the first body
+bytes observed by the gateway, not first-token latency. No completion-start time
+is fabricated from completed items.
+
+`input_complete` means the full native request was captured; it is false when
+content is omitted in usage mode. `output_complete` means response inspection
+finished without capture gaps for the configured mode (including a terminal event
+and all expected completed items for full-mode SSE). It does not promise that the
+provider succeeded or that delivery to the client finished. `capture_complete`
+combines request inspection and output completeness; intentionally omitted usage-mode
+content does not make it false. Request inspection failure does not invalidate a
+fully captured response, and downstream cancellation does not erase completed capture.
+
+The implementation separates shared `ExecutionCapture` lifecycle/timing, the
+`OpenAiResponsesCapture` adapter and bounded `SseDecoder`. A private `ProtocolCapture`
+enum dispatches to the adapter. Finalization hands an owned `InferenceFacts` record
+to `telemetry::record`, which currently only pretty-prints at debug level. Another
+provider can supply native facts through the same interface without changing the
+relay or final sink. No Anthropic adapter or Langfuse observation/span mapping is
+implemented yet. Capture runs independently of log level; only emission is gated.
+
+Capture is limited to 1 MiB request inspection, 1 MiB JSON/SSE event inspection,
+1 MiB retained output and 256 output items per execution. The active-request limit
+bounds the number of captures. Trusted key metadata is bounded by the resolver's
+256 KiB response limit. Oversized input is omitted; oversized output items
+are skipped and completeness is false. Malformed, truncated or compressed bodies
+do not interrupt the relay. Capture buffers are independent of forwarding, so
+these limits never cap the actual provider response. Media is not fetched/uploaded.
+
+Provider credentials, the gateway credential and ingestion tokens are not part of
+the capture record; raw headers are not logged. Full captured content and resolved
+key metadata may themselves be sensitive. Keep debug logging enabled only where
+that content is intended to be recorded. No data is posted to Langfuse yet.
+
## Process lifecycle
- `/health` returns 200 while HTTP is serving. It never probes dependencies.
diff --git a/ai-gateway/src/capture/facts.rs b/ai-gateway/src/capture/facts.rs
new file mode 100644
index 0000000000..ee12dfed8d
--- /dev/null
+++ b/ai-gateway/src/capture/facts.rs
@@ -0,0 +1,42 @@
+//! Owned facts at the capture/telemetry boundary. Content and usage remain native JSON.
+use serde::Serialize;
+use serde_json::{Map, Value};
+
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)]
+#[serde(rename_all = "snake_case")]
+pub(crate) enum RelayOutcome {
+ Eof,
+ Cancelled,
+ Timeout,
+ TransportError,
+}
+
+#[derive(Serialize)]
+pub(crate) struct InferenceFacts {
+ pub api_format: &'static str,
+ pub start_time_unix_ms: u128,
+ pub duration_ms: u128,
+ pub first_byte_ms: Option<u128>,
+ pub http_status: Option<u16>,
+ pub metadata: Value,
+ pub outcome: RelayOutcome,
+ #[serde(flatten)]
+ pub inference: ProviderFacts,
+}
+
+/// Provider adapters fill this common envelope without translating native payloads.
+#[derive(Default, Serialize)]
+pub(crate) struct ProviderFacts {
+ pub model: Option<String>,
+ pub requested_model: Option<String>,
+ pub model_parameters: Map<String, Value>,
+ pub usage_details: Option<Value>,
+ pub input: Option<Value>,
+ pub output: Option<Value>,
+ pub provider_response_id: Option<String>,
+ pub provider_request_id: Option<String>,
+ pub provider_status: Option<String>,
+ pub input_complete: bool,
+ pub output_complete: bool,
+ pub capture_complete: bool,
+}
diff --git a/ai-gateway/src/capture/mod.rs b/ai-gateway/src/capture/mod.rs
new file mode 100644
index 0000000000..2ceda4752e
--- /dev/null
+++ b/ai-gateway/src/capture/mod.rs
@@ -0,0 +1,168 @@
+//! Best-effort capture around the existing relay lifecycle.
+mod facts;
+mod openai_responses;
+mod sse;
+
+use std::time::{SystemTime, UNIX_EPOCH};
+
+use axum::http::{HeaderMap, header};
+use serde_json::{Map, Value, json};
+use tokio::time::Instant;
+
+use crate::{
+ resolution::{IngestionMode, MetadataValue, ResolvedRequestContext},
+ telemetry,
+};
+use facts::ProviderFacts;
+pub(crate) use facts::{InferenceFacts, RelayOutcome};
+use openai_responses::OpenAiResponsesCapture;
+
+const MAX_CAPTURE_BYTES: usize = 1024 * 1024;
+const MAX_ITEMS: usize = 256;
+const MAX_FACT_STRING: usize = 512;
+
+/// Add another adapter here when another API is supported; the relay stays shared.
+enum ProtocolCapture {
+ OpenAiResponses(OpenAiResponsesCapture),
+}
+
+impl ProtocolCapture {
+ fn response(&mut self, headers: &HeaderMap) {
+ match self {
+ Self::OpenAiResponses(capture) => capture.response(headers),
+ }
+ }
+
+ fn bytes(&mut self, bytes: &[u8]) {
+ match self {
+ Self::OpenAiResponses(capture) => capture.bytes(bytes),
+ }
+ }
+
+ fn end_body(&mut self) {
+ match self {
+ Self::OpenAiResponses(capture) => capture.end_body(),
+ }
+ }
+
+ fn into_facts(self) -> (&'static str, ProviderFacts) {
+ match self {
+ Self::OpenAiResponses(capture) => ("openai.responses", capture.into_facts()),
+ }
+ }
+}
+
+/// Owned before dispatch and moved into the response body. Drop also covers a
+/// cancelled provider future before response headers have arrived.
+pub(crate) struct ExecutionCapture {
+ protocol: Option<ProtocolCapture>,
+ started: Instant,
+ start_time_unix_ms: u128,
+ first_byte_ms: Option<u128>,
+ http_status: Option<u16>,
+ metadata: Value,
+}
+
+impl ExecutionCapture {
+ pub fn openai_responses(
+ context: &ResolvedRequestContext,
+ headers: &HeaderMap,
+ body: &[u8],
+ ) -> Self {
+ let started = Instant::now();
+ let start_time_unix_ms = SystemTime::now()
+ .duration_since(UNIX_EPOCH)
+ .unwrap_or_default()
+ .as_millis();
+ let attribution = context.attribution();
+ // Copy only attribution; never retain provider or ingestion credentials.
+ // The entire resolver response, including key metadata, is bounded at 256 KiB.
+ let key_metadata: Map<String, Value> = attribution
+ .key_metadata()
+ .iter()
+ .map(|(key, value)| {
+ let value = match value {
+ MetadataValue::String(value) => Value::String(value.clone()),
+ MetadataValue::Number(value) => Value::Number(value.clone()),
+ MetadataValue::Bool(value) => Value::Bool(*value),
+ };
+ (key.clone(), value)
+ })
+ .collect();
+ let full = context.ingestion_mode() == IngestionMode::Full;
+ Self {
+ protocol: Some(ProtocolCapture::OpenAiResponses(
+ OpenAiResponsesCapture::new(headers, body, context.ingestion_mode()),
+ )),
+ started,
+ start_time_unix_ms,
+ first_byte_ms: None,
+ http_status: None,
+ metadata: json!({
+ "organization_id": attribution.organization_id(),
+ "project_id": attribution.project_id(),
+ "key_id": attribution.key_id(),
+ "provider_connection_id": attribution.provider_connection_id(),
+ "key_metadata": key_metadata,
+ "ingestion_mode": if full { "full" } else { "usage" },
+ }),
+ }
+ }
+
+ pub fn response(&mut self, status: u16, headers: &HeaderMap) {
+ if let Some(protocol) = &mut self.protocol {
+ self.http_status = Some(status);
+ protocol.response(headers);
+ }
+ }
+
+ pub fn bytes(&mut self, bytes: &[u8]) {
+ if let Some(protocol) = &mut self.protocol {
+ if !bytes.is_empty() {
+ self.first_byte_ms
+ .get_or_insert_with(|| self.started.elapsed().as_millis());
+ }
+ protocol.bytes(bytes);
+ }
+ }
+
+ pub fn end_body(&mut self) {
+ if let Some(protocol) = &mut self.protocol {
+ protocol.end_body();
+ }
+ }
+
+ pub fn finish(&mut self, outcome: RelayOutcome) {
+ let Some(protocol) = self.protocol.take() else {
+ return;
+ };
+ let (api_format, inference) = protocol.into_facts();
+ telemetry::record(InferenceFacts {
+ api_format,
+ start_time_unix_ms: self.start_time_unix_ms,
+ duration_ms: self.started.elapsed().as_millis(),
+ first_byte_ms: self.first_byte_ms,
+ http_status: self.http_status,
+ metadata: std::mem::take(&mut self.metadata),
+ outcome,
+ inference,
+ });
+ }
+}
+
+impl Drop for ExecutionCapture {
+ fn drop(&mut self) {
+ self.finish(RelayOutcome::Cancelled);
+ }
+}
+
+fn bounded_string(value: &str) -> Option<String> {
+ (value.len() <= MAX_FACT_STRING).then(|| value.to_owned())
+}
+
+fn identity_encoding(headers: &HeaderMap) -> bool {
+ headers
+ .get_all(header::CONTENT_ENCODING)
+ .iter()
+ .all(|v| v.to_str().is_ok_and(|v| v.eq_ignore_ascii_case("identity")))
+}
diff --git a/ai-gateway/src/capture/openai_responses.rs b/ai-gateway/src/capture/openai_responses.rs
new file mode 100644
index 0000000000..b64419a086
--- /dev/null
+++ b/ai-gateway/src/capture/openai_responses.rs
@@ -0,0 +1,262 @@
+//! `OpenAI` Responses request, JSON response and completed SSE item capture.
+use super::{
+ MAX_CAPTURE_BYTES, MAX_FACT_STRING, MAX_ITEMS, bounded_string, facts::ProviderFacts,
+ identity_encoding, sse::SseDecoder,
+};
+use crate::resolution::IngestionMode;
+use axum::http::{HeaderMap, header};
+use serde_json::{Map, Value};
+use std::collections::BTreeMap;
+
+enum ResponseBody {
+ Unknown,
+ Json(Vec<u8>),
+ Sse(SseDecoder),
+ Unavailable,
+}
+
+pub(super) struct OpenAiResponsesCapture {
+ facts: ProviderFacts,
+ mode: IngestionMode,
+ body: ResponseBody,
+ items: BTreeMap<u64, Value>,
+ output_bytes: usize,
+ terminal: bool,
+ expected_items: Option<usize>,
+ request_complete: bool,
+ response_valid: bool,
+}
+
+impl OpenAiResponsesCapture {
+ pub fn new(headers: &HeaderMap, body: &[u8], mode: IngestionMode) -> Self {
+ let mut capture = Self {
+ facts: ProviderFacts::default(),
+ mode,
+ body: ResponseBody::Unknown,
+ items: BTreeMap::new(),
+ output_bytes: 0,
+ terminal: false,
+ expected_items: None,
+ request_complete: false,
+ response_valid: true,
+ };
+ if identity_encoding(headers)
+ && body.len() <= MAX_CAPTURE_BYTES
+ && let Ok(Value::Object(request)) = serde_json::from_slice(body)
+ {
+ capture.request(request);
+ capture.request_complete = true;
+ }
+ capture
+ }
+
+ pub fn response(&mut self, headers: &HeaderMap) {
+ self.facts.provider_request_id = headers
+ .get("x-request-id")
+ .and_then(|v| v.to_str().ok())
+ .and_then(bounded_string);
+ let content_type = headers
+ .get(header::CONTENT_TYPE)
+ .and_then(|v| v.to_str().ok())
+ .unwrap_or("")
+ .split(';')
+ .next()
+ .unwrap_or("")
+ .trim();
+ self.body = if !identity_encoding(headers) {
+ ResponseBody::Unavailable
+ } else if content_type.eq_ignore_ascii_case("text/event-stream") {
+ ResponseBody::Sse(SseDecoder::default())
+ } else if content_type.eq_ignore_ascii_case("application/json") {
+ ResponseBody::Json(Vec::new())
+ } else {
+ ResponseBody::Unavailable
+ };
+ if matches!(self.body, ResponseBody::Unavailable) {
+ self.response_valid = false;
+ }
+ }
+
+ pub fn bytes(&mut self, bytes: &[u8]) {
+ let mut body = std::mem::replace(&mut self.body, ResponseBody::Unavailable);
+ match &mut body {
+ ResponseBody::Sse(sse) => {
+ sse.push(bytes, MAX_CAPTURE_BYTES, |event| self.event(event));
+ }
+ ResponseBody::Json(buffer)
+ if buffer.len().saturating_add(bytes.len()) <= MAX_CAPTURE_BYTES =>
+ {
+ buffer.extend_from_slice(bytes);
+ }
+ ResponseBody::Json(_) => {
+ body = ResponseBody::Unavailable;
+ self.response_valid = false;
+ }
+ _ => {}
+ }
+ self.body = body;
+ }
+
+ pub fn end_body(&mut self) {
+ match std::mem::replace(&mut self.body, ResponseBody::Unavailable) {
+ ResponseBody::Json(bytes) => match serde_json::from_slice::<Value>(&bytes) {
+ Ok(Value::Object(response)) => {
+ self.response_facts(&response);
+ if let Some(output) = response.get("output").and_then(Value::as_array) {
+ for (index, item) in output.iter().enumerate() {
+ self.item(index as u64, item.clone());
+ }
+ self.facts.output_complete = self.response_valid;
+ }
+ }
+ _ => self.response_valid = false,
+ },
+ ResponseBody::Sse(sse) => {
+ self.response_valid &= sse.finish();
+ self.facts.output_complete = self.terminal
+ && self.response_valid
+ && (self.mode != IngestionMode::Full
+ || self.expected_items.is_some_and(|count| {
+ count == self.items.len()
+ && self.items.keys().copied().eq(0..count as u64)
+ }));
+ }
+ _ => {}
+ }
+ self.facts.capture_complete = self.request_complete && self.facts.output_complete;
+ }
+
+ pub fn into_facts(mut self) -> ProviderFacts {
+ if self.mode == IngestionMode::Full {
+ self.facts.output = Some(Value::Array(self.items.into_values().collect()));
+ }
+ self.facts
+ }
+
+ fn request(&mut self, request: Map<String, Value>) {
+ self.facts.requested_model = request
+ .get("model")
+ .and_then(Value::as_str)
+ .and_then(bounded_string);
+ self.facts.model.clone_from(&self.facts.requested_model);
+ for key in ["temperature", "top_p", "max_output_tokens", "service_tier"] {
+ if let Some(value) = request
+ .get(key)
+ .filter(|v| v.is_number() || v.as_str().is_some_and(|s| s.len() <= MAX_FACT_STRING))
+ {
+ self.facts
+ .model_parameters
+ .insert(key.to_owned(), value.clone());
+ }
+ }
+ if self.mode == IngestionMode::Full {
+ for key in ["reasoning", "text"] {
+ if let Some(value) = request.get(key) {
+ self.facts
+ .model_parameters
+ .insert(key.to_owned(), value.clone());
+ }
+ }
+ // Keep the original request object, including native instructions/input
+ // and unknown fields. Authentication headers are never part of this value.
+ self.facts.input = Some(Value::Object(request));
+ self.facts.input_complete = true;
+ }
+ }
+
+ fn event(&mut self, bytes: &[u8]) {
+ let Ok(Value::Object(mut event)) = serde_json::from_slice(bytes) else {
+ self.response_valid = false;
+ return;
+ };
+ match event.get("type").and_then(Value::as_str) {
+ Some("response.output_item.done") if self.mode == IngestionMode::Full => {
+ if let (Some(index), Some(item)) = (
+ event.get("output_index").and_then(Value::as_u64),
+ event.remove("item").filter(Value::is_object),
+ ) {
+ self.item(index, item);
+ } else {
+ self.response_valid = false;
+ }
+ }
+ Some(
+ "response.created"
+ | "response.in_progress"
+ | "response.completed"
+ | "response.failed"
+ | "response.incomplete",
+ ) => {
+ let terminal = matches!(
+ event.get("type").and_then(Value::as_str),
+ Some("response.completed" | "response.failed" | "response.incomplete")
+ );
+ if let Some(response) = event.get("response").and_then(Value::as_object) {
+ self.response_facts(response);
+ if terminal {
+ self.terminal = true;
+ self.expected_items = response
+ .get("output")
+ .and_then(Value::as_array)
+ .map(Vec::len);
+ }
+ } else {
+ self.response_valid = false;
+ }
+ }
+ Some("error") => self.facts.provider_status = Some("failed".to_owned()),
+ _ => {}
+ }
+ }
+
+ fn item(&mut self, index: u64, item: Value) {
+ if self.mode != IngestionMode::Full {
+ return;
+ }
+ let size = item.to_string().len();
+ let previous = self
+ .items
+ .get(&index)
+ .map_or(0, |item| item.to_string().len());
+ if self.output_bytes - previous + size > MAX_CAPTURE_BYTES
+ || (!self.items.contains_key(&index) && self.items.len() >= MAX_ITEMS)
+ {
+ self.response_valid = false;
+ return;
+ }
+ self.output_bytes = self.output_bytes - previous + size;
+ self.items.insert(index, item);
+ }
+
+ fn response_facts(&mut self, response: &Map<String, Value>) {
+ for (key, target) in [
+ ("id", &mut self.facts.provider_response_id),
+ ("model", &mut self.facts.model),
+ ("status", &mut self.facts.provider_status),
+ ] {
+ if let Some(value) = response
+ .get(key)
+ .and_then(Value::as_str)
+ .and_then(bounded_string)
+ {
+ *target = Some(value);
+ }
+ }
+ if let Some(tier) = response
+ .get("service_tier")
+ .and_then(Value::as_str)
+ .and_then(bounded_string)
+ {
+ self.facts
+ .model_parameters
+ .insert("service_tier".to_owned(), Value::String(tier));
+ }
+ if let Some(usage) = response.get("usage").filter(|v| v.is_object()) {
+ // The enclosing JSON body or SSE event already enforces the byte limit.
+ self.facts.usage_details = Some(usage.clone());
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests;
diff --git a/ai-gateway/src/capture/openai_responses/tests.rs b/ai-gateway/src/capture/openai_responses/tests.rs
new file mode 100644
index 0000000000..89c78611d4
--- /dev/null
+++ b/ai-gateway/src/capture/openai_responses/tests.rs
@@ -0,0 +1,667 @@
+use super::*;
+use crate::capture::{ExecutionCapture, ProtocolCapture, RelayOutcome};
+use crate::{
+ providers::openai::{OpenAiProvider, ProviderLimits},
+ test_support::{FakeServer, resolved_request_context_with_mode},
+};
+use axum::{
+ body::{Body, Bytes, to_bytes},
+ http::{HeaderValue, Response},
+};
+use serde_json::json;
+use std::{
+ convert::Infallible,
+ future::pending,
+ sync::{Arc, Mutex},
+ time::Duration,
+};
+use tokio::sync::Notify;
+
+fn captured(execution: &ExecutionCapture) -> &OpenAiResponsesCapture {
+ let ProtocolCapture::OpenAiResponses(capture) = execution.protocol.as_ref().unwrap();
+ capture
+}
+
+async fn observer(mode: &'static str) -> ExecutionCapture {
+ let context = resolved_request_context_with_mode("provider-secret", mode).await;
+ ExecutionCapture::openai_responses(&context, &HeaderMap::new(), br#"{"model":"requested","instructions":"prompt-canary","input":[{"role":"user","content":"hello"}],"tools":[{"type":"function","name":"weather","parameters":{"type":"object"}}],"text":{"format":{"type":"json_schema","schema":{"description":"schema-canary"}}},"service_tier":"auto"}"#)
+}
+
+fn response(observer: &mut ExecutionCapture, content_type: &'static str) {
+ let mut headers = HeaderMap::new();
+ headers.insert(header::CONTENT_TYPE, HeaderValue::from_static(content_type));
+ headers.insert(
+ "x-request-id",
+ HeaderValue::from_static("provider-request-1"),
+ );
+ observer.response(200, &headers);
+}
+
+fn event(kind: &str, fields: Value) -> String {
+ let mut value = fields;
+ value["type"] = kind.into();
+ format!("event: {kind}\ndata: {value}\n\n")
+}
+
+fn completed_item(index: u64, text: &str) -> String {
+ event(
+ "response.output_item.done",
+ json!({"output_index":index,"item":{"type":"message","role":"assistant","content":[{"type":"output_text","text":text}],"future":{"preserved":true}}}),
+ )
+}
+
+fn terminal(status: &str, count: usize) -> String {
+ event(
+ &format!("response.{status}"),
+ json!({"response":{"id":"resp-1","model":"actual","status":status,"service_tier":"priority","usage":{"input_tokens":100,"output_tokens":30,"total_tokens":130,"input_tokens_details":{"cached_tokens":40},"output_tokens_details":{"reasoning_tokens":10}},"output":vec![json!({"text":"terminal-output-must-not-be-copied"});count]}}),
+ )
+}
+
+#[tokio::test]
+async fn captures_completed_items_only_in_order_without_duplication() {
+ let mut observer = observer("full").await;
+ response(&mut observer, "text/event-stream; charset=utf-8");
+ let body = format!(
+ ": ping\n\n{}{}{}{}{}",
+ event(
+ "response.output_text.delta",
+ json!({"delta":"unfinished-canary"})
+ ),
+ completed_item(1, "second"),
+ completed_item(0, "héllo 🌍"),
+ completed_item(0, "héllo 🌍"),
+ terminal("completed", 2)
+ );
+ for byte in body.as_bytes() {
+ observer.bytes(&[*byte]);
+ }
+ observer.end_body();
+ let capture = captured(&observer);
+ assert_eq!(capture.items.len(), 2);
+ assert_eq!(capture.items[&0]["content"][0]["text"], "héllo 🌍");
+ assert_eq!(capture.items[&1]["content"][0]["text"], "second");
+ assert_eq!(capture.items[&0]["future"]["preserved"], true);
+ assert_eq!(capture.facts.model.as_deref(), Some("actual"));
+ assert_eq!(capture.facts.model_parameters["service_tier"], "priority");
+ assert_eq!(
+ capture.facts.usage_details.as_ref().unwrap()["input_tokens_details"]["cached_tokens"],
+ 40
+ );
+ assert_eq!(
+ capture.facts.input.as_ref().unwrap()["instructions"],
+ "prompt-canary"
+ );
+ assert!(capture.facts.output_complete);
+ let stored = json!({"record":capture.facts,"items":capture.items}).to_string();
+ assert!(!stored.contains("unfinished-canary"));
+ assert!(!stored.contains("terminal-output-must-not-be-copied"));
+}
+
+#[tokio::test]
+async fn eof_and_item_completion_are_not_provider_completion() {
+ let mut observer = observer("full").await;
+ response(&mut observer, "text/event-stream");
+ observer.bytes(completed_item(0, "completed item").as_bytes());
+ observer.bytes(event("response.output_text.delta", json!({"delta":"unfinished"})).as_bytes());
+ observer.end_body();
+ let capture = captured(&observer);
+ assert_eq!(capture.items.len(), 1);
+ assert!(!capture.facts.output_complete);
+ assert!(capture.facts.usage_details.is_none());
+ assert!(capture.facts.provider_status.is_none());
+}
+
+#[tokio::test]
+async fn failed_and_incomplete_responses_keep_usage_and_native_status() {
+ for status in ["failed", "incomplete"] {
+ let mut observer = observer("full").await;
+ response(&mut observer, "text/event-stream");
+ observer.bytes(terminal(status, 0).as_bytes());
+ observer.end_body();
+ let capture = captured(&observer);
+ assert_eq!(capture.facts.provider_status.as_deref(), Some(status));
+ assert_eq!(
+ capture.facts.usage_details.as_ref().unwrap()["total_tokens"],
+ 130
+ );
+ assert!(capture.facts.output_complete);
+ }
+}
+
+#[tokio::test]
+async fn json_and_streaming_capture_equivalent_native_items_and_facts() {
+ let item = json!({"type":"function_call","id":"item-1","call_id":"call-1","name":"weather","arguments":"{\"city\":\"Berlin\"}"});
+ let mut json_observer = observer("full").await;
+ response(&mut json_observer, "application/json");
+ let body = json!({"id":"resp-1","model":"actual","status":"completed","output":[item.clone()],"usage":{"input_tokens":5,"output_tokens":2,"total_tokens":7}}).to_string();
+ for chunk in body.as_bytes().chunks(3) {
+ json_observer.bytes(chunk);
+ }
+ json_observer.end_body();
+ let captured = captured(&json_observer);
+ assert_eq!(captured.items[&0], item);
+ assert!(captured.facts.output_complete);
+ assert_eq!(
+ captured.facts.usage_details.as_ref().unwrap()["total_tokens"],
+ 7
+ );
+}
+
+#[tokio::test]
+async fn usage_mode_excludes_content_and_context_credentials() {
+ let mut observer = observer("usage").await;
+ response(&mut observer, "text/event-stream");
+ observer.bytes(completed_item(0, "output-canary").as_bytes());
+ observer.bytes(terminal("completed", 1).as_bytes());
+ observer.end_body();
+ let capture = captured(&observer);
+ assert!(capture.facts.input.is_none());
+ assert!(capture.items.is_empty());
+ assert!(capture.facts.usage_details.is_some());
+ let text = json!(capture.facts).to_string();
+ for secret in [
+ "prompt-canary",
+ "schema-canary",
+ "output-canary",
+ "provider-secret",
+ "private-ingestion-token",
+ "gateway-secret",
+ ] {
+ assert!(!text.contains(secret), "leaked {secret}");
+ }
+}
+
+#[tokio::test]
+async fn oversized_and_malformed_events_do_not_prevent_later_facts() {
+ let mut observer = observer("full").await;
+ response(&mut observer, "text/event-stream");
+ observer.bytes(b"data: malformed\n\n");
+ observer.bytes(format!("data: {}\n\n", "x".repeat(MAX_CAPTURE_BYTES + 1)).as_bytes());
+ observer.bytes(terminal("completed", 0).as_bytes());
+ observer.end_body();
+ let capture = captured(&observer);
+ assert!(!capture.facts.capture_complete);
+ assert!(!capture.facts.output_complete);
+ assert_eq!(capture.facts.provider_status.as_deref(), Some("completed"));
+ assert_eq!(
+ capture.facts.usage_details.as_ref().unwrap()["total_tokens"],
+ 130
+ );
+}
+
+#[tokio::test]
+async fn capture_limits_and_encoded_bodies_are_explicit() {
+ let mut observer = observer("full").await;
+ response(&mut observer, "text/event-stream");
+ for index in 0..=MAX_ITEMS {
+ observer.bytes(completed_item(index as u64, "x").as_bytes());
+ }
+ assert_eq!(captured(&observer).items.len(), MAX_ITEMS);
+ assert!(!captured(&observer).facts.capture_complete);
+ let mut encoded = observer;
+ let mut headers = HeaderMap::new();
+ headers.insert(
+ header::CONTENT_TYPE,
+ HeaderValue::from_static("application/json"),
+ );
+ headers.insert(header::CONTENT_ENCODING, HeaderValue::from_static("gzip"));
+ encoded.response(200, &headers);
+ encoded.bytes(b"opaque compressed bytes");
+ encoded.end_body();
+ assert!(matches!(captured(&encoded).body, ResponseBody::Unavailable));
+ assert!(!captured(&encoded).facts.output_complete);
+}
+
+#[test]
+fn sse_handles_crlf_multiline_data_and_truncated_eof() {
+ let mut decoder = SseDecoder::default();
+ let mut events = Vec::new();
+ for byte in b": ping\r\nevent: anything\r\ndata: {\r\ndata: \"ok\": true}\r\n\r\n" {
+ decoder.push(&[*byte], 1024, |event| events.push(event.to_vec()));
+ }
+ assert_eq!(events, vec![b"{\n\"ok\": true}".to_vec()]);
+ assert!(decoder.finish());
+ decoder.push(b"data: {\"unfinished\":true}", 1024, |_| {
+ panic!("must wait for delimiter")
+ });
+ assert!(!decoder.finish());
+}
+
+#[derive(Clone, Default)]
+struct LogWriter(Arc<Mutex<Vec<u8>>>);
+impl std::io::Write for LogWriter {
+ fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
+ self.0.lock().unwrap().extend_from_slice(bytes);
+ Ok(bytes.len())
+ }
+ fn flush(&mut self) -> std::io::Result<()> {
+ Ok(())
+ }
+}
+impl<'a> tracing_subscriber::fmt::MakeWriter<'a> for LogWriter {
+ type Writer = Self;
+ fn make_writer(&'a self) -> Self::Writer {
+ self.clone()
+ }
+}
+
+#[tokio::test]
+async fn debug_record_is_emitted_once_and_only_at_debug_level() {
+ for level in [tracing::Level::DEBUG, tracing::Level::INFO] {
+ let mut observer = observer("full").await;
+ response(&mut observer, "text/event-stream");
+ observer.bytes(completed_item(0, "captured-output").as_bytes());
+ observer.bytes(terminal("completed", 1).as_bytes());
+ observer.end_body();
+ let writer = LogWriter::default();
+ let subscriber = tracing_subscriber::fmt()
+ .json()
+ .with_max_level(level)
+ .with_writer(writer.clone())
+ .finish();
+ tracing::subscriber::with_default(subscriber, || {
+ observer.finish(RelayOutcome::Eof);
+ observer.finish(RelayOutcome::Cancelled);
+ drop(observer);
+ });
+ let text = String::from_utf8(writer.0.lock().unwrap().clone()).unwrap();
+ if level == tracing::Level::DEBUG {
+ assert_eq!(text.lines().count(), 1);
+ let event: Value = serde_json::from_str(&text).unwrap();
+ let capture: Value =
+ serde_json::from_str(event["fields"]["capture"].as_str().unwrap()).unwrap();
+ assert_eq!(
+ capture["output"][0]["content"][0]["text"],
+ "captured-output"
+ );
+ assert_eq!(capture["outcome"], "eof");
+ assert_eq!(capture["input"]["instructions"], "prompt-canary");
+ for secret in [
+ "provider-secret",
+ "private-ingestion-token",
+ "gateway-secret",
+ ] {
+ assert!(!text.contains(secret));
+ }
+ } else {
+ assert!(text.is_empty());
+ }
+ }
+}
+
+fn records(writer: &LogWriter) -> Vec<Value> {
+ String::from_utf8(writer.0.lock().unwrap().clone())
+ .unwrap()
+ .lines()
+ .filter_map(|line| {
+ let event: Value = serde_json::from_str(line).unwrap();
+ event["fields"]["capture"]
+ .as_str()
+ .map(|capture| serde_json::from_str(capture).unwrap())
+ })
+ .collect()
+}
+
+#[tokio::test]
+async fn upstream_read_timeout_is_logged_as_timeout() {
+ let upstream = FakeServer::start(|_| async {
+ Response::builder()
+ .header("content-type", "text/event-stream")
+ .body(Body::from_stream(futures_util::stream::pending::<
+ Result<Bytes, Infallible>,
+ >()))
+ .unwrap()
+ })
+ .await;
+ let context = resolved_request_context_with_mode("provider-secret", "full").await;
+ let provider = OpenAiProvider::for_test(
+ format!("{}/v1/responses", upstream.url),
+ ProviderLimits {
+ active: 1,
+ read_timeout: Duration::from_millis(50),
+ ..ProviderLimits::default()
+ },
+ );
+ let writer = LogWriter::default();
+ let subscriber = tracing_subscriber::fmt()
+ .json()
+ .with_max_level(tracing::Level::DEBUG)
+ .with_writer(writer.clone())
+ .finish();
+ let _guard = tracing::subscriber::set_default(subscriber);
+ let response = provider
+ .forward(
+ provider.try_admit().unwrap(),
+ context,
+ &HeaderMap::new(),
+ Bytes::from_static(b"{}"),
+ )
+ .await
+ .unwrap();
+ assert!(to_bytes(response.into_body(), 4096).await.is_err());
+ let records = records(&writer);
+ assert_eq!(records.len(), 1);
+ assert_eq!(records[0]["outcome"], "timeout");
+ assert!(provider.try_admit().is_ok());
+}
+
+#[tokio::test]
+async fn native_http_relay_logs_once_on_eof_drop_and_unpolled_deadline() {
+ for outcome in [
+ RelayOutcome::Eof,
+ RelayOutcome::Cancelled,
+ RelayOutcome::Timeout,
+ ] {
+ let native = format!(
+ "{}{}",
+ completed_item(0, "native-result"),
+ terminal("completed", 1)
+ );
+ let expected = native.clone();
+ let upstream = FakeServer::start(move |_| {
+ let native = native.clone();
+ async move {
+ Response::builder()
+ .header("content-type", "text/event-stream")
+ .body(Body::from(native))
+ .unwrap()
+ }
+ })
+ .await;
+ let context = resolved_request_context_with_mode("provider-secret", "full").await;
+ let provider = OpenAiProvider::for_test(
+ format!("{}/v1/responses", upstream.url),
+ ProviderLimits {
+ active: 1,
+ execution_timeout: Duration::from_secs(1),
+ ..ProviderLimits::default()
+ },
+ );
+ let writer = LogWriter::default();
+ let subscriber = tracing_subscriber::fmt()
+ .json()
+ .with_max_level(tracing::Level::DEBUG)
+ .with_writer(writer.clone())
+ .finish();
+ let _guard = tracing::subscriber::set_default(subscriber);
+ let response = provider
+ .forward(
+ provider.try_admit().unwrap(),
+ context,
+ &HeaderMap::new(),
+ Bytes::from_static(br#"{"input":"request-content"}"#),
+ )
+ .await
+ .unwrap();
+ match outcome {
+ RelayOutcome::Eof => assert_eq!(
+ to_bytes(response.into_body(), 4096).await.unwrap(),
+ expected
+ ),
+ RelayOutcome::Cancelled => drop(response),
+ RelayOutcome::Timeout => {
+ tokio::time::sleep(Duration::from_millis(1100)).await;
+ assert!(provider.try_admit().is_ok());
+ drop(response);
+ }
+ RelayOutcome::TransportError => unreachable!(),
+ }
+ tokio::task::yield_now().await;
+ let records = records(&writer);
+ assert_eq!(records.len(), 1);
+ assert_eq!(records[0]["outcome"], json!(outcome));
+ if outcome == RelayOutcome::Eof {
+ assert_eq!(
+ records[0]["output"][0]["content"][0]["text"],
+ "native-result"
+ );
+ assert_eq!(records[0]["output_complete"], true);
+ assert!(records[0]["first_byte_ms"].is_number());
+ }
+ assert!(provider.try_admit().is_ok());
+ assert_eq!(upstream.calls(), 1);
+ }
+}
+
+#[tokio::test]
+async fn provider_future_finalizes_on_timeout_and_cancellation_before_headers() {
+ for cancel in [false, true] {
+ let started = Arc::new(Notify::new());
+ let signal = started.clone();
+ let upstream = FakeServer::start(move |_| {
+ let signal = signal.clone();
+ async move {
+ signal.notify_one();
+ pending::<Response<Body>>().await
+ }
+ })
+ .await;
+ let context = resolved_request_context_with_mode("provider-secret", "full").await;
+ let provider = OpenAiProvider::for_test(
+ format!("{}/v1/responses", upstream.url),
+ ProviderLimits {
+ active: 1,
+ headers_timeout: Duration::from_millis(100),
+ ..ProviderLimits::default()
+ },
+ );
+ let writer = LogWriter::default();
+ let subscriber = tracing_subscriber::fmt()
+ .json()
+ .with_max_level(tracing::Level::DEBUG)
+ .with_writer(writer.clone())
+ .finish();
+ let _guard = tracing::subscriber::set_default(subscriber);
+ let headers = HeaderMap::new();
+ let call = provider.forward(
+ provider.try_admit().unwrap(),
+ context,
+ &headers,
+ Bytes::from_static(b"{}"),
+ );
+ if cancel {
+ tokio::select! {
+ _ = call => panic!("provider must still be waiting"),
+ () = started.notified() => {},
+ }
+ } else {
+ assert!(matches!(
+ call.await,
+ Err(crate::transport::ProviderError::Timeout)
+ ));
+ }
+ let records = records(&writer);
+ assert_eq!(records.len(), 1);
+ assert_eq!(
+ records[0]["outcome"],
+ if cancel { "cancelled" } else { "timeout" }
+ );
+ assert!(records[0]["http_status"].is_null());
+ assert!(provider.try_admit().is_ok());
+ }
+}
+
+#[tokio::test]
+async fn client_compression_preferences_do_not_disable_capture() {
+ for streaming in [false, true] {
+ let native = if streaming {
+ format!(
+ "{}{}",
+ completed_item(0, "captured output"),
+ terminal("completed", 1)
+ )
+ } else {
+ json!({"id":"resp-1","status":"completed","model":"actual","output":[{"type":"message","role":"assistant","content":[{"type":"output_text","text":"captured output"}]}],"usage":{"input_tokens":10,"output_tokens":21,"total_tokens":31}}).to_string()
+ };
+ let expected = native.clone();
+ let upstream = FakeServer::start(move |request| {
+ let native = native.clone();
+ async move {
+ assert_eq!(request.headers()[header::ACCEPT_ENCODING], "identity");
+ Response::builder()
+ .header(
+ "content-type",
+ if streaming {
+ "text/event-stream"
+ } else {
+ "application/json"
+ },
+ )
+ .body(Body::from(native))
+ .unwrap()
+ }
+ })
+ .await;
+ let context = resolved_request_context_with_mode("provider-secret", "full").await;
+ let provider = OpenAiProvider::for_test(
+ format!("{}/v1/responses", upstream.url),
+ ProviderLimits::default(),
+ );
+ let writer = LogWriter::default();
+ let subscriber = tracing_subscriber::fmt()
+ .json()
+ .with_max_level(tracing::Level::DEBUG)
+ .with_writer(writer.clone())
+ .finish();
+ let _guard = tracing::subscriber::set_default(subscriber);
+ let mut headers = HeaderMap::new();
+ headers.insert(
+ header::ACCEPT_ENCODING,
+ HeaderValue::from_static("gzip, deflate, br"),
+ );
+ let response = provider
+ .forward(
+ provider.try_admit().unwrap(),
+ context,
+ &headers,
+ Bytes::from(json!({"input":"hello","stream":streaming}).to_string()),
+ )
+ .await
+ .unwrap();
+ assert_eq!(
+ to_bytes(response.into_body(), 4096).await.unwrap(),
+ expected
+ );
+ let records = records(&writer);
+ assert_eq!(records.len(), 1);
+ assert_eq!(records[0]["capture_complete"], true);
+ assert_eq!(records[0]["output_complete"], true);
+ assert_eq!(records[0]["provider_response_id"], "resp-1");
+ assert_eq!(records[0]["provider_status"], "completed");
+ assert!(records[0]["usage_details"]["input_tokens"].is_number());
+ assert_eq!(
+ records[0]["output"][0]["content"][0]["text"],
+ "captured output"
+ );
+ }
+}
+
+#[tokio::test]
+async fn capture_regression_large_request_does_not_invalidate_output() {
+ let context = resolved_request_context_with_mode("provider-secret", "full").await;
+ let request = json!({"input": "x".repeat(MAX_CAPTURE_BYTES)}).to_string();
+ for streaming in [false, true] {
+ let mut observer =
+ ExecutionCapture::openai_responses(&context, &HeaderMap::new(), request.as_bytes());
+ response(
+ &mut observer,
+ if streaming {
+ "text/event-stream"
+ } else {
+ "application/json"
+ },
+ );
+ let body = if streaming {
+ terminal("completed", 0)
+ } else {
+ json!({"status":"completed","output":[]}).to_string()
+ };
+ observer.bytes(body.as_bytes());
+ observer.end_body();
+ let facts = &captured(&observer).facts;
+ assert!(!facts.input_complete);
+ assert!(
+ facts.output_complete,
+ "response capture must be independent of request size"
+ );
+ }
+}
+
+#[tokio::test]
+async fn capture_regression_preserves_native_request_and_usage() {
+ let usage = json!({"input_tokens":5,"output_tokens":2,"input_tokens_details":{"cached_tokens":3,"future_tokens":1},"future_usage_field":{"region":"eu"}});
+ for mode in ["full", "usage"] {
+ let context = resolved_request_context_with_mode("provider-secret", mode).await;
+ for input in [
+ json!("hello"),
+ json!([{"type":"function_call_output","call_id":"call-1","output":"native result"}]),
+ ] {
+ let request = json!({"model":"requested","input":input,"instructions":"be brief","future_request_field":{"enabled":true}});
+ for streaming in [false, true] {
+ let mut execution = ExecutionCapture::openai_responses(
+ &context,
+ &HeaderMap::new(),
+ request.to_string().as_bytes(),
+ );
+ response(
+ &mut execution,
+ if streaming {
+ "text/event-stream"
+ } else {
+ "application/json"
+ },
+ );
+ let native = json!({"output":[],"usage":usage,"status":"completed"});
+ let body = if streaming {
+ event("response.completed", json!({"response":native}))
+ } else {
+ native.to_string()
+ };
+ execution.bytes(body.as_bytes());
+ execution.end_body();
+ let facts = &captured(&execution).facts;
+ assert_eq!(
+ facts.input.as_ref(),
+ if mode == "full" { Some(&request) } else { None }
+ );
+ assert_eq!(facts.usage_details.as_ref(), Some(&usage));
+ assert!(facts.capture_complete);
+ }
+ }
+ }
+}
+
+#[tokio::test]
+async fn downstream_cancellation_does_not_erase_completed_capture() {
+ for streaming in [false, true] {
+ let mut execution = observer("full").await;
+ response(
+ &mut execution,
+ if streaming {
+ "text/event-stream"
+ } else {
+ "application/json"
+ },
+ );
+ let body = if streaming {
+ terminal("completed", 0)
+ } else {
+ json!({"status":"completed","output":[]}).to_string()
+ };
+ execution.bytes(body.as_bytes());
+ execution.end_body();
+ let writer = LogWriter::default();
+ let subscriber = tracing_subscriber::fmt()
+ .json()
+ .with_max_level(tracing::Level::DEBUG)
+ .with_writer(writer.clone())
+ .finish();
+ tracing::subscriber::with_default(subscriber, || execution.finish(RelayOutcome::Cancelled));
+ let facts = records(&writer);
+ assert_eq!(facts.len(), 1);
+ assert_eq!(facts[0]["outcome"], "cancelled");
+ assert_eq!(facts[0]["provider_status"], "completed");
+ assert_eq!(facts[0]["output_complete"], true);
+ assert_eq!(facts[0]["capture_complete"], true);
+ }
+}
diff --git a/ai-gateway/src/capture/sse.rs b/ai-gateway/src/capture/sse.rs
new file mode 100644
index 0000000000..e749b25639
--- /dev/null
+++ b/ai-gateway/src/capture/sse.rs
@@ -0,0 +1,76 @@
+//! Incremental SSE framing. A bad or oversized event only affects capture.
+#[derive(Default)]
+pub(super) struct SseDecoder {
+ line: Vec<u8>,
+ data: Vec<u8>,
+ line_nonempty: bool,
+ skip_lf: bool,
+ state: EventState,
+ pub incomplete: bool,
+}
+
+#[derive(Default)]
+enum EventState {
+ #[default]
+ Collecting,
+ Discarding,
+}
+
+impl SseDecoder {
+ pub fn push(&mut self, bytes: &[u8], limit: usize, mut event: impl FnMut(&[u8])) {
+ for &byte in bytes {
+ if self.skip_lf {
+ self.skip_lf = false;
+ if byte == b'\n' {
+ continue;
+ }
+ }
+ if byte == b'\r' || byte == b'\n' {
+ self.end_line(&mut event);
+ self.skip_lf = byte == b'\r';
+ } else {
+ self.line_nonempty = true;
+ if matches!(self.state, EventState::Collecting) {
+ if self.line.len() + self.data.len() >= limit {
+ self.incomplete = true;
+ self.state = EventState::Discarding;
+ self.line.clear();
+ self.data.clear();
+ } else {
+ self.line.push(byte);
+ }
+ }
+ }
+ }
+ }
+
+ fn end_line(&mut self, event: &mut impl FnMut(&[u8])) {
+ if !self.line_nonempty {
+ if matches!(self.state, EventState::Collecting) && !self.data.is_empty() {
+ event(&self.data[..self.data.len() - 1]);
+ }
+ self.data.clear();
+ self.state = EventState::Collecting;
+ } else if matches!(self.state, EventState::Collecting) {
+ let value = self
+ .line
+ .strip_prefix(b"data:")
+ .map(|value| value.strip_prefix(b" ").unwrap_or(value));
+ if let Some(value) = value {
+ self.data.extend_from_slice(value);
+ self.data.push(b'\n');
+ } else if self.line == b"data" {
+ self.data.push(b'\n');
+ }
+ }
+ self.line.clear();
+ self.line_nonempty = false;
+ }
+
+ pub fn finish(&self) -> bool {
+ !self.incomplete
+ && !self.line_nonempty
+ && self.data.is_empty()
+ && matches!(self.state, EventState::Collecting)
+ }
+}
diff --git a/ai-gateway/src/lib.rs b/ai-gateway/src/lib.rs
index 60740a9181..ce4f00cec5 100644
--- a/ai-gateway/src/lib.rs
+++ b/ai-gateway/src/lib.rs
@@ -1,9 +1,11 @@
+mod capture;
pub mod config;
pub mod http;
pub mod inference;
pub mod providers;
pub mod resolution;
pub mod server;
+mod telemetry;
mod transport;
#[cfg(test)]
diff --git a/ai-gateway/src/providers/openai.rs b/ai-gateway/src/providers/openai.rs
index e8882905b4..71a6568056 100644
--- a/ai-gateway/src/providers/openai.rs
+++ b/ai-gateway/src/providers/openai.rs
@@ -11,7 +11,11 @@ use tokio::{
time::Instant,
};
-use crate::{resolution::ResolvedRequestContext, transport};
+use crate::{
+ capture::{ExecutionCapture, RelayOutcome},
+ resolution::ResolvedRequestContext,
+ transport,
+};
pub use crate::transport::ProviderError;
@@ -124,6 +128,7 @@ impl OpenAiProvider {
HeaderValue::from_str(&format!("Bearer {}", context.connection().provider_token()))
.map_err(|_| ProviderError::Configuration)?;
authorization.set_sensitive(true);
+ let mut capture = ExecutionCapture::openai_responses(&context, headers, &body);
let response = tokio::time::timeout_at(
permit
.deadline
@@ -131,23 +136,40 @@ impl OpenAiProvider {
self.client
.post(endpoint)
.headers(transport::request_headers(headers))
+ // Observe plain JSON/SSE while relaying the provider bytes unchanged.
+ .header(header::ACCEPT_ENCODING, "identity")
.header(header::AUTHORIZATION, authorization)
.body(body)
.send(),
)
.await
- .map_err(|_| ProviderError::Timeout)?
- .map_err(|error| {
- if error.is_timeout() {
- ProviderError::Timeout
- } else {
- ProviderError::Transport
+ .map_err(|_| ProviderError::Timeout)
+ .and_then(|result| {
+ result.map_err(|error| {
+ if error.is_timeout() {
+ ProviderError::Timeout
+ } else {
+ ProviderError::Transport
+ }
+ })
+ });
+ let response = match response {
+ Ok(response) => response,
+ Err(error) => {
+ capture.finish(if matches!(error, ProviderError::Timeout) {
+ RelayOutcome::Timeout
+ } else {
+ RelayOutcome::TransportError
+ });
+ return Err(error);
}
- })?;
+ };
+ capture.response(response.status().as_u16(), response.headers());
let mut downstream = Response::new(Body::empty());
*downstream.status_mut() = response.status();
*downstream.headers_mut() = transport::response_headers(response.headers());
- *downstream.body_mut() = transport::relay(response, permit.deadline, (permit, context));
+ *downstream.body_mut() =
+ transport::relay(response, permit.deadline, (permit, context), capture);
Ok(downstream)
}
diff --git a/ai-gateway/src/providers/openai/tests.rs b/ai-gateway/src/providers/openai/tests.rs
index 4c1bde86b8..8c1d502e72 100644
--- a/ai-gateway/src/providers/openai/tests.rs
+++ b/ai-gateway/src/providers/openai/tests.rs
@@ -27,6 +27,7 @@ async fn preserves_opaque_bytes_and_isolates_request_and_response_headers() {
"Bearer provider-secret"
);
assert_eq!(request.headers()[header::CONTENT_TYPE], "application/json");
+ assert_eq!(request.headers()[header::ACCEPT_ENCODING], "identity");
for name in [
"cookie",
"langfuse-gateway-authorization",
@@ -34,7 +35,6 @@ async fn preserves_opaque_bytes_and_isolates_request_and_response_headers() {
"openai-project",
"x-api-key",
"x-forwarded-host",
- "accept-encoding",
] {
assert!(!request.headers().contains_key(name), "forwarded {name}");
}
diff --git a/ai-gateway/src/telemetry.rs b/ai-gateway/src/telemetry.rs
new file mode 100644
index 0000000000..bedc5daaac
--- /dev/null
+++ b/ai-gateway/src/telemetry.rs
@@ -0,0 +1,7 @@
+//! Final facts delivery. A Langfuse exporter can replace this debug sink later.
+use crate::capture::InferenceFacts;
+
+pub(crate) fn record(facts: InferenceFacts) {
+ // Full mode intentionally includes native request and completed output content.
+ tracing::debug!(capture = %format_args!("{:#}", serde_json::to_value(facts).expect("captured facts must serialize")), "gateway response captured");
+}
diff --git a/ai-gateway/src/test_support.rs b/ai-gateway/src/test_support.rs
index a9541d26f3..7191fbe02b 100644
--- a/ai-gateway/src/test_support.rs
+++ b/ai-gateway/src/test_support.rs
@@ -89,10 +89,27 @@ pub(crate) fn resolution_response(provider_token: &str) -> Response<Body> {
}
pub(crate) async fn resolved_request_context(provider_token: &str) -> ResolvedRequestContext {
+ resolved_request_context_with_mode(provider_token, "usage").await
+}
+
+pub(crate) async fn resolved_request_context_with_mode(
+ provider_token: &str,
+ mode: &'static str,
+) -> ResolvedRequestContext {
let token = provider_token.to_owned();
let web = FakeServer::start(move |_| {
let token = token.clone();
- async move { resolution_response(&token) }
+ async move {
+ let bytes = axum::body::to_bytes(resolution_response(&token).into_body(), 4096)
+ .await
+ .unwrap();
+ let mut body: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
+ body["ingestion_mode"] = mode.into();
+ Response::builder()
+ .header("content-type", "application/json")
+ .body(Body::from(body.to_string()))
+ .unwrap()
+ }
})
.await;
web.control_plane()
diff --git a/ai-gateway/src/transport/mod.rs b/ai-gateway/src/transport/mod.rs
index 33cc6614ea..821a7e2a2b 100644
--- a/ai-gateway/src/transport/mod.rs
+++ b/ai-gateway/src/transport/mod.rs
@@ -9,6 +9,7 @@ use std::{
task::{Context, Poll},
};
+use crate::capture::{ExecutionCapture, RelayOutcome};
use axum::{
body::{Body, Bytes},
http::{HeaderMap, header},
@@ -63,15 +64,7 @@ fn selected_headers(source: &HeaderMap, allowed: &[&'static str]) -> HeaderMap {
}
pub(crate) fn request_headers(source: &HeaderMap) -> HeaderMap {
- selected_headers(
- source,
- &[
- "content-type",
- "content-encoding",
- "accept",
- "accept-encoding",
- ],
- )
+ selected_headers(source, &["content-type", "content-encoding", "accept"])
}
pub(crate) fn response_headers(source: &HeaderMap) -> HeaderMap {
@@ -99,26 +92,38 @@ pub(crate) fn relay<T: Send + 'static>(
upstream: reqwest::Response,
deadline: Instant,
owner: T,
+ capture: ExecutionCapture,
) -> Body {
relay_stream(
- upstream
- .bytes_stream()
- .map(|chunk| chunk.map_err(|_| ProviderError::Transport)),
+ upstream.bytes_stream().map(|chunk| {
+ chunk.map_err(|error| {
+ if error.is_timeout() {
+ ProviderError::Timeout
+ } else {
+ ProviderError::Transport
+ }
+ })
+ }),
deadline,
owner,
+ Some(capture),
)
}
-fn relay_stream<S, T>(upstream: S, deadline: Instant, owner: T) -> Body
+fn relay_stream<S, T>(
+ upstream: S,
+ deadline: Instant,
+ owner: T,
+ capture: Option<ExecutionCapture>,
+) -> Body
where
S: Stream<Item = Result<Bytes, ProviderError>> + Send + 'static,
T: Send + 'static,
{
let (sender, receiver) = mpsc::channel(1);
- let failed = Arc::new(AtomicBool::new(false));
- let failure = failed.clone();
let resources = Arc::new(StreamResources {
- owner: Mutex::new(Some(owner)),
+ owner: Mutex::new(Some((owner, capture))),
+ failed: AtomicBool::new(false),
released: Notify::new(),
});
let pump_resources = resources.clone();
@@ -127,6 +132,7 @@ where
tokio::pin!(upstream);
while let Some(chunk) = upstream.next().await {
let chunk = chunk?;
+ pump_resources.observe(|capture| capture.bytes(&chunk));
// One queued chunk plus one pending send; no per-chunk tasks.
for bytes in chunk.chunks(64 * 1024) {
if sender.send(Bytes::copy_from_slice(bytes)).await.is_err() {
@@ -134,11 +140,19 @@ where
}
}
}
+ pump_resources.observe(ExecutionCapture::end_body);
Ok::<_, ProviderError>(())
};
- if !matches!(tokio::time::timeout_at(deadline, pump).await, Ok(Ok(()))) {
- failure.store(true, Ordering::Release);
- pump_resources.release();
+ match tokio::time::timeout_at(deadline, pump).await {
+ Ok(Ok(())) => {}
+ result => {
+ let outcome = if matches!(result, Err(_) | Ok(Err(ProviderError::Timeout))) {
+ RelayOutcome::Timeout
+ } else {
+ RelayOutcome::TransportError
+ };
+ pump_resources.release(outcome);
+ }
}
drop(sender);
// Upstream EOF alone does not release admission: the downstream may still
@@ -147,35 +161,64 @@ where
.await
.is_err()
{
- failure.store(true, Ordering::Release);
- pump_resources.release();
+ pump_resources.release(RelayOutcome::Timeout);
}
});
Body::from_stream(ResponseStream {
receiver,
task,
- failed,
resources,
done: false,
})
}
struct StreamResources<T> {
- owner: Mutex<Option<T>>,
+ owner: Mutex<Option<(T, Option<ExecutionCapture>)>>,
+ failed: AtomicBool,
released: Notify,
}
impl<T> StreamResources<T> {
- fn release(&self) {
- self.owner.lock().expect("relay owner lock poisoned").take();
+ fn release(&self, outcome: RelayOutcome) {
+ let owner = {
+ let mut owner = self.owner.lock().expect("relay owner lock poisoned");
+ let taken = owner.take();
+ // Only the winning finalizer publishes the body failure. In
+ // particular, a deadline racing downstream EOF cannot change it later.
+ if taken.is_some()
+ && matches!(
+ outcome,
+ RelayOutcome::Timeout | RelayOutcome::TransportError
+ )
+ {
+ self.failed.store(true, Ordering::Release);
+ }
+ taken
+ };
+ if let Some((owner, capture)) = owner {
+ drop(owner);
+ if let Some(mut capture) = capture {
+ capture.finish(outcome);
+ }
+ }
self.released.notify_one();
}
+
+ fn observe(&self, update: impl FnOnce(&mut ExecutionCapture)) {
+ if let Some((_, Some(capture))) = self
+ .owner
+ .lock()
+ .expect("relay owner lock poisoned")
+ .as_mut()
+ {
+ update(capture);
+ }
+ }
}
struct ResponseStream<T> {
receiver: mpsc::Receiver<Bytes>,
task: JoinHandle<()>,
- failed: Arc<AtomicBool>,
resources: Arc<StreamResources<T>>,
done: bool,
}
@@ -191,10 +234,11 @@ impl<T> Stream for ResponseStream<T> {
match this.receiver.poll_recv(cx) {
Poll::Ready(None) => {
this.done = true;
- this.resources.release();
- // The sender publishes a failure before closing its channel.
+ this.resources.release(RelayOutcome::Eof);
+ // Finalization publishes failure under the same lock as its outcome.
Poll::Ready(
- this.failed
+ this.resources
+ .failed
.load(Ordering::Acquire)
.then_some(Err(ProviderError::Transport)),
)
@@ -207,7 +251,7 @@ impl<T> Stream for ResponseStream<T> {
impl<T> Drop for ResponseStream<T> {
fn drop(&mut self) {
self.task.abort();
- self.resources.release();
+ self.resources.release(RelayOutcome::Cancelled);
}
}
@@ -225,6 +269,26 @@ mod tests {
}
}
+ #[test]
+ fn eof_and_deadline_finalizers_cannot_overwrite_each_other() {
+ for first in [RelayOutcome::Eof, RelayOutcome::Timeout] {
+ let resources = StreamResources {
+ owner: Mutex::new(Some(((), None))),
+ failed: AtomicBool::new(false),
+ released: Notify::new(),
+ };
+ resources.release(first);
+ resources.release(RelayOutcome::Timeout);
+ resources.release(RelayOutcome::Eof);
+ resources.release(RelayOutcome::Cancelled);
+ assert_eq!(
+ resources.failed.load(Ordering::Acquire),
+ first == RelayOutcome::Timeout
+ );
+ assert!(resources.owner.lock().unwrap().is_none());
+ }
+ }
+
#[tokio::test]
async fn slow_downstream_bounds_read_ahead_and_drop_releases_owner() {
let polled = Arc::new(AtomicUsize::new(0));
@@ -238,6 +302,7 @@ mod tests {
upstream,
Instant::now() + Duration::from_secs(10),
Owner(released.clone()),
+ None,
);
tokio::task::yield_now().await;
assert_eq!(polled.load(Ordering::SeqCst), 1);
@@ -255,6 +320,7 @@ mod tests {
stream::iter([Ok(bytes.clone()), Err(ProviderError::Transport)]),
Instant::now() + Duration::from_secs(1),
Owner(released.clone()),
+ None,
);
// Wait until the pump has published its failure with the successful chunk queued.
tokio::time::timeout(Duration::from_secs(1), released.notified())
@@ -273,6 +339,7 @@ mod tests {
stream::once(async { Ok(Bytes::from_static(b"final bytes")) }),
Instant::now() + Duration::from_millis(30),
Owner(released.clone()),
+ None,
);
tokio::time::timeout(Duration::from_secs(1), released.notified())
.await