🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
feat(ai-gateway): capture Responses facts in debug logs (#17442)
Hassieb Pakzadaf8f4f192355
diff --git a/ai-gateway/README.md b/ai-gateway/README.mdindex 229e9866ff..336e72e09e 100644--- a/ai-gateway/README.md+++ b/ai-gateway/README.md@@ -2,7 +2,8 @@A standalone Rust gateway for `POST /openai/v1/responses`. Web resolves the gatewaykey to a trusted provider connection; Rust relays native JSON or SSE without-parsing the request or rewriting provider bytes. Customer telemetry is not exported+rewriting provider bytes. A bounded capture layer captures request/response facts and+logs the finalized record at debug level. Customer telemetry is not exportedyet. Building and testing need no real Web, database or provider credentials.## Run locally@@ -81,8 +82,10 @@ Both formats use the same log level and preserve event fields. For example:2026-09-10T13:20:00Z INFO gateway listening address=0.0.0.0:8080```-Invalid values fail startup without echoing their contents. Logs contain service-lifecycle events, not request bodies or headers. For direct+Invalid values fail startup without echoing their contents. At the default `info`+level, logs contain service lifecycle events. Debug logging also includes the+captured request and completed output when Web resolves `ingestion_mode: full`;+see capture details below. For directhost-only development, set `LANGFUSE_AI_GATEWAY_LISTEN_ADDRESS=127.0.0.1:8080`.## Call the gateway@@ -122,7 +125,8 @@ print(response.output_text)```The gateway forwards to the official OpenAI Responses endpoint only. It does not-retry, follow redirects, parse `model`/`stream`, or accept client routing overrides.+retry, follow redirects, or accept client routing overrides. Request parsing is+best-effort capture only; Web still selects the provider connection.Provider errors retain their status and body. Gateway errors use an OpenAI-style`{"error":{"message":"...","type":"...","param":null,"code":"..."}}` envelope.@@ -146,12 +150,79 @@ There is no separate downstream stall timeout; a client that stops reading canretain execution capacity until the overall deadline. A progress-based downstreamstall policy is deferred to a separate change.-Only request content type/encoding and accept/accept-encoding cross the provider-boundary, plus the resolved Bearer token. Response content type/encoding, cache-control, retry-after, request ID and selected OpenAI timing/version/rate-limit+Only request content type/encoding and accept cross the provider boundary, plus+the resolved Bearer token. The gateway sets `Accept-Encoding: identity` upstream+so client compression preferences cannot disable JSON/SSE capture. Response+content type/encoding, cache control, retry-after, request ID and selected OpenAI+timing/version/rate-limitheaders are retained. Cookies, routing overrides, gateway/ingestion credentials,hop-by-hop headers and upstream framing are excluded.+## Response capture++Set `LANGFUSE_LOG_LEVEL=debug` to print one `gateway response captured` event at+execution end. Its `capture` field is JSON with `input`, `output`, model,+parameters, usage, attribution, timestamps, capture completeness and relay outcome.+Text logs pretty-print the captured JSON across multiple lines.+`LANGFUSE_LOG_FORMAT=json` keeps each surrounding log event on one line, with+newlines escaped inside the `capture` string.++Web's resolved ingestion mode controls content capture:++- `full`: input is the native request JSON object, including `input`, `instructions`,+ tools, parameters, context references and unknown fields. Output is an ordered+ array of native completed items. Debug logs intentionally contain this content.+- `usage`: input and output are null. Model, scalar parameters, native usage,+ timing and trusted attribution are retained; request schemas/content are omitted.++In both modes, `usage_details` preserves the provider's entire usage object,+including nested and unknown fields. The gateway does not rename counters,+subtract cached tokens, or synthesize totals. Missing usage stays null.+`api_format` identifies the payload format (`openai.responses` today).++For SSE, only `response.output_item.done` adds output. Terminal Responses events+provide model, service tier, status and usage; their repeated output is not copied.+Deltas are never stitched or retained. A disconnect midway through an item loses+that unfinished item; already completed items remain in the partial record. Item+completion alone is not response completion. JSON responses capture their native+output array at EOF. Missing usage remains null, rather than invented zeros.++`outcome` describes the relay (`eof`, `cancelled`, `timeout`, `transport_error`);+`provider_status` is separate. For example, a completed provider response can+still end with downstream cancellation. `first_byte_ms` measures the first body+bytes observed by the gateway, not first-token latency. No completion-start time+is fabricated from completed items.++`input_complete` means the full native request was captured; it is false when+content is omitted in usage mode. `output_complete` means response inspection+finished without capture gaps for the configured mode (including a terminal event+and all expected completed items for full-mode SSE). It does not promise that the+provider succeeded or that delivery to the client finished. `capture_complete`+combines request inspection and output completeness; intentionally omitted usage-mode+content does not make it false. Request inspection failure does not invalidate a+fully captured response, and downstream cancellation does not erase completed capture.++The implementation separates shared `ExecutionCapture` lifecycle/timing, the+`OpenAiResponsesCapture` adapter and bounded `SseDecoder`. A private `ProtocolCapture`+enum dispatches to the adapter. Finalization hands an owned `InferenceFacts` record+to `telemetry::record`, which currently only pretty-prints at debug level. Another+provider can supply native facts through the same interface without changing the+relay or final sink. No Anthropic adapter or Langfuse observation/span mapping is+implemented yet. Capture runs independently of log level; only emission is gated.++Capture is limited to 1 MiB request inspection, 1 MiB JSON/SSE event inspection,+1 MiB retained output and 256 output items per execution. The active-request limit+bounds the number of captures. Trusted key metadata is bounded by the resolver's+256 KiB response limit. Oversized input is omitted; oversized output items+are skipped and completeness is false. Malformed, truncated or compressed bodies+do not interrupt the relay. Capture buffers are independent of forwarding, so+these limits never cap the actual provider response. Media is not fetched/uploaded.++Provider credentials, the gateway credential and ingestion tokens are not part of+the capture record; raw headers are not logged. Full captured content and resolved+key metadata may themselves be sensitive. Keep debug logging enabled only where+that content is intended to be recorded. No data is posted to Langfuse yet.+## Process lifecycle- `/health` returns 200 while HTTP is serving. It never probes dependencies.diff --git a/ai-gateway/src/capture/facts.rs b/ai-gateway/src/capture/facts.rsnew file mode 100644index 0000000000..ee12dfed8d--- /dev/null+++ b/ai-gateway/src/capture/facts.rs@@ -0,0 +1,42 @@+//! Owned facts at the capture/telemetry boundary. Content and usage remain native JSON.+use serde::Serialize;+use serde_json::{Map, Value};++#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)]+#[serde(rename_all = "snake_case")]+pub(crate) enum RelayOutcome {+ Eof,+ Cancelled,+ Timeout,+ TransportError,+}++#[derive(Serialize)]+pub(crate) struct InferenceFacts {+ pub api_format: &'static str,+ pub start_time_unix_ms: u128,+ pub duration_ms: u128,+ pub first_byte_ms: Option<u128>,+ pub http_status: Option<u16>,+ pub metadata: Value,+ pub outcome: RelayOutcome,+ #[serde(flatten)]+ pub inference: ProviderFacts,+}++/// Provider adapters fill this common envelope without translating native payloads.+#[derive(Default, Serialize)]+pub(crate) struct ProviderFacts {+ pub model: Option<String>,+ pub requested_model: Option<String>,+ pub model_parameters: Map<String, Value>,+ pub usage_details: Option<Value>,+ pub input: Option<Value>,+ pub output: Option<Value>,+ pub provider_response_id: Option<String>,+ pub provider_request_id: Option<String>,+ pub provider_status: Option<String>,+ pub input_complete: bool,+ pub output_complete: bool,+ pub capture_complete: bool,+}diff --git a/ai-gateway/src/capture/mod.rs b/ai-gateway/src/capture/mod.rsnew file mode 100644index 0000000000..2ceda4752e--- /dev/null+++ b/ai-gateway/src/capture/mod.rs@@ -0,0 +1,168 @@+//! Best-effort capture around the existing relay lifecycle.+mod facts;+mod openai_responses;+mod sse;++use std::time::{SystemTime, UNIX_EPOCH};++use axum::http::{HeaderMap, header};+use serde_json::{Map, Value, json};+use tokio::time::Instant;++use crate::{+ resolution::{IngestionMode, MetadataValue, ResolvedRequestContext},+ telemetry,+};+use facts::ProviderFacts;+pub(crate) use facts::{InferenceFacts, RelayOutcome};+use openai_responses::OpenAiResponsesCapture;++const MAX_CAPTURE_BYTES: usize = 1024 * 1024;+const MAX_ITEMS: usize = 256;+const MAX_FACT_STRING: usize = 512;++/// Add another adapter here when another API is supported; the relay stays shared.+enum ProtocolCapture {+ OpenAiResponses(OpenAiResponsesCapture),+}++impl ProtocolCapture {+ fn response(&mut self, headers: &HeaderMap) {+ match self {+ Self::OpenAiResponses(capture) => capture.response(headers),+ }+ }++ fn bytes(&mut self, bytes: &[u8]) {+ match self {+ Self::OpenAiResponses(capture) => capture.bytes(bytes),+ }+ }++ fn end_body(&mut self) {+ match self {+ Self::OpenAiResponses(capture) => capture.end_body(),+ }+ }++ fn into_facts(self) -> (&'static str, ProviderFacts) {+ match self {+ Self::OpenAiResponses(capture) => ("openai.responses", capture.into_facts()),+ }+ }+}++/// Owned before dispatch and moved into the response body. Drop also covers a+/// cancelled provider future before response headers have arrived.+pub(crate) struct ExecutionCapture {+ protocol: Option<ProtocolCapture>,+ started: Instant,+ start_time_unix_ms: u128,+ first_byte_ms: Option<u128>,+ http_status: Option<u16>,+ metadata: Value,+}++impl ExecutionCapture {+ pub fn openai_responses(+ context: &ResolvedRequestContext,+ headers: &HeaderMap,+ body: &[u8],+ ) -> Self {+ let started = Instant::now();+ let start_time_unix_ms = SystemTime::now()+ .duration_since(UNIX_EPOCH)+ .unwrap_or_default()+ .as_millis();+ let attribution = context.attribution();+ // Copy only attribution; never retain provider or ingestion credentials.+ // The entire resolver response, including key metadata, is bounded at 256 KiB.+ let key_metadata: Map<String, Value> = attribution+ .key_metadata()+ .iter()+ .map(|(key, value)| {+ let value = match value {+ MetadataValue::String(value) => Value::String(value.clone()),+ MetadataValue::Number(value) => Value::Number(value.clone()),+ MetadataValue::Bool(value) => Value::Bool(*value),+ };+ (key.clone(), value)+ })+ .collect();+ let full = context.ingestion_mode() == IngestionMode::Full;+ Self {+ protocol: Some(ProtocolCapture::OpenAiResponses(+ OpenAiResponsesCapture::new(headers, body, context.ingestion_mode()),+ )),+ started,+ start_time_unix_ms,+ first_byte_ms: None,+ http_status: None,+ metadata: json!({+ "organization_id": attribution.organization_id(),+ "project_id": attribution.project_id(),+ "key_id": attribution.key_id(),+ "provider_connection_id": attribution.provider_connection_id(),+ "key_metadata": key_metadata,+ "ingestion_mode": if full { "full" } else { "usage" },+ }),+ }+ }++ pub fn response(&mut self, status: u16, headers: &HeaderMap) {+ if let Some(protocol) = &mut self.protocol {+ self.http_status = Some(status);+ protocol.response(headers);+ }+ }++ pub fn bytes(&mut self, bytes: &[u8]) {+ if let Some(protocol) = &mut self.protocol {+ if !bytes.is_empty() {+ self.first_byte_ms+ .get_or_insert_with(|| self.started.elapsed().as_millis());+ }+ protocol.bytes(bytes);+ }+ }++ pub fn end_body(&mut self) {+ if let Some(protocol) = &mut self.protocol {+ protocol.end_body();+ }+ }++ pub fn finish(&mut self, outcome: RelayOutcome) {+ let Some(protocol) = self.protocol.take() else {+ return;+ };+ let (api_format, inference) = protocol.into_facts();+ telemetry::record(InferenceFacts {+ api_format,+ start_time_unix_ms: self.start_time_unix_ms,+ duration_ms: self.started.elapsed().as_millis(),+ first_byte_ms: self.first_byte_ms,+ http_status: self.http_status,+ metadata: std::mem::take(&mut self.metadata),+ outcome,+ inference,+ });+ }+}++impl Drop for ExecutionCapture {+ fn drop(&mut self) {+ self.finish(RelayOutcome::Cancelled);+ }+}++fn bounded_string(value: &str) -> Option<String> {+ (value.len() <= MAX_FACT_STRING).then(|| value.to_owned())+}++fn identity_encoding(headers: &HeaderMap) -> bool {+ headers+ .get_all(header::CONTENT_ENCODING)+ .iter()+ .all(|v| v.to_str().is_ok_and(|v| v.eq_ignore_ascii_case("identity")))+}diff --git a/ai-gateway/src/capture/openai_responses.rs b/ai-gateway/src/capture/openai_responses.rsnew file mode 100644index 0000000000..b64419a086--- /dev/null+++ b/ai-gateway/src/capture/openai_responses.rs@@ -0,0 +1,262 @@+//! `OpenAI` Responses request, JSON response and completed SSE item capture.+use super::{+ MAX_CAPTURE_BYTES, MAX_FACT_STRING, MAX_ITEMS, bounded_string, facts::ProviderFacts,+ identity_encoding, sse::SseDecoder,+};+use crate::resolution::IngestionMode;+use axum::http::{HeaderMap, header};+use serde_json::{Map, Value};+use std::collections::BTreeMap;++enum ResponseBody {+ Unknown,+ Json(Vec<u8>),+ Sse(SseDecoder),+ Unavailable,+}++pub(super) struct OpenAiResponsesCapture {+ facts: ProviderFacts,+ mode: IngestionMode,+ body: ResponseBody,+ items: BTreeMap<u64, Value>,+ output_bytes: usize,+ terminal: bool,+ expected_items: Option<usize>,+ request_complete: bool,+ response_valid: bool,+}++impl OpenAiResponsesCapture {+ pub fn new(headers: &HeaderMap, body: &[u8], mode: IngestionMode) -> Self {+ let mut capture = Self {+ facts: ProviderFacts::default(),+ mode,+ body: ResponseBody::Unknown,+ items: BTreeMap::new(),+ output_bytes: 0,+ terminal: false,+ expected_items: None,+ request_complete: false,+ response_valid: true,+ };+ if identity_encoding(headers)+ && body.len() <= MAX_CAPTURE_BYTES+ && let Ok(Value::Object(request)) = serde_json::from_slice(body)+ {+ capture.request(request);+ capture.request_complete = true;+ }+ capture+ }++ pub fn response(&mut self, headers: &HeaderMap) {+ self.facts.provider_request_id = headers+ .get("x-request-id")+ .and_then(|v| v.to_str().ok())+ .and_then(bounded_string);+ let content_type = headers+ .get(header::CONTENT_TYPE)+ .and_then(|v| v.to_str().ok())+ .unwrap_or("")+ .split(';')+ .next()+ .unwrap_or("")+ .trim();+ self.body = if !identity_encoding(headers) {+ ResponseBody::Unavailable+ } else if content_type.eq_ignore_ascii_case("text/event-stream") {+ ResponseBody::Sse(SseDecoder::default())+ } else if content_type.eq_ignore_ascii_case("application/json") {+ ResponseBody::Json(Vec::new())+ } else {+ ResponseBody::Unavailable+ };+ if matches!(self.body, ResponseBody::Unavailable) {+ self.response_valid = false;+ }+ }++ pub fn bytes(&mut self, bytes: &[u8]) {+ let mut body = std::mem::replace(&mut self.body, ResponseBody::Unavailable);+ match &mut body {+ ResponseBody::Sse(sse) => {+ sse.push(bytes, MAX_CAPTURE_BYTES, |event| self.event(event));+ }+ ResponseBody::Json(buffer)+ if buffer.len().saturating_add(bytes.len()) <= MAX_CAPTURE_BYTES =>+ {+ buffer.extend_from_slice(bytes);+ }+ ResponseBody::Json(_) => {+ body = ResponseBody::Unavailable;+ self.response_valid = false;+ }+ _ => {}+ }+ self.body = body;+ }++ pub fn end_body(&mut self) {+ match std::mem::replace(&mut self.body, ResponseBody::Unavailable) {+ ResponseBody::Json(bytes) => match serde_json::from_slice::<Value>(&bytes) {+ Ok(Value::Object(response)) => {+ self.response_facts(&response);+ if let Some(output) = response.get("output").and_then(Value::as_array) {+ for (index, item) in output.iter().enumerate() {+ self.item(index as u64, item.clone());+ }+ self.facts.output_complete = self.response_valid;+ }+ }+ _ => self.response_valid = false,+ },+ ResponseBody::Sse(sse) => {+ self.response_valid &= sse.finish();+ self.facts.output_complete = self.terminal+ && self.response_valid+ && (self.mode != IngestionMode::Full+ || self.expected_items.is_some_and(|count| {+ count == self.items.len()+ && self.items.keys().copied().eq(0..count as u64)+ }));+ }+ _ => {}+ }+ self.facts.capture_complete = self.request_complete && self.facts.output_complete;+ }++ pub fn into_facts(mut self) -> ProviderFacts {+ if self.mode == IngestionMode::Full {+ self.facts.output = Some(Value::Array(self.items.into_values().collect()));+ }+ self.facts+ }++ fn request(&mut self, request: Map<String, Value>) {+ self.facts.requested_model = request+ .get("model")+ .and_then(Value::as_str)+ .and_then(bounded_string);+ self.facts.model.clone_from(&self.facts.requested_model);+ for key in ["temperature", "top_p", "max_output_tokens", "service_tier"] {+ if let Some(value) = request+ .get(key)+ .filter(|v| v.is_number() || v.as_str().is_some_and(|s| s.len() <= MAX_FACT_STRING))+ {+ self.facts+ .model_parameters+ .insert(key.to_owned(), value.clone());+ }+ }+ if self.mode == IngestionMode::Full {+ for key in ["reasoning", "text"] {+ if let Some(value) = request.get(key) {+ self.facts+ .model_parameters+ .insert(key.to_owned(), value.clone());+ }+ }+ // Keep the original request object, including native instructions/input+ // and unknown fields. Authentication headers are never part of this value.+ self.facts.input = Some(Value::Object(request));+ self.facts.input_complete = true;+ }+ }++ fn event(&mut self, bytes: &[u8]) {+ let Ok(Value::Object(mut event)) = serde_json::from_slice(bytes) else {+ self.response_valid = false;+ return;+ };+ match event.get("type").and_then(Value::as_str) {+ Some("response.output_item.done") if self.mode == IngestionMode::Full => {+ if let (Some(index), Some(item)) = (+ event.get("output_index").and_then(Value::as_u64),+ event.remove("item").filter(Value::is_object),+ ) {+ self.item(index, item);+ } else {+ self.response_valid = false;+ }+ }+ Some(+ "response.created"+ | "response.in_progress"+ | "response.completed"+ | "response.failed"+ | "response.incomplete",+ ) => {+ let terminal = matches!(+ event.get("type").and_then(Value::as_str),+ Some("response.completed" | "response.failed" | "response.incomplete")+ );+ if let Some(response) = event.get("response").and_then(Value::as_object) {+ self.response_facts(response);+ if terminal {+ self.terminal = true;+ self.expected_items = response+ .get("output")+ .and_then(Value::as_array)+ .map(Vec::len);+ }+ } else {+ self.response_valid = false;+ }+ }+ Some("error") => self.facts.provider_status = Some("failed".to_owned()),+ _ => {}+ }+ }++ fn item(&mut self, index: u64, item: Value) {+ if self.mode != IngestionMode::Full {+ return;+ }+ let size = item.to_string().len();+ let previous = self+ .items+ .get(&index)+ .map_or(0, |item| item.to_string().len());+ if self.output_bytes - previous + size > MAX_CAPTURE_BYTES+ || (!self.items.contains_key(&index) && self.items.len() >= MAX_ITEMS)+ {+ self.response_valid = false;+ return;+ }+ self.output_bytes = self.output_bytes - previous + size;+ self.items.insert(index, item);+ }++ fn response_facts(&mut self, response: &Map<String, Value>) {+ for (key, target) in [+ ("id", &mut self.facts.provider_response_id),+ ("model", &mut self.facts.model),+ ("status", &mut self.facts.provider_status),+ ] {+ if let Some(value) = response+ .get(key)+ .and_then(Value::as_str)+ .and_then(bounded_string)+ {+ *target = Some(value);+ }+ }+ if let Some(tier) = response+ .get("service_tier")+ .and_then(Value::as_str)+ .and_then(bounded_string)+ {+ self.facts+ .model_parameters+ .insert("service_tier".to_owned(), Value::String(tier));+ }+ if let Some(usage) = response.get("usage").filter(|v| v.is_object()) {+ // The enclosing JSON body or SSE event already enforces the byte limit.+ self.facts.usage_details = Some(usage.clone());+ }+ }+}++#[cfg(test)]+mod tests;diff --git a/ai-gateway/src/capture/openai_responses/tests.rs b/ai-gateway/src/capture/openai_responses/tests.rsnew file mode 100644index 0000000000..89c78611d4--- /dev/null+++ b/ai-gateway/src/capture/openai_responses/tests.rs@@ -0,0 +1,667 @@+use super::*;+use crate::capture::{ExecutionCapture, ProtocolCapture, RelayOutcome};+use crate::{+ providers::openai::{OpenAiProvider, ProviderLimits},+ test_support::{FakeServer, resolved_request_context_with_mode},+};+use axum::{+ body::{Body, Bytes, to_bytes},+ http::{HeaderValue, Response},+};+use serde_json::json;+use std::{+ convert::Infallible,+ future::pending,+ sync::{Arc, Mutex},+ time::Duration,+};+use tokio::sync::Notify;++fn captured(execution: &ExecutionCapture) -> &OpenAiResponsesCapture {+ let ProtocolCapture::OpenAiResponses(capture) = execution.protocol.as_ref().unwrap();+ capture+}++async fn observer(mode: &'static str) -> ExecutionCapture {+ let context = resolved_request_context_with_mode("provider-secret", mode).await;+ ExecutionCapture::openai_responses(&context, &HeaderMap::new(), br#"{"model":"requested","instructions":"prompt-canary","input":[{"role":"user","content":"hello"}],"tools":[{"type":"function","name":"weather","parameters":{"type":"object"}}],"text":{"format":{"type":"json_schema","schema":{"description":"schema-canary"}}},"service_tier":"auto"}"#)+}++fn response(observer: &mut ExecutionCapture, content_type: &'static str) {+ let mut headers = HeaderMap::new();+ headers.insert(header::CONTENT_TYPE, HeaderValue::from_static(content_type));+ headers.insert(+ "x-request-id",+ HeaderValue::from_static("provider-request-1"),+ );+ observer.response(200, &headers);+}++fn event(kind: &str, fields: Value) -> String {+ let mut value = fields;+ value["type"] = kind.into();+ format!("event: {kind}\ndata: {value}\n\n")+}++fn completed_item(index: u64, text: &str) -> String {+ event(+ "response.output_item.done",+ json!({"output_index":index,"item":{"type":"message","role":"assistant","content":[{"type":"output_text","text":text}],"future":{"preserved":true}}}),+ )+}++fn terminal(status: &str, count: usize) -> String {+ event(+ &format!("response.{status}"),+ json!({"response":{"id":"resp-1","model":"actual","status":status,"service_tier":"priority","usage":{"input_tokens":100,"output_tokens":30,"total_tokens":130,"input_tokens_details":{"cached_tokens":40},"output_tokens_details":{"reasoning_tokens":10}},"output":vec![json!({"text":"terminal-output-must-not-be-copied"});count]}}),+ )+}++#[tokio::test]+async fn captures_completed_items_only_in_order_without_duplication() {+ let mut observer = observer("full").await;+ response(&mut observer, "text/event-stream; charset=utf-8");+ let body = format!(+ ": ping\n\n{}{}{}{}{}",+ event(+ "response.output_text.delta",+ json!({"delta":"unfinished-canary"})+ ),+ completed_item(1, "second"),+ completed_item(0, "héllo 🌍"),+ completed_item(0, "héllo 🌍"),+ terminal("completed", 2)+ );+ for byte in body.as_bytes() {+ observer.bytes(&[*byte]);+ }+ observer.end_body();+ let capture = captured(&observer);+ assert_eq!(capture.items.len(), 2);+ assert_eq!(capture.items[&0]["content"][0]["text"], "héllo 🌍");+ assert_eq!(capture.items[&1]["content"][0]["text"], "second");+ assert_eq!(capture.items[&0]["future"]["preserved"], true);+ assert_eq!(capture.facts.model.as_deref(), Some("actual"));+ assert_eq!(capture.facts.model_parameters["service_tier"], "priority");+ assert_eq!(+ capture.facts.usage_details.as_ref().unwrap()["input_tokens_details"]["cached_tokens"],+ 40+ );+ assert_eq!(+ capture.facts.input.as_ref().unwrap()["instructions"],+ "prompt-canary"+ );+ assert!(capture.facts.output_complete);+ let stored = json!({"record":capture.facts,"items":capture.items}).to_string();+ assert!(!stored.contains("unfinished-canary"));+ assert!(!stored.contains("terminal-output-must-not-be-copied"));+}++#[tokio::test]+async fn eof_and_item_completion_are_not_provider_completion() {+ let mut observer = observer("full").await;+ response(&mut observer, "text/event-stream");+ observer.bytes(completed_item(0, "completed item").as_bytes());+ observer.bytes(event("response.output_text.delta", json!({"delta":"unfinished"})).as_bytes());+ observer.end_body();+ let capture = captured(&observer);+ assert_eq!(capture.items.len(), 1);+ assert!(!capture.facts.output_complete);+ assert!(capture.facts.usage_details.is_none());+ assert!(capture.facts.provider_status.is_none());+}++#[tokio::test]+async fn failed_and_incomplete_responses_keep_usage_and_native_status() {+ for status in ["failed", "incomplete"] {+ let mut observer = observer("full").await;+ response(&mut observer, "text/event-stream");+ observer.bytes(terminal(status, 0).as_bytes());+ observer.end_body();+ let capture = captured(&observer);+ assert_eq!(capture.facts.provider_status.as_deref(), Some(status));+ assert_eq!(+ capture.facts.usage_details.as_ref().unwrap()["total_tokens"],+ 130+ );+ assert!(capture.facts.output_complete);+ }+}++#[tokio::test]+async fn json_and_streaming_capture_equivalent_native_items_and_facts() {+ let item = json!({"type":"function_call","id":"item-1","call_id":"call-1","name":"weather","arguments":"{\"city\":\"Berlin\"}"});+ let mut json_observer = observer("full").await;+ response(&mut json_observer, "application/json");+ let body = json!({"id":"resp-1","model":"actual","status":"completed","output":[item.clone()],"usage":{"input_tokens":5,"output_tokens":2,"total_tokens":7}}).to_string();+ for chunk in body.as_bytes().chunks(3) {+ json_observer.bytes(chunk);+ }+ json_observer.end_body();+ let captured = captured(&json_observer);+ assert_eq!(captured.items[&0], item);+ assert!(captured.facts.output_complete);+ assert_eq!(+ captured.facts.usage_details.as_ref().unwrap()["total_tokens"],+ 7+ );+}++#[tokio::test]+async fn usage_mode_excludes_content_and_context_credentials() {+ let mut observer = observer("usage").await;+ response(&mut observer, "text/event-stream");+ observer.bytes(completed_item(0, "output-canary").as_bytes());+ observer.bytes(terminal("completed", 1).as_bytes());+ observer.end_body();+ let capture = captured(&observer);+ assert!(capture.facts.input.is_none());+ assert!(capture.items.is_empty());+ assert!(capture.facts.usage_details.is_some());+ let text = json!(capture.facts).to_string();+ for secret in [+ "prompt-canary",+ "schema-canary",+ "output-canary",+ "provider-secret",+ "private-ingestion-token",+ "gateway-secret",+ ] {+ assert!(!text.contains(secret), "leaked {secret}");+ }+}++#[tokio::test]+async fn oversized_and_malformed_events_do_not_prevent_later_facts() {+ let mut observer = observer("full").await;+ response(&mut observer, "text/event-stream");+ observer.bytes(b"data: malformed\n\n");+ observer.bytes(format!("data: {}\n\n", "x".repeat(MAX_CAPTURE_BYTES + 1)).as_bytes());+ observer.bytes(terminal("completed", 0).as_bytes());+ observer.end_body();+ let capture = captured(&observer);+ assert!(!capture.facts.capture_complete);+ assert!(!capture.facts.output_complete);+ assert_eq!(capture.facts.provider_status.as_deref(), Some("completed"));+ assert_eq!(+ capture.facts.usage_details.as_ref().unwrap()["total_tokens"],+ 130+ );+}++#[tokio::test]+async fn capture_limits_and_encoded_bodies_are_explicit() {+ let mut observer = observer("full").await;+ response(&mut observer, "text/event-stream");+ for index in 0..=MAX_ITEMS {+ observer.bytes(completed_item(index as u64, "x").as_bytes());+ }+ assert_eq!(captured(&observer).items.len(), MAX_ITEMS);+ assert!(!captured(&observer).facts.capture_complete);+ let mut encoded = observer;+ let mut headers = HeaderMap::new();+ headers.insert(+ header::CONTENT_TYPE,+ HeaderValue::from_static("application/json"),+ );+ headers.insert(header::CONTENT_ENCODING, HeaderValue::from_static("gzip"));+ encoded.response(200, &headers);+ encoded.bytes(b"opaque compressed bytes");+ encoded.end_body();+ assert!(matches!(captured(&encoded).body, ResponseBody::Unavailable));+ assert!(!captured(&encoded).facts.output_complete);+}++#[test]+fn sse_handles_crlf_multiline_data_and_truncated_eof() {+ let mut decoder = SseDecoder::default();+ let mut events = Vec::new();+ for byte in b": ping\r\nevent: anything\r\ndata: {\r\ndata: \"ok\": true}\r\n\r\n" {+ decoder.push(&[*byte], 1024, |event| events.push(event.to_vec()));+ }+ assert_eq!(events, vec![b"{\n\"ok\": true}".to_vec()]);+ assert!(decoder.finish());+ decoder.push(b"data: {\"unfinished\":true}", 1024, |_| {+ panic!("must wait for delimiter")+ });+ assert!(!decoder.finish());+}++#[derive(Clone, Default)]+struct LogWriter(Arc<Mutex<Vec<u8>>>);+impl std::io::Write for LogWriter {+ fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {+ self.0.lock().unwrap().extend_from_slice(bytes);+ Ok(bytes.len())+ }+ fn flush(&mut self) -> std::io::Result<()> {+ Ok(())+ }+}+impl<'a> tracing_subscriber::fmt::MakeWriter<'a> for LogWriter {+ type Writer = Self;+ fn make_writer(&'a self) -> Self::Writer {+ self.clone()+ }+}++#[tokio::test]+async fn debug_record_is_emitted_once_and_only_at_debug_level() {+ for level in [tracing::Level::DEBUG, tracing::Level::INFO] {+ let mut observer = observer("full").await;+ response(&mut observer, "text/event-stream");+ observer.bytes(completed_item(0, "captured-output").as_bytes());+ observer.bytes(terminal("completed", 1).as_bytes());+ observer.end_body();+ let writer = LogWriter::default();+ let subscriber = tracing_subscriber::fmt()+ .json()+ .with_max_level(level)+ .with_writer(writer.clone())+ .finish();+ tracing::subscriber::with_default(subscriber, || {+ observer.finish(RelayOutcome::Eof);+ observer.finish(RelayOutcome::Cancelled);+ drop(observer);+ });+ let text = String::from_utf8(writer.0.lock().unwrap().clone()).unwrap();+ if level == tracing::Level::DEBUG {+ assert_eq!(text.lines().count(), 1);+ let event: Value = serde_json::from_str(&text).unwrap();+ let capture: Value =+ serde_json::from_str(event["fields"]["capture"].as_str().unwrap()).unwrap();+ assert_eq!(+ capture["output"][0]["content"][0]["text"],+ "captured-output"+ );+ assert_eq!(capture["outcome"], "eof");+ assert_eq!(capture["input"]["instructions"], "prompt-canary");+ for secret in [+ "provider-secret",+ "private-ingestion-token",+ "gateway-secret",+ ] {+ assert!(!text.contains(secret));+ }+ } else {+ assert!(text.is_empty());+ }+ }+}++fn records(writer: &LogWriter) -> Vec<Value> {+ String::from_utf8(writer.0.lock().unwrap().clone())+ .unwrap()+ .lines()+ .filter_map(|line| {+ let event: Value = serde_json::from_str(line).unwrap();+ event["fields"]["capture"]+ .as_str()+ .map(|capture| serde_json::from_str(capture).unwrap())+ })+ .collect()+}++#[tokio::test]+async fn upstream_read_timeout_is_logged_as_timeout() {+ let upstream = FakeServer::start(|_| async {+ Response::builder()+ .header("content-type", "text/event-stream")+ .body(Body::from_stream(futures_util::stream::pending::<+ Result<Bytes, Infallible>,+ >()))+ .unwrap()+ })+ .await;+ let context = resolved_request_context_with_mode("provider-secret", "full").await;+ let provider = OpenAiProvider::for_test(+ format!("{}/v1/responses", upstream.url),+ ProviderLimits {+ active: 1,+ read_timeout: Duration::from_millis(50),+ ..ProviderLimits::default()+ },+ );+ let writer = LogWriter::default();+ let subscriber = tracing_subscriber::fmt()+ .json()+ .with_max_level(tracing::Level::DEBUG)+ .with_writer(writer.clone())+ .finish();+ let _guard = tracing::subscriber::set_default(subscriber);+ let response = provider+ .forward(+ provider.try_admit().unwrap(),+ context,+ &HeaderMap::new(),+ Bytes::from_static(b"{}"),+ )+ .await+ .unwrap();+ assert!(to_bytes(response.into_body(), 4096).await.is_err());+ let records = records(&writer);+ assert_eq!(records.len(), 1);+ assert_eq!(records[0]["outcome"], "timeout");+ assert!(provider.try_admit().is_ok());+}++#[tokio::test]+async fn native_http_relay_logs_once_on_eof_drop_and_unpolled_deadline() {+ for outcome in [+ RelayOutcome::Eof,+ RelayOutcome::Cancelled,+ RelayOutcome::Timeout,+ ] {+ let native = format!(+ "{}{}",+ completed_item(0, "native-result"),+ terminal("completed", 1)+ );+ let expected = native.clone();+ let upstream = FakeServer::start(move |_| {+ let native = native.clone();+ async move {+ Response::builder()+ .header("content-type", "text/event-stream")+ .body(Body::from(native))+ .unwrap()+ }+ })+ .await;+ let context = resolved_request_context_with_mode("provider-secret", "full").await;+ let provider = OpenAiProvider::for_test(+ format!("{}/v1/responses", upstream.url),+ ProviderLimits {+ active: 1,+ execution_timeout: Duration::from_secs(1),+ ..ProviderLimits::default()+ },+ );+ let writer = LogWriter::default();+ let subscriber = tracing_subscriber::fmt()+ .json()+ .with_max_level(tracing::Level::DEBUG)+ .with_writer(writer.clone())+ .finish();+ let _guard = tracing::subscriber::set_default(subscriber);+ let response = provider+ .forward(+ provider.try_admit().unwrap(),+ context,+ &HeaderMap::new(),+ Bytes::from_static(br#"{"input":"request-content"}"#),+ )+ .await+ .unwrap();+ match outcome {+ RelayOutcome::Eof => assert_eq!(+ to_bytes(response.into_body(), 4096).await.unwrap(),+ expected+ ),+ RelayOutcome::Cancelled => drop(response),+ RelayOutcome::Timeout => {+ tokio::time::sleep(Duration::from_millis(1100)).await;+ assert!(provider.try_admit().is_ok());+ drop(response);+ }+ RelayOutcome::TransportError => unreachable!(),+ }+ tokio::task::yield_now().await;+ let records = records(&writer);+ assert_eq!(records.len(), 1);+ assert_eq!(records[0]["outcome"], json!(outcome));+ if outcome == RelayOutcome::Eof {+ assert_eq!(+ records[0]["output"][0]["content"][0]["text"],+ "native-result"+ );+ assert_eq!(records[0]["output_complete"], true);+ assert!(records[0]["first_byte_ms"].is_number());+ }+ assert!(provider.try_admit().is_ok());+ assert_eq!(upstream.calls(), 1);+ }+}++#[tokio::test]+async fn provider_future_finalizes_on_timeout_and_cancellation_before_headers() {+ for cancel in [false, true] {+ let started = Arc::new(Notify::new());+ let signal = started.clone();+ let upstream = FakeServer::start(move |_| {+ let signal = signal.clone();+ async move {+ signal.notify_one();+ pending::<Response<Body>>().await+ }+ })+ .await;+ let context = resolved_request_context_with_mode("provider-secret", "full").await;+ let provider = OpenAiProvider::for_test(+ format!("{}/v1/responses", upstream.url),+ ProviderLimits {+ active: 1,+ headers_timeout: Duration::from_millis(100),+ ..ProviderLimits::default()+ },+ );+ let writer = LogWriter::default();+ let subscriber = tracing_subscriber::fmt()+ .json()+ .with_max_level(tracing::Level::DEBUG)+ .with_writer(writer.clone())+ .finish();+ let _guard = tracing::subscriber::set_default(subscriber);+ let headers = HeaderMap::new();+ let call = provider.forward(+ provider.try_admit().unwrap(),+ context,+ &headers,+ Bytes::from_static(b"{}"),+ );+ if cancel {+ tokio::select! {+ _ = call => panic!("provider must still be waiting"),+ () = started.notified() => {},+ }+ } else {+ assert!(matches!(+ call.await,+ Err(crate::transport::ProviderError::Timeout)+ ));+ }+ let records = records(&writer);+ assert_eq!(records.len(), 1);+ assert_eq!(+ records[0]["outcome"],+ if cancel { "cancelled" } else { "timeout" }+ );+ assert!(records[0]["http_status"].is_null());+ assert!(provider.try_admit().is_ok());+ }+}++#[tokio::test]+async fn client_compression_preferences_do_not_disable_capture() {+ for streaming in [false, true] {+ let native = if streaming {+ format!(+ "{}{}",+ completed_item(0, "captured output"),+ terminal("completed", 1)+ )+ } else {+ json!({"id":"resp-1","status":"completed","model":"actual","output":[{"type":"message","role":"assistant","content":[{"type":"output_text","text":"captured output"}]}],"usage":{"input_tokens":10,"output_tokens":21,"total_tokens":31}}).to_string()+ };+ let expected = native.clone();+ let upstream = FakeServer::start(move |request| {+ let native = native.clone();+ async move {+ assert_eq!(request.headers()[header::ACCEPT_ENCODING], "identity");+ Response::builder()+ .header(+ "content-type",+ if streaming {+ "text/event-stream"+ } else {+ "application/json"+ },+ )+ .body(Body::from(native))+ .unwrap()+ }+ })+ .await;+ let context = resolved_request_context_with_mode("provider-secret", "full").await;+ let provider = OpenAiProvider::for_test(+ format!("{}/v1/responses", upstream.url),+ ProviderLimits::default(),+ );+ let writer = LogWriter::default();+ let subscriber = tracing_subscriber::fmt()+ .json()+ .with_max_level(tracing::Level::DEBUG)+ .with_writer(writer.clone())+ .finish();+ let _guard = tracing::subscriber::set_default(subscriber);+ let mut headers = HeaderMap::new();+ headers.insert(+ header::ACCEPT_ENCODING,+ HeaderValue::from_static("gzip, deflate, br"),+ );+ let response = provider+ .forward(+ provider.try_admit().unwrap(),+ context,+ &headers,+ Bytes::from(json!({"input":"hello","stream":streaming}).to_string()),+ )+ .await+ .unwrap();+ assert_eq!(+ to_bytes(response.into_body(), 4096).await.unwrap(),+ expected+ );+ let records = records(&writer);+ assert_eq!(records.len(), 1);+ assert_eq!(records[0]["capture_complete"], true);+ assert_eq!(records[0]["output_complete"], true);+ assert_eq!(records[0]["provider_response_id"], "resp-1");+ assert_eq!(records[0]["provider_status"], "completed");+ assert!(records[0]["usage_details"]["input_tokens"].is_number());+ assert_eq!(+ records[0]["output"][0]["content"][0]["text"],+ "captured output"+ );+ }+}++#[tokio::test]+async fn capture_regression_large_request_does_not_invalidate_output() {+ let context = resolved_request_context_with_mode("provider-secret", "full").await;+ let request = json!({"input": "x".repeat(MAX_CAPTURE_BYTES)}).to_string();+ for streaming in [false, true] {+ let mut observer =+ ExecutionCapture::openai_responses(&context, &HeaderMap::new(), request.as_bytes());+ response(+ &mut observer,+ if streaming {+ "text/event-stream"+ } else {+ "application/json"+ },+ );+ let body = if streaming {+ terminal("completed", 0)+ } else {+ json!({"status":"completed","output":[]}).to_string()+ };+ observer.bytes(body.as_bytes());+ observer.end_body();+ let facts = &captured(&observer).facts;+ assert!(!facts.input_complete);+ assert!(+ facts.output_complete,+ "response capture must be independent of request size"+ );+ }+}++#[tokio::test]+async fn capture_regression_preserves_native_request_and_usage() {+ let usage = json!({"input_tokens":5,"output_tokens":2,"input_tokens_details":{"cached_tokens":3,"future_tokens":1},"future_usage_field":{"region":"eu"}});+ for mode in ["full", "usage"] {+ let context = resolved_request_context_with_mode("provider-secret", mode).await;+ for input in [+ json!("hello"),+ json!([{"type":"function_call_output","call_id":"call-1","output":"native result"}]),+ ] {+ let request = json!({"model":"requested","input":input,"instructions":"be brief","future_request_field":{"enabled":true}});+ for streaming in [false, true] {+ let mut execution = ExecutionCapture::openai_responses(+ &context,+ &HeaderMap::new(),+ request.to_string().as_bytes(),+ );+ response(+ &mut execution,+ if streaming {+ "text/event-stream"+ } else {+ "application/json"+ },+ );+ let native = json!({"output":[],"usage":usage,"status":"completed"});+ let body = if streaming {+ event("response.completed", json!({"response":native}))+ } else {+ native.to_string()+ };+ execution.bytes(body.as_bytes());+ execution.end_body();+ let facts = &captured(&execution).facts;+ assert_eq!(+ facts.input.as_ref(),+ if mode == "full" { Some(&request) } else { None }+ );+ assert_eq!(facts.usage_details.as_ref(), Some(&usage));+ assert!(facts.capture_complete);+ }+ }+ }+}++#[tokio::test]+async fn downstream_cancellation_does_not_erase_completed_capture() {+ for streaming in [false, true] {+ let mut execution = observer("full").await;+ response(+ &mut execution,+ if streaming {+ "text/event-stream"+ } else {+ "application/json"+ },+ );+ let body = if streaming {+ terminal("completed", 0)+ } else {+ json!({"status":"completed","output":[]}).to_string()+ };+ execution.bytes(body.as_bytes());+ execution.end_body();+ let writer = LogWriter::default();+ let subscriber = tracing_subscriber::fmt()+ .json()+ .with_max_level(tracing::Level::DEBUG)+ .with_writer(writer.clone())+ .finish();+ tracing::subscriber::with_default(subscriber, || execution.finish(RelayOutcome::Cancelled));+ let facts = records(&writer);+ assert_eq!(facts.len(), 1);+ assert_eq!(facts[0]["outcome"], "cancelled");+ assert_eq!(facts[0]["provider_status"], "completed");+ assert_eq!(facts[0]["output_complete"], true);+ assert_eq!(facts[0]["capture_complete"], true);+ }+}diff --git a/ai-gateway/src/capture/sse.rs b/ai-gateway/src/capture/sse.rsnew file mode 100644index 0000000000..e749b25639--- /dev/null+++ b/ai-gateway/src/capture/sse.rs@@ -0,0 +1,76 @@+//! Incremental SSE framing. A bad or oversized event only affects capture.+#[derive(Default)]+pub(super) struct SseDecoder {+ line: Vec<u8>,+ data: Vec<u8>,+ line_nonempty: bool,+ skip_lf: bool,+ state: EventState,+ pub incomplete: bool,+}++#[derive(Default)]+enum EventState {+ #[default]+ Collecting,+ Discarding,+}++impl SseDecoder {+ pub fn push(&mut self, bytes: &[u8], limit: usize, mut event: impl FnMut(&[u8])) {+ for &byte in bytes {+ if self.skip_lf {+ self.skip_lf = false;+ if byte == b'\n' {+ continue;+ }+ }+ if byte == b'\r' || byte == b'\n' {+ self.end_line(&mut event);+ self.skip_lf = byte == b'\r';+ } else {+ self.line_nonempty = true;+ if matches!(self.state, EventState::Collecting) {+ if self.line.len() + self.data.len() >= limit {+ self.incomplete = true;+ self.state = EventState::Discarding;+ self.line.clear();+ self.data.clear();+ } else {+ self.line.push(byte);+ }+ }+ }+ }+ }++ fn end_line(&mut self, event: &mut impl FnMut(&[u8])) {+ if !self.line_nonempty {+ if matches!(self.state, EventState::Collecting) && !self.data.is_empty() {+ event(&self.data[..self.data.len() - 1]);+ }+ self.data.clear();+ self.state = EventState::Collecting;+ } else if matches!(self.state, EventState::Collecting) {+ let value = self+ .line+ .strip_prefix(b"data:")+ .map(|value| value.strip_prefix(b" ").unwrap_or(value));+ if let Some(value) = value {+ self.data.extend_from_slice(value);+ self.data.push(b'\n');+ } else if self.line == b"data" {+ self.data.push(b'\n');+ }+ }+ self.line.clear();+ self.line_nonempty = false;+ }++ pub fn finish(&self) -> bool {+ !self.incomplete+ && !self.line_nonempty+ && self.data.is_empty()+ && matches!(self.state, EventState::Collecting)+ }+}diff --git a/ai-gateway/src/lib.rs b/ai-gateway/src/lib.rsindex 60740a9181..ce4f00cec5 100644--- a/ai-gateway/src/lib.rs+++ b/ai-gateway/src/lib.rs@@ -1,9 +1,11 @@+mod capture;pub mod config;pub mod http;pub mod inference;pub mod providers;pub mod resolution;pub mod server;+mod telemetry;mod transport;#[cfg(test)]diff --git a/ai-gateway/src/providers/openai.rs b/ai-gateway/src/providers/openai.rsindex e8882905b4..71a6568056 100644--- a/ai-gateway/src/providers/openai.rs+++ b/ai-gateway/src/providers/openai.rs@@ -11,7 +11,11 @@ use tokio::{time::Instant,};-use crate::{resolution::ResolvedRequestContext, transport};+use crate::{+ capture::{ExecutionCapture, RelayOutcome},+ resolution::ResolvedRequestContext,+ transport,+};pub use crate::transport::ProviderError;@@ -124,6 +128,7 @@ impl OpenAiProvider {HeaderValue::from_str(&format!("Bearer {}", context.connection().provider_token())).map_err(|_| ProviderError::Configuration)?;authorization.set_sensitive(true);+ let mut capture = ExecutionCapture::openai_responses(&context, headers, &body);let response = tokio::time::timeout_at(permit.deadline@@ -131,23 +136,40 @@ impl OpenAiProvider {self.client.post(endpoint).headers(transport::request_headers(headers))+ // Observe plain JSON/SSE while relaying the provider bytes unchanged.+ .header(header::ACCEPT_ENCODING, "identity").header(header::AUTHORIZATION, authorization).body(body).send(),).await- .map_err(|_| ProviderError::Timeout)?- .map_err(|error| {- if error.is_timeout() {- ProviderError::Timeout- } else {- ProviderError::Transport+ .map_err(|_| ProviderError::Timeout)+ .and_then(|result| {+ result.map_err(|error| {+ if error.is_timeout() {+ ProviderError::Timeout+ } else {+ ProviderError::Transport+ }+ })+ });+ let response = match response {+ Ok(response) => response,+ Err(error) => {+ capture.finish(if matches!(error, ProviderError::Timeout) {+ RelayOutcome::Timeout+ } else {+ RelayOutcome::TransportError+ });+ return Err(error);}- })?;+ };+ capture.response(response.status().as_u16(), response.headers());let mut downstream = Response::new(Body::empty());*downstream.status_mut() = response.status();*downstream.headers_mut() = transport::response_headers(response.headers());- *downstream.body_mut() = transport::relay(response, permit.deadline, (permit, context));+ *downstream.body_mut() =+ transport::relay(response, permit.deadline, (permit, context), capture);Ok(downstream)}diff --git a/ai-gateway/src/providers/openai/tests.rs b/ai-gateway/src/providers/openai/tests.rsindex 4c1bde86b8..8c1d502e72 100644--- a/ai-gateway/src/providers/openai/tests.rs+++ b/ai-gateway/src/providers/openai/tests.rs@@ -27,6 +27,7 @@ async fn preserves_opaque_bytes_and_isolates_request_and_response_headers() {"Bearer provider-secret");assert_eq!(request.headers()[header::CONTENT_TYPE], "application/json");+ assert_eq!(request.headers()[header::ACCEPT_ENCODING], "identity");for name in ["cookie","langfuse-gateway-authorization",@@ -34,7 +35,6 @@ async fn preserves_opaque_bytes_and_isolates_request_and_response_headers() {"openai-project","x-api-key","x-forwarded-host",- "accept-encoding",] {assert!(!request.headers().contains_key(name), "forwarded {name}");}diff --git a/ai-gateway/src/telemetry.rs b/ai-gateway/src/telemetry.rsnew file mode 100644index 0000000000..bedc5daaac--- /dev/null+++ b/ai-gateway/src/telemetry.rs@@ -0,0 +1,7 @@+//! Final facts delivery. A Langfuse exporter can replace this debug sink later.+use crate::capture::InferenceFacts;++pub(crate) fn record(facts: InferenceFacts) {+ // Full mode intentionally includes native request and completed output content.+ tracing::debug!(capture = %format_args!("{:#}", serde_json::to_value(facts).expect("captured facts must serialize")), "gateway response captured");+}diff --git a/ai-gateway/src/test_support.rs b/ai-gateway/src/test_support.rsindex a9541d26f3..7191fbe02b 100644--- a/ai-gateway/src/test_support.rs+++ b/ai-gateway/src/test_support.rs@@ -89,10 +89,27 @@ pub(crate) fn resolution_response(provider_token: &str) -> Response<Body> {}pub(crate) async fn resolved_request_context(provider_token: &str) -> ResolvedRequestContext {+ resolved_request_context_with_mode(provider_token, "usage").await+}++pub(crate) async fn resolved_request_context_with_mode(+ provider_token: &str,+ mode: &'static str,+) -> ResolvedRequestContext {let token = provider_token.to_owned();let web = FakeServer::start(move |_| {let token = token.clone();- async move { resolution_response(&token) }+ async move {+ let bytes = axum::body::to_bytes(resolution_response(&token).into_body(), 4096)+ .await+ .unwrap();+ let mut body: serde_json::Value = serde_json::from_slice(&bytes).unwrap();+ body["ingestion_mode"] = mode.into();+ Response::builder()+ .header("content-type", "application/json")+ .body(Body::from(body.to_string()))+ .unwrap()+ }}).await;web.control_plane()diff --git a/ai-gateway/src/transport/mod.rs b/ai-gateway/src/transport/mod.rsindex 33cc6614ea..821a7e2a2b 100644--- a/ai-gateway/src/transport/mod.rs+++ b/ai-gateway/src/transport/mod.rs@@ -9,6 +9,7 @@ use std::{task::{Context, Poll},};+use crate::capture::{ExecutionCapture, RelayOutcome};use axum::{body::{Body, Bytes},http::{HeaderMap, header},@@ -63,15 +64,7 @@ fn selected_headers(source: &HeaderMap, allowed: &[&'static str]) -> HeaderMap {}pub(crate) fn request_headers(source: &HeaderMap) -> HeaderMap {- selected_headers(- source,- &[- "content-type",- "content-encoding",- "accept",- "accept-encoding",- ],- )+ selected_headers(source, &["content-type", "content-encoding", "accept"])}pub(crate) fn response_headers(source: &HeaderMap) -> HeaderMap {@@ -99,26 +92,38 @@ pub(crate) fn relay<T: Send + 'static>(upstream: reqwest::Response,deadline: Instant,owner: T,+ capture: ExecutionCapture,) -> Body {relay_stream(- upstream- .bytes_stream()- .map(|chunk| chunk.map_err(|_| ProviderError::Transport)),+ upstream.bytes_stream().map(|chunk| {+ chunk.map_err(|error| {+ if error.is_timeout() {+ ProviderError::Timeout+ } else {+ ProviderError::Transport+ }+ })+ }),deadline,owner,+ Some(capture),)}-fn relay_stream<S, T>(upstream: S, deadline: Instant, owner: T) -> Body+fn relay_stream<S, T>(+ upstream: S,+ deadline: Instant,+ owner: T,+ capture: Option<ExecutionCapture>,+) -> BodywhereS: Stream<Item = Result<Bytes, ProviderError>> + Send + 'static,T: Send + 'static,{let (sender, receiver) = mpsc::channel(1);- let failed = Arc::new(AtomicBool::new(false));- let failure = failed.clone();let resources = Arc::new(StreamResources {- owner: Mutex::new(Some(owner)),+ owner: Mutex::new(Some((owner, capture))),+ failed: AtomicBool::new(false),released: Notify::new(),});let pump_resources = resources.clone();@@ -127,6 +132,7 @@ wheretokio::pin!(upstream);while let Some(chunk) = upstream.next().await {let chunk = chunk?;+ pump_resources.observe(|capture| capture.bytes(&chunk));// One queued chunk plus one pending send; no per-chunk tasks.for bytes in chunk.chunks(64 * 1024) {if sender.send(Bytes::copy_from_slice(bytes)).await.is_err() {@@ -134,11 +140,19 @@ where}}}+ pump_resources.observe(ExecutionCapture::end_body);Ok::<_, ProviderError>(())};- if !matches!(tokio::time::timeout_at(deadline, pump).await, Ok(Ok(()))) {- failure.store(true, Ordering::Release);- pump_resources.release();+ match tokio::time::timeout_at(deadline, pump).await {+ Ok(Ok(())) => {}+ result => {+ let outcome = if matches!(result, Err(_) | Ok(Err(ProviderError::Timeout))) {+ RelayOutcome::Timeout+ } else {+ RelayOutcome::TransportError+ };+ pump_resources.release(outcome);+ }}drop(sender);// Upstream EOF alone does not release admission: the downstream may still@@ -147,35 +161,64 @@ where.await.is_err(){- failure.store(true, Ordering::Release);- pump_resources.release();+ pump_resources.release(RelayOutcome::Timeout);}});Body::from_stream(ResponseStream {receiver,task,- failed,resources,done: false,})}struct StreamResources<T> {- owner: Mutex<Option<T>>,+ owner: Mutex<Option<(T, Option<ExecutionCapture>)>>,+ failed: AtomicBool,released: Notify,}impl<T> StreamResources<T> {- fn release(&self) {- self.owner.lock().expect("relay owner lock poisoned").take();+ fn release(&self, outcome: RelayOutcome) {+ let owner = {+ let mut owner = self.owner.lock().expect("relay owner lock poisoned");+ let taken = owner.take();+ // Only the winning finalizer publishes the body failure. In+ // particular, a deadline racing downstream EOF cannot change it later.+ if taken.is_some()+ && matches!(+ outcome,+ RelayOutcome::Timeout | RelayOutcome::TransportError+ )+ {+ self.failed.store(true, Ordering::Release);+ }+ taken+ };+ if let Some((owner, capture)) = owner {+ drop(owner);+ if let Some(mut capture) = capture {+ capture.finish(outcome);+ }+ }self.released.notify_one();}++ fn observe(&self, update: impl FnOnce(&mut ExecutionCapture)) {+ if let Some((_, Some(capture))) = self+ .owner+ .lock()+ .expect("relay owner lock poisoned")+ .as_mut()+ {+ update(capture);+ }+ }}struct ResponseStream<T> {receiver: mpsc::Receiver<Bytes>,task: JoinHandle<()>,- failed: Arc<AtomicBool>,resources: Arc<StreamResources<T>>,done: bool,}@@ -191,10 +234,11 @@ impl<T> Stream for ResponseStream<T> {match this.receiver.poll_recv(cx) {Poll::Ready(None) => {this.done = true;- this.resources.release();- // The sender publishes a failure before closing its channel.+ this.resources.release(RelayOutcome::Eof);+ // Finalization publishes failure under the same lock as its outcome.Poll::Ready(- this.failed+ this.resources+ .failed.load(Ordering::Acquire).then_some(Err(ProviderError::Transport)),)@@ -207,7 +251,7 @@ impl<T> Stream for ResponseStream<T> {impl<T> Drop for ResponseStream<T> {fn drop(&mut self) {self.task.abort();- self.resources.release();+ self.resources.release(RelayOutcome::Cancelled);}}@@ -225,6 +269,26 @@ mod tests {}}+ #[test]+ fn eof_and_deadline_finalizers_cannot_overwrite_each_other() {+ for first in [RelayOutcome::Eof, RelayOutcome::Timeout] {+ let resources = StreamResources {+ owner: Mutex::new(Some(((), None))),+ failed: AtomicBool::new(false),+ released: Notify::new(),+ };+ resources.release(first);+ resources.release(RelayOutcome::Timeout);+ resources.release(RelayOutcome::Eof);+ resources.release(RelayOutcome::Cancelled);+ assert_eq!(+ resources.failed.load(Ordering::Acquire),+ first == RelayOutcome::Timeout+ );+ assert!(resources.owner.lock().unwrap().is_none());+ }+ }+#[tokio::test]async fn slow_downstream_bounds_read_ahead_and_drop_releases_owner() {let polled = Arc::new(AtomicUsize::new(0));@@ -238,6 +302,7 @@ mod tests {upstream,Instant::now() + Duration::from_secs(10),Owner(released.clone()),+ None,);tokio::task::yield_now().await;assert_eq!(polled.load(Ordering::SeqCst), 1);@@ -255,6 +320,7 @@ mod tests {stream::iter([Ok(bytes.clone()), Err(ProviderError::Transport)]),Instant::now() + Duration::from_secs(1),Owner(released.clone()),+ None,);// Wait until the pump has published its failure with the successful chunk queued.tokio::time::timeout(Duration::from_secs(1), released.notified())@@ -273,6 +339,7 @@ mod tests {stream::once(async { Ok(Bytes::from_static(b"final bytes")) }),Instant::now() + Duration::from_millis(30),Owner(released.clone()),+ None,);tokio::time::timeout(Duration::from_secs(1), released.notified()).await