🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
fix(ai-gateway): clean up generation fields and streaming timing (#17545)
Hassieb Pakzad84af505e2b7d
diff --git a/ai-gateway/README.md b/ai-gateway/README.mdindex acac378690..ddc70f6675 100644--- a/ai-gateway/README.md+++ b/ai-gateway/README.md@@ -229,12 +229,27 @@ are dropped without re-resolving the execution. Requests explicitly select v4 inEach execution becomes one root generation with generated trace/observation IDs,actual/requested model, model parameters, native usage, and trusted key/connection-attribution. Full mode includes the captured native input/output; usage mode omits-content. First-byte timing supplies Langfuse's completion-start time as the gateway's-first-token approximation. Relay outcome, provider status and capture completeness-remain separate metadata fields. The exporter projects native OpenAI usage into the-receiver's supported shape for pricing and preserves the untouched usage object as-`native_usage` metadata. Missing usage is not reported as zero.+attribution. Full mode includes the captured input/output; usage mode omits content.+Completion-start time is emitted only for upstream `text/event-stream` responses,+on the first nonempty text, reasoning, refusal, tool-input, audio, or partial-image content. JSON responses+and streams without a captured content delta have no completion-start time or TTFT.+The exporter projects native OpenAI usage into the receiver's supported shape for+pricing without duplicating it in metadata. Missing usage is not reported as zero.++Gateway metadata uses `langfuse.gateway.*`: `project_id`, `organization_id`,+`ingestion_mode`, `api_format`, `api-key.id`, and `provider.connection_id`,+`provider.request_id`, `provider.response_id`. API-key attribution entries appear+both as top-level metadata and under `langfuse.gateway.api-key.metadata.*`.+Gateway and OpenTelemetry fields win collisions; the namespaced attribution copy+preserves the original value. `http_status` stays top level. Relay outcome, provider+status, completeness flags and first-byte timing remain internal facts rather than+generation metadata. Ingestion removes mapped observation-attribute duplicates for+the gateway scope while preserving custom attributes, scope and resources.++Provider HTTP failures and failed SSE responses set the generation level to `ERROR`+with the available HTTP status in its status message. Full mode also includes a+bounded provider error code/message; usage mode omits these details because provider+errors may echo request content.Provisional upload limits are 32 concurrent tasks, 4 MiB serialized facts per record,16 MiB total retained serialized-fact/credential bytes, 8 MiB encoded payloads, and@@ -265,12 +280,23 @@ including at debug level. `LANGFUSE_LOG_FORMAT=json` emits one JSON object per lWeb's resolved ingestion mode controls content capture:-- `full`: input is the native request JSON object, including `input`, `instructions`,- tools, parameters, context references and unknown fields. Output is an ordered- array of native completed items. Content is sent only through Langfuse ingestion.+- `full`: input retains native `input`, `instructions`, tools, prompt/context+ references and unknown fields. Model and configuration are projected out of input.+ Output is an ordered array of native completed items. Content is sent only through+ Langfuse ingestion.- `usage`: input and output are null. Model, scalar parameters, native usage,timing and trusted attribution are retained; request schemas/content are omitted.+Full-mode model parameters include sampling/token limits, service tier, reasoning,+text formatting, tool choice/limits, context management, truncation, streaming options,+background/store/include flags, moderation and cache options/retention. Capture retains+JSON types; ingestion applies Langfuse's existing model-parameter normalization+(booleans and structured values become JSON strings). Omitted request parameters+are not filled with assumed defaults.+The response model and service tier take precedence over requested values when present.+Request `metadata`, `prompt_cache_key`, `safety_identifier` and deprecated `user` are+stored under `langfuse.gateway.provider.request.*` only in full mode.+In both modes, `usage_details` preserves the provider's entire usage object,including nested and unknown fields. The gateway does not rename counters,subtract cached tokens, or synthesize totals. Missing usage stays null.@@ -286,10 +312,11 @@ output array at EOF. Missing usage remains null, rather than invented zeros.`outcome` describes the relay (`eof`, `cancelled`, `timeout`, `transport_error`);`provider_status` is separate. For example, a completed provider response canstill end with downstream cancellation. `first_byte_ms` measures the first body-bytes observed by the gateway. Telemetry uses this as the first-token approximation-for Langfuse completion-start time; it does not wait for completed output items.+bytes observed by the gateway for operational metrics. `completion_start_ms` measures+the first captured SSE content delta independently; it ignores response-created+events and keepalives and does not retain or reconstruct delta content.-`input_complete` means the full native request was captured; it is false when+`input_complete` means request input was captured and configuration projected; it is false whencontent is omitted in usage mode. `output_complete` means response inspectionfinished without capture gaps for the configured mode (including a terminal eventand all expected completed items for full-mode SSE). It does not promise that thediff --git a/ai-gateway/src/capture/facts.rs b/ai-gateway/src/capture/facts.rsindex ee12dfed8d..1d3dccd117 100644--- a/ai-gateway/src/capture/facts.rs+++ b/ai-gateway/src/capture/facts.rs@@ -17,6 +17,7 @@ pub(crate) struct InferenceFacts {pub start_time_unix_ms: u128,pub duration_ms: u128,pub first_byte_ms: Option<u128>,+ pub completion_start_ms: Option<u128>,pub http_status: Option<u16>,pub metadata: Value,pub outcome: RelayOutcome,@@ -30,6 +31,8 @@ pub(crate) struct ProviderFacts {pub model: Option<String>,pub requested_model: Option<String>,pub model_parameters: Map<String, Value>,+ pub request_metadata: Map<String, Value>,+ pub error_message: Option<String>,pub usage_details: Option<Value>,pub input: Option<Value>,pub output: Option<Value>,diff --git a/ai-gateway/src/capture/mod.rs b/ai-gateway/src/capture/mod.rsindex 61c9c62c59..f1b86d30ba 100644--- a/ai-gateway/src/capture/mod.rs+++ b/ai-gateway/src/capture/mod.rs@@ -33,7 +33,7 @@ impl ProtocolCapture {}}- fn bytes(&mut self, bytes: &[u8]) {+ fn bytes(&mut self, bytes: &[u8]) -> bool {match self {Self::OpenAiResponses(capture) => capture.bytes(bytes),}@@ -60,6 +60,7 @@ pub(crate) struct ExecutionCapture {started: Instant,start_time_unix_ms: u128,first_byte_ms: Option<u128>,+ completion_start_ms: Option<u128>,http_status: Option<u16>,metadata: Value,delivery: Option<(telemetry::Telemetry, telemetry::DeliveryContext)>,@@ -100,6 +101,7 @@ impl ExecutionCapture {started,start_time_unix_ms,first_byte_ms: None,+ completion_start_ms: None,http_status: None,metadata: json!({"organization_id": attribution.organization_id(),@@ -137,7 +139,10 @@ impl ExecutionCapture {self.first_byte_ms.get_or_insert_with(|| self.started.elapsed().as_millis());}- protocol.bytes(bytes);+ if protocol.bytes(bytes) {+ self.completion_start_ms+ .get_or_insert_with(|| self.started.elapsed().as_millis());+ }}}@@ -159,6 +164,7 @@ impl ExecutionCapture {start_time_unix_ms: self.start_time_unix_ms,duration_ms: self.started.elapsed().as_millis(),first_byte_ms: self.first_byte_ms,+ completion_start_ms: self.completion_start_ms,http_status: self.http_status,metadata: std::mem::take(&mut self.metadata),outcome,diff --git a/ai-gateway/src/capture/openai_responses.rs b/ai-gateway/src/capture/openai_responses.rsindex b64419a086..5803a1ba6c 100644--- a/ai-gateway/src/capture/openai_responses.rs+++ b/ai-gateway/src/capture/openai_responses.rs@@ -77,11 +77,14 @@ impl OpenAiResponsesCapture {}}- pub fn bytes(&mut self, bytes: &[u8]) {+ pub fn bytes(&mut self, bytes: &[u8]) -> bool {let mut body = std::mem::replace(&mut self.body, ResponseBody::Unavailable);+ let mut completion_started = false;match &mut body {ResponseBody::Sse(sse) => {- sse.push(bytes, MAX_CAPTURE_BYTES, |event| self.event(event));+ sse.push(bytes, MAX_CAPTURE_BYTES, |event| {+ completion_started |= self.event(event);+ });}ResponseBody::Json(buffer)if buffer.len().saturating_add(bytes.len()) <= MAX_CAPTURE_BYTES =>@@ -95,6 +98,7 @@ impl OpenAiResponsesCapture {_ => {}}self.body = body;+ completion_started}pub fn end_body(&mut self) {@@ -133,43 +137,92 @@ impl OpenAiResponsesCapture {self.facts}- fn request(&mut self, request: Map<String, Value>) {+ fn request(&mut self, mut request: Map<String, Value>) {self.facts.requested_model = request.get("model").and_then(Value::as_str).and_then(bounded_string);self.facts.model.clone_from(&self.facts.requested_model);- for key in ["temperature", "top_p", "max_output_tokens", "service_tier"] {- if let Some(value) = request- .get(key)- .filter(|v| v.is_number() || v.as_str().is_some_and(|s| s.len() <= MAX_FACT_STRING))- {- self.facts- .model_parameters- .insert(key.to_owned(), value.clone());+ request.remove("model");+ for key in [+ "temperature",+ "top_p",+ "top_logprobs",+ "max_output_tokens",+ "service_tier",+ "parallel_tool_calls",+ "max_tool_calls",+ "truncation",+ "stream",+ "background",+ "store",+ "prompt_cache_retention",+ ] {+ if let Some(value) = request.remove(key).filter(|v| {+ self.mode == IngestionMode::Full+ || v.is_number()+ || v.is_boolean()+ || v.as_str().is_some_and(|s| s.len() <= MAX_FACT_STRING)+ }) {+ self.facts.model_parameters.insert(key.to_owned(), value);}}if self.mode == IngestionMode::Full {- for key in ["reasoning", "text"] {- if let Some(value) = request.get(key) {- self.facts- .model_parameters- .insert(key.to_owned(), value.clone());+ for key in [+ "reasoning",+ "text",+ "tool_choice",+ "context_management",+ "stream_options",+ "include",+ "moderation",+ "prompt_cache_options",+ ] {+ if let Some(value) = request.remove(key) {+ self.facts.model_parameters.insert(key.to_owned(), value);+ }+ }+ for key in ["metadata", "prompt_cache_key", "safety_identifier", "user"] {+ if let Some(value) = request.remove(key) {+ self.facts.request_metadata.insert(key.to_owned(), value);}}- // Keep the original request object, including native instructions/input- // and unknown fields. Authentication headers are never part of this value.+ // Preserve native prompt context and unknown fields after projecting configuration.self.facts.input = Some(Value::Object(request));self.facts.input_complete = true;}}- fn event(&mut self, bytes: &[u8]) {+ fn event(&mut self, bytes: &[u8]) -> bool {let Ok(Value::Object(mut event)) = serde_json::from_slice(bytes) else {self.response_valid = false;- return;+ return false;};match event.get("type").and_then(Value::as_str) {+ Some(+ "response.output_text.delta"+ | "response.refusal.delta"+ | "response.function_call_arguments.delta"+ | "response.custom_tool_call_input.delta"+ | "response.reasoning_text.delta"+ | "response.reasoning_summary_text.delta"+ | "response.mcp_call_arguments.delta"+ | "response.code_interpreter_call_code.delta"+ | "response.audio.delta"+ | "response.audio.transcript.delta"+ | "response.shell_call_command.delta",+ ) => {+ return event+ .get("delta")+ .and_then(Value::as_str)+ .is_some_and(|delta| !delta.is_empty());+ }+ Some("response.image_generation_call.partial_image") => {+ return event+ .get("partial_image_b64")+ .and_then(Value::as_str)+ .is_some_and(|image| !image.is_empty());+ }Some("response.output_item.done") if self.mode == IngestionMode::Full => {if let (Some(index), Some(item)) = (event.get("output_index").and_then(Value::as_u64),@@ -204,9 +257,13 @@ impl OpenAiResponsesCapture {self.response_valid = false;}}- Some("error") => self.facts.provider_status = Some("failed".to_owned()),+ Some("error") => {+ self.facts.provider_status = Some("failed".to_owned());+ self.error(&event);+ }_ => {}}+ false}fn item(&mut self, index: u64, item: Value) {@@ -229,6 +286,9 @@ impl OpenAiResponsesCapture {}fn response_facts(&mut self, response: &Map<String, Value>) {+ if let Some(error) = response.get("error").and_then(Value::as_object) {+ self.error(error);+ }for (key, target) in [("id", &mut self.facts.provider_response_id),("model", &mut self.facts.model),@@ -256,6 +316,24 @@ impl OpenAiResponsesCapture {self.facts.usage_details = Some(usage.clone());}}++ fn error(&mut self, error: &Map<String, Value>) {+ // Provider error messages may echo prompt content, so retain them only in full mode.+ if self.mode != IngestionMode::Full {+ return;+ }+ let detail = ["code", "message"]+ .into_iter()+ .filter_map(|key| error.get(key).and_then(Value::as_str))+ .filter(|value| !value.is_empty())+ .map(|value| &value[..value.floor_char_boundary(MAX_FACT_STRING)])+ .collect::<Vec<_>>()+ .join(": ");+ if !detail.is_empty() {+ self.facts.error_message =+ Some(detail[..detail.floor_char_boundary(MAX_FACT_STRING)].to_owned());+ }+ }}#[cfg(test)]diff --git a/ai-gateway/src/capture/openai_responses/tests.rs b/ai-gateway/src/capture/openai_responses/tests.rsindex d03fdd9a65..1a137c2f56 100644--- a/ai-gateway/src/capture/openai_responses/tests.rs+++ b/ai-gateway/src/capture/openai_responses/tests.rs@@ -24,7 +24,7 @@ fn captured(execution: &ExecutionCapture) -> &OpenAiResponsesCapture {async fn observer(mode: &'static str) -> ExecutionCapture {let context = resolved_request_context_with_mode("provider-secret", mode).await;- ExecutionCapture::openai_responses(&context, &HeaderMap::new(), br#"{"model":"requested","instructions":"prompt-canary","input":[{"role":"user","content":"hello"}],"tools":[{"type":"function","name":"weather","parameters":{"type":"object"}}],"text":{"format":{"type":"json_schema","schema":{"description":"schema-canary"}}},"service_tier":"auto"}"#)+ ExecutionCapture::openai_responses(&context, &HeaderMap::new(), br#"{"model":"requested","instructions":"prompt-canary","input":[{"role":"user","content":"hello"}],"tools":[{"type":"function","name":"weather","parameters":{"type":"object"}}],"text":{"format":{"type":"json_schema","schema":{"description":"schema-canary"}}},"service_tier":"auto","metadata":{"label":"request-metadata-canary"},"safety_identifier":"safety-id-canary","prompt_cache_key":"cache-id-canary","user":"user-id-canary"}"#)}fn response(observer: &mut ExecutionCapture, content_type: &'static str) {@@ -166,6 +166,10 @@ async fn usage_mode_excludes_content_and_context_credentials() {for secret in ["prompt-canary","schema-canary",+ "request-metadata-canary",+ "safety-id-canary",+ "cache-id-canary",+ "user-id-canary","output-canary","provider-secret","private-ingestion-token",@@ -175,6 +179,129 @@ async fn usage_mode_excludes_content_and_context_credentials() {}}+#[tokio::test]+async fn request_configuration_is_projected_out_of_input() {+ let request = json!({+ "model": "requested", "input": "hello", "instructions": "be brief",+ "tools": [{"type": "function", "name": "weather"}],+ "previous_response_id": "resp-previous", "future_field": {"keep": true},+ "temperature": 0.5, "stream": true, "parallel_tool_calls": false,+ "reasoning": {"effort": "high"}, "text": {"format": {"type": "json_object"}},+ "stream_options": {"include_obfuscation": false},+ "metadata": {"purpose": "test"}, "prompt_cache_key": "cache-key"+ });+ let capture = OpenAiResponsesCapture::new(+ &HeaderMap::new(),+ request.to_string().as_bytes(),+ IngestionMode::Full,+ )+ .into_facts();+ assert_eq!(+ capture.input.unwrap(),+ json!({+ "input": "hello", "instructions": "be brief",+ "tools": [{"type": "function", "name": "weather"}],+ "previous_response_id": "resp-previous", "future_field": {"keep": true}+ })+ );+ assert_eq!(capture.model_parameters["stream"], true);+ assert_eq!(capture.model_parameters["parallel_tool_calls"], false);+ assert_eq!(capture.model_parameters["reasoning"], request["reasoning"]);+ assert_eq!(capture.model_parameters["text"], request["text"]);+ assert_eq!(+ capture.model_parameters["stream_options"],+ request["stream_options"]+ );+ assert_eq!(capture.request_metadata["metadata"], request["metadata"]);+ assert_eq!(capture.request_metadata["prompt_cache_key"], "cache-key");+}++#[tokio::test]+async fn completion_time_requires_sse_generated_content_in_either_ingestion_mode() {+ for mode in ["full", "usage"] {+ for (kind, field) in [+ ("response.output_text.delta", "delta"),+ ("response.function_call_arguments.delta", "delta"),+ ("response.reasoning_summary_text.delta", "delta"),+ ("response.custom_tool_call_input.delta", "delta"),+ ("response.audio.delta", "delta"),+ ("response.audio.transcript.delta", "delta"),+ ("response.shell_call_command.delta", "delta"),+ (+ "response.image_generation_call.partial_image",+ "partial_image_b64",+ ),+ ] {+ let mut observer = observer(mode).await;+ response(&mut observer, "text/event-stream; charset=utf-8");+ observer.bytes(b": keepalive\n\n");+ observer.bytes(+ event(+ "response.created",+ json!({"response":{"status":"in_progress"}}),+ )+ .as_bytes(),+ );+ observer.bytes(event(kind, json!({field:""})).as_bytes());+ assert!(observer.first_byte_ms.is_some());+ assert!(observer.completion_start_ms.is_none());+ let delta = event(kind, json!({field:"hello"}));+ let split = delta.len() - 1;+ observer.bytes(&delta.as_bytes()[..split]);+ assert!(observer.completion_start_ms.is_none());+ observer.bytes(&delta.as_bytes()[split..]);+ let first = observer.completion_start_ms.unwrap();+ observer.bytes(event(kind, json!({field:"later"})).as_bytes());+ assert_eq!(observer.completion_start_ms, Some(first));+ }+ // The upstream content type controls timing even if the request asks to stream.+ let context = resolved_request_context_with_mode("provider-secret", mode).await;+ let mut observer = ExecutionCapture::openai_responses(+ &context,+ &HeaderMap::new(),+ br#"{"stream":true,"input":"hello"}"#,+ );+ response(&mut observer, "application/json; charset=utf-8");+ observer.bytes(br#"{"output":[],"status":"completed"}"#);+ observer.end_body();+ assert!(observer.first_byte_ms.is_some());+ assert!(observer.completion_start_ms.is_none());+ }+}++#[tokio::test]+async fn provider_error_details_are_bounded_and_full_mode_only() {+ for mode in ["full", "usage"] {+ for streaming in [false, true] {+ let mut observer = observer(mode).await;+ response(+ &mut observer,+ if streaming {+ "text/event-stream"+ } else {+ "application/json"+ },+ );+ let error = json!({"code":"rate_limit_exceeded","message":"é".repeat(MAX_FACT_STRING)});+ let body = if streaming {+ event("error", error)+ } else {+ json!({"error":error}).to_string()+ };+ observer.bytes(body.as_bytes());+ observer.end_body();+ let facts = &captured(&observer).facts;+ if mode == "full" {+ let message = facts.error_message.as_ref().unwrap();+ assert!(message.starts_with("rate_limit_exceeded: é"));+ assert!(message.len() <= MAX_FACT_STRING);+ } else {+ assert!(facts.error_message.is_none());+ }+ }+ }+}+#[tokio::test]async fn oversized_and_malformed_events_do_not_prevent_later_facts() {let mut observer = observer("full").await;@@ -577,11 +704,15 @@ async fn client_compression_preferences_do_not_disable_capture() {assert_eq!(collector.calls(), 1);let upload = uploaded.lock().unwrap();let metadata = uploaded_attribute(&upload, "langfuse.observation.metadata");- assert_eq!(metadata["capture_complete"], true);- assert_eq!(metadata["output_complete"], true);- assert_eq!(metadata["provider_response_id"], "resp-1");- assert_eq!(metadata["provider_status"], "completed");- assert!(metadata["native_usage"]["input_tokens"].is_number());+ assert!(metadata.get("capture_complete").is_none());+ assert!(metadata.get("output_complete").is_none());+ assert!(metadata.get("provider_status").is_none());+ assert!(metadata.get("native_usage").is_none());+ assert_eq!(metadata["langfuse.gateway.provider.response_id"], "resp-1");+ assert!(+ uploaded_attribute(&upload, "langfuse.observation.usage_details")["input_tokens"]+ .is_number()+ );assert_eq!(uploaded_attribute(&upload, "langfuse.observation.output")[0]["content"][0]["text"],"captured output"@@ -653,9 +784,15 @@ async fn capture_regression_preserves_native_request_and_usage() {execution.bytes(body.as_bytes());execution.end_body();let facts = &captured(&execution).facts;+ let mut expected_input = request.clone();+ expected_input.as_object_mut().unwrap().remove("model");assert_eq!(facts.input.as_ref(),- if mode == "full" { Some(&request) } else { None }+ if mode == "full" {+ Some(&expected_input)+ } else {+ None+ });assert_eq!(facts.usage_details.as_ref(), Some(&usage));assert!(facts.capture_complete);diff --git a/ai-gateway/src/providers/openai/tests.rs b/ai-gateway/src/providers/openai/tests.rsindex 7cbb25a600..4bf74c65fc 100644--- a/ai-gateway/src/providers/openai/tests.rs+++ b/ai-gateway/src/providers/openai/tests.rs@@ -348,7 +348,7 @@ async fn completed_json_and_sse_upload_once_without_waiting_for_ingestion() {),("text/event-stream",- "data: {\"type\":\"response.completed\",\"response\":{\"id\":\"resp-1\",\"model\":\"actual\",\"status\":\"completed\",\"output\":[],\"usage\":{\"input_tokens\":2,\"output_tokens\":1,\"total_tokens\":3}}}\n\n",+ "data: {\"type\":\"response.output_text.delta\",\"delta\":\"hello\"}\n\ndata: {\"type\":\"response.completed\",\"response\":{\"id\":\"resp-1\",\"model\":\"actual\",\"status\":\"completed\",\"output\":[],\"usage\":{\"input_tokens\":2,\"output_tokens\":1,\"total_tokens\":3}}}\n\n",503,),] {@@ -404,7 +404,7 @@ async fn completed_json_and_sse_upload_once_without_waiting_for_ingestion() {.await.unwrap().unwrap();- assert_completed_upload(&payload);+ assert_completed_upload(&payload, content_type == "text/event-stream");release.notify_one();telemetry.shutdown(Instant::now() + Duration::from_secs(1))@@ -488,9 +488,13 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rel).unwrap();assert_eq!(- metadata["relay_outcome"],- if cancelled { "cancelled" } else { "timeout" }+ attrs+ .iter()+ .find(|a| a["key"] == "langfuse.observation.level")+ .unwrap()["value"]["stringValue"],+ if cancelled { "WARNING" } else { "ERROR" });+ assert!(metadata.get("relay_outcome").is_none());telemetry.shutdown(Instant::now() + Duration::from_secs(1)).await;@@ -499,7 +503,7 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rel}}-fn assert_completed_upload(payload: &serde_json::Value) {+fn assert_completed_upload(payload: &serde_json::Value, streaming: bool) {use serde_json::{Value, json};let span = &payload["resourceSpans"][0]["scopeSpans"][0]["spans"][0];let attrs = span["attributes"].as_array().unwrap();@@ -509,10 +513,11 @@ fn assert_completed_upload(payload: &serde_json::Value) {.any(|a| a["key"] == "langfuse.observation.model.name"&& a["value"]["stringValue"] == "actual"));- assert!(+ assert_eq!(attrs.iter()- .any(|a| a["key"] == "langfuse.observation.completion_start_time")+ .any(|a| a["key"] == "langfuse.observation.completion_start_time"),+ streaming,);let metadata: Value = serde_json::from_str(attrs@@ -523,9 +528,20 @@ fn assert_completed_upload(payload: &serde_json::Value) {.unwrap(),).unwrap();- assert_eq!(metadata["relay_outcome"], "eof");+ assert!(metadata.get("relay_outcome").is_none());+ assert!(metadata.get("native_usage").is_none());+ assert_eq!(metadata["langfuse.gateway.provider.response_id"], "resp-1");+ let usage: Value = serde_json::from_str(+ attrs+ .iter()+ .find(|a| a["key"] == "langfuse.observation.usage_details")+ .unwrap()["value"]["stringValue"]+ .as_str()+ .unwrap(),+ )+ .unwrap();assert_eq!(- metadata["native_usage"],+ usage,json!({"input_tokens":2,"output_tokens":1,"total_tokens":3}));for secret in [diff --git a/ai-gateway/src/telemetry/mapping.rs b/ai-gateway/src/telemetry/mapping.rsindex 9b61012e3a..7966dcec9b 100644--- a/ai-gateway/src/telemetry/mapping.rs+++ b/ai-gateway/src/telemetry/mapping.rs@@ -7,27 +7,8 @@ use crate::capture::{InferenceFacts, RelayOutcome};pub(super) fn span(facts: InferenceFacts, trace_id: &str, observation_id: &str) -> Value {let full = facts.metadata.get("ingestion_mode").and_then(Value::as_str) == Some("full");let (level, message) = observation_status(&facts);+ let metadata = generation_metadata(&facts);let inference = facts.inference;- let mut metadata = facts.metadata.as_object().cloned().unwrap_or_default();- metadata.extend([- ("api_format".into(), json!(facts.api_format)),- ("relay_outcome".into(), json!(facts.outcome)),- ("http_status".into(), json!(facts.http_status)),- ("first_byte_ms".into(), json!(facts.first_byte_ms)),- ("requested_model".into(), json!(inference.requested_model)),- (- "provider_response_id".into(),- json!(inference.provider_response_id),- ),- (- "provider_request_id".into(),- json!(inference.provider_request_id),- ),- ("provider_status".into(), json!(inference.provider_status)),- ("input_complete".into(), json!(inference.input_complete)),- ("output_complete".into(), json!(inference.output_complete)),- ("capture_complete".into(), json!(inference.capture_complete)),- ]);let mut attributes = vec![attribute("langfuse.observation.type", "generation")];if let Some(model) = inference.model.or(inference.requested_model) {attributes.push(attribute("langfuse.observation.model.name", model));@@ -38,14 +19,13 @@ pub(super) fn span(facts: InferenceFacts, trace_id: &str, observation_id: &str)Value::Object(inference.model_parameters).to_string(),));}- if let Some(usage) = inference.usage_details {- if let Some(projected) = openai_usage(&usage) {- attributes.push(attribute(- "langfuse.observation.usage_details",- projected.to_string(),- ));- }- metadata.insert("native_usage".into(), usage);+ if let Some(usage) = inference.usage_details+ && let Some(projected) = openai_usage(&usage)+ {+ attributes.push(attribute(+ "langfuse.observation.usage_details",+ projected.to_string(),+ ));}if full {if let Some(input) = inference.input {@@ -56,7 +36,7 @@ pub(super) fn span(facts: InferenceFacts, trace_id: &str, observation_id: &str)}}if let Some(completion_start) = facts- .first_byte_ms+ .completion_start_ms.and_then(|elapsed| facts.start_time_unix_ms.checked_add(elapsed)).and_then(|timestamp| i64::try_from(timestamp).ok()).and_then(DateTime::from_timestamp_millis)@@ -86,24 +66,91 @@ pub(super) fn span(facts: InferenceFacts, trace_id: &str, observation_id: &str)})}-fn observation_status(facts: &InferenceFacts) -> (&'static str, Option<&'static str>) {- let failure = if facts.http_status.is_some_and(|status| status >= 400) {- Some("Provider HTTP error")+fn generation_metadata(facts: &InferenceFacts) -> Map<String, Value> {+ let mut metadata = Map::new();+ if let Some(attribution) = facts+ .metadata+ .get("key_metadata")+ .and_then(Value::as_object)+ {+ for (key, value) in attribution {+ // Attribution remains searchable, but cannot impersonate gateway or OTEL fields.+ if ![+ "langfuse.gateway",+ "http_status",+ "scope",+ "resourceAttributes",+ "attributes",+ ]+ .iter()+ .any(|reserved| key == reserved || key.starts_with(&format!("{reserved}.")))+ {+ metadata.insert(key.clone(), value.clone());+ }+ metadata.insert(+ format!("langfuse.gateway.api-key.metadata.{key}"),+ value.clone(),+ );+ }+ }+ for (source, target) in [+ ("project_id", "project_id"),+ ("organization_id", "organization_id"),+ ("ingestion_mode", "ingestion_mode"),+ ("key_id", "api-key.id"),+ ("provider_connection_id", "provider.connection_id"),+ ] {+ if let Some(value) = facts.metadata.get(source) {+ metadata.insert(format!("langfuse.gateway.{target}"), value.clone());+ }+ }+ metadata.insert(+ "langfuse.gateway.api_format".into(),+ json!(facts.api_format),+ );+ metadata.insert("http_status".into(), json!(facts.http_status));+ for (key, value) in [+ ("response_id", &facts.inference.provider_response_id),+ ("request_id", &facts.inference.provider_request_id),+ ] {+ if let Some(value) = value {+ metadata.insert(format!("langfuse.gateway.provider.{key}"), json!(value));+ }+ }+ for (key, value) in &facts.inference.request_metadata {+ metadata.insert(+ format!("langfuse.gateway.provider.request.{key}"),+ value.clone(),+ );+ }+ metadata+}++fn observation_status(facts: &InferenceFacts) -> (&'static str, Option<String>) {+ let failure = if let Some(status) = facts.http_status.filter(|status| *status >= 400) {+ Some(format!("Provider HTTP error ({status})"))} else if facts.inference.provider_status.as_deref() == Some("failed") {- Some("Provider response failed")+ Some("Provider response failed".into())} else {match facts.outcome {- RelayOutcome::Timeout => Some("Provider response timed out"),- RelayOutcome::TransportError => Some("Provider transport error"),+ RelayOutcome::Timeout => Some("Provider response timed out".into()),+ RelayOutcome::TransportError => Some("Provider transport error".into()),RelayOutcome::Eof | RelayOutcome::Cancelled => None,}};- if let Some(message) = failure {+ if let Some(mut message) = failure {+ if let Some(status) = facts.http_status.filter(|status| *status < 400) {+ message = format!("{message} (HTTP {status})");+ }+ if let Some(error) = &facts.inference.error_message {+ message.push_str(": ");+ message.push_str(error);+ }("ERROR", Some(message))} else if facts.outcome == RelayOutcome::Cancelled {- ("WARNING", Some("Client cancelled the response"))+ ("WARNING", Some("Client cancelled the response".into()))} else if facts.inference.provider_status.as_deref() == Some("incomplete") {- ("WARNING", Some("Provider response incomplete"))+ ("WARNING", Some("Provider response incomplete".into()))} else {("DEFAULT", None)}@@ -114,7 +161,7 @@ fn attribute(key: &str, value: impl Into<String>) -> Value {}/// The receiver's native `OpenAI` usage schema is strict at the top level, but-/// accepts new numeric detail counters. Keep the original payload in metadata.+/// accepts new numeric detail counters.fn openai_usage(usage: &Value) -> Option<Value> {let usage = usage.as_object()?;let mut projected = Map::new();@@ -154,7 +201,8 @@ mod tests {api_format: "openai.responses",start_time_unix_ms: 1_735_689_600_000,duration_ms: 900,- first_byte_ms: Some(125),+ first_byte_ms: Some(50),+ completion_start_ms: Some(125),http_status: Some(200),metadata: json!({"ingestion_mode": "full", "project_id": "project", "key_id": "key", "key_metadata": {"team": "search"}}),outcome: RelayOutcome::Eof,@@ -198,7 +246,7 @@ mod tests {}#[test]- fn full_generation_preserves_native_content_attribution_and_first_byte_time() {+ fn full_generation_preserves_native_content_attribution_and_completion_time() {let span = span(facts(),"0123456789abcdef0123456789abcdef",@@ -223,10 +271,75 @@ mod tests {);assert!(attrs.contains_key("langfuse.observation.output"));let metadata = metadata(&attrs);- assert_eq!(metadata["project_id"], "project");- assert_eq!(metadata["key_metadata"]["team"], "search");- assert_eq!(metadata["requested_model"], "requested-model");- assert_eq!(metadata["capture_complete"], true);+ assert_eq!(+ metadata,+ json!({+ "http_status": 200,+ "team": "search",+ "langfuse.gateway.api-key.metadata.team": "search",+ "langfuse.gateway.api-key.id": "key",+ "langfuse.gateway.project_id": "project",+ "langfuse.gateway.ingestion_mode": "full",+ "langfuse.gateway.api_format": "openai.responses",+ })+ );+ }++ #[test]+ fn attribution_is_searchable_without_overwriting_gateway_or_otel_metadata() {+ let mut facts = facts();+ facts.metadata["organization_id"] = json!("org");+ facts.metadata["provider_connection_id"] = json!("connection");+ facts.metadata["key_metadata"] = json!({+ "team": "search", "enabled": true, "cost_center": 42,+ "http_status": 500,+ "langfuse.gateway.api-key.id": "spoofed-key",+ "langfuse.gateway.provider.request_id": "spoofed-request",+ "langfuse.gateway.future_field": "spoofed-future",+ "langfuse.gateway.api-key.metadata.team": "spoofed-team",+ "scope": "spoofed-scope", "scope.name": "spoofed-scope-name",+ "resourceAttributes": "spoofed-resource",+ "attributes": "spoofed-attributes"+ });+ facts.inference.provider_response_id = Some("response".into());+ let attrs = attributes(&span(facts, "trace", "span"));+ let metadata = metadata(&attrs);+ assert_eq!(metadata["team"], "search");+ assert_eq!(metadata["enabled"], true);+ assert_eq!(metadata["cost_center"], 42);+ assert_eq!(metadata["langfuse.gateway.api-key.metadata.team"], "search");+ assert_eq!(+ metadata["langfuse.gateway.api-key.metadata.http_status"],+ 500+ );+ assert_eq!(metadata["http_status"], 200);+ assert_eq!(metadata["langfuse.gateway.api-key.id"], "key");+ assert_eq!(+ metadata["langfuse.gateway.provider.connection_id"],+ "connection"+ );+ assert_eq!(+ metadata["langfuse.gateway.provider.response_id"],+ "response"+ );+ assert_eq!(metadata["langfuse.gateway.organization_id"], "org");+ for key in [+ "scope",+ "scope.name",+ "resourceAttributes",+ "attributes",+ "langfuse.gateway.future_field",+ ] {+ assert!(metadata.get(key).is_none(), "{key}");+ }+ assert_ne!(+ metadata["langfuse.gateway.provider.request_id"],+ "spoofed-request"+ );+ assert_eq!(+ metadata["langfuse.gateway.api-key.metadata.langfuse.gateway.api-key.metadata.team"],+ "spoofed-team"+ );}#[test]@@ -234,7 +347,7 @@ mod tests {let mut facts = facts();facts.metadata["ingestion_mode"] = json!("usage");facts.inference.model = None;- facts.first_byte_ms = None;+ facts.completion_start_ms = None;let attrs = attributes(&span(facts, "trace", "span"));for key in ["input", "output", "usage_details", "completion_start_time"] {assert!(!attrs.contains_key(&format!("langfuse.observation.{key}")));@@ -254,7 +367,7 @@ mod tests {});facts.inference.usage_details = Some(usage.clone());let attrs = attributes(&span(facts, "trace", "span"));- assert_eq!(metadata(&attrs)["native_usage"], usage);+ assert!(metadata(&attrs).get("native_usage").is_none());let projected: Value = serde_json::from_str(attrs["langfuse.observation.usage_details"].as_str()@@ -272,12 +385,72 @@ mod tests {}#[test]- fn unsupported_usage_remains_metadata_without_invented_totals() {+ fn unsupported_usage_omits_usage_without_invented_totals() {let mut facts = facts();facts.inference.usage_details = Some(json!({"input_tokens": 9}));let attrs = attributes(&span(facts, "trace", "span"));assert!(!attrs.contains_key("langfuse.observation.usage_details"));- assert_eq!(metadata(&attrs)["native_usage"], json!({"input_tokens": 9}));+ assert!(metadata(&attrs).get("native_usage").is_none());+ }++ #[test]+ fn provider_request_metadata_is_namespaced_and_keeps_native_types() {+ let mut facts = facts();+ facts.inference.request_metadata = serde_json::from_value(json!({+ "metadata": {"customer": "customer-1"},+ "prompt_cache_key": "cache-key",+ "safety_identifier": "safety-id",+ "user": "legacy-user"+ }))+ .unwrap();+ let attrs = attributes(&span(facts, "trace", "span"));+ let metadata = metadata(&attrs);+ assert_eq!(+ metadata["langfuse.gateway.provider.request.metadata"],+ json!({"customer": "customer-1"})+ );+ assert_eq!(+ metadata["langfuse.gateway.provider.request.prompt_cache_key"],+ "cache-key"+ );+ assert_eq!(+ metadata["langfuse.gateway.provider.request.safety_identifier"],+ "safety-id"+ );+ assert_eq!(+ metadata["langfuse.gateway.provider.request.user"],+ "legacy-user"+ );+ for key in ["metadata", "prompt_cache_key", "safety_identifier", "user"] {+ assert!(metadata.get(key).is_none());+ }+ }++ #[test]+ fn provider_errors_include_http_status_and_available_details() {+ for (status, provider_status, error, expected) in [+ (429, None, None, "Provider HTTP error (429)"),+ (+ 429,+ None,+ Some("rate_limit_exceeded: Too many requests"),+ "Provider HTTP error (429): rate_limit_exceeded: Too many requests",+ ),+ (+ 200,+ Some("failed"),+ Some("server_error: Please retry"),+ "Provider response failed (HTTP 200): server_error: Please retry",+ ),+ ] {+ let mut facts = facts();+ facts.http_status = Some(status);+ facts.inference.provider_status = provider_status.map(str::to_owned);+ facts.inference.error_message = error.map(str::to_owned);+ let attrs = attributes(&span(facts, "trace", "span"));+ assert_eq!(attrs["langfuse.observation.status_message"], expected);+ assert_eq!(attrs["langfuse.observation.level"], "ERROR");+ }}#[test]@@ -304,7 +477,7 @@ mod tests {let attrs = attributes(&span);assert_eq!(attrs["langfuse.observation.level"], level);assert_eq!(span["status"]["code"], code);- assert_eq!(metadata(&attrs)["relay_outcome"], json!(outcome));+ assert!(metadata(&attrs).get("relay_outcome").is_none());}}}diff --git a/ai-gateway/src/telemetry/tests.rs b/ai-gateway/src/telemetry/tests.rsindex 2b1bd50c3a..116c2788e5 100644--- a/ai-gateway/src/telemetry/tests.rs+++ b/ai-gateway/src/telemetry/tests.rs@@ -20,6 +20,7 @@ fn facts(project: &str) -> InferenceFacts {start_time_unix_ms: 1_735_689_600_000,duration_ms: 100,first_byte_ms: Some(10),+ completion_start_ms: None,http_status: Some(200),metadata: json!({"project_id": project, "ingestion_mode": "usage"}),outcome: RelayOutcome::Eof,@@ -120,7 +121,10 @@ async fn concurrent_projects_keep_their_original_grants_and_attribution() {serde_json::from_str(metadata["value"]["stringValue"].as_str().unwrap()).unwrap();received.lock().unwrap().push((authorization,- metadata["project_id"].as_str().unwrap().to_owned(),+ metadata["langfuse.gateway.project_id"]+ .as_str()+ .unwrap()+ .to_owned(),));response(200, "{}")}diff --git a/packages/shared/src/server/otel/OtelIngestionProcessor.metadataDropped.test.ts b/packages/shared/src/server/otel/OtelIngestionProcessor.metadataDropped.test.tsindex e8b91d8770..b833b8a806 100644--- a/packages/shared/src/server/otel/OtelIngestionProcessor.metadataDropped.test.ts+++ b/packages/shared/src/server/otel/OtelIngestionProcessor.metadataDropped.test.ts@@ -119,6 +119,99 @@ const expectDropTags = (expect(tags?.sdkVersion).toBe("3.8.1");};+describe("gateway metadata", () => {+ it.each([+ ["v3", "langfuse-ai-gateway"],+ ["v4", "langfuse-ai-gateway"],+ ["v3", "other-instrumentation"],+ ["v4", "other-instrumentation"],+ ])(+ "preserves canonical fields and metadata for %s %s",+ async (path, scope) => {+ const completionStartTime = "2025-07-13T05:20:00.500Z";+ const modelParameters = {+ service_tier: "default",+ stream: true,+ reasoning: { effort: "low" },+ };+ const usageDetails = { input: 10, output: 21 };+ const canonicalAttributes = {+ "langfuse.observation.type": "generation",+ "langfuse.observation.level": "ERROR",+ "langfuse.observation.status_message": "HTTP 429: rate limited",+ "langfuse.observation.model.name": "test-model",+ "langfuse.observation.model.parameters":+ JSON.stringify(modelParameters),+ "langfuse.observation.usage_details": JSON.stringify(usageDetails),+ "langfuse.observation.cost_details": JSON.stringify({ total: 0.001 }),+ "langfuse.observation.completion_start_time": completionStartTime,+ };+ const batch = buildBatch(+ Object.entries({+ ...canonicalAttributes,+ "langfuse.observation.input": '[{"role":"user","content":"Hi"}]',+ "langfuse.observation.output": '[{"type":"message","content":[]}]',+ "langfuse.observation.metadata": JSON.stringify({+ "langfuse.gateway.provider.request_id": "req-test",+ }),+ "langfuse.observation.metadata.langfuse.gateway.api-key.id":+ "key-test",+ "custom.attribute": "keep-custom",+ "langfuse.observation.custom": "keep-unknown",+ }).map(([key, value]) => ({ key, value: { stringValue: value } })),+ );+ batch[0].scopeSpans![0].scope!.name = scope;+ const processor = createProcessor();+ const observation =+ path === "v4"+ ? processor.processToEvent(batch)[0]+ : (await processor.processToIngestionEvents(batch)).find(+ (event) => event.type === "generation-create",+ )?.body;++ expect(observation).toMatchObject({+ level: "ERROR",+ statusMessage: "HTTP 429: rate limited",+ modelParameters: {+ service_tier: "default",+ stream: "true",+ reasoning: '{"effort":"low"}',+ },+ completionStartTime,+ input: '[{"role":"user","content":"Hi"}]',+ output: '[{"type":"message","content":[]}]',+ ...(path === "v4"+ ? {+ type: "GENERATION",+ modelName: "test-model",+ providedUsageDetails: usageDetails,+ providedCostDetails: { total: 0.001 },+ }+ : {+ model: "test-model",+ usageDetails,+ costDetails: { total: 0.001 },+ }),+ });+ expect(observation?.metadata).toEqual({+ "langfuse.gateway.provider.request_id": "req-test",+ "langfuse.gateway.api-key.id": "key-test",+ attributes: {+ ...(scope === "langfuse-ai-gateway" ? {} : canonicalAttributes),+ "custom.attribute": "keep-custom",+ "langfuse.observation.custom": "keep-unknown",+ },+ resourceAttributes: { "service.name": "test-svc" },+ scope: {+ name: scope,+ version: "3.8.1",+ attributes: { public_key: "pk-test" },+ },+ });+ },+ );+});+describe("OTel metadata_dropped metric", () => {beforeEach(() => {recordIncrementMock.mockClear();diff --git a/packages/shared/src/server/otel/OtelIngestionProcessor.ts b/packages/shared/src/server/otel/OtelIngestionProcessor.tsindex 55c7cd5ced..3530e37795 100644--- a/packages/shared/src/server/otel/OtelIngestionProcessor.ts+++ b/packages/shared/src/server/otel/OtelIngestionProcessor.ts@@ -1887,6 +1887,16 @@ export class OtelIngestionProcessor {delete rawFilteredAttributes[key];});+ // Gateway observation attributes are represented by canonical fields.+ // Keep unknown attributes available for diagnostics.+ if (instrumentationScopeName === "langfuse-ai-gateway") {+ for (const key of Object.values(LangfuseOtelSpanAttributes)) {+ if (key.startsWith("langfuse.observation.")) {+ delete rawFilteredAttributes[key];+ }+ }+ }+// Delete gen_ai.prompt.*, gen_ai.completion.*, llm.input_messages.*, llm.output_messages.*,// and metadata blob keys (already extracted into top-level metadata by extractMetadata())Object.keys(attributes).forEach((key) => {