Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

fix(ai-gateway): clean up generation fields and streaming timing (#17545)

Hassieb Pakzad84af505e2b7d
diff --git a/ai-gateway/README.md b/ai-gateway/README.md
index acac378690..ddc70f6675 100644
--- a/ai-gateway/README.md
+++ b/ai-gateway/README.md
@@ -229,12 +229,27 @@ are dropped without re-resolving the execution. Requests explicitly select v4 in
Each execution becomes one root generation with generated trace/observation IDs,
actual/requested model, model parameters, native usage, and trusted key/connection
-attribution. Full mode includes the captured native input/output; usage mode omits
-content. First-byte timing supplies Langfuse's completion-start time as the gateway's
-first-token approximation. Relay outcome, provider status and capture completeness
-remain separate metadata fields. The exporter projects native OpenAI usage into the
-receiver's supported shape for pricing and preserves the untouched usage object as
-`native_usage` metadata. Missing usage is not reported as zero.
+attribution. Full mode includes the captured input/output; usage mode omits content.
+Completion-start time is emitted only for upstream `text/event-stream` responses,
+on the first nonempty text, reasoning, refusal, tool-input, audio, or partial-image content. JSON responses
+and streams without a captured content delta have no completion-start time or TTFT.
+The exporter projects native OpenAI usage into the receiver's supported shape for
+pricing without duplicating it in metadata. Missing usage is not reported as zero.
+
+Gateway metadata uses `langfuse.gateway.*`: `project_id`, `organization_id`,
+`ingestion_mode`, `api_format`, `api-key.id`, and `provider.connection_id`,
+`provider.request_id`, `provider.response_id`. API-key attribution entries appear
+both as top-level metadata and under `langfuse.gateway.api-key.metadata.*`.
+Gateway and OpenTelemetry fields win collisions; the namespaced attribution copy
+preserves the original value. `http_status` stays top level. Relay outcome, provider
+status, completeness flags and first-byte timing remain internal facts rather than
+generation metadata. Ingestion removes mapped observation-attribute duplicates for
+the gateway scope while preserving custom attributes, scope and resources.
+
+Provider HTTP failures and failed SSE responses set the generation level to `ERROR`
+with the available HTTP status in its status message. Full mode also includes a
+bounded provider error code/message; usage mode omits these details because provider
+errors may echo request content.
Provisional upload limits are 32 concurrent tasks, 4 MiB serialized facts per record,
16 MiB total retained serialized-fact/credential bytes, 8 MiB encoded payloads, and
@@ -265,12 +280,23 @@ including at debug level. `LANGFUSE_LOG_FORMAT=json` emits one JSON object per l
Web's resolved ingestion mode controls content capture:
-- `full`: input is the native request JSON object, including `input`, `instructions`,
- tools, parameters, context references and unknown fields. Output is an ordered
- array of native completed items. Content is sent only through Langfuse ingestion.
+- `full`: input retains native `input`, `instructions`, tools, prompt/context
+ references and unknown fields. Model and configuration are projected out of input.
+ Output is an ordered array of native completed items. Content is sent only through
+ Langfuse ingestion.
- `usage`: input and output are null. Model, scalar parameters, native usage,
timing and trusted attribution are retained; request schemas/content are omitted.
+Full-mode model parameters include sampling/token limits, service tier, reasoning,
+text formatting, tool choice/limits, context management, truncation, streaming options,
+background/store/include flags, moderation and cache options/retention. Capture retains
+JSON types; ingestion applies Langfuse's existing model-parameter normalization
+(booleans and structured values become JSON strings). Omitted request parameters
+are not filled with assumed defaults.
+The response model and service tier take precedence over requested values when present.
+Request `metadata`, `prompt_cache_key`, `safety_identifier` and deprecated `user` are
+stored under `langfuse.gateway.provider.request.*` only in full mode.
+
In both modes, `usage_details` preserves the provider's entire usage object,
including nested and unknown fields. The gateway does not rename counters,
subtract cached tokens, or synthesize totals. Missing usage stays null.
@@ -286,10 +312,11 @@ output array at EOF. Missing usage remains null, rather than invented zeros.
`outcome` describes the relay (`eof`, `cancelled`, `timeout`, `transport_error`);
`provider_status` is separate. For example, a completed provider response can
still end with downstream cancellation. `first_byte_ms` measures the first body
-bytes observed by the gateway. Telemetry uses this as the first-token approximation
-for Langfuse completion-start time; it does not wait for completed output items.
+bytes observed by the gateway for operational metrics. `completion_start_ms` measures
+the first captured SSE content delta independently; it ignores response-created
+events and keepalives and does not retain or reconstruct delta content.
-`input_complete` means the full native request was captured; it is false when
+`input_complete` means request input was captured and configuration projected; it is false when
content is omitted in usage mode. `output_complete` means response inspection
finished without capture gaps for the configured mode (including a terminal event
and all expected completed items for full-mode SSE). It does not promise that the
diff --git a/ai-gateway/src/capture/facts.rs b/ai-gateway/src/capture/facts.rs
index ee12dfed8d..1d3dccd117 100644
--- a/ai-gateway/src/capture/facts.rs
+++ b/ai-gateway/src/capture/facts.rs
@@ -17,6 +17,7 @@ pub(crate) struct InferenceFacts {
pub start_time_unix_ms: u128,
pub duration_ms: u128,
pub first_byte_ms: Option<u128>,
+ pub completion_start_ms: Option<u128>,
pub http_status: Option<u16>,
pub metadata: Value,
pub outcome: RelayOutcome,
@@ -30,6 +31,8 @@ pub(crate) struct ProviderFacts {
pub model: Option<String>,
pub requested_model: Option<String>,
pub model_parameters: Map<String, Value>,
+ pub request_metadata: Map<String, Value>,
+ pub error_message: Option<String>,
pub usage_details: Option<Value>,
pub input: Option<Value>,
pub output: Option<Value>,
diff --git a/ai-gateway/src/capture/mod.rs b/ai-gateway/src/capture/mod.rs
index 61c9c62c59..f1b86d30ba 100644
--- a/ai-gateway/src/capture/mod.rs
+++ b/ai-gateway/src/capture/mod.rs
@@ -33,7 +33,7 @@ impl ProtocolCapture {
}
}
- fn bytes(&mut self, bytes: &[u8]) {
+ fn bytes(&mut self, bytes: &[u8]) -> bool {
match self {
Self::OpenAiResponses(capture) => capture.bytes(bytes),
}
@@ -60,6 +60,7 @@ pub(crate) struct ExecutionCapture {
started: Instant,
start_time_unix_ms: u128,
first_byte_ms: Option<u128>,
+ completion_start_ms: Option<u128>,
http_status: Option<u16>,
metadata: Value,
delivery: Option<(telemetry::Telemetry, telemetry::DeliveryContext)>,
@@ -100,6 +101,7 @@ impl ExecutionCapture {
started,
start_time_unix_ms,
first_byte_ms: None,
+ completion_start_ms: None,
http_status: None,
metadata: json!({
"organization_id": attribution.organization_id(),
@@ -137,7 +139,10 @@ impl ExecutionCapture {
self.first_byte_ms
.get_or_insert_with(|| self.started.elapsed().as_millis());
}
- protocol.bytes(bytes);
+ if protocol.bytes(bytes) {
+ self.completion_start_ms
+ .get_or_insert_with(|| self.started.elapsed().as_millis());
+ }
}
}
@@ -159,6 +164,7 @@ impl ExecutionCapture {
start_time_unix_ms: self.start_time_unix_ms,
duration_ms: self.started.elapsed().as_millis(),
first_byte_ms: self.first_byte_ms,
+ completion_start_ms: self.completion_start_ms,
http_status: self.http_status,
metadata: std::mem::take(&mut self.metadata),
outcome,
diff --git a/ai-gateway/src/capture/openai_responses.rs b/ai-gateway/src/capture/openai_responses.rs
index b64419a086..5803a1ba6c 100644
--- a/ai-gateway/src/capture/openai_responses.rs
+++ b/ai-gateway/src/capture/openai_responses.rs
@@ -77,11 +77,14 @@ impl OpenAiResponsesCapture {
}
}
- pub fn bytes(&mut self, bytes: &[u8]) {
+ pub fn bytes(&mut self, bytes: &[u8]) -> bool {
let mut body = std::mem::replace(&mut self.body, ResponseBody::Unavailable);
+ let mut completion_started = false;
match &mut body {
ResponseBody::Sse(sse) => {
- sse.push(bytes, MAX_CAPTURE_BYTES, |event| self.event(event));
+ sse.push(bytes, MAX_CAPTURE_BYTES, |event| {
+ completion_started |= self.event(event);
+ });
}
ResponseBody::Json(buffer)
if buffer.len().saturating_add(bytes.len()) <= MAX_CAPTURE_BYTES =>
@@ -95,6 +98,7 @@ impl OpenAiResponsesCapture {
_ => {}
}
self.body = body;
+ completion_started
}
pub fn end_body(&mut self) {
@@ -133,43 +137,92 @@ impl OpenAiResponsesCapture {
self.facts
}
- fn request(&mut self, request: Map<String, Value>) {
+ fn request(&mut self, mut request: Map<String, Value>) {
self.facts.requested_model = request
.get("model")
.and_then(Value::as_str)
.and_then(bounded_string);
self.facts.model.clone_from(&self.facts.requested_model);
- for key in ["temperature", "top_p", "max_output_tokens", "service_tier"] {
- if let Some(value) = request
- .get(key)
- .filter(|v| v.is_number() || v.as_str().is_some_and(|s| s.len() <= MAX_FACT_STRING))
- {
- self.facts
- .model_parameters
- .insert(key.to_owned(), value.clone());
+ request.remove("model");
+ for key in [
+ "temperature",
+ "top_p",
+ "top_logprobs",
+ "max_output_tokens",
+ "service_tier",
+ "parallel_tool_calls",
+ "max_tool_calls",
+ "truncation",
+ "stream",
+ "background",
+ "store",
+ "prompt_cache_retention",
+ ] {
+ if let Some(value) = request.remove(key).filter(|v| {
+ self.mode == IngestionMode::Full
+ || v.is_number()
+ || v.is_boolean()
+ || v.as_str().is_some_and(|s| s.len() <= MAX_FACT_STRING)
+ }) {
+ self.facts.model_parameters.insert(key.to_owned(), value);
}
}
if self.mode == IngestionMode::Full {
- for key in ["reasoning", "text"] {
- if let Some(value) = request.get(key) {
- self.facts
- .model_parameters
- .insert(key.to_owned(), value.clone());
+ for key in [
+ "reasoning",
+ "text",
+ "tool_choice",
+ "context_management",
+ "stream_options",
+ "include",
+ "moderation",
+ "prompt_cache_options",
+ ] {
+ if let Some(value) = request.remove(key) {
+ self.facts.model_parameters.insert(key.to_owned(), value);
+ }
+ }
+ for key in ["metadata", "prompt_cache_key", "safety_identifier", "user"] {
+ if let Some(value) = request.remove(key) {
+ self.facts.request_metadata.insert(key.to_owned(), value);
}
}
- // Keep the original request object, including native instructions/input
- // and unknown fields. Authentication headers are never part of this value.
+ // Preserve native prompt context and unknown fields after projecting configuration.
self.facts.input = Some(Value::Object(request));
self.facts.input_complete = true;
}
}
- fn event(&mut self, bytes: &[u8]) {
+ fn event(&mut self, bytes: &[u8]) -> bool {
let Ok(Value::Object(mut event)) = serde_json::from_slice(bytes) else {
self.response_valid = false;
- return;
+ return false;
};
match event.get("type").and_then(Value::as_str) {
+ Some(
+ "response.output_text.delta"
+ | "response.refusal.delta"
+ | "response.function_call_arguments.delta"
+ | "response.custom_tool_call_input.delta"
+ | "response.reasoning_text.delta"
+ | "response.reasoning_summary_text.delta"
+ | "response.mcp_call_arguments.delta"
+ | "response.code_interpreter_call_code.delta"
+ | "response.audio.delta"
+ | "response.audio.transcript.delta"
+ | "response.shell_call_command.delta",
+ ) => {
+ return event
+ .get("delta")
+ .and_then(Value::as_str)
+ .is_some_and(|delta| !delta.is_empty());
+ }
+ Some("response.image_generation_call.partial_image") => {
+ return event
+ .get("partial_image_b64")
+ .and_then(Value::as_str)
+ .is_some_and(|image| !image.is_empty());
+ }
Some("response.output_item.done") if self.mode == IngestionMode::Full => {
if let (Some(index), Some(item)) = (
event.get("output_index").and_then(Value::as_u64),
@@ -204,9 +257,13 @@ impl OpenAiResponsesCapture {
self.response_valid = false;
}
}
- Some("error") => self.facts.provider_status = Some("failed".to_owned()),
+ Some("error") => {
+ self.facts.provider_status = Some("failed".to_owned());
+ self.error(&event);
+ }
_ => {}
}
+ false
}
fn item(&mut self, index: u64, item: Value) {
@@ -229,6 +286,9 @@ impl OpenAiResponsesCapture {
}
fn response_facts(&mut self, response: &Map<String, Value>) {
+ if let Some(error) = response.get("error").and_then(Value::as_object) {
+ self.error(error);
+ }
for (key, target) in [
("id", &mut self.facts.provider_response_id),
("model", &mut self.facts.model),
@@ -256,6 +316,24 @@ impl OpenAiResponsesCapture {
self.facts.usage_details = Some(usage.clone());
}
}
+
+ fn error(&mut self, error: &Map<String, Value>) {
+ // Provider error messages may echo prompt content, so retain them only in full mode.
+ if self.mode != IngestionMode::Full {
+ return;
+ }
+ let detail = ["code", "message"]
+ .into_iter()
+ .filter_map(|key| error.get(key).and_then(Value::as_str))
+ .filter(|value| !value.is_empty())
+ .map(|value| &value[..value.floor_char_boundary(MAX_FACT_STRING)])
+ .collect::<Vec<_>>()
+ .join(": ");
+ if !detail.is_empty() {
+ self.facts.error_message =
+ Some(detail[..detail.floor_char_boundary(MAX_FACT_STRING)].to_owned());
+ }
+ }
}
#[cfg(test)]
diff --git a/ai-gateway/src/capture/openai_responses/tests.rs b/ai-gateway/src/capture/openai_responses/tests.rs
index d03fdd9a65..1a137c2f56 100644
--- a/ai-gateway/src/capture/openai_responses/tests.rs
+++ b/ai-gateway/src/capture/openai_responses/tests.rs
@@ -24,7 +24,7 @@ fn captured(execution: &ExecutionCapture) -> &OpenAiResponsesCapture {
async fn observer(mode: &'static str) -> ExecutionCapture {
let context = resolved_request_context_with_mode("provider-secret", mode).await;
- ExecutionCapture::openai_responses(&context, &HeaderMap::new(), br#"{"model":"requested","instructions":"prompt-canary","input":[{"role":"user","content":"hello"}],"tools":[{"type":"function","name":"weather","parameters":{"type":"object"}}],"text":{"format":{"type":"json_schema","schema":{"description":"schema-canary"}}},"service_tier":"auto"}"#)
+ ExecutionCapture::openai_responses(&context, &HeaderMap::new(), br#"{"model":"requested","instructions":"prompt-canary","input":[{"role":"user","content":"hello"}],"tools":[{"type":"function","name":"weather","parameters":{"type":"object"}}],"text":{"format":{"type":"json_schema","schema":{"description":"schema-canary"}}},"service_tier":"auto","metadata":{"label":"request-metadata-canary"},"safety_identifier":"safety-id-canary","prompt_cache_key":"cache-id-canary","user":"user-id-canary"}"#)
}
fn response(observer: &mut ExecutionCapture, content_type: &'static str) {
@@ -166,6 +166,10 @@ async fn usage_mode_excludes_content_and_context_credentials() {
for secret in [
"prompt-canary",
"schema-canary",
+ "request-metadata-canary",
+ "safety-id-canary",
+ "cache-id-canary",
+ "user-id-canary",
"output-canary",
"provider-secret",
"private-ingestion-token",
@@ -175,6 +179,129 @@ async fn usage_mode_excludes_content_and_context_credentials() {
}
}
+#[tokio::test]
+async fn request_configuration_is_projected_out_of_input() {
+ let request = json!({
+ "model": "requested", "input": "hello", "instructions": "be brief",
+ "tools": [{"type": "function", "name": "weather"}],
+ "previous_response_id": "resp-previous", "future_field": {"keep": true},
+ "temperature": 0.5, "stream": true, "parallel_tool_calls": false,
+ "reasoning": {"effort": "high"}, "text": {"format": {"type": "json_object"}},
+ "stream_options": {"include_obfuscation": false},
+ "metadata": {"purpose": "test"}, "prompt_cache_key": "cache-key"
+ });
+ let capture = OpenAiResponsesCapture::new(
+ &HeaderMap::new(),
+ request.to_string().as_bytes(),
+ IngestionMode::Full,
+ )
+ .into_facts();
+ assert_eq!(
+ capture.input.unwrap(),
+ json!({
+ "input": "hello", "instructions": "be brief",
+ "tools": [{"type": "function", "name": "weather"}],
+ "previous_response_id": "resp-previous", "future_field": {"keep": true}
+ })
+ );
+ assert_eq!(capture.model_parameters["stream"], true);
+ assert_eq!(capture.model_parameters["parallel_tool_calls"], false);
+ assert_eq!(capture.model_parameters["reasoning"], request["reasoning"]);
+ assert_eq!(capture.model_parameters["text"], request["text"]);
+ assert_eq!(
+ capture.model_parameters["stream_options"],
+ request["stream_options"]
+ );
+ assert_eq!(capture.request_metadata["metadata"], request["metadata"]);
+ assert_eq!(capture.request_metadata["prompt_cache_key"], "cache-key");
+}
+
+#[tokio::test]
+async fn completion_time_requires_sse_generated_content_in_either_ingestion_mode() {
+ for mode in ["full", "usage"] {
+ for (kind, field) in [
+ ("response.output_text.delta", "delta"),
+ ("response.function_call_arguments.delta", "delta"),
+ ("response.reasoning_summary_text.delta", "delta"),
+ ("response.custom_tool_call_input.delta", "delta"),
+ ("response.audio.delta", "delta"),
+ ("response.audio.transcript.delta", "delta"),
+ ("response.shell_call_command.delta", "delta"),
+ (
+ "response.image_generation_call.partial_image",
+ "partial_image_b64",
+ ),
+ ] {
+ let mut observer = observer(mode).await;
+ response(&mut observer, "text/event-stream; charset=utf-8");
+ observer.bytes(b": keepalive\n\n");
+ observer.bytes(
+ event(
+ "response.created",
+ json!({"response":{"status":"in_progress"}}),
+ )
+ .as_bytes(),
+ );
+ observer.bytes(event(kind, json!({field:""})).as_bytes());
+ assert!(observer.first_byte_ms.is_some());
+ assert!(observer.completion_start_ms.is_none());
+ let delta = event(kind, json!({field:"hello"}));
+ let split = delta.len() - 1;
+ observer.bytes(&delta.as_bytes()[..split]);
+ assert!(observer.completion_start_ms.is_none());
+ observer.bytes(&delta.as_bytes()[split..]);
+ let first = observer.completion_start_ms.unwrap();
+ observer.bytes(event(kind, json!({field:"later"})).as_bytes());
+ assert_eq!(observer.completion_start_ms, Some(first));
+ }
+ // The upstream content type controls timing even if the request asks to stream.
+ let context = resolved_request_context_with_mode("provider-secret", mode).await;
+ let mut observer = ExecutionCapture::openai_responses(
+ &context,
+ &HeaderMap::new(),
+ br#"{"stream":true,"input":"hello"}"#,
+ );
+ response(&mut observer, "application/json; charset=utf-8");
+ observer.bytes(br#"{"output":[],"status":"completed"}"#);
+ observer.end_body();
+ assert!(observer.first_byte_ms.is_some());
+ assert!(observer.completion_start_ms.is_none());
+ }
+}
+
+#[tokio::test]
+async fn provider_error_details_are_bounded_and_full_mode_only() {
+ for mode in ["full", "usage"] {
+ for streaming in [false, true] {
+ let mut observer = observer(mode).await;
+ response(
+ &mut observer,
+ if streaming {
+ "text/event-stream"
+ } else {
+ "application/json"
+ },
+ );
+ let error = json!({"code":"rate_limit_exceeded","message":"é".repeat(MAX_FACT_STRING)});
+ let body = if streaming {
+ event("error", error)
+ } else {
+ json!({"error":error}).to_string()
+ };
+ observer.bytes(body.as_bytes());
+ observer.end_body();
+ let facts = &captured(&observer).facts;
+ if mode == "full" {
+ let message = facts.error_message.as_ref().unwrap();
+ assert!(message.starts_with("rate_limit_exceeded: é"));
+ assert!(message.len() <= MAX_FACT_STRING);
+ } else {
+ assert!(facts.error_message.is_none());
+ }
+ }
+ }
+}
+
#[tokio::test]
async fn oversized_and_malformed_events_do_not_prevent_later_facts() {
let mut observer = observer("full").await;
@@ -577,11 +704,15 @@ async fn client_compression_preferences_do_not_disable_capture() {
assert_eq!(collector.calls(), 1);
let upload = uploaded.lock().unwrap();
let metadata = uploaded_attribute(&upload, "langfuse.observation.metadata");
- assert_eq!(metadata["capture_complete"], true);
- assert_eq!(metadata["output_complete"], true);
- assert_eq!(metadata["provider_response_id"], "resp-1");
- assert_eq!(metadata["provider_status"], "completed");
- assert!(metadata["native_usage"]["input_tokens"].is_number());
+ assert!(metadata.get("capture_complete").is_none());
+ assert!(metadata.get("output_complete").is_none());
+ assert!(metadata.get("provider_status").is_none());
+ assert!(metadata.get("native_usage").is_none());
+ assert_eq!(metadata["langfuse.gateway.provider.response_id"], "resp-1");
+ assert!(
+ uploaded_attribute(&upload, "langfuse.observation.usage_details")["input_tokens"]
+ .is_number()
+ );
assert_eq!(
uploaded_attribute(&upload, "langfuse.observation.output")[0]["content"][0]["text"],
"captured output"
@@ -653,9 +784,15 @@ async fn capture_regression_preserves_native_request_and_usage() {
execution.bytes(body.as_bytes());
execution.end_body();
let facts = &captured(&execution).facts;
+ let mut expected_input = request.clone();
+ expected_input.as_object_mut().unwrap().remove("model");
assert_eq!(
facts.input.as_ref(),
- if mode == "full" { Some(&request) } else { None }
+ if mode == "full" {
+ Some(&expected_input)
+ } else {
+ None
+ }
);
assert_eq!(facts.usage_details.as_ref(), Some(&usage));
assert!(facts.capture_complete);
diff --git a/ai-gateway/src/providers/openai/tests.rs b/ai-gateway/src/providers/openai/tests.rs
index 7cbb25a600..4bf74c65fc 100644
--- a/ai-gateway/src/providers/openai/tests.rs
+++ b/ai-gateway/src/providers/openai/tests.rs
@@ -348,7 +348,7 @@ async fn completed_json_and_sse_upload_once_without_waiting_for_ingestion() {
),
(
"text/event-stream",
- "data: {\"type\":\"response.completed\",\"response\":{\"id\":\"resp-1\",\"model\":\"actual\",\"status\":\"completed\",\"output\":[],\"usage\":{\"input_tokens\":2,\"output_tokens\":1,\"total_tokens\":3}}}\n\n",
+ "data: {\"type\":\"response.output_text.delta\",\"delta\":\"hello\"}\n\ndata: {\"type\":\"response.completed\",\"response\":{\"id\":\"resp-1\",\"model\":\"actual\",\"status\":\"completed\",\"output\":[],\"usage\":{\"input_tokens\":2,\"output_tokens\":1,\"total_tokens\":3}}}\n\n",
503,
),
] {
@@ -404,7 +404,7 @@ async fn completed_json_and_sse_upload_once_without_waiting_for_ingestion() {
.await
.unwrap()
.unwrap();
- assert_completed_upload(&payload);
+ assert_completed_upload(&payload, content_type == "text/event-stream");
release.notify_one();
telemetry
.shutdown(Instant::now() + Duration::from_secs(1))
@@ -488,9 +488,13 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rel
)
.unwrap();
assert_eq!(
- metadata["relay_outcome"],
- if cancelled { "cancelled" } else { "timeout" }
+ attrs
+ .iter()
+ .find(|a| a["key"] == "langfuse.observation.level")
+ .unwrap()["value"]["stringValue"],
+ if cancelled { "WARNING" } else { "ERROR" }
);
+ assert!(metadata.get("relay_outcome").is_none());
telemetry
.shutdown(Instant::now() + Duration::from_secs(1))
.await;
@@ -499,7 +503,7 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rel
}
}
-fn assert_completed_upload(payload: &serde_json::Value) {
+fn assert_completed_upload(payload: &serde_json::Value, streaming: bool) {
use serde_json::{Value, json};
let span = &payload["resourceSpans"][0]["scopeSpans"][0]["spans"][0];
let attrs = span["attributes"].as_array().unwrap();
@@ -509,10 +513,11 @@ fn assert_completed_upload(payload: &serde_json::Value) {
.any(|a| a["key"] == "langfuse.observation.model.name"
&& a["value"]["stringValue"] == "actual")
);
- assert!(
+ assert_eq!(
attrs
.iter()
- .any(|a| a["key"] == "langfuse.observation.completion_start_time")
+ .any(|a| a["key"] == "langfuse.observation.completion_start_time"),
+ streaming,
);
let metadata: Value = serde_json::from_str(
attrs
@@ -523,9 +528,20 @@ fn assert_completed_upload(payload: &serde_json::Value) {
.unwrap(),
)
.unwrap();
- assert_eq!(metadata["relay_outcome"], "eof");
+ assert!(metadata.get("relay_outcome").is_none());
+ assert!(metadata.get("native_usage").is_none());
+ assert_eq!(metadata["langfuse.gateway.provider.response_id"], "resp-1");
+ let usage: Value = serde_json::from_str(
+ attrs
+ .iter()
+ .find(|a| a["key"] == "langfuse.observation.usage_details")
+ .unwrap()["value"]["stringValue"]
+ .as_str()
+ .unwrap(),
+ )
+ .unwrap();
assert_eq!(
- metadata["native_usage"],
+ usage,
json!({"input_tokens":2,"output_tokens":1,"total_tokens":3})
);
for secret in [
diff --git a/ai-gateway/src/telemetry/mapping.rs b/ai-gateway/src/telemetry/mapping.rs
index 9b61012e3a..7966dcec9b 100644
--- a/ai-gateway/src/telemetry/mapping.rs
+++ b/ai-gateway/src/telemetry/mapping.rs
@@ -7,27 +7,8 @@ use crate::capture::{InferenceFacts, RelayOutcome};
pub(super) fn span(facts: InferenceFacts, trace_id: &str, observation_id: &str) -> Value {
let full = facts.metadata.get("ingestion_mode").and_then(Value::as_str) == Some("full");
let (level, message) = observation_status(&facts);
+ let metadata = generation_metadata(&facts);
let inference = facts.inference;
- let mut metadata = facts.metadata.as_object().cloned().unwrap_or_default();
- metadata.extend([
- ("api_format".into(), json!(facts.api_format)),
- ("relay_outcome".into(), json!(facts.outcome)),
- ("http_status".into(), json!(facts.http_status)),
- ("first_byte_ms".into(), json!(facts.first_byte_ms)),
- ("requested_model".into(), json!(inference.requested_model)),
- (
- "provider_response_id".into(),
- json!(inference.provider_response_id),
- ),
- (
- "provider_request_id".into(),
- json!(inference.provider_request_id),
- ),
- ("provider_status".into(), json!(inference.provider_status)),
- ("input_complete".into(), json!(inference.input_complete)),
- ("output_complete".into(), json!(inference.output_complete)),
- ("capture_complete".into(), json!(inference.capture_complete)),
- ]);
let mut attributes = vec![attribute("langfuse.observation.type", "generation")];
if let Some(model) = inference.model.or(inference.requested_model) {
attributes.push(attribute("langfuse.observation.model.name", model));
@@ -38,14 +19,13 @@ pub(super) fn span(facts: InferenceFacts, trace_id: &str, observation_id: &str)
Value::Object(inference.model_parameters).to_string(),
));
}
- if let Some(usage) = inference.usage_details {
- if let Some(projected) = openai_usage(&usage) {
- attributes.push(attribute(
- "langfuse.observation.usage_details",
- projected.to_string(),
- ));
- }
- metadata.insert("native_usage".into(), usage);
+ if let Some(usage) = inference.usage_details
+ && let Some(projected) = openai_usage(&usage)
+ {
+ attributes.push(attribute(
+ "langfuse.observation.usage_details",
+ projected.to_string(),
+ ));
}
if full {
if let Some(input) = inference.input {
@@ -56,7 +36,7 @@ pub(super) fn span(facts: InferenceFacts, trace_id: &str, observation_id: &str)
}
}
if let Some(completion_start) = facts
- .first_byte_ms
+ .completion_start_ms
.and_then(|elapsed| facts.start_time_unix_ms.checked_add(elapsed))
.and_then(|timestamp| i64::try_from(timestamp).ok())
.and_then(DateTime::from_timestamp_millis)
@@ -86,24 +66,91 @@ pub(super) fn span(facts: InferenceFacts, trace_id: &str, observation_id: &str)
})
}
-fn observation_status(facts: &InferenceFacts) -> (&'static str, Option<&'static str>) {
- let failure = if facts.http_status.is_some_and(|status| status >= 400) {
- Some("Provider HTTP error")
+fn generation_metadata(facts: &InferenceFacts) -> Map<String, Value> {
+ let mut metadata = Map::new();
+ if let Some(attribution) = facts
+ .metadata
+ .get("key_metadata")
+ .and_then(Value::as_object)
+ {
+ for (key, value) in attribution {
+ // Attribution remains searchable, but cannot impersonate gateway or OTEL fields.
+ if ![
+ "langfuse.gateway",
+ "http_status",
+ "scope",
+ "resourceAttributes",
+ "attributes",
+ ]
+ .iter()
+ .any(|reserved| key == reserved || key.starts_with(&format!("{reserved}.")))
+ {
+ metadata.insert(key.clone(), value.clone());
+ }
+ metadata.insert(
+ format!("langfuse.gateway.api-key.metadata.{key}"),
+ value.clone(),
+ );
+ }
+ }
+ for (source, target) in [
+ ("project_id", "project_id"),
+ ("organization_id", "organization_id"),
+ ("ingestion_mode", "ingestion_mode"),
+ ("key_id", "api-key.id"),
+ ("provider_connection_id", "provider.connection_id"),
+ ] {
+ if let Some(value) = facts.metadata.get(source) {
+ metadata.insert(format!("langfuse.gateway.{target}"), value.clone());
+ }
+ }
+ metadata.insert(
+ "langfuse.gateway.api_format".into(),
+ json!(facts.api_format),
+ );
+ metadata.insert("http_status".into(), json!(facts.http_status));
+ for (key, value) in [
+ ("response_id", &facts.inference.provider_response_id),
+ ("request_id", &facts.inference.provider_request_id),
+ ] {
+ if let Some(value) = value {
+ metadata.insert(format!("langfuse.gateway.provider.{key}"), json!(value));
+ }
+ }
+ for (key, value) in &facts.inference.request_metadata {
+ metadata.insert(
+ format!("langfuse.gateway.provider.request.{key}"),
+ value.clone(),
+ );
+ }
+ metadata
+}
+
+fn observation_status(facts: &InferenceFacts) -> (&'static str, Option<String>) {
+ let failure = if let Some(status) = facts.http_status.filter(|status| *status >= 400) {
+ Some(format!("Provider HTTP error ({status})"))
} else if facts.inference.provider_status.as_deref() == Some("failed") {
- Some("Provider response failed")
+ Some("Provider response failed".into())
} else {
match facts.outcome {
- RelayOutcome::Timeout => Some("Provider response timed out"),
- RelayOutcome::TransportError => Some("Provider transport error"),
+ RelayOutcome::Timeout => Some("Provider response timed out".into()),
+ RelayOutcome::TransportError => Some("Provider transport error".into()),
RelayOutcome::Eof | RelayOutcome::Cancelled => None,
}
};
- if let Some(message) = failure {
+ if let Some(mut message) = failure {
+ if let Some(status) = facts.http_status.filter(|status| *status < 400) {
+ message = format!("{message} (HTTP {status})");
+ }
+ if let Some(error) = &facts.inference.error_message {
+ message.push_str(": ");
+ message.push_str(error);
+ }
("ERROR", Some(message))
} else if facts.outcome == RelayOutcome::Cancelled {
- ("WARNING", Some("Client cancelled the response"))
+ ("WARNING", Some("Client cancelled the response".into()))
} else if facts.inference.provider_status.as_deref() == Some("incomplete") {
- ("WARNING", Some("Provider response incomplete"))
+ ("WARNING", Some("Provider response incomplete".into()))
} else {
("DEFAULT", None)
}
@@ -114,7 +161,7 @@ fn attribute(key: &str, value: impl Into<String>) -> Value {
}
/// The receiver's native `OpenAI` usage schema is strict at the top level, but
-/// accepts new numeric detail counters. Keep the original payload in metadata.
+/// accepts new numeric detail counters.
fn openai_usage(usage: &Value) -> Option<Value> {
let usage = usage.as_object()?;
let mut projected = Map::new();
@@ -154,7 +201,8 @@ mod tests {
api_format: "openai.responses",
start_time_unix_ms: 1_735_689_600_000,
duration_ms: 900,
- first_byte_ms: Some(125),
+ first_byte_ms: Some(50),
+ completion_start_ms: Some(125),
http_status: Some(200),
metadata: json!({"ingestion_mode": "full", "project_id": "project", "key_id": "key", "key_metadata": {"team": "search"}}),
outcome: RelayOutcome::Eof,
@@ -198,7 +246,7 @@ mod tests {
}
#[test]
- fn full_generation_preserves_native_content_attribution_and_first_byte_time() {
+ fn full_generation_preserves_native_content_attribution_and_completion_time() {
let span = span(
facts(),
"0123456789abcdef0123456789abcdef",
@@ -223,10 +271,75 @@ mod tests {
);
assert!(attrs.contains_key("langfuse.observation.output"));
let metadata = metadata(&attrs);
- assert_eq!(metadata["project_id"], "project");
- assert_eq!(metadata["key_metadata"]["team"], "search");
- assert_eq!(metadata["requested_model"], "requested-model");
- assert_eq!(metadata["capture_complete"], true);
+ assert_eq!(
+ metadata,
+ json!({
+ "http_status": 200,
+ "team": "search",
+ "langfuse.gateway.api-key.metadata.team": "search",
+ "langfuse.gateway.api-key.id": "key",
+ "langfuse.gateway.project_id": "project",
+ "langfuse.gateway.ingestion_mode": "full",
+ "langfuse.gateway.api_format": "openai.responses",
+ })
+ );
+ }
+
+ #[test]
+ fn attribution_is_searchable_without_overwriting_gateway_or_otel_metadata() {
+ let mut facts = facts();
+ facts.metadata["organization_id"] = json!("org");
+ facts.metadata["provider_connection_id"] = json!("connection");
+ facts.metadata["key_metadata"] = json!({
+ "team": "search", "enabled": true, "cost_center": 42,
+ "http_status": 500,
+ "langfuse.gateway.api-key.id": "spoofed-key",
+ "langfuse.gateway.provider.request_id": "spoofed-request",
+ "langfuse.gateway.future_field": "spoofed-future",
+ "langfuse.gateway.api-key.metadata.team": "spoofed-team",
+ "scope": "spoofed-scope", "scope.name": "spoofed-scope-name",
+ "resourceAttributes": "spoofed-resource",
+ "attributes": "spoofed-attributes"
+ });
+ facts.inference.provider_response_id = Some("response".into());
+ let attrs = attributes(&span(facts, "trace", "span"));
+ let metadata = metadata(&attrs);
+ assert_eq!(metadata["team"], "search");
+ assert_eq!(metadata["enabled"], true);
+ assert_eq!(metadata["cost_center"], 42);
+ assert_eq!(metadata["langfuse.gateway.api-key.metadata.team"], "search");
+ assert_eq!(
+ metadata["langfuse.gateway.api-key.metadata.http_status"],
+ 500
+ );
+ assert_eq!(metadata["http_status"], 200);
+ assert_eq!(metadata["langfuse.gateway.api-key.id"], "key");
+ assert_eq!(
+ metadata["langfuse.gateway.provider.connection_id"],
+ "connection"
+ );
+ assert_eq!(
+ metadata["langfuse.gateway.provider.response_id"],
+ "response"
+ );
+ assert_eq!(metadata["langfuse.gateway.organization_id"], "org");
+ for key in [
+ "scope",
+ "scope.name",
+ "resourceAttributes",
+ "attributes",
+ "langfuse.gateway.future_field",
+ ] {
+ assert!(metadata.get(key).is_none(), "{key}");
+ }
+ assert_ne!(
+ metadata["langfuse.gateway.provider.request_id"],
+ "spoofed-request"
+ );
+ assert_eq!(
+ metadata["langfuse.gateway.api-key.metadata.langfuse.gateway.api-key.metadata.team"],
+ "spoofed-team"
+ );
}
#[test]
@@ -234,7 +347,7 @@ mod tests {
let mut facts = facts();
facts.metadata["ingestion_mode"] = json!("usage");
facts.inference.model = None;
- facts.first_byte_ms = None;
+ facts.completion_start_ms = None;
let attrs = attributes(&span(facts, "trace", "span"));
for key in ["input", "output", "usage_details", "completion_start_time"] {
assert!(!attrs.contains_key(&format!("langfuse.observation.{key}")));
@@ -254,7 +367,7 @@ mod tests {
});
facts.inference.usage_details = Some(usage.clone());
let attrs = attributes(&span(facts, "trace", "span"));
- assert_eq!(metadata(&attrs)["native_usage"], usage);
+ assert!(metadata(&attrs).get("native_usage").is_none());
let projected: Value = serde_json::from_str(
attrs["langfuse.observation.usage_details"]
.as_str()
@@ -272,12 +385,72 @@ mod tests {
}
#[test]
- fn unsupported_usage_remains_metadata_without_invented_totals() {
+ fn unsupported_usage_omits_usage_without_invented_totals() {
let mut facts = facts();
facts.inference.usage_details = Some(json!({"input_tokens": 9}));
let attrs = attributes(&span(facts, "trace", "span"));
assert!(!attrs.contains_key("langfuse.observation.usage_details"));
- assert_eq!(metadata(&attrs)["native_usage"], json!({"input_tokens": 9}));
+ assert!(metadata(&attrs).get("native_usage").is_none());
+ }
+
+ #[test]
+ fn provider_request_metadata_is_namespaced_and_keeps_native_types() {
+ let mut facts = facts();
+ facts.inference.request_metadata = serde_json::from_value(json!({
+ "metadata": {"customer": "customer-1"},
+ "prompt_cache_key": "cache-key",
+ "safety_identifier": "safety-id",
+ "user": "legacy-user"
+ }))
+ .unwrap();
+ let attrs = attributes(&span(facts, "trace", "span"));
+ let metadata = metadata(&attrs);
+ assert_eq!(
+ metadata["langfuse.gateway.provider.request.metadata"],
+ json!({"customer": "customer-1"})
+ );
+ assert_eq!(
+ metadata["langfuse.gateway.provider.request.prompt_cache_key"],
+ "cache-key"
+ );
+ assert_eq!(
+ metadata["langfuse.gateway.provider.request.safety_identifier"],
+ "safety-id"
+ );
+ assert_eq!(
+ metadata["langfuse.gateway.provider.request.user"],
+ "legacy-user"
+ );
+ for key in ["metadata", "prompt_cache_key", "safety_identifier", "user"] {
+ assert!(metadata.get(key).is_none());
+ }
+ }
+
+ #[test]
+ fn provider_errors_include_http_status_and_available_details() {
+ for (status, provider_status, error, expected) in [
+ (429, None, None, "Provider HTTP error (429)"),
+ (
+ 429,
+ None,
+ Some("rate_limit_exceeded: Too many requests"),
+ "Provider HTTP error (429): rate_limit_exceeded: Too many requests",
+ ),
+ (
+ 200,
+ Some("failed"),
+ Some("server_error: Please retry"),
+ "Provider response failed (HTTP 200): server_error: Please retry",
+ ),
+ ] {
+ let mut facts = facts();
+ facts.http_status = Some(status);
+ facts.inference.provider_status = provider_status.map(str::to_owned);
+ facts.inference.error_message = error.map(str::to_owned);
+ let attrs = attributes(&span(facts, "trace", "span"));
+ assert_eq!(attrs["langfuse.observation.status_message"], expected);
+ assert_eq!(attrs["langfuse.observation.level"], "ERROR");
+ }
}
#[test]
@@ -304,7 +477,7 @@ mod tests {
let attrs = attributes(&span);
assert_eq!(attrs["langfuse.observation.level"], level);
assert_eq!(span["status"]["code"], code);
- assert_eq!(metadata(&attrs)["relay_outcome"], json!(outcome));
+ assert!(metadata(&attrs).get("relay_outcome").is_none());
}
}
}
diff --git a/ai-gateway/src/telemetry/tests.rs b/ai-gateway/src/telemetry/tests.rs
index 2b1bd50c3a..116c2788e5 100644
--- a/ai-gateway/src/telemetry/tests.rs
+++ b/ai-gateway/src/telemetry/tests.rs
@@ -20,6 +20,7 @@ fn facts(project: &str) -> InferenceFacts {
start_time_unix_ms: 1_735_689_600_000,
duration_ms: 100,
first_byte_ms: Some(10),
+ completion_start_ms: None,
http_status: Some(200),
metadata: json!({"project_id": project, "ingestion_mode": "usage"}),
outcome: RelayOutcome::Eof,
@@ -120,7 +121,10 @@ async fn concurrent_projects_keep_their_original_grants_and_attribution() {
serde_json::from_str(metadata["value"]["stringValue"].as_str().unwrap()).unwrap();
received.lock().unwrap().push((
authorization,
- metadata["project_id"].as_str().unwrap().to_owned(),
+ metadata["langfuse.gateway.project_id"]
+ .as_str()
+ .unwrap()
+ .to_owned(),
));
response(200, "{}")
}
diff --git a/packages/shared/src/server/otel/OtelIngestionProcessor.metadataDropped.test.ts b/packages/shared/src/server/otel/OtelIngestionProcessor.metadataDropped.test.ts
index e8b91d8770..b833b8a806 100644
--- a/packages/shared/src/server/otel/OtelIngestionProcessor.metadataDropped.test.ts
+++ b/packages/shared/src/server/otel/OtelIngestionProcessor.metadataDropped.test.ts
@@ -119,6 +119,99 @@ const expectDropTags = (
expect(tags?.sdkVersion).toBe("3.8.1");
};
+describe("gateway metadata", () => {
+ it.each([
+ ["v3", "langfuse-ai-gateway"],
+ ["v4", "langfuse-ai-gateway"],
+ ["v3", "other-instrumentation"],
+ ["v4", "other-instrumentation"],
+ ])(
+ "preserves canonical fields and metadata for %s %s",
+ async (path, scope) => {
+ const completionStartTime = "2025-07-13T05:20:00.500Z";
+ const modelParameters = {
+ service_tier: "default",
+ stream: true,
+ reasoning: { effort: "low" },
+ };
+ const usageDetails = { input: 10, output: 21 };
+ const canonicalAttributes = {
+ "langfuse.observation.type": "generation",
+ "langfuse.observation.level": "ERROR",
+ "langfuse.observation.status_message": "HTTP 429: rate limited",
+ "langfuse.observation.model.name": "test-model",
+ "langfuse.observation.model.parameters":
+ JSON.stringify(modelParameters),
+ "langfuse.observation.usage_details": JSON.stringify(usageDetails),
+ "langfuse.observation.cost_details": JSON.stringify({ total: 0.001 }),
+ "langfuse.observation.completion_start_time": completionStartTime,
+ };
+ const batch = buildBatch(
+ Object.entries({
+ ...canonicalAttributes,
+ "langfuse.observation.input": '[{"role":"user","content":"Hi"}]',
+ "langfuse.observation.output": '[{"type":"message","content":[]}]',
+ "langfuse.observation.metadata": JSON.stringify({
+ "langfuse.gateway.provider.request_id": "req-test",
+ }),
+ "langfuse.observation.metadata.langfuse.gateway.api-key.id":
+ "key-test",
+ "custom.attribute": "keep-custom",
+ "langfuse.observation.custom": "keep-unknown",
+ }).map(([key, value]) => ({ key, value: { stringValue: value } })),
+ );
+ batch[0].scopeSpans![0].scope!.name = scope;
+ const processor = createProcessor();
+ const observation =
+ path === "v4"
+ ? processor.processToEvent(batch)[0]
+ : (await processor.processToIngestionEvents(batch)).find(
+ (event) => event.type === "generation-create",
+ )?.body;
+
+ expect(observation).toMatchObject({
+ level: "ERROR",
+ statusMessage: "HTTP 429: rate limited",
+ modelParameters: {
+ service_tier: "default",
+ stream: "true",
+ reasoning: '{"effort":"low"}',
+ },
+ completionStartTime,
+ input: '[{"role":"user","content":"Hi"}]',
+ output: '[{"type":"message","content":[]}]',
+ ...(path === "v4"
+ ? {
+ type: "GENERATION",
+ modelName: "test-model",
+ providedUsageDetails: usageDetails,
+ providedCostDetails: { total: 0.001 },
+ }
+ : {
+ model: "test-model",
+ usageDetails,
+ costDetails: { total: 0.001 },
+ }),
+ });
+ expect(observation?.metadata).toEqual({
+ "langfuse.gateway.provider.request_id": "req-test",
+ "langfuse.gateway.api-key.id": "key-test",
+ attributes: {
+ ...(scope === "langfuse-ai-gateway" ? {} : canonicalAttributes),
+ "custom.attribute": "keep-custom",
+ "langfuse.observation.custom": "keep-unknown",
+ },
+ resourceAttributes: { "service.name": "test-svc" },
+ scope: {
+ name: scope,
+ version: "3.8.1",
+ attributes: { public_key: "pk-test" },
+ },
+ });
+ },
+ );
+});
+
describe("OTel metadata_dropped metric", () => {
beforeEach(() => {
recordIncrementMock.mockClear();
diff --git a/packages/shared/src/server/otel/OtelIngestionProcessor.ts b/packages/shared/src/server/otel/OtelIngestionProcessor.ts
index 55c7cd5ced..3530e37795 100644
--- a/packages/shared/src/server/otel/OtelIngestionProcessor.ts
+++ b/packages/shared/src/server/otel/OtelIngestionProcessor.ts
@@ -1887,6 +1887,16 @@ export class OtelIngestionProcessor {
delete rawFilteredAttributes[key];
});
+ // Gateway observation attributes are represented by canonical fields.
+ // Keep unknown attributes available for diagnostics.
+ if (instrumentationScopeName === "langfuse-ai-gateway") {
+ for (const key of Object.values(LangfuseOtelSpanAttributes)) {
+ if (key.startsWith("langfuse.observation.")) {
+ delete rawFilteredAttributes[key];
+ }
+ }
+ }
+
// Delete gen_ai.prompt.*, gen_ai.completion.*, llm.input_messages.*, llm.output_messages.*,
// and metadata blob keys (already extracted into top-level metadata by extractMetadata())
Object.keys(attributes).forEach((key) => {