🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
feat(ai-gateway): record model, key owner, byte and upstream error facts on generations (#19161)
Hassieb Pakzada4348c9d2447
diff --git a/ai-gateway/README.md b/ai-gateway/README.mdindex b0e5d296d6..87928811a2 100644--- a/ai-gateway/README.md+++ b/ai-gateway/README.md@@ -2,10 +2,10 @@A standalone Rust gateway for native provider APIs under two namespaces:-| Namespace | Endpoints | Generation ingested |-| --- | --- | --- |-| `/openai/v1` | `POST /responses`, `POST /responses/compact`, `POST /chat/completions`, `GET /models` | Responses, compact and Chat Completions |-| `/anthropic/v1` | `POST /messages`, `POST /messages/count_tokens`, `GET /models` | Messages |+| Namespace | Endpoints | Generation ingested |+| --------------- | ------------------------------------------------------------------------------------- | --------------------------------------- |+| `/openai/v1` | `POST /responses`, `POST /responses/compact`, `POST /chat/completions`, `GET /models` | Responses, compact and Chat Completions |+| `/anthropic/v1` | `POST /messages`, `POST /messages/count_tokens`, `GET /models` | Messages |Web resolves the gateway key to a trusted provider connection for the namespace's APIformat; Rust relays native JSON or SSE without rewriting provider bytes, except that a@@ -121,13 +121,13 @@ top-level `request_id`; see [Request IDs](#request-ids).sanitized capture details. Other log events do not automatically include trace IDs. Dependency logs stay at `warn`. Log severity does not controltrace sampling. `trace` is not an accepted gateway log level.-| Variable | Default | Purpose |-| --- | --- | --- |-| `OTEL_EXPORTER_OTLP_ENDPOINT` | unset | Enables operational traces and metrics; HTTP/protobuf base URL, e.g. `http://localhost:4318` |-| `OTEL_SERVICE_NAME` | `ai-gateway` | Operational service name |-| `DD_ENV` | `development` | Deployment environment resource attribute |-| `BUILD_ID` | Cargo package version | Service version; set to the deployed image's commit SHA |-| `OTEL_TRACES_SAMPLER_ARG` | `1` | Sampling ratio from 0 to 1 for independent operational root traces |+| Variable | Default | Purpose |+| ----------------------------- | --------------------- | -------------------------------------------------------------------------------------------- |+| `OTEL_EXPORTER_OTLP_ENDPOINT` | unset | Enables operational traces and metrics; HTTP/protobuf base URL, e.g. `http://localhost:4318` |+| `OTEL_SERVICE_NAME` | `ai-gateway` | Operational service name |+| `DD_ENV` | `development` | Deployment environment resource attribute |+| `BUILD_ID` | Cargo package version | Service version; set to the deployed image's commit SHA |+| `OTEL_TRACES_SAMPLER_ARG` | `1` | Sampling ratio from 0 to 1 for independent operational root traces |Tower's `TraceLayer` keeps the server span alive through the response body.The HTTP response log and `http.server.request.duration` measure time to response@@ -145,14 +145,14 @@ Every inference request gets a fresh UUIDv7 gateway request ID (time-ordered, noauthentication, so it exists for every response, including rejections beforeresolution, 4xx/5xx gateway errors and streams. It appears in:-| Where | Field |-| --- | --- |-| Every response | `langfuse-request-id` header, sent with the initial headers of a stream |-| Gateway JSON error bodies | Top-level `request_id` in both native envelopes |-| Gateway JSON log lines for the request | Top-level `request_id` |-| Operational server span | `gateway.request.id` |-| Web resolution call | `langfuse-gateway-request-id` header; Web adds it to its log context |-| Langfuse generation | `langfuse.gateway.request.id` metadata |+| Where | Field |+| -------------------------------------- | ----------------------------------------------------------------------- |+| Every response | `langfuse-request-id` header, sent with the initial headers of a stream |+| Gateway JSON error bodies | Top-level `request_id` in both native envelopes |+| Gateway JSON log lines for the request | Top-level `request_id` |+| Operational server span | `gateway.request.id` |+| Web resolution call | `langfuse-gateway-request-id` header; Web adds it to its log context |+| Langfuse generation | `langfuse.gateway.request.id` metadata |A caller's `x-request-id` never becomes the gateway request ID. A single printablevalue of at most 256 bytes is recorded separately as the server span's@@ -181,14 +181,14 @@ sent upstream, the provider connection it was sent to. Requests rejected beforeresolution, including invalid or oversized bodies, carry none of them. Provider failuresbefore response headers (502/504) still name the connection that was tried.-| Response header | Server span field | JSON log key | Value |-| --- | --- | --- | --- |-| `langfuse-organization-id` | `langfuse.organization.id` | `organization_id` | Organization of the gateway key |-| `langfuse-project-id` | `langfuse.project.id` | `project_id` | Ingestion project |-| `langfuse-provider` | `gateway.provider` | `provider` | Stable lowercase provider type, `openai` or `anthropic` |-| `langfuse-provider-connection-id` | `gateway.provider.connection.id` | `provider_connection_id` | Opaque provider connection ID |-| — | `gateway.provider.connection.name` | `provider_connection_name` | The connection's user-chosen name; never sent to callers |-| — | `gateway.upstream.model` | `upstream_model` | The request's `model`, which the relay forwards unchanged; never sent to callers |+| Response header | Server span field | JSON log key | Value |+| --------------------------------- | ---------------------------------- | -------------------------- | -------------------------------------------------------------------------------- |+| `langfuse-organization-id` | `langfuse.organization.id` | `organization_id` | Organization of the gateway key |+| `langfuse-project-id` | `langfuse.project.id` | `project_id` | Ingestion project |+| `langfuse-provider` | `gateway.provider` | `provider` | Stable lowercase provider type, `openai` or `anthropic` |+| `langfuse-provider-connection-id` | `gateway.provider.connection.id` | `provider_connection_id` | Opaque provider connection ID |+| — | `gateway.provider.connection.name` | `provider_connection_name` | The connection's user-chosen name; never sent to callers |+| — | `gateway.upstream.model` | `upstream_model` | The request's `model`, which the relay forwards unchanged; never sent to callers |The provider fields describe the latest upstream attempt, so with fallbacks theyname the connection that served the response, or the last one tried. The model@@ -202,13 +202,13 @@ Apart from the bounded `gateway.provider`, they are never metric attributes.Each inference request phase has a child span of the server span so a waterfallhas no unattributed wall time:-| Span | Covers |-| --- | --- |-| `resolution` | The `resolver` Web call; `gateway.outcome` is `admitted` or `resolution_failed` |-| `request.body` | Buffering the caller's request body, mostly upload time; `http.request.body.size` |-| `request.capture` | Parsing the request for inference telemetry; CPU-bound and proportional to `http.request.body.size` |-| `provider.headers` | The provider HTTP send through response headers, including connection setup |-| `provider.stream` | Relaying the provider body to the caller until the relay is finalized; `http.response.body.size`, `gateway.chunks`, `gateway.outcome`, and an error status on timeout or transport failure |+| Span | Covers |+| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |+| `resolution` | The `resolver` Web call; `gateway.outcome` is `admitted` or `resolution_failed` |+| `request.body` | Buffering the caller's request body, mostly upload time; `http.request.body.size` |+| `request.capture` | Parsing the request for inference telemetry; CPU-bound and proportional to `http.request.body.size` |+| `provider.headers` | The provider HTTP send through response headers, including connection setup |+| `provider.stream` | Relaying the provider body to the caller until the relay is finalized; `http.response.body.size`, `gateway.chunks`, `gateway.outcome`, and an error status on timeout or transport failure |Inference-telemetry uploads carry records from many requests, so each runs in itsown trace: a root `telemetry.batch` span with `gateway.telemetry.records` and a span@@ -482,9 +482,8 @@ Ingestion JWTs stay outside serializable capture facts and debug logs. Expired gare dropped without re-resolving the execution. Requests explicitly select v4 ingestion.Each execution becomes one generation with a new observation ID and either the-caller's trace/parent IDs or a generated root trace ID,-actual/requested model, model parameters, native usage, and trusted key/connection-attribution. Full mode includes the captured input/output; usage mode omits content.+caller's trace/parent IDs or a generated root trace ID, the model, model parameters,+native usage, and trusted key/connection attribution. Full mode includes the captured input/output; usage mode omits content.Completion-start time is emitted only for upstream `text/event-stream` responses,on the first nonempty text, reasoning, refusal, tool-input, audio, or partial-image content. JSON responsesand streams without a captured content delta have no completion-start time or TTFT.@@ -494,38 +493,66 @@ plus `total_tokens`), keeping their nested detail counters. Anthropic usage is uit; ingestion decides what to price. Usage is not duplicated in metadata. Missing usageis not reported as zero.-Gateway metadata uses `langfuse.gateway.*`, grouped by the resource or exchange-each field describes:--| Suffix | Meaning |-| ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |-| `organization.id`, `project.id` | Resolved Langfuse organization and ingestion project |-| `api_key.id`, `connection.id` | Authenticated gateway key and selected Langfuse connection |-| `api_key.user.id` | Langfuse user the gateway key belongs to; null for keys created before keys were bound to users |-| `ingestion.mode` | Effective capture mode |-| `request.api_format` | Native API contract: `openai.responses`, `openai.chat-completions` or `anthropic.messages` |-| `request.metadata`, `request.prompt_cache_key`, `request.safety_identifier`, `request.user` | Native caller-supplied fields, captured only in full mode |-| `response.id` | Native response object's ID |-| `response.status_code` | HTTP status returned to the caller |-| `request.id` | Gateway request ID, also returned as the `langfuse-request-id` response header |-| `client.request.id` | Caller's `x-request-id`, when valid |-| `upstream.request.id` | Provider request ID from the upstream `x-request-id` (OpenAI) or `request-id` (Anthropic) header |--The client and upstream request IDs are omitted when unavailable.-API-key attribution entries appear both as top-level metadata and under-`langfuse.gateway.api_key.metadata.*`.-Gateway, agent and OpenTelemetry fields win collisions; the namespaced attribution-copy preserves the original value. `agent.*` and native request field names retain-their existing spelling. Metadata names apply to newly captured generations;-historical traces are not rewritten.--`response.status_code` includes gateway-generated 502/504 errors before upstream-headers arrive. After headers arrive it retains the relayed status, even if a-stream subsequently fails; cancellation before headers leaves the status unknown.-Relay outcome, provider status, completeness flags and first-byte timing remain-internal facts rather than generation metadata. For the gateway scope, ingestion removes-Langfuse attributes already mapped to trace and observation fields, such as user, session,-trace name, tags and environment, while preserving custom attributes, scope and resources.+### Generation fields++The gateway sets these generation fields directly:++| Generation field | OTLP attribute | Value |+| ------------------------------------------------------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| Model (`provided_model_name`) | `langfuse.observation.model.name` | The first known of `model.response`, `model.upstream` and `model.requested` below |+| User ID | `user.id` | The caller's `langfuse-user-id` header or `langfuse_user_id` baggage. Otherwise, only when the gateway starts the trace (no valid `traceparent`), the gateway key owner's user ID. The source is in `user_id_source` |+| Session ID, trace name, environment, tags | `session.id`, `langfuse.trace.name`, `langfuse.environment`, `langfuse.trace.tags` | Caller headers, baggage or a recognized coding agent; see [Caller tracing context](#caller-tracing-context) |+| Level, status message | `langfuse.observation.level`, `langfuse.observation.status_message` | Derived from the HTTP status, provider status and relay outcome |+| Model parameters, usage, input, output, completion start time | `langfuse.observation.*` | See [Response capture](#response-capture) |+| Metadata | `langfuse.observation.metadata` | Caller metadata plus the gateway keys below |++The generation never emits `user.name` or `gen_ai.*` attributes: ingestion reads them as user ID and model fallbacks.++### Generation metadata++Every gateway fact is a metadata key under `langfuse.gateway.`, grouped by the resource or exchange it describes. The table lists the suffix after that prefix. Source is who supplies the value: the gateway itself, Web at resolve time, or the caller. Caller-sourced values are recorded as received, within the stated bounds; the gateway does not vouch for them.++| Key | Type | Source | Present |+| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |+| `organization.id` | string | Web | Always |+| `ingestion.mode` | string: `full` or `usage` | Web | Always |+| `api_key.id` | string | Web | Always |+| `api_key.name` | string | Web: the key's note | When the key has a note |+| `api_key.owner.user_id` | string | Web: the user the key is bound to | When the key is bound to a user; keys created before keys were bound to users have none |+| `api_key.owner.name` | string | Web: the bound user's display name, never the email | When the key is bound to a user with a name |+| `api_key.metadata.<key>` | string, number or boolean | Web: the key's metadata | One entry per key metadata entry |+| `connection.id` | string | Web | Always |+| `connection.name` | string | Web: the connection's display name | When the connection has a name |+| `connection.provider` | string: `openai` or `anthropic` | Web | Always |+| `model.requested` | string | Caller: request body `model` | When the request body is captured JSON with a model |+| `model.upstream` | string | Web: the connection's provider model, sent upstream | When the request reached the provider path |+| `model.response` | string | Provider: response `model` | When the provider reported one |+| `user_id_source` | string: `client` or `api_key_owner` | Gateway | When the generation has a user ID |+| `request.id` | string | Gateway | Always; also the `langfuse-request-id` response header |+| `request.api_format` | string: `openai.responses`, `openai.chat-completions` or `anthropic.messages` | Gateway | Always |+| `request.body_bytes` | integer | Gateway: the caller's uncompressed request body | Always |+| `request.input_omitted` | string: `size_limit`, `content_encoding`, `invalid_json`, `record_limit` or `telemetry_buffer` | Gateway | Full mode, when the input is not recorded |+| `request.input_limit_bytes` | integer | Gateway | With `input_omitted` `size_limit` |+| `client.request.id` | string | Caller: `x-request-id` header | When valid |+| `client.user_agent` | string, at most 256 bytes | Caller: `User-Agent` header, trimmed | When present, single and free of control characters |+| `client.request.metadata` | JSON value as sent | Caller: request body `metadata` | Full mode, when sent |+| `client.request.user` | JSON value as sent | Caller: request body `user` (deprecated OpenAI field) | Full mode, when sent |+| `client.request.safety_identifier` | JSON value as sent | Caller: request body `safety_identifier` | Full mode, when sent |+| `client.request.prompt_cache_key` | JSON value as sent | Caller: request body `prompt_cache_key` | Full mode, when sent |+| `upstream.request.id` | string | Provider: `x-request-id` (OpenAI) or `request-id` (Anthropic) header | When sent |+| `upstream.response.id` | string | Provider: response object `id` | When sent |+| `upstream.status_code` | integer | Provider | When the provider sent response headers |+| `upstream.retry_after_ms` | integer | Provider: `Retry-After` in seconds, converted to milliseconds | When the header is an integer; HTTP dates are ignored |+| `response.status_code` | integer or null | Gateway | Always; null when the caller cancelled before response headers |+| `response.body_bytes` | integer | Gateway: provider body bytes relayed to the caller | When the provider sent response headers |+| `error.origin` | string: `provider` or `provider_transport` | Gateway | When the generation failed |+| `error.type` | string: `provider_rate_limited`, `provider_auth`, `provider_4xx`, `provider_5xx`, `provider_response_failed`, `timeout` or `transport` | Gateway | With `error.origin` |++`error.origin` and `error.type` use the same values as the gateway's operational `error.origin`/`error.reason`, so a generation can be matched to the request metrics. A cancellation is not an error. Byte counts are uncompressed: compressed request bodies are rejected, and the provider is asked for an identity-encoded response.++Key metadata appears only under `api_key.metadata.*`, never as bare top-level metadata keys. Caller metadata, key metadata and the `agent.*` keys cannot write `langfuse.gateway.*`, `scope`, `resourceAttributes` or `attributes` keys; the gateway's own keys always win. `agent.*` keys and native request field names keep their existing spelling. Metadata names apply to newly captured generations; historical traces are not rewritten.++`response.status_code` includes gateway-generated 502/504 errors before upstream headers arrive. After headers arrive it retains the relayed status, even if a stream subsequently fails. Relay outcome, provider status, completeness flags and first-byte timing remain internal facts rather than generation metadata. For the gateway scope, ingestion removes Langfuse attributes already mapped to trace and observation fields, such as user, session, trace name, tags and environment, while preserving custom attributes, scope and resources. Ingestion never estimates usage with a tokenizer for gateway generations; it recognizes them by their gateway ingestion credential, not by the instrumentation scope name, which any OTLP sender can set.Provider HTTP failures and failed SSE responses set the generation level to `ERROR`with the available HTTP status in its status message. Full mode also includes a@@ -569,14 +596,14 @@ Operational trace sampling and best-effort ingestion remain independent.The following optional headers enrich the generation, including in usage mode:-| Header | Format | Python SDK baggage key |-| --- | --- | --- |-| `langfuse-trace-name` | String | `langfuse_trace_name` |-| `langfuse-session-id` | String | `langfuse_session_id` |-| `langfuse-user-id` | String | `langfuse_user_id` |-| `langfuse-environment` | Environment name | `langfuse_environment` |-| `langfuse-tags` | Comma-separated strings | `langfuse_tags` |-| `langfuse-metadata` | Comma-separated `key:value` entries | `langfuse_metadata_<key>` |+| Header | Format | Python SDK baggage key |+| ---------------------- | ----------------------------------- | ------------------------- |+| `langfuse-trace-name` | String | `langfuse_trace_name` |+| `langfuse-session-id` | String | `langfuse_session_id` |+| `langfuse-user-id` | String | `langfuse_user_id` |+| `langfuse-environment` | Environment name | `langfuse_environment` |+| `langfuse-tags` | Comma-separated strings | `langfuse_tags` |+| `langfuse-metadata` | Comma-separated `key:value` entries | `langfuse_metadata_<key>` |For example:@@ -618,12 +645,12 @@ override trace identity or project selection.The gateway also recognizes correlation headers emitted by coding agents:-| Agent | Session source | Turn source |-| --- | --- | --- |-| Claude Code | `x-claude-code-session-id` | Not available |-| Codex | `thread_id` in the Codex turn snapshot | `turn_id` in the Codex turn snapshot |-| OpenCode | `x-opencode-session`, falling back to `x-session-id` or `x-session-affinity` for an OpenCode user agent | `x-opencode-request` |-| Pi | `x-session-id`, `session_id`, `x-session-affinity`, or `x-client-request-id` for a Pi user agent | Not available |+| Agent | Session source | Turn source |+| ----------- | ------------------------------------------------------------------------------------------------------- | ------------------------------------ |+| Claude Code | `x-claude-code-session-id` | Not available |+| Codex | `thread_id` in the Codex turn snapshot | `turn_id` in the Codex turn snapshot |+| OpenCode | `x-opencode-session`, falling back to `x-session-id` or `x-session-affinity` for an OpenCode user agent | `x-opencode-request` |+| Pi | `x-session-id`, `session_id`, `x-session-affinity`, or `x-client-request-id` for a Pi user agent | Not available |An inferred session becomes `<agent>:<session>` and an inferred turn deterministicallyselects the generation trace ID, grouping the requests made before and after tool@@ -693,9 +720,11 @@ background/store/include flags, moderation and cache options/retention. CaptureJSON types; ingestion applies Langfuse's existing model-parameter normalization(booleans and structured values become JSON strings). Omitted request parametersare not filled with assumed defaults.-The response model and service tier take precedence over requested values when present.+The response service tier takes precedence over the requested value when present.+The requested, upstream and response models are all recorded; see+[Generation metadata](#generation-metadata).Request `metadata`, `prompt_cache_key`, `safety_identifier` and deprecated `user` are-stored under `langfuse.gateway.request.*` only in full mode.+stored under `langfuse.gateway.client.request.*` only in full mode.In both modes, `usage_details` preserves the provider's entire usage object,including nested and unknown fields. The gateway does not rename counters,@@ -718,7 +747,7 @@ fast-mode pricing tier.In full mode the input is the native request with `model` and parameters projectedout: `stop_sequences`, `thinking`, `tool_choice`, `context_management` and`output_config` become model parameters and `metadata` (where Claude Code stores its-session identifiers) becomes `request.metadata`; `system`, `messages`, `tools` and+session identifiers) becomes `langfuse.gateway.client.request.metadata`; `system`, `messages`, `tools` andunknown fields are kept as sent. The output is the native assistant message`{"role":"assistant","content":[...],"stop_reason":...}`. JSON responses contributetheir `content` array at EOF. Streams rebuild each block from `content_block_start`,@@ -737,7 +766,7 @@ For OpenAI Chat Completions, scalar model parameters (`temperature`, `top_p`, `nIn full mode `stop`, `response_format`, `tool_choice`, `function_call`, `stream_options`,`logit_bias`, `modalities`, `audio`, `prediction` and `web_search_options` also becomemodel parameters, `metadata`, `prompt_cache_key`, `safety_identifier` and `user` become-`langfuse.gateway.request.*`, and `messages`, `tools`, legacy `functions` and unknown+`langfuse.gateway.client.request.*`, and `messages`, `tools`, legacy `functions` and unknownfields remain the input. The output is the native `{"choices":[...]}` list: JSONresponses contribute their choices at EOF; streams rebuild each choice from its`delta` chunks. Text fragments (`content`, `refusal`, tool and function `arguments`,@@ -790,12 +819,11 @@ bounds the number of captures. Trusted key metadata is bounded by the resolver's256 KiB response limit. In full mode, an input that is not recorded (over 5 MiB,content-encoded or not a JSON object) is explained in generation metadata:`langfuse.gateway.request.input_omitted` (`size_limit`, `content_encoding` or-`invalid_json`), `langfuse.gateway.request.body_bytes`, and, for `size_limit`,-`langfuse.gateway.request.input_limit_bytes`. If a generation with its input would+`invalid_json`) and, for `size_limit`, `langfuse.gateway.request.input_limit_bytes`;+`langfuse.gateway.request.body_bytes` is always present. If a generation with its input wouldexceed the telemetry record limit (capped at the retained buffer size) or the bufferis full, the input is dropped from that record and the generation is still uploaded,-with `input_omitted` set to `record_limit` or `telemetry_buffer` and-`langfuse.gateway.request.input_bytes`. Oversized output items+with `input_omitted` set to `record_limit` or `telemetry_buffer`. Oversized output itemsare skipped and completeness is false. Malformed, truncated or compressed bodiesdo not interrupt the relay. Capture buffers are independent of forwarding, sothese limits never cap the actual provider response. Media is not fetched/uploaded.@@ -893,7 +921,10 @@ The whole HTTP exchange has a five-second deadline and a 256 KiB response limit,including chunked responses. Redirects, automatic retries, ambient proxy settingsand transparent decompression are disabled. Errors expose fixed categories;upstream error bodies and transport details are discarded. A successful response-must match the strict v1 schema and carry an unexpired project ingestion grant.+must match the v1 schema and carry an unexpired project ingestion grant. Unknown+fields are ignored everywhere except in the provider `auth` object, so Web can add+response fields before the gateway reads them; a credential with an unexpected+field is still an invalid response.The grant's JWT payload must name the attribution's `organization_id` and`project_id`: Web writes uploads into the token's project, and telemetry batches maysend any record of a project with any of that project's grants. The gateway reads@@ -903,13 +934,12 @@ serves the requested `api_format`, `base_url` is that provider's official originand `auth` uses that provider's credential scheme. Any other combination is aninvalid response, even if Web selected it.-| Provider | API format | Official origin | Credential |-| --- | --- | --- | --- |-| `openai` | `openai.responses`, `openai.chat-completions` | `https://api.openai.com/v1` | `{"type":"Bearer","token":…}` → `Authorization: Bearer` |-| `anthropic` | `anthropic.messages` | `https://api.anthropic.com/v1` | `{"type":"x-api-key","header":"x-api-key","value":…}` → `x-api-key` |+| Provider | API format | Official origin | Credential |+| ----------- | --------------------------------------------- | ------------------------------ | ------------------------------------------------------------------- |+| `openai` | `openai.responses`, `openai.chat-completions` | `https://api.openai.com/v1` | `{"type":"Bearer","token":…}` → `Authorization: Bearer` |+| `anthropic` | `anthropic.messages` | `https://api.anthropic.com/v1` | `{"type":"x-api-key","header":"x-api-key","value":…}` → `x-api-key` |-The connection's `name` is optional in the contract, so the gateway also resolves-against a Web deployment that predates it. Web always sends it.+The connection's `name` and the attribution's `user_id`, `key_name` and `user_name` are optional in the contract, so the gateway also resolves against a Web deployment that predates them. `user_id` is the user the key is bound to and is the key's owner; it must not be blank when present. Blank `name`, `key_name` and `user_name` count as absent. They are snapshots taken at resolve time, recorded on generations as described in [Langfuse uploads](#langfuse-uploads).`ProviderConnection::credential()` exposes the secret in its header position(`ProviderCredential::Bearer` or `ProviderCredential::XApiKey`) and has no `Debug`diff --git a/ai-gateway/src/capture/anthropic_messages.rs b/ai-gateway/src/capture/anthropic_messages.rsindex 3de2db0539..6223716fab 100644--- a/ai-gateway/src/capture/anthropic_messages.rs+++ b/ai-gateway/src/capture/anthropic_messages.rs@@ -295,7 +295,6 @@ impl AnthropicMessagesCapture {.as_ref().and_then(Value::as_str).and_then(bounded_string);- self.facts.model.clone_from(&self.facts.requested_model);for key in SCALAR_PARAMETERS {if let Some(value) = request.remove(key).filter(|v| {v.is_number()diff --git a/ai-gateway/src/capture/anthropic_messages/tests.rs b/ai-gateway/src/capture/anthropic_messages/tests.rsindex cf02f50430..f5273d6495 100644--- a/ai-gateway/src/capture/anthropic_messages/tests.rs+++ b/ai-gateway/src/capture/anthropic_messages/tests.rs@@ -593,7 +593,7 @@ async fn streamed_messages_upload_one_generation_with_native_usage() {metadata["langfuse.gateway.request.api_format"],"anthropic.messages");- assert_eq!(metadata["langfuse.gateway.response.id"], "msg_1");+ assert_eq!(metadata["langfuse.gateway.upstream.response.id"], "msg_1");assert_eq!(metadata["langfuse.gateway.upstream.request.id"],"req_upstream"diff --git a/ai-gateway/src/capture/facts.rs b/ai-gateway/src/capture/facts.rsindex 6dfd26a347..ed4a02f98c 100644--- a/ai-gateway/src/capture/facts.rs+++ b/ai-gateway/src/capture/facts.rs@@ -51,6 +51,10 @@ pub(crate) struct InferenceFacts {pub first_byte_ms: Option<u128>,pub completion_start_ms: Option<u128>,pub http_status: Option<u16>,+ pub retry_after_ms: Option<u64>,+ pub upstream_model: Option<String>,+ pub request_body_bytes: usize,+ pub response_body_bytes: u64,pub metadata: Value,pub outcome: RelayOutcome,#[serde(flatten)]diff --git a/ai-gateway/src/capture/mod.rs b/ai-gateway/src/capture/mod.rsindex d98f0b2254..cf5b9a0458 100644--- a/ai-gateway/src/capture/mod.rs+++ b/ai-gateway/src/capture/mod.rs@@ -13,7 +13,7 @@ use tokio::time::Instant;use crate::{correlation::RequestCorrelation,- resolution::{ApiFormat, IngestionMode, MetadataValue, ProjectId, ResolvedRequestContext},+ resolution::{ApiFormat, IngestionMode, MetadataValue, ResolvedRequestContext},telemetry,};use anthropic_messages::AnthropicMessagesCapture;@@ -100,6 +100,9 @@ pub(crate) struct ExecutionCapture {first_byte_ms: Option<u128>,completion_start_ms: Option<u128>,http_status: Option<u16>,+ retry_after_ms: Option<u64>,+ request_body_bytes: usize,+ response_body_bytes: u64,metadata: Value,provider_model: Option<String>,delivery: Option<(telemetry::Telemetry, telemetry::DeliveryContext)>,@@ -118,6 +121,9 @@ impl ExecutionCapture {first_byte_ms: None,completion_start_ms: None,http_status: None,+ retry_after_ms: None,+ request_body_bytes: 0,+ response_body_bytes: 0,metadata: json!({}),provider_model: None,delivery: None,@@ -166,12 +172,18 @@ impl ExecutionCapture {first_byte_ms: None,completion_start_ms: None,http_status: None,+ retry_after_ms: None,+ request_body_bytes: body.len(),+ response_body_bytes: 0,metadata: json!({"organization_id": attribution.organization_id().as_str(),- "project_id": attribution.project_id().map(ProjectId::as_str),"key_id": attribution.key_id(),- "user_id": attribution.user_id(),+ "key_name": attribution.key_name().and_then(bounded_string),+ "key_owner_user_id": attribution.user_id(),+ "key_owner_name": attribution.user_name().and_then(bounded_string),"provider_connection_id": attribution.provider_connection_id(),+ "connection_name": context.connection().name().and_then(bounded_string),+ "provider": context.connection().provider().as_str(),"key_metadata": key_metadata,"ingestion_mode": context.ingestion_mode().as_str(),}),@@ -211,12 +223,16 @@ impl ExecutionCapture {pub(crate) fn record_response(&mut self, status: u16, headers: &HeaderMap) {if let Some(protocol) = &mut self.protocol {self.http_status = Some(status);+ self.retry_after_ms = retry_after_ms(headers);protocol.record_response(headers);}}pub(crate) fn push_bytes(&mut self, bytes: &[u8]) {if let Some(protocol) = &mut self.protocol {+ self.response_body_bytes = self+ .response_body_bytes+ .saturating_add(u64::try_from(bytes.len()).unwrap_or(u64::MAX));if !bytes.is_empty() {self.first_byte_ms.get_or_insert_with(|| self.started.elapsed().as_millis());@@ -241,13 +257,7 @@ impl ExecutionCapture {let Some(protocol) = self.protocol.take() else {return;};- let (api_format, mut inference) = protocol.into_facts();- if let Some(provider_model) = self.provider_model.take() {- if inference.model == inference.requested_model {- inference.model = Some(provider_model.clone());- }- inference.requested_model = Some(provider_model);- }+ let (api_format, inference) = protocol.into_facts();let facts = InferenceFacts {api_format,start_time_unix_ms: self.start_time_unix_ms,@@ -255,6 +265,10 @@ impl ExecutionCapture {first_byte_ms: self.first_byte_ms,completion_start_ms: self.completion_start_ms,http_status: self.http_status,+ retry_after_ms: self.retry_after_ms,+ upstream_model: self.provider_model.take(),+ request_body_bytes: self.request_body_bytes,+ response_body_bytes: self.response_body_bytes,metadata: std::mem::take(&mut self.metadata),outcome,inference,@@ -289,6 +303,18 @@ fn parse_request(headers: &HeaderMap, body: &[u8]) -> Result<Map<String, Value>,})}+/// Only the delay-seconds form; an HTTP-date would make the value depend on clock skew.+fn retry_after_ms(headers: &HeaderMap) -> Option<u64> {+ headers+ .get(header::RETRY_AFTER)?+ .to_str()+ .ok()?+ .trim()+ .parse::<u64>()+ .ok()+ .map(|seconds| seconds.saturating_mul(1000))+}+fn bounded_string(value: &str) -> Option<String> {(value.len() <= MAX_FACT_STRING).then(|| value.to_owned())}@@ -299,3 +325,26 @@ fn identity_encoding(headers: &HeaderMap) -> bool {.iter().all(|v| v.to_str().is_ok_and(|v| v.eq_ignore_ascii_case("identity")))}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn retry_after_accepts_only_delay_seconds() {+ for (value, expected) in [+ (Some("30"), Some(30_000)),+ (Some(" 0 "), Some(0)),+ (Some("Wed, 21 Oct 2026 07:28:00 GMT"), None),+ (Some("-1"), None),+ (Some("1.5"), None),+ (None, None),+ ] {+ let mut headers = HeaderMap::new();+ if let Some(value) = value {+ headers.insert(header::RETRY_AFTER, value.parse().unwrap());+ }+ assert_eq!(retry_after_ms(&headers), expected, "{value:?}");+ }+ }+}diff --git a/ai-gateway/src/capture/openai_chat_completions.rs b/ai-gateway/src/capture/openai_chat_completions.rsindex 84fc8c2f56..5969489e31 100644--- a/ai-gateway/src/capture/openai_chat_completions.rs+++ b/ai-gateway/src/capture/openai_chat_completions.rs@@ -355,7 +355,6 @@ impl OpenAiChatCompletionsCapture {.as_ref().and_then(Value::as_str).and_then(bounded_string);- self.facts.model.clone_from(&self.facts.requested_model);for key in SCALAR_PARAMETERS {if let Some(value) = request.remove(key).filter(|v| {self.mode == IngestionMode::Fulldiff --git a/ai-gateway/src/capture/openai_responses.rs b/ai-gateway/src/capture/openai_responses.rsindex 4259918673..c170629bbb 100644--- a/ai-gateway/src/capture/openai_responses.rs+++ b/ai-gateway/src/capture/openai_responses.rs@@ -120,7 +120,6 @@ impl OpenAiResponsesCapture {.get("model").and_then(Value::as_str).and_then(bounded_string);- self.facts.model.clone_from(&self.facts.requested_model);request.remove("model");// Agent identifiers are recorded as `agent.*` metadata in both ingestion modes.self.client_metadata = telemetry::take_agent_client_metadata(&mut request);diff --git a/ai-gateway/src/capture/openai_responses/tests.rs b/ai-gateway/src/capture/openai_responses/tests.rsindex 7fbaf07cdd..72051b9f25 100644--- a/ai-gateway/src/capture/openai_responses/tests.rs+++ b/ai-gateway/src/capture/openai_responses/tests.rs@@ -765,7 +765,7 @@ async fn client_compression_preferences_do_not_disable_capture() {assert!(metadata.get("output_complete").is_none());assert!(metadata.get("provider_status").is_none());assert!(metadata.get("native_usage").is_none());- assert_eq!(metadata["langfuse.gateway.response.id"], "resp-1");+ assert_eq!(metadata["langfuse.gateway.upstream.response.id"], "resp-1");assert!(uploaded_attribute(&upload, "langfuse.observation.usage_details")["input_tokens"].is_number()@@ -858,7 +858,7 @@ async fn codex_body_metadata_reaches_the_generation_without_agent_headers() {assert_eq!(metadata["agent.request_kind"], "turn");assert_eq!(metadata["agent.sandbox_mode"], "workspace-write");assert!(metadata.get("agent.tool_namespaces_info").is_none());- assert_eq!(metadata["langfuse.gateway.response.id"], "resp-1");+ assert_eq!(metadata["langfuse.gateway.upstream.response.id"], "resp-1");assert_eq!(uploaded_attribute(&upload, "langfuse.observation.input"),json!({"input": "hello"})diff --git a/ai-gateway/src/providers/tests.rs b/ai-gateway/src/providers/tests.rsindex acb2cc8c1f..67de442f91 100644--- a/ai-gateway/src/providers/tests.rs+++ b/ai-gateway/src/providers/tests.rs@@ -809,7 +809,7 @@ fn assert_completed_upload(payload: &serde_json::Value, streaming: bool) {.unwrap();assert!(metadata.get("relay_outcome").is_none());assert!(metadata.get("native_usage").is_none());- assert_eq!(metadata["langfuse.gateway.response.id"], "resp-1");+ assert_eq!(metadata["langfuse.gateway.upstream.response.id"], "resp-1");assert_eq!(metadata["langfuse.gateway.upstream.request.id"], "req-1");assert!(metadata["langfuse.gateway.request.id"].is_string());assert!(metadata.get("langfuse.gateway.client.request.id").is_none());@@ -1221,8 +1221,26 @@ fn assert_chat_upload(payload: &serde_json::Value, request: &str) {metadata["langfuse.gateway.request.api_format"],"openai.chat-completions");- assert_eq!(metadata["langfuse.gateway.response.id"], "chatcmpl-1");+ assert_eq!(+ metadata["langfuse.gateway.upstream.response.id"],+ "chatcmpl-1"+ );assert_eq!(metadata["langfuse.gateway.upstream.request.id"], "req-chat");+ for (key, value) in [+ ("langfuse.gateway.model.requested", json!("gpt-4.1-mini")),+ ("langfuse.gateway.model.upstream", json!("provider-model")),+ (+ "langfuse.gateway.model.response",+ json!("gpt-4.1-mini-2025-04-14"),+ ),+ ("langfuse.gateway.connection.name", json!("Production key")),+ ("langfuse.gateway.connection.provider", json!("openai")),+ ("langfuse.gateway.upstream.status_code", json!(200)),+ ("langfuse.gateway.request.body_bytes", json!(request.len())),+ ] {+ assert_eq!(metadata[key], value, "{key}");+ }+ assert!(metadata["langfuse.gateway.response.body_bytes"].as_u64() > Some(0));for secret in ["provider-secret","gateway-secret",diff --git a/ai-gateway/src/resolution/contracts.rs b/ai-gateway/src/resolution/contracts.rsindex bc3ae3ccb4..e6fc53d554 100644--- a/ai-gateway/src/resolution/contracts.rs+++ b/ai-gateway/src/resolution/contracts.rs@@ -167,7 +167,6 @@ impl Auth {}#[derive(Deserialize)]-#[serde(deny_unknown_fields)]pub struct ProviderConnection {id: String,// Optional so the gateway keeps resolving against a Web that predates it.@@ -188,7 +187,7 @@ impl ProviderConnection {}/// The user-chosen display name. Never send it to callers; it is for operators.pub fn name(&self) -> Option<&str> {- self.name.as_deref().filter(|name| !name.trim().is_empty())+ non_blank(self.name.as_deref())}pub fn provider(&self) -> Provider {self.provider@@ -238,16 +237,20 @@ impl From<&str> for ProjectId {}#[derive(Deserialize)]-#[serde(deny_unknown_fields)]pub struct RequestAttribution {organization_id: OrganizationId,/// Absent only in [`IngestionMode::None`], which needs no ingestion project.project_id: Option<ProjectId>,key_id: String,- /// The user the key is bound to. Absent for keys created before keys were- /// bound to users, and from Web versions that predate the field.+ /// The user the key is bound to, which is the key's owner. Absent for keys+ /// created before keys were bound to users, and from Web versions that+ /// predate the field.#[serde(default)]user_id: Option<String>,+ #[serde(default)]+ key_name: Option<String>,+ #[serde(default)]+ user_name: Option<String>,key_metadata: BTreeMap<String, MetadataValue>,provider_connection_id: String,}@@ -265,6 +268,12 @@ impl RequestAttribution {pub fn user_id(&self) -> Option<&str> {self.user_id.as_deref()}+ pub fn key_name(&self) -> Option<&str> {+ non_blank(self.key_name.as_deref())+ }+ pub fn user_name(&self) -> Option<&str> {+ non_blank(self.user_name.as_deref())+ }pub fn key_metadata(&self) -> &BTreeMap<String, MetadataValue> {&self.key_metadata}@@ -274,7 +283,6 @@ impl RequestAttribution {}#[derive(Deserialize)]-#[serde(deny_unknown_fields)]pub struct IngestionGrant {access_token: SecretString,#[serde(rename = "token_type", deserialize_with = "string_enum")]@@ -292,7 +300,6 @@ impl IngestionGrant {}#[derive(Deserialize)]-#[serde(deny_unknown_fields)]struct ResolutionResponse {version: u8,connection: ProviderConnection,@@ -306,6 +313,10 @@ struct ResolutionResponse {/// A validated v1 execution. Construction is restricted to successful resolution.pub struct ResolvedRequestContext(ResolutionResponse);+fn non_blank(value: Option<&str>) -> Option<&str> {+ value.filter(|value| !value.trim().is_empty())+}+// Web enums are JSON strings. Serde's externally tagged enums also accept objects.fn string_enum<'de, D: Deserializer<'de>, T: DeserializeOwned>(deserializer: D,diff --git a/ai-gateway/src/resolution/tests.rs b/ai-gateway/src/resolution/tests.rsindex 5cb50f061b..a732513306 100644--- a/ai-gateway/src/resolution/tests.rs+++ b/ai-gateway/src/resolution/tests.rs@@ -765,9 +765,68 @@ async fn rejects_incompatible_or_incomplete_execution_contexts() {body.as_object_mut().unwrap().remove(field);assert_invalid_context(body).await;}+}++#[tokio::test]+async fn accepts_unknown_response_fields_outside_provider_auth() {let mut body = success("project-1", "provider-token");body["unexpected"] = json!(true);- assert_invalid_context(body).await;+ body["connection"]["unexpected"] = json!({"nested": true});+ body["attribution"]["unexpected"] = json!("value");+ body["ingestion"]["unexpected"] = json!(1);+ body["attribution"]["key_name"] = json!("CI key");+ body["attribution"]["user_name"] = json!("Alice");+ let web = FakeWeb::start(move |_| {+ let body = body.clone();+ async move { response(body) }+ })+ .await;+ let context = web+ .control_plane()+ .resolve_at(+ GATEWAY_KEY,+ ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",+ NOW_TIME,+ REQUEST_ID,+ )+ .await+ .unwrap();+ let attribution = context.attribution();+ assert_eq!(attribution.key_name(), Some("CI key"));+ assert_eq!(attribution.user_id(), Some("user-1"));+ assert_eq!(attribution.user_name(), Some("Alice"));+}++#[tokio::test]+async fn treats_missing_or_blank_key_snapshots_as_absent() {+ for value in [None, Some(json!("")), Some(json!(" ")), Some(Value::Null)] {+ let mut body = success("project-1", "provider-token");+ if let Some(value) = value {+ for field in ["key_name", "user_name"] {+ body["attribution"][field] = value.clone();+ }+ }+ let web = FakeWeb::start(move |_| {+ let body = body.clone();+ async move { response(body) }+ })+ .await;+ let context = web+ .control_plane()+ .resolve_at(+ GATEWAY_KEY,+ ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",+ NOW_TIME,+ REQUEST_ID,+ )+ .await+ .unwrap();+ let attribution = context.attribution();+ assert_eq!(attribution.key_name(), None);+ assert_eq!(attribution.user_name(), None);+ }}#[tokio::test]diff --git a/ai-gateway/src/telemetry/context.rs b/ai-gateway/src/telemetry/context.rsindex 1e67b58625..bf2f27c417 100644--- a/ai-gateway/src/telemetry/context.rs+++ b/ai-gateway/src/telemetry/context.rs@@ -14,6 +14,7 @@ use sha2::{Digest, Sha256};const MAX_HEADER_BYTES: usize = 8 * 1024;const MAX_FIELD_BYTES: usize = 1024;+const MAX_USER_AGENT_BYTES: usize = 256;const MAX_ENTRIES: usize = 64;const MAX_AGENT_HEADER_BYTES: usize = 8 * 1024;const MAX_AGENT_METADATA_BYTES: usize = 4 * 1024;@@ -116,6 +117,7 @@ pub(super) struct GenerationContext {pub trace_state: String,pub attributes: Map<String, Value>,pub metadata: Map<String, Value>,+ pub user_agent: Option<String>,}impl GenerationContext {@@ -138,6 +140,7 @@ impl GenerationContext {trace_state: String::new(),attributes: Map::new(),metadata: Map::new(),+ user_agent: client_user_agent(headers),};let bytes = HEADERS.iter()@@ -486,6 +489,18 @@ fn user_agent(headers: &HeaderMap) -> Option<String> {agent_header(headers, "user-agent").map(|value| value.to_ascii_lowercase())}+fn client_user_agent(headers: &HeaderMap) -> Option<String> {+ let value = single_header(headers, "user-agent")?.trim();+ if value.is_empty() || value.chars().any(char::is_control) {+ return None;+ }+ let mut end = value.len().min(MAX_USER_AGENT_BYTES);+ while !value.is_char_boundary(end) {+ end -= 1;+ }+ Some(value[..end].to_owned())+}+fn agent_trace_id(name: &str, session_id: Option<&str>, turn_id: &str) -> String {let mut hasher = Sha256::new();hasher.update(b"langfuse-agent-turn-v1\0");diff --git a/ai-gateway/src/telemetry/context/tests.rs b/ai-gateway/src/telemetry/context/tests.rsindex 78131bc1b1..00da523fd8 100644--- a/ai-gateway/src/telemetry/context/tests.rs+++ b/ai-gateway/src/telemetry/context/tests.rs@@ -613,3 +613,48 @@ fn python_string_parser_accepts_escapes_but_no_expressions() {assert!(parse_tags(invalid).is_none(), "{invalid}");}}++#[test]+fn user_agent_is_kept_raw_and_bounded_even_when_trace_headers_are_rejected() {+ let mut headers = HeaderMap::new();+ headers.insert("user-agent", " Codex_CLI_rs/0.1 (Mac OS) ".parse().unwrap());+ assert_eq!(+ GenerationContext::from_headers(&headers)+ .user_agent+ .as_deref(),+ Some("Codex_CLI_rs/0.1 (Mac OS)")+ );++ headers.insert(+ "user-agent",+ HeaderValue::from_str(&"a".repeat(MAX_USER_AGENT_BYTES + 10)).unwrap(),+ );+ headers.insert(+ "baggage",+ HeaderValue::from_str(&"a".repeat(MAX_HEADER_BYTES + 1)).unwrap(),+ );+ assert_eq!(+ GenerationContext::from_headers(&headers)+ .user_agent+ .map(|value| value.len()),+ Some(MAX_USER_AGENT_BYTES)+ );++ for value in ["", " ", "agent\tname"] {+ let mut headers = HeaderMap::new();+ headers.insert("user-agent", value.parse().unwrap());+ assert!(+ GenerationContext::from_headers(&headers)+ .user_agent+ .is_none()+ );+ }+ let mut headers = HeaderMap::new();+ headers.append("user-agent", "one".parse().unwrap());+ headers.append("user-agent", "two".parse().unwrap());+ assert!(+ GenerationContext::from_headers(&headers)+ .user_agent+ .is_none()+ );+}diff --git a/ai-gateway/src/telemetry/mapping.rs b/ai-gateway/src/telemetry/mapping.rsindex dcb2a41d5f..a59f5a62df 100644--- a/ai-gateway/src/telemetry/mapping.rs+++ b/ai-gateway/src/telemetry/mapping.rs@@ -3,7 +3,12 @@ use chrono::{DateTime, SecondsFormat};use serde_json::{Map, Value, json};use super::context::GenerationContext;-use crate::capture::{InferenceFacts, InputOmissionReason, MAX_INPUT_CAPTURE_BYTES, RelayOutcome};+use crate::{+ capture::{InferenceFacts, InputOmissionReason, MAX_INPUT_CAPTURE_BYTES, RelayOutcome},+ observability::{ErrorOrigin, provider_status_reason},+};++const USER_ID_ATTRIBUTE: &str = "user.id";pub(super) fn span(facts: InferenceFacts, context: &GenerationContext) -> Value {let full = facts.metadata.get("ingestion_mode").and_then(Value::as_str) == Some("full");@@ -16,7 +21,12 @@ pub(super) fn span(facts: InferenceFacts, context: &GenerationContext) -> Value.collect();// Trusted key attribution and gateway facts take precedence over caller metadata.metadata.extend(generation_metadata(&facts));- let inference = facts.inference;+ if let Some(user_agent) = &context.user_agent {+ metadata.insert(+ "langfuse.gateway.client.user_agent".into(),+ json!(user_agent),+ );+ }let mut attributes = vec![attribute("langfuse.observation.type", "generation")];attributes.extend(context.attributes.iter().map(|(key, value)| {attribute(@@ -26,7 +36,18 @@ pub(super) fn span(facts: InferenceFacts, context: &GenerationContext) -> Value.map_or_else(|| value.to_string(), str::to_owned),)}));- if let Some(model) = inference.model.or(inference.requested_model) {+ if let Some((source, user_id)) = user_id_source(&facts, context) {+ if let Some(user_id) = user_id {+ attributes.push(attribute(USER_ID_ATTRIBUTE, user_id));+ }+ metadata.insert("langfuse.gateway.user_id_source".into(), json!(source));+ }+ let inference = facts.inference;+ if let Some(model) = inference+ .model+ .or(facts.upstream_model)+ .or(inference.requested_model)+ {attributes.push(attribute("langfuse.observation.model.name", model));}if !inference.model_parameters.is_empty() {@@ -91,6 +112,25 @@ pub(super) fn span(facts: InferenceFacts, context: &GenerationContext) -> Valuespan}+/// The caller's user ID wins. A caller's trace may already name its user, so+/// only a trace the gateway starts defaults to the key owner.+fn user_id_source<'a>(+ facts: &'a InferenceFacts,+ context: &GenerationContext,+) -> Option<(&'static str, Option<&'a str>)> {+ if context.attributes.contains_key(USER_ID_ATTRIBUTE) {+ return Some(("client", None));+ }+ if context.parent_span_id.is_some() {+ return None;+ }+ let owner = facts+ .metadata+ .get("key_owner_user_id")+ .and_then(Value::as_str)?;+ Some(("api_key_owner", Some(owner)))+}+pub(super) fn omit_input(span: &mut Value, reason: InputOmissionReason) -> bool {let Some(attributes) = span["attributes"].as_array_mut() else {return false;@@ -101,8 +141,7 @@ pub(super) fn omit_input(span: &mut Value, reason: InputOmissionReason) -> boolelse {return false;};- let input = attributes.remove(index);- let input_bytes = input["value"]["stringValue"].as_str().map_or(0, str::len);+ attributes.remove(index);if let Some(value) = attributes.iter_mut().find(|attribute| attribute["key"] == "langfuse.observation.metadata")@@ -115,10 +154,6 @@ pub(super) fn omit_input(span: &mut Value, reason: InputOmissionReason) -> bool"langfuse.gateway.request.input_omitted".into(),json!(reason),);- metadata.insert(- "langfuse.gateway.request.input_bytes".into(),- json!(input_bytes),- );*value = Value::String(Value::Object(metadata).to_string());}true@@ -143,10 +178,6 @@ fn generation_metadata(facts: &InferenceFacts) -> Map<String, Value> {.and_then(Value::as_object){for (key, value) in attribution {- // Attribution remains searchable, but cannot impersonate gateway, agent or OTEL fields.- if !reserved_metadata(key) && !agent_metadata(key) {- metadata.insert(key.clone(), value.clone());- }metadata.insert(format!("langfuse.gateway.api_key.metadata.{key}"),value.clone(),@@ -154,17 +185,29 @@ fn generation_metadata(facts: &InferenceFacts) -> Map<String, Value> {}}for (source, target) in [- ("project_id", "project.id"),("organization_id", "organization.id"),("ingestion_mode", "ingestion.mode"),("key_id", "api_key.id"),- ("user_id", "api_key.user.id"),+ ("key_name", "api_key.name"),+ ("key_owner_user_id", "api_key.owner.user_id"),+ ("key_owner_name", "api_key.owner.name"),("provider_connection_id", "connection.id"),+ ("connection_name", "connection.name"),+ ("provider", "connection.provider"),] {- if let Some(value) = facts.metadata.get(source) {+ if let Some(value) = facts.metadata.get(source).filter(|value| !value.is_null()) {metadata.insert(format!("langfuse.gateway.{target}"), value.clone());}}+ for (key, value) in [+ ("model.requested", &facts.inference.requested_model),+ ("model.upstream", &facts.upstream_model),+ ("model.response", &facts.inference.model),+ ] {+ if let Some(value) = value {+ metadata.insert(format!("langfuse.gateway.{key}"), json!(value));+ }+ }metadata.insert("langfuse.gateway.request.api_format".into(),json!(facts.api_format),@@ -180,16 +223,12 @@ fn generation_metadata(facts: &InferenceFacts) -> Map<String, Value> {"langfuse.gateway.response.status_code".into(),json!(response_status),);- for (key, value) in [- ("response.id", &facts.inference.provider_response_id),- ("upstream.request.id", &facts.inference.provider_request_id),- ] {- if let Some(value) = value {- metadata.insert(format!("langfuse.gateway.{key}"), json!(value));- }- }+ metadata.extend(upstream_metadata(facts));for (key, value) in &facts.inference.request_metadata {- metadata.insert(format!("langfuse.gateway.request.{key}"), value.clone());+ metadata.insert(+ format!("langfuse.gateway.client.request.{key}"),+ value.clone(),+ );}// Inserted after caller request metadata so the gateway's own ID always wins.for (source, target) in [@@ -205,10 +244,6 @@ fn generation_metadata(facts: &InferenceFacts) -> Map<String, Value> {"langfuse.gateway.request.input_omitted".into(),json!(omission.reason),);- metadata.insert(- "langfuse.gateway.request.body_bytes".into(),- json!(omission.body_bytes),- );if omission.reason == InputOmissionReason::SizeLimit {metadata.insert("langfuse.gateway.request.input_limit_bytes".into(),@@ -219,8 +254,66 @@ fn generation_metadata(facts: &InferenceFacts) -> Map<String, Value> {metadata}-fn agent_metadata(key: &str) -> bool {- key == "agent" || key.starts_with("agent.")+/// Facts about the provider's answer, or why there was none.+fn upstream_metadata(facts: &InferenceFacts) -> Map<String, Value> {+ let mut metadata = Map::new();+ if let Some(status) = facts.http_status {+ metadata.insert(+ "langfuse.gateway.upstream.status_code".into(),+ json!(status),+ );+ }+ if let Some(retry_after_ms) = facts.retry_after_ms {+ metadata.insert(+ "langfuse.gateway.upstream.retry_after_ms".into(),+ json!(retry_after_ms),+ );+ }+ if let Some((origin, kind)) = error_classification(facts) {+ metadata.insert(+ "langfuse.gateway.error.origin".into(),+ json!(origin.as_str()),+ );+ metadata.insert("langfuse.gateway.error.type".into(), json!(kind));+ }+ metadata.insert(+ "langfuse.gateway.request.body_bytes".into(),+ json!(facts.request_body_bytes),+ );+ if facts.http_status.is_some() {+ metadata.insert(+ "langfuse.gateway.response.body_bytes".into(),+ json!(facts.response_body_bytes),+ );+ }+ for (key, value) in [+ (+ "upstream.response.id",+ &facts.inference.provider_response_id,+ ),+ ("upstream.request.id", &facts.inference.provider_request_id),+ ] {+ if let Some(value) = value {+ metadata.insert(format!("langfuse.gateway.{key}"), json!(value));+ }+ }+ metadata+}++/// Mirrors the failure order of [`observation_status`] and the bounded+/// `error.origin`/`error.reason` values on the gateway's own request telemetry.+fn error_classification(facts: &InferenceFacts) -> Option<(ErrorOrigin, &'static str)> {+ if let Some(status) = facts.http_status.filter(|status| *status >= 400) {+ Some((ErrorOrigin::Provider, provider_status_reason(status)))+ } else if facts.inference.provider_status.as_deref() == Some("failed") {+ Some((ErrorOrigin::Provider, "provider_response_failed"))+ } else {+ match facts.outcome {+ RelayOutcome::Timeout => Some((ErrorOrigin::ProviderTransport, "timeout")),+ RelayOutcome::TransportError => Some((ErrorOrigin::ProviderTransport, "transport")),+ RelayOutcome::Eof | RelayOutcome::Cancelled => None,+ }+ }}fn observation_status(facts: &InferenceFacts) -> (&'static str, Option<String>) {@@ -328,7 +421,11 @@ mod tests {first_byte_ms: Some(50),completion_start_ms: Some(125),http_status: Some(200),- metadata: json!({"ingestion_mode": "full", "project_id": "project", "key_id": "key", "key_metadata": {"team": "search"}}),+ retry_after_ms: None,+ upstream_model: None,+ request_body_bytes: 64,+ response_body_bytes: 256,+ metadata: json!({"ingestion_mode": "full", "key_id": "key", "key_metadata": {"team": "search"}}),outcome: RelayOutcome::Eof,inference: crate::capture::ProviderFacts::default(),};@@ -417,9 +514,8 @@ mod tests {);let metadata = metadata(&attrs);assert_eq!(metadata["custom"], "value");- assert_eq!(metadata["team"], "search");+ assert_eq!(metadata["team"], "untrusted");assert_eq!(metadata["langfuse.gateway.response.status_code"], 200);- assert_eq!(metadata["langfuse.gateway.project.id"], "project");assert_eq!(metadata["langfuse.gateway.api_key.id"], "key");assert_eq!(metadata["langfuse.gateway.api_key.metadata.team"], "search");for key in [@@ -427,6 +523,7 @@ mod tests {"resourceAttributes.secret","attributes","langfuse.gateway.request.fake",+ "langfuse.gateway.project.id",] {assert!(metadata.get(key).is_none(), "caller forged {key}");}@@ -459,27 +556,30 @@ mod tests {metadata,json!({"langfuse.gateway.response.status_code": 200,- "team": "search",+ "langfuse.gateway.upstream.status_code": 200,"langfuse.gateway.api_key.metadata.team": "search","langfuse.gateway.api_key.id": "key",- "langfuse.gateway.project.id": "project","langfuse.gateway.ingestion.mode": "full","langfuse.gateway.request.api_format": "openai.responses",+ "langfuse.gateway.request.body_bytes": 64,+ "langfuse.gateway.response.body_bytes": 256,+ "langfuse.gateway.model.requested": "requested-model",+ "langfuse.gateway.model.response": "actual-model",}));}#[test]- fn attribution_is_searchable_without_overwriting_gateway_or_otel_metadata() {+ fn key_metadata_is_only_namespaced_and_cannot_overwrite_gateway_or_otel_metadata() {let mut facts = facts();facts.metadata["organization_id"] = json!("org");facts.metadata["provider_connection_id"] = json!("connection");- facts.metadata["user_id"] = json!("user");+ facts.metadata["key_owner_user_id"] = json!("user");facts.metadata["key_metadata"] = json!({"team": "search", "enabled": true, "cost_center": 42,"langfuse.gateway.response.status_code": 500,"langfuse.gateway.api_key.id": "spoofed-key",- "langfuse.gateway.api_key.user.id": "spoofed-user",+ "langfuse.gateway.api_key.owner.user_id": "spoofed-user","langfuse.gateway.upstream.request.id": "spoofed-request","langfuse.gateway.future_field": "spoofed-future","langfuse.gateway.api_key.metadata.team": "spoofed-team",@@ -495,19 +595,29 @@ mod tests {.insert("agent.name".into(), json!("opencode"));let attrs = attributes(&span(facts, &context));let metadata = metadata(&attrs);- assert_eq!(metadata["team"], "search");- assert_eq!(metadata["enabled"], true);- assert_eq!(metadata["cost_center"], 42);+ for key in ["team", "enabled", "cost_center"] {+ assert!(metadata.get(key).is_none(), "{key}");+ }assert_eq!(metadata["langfuse.gateway.api_key.metadata.team"], "search");+ assert_eq!(metadata["langfuse.gateway.api_key.metadata.enabled"], true);+ assert_eq!(+ metadata["langfuse.gateway.api_key.metadata.cost_center"],+ 42+ );assert_eq!(metadata["langfuse.gateway.api_key.metadata.langfuse.gateway.response.status_code"],500);assert_eq!(metadata["langfuse.gateway.response.status_code"], 200);assert_eq!(metadata["langfuse.gateway.api_key.id"], "key");- assert_eq!(metadata["langfuse.gateway.api_key.user.id"], "user");+ assert_eq!(metadata["langfuse.gateway.api_key.owner.user_id"], "user");+ assert!(metadata.get("langfuse.gateway.api_key.user.id").is_none());assert_eq!(metadata["langfuse.gateway.connection.id"], "connection");- assert_eq!(metadata["langfuse.gateway.response.id"], "response");+ assert_eq!(+ metadata["langfuse.gateway.upstream.response.id"],+ "response"+ );+ assert!(metadata.get("langfuse.gateway.response.id").is_none());assert_eq!(metadata["langfuse.gateway.organization.id"], "org");assert_eq!(metadata["agent.name"], "opencode");assert_eq!(@@ -547,6 +657,197 @@ mod tests {assert!(metadata(&attrs).get("native_usage").is_none());}+ #[test]+ fn event_model_prefers_response_then_upstream_then_requested() {+ for (response, upstream, expected) in [+ (+ Some("response-model"),+ Some("upstream-model"),+ "response-model",+ ),+ (None, Some("upstream-model"), "upstream-model"),+ (None, None, "requested-model"),+ ] {+ let mut facts = facts();+ facts.inference.model = response.map(str::to_owned);+ facts.upstream_model = upstream.map(str::to_owned);+ let attrs = attributes(&span(facts, &context()));+ assert_eq!(attrs["langfuse.observation.model.name"], expected);+ let metadata = metadata(&attrs);+ assert_eq!(+ metadata["langfuse.gateway.model.requested"],+ "requested-model"+ );+ assert_eq!(+ metadata.get("langfuse.gateway.model.upstream"),+ upstream.map(|model| json!(model)).as_ref()+ );+ assert_eq!(+ metadata.get("langfuse.gateway.model.response"),+ response.map(|model| json!(model)).as_ref()+ );+ }+ }++ #[test]+ fn key_and_connection_snapshots_use_gateway_names_and_skip_absent_values() {+ let mut facts = facts();+ facts.metadata["organization_id"] = json!("org");+ facts.metadata["key_name"] = json!("CI key");+ facts.metadata["key_owner_user_id"] = json!("user-1");+ facts.metadata["key_owner_name"] = json!("Alice");+ facts.metadata["provider_connection_id"] = json!("connection");+ facts.metadata["connection_name"] = Value::Null;+ facts.metadata["provider"] = json!("openai");+ let metadata = metadata(&attributes(&span(facts, &context())));+ assert_eq!(metadata["langfuse.gateway.organization.id"], "org");+ assert_eq!(metadata["langfuse.gateway.api_key.name"], "CI key");+ assert_eq!(metadata["langfuse.gateway.api_key.owner.user_id"], "user-1");+ assert_eq!(metadata["langfuse.gateway.api_key.owner.name"], "Alice");+ assert_eq!(metadata["langfuse.gateway.connection.id"], "connection");+ assert_eq!(metadata["langfuse.gateway.connection.provider"], "openai");+ assert!(metadata.get("langfuse.gateway.connection.name").is_none());+ }++ #[test]+ fn user_id_defaults_to_the_key_owner_only_on_gateway_rooted_traces() {+ let mut owned = facts();+ owned.metadata["key_owner_user_id"] = json!("owner");++ let attrs = attributes(&span(owned, &context()));+ assert_eq!(attrs["user.id"], "owner");+ assert_eq!(+ metadata(&attrs)["langfuse.gateway.user_id_source"],+ "api_key_owner"+ );++ let mut caller = context();+ caller.attributes.insert("user.id".into(), json!("caller"));+ let mut owned = facts();+ owned.metadata["key_owner_user_id"] = json!("owner");+ let attrs = attributes(&span(owned, &caller));+ assert_eq!(attrs["user.id"], "caller");+ assert_eq!(+ metadata(&attrs)["langfuse.gateway.user_id_source"],+ "client"+ );++ let mut child = context();+ child.parent_span_id = Some("fedcba9876543210".into());+ let mut owned = facts();+ owned.metadata["key_owner_user_id"] = json!("owner");+ let attrs = attributes(&span(owned, &child));+ assert!(!attrs.contains_key("user.id"));+ assert!(+ metadata(&attrs)+ .get("langfuse.gateway.user_id_source")+ .is_none()+ );++ let attrs = attributes(&span(facts(), &context()));+ assert!(!attrs.contains_key("user.id"));+ assert!(+ metadata(&attrs)+ .get("langfuse.gateway.user_id_source")+ .is_none()+ );+ }++ #[test]+ fn client_user_agent_is_recorded_as_gateway_metadata() {+ let mut context = context();+ context.user_agent = Some("codex_cli_rs/0.1".into());+ let metadata = metadata(&attributes(&span(facts(), &context)));+ assert_eq!(+ metadata["langfuse.gateway.client.user_agent"],+ "codex_cli_rs/0.1"+ );+ }++ #[test]+ fn failures_carry_error_origin_type_and_upstream_rate_limit_details() {+ for (http_status, provider_status, outcome, expected) in [+ (+ Some(429),+ None,+ RelayOutcome::Eof,+ Some(("provider", "provider_rate_limited")),+ ),+ (+ Some(401),+ None,+ RelayOutcome::Eof,+ Some(("provider", "provider_auth")),+ ),+ (+ Some(400),+ None,+ RelayOutcome::Eof,+ Some(("provider", "provider_4xx")),+ ),+ (+ Some(503),+ None,+ RelayOutcome::Eof,+ Some(("provider", "provider_5xx")),+ ),+ (+ Some(200),+ Some("failed"),+ RelayOutcome::Eof,+ Some(("provider", "provider_response_failed")),+ ),+ (+ None,+ None,+ RelayOutcome::Timeout,+ Some(("provider_transport", "timeout")),+ ),+ (+ Some(200),+ None,+ RelayOutcome::TransportError,+ Some(("provider_transport", "transport")),+ ),+ (Some(200), None, RelayOutcome::Cancelled, None),+ (Some(200), None, RelayOutcome::Eof, None),+ ] {+ let mut facts = facts();+ facts.http_status = http_status;+ facts.inference.provider_status = provider_status.map(str::to_owned);+ facts.outcome = outcome;+ let metadata = metadata(&attributes(&span(facts, &context())));+ assert_eq!(+ metadata.get("langfuse.gateway.error.origin").cloned(),+ expected.map(|(origin, _)| json!(origin)),+ "{http_status:?} {outcome:?}"+ );+ assert_eq!(+ metadata.get("langfuse.gateway.error.type").cloned(),+ expected.map(|(_, kind)| json!(kind)),+ "{http_status:?} {outcome:?}"+ );+ assert_eq!(+ metadata+ .get("langfuse.gateway.upstream.status_code")+ .cloned(),+ http_status.map(|status| json!(status))+ );+ assert_eq!(+ metadata+ .get("langfuse.gateway.response.body_bytes")+ .is_some(),+ http_status.is_some()+ );+ }++ let mut facts = facts();+ facts.http_status = Some(429);+ facts.retry_after_ms = Some(30_000);+ let metadata = metadata(&attributes(&span(facts, &context())));+ assert_eq!(metadata["langfuse.gateway.upstream.retry_after_ms"], 30_000);+ }+#[test]fn usage_projection_preserves_new_numeric_details_without_unknown_top_level_fields() {let mut facts = facts();@@ -668,20 +969,28 @@ mod tests {let attrs = attributes(&span(facts, &context()));let metadata = metadata(&attrs);assert_eq!(- metadata["langfuse.gateway.request.metadata"],+ metadata["langfuse.gateway.client.request.metadata"],json!({"customer": "customer-1"}));assert_eq!(- metadata["langfuse.gateway.request.prompt_cache_key"],+ metadata["langfuse.gateway.client.request.prompt_cache_key"],"cache-key");assert_eq!(- metadata["langfuse.gateway.request.safety_identifier"],+ metadata["langfuse.gateway.client.request.safety_identifier"],"safety-id");- assert_eq!(metadata["langfuse.gateway.request.user"], "legacy-user");+ assert_eq!(+ metadata["langfuse.gateway.client.request.user"],+ "legacy-user"+ );for key in ["metadata", "prompt_cache_key", "safety_identifier", "user"] {assert!(metadata.get(key).is_none());+ assert!(+ metadata+ .get(format!("langfuse.gateway.request.{key}"))+ .is_none()+ );}}@@ -689,6 +998,7 @@ mod tests {fn omitted_input_is_explained_in_gateway_metadata() {let mut facts = facts();facts.inference.input = None;+ facts.request_body_bytes = MAX_INPUT_CAPTURE_BYTES + 1;facts.inference.input_omission = Some(crate::capture::InputOmission {reason: InputOmissionReason::SizeLimit,body_bytes: MAX_INPUT_CAPTURE_BYTES + 1,diff --git a/ai-gateway/src/telemetry/tests.rs b/ai-gateway/src/telemetry/tests.rsindex c03509d622..93545c1c79 100644--- a/ai-gateway/src/telemetry/tests.rs+++ b/ai-gateway/src/telemetry/tests.rs@@ -27,7 +27,11 @@ fn facts(project: &str) -> InferenceFacts {first_byte_ms: Some(10),completion_start_ms: None,http_status: Some(200),- metadata: json!({"project_id": project, "ingestion_mode": "usage"}),+ retry_after_ms: None,+ upstream_model: None,+ request_body_bytes: 0,+ response_body_bytes: 0,+ metadata: json!({"key_id": project, "ingestion_mode": "usage"}),outcome: RelayOutcome::Eof,inference: crate::capture::ProviderFacts::default(),}@@ -146,7 +150,7 @@ async fn concurrent_projects_keep_their_original_grants_and_attribution() {serde_json::from_str(metadata["value"]["stringValue"].as_str().unwrap()).unwrap();received.lock().unwrap().push((authorization,- metadata["langfuse.gateway.project.id"]+ metadata["langfuse.gateway.api_key.id"].as_str().unwrap().to_owned(),@@ -374,7 +378,7 @@ fn span_projects(payload: &Value) -> Vec<String> {.unwrap();let metadata: Value =serde_json::from_str(metadata["value"]["stringValue"].as_str().unwrap()).unwrap();- metadata["langfuse.gateway.project.id"]+ metadata["langfuse.gateway.api_key.id"].as_str().unwrap().to_owned()@@ -770,10 +774,6 @@ async fn input_that_does_not_fit_is_omitted_instead_of_dropping_the_generation()).unwrap();assert_eq!(metadata["langfuse.gateway.request.input_omitted"], reason);- assert_eq!(- metadata["langfuse.gateway.request.input_bytes"],- input_size + 2- );}}diff --git a/packages/shared/src/server/otel/OtelIngestionProcessor.metadataDropped.test.ts b/packages/shared/src/server/otel/OtelIngestionProcessor.metadataDropped.test.tsindex 132fc4db96..91dee568be 100644--- a/packages/shared/src/server/otel/OtelIngestionProcessor.metadataDropped.test.ts+++ b/packages/shared/src/server/otel/OtelIngestionProcessor.metadataDropped.test.ts@@ -185,12 +185,12 @@ describe("gateway metadata", () => {"langfuse.observation.input": '[{"role":"user","content":"Hi"}]',"langfuse.observation.output": '[{"type":"message","content":[]}]',"langfuse.observation.metadata": JSON.stringify({- "langfuse.gateway.provider.request.id": "req-test",+ "langfuse.gateway.upstream.response.id": "resp-test","langfuse.gateway.request.id": gatewayRequestIds.request,"langfuse.gateway.client.request.id": gatewayRequestIds.client,"langfuse.gateway.upstream.request.id": gatewayRequestIds.upstream,}),- "langfuse.observation.metadata.langfuse.gateway.api-key.id":+ "langfuse.observation.metadata.langfuse.gateway.api_key.id":"key-test","custom.attribute": "keep-custom","langfuse.observation.custom": "keep-unknown",@@ -235,11 +235,11 @@ describe("gateway metadata", () => {}),});expect(observation?.metadata).toEqual({- "langfuse.gateway.provider.request.id": "req-test",+ "langfuse.gateway.upstream.response.id": "resp-test","langfuse.gateway.request.id": gatewayRequestIds.request,"langfuse.gateway.client.request.id": gatewayRequestIds.client,"langfuse.gateway.upstream.request.id": gatewayRequestIds.upstream,- "langfuse.gateway.api-key.id": "key-test",+ "langfuse.gateway.api_key.id": "key-test",attributes: {...(scope === "langfuse-ai-gateway" ? {} : canonicalAttributes),"custom.attribute": "keep-custom",diff --git a/packages/shared/src/server/otel/OtelIngestionProcessor.ts b/packages/shared/src/server/otel/OtelIngestionProcessor.tsindex 791ca88de9..6f265a3a76 100644--- a/packages/shared/src/server/otel/OtelIngestionProcessor.ts+++ b/packages/shared/src/server/otel/OtelIngestionProcessor.ts@@ -47,6 +47,21 @@ import { defineOwnValue } from "../../utils/objects";export const AI_GATEWAY_INSTRUMENTATION_SCOPE_NAME = "langfuse-ai-gateway";+const AI_GATEWAY_INGESTION_API_KEY_PREFIX = "gateway:";++/**+ * The ingestion API key Web records for an upload authorized by a gateway+ * ingestion grant. Unlike the instrumentation scope name, an OTLP sender cannot+ * choose it, so it is the marker for behavior that trusts the gateway.+ */+export const aiGatewayIngestionApiKey = (grantId: string) =>+ `${AI_GATEWAY_INGESTION_API_KEY_PREFIX}${grantId}`;++export const isAiGatewayIngestionApiKey = (+ ingestionApiKey: string | null | undefined,+): boolean =>+ ingestionApiKey?.startsWith(AI_GATEWAY_INGESTION_API_KEY_PREFIX) ?? false;+// Foreign level vocabularies observed from OTel senders (OTel severity// names, python logging, loguru, console) mapped onto the Langfuse enum.// The classic ingestion API keeps its strict enum; only thediff --git a/packages/shared/src/utils/normalized-io/conventions/providers/openai/fixtures.ts b/packages/shared/src/utils/normalized-io/conventions/providers/openai/fixtures.tsindex 3f25673dcf..8a0cd0b158 100644--- a/packages/shared/src/utils/normalized-io/conventions/providers/openai/fixtures.ts+++ b/packages/shared/src/utils/normalized-io/conventions/providers/openai/fixtures.ts@@ -4132,19 +4132,19 @@ export const capturedTraceFixtures: NormalizedIOFixture[] = [output:'[{"content":[],"encrypted_content":"anonymized-encrypted-reasoning","id":"rs_01989159bdac1f03016aaac3e381b887d2ab6730f292a7a587","summary":[{"text":"**Inspecting the implementation**\\n\\nI will search the sample project for the retry decorator and its tests, then explain how it handles failed calls.","type":"summary_text"}],"type":"reasoning"},{"arguments":"{\\"command\\":\\"rg -n \\\\\\"def retry|retry\\\\\\" example_sdk tests/unit | head -80\\",\\"timeout\\":10}","call_id":"call_0mE4wfJ7mEy41Nj9HfXjidem","id":"fc_01989159bdac1f03016aaac3e49de487d29427a84b52330077","name":"bash","status":"completed","type":"function_call"}]',metadata: {- "langfuse.gateway.provider.response_id":+ "langfuse.gateway.upstream.response.id":"resp_01989159bdac1f03016aaac3e2b53887d2bf05419f5a5c9409",- "langfuse.gateway.provider.request_id":+ "langfuse.gateway.upstream.request.id":"req_3bc6e7143cb747cc80876011ee0ac473",- "langfuse.gateway.provider.request.prompt_cache_key":+ "langfuse.gateway.client.request.prompt_cache_key":"01a0ab0a-84b2-7775-866d-e0a8ddf0e3b8",- "langfuse.gateway.provider.connection_id": "cmu3uqinh0004ad0ld88jrfe9",- "langfuse.gateway.project_id": "cmtvtpilr0004ad0lxphzbd5i",- "langfuse.gateway.organization_id": "cmtvmaywg0000ad0ly0x9tono",- "langfuse.gateway.ingestion_mode": "full",- "langfuse.gateway.api_format": "openai.responses",- "langfuse.gateway.api-key.id": "cmu3uqwsb0007ad0lwaua4ezv",- http_status: 200,+ "langfuse.gateway.connection.id": "cmu3uqinh0004ad0ld88jrfe9",+ "langfuse.gateway.connection.provider": "openai",+ "langfuse.gateway.organization.id": "cmtvmaywg0000ad0ly0x9tono",+ "langfuse.gateway.ingestion.mode": "full",+ "langfuse.gateway.request.api_format": "openai.responses",+ "langfuse.gateway.api_key.id": "cmu3uqwsb0007ad0lwaua4ezv",+ "langfuse.gateway.response.status_code": 200,"scope.version": "0.1.0","scope.name": "langfuse-ai-gateway","resourceAttributes.service.name": "langfuse-ai-gateway",diff --git a/web/src/__e2e__/ai-gateway.gatewaye2e.ts b/web/src/__e2e__/ai-gateway.gatewaye2e.tsindex 829639167a..33395c106b 100644--- a/web/src/__e2e__/ai-gateway.gatewaye2e.ts+++ b/web/src/__e2e__/ai-gateway.gatewaye2e.ts@@ -608,7 +608,9 @@ describe("AI gateway live end-to-end", () => {organization_id: ORGANIZATION_ID,project_id: expect.stringMatching(/\S/),key_id: keyId,- user_id: expect.stringMatching(/\S/),+ user_id: admin.userId,+ key_name: `Gateway E2E ${provider} ${apiFormat}`,+ user_name: admin.userName || null,key_metadata: {test: true,provider,@@ -784,7 +786,11 @@ async function createGatewayAdmin() {environment: {} as Session["environment"],};const context = createInnerTRPCContext({ session, headers: {} });- return { caller: appRouter.createCaller({ ...context, prisma }) };+ return {+ caller: appRouter.createCaller({ ...context, prisma }),+ userId: user.id,+ userName: user.name,+ };}async function selectOnlyConnection(targetConnectionId: string) {diff --git a/web/src/__tests__/server/organizations-api-key-trpc.servertest.ts b/web/src/__tests__/server/organizations-api-key-trpc.servertest.tsindex 5c89d78710..a843e250fd 100644--- a/web/src/__tests__/server/organizations-api-key-trpc.servertest.ts+++ b/web/src/__tests__/server/organizations-api-key-trpc.servertest.ts@@ -438,6 +438,49 @@ describe("organization API keys trpc", () => {expect(updatedKey?.note).toBe("Updated Note");});+ it("evicts a renamed gateway key from the gateway cache", async () => {+ Object.assign(env, { LANGFUSE_AI_GATEWAY_CACHE_RESOLVE_ENABLED: "true" });+ const gatewayKey = await createApiKey(prisma, {+ tenant: OrganizationId(organizationId),+ scope: OrganizationId(organizationId),+ role: SystemRoleId("AI_GATEWAY"),+ createdBy: UserId("user-1"),+ name: "Original gateway key",+ });+ const { fastHashedSecretKey } = await prisma.apiKey.findUniqueOrThrow({+ where: { id: gatewayKey.id },+ select: { fastHashedSecretKey: true },+ });+ const cache = new GatewayResolveCache();+ const cacheEntry = {+ fastHashedSecretKey: fastHashedSecretKey!,+ apiFormat: gatewayApiFormats[0],+ };+ await cache.set({+ ...cacheEntry,+ context: {+ organizationId,+ apiKeyId: gatewayKey.id,+ userId: null,+ apiKeyName: "Original gateway key",+ userName: null,+ keyMetadata: {},+ ingestionProjectId: null,+ ingestionMode: null,+ connections: [],+ },+ });+ expect(await cache.get(cacheEntry)).not.toBeNull();++ await ownerCaller.organizationApiKeys.updateName({+ orgId: organizationId,+ keyId: gatewayKey.id,+ name: "Renamed gateway key",+ });++ expect(await cache.get(cacheEntry)).toBeNull();+ });+it("returns NOT_FOUND for a missing API key", async () => {await expect(ownerCaller.organizationApiKeys.updateName({@@ -554,6 +597,8 @@ describe("organization API keys trpc", () => {organizationId,apiKeyId: gatewayKey.id,userId: null,+ apiKeyName: "Cached gateway key",+ userName: null,keyMetadata: {},ingestionProjectId: null,ingestionMode: null,diff --git a/web/src/__tests__/server/unit/gatewayResolveCache.servertest.ts b/web/src/__tests__/server/unit/gatewayResolveCache.servertest.tsindex 340bcc31ce..5639a117a9 100644--- a/web/src/__tests__/server/unit/gatewayResolveCache.servertest.ts+++ b/web/src/__tests__/server/unit/gatewayResolveCache.servertest.ts@@ -17,6 +17,8 @@ const context: CachedResolveContext = {organizationId: "org-1",apiKeyId: "key-1",userId: "user-1",+ apiKeyName: "CI key",+ userName: "Alice",keyMetadata: {},ingestionProjectId: null,ingestionMode: null,@@ -105,6 +107,36 @@ describe("GatewayResolveCache", () => {expect(redis.expire).not.toHaveBeenCalled();});+ it("reads entries cached before the key snapshot fields existed", async () => {+ const redis = createStatefulRedis();+ const cache = new GatewayResolveCache(redis);+ const legacy = {+ organizationId: "org-1",+ apiKeyId: "key-1",+ userId: "user-1",+ keyMetadata: {},+ ingestionProjectId: null,+ ingestionMode: null,+ connections: [],+ };+ await cache.set({+ fastHashedSecretKey: "hashed-key",+ apiFormat: "openai.responses",+ context: legacy as unknown as CachedResolveContext,+ });++ await expect(+ cache.get({+ fastHashedSecretKey: "hashed-key",+ apiFormat: "openai.responses",+ }),+ ).resolves.toEqual({+ ...legacy,+ apiKeyName: null,+ userName: null,+ });+ });+it("invalidates cached contexts for an organization", async () => {const cache = new GatewayResolveCache(createStatefulRedis());diff --git a/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts b/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.tsindex 15f29f5024..78b57665a6 100644--- a/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts+++ b/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts@@ -150,6 +150,8 @@ export class GatewayApiKeyAuthenticator {organizationId,apiKeyId: row.id,userId: row.userId,+ apiKeyName: row.note || null,+ userName: row.user?.name || null,keyMetadata: toKeyMetadata(row.gatewayAssociation?.metadata),ingestionProjectId: projectIsUsable? (gatewayConfig?.defaultIngestionProjectId ?? null)diff --git a/web/src/features/ai-gateway/server/auth/ingestionTokenVerifier.ts b/web/src/features/ai-gateway/server/auth/ingestionTokenVerifier.tsindex 9b8577520c..2869381026 100644--- a/web/src/features/ai-gateway/server/auth/ingestionTokenVerifier.ts+++ b/web/src/features/ai-gateway/server/auth/ingestionTokenVerifier.ts@@ -4,6 +4,7 @@ import { getOrganizationPlanServerSide } from "@/src/features/entitlements/serveimport { CloudConfigSchema, UnauthorizedError } from "@langfuse/shared";import { prisma, type PrismaClient } from "@langfuse/shared/src/db";import {+ aiGatewayIngestionApiKey,type ApiAccessScopeWithOptionalApiKeyId,logger,} from "@langfuse/shared/src/server";@@ -192,7 +193,7 @@ export async function verifyGatewayIngestionAuthorization(orgId: project.organization.id,plan: getOrganizationPlanServerSide(cloudConfig),rateLimitOverrides: cloudConfig?.rateLimitOverrides ?? [],- publicKey: `gateway:${claims.jti}`,+ publicKey: aiGatewayIngestionApiKey(claims.jti),isIngestionSuspended:project.organization.cloudFreeTierUsageThresholdState === "BLOCKED",isInAppAgentKey: false,diff --git a/web/src/features/ai-gateway/server/provider/registry.ts b/web/src/features/ai-gateway/server/provider/registry.tsindex 48794b22e7..70a8dcafb1 100644--- a/web/src/features/ai-gateway/server/provider/registry.ts+++ b/web/src/features/ai-gateway/server/provider/registry.ts@@ -48,8 +48,11 @@ export const GatewayResolveResponseSchema = z// Null only in "none" mode, which needs no ingestion project.project_id: z.string().nullable(),key_id: z.string(),- // Null for keys created before keys were bound to users.+ // The bound user is the key's owner. Null for keys created before+ // keys were bound to users.user_id: z.string().nullable(),+ key_name: z.string().nullable(),+ user_name: z.string().nullable(),key_metadata: GatewayMetadataSchema,provider_connection_id: z.string(),})diff --git a/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts b/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.tsindex bc7e81bb06..b909a55457 100644--- a/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts+++ b/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts@@ -27,6 +27,9 @@ export const CachedResolveContext = z.object({organizationId: z.string(),apiKeyId: z.string(),userId: z.string().nullable(),+ // Defaulted so entries cached before these fields existed still parse.+ apiKeyName: z.string().nullable().default(null),+ userName: z.string().nullable().default(null),keyMetadata: GatewayMetadataSchema,ingestionProjectId: z.string().nullable(),ingestionMode: z.enum(["NONE", "USAGE", "FULL"]).nullable(),diff --git a/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts b/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.tsindex fdac1a33dd..e59eb050f1 100644--- a/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts+++ b/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts@@ -21,6 +21,8 @@ export class GatewayResolveRepository {orgId: true,userId: true,expiresAt: true,+ note: true,+ user: { select: { name: true } },gatewayAssociation: { select: { metadata: true } },organization: {select: {diff --git a/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts b/web/src/features/ai-gateway/server/resolve/resolveService.servertest.tsindex 2c8dd9f60d..d6e7e9b8b3 100644--- a/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts+++ b/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts@@ -65,11 +65,15 @@ const row = (overrides: {withoutProject?: boolean;metadata?: unknown;userId?: string | null;+ note?: string | null;+ user?: { name: string | null } | null;}) => ({id: "key-1",orgId: "org-1",userId: overrides.userId === undefined ? "user-1" : overrides.userId,expiresAt: null,+ note: overrides.note === undefined ? "CI key" : overrides.note,+ user: overrides.user === undefined ? { name: "Alice" } : overrides.user,gatewayAssociation:overrides.metadata === undefined ? null : { metadata: overrides.metadata },organization: {@@ -127,6 +131,8 @@ describe("GatewayResolveService", () => {project_id: "project-1",key_id: "key-1",user_id: "user-1",+ key_name: "CI key",+ user_name: "Alice",key_metadata: { team: "platform", project_id: "spoofed-project" },provider_connection_id: "connection-1",},@@ -139,6 +145,32 @@ describe("GatewayResolveService", () => {});});+ it.each([+ { note: "", userId: null, user: null },+ { note: null, userId: "user-1", user: { name: null } },+ { note: null, userId: "user-1", user: { name: "" } },+ ])(+ "sends absent key snapshots as null (%o)",+ async ({ note, userId, user }) => {+ const service = serviceWith(() =>+ Promise.resolve(row({ note, userId, user })),+ );++ await expect(+ service.resolve({+ fastHashedSecretKey: "hashed-secret-key",+ apiFormat: "openai.responses",+ }),+ ).resolves.toMatchObject({+ attribution: {+ key_name: null,+ user_id: userId,+ user_name: null,+ },+ });+ },+ );+it("drops key metadata that is not a flat object of scalars", async () => {const service = serviceWith(() =>Promise.resolve(row({ metadata: [{ team: "platform" }] })),diff --git a/web/src/features/ai-gateway/server/resolve/resolveService.ts b/web/src/features/ai-gateway/server/resolve/resolveService.tsindex f8e8dca50c..bce51ef904 100644--- a/web/src/features/ai-gateway/server/resolve/resolveService.ts+++ b/web/src/features/ai-gateway/server/resolve/resolveService.ts@@ -116,6 +116,8 @@ export class GatewayResolveService {project_id: ingestionProjectId,key_id: context.apiKeyId,user_id: context.userId,+ key_name: context.apiKeyName,+ user_name: context.userName,key_metadata: context.keyMetadata,provider_connection_id: context.connection.id,},diff --git a/web/src/features/public-api/server/organizationApiKeyRouter.ts b/web/src/features/public-api/server/organizationApiKeyRouter.tsindex 4b7c784f2a..518a70a681 100644--- a/web/src/features/public-api/server/organizationApiKeyRouter.ts+++ b/web/src/features/public-api/server/organizationApiKeyRouter.ts@@ -160,6 +160,12 @@ export const organizationApiKeysRouter = createTRPCRouter({note: input.name,},});+ // Gateway resolve contexts snapshot the key's name onto generations.+ if (apiKey.fastHashedSecretKey) {+ await invalidateGatewayResolveCacheForApiKey(+ apiKey.fastHashedSecretKey,+ );+ }// do not return the api keyreturn;diff --git a/worker/src/services/IngestionService/index.ts b/worker/src/services/IngestionService/index.tsindex 10a5587c86..c1e89cadf1 100644--- a/worker/src/services/IngestionService/index.ts+++ b/worker/src/services/IngestionService/index.ts@@ -75,6 +75,7 @@ import {convertPostgresJsonToMetadataRecord,convertRecordValuesToString,hasAiGatewayScope,+ isAiGatewayGenerationEvent,overwriteObject,} from "./utils";import { randomUUID } from "crypto";@@ -364,8 +365,11 @@ export class IngestionService {modelParameters,metadata,},- isAiGatewayGeneration:- eventData.scopeName === AI_GATEWAY_INSTRUMENTATION_SCOPE_NAME,+ isAiGatewayGeneration: isAiGatewayGenerationEvent({+ ingestionApiKey: eventData.ingestionApiKey,+ hasGatewayScope: () =>+ eventData.scopeName === AI_GATEWAY_INSTRUMENTATION_SCOPE_NAME,+ }),observationRecord: {id: eventData.spanId,project_id: eventData.projectId,@@ -1106,9 +1110,13 @@ export class IngestionService {const generationUsage = await this.getGenerationUsage({projectId,pricingMatchAttributeValues,- isAiGatewayGeneration: timeSortedEvents.some((event) =>- hasAiGatewayScope(event.body?.metadata),- ),+ isAiGatewayGeneration: isAiGatewayGenerationEvent({+ ingestionApiKey: attribution.ingestionApiKey,+ hasGatewayScope: () =>+ timeSortedEvents.some((event) =>+ hasAiGatewayScope(event.body?.metadata),+ ),+ }),observationRecord: mergedObservationRecord,});const finalObservationRecord = {diff --git a/worker/src/services/IngestionService/tests/IngestionService.integration.test.ts b/worker/src/services/IngestionService/tests/IngestionService.integration.test.tsindex 7796da84e0..09cb72a698 100644--- a/worker/src/services/IngestionService/tests/IngestionService.integration.test.ts+++ b/worker/src/services/IngestionService/tests/IngestionService.integration.test.ts@@ -798,12 +798,14 @@ describe("Ingestion end-to-end tests", () => {traceEventList,}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: spanId,createdAtTimestamp: new Date(),observationEventList: spanEventList,}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),@@ -1218,6 +1220,7 @@ describe("Ingestion end-to-end tests", () => {traceEventList,}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),@@ -1382,18 +1385,21 @@ describe("Ingestion end-to-end tests", () => {traceEventList,}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: spanId,createdAtTimestamp: new Date(),observationEventList: spanEventList,}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),observationEventList: generationEventList,}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: eventId,createdAtTimestamp: new Date(),@@ -1853,6 +1859,7 @@ describe("Ingestion end-to-end tests", () => {];await ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: observationId,createdAtTimestamp: new Date(),@@ -1991,6 +1998,7 @@ describe("Ingestion end-to-end tests", () => {];await ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: observationId,createdAtTimestamp: new Date(),@@ -2046,6 +2054,7 @@ describe("Ingestion end-to-end tests", () => {},];await ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: observationId,createdAtTimestamp: new Date(),@@ -2069,6 +2078,7 @@ describe("Ingestion end-to-end tests", () => {},];await ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: observationId,createdAtTimestamp: new Date(),@@ -2121,6 +2131,7 @@ describe("Ingestion end-to-end tests", () => {];await ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),@@ -2191,6 +2202,7 @@ describe("Ingestion end-to-end tests", () => {traceEventList,}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),@@ -2233,6 +2245,7 @@ describe("Ingestion end-to-end tests", () => {];await ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),@@ -2335,6 +2348,7 @@ describe("Ingestion end-to-end tests", () => {traceEventList,}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),@@ -2441,6 +2455,7 @@ describe("Ingestion end-to-end tests", () => {traceEventList,}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),@@ -2632,6 +2647,7 @@ describe("Ingestion end-to-end tests", () => {traceEventList,}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),@@ -2821,6 +2837,7 @@ describe("Ingestion end-to-end tests", () => {];await ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),@@ -2948,6 +2965,7 @@ describe("Ingestion end-to-end tests", () => {],}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),@@ -3088,6 +3106,7 @@ describe("Ingestion end-to-end tests", () => {],}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),@@ -3217,6 +3236,7 @@ describe("Ingestion end-to-end tests", () => {],}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),@@ -3338,6 +3358,7 @@ describe("Ingestion end-to-end tests", () => {],}),ingestionService.processObservationEventList({+ attribution: testIngestionAttribution,projectId,entityId: generationId,createdAtTimestamp: new Date(),diff --git a/worker/src/services/IngestionService/tests/IngestionService.unit.test.ts b/worker/src/services/IngestionService/tests/IngestionService.unit.test.tsindex af46f770ad..081a593a33 100644--- a/worker/src/services/IngestionService/tests/IngestionService.unit.test.ts+++ b/worker/src/services/IngestionService/tests/IngestionService.unit.test.ts@@ -595,6 +595,11 @@ describe("IngestionService unit tests", () => {createdAtTimestamp: new Date(timestamp),observationEventList,writeToStagingTables: false,+ attribution: {+ ingestionApiKey: "pk-lf-test",+ ingestionSdkName: "langfuse-test",+ ingestionSdkVersion: "0.0.0",+ },});const observationRecord = addToQueue.mock.calls.find(diff --git a/worker/src/services/IngestionService/tests/calculateTokenCost.unit.test.ts b/worker/src/services/IngestionService/tests/calculateTokenCost.unit.test.tsindex 8a30d5aa05..12e3a74c69 100644--- a/worker/src/services/IngestionService/tests/calculateTokenCost.unit.test.ts+++ b/worker/src/services/IngestionService/tests/calculateTokenCost.unit.test.ts@@ -72,6 +72,12 @@ vi.mock("../../../features/tokenisation/usage", async (importOriginal) => {};});+const sdkAttribution = {+ ingestionApiKey: "pk-lf-test",+ ingestionSdkName: "unknown",+ ingestionSdkVersion: "unknown",+};+const mockIngestionService = new IngestionService(null as any,prisma,@@ -531,6 +537,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});expect(mockAddToClickhouseWriter).toHaveBeenCalled();const args = mockAddToClickhouseWriter.mock.calls[0];@@ -612,6 +619,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});expect(mockAddToClickhouseWriter).toHaveBeenCalled();const args = mockAddToClickhouseWriter.mock.calls[0];@@ -691,6 +699,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});expect(mockAddToClickhouseWriter).toHaveBeenCalled();const args = mockAddToClickhouseWriter.mock.calls[0];@@ -772,6 +781,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});expect(mockAddToClickhouseWriter).toHaveBeenCalled();const args = mockAddToClickhouseWriter.mock.calls[0];@@ -856,6 +866,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});expect(mockAddToClickhouseWriter).toHaveBeenCalled();const args = mockAddToClickhouseWriter.mock.calls[0];@@ -946,6 +957,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});expect(mockAddToClickhouseWriter).toHaveBeenCalled();const args = mockAddToClickhouseWriter.mock.calls[0];@@ -1031,6 +1043,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});expect(mockAddToClickhouseWriter).toHaveBeenCalled();const args = mockAddToClickhouseWriter.mock.calls[0];@@ -1113,6 +1126,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});expect(mockAddToClickhouseWriter).toHaveBeenCalled();const args = mockAddToClickhouseWriter.mock.calls[0];@@ -1200,6 +1214,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});expect(mockAddToClickhouseWriter).toHaveBeenCalled();const args = mockAddToClickhouseWriter.mock.calls[0];@@ -1272,6 +1287,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});expect(mockAddToClickhouseWriter).toHaveBeenCalled();const args = mockAddToClickhouseWriter.mock.calls[0];@@ -1332,6 +1348,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});expect(mockAddToClickhouseWriter).toHaveBeenCalled();@@ -1364,6 +1381,10 @@ describe("Token Cost Calculation", () => {describe("tokenizer inference on failed and cancelled generations", () => {const gatewayScope = { name: "langfuse-ai-gateway", version: "0.1.0" };+ const gatewayAttribution = {+ ...sdkAttribution,+ ingestionApiKey: "gateway:grant-1",+ };beforeEach(() => {tokenisationMocks.tokenCountAsyncOverride = async () => 7;@@ -1430,6 +1451,7 @@ describe("Token Cost Calculation", () => {statusMessage: "Client cancelled the response",scopeName: gatewayScope.name,metadata: { scope: gatewayScope },+ ingestionApiKey: gatewayAttribution.ingestionApiKey,input: "hello world",output: "[]",},@@ -1441,6 +1463,32 @@ describe("Token Cost Calculation", () => {expect(eventRecord.cost_details).toEqual({});});+ it("should tokenize a generation that only claims the gateway scope on the events table", async () => {+ const eventRecord = await (mockIngestionService as any).createEventRecord(+ {+ spanId: generationId,+ traceId: uuidv4(),+ projectId,+ startTimeISO: new Date().toISOString(),+ type: "GENERATION",+ modelName,+ level: "WARNING",+ scopeName: gatewayScope.name,+ metadata: { scope: gatewayScope },+ ingestionApiKey: sdkAttribution.ingestionApiKey,+ input: "hello world",+ output: "hey whassup",+ },+ "testfile.txt",+ );++ expect(eventRecord.usage_details).toEqual({+ input: 7,+ output: 7,+ total: 14,+ });+ });+it("should cost provider-reported usage on a cancelled gateway generation", async () => {const eventRecord = await (mockIngestionService as any).createEventRecord({@@ -1453,6 +1501,7 @@ describe("Token Cost Calculation", () => {level: "WARNING",scopeName: gatewayScope.name,metadata: { scope: gatewayScope },+ ingestionApiKey: gatewayAttribution.ingestionApiKey,providedUsageDetails: { input: 10, output: 5 },input: "hello world",output: "[]",@@ -1493,6 +1542,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: gatewayAttribution,});const [tableName, generation] = mockAddToClickhouseWriter.mock.calls[0];@@ -1501,6 +1551,99 @@ describe("Token Cost Calculation", () => {expect(generation.usage_details).toEqual({});expect(generation.cost_details).toEqual({});});++ it("should tokenize a generation that only claims the gateway scope on the observations table", async () => {+ const events = [+ {+ id: uuidv4(),+ type: "generation-create",+ timestamp: new Date().toISOString(),+ body: {+ id: generationId,+ startTime: new Date().toISOString(),+ model: modelName,+ level: "WARNING",+ statusMessage: "Client cancelled the response",+ metadata: { scope: gatewayScope },+ input: "hello world",+ output: "hey whassup",+ },+ },+ ];++ await (mockIngestionService as any).processObservationEventList({+ projectId,+ entityId: generationId,+ createdAtTimestamp: new Date(),+ observationEventList: events,+ attribution: sdkAttribution,+ });++ const [tableName, generation] = mockAddToClickhouseWriter.mock.calls[0];+ expect(tableName).toBe("observations");+ expect(generation.internal_model_id).toBe(tokenModelData.id);+ expect(generation.usage_details).toEqual({+ input: 7,+ output: 7,+ total: 14,+ });+ });++ it("should not tokenize a replayed cancelled gateway generation on the events table", async () => {+ const eventRecord = await (mockIngestionService as any).createEventRecord(+ {+ spanId: generationId,+ traceId: uuidv4(),+ projectId,+ startTimeISO: new Date().toISOString(),+ type: "GENERATION",+ modelName,+ level: "WARNING",+ statusMessage: "Client cancelled the response",+ scopeName: gatewayScope.name,+ metadata: { scope: gatewayScope },+ ingestionApiKey: "",+ input: "hello world",+ output: "[]",+ },+ "testfile.txt",+ );++ expect(eventRecord.usage_details).toEqual({});+ expect(eventRecord.cost_details).toEqual({});+ });++ it("should not tokenize a replayed cancelled gateway generation on the observations table", async () => {+ const events = [+ {+ id: uuidv4(),+ type: "generation-create",+ timestamp: new Date().toISOString(),+ body: {+ id: generationId,+ startTime: new Date().toISOString(),+ model: modelName,+ level: "WARNING",+ statusMessage: "Client cancelled the response",+ metadata: { scope: gatewayScope },+ input: "hello world",+ output: "[]",+ },+ },+ ];++ await (mockIngestionService as any).processObservationEventList({+ projectId,+ entityId: generationId,+ createdAtTimestamp: new Date(),+ observationEventList: events,+ attribution: { ...sdkAttribution, ingestionApiKey: "" },+ });++ const [, generation] = mockAddToClickhouseWriter.mock.calls[0];+ expect(generation.usage_details).toEqual({});+ expect(generation.cost_details).toEqual({});+ });});it("should skip tokenization and leave usage details blank when cost details are provided", async () => {@@ -1533,6 +1676,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});expect(mockAddToClickhouseWriter).toHaveBeenCalled();@@ -1679,6 +1823,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});// The event loop must never run tokenization on the main thread@@ -1897,6 +2042,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});const mismatchWarnings = debugSpy.mock.calls.filter(([message]) =>@@ -1936,6 +2082,7 @@ describe("Token Cost Calculation", () => {entityId: generationId,createdAtTimestamp: new Date(),observationEventList: events,+ attribution: sdkAttribution,});const mismatchWarnings = debugSpy.mock.calls.filter(([message]) =>diff --git a/worker/src/services/IngestionService/utils.ts b/worker/src/services/IngestionService/utils.tsindex cf43e2bf0c..9a63bcc882 100644--- a/worker/src/services/IngestionService/utils.ts+++ b/worker/src/services/IngestionService/utils.ts@@ -1,6 +1,9 @@/* eslint-disable no-nested-ternary, @typescript-eslint/no-use-before-define */import { JsonNested, Prisma } from "@langfuse/shared";-import { AI_GATEWAY_INSTRUMENTATION_SCOPE_NAME } from "@langfuse/shared/src/server";+import {+ AI_GATEWAY_INSTRUMENTATION_SCOPE_NAME,+ isAiGatewayIngestionApiKey,+} from "@langfuse/shared/src/server";import { mergeWith, merge } from "lodash";// Theoretically this returns Record<string, unknown>, but it would be hard to align the typing accordingly.@@ -74,6 +77,16 @@ export const hasAiGatewayScope = (metadata: unknown): boolean => {);};+// Any OTLP sender can set the instrumentation scope, so the ingestion API key+// decides. Replayed events carry no key; only those fall back to the scope.+export const isAiGatewayGenerationEvent = (params: {+ ingestionApiKey: string | null | undefined;+ hasGatewayScope: () => boolean;+}): boolean =>+ params.ingestionApiKey+ ? isAiGatewayIngestionApiKey(params.ingestionApiKey)+ : params.hasGatewayScope();+export function overwriteObject(a: {id: string;