Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

feat(ai-gateway): expose resolution context in response headers, spans and logs (#18247)

Hassieb Pakzadd4574d17206c
diff --git a/ai-gateway/README.md b/ai-gateway/README.md
index cb7f04150b..09029b52fc 100644
--- a/ai-gateway/README.md
+++ b/ai-gateway/README.md
@@ -102,7 +102,9 @@ host-only development, set `LANGFUSE_AI_GATEWAY_LISTEN_ADDRESS=127.0.0.1:8080`.
Logs and operational OpenTelemetry exports describe gateway health independently
of the inference generations sent to Langfuse. They do not include request/response
-content, credentials, key metadata, or URL query strings. Health probes are excluded
+content, credentials, key metadata, or URL query strings. They do carry the
+[resolution context](#resolution-context), including the provider connection name
+and the requested model. Health probes are excluded
from request logs and spans.
Use `LANGFUSE_LOG_FORMAT=text` locally and `json` for log collectors. Built-in `tracing-subscriber` formatters handle both modes. JSON event fields
@@ -167,6 +169,33 @@ as authentication failures, and model or token-count routes omit them. Upload is
best-effort, so a dropped generation can leave these IDs unresolved. The gateway
does not set CORS headers, so no `Access-Control-Expose-Headers` is involved.
+### Resolution context
+
+Once Web resolution succeeds, the request also carries its tenant, and once it is
+sent upstream, the provider connection it was sent to. Requests rejected
+before resolution carry none of them. A resolved request that fails before reaching
+the provider, such as a 413 body or 503 execution capacity rejection, carries only
+the tenant. Provider failures before response headers (502/504) still name the
+connection that was tried.
+
+| Response header | Server span field | JSON log key | Value |
+| --- | --- | --- | --- |
+| `langfuse-organization-id` | `langfuse.organization.id` | `organization_id` | Organization of the gateway key |
+| `langfuse-project-id` | `langfuse.project.id` | `project_id` | Ingestion project |
+| `langfuse-provider` | `gateway.provider` | `provider` | Stable lowercase provider type, `openai` or `anthropic` |
+| `langfuse-provider-connection-id` | `gateway.provider.connection.id` | `provider_connection_id` | Opaque provider connection ID |
+| — | `gateway.provider.connection.name` | `provider_connection_name` | The connection's user-chosen name; never sent to callers |
+| — | `gateway.upstream.model` | `upstream_model` | The request's `model`, which the relay forwards unchanged; never sent to callers |
+
+The provider fields describe the latest upstream attempt, so with fallbacks they
+name the connection that served the response, or the last one tried. The model
+comes from the request capture's existing parse, so it is recorded only on the
+generation routes listed above, when the body is uncompressed JSON of at most 5 MiB.
+The span fields are declared empty on the `http.server` span and recorded after
+resolution; child spans do not repeat them. JSON log lines emitted inside the
+request after they are recorded repeat them as top-level keys, like `request_id`.
+They are never metric attributes.
+
Each inference request phase has a child span of the server span so a waterfall
has no unattributed wall time:
@@ -822,6 +851,9 @@ invalid response, even if Web selected it.
| `openai` | `openai.responses`, `openai.chat-completions` | `https://api.openai.com/v1` | `{"type":"Bearer","token":…}` → `Authorization: Bearer` |
| `anthropic` | `anthropic.messages` | `https://api.anthropic.com/v1` | `{"type":"x-api-key","header":"x-api-key","value":…}` → `x-api-key` |
+The connection's `name` is optional in the contract, so the gateway also resolves
+against a Web deployment that predates it. Web always sends it.
+
`ProviderConnection::credential()` exposes the secret in its header position
(`ProviderCredential::Bearer` or `ProviderCredential::XApiKey`) and has no `Debug`
output. Ingestion tokens remain opaque. Resolution caching is a separate slice.
diff --git a/ai-gateway/src/capture/anthropic_messages.rs b/ai-gateway/src/capture/anthropic_messages.rs
index 7a7f44f450..3d158a1210 100644
--- a/ai-gateway/src/capture/anthropic_messages.rs
+++ b/ai-gateway/src/capture/anthropic_messages.rs
@@ -280,6 +280,10 @@ impl AnthropicMessagesCapture {
self.facts.capture_complete = self.request_complete && self.facts.output_complete;
}
+ pub fn requested_model(&self) -> Option<&str> {
+ self.facts.requested_model.as_deref()
+ }
+
pub fn into_facts(mut self) -> ProviderFacts {
if matches!(self.body, ResponseBody::Sse(_)) {
self.end_body();
diff --git a/ai-gateway/src/capture/mod.rs b/ai-gateway/src/capture/mod.rs
index dfedf15781..70337a9813 100644
--- a/ai-gateway/src/capture/mod.rs
+++ b/ai-gateway/src/capture/mod.rs
@@ -82,6 +82,14 @@ impl ProtocolCapture {
}
}
+ fn requested_model(&self) -> Option<&str> {
+ match self {
+ Self::OpenAiResponses(capture) => capture.requested_model(),
+ Self::OpenAiChatCompletions(capture) => capture.requested_model(),
+ Self::AnthropicMessages(capture) => capture.requested_model(),
+ }
+ }
+
fn client_metadata(&self) -> Option<&Map<String, Value>> {
match self {
Self::OpenAiResponses(capture) => capture.client_metadata(),
@@ -197,6 +205,13 @@ impl ExecutionCapture {
self.delivery = Some((telemetry, delivery));
}
+ /// The request's `model`, if the capture could parse it.
+ pub fn requested_model(&self) -> Option<&str> {
+ self.protocol
+ .as_ref()
+ .and_then(ProtocolCapture::requested_model)
+ }
+
pub fn record_response(&mut self, status: u16, headers: &HeaderMap) {
if let Some(protocol) = &mut self.protocol {
self.http_status = Some(status);
diff --git a/ai-gateway/src/capture/openai_chat_completions.rs b/ai-gateway/src/capture/openai_chat_completions.rs
index bb29946883..a7ce3fdef1 100644
--- a/ai-gateway/src/capture/openai_chat_completions.rs
+++ b/ai-gateway/src/capture/openai_chat_completions.rs
@@ -341,6 +341,10 @@ impl OpenAiChatCompletionsCapture {
self.facts.capture_complete = self.request_complete && self.facts.output_complete;
}
+ pub fn requested_model(&self) -> Option<&str> {
+ self.facts.requested_model.as_deref()
+ }
+
pub fn into_facts(mut self) -> ProviderFacts {
if matches!(self.body, ResponseBody::Sse(_)) {
self.end_body();
diff --git a/ai-gateway/src/capture/openai_responses.rs b/ai-gateway/src/capture/openai_responses.rs
index 5270cf489e..34cb05ba62 100644
--- a/ai-gateway/src/capture/openai_responses.rs
+++ b/ai-gateway/src/capture/openai_responses.rs
@@ -100,6 +100,10 @@ impl OpenAiResponsesCapture {
self.facts.capture_complete = self.request_complete && self.facts.output_complete;
}
+ pub fn requested_model(&self) -> Option<&str> {
+ self.facts.requested_model.as_deref()
+ }
+
pub fn into_facts(mut self) -> ProviderFacts {
if matches!(self.body, ResponseBody::Sse(_)) {
self.end_body();
diff --git a/ai-gateway/src/correlation.rs b/ai-gateway/src/correlation.rs
index 62542dbbd0..6416e7583f 100644
--- a/ai-gateway/src/correlation.rs
+++ b/ai-gateway/src/correlation.rs
@@ -1,5 +1,7 @@
//! One gateway request ID per inference request, shared by the caller's response,
//! logs, operational spans, the Web resolution call and the Langfuse generation.
+//! Once resolution succeeds, the request also carries its tenant and the provider
+//! connection it was sent to.
use axum::{
extract::Request,
http::{HeaderMap, HeaderName, HeaderValue},
@@ -7,9 +9,16 @@ use axum::{
response::Response,
};
+use crate::resolution::{ProviderConnection, ResolvedRequestContext};
+
pub(crate) const REQUEST_ID_HEADER: HeaderName = HeaderName::from_static("langfuse-request-id");
const TRACE_ID_HEADER: HeaderName = HeaderName::from_static("langfuse-trace-id");
const OBSERVATION_ID_HEADER: HeaderName = HeaderName::from_static("langfuse-observation-id");
+const ORGANIZATION_ID_HEADER: HeaderName = HeaderName::from_static("langfuse-organization-id");
+const PROJECT_ID_HEADER: HeaderName = HeaderName::from_static("langfuse-project-id");
+const PROVIDER_HEADER: HeaderName = HeaderName::from_static("langfuse-provider");
+const PROVIDER_CONNECTION_ID_HEADER: HeaderName =
+ HeaderName::from_static("langfuse-provider-connection-id");
const CLIENT_REQUEST_ID_HEADER: &str = "x-request-id";
const MAX_CLIENT_REQUEST_ID_BYTES: usize = 256;
@@ -21,11 +30,43 @@ pub(crate) struct GenerationIds {
pub observation_id: String,
}
+#[derive(Clone, Debug)]
+struct Tenant {
+ organization_id: String,
+ project_id: String,
+}
+
+/// The provider connection and model of an upstream attempt. With several
+/// attempts the latest one wins, so it names the connection that served the
+/// response, or the last one tried.
+#[derive(Clone, Debug)]
+pub(crate) struct UpstreamTarget {
+ provider: &'static str,
+ connection_id: String,
+ connection_name: Option<String>,
+ model: Option<String>,
+}
+
+impl UpstreamTarget {
+ pub fn for_connection(connection: &ProviderConnection, model: Option<String>) -> Self {
+ Self {
+ provider: connection.provider().as_str(),
+ connection_id: connection.id().to_owned(),
+ connection_name: connection.name().map(str::to_owned),
+ model,
+ }
+ }
+}
+
#[derive(Clone, Debug)]
pub(crate) struct RequestCorrelation {
id: String,
client_id: Option<String>,
generation: Option<GenerationIds>,
+ tenant: Option<Tenant>,
+ upstream: Option<UpstreamTarget>,
+ /// The request's `http.server` span, which outlives the phase spans.
+ server_span: tracing::Span,
}
impl Default for RequestCorrelation {
@@ -34,6 +75,9 @@ impl Default for RequestCorrelation {
id: uuid::Uuid::now_v7().hyphenated().to_string(),
client_id: None,
generation: None,
+ tenant: None,
+ upstream: None,
+ server_span: tracing::Span::none(),
}
}
}
@@ -75,6 +119,55 @@ impl RequestCorrelation {
self.generation = Some(ids);
}
+ pub fn record_resolution(&mut self, context: &ResolvedRequestContext) {
+ let attribution = context.attribution();
+ self.server_span
+ .record("langfuse.organization.id", attribution.organization_id());
+ self.server_span
+ .record("langfuse.project.id", attribution.project_id());
+ self.tenant = Some(Tenant {
+ organization_id: attribution.organization_id().to_owned(),
+ project_id: attribution.project_id().to_owned(),
+ });
+ }
+
+ pub fn record_upstream(&mut self, target: UpstreamTarget) {
+ let span = &self.server_span;
+ span.record("gateway.provider", target.provider);
+ span.record(
+ "gateway.provider.connection.id",
+ target.connection_id.as_str(),
+ );
+ if let Some(name) = &target.connection_name {
+ span.record("gateway.provider.connection.name", name.as_str());
+ }
+ if let Some(model) = &target.model {
+ span.record("gateway.upstream.model", model.as_str());
+ }
+ self.upstream = Some(target);
+ }
+
+ /// Only a resolved request advertises its tenant, and only a request sent
+ /// upstream its provider. The connection name and model stay out of responses.
+ pub fn apply_resolution_headers(&self, headers: &mut HeaderMap) {
+ let Some(tenant) = &self.tenant else {
+ return;
+ };
+ let mut values = vec![
+ (ORGANIZATION_ID_HEADER, tenant.organization_id.as_str()),
+ (PROJECT_ID_HEADER, tenant.project_id.as_str()),
+ ];
+ if let Some(upstream) = &self.upstream {
+ values.push((PROVIDER_HEADER, upstream.provider));
+ values.push((PROVIDER_CONNECTION_ID_HEADER, &upstream.connection_id));
+ }
+ for (name, value) in values {
+ if let Ok(value) = HeaderValue::from_str(value) {
+ headers.insert(name, value);
+ }
+ }
+ }
+
/// Only a request that will produce a Langfuse generation advertises its IDs.
pub fn apply_generation_headers(&self, headers: &mut HeaderMap) {
let Some(generation) = &self.generation else {
@@ -94,12 +187,13 @@ impl RequestCorrelation {
/// Runs inside the server span before any rejection, so every response, log line
/// and phase span of the request can carry the ID.
pub(crate) async fn assign_request_id(mut request: Request, next: Next) -> Response {
- let correlation = RequestCorrelation::from_headers(request.headers());
+ let mut correlation = RequestCorrelation::from_headers(request.headers());
let span = tracing::Span::current();
span.record("gateway.request.id", correlation.id());
if let Some(client_id) = correlation.client_id() {
span.record("gateway.client.request.id", client_id);
}
+ correlation.server_span = span;
let value = HeaderValue::from_str(correlation.id()).expect("a UUID is a valid header value");
request.extensions_mut().insert(correlation);
let mut response = next.run(request).await;
diff --git a/ai-gateway/src/http.rs b/ai-gateway/src/http.rs
index f1b803a58b..261b929a60 100644
--- a/ai-gateway/src/http.rs
+++ b/ai-gateway/src/http.rs
@@ -105,6 +105,7 @@ async fn handle(state: InferenceRouteState, mut request: Request, route: Route)
Err(error) => error.into_native_response(route.api_format(), correlation.id()),
};
correlation.apply_generation_headers(response.headers_mut());
+ correlation.apply_resolution_headers(response.headers_mut());
response
}
@@ -121,7 +122,7 @@ async fn relay(
.ok_or(InferenceHttpError::Unavailable)?;
let gateway_key = gateway_key(request.headers(), route.api_format())?.to_owned();
let (permit, context) = inference
- .resolve_and_admit(&gateway_key, route.api_format(), correlation.id())
+ .resolve_and_admit(&gateway_key, route.api_format(), correlation)
.await
.map_err(|error| match error {
RequestPreparationError::Resolution(error) => InferenceHttpError::Resolution(error),
diff --git a/ai-gateway/src/http/tests.rs b/ai-gateway/src/http/tests.rs
index ed83c74f62..267b4f2358 100644
--- a/ai-gateway/src/http/tests.rs
+++ b/ai-gateway/src/http/tests.rs
@@ -216,6 +216,23 @@ fn request_id(response: &reqwest::Response) -> String {
id
}
+/// A request that never reaches the provider has no generation, and names only
+/// its tenant, and only if it was resolved.
+fn assert_pre_upstream_headers(headers: &reqwest::header::HeaderMap, resolved: bool) {
+ assert!(!headers.contains_key("langfuse-trace-id"));
+ assert!(!headers.contains_key("langfuse-observation-id"));
+ for name in ["langfuse-organization-id", "langfuse-project-id"] {
+ assert_eq!(headers.contains_key(name), resolved, "{name}");
+ }
+ for name in [
+ "langfuse-provider",
+ "langfuse-provider-connection-id",
+ "langfuse-model",
+ ] {
+ assert!(!headers.contains_key(name), "{name}");
+ }
+}
+
#[tokio::test]
async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body() {
let web = FakeServer::start(|request| async move {
@@ -240,23 +257,31 @@ async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body()
let unconfigured = Gateway::start(None).await;
let configured = Gateway::start(Some(inference(&web, &provider))).await;
let mut seen = std::collections::HashSet::new();
- for (request, status, anthropic) in [
- (unconfigured.post(), StatusCode::SERVICE_UNAVAILABLE, false),
+ for (request, status, anthropic, resolved) in [
+ (
+ unconfigured.post(),
+ StatusCode::SERVICE_UNAVAILABLE,
+ false,
+ false,
+ ),
(
unconfigured.messages(),
StatusCode::SERVICE_UNAVAILABLE,
true,
+ false,
),
- (configured.post(), StatusCode::UNAUTHORIZED, false),
+ (configured.post(), StatusCode::UNAUTHORIZED, false, false),
(
unconfigured.chat_completions(),
StatusCode::SERVICE_UNAVAILABLE,
false,
+ false,
),
(
configured.chat_completions().bearer_auth("rejected"),
StatusCode::UNAUTHORIZED,
false,
+ false,
),
(
configured
@@ -265,16 +290,19 @@ async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body()
.body(vec![b'x'; MAX_REQUEST_BYTES + 1]),
StatusCode::PAYLOAD_TOO_LARGE,
false,
+ true,
),
(
configured.post().bearer_auth("rejected"),
StatusCode::UNAUTHORIZED,
false,
+ false,
),
(
configured.messages().header("x-api-key", "rejected"),
StatusCode::UNAUTHORIZED,
true,
+ false,
),
(
configured
@@ -283,6 +311,7 @@ async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body()
.body(vec![b'x'; MAX_REQUEST_BYTES + 1]),
StatusCode::PAYLOAD_TOO_LARGE,
false,
+ true,
),
] {
let response = request
@@ -292,8 +321,7 @@ async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body()
.unwrap();
assert_eq!(response.status(), status);
let id = request_id(&response);
- assert!(!response.headers().contains_key("langfuse-trace-id"));
- assert!(!response.headers().contains_key("langfuse-observation-id"));
+ assert_pre_upstream_headers(response.headers(), resolved);
let body: serde_json::Value =
serde_json::from_slice(&response.bytes().await.unwrap()).unwrap();
assert_eq!(body["request_id"], id.as_str());
@@ -308,6 +336,65 @@ async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body()
assert_eq!(provider.calls(), 0);
}
+#[tokio::test]
+async fn resolved_requests_name_their_tenant_and_the_provider_connection_sent_to() {
+ let web = FakeServer::start(|request| async move {
+ let body = to_bytes(request.into_body(), 1024).await.unwrap();
+ let api_format =
+ serde_json::from_slice::<serde_json::Value>(&body).unwrap()["apiFormat"].clone();
+ resolution_response_for(
+ serde_json::from_value(api_format).unwrap(),
+ "provider-secret",
+ )
+ })
+ .await;
+ let provider = FakeServer::start(|_| async {
+ Response::builder()
+ .header("content-type", "application/json")
+ .body(Body::from("{}"))
+ .unwrap()
+ })
+ .await;
+ let gateway = Gateway::start(Some(inference(&web, &provider))).await;
+ let client = reqwest::Client::new();
+ let anthropic = |path: &str, body: &'static str| {
+ client
+ .post(format!("{}/anthropic/v1/{path}", gateway.url))
+ .header("x-api-key", "gateway-anthropic")
+ .header("content-type", "application/json")
+ .body(body)
+ };
+ for (request, provider_type) in [
+ (
+ anthropic(
+ "messages/count_tokens",
+ r#"{"messages":[],"model":"claude-opus-4-1"}"#,
+ ),
+ "anthropic",
+ ),
+ (gateway.models().bearer_auth("gateway-openai"), "openai"),
+ (
+ anthropic("messages", r#"{"model":"claude-opus-4-1","messages":[]}"#),
+ "anthropic",
+ ),
+ ] {
+ let response = request.send().await.unwrap();
+ assert_eq!(response.status(), StatusCode::OK);
+ let headers = response.headers();
+ let header = |name: &str| headers.get(name).map(|value| value.to_str().unwrap());
+ assert_eq!(header("langfuse-organization-id"), Some("org-1"));
+ assert_eq!(header("langfuse-project-id"), Some("project-1"));
+ assert_eq!(header("langfuse-provider"), Some(provider_type));
+ assert_eq!(
+ header("langfuse-provider-connection-id"),
+ Some("connection-1")
+ );
+ // The model is an operator-side span and log field only.
+ assert_eq!(header("langfuse-model"), None);
+ }
+ assert_eq!(provider.calls(), 3);
+}
+
impl Drop for Gateway {
fn drop(&mut self) {
self.task.abort();
diff --git a/ai-gateway/src/inference.rs b/ai-gateway/src/inference.rs
index 0a963c9054..4c13ca773f 100644
--- a/ai-gateway/src/inference.rs
+++ b/ai-gateway/src/inference.rs
@@ -57,11 +57,12 @@ impl InferenceService {
/// Authenticate with a separate bounded budget before reserving execution capacity.
/// The API format comes from the public route, so Web selects a compatible connection.
+ /// A successful resolution is recorded even if execution admission then fails.
pub(crate) async fn resolve_and_admit(
&self,
gateway_key: &str,
api_format: ApiFormat,
- request_id: &str,
+ correlation: &mut RequestCorrelation,
) -> Result<(RequestPermit, ResolvedRequestContext), RequestPreparationError> {
let span = tracing::info_span!(
"resolution",
@@ -69,7 +70,7 @@ impl InferenceService {
gateway.outcome = tracing::field::Empty
);
async {
- let prepared = self.prepare(gateway_key, api_format, request_id).await;
+ let prepared = self.prepare(gateway_key, api_format, correlation).await;
tracing::Span::current().record(
"gateway.outcome",
match &prepared {
@@ -88,7 +89,7 @@ impl InferenceService {
&self,
gateway_key: &str,
api_format: ApiFormat,
- request_id: &str,
+ correlation: &mut RequestCorrelation,
) -> Result<(RequestPermit, ResolvedRequestContext), RequestPreparationError> {
let context = {
let _permit = self.resolution_capacity.try_acquire().map_err(|_| {
@@ -97,10 +98,11 @@ impl InferenceService {
})?;
let _active = crate::observability::Active::new("resolution");
self.control_plane
- .resolve(gateway_key, api_format, request_id)
+ .resolve(gateway_key, api_format, correlation.id())
.await
.map_err(RequestPreparationError::Resolution)?
};
+ correlation.record_resolution(&context);
let permit = self
.provider
.try_admit()
diff --git a/ai-gateway/src/observability/http.rs b/ai-gateway/src/observability/http.rs
index e379e6e09e..67c116281a 100644
--- a/ai-gateway/src/observability/http.rs
+++ b/ai-gateway/src/observability/http.rs
@@ -21,7 +21,7 @@ pub fn instrument(router: Router) -> Router {
TraceLayer::new_for_http()
.make_span_with(|request: &Request<Body>| {
let (method, route) = request_labels(request);
- let span = tracing::info_span!(parent: None, "http.server", otel.name = %format_args!("{method} {route}"), otel.kind = "server", http.request.method = method, http.route = route, http.request.body.size = tracing::field::Empty, http.response.status_code = tracing::field::Empty, otel.status_code = tracing::field::Empty, gateway.request.id = tracing::field::Empty, gateway.client.request.id = tracing::field::Empty, provider_request_id = tracing::field::Empty, gateway.outcome = tracing::field::Empty, gateway.first_byte_ms = tracing::field::Empty);
+ let span = tracing::info_span!(parent: None, "http.server", otel.name = %format_args!("{method} {route}"), otel.kind = "server", http.request.method = method, http.route = route, http.request.body.size = tracing::field::Empty, http.response.status_code = tracing::field::Empty, otel.status_code = tracing::field::Empty, gateway.request.id = tracing::field::Empty, gateway.client.request.id = tracing::field::Empty, langfuse.organization.id = tracing::field::Empty, langfuse.project.id = tracing::field::Empty, gateway.provider = tracing::field::Empty, gateway.provider.connection.id = tracing::field::Empty, gateway.provider.connection.name = tracing::field::Empty, gateway.upstream.model = tracing::field::Empty, provider_request_id = tracing::field::Empty, gateway.outcome = tracing::field::Empty, gateway.first_byte_ms = tracing::field::Empty);
// Operational tracing has no caller or ambient trace context.
let _ = span.set_parent(Context::new());
span
diff --git a/ai-gateway/src/observability/http/tests.rs b/ai-gateway/src/observability/http/tests.rs
index fb90de8281..25fd158371 100644
--- a/ai-gateway/src/observability/http/tests.rs
+++ b/ai-gateway/src/observability/http/tests.rs
@@ -57,11 +57,11 @@ impl Recording {
tracing_subscriber::fmt::layer()
.json()
.flatten_event(true)
- .map_event_format(crate::observability::logs::WithRequestId)
+ .map_event_format(crate::observability::logs::WithCorrelationFields)
.with_writer(move || buffer.clone()),
)
.with(tracing_opentelemetry::layer().with_tracer(provider.tracer("test")))
- .with(crate::observability::logs::RequestIdSpans);
+ .with(crate::observability::logs::CorrelationSpans);
Self {
_guard: tracing::subscriber::set_default(subscriber),
_provider: provider,
@@ -510,6 +510,39 @@ fn assert_request_correlation(
assert_attribute(server, "gateway.request.id", request_id.to_owned());
assert_attribute(server, "gateway.client.request.id", "client-request-canary");
assert_attribute(server, "provider_request_id", "req_upstream");
+ let resolution = [
+ ("langfuse.organization.id", "organization_id", "org-1"),
+ ("langfuse.project.id", "project_id", "project-1"),
+ ("gateway.provider", "provider", "openai"),
+ (
+ "gateway.provider.connection.id",
+ "provider_connection_id",
+ "connection-1",
+ ),
+ (
+ "gateway.provider.connection.name",
+ "provider_connection_name",
+ "Production key",
+ ),
+ ("gateway.upstream.model", "upstream_model", "test-model"),
+ ];
+ for (attribute, _, expected) in resolution {
+ assert_attribute(server, attribute, expected);
+ }
+ for (name, expected) in [
+ ("langfuse-organization-id", "org-1"),
+ ("langfuse-project-id", "project-1"),
+ ("langfuse-provider", "openai"),
+ ("langfuse-provider-connection-id", "connection-1"),
+ ] {
+ assert_eq!(header(name), expected, "{name}");
+ }
+ assert!(response.values().all(|value| {
+ value
+ .to_str()
+ .is_ok_and(|value| !value.contains("Production key"))
+ }));
+ assert!(!response.contains_key("langfuse-model"));
let events: Vec<_> = recording
.events()
.into_iter()
@@ -525,6 +558,11 @@ fn assert_request_correlation(
}
for event in &events {
assert_eq!(event["request_id"], request_id, "{event}");
+ if event["message"] == "gateway response started" {
+ for (_, key, expected) in resolution {
+ assert_eq!(event[key], expected, "{event}");
+ }
+ }
}
for (name, headers, payload) in observed {
assert_eq!(
diff --git a/ai-gateway/src/observability/logs.rs b/ai-gateway/src/observability/logs.rs
index 195da09237..8097079ba8 100644
--- a/ai-gateway/src/observability/logs.rs
+++ b/ai-gateway/src/observability/logs.rs
@@ -1,5 +1,6 @@
-//! JSON span fields are nested, so the request ID is repeated as a top-level
-//! `request_id` on every event emitted inside a request, for log search.
+//! JSON span fields are nested, so the request's correlation fields are repeated
+//! as top-level keys, such as `request_id`, on every event emitted inside a
+//! request, for log search.
use std::fmt;
use tracing::{
@@ -13,55 +14,85 @@ use tracing_subscriber::{
registry::LookupSpan,
};
-const REQUEST_ID_FIELD: &str = "gateway.request.id";
+/// Span field and its top-level log key. All are recorded on the server span.
+const CORRELATION_FIELDS: [(&str, &str); 7] = [
+ ("gateway.request.id", "request_id"),
+ ("langfuse.organization.id", "organization_id"),
+ ("langfuse.project.id", "project_id"),
+ ("gateway.provider", "provider"),
+ ("gateway.provider.connection.id", "provider_connection_id"),
+ (
+ "gateway.provider.connection.name",
+ "provider_connection_name",
+ ),
+ ("gateway.upstream.model", "upstream_model"),
+];
-struct LoggedRequestId(String);
+#[derive(Default)]
+struct LoggedFields([Option<String>; CORRELATION_FIELDS.len()]);
-/// Keeps each span's `gateway.request.id` where [`WithRequestId`] can find it.
-pub(crate) struct RequestIdSpans;
+/// Keeps each span's correlation fields where [`WithCorrelationFields`] can find them.
+pub(crate) struct CorrelationSpans;
-impl<S> Layer<S> for RequestIdSpans
+impl<S> Layer<S> for CorrelationSpans
where
S: Subscriber + for<'a> LookupSpan<'a>,
{
fn on_new_span(&self, attributes: &span::Attributes<'_>, id: &span::Id, ctx: Context<'_, S>) {
- let mut visitor = RequestIdVisitor(None);
+ let mut visitor = CorrelationVisitor::default();
attributes.record(&mut visitor);
- store(id, &ctx, visitor.0);
+ store(id, &ctx, visitor);
}
fn on_record(&self, id: &span::Id, values: &span::Record<'_>, ctx: Context<'_, S>) {
- let mut visitor = RequestIdVisitor(None);
+ let mut visitor = CorrelationVisitor::default();
values.record(&mut visitor);
- store(id, &ctx, visitor.0);
+ store(id, &ctx, visitor);
}
}
-fn store<S>(id: &span::Id, ctx: &Context<'_, S>, request_id: Option<String>)
+fn store<S>(id: &span::Id, ctx: &Context<'_, S>, visitor: CorrelationVisitor)
where
S: Subscriber + for<'a> LookupSpan<'a>,
{
- if let (Some(request_id), Some(span)) = (request_id, ctx.span(id)) {
- span.extensions_mut().replace(LoggedRequestId(request_id));
+ if visitor.0.0.iter().all(Option::is_none) {
+ return;
}
+ let Some(span) = ctx.span(id) else {
+ return;
+ };
+ let mut extensions = span.extensions_mut();
+ if let Some(logged) = extensions.get_mut::<LoggedFields>() {
+ for (slot, value) in logged.0.iter_mut().zip(visitor.0.0) {
+ if value.is_some() {
+ *slot = value;
+ }
+ }
+ return;
+ }
+ extensions.insert(visitor.0);
}
-struct RequestIdVisitor(Option<String>);
+#[derive(Default)]
+struct CorrelationVisitor(LoggedFields);
-impl Visit for RequestIdVisitor {
+impl Visit for CorrelationVisitor {
fn record_str(&mut self, field: &Field, value: &str) {
- if field.name() == REQUEST_ID_FIELD {
- self.0 = Some(value.to_owned());
+ if let Some(index) = CORRELATION_FIELDS
+ .iter()
+ .position(|(name, _)| *name == field.name())
+ {
+ self.0.0[index] = Some(value.to_owned());
}
}
fn record_debug(&mut self, _: &Field, _: &dyn fmt::Debug) {}
}
-/// Wraps a JSON event formatter and prepends the innermost span's request ID.
-pub(crate) struct WithRequestId<F>(pub F);
+/// Wraps a JSON event formatter and prepends the innermost span's correlation fields.
+pub(crate) struct WithCorrelationFields<F>(pub F);
-impl<S, N, F> FormatEvent<S, N> for WithRequestId<F>
+impl<S, N, F> FormatEvent<S, N> for WithCorrelationFields<F>
where
S: Subscriber + for<'a> LookupSpan<'a>,
N: for<'a> FormatFields<'a> + 'static,
@@ -73,14 +104,14 @@ where
mut writer: Writer<'_>,
event: &Event<'_>,
) -> fmt::Result {
- let request_id = ctx.event_scope().and_then(|scope| {
+ let fields = ctx.event_scope().and_then(|scope| {
scope.into_iter().find_map(|span| {
span.extensions()
- .get::<LoggedRequestId>()
- .map(|id| id.0.clone())
+ .get::<LoggedFields>()
+ .map(|fields| fields.0.clone())
})
});
- let Some(request_id) = request_id else {
+ let Some(fields) = fields else {
return self.0.format_event(ctx, writer, event);
};
let mut line = String::new();
@@ -88,9 +119,20 @@ where
let Some(rest) = line.strip_prefix('{') else {
return writer.write_str(&line);
};
- let request_id = serde_json::to_string(&request_id).map_err(|_| fmt::Error)?;
- write!(writer, "{{\"request_id\":{request_id}")?;
- if !rest.starts_with('}') {
+ writer.write_char('{')?;
+ let mut first = true;
+ for ((_, key), value) in CORRELATION_FIELDS.iter().zip(fields) {
+ let Some(value) = value else {
+ continue;
+ };
+ if !first {
+ writer.write_char(',')?;
+ }
+ first = false;
+ let value = serde_json::to_string(&value).map_err(|_| fmt::Error)?;
+ write!(writer, "\"{key}\":{value}")?;
+ }
+ if !first && !rest.starts_with('}') {
writer.write_char(',')?;
}
writer.write_str(rest)
diff --git a/ai-gateway/src/observability/mod.rs b/ai-gateway/src/observability/mod.rs
index 1372de5436..45012fa99e 100644
--- a/ai-gateway/src/observability/mod.rs
+++ b/ai-gateway/src/observability/mod.rs
@@ -103,7 +103,7 @@ pub fn init(config: &GatewayConfig) -> Result<Observability, Box<dyn Error>> {
LogFormat::Json => tracing_subscriber::fmt::layer()
.json()
.flatten_event(true)
- .map_event_format(logs::WithRequestId)
+ .map_event_format(logs::WithCorrelationFields)
.with_filter(log_filter)
.boxed(),
LogFormat::Text => tracing_subscriber::fmt::layer()
@@ -124,7 +124,7 @@ pub fn init(config: &GatewayConfig) -> Result<Observability, Box<dyn Error>> {
tracing_subscriber::registry()
.with(logger)
.with(spans)
- .with(logs::RequestIdSpans)
+ .with(logs::CorrelationSpans)
.try_init()
.map_err(|_| "failed to initialize gateway logging")?;
Ok(Observability { traces, metrics })
diff --git a/ai-gateway/src/providers/mod.rs b/ai-gateway/src/providers/mod.rs
index 5e5871627b..3c0202f94b 100644
--- a/ai-gateway/src/providers/mod.rs
+++ b/ai-gateway/src/providers/mod.rs
@@ -17,7 +17,7 @@ use tokio::{
pub use crate::transport::ProviderError;
use crate::{
capture::{ExecutionCapture, RelayOutcome},
- correlation::RequestCorrelation,
+ correlation::{RequestCorrelation, UpstreamTarget},
resolution::{ApiFormat, Provider, ProviderCredential, ResolvedRequestContext},
transport,
};
@@ -233,6 +233,9 @@ impl ProviderTransport {
}
_ => body,
};
+ // The relay forwards the caller's `model` unchanged; only a captured route has parsed it.
+ let model = capture.requested_model().map(str::to_owned);
+ correlation.record_upstream(UpstreamTarget::for_connection(context.connection(), model));
let mut upstream = self
.client
.request(route.method(), self.request_url(route, query))
diff --git a/ai-gateway/src/resolution/contracts.rs b/ai-gateway/src/resolution/contracts.rs
index c296f2f00c..5e3ec18975 100644
--- a/ai-gateway/src/resolution/contracts.rs
+++ b/ai-gateway/src/resolution/contracts.rs
@@ -24,6 +24,14 @@ pub enum Provider {
}
impl Provider {
+ /// The stable lowercase provider type, as Web sends it.
+ pub fn as_str(self) -> &'static str {
+ match self {
+ Self::OpenAi => "openai",
+ Self::Anthropic => "anthropic",
+ }
+ }
+
pub fn official_origin(self) -> &'static str {
match self {
Self::OpenAi => "https://api.openai.com/v1",
@@ -130,6 +138,9 @@ impl Auth {
#[serde(deny_unknown_fields)]
pub struct ProviderConnection {
id: String,
+ // Optional so the gateway keeps resolving against a Web that predates it.
+ #[serde(default)]
+ name: Option<String>,
#[serde(deserialize_with = "string_enum")]
provider: Provider,
#[serde(deserialize_with = "string_enum")]
@@ -142,6 +153,10 @@ impl ProviderConnection {
pub fn id(&self) -> &str {
&self.id
}
+ /// The user-chosen display name. Never send it to callers; it is for operators.
+ pub fn name(&self) -> Option<&str> {
+ self.name.as_deref().filter(|name| !name.trim().is_empty())
+ }
pub fn provider(&self) -> Provider {
self.provider
}
diff --git a/ai-gateway/src/resolution/tests.rs b/ai-gateway/src/resolution/tests.rs
index 59aa61e038..009cc19513 100644
--- a/ai-gateway/src/resolution/tests.rs
+++ b/ai-gateway/src/resolution/tests.rs
@@ -95,6 +95,7 @@ fn success(project: &str, provider_token: &str) -> Value {
"version": 1,
"connection": {
"id": "connection-1",
+ "name": "Primary OpenAI",
"provider": "openai",
"api_format": "openai.responses",
"base_url": "https://api.openai.com/v1",
@@ -138,7 +139,9 @@ async fn signs_the_exact_key_and_sends_only_the_api_format() {
.await
.unwrap();
assert_eq!(context.connection().id(), "connection-1");
+ assert_eq!(context.connection().name(), Some("Primary OpenAI"));
assert_eq!(context.connection().provider(), Provider::OpenAi);
+ assert_eq!(context.connection().provider().as_str(), "openai");
assert!(matches!(
context.connection().credential(),
ProviderCredential::Bearer("private-provider-token")
@@ -407,7 +410,10 @@ async fn resolves_anthropic_messages_with_the_native_credential_scheme() {
.await
.unwrap();
assert_eq!(context.connection().id(), "connection-anthropic");
+ // Web deployments that predate connection names omit the field.
+ assert_eq!(context.connection().name(), None);
assert_eq!(context.connection().provider(), Provider::Anthropic);
+ assert_eq!(context.connection().provider().as_str(), "anthropic");
assert_eq!(
context.connection().api_format(),
ApiFormat::AnthropicMessages
diff --git a/ai-gateway/src/test_support.rs b/ai-gateway/src/test_support.rs
index 23606f7f9b..37b59075bb 100644
--- a/ai-gateway/src/test_support.rs
+++ b/ai-gateway/src/test_support.rs
@@ -99,17 +99,17 @@ pub(crate) fn resolution_response_for(
+ 300;
let connection = match api_format {
ApiFormat::OpenAiResponses => json!({
- "id": "connection-1", "provider": "openai", "api_format": "openai.responses",
+ "id": "connection-1", "name": "Production key", "provider": "openai", "api_format": "openai.responses",
"base_url": "https://api.openai.com/v1",
"auth": {"type": "Bearer", "token": provider_secret}
}),
ApiFormat::OpenAiChatCompletions => json!({
- "id": "connection-1", "provider": "openai", "api_format": "openai.chat-completions",
+ "id": "connection-1", "name": "Production key", "provider": "openai", "api_format": "openai.chat-completions",
"base_url": "https://api.openai.com/v1",
"auth": {"type": "Bearer", "token": provider_secret}
}),
ApiFormat::AnthropicMessages => json!({
- "id": "connection-1", "provider": "anthropic", "api_format": "anthropic.messages",
+ "id": "connection-1", "name": "Production key", "provider": "anthropic", "api_format": "anthropic.messages",
"base_url": "https://api.anthropic.com/v1",
"auth": {"type": "x-api-key", "header": "x-api-key", "value": provider_secret}
}),
diff --git a/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts b/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts
index 50d5b68cfe..b4409e8b89 100644
--- a/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts
+++ b/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts
@@ -554,6 +554,7 @@ describe("AI gateway control plane", () => {
expect(openAiPrimary.routingPriority).toBe(0);
expect(result.connection).toEqual({
id: openAiPrimary.id,
+ name: "OpenAI first",
provider: "openai",
api_format: apiFormat,
base_url: "https://api.openai.com/v1",
diff --git a/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts b/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts
index 9819044ce1..44e22d1a30 100644
--- a/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts
+++ b/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts
@@ -94,6 +94,7 @@ export class GatewayApiKeyAuthenticator {
connection: connection
? {
id: connection.id,
+ name: connection.name,
provider: connection.provider,
encryptedCredential: connection.encryptedCredential,
}
diff --git a/web/src/features/ai-gateway/server/provider/registry.ts b/web/src/features/ai-gateway/server/provider/registry.ts
index e0e6f51c2d..c9bb3b20df 100644
--- a/web/src/features/ai-gateway/server/provider/registry.ts
+++ b/web/src/features/ai-gateway/server/provider/registry.ts
@@ -25,6 +25,7 @@ export const GatewayResolveResponseSchema = z
connection: z
.object({
id: z.string(),
+ name: z.string(),
provider: z.enum(gatewayProviderIds),
api_format: GatewayApiFormatSchema,
base_url: z.url(),
diff --git a/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts b/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts
index e8c4722cd0..5186f1ce98 100644
--- a/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts
+++ b/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts
@@ -32,6 +32,7 @@ export const CachedResolveContext = z.object({
connection: z
.object({
id: z.string(),
+ name: z.string(),
provider: z.enum(["OPENAI", "ANTHROPIC"]),
encryptedCredential: z.string(),
})
diff --git a/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts b/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts
index c9c20502c2..0351dde840 100644
--- a/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts
+++ b/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts
@@ -44,6 +44,7 @@ export class GatewayResolveRepository {
take: 1,
select: {
id: true,
+ name: true,
provider: true,
encryptedCredential: true,
},
diff --git a/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts b/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts
index 001f5255d6..faf755d079 100644
--- a/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts
+++ b/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts
@@ -70,6 +70,7 @@ const row = (overrides: {
gatewayAiConnections: [
{
id: "connection-1",
+ name: "Production OpenAI",
provider: "OPENAI",
encryptedCredential: "encrypted-credential",
},
@@ -95,6 +96,7 @@ describe("GatewayResolveService", () => {
version: 1,
connection: {
id: "connection-1",
+ name: "Production OpenAI",
provider: "openai",
api_format: "openai.responses",
base_url: "https://api.openai.com/v1",
diff --git a/web/src/features/ai-gateway/server/resolve/resolveService.ts b/web/src/features/ai-gateway/server/resolve/resolveService.ts
index 668c3a594e..6c774d0d3a 100644
--- a/web/src/features/ai-gateway/server/resolve/resolveService.ts
+++ b/web/src/features/ai-gateway/server/resolve/resolveService.ts
@@ -95,6 +95,7 @@ export class GatewayResolveService {
version: 1 as const,
connection: {
id: context.connection.id,
+ name: context.connection.name,
provider:
context.connection.provider.toLowerCase() as GatewayProviderId,
api_format: params.apiFormat,