Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

feat(ai-gateway): batch inference telemetry per project and retry transient failures (#17855)

Hassieb Pakzad943272ee1845
diff --git a/ai-gateway/Cargo.lock b/ai-gateway/Cargo.lock
index 54ac5b80a9..6ee528b0f3 100644
--- a/ai-gateway/Cargo.lock
+++ b/ai-gateway/Cargo.lock
@@ -2,6 +2,12 @@
# It is not intended for manual editing.
version = 4
+[[package]]
+name = "adler2"
+version = "2.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
+
[[package]]
name = "aho-corasick"
version = "1.1.5"
@@ -16,7 +22,9 @@ name = "ai-gateway"
version = "0.1.0"
dependencies = [
"axum",
+ "base64 0.22.1",
"chrono",
+ "flate2",
"futures-util",
"hmac",
"http-body",
@@ -280,6 +288,15 @@ dependencies = [
"libc",
]
+[[package]]
+name = "crc32fast"
+version = "1.5.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "01a7799fd6b852db0e61728dde9a204c423b44d689dbd432522543614b490e78"
+dependencies = [
+ "cfg-if",
+]
+
[[package]]
name = "crypto-common"
version = "0.2.2"
@@ -349,6 +366,17 @@ version = "0.1.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3e0f1c7c3a72c66fd80abe965175f7523475c0489a87d3ff9d6e8c87d87a9d2d"
+[[package]]
+name = "flate2"
+version = "1.1.10"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb"
+dependencies = [
+ "crc32fast",
+ "miniz_oxide",
+ "zlib-rs",
+]
+
[[package]]
name = "form_urlencoded"
version = "1.2.2"
@@ -849,6 +877,16 @@ version = "0.3.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
+[[package]]
+name = "miniz_oxide"
+version = "0.9.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b63fbc4a50860e98e7b2aa7804ded1db5cbc3aff9193adaff57a6931bf7c4b4c"
+dependencies = [
+ "adler2",
+ "simd-adler32",
+]
+
[[package]]
name = "mio"
version = "1.2.3"
@@ -1494,6 +1532,12 @@ dependencies = [
"libc",
]
+[[package]]
+name = "simd-adler32"
+version = "0.3.10"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea"
+
[[package]]
name = "simd_cesu8"
version = "1.2.0"
@@ -2230,6 +2274,12 @@ dependencies = [
"syn 3.0.5",
]
+[[package]]
+name = "zlib-rs"
+version = "0.6.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b268e58e7c693d7c271f93ffc4ba3b380412554231c85bf61ca7af91042a4112"
+
[[package]]
name = "zmij"
version = "1.0.23"
diff --git a/ai-gateway/Cargo.toml b/ai-gateway/Cargo.toml
index 409ea68988..3090041c22 100644
--- a/ai-gateway/Cargo.toml
+++ b/ai-gateway/Cargo.toml
@@ -7,7 +7,9 @@ license = "MIT"
[dependencies]
axum = { version = "0.8.9", default-features = false, features = ["http1", "json", "matched-path", "tokio"] }
+base64 = "0.22"
chrono = { version = "0.4", default-features = false, features = ["std"] }
+flate2 = "1.1"
hmac = "0.13"
http-body-util = "0.1"
futures-util = "0.3"
diff --git a/ai-gateway/README.md b/ai-gateway/README.md
index ae03501ac5..5d84a98d69 100644
--- a/ai-gateway/README.md
+++ b/ai-gateway/README.md
@@ -12,8 +12,8 @@ format; Rust relays native JSON or SSE without rewriting provider bytes. Models
are GET proxies of the provider catalog and token counting returns the native count;
neither is ingested as a generation. A bounded capture layer captures request/response
facts and logs capture completeness at debug level. Each finalized generation is
-immediately uploaded as a Langfuse generation through OTLP. Building and testing need
-no real Web, database or provider credentials.
+uploaded as a Langfuse generation through OTLP, batched per project. Building and
+testing need no real Web, database or provider credentials.
## Run locally
@@ -79,6 +79,7 @@ do not load dotenv files:
| `LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS` | `10` | Integer from 1 to 300 |
| `LANGFUSE_AI_GATEWAY_MAX_ACTIVE_REQUESTS` | `128` | Positive integer up to Tokio's semaphore capacity; authenticated requests per instance |
| `LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS` | `128` | Positive integer up to Tokio's semaphore capacity; concurrent Web resolutions per instance |
+| `LANGFUSE_AI_GATEWAY_TELEMETRY_BUFFER_BYTES` | `67108864` (64 MiB) | Integer from 4194304 (4 MiB) to 4294967296 (4 GiB); span and credential bytes an instance holds for upload before dropping new records |
The gateway shares `LANGFUSE_LOG_LEVEL` with Web and worker. Values are lowercase;
`fatal` maps to Rust's `error` level and therefore includes ordinary error logs.
@@ -138,7 +139,10 @@ has no unattributed wall time:
| `request.capture` | Parsing the request for inference telemetry; CPU-bound and proportional to `http.request.body.size` |
| `provider.headers` | The provider HTTP send through response headers, including connection setup |
| `provider.stream` | Relaying the provider body to the caller until the relay is finalized; `http.response.body.size`, `gateway.chunks`, `gateway.outcome`, and an error status on timeout or transport failure |
-| `ingestion` | The inference-telemetry upload, scheduled after the relay finishes |
+
+Inference-telemetry uploads carry records from many requests, so each runs in its
+own trace: a root `telemetry.batch` span with `gateway.telemetry.records` and a span
+link to every request it carries, and an `ingestion` client span per HTTP attempt.
`reqwest-tracing` instruments resolver, provider-header, and ingestion requests.
Each request starts a fresh operational trace, independent of incoming trace IDs,
@@ -308,14 +312,34 @@ headers, hop-by-hop headers and upstream framing are excluded.
## Langfuse uploads
-When inference is configured, each finalized execution starts one background POST
-to the Web base URL's `/api/public/otel/v1/traces` endpoint. The client response and
-provider admission never wait for ingestion. There is no batching, waiting queue,
-retry, or durable delivery. A successful upload acknowledges ingestion acceptance;
-storage and cost processing still happen asynchronously in Langfuse.
-
-The original resolver-issued project grant authenticates the upload, together with
-a fresh gateway HMAC signature over that grant. The uploader uses the existing Web
+When inference is configured, each finalized execution is mapped to a generation
+span and queued without waiting. A single delivery worker groups spans by project
+and POSTs each batch to the Web base URL's `/api/public/otel/v1/traces` endpoint
+when it reaches 512 spans or 4 MiB, five seconds after it opened, 30 seconds before its
+grant expires, when more than 256 projects have open batches (the batch due soonest
+goes first), or at shutdown. The client response and provider admission never wait
+for ingestion. There is no durable delivery. A successful upload acknowledges
+ingestion acceptance; storage and cost processing still happen asynchronously in
+Langfuse.
+
+A batch gets up to three attempts. Only failures a resend can fix are retried:
+transport errors and timeouts, and HTTP 408, 429, 500, 502, 503 or 504. The second
+attempt waits 250–500 ms and the third 1–2 s (the upper half of a 0.5 s × 4ⁿ step,
+randomized), or longer when `Retry-After` asks for up to 10 s; a longer
+`Retry-After`, or a wait that would leave under a second of grant lifetime, fails
+the batch instead. Other statuses, expired grants, invalid responses and partial
+OTLP rejections are not retried. Each batch is serialized and gzip-compressed once,
+off the async runtime, and every attempt resends the identical payload: Web
+stores spans by span ID, so a span already ingested by a timed-out attempt is
+replaced rather than duplicated. A batch holds an upload slot only while an attempt
+runs, but keeps its retained bytes until its last attempt settles, so a failing
+ingestion endpoint fills the byte budget and new records are dropped instead of
+queuing without bound.
+
+A resolver-issued project grant authenticates the upload, together with a fresh
+gateway HMAC signature over that grant. Web authorizes a grant by its organization
+and project alone, so a batch uses the latest-expiring grant among its records;
+ingestion mode is already applied while mapping each span. The uploader uses the existing Web
URL and service key, preserves deployment prefixes, disables redirects and proxies,
and never sends the provider credential or the client's gateway key to ingestion.
Ingestion JWTs stay outside serializable capture facts and debug logs. Expired grants
@@ -369,25 +393,30 @@ with the available HTTP status in its status message. Full mode also includes a
bounded provider error code/message; usage mode omits these details because provider
errors may echo request content.
-Provisional upload limits are 32 concurrent tasks, 4 MiB serialized facts per record,
-16 MiB total retained serialized-fact/credential bytes, 8 MiB encoded payloads, and
+Provisional upload limits are 32 concurrent uploads, 1024 queued records, 4 MiB
+serialized span and credentials per record, 64 MiB total retained span/credential
+bytes by default (`LANGFUSE_AI_GATEWAY_TELEMETRY_BUFFER_BYTES`), 8 MiB of OTLP JSON per
+payload before gzip compression, and
64 KiB ingestion responses. Serialization and mapping have additional bounded memory
overhead; these byte budgets are not an RSS limit. Uploads have a two-second connect
-timeout and a five-second total timeout, shortened to the grant's remaining lifetime.
+timeout and a 30-second total timeout per attempt, enough for a full payload over a
+slow link and shortened to the grant's remaining lifetime.
Capacity exhaustion, oversized payloads, auth/HTTP errors, rejected OTLP spans, and
-transport failures are reported without changing inference results. Sanitized logs
+transport failures are reported per record without changing inference results; a
+batch that fails its last attempt fails every record in it. The `telemetry.batch`
+span records `gateway.telemetry.attempts`. Sanitized logs
record failures/drops; shutdown reports accepted, failed and dropped counts.
-SIGTERM marks the gateway unready, drains inference, then finishes uploads within
-the remaining shared shutdown budget. At the deadline, unfinished uploads are
-cancelled and counted as drops. Process crashes or forced shutdown can lose telemetry;
+SIGTERM marks the gateway unready, drains inference, then flushes open batches and
+finishes uploads within the remaining shared shutdown budget. At the deadline, unfinished uploads,
+including batches waiting to retry, are cancelled and counted as drops. Process crashes or forced shutdown can lose telemetry;
these records are not a durable accounting ledger.
-`telemetry/mod.rs` owns admission and task lifecycle, `telemetry/mapping.rs` maps facts,
-and `telemetry/otlp.rs` owns encoding and HTTP delivery. The uploader already accepts
-a span collection; future project batching can replace immediate scheduling while
-retaining each execution's original attribution/content policy and selecting a valid
-compatible grant. Capture and provider byte forwarding do not need to change.
+`telemetry/mod.rs` owns admission and shutdown, `telemetry/mapping.rs` maps facts,
+`telemetry/batch.rs` is the pure per-project grouping and flush policy,
+`telemetry/retry.rs` is the pure retry decision,
+`telemetry/worker.rs` is the delivery actor that owns batches and upload tasks, and
+`telemetry/otlp.rs` owns encoding and HTTP delivery.
## Caller tracing context
@@ -581,7 +610,7 @@ The implementation separates shared `ExecutionCapture` lifecycle/timing, the
`ResponseBody` content-type sniffing and bounded `SseDecoder`. A private
`ProtocolCapture` enum dispatches to the adapter selected by API format. Finalization
hands an owned `InferenceFacts` record to telemetry for a safe debug summary and
-immediate upload. Capture and upload run independently of log level; debug emission
+batched upload. Capture and upload run independently of log level; debug emission
alone is gated.
Capture is limited to 1 MiB request inspection, 1 MiB JSON/SSE event inspection,
@@ -672,6 +701,10 @@ including chunked responses. Redirects, automatic retries, ambient proxy setting
and transparent decompression are disabled. Errors expose fixed categories;
upstream error bodies and transport details are discarded. A successful response
must match the strict v1 schema and carry an unexpired project ingestion grant.
+The grant's JWT payload must name the attribution's `organization_id` and
+`project_id`: Web writes uploads into the token's project, and telemetry batches may
+send any record of a project with any of that project's grants. The gateway reads
+these claims without verifying the signature, which Web checks on ingestion.
Its connection must also be one of the known official pairings: the `provider`
serves the requested `api_format`, `base_url` is that provider's official origin,
and `auth` uses that provider's credential scheme. Any other combination is an
diff --git a/ai-gateway/src/capture/anthropic_messages/tests.rs b/ai-gateway/src/capture/anthropic_messages/tests.rs
index 050abcbfd5..aed48421a3 100644
--- a/ai-gateway/src/capture/anthropic_messages/tests.rs
+++ b/ai-gateway/src/capture/anthropic_messages/tests.rs
@@ -504,7 +504,7 @@ async fn collector() -> (FakeServer, Arc<Mutex<Value>>) {
let received = received.clone();
async move {
let bytes = to_bytes(request.into_body(), 64 * 1024).await.unwrap();
- *received.lock().unwrap() = serde_json::from_slice(&bytes).unwrap();
+ *received.lock().unwrap() = crate::test_support::upload_json(&bytes);
Response::new(Body::from("{}"))
}
})
@@ -535,6 +535,7 @@ async fn streamed_messages_upload_one_generation_with_native_usage() {
let (collector, uploaded) = collector().await;
let telemetry = crate::telemetry::Telemetry::new(
&crate::resolution::ControlPlaneConfig::new(&collector.url, "test-service-key").unwrap(),
+ crate::telemetry::DEFAULT_RETAINED_BYTES,
)
.unwrap();
let context =
diff --git a/ai-gateway/src/capture/openai_responses/tests.rs b/ai-gateway/src/capture/openai_responses/tests.rs
index 072132b968..e854997c38 100644
--- a/ai-gateway/src/capture/openai_responses/tests.rs
+++ b/ai-gateway/src/capture/openai_responses/tests.rs
@@ -716,7 +716,7 @@ async fn client_compression_preferences_do_not_disable_capture() {
let received = received.clone();
async move {
let bytes = to_bytes(request.into_body(), 64 * 1024).await.unwrap();
- *received.lock().unwrap() = serde_json::from_slice(&bytes).unwrap();
+ *received.lock().unwrap() = crate::test_support::upload_json(&bytes);
Response::new(Body::from("{}"))
}
})
@@ -724,6 +724,7 @@ async fn client_compression_preferences_do_not_disable_capture() {
let telemetry = crate::telemetry::Telemetry::new(
&crate::resolution::ControlPlaneConfig::new(&collector.url, "test-service-key")
.unwrap(),
+ crate::telemetry::DEFAULT_RETAINED_BYTES,
)
.unwrap();
let context = resolved_request_context_with_mode("provider-secret", "full").await;
@@ -787,13 +788,14 @@ async fn codex_body_metadata_reaches_the_generation_without_agent_headers() {
let received = received.clone();
async move {
let bytes = to_bytes(request.into_body(), 64 * 1024).await.unwrap();
- *received.lock().unwrap() = serde_json::from_slice(&bytes).unwrap();
+ *received.lock().unwrap() = crate::test_support::upload_json(&bytes);
Response::new(Body::from("{}"))
}
})
.await;
let telemetry = crate::telemetry::Telemetry::new(
&crate::resolution::ControlPlaneConfig::new(&collector.url, "test-service-key").unwrap(),
+ crate::telemetry::DEFAULT_RETAINED_BYTES,
)
.unwrap();
let context = resolved_request_context_with_mode("provider-secret", "full").await;
diff --git a/ai-gateway/src/config.rs b/ai-gateway/src/config.rs
index aa8a2fd709..3061b9768c 100644
--- a/ai-gateway/src/config.rs
+++ b/ai-gateway/src/config.rs
@@ -11,6 +11,7 @@ pub struct GatewayConfig {
pub log_format: LogFormat,
pub max_active_requests: usize,
pub max_concurrent_resolutions: usize,
+ pub telemetry_buffer_bytes: usize,
}
#[derive(Debug, PartialEq, Eq)]
@@ -47,6 +48,9 @@ impl GatewayConfig {
read_env("LANGFUSE_AI_GATEWAY_MAX_ACTIVE_REQUESTS")?.as_deref(),
read_env("LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS")?.as_deref(),
)?;
+ config.telemetry_buffer_bytes = telemetry_buffer_bytes(
+ read_env("LANGFUSE_AI_GATEWAY_TELEMETRY_BUFFER_BYTES")?.as_deref(),
+ )?;
if let Some(web_url) =
read_env("LANGFUSE_AI_GATEWAY_WEB_URL")?.filter(|url| !url.is_empty())
{
@@ -140,6 +144,7 @@ impl GatewayConfig {
max_concurrent_resolutions,
"LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS must be a positive integer within the semaphore capacity",
)?,
+ telemetry_buffer_bytes: crate::telemetry::DEFAULT_RETAINED_BYTES,
})
}
}
@@ -156,6 +161,20 @@ fn concurrency_limit(
.ok_or(GatewayConfigError(message))
}
+fn telemetry_buffer_bytes(value: Option<&str>) -> Result<usize, GatewayConfigError> {
+ const MIB: usize = 1024 * 1024;
+ let Some(value) = value else {
+ return Ok(crate::telemetry::DEFAULT_RETAINED_BYTES);
+ };
+ value
+ .parse::<usize>()
+ .ok()
+ .filter(|bytes| (4 * MIB..=4096 * MIB).contains(bytes))
+ .ok_or(GatewayConfigError(
+ "LANGFUSE_AI_GATEWAY_TELEMETRY_BUFFER_BYTES must be an integer from 4194304 (4 MiB) to 4294967296 (4 GiB)",
+ ))
+}
+
fn read_env(name: &'static str) -> Result<Option<String>, GatewayConfigError> {
match env::var(name) {
Ok(value) => Ok(Some(value)),
@@ -188,6 +207,7 @@ mod tests {
assert_eq!(default.log_format, LogFormat::Text);
assert_eq!(default.max_active_requests, 128);
assert_eq!(default.max_concurrent_resolutions, 128);
+ assert_eq!(default.telemetry_buffer_bytes, 64 * 1024 * 1024);
let custom = GatewayConfig::from_values(
Some("[::1]:9000"),
Some("true"),
@@ -315,4 +335,32 @@ mod tests {
}
}
}
+
+ #[test]
+ fn telemetry_buffer_is_read_in_bytes_within_bounds() {
+ assert_eq!(telemetry_buffer_bytes(None).unwrap(), 64 * 1024 * 1024);
+ for value in [4_194_304, 67_108_864, 4_294_967_296] {
+ assert_eq!(
+ telemetry_buffer_bytes(Some(&value.to_string())).unwrap(),
+ value
+ );
+ }
+ for value in [
+ "",
+ "0",
+ "64",
+ "4194303",
+ "4294967297",
+ "-1",
+ "1.5",
+ "64MiB",
+ "secret-that-must-not-appear",
+ ] {
+ let error = telemetry_buffer_bytes(Some(value)).err().unwrap();
+ assert_eq!(
+ error.to_string(),
+ "LANGFUSE_AI_GATEWAY_TELEMETRY_BUFFER_BYTES must be an integer from 4194304 (4 MiB) to 4294967296 (4 GiB)"
+ );
+ }
+ }
}
diff --git a/ai-gateway/src/http/tests.rs b/ai-gateway/src/http/tests.rs
index baa8116980..1aae02545a 100644
--- a/ai-gateway/src/http/tests.rs
+++ b/ai-gateway/src/http/tests.rs
@@ -509,7 +509,7 @@ async fn pre_header_provider_failures_export_the_status_returned_to_the_caller()
let sent = sent.clone();
async move {
let bytes = to_bytes(request.into_body(), 65536).await.unwrap();
- sent.send(serde_json::from_slice::<Value>(&bytes).unwrap())
+ sent.send(crate::test_support::upload_json(&bytes))
.await
.unwrap();
Response::new(Body::from("{}"))
@@ -517,7 +517,7 @@ async fn pre_header_provider_failures_export_the_status_returned_to_the_caller()
})
.await;
let telemetry =
- Telemetry::new(&ControlPlaneConfig::new(&sink.url, "service-key").unwrap()).unwrap();
+ Telemetry::for_test(&ControlPlaneConfig::new(&sink.url, "service-key").unwrap());
let upstream = TcpListener::bind("127.0.0.1:0").await.unwrap();
let upstream_url = format!("http://{}", upstream.local_addr().unwrap());
let upstream_task = tokio::spawn(async move {
diff --git a/ai-gateway/src/inference.rs b/ai-gateway/src/inference.rs
index 36d8f92306..c4eda64ffc 100644
--- a/ai-gateway/src/inference.rs
+++ b/ai-gateway/src/inference.rs
@@ -34,11 +34,12 @@ impl InferenceService {
config: ControlPlaneConfig,
max_active_requests: usize,
max_concurrent_resolutions: usize,
+ telemetry_buffer_bytes: usize,
) -> Result<Self, ResolutionError> {
if !(1..=Semaphore::MAX_PERMITS).contains(&max_concurrent_resolutions) {
return Err(ResolutionError::Configuration);
}
- let telemetry = crate::telemetry::Telemetry::new(&config)?;
+ let telemetry = crate::telemetry::Telemetry::new(&config, telemetry_buffer_bytes)?;
Ok(Self {
control_plane: ControlPlaneClient::new(config)?,
provider: ProviderTransport::new(max_active_requests)
diff --git a/ai-gateway/src/main.rs b/ai-gateway/src/main.rs
index 71c7193383..8fdacd8e27 100644
--- a/ai-gateway/src/main.rs
+++ b/ai-gateway/src/main.rs
@@ -30,6 +30,7 @@ async fn run() -> Result<(), Box<dyn Error>> {
control_plane,
config.max_active_requests,
config.max_concurrent_resolutions,
+ config.telemetry_buffer_bytes,
)
})
.transpose()?;
diff --git a/ai-gateway/src/observability/http/tests.rs b/ai-gateway/src/observability/http/tests.rs
index 33be8af730..d6457757f4 100644
--- a/ai-gateway/src/observability/http/tests.rs
+++ b/ai-gateway/src/observability/http/tests.rs
@@ -247,7 +247,11 @@ async fn generation_context_is_isolated_from_operational_spans_and_outbound_head
web_calls.lock().unwrap().push((
if ingestion { "ingestion" } else { "resolver" },
parts.headers,
- serde_json::from_slice::<Value>(&bytes).unwrap(),
+ if ingestion {
+ crate::test_support::upload_json(&bytes)
+ } else {
+ serde_json::from_slice::<Value>(&bytes).unwrap()
+ },
));
if ingestion {
Response::new(Body::from("{}"))
@@ -272,8 +276,11 @@ async fn generation_context_is_isolated_from_operational_spans_and_outbound_head
}
})
.await;
- let telemetry =
- Telemetry::new(&ControlPlaneConfig::new(&web.url, "test-service-key").unwrap()).unwrap();
+ let telemetry = Telemetry::new(
+ &ControlPlaneConfig::new(&web.url, "test-service-key").unwrap(),
+ crate::telemetry::DEFAULT_RETAINED_BYTES,
+ )
+ .unwrap();
let service = InferenceService::for_test(
web.control_plane(),
ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default())
@@ -325,10 +332,11 @@ async fn generation_context_is_isolated_from_operational_spans_and_outbound_head
type SpanData = opentelemetry_sdk::trace::SpanData;
/// Every phase between the caller's headers and the last body byte has a span in
-/// the server trace, so a waterfall shows no unattributed wall time.
+/// the server trace, so a waterfall shows no unattributed wall time. Batched
+/// ingestion runs in its own trace, linked back to each request it carries.
fn assert_phase_spans(spans: &[SpanData], server: &SpanData) {
let named = |name: &str| spans.iter().find(|span| span.name == name).unwrap();
- assert_eq!(spans.len(), 8);
+ assert_eq!(spans.len(), 9);
for (name, parent) in [
("resolution", server),
("resolver", named("resolution")),
@@ -336,7 +344,6 @@ fn assert_phase_spans(spans: &[SpanData], server: &SpanData) {
("request.capture", server),
("provider.headers", server),
("provider.stream", server),
- ("ingestion", server),
] {
let child = named(name);
assert_eq!(
@@ -351,6 +358,28 @@ fn assert_phase_spans(spans: &[SpanData], server: &SpanData) {
"{name}"
);
}
+ let batch = named("telemetry.batch");
+ assert_eq!(batch.parent_span_id, opentelemetry::trace::SpanId::INVALID);
+ assert_ne!(
+ batch.span_context.trace_id(),
+ server.span_context.trace_id()
+ );
+ assert_eq!(
+ batch
+ .links
+ .iter()
+ .map(|link| link.span_context.clone())
+ .collect::<Vec<_>>(),
+ std::slice::from_ref(&server.span_context)
+ );
+ assert_attribute(batch, "gateway.telemetry.records", 1i64);
+ assert_attribute(batch, "gateway.telemetry.attempts", 1i64);
+ let ingestion = named("ingestion");
+ assert_eq!(ingestion.parent_span_id, batch.span_context.span_id());
+ assert_eq!(
+ ingestion.span_context.trace_id(),
+ batch.span_context.trace_id()
+ );
let request_bytes = i64::try_from(GENERATION_REQUEST.len()).unwrap();
assert_attribute(server, "http.request.body.size", request_bytes);
assert_attribute(server, "gateway.outcome", "eof");
diff --git a/ai-gateway/src/observability/metrics.rs b/ai-gateway/src/observability/metrics.rs
index 0e33e9363d..a68fd9e729 100644
--- a/ai-gateway/src/observability/metrics.rs
+++ b/ai-gateway/src/observability/metrics.rs
@@ -83,9 +83,9 @@ pub(crate) fn rejected(phase: &'static str) {
METRICS.rejections.add(1, &[KeyValue::new("phase", phase)]);
}
-pub(crate) fn delivery(outcome: &'static str, reason: &'static str) {
+pub(crate) fn delivery(outcome: &'static str, reason: &'static str, records: u64) {
METRICS.delivery.add(
- 1,
+ records,
&[
KeyValue::new("outcome", outcome),
KeyValue::new("reason", reason),
diff --git a/ai-gateway/src/providers/tests.rs b/ai-gateway/src/providers/tests.rs
index dc7955f87e..1de6fcfa94 100644
--- a/ai-gateway/src/providers/tests.rs
+++ b/ai-gateway/src/providers/tests.rs
@@ -1,5 +1,7 @@
use super::*;
-use crate::test_support::{FakeServer, resolved_request_context, resolved_request_context_for};
+use crate::test_support::{
+ FakeServer, ingestion_token, resolved_request_context, resolved_request_context_for,
+};
use axum::{
body::to_bytes,
http::{HeaderName, StatusCode},
@@ -360,7 +362,6 @@ async fn completed_json_and_sse_upload_once_without_waiting_for_ingestion() {
resolution::ControlPlaneConfig, telemetry::Telemetry,
test_support::resolved_request_context_with_mode,
};
- use serde_json::Value;
const JSON: &str = r#"{"id":"resp-1","model":"actual","status":"completed","output":[],"usage":{"input_tokens":2,"output_tokens":1,"total_tokens":3}}"#;
const SSE: &str = "data: {\"type\":\"response.output_text.delta\",\"delta\":\"hello\"}\n\ndata: {\"type\":\"response.completed\",\"response\":{\"id\":\"resp-1\",\"model\":\"actual\",\"status\":\"completed\",\"output\":[],\"usage\":{\"input_tokens\":2,\"output_tokens\":1,\"total_tokens\":3}}}\n\n";
@@ -377,7 +378,7 @@ async fn completed_json_and_sse_upload_once_without_waiting_for_ingestion() {
let release = sink_release.clone();
async move {
let bytes = to_bytes(request.into_body(), 65536).await.unwrap();
- sent.send(serde_json::from_slice::<Value>(&bytes).unwrap())
+ sent.send(crate::test_support::upload_json(&bytes))
.await
.unwrap();
release.notified().await;
@@ -389,7 +390,7 @@ async fn completed_json_and_sse_upload_once_without_waiting_for_ingestion() {
})
.await;
let telemetry =
- Telemetry::new(&ControlPlaneConfig::new(&sink.url, "service-key").unwrap()).unwrap();
+ Telemetry::for_test(&ControlPlaneConfig::new(&sink.url, "service-key").unwrap());
let upstream = FakeServer::start(move |_| async move {
let body = if close_before_eof {
Body::from_stream(
@@ -472,10 +473,10 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rel
async move {
assert_eq!(
request.headers()[header::AUTHORIZATION],
- "Bearer private-ingestion-token"
+ format!("Bearer {}", ingestion_token("org-1", "project-1")).as_str()
);
let bytes = to_bytes(request.into_body(), 65536).await.unwrap();
- sent.send(serde_json::from_slice::<Value>(&bytes).unwrap())
+ sent.send(crate::test_support::upload_json(&bytes))
.await
.unwrap();
Response::new(Body::from("{}"))
@@ -483,7 +484,7 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rel
})
.await;
let telemetry =
- Telemetry::new(&ControlPlaneConfig::new(&sink.url, "service-key").unwrap()).unwrap();
+ Telemetry::for_test(&ControlPlaneConfig::new(&sink.url, "service-key").unwrap());
let dropped = Arc::new(Notify::new());
let upstream = stalled_provider(dropped).await;
let relay = ProviderTransport::for_test(
@@ -552,7 +553,6 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rel
#[tokio::test]
async fn compact_posts_compact_path_and_models_get_skips_ingestion() {
use crate::{resolution::ControlPlaneConfig, telemetry::Telemetry};
- use serde_json::Value;
const COMPACT: &[u8] =
br#"{"model":"gpt-4.1","input":[{"encrypted_content":"opaque-ciphertext"}]}"#;
@@ -565,7 +565,7 @@ async fn compact_posts_compact_path_and_models_get_skips_ingestion() {
let sent = sent.clone();
async move {
let bytes = to_bytes(request.into_body(), 65536).await.unwrap();
- sent.send(serde_json::from_slice::<Value>(&bytes).unwrap())
+ sent.send(crate::test_support::upload_json(&bytes))
.await
.unwrap();
Response::new(Body::from("{}"))
@@ -573,7 +573,7 @@ async fn compact_posts_compact_path_and_models_get_skips_ingestion() {
})
.await;
let telemetry =
- Telemetry::new(&ControlPlaneConfig::new(&sink.url, "service-key").unwrap()).unwrap();
+ Telemetry::for_test(&ControlPlaneConfig::new(&sink.url, "service-key").unwrap());
let upstream = FakeServer::start(|request| async move {
let uri = request.uri().to_string();
let method = request.method().clone();
diff --git a/ai-gateway/src/resolution/contracts.rs b/ai-gateway/src/resolution/contracts.rs
index 0a3b744556..581e1166a8 100644
--- a/ai-gateway/src/resolution/contracts.rs
+++ b/ai-gateway/src/resolution/contracts.rs
@@ -1,4 +1,5 @@
use super::{ResolutionError, valid_token};
+use base64::{Engine, engine::general_purpose::URL_SAFE_NO_PAD};
use serde::{
Deserialize, Deserializer, Serialize,
de::{DeserializeOwned, IntoDeserializer},
@@ -240,6 +241,32 @@ impl fmt::Debug for ResolvedRequestContext {
}
}
+#[derive(Deserialize)]
+struct IngestionClaims {
+ organization_id: String,
+ project_id: String,
+}
+
+fn ingestion_claims_match(token: &str, attribution: &RequestAttribution) -> bool {
+ let mut segments = token.split('.');
+ let (Some(_header), Some(payload), Some(_signature), None) = (
+ segments.next(),
+ segments.next(),
+ segments.next(),
+ segments.next(),
+ ) else {
+ return false;
+ };
+ URL_SAFE_NO_PAD
+ .decode(payload)
+ .ok()
+ .and_then(|bytes| serde_json::from_slice::<IngestionClaims>(&bytes).ok())
+ .is_some_and(|claims| {
+ claims.organization_id == attribution.organization_id
+ && claims.project_id == attribution.project_id
+ })
+}
+
pub(super) fn decode(
bytes: &[u8],
expected_format: ApiFormat,
@@ -257,6 +284,7 @@ pub(super) fn decode(
|| !provider.accepts(connection.credential())
|| !valid_token(connection.credential().secret())
|| !valid_token(&response.ingestion.access_token)
+ || !ingestion_claims_match(&response.ingestion.access_token, attribution)
|| response.ingestion.expires_at <= now
|| [
&connection.id,
diff --git a/ai-gateway/src/resolution/tests.rs b/ai-gateway/src/resolution/tests.rs
index 96db6264dc..2959de4301 100644
--- a/ai-gateway/src/resolution/tests.rs
+++ b/ai-gateway/src/resolution/tests.rs
@@ -17,6 +17,8 @@ use std::{
};
use tokio::{net::TcpListener, task::JoinHandle};
+use crate::test_support::ingestion_token;
+
const NOW: u64 = 1_800_000_000;
const NOW_TIME: Duration = Duration::from_secs(NOW);
const SERVICE_KEY: &str = "gateway-web-test-secret-not-for-production";
@@ -105,7 +107,7 @@ fn success(project: &str, provider_token: &str) -> Value {
"provider_connection_id": "connection-1"
},
"ingestion_mode": "usage",
- "ingestion": {"access_token": "private-ingestion-token", "token_type": "Bearer", "expires_at": NOW + 300}
+ "ingestion": {"access_token": ingestion_token("org-1", project), "token_type": "Bearer", "expires_at": NOW + 300}
})
}
@@ -141,7 +143,7 @@ async fn signs_the_exact_key_and_sends_only_the_api_format() {
);
assert_eq!(
context.ingestion().access_token(),
- "private-ingestion-token"
+ ingestion_token("org-1", "project-1")
);
let debug = format!("{context:?}");
for sensitive in [
@@ -500,6 +502,50 @@ async fn rejects_incompatible_or_incomplete_execution_contexts() {
assert_invalid_context(body).await;
}
+#[tokio::test]
+async fn rejects_ingestion_tokens_that_authorize_a_different_tenant() {
+ use base64::{Engine, engine::general_purpose::URL_SAFE_NO_PAD};
+ let payload = |claims: Value| URL_SAFE_NO_PAD.encode(claims.to_string());
+ let claims = json!({"organization_id": "org-1", "project_id": "project-1"});
+ for token in [
+ ingestion_token("org-1", "project-2"),
+ ingestion_token("org-2", "project-1"),
+ "opaque-ingestion-token".to_owned(),
+ format!("header.{}.signature.extra", payload(claims.clone())),
+ format!("header.{}", payload(claims.clone())),
+ format!("header.{}=.signature", payload(claims)),
+ format!(
+ "header.{}.signature",
+ payload(json!({"project_id": "project-1"}))
+ ),
+ format!(
+ "header.{}.signature",
+ payload(json!({"organization_id": "org-1", "project_id": 1}))
+ ),
+ "header.not-base64!.signature".to_owned(),
+ ] {
+ let mut body = success("project-1", "provider-token");
+ body["ingestion"]["access_token"] = json!(token);
+ assert_invalid_context(body).await;
+ }
+ let mut body = success("project-1", "provider-token");
+ body["ingestion"]["access_token"] = json!(format!(
+ "header.{}.signature",
+ payload(json!({"organization_id": "org-1", "project_id": "project-1", "extra": true}))
+ ));
+ let web = FakeWeb::start(move |_| {
+ let body = body.clone();
+ async move { response(body) }
+ })
+ .await;
+ assert!(
+ web.control_plane()
+ .resolve_at(GATEWAY_KEY, ApiFormat::OpenAiResponses, NOW_TIME)
+ .await
+ .is_ok()
+ );
+}
+
async fn assert_invalid_context(body: Value) {
assert_invalid_context_for(body, ApiFormat::OpenAiResponses).await;
}
diff --git a/ai-gateway/src/telemetry/batch.rs b/ai-gateway/src/telemetry/batch.rs
new file mode 100644
index 0000000000..42ba72faf3
--- /dev/null
+++ b/ai-gateway/src/telemetry/batch.rs
@@ -0,0 +1,175 @@
+use std::{
+ collections::{HashMap, hash_map::Entry},
+ time::Duration,
+};
+
+use opentelemetry::trace::SpanContext;
+use serde_json::Value;
+use tokio::{sync::OwnedSemaphorePermit, time::Instant};
+
+use super::Grant;
+
+#[derive(Clone, Copy)]
+pub(super) struct BatchPolicy {
+ pub max_records: usize,
+ pub max_bytes: usize,
+ pub linger: Duration,
+ pub expiry_margin: Duration,
+ pub max_open: usize,
+}
+
+impl Default for BatchPolicy {
+ fn default() -> Self {
+ Self {
+ max_records: 512,
+ max_bytes: 4 * 1024 * 1024,
+ linger: Duration::from_secs(5),
+ expiry_margin: Duration::from_secs(30),
+ max_open: 256,
+ }
+ }
+}
+
+pub(super) struct Pending {
+ pub span: Value,
+ pub bytes: usize,
+ pub link: SpanContext,
+ _retained: OwnedSemaphorePermit,
+}
+
+impl Pending {
+ pub fn new(
+ span: Value,
+ bytes: usize,
+ link: SpanContext,
+ retained: OwnedSemaphorePermit,
+ ) -> Self {
+ Self {
+ span,
+ bytes,
+ link,
+ _retained: retained,
+ }
+ }
+}
+
+pub(super) struct Flush {
+ pub grant: Grant,
+ pub items: Vec<Pending>,
+}
+
+struct Batch {
+ grant: Grant,
+ items: Vec<Pending>,
+ bytes: usize,
+ opened: Instant,
+ due: Instant,
+}
+
+impl Batch {
+ fn open(grant: Grant, now: Instant, policy: &BatchPolicy) -> Self {
+ let due = due_at(&grant, now, now, policy);
+ Self {
+ grant,
+ items: Vec::new(),
+ bytes: 0,
+ opened: now,
+ due,
+ }
+ }
+
+ fn adopt_grant(&mut self, grant: Grant, now: Instant, policy: &BatchPolicy) {
+ if grant.expires_at > self.grant.expires_at {
+ self.due = due_at(&grant, self.opened, now, policy);
+ self.grant = grant;
+ }
+ }
+
+ fn into_flush(self) -> Flush {
+ Flush {
+ grant: self.grant,
+ items: self.items,
+ }
+ }
+}
+
+fn due_at(grant: &Grant, opened: Instant, now: Instant, policy: &BatchPolicy) -> Instant {
+ let usable = grant.remaining().saturating_sub(policy.expiry_margin);
+ (opened + policy.linger).min(now + usable.min(policy.linger))
+}
+
+pub(super) struct Batches {
+ policy: BatchPolicy,
+ open: HashMap<String, Batch>,
+}
+
+impl Batches {
+ pub fn new(policy: BatchPolicy) -> Self {
+ Self {
+ policy,
+ open: HashMap::new(),
+ }
+ }
+
+ pub fn push_record(&mut self, grant: Grant, item: Pending, now: Instant) -> Vec<Flush> {
+ let mut ready = Vec::new();
+ let project = grant.project_id.clone();
+ if self
+ .open
+ .get(&project)
+ .is_some_and(|batch| batch.bytes + item.bytes > self.policy.max_bytes)
+ {
+ ready.extend(self.take_project(&project));
+ }
+ if !self.open.contains_key(&project) && self.open.len() >= self.policy.max_open {
+ let soonest = self
+ .open
+ .iter()
+ .min_by_key(|(_, batch)| batch.due)
+ .map(|(project, _)| project.clone());
+ if let Some(soonest) = soonest {
+ ready.extend(self.take_project(&soonest));
+ }
+ }
+ let policy = self.policy;
+ let batch = match self.open.entry(project.clone()) {
+ Entry::Occupied(entry) => {
+ let batch = entry.into_mut();
+ batch.adopt_grant(grant, now, &policy);
+ batch
+ }
+ Entry::Vacant(entry) => entry.insert(Batch::open(grant, now, &policy)),
+ };
+ batch.bytes += item.bytes;
+ batch.items.push(item);
+ if batch.items.len() >= policy.max_records || batch.bytes >= policy.max_bytes {
+ ready.extend(self.take_project(&project));
+ }
+ ready
+ }
+
+ pub fn take_due(&mut self, now: Instant) -> Vec<Flush> {
+ self.open
+ .extract_if(|_, batch| batch.due <= now)
+ .map(|(_, batch)| batch.into_flush())
+ .collect()
+ }
+
+ pub fn take_all(&mut self) -> Vec<Flush> {
+ self.open
+ .drain()
+ .map(|(_, batch)| batch.into_flush())
+ .collect()
+ }
+
+ pub fn next_due(&self) -> Option<Instant> {
+ self.open.values().map(|batch| batch.due).min()
+ }
+
+ fn take_project(&mut self, project: &str) -> Option<Flush> {
+ self.open.remove(project).map(Batch::into_flush)
+ }
+}
+
+#[cfg(test)]
+mod tests;
diff --git a/ai-gateway/src/telemetry/batch/tests.rs b/ai-gateway/src/telemetry/batch/tests.rs
new file mode 100644
index 0000000000..ba7fc29e0e
--- /dev/null
+++ b/ai-gateway/src/telemetry/batch/tests.rs
@@ -0,0 +1,137 @@
+use std::{
+ sync::Arc,
+ time::{Duration, SystemTime, UNIX_EPOCH},
+};
+
+use opentelemetry::trace::SpanContext;
+use serde_json::json;
+use tokio::{sync::Semaphore, time::Instant};
+
+use super::*;
+
+const LINGER: Duration = Duration::from_secs(1);
+
+fn policy() -> BatchPolicy {
+ BatchPolicy {
+ max_records: 3,
+ max_bytes: 100,
+ linger: LINGER,
+ expiry_margin: Duration::from_secs(30),
+ max_open: 2,
+ }
+}
+
+fn grant(project: &str, token: &str, expires_in: u64) -> Grant {
+ let unix_now = SystemTime::now()
+ .duration_since(UNIX_EPOCH)
+ .unwrap()
+ .as_secs();
+ Grant {
+ project_id: project.into(),
+ access_token: token.into(),
+ expires_at: unix_now + expires_in,
+ }
+}
+
+fn item(bytes: usize) -> Pending {
+ let permit = Arc::new(Semaphore::new(1)).try_acquire_owned().unwrap();
+ Pending::new(json!({}), bytes, SpanContext::empty_context(), permit)
+}
+
+fn projects(flushes: &[Flush]) -> Vec<(&str, usize)> {
+ let mut projects: Vec<_> = flushes
+ .iter()
+ .map(|flush| (flush.grant.project_id.as_str(), flush.items.len()))
+ .collect();
+ projects.sort_unstable();
+ projects
+}
+
+#[test]
+fn reaching_the_record_limit_flushes_only_that_project() {
+ let mut batches = Batches::new(policy());
+ let now = Instant::now();
+ for _ in 0..2 {
+ assert!(
+ batches
+ .push_record(grant("a", "t", 900), item(1), now)
+ .is_empty()
+ );
+ }
+ assert!(
+ batches
+ .push_record(grant("b", "t", 900), item(1), now)
+ .is_empty()
+ );
+ let ready = batches.push_record(grant("a", "t", 900), item(1), now);
+ assert_eq!(projects(&ready), [("a", 3)]);
+ assert_eq!(projects(&batches.take_all()), [("b", 1)]);
+}
+
+#[test]
+fn a_span_that_would_overflow_the_byte_limit_starts_a_new_batch() {
+ let mut batches = Batches::new(policy());
+ let now = Instant::now();
+ assert!(
+ batches
+ .push_record(grant("a", "t", 900), item(60), now)
+ .is_empty()
+ );
+ let ready = batches.push_record(grant("a", "t", 900), item(60), now);
+ assert_eq!(projects(&ready), [("a", 1)]);
+ assert_eq!(projects(&batches.take_all()), [("a", 1)]);
+}
+
+#[test]
+fn a_span_at_the_byte_limit_is_uploaded_alone() {
+ let mut batches = Batches::new(policy());
+ let ready = batches.push_record(grant("a", "t", 900), item(150), Instant::now());
+ assert_eq!(projects(&ready), [("a", 1)]);
+ assert!(batches.next_due().is_none());
+}
+
+#[test]
+fn batches_become_due_after_the_linger() {
+ let mut batches = Batches::new(policy());
+ let now = Instant::now();
+ batches.push_record(grant("a", "t", 900), item(1), now);
+ batches.push_record(grant("b", "t", 900), item(1), now + LINGER / 2);
+ assert_eq!(batches.next_due(), Some(now + LINGER));
+ assert!(batches.take_due(now + LINGER / 2).is_empty());
+ assert_eq!(projects(&batches.take_due(now + LINGER)), [("a", 1)]);
+ assert_eq!(batches.next_due(), Some(now + LINGER + LINGER / 2));
+}
+
+#[test]
+fn grants_close_to_expiry_are_used_immediately() {
+ let mut batches = Batches::new(policy());
+ let now = Instant::now();
+ batches.push_record(grant("a", "t", 10), item(1), now);
+ assert_eq!(batches.next_due(), Some(now));
+}
+
+#[test]
+fn a_later_expiring_grant_replaces_the_batch_grant() {
+ let mut batches = Batches::new(BatchPolicy {
+ max_records: 10,
+ ..policy()
+ });
+ let now = Instant::now();
+ batches.push_record(grant("a", "soon", 10), item(1), now);
+ batches.push_record(grant("a", "later", 900), item(1), now);
+ assert_eq!(batches.next_due(), Some(now + LINGER));
+ batches.push_record(grant("a", "earlier", 5), item(1), now);
+ let flush = batches.take_all().pop().unwrap();
+ assert_eq!(flush.grant.access_token, "later");
+}
+
+#[test]
+fn opening_beyond_the_project_limit_flushes_the_batch_due_soonest() {
+ let mut batches = Batches::new(policy());
+ let now = Instant::now();
+ batches.push_record(grant("a", "t", 900), item(1), now + LINGER / 2);
+ batches.push_record(grant("b", "t", 900), item(1), now);
+ let ready = batches.push_record(grant("c", "t", 900), item(1), now + LINGER / 2);
+ assert_eq!(projects(&ready), [("b", 1)]);
+ assert_eq!(projects(&batches.take_all()), [("a", 1), ("c", 1)]);
+}
diff --git a/ai-gateway/src/telemetry/mod.rs b/ai-gateway/src/telemetry/mod.rs
index e9ef212d68..8f3e78435c 100644
--- a/ai-gateway/src/telemetry/mod.rs
+++ b/ai-gateway/src/telemetry/mod.rs
@@ -1,7 +1,10 @@
-//! Immediate, best-effort delivery of finalized inference facts.
+//! Best-effort delivery of finalized inference facts, batched per project.
+mod batch;
mod context;
mod mapping;
mod otlp;
+mod retry;
+mod worker;
use std::{
io::{self, Write},
@@ -9,32 +12,52 @@ use std::{
Arc, Mutex,
atomic::{AtomicU64, Ordering},
},
+ time::{Duration, SystemTime, UNIX_EPOCH},
};
use axum::http::HeaderMap;
-use opentelemetry::trace::{FutureExt, TraceContextExt};
+use opentelemetry::trace::TraceContextExt;
use serde_json::{Map, Value};
-use tokio::{sync::Semaphore, task::JoinSet, time::Instant};
+use tokio::{
+ sync::{Semaphore, mpsc, mpsc::error::TrySendError},
+ task::JoinHandle,
+ time::Instant,
+};
use tracing_opentelemetry::OpenTelemetrySpanExt;
use crate::{
capture::InferenceFacts,
resolution::{ControlPlaneConfig, ResolutionError, ResolvedRequestContext},
};
+use batch::{BatchPolicy, Batches, Pending};
use context::GenerationContext;
pub(crate) use context::take_agent_client_metadata;
use otlp::Uploader;
+use retry::RetryPolicy;
+use worker::{Message, Uploads};
const MAX_UPLOADS: usize = 32;
-const MAX_FACT_BYTES: usize = 4 * 1024 * 1024;
-const MAX_RETAINED_BYTES: usize = 16 * 1024 * 1024;
+const MAX_RECORD_BYTES: usize = 4 * 1024 * 1024;
+pub(crate) const DEFAULT_RETAINED_BYTES: usize = 64 * 1024 * 1024;
+const MAX_QUEUED_RECORDS: usize = 1024;
+
+struct Grant {
+ project_id: String,
+ access_token: String,
+ expires_at: u64,
+}
+
+impl Grant {
+ fn remaining(&self) -> Duration {
+ let now = SystemTime::now()
+ .duration_since(UNIX_EPOCH)
+ .unwrap_or_default();
+ Duration::from_secs(self.expires_at).saturating_sub(now)
+ }
+}
-/// Credentials are deliberately not serializable or printable. Project attribution
-/// stays attached to each execution even when delivery outlives the response body.
pub(crate) struct DeliveryContext {
- pub project_id: String,
- pub access_token: String,
- pub expires_at: u64,
+ grant: Grant,
generation: GenerationContext,
}
@@ -45,9 +68,11 @@ impl DeliveryContext {
client_metadata: Option<&Map<String, Value>>,
) -> Self {
Self {
- project_id: context.attribution().project_id().to_owned(),
- access_token: context.ingestion().access_token().to_owned(),
- expires_at: context.ingestion().expires_at(),
+ grant: Grant {
+ project_id: context.attribution().project_id().to_owned(),
+ access_token: context.ingestion().access_token().to_owned(),
+ expires_at: context.ingestion().expires_at(),
+ },
generation: GenerationContext::from_request(headers, client_metadata),
}
}
@@ -60,148 +85,142 @@ struct Stats {
dropped: AtomicU64,
}
+impl Stats {
+ fn record_accepted(&self, records: u64) {
+ self.accepted.fetch_add(records, Ordering::Relaxed);
+ crate::observability::delivery("accepted", "success", records);
+ }
+
+ fn record_failed(&self, records: u64, reason: &'static str) -> bool {
+ let before = self.failed.fetch_add(records, Ordering::Relaxed);
+ crate::observability::delivery("failed", reason, records);
+ crosses_report_threshold(before, before + records)
+ }
+
+ fn record_dropped(&self, records: u64, reason: &'static str) {
+ let before = self.dropped.fetch_add(records, Ordering::Relaxed);
+ crate::observability::delivery("dropped", reason, records);
+ let dropped = before + records;
+ if crosses_report_threshold(before, dropped) {
+ tracing::warn!(reason, dropped, "gateway telemetry dropped");
+ }
+ }
+}
+
+fn crosses_report_threshold(before: u64, after: u64) -> bool {
+ before == 0 || before / 100 != after / 100
+}
+
struct Delivery {
- uploader: Arc<Uploader>,
- capacity: Arc<Semaphore>,
- bytes: Arc<Semaphore>,
- tasks: Mutex<Option<JoinSet<()>>>,
+ queue: mpsc::Sender<Message>,
+ retained: Arc<Semaphore>,
+ worker: Mutex<Option<JoinHandle<()>>>,
stats: Arc<Stats>,
}
-/// Shared delivery handle. Shutdown closes submission and drains already admitted work.
#[derive(Clone)]
pub struct Telemetry(Arc<Delivery>);
impl Telemetry {
- pub(crate) fn new(config: &ControlPlaneConfig) -> Result<Self, ResolutionError> {
+ pub(crate) fn new(
+ config: &ControlPlaneConfig,
+ retained_bytes: usize,
+ ) -> Result<Self, ResolutionError> {
Ok(Self::with_uploader(
Uploader::new(config)?,
MAX_UPLOADS,
- MAX_RETAINED_BYTES,
+ retained_bytes,
+ BatchPolicy::default(),
+ RetryPolicy::default(),
))
}
- fn with_uploader(uploader: Uploader, uploads: usize, bytes: usize) -> Self {
+ #[cfg(test)]
+ pub(crate) fn for_test(config: &ControlPlaneConfig) -> Self {
+ Self::with_uploader(
+ Uploader::new(config).expect("test uploader"),
+ MAX_UPLOADS,
+ DEFAULT_RETAINED_BYTES,
+ BatchPolicy {
+ linger: Duration::from_millis(10),
+ ..BatchPolicy::default()
+ },
+ RetryPolicy {
+ max_attempts: 1,
+ ..RetryPolicy::default()
+ },
+ )
+ }
+
+ fn with_uploader(
+ uploader: Uploader,
+ uploads: usize,
+ bytes: usize,
+ policy: BatchPolicy,
+ retry: RetryPolicy,
+ ) -> Self {
+ let (queue, receiver) = mpsc::channel(MAX_QUEUED_RECORDS);
+ let stats = Arc::new(Stats::default());
+ let worker = tokio::spawn(worker::run_worker(
+ receiver,
+ Batches::new(policy),
+ Uploads::new(uploader, uploads, retry, stats.clone()),
+ ));
Self(Arc::new(Delivery {
- uploader: Arc::new(uploader),
- capacity: Arc::new(Semaphore::new(uploads)),
- bytes: Arc::new(Semaphore::new(bytes)),
- tasks: Mutex::new(Some(JoinSet::new())),
- stats: Arc::new(Stats::default()),
+ queue,
+ retained: Arc::new(Semaphore::new(bytes)),
+ worker: Mutex::new(Some(worker)),
+ stats,
}))
}
- /// No waiting queue and no network work on the caller's response/drop path.
pub(crate) fn record(&self, context: DeliveryContext, facts: InferenceFacts) {
- let Ok(capacity) = self.0.capacity.clone().try_acquire_owned() else {
- self.drop_record("capacity");
- return;
- };
+ let link = tracing::Span::current()
+ .context()
+ .span()
+ .span_context()
+ .clone();
+ let span = mapping::span(facts, &context.generation);
+ let credentials = context.grant.access_token.len() + context.grant.project_id.len();
let mut size = SizeCounter {
- bytes: context.access_token.len() + context.project_id.len(),
- limit: MAX_FACT_BYTES,
+ bytes: 0,
+ limit: MAX_RECORD_BYTES.saturating_sub(credentials),
};
- if serde_json::to_writer(&mut size, &context.generation).is_err()
- || serde_json::to_writer(&mut size, &facts).is_err()
- {
- self.drop_record("size");
+ if serde_json::to_writer(&mut size, &span).is_err() {
+ self.0.stats.record_dropped(1, "size");
return;
}
- let Ok(bytes) = self
- .0
- .bytes
- .clone()
- .try_acquire_many_owned(u32::try_from(size.bytes).expect("bounded fact bytes"))
- else {
- self.drop_record("bytes");
- return;
- };
- let mut tasks = self.0.tasks.lock().expect("telemetry task lock poisoned");
- let Some(tasks) = tasks.as_mut() else {
- self.drop_record("shutdown");
+ let Ok(retained) = self.0.retained.clone().try_acquire_many_owned(
+ u32::try_from(size.bytes + credentials).expect("bounded record bytes"),
+ ) else {
+ self.0.stats.record_dropped(1, "bytes");
return;
};
- // Reap completed handles on submission so the task registry stays bounded.
- while tasks.try_join_next().is_some() {}
- let uploader = self.0.uploader.clone();
- let stats = self.0.stats.clone();
- let parent = tracing::Span::current().context();
- tasks.spawn(
- async move {
- let _capacity = capacity;
- let _bytes = bytes;
- let span = mapping::span(facts, &context.generation);
- match uploader.export(&context, &[span]).await {
- Ok(()) => {
- stats.accepted.fetch_add(1, Ordering::Relaxed);
- crate::observability::delivery("accepted", "success");
- tracing::debug!("gateway telemetry accepted");
- }
- Err(error) => {
- let failed = stats.failed.fetch_add(1, Ordering::Relaxed) + 1;
- crate::observability::delivery("failed", error.reason());
- // Error categories contain no URLs, credentials, response bodies or content.
- if failed == 1 || failed.is_multiple_of(100) {
- let context = opentelemetry::Context::current();
- let span = context.span();
- let context = span.span_context();
- tracing::warn!(
- trace_id = %context.trace_id(),
- span_id = %context.span_id(),
- reason = error.reason(),
- failed,
- "gateway telemetry upload failed"
- );
- }
- }
- }
- }
- .with_context(parent),
- );
- }
-
- fn drop_record(&self, reason: &'static str) {
- let dropped = self.0.stats.dropped.fetch_add(1, Ordering::Relaxed) + 1;
- crate::observability::delivery("dropped", reason);
- if dropped == 1 || dropped.is_multiple_of(100) {
- tracing::warn!(reason, dropped, "gateway telemetry dropped");
+ let item = Pending::new(span, size.bytes, link, retained);
+ match self.0.queue.try_send(Message::Record(context.grant, item)) {
+ Ok(()) => {}
+ Err(TrySendError::Full(_)) => self.0.stats.record_dropped(1, "capacity"),
+ Err(TrySendError::Closed(_)) => self.0.stats.record_dropped(1, "shutdown"),
}
}
- /// Stop accepting work and finish uploads within the existing process drain deadline.
+ /// Stop accepting work, flush open batches, and finish uploads within the existing
+ /// process drain deadline.
///
/// # Panics
- /// Panics if the upload task lock was poisoned.
+ /// Panics if the worker handle lock was poisoned.
pub async fn shutdown(&self, deadline: Instant) {
- self.0.capacity.close();
- let tasks = self
+ let worker = self
.0
- .tasks
+ .worker
.lock()
- .expect("telemetry task lock poisoned")
+ .expect("telemetry worker lock poisoned")
.take();
- let Some(mut tasks) = tasks else {
+ let Some(worker) = worker else {
return;
};
- loop {
- match tokio::time::timeout_at(deadline, tasks.join_next()).await {
- Ok(Some(Ok(()))) => {}
- Ok(Some(Err(_))) => {
- self.0.stats.failed.fetch_add(1, Ordering::Relaxed);
- crate::observability::delivery("failed", "task");
- }
- Ok(None) => break,
- Err(_) => {
- tasks.abort_all();
- while let Some(result) = tasks.join_next().await {
- if result.is_err() {
- self.0.stats.dropped.fetch_add(1, Ordering::Relaxed);
- crate::observability::delivery("dropped", "shutdown");
- }
- }
- break;
- }
- }
+ if self.0.queue.send(Message::Shutdown(deadline)).await.is_ok() {
+ let _ = worker.await;
}
tracing::info!(
accepted = self.0.stats.accepted.load(Ordering::Relaxed),
@@ -212,7 +231,7 @@ impl Telemetry {
}
}
-/// Count serialized facts without allocating a second copy on the finalization path.
+/// Count serialized bytes without allocating a copy on the finalization path.
struct SizeCounter {
bytes: usize,
limit: usize,
diff --git a/ai-gateway/src/telemetry/otlp.rs b/ai-gateway/src/telemetry/otlp.rs
index 66a2cced08..71f7da88a8 100644
--- a/ai-gateway/src/telemetry/otlp.rs
+++ b/ai-gateway/src/telemetry/otlp.rs
@@ -1,20 +1,25 @@
use std::{
- io::{self, Write},
+ io::{self, BufWriter, Write},
time::{Duration, SystemTime, UNIX_EPOCH},
};
-use axum::http::{HeaderValue, header};
+use axum::{
+ body::Bytes,
+ http::{HeaderValue, header},
+};
+use flate2::{Compression, write::GzEncoder};
use reqwest::{Client, Url};
use reqwest_middleware::ClientWithMiddleware;
+use serde::Serialize;
use serde_json::{Value, json};
-use super::DeliveryContext;
+use super::Grant;
use crate::resolution::{ControlPlaneConfig, ResolutionError, signing};
const INGESTION_PATH: &str = "/api/public/otel/v1/traces";
const MAX_PAYLOAD_BYTES: usize = 8 * 1024 * 1024;
const MAX_RESPONSE_BYTES: usize = 64 * 1024;
-const UPLOAD_TIMEOUT: Duration = Duration::from_secs(5);
+const UPLOAD_TIMEOUT: Duration = Duration::from_secs(30);
pub(super) struct Uploader {
client: ClientWithMiddleware,
@@ -27,7 +32,10 @@ pub(super) enum ExportError {
Expired,
Payload,
Transport,
- Rejected,
+ Rejected {
+ status: u16,
+ retry_after: Option<Duration>,
+ },
Response,
Partial,
}
@@ -38,11 +46,26 @@ impl ExportError {
Self::Expired => "expired_grant",
Self::Payload => "payload",
Self::Transport => "transport",
- Self::Rejected => "http_status",
+ Self::Rejected { .. } => "http_status",
Self::Response => "invalid_response",
Self::Partial => "rejected_spans",
}
}
+
+ pub fn is_transient(&self) -> bool {
+ match self {
+ Self::Transport => true,
+ Self::Rejected { status, .. } => matches!(status, 408 | 429 | 500 | 502 | 503 | 504),
+ _ => false,
+ }
+ }
+
+ pub fn retry_after(&self) -> Option<Duration> {
+ match self {
+ Self::Rejected { retry_after, .. } => *retry_after,
+ _ => None,
+ }
+ }
}
impl Uploader {
@@ -67,18 +90,7 @@ impl Uploader {
})
}
- /// A collection allows project batching without changing the HTTP transport.
- pub async fn export(
- &self,
- grant: &DeliveryContext,
- spans: &[Value],
- ) -> Result<(), ExportError> {
- let payload = json!({"resourceSpans": [{
- "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "langfuse-ai-gateway"}}]},
- "scopeSpans": [{"scope": {"name": "langfuse-ai-gateway", "version": env!("CARGO_PKG_VERSION")}, "spans": spans}]
- }]});
- let mut body = LimitedBody(Vec::new());
- serde_json::to_writer(&mut body, &payload).map_err(|_| ExportError::Payload)?;
+ pub async fn export(&self, grant: &Grant, payload: &Payload) -> Result<(), ExportError> {
let now = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map_err(|_| ExportError::Expired)?;
@@ -102,17 +114,28 @@ impl Uploader {
.header(header::AUTHORIZATION, authorization)
.header("langfuse-gateway-authorization", signature)
.header(header::CONTENT_TYPE, "application/json")
+ .header(header::CONTENT_ENCODING, "gzip")
.header(header::ACCEPT_ENCODING, "identity")
.header("x-langfuse-sdk-name", "langfuse-ai-gateway")
.header("x-langfuse-sdk-version", env!("CARGO_PKG_VERSION"))
.header("x-langfuse-ingestion-version", "4")
.timeout(UPLOAD_TIMEOUT.min(remaining))
- .body(body.0)
+ .body(payload.0.clone())
.send()
.await
.map_err(|_| ExportError::Transport)?;
- if response.status().as_u16() != 200 {
- return Err(ExportError::Rejected);
+ let status = response.status().as_u16();
+ if status != 200 {
+ let retry_after = response
+ .headers()
+ .get(header::RETRY_AFTER)
+ .and_then(|value| value.to_str().ok())
+ .and_then(|value| value.trim().parse().ok())
+ .map(Duration::from_secs);
+ return Err(ExportError::Rejected {
+ status,
+ retry_after,
+ });
}
if response
.content_length()
@@ -121,7 +144,7 @@ impl Uploader {
return Err(ExportError::Response);
}
let mut bytes = Vec::new();
- while let Some(chunk) = response.chunk().await.map_err(|_| ExportError::Transport)? {
+ while let Some(chunk) = response.chunk().await.map_err(|_| ExportError::Response)? {
if chunk.len() > MAX_RESPONSE_BYTES - bytes.len() {
return Err(ExportError::Response);
}
@@ -150,17 +173,43 @@ impl Uploader {
}
}
-struct LimitedBody(Vec<u8>);
+pub(super) struct Payload(Bytes);
+
+impl Payload {
+ pub fn encode(spans: &[impl Serialize]) -> Result<Self, ExportError> {
+ let payload = json!({"resourceSpans": [{
+ "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "langfuse-ai-gateway"}}]},
+ "scopeSpans": [{"scope": {"name": "langfuse-ai-gateway", "version": env!("CARGO_PKG_VERSION")}, "spans": spans}]
+ }]});
+ let mut writer = BufWriter::with_capacity(
+ 64 * 1024,
+ LimitedBody {
+ written: 0,
+ encoder: GzEncoder::new(Vec::new(), Compression::fast()),
+ },
+ );
+ serde_json::to_writer(&mut writer, &payload).map_err(|_| ExportError::Payload)?;
+ let body = writer.into_inner().map_err(|_| ExportError::Payload)?;
+ let compressed = body.encoder.finish().map_err(|_| ExportError::Payload)?;
+ Ok(Self(compressed.into()))
+ }
+}
+
+struct LimitedBody {
+ written: usize,
+ encoder: GzEncoder<Vec<u8>>,
+}
impl Write for LimitedBody {
fn write(&mut self, bytes: &[u8]) -> io::Result<usize> {
- if bytes.len() > MAX_PAYLOAD_BYTES - self.0.len() {
+ if bytes.len() > MAX_PAYLOAD_BYTES - self.written {
return Err(io::Error::other("telemetry payload limit"));
}
- self.0.extend_from_slice(bytes);
+ self.encoder.write_all(bytes)?;
+ self.written += bytes.len();
Ok(bytes.len())
}
fn flush(&mut self) -> io::Result<()> {
- Ok(())
+ self.encoder.flush()
}
}
diff --git a/ai-gateway/src/telemetry/retry.rs b/ai-gateway/src/telemetry/retry.rs
new file mode 100644
index 0000000000..b338e0a921
--- /dev/null
+++ b/ai-gateway/src/telemetry/retry.rs
@@ -0,0 +1,57 @@
+use std::time::Duration;
+
+use super::otlp::ExportError;
+
+#[derive(Clone, Copy)]
+pub(super) struct RetryPolicy {
+ pub max_attempts: u32,
+ pub base_delay: Duration,
+ pub max_delay: Duration,
+}
+
+impl Default for RetryPolicy {
+ fn default() -> Self {
+ Self {
+ max_attempts: 3,
+ base_delay: Duration::from_millis(500),
+ max_delay: Duration::from_secs(10),
+ }
+ }
+}
+
+impl RetryPolicy {
+ pub fn backoff(
+ &self,
+ attempt: u32,
+ error: &ExportError,
+ grant_remaining: Duration,
+ random: u64,
+ ) -> Option<Duration> {
+ if attempt >= self.max_attempts || !error.is_transient() {
+ return None;
+ }
+ let step = self
+ .base_delay
+ .saturating_mul(4u32.saturating_pow(attempt.saturating_sub(1)))
+ .min(self.max_delay);
+ let half = step / 2;
+ let spread = u64::try_from(half.as_nanos()).unwrap_or(u64::MAX);
+ let jittered = half + Duration::from_nanos(random % spread.saturating_add(1));
+ let delay = match error.retry_after() {
+ Some(hint) if hint > self.max_delay => return None,
+ Some(hint) => hint.max(jittered),
+ None => jittered,
+ };
+ (delay + MIN_SEND_WINDOW <= grant_remaining).then_some(delay)
+ }
+}
+
+const MIN_SEND_WINDOW: Duration = Duration::from_secs(1);
+
+pub(super) fn random() -> u64 {
+ use std::hash::BuildHasher;
+ std::hash::RandomState::new().hash_one(std::time::Instant::now())
+}
+
+#[cfg(test)]
+mod tests;
diff --git a/ai-gateway/src/telemetry/retry/tests.rs b/ai-gateway/src/telemetry/retry/tests.rs
new file mode 100644
index 0000000000..4ebae33be2
--- /dev/null
+++ b/ai-gateway/src/telemetry/retry/tests.rs
@@ -0,0 +1,101 @@
+use std::time::Duration;
+
+use super::*;
+
+const GRANT: Duration = Duration::from_mins(15);
+
+fn status(status: u16) -> ExportError {
+ ExportError::Rejected {
+ status,
+ retry_after: None,
+ }
+}
+
+#[test]
+fn transient_failures_back_off_four_times_longer_within_the_upper_half() {
+ let policy = RetryPolicy::default();
+ for (attempt, low_ms, high_ms) in [(1, 250, 500), (2, 1_000, 2_000)] {
+ let low = Duration::from_millis(low_ms);
+ let high = Duration::from_millis(high_ms);
+ assert_eq!(
+ policy.backoff(attempt, &ExportError::Transport, GRANT, 0),
+ Some(low)
+ );
+ let spread = (high_ms - low_ms) * 1_000_000;
+ assert_eq!(
+ policy.backoff(attempt, &ExportError::Transport, GRANT, spread),
+ Some(high)
+ );
+ for random in [1, 7_777_777, u64::MAX] {
+ let delay = policy
+ .backoff(attempt, &ExportError::Transport, GRANT, random)
+ .unwrap();
+ assert!((low..=high).contains(&delay), "{delay:?}");
+ }
+ }
+ assert_eq!(policy.backoff(3, &ExportError::Transport, GRANT, 0), None);
+}
+
+#[test]
+fn only_failures_that_a_resend_can_fix_are_retried() {
+ let policy = RetryPolicy::default();
+ for error in [
+ status(408),
+ status(429),
+ status(500),
+ status(502),
+ status(503),
+ status(504),
+ ] {
+ assert!(policy.backoff(1, &error, GRANT, 0).is_some(), "{error:?}");
+ }
+ for error in [
+ status(307),
+ status(400),
+ status(401),
+ status(403),
+ status(413),
+ status(501),
+ ExportError::Expired,
+ ExportError::Payload,
+ ExportError::Response,
+ ExportError::Partial,
+ ] {
+ assert_eq!(policy.backoff(1, &error, GRANT, 0), None, "{error:?}");
+ }
+}
+
+#[test]
+fn retry_after_extends_the_backoff_but_never_beyond_the_maximum_wait() {
+ let policy = RetryPolicy::default();
+ let throttled = |seconds| ExportError::Rejected {
+ status: 429,
+ retry_after: Some(Duration::from_secs(seconds)),
+ };
+ assert_eq!(
+ policy.backoff(1, &throttled(7), GRANT, 0),
+ Some(Duration::from_secs(7))
+ );
+ assert_eq!(
+ policy.backoff(1, &throttled(0), GRANT, 0),
+ Some(Duration::from_millis(250))
+ );
+ assert_eq!(policy.backoff(1, &throttled(11), GRANT, 0), None);
+}
+
+#[test]
+fn retries_stop_when_the_grant_would_expire_before_the_resend() {
+ let policy = RetryPolicy::default();
+ let throttled = ExportError::Rejected {
+ status: 503,
+ retry_after: Some(Duration::from_secs(5)),
+ };
+ assert_eq!(
+ policy.backoff(1, &throttled, Duration::from_secs(6), 0),
+ Some(Duration::from_secs(5))
+ );
+ assert_eq!(
+ policy.backoff(1, &throttled, Duration::from_millis(5_999), 0),
+ None
+ );
+}
diff --git a/ai-gateway/src/telemetry/tests.rs b/ai-gateway/src/telemetry/tests.rs
index edb6554665..fc908ec9de 100644
--- a/ai-gateway/src/telemetry/tests.rs
+++ b/ai-gateway/src/telemetry/tests.rs
@@ -7,11 +7,16 @@ use axum::{
use serde_json::{Value, json};
use tokio::sync::Notify;
-use super::{otlp::ExportError, *};
+use super::{
+ otlp::{ExportError, Payload},
+ *,
+};
use crate::{
capture::RelayOutcome,
resolution::signing,
- test_support::{FakeServer, resolved_request_context},
+ test_support::{
+ FakeServer, ingestion_token, resolved_request_context, upload_json, upload_text,
+ },
};
fn facts(project: &str) -> InferenceFacts {
@@ -40,6 +45,18 @@ fn uploader(url: &str) -> Uploader {
Uploader::new(&ControlPlaneConfig::new(url, "test-service-key").unwrap()).unwrap()
}
+fn fast_retry() -> RetryPolicy {
+ RetryPolicy {
+ max_attempts: 3,
+ base_delay: Duration::from_millis(2),
+ max_delay: Duration::from_millis(10),
+ }
+}
+
+fn empty_payload() -> Payload {
+ Payload::encode(&[json!({})]).unwrap()
+}
+
fn response(status: u16, body: impl Into<Body>) -> Response<Body> {
Response::builder()
.status(status)
@@ -48,12 +65,13 @@ fn response(status: u16, body: impl Into<Body>) -> Response<Body> {
}
#[tokio::test]
-async fn immediate_upload_uses_prefixed_path_signed_grant_and_gateway_sdk_headers() {
+async fn upload_uses_prefixed_path_signed_grant_and_gateway_sdk_headers() {
let web = FakeServer::start(|request| async move {
assert_eq!(request.method(), "POST");
assert_eq!(request.uri().path(), "/app/api/public/otel/v1/traces");
let headers = request.headers();
- assert_eq!(headers["authorization"], "Bearer private-ingestion-token");
+ let token = ingestion_token("org-1", "project-1");
+ assert_eq!(headers["authorization"], format!("Bearer {token}").as_str());
let signature = headers["langfuse-gateway-authorization"].to_str().unwrap();
let timestamp = signature
.strip_prefix("HMAC timestamp=")
@@ -65,21 +83,22 @@ async fn immediate_upload_uses_prefixed_path_signed_grant_and_gateway_sdk_header
.unwrap();
assert_eq!(
signature,
- signing::authorization("test-service-key", "private-ingestion-token", timestamp)
+ signing::authorization("test-service-key", &token, timestamp)
);
assert_eq!(headers["content-type"], "application/json");
+ assert_eq!(headers["content-encoding"], "gzip");
assert_eq!(headers["x-langfuse-sdk-name"], "langfuse-ai-gateway");
assert_eq!(headers["x-langfuse-sdk-version"], env!("CARGO_PKG_VERSION"));
assert_eq!(headers["x-langfuse-ingestion-version"], "4");
let bytes = to_bytes(request.into_body(), 64 * 1024).await.unwrap();
- let payload: Value = serde_json::from_slice(&bytes).unwrap();
+ let payload = upload_json(&bytes);
let spans = payload["resourceSpans"][0]["scopeSpans"][0]["spans"]
.as_array()
.unwrap();
assert_eq!(spans.len(), 1);
assert_eq!(spans[0]["traceId"].as_str().unwrap().len(), 32);
assert_eq!(spans[0]["spanId"].as_str().unwrap().len(), 16);
- let body = String::from_utf8(bytes.to_vec()).unwrap();
+ let body = upload_text(&bytes);
assert!(!body.contains("provider-secret"));
assert!(!body.contains("private-ingestion-token"));
response(200, "{}")
@@ -87,6 +106,7 @@ async fn immediate_upload_uses_prefixed_path_signed_grant_and_gateway_sdk_header
.await;
let telemetry = Telemetry::new(
&ControlPlaneConfig::new(&format!("{}/app", web.url), "test-service-key").unwrap(),
+ DEFAULT_RETAINED_BYTES,
)
.unwrap();
telemetry.record(grant().await, facts("project-1"));
@@ -113,7 +133,7 @@ async fn concurrent_projects_keep_their_original_grants_and_attribution() {
.unwrap()
.to_owned();
let bytes = to_bytes(request.into_body(), 64 * 1024).await.unwrap();
- let payload: Value = serde_json::from_slice(&bytes).unwrap();
+ let payload = upload_json(&bytes);
let attributes = payload["resourceSpans"][0]["scopeSpans"][0]["spans"][0]["attributes"]
.as_array()
.unwrap();
@@ -134,11 +154,17 @@ async fn concurrent_projects_keep_their_original_grants_and_attribution() {
}
})
.await;
- let telemetry = Telemetry::with_uploader(uploader(&web.url), 2, MAX_RETAINED_BYTES);
+ let telemetry = Telemetry::with_uploader(
+ uploader(&web.url),
+ 2,
+ DEFAULT_RETAINED_BYTES,
+ BatchPolicy::default(),
+ fast_retry(),
+ );
for project in ["project-a", "project-b"] {
let mut grant = grant().await;
- grant.project_id = project.into();
- grant.access_token = format!("token-{project}");
+ grant.grant.project_id = project.into();
+ grant.grant.access_token = format!("token-{project}");
telemetry.record(grant, facts(project));
}
telemetry
@@ -157,10 +183,17 @@ async fn concurrent_projects_keep_their_original_grants_and_attribution() {
}
#[tokio::test]
-async fn upload_response_failures_are_reported_without_retries() {
+async fn upload_response_failures_are_classified_per_attempt() {
let grant = grant().await;
for (status, body, expected) in [
- (500, "{}", Err(ExportError::Rejected)),
+ (
+ 500,
+ "{}",
+ Err(ExportError::Rejected {
+ status: 500,
+ retry_after: None,
+ }),
+ ),
(
200,
r#"{"partialSuccess":{"rejectedSpans":"1"}}"#,
@@ -182,7 +215,9 @@ async fn upload_response_failures_are_reported_without_retries() {
] {
let web = FakeServer::start(move |_| async move { response(status, body) }).await;
assert_eq!(
- uploader(&web.url).export(&grant, &[json!({})]).await,
+ uploader(&web.url)
+ .export(&grant.grant, &empty_payload())
+ .await,
expected
);
assert_eq!(web.calls(), 1);
@@ -194,10 +229,29 @@ async fn upload_response_failures_are_reported_without_retries() {
})
.await;
assert_eq!(
- uploader(&web.url).export(&grant, &[json!({})]).await,
+ uploader(&web.url)
+ .export(&grant.grant, &empty_payload())
+ .await,
Err(ExportError::Response)
);
assert_eq!(web.calls(), 1);
+ let web = FakeServer::start(|_| async {
+ Response::builder()
+ .status(429)
+ .header("retry-after", " 7 ")
+ .body(Body::empty())
+ .unwrap()
+ })
+ .await;
+ assert_eq!(
+ uploader(&web.url)
+ .export(&grant.grant, &empty_payload())
+ .await,
+ Err(ExportError::Rejected {
+ status: 429,
+ retry_after: Some(Duration::from_secs(7)),
+ })
+ );
}
#[tokio::test]
@@ -217,9 +271,12 @@ async fn redirects_do_not_forward_the_ingestion_credentials() {
.await;
assert_eq!(
uploader(&web.url)
- .export(&grant().await, &[json!({})])
+ .export(&grant().await.grant, &empty_payload())
.await,
- Err(ExportError::Rejected)
+ Err(ExportError::Rejected {
+ status: 307,
+ retry_after: None,
+ })
);
assert_eq!(web.calls(), 1);
assert_eq!(destination.calls(), 0);
@@ -230,21 +287,285 @@ async fn expired_grants_and_oversized_payloads_never_reach_the_network() {
let web = FakeServer::start(|_| async { response(200, "{}") }).await;
let uploader = uploader(&web.url);
let mut expired = grant().await;
- expired.expires_at = 1;
+ expired.grant.expires_at = 1;
assert_eq!(
- uploader.export(&expired, &[json!({})]).await,
+ uploader.export(&expired.grant, &empty_payload()).await,
Err(ExportError::Expired)
);
let oversized = json!({"oversized": "x".repeat(8 * 1024 * 1024)});
assert_eq!(
- uploader.export(&grant().await, &[oversized]).await,
- Err(ExportError::Payload)
+ Payload::encode(&[oversized]).err(),
+ Some(ExportError::Payload)
);
assert_eq!(web.calls(), 0);
}
+fn record_bytes(context: &DeliveryContext, facts: InferenceFacts) -> usize {
+ serde_json::to_vec(&mapping::span(facts, &context.generation))
+ .unwrap()
+ .len()
+ + context.grant.access_token.len()
+ + context.grant.project_id.len()
+}
+
+fn exported_spans(payload: &Value) -> usize {
+ payload["resourceSpans"][0]["scopeSpans"][0]["spans"]
+ .as_array()
+ .unwrap()
+ .len()
+}
+
#[tokio::test]
-async fn upload_capacity_and_shutdown_deadline_bound_hanging_delivery() {
+async fn records_of_one_project_share_an_upload_with_the_latest_expiring_grant() {
+ let observed = Arc::new(Mutex::new(Vec::new()));
+ let received = observed.clone();
+ let web = FakeServer::start(move |request| {
+ let received = received.clone();
+ async move {
+ let authorization = request.headers()["authorization"]
+ .to_str()
+ .unwrap()
+ .to_owned();
+ let bytes = to_bytes(request.into_body(), 64 * 1024).await.unwrap();
+ let payload = upload_json(&bytes);
+ received
+ .lock()
+ .unwrap()
+ .push((authorization, exported_spans(&payload)));
+ response(200, "{}")
+ }
+ })
+ .await;
+ let telemetry = Telemetry::with_uploader(
+ uploader(&web.url),
+ 2,
+ DEFAULT_RETAINED_BYTES,
+ BatchPolicy::default(),
+ fast_retry(),
+ );
+ for (token, extra_seconds) in [("token-a", 0), ("token-b", 60), ("token-c", 30)] {
+ let mut context = grant().await;
+ context.grant.access_token = token.into();
+ context.grant.expires_at += extra_seconds;
+ telemetry.record(context, facts("project-1"));
+ }
+ telemetry
+ .shutdown(Instant::now() + Duration::from_secs(2))
+ .await;
+ assert_eq!(
+ *observed.lock().unwrap(),
+ vec![("Bearer token-b".to_owned(), 3)]
+ );
+ assert_eq!(telemetry.0.stats.accepted.load(Ordering::Relaxed), 3);
+}
+
+fn span_projects(payload: &Value) -> Vec<String> {
+ payload["resourceSpans"][0]["scopeSpans"][0]["spans"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .map(|span| {
+ let metadata = span["attributes"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .find(|attribute| attribute["key"] == "langfuse.observation.metadata")
+ .unwrap();
+ let metadata: Value =
+ serde_json::from_str(metadata["value"]["stringValue"].as_str().unwrap()).unwrap();
+ metadata["langfuse.gateway.project.id"]
+ .as_str()
+ .unwrap()
+ .to_owned()
+ })
+ .collect()
+}
+
+#[tokio::test]
+async fn interleaved_projects_never_share_an_upload_or_a_grant() {
+ let observed = Arc::new(Mutex::new(Vec::new()));
+ let received = observed.clone();
+ let web = FakeServer::start(move |request| {
+ let received = received.clone();
+ async move {
+ let authorization = request.headers()["authorization"]
+ .to_str()
+ .unwrap()
+ .to_owned();
+ let bytes = to_bytes(request.into_body(), 64 * 1024).await.unwrap();
+ let payload = upload_json(&bytes);
+ received
+ .lock()
+ .unwrap()
+ .push((authorization, span_projects(&payload)));
+ response(200, "{}")
+ }
+ })
+ .await;
+ let telemetry = Telemetry::with_uploader(
+ uploader(&web.url),
+ 2,
+ DEFAULT_RETAINED_BYTES,
+ BatchPolicy {
+ max_records: 3,
+ ..BatchPolicy::default()
+ },
+ fast_retry(),
+ );
+ // Later records carry later-expiring grants, so each batch adopts a new grant
+ // while the other project's batch is open.
+ for round in 0..4u64 {
+ for project in ["project-a", "project-b"] {
+ let mut context = grant().await;
+ context.grant.project_id = project.into();
+ context.grant.access_token = format!("{project}-token-{round}");
+ context.grant.expires_at += round * 10;
+ telemetry.record(context, facts(project));
+ }
+ }
+ telemetry
+ .shutdown(Instant::now() + Duration::from_secs(2))
+ .await;
+ let mut uploads = observed.lock().unwrap().clone();
+ uploads.sort();
+ let spans = |project: &str, count| vec![project.to_owned(); count];
+ assert_eq!(
+ uploads,
+ vec![
+ ("Bearer project-a-token-2".into(), spans("project-a", 3)),
+ ("Bearer project-a-token-3".into(), spans("project-a", 1)),
+ ("Bearer project-b-token-2".into(), spans("project-b", 3)),
+ ("Bearer project-b-token-3".into(), spans("project-b", 1)),
+ ]
+ );
+ assert_eq!(telemetry.0.stats.accepted.load(Ordering::Relaxed), 8);
+}
+
+#[tokio::test]
+async fn open_batches_upload_once_their_linger_elapses() {
+ let web = FakeServer::start(|_| async { response(200, "{}") }).await;
+ let telemetry = Telemetry::with_uploader(
+ uploader(&web.url),
+ 1,
+ DEFAULT_RETAINED_BYTES,
+ BatchPolicy {
+ linger: Duration::from_millis(20),
+ ..BatchPolicy::default()
+ },
+ fast_retry(),
+ );
+ telemetry.record(grant().await, facts("project-1"));
+ tokio::time::timeout(Duration::from_secs(2), async {
+ while telemetry.0.stats.accepted.load(Ordering::Relaxed) == 0 {
+ tokio::time::sleep(Duration::from_millis(5)).await;
+ }
+ })
+ .await
+ .unwrap();
+ assert_eq!(web.calls(), 1);
+ telemetry
+ .shutdown(Instant::now() + Duration::from_secs(2))
+ .await;
+ assert_eq!(web.calls(), 1);
+}
+
+async fn deliver_three_records(web: &FakeServer) -> Telemetry {
+ let telemetry = Telemetry::with_uploader(
+ uploader(&web.url),
+ 1,
+ DEFAULT_RETAINED_BYTES,
+ BatchPolicy::default(),
+ fast_retry(),
+ );
+ for _ in 0..3 {
+ telemetry.record(grant().await, facts("project-1"));
+ }
+ telemetry
+ .shutdown(Instant::now() + Duration::from_secs(2))
+ .await;
+ telemetry
+}
+
+#[tokio::test]
+async fn transient_failures_resend_the_identical_batch_until_accepted() {
+ let bodies = Arc::new(Mutex::new(Vec::new()));
+ let received = bodies.clone();
+ let web = FakeServer::start(move |request| {
+ let received = received.clone();
+ async move {
+ let bytes = to_bytes(request.into_body(), 64 * 1024).await.unwrap();
+ let mut received = received.lock().unwrap();
+ received.push(bytes);
+ if received.len() == 1 {
+ response(503, "{}")
+ } else {
+ response(200, "{}")
+ }
+ }
+ })
+ .await;
+ let telemetry = deliver_three_records(&web).await;
+ assert_eq!(web.calls(), 2);
+ let bodies = bodies.lock().unwrap();
+ assert_eq!(bodies[0], bodies[1]);
+ assert_eq!(telemetry.0.stats.accepted.load(Ordering::Relaxed), 3);
+ assert_eq!(telemetry.0.stats.failed.load(Ordering::Relaxed), 0);
+ assert_eq!(
+ telemetry.0.retained.available_permits(),
+ DEFAULT_RETAINED_BYTES
+ );
+}
+
+#[tokio::test]
+async fn batches_that_keep_failing_count_every_record_once_after_the_last_attempt() {
+ let web = FakeServer::start(|_| async { response(500, "{}") }).await;
+ let telemetry = deliver_three_records(&web).await;
+ assert_eq!(web.calls(), 3);
+ assert_eq!(telemetry.0.stats.failed.load(Ordering::Relaxed), 3);
+ assert_eq!(telemetry.0.stats.accepted.load(Ordering::Relaxed), 0);
+ assert_eq!(
+ telemetry.0.retained.available_permits(),
+ DEFAULT_RETAINED_BYTES
+ );
+}
+
+#[tokio::test]
+async fn permanent_rejections_are_not_retried() {
+ let web = FakeServer::start(|_| async { response(400, "{}") }).await;
+ let telemetry = deliver_three_records(&web).await;
+ assert_eq!(web.calls(), 1);
+ assert_eq!(telemetry.0.stats.failed.load(Ordering::Relaxed), 3);
+}
+
+#[tokio::test]
+async fn shutdown_deadline_cuts_retry_backoff_short() {
+ let web = FakeServer::start(|_| async { response(503, "{}") }).await;
+ let telemetry = Telemetry::with_uploader(
+ uploader(&web.url),
+ 1,
+ DEFAULT_RETAINED_BYTES,
+ BatchPolicy::default(),
+ RetryPolicy {
+ max_attempts: 3,
+ base_delay: Duration::from_secs(60),
+ max_delay: Duration::from_secs(60),
+ },
+ );
+ telemetry.record(grant().await, facts("project-1"));
+ telemetry
+ .shutdown(Instant::now() + Duration::from_millis(200))
+ .await;
+ assert_eq!(web.calls(), 1);
+ assert_eq!(telemetry.0.stats.dropped.load(Ordering::Relaxed), 1);
+ assert_eq!(telemetry.0.stats.failed.load(Ordering::Relaxed), 0);
+ assert_eq!(
+ telemetry.0.retained.available_permits(),
+ DEFAULT_RETAINED_BYTES
+ );
+}
+
+#[tokio::test]
+async fn shutdown_deadline_bounds_hanging_delivery_and_closes_admission() {
let started = Arc::new(Notify::new());
let received = started.clone();
let web = FakeServer::start(move |_| {
@@ -256,17 +577,29 @@ async fn upload_capacity_and_shutdown_deadline_bound_hanging_delivery() {
}
})
.await;
- let telemetry = Telemetry::with_uploader(uploader(&web.url), 1, MAX_RETAINED_BYTES);
+ let telemetry = Telemetry::with_uploader(
+ uploader(&web.url),
+ 1,
+ DEFAULT_RETAINED_BYTES,
+ BatchPolicy {
+ max_records: 1,
+ ..BatchPolicy::default()
+ },
+ fast_retry(),
+ );
telemetry.record(grant().await, facts("project-1"));
tokio::time::timeout(Duration::from_secs(2), started.notified())
.await
.unwrap();
- telemetry.record(grant().await, facts("project-1"));
- assert_eq!(telemetry.0.stats.dropped.load(Ordering::Relaxed), 1);
+ // Waits for the only upload slot behind the hanging upload.
+ telemetry.record(grant().await, facts("project-2"));
+ assert_eq!(telemetry.0.stats.dropped.load(Ordering::Relaxed), 0);
telemetry.shutdown(Instant::now()).await;
assert_eq!(telemetry.0.stats.dropped.load(Ordering::Relaxed), 2);
- assert_eq!(telemetry.0.capacity.available_permits(), 1);
- assert_eq!(telemetry.0.bytes.available_permits(), MAX_RETAINED_BYTES);
+ assert_eq!(
+ telemetry.0.retained.available_permits(),
+ DEFAULT_RETAINED_BYTES
+ );
telemetry.record(grant().await, facts("project-1"));
assert_eq!(telemetry.0.stats.dropped.load(Ordering::Relaxed), 3);
assert_eq!(web.calls(), 1);
@@ -289,16 +622,22 @@ async fn retained_byte_budget_is_released_after_successful_drain() {
})
.await;
let grant = grant().await;
- let budget = serde_json::to_vec(&facts("project-1")).unwrap().len()
- + grant.access_token.len()
- + grant.project_id.len()
- + serde_json::to_vec(&grant.generation).unwrap().len();
- let telemetry = Telemetry::with_uploader(uploader(&web.url), 2, budget);
+ let budget = record_bytes(&grant, facts("project-1"));
+ let telemetry = Telemetry::with_uploader(
+ uploader(&web.url),
+ 2,
+ budget,
+ BatchPolicy {
+ max_records: 1,
+ ..BatchPolicy::default()
+ },
+ fast_retry(),
+ );
telemetry.record(grant, facts("project-1"));
tokio::time::timeout(Duration::from_secs(2), started.notified())
.await
.unwrap();
- assert_eq!(telemetry.0.bytes.available_permits(), 0);
+ assert_eq!(telemetry.0.retained.available_permits(), 0);
let second = DeliveryContext::from_resolved(
&resolved_request_context("provider-secret").await,
&HeaderMap::new(),
@@ -310,22 +649,29 @@ async fn retained_byte_budget_is_released_after_successful_drain() {
telemetry
.shutdown(Instant::now() + Duration::from_secs(2))
.await;
- assert_eq!(telemetry.0.bytes.available_permits(), budget);
- assert_eq!(telemetry.0.capacity.available_permits(), 2);
+ assert_eq!(telemetry.0.retained.available_permits(), budget);
assert_eq!(telemetry.0.stats.accepted.load(Ordering::Relaxed), 1);
assert_eq!(web.calls(), 1);
}
#[tokio::test]
-async fn oversized_facts_are_dropped_before_task_admission() {
+async fn oversized_records_are_dropped_before_admission() {
let web = FakeServer::start(|_| async { response(200, "{}") }).await;
- let telemetry = Telemetry::with_uploader(uploader(&web.url), 1, MAX_RETAINED_BYTES);
+ let telemetry = Telemetry::with_uploader(
+ uploader(&web.url),
+ 1,
+ DEFAULT_RETAINED_BYTES,
+ BatchPolicy::default(),
+ fast_retry(),
+ );
let mut oversized = facts("project-1");
- oversized.metadata["oversized"] = json!("x".repeat(MAX_FACT_BYTES));
+ oversized.metadata["key_metadata"] = json!({"oversized": "x".repeat(MAX_RECORD_BYTES)});
telemetry.record(grant().await, oversized);
assert_eq!(telemetry.0.stats.dropped.load(Ordering::Relaxed), 1);
- assert_eq!(telemetry.0.capacity.available_permits(), 1);
- assert_eq!(telemetry.0.bytes.available_permits(), MAX_RETAINED_BYTES);
+ assert_eq!(
+ telemetry.0.retained.available_permits(),
+ DEFAULT_RETAINED_BYTES
+ );
telemetry
.shutdown(Instant::now() + Duration::from_secs(2))
.await;
diff --git a/ai-gateway/src/telemetry/worker.rs b/ai-gateway/src/telemetry/worker.rs
new file mode 100644
index 0000000000..321380608c
--- /dev/null
+++ b/ai-gateway/src/telemetry/worker.rs
@@ -0,0 +1,208 @@
+use std::sync::Arc;
+
+use opentelemetry::trace::TraceContextExt;
+use tokio::{
+ sync::{Semaphore, mpsc},
+ task::JoinSet,
+ time::Instant,
+};
+use tracing::Instrument;
+use tracing_opentelemetry::OpenTelemetrySpanExt;
+
+use super::{
+ Grant, Stats,
+ batch::{Batches, Flush, Pending},
+ otlp::{ExportError, Payload, Uploader},
+ retry::{RetryPolicy, random},
+};
+
+pub(super) enum Message {
+ Record(Grant, Pending),
+ Shutdown(Instant),
+}
+
+pub(super) async fn run_worker(
+ mut queue: mpsc::Receiver<Message>,
+ mut batches: Batches,
+ mut uploads: Uploads,
+) {
+ let deadline = loop {
+ let due = batches.next_due();
+ tokio::select! {
+ message = queue.recv() => match message {
+ Some(Message::Record(grant, item)) => {
+ uploads.start_flushes(batches.push_record(grant, item, Instant::now()));
+ }
+ Some(Message::Shutdown(deadline)) => break deadline,
+ None => break Instant::now(),
+ },
+ () = sleep_until_due(due) => uploads.start_flushes(batches.take_due(Instant::now())),
+ }
+ };
+ queue.close();
+ while let Some(message) = queue.recv().await {
+ if let Message::Record(grant, item) = message {
+ uploads.start_flushes(batches.push_record(grant, item, Instant::now()));
+ }
+ }
+ uploads.start_flushes(batches.take_all());
+ uploads.drain_until(deadline).await;
+}
+
+async fn sleep_until_due(due: Option<Instant>) {
+ match due {
+ Some(due) => tokio::time::sleep_until(due).await,
+ None => std::future::pending().await,
+ }
+}
+
+pub(super) struct Uploads {
+ uploader: Arc<Uploader>,
+ slots: Arc<Semaphore>,
+ retry: RetryPolicy,
+ tasks: JoinSet<()>,
+ stats: Arc<Stats>,
+}
+
+impl Uploads {
+ pub fn new(uploader: Uploader, slots: usize, retry: RetryPolicy, stats: Arc<Stats>) -> Self {
+ Self {
+ uploader: Arc::new(uploader),
+ slots: Arc::new(Semaphore::new(slots)),
+ retry,
+ tasks: JoinSet::new(),
+ stats,
+ }
+ }
+
+ fn start_flushes(&mut self, flushes: Vec<Flush>) {
+ for flush in flushes {
+ self.start_flush(flush);
+ }
+ }
+
+ fn start_flush(&mut self, flush: Flush) {
+ while self.tasks.try_join_next().is_some() {}
+ let receipt = Receipt::new(flush.items.len(), self.stats.clone());
+ let span = tracing::info_span!(
+ parent: None,
+ "telemetry.batch",
+ otel.kind = "internal",
+ gateway.telemetry.records = i64::try_from(flush.items.len()).unwrap_or(i64::MAX),
+ gateway.telemetry.attempts = tracing::field::Empty
+ );
+ for item in &flush.items {
+ if item.link.is_valid() {
+ span.add_link(item.link.clone());
+ }
+ }
+ let uploader = self.uploader.clone();
+ let slots = self.slots.clone();
+ let retry = self.retry;
+ self.tasks.spawn(
+ async move {
+ let mut flush = flush;
+ let payload = match encode(&mut flush.items).await {
+ Ok(payload) => payload,
+ Err(error) => return receipt.settle_failed(error.reason()),
+ };
+ let mut attempt = 1;
+ let outcome = loop {
+ let Ok(slot) = slots.acquire().await else {
+ return;
+ };
+ let result = uploader.export(&flush.grant, &payload).await;
+ drop(slot);
+ let Err(error) = result else { break Ok(()) };
+ let Some(delay) =
+ retry.backoff(attempt, &error, flush.grant.remaining(), random())
+ else {
+ break Err(error);
+ };
+ tracing::debug!(
+ attempt,
+ reason = error.reason(),
+ delay_ms = u64::try_from(delay.as_millis()).unwrap_or(u64::MAX),
+ "gateway telemetry upload retrying"
+ );
+ tokio::time::sleep(delay).await;
+ attempt += 1;
+ };
+ tracing::Span::current().record("gateway.telemetry.attempts", i64::from(attempt));
+ match outcome {
+ Ok(()) => receipt.settle_accepted(),
+ Err(error) => receipt.settle_failed(error.reason()),
+ }
+ }
+ .instrument(span),
+ );
+ }
+
+ async fn drain_until(mut self, deadline: Instant) {
+ loop {
+ match tokio::time::timeout_at(deadline, self.tasks.join_next()).await {
+ Ok(Some(_)) => {}
+ Ok(None) => break,
+ Err(_) => {
+ self.tasks.abort_all();
+ while self.tasks.join_next().await.is_some() {}
+ break;
+ }
+ }
+ }
+ }
+}
+
+async fn encode(items: &mut [Pending]) -> Result<Payload, ExportError> {
+ let spans: Vec<_> = items.iter_mut().map(|item| item.span.take()).collect();
+ tokio::task::spawn_blocking(move || Payload::encode(&spans))
+ .await
+ .unwrap_or(Err(ExportError::Payload))
+}
+
+struct Receipt {
+ records: u64,
+ stats: Arc<Stats>,
+ settled: bool,
+}
+
+impl Receipt {
+ fn new(records: usize, stats: Arc<Stats>) -> Self {
+ Self {
+ records: u64::try_from(records).unwrap_or(u64::MAX),
+ stats,
+ settled: false,
+ }
+ }
+
+ fn settle_accepted(mut self) {
+ self.settled = true;
+ self.stats.record_accepted(self.records);
+ tracing::debug!(records = self.records, "gateway telemetry accepted");
+ }
+
+ fn settle_failed(mut self, reason: &'static str) {
+ self.settled = true;
+ if self.stats.record_failed(self.records, reason) {
+ let context = tracing::Span::current().context();
+ let span = context.span();
+ let context = span.span_context();
+ tracing::warn!(
+ trace_id = %context.trace_id(),
+ span_id = %context.span_id(),
+ reason,
+ records = self.records,
+ failed = self.stats.failed.load(std::sync::atomic::Ordering::Relaxed),
+ "gateway telemetry upload failed"
+ );
+ }
+ }
+}
+
+impl Drop for Receipt {
+ fn drop(&mut self) {
+ if !self.settled {
+ self.stats.record_dropped(self.records, "shutdown");
+ }
+ }
+}
diff --git a/ai-gateway/src/test_support.rs b/ai-gateway/src/test_support.rs
index 1e38c7b376..b389a4b454 100644
--- a/ai-gateway/src/test_support.rs
+++ b/ai-gateway/src/test_support.rs
@@ -62,6 +62,32 @@ impl Drop for FakeServer {
}
}
+pub(crate) fn upload_text(body: &[u8]) -> String {
+ let mut text = String::new();
+ std::io::Read::read_to_string(&mut flate2::read::GzDecoder::new(body), &mut text).unwrap();
+ text
+}
+
+pub(crate) fn upload_json(body: &[u8]) -> serde_json::Value {
+ serde_json::from_str(&upload_text(body)).unwrap()
+}
+
+pub(crate) fn ingestion_token(organization: &str, project: &str) -> String {
+ use base64::{Engine, engine::general_purpose::URL_SAFE_NO_PAD};
+ let segment = |value: serde_json::Value| URL_SAFE_NO_PAD.encode(value.to_string());
+ format!(
+ "{}.{}.private-ingestion-token",
+ segment(json!({"alg": "ES256", "typ": "JWT"})),
+ segment(json!({
+ "version": 1,
+ "organization_id": organization,
+ "project_id": project,
+ "ingestion_mode": "usage",
+ "scope": "gateway-ingest",
+ })),
+ )
+}
+
pub(crate) fn resolution_response_for(
api_format: ApiFormat,
provider_secret: &str,
@@ -91,7 +117,7 @@ pub(crate) fn resolution_response_for(
"provider_connection_id": "provider-connection-1"
},
"ingestion_mode": "usage",
- "ingestion": {"access_token": "private-ingestion-token", "token_type": "Bearer", "expires_at": expires_at}
+ "ingestion": {"access_token": ingestion_token("org-1", "project-1"), "token_type": "Bearer", "expires_at": expires_at}
});
Response::builder()
.header("content-type", "application/json")
diff --git a/web/src/__e2e__/ai-gateway.gatewaye2e.ts b/web/src/__e2e__/ai-gateway.gatewaye2e.ts
index 037a2dcfe8..dad59829fc 100644
--- a/web/src/__e2e__/ai-gateway.gatewaye2e.ts
+++ b/web/src/__e2e__/ai-gateway.gatewaye2e.ts
@@ -27,6 +27,7 @@ import {
import {
getObservationByIdFromEventsTable,
getTraceByIdFromEventsTable,
+ queryClickhouse,
redis,
} from "@langfuse/shared/src/server";
import { getDisplaySecretKey } from "@langfuse/shared/src/server/auth/apiKeys";
@@ -584,12 +585,20 @@ async function ingestTrace(
const traceId = traceIdBytes.toString("hex");
const observationId = observationIdBytes.toString("hex");
const startTimeUnixNano = BigInt(Date.now()) * 1_000_000n;
+ // The payload claims another project everywhere a span can; ingestion must still
+ // write only into the project named by the gateway token.
+ const spoofedProjectId = `spoofed-${randomBytes(8).toString("hex")}`;
+ const spoofedProject = {
+ key: "langfuse.project.id",
+ value: { stringValue: spoofedProjectId },
+ };
const ExportTraceServiceRequest =
$root.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest;
const requestBody = ExportTraceServiceRequest.encode(
ExportTraceServiceRequest.fromObject({
resourceSpans: [
{
+ resource: { attributes: [spoofedProject] },
scopeSpans: [
{
scope: { name: "langfuse-ai-gateway-e2e", version: "1.0.0" },
@@ -610,6 +619,15 @@ async function ingestTrace(
key: "gen_ai.request.model",
value: { stringValue: "ai-gateway-e2e" },
},
+ spoofedProject,
+ {
+ key: "langfuse.observation.metadata",
+ value: {
+ stringValue: JSON.stringify({
+ "langfuse.gateway.project.id": spoofedProjectId,
+ }),
+ },
+ },
],
status: { code: 1 },
},
@@ -649,6 +667,14 @@ async function ingestTrace(
]);
expect(trace?.id).toBe(traceId);
expect(observation?.id).toBe(observationId);
+ const projects = await queryClickhouse<{ project_id: string }>({
+ query: `SELECT DISTINCT project_id FROM events_core
+ WHERE span_id = {observationId: String} AND trace_id = {traceId: String}`,
+ params: { observationId, traceId },
+ });
+ expect(projects.map(({ project_id }) => project_id)).toEqual([
+ defaultProjectId,
+ ]);
},
40_000,
1_000,
diff --git a/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts b/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts
index ec9366f07f..50d5b68cfe 100644
--- a/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts
+++ b/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts
@@ -571,6 +571,13 @@ describe("AI gateway control plane", () => {
project_id: project.id,
});
expect(rawClaims).not.toHaveProperty("api_key_id");
+ // The gateway batches uploads by attribution but ingestion writes into the
+ // token's project, so both must name the same tenant.
+ expect(result.attribution).toMatchObject({
+ organization_id: ingestionClaims.organization_id,
+ project_id: ingestionClaims.project_id,
+ });
+ expect(result.ingestion_mode).toBe(rawClaims.ingestion_mode);
expect(result.ingestion_mode).toBe("full");
expect(result.ingestion?.expires_at).toBe(ingestionClaims.exp);
});