🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
feat(ai-gateway): admit requests by memory budget only (#19020)
Tobias Wochinger24876f05405d
diff --git a/.env.dev.example b/.env.dev.exampleindex 3f8b0c68de..5980af1414 100644--- a/.env.dev.example+++ b/.env.dev.example@@ -212,12 +212,11 @@ ENCRYPTION_KEY=0000000000000000000000000000000000000000000000000000000000000000# Rust gateway: set the Web base URL to enable inference; include any Web base path.# The service key below must match Web. Without this URL, readiness/inference return 503.# LANGFUSE_AI_GATEWAY_WEB_URL=http://localhost:3000-# Independent per-instance concurrency limits; provisional defaults, tune with load tests.-# LANGFUSE_AI_GATEWAY_MAX_ACTIVE_REQUESTS=128-# LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS=128-# Largest request body, and the body bytes all in-flight requests may buffer together.+# Largest request body. Each in-flight request reserves a fixed overhead plus its+# body from a per-instance memory budget; an exhausted budget returns 503.# LANGFUSE_AI_GATEWAY_MAX_REQUEST_BYTES=16777216-# LANGFUSE_AI_GATEWAY_SHARED_REQUEST_BUFFER_BYTES=1073741824+# LANGFUSE_AI_GATEWAY_MEMORY_BUDGET_BYTES=1073741824+# LANGFUSE_AI_GATEWAY_REQUEST_MEMORY_OVERHEAD_BYTES=2097152# Execution ceiling and the longest wait without stream progress.# LANGFUSE_AI_GATEWAY_MAX_EXECUTION_SECONDS=600# LANGFUSE_AI_GATEWAY_STALL_TIMEOUT_SECONDS=120diff --git a/.env.prod.example b/.env.prod.exampleindex 945e2a82fb..09b2f7fc43 100644--- a/.env.prod.example+++ b/.env.prod.example@@ -591,12 +591,11 @@ LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG=true# LANGFUSE_AI_GATEWAY_WEB_URL=https://langfuse.example.com/app# Same shared service key configured in Web. Provider keys are resolved through Web.# LANGFUSE_AI_GATEWAY_SERVICE_KEY=-# Independent per-instance concurrency limits; provisional defaults, tune with load tests.-# LANGFUSE_AI_GATEWAY_MAX_ACTIVE_REQUESTS=128-# LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS=128-# Largest request body, and the body bytes all in-flight requests may buffer together.+# Largest request body. Each in-flight request reserves a fixed overhead plus its+# body from a per-instance memory budget; an exhausted budget returns 503.# LANGFUSE_AI_GATEWAY_MAX_REQUEST_BYTES=16777216-# LANGFUSE_AI_GATEWAY_SHARED_REQUEST_BUFFER_BYTES=1073741824+# LANGFUSE_AI_GATEWAY_MEMORY_BUDGET_BYTES=1073741824+# LANGFUSE_AI_GATEWAY_REQUEST_MEMORY_OVERHEAD_BYTES=2097152# Execution ceiling and the longest wait without stream progress.# LANGFUSE_AI_GATEWAY_MAX_EXECUTION_SECONDS=600# LANGFUSE_AI_GATEWAY_STALL_TIMEOUT_SECONDS=120diff --git a/ai-gateway/README.md b/ai-gateway/README.mdindex 260c952561..87162344a0 100644--- a/ai-gateway/README.md+++ b/ai-gateway/README.md@@ -78,11 +78,10 @@ do not load dotenv files:| `LANGFUSE_LOG_FORMAT` | `text` | `text`, `json` || `LANGFUSE_LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error`, `fatal` || `LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS` | `10` | Integer from 1 to 300 |-| `LANGFUSE_AI_GATEWAY_MAX_ACTIVE_REQUESTS` | `128` | Positive integer up to Tokio's semaphore capacity; buffered and active requests per instance |-| `LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS` | `128` | Positive integer up to Tokio's semaphore capacity; separate model-catalog and Web-resolution budgets per instance || `LANGFUSE_AI_GATEWAY_TELEMETRY_BUFFER_BYTES` | `67108864` (64 MiB) | Integer from 4194304 (4 MiB) to 4294967296 (4 GiB); span and credential bytes an instance holds for upload before dropping new records || `LANGFUSE_AI_GATEWAY_MAX_REQUEST_BYTES` | `16777216` (16 MiB) | Integer from 1048576 (1 MiB) to 4294967295; largest accepted request body |-| `LANGFUSE_AI_GATEWAY_SHARED_REQUEST_BUFFER_BYTES` | `1073741824` (1 GiB) | Integer from the request limit up to Tokio's semaphore capacity; request body bytes all in-flight requests of an instance may buffer together |+| `LANGFUSE_AI_GATEWAY_MEMORY_BUDGET_BYTES` | `1073741824` (1 GiB) | Integer from the request limit plus the overhead up to Tokio's semaphore capacity; bytes all in-flight requests and model catalogs of an instance may reserve together |+| `LANGFUSE_AI_GATEWAY_REQUEST_MEMORY_OVERHEAD_BYTES` | `2097152` (2 MiB) | Integer from 0 to 4294967295; fixed bytes each request reserves on top of its body for response capture and buffers || `LANGFUSE_AI_GATEWAY_MAX_EXECUTION_SECONDS` | `600` | Integer of at least 60; ceiling from execution admission until the response completes. Web ingestion tokens last 15 minutes, so traces of longer executions may be dropped || `LANGFUSE_AI_GATEWAY_STALL_TIMEOUT_SECONDS` | `120` | Integer from 1 to the execution ceiling; longest wait without progress |@@ -177,8 +176,7 @@ does not set CORS headers, so no `Access-Control-Expose-Headers` is involved.Once Web resolution succeeds, the request also carries its tenant, and once it issent upstream, the provider connection it was sent to. Requests rejected before-resolution, including invalid or oversized bodies, carry none of them. A resolved-request rejected by execution admission carries only the tenant. Provider failures+resolution, including invalid or oversized bodies, carry none of them. Provider failuresbefore response headers (502/504) still name the connection that was tried.| Response header | Server span field | JSON log key | Value |@@ -204,7 +202,7 @@ has no unattributed wall time:| Span | Covers || --- | --- |-| `resolution` | Resolution admission, the `resolver` Web call, and execution admission; `gateway.outcome` is `admitted`, `resolution_failed`, or `busy` |+| `resolution` | The `resolver` Web call; `gateway.outcome` is `admitted` or `resolution_failed` || `request.body` | Buffering the caller's request body, mostly upload time; `http.request.body.size` || `request.capture` | Parsing the request for inference telemetry; CPU-bound and proportional to `http.request.body.size` || `provider.headers` | The provider HTTP send through response headers, including connection setup |@@ -244,11 +242,15 @@ Gateway-specific metrics:`http.server.response.body.size` (`gateway.provider`), in bytes: the bufferedrequest body, and the bytes relayed from the provider, including truncated orcancelled streams.-- `gateway.active`, `gateway.admission.capacity`, and `gateway.active.peak`- (`phase`): in-flight, configured, and most-held-since-last-export slots for- model_catalog/resolution/execution.+- `gateway.active` and `gateway.active.peak` (`phase`): in-flight and+ most-in-flight-since-last-export model_catalog/resolution/execution work.+- `gateway.memory.budget`, `gateway.memory.reserved`, and+ `gateway.memory.reserved.peak`, in bytes: the configured memory budget, the bytes+ in-flight requests hold, and the most held since the last export. Peak utilisation+ is `gateway.memory.reserved.peak / gateway.memory.budget`.- `gateway.phase.duration` (execution/provider.first_byte), `gateway.executions`,- `gateway.admission.rejected`, and `gateway.telemetry.records`.+ `gateway.admission.rejected` (`phase`: `request`, `request_body` or+ `model_catalog`), and `gateway.telemetry.records`.Durations use seconds; first-byte timing measures provider bytes arriving at thegateway, not first-token delivery to the caller. Metric attributes contain@@ -394,22 +396,28 @@ error type follows the status (`authentication_error`, `permission_error`,`not_found_error`, `request_too_large`, `invalid_request_error`, `overloaded_error`,`api_error`).-After validating the credential header, the gateway buffers and validates the request-under the active-request admission budget and a shared request-byte budget, then resolves the credential and-model-aware route through Web under a separate budget. A declared `Content-Length` is-checked against the request limit before reading. The byte budget covers the buffer's-allocation and grows as bytes arrive, so a client that declares a large body and stalls-holds only what it sent. The body slot and its bytes stay reserved with the execution-until the response ends. Model catalogs use another budget so slow catalog discovery-cannot block inference resolution.-Execution capacity remains reserved through response completion. An exhausted budget-returns 503 immediately without a waiting queue. Request-body and execution budgets-use the active-request limit, and buffered bodies also share-`LANGFUSE_AI_GATEWAY_SHARED_REQUEST_BUFFER_BYTES`; model-catalog and Web-call budgets use the resolution-limit. Their defaults of 128 are provisional-guardrails, not measured capacity: tune them using load tests for the instance-resources, request sizes and stream durations. These limits bound work; they do not-guarantee fairness between clients or tenants.+Admission is bounded by memory only. After validating the credential header, the+gateway reserves a fixed per-request overhead from the instance's memory budget, then+buffers and validates the request and resolves the credential and model-aware route+through Web. A declared `Content-Length` is checked against the request limit before+reading. The reservation grows with the body buffer's allocation as bytes arrive, so a+client that declares a large body and stalls holds only the overhead and what it sent.+The whole reservation stays held until the response ends. Model catalog requests+reserve the overhead, grow it with Web's catalog buffer as it arrives, then to three+times the catalog for parsing and the provider-shaped response, and hold it until the+client has read that response; the response is relayed under the same stall timeout and+execution deadline as provider responses. An exhausted budget returns 503 immediately without a+waiting queue. There is no request-count limit; file descriptors are bounded by the+container.++The overhead stands in for what a request holds besides its body: retained capture+output (up to 1 MiB), one inspected response event (up to 1 MiB), the relay chunk and+resolution context. The 2 MiB default sits above the typical request: on staging,+responses average 8 KB (Anthropic) to 210 KB (OpenAI), with a largest of 3 MB. With+the 1 GiB budget it admits about 400 typical concurrent requests. The parsed request+input kept for capture is covered by the body's reservation, which outlives the+buffered bytes. These limits bound work; they do not guarantee fairness between+clients or tenants.Request bodies are limited to 16 MiB by default in both namespaces. The body read deadlinescales with that limit: at least 30 seconds, and enough for the maximal body at 10 MiB per@@ -421,7 +429,7 @@ upstream chunk, without the downstream accepting a chunk, or, for streamed callsheaders. A non-streamed generation waits for headers until the ceiling, becauseproviders send nothing until the response is generated. Response size is not capped: a single task pumps chunksthrough a one-slot channel, with chunks at most 64 KiB. It stops reading when that-channel fills. Completion, disconnect, stall and ceiling release admission and context;+channel fills. Completion, disconnect, stall and ceiling release the memory reservation and context;both timeouts run even when the downstream stops polling. A failure after headersterminates the response body without adding an error event or retrying.@@ -774,7 +782,7 @@ batched upload. Capture and upload run independently of log level; debug emissioalone is gated.Capture is limited to 5 MiB request inspection, 1 MiB JSON/SSE event inspection,-1 MiB retained output and 256 output items per execution. The active-request limit+1 MiB retained output and 256 output items per execution. The memory budgetbounds the number of captures. Trusted key metadata is bounded by the resolver's256 KiB response limit. In full mode, an input that is not recorded (over 5 MiB,content-encoded or not a JSON object) is explained in generation metadata:@@ -939,7 +947,7 @@ cargo test --locked- `main.rs`: configuration, logging, signal registration and process exit.- `http.rs`: credential extraction, bounded body reads and gateway error envelopes.- `inference.rs`: `InferenceService` coordinates resolution, admission and forwarding.-- `providers/mod.rs`: `ProviderTransport` with shared admission, the `Route` enum of+- `providers/mod.rs`: `ProviderTransport`, the `Route` enum ofofficial operations (method, upstream path and capture), model rewriting and credentialplacement.- `providers/model_list`: provider-native OpenAI and Anthropic model catalog responses.diff --git a/ai-gateway/src/capture/anthropic_messages/tests.rs b/ai-gateway/src/capture/anthropic_messages/tests.rsindex 0e26b4637e..cf02f50430 100644--- a/ai-gateway/src/capture/anthropic_messages/tests.rs+++ b/ai-gateway/src/capture/anthropic_messages/tests.rs@@ -554,7 +554,7 @@ async fn streamed_messages_upload_one_generation_with_native_usage() {);let forwarded = provider.forward_route(- provider.try_admit().unwrap(),+ provider.admit(),context,&headers,crate::providers::RequestBody {diff --git a/ai-gateway/src/capture/openai_responses/tests.rs b/ai-gateway/src/capture/openai_responses/tests.rsindex 55f49121ed..7fbaf07cdd 100644--- a/ai-gateway/src/capture/openai_responses/tests.rs+++ b/ai-gateway/src/capture/openai_responses/tests.rs@@ -513,7 +513,6 @@ async fn upstream_read_timeout_is_logged_as_timeout() {let provider = ProviderTransport::for_test(format!("{}/v1", upstream.url),ProviderLimits {- active: 1,execution: ExecutionLimits {stall_timeout: Duration::from_millis(50),..ExecutionLimits::default()@@ -529,7 +528,7 @@ async fn upstream_read_timeout_is_logged_as_timeout() {let _guard = tracing::subscriber::set_default(subscriber);let response = provider.forward(- provider.try_admit().unwrap(),+ provider.admit(),context,&HeaderMap::new(),Bytes::from_static(br#"{"model":"requested"}"#),@@ -540,7 +539,6 @@ async fn upstream_read_timeout_is_logged_as_timeout() {let records = records(&writer);assert_eq!(records.len(), 1);assert_eq!(records[0]["outcome"], "timeout");- assert!(provider.try_admit().is_ok());}#[tokio::test]@@ -570,7 +568,6 @@ async fn native_http_relay_logs_once_on_eof_drop_and_unpolled_deadline() {let provider = ProviderTransport::for_test(format!("{}/v1", upstream.url),ProviderLimits {- active: 1,execution: ExecutionLimits {max_execution_time: Duration::from_secs(1),..ExecutionLimits::default()@@ -586,7 +583,7 @@ async fn native_http_relay_logs_once_on_eof_drop_and_unpolled_deadline() {let _guard = tracing::subscriber::set_default(subscriber);let response = provider.forward(- provider.try_admit().unwrap(),+ provider.admit(),context,&HeaderMap::new(),Bytes::from_static(br#"{"model":"requested","input":"request-content"}"#),@@ -601,7 +598,6 @@ async fn native_http_relay_logs_once_on_eof_drop_and_unpolled_deadline() {RelayOutcome::Cancelled => drop(response),RelayOutcome::Timeout => {tokio::time::sleep(Duration::from_millis(1100)).await;- assert!(provider.try_admit().is_ok());drop(response);}RelayOutcome::TransportError => unreachable!(),@@ -614,7 +610,6 @@ async fn native_http_relay_logs_once_on_eof_drop_and_unpolled_deadline() {assert_eq!(records[0]["output_complete"], true);assert!(records[0]["first_byte_ms"].is_number());}- assert!(provider.try_admit().is_ok());assert_eq!(upstream.calls(), 1);}}@@ -636,7 +631,6 @@ async fn provider_future_finalizes_on_timeout_and_cancellation_before_headers()let provider = ProviderTransport::for_test(format!("{}/v1", upstream.url),ProviderLimits {- active: 1,execution: ExecutionLimits {stall_timeout: Duration::from_millis(100),..ExecutionLimits::default()@@ -652,7 +646,7 @@ async fn provider_future_finalizes_on_timeout_and_cancellation_before_headers()let _guard = tracing::subscriber::set_default(subscriber);let headers = HeaderMap::new();let call = provider.forward(- provider.try_admit().unwrap(),+ provider.admit(),context,&headers,Bytes::from_static(br#"{"model":"requested"}"#),@@ -675,7 +669,6 @@ async fn provider_future_finalizes_on_timeout_and_cancellation_before_headers()if cancel { "cancelled" } else { "timeout" });assert!(records[0]["http_status"].is_null());- assert!(provider.try_admit().is_ok());}}@@ -749,7 +742,7 @@ async fn client_compression_preferences_do_not_disable_capture() {);let forwarded = provider.forward(- provider.try_admit().unwrap(),+ provider.admit(),context,&headers,Bytes::from(@@ -831,7 +824,7 @@ async fn codex_body_metadata_reaches_the_generation_without_agent_headers() {});let forwarded = provider.forward(- provider.try_admit().unwrap(),+ provider.admit(),context,&HeaderMap::new(),Bytes::from(body.to_string()),diff --git a/ai-gateway/src/config.rs b/ai-gateway/src/config.rsindex c801bf4af9..f5fabd7e3b 100644--- a/ai-gateway/src/config.rs+++ b/ai-gateway/src/config.rs@@ -9,33 +9,37 @@ pub struct GatewayConfig {pub shutdown_timeout: Duration,pub log_level: LevelFilter,pub log_format: LogFormat,- pub max_active_requests: usize,- pub max_concurrent_resolutions: usize,pub telemetry_buffer_bytes: usize,pub request_limits: RequestLimits,pub execution_limits: ExecutionLimits,}const MIB: usize = 1024 * 1024;-const DEFAULT_CONCURRENCY: usize = 128;const SEMAPHORE_CAPACITY: usize = tokio::sync::Semaphore::MAX_PERMITS;pub const DEFAULT_MAX_REQUEST_BYTES: usize = 16 * MIB;-const DEFAULT_SHARED_REQUEST_BUFFER_BYTES: usize = 1024 * MIB;+const DEFAULT_MEMORY_BUDGET_BYTES: usize = 1024 * MIB;+/// Covers response capture (up to 1 MiB retained output plus one inspected event of up+/// to 1 MiB) and connection buffers. Staging responses average 8 KB (Anthropic) to+/// 210 KB (`OpenAI`), so this is above the typical request.+const DEFAULT_REQUEST_OVERHEAD_BYTES: usize = 2 * MIB;const DEFAULT_MAX_EXECUTION_SECONDS: u32 = 10 * 60;const DEFAULT_STALL_TIMEOUT_SECONDS: u32 = 120;-/// Buffered request body bounds per instance.+/// Memory bounds per instance. Admission is limited only by the memory budget.#[derive(Clone, Copy, Debug, PartialEq, Eq)]pub struct RequestLimits {pub max_request_bytes: usize,- /// Request body bytes all in-flight requests may buffer together.- pub shared_buffer_bytes: usize,+ /// Bytes all in-flight requests may reserve together.+ pub memory_budget_bytes: usize,+ /// Fixed bytes every in-flight request reserves on top of its body.+ pub request_overhead_bytes: usize,}impl Default for RequestLimits {fn default() -> Self {Self {max_request_bytes: DEFAULT_MAX_REQUEST_BYTES,- shared_buffer_bytes: DEFAULT_SHARED_REQUEST_BUFFER_BYTES,+ memory_budget_bytes: DEFAULT_MEMORY_BUDGET_BYTES,+ request_overhead_bytes: DEFAULT_REQUEST_OVERHEAD_BYTES,}}}@@ -82,15 +86,14 @@ impl GatewayConfig {read_env("LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS")?.as_deref(),read_env("LANGFUSE_LOG_LEVEL")?.as_deref(),read_env("LANGFUSE_LOG_FORMAT")?.as_deref(),- read_env("LANGFUSE_AI_GATEWAY_MAX_ACTIVE_REQUESTS")?.as_deref(),- read_env("LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS")?.as_deref(),)?;config.telemetry_buffer_bytes = telemetry_buffer_bytes(read_env("LANGFUSE_AI_GATEWAY_TELEMETRY_BUFFER_BYTES")?.as_deref(),)?;config.request_limits = request_limits(read_env("LANGFUSE_AI_GATEWAY_MAX_REQUEST_BYTES")?.as_deref(),- read_env("LANGFUSE_AI_GATEWAY_SHARED_REQUEST_BUFFER_BYTES")?.as_deref(),+ read_env("LANGFUSE_AI_GATEWAY_MEMORY_BUDGET_BYTES")?.as_deref(),+ read_env("LANGFUSE_AI_GATEWAY_REQUEST_MEMORY_OVERHEAD_BYTES")?.as_deref(),)?;config.execution_limits = execution_limits(read_env("LANGFUSE_AI_GATEWAY_MAX_EXECUTION_SECONDS")?.as_deref(),@@ -118,16 +121,13 @@ impl GatewayConfig {/// # Errors/// Returns an error for an invalid IP address and port, a non-boolean port/// auto-increment setting, a shutdown timeout outside 1–300 integer seconds,- /// an unsupported log level or format, or concurrency- /// limits outside 1 through [`tokio::sync::Semaphore::MAX_PERMITS`].+ /// or an unsupported log level or format.pub fn from_values(listen_address: Option<&str>,auto_increment_listen_port: Option<&str>,shutdown_timeout: Option<&str>,log_level: Option<&str>,log_format: Option<&str>,- max_active_requests: Option<&str>,- max_concurrent_resolutions: Option<&str>,) -> Result<Self, GatewayConfigError> {let listen_address = listen_address.unwrap_or("0.0.0.0:8080")@@ -177,18 +177,6 @@ impl GatewayConfig {shutdown_timeout: Duration::from_secs(seconds),log_level,log_format,- max_active_requests: integer_setting(- max_active_requests,- DEFAULT_CONCURRENCY,- 1..=SEMAPHORE_CAPACITY,- "LANGFUSE_AI_GATEWAY_MAX_ACTIVE_REQUESTS must be a positive integer within the semaphore capacity",- )?,- max_concurrent_resolutions: integer_setting(- max_concurrent_resolutions,- DEFAULT_CONCURRENCY,- 1..=SEMAPHORE_CAPACITY,- "LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS must be a positive integer within the semaphore capacity",- )?,telemetry_buffer_bytes: crate::telemetry::DEFAULT_RETAINED_BYTES,request_limits: RequestLimits::default(),execution_limits: ExecutionLimits::default(),@@ -198,10 +186,11 @@ impl GatewayConfig {fn request_limits(max_request_bytes: Option<&str>,- shared_buffer_bytes: Option<&str>,+ memory_budget_bytes: Option<&str>,+ request_overhead_bytes: Option<&str>,) -> Result<RequestLimits, GatewayConfigError> {let defaults = RequestLimits::default();- // One request reserves its body in a single semaphore acquisition, which takes a u32.+ // The overhead and each body growth are single semaphore acquisitions, which take a u32.let max_reservation = usize::try_from(u32::MAX).unwrap_or(usize::MAX).min(SEMAPHORE_CAPACITY);@@ -211,15 +200,24 @@ fn request_limits(MIB..=max_reservation,"LANGFUSE_AI_GATEWAY_MAX_REQUEST_BYTES must be an integer of at least 1048576 (1 MiB) and within a single semaphore reservation",)?;- let shared_buffer_bytes = integer_setting(- shared_buffer_bytes,- defaults.shared_buffer_bytes,- max_request_bytes..=SEMAPHORE_CAPACITY,- "LANGFUSE_AI_GATEWAY_SHARED_REQUEST_BUFFER_BYTES must be an integer of at least LANGFUSE_AI_GATEWAY_MAX_REQUEST_BYTES within the semaphore capacity",+ let request_overhead_bytes = integer_setting(+ request_overhead_bytes,+ defaults.request_overhead_bytes,+ 0..=max_reservation,+ "LANGFUSE_AI_GATEWAY_REQUEST_MEMORY_OVERHEAD_BYTES must be a non-negative integer within a single semaphore reservation",+ )?;+ // A maximal request must fit, or it could never be admitted.+ let largest_request = max_request_bytes.saturating_add(request_overhead_bytes);+ let memory_budget_bytes = integer_setting(+ memory_budget_bytes,+ defaults.memory_budget_bytes,+ largest_request..=SEMAPHORE_CAPACITY,+ "LANGFUSE_AI_GATEWAY_MEMORY_BUDGET_BYTES must be an integer of at least LANGFUSE_AI_GATEWAY_MAX_REQUEST_BYTES plus LANGFUSE_AI_GATEWAY_REQUEST_MEMORY_OVERHEAD_BYTES within the semaphore capacity",)?;Ok(RequestLimits {max_request_bytes,- shared_buffer_bytes,+ memory_budget_bytes,+ request_overhead_bytes,})}@@ -292,14 +290,12 @@ mod tests {#[test]fn configuration_parses_defaults_and_overrides() {- let default = GatewayConfig::from_values(None, None, None, None, None, None, None).unwrap();+ let default = GatewayConfig::from_values(None, None, None, None, None).unwrap();assert_eq!(default.listen_address, "0.0.0.0:8080".parse().unwrap());assert!(!default.auto_increment_listen_port);assert_eq!(default.shutdown_timeout, Duration::from_secs(10));assert_eq!(default.log_level, LevelFilter::INFO);assert_eq!(default.log_format, LogFormat::Text);- assert_eq!(default.max_active_requests, 128);- assert_eq!(default.max_concurrent_resolutions, 128);assert_eq!(default.telemetry_buffer_bytes, 64 * 1024 * 1024);let custom = GatewayConfig::from_values(Some("[::1]:9000"),@@ -307,8 +303,6 @@ mod tests {Some("30"),Some("debug"),Some("json"),- Some("256"),- Some("32"),).unwrap();assert_eq!(custom.listen_address, "[::1]:9000".parse().unwrap());@@ -316,22 +310,51 @@ mod tests {assert_eq!(custom.shutdown_timeout, Duration::from_secs(30));assert_eq!(custom.log_level, LevelFilter::DEBUG);assert_eq!(custom.log_format, LogFormat::Json);- assert_eq!(custom.max_active_requests, 256);- assert_eq!(custom.max_concurrent_resolutions, 32);+ }++ #[test]+ fn memory_budget_must_fit_a_maximal_request_with_its_overhead() {+ assert_eq!(+ request_limits(None, None, None).unwrap(),+ RequestLimits {+ max_request_bytes: 16 * MIB,+ memory_budget_bytes: 1024 * MIB,+ request_overhead_bytes: 2 * MIB,+ }+ );+ let fitting = request_limits(Some("1048576"), Some("1048576"), Some("0")).unwrap();+ assert_eq!(fitting.memory_budget_bytes, MIB);+ assert_eq!(fitting.request_overhead_bytes, 0);+ let budget = "LANGFUSE_AI_GATEWAY_MEMORY_BUDGET_BYTES";+ let overhead = "LANGFUSE_AI_GATEWAY_REQUEST_MEMORY_OVERHEAD_BYTES";+ for (max_request, budget_value, overhead_value, name) in [+ (Some("1048576"), Some("1048576"), Some("1"), budget),+ // One byte short of the default 16 MiB request plus 2 MiB overhead.+ (None, Some("18874367"), None, budget),+ (None, None, Some("-1"), overhead),+ (None, None, Some("secret-that-must-not-appear"), overhead),+ (None, Some("secret-that-must-not-appear"), None, budget),+ ] {+ let error = request_limits(max_request, budget_value, overhead_value)+ .err()+ .unwrap();+ assert!(error.to_string().starts_with(name), "{error}");+ assert!(!error.to_string().contains("secret-that-must-not-appear"));+ }}#[test]fn validates_shared_log_format() {for (value, expected) in [("text", LogFormat::Text), ("json", LogFormat::Json)] {assert_eq!(- GatewayConfig::from_values(None, None, None, None, Some(value), None, None)+ GatewayConfig::from_values(None, None, None, None, Some(value)).unwrap().log_format,expected);}for value in ["", "TEXT", "pretty", "0", "secret-that-must-not-appear"] {- let error = GatewayConfig::from_values(None, None, None, None, Some(value), None, None)+ let error = GatewayConfig::from_values(None, None, None, None, Some(value)).err().unwrap();assert_eq!(@@ -350,9 +373,7 @@ mod tests {("error", LevelFilter::ERROR),("fatal", LevelFilter::ERROR),] {- let config =- GatewayConfig::from_values(None, None, None, Some(value), None, None, None)- .unwrap();+ let config = GatewayConfig::from_values(None, None, None, Some(value), None).unwrap();assert_eq!(config.log_level, expected, "{value}");}}@@ -374,14 +395,14 @@ mod tests {(None, Some("-1"), None),(Some("localhost:8080"), None, None),] {- let error = GatewayConfig::from_values(address, None, timeout, level, None, None, None)+ let error = GatewayConfig::from_values(address, None, timeout, level, None).err().unwrap();assert!(!error.to_string().contains(sensitive_input));assert!(!format!("{error:?}").contains(sensitive_input));}for value in ["", "TRUE", "1", sensitive_input] {- let error = GatewayConfig::from_values(None, Some(value), None, None, None, None, None)+ let error = GatewayConfig::from_values(None, Some(value), None, None, None).err().unwrap();assert!(!error.to_string().contains(sensitive_input));@@ -389,46 +410,6 @@ mod tests {}}- #[test]- fn concurrency_limits_reject_invalid_values_and_accept_boundaries() {- let maximum = tokio::sync::Semaphore::MAX_PERMITS.to_string();- let overflow = (tokio::sync::Semaphore::MAX_PERMITS + 1).to_string();- for value in ["1", maximum.as_str()] {- let config =- GatewayConfig::from_values(None, None, None, None, None, Some(value), Some(value))- .unwrap();- assert_eq!(config.max_active_requests, value.parse::<usize>().unwrap());- assert_eq!(- config.max_concurrent_resolutions,- value.parse::<usize>().unwrap()- );- }- for value in [- "0",- "-1",- "1.5",- "",- "secret-that-must-not-appear",- overflow.as_str(),- ] {- for (active, resolutions, name) in [- (Some(value), None, "LANGFUSE_AI_GATEWAY_MAX_ACTIVE_REQUESTS"),- (- None,- Some(value),- "LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS",- ),- ] {- let error =- GatewayConfig::from_values(None, None, None, None, None, active, resolutions)- .err()- .unwrap();- assert!(error.to_string().contains(name));- assert!(!error.to_string().contains("secret-that-must-not-appear"));- }- }- }-#[test]fn telemetry_buffer_is_read_in_bytes_within_bounds() {assert_eq!(telemetry_buffer_bytes(None).unwrap(), 64 * 1024 * 1024);diff --git a/ai-gateway/src/http.rs b/ai-gateway/src/http.rsindex 90f5d5e738..58cb9e58b4 100644--- a/ai-gateway/src/http.rs+++ b/ai-gateway/src/http.rs@@ -16,10 +16,7 @@ use tracing::Instrument;use crate::{correlation::{RequestCorrelation, assign_request_id},- inference::{- InferenceService, RequestBodyPermit, RequestPreparationError, provider_origin,- resolution_origin,- },+ inference::{InferenceService, MemoryReservation, provider_origin, resolution_origin},observability::{ErrorOrigin, Wait},providers::{ProviderError, RequestBody, Route},resolution::{ApiFormat, ResolutionError},@@ -207,22 +204,22 @@ async fn relay(return Err(InferenceHttpError::UnsupportedEncoding);}// Held with the execution and released once the response ends.- let body_permit = inference- .try_reserve_request_body()- .map_err(InferenceHttpError::from)?;+ let reservation = inference+ .try_reserve_memory("request")+ .map_err(InferenceHttpError::Resolution)?;let (parts, body) = request.into_parts();let started = Instant::now();let read = read_request_body(body,&parts.headers,inference.max_request_bytes(),- body_permit,+ reservation,).await;correlation.observation().record_wait(Wait::RequestBody, started.elapsed());- let (bytes, body_permit) = read?;+ let (bytes, reservation) = read?;correlation.observation().record_request_bytes(bytes.len());let request_body = request_body_and_model(bytes)?;let (permit, context) = inference@@ -233,7 +230,7 @@ async fn relay(correlation,).await- .map_err(InferenceHttpError::from)?;+ .map_err(InferenceHttpError::Resolution)?;inference.forward(permit,@@ -243,7 +240,7 @@ async fn relay(bytes: request_body.bytes,model_range: request_body.model_range,streaming: request_body.streaming,- reservation: Some(Box::new(body_permit)),+ reservation: Some(Box::new(reservation)),},route,correlation,@@ -295,7 +292,7 @@ fn request_body_and_model(bytes: Bytes) -> Result<InferenceRequestBody, Inferenc})}-/// Buffer the client body, reserving its allocation from the shared budget as bytes+/// Buffer the client body, reserving its allocation from the memory budget as bytes/// arrive. A declared length is checked before reading and caps buffer growth, but is/// not reserved up front, so a client that declares a large body and stalls holds only/// what it actually sent. The router's body deadline bounds the read. The wait is mostly@@ -305,8 +302,8 @@ async fn read_request_body(body: Body,headers: &HeaderMap,max_bytes: usize,- mut permit: RequestBodyPermit,-) -> Result<(Bytes, RequestBodyPermit), InferenceHttpError> {+ mut reservation: MemoryReservation,+) -> Result<(Bytes, MemoryReservation), InferenceHttpError> {let server = tracing::Span::current();let span = tracing::info_span!("request.body",@@ -337,8 +334,8 @@ async fn read_request_body(.capacity().saturating_mul(2).clamp(total, capacity_limit.max(total));- permit- .try_reserve_bytes(capacity)+ reservation+ .try_reserve_body(capacity, "request_body").map_err(InferenceHttpError::Resolution)?;buffer.reserve_exact(capacity - buffer.len());}@@ -351,7 +348,7 @@ async fn read_request_body(.instrument(span).await?;server.record("http.request.body.size", byte_count(bytes.len()));- Ok((bytes, permit))+ Ok((bytes, reservation))}/// The router's body deadline surfaces as a body error.@@ -428,15 +425,6 @@ enum InferenceHttpError {Provider(ProviderError),}-impl From<RequestPreparationError> for InferenceHttpError {- fn from(error: RequestPreparationError) -> Self {- match error {- RequestPreparationError::Resolution(error) => Self::Resolution(error),- RequestPreparationError::Provider(error) => Self::Provider(error),- }- }-}-#[derive(Serialize)]struct OpenAiErrorResponse<'a> {error: OpenAiErrorDetail,@@ -496,7 +484,6 @@ impl InferenceHttpError {Self::Provider(error) => ("provider",match error {- ProviderError::Busy => "capacity",ProviderError::Timeout => "timeout",ProviderError::Transport => "transport",ProviderError::Configuration => "configuration",@@ -570,7 +557,7 @@ impl InferenceHttpError {),Self::Unavailable| Self::Resolution(R::Unavailable | R::Configuration | R::Busy)- | Self::Provider(ProviderError::Configuration | ProviderError::Busy) => (+ | Self::Provider(ProviderError::Configuration) => (StatusCode::SERVICE_UNAVAILABLE,"Gateway temporarily unavailable","server_error",diff --git a/ai-gateway/src/http/tests.rs b/ai-gateway/src/http/tests.rsindex 7c993826a8..9518b87eaa 100644--- a/ai-gateway/src/http/tests.rs+++ b/ai-gateway/src/http/tests.rs@@ -16,49 +16,13 @@ use std::{use tokio::{net::TcpListener, sync::oneshot, task::JoinHandle};use tower::ServiceExt;-#[tokio::test]-async fn control_plane_rejections_do_not_consume_provider_capacity() {- let web = FakeServer::start(|_| async {- Response::builder()- .status(StatusCode::UNAUTHORIZED)- .body(Body::empty())- .unwrap()- })- .await;- let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;- let provider_client = ProviderTransport::for_test(- provider.url.clone(),- ProviderLimits {- active: 1,- ..ProviderLimits::default()- },- );- let permit = provider_client.try_admit().unwrap();- let gateway = Gateway::start(Some(InferenceService::for_test(- web.control_plane(),- provider_client,- 1,- )))- .await;- // Exercise both an exhausted and an available execution pool.- for occupied in [Some(permit), None] {- let response = tokio::time::timeout(- Duration::from_secs(2),- gateway.app.clone().oneshot(- Request::post("/openai/v1/responses")- .header(header::AUTHORIZATION, "Bearer invalid-key")- .body(Body::from(r#"{"model":"Production key:test"}"#))- .unwrap(),- ),- )- .await- .unwrap()- .unwrap();- assert_eq!(response.status(), StatusCode::UNAUTHORIZED);- drop(occupied);+/// The budget fits one request: any held reservation leaves less than the overhead.+fn single_request_memory() -> RequestLimits {+ RequestLimits {+ max_request_bytes: MAX_REQUEST_BYTES,+ memory_budget_bytes: 2 * MAX_REQUEST_BYTES + 1,+ request_overhead_bytes: MAX_REQUEST_BYTES + 1,}- assert_eq!(web.calls(), 2);- assert_eq!(provider.calls(), 0);}struct Gateway {@@ -69,245 +33,172 @@ struct Gateway {}#[tokio::test]-async fn resolution_capacity_is_bounded_and_released_on_cancellation_and_failure() {+async fn held_memory_rejects_others_until_its_holder_ends() {let entered = Arc::new(tokio::sync::Notify::new());let started = entered.clone();let web = FakeServer::start(move |request| {let started = started.clone();async move {- match request.headers()[header::AUTHORIZATION].to_str().unwrap() {- "Bearer stalled" => {- started.notify_one();- std::future::pending().await- }- "Bearer invalid" => Response::builder()+ let authorization = request.headers()[header::AUTHORIZATION].to_str().unwrap();+ if request.uri().path() == "/api/internal/ai-gateway/v1/models"+ || authorization == "Bearer stalled"+ {+ started.notify_one();+ return std::future::pending().await;+ }+ if authorization == "Bearer invalid" {+ return Response::builder().status(StatusCode::UNAUTHORIZED).body(Body::empty())- .unwrap(),- _ => resolution_response("provider"),+ .unwrap();}+ resolution_response("provider")}}).await;let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;- let service = InferenceService::for_test(- web.control_plane(),- ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()),- 1,- );- let gateway = Gateway::start(Some(service)).await;- let request = |key| {+ let gateway = Gateway::start(Some(+ InferenceService::for_test(+ web.control_plane(),+ ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()),+ )+ .with_request_limits(single_request_memory()),+ ))+ .await;+ let inference = |key: &str, body: Body| {Request::post("/openai/v1/responses").header(header::AUTHORIZATION, key)- .body(Body::from(r#"{"model":"Production key:test"}"#))+ .body(body).unwrap()};- let mut stalled = Box::pin(gateway.app.clone().oneshot(request("Bearer stalled")));- tokio::time::timeout(Duration::from_secs(2), async {- tokio::select! {- () = entered.notified() => {},- _ = &mut stalled => panic!("resolution should be pending"),- }- })- .await- .unwrap();- let body_polled = Arc::new(AtomicBool::new(false));- let observed = body_polled.clone();- let body = Body::from_stream(stream::once(async move {- observed.store(true, Ordering::SeqCst);- Ok::<_, Infallible>(Bytes::from_static(br#"{"model":"Production key:test"}"#))- }));- let busy = tokio::time::timeout(- Duration::from_secs(2),- gateway.app.clone().oneshot(- Request::post("/openai/v1/responses")- .header(header::AUTHORIZATION, "Bearer valid")- .body(body)+ let model = || Body::from(r#"{"model":"Production key:test"}"#);+ let stalled_upload = || {+ let entered = entered.clone();+ Body::from_stream(stream::once(async move {+ entered.notify_one();+ std::future::pending::<Result<Bytes, Infallible>>().await+ }))+ };+ let holders = [+ ("resolution", inference("Bearer stalled", model())),+ ("upload", inference("Bearer upload", stalled_upload())),+ (+ "model catalog",+ Request::get("/openai/v1/models")+ .header(header::AUTHORIZATION, "Bearer models")+ .body(Body::empty()).unwrap(),),- )- .await- .unwrap()- .unwrap();- assert_eq!(busy.status(), StatusCode::SERVICE_UNAVAILABLE);- assert!(!body_polled.load(Ordering::SeqCst));- assert_eq!(web.calls(), 1);- assert_eq!(provider.calls(), 0);+ ];- drop(stalled);- for (key, status) in [- ("Bearer invalid", StatusCode::UNAUTHORIZED),- ("Bearer valid", StatusCode::OK),- ] {- let response = tokio::time::timeout(- Duration::from_secs(2),- gateway.app.clone().oneshot(request(key)),- )+ for (calls, (holder, request)) in (1..).zip(holders) {+ let mut held = Box::pin(gateway.app.clone().oneshot(request));+ tokio::time::timeout(Duration::from_secs(2), async {+ tokio::select! {+ () = entered.notified() => {},+ _ = &mut held => panic!("{holder} should remain pending"),+ }+ }).await- .unwrap().unwrap();- assert_eq!(response.status(), status);+ let body_polled = Arc::new(AtomicBool::new(false));+ let observed = body_polled.clone();+ let body = Body::from_stream(stream::once(async move {+ observed.store(true, Ordering::SeqCst);+ Ok::<_, Infallible>(Bytes::from_static(br#"{"model":"Production key:test"}"#))+ }));+ let busy = gateway+ .app+ .clone()+ .oneshot(inference("Bearer valid", body))+ .await+ .unwrap();+ assert_eq!(busy.status(), StatusCode::SERVICE_UNAVAILABLE, "{holder}");+ assert!(!body_polled.load(Ordering::SeqCst), "{holder}");++ drop(held);+ // A rejected request must release its memory too, or the next call gets 503.+ for (key, status) in [+ ("Bearer invalid", StatusCode::UNAUTHORIZED),+ ("Bearer valid", StatusCode::OK),+ ] {+ let response = gateway+ .app+ .clone()+ .oneshot(inference(key, model()))+ .await+ .unwrap();+ assert_eq!(response.status(), status, "{holder}");+ }+ assert_eq!(provider.calls(), calls, "{holder}");}- assert_eq!(web.calls(), 3);- assert_eq!(provider.calls(), 1);}#[tokio::test]-async fn slow_body_reads_do_not_consume_resolution_capacity() {- let web = FakeServer::start(|request| async move {- if request.uri().path() == "/api/internal/ai-gateway/v1/models" {- return Response::new(Body::from(- r#"{"data":[{"id":"Production key:gpt-4.1","provider":"OPENAI","canonicalSlug":"gpt-4.1","displayName":"GPT-4.1","createdAt":"1970-01-01T00:00:10.000Z"}]}"#,- ));+async fn model_catalog_memory_covers_the_catalog_until_it_is_read_or_stalls() {+ let entry = r#"{"id":"Production key:gpt-4.1","provider":"OPENAI","canonicalSlug":"gpt-4.1","displayName":"GPT-4.1","createdAt":"1970-01-01T00:00:10.000Z"}"#;+ let catalog = format!(r#"{{"data":[{}]}}"#, vec![entry; 2000].join(","));+ let catalog_bytes = catalog.len();+ let web = FakeServer::start(move |_| {+ let catalog = catalog.clone();+ async move {+ Response::builder()+ .header("content-type", "application/json")+ .body(Body::from(catalog))+ .unwrap()}- resolution_response("provider-secret")}).await;let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;- let gateway = Gateway::start(Some(InferenceService::for_test(- web.control_plane(),- ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()),- 1,- )))+ let overhead = 1024;+ let gateway = Gateway::start(Some(+ InferenceService::for_test(+ web.control_plane(),+ ProviderTransport::for_test(+ provider.url.clone(),+ ProviderLimits {+ execution: ExecutionLimits {+ stall_timeout: Duration::from_millis(500),+ ..ExecutionLimits::default()+ },+ },+ ),+ )+ .with_request_limits(RequestLimits {+ max_request_bytes: MAX_REQUEST_BYTES,+ memory_budget_bytes: 2 * overhead + 3 * catalog_bytes - 1,+ request_overhead_bytes: overhead,+ }),+ )).await;- let body_entered = Arc::new(tokio::sync::Notify::new());- let entered = body_entered.clone();- let held_body = Body::from_stream(stream::once(async move {- entered.notify_one();- std::future::pending::<Result<Bytes, Infallible>>().await- }));- let mut held = Box::pin(+ let catalog_request = || {gateway.app.clone().oneshot(- Request::post("/openai/v1/responses")- .header(header::AUTHORIZATION, "Bearer held")- .body(held_body)- .unwrap(),- ),- );- tokio::time::timeout(Duration::from_secs(2), async {- tokio::select! {- () = body_entered.notified() => {},- _ = &mut held => panic!("body read should remain pending"),- }- })- .await- .unwrap();- let body_polled = Arc::new(AtomicBool::new(false));- let observed = body_polled.clone();- let body = Body::from_stream(stream::once(async move {- observed.store(true, Ordering::SeqCst);- Ok::<_, Infallible>(Bytes::from_static(br#"{"model":"connection:model"}"#))- }));- let busy = gateway- .app- .clone()- .oneshot(- Request::post("/openai/v1/responses")- .header(header::AUTHORIZATION, "Bearer busy")- .body(body)- .unwrap(),- )- .await- .unwrap();- assert_eq!(busy.status(), StatusCode::SERVICE_UNAVAILABLE);- assert!(!body_polled.load(Ordering::SeqCst));- let models = gateway- .app- .clone()- .oneshot(Request::get("/openai/v1/models").header(header::AUTHORIZATION, "Bearer models").body(Body::empty()).unwrap(),)- .await- .unwrap();- assert_eq!(models.status(), StatusCode::OK);- assert_eq!(web.calls(), 1);- assert_eq!(provider.calls(), 0);- drop(held);- let response = gateway- .app- .clone()- .oneshot(- Request::post("/openai/v1/responses")- .header(header::AUTHORIZATION, "Bearer released")- .body(Body::from(r#"{"model":"connection:model"}"#))- .unwrap(),- )- .await- .unwrap();- assert_eq!(response.status(), StatusCode::OK);- assert_eq!(web.calls(), 2);- assert_eq!(provider.calls(), 1);-}+ };-#[tokio::test]-async fn slow_model_catalogs_do_not_consume_resolution_capacity() {- let entered = Arc::new(tokio::sync::Notify::new());- let started = entered.clone();- let web = FakeServer::start(move |request| {- let started = started.clone();- async move {- if request.uri().path() == "/api/internal/ai-gateway/v1/models" {- started.notify_one();- return std::future::pending().await;- }- resolution_response("provider-secret")- }- })- .await;- let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;- let gateway = Gateway::start(Some(InferenceService::for_test(- web.control_plane(),- ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()),- 1,- )))- .await;- let mut held = Box::pin(- gateway.app.clone().oneshot(- Request::get("/openai/v1/models")- .header(header::AUTHORIZATION, "Bearer models-held")- .body(Body::empty())- .unwrap(),- ),+ let unread = catalog_request().await.unwrap();+ assert_eq!(unread.status(), StatusCode::OK);+ assert_eq!(+ catalog_request().await.unwrap().status(),+ StatusCode::SERVICE_UNAVAILABLE+ );+ let body = to_bytes(unread.into_body(), usize::MAX).await.unwrap();+ assert_eq!(+ serde_json::from_slice::<serde_json::Value>(&body).unwrap()["data"]+ .as_array()+ .unwrap()+ .len(),+ 2000);- tokio::time::timeout(Duration::from_secs(2), async {- tokio::select! {- () = entered.notified() => {},- _ = &mut held => panic!("model catalog should remain pending"),- }- })- .await- .unwrap();- let inference = gateway- .app- .clone()- .oneshot(- Request::post("/openai/v1/responses")- .header(header::AUTHORIZATION, "Bearer inference")- .body(Body::from(r#"{"model":"connection:model"}"#))- .unwrap(),- )- .await- .unwrap();- assert_eq!(inference.status(), StatusCode::OK);- let busy = gateway- .app- .clone()- .oneshot(- Request::get("/openai/v1/models")- .header(header::AUTHORIZATION, "Bearer models-busy")- .body(Body::empty())- .unwrap(),- )- .await- .unwrap();- assert_eq!(busy.status(), StatusCode::SERVICE_UNAVAILABLE);- assert_eq!(web.calls(), 2);- assert_eq!(provider.calls(), 1);- drop(held);+ let stalled = catalog_request().await.unwrap();+ tokio::time::sleep(Duration::from_millis(700)).await;+ assert_eq!(catalog_request().await.unwrap().status(), StatusCode::OK);+ assert!(to_bytes(stalled.into_body(), usize::MAX).await.is_err());}impl Gateway {@@ -592,7 +483,6 @@ fn inference(web: &FakeServer, provider: &FakeServer) -> InferenceService {InferenceService::for_test(web.control_plane(),ProviderTransport::for_test(format!("{}/v1", provider.url), ProviderLimits::default()),- 128,)}@@ -804,20 +694,14 @@ async fn model_routes_request_the_matching_control_plane_catalog() {}#[tokio::test]-async fn malformed_credentials_skip_resolution_and_oversized_body_releases_capacity() {+async fn malformed_credentials_skip_resolution_and_oversized_body_releases_memory() {let web = FakeServer::start(|_| async { resolution_response("provider-secret") }).await;let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;let service = InferenceService::for_test(web.control_plane(),- ProviderTransport::for_test(- provider.url.clone(),- ProviderLimits {- active: 1,- ..ProviderLimits::default()- },- ),- 1,- );+ ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()),+ )+ .with_request_limits(single_request_memory());let gateway = Gateway::start(Some(service)).await;for authorization in [None,@@ -1066,14 +950,12 @@ async fn pre_header_provider_failures_export_the_status_returned_to_the_caller()stall_timeout: Duration::from_millis(100),..ExecutionLimits::default()},- ..ProviderLimits::default()},).with_telemetry(telemetry.clone());let gateway = Gateway::start(Some(InferenceService::for_test(web.control_plane(),provider,- 1,))).await;let response = gateway@@ -1165,7 +1047,6 @@ async fn chat_completions_streams_carry_the_request_and_generation_ids() {web.control_plane(),ProviderTransport::for_test(format!("{}/v1", provider.url), ProviderLimits::default()).with_telemetry(telemetry.clone()),- 128,))).await;let response = gateway@@ -1273,7 +1154,7 @@ async fn unconfigured_gateway_is_live_but_not_ready_and_inference_is_unavailable}#[tokio::test]-async fn active_admission_is_held_through_stream_and_shutdown_drains_it() {+async fn request_memory_is_held_through_stream_and_shutdown_drains_it() {let (release, released) = tokio::sync::watch::channel(false);let web = FakeServer::start(|_| async { resolution_response("provider") }).await;let provider = FakeServer::start(move |_| {@@ -1294,15 +1175,9 @@ async fn active_admission_is_held_through_stream_and_shutdown_drains_it() {.await;let service = InferenceService::for_test(web.control_plane(),- ProviderTransport::for_test(- provider.url.clone(),- ProviderLimits {- active: 1,- ..ProviderLimits::default()- },- ),- 1,- );+ ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()),+ )+ .with_request_limits(single_request_memory());let mut gateway = Gateway::start(Some(service)).await;let mut response = gateway.post()@@ -1339,7 +1214,7 @@ async fn active_admission_is_held_through_stream_and_shutdown_drains_it() {}#[tokio::test]-async fn client_disconnect_releases_capacity_for_the_next_request() {+async fn client_disconnect_releases_memory_for_the_next_request() {let web = FakeServer::start(|_| async { resolution_response("provider") }).await;let provider = FakeServer::start(|_| async {let stream =@@ -1352,15 +1227,9 @@ async fn client_disconnect_releases_capacity_for_the_next_request() {.await;let service = InferenceService::for_test(web.control_plane(),- ProviderTransport::for_test(- provider.url.clone(),- ProviderLimits {- active: 1,- ..ProviderLimits::default()- },- ),- 1,- );+ ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()),+ )+ .with_request_limits(single_request_memory());let gateway = Gateway::start(Some(service)).await;let mut first = gateway.post()@@ -1603,24 +1472,17 @@ async fn model_policy_rejections_use_provider_native_permission_errors() {#[tokio::test]async fn anthropic_capacity_rejections_use_the_native_envelope() {let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;- let busy = ProviderTransport::for_test(- format!("{}/v1", provider.url),- ProviderLimits {- active: 1,- ..ProviderLimits::default()- },- );- let _held = busy.try_admit().unwrap();let web = FakeServer::start(|_| async {resolution_response_for(ApiFormat::AnthropicMessages, "provider-anthropic")}).await;- let gateway = Gateway::start(Some(InferenceService::for_test(+ let service = InferenceService::for_test(web.control_plane(),- busy,- 1,- )))- .await;+ ProviderTransport::for_test(format!("{}/v1", provider.url), ProviderLimits::default()),+ )+ .with_request_limits(single_request_memory());+ let _held = service.try_reserve_memory("request").unwrap();+ let gateway = Gateway::start(Some(service)).await;let response = reqwest::Client::new().post(format!("{}/anthropic/v1/messages", gateway.url)).header("x-api-key", "gateway-anthropic")@@ -1772,118 +1634,54 @@ async fn anthropic_routes_share_the_request_body_limit() {}#[tokio::test]-async fn buffered_request_bytes_are_released_once_the_response_ends() {- let (release, released) = tokio::sync::watch::channel(false);- let web = FakeServer::start(|_| async { resolution_response("provider") }).await;- let provider = FakeServer::start(move |request: Request| {- let mut released = released.clone();- async move {- to_bytes(request.into_body(), usize::MAX).await.unwrap();- released.wait_for(|done| *done).await.unwrap();- Response::new(Body::empty())- }- })- .await;- let service = inference(&web, &provider).with_request_limits(RequestLimits {- max_request_bytes: 64,- shared_buffer_bytes: 100,- });- let gateway = Gateway::start(Some(service)).await;- let body = |size: usize| {- let prefix = r#"{"model":"Production key:test","padding":""#;- format!("{prefix}{}\"}}", "x".repeat(size - prefix.len() - 2))- };- let oversized = gateway.post().bearer_auth("key").body(body(65)).send();- assert_eq!(- oversized.await.unwrap().status(),- StatusCode::PAYLOAD_TOO_LARGE- );- // Two 60-byte calls do not fit the 100-byte budget at once.- let first = tokio::spawn(gateway.post().bearer_auth("key").body(body(60)).send());- while provider.calls() == 0 {- tokio::task::yield_now().await;- }- let second = gateway.post().bearer_auth("key").body(body(60)).send();- let second = tokio::time::timeout(Duration::from_secs(5), second)- .await- .expect("the first call still holds its bytes while it waits for headers");- assert_eq!(second.unwrap().status(), StatusCode::SERVICE_UNAVAILABLE);- release.send(true).unwrap();- let first = first.await.unwrap().unwrap();- assert_eq!(first.status(), StatusCode::OK);- first.bytes().await.unwrap();- tokio::time::timeout(Duration::from_secs(5), async {- loop {- let next = gateway.post().bearer_auth("key").body(body(60)).send();- if next.await.unwrap().status() == StatusCode::OK {- break;- }- tokio::task::yield_now().await;- }- })- .await- .expect("the first call's bytes are released once its response ends");-}--async fn budgeted_service() -> (FakeServer, FakeServer, InferenceService) {+async fn body_reservations_follow_the_allocation_on_top_of_the_overhead() {let web = FakeServer::start(|_| async { resolution_response("provider") }).await;let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;let service = inference(&web, &provider).with_request_limits(RequestLimits {max_request_bytes: 64,- shared_buffer_bytes: 100,+ memory_budget_bytes: 100,+ request_overhead_bytes: 10,});- (web, provider, service)-}--fn streamed_body(chunks: Vec<&'static [u8]>, stall: bool) -> Body {- let chunks = stream::iter(chunks.into_iter().map(Ok::<_, Infallible>));- if stall {- Body::from_stream(chunks.chain(stream::pending()))- } else {- Body::from_stream(chunks)- }-}+ let body = |chunks: Vec<&'static [u8]>| {+ Body::from_stream(stream::iter(chunks.into_iter().map(Ok::<_, Infallible>)))+ };-#[tokio::test]-async fn a_stalled_upload_holds_only_the_bytes_it_sent() {- let (_web, _provider, service) = budgeted_service().await;- let mut headers = HeaderMap::new();- headers.insert(header::CONTENT_LENGTH, "60".parse().unwrap());- let stalled = service.try_reserve_request_body().ok().unwrap();- let read = read_request_body(- streamed_body(vec![&[b'x'; 10]], true),- &headers,+ // A declared length is not reserved up front: a stalled upload holds what it sent.+ let mut declared = HeaderMap::new();+ declared.insert(header::CONTENT_LENGTH, "60".parse().unwrap());+ let mut stalled = Box::pin(read_request_body(+ Body::from_stream(+ body(vec![&[b'x'; 10]])+ .into_data_stream()+ .chain(stream::pending()),+ ),+ &declared,64,- stalled,- );- tokio::pin!(read);+ service.try_reserve_memory("request").ok().unwrap(),+ ));assert!(- tokio::time::timeout(Duration::from_millis(50), &mut read)+ tokio::time::timeout(Duration::from_millis(50), &mut stalled).await.is_err());- let mut next = service.try_reserve_request_body().ok().unwrap();- assert!(next.try_reserve_bytes(91).is_err());- assert!(next.try_reserve_bytes(90).is_ok());-}+ assert_eq!(service.available_memory(), 100 - 10 - 10);+ drop(stalled);+ assert_eq!(service.available_memory(), 100);-#[tokio::test]-async fn buffered_bytes_reserve_the_allocation_not_just_the_length() {- let (_web, _provider, service) = budgeted_service().await;- let held = service.try_reserve_request_body().ok().unwrap();- let (bytes, _held) = read_request_body(- streamed_body(vec![&[b'x'; 32], b"x"], false),+ // Growth reserves the buffer's allocation, not just its length.+ let (bytes, held) = read_request_body(+ body(vec![&[b'x'; 32], b"x"]),&HeaderMap::new(),64,- held,+ service.try_reserve_memory("request").ok().unwrap(),).await.ok().unwrap();assert_eq!(bytes.len(), 33);- let mut next = service.try_reserve_request_body().ok().unwrap();- assert!(next.try_reserve_bytes(37).is_err());- assert!(next.try_reserve_bytes(36).is_ok());+ assert_eq!(service.available_memory(), 100 - 10 - 64);+ drop(held);+ assert_eq!(service.available_memory(), 100);}#[test]@@ -1933,10 +1731,8 @@ async fn only_non_streamed_generations_wait_past_the_stall_for_provider_headers(stall_timeout: Duration::from_millis(100),..ExecutionLimits::default()},- ..ProviderLimits::default()},),- 128,);let gateway = Gateway::start(Some(service)).await;let client = reqwest::Client::new();diff --git a/ai-gateway/src/inference.rs b/ai-gateway/src/inference.rsindex 3162477d4f..1357e2fd9e 100644--- a/ai-gateway/src/inference.rs+++ b/ai-gateway/src/inference.rs@@ -5,7 +5,7 @@ use axum::{body::Body,http::{HeaderMap, Response},};-use tokio::sync::{OwnedSemaphorePermit, Semaphore, SemaphorePermit};+use tokio::sync::{OwnedSemaphorePermit, Semaphore};use tracing::Instrument;use crate::{@@ -21,28 +21,11 @@ use crate::{pub struct InferenceService {control_plane: ControlPlaneClient,provider: ProviderTransport,- request_body_capacity: Arc<Semaphore>,request_limits: RequestLimits,- request_bytes_capacity: Arc<Semaphore>,- model_catalog_capacity: Semaphore,- resolution_capacity: Semaphore,+ memory_budget: Arc<Semaphore>,telemetry: Option<crate::telemetry::Telemetry>,}-pub(crate) enum RequestPreparationError {- Resolution(ResolutionError),- Provider(ProviderError),-}--impl RequestPreparationError {- pub(crate) fn origin(&self) -> (ErrorOrigin, &'static str) {- match *self {- Self::Resolution(error) => resolution_origin(error),- Self::Provider(error) => provider_origin(error),- }- }-}-/// `error.origin` and `error.reason` for a failed resolution or catalog fetch.pub(crate) fn resolution_origin(error: ResolutionError) -> (ErrorOrigin, &'static str) {use ResolutionError as R;@@ -66,27 +49,34 @@ pub(crate) fn resolution_origin(error: ResolutionError) -> (ErrorOrigin, &'statipub(crate) fn provider_origin(error: ProviderError) -> (ErrorOrigin, &'static str) {match error {ProviderError::Configuration => (ErrorOrigin::Gateway, "configuration"),- ProviderError::Busy => (ErrorOrigin::Gateway, "capacity"),ProviderError::Timeout => (ErrorOrigin::ProviderTransport, "timeout"),ProviderError::Transport => (ErrorOrigin::ProviderTransport, "transport"),}}-/// One request-body slot plus the body bytes reserved so far from the shared budget.-/// It is held with the execution and released once the response ends.-pub(crate) struct RequestBodyPermit {- _slot: OwnedSemaphorePermit,+/// Bytes one in-flight request holds from the shared memory budget: a fixed overhead+/// for response capture and connection buffers, plus its body as it arrives. It is held+/// with the execution and released once the response ends.+pub(crate) struct MemoryReservation {bytes: OwnedSemaphorePermit,+ overhead: usize,budget: Arc<Semaphore>,}-impl RequestBodyPermit {- /// Grow the reservation to `total` body bytes without waiting.+impl MemoryReservation {+ /// Grow the reservation to the overhead plus `body` bytes without waiting.////// # Errors- /// Returns [`ResolutionError::Busy`] when the shared budget cannot cover `total`.- pub(crate) fn try_reserve_bytes(&mut self, total: usize) -> Result<(), ResolutionError> {- let Some(missing) = total+ /// Returns [`ResolutionError::Busy`] when the shared budget cannot cover `body`,+ /// recording the rejection under `phase`.+ pub(crate) fn try_reserve_body(+ &mut self,+ body: usize,+ phase: &'static str,+ ) -> Result<(), ResolutionError> {+ let Some(missing) = self+ .overhead+ .saturating_add(body).checked_sub(self.bytes.num_permits()).filter(|missing| *missing > 0)else {@@ -100,14 +90,21 @@ impl RequestBodyPermit {.ok()}).ok_or_else(|| {- crate::observability::rejected("request_bytes");+ crate::observability::rejected(phase);ResolutionError::Busy})?;+ crate::observability::memory_reserved(missing);self.bytes.merge(reserved);Ok(())}}+impl Drop for MemoryReservation {+ fn drop(&mut self) {+ crate::observability::memory_released(self.bytes.num_permits());+ }+}+/// Uploads slower than 10 MiB per 30 seconds on average miss the read deadline.const MIN_REQUEST_UPLOAD_BYTES_PER_SECOND: usize = 10 * 1024 * 1024 / 30;const MIN_REQUEST_READ_TIMEOUT: Duration = Duration::from_secs(30);@@ -116,31 +113,24 @@ impl InferenceService {/// Initialize pooled clients for resolution and provider requests.////// # Errors- /// Returns a sanitized configuration error for invalid capacities or client setup.+ /// Returns a sanitized configuration error for invalid limits or client setup.pub fn new(control_plane: ControlPlaneConfig,config: &GatewayConfig,) -> Result<Self, ResolutionError> {- let max_concurrent_resolutions = config.max_concurrent_resolutions;- if !(1..=Semaphore::MAX_PERMITS).contains(&max_concurrent_resolutions) {+ if config.request_limits.memory_budget_bytes > Semaphore::MAX_PERMITS {return Err(ResolutionError::Configuration);}let telemetry =crate::telemetry::Telemetry::new(&control_plane, config.telemetry_buffer_bytes)?;- crate::observability::record_capacity("model_catalog", max_concurrent_resolutions);- crate::observability::record_capacity("resolution", max_concurrent_resolutions);+ crate::observability::record_memory_budget(config.request_limits.memory_budget_bytes);Ok(Self {control_plane: ControlPlaneClient::new(control_plane)?,- provider: ProviderTransport::new(config.max_active_requests, config.execution_limits)+ provider: ProviderTransport::new(config.execution_limits).map_err(|_| ResolutionError::Configuration)?.with_telemetry(telemetry.clone()),- request_body_capacity: Arc::new(Semaphore::new(config.max_active_requests)),request_limits: config.request_limits,- request_bytes_capacity: Arc::new(Semaphore::new(- config.request_limits.shared_buffer_bytes,- )),- model_catalog_capacity: Semaphore::new(max_concurrent_resolutions),- resolution_capacity: Semaphore::new(max_concurrent_resolutions),+ memory_budget: Arc::new(Semaphore::new(config.request_limits.memory_budget_bytes)),telemetry: Some(telemetry),})}@@ -149,29 +139,31 @@ impl InferenceService {self.telemetry.clone()}- fn try_reserve_resolution(&self) -> Result<SemaphorePermit<'_>, ResolutionError> {- self.resolution_capacity.try_acquire().map_err(|_| {- crate::observability::rejected("resolution");- ResolutionError::Busy- })- }-- pub(crate) fn try_reserve_request_body(+ /// Reserve the fixed per-request overhead without waiting. `phase` labels a rejection.+ ///+ /// # Errors+ /// Returns [`ResolutionError::Busy`] when the shared budget cannot cover the overhead.+ pub(crate) fn try_reserve_memory(&self,- ) -> Result<RequestBodyPermit, RequestPreparationError> {- let slot = Arc::clone(&self.request_body_capacity)- .try_acquire_owned()- .map_err(|_| {- crate::observability::rejected("request_body");- RequestPreparationError::Resolution(ResolutionError::Busy)+ phase: &'static str,+ ) -> Result<MemoryReservation, ResolutionError> {+ let overhead = self.request_limits.request_overhead_bytes;+ let bytes = u32::try_from(overhead)+ .ok()+ .and_then(|overhead| {+ Arc::clone(&self.memory_budget)+ .try_acquire_many_owned(overhead)+ .ok()+ })+ .ok_or_else(|| {+ crate::observability::rejected(phase);+ ResolutionError::Busy})?;- let bytes = Arc::clone(&self.request_bytes_capacity)- .try_acquire_many_owned(0)- .map_err(|_| RequestPreparationError::Resolution(ResolutionError::Unavailable))?;- Ok(RequestBodyPermit {- _slot: slot,+ crate::observability::memory_reserved(overhead);+ Ok(MemoryReservation {bytes,- budget: Arc::clone(&self.request_bytes_capacity),+ overhead,+ budget: Arc::clone(&self.memory_budget),})}@@ -195,29 +187,32 @@ impl InferenceService {gateway_key: &str,api_format: ApiFormat,correlation: &RequestCorrelation,- ) -> Result<Vec<u8>, ResolutionError> {- let _permit = self.model_catalog_capacity.try_acquire().map_err(|_| {- crate::observability::rejected("model_catalog");- ResolutionError::Busy- })?;+ ) -> Result<Body, ResolutionError> {+ let mut reservation = self.try_reserve_memory("model_catalog")?;let _active = crate::observability::Active::new("model_catalog");let body = self.control_plane- .models(gateway_key, api_format, correlation.id())+ .models(gateway_key, api_format, correlation.id(), &mut |capacity| {+ reservation.try_reserve_body(capacity, "model_catalog")+ }).await?;- crate::providers::model_list::provider_response(&body, api_format)+ // Parsing holds the typed catalog and the provider-shaped output beside the body.+ reservation.try_reserve_body(body.len().saturating_mul(3), "model_catalog")?;+ let catalog = crate::providers::model_list::provider_response(&body, api_format)?;+ drop(body);+ // The reservation is released once the client has read the catalog.+ Ok(self.provider.relay_buffered(catalog, reservation))}- /// Authenticate, then reserve execution capacity. The API format comes from the public route,- /// so Web selects a compatible connection. A successful resolution is recorded even if- /// execution admission then fails.+ /// Authenticate, then start the execution. The API format comes from the public route,+ /// so Web selects a compatible connection. Memory is already reserved by the caller.pub(crate) async fn resolve_and_admit(&self,gateway_key: &str,api_format: ApiFormat,model: &str,correlation: &mut RequestCorrelation,- ) -> Result<(RequestPermit, ResolvedRequestContext), RequestPreparationError> {+ ) -> Result<(RequestPermit, ResolvedRequestContext), ResolutionError> {let span = tracing::info_span!("resolution",otel.kind = "internal",@@ -226,59 +221,38 @@ impl InferenceService {error.type = tracing::field::Empty);async {- let prepared = self- .prepare(gateway_key, api_format, model, correlation)- .await;+ let resolved = {+ let _active = crate::observability::Active::new("resolution");+ self.control_plane+ .resolve(gateway_key, api_format, model, correlation.id())+ .await+ };let span = tracing::Span::current();span.record("gateway.outcome",- match &prepared {- Ok(_) => "admitted",- Err(RequestPreparationError::Resolution(_)) => "resolution_failed",- Err(RequestPreparationError::Provider(_)) => "busy",+ if resolved.is_ok() {+ "admitted"+ } else {+ "resolution_failed"},);// Caller-caused failures (bad key, forbidden model) stay unset so// Datadog error rates track gateway and control-plane faults only.- if let Err(error) = &prepared- && let (origin, reason) = error.origin()+ if let Err(error) = resolved+ && let (origin, reason) = resolution_origin(error)&& origin.is_ours(){span.record("otel.status_code", "ERROR");span.record("error.type", reason);}- prepared+ let context = resolved?;+ correlation.record_resolution(&context);+ Ok((self.provider.admit(), context))}.instrument(span).await}- async fn prepare(- &self,- gateway_key: &str,- api_format: ApiFormat,- model: &str,- correlation: &mut RequestCorrelation,- ) -> Result<(RequestPermit, ResolvedRequestContext), RequestPreparationError> {- let resolution_permit = self- .try_reserve_resolution()- .map_err(RequestPreparationError::Resolution)?;- let context = {- let _active = crate::observability::Active::new("resolution");- self.control_plane- .resolve(gateway_key, api_format, model, correlation.id())- .await- .map_err(RequestPreparationError::Resolution)?- };- drop(resolution_permit);- correlation.record_resolution(&context);- let permit = self- .provider- .try_admit()- .map_err(RequestPreparationError::Provider)?;- Ok((permit, context))- }-pub(crate) async fn forward(&self,permit: RequestPermit,@@ -294,29 +268,25 @@ impl InferenceService {}#[cfg(test)]- pub(crate) fn for_test(- control_plane: ControlPlaneClient,- provider: ProviderTransport,- resolutions: usize,- ) -> Self {+ pub(crate) fn for_test(control_plane: ControlPlaneClient, provider: ProviderTransport) -> Self {Self {control_plane,provider,- request_body_capacity: Arc::new(Semaphore::new(resolutions)),request_limits: RequestLimits::default(),- request_bytes_capacity: Arc::new(Semaphore::new(- RequestLimits::default().shared_buffer_bytes,- )),- model_catalog_capacity: Semaphore::new(resolutions),- resolution_capacity: Semaphore::new(resolutions),+ memory_budget: Arc::new(Semaphore::new(RequestLimits::default().memory_budget_bytes)),telemetry: None,}}#[cfg(test)]pub(crate) fn with_request_limits(mut self, request_limits: RequestLimits) -> Self {- self.request_bytes_capacity = Arc::new(Semaphore::new(request_limits.shared_buffer_bytes));+ self.memory_budget = Arc::new(Semaphore::new(request_limits.memory_budget_bytes));self.request_limits = request_limits;self}++ #[cfg(test)]+ pub(crate) fn available_memory(&self) -> usize {+ self.memory_budget.available_permits()+ }}diff --git a/ai-gateway/src/main.rs b/ai-gateway/src/main.rsindex b32b21f87d..307242c32c 100644--- a/ai-gateway/src/main.rs+++ b/ai-gateway/src/main.rs@@ -45,7 +45,7 @@ async fn run() -> Result<(), Box<dyn Error>> {)));let shutdown = shutdown_signal()?;let listener = bind_listener(config.listen_address, config.auto_increment_listen_port).await?;- tracing::info!(address = %listener.local_addr()?, inference_enabled, max_active_requests = config.max_active_requests, max_concurrent_resolutions = config.max_concurrent_resolutions, max_request_bytes = config.request_limits.max_request_bytes, shared_request_buffer_bytes = config.request_limits.shared_buffer_bytes, max_execution_seconds = config.execution_limits.max_execution_time.as_secs(), stall_timeout_seconds = config.execution_limits.stall_timeout.as_secs(), shutdown_timeout_seconds = config.shutdown_timeout.as_secs(), "gateway listening");+ tracing::info!(address = %listener.local_addr()?, inference_enabled, max_request_bytes = config.request_limits.max_request_bytes, memory_budget_bytes = config.request_limits.memory_budget_bytes, request_memory_overhead_bytes = config.request_limits.request_overhead_bytes, max_execution_seconds = config.execution_limits.max_execution_time.as_secs(), stall_timeout_seconds = config.execution_limits.stall_timeout.as_secs(), shutdown_timeout_seconds = config.shutdown_timeout.as_secs(), "gateway listening");let result = server::serve(listener,app,diff --git a/ai-gateway/src/observability/http/tests.rs b/ai-gateway/src/observability/http/tests.rsindex d905df4d7f..f1a774d276 100644--- a/ai-gateway/src/observability/http/tests.rs+++ b/ai-gateway/src/observability/http/tests.rs@@ -1,5 +1,6 @@use super::*;use crate::{+ config::RequestLimits,inference::InferenceService,providers::{ProviderLimits, ProviderTransport},resolution::ControlPlaneConfig,@@ -202,7 +203,7 @@ async fn provider_http_errors_are_traced_without_changing_the_response() {let recording = Recording::start();let response = provider.forward(- provider.try_admit().unwrap(),+ provider.admit(),context,&HeaderMap::new(),Bytes::from_static(br#"{"model":"requested"}"#),@@ -242,10 +243,11 @@ async fn provider_http_errors_are_traced_without_changing_the_response() {/// `error.origin`; only 5xx responses mark the span as an error.#[tokio::test]async fn failed_responses_name_their_origin_on_the_server_span() {- for (provider_status, capacity, origin, reason) in [- (429, 1, "provider", "provider_rate_limited"),- (401, 1, "provider", "provider_auth"),- (500, 1, "provider", "provider_5xx"),+ let budget = RequestLimits::default().memory_budget_bytes;+ for (provider_status, memory_budget_bytes, origin, reason) in [+ (429, budget, "provider", "provider_rate_limited"),+ (401, budget, "provider", "provider_auth"),+ (500, budget, "provider", "provider_5xx"),(200, 0, "gateway", "capacity"),] {let web = FakeServer::start(|_| async { resolution_response("provider-secret") }).await;@@ -259,8 +261,11 @@ async fn failed_responses_name_their_origin_on_the_server_span() {let service = InferenceService::for_test(web.control_plane(),ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()),- capacity,- );+ )+ .with_request_limits(RequestLimits {+ memory_budget_bytes,+ ..RequestLimits::default()+ });let recording = Recording::start();let lifecycle = GatewayLifecycleState::default();let app = instrument(crate::http::router(Some(service), lifecycle.clone()));@@ -290,7 +295,7 @@ async fn failed_responses_name_their_origin_on_the_server_span() {status.is_server_error(),"{reason}");- if capacity == 0 {+ if memory_budget_bytes == 0 {assert_eq!(status, 503);assert_eq!(web.calls() + provider.calls(), 0);} else {@@ -326,7 +331,6 @@ async fn late_body_failures_fail_the_server_span_without_a_generation() {let service = InferenceService::for_test(web.control_plane(),ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()),- 1,);let recording = Recording::start();let lifecycle = GatewayLifecycleState::default();@@ -422,7 +426,6 @@ async fn generation_context_is_isolated_from_operational_spans_and_outbound_headweb.control_plane(),ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()).with_telemetry(telemetry.clone()),- 1,);let recording = Recording::start();let lifecycle = GatewayLifecycleState::default();diff --git a/ai-gateway/src/observability/metrics.rs b/ai-gateway/src/observability/metrics.rsindex 9c5841e727..fad50b911c 100644--- a/ai-gateway/src/observability/metrics.rs+++ b/ai-gateway/src/observability/metrics.rs@@ -127,83 +127,103 @@ pub(crate) fn response_body_size(provider: &'static str, bytes: u64) {.record(bytes, &[KeyValue::new("gateway.provider", provider)]);}-/// In-use, high-water, and configured slots of one admission semaphore.-struct Slots {- phase: &'static str,- in_use: AtomicU64,+/// A current level and its high-water mark since the previous export.+struct Level {+ current: AtomicU64,peak: AtomicU64,- capacity: AtomicU64,}-impl Slots {- const fn new(phase: &'static str) -> Self {+impl Level {+ const fn new() -> Self {Self {- phase,- in_use: AtomicU64::new(0),+ current: AtomicU64::new(0),peak: AtomicU64::new(0),- capacity: AtomicU64::new(0),}}- fn acquire(&self) {- let in_use = self.in_use.fetch_add(1, Ordering::Relaxed) + 1;- self.peak.fetch_max(in_use, Ordering::Relaxed);+ fn add(&self, amount: u64) {+ let current = self.current.fetch_add(amount, Ordering::Relaxed) + amount;+ self.peak.fetch_max(current, Ordering::Relaxed);}- fn release(&self) {- self.in_use.fetch_sub(1, Ordering::Relaxed);+ fn sub(&self, amount: u64) {+ self.current.fetch_sub(amount, Ordering::Relaxed);}- /// The highest in-use count since the previous read. The next interval starts- /// from the slots still held, so a long request is not reported as idle.+ /// The highest level since the previous read. The next interval starts from the+ /// level still held, so a long request is not reported as idle.fn take_peak(&self) -> u64 {let peak = self.peak- .swap(self.in_use.load(Ordering::Relaxed), Ordering::Relaxed);+ .swap(self.current.load(Ordering::Relaxed), Ordering::Relaxed);self.peak- .fetch_max(self.in_use.load(Ordering::Relaxed), Ordering::Relaxed);+ .fetch_max(self.current.load(Ordering::Relaxed), Ordering::Relaxed);peak}}-static SLOTS: [Slots; 3] = [- Slots::new("model_catalog"),- Slots::new("resolution"),- Slots::new("execution"),+struct Phase {+ name: &'static str,+ in_flight: Level,+}++static PHASES: [Phase; 3] = [+ Phase {+ name: "model_catalog",+ in_flight: Level::new(),+ },+ Phase {+ name: "resolution",+ in_flight: Level::new(),+ },+ Phase {+ name: "execution",+ in_flight: Level::new(),+ },];-fn slots(phase: &str) -> Option<&'static Slots> {- SLOTS.iter().find(|slots| slots.phase == phase)+fn phase(name: &str) -> Option<&'static Phase> {+ PHASES.iter().find(|phase| phase.name == name)}-/// Report a semaphore's configured size for `gateway.admission.capacity`. The-/// gateway runs one inference service per process, so the latest value wins.-pub(crate) fn record_capacity(phase: &'static str, capacity: usize) {- if let Some(slots) = slots(phase) {- slots.capacity.store(- u64::try_from(capacity).unwrap_or(u64::MAX),- Ordering::Relaxed,- );- }+static MEMORY_RESERVED: Level = Level::new();+static MEMORY_BUDGET: AtomicU64 = AtomicU64::new(0);++fn bytes(amount: usize) -> u64 {+ u64::try_from(amount).unwrap_or(u64::MAX)+}++/// Report the configured budget for `gateway.memory.budget`. The gateway runs one+/// inference service per process, so the latest value wins.+pub(crate) fn record_memory_budget(budget: usize) {+ MEMORY_BUDGET.store(bytes(budget), Ordering::Relaxed);+}++pub(crate) fn memory_reserved(amount: usize) {+ MEMORY_RESERVED.add(bytes(amount));+}++pub(crate) fn memory_released(amount: usize) {+ MEMORY_RESERVED.sub(bytes(amount));}pub(crate) struct Active(&'static str);impl Active {- pub(crate) fn new(phase: &'static str) -> Self {- METRICS.active.add(1, &[KeyValue::new("phase", phase)]);- if let Some(slots) = slots(phase) {- slots.acquire();+ pub(crate) fn new(name: &'static str) -> Self {+ METRICS.active.add(1, &[KeyValue::new("phase", name)]);+ if let Some(phase) = phase(name) {+ phase.in_flight.add(1);}- Self(phase)+ Self(name)}}impl Drop for Active {fn drop(&mut self) {METRICS.active.add(-1, &[KeyValue::new("phase", self.0)]);- if let Some(slots) = slots(self.0) {- slots.release();+ if let Some(phase) = phase(self.0) {+ phase.in_flight.sub(1);}}}@@ -279,58 +299,74 @@ pub(crate) fn execution_failed(outcome: RelayOutcome) {/// Keeps the admission callbacks registered for the life of the meter provider.pub(super) struct AdmissionInstruments {- _capacity: ObservableGauge<u64>,- _peak: ObservableGauge<u64>,+ _active_peak: ObservableGauge<u64>,+ _memory_budget: ObservableGauge<u64>,+ _memory_reserved: ObservableGauge<u64>,+ _memory_reserved_peak: ObservableGauge<u64>,}-/// Free slots are `gateway.admission.capacity - gateway.active`. The 30-second-/// average of `gateway.active` hides bursts, so `gateway.active.peak` reports the-/// most slots held at once since the previous export.+/// Free memory is `gateway.memory.budget - gateway.memory.reserved`. A 30-second+/// average hides bursts, so the `.peak` gauges report the highest level since the+/// previous export.pub(super) fn register_admission() -> AdmissionInstruments {let meter = global::meter("ai-gateway");- let capacity = meter- .u64_observable_gauge("gateway.admission.capacity")+ // The SDK reads observable instruments once per collection, which is once+ // per export with the single periodic reader.+ let active_peak = meter+ .u64_observable_gauge("gateway.active.peak").with_callback(|observer| {- for slots in &SLOTS {- let capacity = slots.capacity.load(Ordering::Relaxed);- if capacity > 0 {- observer.observe(capacity, &[KeyValue::new("phase", slots.phase)]);- }+ for phase in &PHASES {+ observer.observe(+ phase.in_flight.take_peak(),+ &[KeyValue::new("phase", phase.name)],+ );}}).build();- // The SDK reads observable instruments once per collection, which is once- // per export with the single periodic reader.- let peak = meter- .u64_observable_gauge("gateway.active.peak")+ let memory_budget = meter+ .u64_observable_gauge("gateway.memory.budget")+ .with_unit("By").with_callback(|observer| {- for slots in &SLOTS {- observer.observe(slots.take_peak(), &[KeyValue::new("phase", slots.phase)]);+ let budget = MEMORY_BUDGET.load(Ordering::Relaxed);+ if budget > 0 {+ observer.observe(budget, &[]);}}).build();+ let memory_reserved = meter+ .u64_observable_gauge("gateway.memory.reserved")+ .with_unit("By")+ .with_callback(|observer| {+ observer.observe(MEMORY_RESERVED.current.load(Ordering::Relaxed), &[]);+ })+ .build();+ let memory_reserved_peak = meter+ .u64_observable_gauge("gateway.memory.reserved.peak")+ .with_unit("By")+ .with_callback(|observer| observer.observe(MEMORY_RESERVED.take_peak(), &[]))+ .build();AdmissionInstruments {- _capacity: capacity,- _peak: peak,+ _active_peak: active_peak,+ _memory_budget: memory_budget,+ _memory_reserved: memory_reserved,+ _memory_reserved_peak: memory_reserved_peak,}}#[cfg(test)]mod tests {- use super::Slots;+ use super::Level;#[test]- fn peak_reports_the_interval_burst_then_restarts_from_held_slots() {- let slots = Slots::new("execution");- for _ in 0..3 {- slots.acquire();- }- slots.release();- slots.release();- assert_eq!(slots.take_peak(), 3, "a burst between exports is kept");- assert_eq!(slots.take_peak(), 1, "a slot still held stays visible");- slots.release();- assert_eq!(slots.take_peak(), 1);- assert_eq!(slots.take_peak(), 0);+ fn peak_reports_the_interval_burst_then_restarts_from_the_held_level() {+ let level = Level::new();+ level.add(1);+ level.add(2);+ level.sub(2);+ assert_eq!(level.take_peak(), 3, "a burst between exports is kept");+ assert_eq!(level.take_peak(), 1, "a level still held stays visible");+ level.sub(1);+ assert_eq!(level.take_peak(), 1);+ assert_eq!(level.take_peak(), 0);}}diff --git a/ai-gateway/src/observability/mod.rs b/ai-gateway/src/observability/mod.rsindex a4189edccf..53936a7f64 100644--- a/ai-gateway/src/observability/mod.rs+++ b/ai-gateway/src/observability/mod.rs@@ -7,8 +7,8 @@ mod request;pub(crate) use client::instrument_client;pub use http::instrument;pub(crate) use metrics::{- Active, delivery, execution_failed, execution_finished, record_capacity, rejected,- response_body_size,+ Active, delivery, execution_failed, execution_finished, memory_released, memory_reserved,+ record_memory_budget, rejected, response_body_size,};pub(crate) use request::{ErrorOrigin, RequestObservation, Wait, provider_status_reason};diff --git a/ai-gateway/src/providers/mod.rs b/ai-gateway/src/providers/mod.rsindex a8cdbf9f11..426ff66ede 100644--- a/ai-gateway/src/providers/mod.rs+++ b/ai-gateway/src/providers/mod.rs@@ -1,7 +1,7 @@mod chat_completions;pub(crate) mod model_list;-use std::{ops::Range, sync::Arc, time::Duration};+use std::{ops::Range, time::Duration};use axum::{body::{Body, Bytes},@@ -10,10 +10,7 @@ use axum::{use reqwest::Client;use reqwest_middleware::ClientWithMiddleware;use reqwest_tracing::DisableOtelPropagation;-use tokio::{- sync::{OwnedSemaphorePermit, Semaphore},- time::Instant,-};+use tokio::time::Instant;pub use crate::transport::ProviderError;use crate::{@@ -75,22 +72,14 @@ impl Route {}}+#[derive(Default)]pub(crate) struct ProviderLimits {- pub active: usize,pub execution: ExecutionLimits,}-impl Default for ProviderLimits {- fn default() -> Self {- Self {- active: 128,- execution: ExecutionLimits::default(),- }- }-}-+/// An admitted execution: its deadline and in-flight accounting. Memory is reserved+/// separately, before the body is read.pub struct RequestPermit {- _permit: OwnedSemaphorePermit,_active: crate::observability::Active,deadline: Instant,}@@ -109,7 +98,6 @@ pub(crate) type BodyReservation = Box<dyn Send + Sync>;pub struct ProviderTransport {client: ClientWithMiddleware,- capacity: Arc<Semaphore>,limits: ProviderLimits,telemetry: Option<crate::telemetry::Telemetry>,origin_override: Option<String>,@@ -118,20 +106,11 @@ pub struct ProviderTransport {impl ProviderTransport {/// # Errors/// Returns [`ProviderError::Configuration`] when the HTTPS client cannot be initialized.- pub fn new(- max_active_requests: usize,- execution: ExecutionLimits,- ) -> Result<Self, ProviderError> {- Self::with_limits(ProviderLimits {- active: max_active_requests,- execution,- })+ pub fn new(execution: ExecutionLimits) -> Result<Self, ProviderError> {+ Self::with_limits(ProviderLimits { execution })}fn with_limits(limits: ProviderLimits) -> Result<Self, ProviderError> {- if !(1..=Semaphore::MAX_PERMITS).contains(&limits.active) {- return Err(ProviderError::Configuration);- }let client = Client::builder().redirect(reqwest::redirect::Policy::none()).retry(reqwest::retry::never())@@ -146,10 +125,8 @@ impl ProviderTransport {.pool_max_idle_per_host(16).build().map_err(|_| ProviderError::Configuration)?;- crate::observability::record_capacity("execution", limits.active);Ok(Self {client: crate::observability::instrument_client(client, "provider.headers"),- capacity: Arc::new(Semaphore::new(limits.active)),limits,telemetry: None,origin_override: None,@@ -161,18 +138,12 @@ impl ProviderTransport {self}- /// # Errors- /// Returns [`ProviderError::Busy`] immediately when all execution slots are occupied.- pub fn try_admit(&self) -> Result<RequestPermit, ProviderError> {- let permit = self.capacity.clone().try_acquire_owned().map_err(|_| {- crate::observability::rejected("execution");- ProviderError::Busy- })?;- Ok(RequestPermit {- _permit: permit,+ /// Start the execution deadline.+ pub fn admit(&self) -> RequestPermit {+ RequestPermit {_active: crate::observability::Active::new("execution"),deadline: Instant::now() + self.limits.execution.max_execution_time,- })+ }}/// Forward as a streamed `OpenAI` Responses call.@@ -314,6 +285,17 @@ impl ProviderTransport {Ok(downstream)}+ /// Relay a gateway-built response body under the execution deadlines, holding+ /// `owner` until the client has read it or the relay ends.+ pub(crate) fn relay_buffered<T: Send + 'static>(&self, body: Vec<u8>, owner: T) -> Body {+ transport::relay_buffered(+ body,+ Instant::now() + self.limits.execution.max_execution_time,+ self.limits.execution.stall_timeout,+ owner,+ )+ }+fn request_url(&self, route: Route) -> String {let base = self.origin_overridediff --git a/ai-gateway/src/providers/tests.rs b/ai-gateway/src/providers/tests.rsindex 8997824fef..f7c52d00dc 100644--- a/ai-gateway/src/providers/tests.rs+++ b/ai-gateway/src/providers/tests.rs@@ -8,17 +8,11 @@ use axum::{http::{HeaderName, StatusCode},};use futures_util::{StreamExt, stream};-use std::{convert::Infallible, future::pending};+use std::{convert::Infallible, future::pending, sync::Arc};use tokio::sync::Notify;-fn provider(server: &FakeServer, active: usize) -> ProviderTransport {- ProviderTransport::for_test(- format!("{}/v1", server.url),- ProviderLimits {- active,- ..ProviderLimits::default()- },- )+fn provider(server: &FakeServer) -> ProviderTransport {+ ProviderTransport::for_test(format!("{}/v1", server.url), ProviderLimits::default())}/// Consumed as generation context; never forwarded upstream.@@ -64,10 +58,10 @@ fn model_rewrite_fails_closed_for_an_invalid_model_range() {#[tokio::test]async fn invalid_model_rewrite_never_reaches_the_provider() {let upstream = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;- let relay = provider(&upstream, 1);+ let relay = provider(&upstream);let result = relay.forward_route(- relay.try_admit().unwrap(),+ relay.admit(),resolved_request_context("provider-secret").await,&HeaderMap::new(),RequestBody {@@ -177,7 +171,7 @@ async fn rewrites_the_model_and_isolates_request_and_response_headers() {.unwrap()}).await;- let relay = provider(&upstream, 1);+ let relay = provider(&upstream);let mut headers = HeaderMap::new();for (name, value) in [("authorization", "Bearer gateway-secret"),@@ -201,7 +195,7 @@ async fn rewrites_the_model_and_isolates_request_and_response_headers() {}let response = relay.forward(- relay.try_admit().unwrap(),+ relay.admit(),resolved_request_context("provider-secret").await,&headers,Bytes::from_static(REQUEST),@@ -224,7 +218,6 @@ async fn rewrites_the_model_and_isolates_request_and_response_headers() {to_bytes(response.into_body(), 1024).await.unwrap(),RESPONSE);- assert!(relay.try_admit().is_ok());assert_eq!(upstream.calls(), 1);}@@ -245,10 +238,10 @@ async fn preserves_provider_failures_without_retries_or_redirects() {}}).await;- let relay = provider(&upstream, 1);+ let relay = provider(&upstream);let response = relay.forward(- relay.try_admit().unwrap(),+ relay.admit(),resolved_request_context("provider-secret").await,&HeaderMap::new(),Bytes::from_static(br#"{"model":"requested"}"#),@@ -268,7 +261,7 @@ async fn preserves_provider_failures_without_retries_or_redirects() {}#[tokio::test]-async fn streams_sse_incrementally_and_holds_admission_until_eof() {+async fn streams_sse_incrementally() {let release = Arc::new(Notify::new());let upstream_release = release.clone();let upstream = FakeServer::start(move |_| {@@ -287,10 +280,10 @@ async fn streams_sse_incrementally_and_holds_admission_until_eof() {}}).await;- let relay = provider(&upstream, 1);+ let relay = provider(&upstream);let response = relay.forward(- relay.try_admit().unwrap(),+ relay.admit(),resolved_request_context("provider-secret").await,&HeaderMap::new(),Bytes::from_static(br#"{"model":"requested"}"#),@@ -307,14 +300,12 @@ async fn streams_sse_incrementally_and_holds_admission_until_eof() {.unwrap(),"data: first\n\n");- assert!(matches!(relay.try_admit(), Err(ProviderError::Busy)));release.notify_one();let mut remaining = Vec::new();while let Some(chunk) = body.next().await {remaining.extend_from_slice(&chunk.unwrap());}assert_eq!(remaining, b"data: [DONE]\n\n");- assert!(relay.try_admit().is_ok());}#[tokio::test]@@ -331,19 +322,19 @@ async fn simultaneous_requests_keep_provider_credentials_and_bodies_isolated() {)))}).await;- let relay = provider(&upstream, 2);+ let relay = provider(&upstream);let alice = resolved_request_context("alice-secret").await;let bob = resolved_request_context("bob-secret").await;let headers = HeaderMap::new();let (alice, bob) = tokio::join!(relay.forward(- relay.try_admit().unwrap(),+ relay.admit(),alice,&headers,Bytes::from_static(br#"{"model":"requested","owner":"alice"}"#)),relay.forward(- relay.try_admit().unwrap(),+ relay.admit(),bob,&headers,Bytes::from_static(br#"{"model":"requested","owner":"bob"}"#)@@ -384,13 +375,13 @@ async fn stalled_provider(dropped: Arc<Notify>) -> FakeServer {}#[tokio::test]-async fn dropping_downstream_cancels_upstream_and_releases_admission() {+async fn dropping_downstream_cancels_upstream() {let dropped = Arc::new(Notify::new());let upstream = stalled_provider(dropped.clone()).await;- let relay = provider(&upstream, 1);+ let relay = provider(&upstream);let response = relay.forward(- relay.try_admit().unwrap(),+ relay.admit(),resolved_request_context("provider-secret").await,&HeaderMap::new(),Bytes::from_static(br#"{"model":"requested"}"#),@@ -399,9 +390,7 @@ async fn dropping_downstream_cancels_upstream_and_releases_admission() {.unwrap();let mut body = response.into_body().into_data_stream();assert_eq!(body.next().await.unwrap().unwrap(), "first");- assert!(matches!(relay.try_admit(), Err(ProviderError::Busy)));drop(body);- assert!(relay.try_admit().is_ok());tokio::time::timeout(Duration::from_secs(1), dropped.notified()).await.expect("upstream body should be dropped on cancellation");@@ -421,19 +410,17 @@ async fn deadlines_bound_headers_and_stalled_bodies_without_exposing_transport_dstall_timeout: Duration::from_millis(30),..ExecutionLimits::default()},- ..ProviderLimits::default()},);let response = relay.forward(- relay.try_admit().unwrap(),+ relay.admit(),resolved_request_context("provider-secret").await,&HeaderMap::new(),Bytes::from_static(br#"{"model":"requested"}"#),).await;assert!(matches!(response, Err(ProviderError::Timeout)));- assert!(relay.try_admit().is_ok());let dropped = Arc::new(Notify::new());let upstream = stalled_provider(dropped.clone()).await;@@ -444,13 +431,12 @@ async fn deadlines_bound_headers_and_stalled_bodies_without_exposing_transport_dmax_execution_time: Duration::from_millis(100),..ExecutionLimits::default()},- ..ProviderLimits::default()},);let context = resolved_request_context("provider-secret").await;let response = relay.forward(- relay.try_admit().unwrap(),+ relay.admit(),context,&HeaderMap::new(),Bytes::from_static(br#"{"model":"requested"}"#),@@ -464,7 +450,6 @@ async fn deadlines_bound_headers_and_stalled_bodies_without_exposing_transport_dlet error = to_bytes(response.into_body(), 1024).await.unwrap_err();assert!(!error.to_string().contains(&upstream.url));assert!(!error.to_string().contains("provider-secret"));- assert!(relay.try_admit().is_ok());}#[tokio::test]@@ -518,7 +503,7 @@ async fn completed_json_and_sse_upload_once_without_waiting_for_ingestion() {.unwrap()}).await;- let relay = provider(&upstream, 1).with_telemetry(telemetry.clone());+ let relay = provider(&upstream).with_telemetry(telemetry.clone());let headers = HeaderMap::from_iter([(axum::http::HeaderName::from_static("traceparent"),@@ -535,7 +520,7 @@ async fn completed_json_and_sse_upload_once_without_waiting_for_ingestion() {]);let response = relay.forward(- relay.try_admit().unwrap(),+ relay.admit(),resolved_request_context_with_mode("provider-secret", "full").await,&headers,Bytes::from_static(br#"{"model":"requested","input":"hello"}"#),@@ -556,7 +541,6 @@ async fn completed_json_and_sse_upload_once_without_waiting_for_ingestion() {}drop(body);assert_eq!(forwarded, native.as_bytes());- assert!(relay.try_admit().is_ok());let payload = tokio::time::timeout(Duration::from_secs(1), received.recv()).await.unwrap()@@ -601,7 +585,6 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rellet relay = ProviderTransport::for_test(upstream.url.clone(),ProviderLimits {- active: 1,execution: ExecutionLimits {max_execution_time: Duration::from_millis(150),..ExecutionLimits::default()@@ -612,7 +595,7 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rellet context = resolved_request_context("provider-secret").await;let response = relay.forward(- relay.try_admit().unwrap(),+ relay.admit(),context,&HeaderMap::new(),Bytes::from_static(br#"{"model":"requested","input":"prompt-canary"}"#),@@ -632,7 +615,6 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rel.await.unwrap().unwrap();- assert!(relay.try_admit().is_ok());assert!(!payload.to_string().contains("prompt-canary"));let attrs = payload["resourceSpans"][0]["scopeSpans"][0]["spans"][0]["attributes"].as_array()@@ -705,10 +687,10 @@ async fn compact_posts_compact_path_and_captures_ingestion() {.unwrap()}).await;- let relay = provider(&upstream, 1).with_telemetry(telemetry.clone());+ let relay = provider(&upstream).with_telemetry(telemetry.clone());let compact = relay.forward_route(- relay.try_admit().unwrap(),+ relay.admit(),resolved_request_context("provider-secret").await,&HeaderMap::new(),RequestBody {@@ -819,13 +801,7 @@ fn assert_completed_upload(payload: &serde_json::Value, streaming: bool) {}fn transport(server: &FakeServer) -> ProviderTransport {- ProviderTransport::for_test(- format!("{}/v1", server.url),- ProviderLimits {- active: 1,- ..ProviderLimits::default()- },- )+ ProviderTransport::for_test(format!("{}/v1", server.url), ProviderLimits::default())}/// Headers the gateway consumes for attribution or that belong to the client@@ -900,7 +876,7 @@ async fn sends_the_x_api_key_credential_and_forwards_the_open_anthropic_header_f.await;let response = relay.forward_route(- relay.try_admit().unwrap(),+ relay.admit(),context,&headers(&[@@ -975,7 +951,7 @@ async fn count_tokens_relays_the_body() {.await;let response = relay.forward_route(- relay.try_admit().unwrap(),+ relay.admit(),context,&headers(&[("content-type", "application/json")]),RequestBody {@@ -998,24 +974,6 @@ async fn count_tokens_relays_the_body() {assert_eq!(upstream.calls(), 1);}-#[test]-fn openai_and_anthropic_admissions_draw_from_one_execution_budget() {- let relay = ProviderTransport::for_test(- "http://127.0.0.1:9".to_owned(),- ProviderLimits {- active: 1,- ..ProviderLimits::default()- },- );- let held = relay.try_admit().unwrap();- assert!(relay.try_admit().is_err());- drop(held);- let held = relay.try_admit().unwrap();- assert!(relay.try_admit().is_err());- drop(held);- assert!(relay.try_admit().is_ok());-}-const CHAT_CONTENT_CHUNK: &str = "data: {\"id\":\"chatcmpl-1\",\"object\":\"chat.completion.chunk\",\"model\":\"gpt-4.1-mini-2025-04-14\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"Hello\"},\"finish_reason\":\"stop\"}],\"usage\":null}\n\n";const CHAT_USAGE_CHUNK: &str = "data: {\"id\":\"chatcmpl-1\",\"object\":\"chat.completion.chunk\",\"model\":\"gpt-4.1-mini-2025-04-14\",\"choices\":[],\"usage\":{\"prompt_tokens\":9,\"completion_tokens\":1,\"total_tokens\":10,\"prompt_tokens_details\":{\"cached_tokens\":4},\"completion_tokens_details\":{\"reasoning_tokens\":0}}}\n\n";const CHAT_DONE: &str = "data: [DONE]\n\n";@@ -1084,10 +1042,10 @@ async fn chat_completions_streams_request_usage_and_relay_the_usage_chunk() {.unwrap()}).await;- let relay = provider(&upstream, 1).with_telemetry(telemetry.clone());+ let relay = provider(&upstream).with_telemetry(telemetry.clone());let response = relay.forward_route(- relay.try_admit().unwrap(),+ relay.admit(),resolved_request_context_for(ApiFormat::OpenAiChatCompletions,"provider-secret",@@ -1145,7 +1103,7 @@ async fn chat_completions_rewrite_non_streamed_models() {let relay = transport(&upstream);let response = relay.forward_route(- relay.try_admit().unwrap(),+ relay.admit(),resolved_request_context_for(ApiFormat::OpenAiChatCompletions,"provider-secret",diff --git a/ai-gateway/src/resolution/mod.rs b/ai-gateway/src/resolution/mod.rsindex 9659b46cb2..ec9f06a2e2 100644--- a/ai-gateway/src/resolution/mod.rs+++ b/ai-gateway/src/resolution/mod.rs@@ -144,15 +144,18 @@ impl ControlPlaneClient {.await}- /// Fetch the caller-visible model catalog from trusted Web.+ /// Fetch the caller-visible model catalog from trusted Web. `reserve` is called+ /// with the buffer's new capacity before the buffer grows to it.////// # Errors- /// Returns the same sanitized errors as [`Self::resolve`].+ /// Returns the same sanitized errors as [`Self::resolve`], or the error `reserve`+ /// returns.pub async fn models(&self,gateway_key: &str,api_format: ApiFormat,request_id: &str,+ reserve: &mut (dyn FnMut(usize) -> Result<(), ResolutionError> + Send),) -> Result<Vec<u8>, ResolutionError> {let timestamp = SystemTime::now().duration_since(UNIX_EPOCH)@@ -168,7 +171,8 @@ impl ControlPlaneClient {timestamp.as_secs(),request_id,)?;- self.execute(request, self.config.models_limits).await+ self.execute(request, self.config.models_limits, reserve)+ .await}async fn resolve_at(@@ -195,7 +199,9 @@ impl ControlPlaneClient {)?.header(CONTENT_TYPE, "application/json").body(body);- let body = self.execute(request, self.config.resolve_limits).await?;+ let body = self+ .execute(request, self.config.resolve_limits, &mut |_| Ok(()))+ .await?;contracts::decode(&body,api_format,@@ -236,13 +242,14 @@ impl ControlPlaneClient {&self,request: RequestBuilder,limits: ControlPlaneLimits,+ reserve: &mut (dyn FnMut(usize) -> Result<(), ResolutionError> + Send),) -> Result<Vec<u8>, ResolutionError> {tokio::time::timeout(limits.timeout, async {let response = request.send().await.map_err(|_| ResolutionError::Transport)?;- read_response(response, limits.max_response_bytes).await+ read_response(response, limits.max_response_bytes, reserve).await}).await.map_err(|_| ResolutionError::Timeout)?@@ -252,6 +259,7 @@ impl ControlPlaneClient {async fn read_response(mut response: reqwest::Response,limit: usize,+ reserve: &mut (dyn FnMut(usize) -> Result<(), ResolutionError> + Send),) -> Result<Vec<u8>, ResolutionError> {match response.status().as_u16() {200 => {}@@ -277,12 +285,13 @@ async fn read_response(429 | 500..=599 => return Err(ResolutionError::Unavailable),_ => return Err(ResolutionError::InvalidResponse),}- if response+ let declared = response.content_length()- .is_some_and(|size| size > limit as u64)- {+ .map(|size| usize::try_from(size).unwrap_or(usize::MAX));+ if declared.is_some_and(|size| size > limit) {return Err(ResolutionError::ResponseTooLarge);}+ let capacity_limit = declared.unwrap_or(limit);let mut bytes = Vec::new();while let Some(chunk) = response.chunk()@@ -292,6 +301,15 @@ async fn read_response(if chunk.len() > limit - bytes.len() {return Err(ResolutionError::ResponseTooLarge);}+ let total = bytes.len() + chunk.len();+ if total > bytes.capacity() {+ let capacity = bytes+ .capacity()+ .saturating_mul(2)+ .clamp(total, capacity_limit.max(total));+ reserve(capacity)?;+ bytes.reserve_exact(capacity - bytes.len());+ }bytes.extend_from_slice(&chunk);}Ok(bytes)@@ -318,7 +336,7 @@ pub enum ResolutionError {ModelNotAllowed,#[error("no eligible connection")]NoRoute,- /// The gateway's own resolution or catalog capacity is exhausted.+ /// The gateway's memory budget cannot cover another request.#[error("gateway at capacity")]Busy,#[error("resolution unavailable")]diff --git a/ai-gateway/src/resolution/tests.rs b/ai-gateway/src/resolution/tests.rsindex e9dc39250e..5e77af0eea 100644--- a/ai-gateway/src/resolution/tests.rs+++ b/ai-gateway/src/resolution/tests.rs@@ -206,7 +206,12 @@ async fn fetches_models_from_the_control_plane() {let body = web.control_plane()- .models(GATEWAY_KEY, ApiFormat::OpenAiResponses, REQUEST_ID)+ .models(+ GATEWAY_KEY,+ ApiFormat::OpenAiResponses,+ REQUEST_ID,+ &mut |_| Ok(()),+ ).await.unwrap();assert_eq!(@@ -237,7 +242,12 @@ async fn model_catalogs_have_a_larger_response_limit_than_resolution() {assert!(matches!(resolution, Err(ResolutionError::ResponseTooLarge)));let models = control_plane- .models(GATEWAY_KEY, ApiFormat::OpenAiResponses, REQUEST_ID)+ .models(+ GATEWAY_KEY,+ ApiFormat::OpenAiResponses,+ REQUEST_ID,+ &mut |_| Ok(()),+ ).await.unwrap();assert_eq!(models.len(), 300 * 1024);diff --git a/ai-gateway/src/transport/mod.rs b/ai-gateway/src/transport/mod.rsindex c243839656..8d40f39b6c 100644--- a/ai-gateway/src/transport/mod.rs+++ b/ai-gateway/src/transport/mod.rs@@ -29,8 +29,6 @@ use tokio::{pub enum ProviderError {#[error("invalid provider configuration")]Configuration,- #[error("gateway execution capacity exhausted")]- Busy,#[error("provider deadline exceeded")]Timeout,#[error("provider transport failed")]@@ -158,7 +156,7 @@ pub(crate) fn relay<T: Send + 'static>(}})}),- provider,+ Some(provider),deadline,stall_timeout,owner,@@ -166,11 +164,29 @@ pub(crate) fn relay<T: Send + 'static>()}+/// Relay an already buffered body under the same deadlines, without provider telemetry.+pub(crate) fn relay_buffered<T: Send + 'static>(+ body: Vec<u8>,+ deadline: Instant,+ stall_timeout: Duration,+ owner: T,+) -> Body {+ relay_stream(+ futures_util::stream::once(async move { Ok(Bytes::from(body)) }),+ None,+ deadline,+ stall_timeout,+ owner,+ None,+ )+}+/// Relay until EOF, the overall `deadline`, or `stall_timeout` without progress/// in either direction: no upstream chunk, or a downstream that stopped reading.+/// Without a `provider`, the relay records no stream span or body size metric.fn relay_stream<S, T>(upstream: S,- provider: &'static str,+ provider: Option<&'static str>,deadline: Instant,stall_timeout: Duration,owner: T,@@ -183,15 +199,17 @@ wherelet (sender, receiver) = mpsc::channel(1);// Covers the response body after provider headers until the relay is// finalized, which the header-scoped client span cannot see.- let span = tracing::info_span!(- "provider.stream",- otel.kind = "internal",- otel.status_code = tracing::field::Empty,- error.type = tracing::field::Empty,- http.response.body.size = tracing::field::Empty,- gateway.chunks = tracing::field::Empty,- gateway.outcome = tracing::field::Empty,- );+ let span = provider.map_or_else(tracing::Span::none, |_| {+ tracing::info_span!(+ "provider.stream",+ otel.kind = "internal",+ otel.status_code = tracing::field::Empty,+ error.type = tracing::field::Empty,+ http.response.body.size = tracing::field::Empty,+ gateway.chunks = tracing::field::Empty,+ gateway.outcome = tracing::field::Empty,+ )+ });let resources = Arc::new(StreamResources {relay: Mutex::new(Some(Relay {provider,@@ -269,7 +287,7 @@ where/// Everything the winning finalizer releases at once: admission and trusted/// context, the capture, and the stream span, which ends when it is dropped here.struct Relay<T> {- provider: &'static str,+ provider: Option<&'static str>,owner: T,capture: Option<ExecutionCapture>,span: tracing::Span,@@ -308,10 +326,12 @@ impl<T> StreamResources<T> {}) = relay{self.record_outcome(&span, outcome);- crate::observability::response_body_size(- provider,- u64::try_from(self.bytes.load(Ordering::Relaxed)).unwrap_or(u64::MAX),- );+ if let Some(provider) = provider {+ crate::observability::response_body_size(+ provider,+ u64::try_from(self.bytes.load(Ordering::Relaxed)).unwrap_or(u64::MAX),+ );+ }drop(span);drop(owner);if let Some(mut capture) = capture {@@ -404,7 +424,7 @@ mod tests {for first in [RelayOutcome::Eof, RelayOutcome::Timeout] {let resources = StreamResources {relay: Mutex::new(Some(Relay {- provider: "test",+ provider: Some("test"),owner: (),capture: None,span: tracing::Span::none(),@@ -430,7 +450,7 @@ mod tests {fn a_panic_while_updating_the_capture_fails_the_relay() {let resources = StreamResources {relay: Mutex::new(Some(Relay {- provider: "test",+ provider: Some("test"),owner: (),capture: None,span: tracing::Span::none(),@@ -478,7 +498,7 @@ mod tests {Ok(Bytes::from_static(b"partial")),Err(ProviderError::Transport),]),- "test",+ Some("test"),Instant::now() + Duration::from_secs(1),Duration::from_secs(10),Owner(released.clone()),@@ -520,7 +540,7 @@ mod tests {let released = Arc::new(Notify::new());let body = relay_stream(upstream,- "test",+ Some("test"),Instant::now() + Duration::from_secs(10),Duration::from_secs(10),Owner(released.clone()),@@ -540,7 +560,7 @@ mod tests {let bytes = Bytes::from_static(b"event: response.output_text.delta\ndata: hello\n\n");let body = relay_stream(stream::iter([Ok(bytes.clone()), Err(ProviderError::Transport)]),- "test",+ Some("test"),Instant::now() + Duration::from_secs(1),Duration::from_secs(10),Owner(released.clone()),@@ -561,7 +581,7 @@ mod tests {let released = Arc::new(Notify::new());let body = relay_stream(stream::once(async { Ok(Bytes::from_static(b"final bytes")) }),- "test",+ Some("test"),Instant::now() + Duration::from_millis(30),Duration::from_secs(10),Owner(released.clone()),@@ -579,7 +599,7 @@ mod tests {let started = Instant::now();let body = relay_stream(stream::once(async { Ok(Bytes::from_static(b"first")) }).chain(stream::pending()),- "test",+ Some("test"),started + Duration::from_secs(3600),Duration::from_secs(120),Owner(released.clone()),@@ -604,7 +624,7 @@ mod tests {});let _body = relay_stream(upstream,- "test",+ Some("test"),started + Duration::from_secs(3600),Duration::from_secs(120),Owner(released.clone()),diff --git a/ai-gateway/tests/startup.rs b/ai-gateway/tests/startup.rsindex 813e0429cd..e48c054775 100644--- a/ai-gateway/tests/startup.rs+++ b/ai-gateway/tests/startup.rs@@ -10,10 +10,9 @@ fn invalid_configuration_exits_without_disclosing_the_value() {"LANGFUSE_LOG_LEVEL","LANGFUSE_LOG_FORMAT","LANGFUSE_AI_GATEWAY_WEB_URL",- "LANGFUSE_AI_GATEWAY_MAX_ACTIVE_REQUESTS",- "LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS","LANGFUSE_AI_GATEWAY_MAX_REQUEST_BYTES",- "LANGFUSE_AI_GATEWAY_SHARED_REQUEST_BUFFER_BYTES",+ "LANGFUSE_AI_GATEWAY_MEMORY_BUDGET_BYTES",+ "LANGFUSE_AI_GATEWAY_REQUEST_MEMORY_OVERHEAD_BYTES","LANGFUSE_AI_GATEWAY_MAX_EXECUTION_SECONDS","LANGFUSE_AI_GATEWAY_STALL_TIMEOUT_SECONDS",] {