🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
fix(ai-gateway): give telemetry flush its own shutdown budget (#19082)
Tobias Wochinger12191aae2b81
diff --git a/ai-gateway/README.md b/ai-gateway/README.mdindex b72875acf1..b0e5d296d6 100644--- a/ai-gateway/README.md+++ b/ai-gateway/README.md@@ -78,7 +78,8 @@ do not load dotenv files:| `LANGFUSE_AI_GATEWAY_AUTO_INCREMENT_LISTEN_PORT` | `false` | `true`, `false`; the pnpm dev task sets `true` || `LANGFUSE_LOG_FORMAT` | `text` | `text`, `json` || `LANGFUSE_LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error`, `fatal` |-| `LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS` | `10` | Integer from 1 to 300 |+| `LANGFUSE_AI_GATEWAY_CONNECTION_DRAIN_TIMEOUT_SECONDS` | `10` | Integer from 1 to 300; connection drain after the shutdown signal. Replaces the deprecated `LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS`, which is still read when the new name is unset. On ECS behind an ALB, the deregistration delay drains streams before SIGTERM, so use `1`. On Kubernetes the gateway drains streams itself, so set it to the stream time to protect (for example `300`) |+| `LANGFUSE_AI_GATEWAY_TELEMETRY_SHUTDOWN_TIMEOUT_SECONDS` | `10` | Integer from 1 to 300; telemetry flush after the drain ends. On ECS use most of `stopTimeout` (max 120), for example `115`. On Kubernetes `30` is enough; set `terminationGracePeriodSeconds` above the `preStop` sleep plus both timeouts || `LANGFUSE_AI_GATEWAY_TELEMETRY_BUFFER_BYTES` | `67108864` (64 MiB) | Integer from 4194304 (4 MiB) to 4294967296 (4 GiB); span and credential bytes an instance holds for upload before dropping new records || `LANGFUSE_AI_GATEWAY_MAX_REQUEST_BYTES` | `16777216` (16 MiB) | Integer from 1048576 (1 MiB) to 4294967295; largest accepted request body || `LANGFUSE_AI_GATEWAY_MEMORY_BUDGET_BYTES` | `1073741824` (1 GiB) | Integer from the request limit plus the overhead up to Tokio's semaphore capacity; bytes all in-flight requests and model catalogs of an instance may reserve together |@@ -259,7 +260,7 @@ bounded categories, not tenant or request identifiers.Trace export uses a 512-span queue, batches of up to 64, and a one-second interval.Metrics export every 30 seconds. Export calls use a three-second timeout; failures-do not fail inference. Shutdown flushes within the remaining shared drain deadline.+do not fail inference. Shutdown flushes within the remaining telemetry shutdown budget.Telemetry delivery warnings report the first failure/drop and every hundredth;metrics count each occurrence. This is best-effort operational telemetry.@@ -547,7 +548,7 @@ span records `gateway.telemetry.attempts`. Sanitized logsrecord failures/drops; shutdown reports accepted, failed and dropped counts.SIGTERM marks the gateway unready, drains inference, then flushes open batches and-finishes uploads within the remaining shared shutdown budget. At the deadline, unfinished uploads,+finishes uploads within the telemetry shutdown budget, which starts when draining ends. At the deadline, unfinished uploads,including batches waiting to retry, are cancelled and counted as drops. Process crashes or forced shutdown can lose telemetry;these records are not a durable accounting ledger.@@ -814,8 +815,8 @@ key metadata remain in Langfuse ingestion and are excluded from operational logs- SIGTERM or Ctrl-C starts graceful shutdown. In-flight requests may finishwithin the configured timeout. Deadline expiry exits nonzero and terminatesremaining runtime tasks; ordinary shutdown exits zero.-- The shutdown deadline starts on the shutdown signal. New inference requests- are rejected while draining; existing response streams share this budget.+- The drain deadline (`LANGFUSE_AI_GATEWAY_CONNECTION_DRAIN_TIMEOUT_SECONDS`) starts on the shutdown signal. New inference requests are rejected while draining; existing response streams share this budget.+- Inference telemetry and operational OTLP export then flush within `LANGFUSE_AI_GATEWAY_TELEMETRY_SHUTDOWN_TIMEOUT_SECONDS`, starting when draining ends, so a drain that hits its deadline does not drop open batches.## Container@@ -836,7 +837,7 @@ docker build --target runtime \docker run --rm --name langfuse-ai-gateway-dev \-e LANGFUSE_LOG_FORMAT=json -p 127.0.0.1:8080:8080 langfuse-ai-gateway:dev# In another terminal:-docker stop --time 15 langfuse-ai-gateway-dev+docker stop --time 25 langfuse-ai-gateway-devbash ai-gateway/scripts/smoke-image.sh langfuse-ai-gateway:dev```@@ -846,7 +847,7 @@ package version, or override `BUILD_ID` at runtime.The runtime image runs as UID/GID 10001, includes CA certificates, and executesthe binary directly so it receives signals. Set the container/orchestrator stop-grace period longer than the gateway shutdown timeout. The smoke test uses an+grace period longer than the gateway connection drain timeout plus the telemetry shutdown timeout. The smoke test uses anisolated container and an ephemeral host port, checks read-only/non-root startup,probes, rejection of unauthenticated inference and a clean SIGTERM exit, then removes it.diff --git a/ai-gateway/scripts/smoke-image.sh b/ai-gateway/scripts/smoke-image.shindex 3eb51ae0d0..b29acc7e2e 100644--- a/ai-gateway/scripts/smoke-image.sh+++ b/ai-gateway/scripts/smoke-image.sh@@ -3,7 +3,7 @@ set -euo pipefailimage="${1:?Usage: bash scripts/smoke-image.sh IMAGE}"container="$(docker run -d --read-only --cap-drop ALL --security-opt no-new-privileges \- -e LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS=2 \+ -e LANGFUSE_AI_GATEWAY_CONNECTION_DRAIN_TIMEOUT_SECONDS=2 \-e LANGFUSE_AI_GATEWAY_WEB_URL=https://web.example \-e LANGFUSE_AI_GATEWAY_SERVICE_KEY=smoke-test-service-key -p 127.0.0.1::8080 "$image")"cleanup() {diff --git a/ai-gateway/src/config.rs b/ai-gateway/src/config.rsindex 11b1750eb9..879d90791b 100644--- a/ai-gateway/src/config.rs+++ b/ai-gateway/src/config.rs@@ -6,7 +6,14 @@ pub struct GatewayConfig {pub control_plane: Option<ControlPlaneConfig>,pub listen_address: SocketAddr,pub auto_increment_listen_port: bool,- pub shutdown_timeout: Duration,+ /// Connection drain after the shutdown signal. Behind an ALB, deregistration has+ /// already drained streams on SIGTERM; on Kubernetes this drain is the only one.+ pub connection_drain_timeout: Duration,+ /// The connection drain timeout was read from its deprecated name.+ pub uses_deprecated_connection_drain_timeout_name: bool,+ /// Budget for flushing telemetry, starting once connection draining ends. The+ /// orchestrator grace period must cover both timeouts (ECS `stopTimeout` is at most 120 s).+ pub telemetry_shutdown_timeout: Duration,pub log_level: LevelFilter,pub log_format: LogFormat,pub telemetry_buffer_bytes: usize,@@ -24,6 +31,7 @@ const DEFAULT_MEMORY_BUDGET_BYTES: usize = 1024 * MIB;const DEFAULT_REQUEST_OVERHEAD_BYTES: usize = 2 * MIB;const DEFAULT_MAX_EXECUTION_SECONDS: u32 = 10 * 60;const DEFAULT_STALL_TIMEOUT_SECONDS: u32 = 120;+const DEFAULT_SHUTDOWN_TIMEOUT_SECONDS: u64 = 10;/// Memory bounds per instance. Admission is limited only by the memory budget.#[derive(Clone, Copy, Debug, PartialEq, Eq)]pub struct RequestLimits {@@ -81,16 +89,26 @@ impl GatewayConfig {/// or missing/blank service key, or a non-boolean HTTP Web URL opt-in. Error/// messages never include supplied values.pub fn from_env() -> Result<Self, GatewayConfigError> {+ let (connection_drain_timeout, uses_deprecated_connection_drain_timeout_name) =+ connection_drain_timeout_value(+ read_env("LANGFUSE_AI_GATEWAY_CONNECTION_DRAIN_TIMEOUT_SECONDS")?,+ read_env("LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS")?,+ );let mut config = Self::from_values(read_env("LANGFUSE_AI_GATEWAY_LISTEN_ADDRESS")?.as_deref(),read_env("LANGFUSE_AI_GATEWAY_AUTO_INCREMENT_LISTEN_PORT")?.as_deref(),- read_env("LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS")?.as_deref(),+ connection_drain_timeout.as_deref(),read_env("LANGFUSE_LOG_LEVEL")?.as_deref(),read_env("LANGFUSE_LOG_FORMAT")?.as_deref(),)?;+ config.uses_deprecated_connection_drain_timeout_name =+ uses_deprecated_connection_drain_timeout_name;config.telemetry_buffer_bytes = telemetry_buffer_bytes(read_env("LANGFUSE_AI_GATEWAY_TELEMETRY_BUFFER_BYTES")?.as_deref(),)?;+ config.telemetry_shutdown_timeout = telemetry_shutdown_timeout(+ read_env("LANGFUSE_AI_GATEWAY_TELEMETRY_SHUTDOWN_TIMEOUT_SECONDS")?.as_deref(),+ )?;config.request_limits = request_limits(read_env("LANGFUSE_AI_GATEWAY_MAX_REQUEST_BYTES")?.as_deref(),read_env("LANGFUSE_AI_GATEWAY_MEMORY_BUDGET_BYTES")?.as_deref(),@@ -112,12 +130,12 @@ impl GatewayConfig {////// # Errors/// Returns an error for an invalid IP address and port, a non-boolean port- /// auto-increment setting, a shutdown timeout outside 1–300 integer seconds,+ /// auto-increment setting, a connection drain timeout outside 1–300 integer seconds,/// or an unsupported log level or format.pub fn from_values(listen_address: Option<&str>,auto_increment_listen_port: Option<&str>,- shutdown_timeout: Option<&str>,+ connection_drain_timeout: Option<&str>,log_level: Option<&str>,log_format: Option<&str>,) -> Result<Self, GatewayConfigError> {@@ -136,10 +154,10 @@ impl GatewayConfig {"LANGFUSE_AI_GATEWAY_AUTO_INCREMENT_LISTEN_PORT must be true or false",))?;let seconds = integer_setting(- shutdown_timeout,- 10,+ connection_drain_timeout,+ DEFAULT_SHUTDOWN_TIMEOUT_SECONDS,1..=300,- "LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS must be an integer from 1 to 300",+ "LANGFUSE_AI_GATEWAY_CONNECTION_DRAIN_TIMEOUT_SECONDS (or the deprecated LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS) must be an integer from 1 to 300",)?;let log_level = match log_level.unwrap_or("info") {"debug" => LevelFilter::DEBUG,@@ -166,7 +184,9 @@ impl GatewayConfig {control_plane: None,listen_address,auto_increment_listen_port,- shutdown_timeout: Duration::from_secs(seconds),+ connection_drain_timeout: Duration::from_secs(seconds),+ uses_deprecated_connection_drain_timeout_name: false,+ telemetry_shutdown_timeout: Duration::from_secs(DEFAULT_SHUTDOWN_TIMEOUT_SECONDS),log_level,log_format,telemetry_buffer_bytes: crate::telemetry::DEFAULT_RETAINED_BYTES,@@ -269,6 +289,28 @@ fn telemetry_buffer_bytes(value: Option<&str>) -> Result<usize, GatewayConfigErr)}+/// The current name wins; the deprecated name keeps released deployments working.+fn connection_drain_timeout_value(+ current: Option<String>,+ deprecated: Option<String>,+) -> (Option<String>, bool) {+ match (current, deprecated) {+ (Some(value), _) => (Some(value), false),+ (None, Some(value)) => (Some(value), true),+ (None, None) => (None, false),+ }+}++fn telemetry_shutdown_timeout(value: Option<&str>) -> Result<Duration, GatewayConfigError> {+ integer_setting(+ value,+ DEFAULT_SHUTDOWN_TIMEOUT_SECONDS,+ 1..=300,+ "LANGFUSE_AI_GATEWAY_TELEMETRY_SHUTDOWN_TIMEOUT_SECONDS must be an integer from 1 to 300",+ )+ .map(Duration::from_secs)+}+/// An unset value takes `default`. Errors carry only `message`, never the supplied value.fn integer_setting<T: std::str::FromStr + PartialOrd>(value: Option<&str>,@@ -309,7 +351,7 @@ mod tests {let default = GatewayConfig::from_values(None, None, None, None, None).unwrap();assert_eq!(default.listen_address, "0.0.0.0:8080".parse().unwrap());assert!(!default.auto_increment_listen_port);- assert_eq!(default.shutdown_timeout, Duration::from_secs(10));+ assert_eq!(default.connection_drain_timeout, Duration::from_secs(10));assert_eq!(default.log_level, LevelFilter::INFO);assert_eq!(default.log_format, LogFormat::Text);assert_eq!(default.telemetry_buffer_bytes, 64 * 1024 * 1024);@@ -323,7 +365,7 @@ mod tests {.unwrap();assert_eq!(custom.listen_address, "[::1]:9000".parse().unwrap());assert!(custom.auto_increment_listen_port);- assert_eq!(custom.shutdown_timeout, Duration::from_secs(30));+ assert_eq!(custom.connection_drain_timeout, Duration::from_secs(30));assert_eq!(custom.log_level, LevelFilter::DEBUG);assert_eq!(custom.log_format, LogFormat::Json);}@@ -491,4 +533,49 @@ mod tests {);}}++ #[test]+ fn connection_drain_timeout_falls_back_to_its_deprecated_name() {+ let value = |text: &str| Some(text.to_owned());+ assert_eq!(connection_drain_timeout_value(None, None), (None, false));+ assert_eq!(+ connection_drain_timeout_value(value("5"), None),+ (value("5"), false)+ );+ assert_eq!(+ connection_drain_timeout_value(None, value("7")),+ (value("7"), true)+ );+ assert_eq!(+ connection_drain_timeout_value(value("5"), value("7")),+ (value("5"), false)+ );+ }++ #[test]+ fn telemetry_shutdown_timeout_is_read_in_seconds_within_bounds() {+ assert_eq!(+ GatewayConfig::from_values(None, None, None, None, None)+ .unwrap()+ .telemetry_shutdown_timeout,+ Duration::from_secs(10)+ );+ assert_eq!(+ telemetry_shutdown_timeout(None).unwrap(),+ Duration::from_secs(10)+ );+ for value in [1, 120, 300] {+ assert_eq!(+ telemetry_shutdown_timeout(Some(&value.to_string())).unwrap(),+ Duration::from_secs(value)+ );+ }+ for value in ["", "0", "301", "-1", "1.5", "secret-that-must-not-appear"] {+ let error = telemetry_shutdown_timeout(Some(value)).err().unwrap();+ assert_eq!(+ error.to_string(),+ "LANGFUSE_AI_GATEWAY_TELEMETRY_SHUTDOWN_TIMEOUT_SECONDS must be an integer from 1 to 300"+ );+ }+ }}diff --git a/ai-gateway/src/main.rs b/ai-gateway/src/main.rsindex d83caa9b93..f0aed9e1b0 100644--- a/ai-gateway/src/main.rs+++ b/ai-gateway/src/main.rs@@ -28,6 +28,11 @@ async fn main() -> ExitCode {async fn run() -> Result<(), Box<dyn Error>> {let mut config = GatewayConfig::from_env()?;let observability = observability::init(&config)?;+ if config.uses_deprecated_connection_drain_timeout_name {+ tracing::warn!(+ "LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS is deprecated; set LANGFUSE_AI_GATEWAY_CONNECTION_DRAIN_TIMEOUT_SECONDS instead"+ );+ }if config.control_plane.as_ref()@@ -55,36 +60,26 @@ async fn run() -> Result<(), Box<dyn Error>> {)));let shutdown = shutdown_signal()?;let listener = bind_listener(config.listen_address, config.auto_increment_listen_port).await?;- tracing::info!(address = %listener.local_addr()?, inference_enabled, max_request_bytes = config.request_limits.max_request_bytes, memory_budget_bytes = config.request_limits.memory_budget_bytes, request_memory_overhead_bytes = config.request_limits.request_overhead_bytes, max_execution_seconds = config.execution_limits.max_execution_time.as_secs(), stall_timeout_seconds = config.execution_limits.stall_timeout.as_secs(), shutdown_timeout_seconds = config.shutdown_timeout.as_secs(), "gateway listening");+ tracing::info!(address = %listener.local_addr()?, inference_enabled, max_request_bytes = config.request_limits.max_request_bytes, memory_budget_bytes = config.request_limits.memory_budget_bytes, request_memory_overhead_bytes = config.request_limits.request_overhead_bytes, max_execution_seconds = config.execution_limits.max_execution_time.as_secs(), stall_timeout_seconds = config.execution_limits.stall_timeout.as_secs(), connection_drain_timeout_seconds = config.connection_drain_timeout.as_secs(), telemetry_shutdown_timeout_seconds = config.telemetry_shutdown_timeout.as_secs(), "gateway listening");let result = server::serve(listener,app,state.clone(),shutdown,- config.shutdown_timeout,+ config.connection_drain_timeout,).await;+ // Starts after draining so a drain that hits its deadline cannot consume the flush.+ let flush_deadline = tokio::time::Instant::now() + config.telemetry_shutdown_timeout;if let Some(telemetry) = telemetry {- telemetry- .shutdown(- state- .drain_deadline()- .unwrap_or_else(tokio::time::Instant::now),- )- .await;+ telemetry.shutdown(flush_deadline).await;}if result.is_err() {tracing::error!("gateway shutdown failed");} else {tracing::info!("gateway stopped");}- observability- .shutdown(- state- .drain_deadline()- .unwrap_or_else(tokio::time::Instant::now),- )- .await;+ observability.shutdown(flush_deadline).await;result.map_err(Into::into)}diff --git a/ai-gateway/src/server.rs b/ai-gateway/src/server.rsindex ab7aa6c61f..b18dd64e4d 100644--- a/ai-gateway/src/server.rs+++ b/ai-gateway/src/server.rs@@ -2,7 +2,7 @@ use std::{future::{Future, IntoFuture},io,sync::{- Arc, Mutex, PoisonError,+ Arc,atomic::{AtomicBool, Ordering},},time::Duration,@@ -20,18 +20,9 @@ pub(crate) use connection::ConnectionId;pub struct GatewayLifecycleState {ready: Arc<AtomicBool>,disabled: bool,- drain_deadline: Arc<Mutex<Option<tokio::time::Instant>>>,}impl GatewayLifecycleState {- /// Deadline shared by connection draining and telemetry delivery.- pub fn drain_deadline(&self) -> Option<tokio::time::Instant> {- *self- .drain_deadline- .lock()- .unwrap_or_else(PoisonError::into_inner)- }-pub fn is_ready(&self) -> bool {!self.disabled && self.ready.load(Ordering::Acquire)}@@ -79,13 +70,13 @@ async fn readiness(State(state): State<GatewayLifecycleState>) -> (StatusCode, J////// # Errors/// Returns the server's I/O error, or [`io::ErrorKind::TimedOut`] if connections-/// do not finish draining within `shutdown_timeout`.+/// do not finish draining within `connection_drain_timeout`.pub async fn serve(listener: TcpListener,app: Router,state: GatewayLifecycleState,shutdown: impl Future<Output = ()> + Send + 'static,- shutdown_timeout: Duration,+ connection_drain_timeout: Duration,) -> io::Result<()> {let (draining_tx, draining_rx) = oneshot::channel();let shutdown_state = state.clone();@@ -96,11 +87,6 @@ pub async fn serve(.with_graceful_shutdown(async move {shutdown.await;shutdown_state.ready.store(false, Ordering::Release);- *shutdown_state- .drain_deadline- .lock()- .unwrap_or_else(PoisonError::into_inner) =- Some(tokio::time::Instant::now() + shutdown_timeout);tracing::info!("gateway draining");let _ = draining_tx.send(());})@@ -111,8 +97,8 @@ pub async fn serve(result = &mut server => result,() = async {let _ = draining_rx.await;- tokio::time::sleep(shutdown_timeout).await;- } => Err(io::Error::new(io::ErrorKind::TimedOut, "gateway shutdown deadline exceeded")),+ tokio::time::sleep(connection_drain_timeout).await;+ } => Err(io::Error::new(io::ErrorKind::TimedOut, "gateway drain deadline exceeded")),};state.ready.store(false, Ordering::Release);resultdiff --git a/ai-gateway/src/telemetry/mod.rs b/ai-gateway/src/telemetry/mod.rsindex 8e3b75a4c0..6572064990 100644--- a/ai-gateway/src/telemetry/mod.rs+++ b/ai-gateway/src/telemetry/mod.rs@@ -233,8 +233,7 @@ impl Telemetry {}}- /// Stop accepting work, flush open batches, and finish uploads within the existing- /// process drain deadline.+ /// Stop accepting work, flush open batches, and finish uploads before `deadline`.pub async fn shutdown(&self, deadline: Instant) {let worker = self.0diff --git a/ai-gateway/tests/observability.rs b/ai-gateway/tests/observability.rsindex 233249f876..7b17806633 100644--- a/ai-gateway/tests/observability.rs+++ b/ai-gateway/tests/observability.rs@@ -13,17 +13,17 @@ use prost::Message;use serde_json::Value;use std::{io::{BufRead, BufReader},- process::{Child, Command, Stdio},+ process::{Child, Command, ExitStatus, Stdio},sync::{Arc, Mutex, mpsc},- thread::JoinHandle,time::{Duration, Instant},};+use tokio::{io::AsyncWriteExt, task::JoinHandle};struct Gateway {child: Child,url: String,logs: Arc<Mutex<Vec<Value>>>,- reader: Option<JoinHandle<()>>,+ reader: Option<std::thread::JoinHandle<()>>,}impl Gateway {@@ -32,13 +32,22 @@ impl Gateway {clippy::expect_used,reason = "test helper outside #[test]; a panic fails the test")]- fn start(endpoint: &str, shutdown_seconds: &str, level: &str) -> Self {+ fn start(+ endpoint: &str,+ drain_seconds: &str,+ telemetry_shutdown_seconds: &str,+ level: &str,+ ) -> Self {let mut child = Command::new(env!("CARGO_BIN_EXE_ai-gateway")).env_clear().env("LANGFUSE_AI_GATEWAY_LISTEN_ADDRESS", "127.0.0.1:0").env(- "LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS",- shutdown_seconds,+ "LANGFUSE_AI_GATEWAY_CONNECTION_DRAIN_TIMEOUT_SECONDS",+ drain_seconds,+ )+ .env(+ "LANGFUSE_AI_GATEWAY_TELEMETRY_SHUTDOWN_TIMEOUT_SECONDS",+ telemetry_shutdown_seconds,).env("LANGFUSE_LOG_FORMAT", "json").env("LANGFUSE_LOG_LEVEL", level)@@ -117,7 +126,7 @@ impl Gateway {clippy::expect_used,reason = "test helper outside #[test]; a panic fails the test")]- async fn stop(&mut self) {+ async fn terminate(&mut self) -> ExitStatus {assert!(Command::new("/bin/kill").args(["-TERM", &self.child.id().to_string()])@@ -125,11 +134,10 @@ impl Gateway {.unwrap().success());- tokio::time::timeout(Duration::from_secs(6), async {+ let status = tokio::time::timeout(Duration::from_secs(6), async {loop {if let Some(status) = self.child.try_wait().unwrap() {- assert!(status.success());- break;+ break status;}tokio::time::sleep(Duration::from_millis(10)).await;}@@ -137,18 +145,20 @@ impl Gateway {.await.expect("gateway shutdown deadline");self.reader.take().unwrap().join().unwrap();+ status}-}-impl Drop for Gateway {- fn drop(&mut self) {- let _ = self.child.kill();- let _ = self.child.wait();+ async fn stop(&mut self) {+ assert!(self.terminate().await.success());}}-#[tokio::test]-async fn exports_otlp_traces_metrics_and_correlated_content_free_logs() {+/// Record every OTLP export request as `(path, body)`.+#[expect(+ clippy::unwrap_used,+ reason = "test helper outside #[test]; a panic fails the test"+)]+async fn start_collector() -> (String, Arc<Mutex<Vec<(String, Bytes)>>>, JoinHandle<()>) {let captured = Arc::new(Mutex::new(Vec::new()));let receiver = captured.clone();let app = Router::new().route(@@ -168,7 +178,20 @@ async fn exports_otlp_traces_metrics_and_correlated_content_free_logs() {let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();let endpoint = format!("http://{}", listener.local_addr().unwrap());let server = tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });- let mut gateway = Gateway::start(&endpoint, "5", "debug");+ (endpoint, captured, server)+}++impl Drop for Gateway {+ fn drop(&mut self) {+ let _ = self.child.kill();+ let _ = self.child.wait();+ }+}++#[tokio::test]+async fn exports_otlp_traces_metrics_and_correlated_content_free_logs() {+ let (endpoint, captured, server) = start_collector().await;+ let mut gateway = Gateway::start(&endpoint, "5", "5", "debug");let request_ids = [gateway.request(true).await, gateway.request(false).await];assert_ne!(request_ids[0], request_ids[1]);assert_eq!(@@ -391,10 +414,37 @@ async fn an_unresponsive_collector_does_not_extend_the_shutdown_deadline() {let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();let endpoint = format!("http://{}", listener.local_addr().unwrap());let server = tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });- let mut gateway = Gateway::start(&endpoint, "1", "info");+ let mut gateway = Gateway::start(&endpoint, "1", "1", "info");gateway.request(true).await;let start = Instant::now();gateway.stop().await;assert!(start.elapsed() < Duration::from_secs(2));server.abort();}++#[tokio::test]+async fn a_drain_timeout_leaves_the_telemetry_flush_its_own_budget() {+ let (endpoint, captured, server) = start_collector().await;+ let mut gateway = Gateway::start(&endpoint, "1", "3", "info");+ gateway.request(true).await;+ // A request whose headers never complete keeps its connection open past the drain deadline.+ let mut stalled = tokio::net::TcpStream::connect(gateway.url.trim_start_matches("http://"))+ .await+ .unwrap();+ stalled+ .write_all(b"POST /openai/v1/responses HTTP/1.1\r\nhost: gateway\r\n")+ .await+ .unwrap();+ let start = Instant::now();+ assert!(!gateway.terminate().await.success());+ assert!(start.elapsed() >= Duration::from_secs(1));+ server.abort();+ assert!(+ captured+ .lock()+ .unwrap()+ .iter()+ .any(|(path, _)| path == "/v1/metrics"),+ "metrics export only on shutdown, so they must flush after the drain timed out"+ );+}diff --git a/ai-gateway/tests/startup.rs b/ai-gateway/tests/startup.rsindex e48c054775..b2a559ccbf 100644--- a/ai-gateway/tests/startup.rs+++ b/ai-gateway/tests/startup.rs@@ -6,7 +6,9 @@ fn invalid_configuration_exits_without_disclosing_the_value() {let listener = TcpListener::bind("127.0.0.1:0").unwrap();for name in ["LANGFUSE_AI_GATEWAY_AUTO_INCREMENT_LISTEN_PORT",+ "LANGFUSE_AI_GATEWAY_CONNECTION_DRAIN_TIMEOUT_SECONDS","LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS",+ "LANGFUSE_AI_GATEWAY_TELEMETRY_SHUTDOWN_TIMEOUT_SECONDS","LANGFUSE_LOG_LEVEL","LANGFUSE_LOG_FORMAT","LANGFUSE_AI_GATEWAY_WEB_URL",diff --git a/web/src/__e2e__/ai-gateway.gatewaye2e.ts b/web/src/__e2e__/ai-gateway.gatewaye2e.tsindex 25255c632e..829639167a 100644--- a/web/src/__e2e__/ai-gateway.gatewaye2e.ts+++ b/web/src/__e2e__/ai-gateway.gatewaye2e.ts@@ -1199,7 +1199,7 @@ async function startGatewayProcess() {LANGFUSE_AI_GATEWAY_SERVICE_KEY: env.LANGFUSE_AI_GATEWAY_SERVICE_KEY,LANGFUSE_AI_GATEWAY_LISTEN_ADDRESS: `127.0.0.1:${port}`,LANGFUSE_AI_GATEWAY_AUTO_INCREMENT_LISTEN_PORT: "false",- LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS: "5",+ LANGFUSE_AI_GATEWAY_CONNECTION_DRAIN_TIMEOUT_SECONDS: "5",LANGFUSE_LOG_FORMAT: "json",LANGFUSE_LOG_LEVEL: "info",},