:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(flags): drain on the pre-stop file before shutting down (#125490)
Matheus Batista9b61c96a130e
diff --git a/rust/Cargo.lock b/rust/Cargo.lockindex 7b6790256d6c..7391b08a2c17 100644--- a/rust/Cargo.lock+++ b/rust/Cargo.lock@@ -4791,6 +4791,7 @@ dependencies = ["sqlx","strum 0.28.0","subtle",+ "tempfile","test-case","thiserror 2.0.18","tokio",@@ -7501,6 +7502,7 @@ dependencies = ["axum 0.7.9","common-liveness","metrics",+ "tempfile","thiserror 2.0.18","tokio","tokio-util",diff --git a/rust/common/lifecycle/Cargo.toml b/rust/common/lifecycle/Cargo.tomlindex 86d63fb4faed..13ed25dd1159 100644--- a/rust/common/lifecycle/Cargo.toml+++ b/rust/common/lifecycle/Cargo.toml@@ -14,3 +14,6 @@ thiserror = { workspace = true }tokio = { workspace = true }tokio-util = { workspace = true }tracing = { workspace = true }++[dev-dependencies]+tempfile = "3"diff --git a/rust/common/lifecycle/README.md b/rust/common/lifecycle/README.mdindex 6259bb211fad..38fe04e79c1a 100644--- a/rust/common/lifecycle/README.md+++ b/rust/common/lifecycle/README.md@@ -105,6 +105,7 @@ All options except `name` have sensible defaults.| `.with_global_shutdown_timeout(duration)` | Hard ceiling on total shutdown. Monitor returns `ShutdownTimeout` if exceeded. Prevents indefinite hangs if a component doesn't check for cancellation. (see test `global_timeout_fires_when_component_hangs`) | `60s` || `.with_trap_signals(bool)` | Install SIGINT/SIGTERM handlers. Set `false` in tests. | `true` || `.with_prestop_check(bool)` | Poll for pre-stop shutdown file (K8s pre-stop hook pattern). | `true` |+| `.with_prestop_action(action)` | What the pre-stop file does. `PrestopAction::Shutdown` starts graceful shutdown. `PrestopAction::Drain { max }` fails readiness and keeps serving until SIGTERM, for at most `max`. Turns the pre-stop check on. See [Draining before shutdown](#draining-before-shutdown). (see test `prestop_drain_serves_until_max`) | `Shutdown` || `.with_prestop_path(path)` | Override the pre-stop file path. | `/tmp/shutdown` || `.with_health_poll_interval(duration)` | Override health monitor poll frequency. The health monitor is automatically active when any component has `with_liveness_deadline`. (see test `stall_triggers_shutdown`) | `5s` || `.with_shutdown_token(token)` | Caller supplies an external `CancellationToken`. Use in tests to trigger deterministic, app-global shutdown (simulate `SIGTERM` from k8s etc.) | `None` |@@ -125,7 +126,7 @@ All options except `name` have sensible defaults.## K8s readiness and liveness-**Readiness** (`/_readiness`) returns 200 until shutdown begins, then 503. K8s uses this to stop routing traffic to the pod. No per-component logic — it's purely "is the app accepting work?" (see test `readiness_200_until_shutdown_then_503`)+**Readiness** (`/_readiness`) returns 200 until the app drains or shutdown begins, then 503. K8s uses this to stop routing traffic to the pod. No per-component logic — it's purely "is the app accepting work?" (see test `readiness_200_until_shutdown_then_503`)**Liveness** (`/_liveness`) always returns 200 — it means "the process is reachable." Health monitoring is handled internally by the manager's health poll task, not by K8s. When a component's heartbeat deadline expires, the health monitor increments a stall counter. After `stall_threshold` consecutive stalled checks, the manager triggers global shutdown via the same `ComponentEvent::Failure` path as `signal_failure()`. This ensures the app always gets coordinated graceful shutdown instead of K8s surprise-killing the pod. (see tests `stall_triggers_shutdown`, `stall_threshold_exceeded_triggers_shutdown`, `stall_threshold_allows_recovery`)@@ -162,6 +163,30 @@ metrics_handle.work_completed();guard.wait().await?;```+### Draining before shutdown++By default the pre-stop file starts graceful shutdown at once, the same as SIGTERM.+A server can close connections that clients still use, because clients do not see the pod leave until K8s removes it from the Service endpoints.+A client behind a proxy that reaches the pod through one ClusterIP never sees the pod leave, and keeps its pooled connections on it.++`.with_prestop_action(PrestopAction::Drain { max })` turns the pre-stop window into a drain:++- The pre-stop file fails readiness, and components keep serving. The manager logs `Lifecycle: drain started, prestop file detected`.+- SIGTERM, SIGINT, a component failure or a liveness stall starts shutdown as usual. If none of them comes within `max` of the drain start, the manager starts shutdown itself and logs `Lifecycle: shutdown initiated, drain limit reached`.+- Set the pod's preStop sleep to `max`, because kubelet sends SIGTERM only after the sleep ends.+- Set the pod's `terminationGracePeriodSeconds` above `max` plus `with_global_shutdown_timeout`.++```rust+let mut manager = Manager::builder("my-service")+ .with_prestop_action(PrestopAction::Drain { max: Duration::from_secs(20) })+ .build();+```++An HTTP/1.1 server can send `Connection: close` on each response while `ReadinessHandler::is_ready()` returns false.+Each client that sends a request during the drain then opens its next connection through the Service to a pod that stays.+A pooled connection that carries no request during the drain still closes at shutdown, so clients must still retry a request that fails on a reused connection.+See `close_connection_unless_ready` in `rust/feature-flags/src/router.rs` (see test `drain_closes_keep_alive_connections_after_the_response`).+## Using the handle### With `process_scope()` (struct-held handle)diff --git a/rust/common/lifecycle/src/lib.rs b/rust/common/lifecycle/src/lib.rsindex 34f57f20ac2a..a020416393cc 100644--- a/rust/common/lifecycle/src/lib.rs+++ b/rust/common/lifecycle/src/lib.rs@@ -14,5 +14,5 @@ mod signals;pub use error::LifecycleError;pub use handle::{Handle, ProcessScopeGuard};pub use liveness::{LivenessHandler, LivenessStatus};-pub use manager::{ComponentOptions, Manager, ManagerBuilder, MonitorGuard};+pub use manager::{ComponentOptions, Manager, ManagerBuilder, MonitorGuard, PrestopAction};pub use readiness::ReadinessHandler;diff --git a/rust/common/lifecycle/src/manager.rs b/rust/common/lifecycle/src/manager.rsindex 50796eda16fe..a76e78a6d136 100644--- a/rust/common/lifecycle/src/manager.rs+++ b/rust/common/lifecycle/src/manager.rs@@ -21,6 +21,20 @@ use crate::metrics;use crate::readiness::ReadinessHandler;use crate::signals;+/// What the manager does when the pre-stop file appears.+/// (see tests `prestop_file_starts_shutdown_by_default`, `prestop_drain_serves_until_max`)+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]+pub enum PrestopAction {+ /// Start graceful shutdown, the same as SIGTERM.+ #[default]+ Shutdown,+ /// Fail readiness and keep serving, so K8s removes the pod from the Service+ /// endpoints while components still answer. Shutdown starts on SIGTERM or any+ /// other trigger, and at the latest `max` after the drain starts, so a lost+ /// SIGTERM cannot keep the pod in the drain until SIGKILL.+ Drain { max: Duration },+}+/// Builder for [`Manager`]. Start with [`Manager::builder("name")`](Manager::builder),/// chain `.with_*()` calls, then call [`.build()`](ManagerBuilder::build).#[derive(Debug)]@@ -28,7 +42,7 @@ pub struct ManagerBuilder {name: String,global_shutdown_timeout: Duration,trap_signals: bool,- enable_prestop_check: bool,+ prestop: Option<PrestopAction>,prestop_path: PathBuf,health_poll_interval: Duration,external_shutdown_token: Option<CancellationToken>,@@ -51,7 +65,16 @@ impl ManagerBuilder {/// Poll for pre-stop shutdown file (K8s pre-stop hook pattern). Default: true.pub fn with_prestop_check(mut self, enabled: bool) -> Self {- self.enable_prestop_check = enabled;+ self.prestop = enabled.then_some(self.prestop.unwrap_or_default());+ self+ }++ /// What to do when the pre-stop file appears. Default: [`PrestopAction::Shutdown`].+ /// Also turns the pre-stop check on.+ /// Use [`PrestopAction::Drain`] for a server whose clients keep connections to+ /// the pod after K8s removes it from the Service endpoints.+ pub fn with_prestop_action(mut self, action: PrestopAction) -> Self {+ self.prestop = Some(action);self}@@ -79,14 +102,16 @@ impl ManagerBuilder {/// Consume the builder and produce a [`Manager`].pub fn build(self) -> Manager {+ let shutdown_token = self.external_shutdown_token.unwrap_or_default();Manager {name: self.name,global_shutdown_timeout: self.global_shutdown_timeout,trap_signals: self.trap_signals,- enable_prestop_check: self.enable_prestop_check,+ prestop: self.prestop,prestop_path: self.prestop_path,health_poll_interval: self.health_poll_interval,- shutdown_token: self.external_shutdown_token.unwrap_or_default(),+ drain_token: shutdown_token.child_token(),+ shutdown_token,observability_shutdown_token: CancellationToken::new(),phase_tokens: HashMap::new(),event_tx_slot: Arc::new(OnceLock::new()),@@ -238,9 +263,12 @@ pub struct Manager {name: String,global_shutdown_timeout: Duration,trap_signals: bool,- enable_prestop_check: bool,+ prestop: Option<PrestopAction>,prestop_path: PathBuf,health_poll_interval: Duration,+ /// Child of `shutdown_token`, so readiness, which reads only this token, fails+ /// both when the app drains and when it shuts down.+ drain_token: CancellationToken,shutdown_token: CancellationToken,observability_shutdown_token: CancellationToken,/// Tokens for shutdown phases above 0, cancelled in ascending phase@@ -265,7 +293,7 @@ impl Manager {name: name.into(),global_shutdown_timeout: Duration::from_secs(60),trap_signals: true,- enable_prestop_check: true,+ prestop: Some(PrestopAction::default()),prestop_path: PathBuf::from(DEFAULT_PRESTOP_PATH),health_poll_interval: Duration::from_secs(5),external_shutdown_token: None,@@ -403,11 +431,11 @@ impl Manager {Handle { inner }}- /// Readiness probe handler (`/_readiness`). Returns 200 while running, 503 after shutdown- /// begins. K8s uses this to stop routing traffic to the pod.+ /// Readiness probe handler (`/_readiness`). Returns 200 while running, 503 once the app+ /// drains or shuts down. K8s uses this to stop routing traffic to the pod./// (see test `readiness_200_until_shutdown_then_503`)pub fn readiness_handler(&self) -> ReadinessHandler {- ReadinessHandler::new(self.shutdown_token.clone())+ ReadinessHandler::new(self.drain_token.clone())}/// Liveness probe handler (`/_liveness`). **Intentionally always returns 200** — the@@ -483,7 +511,6 @@ impl Manager {let _span = tracing::info_span!("lifecycle", app = %self.name).entered();let name = self.name.clone();let trap_signals = self.trap_signals;- let enable_prestop = self.enable_prestop_check;let health_poll_interval = self.health_poll_interval;let global_timeout = self.global_shutdown_timeout;let shutdown_token = self.shutdown_token.clone();@@ -502,21 +529,46 @@ impl Manager {});}- if enable_prestop {+ if let Some(prestop_action) = self.prestop {let token = shutdown_token.clone();+ let drain_token = self.drain_token.clone();let prestop_path = self.prestop_path.clone();tokio::spawn(async move {let mut interval = tokio::time::interval(Duration::from_secs(1));loop {- interval.tick().await;+ tokio::select! {+ _ = interval.tick() => {}+ _ = token.cancelled() => return,+ }if prestop_path.exists() {+ break;+ }+ }+ match prestop_action {+ PrestopAction::Shutdown => {info!(trigger_reason = "prestop",path = %prestop_path.display(),"Lifecycle: shutdown initiated, prestop file detected");token.cancel();- break;+ }+ PrestopAction::Drain { max } => {+ info!(+ path = %prestop_path.display(),+ "Lifecycle: drain started, prestop file detected"+ );+ drain_token.cancel();+ tokio::select! {+ _ = tokio::time::sleep(max) => {+ info!(+ trigger_reason = "prestop",+ "Lifecycle: shutdown initiated, drain limit reached"+ );+ token.cancel();+ }+ _ = token.cancelled() => {}+ }}}});diff --git a/rust/common/lifecycle/src/readiness.rs b/rust/common/lifecycle/src/readiness.rsindex 9c9dfb8a7dee..c63c22a5e3af 100644--- a/rust/common/lifecycle/src/readiness.rs+++ b/rust/common/lifecycle/src/readiness.rs@@ -3,26 +3,32 @@use axum::http::StatusCode;use tokio_util::sync::CancellationToken;-/// K8s readiness probe handler. Returns 200 while the app is running, 503 after shutdown-/// begins. K8s stops routing traffic to the pod when readiness fails. No per-component-/// logic — readiness is purely "is the app accepting work?"-/// (see test `readiness_200_until_shutdown_then_503`)+/// K8s readiness probe handler. Returns 200 while the app is running, 503 once it+/// drains or shuts down. K8s stops routing traffic to the pod when readiness fails.+/// No per-component logic — readiness is purely "is the app accepting work?"+/// (see tests `readiness_200_until_shutdown_then_503`, `prestop_drain_serves_until_max`)#[derive(Clone)]pub struct ReadinessHandler {- shutdown_token: CancellationToken,+ drain_token: CancellationToken,}impl ReadinessHandler {- pub(crate) fn new(shutdown_token: CancellationToken) -> Self {- Self { shutdown_token }+ pub(crate) fn new(drain_token: CancellationToken) -> Self {+ Self { drain_token }}- /// Returns OK or SERVICE_UNAVAILABLE based on shutdown token; no I/O.+ /// False once the app drains or shuts down; no I/O. A server can use it to ask+ /// clients to reconnect through the Service to a pod that stays.+ pub fn is_ready(&self) -> bool {+ !self.drain_token.is_cancelled()+ }++ /// Returns OK or SERVICE_UNAVAILABLE based on [`is_ready`](Self::is_ready); no I/O.pub async fn check(&self) -> StatusCode {- if self.shutdown_token.is_cancelled() {- StatusCode::SERVICE_UNAVAILABLE- } else {+ if self.is_ready() {StatusCode::OK+ } else {+ StatusCode::SERVICE_UNAVAILABLE}}}diff --git a/rust/common/lifecycle/tests/lifecycle_tests.rs b/rust/common/lifecycle/tests/lifecycle_tests.rsindex 5b60c11724af..096b6ce7aca9 100644--- a/rust/common/lifecycle/tests/lifecycle_tests.rs+++ b/rust/common/lifecycle/tests/lifecycle_tests.rs@@ -2,7 +2,7 @@ use std::sync::atomic::{AtomicBool, Ordering};use std::sync::Arc;use std::time::Duration;-use lifecycle::{ComponentOptions, LifecycleError, Manager};+use lifecycle::{ComponentOptions, LifecycleError, Manager, PrestopAction};use tokio_util::sync::CancellationToken;// ---------------------------------------------------------------------------@@ -1879,3 +1879,80 @@ async fn later_phase_graceful_window_starts_at_its_phase() {"phase-1 component was timed out against a window measured from shutdown start");}++// ---------------------------------------------------------------------------+// Pre-stop file+// ---------------------------------------------------------------------------++/// By default the pre-stop file starts shutdown, the same as SIGTERM.+#[tokio::test]+async fn prestop_file_starts_shutdown_by_default() {+ let prestop_dir = tempfile::tempdir().unwrap();+ let prestop_path = prestop_dir.path().join("shutdown");+ std::fs::write(&prestop_path, "").unwrap();+ let mut manager = Manager::builder("test")+ .with_trap_signals(false)+ .with_prestop_path(&prestop_path)+ .with_global_shutdown_timeout(Duration::from_secs(5))+ .build();++ let handle = manager.register("worker", ComponentOptions::new());+ let readiness = manager.readiness_handler();+ let guard = manager.monitor_background();++ let comp = ComponentA { handle };+ tokio::spawn(async move {+ comp.process().await;+ });++ tokio::time::timeout(Duration::from_secs(10), guard.wait())+ .await+ .expect("pre-stop file did not start shutdown")+ .unwrap();+ assert_eq!(readiness.check().await.as_u16(), 503);+}++/// Under `PrestopAction::Drain` the pre-stop file fails readiness, and components+/// keep serving. With no signal, shutdown starts once the drain reaches `max`.+#[tokio::test]+async fn prestop_drain_serves_until_max() {+ let prestop_dir = tempfile::tempdir().unwrap();+ let prestop_path = prestop_dir.path().join("shutdown");+ std::fs::write(&prestop_path, "").unwrap();+ let mut manager = Manager::builder("test")+ .with_trap_signals(false)+ .with_prestop_path(&prestop_path)+ .with_prestop_action(PrestopAction::Drain {+ max: Duration::from_secs(1),+ })+ .with_global_shutdown_timeout(Duration::from_secs(5))+ .build();++ let handle = manager.register("worker", ComponentOptions::new());+ let readiness = manager.readiness_handler();+ let guard = manager.monitor_background();++ let comp = ComponentA {+ handle: handle.clone(),+ };+ tokio::spawn(async move {+ comp.process().await;+ });++ tokio::time::timeout(Duration::from_secs(5), async {+ while readiness.is_ready() {+ tokio::time::sleep(Duration::from_millis(10)).await;+ }+ })+ .await+ .expect("pre-stop file did not start the drain");+ assert!(+ !handle.is_shutting_down(),+ "a drain must not signal shutdown to components"+ );++ tokio::time::timeout(Duration::from_secs(10), guard.wait())+ .await+ .expect("the drain did not end at max")+ .unwrap();+}diff --git a/rust/feature-flags/Cargo.toml b/rust/feature-flags/Cargo.tomlindex 887555c1ffbf..aa2472fe62ad 100644--- a/rust/feature-flags/Cargo.toml+++ b/rust/feature-flags/Cargo.toml@@ -121,6 +121,7 @@ assert-json-diff = { workspace = true }common-s3 = { path = "../common/s3", features = ["mock-client"] }reqwest = { workspace = true }futures = "0.3.30"+tempfile = "3"test-case = "3.3.1"tokio = { workspace = true, features = ["test-util"] }tokio-util = { workspace = true }diff --git a/rust/feature-flags/src/main.rs b/rust/feature-flags/src/main.rsindex a0beb94175d5..7ba1fcebb8d6 100644--- a/rust/feature-flags/src/main.rs+++ b/rust/feature-flags/src/main.rs@@ -145,8 +145,15 @@ async fn async_main(mut config: Config, rayon_dispatcher: RayonDispatcher) {}};+ // Drain for 20 s after the preStop file appears, then shut down. Kubelet sends SIGTERM+ // only after the preStop sleep, so keep that sleep at 20 s: a shorter sleep can end the+ // drain early, and a longer one can outlive the process and fail the preStop hook.+ // The termination grace period must cover the 20 s drain plus the 45 s shutdown timeout.let mut manager = lifecycle::Manager::builder("feature-flags").with_global_shutdown_timeout(Duration::from_secs(45))+ .with_prestop_action(lifecycle::PrestopAction::Drain {+ max: Duration::from_secs(20),+ }).build();let handles = register_components(&mut manager);diff --git a/rust/feature-flags/src/router.rs b/rust/feature-flags/src/router.rsindex 4dfd9c8f6b40..72ef0e5c8d5b 100644--- a/rust/feature-flags/src/router.rs+++ b/rust/feature-flags/src/router.rs@@ -10,8 +10,10 @@ use crate::billing::{BillingAggregator, FeatureFlagsLimiter, SessionReplayLimiteuse crate::database_pools::DatabasePools;use axum::{error_handling::HandleErrorLayer,- extract::{DefaultBodyLimit, Extension},- http::{Method, StatusCode},+ extract::{DefaultBodyLimit, Extension, Request},+ http::{header, HeaderValue, Method, StatusCode},+ middleware::Next,+ response::Response,routing::{any, get, post},Router,};@@ -458,9 +460,12 @@ where.route("/", get(index)).route("/_readiness",- get(move || {+ get({let r = readiness.clone();- async move { r.check().await }+ move || {+ let r = r.clone();+ async move { r.check().await }+ }}),).route(@@ -606,6 +611,10 @@ where.merge(internal_endpoints).layer(TraceLayer::new_for_http()).layer(cors)+ .layer(axum::middleware::from_fn_with_state(+ readiness,+ close_connection_unless_ready,+ )).with_state(state);// Don't install metrics unless asked to@@ -726,6 +735,22 @@ async fn test_pool_connection(pool: &PgPool, name: &str, skip_query: bool) -> ReOk(())}+/// Asks HTTP/1.1 clients to close the connection after this response once the pod is not+/// ready, so their next request opens a new connection through the Service to a pod that stays.+async fn close_connection_unless_ready(+ axum::extract::State(readiness): axum::extract::State<ReadinessHandler>,+ request: Request,+ next: Next,+) -> Response {+ let mut response = next.run(request).await;+ if !readiness.is_ready() {+ response+ .headers_mut()+ .insert(header::CONNECTION, HeaderValue::from_static("close"));+ }+ response+}+pub async fn index() -> &'static str {"feature flags"}diff --git a/rust/feature-flags/tests/common/mod.rs b/rust/feature-flags/tests/common/mod.rsindex dac41efe5c27..831aaa99f267 100644--- a/rust/feature-flags/tests/common/mod.rs+++ b/rust/feature-flags/tests/common/mod.rs@@ -1,4 +1,5 @@use std::net::SocketAddr;+use std::path::Path;use std::sync::Arc;use std::time::Duration;@@ -9,7 +10,7 @@ use common_redis::MockRedisClient;use feature_flags::team::team_models::Team;use feature_flags::utils::test_utils::team_token_hypercache_key;use governor::clock;-use lifecycle::Manager;+use lifecycle::{Manager, ManagerBuilder, PrestopAction};use limiters::redis::QUOTA_LIMITER_CACHE_KEY;use reqwest::header::CONTENT_TYPE;use tokio::net::TcpListener;@@ -34,13 +35,11 @@ impl ServerHandle {}}-fn build_test_manager(token: CancellationToken) -> Manager {+fn test_manager_builder() -> ManagerBuilder {Manager::builder("feature-flags-test").with_trap_signals(false).with_prestop_check(false)- .with_shutdown_token(token).with_global_shutdown_timeout(Duration::from_secs(10))- .build()}impl ServerHandle {@@ -58,12 +57,34 @@ impl ServerHandle {) -> ServerHandle {Self::for_config_with_s3_and_rate_limiter_clock(config,+ test_manager_builder(),flags_with_cohorts_s3,clock::DefaultClock::default(),).await}+ /// Like `for_config`, but the server drains once `prestop_path` exists, as in production.+ #[allow(dead_code)]+ pub async fn for_config_with_prestop_drain(+ config: Config,+ prestop_path: &Path,+ ) -> ServerHandle {+ let manager_builder = test_manager_builder()+ .with_prestop_path(prestop_path)+ .with_prestop_action(PrestopAction::Drain {+ // Longer than any test, so the drain limit never starts shutdown mid-test.+ max: Duration::from_secs(60),+ });+ Self::for_config_with_s3_and_rate_limiter_clock(+ config,+ manager_builder,+ None,+ clock::DefaultClock::default(),+ )+ .await+ }+#[allow(dead_code)]pub async fn for_config_with_rate_limiter_clock<C>(config: Config,@@ -72,11 +93,18 @@ impl ServerHandle {whereC: clock::Clock + Clone + Send + Sync + 'static,{- Self::for_config_with_s3_and_rate_limiter_clock(config, None, rate_limiter_clock).await+ Self::for_config_with_s3_and_rate_limiter_clock(+ config,+ test_manager_builder(),+ None,+ rate_limiter_clock,+ )+ .await}async fn for_config_with_s3_and_rate_limiter_clock<C>(config: Config,+ manager_builder: ManagerBuilder,flags_with_cohorts_s3: Option<Arc<dyn common_hypercache::S3Client + Send + Sync>>,rate_limiter_clock: C,) -> ServerHandle@@ -86,7 +114,9 @@ impl ServerHandle {let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();let addr = listener.local_addr().unwrap();let shutdown = CancellationToken::new();- let mut manager = build_test_manager(shutdown.clone());+ let mut manager = manager_builder+ .with_shutdown_token(shutdown.clone())+ .build();let handles = register_components(&mut manager);let monitor_guard = manager.monitor_background();@@ -152,7 +182,9 @@ impl ServerHandle {let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();let addr = listener.local_addr().unwrap();let shutdown = CancellationToken::new();- let mut manager = build_test_manager(shutdown.clone());+ let mut manager = test_manager_builder()+ .with_shutdown_token(shutdown.clone())+ .build();// `monitor_background()` is intentionally not called in this path:// ReadinessHandler reads the token directly and `trap_signals=false`// means no supervisor is required. `manager` is dropped at the end ofdiff --git a/rust/feature-flags/tests/test_lifecycle.rs b/rust/feature-flags/tests/test_lifecycle.rsindex 380fa8b75ea2..290abbf2410c 100644--- a/rust/feature-flags/tests/test_lifecycle.rs+++ b/rust/feature-flags/tests/test_lifecycle.rs@@ -1,5 +1,9 @@-use anyhow::Result;+use std::time::Duration;++use anyhow::{bail, Context, Result};use reqwest::StatusCode;+use tokio::io::{AsyncBufReadExt, AsyncReadExt, AsyncWriteExt, BufReader};+use tokio::net::TcpStream;use crate::common::*;use feature_flags::config::DEFAULT_TEST_CONFIG;@@ -12,6 +16,32 @@ async fn get_status(addr: &std::net::SocketAddr, path: &str) -> Result<StatusCodOk(resp.status())}+/// Sends `GET path` on an open HTTP/1.1 connection and returns the response head as+/// lowercase lines. Reads the body too, so the next request starts on a clean boundary.+async fn get_on(conn: &mut BufReader<TcpStream>, path: &str) -> Result<Vec<String>> {+ conn.get_mut()+ .write_all(format!("GET {path} HTTP/1.1\r\nHost: localhost\r\n\r\n").as_bytes())+ .await?;+ let mut head = Vec::new();+ loop {+ let mut line = String::new();+ if conn.read_line(&mut line).await? == 0 {+ bail!("connection closed before the response head ended");+ }+ let line = line.trim_end().to_ascii_lowercase();+ if line.is_empty() {+ break;+ }+ head.push(line);+ }+ let content_length = head+ .iter()+ .find_map(|line| line.strip_prefix("content-length:"))+ .map_or(Ok(0), |len| len.trim().parse())?;+ conn.read_exact(&mut vec![0; content_length]).await?;+ Ok(head)+}+/// Readiness returns 200 while running, then 503 once the shutdown token is cancelled.#[tokio::test]async fn readiness_flips_to_503_after_shutdown() -> Result<()> {@@ -110,3 +140,50 @@ async fn drop_triggers_shutdown() -> Result<()> {}panic!("server still accepting /_readiness requests ~10s after drop (last status: {last_status:?})");}++/// While the pod drains, the server still answers on a keep-alive connection opened+/// before the drain, then closes it, so the client reconnects to a pod that stays.+#[tokio::test]+async fn drain_closes_keep_alive_connections_after_the_response() -> Result<()> {+ let prestop_dir = tempfile::tempdir()?;+ let prestop_path = prestop_dir.path().join("shutdown");+ let server =+ ServerHandle::for_config_with_prestop_drain(DEFAULT_TEST_CONFIG.clone(), &prestop_path)+ .await;+ server.wait_until_ready().await;++ let mut conn = BufReader::new(TcpStream::connect(server.addr).await?);+ for _ in 0..2 {+ let head = get_on(&mut conn, "/flags").await?;+ assert!(+ !head.iter().any(|line| line == "connection: close"),+ "a serving pod must keep the connection open, got {head:?}"+ );+ }++ std::fs::write(&prestop_path, "")?;+ // The listener stays open while draining, so readiness answers 503 rather than+ // refusing the connection.+ tokio::time::timeout(Duration::from_secs(5), async {+ while get_status(&server.addr, "/_readiness").await? != StatusCode::SERVICE_UNAVAILABLE {+ tokio::time::sleep(Duration::from_millis(20)).await;+ }+ anyhow::Ok(())+ })+ .await+ .context("readiness did not fail after the pre-stop file appeared")??;++ let head = get_on(&mut conn, "/flags").await?;+ assert!(+ head.iter().any(|line| line == "connection: close"),+ "a draining pod must ask the client to close the connection, got {head:?}"+ );+ let closed = tokio::time::timeout(Duration::from_secs(5), conn.read(&mut [0; 1]))+ .await+ .context("the server kept the connection open after the response")??;+ assert_eq!(+ closed, 0,+ "the server must close the connection after the response"+ );+ Ok(())+}