Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(flags): drain on the pre-stop file before shutting down (#125490)

Matheus Batista9b61c96a130e
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index 7b6790256d6c..7391b08a2c17 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -4791,6 +4791,7 @@ dependencies = [
"sqlx",
"strum 0.28.0",
"subtle",
+ "tempfile",
"test-case",
"thiserror 2.0.18",
"tokio",
@@ -7501,6 +7502,7 @@ dependencies = [
"axum 0.7.9",
"common-liveness",
"metrics",
+ "tempfile",
"thiserror 2.0.18",
"tokio",
"tokio-util",
diff --git a/rust/common/lifecycle/Cargo.toml b/rust/common/lifecycle/Cargo.toml
index 86d63fb4faed..13ed25dd1159 100644
--- a/rust/common/lifecycle/Cargo.toml
+++ b/rust/common/lifecycle/Cargo.toml
@@ -14,3 +14,6 @@ thiserror = { workspace = true }
tokio = { workspace = true }
tokio-util = { workspace = true }
tracing = { workspace = true }
+
+[dev-dependencies]
+tempfile = "3"
diff --git a/rust/common/lifecycle/README.md b/rust/common/lifecycle/README.md
index 6259bb211fad..38fe04e79c1a 100644
--- a/rust/common/lifecycle/README.md
+++ b/rust/common/lifecycle/README.md
@@ -105,6 +105,7 @@ All options except `name` have sensible defaults.
| `.with_global_shutdown_timeout(duration)` | Hard ceiling on total shutdown. Monitor returns `ShutdownTimeout` if exceeded. Prevents indefinite hangs if a component doesn't check for cancellation. (see test `global_timeout_fires_when_component_hangs`) | `60s` |
| `.with_trap_signals(bool)` | Install SIGINT/SIGTERM handlers. Set `false` in tests. | `true` |
| `.with_prestop_check(bool)` | Poll for pre-stop shutdown file (K8s pre-stop hook pattern). | `true` |
+| `.with_prestop_action(action)` | What the pre-stop file does. `PrestopAction::Shutdown` starts graceful shutdown. `PrestopAction::Drain { max }` fails readiness and keeps serving until SIGTERM, for at most `max`. Turns the pre-stop check on. See [Draining before shutdown](#draining-before-shutdown). (see test `prestop_drain_serves_until_max`) | `Shutdown` |
| `.with_prestop_path(path)` | Override the pre-stop file path. | `/tmp/shutdown` |
| `.with_health_poll_interval(duration)` | Override health monitor poll frequency. The health monitor is automatically active when any component has `with_liveness_deadline`. (see test `stall_triggers_shutdown`) | `5s` |
| `.with_shutdown_token(token)` | Caller supplies an external `CancellationToken`. Use in tests to trigger deterministic, app-global shutdown (simulate `SIGTERM` from k8s etc.) | `None` |
@@ -125,7 +126,7 @@ All options except `name` have sensible defaults.
## K8s readiness and liveness
-**Readiness** (`/_readiness`) returns 200 until shutdown begins, then 503. K8s uses this to stop routing traffic to the pod. No per-component logic — it's purely "is the app accepting work?" (see test `readiness_200_until_shutdown_then_503`)
+**Readiness** (`/_readiness`) returns 200 until the app drains or shutdown begins, then 503. K8s uses this to stop routing traffic to the pod. No per-component logic — it's purely "is the app accepting work?" (see test `readiness_200_until_shutdown_then_503`)
**Liveness** (`/_liveness`) always returns 200 — it means "the process is reachable." Health monitoring is handled internally by the manager's health poll task, not by K8s. When a component's heartbeat deadline expires, the health monitor increments a stall counter. After `stall_threshold` consecutive stalled checks, the manager triggers global shutdown via the same `ComponentEvent::Failure` path as `signal_failure()`. This ensures the app always gets coordinated graceful shutdown instead of K8s surprise-killing the pod. (see tests `stall_triggers_shutdown`, `stall_threshold_exceeded_triggers_shutdown`, `stall_threshold_allows_recovery`)
@@ -162,6 +163,30 @@ metrics_handle.work_completed();
guard.wait().await?;
```
+### Draining before shutdown
+
+By default the pre-stop file starts graceful shutdown at once, the same as SIGTERM.
+A server can close connections that clients still use, because clients do not see the pod leave until K8s removes it from the Service endpoints.
+A client behind a proxy that reaches the pod through one ClusterIP never sees the pod leave, and keeps its pooled connections on it.
+
+`.with_prestop_action(PrestopAction::Drain { max })` turns the pre-stop window into a drain:
+
+- The pre-stop file fails readiness, and components keep serving. The manager logs `Lifecycle: drain started, prestop file detected`.
+- SIGTERM, SIGINT, a component failure or a liveness stall starts shutdown as usual. If none of them comes within `max` of the drain start, the manager starts shutdown itself and logs `Lifecycle: shutdown initiated, drain limit reached`.
+- Set the pod's preStop sleep to `max`, because kubelet sends SIGTERM only after the sleep ends.
+- Set the pod's `terminationGracePeriodSeconds` above `max` plus `with_global_shutdown_timeout`.
+
+```rust
+let mut manager = Manager::builder("my-service")
+ .with_prestop_action(PrestopAction::Drain { max: Duration::from_secs(20) })
+ .build();
+```
+
+An HTTP/1.1 server can send `Connection: close` on each response while `ReadinessHandler::is_ready()` returns false.
+Each client that sends a request during the drain then opens its next connection through the Service to a pod that stays.
+A pooled connection that carries no request during the drain still closes at shutdown, so clients must still retry a request that fails on a reused connection.
+See `close_connection_unless_ready` in `rust/feature-flags/src/router.rs` (see test `drain_closes_keep_alive_connections_after_the_response`).
+
## Using the handle
### With `process_scope()` (struct-held handle)
diff --git a/rust/common/lifecycle/src/lib.rs b/rust/common/lifecycle/src/lib.rs
index 34f57f20ac2a..a020416393cc 100644
--- a/rust/common/lifecycle/src/lib.rs
+++ b/rust/common/lifecycle/src/lib.rs
@@ -14,5 +14,5 @@ mod signals;
pub use error::LifecycleError;
pub use handle::{Handle, ProcessScopeGuard};
pub use liveness::{LivenessHandler, LivenessStatus};
-pub use manager::{ComponentOptions, Manager, ManagerBuilder, MonitorGuard};
+pub use manager::{ComponentOptions, Manager, ManagerBuilder, MonitorGuard, PrestopAction};
pub use readiness::ReadinessHandler;
diff --git a/rust/common/lifecycle/src/manager.rs b/rust/common/lifecycle/src/manager.rs
index 50796eda16fe..a76e78a6d136 100644
--- a/rust/common/lifecycle/src/manager.rs
+++ b/rust/common/lifecycle/src/manager.rs
@@ -21,6 +21,20 @@ use crate::metrics;
use crate::readiness::ReadinessHandler;
use crate::signals;
+/// What the manager does when the pre-stop file appears.
+/// (see tests `prestop_file_starts_shutdown_by_default`, `prestop_drain_serves_until_max`)
+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
+pub enum PrestopAction {
+ /// Start graceful shutdown, the same as SIGTERM.
+ #[default]
+ Shutdown,
+ /// Fail readiness and keep serving, so K8s removes the pod from the Service
+ /// endpoints while components still answer. Shutdown starts on SIGTERM or any
+ /// other trigger, and at the latest `max` after the drain starts, so a lost
+ /// SIGTERM cannot keep the pod in the drain until SIGKILL.
+ Drain { max: Duration },
+}
+
/// Builder for [`Manager`]. Start with [`Manager::builder("name")`](Manager::builder),
/// chain `.with_*()` calls, then call [`.build()`](ManagerBuilder::build).
#[derive(Debug)]
@@ -28,7 +42,7 @@ pub struct ManagerBuilder {
name: String,
global_shutdown_timeout: Duration,
trap_signals: bool,
- enable_prestop_check: bool,
+ prestop: Option<PrestopAction>,
prestop_path: PathBuf,
health_poll_interval: Duration,
external_shutdown_token: Option<CancellationToken>,
@@ -51,7 +65,16 @@ impl ManagerBuilder {
/// Poll for pre-stop shutdown file (K8s pre-stop hook pattern). Default: true.
pub fn with_prestop_check(mut self, enabled: bool) -> Self {
- self.enable_prestop_check = enabled;
+ self.prestop = enabled.then_some(self.prestop.unwrap_or_default());
+ self
+ }
+
+ /// What to do when the pre-stop file appears. Default: [`PrestopAction::Shutdown`].
+ /// Also turns the pre-stop check on.
+ /// Use [`PrestopAction::Drain`] for a server whose clients keep connections to
+ /// the pod after K8s removes it from the Service endpoints.
+ pub fn with_prestop_action(mut self, action: PrestopAction) -> Self {
+ self.prestop = Some(action);
self
}
@@ -79,14 +102,16 @@ impl ManagerBuilder {
/// Consume the builder and produce a [`Manager`].
pub fn build(self) -> Manager {
+ let shutdown_token = self.external_shutdown_token.unwrap_or_default();
Manager {
name: self.name,
global_shutdown_timeout: self.global_shutdown_timeout,
trap_signals: self.trap_signals,
- enable_prestop_check: self.enable_prestop_check,
+ prestop: self.prestop,
prestop_path: self.prestop_path,
health_poll_interval: self.health_poll_interval,
- shutdown_token: self.external_shutdown_token.unwrap_or_default(),
+ drain_token: shutdown_token.child_token(),
+ shutdown_token,
observability_shutdown_token: CancellationToken::new(),
phase_tokens: HashMap::new(),
event_tx_slot: Arc::new(OnceLock::new()),
@@ -238,9 +263,12 @@ pub struct Manager {
name: String,
global_shutdown_timeout: Duration,
trap_signals: bool,
- enable_prestop_check: bool,
+ prestop: Option<PrestopAction>,
prestop_path: PathBuf,
health_poll_interval: Duration,
+ /// Child of `shutdown_token`, so readiness, which reads only this token, fails
+ /// both when the app drains and when it shuts down.
+ drain_token: CancellationToken,
shutdown_token: CancellationToken,
observability_shutdown_token: CancellationToken,
/// Tokens for shutdown phases above 0, cancelled in ascending phase
@@ -265,7 +293,7 @@ impl Manager {
name: name.into(),
global_shutdown_timeout: Duration::from_secs(60),
trap_signals: true,
- enable_prestop_check: true,
+ prestop: Some(PrestopAction::default()),
prestop_path: PathBuf::from(DEFAULT_PRESTOP_PATH),
health_poll_interval: Duration::from_secs(5),
external_shutdown_token: None,
@@ -403,11 +431,11 @@ impl Manager {
Handle { inner }
}
- /// Readiness probe handler (`/_readiness`). Returns 200 while running, 503 after shutdown
- /// begins. K8s uses this to stop routing traffic to the pod.
+ /// Readiness probe handler (`/_readiness`). Returns 200 while running, 503 once the app
+ /// drains or shuts down. K8s uses this to stop routing traffic to the pod.
/// (see test `readiness_200_until_shutdown_then_503`)
pub fn readiness_handler(&self) -> ReadinessHandler {
- ReadinessHandler::new(self.shutdown_token.clone())
+ ReadinessHandler::new(self.drain_token.clone())
}
/// Liveness probe handler (`/_liveness`). **Intentionally always returns 200** — the
@@ -483,7 +511,6 @@ impl Manager {
let _span = tracing::info_span!("lifecycle", app = %self.name).entered();
let name = self.name.clone();
let trap_signals = self.trap_signals;
- let enable_prestop = self.enable_prestop_check;
let health_poll_interval = self.health_poll_interval;
let global_timeout = self.global_shutdown_timeout;
let shutdown_token = self.shutdown_token.clone();
@@ -502,21 +529,46 @@ impl Manager {
});
}
- if enable_prestop {
+ if let Some(prestop_action) = self.prestop {
let token = shutdown_token.clone();
+ let drain_token = self.drain_token.clone();
let prestop_path = self.prestop_path.clone();
tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(1));
loop {
- interval.tick().await;
+ tokio::select! {
+ _ = interval.tick() => {}
+ _ = token.cancelled() => return,
+ }
if prestop_path.exists() {
+ break;
+ }
+ }
+ match prestop_action {
+ PrestopAction::Shutdown => {
info!(
trigger_reason = "prestop",
path = %prestop_path.display(),
"Lifecycle: shutdown initiated, prestop file detected"
);
token.cancel();
- break;
+ }
+ PrestopAction::Drain { max } => {
+ info!(
+ path = %prestop_path.display(),
+ "Lifecycle: drain started, prestop file detected"
+ );
+ drain_token.cancel();
+ tokio::select! {
+ _ = tokio::time::sleep(max) => {
+ info!(
+ trigger_reason = "prestop",
+ "Lifecycle: shutdown initiated, drain limit reached"
+ );
+ token.cancel();
+ }
+ _ = token.cancelled() => {}
+ }
}
}
});
diff --git a/rust/common/lifecycle/src/readiness.rs b/rust/common/lifecycle/src/readiness.rs
index 9c9dfb8a7dee..c63c22a5e3af 100644
--- a/rust/common/lifecycle/src/readiness.rs
+++ b/rust/common/lifecycle/src/readiness.rs
@@ -3,26 +3,32 @@
use axum::http::StatusCode;
use tokio_util::sync::CancellationToken;
-/// K8s readiness probe handler. Returns 200 while the app is running, 503 after shutdown
-/// begins. K8s stops routing traffic to the pod when readiness fails. No per-component
-/// logic — readiness is purely "is the app accepting work?"
-/// (see test `readiness_200_until_shutdown_then_503`)
+/// K8s readiness probe handler. Returns 200 while the app is running, 503 once it
+/// drains or shuts down. K8s stops routing traffic to the pod when readiness fails.
+/// No per-component logic — readiness is purely "is the app accepting work?"
+/// (see tests `readiness_200_until_shutdown_then_503`, `prestop_drain_serves_until_max`)
#[derive(Clone)]
pub struct ReadinessHandler {
- shutdown_token: CancellationToken,
+ drain_token: CancellationToken,
}
impl ReadinessHandler {
- pub(crate) fn new(shutdown_token: CancellationToken) -> Self {
- Self { shutdown_token }
+ pub(crate) fn new(drain_token: CancellationToken) -> Self {
+ Self { drain_token }
}
- /// Returns OK or SERVICE_UNAVAILABLE based on shutdown token; no I/O.
+ /// False once the app drains or shuts down; no I/O. A server can use it to ask
+ /// clients to reconnect through the Service to a pod that stays.
+ pub fn is_ready(&self) -> bool {
+ !self.drain_token.is_cancelled()
+ }
+
+ /// Returns OK or SERVICE_UNAVAILABLE based on [`is_ready`](Self::is_ready); no I/O.
pub async fn check(&self) -> StatusCode {
- if self.shutdown_token.is_cancelled() {
- StatusCode::SERVICE_UNAVAILABLE
- } else {
+ if self.is_ready() {
StatusCode::OK
+ } else {
+ StatusCode::SERVICE_UNAVAILABLE
}
}
}
diff --git a/rust/common/lifecycle/tests/lifecycle_tests.rs b/rust/common/lifecycle/tests/lifecycle_tests.rs
index 5b60c11724af..096b6ce7aca9 100644
--- a/rust/common/lifecycle/tests/lifecycle_tests.rs
+++ b/rust/common/lifecycle/tests/lifecycle_tests.rs
@@ -2,7 +2,7 @@ use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::Duration;
-use lifecycle::{ComponentOptions, LifecycleError, Manager};
+use lifecycle::{ComponentOptions, LifecycleError, Manager, PrestopAction};
use tokio_util::sync::CancellationToken;
// ---------------------------------------------------------------------------
@@ -1879,3 +1879,80 @@ async fn later_phase_graceful_window_starts_at_its_phase() {
"phase-1 component was timed out against a window measured from shutdown start"
);
}
+
+// ---------------------------------------------------------------------------
+// Pre-stop file
+// ---------------------------------------------------------------------------
+
+/// By default the pre-stop file starts shutdown, the same as SIGTERM.
+#[tokio::test]
+async fn prestop_file_starts_shutdown_by_default() {
+ let prestop_dir = tempfile::tempdir().unwrap();
+ let prestop_path = prestop_dir.path().join("shutdown");
+ std::fs::write(&prestop_path, "").unwrap();
+ let mut manager = Manager::builder("test")
+ .with_trap_signals(false)
+ .with_prestop_path(&prestop_path)
+ .with_global_shutdown_timeout(Duration::from_secs(5))
+ .build();
+
+ let handle = manager.register("worker", ComponentOptions::new());
+ let readiness = manager.readiness_handler();
+ let guard = manager.monitor_background();
+
+ let comp = ComponentA { handle };
+ tokio::spawn(async move {
+ comp.process().await;
+ });
+
+ tokio::time::timeout(Duration::from_secs(10), guard.wait())
+ .await
+ .expect("pre-stop file did not start shutdown")
+ .unwrap();
+ assert_eq!(readiness.check().await.as_u16(), 503);
+}
+
+/// Under `PrestopAction::Drain` the pre-stop file fails readiness, and components
+/// keep serving. With no signal, shutdown starts once the drain reaches `max`.
+#[tokio::test]
+async fn prestop_drain_serves_until_max() {
+ let prestop_dir = tempfile::tempdir().unwrap();
+ let prestop_path = prestop_dir.path().join("shutdown");
+ std::fs::write(&prestop_path, "").unwrap();
+ let mut manager = Manager::builder("test")
+ .with_trap_signals(false)
+ .with_prestop_path(&prestop_path)
+ .with_prestop_action(PrestopAction::Drain {
+ max: Duration::from_secs(1),
+ })
+ .with_global_shutdown_timeout(Duration::from_secs(5))
+ .build();
+
+ let handle = manager.register("worker", ComponentOptions::new());
+ let readiness = manager.readiness_handler();
+ let guard = manager.monitor_background();
+
+ let comp = ComponentA {
+ handle: handle.clone(),
+ };
+ tokio::spawn(async move {
+ comp.process().await;
+ });
+
+ tokio::time::timeout(Duration::from_secs(5), async {
+ while readiness.is_ready() {
+ tokio::time::sleep(Duration::from_millis(10)).await;
+ }
+ })
+ .await
+ .expect("pre-stop file did not start the drain");
+ assert!(
+ !handle.is_shutting_down(),
+ "a drain must not signal shutdown to components"
+ );
+
+ tokio::time::timeout(Duration::from_secs(10), guard.wait())
+ .await
+ .expect("the drain did not end at max")
+ .unwrap();
+}
diff --git a/rust/feature-flags/Cargo.toml b/rust/feature-flags/Cargo.toml
index 887555c1ffbf..aa2472fe62ad 100644
--- a/rust/feature-flags/Cargo.toml
+++ b/rust/feature-flags/Cargo.toml
@@ -121,6 +121,7 @@ assert-json-diff = { workspace = true }
common-s3 = { path = "../common/s3", features = ["mock-client"] }
reqwest = { workspace = true }
futures = "0.3.30"
+tempfile = "3"
test-case = "3.3.1"
tokio = { workspace = true, features = ["test-util"] }
tokio-util = { workspace = true }
diff --git a/rust/feature-flags/src/main.rs b/rust/feature-flags/src/main.rs
index a0beb94175d5..7ba1fcebb8d6 100644
--- a/rust/feature-flags/src/main.rs
+++ b/rust/feature-flags/src/main.rs
@@ -145,8 +145,15 @@ async fn async_main(mut config: Config, rayon_dispatcher: RayonDispatcher) {
}
};
+ // Drain for 20 s after the preStop file appears, then shut down. Kubelet sends SIGTERM
+ // only after the preStop sleep, so keep that sleep at 20 s: a shorter sleep can end the
+ // drain early, and a longer one can outlive the process and fail the preStop hook.
+ // The termination grace period must cover the 20 s drain plus the 45 s shutdown timeout.
let mut manager = lifecycle::Manager::builder("feature-flags")
.with_global_shutdown_timeout(Duration::from_secs(45))
+ .with_prestop_action(lifecycle::PrestopAction::Drain {
+ max: Duration::from_secs(20),
+ })
.build();
let handles = register_components(&mut manager);
diff --git a/rust/feature-flags/src/router.rs b/rust/feature-flags/src/router.rs
index 4dfd9c8f6b40..72ef0e5c8d5b 100644
--- a/rust/feature-flags/src/router.rs
+++ b/rust/feature-flags/src/router.rs
@@ -10,8 +10,10 @@ use crate::billing::{BillingAggregator, FeatureFlagsLimiter, SessionReplayLimite
use crate::database_pools::DatabasePools;
use axum::{
error_handling::HandleErrorLayer,
- extract::{DefaultBodyLimit, Extension},
- http::{Method, StatusCode},
+ extract::{DefaultBodyLimit, Extension, Request},
+ http::{header, HeaderValue, Method, StatusCode},
+ middleware::Next,
+ response::Response,
routing::{any, get, post},
Router,
};
@@ -458,9 +460,12 @@ where
.route("/", get(index))
.route(
"/_readiness",
- get(move || {
+ get({
let r = readiness.clone();
- async move { r.check().await }
+ move || {
+ let r = r.clone();
+ async move { r.check().await }
+ }
}),
)
.route(
@@ -606,6 +611,10 @@ where
.merge(internal_endpoints)
.layer(TraceLayer::new_for_http())
.layer(cors)
+ .layer(axum::middleware::from_fn_with_state(
+ readiness,
+ close_connection_unless_ready,
+ ))
.with_state(state);
// Don't install metrics unless asked to
@@ -726,6 +735,22 @@ async fn test_pool_connection(pool: &PgPool, name: &str, skip_query: bool) -> Re
Ok(())
}
+/// Asks HTTP/1.1 clients to close the connection after this response once the pod is not
+/// ready, so their next request opens a new connection through the Service to a pod that stays.
+async fn close_connection_unless_ready(
+ axum::extract::State(readiness): axum::extract::State<ReadinessHandler>,
+ request: Request,
+ next: Next,
+) -> Response {
+ let mut response = next.run(request).await;
+ if !readiness.is_ready() {
+ response
+ .headers_mut()
+ .insert(header::CONNECTION, HeaderValue::from_static("close"));
+ }
+ response
+}
+
pub async fn index() -> &'static str {
"feature flags"
}
diff --git a/rust/feature-flags/tests/common/mod.rs b/rust/feature-flags/tests/common/mod.rs
index dac41efe5c27..831aaa99f267 100644
--- a/rust/feature-flags/tests/common/mod.rs
+++ b/rust/feature-flags/tests/common/mod.rs
@@ -1,4 +1,5 @@
use std::net::SocketAddr;
+use std::path::Path;
use std::sync::Arc;
use std::time::Duration;
@@ -9,7 +10,7 @@ use common_redis::MockRedisClient;
use feature_flags::team::team_models::Team;
use feature_flags::utils::test_utils::team_token_hypercache_key;
use governor::clock;
-use lifecycle::Manager;
+use lifecycle::{Manager, ManagerBuilder, PrestopAction};
use limiters::redis::QUOTA_LIMITER_CACHE_KEY;
use reqwest::header::CONTENT_TYPE;
use tokio::net::TcpListener;
@@ -34,13 +35,11 @@ impl ServerHandle {
}
}
-fn build_test_manager(token: CancellationToken) -> Manager {
+fn test_manager_builder() -> ManagerBuilder {
Manager::builder("feature-flags-test")
.with_trap_signals(false)
.with_prestop_check(false)
- .with_shutdown_token(token)
.with_global_shutdown_timeout(Duration::from_secs(10))
- .build()
}
impl ServerHandle {
@@ -58,12 +57,34 @@ impl ServerHandle {
) -> ServerHandle {
Self::for_config_with_s3_and_rate_limiter_clock(
config,
+ test_manager_builder(),
flags_with_cohorts_s3,
clock::DefaultClock::default(),
)
.await
}
+ /// Like `for_config`, but the server drains once `prestop_path` exists, as in production.
+ #[allow(dead_code)]
+ pub async fn for_config_with_prestop_drain(
+ config: Config,
+ prestop_path: &Path,
+ ) -> ServerHandle {
+ let manager_builder = test_manager_builder()
+ .with_prestop_path(prestop_path)
+ .with_prestop_action(PrestopAction::Drain {
+ // Longer than any test, so the drain limit never starts shutdown mid-test.
+ max: Duration::from_secs(60),
+ });
+ Self::for_config_with_s3_and_rate_limiter_clock(
+ config,
+ manager_builder,
+ None,
+ clock::DefaultClock::default(),
+ )
+ .await
+ }
+
#[allow(dead_code)]
pub async fn for_config_with_rate_limiter_clock<C>(
config: Config,
@@ -72,11 +93,18 @@ impl ServerHandle {
where
C: clock::Clock + Clone + Send + Sync + 'static,
{
- Self::for_config_with_s3_and_rate_limiter_clock(config, None, rate_limiter_clock).await
+ Self::for_config_with_s3_and_rate_limiter_clock(
+ config,
+ test_manager_builder(),
+ None,
+ rate_limiter_clock,
+ )
+ .await
}
async fn for_config_with_s3_and_rate_limiter_clock<C>(
config: Config,
+ manager_builder: ManagerBuilder,
flags_with_cohorts_s3: Option<Arc<dyn common_hypercache::S3Client + Send + Sync>>,
rate_limiter_clock: C,
) -> ServerHandle
@@ -86,7 +114,9 @@ impl ServerHandle {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let shutdown = CancellationToken::new();
- let mut manager = build_test_manager(shutdown.clone());
+ let mut manager = manager_builder
+ .with_shutdown_token(shutdown.clone())
+ .build();
let handles = register_components(&mut manager);
let monitor_guard = manager.monitor_background();
@@ -152,7 +182,9 @@ impl ServerHandle {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let shutdown = CancellationToken::new();
- let mut manager = build_test_manager(shutdown.clone());
+ let mut manager = test_manager_builder()
+ .with_shutdown_token(shutdown.clone())
+ .build();
// `monitor_background()` is intentionally not called in this path:
// ReadinessHandler reads the token directly and `trap_signals=false`
// means no supervisor is required. `manager` is dropped at the end of
diff --git a/rust/feature-flags/tests/test_lifecycle.rs b/rust/feature-flags/tests/test_lifecycle.rs
index 380fa8b75ea2..290abbf2410c 100644
--- a/rust/feature-flags/tests/test_lifecycle.rs
+++ b/rust/feature-flags/tests/test_lifecycle.rs
@@ -1,5 +1,9 @@
-use anyhow::Result;
+use std::time::Duration;
+
+use anyhow::{bail, Context, Result};
use reqwest::StatusCode;
+use tokio::io::{AsyncBufReadExt, AsyncReadExt, AsyncWriteExt, BufReader};
+use tokio::net::TcpStream;
use crate::common::*;
use feature_flags::config::DEFAULT_TEST_CONFIG;
@@ -12,6 +16,32 @@ async fn get_status(addr: &std::net::SocketAddr, path: &str) -> Result<StatusCod
Ok(resp.status())
}
+/// Sends `GET path` on an open HTTP/1.1 connection and returns the response head as
+/// lowercase lines. Reads the body too, so the next request starts on a clean boundary.
+async fn get_on(conn: &mut BufReader<TcpStream>, path: &str) -> Result<Vec<String>> {
+ conn.get_mut()
+ .write_all(format!("GET {path} HTTP/1.1\r\nHost: localhost\r\n\r\n").as_bytes())
+ .await?;
+ let mut head = Vec::new();
+ loop {
+ let mut line = String::new();
+ if conn.read_line(&mut line).await? == 0 {
+ bail!("connection closed before the response head ended");
+ }
+ let line = line.trim_end().to_ascii_lowercase();
+ if line.is_empty() {
+ break;
+ }
+ head.push(line);
+ }
+ let content_length = head
+ .iter()
+ .find_map(|line| line.strip_prefix("content-length:"))
+ .map_or(Ok(0), |len| len.trim().parse())?;
+ conn.read_exact(&mut vec![0; content_length]).await?;
+ Ok(head)
+}
+
/// Readiness returns 200 while running, then 503 once the shutdown token is cancelled.
#[tokio::test]
async fn readiness_flips_to_503_after_shutdown() -> Result<()> {
@@ -110,3 +140,50 @@ async fn drop_triggers_shutdown() -> Result<()> {
}
panic!("server still accepting /_readiness requests ~10s after drop (last status: {last_status:?})");
}
+
+/// While the pod drains, the server still answers on a keep-alive connection opened
+/// before the drain, then closes it, so the client reconnects to a pod that stays.
+#[tokio::test]
+async fn drain_closes_keep_alive_connections_after_the_response() -> Result<()> {
+ let prestop_dir = tempfile::tempdir()?;
+ let prestop_path = prestop_dir.path().join("shutdown");
+ let server =
+ ServerHandle::for_config_with_prestop_drain(DEFAULT_TEST_CONFIG.clone(), &prestop_path)
+ .await;
+ server.wait_until_ready().await;
+
+ let mut conn = BufReader::new(TcpStream::connect(server.addr).await?);
+ for _ in 0..2 {
+ let head = get_on(&mut conn, "/flags").await?;
+ assert!(
+ !head.iter().any(|line| line == "connection: close"),
+ "a serving pod must keep the connection open, got {head:?}"
+ );
+ }
+
+ std::fs::write(&prestop_path, "")?;
+ // The listener stays open while draining, so readiness answers 503 rather than
+ // refusing the connection.
+ tokio::time::timeout(Duration::from_secs(5), async {
+ while get_status(&server.addr, "/_readiness").await? != StatusCode::SERVICE_UNAVAILABLE {
+ tokio::time::sleep(Duration::from_millis(20)).await;
+ }
+ anyhow::Ok(())
+ })
+ .await
+ .context("readiness did not fail after the pre-stop file appeared")??;
+
+ let head = get_on(&mut conn, "/flags").await?;
+ assert!(
+ head.iter().any(|line| line == "connection: close"),
+ "a draining pod must ask the client to close the connection, got {head:?}"
+ );
+ let closed = tokio::time::timeout(Duration::from_secs(5), conn.read(&mut [0; 1]))
+ .await
+ .context("the server kept the connection open after the response")??;
+ assert_eq!(
+ closed, 0,
+ "the server must close the connection after the response"
+ );
+ Ok(())
+}