:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(personhog): stay up through an etcd outage (#123655)
Zach Brown805a6d2bf9bc
diff --git a/rust/common/assignment-coordination/src/failover.rs b/rust/common/assignment-coordination/src/failover.rsindex 13c1a050b16c..33058c0b74ba 100644--- a/rust/common/assignment-coordination/src/failover.rs+++ b/rust/common/assignment-coordination/src/failover.rs@@ -33,6 +33,12 @@ pub fn is_member_level(err: &etcd_client::Error) -> bool {}}+/// The Unavailable error etcd answers a require-leader request with when the+/// member has no leader, for the same condition reported another way.+pub fn no_leader_error(message: impl Into<String>) -> etcd_client::Error {+ etcd_client::Error::GRpcStatus(tonic::Status::unavailable(message))+}+/// Whether the client raised this status for a failed connection or HTTP/2/// stream, whatever its code, rather than etcd sending it.fn raised_by_connection(status: &tonic::Status) -> bool {diff --git a/rust/personhog-changelog-relay/src/main.rs b/rust/personhog-changelog-relay/src/main.rsindex 8c543abdab71..45e986917264 100644--- a/rust/personhog-changelog-relay/src/main.rs+++ b/rust/personhog-changelog-relay/src/main.rs@@ -1,8 +1,9 @@+use std::sync::atomic::{AtomicBool, Ordering};use std::sync::Arc;use std::time::Duration;use assignment_coordination::store::{EtcdStore, StoreConfig};-use axum::{routing::get, Router};+use axum::{http::StatusCode, routing::get, Router};use common_kafka::kafka_producer::{create_kafka_producer, spawn_bounded_flush_on_shutdown};use common_metrics::setup_metrics_routes;use envconfig::Envconfig;@@ -67,6 +68,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {ComponentOptions::new().is_observability(true),);let readiness = manager.readiness_handler();+ let config_loaded = Arc::new(AtomicBool::new(false));let liveness = manager.liveness_handler();let monitor = manager.monitor_background();@@ -75,9 +77,19 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {Router::new().route("/_readiness",- get(move || {- let r = readiness.clone();- async move { r.check().await }+ get({+ let config_loaded = Arc::clone(&config_loaded);+ move || {+ let r = readiness.clone();+ let loaded = config_loaded.load(Ordering::SeqCst);+ async move {+ if loaded {+ r.check().await+ } else {+ StatusCode::SERVICE_UNAVAILABLE+ }+ }+ }}),).route("/_liveness", get(move || async move { liveness.check() })),@@ -100,7 +112,16 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {}).await?;let store = Arc::new(PersonhogStore::new(store));- let loaded = load_changelog_config(&store, CHANGELOG_BOOT_POLL_INTERVAL).await?;+ let Some(loaded) = load_changelog_config(+ &store,+ CHANGELOG_BOOT_POLL_INTERVAL,+ &changelog_handle.shutdown_token(),+ )+ .await?+ else {+ return Ok(());+ };+ config_loaded.store(true, Ordering::SeqCst);// Retire waits only on the groups the config lists, so relaying under any// other group could see a generation deleted before it is read.if !loaded.retire_groups.contains(&config.kafka_consumer_group) {diff --git a/rust/personhog-coordination/README.md b/rust/personhog-coordination/README.mdindex 1f7dbb57d2ac..dfa7171b91b8 100644--- a/rust/personhog-coordination/README.md+++ b/rust/personhog-coordination/README.md@@ -96,7 +96,7 @@ Several failure modes are handled explicitly:- **Reconcile tick**: Beyond the startup reconcile, the coordinator re-evaluates every in-flight handoff on a fixed interval (`reconcile_interval`, default 5 s). This is the liveness backstop for state changes that fire no watched event — e.g. a router departing (nothing watches router registrations) newly satisfying a freeze quorum. Everything the tick drives is idempotent: phase transitions use CAS and cleanup tolerates already-deleted records.- **Lease loss (self-fencing)**: Connection trouble and lease loss are different things. A broken or silent keepalive stream is a fact about one connection — the lease lives in etcd's replicated keyspace and stays valid until its TTL passes without renewal — so ambiguous keepalive failures rebuild the stream and retry (the store moves to another member when its own keeps failing; see `assignment_coordination::failover`) for as long as the last confirmed renewal is recent; a routine single-member restart costs some retries (`personhog_coordination_keepalive_retries_total{component}`), not a fence. The keepalive declares loss on exactly two signals: etcd answering a round with TTL ≤ 0 (authoritative — the lease is revoked or expired), or the renewal margin running out (two thirds of the TTL without a confirmed renewal — the final third is reserved so the fence completes before the coordinator can possibly treat the lease as expired; retrying past it would gamble split-brain against availability). On loss the component ends the coordination attempt immediately, skipping the graceful drain: the coordinator already considers it dead and is reassigning its partitions via the dead-owner path, so a graceful drain would only race that. A pod self-fences *locally* before anything else: every held partition is drained and, once quiesced, released (dropping its cache and serving authority — re-acquisition always re-warms). A partition whose drain cannot quiesce is deliberately **not** released — release is what unfences admissions and drops the cache, and a leaseless pod must do neither over live in-flight work — so it stays fenced and held, in-place recovery is refused, and the process restart clears it by death. A zombie owner therefore never keeps serving partitions the protocol has reassigned beyond the window that restart closes. Lease loss is observed at every await the session makes while holding a lease — the attempt select, the backoff nap between failed attempts, and the graceful drain all race the keepalive — so once the keepalive declares loss, the fence is never deferred by supervisor state. Every other session end fences too — budget exhaustion and shutdown revoke the lease, which deletes the registration instantly, so the local fence runs first (a no-op after a completed graceful drain) and the dead-owner path never meets a still-serving owner. On lease loss the fence's drain budget is the runway the keepalive margin reserved (a third of the TTL), not the full graceful drain timeout; overshoot poisons recovery and the process restart clears stragglers by death.- **etcd member failover**: Each process keeps one connection to whichever etcd member it reached first. Three member-level failures in a row (Unavailable, DeadlineExceeded, a broken connection), or one call left unanswered for at least 2 s, start a switch: the store probes the other members from its cached member list with a linearizable read and moves new calls to the first that answers. Requests carry etcd's require-leader flag, so a member without a leader rejects them at once. Watches and keepalive streams already open stay on their connection until it closes or etcd cancels them. A renewal waits up to the heartbeat interval for an answer, so a member that stops answering without closing its connections still costs that pod one renewal margin, and with it the pod's partitions.-- **Supervised coordination runs (etcd resilience)**: A failed coordination attempt — a broken watch stream, a failed etcd write, an exhausted reconcile budget — no longer kills the process; the run supervisors tear the attempt down cleanly, back off exponentially, and rebuild through the same bootstrap that recovers a restarted process, while the data plane keeps serving. The two participant classes recover *differently*, on purpose. A **router** sheds its registration on every failed attempt (its registration is a freeze-quorum vote, and a registered but non-acking router would stall every quorum until the phase deadline); serving through its gap is safe because handoffs created while it is unregistered exclude it from their quorums and the leader-side fences reject its stale forwards. A **pod** does the opposite: its lease session (grant, register, heartbeat) lives *above* the attempt loop and survives attempt failures, so the coordinator keeps seeing a live pod and never runs the dead-owner path against a process that is still serving — the invariant is that serving authority is held exactly as long as the lease-backed registration behind it. This matters because the dead-owner path has no fence: its safety rests entirely on a deregistered owner being unable to produce, so a pod must never voluntarily deregister while it still serves. Only lease loss ends a pod session, self-fencing locally (every held partition released; re-acquisition re-warms) before a fresh session registers anew. A consecutive-failure budget (`ROUTER_RUN_RETRY_BUDGET`, default 10; same default on pods) keeps a genuinely wedged participant on the process-restart path; an attempt that made real progress — a convergence applied, a reconcile pass completed — resets the count, and the reset is keyed on measured progress rather than elapsed time because a time threshold silently exempts every failure detector slower than it (the reconcile budget and the stall watchdog both take a minute to fire), and `personhog_coordination_run_restarts_total{component}` counts every rebuild.+- **Supervised coordination runs (etcd resilience)**: A failed coordination attempt — a broken watch stream, a failed etcd write, an exhausted reconcile budget — no longer kills the process; the run supervisors tear the attempt down cleanly, back off exponentially, and rebuild through the same bootstrap that recovers a restarted process, while the data plane keeps serving. The two participant classes recover *differently*, on purpose. A **router** sheds its registration on every failed attempt (its registration is a freeze-quorum vote, and a registered but non-acking router would stall every quorum until the phase deadline); serving through its gap is safe because handoffs created while it is unregistered exclude it from their quorums and the leader-side fences reject its stale forwards. A **pod** does the opposite: its lease session (grant, register, heartbeat) lives *above* the attempt loop and survives attempt failures, so the coordinator keeps seeing a live pod and never runs the dead-owner path against a process that is still serving — the invariant is that serving authority is held exactly as long as the lease-backed registration behind it. This matters because the dead-owner path has no fence: its safety rests entirely on a deregistered owner being unable to produce, so a pod must never voluntarily deregister while it still serves. Only lease loss ends a pod session, self-fencing locally (every held partition released; re-acquisition re-warms) before a fresh session registers anew. A consecutive-failure budget (`ROUTER_RUN_RETRY_BUDGET`, default 10; same default on pods) keeps a genuinely wedged participant on the process-restart path. Failures a restart cannot fix back off like any other but spend nothing: etcd could not be reached (Unavailable, DeadlineExceeded, a broken connection, or a watch cancelled because its member has no leader), or the session's lease was lost, by expiring in etcd or by its renewal margin running out. A new attempt is already the remedy for these, through a fresh session and, for connection trouble, member failover; through an outage the participant stays up and rebuilds once etcd answers. An attempt that made real progress — a convergence applied, a reconcile pass completed — resets the count, and the reset is keyed on measured progress rather than elapsed time because a time threshold silently exempts every failure detector slower than it (the reconcile budget and the stall watchdog both take a minute to fire), and `personhog_coordination_run_restarts_total{component}` counts every rebuild.### Concurrency and atomicitydiff --git a/rust/personhog-coordination/src/changelog.rs b/rust/personhog-coordination/src/changelog.rsindex 194071661927..80eb2e52bc91 100644--- a/rust/personhog-coordination/src/changelog.rs+++ b/rust/personhog-coordination/src/changelog.rs@@ -13,21 +13,43 @@ use crate::store::PersonhogStore;use crate::types::ChangelogConfig;use crate::util;+/// Whether boot keeps polling through `err`: any etcd error, since there is+/// nothing to do before the config loads, but not a malformed config value.+fn waits_at_boot(err: &Error) -> bool {+ err.etcd_error().is_some()+}+/// Polls until `config/changelog` is seeded; it is the only source of the topic and count.+/// Keeps polling through etcd being unreachable, and returns `None` once `shutdown` fires.pub async fn load_changelog_config(store: &PersonhogStore,poll_interval: Duration,-) -> Result<ChangelogConfig> {+ shutdown: &CancellationToken,+) -> Result<Option<ChangelogConfig>> {let mut polls: u64 = 0;loop {- if let Some(config) = store.get_changelog_config().await? {- return Ok(config);- }- if polls.is_multiple_of(10) {- tracing::warn!("config/changelog is not seeded; waiting for the control plane");+ let read = tokio::select! {+ _ = shutdown.cancelled() => return Ok(None),+ read = store.get_changelog_config() => read,+ };+ match read {+ Ok(Some(config)) => return Ok(Some(config)),+ Ok(None) if polls.is_multiple_of(10) => {+ tracing::warn!("config/changelog is not seeded; waiting for the control plane");+ }+ Err(e) if waits_at_boot(&e) => {+ if polls.is_multiple_of(10) {+ tracing::warn!(error = %e, "etcd is unreachable; waiting to load config/changelog");+ }+ }+ Err(e) => return Err(e),+ Ok(None) => {}}polls += 1;- tokio::time::sleep(poll_interval).await;+ tokio::select! {+ _ = shutdown.cancelled() => return Ok(None),+ _ = tokio::time::sleep(poll_interval) => {}+ }}}@@ -363,6 +385,17 @@ mod tests {use super::*;use crate::types::Generation;+ #[test]+ fn boot_waits_through_any_etcd_error_but_not_a_malformed_config() {+ let not_member_level: Error =+ etcd_client::Error::WatchError("member stopping".to_string()).into();+ assert!(waits_at_boot(¬_member_level));+ let malformed: Error = serde_json::from_str::<ChangelogConfig>("{")+ .unwrap_err()+ .into();+ assert!(!waits_at_boot(&malformed));+ }+fn config(partitions: u32, draining: Vec<(u32, u64)>) -> ChangelogConfig {ChangelogConfig {topic: "new".to_string(),diff --git a/rust/personhog-coordination/src/error.rs b/rust/personhog-coordination/src/error.rsindex 1685c896016d..b48ff9ac46c2 100644--- a/rust/personhog-coordination/src/error.rs+++ b/rust/personhog-coordination/src/error.rs@@ -10,6 +10,9 @@ pub enum Error {#[error("handoff failed for partition {partition}: {reason}")]HandoffFailed { partition: u32, reason: String },++ #[error("lease renewal margin exhausted: no confirmed renewal in {0:?}")]+ LeaseMarginExhausted(std::time::Duration),}impl From<serde_json::Error> for Error {@@ -52,6 +55,12 @@ impl Error {Self::Store(assignment_coordination::error::Error::LeadershipLost))}++ /// Whether a lease ended the session, by expiring in etcd or by its+ /// renewal margin running out. A new session is the remedy.+ pub fn is_lease_lost(&self) -> bool {+ self.is_leadership_lost() || matches!(self, Self::LeaseMarginExhausted(_))+ }}pub type Result<T> = std::result::Result<T, Error>;diff --git a/rust/personhog-coordination/src/pod.rs b/rust/personhog-coordination/src/pod.rsindex 9b896ce9c57c..a7603591a6a0 100644--- a/rust/personhog-coordination/src/pod.rs+++ b/rust/personhog-coordination/src/pod.rs@@ -458,7 +458,7 @@ impl PodHandle {/// stall every freeze quorum until the phase deadline.pub async fn run(&self, cancel: CancellationToken) -> Result<()> {util::preregister_pod_metrics();- let mut consecutive_failures: u32 = 0;+ let mut consecutive_failures = util::RunFailures::default();// Set by the coordination loop whenever it applies real work// (a convergence completed); consumed by each failure note to// decide crash-loop vs fresh failure.@@ -476,7 +476,10 @@ impl PodHandle {if !self.note_run_failure(&mut consecutive_failures, &progress, &e) {return Err(e);}- if self.run_backoff(&cancel, consecutive_failures).await {+ if self+ .run_backoff(&cancel, consecutive_failures.consecutive())+ .await+ {return Ok(());}continue;@@ -589,7 +592,7 @@ impl PodHandle {// coordinator may already be reassigning while the pod// serves, unaware, until the nap ends.tokio::select! {- stop = self.run_backoff(&cancel, consecutive_failures) => {+ stop = self.run_backoff(&cancel, consecutive_failures.consecutive()) => {if stop {break;}@@ -753,7 +756,10 @@ impl PodHandle {if !self.note_run_failure(&mut consecutive_failures, &progress, &e) {return Err(e);}- if self.run_backoff(&cancel, consecutive_failures).await {+ if self+ .run_backoff(&cancel, consecutive_failures.consecutive())+ .await+ {return Ok(());}// New session: fresh lease, fresh registration,@@ -813,9 +819,14 @@ impl PodHandle {/// See `util::note_run_failure` for the progress-based reset/// semantics.- fn note_run_failure(&self, consecutive: &mut u32, progress: &AtomicBool, err: &Error) -> bool {+ fn note_run_failure(+ &self,+ failures: &mut util::RunFailures,+ progress: &AtomicBool,+ err: &Error,+ ) -> bool {util::note_run_failure(- consecutive,+ failures,progress,self.config.run_retry_budget,"pod",diff --git a/rust/personhog-coordination/src/routing_table.rs b/rust/personhog-coordination/src/routing_table.rsindex 5a016b4ae902..f6ac042c4e5f 100644--- a/rust/personhog-coordination/src/routing_table.rs+++ b/rust/personhog-coordination/src/routing_table.rs@@ -367,7 +367,7 @@ impl RoutingTable {) -> Result<()> {const BACKOFF_CAP: Duration = Duration::from_secs(15);- let mut consecutive_failures: u32 = 0;+ let mut consecutive_failures = util::RunFailures::default();// Set by the coordination loop whenever it does real work;// consumed by each failure note to decide crash-loop vs fresh// failure. Arc because the watch loop runs as a spawned task.@@ -398,7 +398,9 @@ impl RoutingTable {let backoff = self.config.run_retry_backoff- .saturating_mul(2u32.saturating_pow(consecutive_failures.saturating_sub(1)))+ .saturating_mul(+ 2u32.saturating_pow(consecutive_failures.consecutive().saturating_sub(1)),+ ).min(BACKOFF_CAP);tokio::select! {_ = cancel.cancelled() => return Ok(()),diff --git a/rust/personhog-coordination/src/util.rs b/rust/personhog-coordination/src/util.rsindex 545556080936..6cd203e2c769 100644--- a/rust/personhog-coordination/src/util.rs+++ b/rust/personhog-coordination/src/util.rs@@ -24,16 +24,38 @@ pub fn new_handoff_id() -> String {format!("{}-{}", millis, Uuid::new_v4())}-/// Record one supervisor-level failure against the consecutive budget;-/// returns `false` when it is exhausted. Progress resets the count-/// first, and INVARIANT: progress means *applied* work (a completed-/// convergence, an applied snapshot or event) — never a read that-/// changed nothing, which stays available in exactly the wedges the-/// budget exists for. Measured rather than time-based, because a-/// "healthy after N seconds" threshold exempts every failure detector-/// slower than N.+/// A supervisor's consecutive failures since its last applied work.+#[derive(Debug, Default)]+pub(crate) struct RunFailures {+ /// Every failure; paces the backoff.+ consecutive: u32,+ /// Failures that spend the budget.+ budgeted: u32,+}++impl RunFailures {+ pub(crate) fn consecutive(&self) -> u32 {+ self.consecutive+ }+}++/// Whether a coordination failure only says etcd could not be reached.+pub(crate) fn is_etcd_unavailable(err: &Error) -> bool {+ err.etcd_error()+ .is_some_and(assignment_coordination::failover::is_member_level)+}++/// Whether a restart would fix nothing a new attempt does not: etcd could+/// not be reached, or a lease was lost. These do not spend the budget.+fn spends_budget(err: &Error) -> bool {+ !(is_etcd_unavailable(err) || err.is_lease_lost())+}++/// Records one supervisor failure; returns `false` once the budget is spent.+/// INVARIANT: only applied work resets the counts, never a read that changed+/// nothing. Failures a restart cannot fix spend no budget: see `spends_budget`.pub(crate) fn note_run_failure(- consecutive: &mut u32,+ failures: &mut RunFailures,progress: &AtomicBool,budget: u32,component: &'static str,@@ -41,9 +63,12 @@ pub(crate) fn note_run_failure(err: &Error,) -> bool {if progress.swap(false, Ordering::SeqCst) {- *consecutive = 1;- } else {- *consecutive += 1;+ *failures = RunFailures::default();+ }+ failures.consecutive += 1;+ let spends_budget = spends_budget(err);+ if spends_budget {+ failures.budgeted += 1;}counter!("personhog_coordination_run_restarts_total",@@ -54,11 +79,13 @@ pub(crate) fn note_run_failure(component,name,error = %err,- consecutive = *consecutive,+ consecutive = failures.consecutive,+ budget_spent = failures.budgeted,budget,+ spends_budget,"coordination run failed; rebuilding in place while the data plane keeps serving");- *consecutive < budget+ failures.budgeted < budget}/// Log and count a coordination-run failure for a component with no@@ -135,11 +162,7 @@ pub async fn run_lease_keepalive(// however long registration took.let mut last_renewed = granted_at;- let margin_exhausted = || {- Error::invalid_state(format!(- "lease renewal margin exhausted: no confirmed renewal in {renewal_margin:?}"- ))- };+ let margin_exhausted = || Error::LeaseMarginExhausted(renewal_margin);'stream: loop {// (Re)establish the keepalive stream, bounded by the margin.@@ -305,15 +328,34 @@ pub(crate) fn live_watch_response() -> Result<etcd_client::WatchResponse> {let resp = resp.ok_or_else(|| Error::invalid_state(format!("{what} watch stream ended")))?;if resp.canceled() {- return Err(Error::invalid_state(format!(- "{what} watcher cancelled by etcd (reason: {:?}, compact_revision: {})",+ return Err(watch_cancelled(+ what,resp.cancel_reason(),resp.compact_revision(),- )));+ ));}Ok(resp)}+/// The cancel reason etcd sends when the member serving a require-leader+/// watch loses its leader (rpctypes.ErrNoLeader).+const NO_LEADER_CANCEL_REASON: &str = "etcdserver: no leader";++/// A member without a leader cancels the watch rather than failing it, so+/// that case maps to the same Unavailable status etcd answers a+/// require-leader request with.+fn watch_cancelled(what: &str, reason: &str, compact_revision: i64) -> Error {+ if reason == NO_LEADER_CANCEL_REASON {+ return assignment_coordination::failover::no_leader_error(format!(+ "{what} watcher cancelled by etcd: {reason}"+ ))+ .into();+ }+ Error::invalid_state(format!(+ "{what} watcher cancelled by etcd (reason: {reason:?}, compact_revision: {compact_revision})"+ ))+}+/// Count one resolution that found no record — not one record lost: a/// single missing record resolves once per frozen partition per pass,/// so read it as a rate (how much work is degraded to the fallback),@@ -529,7 +571,7 @@ mod tests {use std::collections::HashSet;use std::sync::atomic::{AtomicBool, Ordering};- use super::{new_handoff_id, note_run_failure};+ use super::{new_handoff_id, note_run_failure, watch_cancelled, RunFailures};use crate::error::Error;/// Losing the election lease ends a leadership term but says nothing@@ -552,22 +594,22 @@ mod tests {#[test]fn applied_work_clears_the_failure_count() {let progress = AtomicBool::new(false);- let err = Error::invalid_state("etcd unreachable");- let mut consecutive = 0u32;+ let err = Error::invalid_state("list_handoffs failed");+ let mut failures = RunFailures::default();for attempt in 1..3 {assert!(- note_run_failure(&mut consecutive, &progress, 3, "pod", "p", &err),+ note_run_failure(&mut failures, &progress, 3, "pod", "p", &err),"attempt {attempt} is within budget");}progress.store(true, Ordering::SeqCst);assert!(- note_run_failure(&mut consecutive, &progress, 3, "pod", "p", &err),+ note_run_failure(&mut failures, &progress, 3, "pod", "p", &err),"applied work resets the count");- assert_eq!(consecutive, 1);+ assert_eq!(failures.consecutive(), 1);}/// And it must escalate when nothing succeeds in between — this is@@ -577,10 +619,10 @@ mod tests {fn an_unbroken_run_of_failures_exhausts_the_budget() {let progress = AtomicBool::new(false);let err = Error::invalid_state("list_handoffs failed");- let mut consecutive = 0u32;+ let mut failures = RunFailures::default();assert!(note_run_failure(- &mut consecutive,+ &mut failures,&progress,2,"pod",@@ -588,11 +630,73 @@ mod tests {&err));assert!(- !note_run_failure(&mut consecutive, &progress, 2, "pod", "p", &err),+ !note_run_failure(&mut failures, &progress, 2, "pod", "p", &err),"the budget is spent, so the caller must stop retrying");}+ #[test]+ fn failures_a_restart_cannot_fix_back_off_without_spending_the_budget() {+ let unfixable_by_restart: Vec<(&str, Error)> = vec![+ (+ "etcd unreachable",+ etcd_client::Error::IoError(std::io::Error::from(+ std::io::ErrorKind::ConnectionRefused,+ ))+ .into(),+ ),+ ("lease expired in etcd", Error::leadership_lost()),+ (+ "renewal margin exhausted",+ Error::LeaseMarginExhausted(std::time::Duration::from_secs(20)),+ ),+ (+ "watch cancelled for no leader",+ watch_cancelled("routing table", "etcdserver: no leader", 0),+ ),+ ];+ for (case, err) in unfixable_by_restart {+ let progress = AtomicBool::new(false);+ let mut failures = RunFailures::default();+ for _ in 0..50 {+ assert!(+ note_run_failure(&mut failures, &progress, 2, "pod", "p", &err),+ "{case} spent the budget"+ );+ }+ assert_eq!(+ failures.consecutive(),+ 50,+ "{case}: the backoff still grows"+ );+ }++ let wedges = [+ Error::invalid_state("list_handoffs failed"),+ watch_cancelled(+ "routing table",+ "etcdserver: mvcc: required revision has been compacted",+ 42,+ ),+ ];+ for wedge in wedges {+ let progress = AtomicBool::new(false);+ let mut failures = RunFailures::default();+ assert!(note_run_failure(+ &mut failures,+ &progress,+ 2,+ "pod",+ "p",+ &wedge+ ));+ assert!(+ !note_run_failure(&mut failures, &progress, 2, "pod", "p", &wedge),+ "{wedge} must still exhaust the budget"+ );+ }+ }+/// Quorum correlation and cancellation detection hang off id/// uniqueness; ids minted in the same instant (a handoff cancelled/// and recreated within one millisecond) must never collide.diff --git a/rust/personhog-coordination/tests/changelog.rs b/rust/personhog-coordination/tests/changelog.rsindex 282c77bfa8a0..483d5639051b 100644--- a/rust/personhog-coordination/tests/changelog.rs+++ b/rust/personhog-coordination/tests/changelog.rs@@ -9,7 +9,7 @@ use personhog_coordination::types::{ChangelogConfig, Generation};use tokio::sync::mpsc;use tokio_util::sync::CancellationToken;-use common::{store_at, test_store, FlakyProxy, ETCD_ENDPOINT};+use common::{store_at, test_store, FlakyProxy, ETCD_ENDPOINT, WAIT_TIMEOUT};#[tokio::test]async fn a_second_seed_is_refused() {@@ -47,14 +47,65 @@ async fn boot_polls_until_the_config_is_seeded() {});let loaded = tokio::time::timeout(Duration::from_secs(5),- load_changelog_config(&store, Duration::from_millis(20)),+ load_changelog_config(&store, Duration::from_millis(20), &CancellationToken::new()),).await.expect("boot polls until the config is seeded")- .unwrap();+ .unwrap()+ .expect("no shutdown was requested");assert_eq!((loaded.partitions, loaded.epoch), (64, 1));}+#[tokio::test]+async fn boot_waits_out_an_unreachable_etcd() {+ let seeded = test_store("changelog-boot-outage").await;+ seeded+ .seed_changelog_config("personhog_updates_64", 64, "personhog-writer")+ .await+ .unwrap();+ let proxy = FlakyProxy::start("127.0.0.1:2379").await;+ proxy.set_blackholed(true);+ let store = store_at(&proxy.endpoint, seeded.inner().prefix()).await;++ let boot = tokio::spawn(async move {+ load_changelog_config(&store, Duration::from_millis(20), &CancellationToken::new()).await+ });+ tokio::time::sleep(Duration::from_secs(1)).await;+ assert!(+ !boot.is_finished(),+ "boot keeps waiting while etcd is unreachable"+ );++ proxy.set_blackholed(false);+ let loaded = tokio::time::timeout(WAIT_TIMEOUT, boot)+ .await+ .expect("boot loads the config once etcd is back")+ .unwrap()+ .unwrap()+ .expect("no shutdown was requested");+ assert_eq!(loaded.partitions, 64);+}++#[tokio::test]+async fn boot_stops_waiting_on_shutdown() {+ let store = test_store("changelog-boot-shutdown").await;+ let shutdown = CancellationToken::new();+ let waiting = {+ let shutdown = shutdown.clone();+ tokio::spawn(async move {+ load_changelog_config(&store, Duration::from_millis(20), &shutdown).await+ })+ };+ tokio::time::sleep(Duration::from_millis(100)).await;+ shutdown.cancel();+ let loaded = tokio::time::timeout(Duration::from_secs(2), waiting)+ .await+ .expect("shutdown ends the wait")+ .unwrap()+ .unwrap();+ assert!(loaded.is_none());+}+/// What a loop starts from before it has adopted anything.fn nothing_adopted() -> ChangelogConfig {ChangelogConfig {diff --git a/rust/personhog-identity/src/main.rs b/rust/personhog-identity/src/main.rsindex 8446092f723c..30db8d126c81 100644--- a/rust/personhog-identity/src/main.rs+++ b/rust/personhog-identity/src/main.rs@@ -155,9 +155,17 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {.await.expect("Failed to connect to etcd");let store = Arc::new(PersonhogStore::new(etcd_store));- let loaded = load_changelog_config(&store, CHANGELOG_BOOT_POLL_INTERVAL)- .await- .expect("Failed to load the changelog config from etcd");+ // No signal handler is installed until the lifecycle monitor starts,+ // so SIGTERM ends the process during this wait.+ let Some(loaded) = load_changelog_config(+ &store,+ CHANGELOG_BOOT_POLL_INTERVAL,+ &CancellationToken::new(),+ )+ .await+ .expect("Failed to load the changelog config from etcd") else {+ return Ok(());+ };tracing::info!(num_partitions = loaded.partitions,epoch = loaded.epoch,diff --git a/rust/personhog-leader/src/main.rs b/rust/personhog-leader/src/main.rsindex db0c09d6bfbb..f470fa555eda 100644--- a/rust/personhog-leader/src/main.rs+++ b/rust/personhog-leader/src/main.rs@@ -367,9 +367,15 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {let store = Arc::new(PersonhogStore::new(etcd_store));// The same source the router hashes against.- let loaded = load_changelog_config(&store, CHANGELOG_BOOT_POLL_INTERVAL)- .await- .expect("Failed to load the changelog config from etcd");+ let Some(loaded) = load_changelog_config(+ &store,+ CHANGELOG_BOOT_POLL_INTERVAL,+ &changelog_handle.shutdown_token(),+ )+ .await+ .expect("Failed to load the changelog config from etcd") else {+ return Ok(());+ };let changelog_topic = loaded.topic.clone();let num_partitions = loaded.partitions;let topic_partitions =diff --git a/rust/personhog-router/src/main.rs b/rust/personhog-router/src/main.rsindex dfd599040890..08751f27c71c 100644--- a/rust/personhog-router/src/main.rs+++ b/rust/personhog-router/src/main.rs@@ -258,9 +258,17 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {.expect("Failed to connect to etcd");let store = Arc::new(PersonhogStore::new(etcd_store));- let loaded = load_changelog_config(&store, CHANGELOG_BOOT_POLL_INTERVAL)- .await- .expect("Failed to load the changelog config from etcd");+ let boot_shutdown = changelog_handle+ .as_ref()+ .expect("changelog handle must be registered in leader mode")+ .shutdown_token();+ let Some(loaded) =+ load_changelog_config(&store, CHANGELOG_BOOT_POLL_INTERVAL, &boot_shutdown)+ .await+ .expect("Failed to load the changelog config from etcd")+ else {+ return Ok(());+ };let num_partitions = loaded.partitions;tracing::info!(num_partitions,diff --git a/rust/personhog-writer/src/main.rs b/rust/personhog-writer/src/main.rsindex 659541589f8b..89d04660c269 100644--- a/rust/personhog-writer/src/main.rs+++ b/rust/personhog-writer/src/main.rs@@ -61,7 +61,17 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {}).await?;let store = Arc::new(PersonhogStore::new(store));- let loaded = load_changelog_config(&store, CHANGELOG_BOOT_POLL_INTERVAL).await?;+ // No signal handler is installed until the lifecycle monitor starts,+ // so SIGTERM ends the process during this wait.+ let Some(loaded) = load_changelog_config(+ &store,+ CHANGELOG_BOOT_POLL_INTERVAL,+ &CancellationToken::new(),+ )+ .await?+ else {+ return Ok(());+ };if loaded.writer_group != config.kafka_consumer_group {return Err(format!("config/changelog names writer group {} but KAFKA_CONSUMER_GROUP is {}",