Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(personhog): stay up through an etcd outage (#123655)

Zach Brown805a6d2bf9bc
diff --git a/rust/common/assignment-coordination/src/failover.rs b/rust/common/assignment-coordination/src/failover.rs
index 13c1a050b16c..33058c0b74ba 100644
--- a/rust/common/assignment-coordination/src/failover.rs
+++ b/rust/common/assignment-coordination/src/failover.rs
@@ -33,6 +33,12 @@ pub fn is_member_level(err: &etcd_client::Error) -> bool {
}
}
+/// The Unavailable error etcd answers a require-leader request with when the
+/// member has no leader, for the same condition reported another way.
+pub fn no_leader_error(message: impl Into<String>) -> etcd_client::Error {
+ etcd_client::Error::GRpcStatus(tonic::Status::unavailable(message))
+}
+
/// Whether the client raised this status for a failed connection or HTTP/2
/// stream, whatever its code, rather than etcd sending it.
fn raised_by_connection(status: &tonic::Status) -> bool {
diff --git a/rust/personhog-changelog-relay/src/main.rs b/rust/personhog-changelog-relay/src/main.rs
index 8c543abdab71..45e986917264 100644
--- a/rust/personhog-changelog-relay/src/main.rs
+++ b/rust/personhog-changelog-relay/src/main.rs
@@ -1,8 +1,9 @@
+use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::Duration;
use assignment_coordination::store::{EtcdStore, StoreConfig};
-use axum::{routing::get, Router};
+use axum::{http::StatusCode, routing::get, Router};
use common_kafka::kafka_producer::{create_kafka_producer, spawn_bounded_flush_on_shutdown};
use common_metrics::setup_metrics_routes;
use envconfig::Envconfig;
@@ -67,6 +68,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
ComponentOptions::new().is_observability(true),
);
let readiness = manager.readiness_handler();
+ let config_loaded = Arc::new(AtomicBool::new(false));
let liveness = manager.liveness_handler();
let monitor = manager.monitor_background();
@@ -75,9 +77,19 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
Router::new()
.route(
"/_readiness",
- get(move || {
- let r = readiness.clone();
- async move { r.check().await }
+ get({
+ let config_loaded = Arc::clone(&config_loaded);
+ move || {
+ let r = readiness.clone();
+ let loaded = config_loaded.load(Ordering::SeqCst);
+ async move {
+ if loaded {
+ r.check().await
+ } else {
+ StatusCode::SERVICE_UNAVAILABLE
+ }
+ }
+ }
}),
)
.route("/_liveness", get(move || async move { liveness.check() })),
@@ -100,7 +112,16 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
})
.await?;
let store = Arc::new(PersonhogStore::new(store));
- let loaded = load_changelog_config(&store, CHANGELOG_BOOT_POLL_INTERVAL).await?;
+ let Some(loaded) = load_changelog_config(
+ &store,
+ CHANGELOG_BOOT_POLL_INTERVAL,
+ &changelog_handle.shutdown_token(),
+ )
+ .await?
+ else {
+ return Ok(());
+ };
+ config_loaded.store(true, Ordering::SeqCst);
// Retire waits only on the groups the config lists, so relaying under any
// other group could see a generation deleted before it is read.
if !loaded.retire_groups.contains(&config.kafka_consumer_group) {
diff --git a/rust/personhog-coordination/README.md b/rust/personhog-coordination/README.md
index 1f7dbb57d2ac..dfa7171b91b8 100644
--- a/rust/personhog-coordination/README.md
+++ b/rust/personhog-coordination/README.md
@@ -96,7 +96,7 @@ Several failure modes are handled explicitly:
- **Reconcile tick**: Beyond the startup reconcile, the coordinator re-evaluates every in-flight handoff on a fixed interval (`reconcile_interval`, default 5 s). This is the liveness backstop for state changes that fire no watched event — e.g. a router departing (nothing watches router registrations) newly satisfying a freeze quorum. Everything the tick drives is idempotent: phase transitions use CAS and cleanup tolerates already-deleted records.
- **Lease loss (self-fencing)**: Connection trouble and lease loss are different things. A broken or silent keepalive stream is a fact about one connection — the lease lives in etcd's replicated keyspace and stays valid until its TTL passes without renewal — so ambiguous keepalive failures rebuild the stream and retry (the store moves to another member when its own keeps failing; see `assignment_coordination::failover`) for as long as the last confirmed renewal is recent; a routine single-member restart costs some retries (`personhog_coordination_keepalive_retries_total{component}`), not a fence. The keepalive declares loss on exactly two signals: etcd answering a round with TTL ≤ 0 (authoritative — the lease is revoked or expired), or the renewal margin running out (two thirds of the TTL without a confirmed renewal — the final third is reserved so the fence completes before the coordinator can possibly treat the lease as expired; retrying past it would gamble split-brain against availability). On loss the component ends the coordination attempt immediately, skipping the graceful drain: the coordinator already considers it dead and is reassigning its partitions via the dead-owner path, so a graceful drain would only race that. A pod self-fences *locally* before anything else: every held partition is drained and, once quiesced, released (dropping its cache and serving authority — re-acquisition always re-warms). A partition whose drain cannot quiesce is deliberately **not** released — release is what unfences admissions and drops the cache, and a leaseless pod must do neither over live in-flight work — so it stays fenced and held, in-place recovery is refused, and the process restart clears it by death. A zombie owner therefore never keeps serving partitions the protocol has reassigned beyond the window that restart closes. Lease loss is observed at every await the session makes while holding a lease — the attempt select, the backoff nap between failed attempts, and the graceful drain all race the keepalive — so once the keepalive declares loss, the fence is never deferred by supervisor state. Every other session end fences too — budget exhaustion and shutdown revoke the lease, which deletes the registration instantly, so the local fence runs first (a no-op after a completed graceful drain) and the dead-owner path never meets a still-serving owner. On lease loss the fence's drain budget is the runway the keepalive margin reserved (a third of the TTL), not the full graceful drain timeout; overshoot poisons recovery and the process restart clears stragglers by death.
- **etcd member failover**: Each process keeps one connection to whichever etcd member it reached first. Three member-level failures in a row (Unavailable, DeadlineExceeded, a broken connection), or one call left unanswered for at least 2 s, start a switch: the store probes the other members from its cached member list with a linearizable read and moves new calls to the first that answers. Requests carry etcd's require-leader flag, so a member without a leader rejects them at once. Watches and keepalive streams already open stay on their connection until it closes or etcd cancels them. A renewal waits up to the heartbeat interval for an answer, so a member that stops answering without closing its connections still costs that pod one renewal margin, and with it the pod's partitions.
-- **Supervised coordination runs (etcd resilience)**: A failed coordination attempt — a broken watch stream, a failed etcd write, an exhausted reconcile budget — no longer kills the process; the run supervisors tear the attempt down cleanly, back off exponentially, and rebuild through the same bootstrap that recovers a restarted process, while the data plane keeps serving. The two participant classes recover *differently*, on purpose. A **router** sheds its registration on every failed attempt (its registration is a freeze-quorum vote, and a registered but non-acking router would stall every quorum until the phase deadline); serving through its gap is safe because handoffs created while it is unregistered exclude it from their quorums and the leader-side fences reject its stale forwards. A **pod** does the opposite: its lease session (grant, register, heartbeat) lives *above* the attempt loop and survives attempt failures, so the coordinator keeps seeing a live pod and never runs the dead-owner path against a process that is still serving — the invariant is that serving authority is held exactly as long as the lease-backed registration behind it. This matters because the dead-owner path has no fence: its safety rests entirely on a deregistered owner being unable to produce, so a pod must never voluntarily deregister while it still serves. Only lease loss ends a pod session, self-fencing locally (every held partition released; re-acquisition re-warms) before a fresh session registers anew. A consecutive-failure budget (`ROUTER_RUN_RETRY_BUDGET`, default 10; same default on pods) keeps a genuinely wedged participant on the process-restart path; an attempt that made real progress — a convergence applied, a reconcile pass completed — resets the count, and the reset is keyed on measured progress rather than elapsed time because a time threshold silently exempts every failure detector slower than it (the reconcile budget and the stall watchdog both take a minute to fire), and `personhog_coordination_run_restarts_total{component}` counts every rebuild.
+- **Supervised coordination runs (etcd resilience)**: A failed coordination attempt — a broken watch stream, a failed etcd write, an exhausted reconcile budget — no longer kills the process; the run supervisors tear the attempt down cleanly, back off exponentially, and rebuild through the same bootstrap that recovers a restarted process, while the data plane keeps serving. The two participant classes recover *differently*, on purpose. A **router** sheds its registration on every failed attempt (its registration is a freeze-quorum vote, and a registered but non-acking router would stall every quorum until the phase deadline); serving through its gap is safe because handoffs created while it is unregistered exclude it from their quorums and the leader-side fences reject its stale forwards. A **pod** does the opposite: its lease session (grant, register, heartbeat) lives *above* the attempt loop and survives attempt failures, so the coordinator keeps seeing a live pod and never runs the dead-owner path against a process that is still serving — the invariant is that serving authority is held exactly as long as the lease-backed registration behind it. This matters because the dead-owner path has no fence: its safety rests entirely on a deregistered owner being unable to produce, so a pod must never voluntarily deregister while it still serves. Only lease loss ends a pod session, self-fencing locally (every held partition released; re-acquisition re-warms) before a fresh session registers anew. A consecutive-failure budget (`ROUTER_RUN_RETRY_BUDGET`, default 10; same default on pods) keeps a genuinely wedged participant on the process-restart path. Failures a restart cannot fix back off like any other but spend nothing: etcd could not be reached (Unavailable, DeadlineExceeded, a broken connection, or a watch cancelled because its member has no leader), or the session's lease was lost, by expiring in etcd or by its renewal margin running out. A new attempt is already the remedy for these, through a fresh session and, for connection trouble, member failover; through an outage the participant stays up and rebuilds once etcd answers. An attempt that made real progress — a convergence applied, a reconcile pass completed — resets the count, and the reset is keyed on measured progress rather than elapsed time because a time threshold silently exempts every failure detector slower than it (the reconcile budget and the stall watchdog both take a minute to fire), and `personhog_coordination_run_restarts_total{component}` counts every rebuild.
### Concurrency and atomicity
diff --git a/rust/personhog-coordination/src/changelog.rs b/rust/personhog-coordination/src/changelog.rs
index 194071661927..80eb2e52bc91 100644
--- a/rust/personhog-coordination/src/changelog.rs
+++ b/rust/personhog-coordination/src/changelog.rs
@@ -13,21 +13,43 @@ use crate::store::PersonhogStore;
use crate::types::ChangelogConfig;
use crate::util;
+/// Whether boot keeps polling through `err`: any etcd error, since there is
+/// nothing to do before the config loads, but not a malformed config value.
+fn waits_at_boot(err: &Error) -> bool {
+ err.etcd_error().is_some()
+}
+
/// Polls until `config/changelog` is seeded; it is the only source of the topic and count.
+/// Keeps polling through etcd being unreachable, and returns `None` once `shutdown` fires.
pub async fn load_changelog_config(
store: &PersonhogStore,
poll_interval: Duration,
-) -> Result<ChangelogConfig> {
+ shutdown: &CancellationToken,
+) -> Result<Option<ChangelogConfig>> {
let mut polls: u64 = 0;
loop {
- if let Some(config) = store.get_changelog_config().await? {
- return Ok(config);
- }
- if polls.is_multiple_of(10) {
- tracing::warn!("config/changelog is not seeded; waiting for the control plane");
+ let read = tokio::select! {
+ _ = shutdown.cancelled() => return Ok(None),
+ read = store.get_changelog_config() => read,
+ };
+ match read {
+ Ok(Some(config)) => return Ok(Some(config)),
+ Ok(None) if polls.is_multiple_of(10) => {
+ tracing::warn!("config/changelog is not seeded; waiting for the control plane");
+ }
+ Err(e) if waits_at_boot(&e) => {
+ if polls.is_multiple_of(10) {
+ tracing::warn!(error = %e, "etcd is unreachable; waiting to load config/changelog");
+ }
+ }
+ Err(e) => return Err(e),
+ Ok(None) => {}
}
polls += 1;
- tokio::time::sleep(poll_interval).await;
+ tokio::select! {
+ _ = shutdown.cancelled() => return Ok(None),
+ _ = tokio::time::sleep(poll_interval) => {}
+ }
}
}
@@ -363,6 +385,17 @@ mod tests {
use super::*;
use crate::types::Generation;
+ #[test]
+ fn boot_waits_through_any_etcd_error_but_not_a_malformed_config() {
+ let not_member_level: Error =
+ etcd_client::Error::WatchError("member stopping".to_string()).into();
+ assert!(waits_at_boot(&not_member_level));
+ let malformed: Error = serde_json::from_str::<ChangelogConfig>("{")
+ .unwrap_err()
+ .into();
+ assert!(!waits_at_boot(&malformed));
+ }
+
fn config(partitions: u32, draining: Vec<(u32, u64)>) -> ChangelogConfig {
ChangelogConfig {
topic: "new".to_string(),
diff --git a/rust/personhog-coordination/src/error.rs b/rust/personhog-coordination/src/error.rs
index 1685c896016d..b48ff9ac46c2 100644
--- a/rust/personhog-coordination/src/error.rs
+++ b/rust/personhog-coordination/src/error.rs
@@ -10,6 +10,9 @@ pub enum Error {
#[error("handoff failed for partition {partition}: {reason}")]
HandoffFailed { partition: u32, reason: String },
+
+ #[error("lease renewal margin exhausted: no confirmed renewal in {0:?}")]
+ LeaseMarginExhausted(std::time::Duration),
}
impl From<serde_json::Error> for Error {
@@ -52,6 +55,12 @@ impl Error {
Self::Store(assignment_coordination::error::Error::LeadershipLost)
)
}
+
+ /// Whether a lease ended the session, by expiring in etcd or by its
+ /// renewal margin running out. A new session is the remedy.
+ pub fn is_lease_lost(&self) -> bool {
+ self.is_leadership_lost() || matches!(self, Self::LeaseMarginExhausted(_))
+ }
}
pub type Result<T> = std::result::Result<T, Error>;
diff --git a/rust/personhog-coordination/src/pod.rs b/rust/personhog-coordination/src/pod.rs
index 9b896ce9c57c..a7603591a6a0 100644
--- a/rust/personhog-coordination/src/pod.rs
+++ b/rust/personhog-coordination/src/pod.rs
@@ -458,7 +458,7 @@ impl PodHandle {
/// stall every freeze quorum until the phase deadline.
pub async fn run(&self, cancel: CancellationToken) -> Result<()> {
util::preregister_pod_metrics();
- let mut consecutive_failures: u32 = 0;
+ let mut consecutive_failures = util::RunFailures::default();
// Set by the coordination loop whenever it applies real work
// (a convergence completed); consumed by each failure note to
// decide crash-loop vs fresh failure.
@@ -476,7 +476,10 @@ impl PodHandle {
if !self.note_run_failure(&mut consecutive_failures, &progress, &e) {
return Err(e);
}
- if self.run_backoff(&cancel, consecutive_failures).await {
+ if self
+ .run_backoff(&cancel, consecutive_failures.consecutive())
+ .await
+ {
return Ok(());
}
continue;
@@ -589,7 +592,7 @@ impl PodHandle {
// coordinator may already be reassigning while the pod
// serves, unaware, until the nap ends.
tokio::select! {
- stop = self.run_backoff(&cancel, consecutive_failures) => {
+ stop = self.run_backoff(&cancel, consecutive_failures.consecutive()) => {
if stop {
break;
}
@@ -753,7 +756,10 @@ impl PodHandle {
if !self.note_run_failure(&mut consecutive_failures, &progress, &e) {
return Err(e);
}
- if self.run_backoff(&cancel, consecutive_failures).await {
+ if self
+ .run_backoff(&cancel, consecutive_failures.consecutive())
+ .await
+ {
return Ok(());
}
// New session: fresh lease, fresh registration,
@@ -813,9 +819,14 @@ impl PodHandle {
/// See `util::note_run_failure` for the progress-based reset
/// semantics.
- fn note_run_failure(&self, consecutive: &mut u32, progress: &AtomicBool, err: &Error) -> bool {
+ fn note_run_failure(
+ &self,
+ failures: &mut util::RunFailures,
+ progress: &AtomicBool,
+ err: &Error,
+ ) -> bool {
util::note_run_failure(
- consecutive,
+ failures,
progress,
self.config.run_retry_budget,
"pod",
diff --git a/rust/personhog-coordination/src/routing_table.rs b/rust/personhog-coordination/src/routing_table.rs
index 5a016b4ae902..f6ac042c4e5f 100644
--- a/rust/personhog-coordination/src/routing_table.rs
+++ b/rust/personhog-coordination/src/routing_table.rs
@@ -367,7 +367,7 @@ impl RoutingTable {
) -> Result<()> {
const BACKOFF_CAP: Duration = Duration::from_secs(15);
- let mut consecutive_failures: u32 = 0;
+ let mut consecutive_failures = util::RunFailures::default();
// Set by the coordination loop whenever it does real work;
// consumed by each failure note to decide crash-loop vs fresh
// failure. Arc because the watch loop runs as a spawned task.
@@ -398,7 +398,9 @@ impl RoutingTable {
let backoff = self
.config
.run_retry_backoff
- .saturating_mul(2u32.saturating_pow(consecutive_failures.saturating_sub(1)))
+ .saturating_mul(
+ 2u32.saturating_pow(consecutive_failures.consecutive().saturating_sub(1)),
+ )
.min(BACKOFF_CAP);
tokio::select! {
_ = cancel.cancelled() => return Ok(()),
diff --git a/rust/personhog-coordination/src/util.rs b/rust/personhog-coordination/src/util.rs
index 545556080936..6cd203e2c769 100644
--- a/rust/personhog-coordination/src/util.rs
+++ b/rust/personhog-coordination/src/util.rs
@@ -24,16 +24,38 @@ pub fn new_handoff_id() -> String {
format!("{}-{}", millis, Uuid::new_v4())
}
-/// Record one supervisor-level failure against the consecutive budget;
-/// returns `false` when it is exhausted. Progress resets the count
-/// first, and INVARIANT: progress means *applied* work (a completed
-/// convergence, an applied snapshot or event) — never a read that
-/// changed nothing, which stays available in exactly the wedges the
-/// budget exists for. Measured rather than time-based, because a
-/// "healthy after N seconds" threshold exempts every failure detector
-/// slower than N.
+/// A supervisor's consecutive failures since its last applied work.
+#[derive(Debug, Default)]
+pub(crate) struct RunFailures {
+ /// Every failure; paces the backoff.
+ consecutive: u32,
+ /// Failures that spend the budget.
+ budgeted: u32,
+}
+
+impl RunFailures {
+ pub(crate) fn consecutive(&self) -> u32 {
+ self.consecutive
+ }
+}
+
+/// Whether a coordination failure only says etcd could not be reached.
+pub(crate) fn is_etcd_unavailable(err: &Error) -> bool {
+ err.etcd_error()
+ .is_some_and(assignment_coordination::failover::is_member_level)
+}
+
+/// Whether a restart would fix nothing a new attempt does not: etcd could
+/// not be reached, or a lease was lost. These do not spend the budget.
+fn spends_budget(err: &Error) -> bool {
+ !(is_etcd_unavailable(err) || err.is_lease_lost())
+}
+
+/// Records one supervisor failure; returns `false` once the budget is spent.
+/// INVARIANT: only applied work resets the counts, never a read that changed
+/// nothing. Failures a restart cannot fix spend no budget: see `spends_budget`.
pub(crate) fn note_run_failure(
- consecutive: &mut u32,
+ failures: &mut RunFailures,
progress: &AtomicBool,
budget: u32,
component: &'static str,
@@ -41,9 +63,12 @@ pub(crate) fn note_run_failure(
err: &Error,
) -> bool {
if progress.swap(false, Ordering::SeqCst) {
- *consecutive = 1;
- } else {
- *consecutive += 1;
+ *failures = RunFailures::default();
+ }
+ failures.consecutive += 1;
+ let spends_budget = spends_budget(err);
+ if spends_budget {
+ failures.budgeted += 1;
}
counter!(
"personhog_coordination_run_restarts_total",
@@ -54,11 +79,13 @@ pub(crate) fn note_run_failure(
component,
name,
error = %err,
- consecutive = *consecutive,
+ consecutive = failures.consecutive,
+ budget_spent = failures.budgeted,
budget,
+ spends_budget,
"coordination run failed; rebuilding in place while the data plane keeps serving"
);
- *consecutive < budget
+ failures.budgeted < budget
}
/// Log and count a coordination-run failure for a component with no
@@ -135,11 +162,7 @@ pub async fn run_lease_keepalive(
// however long registration took.
let mut last_renewed = granted_at;
- let margin_exhausted = || {
- Error::invalid_state(format!(
- "lease renewal margin exhausted: no confirmed renewal in {renewal_margin:?}"
- ))
- };
+ let margin_exhausted = || Error::LeaseMarginExhausted(renewal_margin);
'stream: loop {
// (Re)establish the keepalive stream, bounded by the margin.
@@ -305,15 +328,34 @@ pub(crate) fn live_watch_response(
) -> Result<etcd_client::WatchResponse> {
let resp = resp.ok_or_else(|| Error::invalid_state(format!("{what} watch stream ended")))?;
if resp.canceled() {
- return Err(Error::invalid_state(format!(
- "{what} watcher cancelled by etcd (reason: {:?}, compact_revision: {})",
+ return Err(watch_cancelled(
+ what,
resp.cancel_reason(),
resp.compact_revision(),
- )));
+ ));
}
Ok(resp)
}
+/// The cancel reason etcd sends when the member serving a require-leader
+/// watch loses its leader (rpctypes.ErrNoLeader).
+const NO_LEADER_CANCEL_REASON: &str = "etcdserver: no leader";
+
+/// A member without a leader cancels the watch rather than failing it, so
+/// that case maps to the same Unavailable status etcd answers a
+/// require-leader request with.
+fn watch_cancelled(what: &str, reason: &str, compact_revision: i64) -> Error {
+ if reason == NO_LEADER_CANCEL_REASON {
+ return assignment_coordination::failover::no_leader_error(format!(
+ "{what} watcher cancelled by etcd: {reason}"
+ ))
+ .into();
+ }
+ Error::invalid_state(format!(
+ "{what} watcher cancelled by etcd (reason: {reason:?}, compact_revision: {compact_revision})"
+ ))
+}
+
/// Count one resolution that found no record — not one record lost: a
/// single missing record resolves once per frozen partition per pass,
/// so read it as a rate (how much work is degraded to the fallback),
@@ -529,7 +571,7 @@ mod tests {
use std::collections::HashSet;
use std::sync::atomic::{AtomicBool, Ordering};
- use super::{new_handoff_id, note_run_failure};
+ use super::{new_handoff_id, note_run_failure, watch_cancelled, RunFailures};
use crate::error::Error;
/// Losing the election lease ends a leadership term but says nothing
@@ -552,22 +594,22 @@ mod tests {
#[test]
fn applied_work_clears_the_failure_count() {
let progress = AtomicBool::new(false);
- let err = Error::invalid_state("etcd unreachable");
- let mut consecutive = 0u32;
+ let err = Error::invalid_state("list_handoffs failed");
+ let mut failures = RunFailures::default();
for attempt in 1..3 {
assert!(
- note_run_failure(&mut consecutive, &progress, 3, "pod", "p", &err),
+ note_run_failure(&mut failures, &progress, 3, "pod", "p", &err),
"attempt {attempt} is within budget"
);
}
progress.store(true, Ordering::SeqCst);
assert!(
- note_run_failure(&mut consecutive, &progress, 3, "pod", "p", &err),
+ note_run_failure(&mut failures, &progress, 3, "pod", "p", &err),
"applied work resets the count"
);
- assert_eq!(consecutive, 1);
+ assert_eq!(failures.consecutive(), 1);
}
/// And it must escalate when nothing succeeds in between — this is
@@ -577,10 +619,10 @@ mod tests {
fn an_unbroken_run_of_failures_exhausts_the_budget() {
let progress = AtomicBool::new(false);
let err = Error::invalid_state("list_handoffs failed");
- let mut consecutive = 0u32;
+ let mut failures = RunFailures::default();
assert!(note_run_failure(
- &mut consecutive,
+ &mut failures,
&progress,
2,
"pod",
@@ -588,11 +630,73 @@ mod tests {
&err
));
assert!(
- !note_run_failure(&mut consecutive, &progress, 2, "pod", "p", &err),
+ !note_run_failure(&mut failures, &progress, 2, "pod", "p", &err),
"the budget is spent, so the caller must stop retrying"
);
}
+ #[test]
+ fn failures_a_restart_cannot_fix_back_off_without_spending_the_budget() {
+ let unfixable_by_restart: Vec<(&str, Error)> = vec![
+ (
+ "etcd unreachable",
+ etcd_client::Error::IoError(std::io::Error::from(
+ std::io::ErrorKind::ConnectionRefused,
+ ))
+ .into(),
+ ),
+ ("lease expired in etcd", Error::leadership_lost()),
+ (
+ "renewal margin exhausted",
+ Error::LeaseMarginExhausted(std::time::Duration::from_secs(20)),
+ ),
+ (
+ "watch cancelled for no leader",
+ watch_cancelled("routing table", "etcdserver: no leader", 0),
+ ),
+ ];
+ for (case, err) in unfixable_by_restart {
+ let progress = AtomicBool::new(false);
+ let mut failures = RunFailures::default();
+ for _ in 0..50 {
+ assert!(
+ note_run_failure(&mut failures, &progress, 2, "pod", "p", &err),
+ "{case} spent the budget"
+ );
+ }
+ assert_eq!(
+ failures.consecutive(),
+ 50,
+ "{case}: the backoff still grows"
+ );
+ }
+
+ let wedges = [
+ Error::invalid_state("list_handoffs failed"),
+ watch_cancelled(
+ "routing table",
+ "etcdserver: mvcc: required revision has been compacted",
+ 42,
+ ),
+ ];
+ for wedge in wedges {
+ let progress = AtomicBool::new(false);
+ let mut failures = RunFailures::default();
+ assert!(note_run_failure(
+ &mut failures,
+ &progress,
+ 2,
+ "pod",
+ "p",
+ &wedge
+ ));
+ assert!(
+ !note_run_failure(&mut failures, &progress, 2, "pod", "p", &wedge),
+ "{wedge} must still exhaust the budget"
+ );
+ }
+ }
+
/// Quorum correlation and cancellation detection hang off id
/// uniqueness; ids minted in the same instant (a handoff cancelled
/// and recreated within one millisecond) must never collide.
diff --git a/rust/personhog-coordination/tests/changelog.rs b/rust/personhog-coordination/tests/changelog.rs
index 282c77bfa8a0..483d5639051b 100644
--- a/rust/personhog-coordination/tests/changelog.rs
+++ b/rust/personhog-coordination/tests/changelog.rs
@@ -9,7 +9,7 @@ use personhog_coordination::types::{ChangelogConfig, Generation};
use tokio::sync::mpsc;
use tokio_util::sync::CancellationToken;
-use common::{store_at, test_store, FlakyProxy, ETCD_ENDPOINT};
+use common::{store_at, test_store, FlakyProxy, ETCD_ENDPOINT, WAIT_TIMEOUT};
#[tokio::test]
async fn a_second_seed_is_refused() {
@@ -47,14 +47,65 @@ async fn boot_polls_until_the_config_is_seeded() {
});
let loaded = tokio::time::timeout(
Duration::from_secs(5),
- load_changelog_config(&store, Duration::from_millis(20)),
+ load_changelog_config(&store, Duration::from_millis(20), &CancellationToken::new()),
)
.await
.expect("boot polls until the config is seeded")
- .unwrap();
+ .unwrap()
+ .expect("no shutdown was requested");
assert_eq!((loaded.partitions, loaded.epoch), (64, 1));
}
+#[tokio::test]
+async fn boot_waits_out_an_unreachable_etcd() {
+ let seeded = test_store("changelog-boot-outage").await;
+ seeded
+ .seed_changelog_config("personhog_updates_64", 64, "personhog-writer")
+ .await
+ .unwrap();
+ let proxy = FlakyProxy::start("127.0.0.1:2379").await;
+ proxy.set_blackholed(true);
+ let store = store_at(&proxy.endpoint, seeded.inner().prefix()).await;
+
+ let boot = tokio::spawn(async move {
+ load_changelog_config(&store, Duration::from_millis(20), &CancellationToken::new()).await
+ });
+ tokio::time::sleep(Duration::from_secs(1)).await;
+ assert!(
+ !boot.is_finished(),
+ "boot keeps waiting while etcd is unreachable"
+ );
+
+ proxy.set_blackholed(false);
+ let loaded = tokio::time::timeout(WAIT_TIMEOUT, boot)
+ .await
+ .expect("boot loads the config once etcd is back")
+ .unwrap()
+ .unwrap()
+ .expect("no shutdown was requested");
+ assert_eq!(loaded.partitions, 64);
+}
+
+#[tokio::test]
+async fn boot_stops_waiting_on_shutdown() {
+ let store = test_store("changelog-boot-shutdown").await;
+ let shutdown = CancellationToken::new();
+ let waiting = {
+ let shutdown = shutdown.clone();
+ tokio::spawn(async move {
+ load_changelog_config(&store, Duration::from_millis(20), &shutdown).await
+ })
+ };
+ tokio::time::sleep(Duration::from_millis(100)).await;
+ shutdown.cancel();
+ let loaded = tokio::time::timeout(Duration::from_secs(2), waiting)
+ .await
+ .expect("shutdown ends the wait")
+ .unwrap()
+ .unwrap();
+ assert!(loaded.is_none());
+}
+
/// What a loop starts from before it has adopted anything.
fn nothing_adopted() -> ChangelogConfig {
ChangelogConfig {
diff --git a/rust/personhog-identity/src/main.rs b/rust/personhog-identity/src/main.rs
index 8446092f723c..30db8d126c81 100644
--- a/rust/personhog-identity/src/main.rs
+++ b/rust/personhog-identity/src/main.rs
@@ -155,9 +155,17 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
.await
.expect("Failed to connect to etcd");
let store = Arc::new(PersonhogStore::new(etcd_store));
- let loaded = load_changelog_config(&store, CHANGELOG_BOOT_POLL_INTERVAL)
- .await
- .expect("Failed to load the changelog config from etcd");
+ // No signal handler is installed until the lifecycle monitor starts,
+ // so SIGTERM ends the process during this wait.
+ let Some(loaded) = load_changelog_config(
+ &store,
+ CHANGELOG_BOOT_POLL_INTERVAL,
+ &CancellationToken::new(),
+ )
+ .await
+ .expect("Failed to load the changelog config from etcd") else {
+ return Ok(());
+ };
tracing::info!(
num_partitions = loaded.partitions,
epoch = loaded.epoch,
diff --git a/rust/personhog-leader/src/main.rs b/rust/personhog-leader/src/main.rs
index db0c09d6bfbb..f470fa555eda 100644
--- a/rust/personhog-leader/src/main.rs
+++ b/rust/personhog-leader/src/main.rs
@@ -367,9 +367,15 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
let store = Arc::new(PersonhogStore::new(etcd_store));
// The same source the router hashes against.
- let loaded = load_changelog_config(&store, CHANGELOG_BOOT_POLL_INTERVAL)
- .await
- .expect("Failed to load the changelog config from etcd");
+ let Some(loaded) = load_changelog_config(
+ &store,
+ CHANGELOG_BOOT_POLL_INTERVAL,
+ &changelog_handle.shutdown_token(),
+ )
+ .await
+ .expect("Failed to load the changelog config from etcd") else {
+ return Ok(());
+ };
let changelog_topic = loaded.topic.clone();
let num_partitions = loaded.partitions;
let topic_partitions =
diff --git a/rust/personhog-router/src/main.rs b/rust/personhog-router/src/main.rs
index dfd599040890..08751f27c71c 100644
--- a/rust/personhog-router/src/main.rs
+++ b/rust/personhog-router/src/main.rs
@@ -258,9 +258,17 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
.expect("Failed to connect to etcd");
let store = Arc::new(PersonhogStore::new(etcd_store));
- let loaded = load_changelog_config(&store, CHANGELOG_BOOT_POLL_INTERVAL)
- .await
- .expect("Failed to load the changelog config from etcd");
+ let boot_shutdown = changelog_handle
+ .as_ref()
+ .expect("changelog handle must be registered in leader mode")
+ .shutdown_token();
+ let Some(loaded) =
+ load_changelog_config(&store, CHANGELOG_BOOT_POLL_INTERVAL, &boot_shutdown)
+ .await
+ .expect("Failed to load the changelog config from etcd")
+ else {
+ return Ok(());
+ };
let num_partitions = loaded.partitions;
tracing::info!(
num_partitions,
diff --git a/rust/personhog-writer/src/main.rs b/rust/personhog-writer/src/main.rs
index 659541589f8b..89d04660c269 100644
--- a/rust/personhog-writer/src/main.rs
+++ b/rust/personhog-writer/src/main.rs
@@ -61,7 +61,17 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
})
.await?;
let store = Arc::new(PersonhogStore::new(store));
- let loaded = load_changelog_config(&store, CHANGELOG_BOOT_POLL_INTERVAL).await?;
+ // No signal handler is installed until the lifecycle monitor starts,
+ // so SIGTERM ends the process during this wait.
+ let Some(loaded) = load_changelog_config(
+ &store,
+ CHANGELOG_BOOT_POLL_INTERVAL,
+ &CancellationToken::new(),
+ )
+ .await?
+ else {
+ return Ok(());
+ };
if loaded.writer_group != config.kafka_consumer_group {
return Err(format!(
"config/changelog names writer group {} but KAFKA_CONSUMER_GROUP is {}",