:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(personhog): recover coordinator and router registrations after an etcd wipe (#123626)
Zach Browna9bd34eb85f2
diff --git a/rust/personhog-coordination/README.md b/rust/personhog-coordination/README.mdindex dfa7171b91b8..fcb7dc4dd7a1 100644--- a/rust/personhog-coordination/README.md+++ b/rust/personhog-coordination/README.md@@ -93,6 +93,7 @@ Several failure modes are handled explicitly:- **Level-triggered reconciliation everywhere**: Watches are the latency path; every participant also has a truth path that re-derives its state from a fresh snapshot. The coordinator's tick re-evaluates handoffs; the router's reconcile pass (an arm of its watch loop, `ROUTER_RECONCILE_SECS`, default 5 s) re-asserts stashes and freeze acks, converges its table to the assignment authority (dropping routes at or above the live partition count), and drains any stash whose partition has no handoff: to its assignment owner, or with no owner when no assignment names it — which heals a missed Complete, a silently dead watch stream, or a deleted handoff; the pod's tick re-converges every involved partition, including locally-held state the durable world no longer mentions. Every reconcile action is idempotent against the event path. Reconcile failures are tolerated, not fatal: a failed pass leaves the participant exactly as stale as the previous tick (the watch-driven steady state), so brief etcd blips never restart the fleet — sustained outage self-fences through the lease keepalive instead. A consecutive-failure budget (`ROUTER_RECONCILE_FAILURE_BUDGET`, default 12 ticks; same default on pods) bounds the partial mode where snapshot reads fail while the lease stays healthy, and `personhog_coordination_reconcile_failures_total{component}` counts every failure.- **Warm provenance**: A pod records what each partition warm was installed for (serving, or a specific handoff id). Warming honors only a warm installed for the *current* handoff — one left over from an earlier ownership era is released and rebuilt, because its cache can hold values that predate writes the changelog accepted since, and a fresh warm replays only from the writer's committed offset. Without this, a returning owner whose `Released` convergence never ran could ack a warm it never performed and lose acknowledged writes — a counterexample the stateright model found and now pins.- **Coordinator restart**: On startup, the coordinator runs `reconcile_pending_handoffs`. For every existing handoff it applies cleanup if `Complete`, otherwise calls `check_phase_advance` to nudge the state machine. This handles the case where acks were written before the coordinator was up — the ack-watch only fires on Put events, so without reconcile those acks would be invisible.+- **Lost coordination keys**: The coordinator watches its own election key and abdicates, then campaigns again, when the key is deleted or rewritten under another lease; it never deletes the new holder's key. A router watches its own registration and, when it is deleted, ends the attempt and registers again on a fresh lease. Each watcher keeps one watch open and re-reads the key only after a failed call or an ended watch.- **Reconcile tick**: Beyond the startup reconcile, the coordinator re-evaluates every in-flight handoff on a fixed interval (`reconcile_interval`, default 5 s). This is the liveness backstop for state changes that fire no watched event — e.g. a router departing (nothing watches router registrations) newly satisfying a freeze quorum. Everything the tick drives is idempotent: phase transitions use CAS and cleanup tolerates already-deleted records.- **Lease loss (self-fencing)**: Connection trouble and lease loss are different things. A broken or silent keepalive stream is a fact about one connection — the lease lives in etcd's replicated keyspace and stays valid until its TTL passes without renewal — so ambiguous keepalive failures rebuild the stream and retry (the store moves to another member when its own keeps failing; see `assignment_coordination::failover`) for as long as the last confirmed renewal is recent; a routine single-member restart costs some retries (`personhog_coordination_keepalive_retries_total{component}`), not a fence. The keepalive declares loss on exactly two signals: etcd answering a round with TTL ≤ 0 (authoritative — the lease is revoked or expired), or the renewal margin running out (two thirds of the TTL without a confirmed renewal — the final third is reserved so the fence completes before the coordinator can possibly treat the lease as expired; retrying past it would gamble split-brain against availability). On loss the component ends the coordination attempt immediately, skipping the graceful drain: the coordinator already considers it dead and is reassigning its partitions via the dead-owner path, so a graceful drain would only race that. A pod self-fences *locally* before anything else: every held partition is drained and, once quiesced, released (dropping its cache and serving authority — re-acquisition always re-warms). A partition whose drain cannot quiesce is deliberately **not** released — release is what unfences admissions and drops the cache, and a leaseless pod must do neither over live in-flight work — so it stays fenced and held, in-place recovery is refused, and the process restart clears it by death. A zombie owner therefore never keeps serving partitions the protocol has reassigned beyond the window that restart closes. Lease loss is observed at every await the session makes while holding a lease — the attempt select, the backoff nap between failed attempts, and the graceful drain all race the keepalive — so once the keepalive declares loss, the fence is never deferred by supervisor state. Every other session end fences too — budget exhaustion and shutdown revoke the lease, which deletes the registration instantly, so the local fence runs first (a no-op after a completed graceful drain) and the dead-owner path never meets a still-serving owner. On lease loss the fence's drain budget is the runway the keepalive margin reserved (a third of the TTL), not the full graceful drain timeout; overshoot poisons recovery and the process restart clears stragglers by death.- **etcd member failover**: Each process keeps one connection to whichever etcd member it reached first. Three member-level failures in a row (Unavailable, DeadlineExceeded, a broken connection), or one call left unanswered for at least 2 s, start a switch: the store probes the other members from its cached member list with a linearizable read and moves new calls to the first that answers. Requests carry etcd's require-leader flag, so a member without a leader rejects them at once. Watches and keepalive streams already open stay on their connection until it closes or etcd cancels them. A renewal waits up to the heartbeat interval for an answer, so a member that stops answering without closing its connections still costs that pod one renewal margin, and with it the pod's partitions.diff --git a/rust/personhog-coordination/src/coordinator.rs b/rust/personhog-coordination/src/coordinator.rsindex 766e14fdb678..34be4a44e0d8 100644--- a/rust/personhog-coordination/src/coordinator.rs+++ b/rust/personhog-coordination/src/coordinator.rs@@ -627,6 +627,21 @@ impl Coordinator {// still runs, which is what spares the successor the TTL._ = cancel.cancelled() => Ok(()),_ = lease_lost.cancelled() => Err(Error::leadership_lost()),+ // A live lease does not prove the seat: a deleted election key+ // lets another candidate win while this one keeps coordinating.+ lost = util::await_held_key_lost(+ &self.store,+ util::HeldKey::Leader,+ lease_id,+ self.config.standby_poll_interval,+ &cancel,+ ) => match lost {+ Some(how) => {+ tracing::error!(name = %self.config.name, how, "election key lost; abdicating");+ Err(Error::leadership_lost())+ }+ None => Ok(()),+ },result = self.run_coordination_loop(cancel.clone()) => result,};diff --git a/rust/personhog-coordination/src/routing_table.rs b/rust/personhog-coordination/src/routing_table.rsindex f6ac042c4e5f..6e0cb80c60cb 100644--- a/rust/personhog-coordination/src/routing_table.rs+++ b/rust/personhog-coordination/src/routing_table.rs@@ -563,6 +563,31 @@ impl RoutingTable {});}+ // A deleted registration leaves this router out of every freeze+ // quorum until it re-registers, which ending the run does.+ {+ let store = Arc::clone(&self.store);+ let router_name = self.config.router_name.clone();+ let retry_interval = self.config.heartbeat_interval;+ let token = cancel.child_token();+ tasks.spawn(async move {+ let lost = util::await_held_key_lost(+ &store,+ util::HeldKey::Router(&router_name),+ lease_id,+ retry_interval,+ &token,+ )+ .await;+ match lost {+ Some(how) => Err(Error::invalid_state(format!(+ "registration of router {router_name} {how}; re-registering"+ ))),+ None => Ok(()),+ }+ });+ }+{let addresses = Arc::clone(&self.addresses);let token = cancel.child_token();diff --git a/rust/personhog-coordination/src/store.rs b/rust/personhog-coordination/src/store.rsindex 4a2fa6ca65a9..fd4a88bf0f81 100644--- a/rust/personhog-coordination/src/store.rs+++ b/rust/personhog-coordination/src/store.rs@@ -360,6 +360,25 @@ impl PersonhogStore {Ok(self.inner.put(&key, router, Some(lease_id)).await?)}+ pub async fn get_router_with_revision(+ &self,+ router_name: &str,+ ) -> Result<(Option<RegisteredRouter>, i64)> {+ count_call("get_router_with_revision");+ let key = self.key(StoreKey::Router(router_name));+ Ok(self.inner.get_with_revision(&key).await?)+ }++ pub async fn watch_router_from(+ &self,+ router_name: &str,+ start_revision: i64,+ ) -> Result<WatchStream> {+ count_call("watch_router_from");+ let key = self.key(StoreKey::Router(router_name));+ Ok(self.inner.watch_key_from(&key, start_revision).await?)+ }+pub async fn list_routers(&self) -> Result<Vec<RegisteredRouter>> {count_call("list_routers");let key = self.key(StoreKey::RoutersPrefix);diff --git a/rust/personhog-coordination/src/util.rs b/rust/personhog-coordination/src/util.rsindex 6cd203e2c769..b162912c8ca3 100644--- a/rust/personhog-coordination/src/util.rs+++ b/rust/personhog-coordination/src/util.rs@@ -2,6 +2,7 @@ use std::sync::atomic::{AtomicBool, Ordering};use std::sync::Arc;use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};+use etcd_client::{EventType, WatchStream};use metrics::{counter, histogram};use tokio_util::sync::CancellationToken;use uuid::Uuid;@@ -113,6 +114,118 @@ pub(crate) fn record_run_failure();}+#[derive(Debug, Clone, Copy)]+pub(crate) enum HeldKey<'a> {+ Leader,+ Router(&'a str),+}++impl HeldKey<'_> {+ fn label(self) -> &'static str {+ match self {+ HeldKey::Leader => "leader",+ HeldKey::Router(_) => "router",+ }+ }++ /// (exists, the lease the value records if it records one, read revision)+ async fn read(self, store: &PersonhogStore) -> Result<(bool, Option<i64>, i64)> {+ match self {+ HeldKey::Leader => {+ let (leader, revision) = store.get_leader_with_revision().await?;+ Ok((+ leader.is_some(),+ leader.map(|leader| leader.lease_id),+ revision,+ ))+ }+ HeldKey::Router(name) => {+ let (router, revision) = store.get_router_with_revision(name).await?;+ Ok((router.is_some(), None, revision))+ }+ }+ }++ async fn watch_from(self, store: &PersonhogStore, start_revision: i64) -> Result<WatchStream> {+ match self {+ HeldKey::Leader => store.watch_leader_from(start_revision).await,+ HeldKey::Router(name) => store.watch_router_from(name, start_revision).await,+ }+ }+}++/// Resolve once `key` is deleted or rewritten under a lease other than+/// `lease_id`; `None` on cancellation. One watch stays open; a failed call or+/// an ended watch re-reads after `retry_interval`.+pub(crate) async fn await_held_key_lost(+ store: &PersonhogStore,+ key: HeldKey<'_>,+ lease_id: i64,+ retry_interval: Duration,+ cancel: &CancellationToken,+) -> Option<&'static str> {+ let lost = |how: &'static str| {+ counter!(+ "personhog_coordination_held_key_lost_total",+ "key" => key.label(),+ "how" => how+ )+ .increment(1);+ Some(how)+ };+ loop {+ let earliest_retry = tokio::time::Instant::now() + retry_interval;+ let read = tokio::select! {+ _ = cancel.cancelled() => return None,+ read = key.read(store) => read,+ };+ let revision = match read {+ Ok((false, _, _)) => return lost("deleted"),+ Ok((true, Some(holder), _)) if holder != lease_id => return lost("replaced"),+ Ok((true, _, revision)) => Some(revision),+ Err(e) => {+ tracing::warn!(key = key.label(), error = %e, "held key read failed; retrying");+ None+ }+ };+ let stream = match revision {+ Some(revision) => tokio::select! {+ _ = cancel.cancelled() => return None,+ stream = key.watch_from(store, revision + 1) => stream.map_err(|e| {+ tracing::warn!(key = key.label(), error = %e, "held key watch failed; retrying");+ }).ok(),+ },+ None => None,+ };+ if let Some(mut stream) = stream {+ loop {+ let message = tokio::select! {+ _ = cancel.cancelled() => return None,+ message = stream.message() => message,+ };+ let Ok(Some(response)) = message else { break };+ if response.canceled() {+ break;+ }+ for event in response.events() {+ match event.event_type() {+ EventType::Delete => return lost("deleted"),+ EventType::Put => {+ if event.kv().is_some_and(|kv| kv.lease() != lease_id) {+ return lost("replaced");+ }+ }+ }+ }+ }+ }+ tokio::select! {+ _ = cancel.cancelled() => return None,+ _ = tokio::time::sleep_until(earliest_retry) => {}+ }+ }+}+/// Maintain a lease keepalive until cancelled, treating connection/// trouble and lease loss as the different things they are. A broken or/// silent keepalive stream is evidence about one connection — the lease@@ -426,6 +539,14 @@ pub fn preregister_coordinator_metrics() {metrics::counter!("personhog_coordination_elections_won_total").increment(0);metrics::counter!("personhog_coordination_election_campaigns_total").increment(0);metrics::counter!("personhog_coordination_abdications_total").increment(0);+ for how in ["deleted", "replaced"] {+ metrics::counter!(+ "personhog_coordination_held_key_lost_total",+ "key" => "leader",+ "how" => how+ )+ .increment(0);+ }metrics::counter!("personhog_coordination_election_observation_failures_total").increment(0);metrics::counter!("personhog_coordination_election_watch_interruptions_total").increment(0);for stage in ["list", "delete"] {@@ -564,6 +685,14 @@ pub fn preregister_router_coordination_metrics() {.increment(0);}metrics::counter!("personhog_coordination_freeze_acks_written_total").increment(0);+ for how in ["deleted", "replaced"] {+ metrics::counter!(+ "personhog_coordination_held_key_lost_total",+ "key" => "router",+ "how" => how+ )+ .increment(0);+ }}#[cfg(test)]diff --git a/rust/personhog-coordination/tests/protocol_hardening.rs b/rust/personhog-coordination/tests/protocol_hardening.rsindex 657635c17e60..f0cdaed48725 100644--- a/rust/personhog-coordination/tests/protocol_hardening.rs+++ b/rust/personhog-coordination/tests/protocol_hardening.rs@@ -38,8 +38,8 @@ use personhog_coordination::routing_table::{RoutingTable, RoutingTableConfig, Stuse personhog_coordination::store::PersonhogStore;use personhog_coordination::strategy::StickyBalancedStrategy;use personhog_coordination::types::{- AssignmentStatus, HandoffPhase, HandoffState, PartitionAssignment, PodStatus, RegisteredPod,- RegisteredRouter, RouterFreezeAck,+ AssignmentStatus, HandoffPhase, HandoffState, LeaderInfo, PartitionAssignment, PodStatus,+ RegisteredPod, RegisteredRouter, RouterFreezeAck,};/// Write a handoff record directly, bypassing the coordinator — gives the@@ -295,6 +295,84 @@ async fn coordinator_abdicates_and_recampaigns_when_election_lease_revoked() {.expect("graceful cancellation must exit cleanly after an abdication cycle");}+/// A leader must abdicate when its election key is deleted or taken under+/// another lease. With one candidate, a new lease on the key proves it did.+#[tokio::test]+async fn a_coordinator_whose_election_key_is_lost_abdicates() {+ for replace in [false, true] {+ let (store, prefix) = test_store_with_prefix("coordinator-key-lost").await;+ let key = format!("{prefix}coordinator/leader");+ let cancel = CancellationToken::new();+ let handle = start_coordinator_named(+ Arc::clone(&store),+ "key-lost-coordinator-0",+ 60,+ Arc::new(StickyBalancedStrategy),+ cancel.clone(),+ );++ let check_store = Arc::clone(&store);+ wait_for_condition(WAIT_TIMEOUT, POLL_INTERVAL, || {+ let store = Arc::clone(&check_store);+ async move { store.get_leader().await.ok().flatten().is_some() }+ })+ .await;+ let first = store.get_leader().await.unwrap().unwrap();++ let mut client = etcd_client::Client::connect([ETCD_ENDPOINT], None)+ .await+ .unwrap();+ let intruder = if replace {+ let lease = client.lease_grant(60, None).await.unwrap().id();+ let value = serde_json::to_vec(&LeaderInfo {+ holder: "intruder".to_string(),+ lease_id: lease,+ })+ .unwrap();+ client+ .put(+ key.as_str(),+ value,+ Some(etcd_client::PutOptions::new().with_lease(lease)),+ )+ .await+ .unwrap();+ Some(lease)+ } else {+ client.delete(key.as_str(), None).await.unwrap();+ None+ };+ if let Some(lease) = intruder {+ tokio::time::sleep(Duration::from_secs(2)).await;+ client.lease_revoke(lease).await.unwrap();+ }++ let check_store = Arc::clone(&store);+ let first_lease = first.lease_id;+ wait_for_condition(Duration::from_secs(10), POLL_INTERVAL, || {+ let store = Arc::clone(&check_store);+ async move {+ store+ .get_leader()+ .await+ .ok()+ .flatten()+ .is_some_and(|leader| {+ leader.holder == "key-lost-coordinator-0" && leader.lease_id != first_lease+ })+ }+ })+ .await;++ cancel.cancel();+ tokio::time::timeout(Duration::from_secs(10), handle)+ .await+ .expect("coordinator must exit promptly on cancellation")+ .expect("coordinator task must not panic")+ .expect("graceful cancellation must exit cleanly after an abdication");+ }+}+// ============================================================// Cancelled handoffs converge on the durable state// ============================================================@@ -3693,6 +3771,52 @@ async fn a_deleted_registration_starts_a_new_session() {cancel.cancel();}+/// A router whose registration is deleted under a live lease must register again.+#[tokio::test]+async fn a_router_whose_registration_is_deleted_registers_again() {+ let (store, prefix) = test_store_with_prefix("router-registration-lost").await;+ let cancel = CancellationToken::new();+ let _router = start_router_with_lease_ttl(+ Arc::clone(&store),+ "registration-lost-router",+ 60,+ cancel.clone(),+ );++ let registered = |store: Arc<PersonhogStore>| async move {+ store+ .list_routers()+ .await+ .map(|routers| {+ routers+ .iter()+ .any(|r| r.router_name == "registration-lost-router")+ })+ .unwrap_or(false)+ };+ let check = Arc::clone(&store);+ wait_for_condition(WAIT_TIMEOUT, POLL_INTERVAL, || {+ registered(Arc::clone(&check))+ })+ .await;++ let mut client = etcd_client::Client::connect([ETCD_ENDPOINT], None)+ .await+ .unwrap();+ client+ .delete(format!("{prefix}routers/registration-lost-router"), None)+ .await+ .unwrap();++ let check = Arc::clone(&store);+ wait_for_condition(Duration::from_secs(20), POLL_INTERVAL, || {+ registered(Arc::clone(&check))+ })+ .await;++ cancel.cancel();+}+/// The watch is a prefix watch, so it sees every deletion under/// `pods/`. Only the pod's own registration key may cost it the session/// — matching anything looser (say, a final path segment) would let an