Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(personhog): recover coordinator and router registrations after an etcd wipe (#123626)

Zach Browna9bd34eb85f2
diff --git a/rust/personhog-coordination/README.md b/rust/personhog-coordination/README.md
index dfa7171b91b8..fcb7dc4dd7a1 100644
--- a/rust/personhog-coordination/README.md
+++ b/rust/personhog-coordination/README.md
@@ -93,6 +93,7 @@ Several failure modes are handled explicitly:
- **Level-triggered reconciliation everywhere**: Watches are the latency path; every participant also has a truth path that re-derives its state from a fresh snapshot. The coordinator's tick re-evaluates handoffs; the router's reconcile pass (an arm of its watch loop, `ROUTER_RECONCILE_SECS`, default 5 s) re-asserts stashes and freeze acks, converges its table to the assignment authority (dropping routes at or above the live partition count), and drains any stash whose partition has no handoff: to its assignment owner, or with no owner when no assignment names it — which heals a missed Complete, a silently dead watch stream, or a deleted handoff; the pod's tick re-converges every involved partition, including locally-held state the durable world no longer mentions. Every reconcile action is idempotent against the event path. Reconcile failures are tolerated, not fatal: a failed pass leaves the participant exactly as stale as the previous tick (the watch-driven steady state), so brief etcd blips never restart the fleet — sustained outage self-fences through the lease keepalive instead. A consecutive-failure budget (`ROUTER_RECONCILE_FAILURE_BUDGET`, default 12 ticks; same default on pods) bounds the partial mode where snapshot reads fail while the lease stays healthy, and `personhog_coordination_reconcile_failures_total{component}` counts every failure.
- **Warm provenance**: A pod records what each partition warm was installed for (serving, or a specific handoff id). Warming honors only a warm installed for the *current* handoff — one left over from an earlier ownership era is released and rebuilt, because its cache can hold values that predate writes the changelog accepted since, and a fresh warm replays only from the writer's committed offset. Without this, a returning owner whose `Released` convergence never ran could ack a warm it never performed and lose acknowledged writes — a counterexample the stateright model found and now pins.
- **Coordinator restart**: On startup, the coordinator runs `reconcile_pending_handoffs`. For every existing handoff it applies cleanup if `Complete`, otherwise calls `check_phase_advance` to nudge the state machine. This handles the case where acks were written before the coordinator was up — the ack-watch only fires on Put events, so without reconcile those acks would be invisible.
+- **Lost coordination keys**: The coordinator watches its own election key and abdicates, then campaigns again, when the key is deleted or rewritten under another lease; it never deletes the new holder's key. A router watches its own registration and, when it is deleted, ends the attempt and registers again on a fresh lease. Each watcher keeps one watch open and re-reads the key only after a failed call or an ended watch.
- **Reconcile tick**: Beyond the startup reconcile, the coordinator re-evaluates every in-flight handoff on a fixed interval (`reconcile_interval`, default 5 s). This is the liveness backstop for state changes that fire no watched event — e.g. a router departing (nothing watches router registrations) newly satisfying a freeze quorum. Everything the tick drives is idempotent: phase transitions use CAS and cleanup tolerates already-deleted records.
- **Lease loss (self-fencing)**: Connection trouble and lease loss are different things. A broken or silent keepalive stream is a fact about one connection — the lease lives in etcd's replicated keyspace and stays valid until its TTL passes without renewal — so ambiguous keepalive failures rebuild the stream and retry (the store moves to another member when its own keeps failing; see `assignment_coordination::failover`) for as long as the last confirmed renewal is recent; a routine single-member restart costs some retries (`personhog_coordination_keepalive_retries_total{component}`), not a fence. The keepalive declares loss on exactly two signals: etcd answering a round with TTL ≤ 0 (authoritative — the lease is revoked or expired), or the renewal margin running out (two thirds of the TTL without a confirmed renewal — the final third is reserved so the fence completes before the coordinator can possibly treat the lease as expired; retrying past it would gamble split-brain against availability). On loss the component ends the coordination attempt immediately, skipping the graceful drain: the coordinator already considers it dead and is reassigning its partitions via the dead-owner path, so a graceful drain would only race that. A pod self-fences *locally* before anything else: every held partition is drained and, once quiesced, released (dropping its cache and serving authority — re-acquisition always re-warms). A partition whose drain cannot quiesce is deliberately **not** released — release is what unfences admissions and drops the cache, and a leaseless pod must do neither over live in-flight work — so it stays fenced and held, in-place recovery is refused, and the process restart clears it by death. A zombie owner therefore never keeps serving partitions the protocol has reassigned beyond the window that restart closes. Lease loss is observed at every await the session makes while holding a lease — the attempt select, the backoff nap between failed attempts, and the graceful drain all race the keepalive — so once the keepalive declares loss, the fence is never deferred by supervisor state. Every other session end fences too — budget exhaustion and shutdown revoke the lease, which deletes the registration instantly, so the local fence runs first (a no-op after a completed graceful drain) and the dead-owner path never meets a still-serving owner. On lease loss the fence's drain budget is the runway the keepalive margin reserved (a third of the TTL), not the full graceful drain timeout; overshoot poisons recovery and the process restart clears stragglers by death.
- **etcd member failover**: Each process keeps one connection to whichever etcd member it reached first. Three member-level failures in a row (Unavailable, DeadlineExceeded, a broken connection), or one call left unanswered for at least 2 s, start a switch: the store probes the other members from its cached member list with a linearizable read and moves new calls to the first that answers. Requests carry etcd's require-leader flag, so a member without a leader rejects them at once. Watches and keepalive streams already open stay on their connection until it closes or etcd cancels them. A renewal waits up to the heartbeat interval for an answer, so a member that stops answering without closing its connections still costs that pod one renewal margin, and with it the pod's partitions.
diff --git a/rust/personhog-coordination/src/coordinator.rs b/rust/personhog-coordination/src/coordinator.rs
index 766e14fdb678..34be4a44e0d8 100644
--- a/rust/personhog-coordination/src/coordinator.rs
+++ b/rust/personhog-coordination/src/coordinator.rs
@@ -627,6 +627,21 @@ impl Coordinator {
// still runs, which is what spares the successor the TTL.
_ = cancel.cancelled() => Ok(()),
_ = lease_lost.cancelled() => Err(Error::leadership_lost()),
+ // A live lease does not prove the seat: a deleted election key
+ // lets another candidate win while this one keeps coordinating.
+ lost = util::await_held_key_lost(
+ &self.store,
+ util::HeldKey::Leader,
+ lease_id,
+ self.config.standby_poll_interval,
+ &cancel,
+ ) => match lost {
+ Some(how) => {
+ tracing::error!(name = %self.config.name, how, "election key lost; abdicating");
+ Err(Error::leadership_lost())
+ }
+ None => Ok(()),
+ },
result = self.run_coordination_loop(cancel.clone()) => result,
};
diff --git a/rust/personhog-coordination/src/routing_table.rs b/rust/personhog-coordination/src/routing_table.rs
index f6ac042c4e5f..6e0cb80c60cb 100644
--- a/rust/personhog-coordination/src/routing_table.rs
+++ b/rust/personhog-coordination/src/routing_table.rs
@@ -563,6 +563,31 @@ impl RoutingTable {
});
}
+ // A deleted registration leaves this router out of every freeze
+ // quorum until it re-registers, which ending the run does.
+ {
+ let store = Arc::clone(&self.store);
+ let router_name = self.config.router_name.clone();
+ let retry_interval = self.config.heartbeat_interval;
+ let token = cancel.child_token();
+ tasks.spawn(async move {
+ let lost = util::await_held_key_lost(
+ &store,
+ util::HeldKey::Router(&router_name),
+ lease_id,
+ retry_interval,
+ &token,
+ )
+ .await;
+ match lost {
+ Some(how) => Err(Error::invalid_state(format!(
+ "registration of router {router_name} {how}; re-registering"
+ ))),
+ None => Ok(()),
+ }
+ });
+ }
+
{
let addresses = Arc::clone(&self.addresses);
let token = cancel.child_token();
diff --git a/rust/personhog-coordination/src/store.rs b/rust/personhog-coordination/src/store.rs
index 4a2fa6ca65a9..fd4a88bf0f81 100644
--- a/rust/personhog-coordination/src/store.rs
+++ b/rust/personhog-coordination/src/store.rs
@@ -360,6 +360,25 @@ impl PersonhogStore {
Ok(self.inner.put(&key, router, Some(lease_id)).await?)
}
+ pub async fn get_router_with_revision(
+ &self,
+ router_name: &str,
+ ) -> Result<(Option<RegisteredRouter>, i64)> {
+ count_call("get_router_with_revision");
+ let key = self.key(StoreKey::Router(router_name));
+ Ok(self.inner.get_with_revision(&key).await?)
+ }
+
+ pub async fn watch_router_from(
+ &self,
+ router_name: &str,
+ start_revision: i64,
+ ) -> Result<WatchStream> {
+ count_call("watch_router_from");
+ let key = self.key(StoreKey::Router(router_name));
+ Ok(self.inner.watch_key_from(&key, start_revision).await?)
+ }
+
pub async fn list_routers(&self) -> Result<Vec<RegisteredRouter>> {
count_call("list_routers");
let key = self.key(StoreKey::RoutersPrefix);
diff --git a/rust/personhog-coordination/src/util.rs b/rust/personhog-coordination/src/util.rs
index 6cd203e2c769..b162912c8ca3 100644
--- a/rust/personhog-coordination/src/util.rs
+++ b/rust/personhog-coordination/src/util.rs
@@ -2,6 +2,7 @@ use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
+use etcd_client::{EventType, WatchStream};
use metrics::{counter, histogram};
use tokio_util::sync::CancellationToken;
use uuid::Uuid;
@@ -113,6 +114,118 @@ pub(crate) fn record_run_failure(
);
}
+#[derive(Debug, Clone, Copy)]
+pub(crate) enum HeldKey<'a> {
+ Leader,
+ Router(&'a str),
+}
+
+impl HeldKey<'_> {
+ fn label(self) -> &'static str {
+ match self {
+ HeldKey::Leader => "leader",
+ HeldKey::Router(_) => "router",
+ }
+ }
+
+ /// (exists, the lease the value records if it records one, read revision)
+ async fn read(self, store: &PersonhogStore) -> Result<(bool, Option<i64>, i64)> {
+ match self {
+ HeldKey::Leader => {
+ let (leader, revision) = store.get_leader_with_revision().await?;
+ Ok((
+ leader.is_some(),
+ leader.map(|leader| leader.lease_id),
+ revision,
+ ))
+ }
+ HeldKey::Router(name) => {
+ let (router, revision) = store.get_router_with_revision(name).await?;
+ Ok((router.is_some(), None, revision))
+ }
+ }
+ }
+
+ async fn watch_from(self, store: &PersonhogStore, start_revision: i64) -> Result<WatchStream> {
+ match self {
+ HeldKey::Leader => store.watch_leader_from(start_revision).await,
+ HeldKey::Router(name) => store.watch_router_from(name, start_revision).await,
+ }
+ }
+}
+
+/// Resolve once `key` is deleted or rewritten under a lease other than
+/// `lease_id`; `None` on cancellation. One watch stays open; a failed call or
+/// an ended watch re-reads after `retry_interval`.
+pub(crate) async fn await_held_key_lost(
+ store: &PersonhogStore,
+ key: HeldKey<'_>,
+ lease_id: i64,
+ retry_interval: Duration,
+ cancel: &CancellationToken,
+) -> Option<&'static str> {
+ let lost = |how: &'static str| {
+ counter!(
+ "personhog_coordination_held_key_lost_total",
+ "key" => key.label(),
+ "how" => how
+ )
+ .increment(1);
+ Some(how)
+ };
+ loop {
+ let earliest_retry = tokio::time::Instant::now() + retry_interval;
+ let read = tokio::select! {
+ _ = cancel.cancelled() => return None,
+ read = key.read(store) => read,
+ };
+ let revision = match read {
+ Ok((false, _, _)) => return lost("deleted"),
+ Ok((true, Some(holder), _)) if holder != lease_id => return lost("replaced"),
+ Ok((true, _, revision)) => Some(revision),
+ Err(e) => {
+ tracing::warn!(key = key.label(), error = %e, "held key read failed; retrying");
+ None
+ }
+ };
+ let stream = match revision {
+ Some(revision) => tokio::select! {
+ _ = cancel.cancelled() => return None,
+ stream = key.watch_from(store, revision + 1) => stream.map_err(|e| {
+ tracing::warn!(key = key.label(), error = %e, "held key watch failed; retrying");
+ }).ok(),
+ },
+ None => None,
+ };
+ if let Some(mut stream) = stream {
+ loop {
+ let message = tokio::select! {
+ _ = cancel.cancelled() => return None,
+ message = stream.message() => message,
+ };
+ let Ok(Some(response)) = message else { break };
+ if response.canceled() {
+ break;
+ }
+ for event in response.events() {
+ match event.event_type() {
+ EventType::Delete => return lost("deleted"),
+ EventType::Put => {
+ if event.kv().is_some_and(|kv| kv.lease() != lease_id) {
+ return lost("replaced");
+ }
+ }
+ }
+ }
+ }
+ }
+ tokio::select! {
+ _ = cancel.cancelled() => return None,
+ _ = tokio::time::sleep_until(earliest_retry) => {}
+ }
+ }
+}
+
/// Maintain a lease keepalive until cancelled, treating connection
/// trouble and lease loss as the different things they are. A broken or
/// silent keepalive stream is evidence about one connection — the lease
@@ -426,6 +539,14 @@ pub fn preregister_coordinator_metrics() {
metrics::counter!("personhog_coordination_elections_won_total").increment(0);
metrics::counter!("personhog_coordination_election_campaigns_total").increment(0);
metrics::counter!("personhog_coordination_abdications_total").increment(0);
+ for how in ["deleted", "replaced"] {
+ metrics::counter!(
+ "personhog_coordination_held_key_lost_total",
+ "key" => "leader",
+ "how" => how
+ )
+ .increment(0);
+ }
metrics::counter!("personhog_coordination_election_observation_failures_total").increment(0);
metrics::counter!("personhog_coordination_election_watch_interruptions_total").increment(0);
for stage in ["list", "delete"] {
@@ -564,6 +685,14 @@ pub fn preregister_router_coordination_metrics() {
.increment(0);
}
metrics::counter!("personhog_coordination_freeze_acks_written_total").increment(0);
+ for how in ["deleted", "replaced"] {
+ metrics::counter!(
+ "personhog_coordination_held_key_lost_total",
+ "key" => "router",
+ "how" => how
+ )
+ .increment(0);
+ }
}
#[cfg(test)]
diff --git a/rust/personhog-coordination/tests/protocol_hardening.rs b/rust/personhog-coordination/tests/protocol_hardening.rs
index 657635c17e60..f0cdaed48725 100644
--- a/rust/personhog-coordination/tests/protocol_hardening.rs
+++ b/rust/personhog-coordination/tests/protocol_hardening.rs
@@ -38,8 +38,8 @@ use personhog_coordination::routing_table::{RoutingTable, RoutingTableConfig, St
use personhog_coordination::store::PersonhogStore;
use personhog_coordination::strategy::StickyBalancedStrategy;
use personhog_coordination::types::{
- AssignmentStatus, HandoffPhase, HandoffState, PartitionAssignment, PodStatus, RegisteredPod,
- RegisteredRouter, RouterFreezeAck,
+ AssignmentStatus, HandoffPhase, HandoffState, LeaderInfo, PartitionAssignment, PodStatus,
+ RegisteredPod, RegisteredRouter, RouterFreezeAck,
};
/// Write a handoff record directly, bypassing the coordinator — gives the
@@ -295,6 +295,84 @@ async fn coordinator_abdicates_and_recampaigns_when_election_lease_revoked() {
.expect("graceful cancellation must exit cleanly after an abdication cycle");
}
+/// A leader must abdicate when its election key is deleted or taken under
+/// another lease. With one candidate, a new lease on the key proves it did.
+#[tokio::test]
+async fn a_coordinator_whose_election_key_is_lost_abdicates() {
+ for replace in [false, true] {
+ let (store, prefix) = test_store_with_prefix("coordinator-key-lost").await;
+ let key = format!("{prefix}coordinator/leader");
+ let cancel = CancellationToken::new();
+ let handle = start_coordinator_named(
+ Arc::clone(&store),
+ "key-lost-coordinator-0",
+ 60,
+ Arc::new(StickyBalancedStrategy),
+ cancel.clone(),
+ );
+
+ let check_store = Arc::clone(&store);
+ wait_for_condition(WAIT_TIMEOUT, POLL_INTERVAL, || {
+ let store = Arc::clone(&check_store);
+ async move { store.get_leader().await.ok().flatten().is_some() }
+ })
+ .await;
+ let first = store.get_leader().await.unwrap().unwrap();
+
+ let mut client = etcd_client::Client::connect([ETCD_ENDPOINT], None)
+ .await
+ .unwrap();
+ let intruder = if replace {
+ let lease = client.lease_grant(60, None).await.unwrap().id();
+ let value = serde_json::to_vec(&LeaderInfo {
+ holder: "intruder".to_string(),
+ lease_id: lease,
+ })
+ .unwrap();
+ client
+ .put(
+ key.as_str(),
+ value,
+ Some(etcd_client::PutOptions::new().with_lease(lease)),
+ )
+ .await
+ .unwrap();
+ Some(lease)
+ } else {
+ client.delete(key.as_str(), None).await.unwrap();
+ None
+ };
+ if let Some(lease) = intruder {
+ tokio::time::sleep(Duration::from_secs(2)).await;
+ client.lease_revoke(lease).await.unwrap();
+ }
+
+ let check_store = Arc::clone(&store);
+ let first_lease = first.lease_id;
+ wait_for_condition(Duration::from_secs(10), POLL_INTERVAL, || {
+ let store = Arc::clone(&check_store);
+ async move {
+ store
+ .get_leader()
+ .await
+ .ok()
+ .flatten()
+ .is_some_and(|leader| {
+ leader.holder == "key-lost-coordinator-0" && leader.lease_id != first_lease
+ })
+ }
+ })
+ .await;
+
+ cancel.cancel();
+ tokio::time::timeout(Duration::from_secs(10), handle)
+ .await
+ .expect("coordinator must exit promptly on cancellation")
+ .expect("coordinator task must not panic")
+ .expect("graceful cancellation must exit cleanly after an abdication");
+ }
+}
+
// ============================================================
// Cancelled handoffs converge on the durable state
// ============================================================
@@ -3693,6 +3771,52 @@ async fn a_deleted_registration_starts_a_new_session() {
cancel.cancel();
}
+/// A router whose registration is deleted under a live lease must register again.
+#[tokio::test]
+async fn a_router_whose_registration_is_deleted_registers_again() {
+ let (store, prefix) = test_store_with_prefix("router-registration-lost").await;
+ let cancel = CancellationToken::new();
+ let _router = start_router_with_lease_ttl(
+ Arc::clone(&store),
+ "registration-lost-router",
+ 60,
+ cancel.clone(),
+ );
+
+ let registered = |store: Arc<PersonhogStore>| async move {
+ store
+ .list_routers()
+ .await
+ .map(|routers| {
+ routers
+ .iter()
+ .any(|r| r.router_name == "registration-lost-router")
+ })
+ .unwrap_or(false)
+ };
+ let check = Arc::clone(&store);
+ wait_for_condition(WAIT_TIMEOUT, POLL_INTERVAL, || {
+ registered(Arc::clone(&check))
+ })
+ .await;
+
+ let mut client = etcd_client::Client::connect([ETCD_ENDPOINT], None)
+ .await
+ .unwrap();
+ client
+ .delete(format!("{prefix}routers/registration-lost-router"), None)
+ .await
+ .unwrap();
+
+ let check = Arc::clone(&store);
+ wait_for_condition(Duration::from_secs(20), POLL_INTERVAL, || {
+ registered(Arc::clone(&check))
+ })
+ .await;
+
+ cancel.cancel();
+}
+
/// The watch is a prefix watch, so it sees every deletion under
/// `pods/`. Only the pod's own registration key may cost it the session
/// — matching anything looser (say, a final path segment) would let an