Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

38.2 KB
Raw Blame

personhog-coordination

Etcd-based coordination primitives for dynamically assigning Kafka partitions to PersonHog writer pods and orchestrating zero-downtime handoffs when pods join or leave.

Why this crate exists

PersonHog processes person data sharded across Kafka partitions. Each partition must be owned by exactly one writer pod at a time (single-writer invariant). When pods scale up, scale down, or crash, partitions need to move between pods without losing inflight requests.

This crate provides the coordination layer: a coordinator assigns partitions to pods, and when assignments change, it orchestrates a handoff protocol that moves traffic through routers without dropping requests.

Why etcd

Etcd gives us the building blocks we need without rolling our own consensus:

  • Leases for failure detection: pod keys auto-delete when a pod crashes, triggering reassignment
  • Watches for reactivity: components react to state changes instead of polling
  • Transactions (CAS) for leader election: only one coordinator runs at a time. Standby candidates watch the leader key rather than campaigning on a timer, so election traffic tracks how often leadership changes instead of how many candidates are running
  • Strong consistency: all participants see the same state

Components

Component Crate Role
Coordinator personhog-coordination Leader-elected singleton. Watches pods, computes partition assignments, orchestrates handoffs.
Writer pod personhog-leader Owns partitions, processes person data. Registers with etcd, responds to handoff events (warm cache, release partition).
Router personhog-router Routes requests to the correct writer pod. Maintains a local routing table from etcd watches. Executes traffic cutover during handoffs.
etcd (external) Source of truth for pod registrations, partition assignments, and handoff state.

How it works

Registration

Each writer pod and router registers in etcd with a lease-backed key. If a pod crashes, the lease expires, the key is deleted, and the coordinator reacts.

Assignment

When the set of active pods changes, the coordinator computes a new partition-to-pod mapping using a pluggable AssignmentStrategy (jump consistent hash or sticky balanced). It diffs the result against current assignments to determine which partitions need to move.

Handoff

Partitions that need to move go through a four-phase state machine. The protocol is designed so that no acknowledged write is ever lost across a cutover, even though four independent components (coordinator, old owner, new owner, routers) participate.

Freezing --> Draining --> Warming --> Complete --> (deleted)
  1. Freezing: Coordinator creates the handoff, snapshotting the then-registered routers into a membership record the handoff refers to by freeze_quorum_ref. Routers begin buffering ("stashing") incoming writes for the partition and write a RouterFreezeAck. The old owner does not yet drain — it continues serving until the quorum has acked, because while a single router still forwards traffic to the old owner the inflight count cannot meaningfully be observed as zero. The required ackers are the creation-time snapshot intersected with the live registry, so the requirement only ever shrinks: a router that dies drops out, and one that registers later is never added (it anchors its watch after the handoff and its table starts fail-closed, so it cannot be routing to the old owner and its ack is not needed — it writes one anyway as it bootstraps, which is what keeps the pre-membership fallback satisfiable). Once every required router has written an ack matching the current handoff's handoff_id, the coordinator advances to Draining via a compare-and-swap on the handoff key's mod_revision.

  2. Draining: No router can forward new requests to the old owner anymore (every router is stashing). The old owner waits for its inflight request handlers to complete, then writes a PodDrainedAck. Because the leader's produce path awaits the Kafka delivery future before returning success, "no inflight" implies "every write this pod ever acked is durable in Kafka." When the ack arrives (or the old owner is no longer registered), the coordinator advances to Warming.

  3. Warming: Kafka high-water mark for the partition is now stable — no producer can append. The new owner consumes the changelog from the writer's last committed offset up to the current HWM, populating its cache, then writes a PodWarmedAck. The coordinator atomically writes both phase=Complete and the new PartitionAssignment in a single etcd transaction.

  4. Complete: Routers observe the handoff completion, drain their stashed writes to the new owner, and resume routing through the standard table lookup (which now points at the new owner). The old owner observes Complete and releases its partition cache. The coordinator deletes the handoff and ack keys.

The handoff record carries old_owner: Option<String>. None denotes an initial assignment with no prior owner — there's nothing to drain, so the protocol short-circuits and skips Draining entirely, advancing Freezing → Warming once router quorum is met.

Every handoff carries a unique handoff_id, and every ack (RouterFreezeAck, PodDrainedAck, PodWarmedAck) echoes it. Quorum checks count only acks whose id matches the current handoff, so a stale ack left over from an earlier handoff of the same partition can never satisfy a later one.

Where the freeze quorum's membership lives

Every handoff a plan creates shares one membership — the router registry as the coordinator saw it — so the plan writes it once under freeze_quorums/{id} and each handoff carries only that id. Written into each handoff instead, it grew with the fleet, and a plan transaction grew with the fleet times the partition count: at a few hundred of each it passed etcd's --max-request-bytes, which rejects the whole transaction, and it was paid again by every list of handoffs. The coordinator's reconcile tick deletes membership records no live handoff refers to, listing the ids before the handoffs so a record written mid-sweep is never a candidate.

Resolution fails closed. An id whose record is missing reads as unknown membership, which falls back to requiring every live router — the same rule as a record that predates the field. Both are stricter than any recorded membership, so a lost or prematurely collected record can only delay a handoff. The distinction that matters is against empty membership, which requires nobody: that is a real snapshot (zero routers registered at creation), and it must never be what an unresolvable reference degrades to.

Only the coordinator judges the quorum, and only one coordinator leads at a time, which is what makes the roll safe. A new leader reading a record written before this change uses the membership it carries inline. An old leader reading a record written after it sees no membership at all and falls back to requiring every live router — satisfiable, since a router that joins late acks in-progress handoffs as it bootstraps, so a mid-roll failover slows a handoff rather than wedging it.

Accepted degradations

Each is bounded by something that already exists, and none of them can advance a handoff early or lose an acked write. Detail lives at the code site.

What degrades When it happens What bounds it
A handoff loses its membership reference and falls back to requiring every live router An old-binary leader advances that handoff mid-roll: phase advances rewrite the whole record, and its struct has nowhere to carry the reference through The roll — a finished one cannot produce it. The fallback is the stricter rule, so it only ever delays a handoff. Not counted: unresolved_freeze_quorums_total sees references that fail to resolve, never ones stripped
A pod stays registered as Draining, and graceful handoffs toward it park until it expires Both the drain's bounded setup and the bounded revoke time out against a hung etcd One lease TTL, the same window a pod crashing right after registering already costs
A cancelled session leaves a registered phantom — a pod phantom parks writes behind plans toward it, a router phantom stalls freezes created in its window A session begin is cancelled after its registration may have landed, and the bounded revoke times out too One lease TTL, as above

A watcher etcd cancels — compaction, watcher pressure — is delivered as an ordinary response with canceled set and nothing ever after it. Every watch loop that must stay live treats that response as a failed attempt (live_watch_response), so it surfaces in the same restart-and-rebuild path as a broken stream rather than parking the loop on a dead watcher. Two loops handle it differently: the pod's own-registration watch — best-effort acceleration of a teardown that completes without it — ends for the session instead, and the standby's leader watch waits out its fallback window and re-reads, so a cancelled watch never observes an opening slower than a stalled one and no failure mode costs etcd more than the healthy cadence.

Why the Freezing/Draining split matters

If a single phase served both purposes (router stash and old-owner drain in parallel), a slow router could send a final write to the old owner after the old owner observed inflight=0 momentarily and wrote DrainedAck. The coordinator would then advance to Warming based on a stale ack. The old owner would still process that final write, advancing Kafka HWM past the point warming snapshots, and the new owner's cache would be missing the record — silently stale. Sequencing the phases ensures "no inflight" actually means "no producer can append more."

Recovery

Several failure modes are handled explicitly:

  • Dead new_owner: If the new owner's etcd registration disappears (lease expired, deregistered) before it can ack warming, the planner cancels the handoff by replacement, never by naked deletion: it picks a replacement owner and rewrites the record toward it, reaffirms the current owner (old_owner: None, phase Complete) when the partition should stay put, and only deletes — handoff plus all of its acks in one transaction guarded on the record's mod_revision — when nothing is placeable at all. The same machinery serves the second cancellation trigger, a handoff that outstays its phase deadline (past_phase_deadline). Either way a concurrent actor loses the CAS cleanly, and routers never observe a gap between owners.
  • Dead old_owner: A dead old owner is deliberately not grounds for cleanup. A pod that is no longer registered can't produce writes, so the drain requirement is vacuously satisfied — check_phase_advance treats a handoff whose old_owner is absent from the registered set as drained and advances it in place toward Complete. An old_owner that is still registered must write its DrainedAck before the handoff advances.
  • Cancellation by replacement: A handoff record is never deleted while non-terminal. To cancel one — its new owner's registration disappeared, or it outstayed its phase's deadline — the planner atomically replaces the record, under its mod_revision and with the predecessor's acks deleted in the same transaction, with whatever resolves the routers' stashes: a successor Freezing handoff when the plan re-places the partition (stashes ride into it without observing a gap), or a reaffirm — Complete toward the live current owner, old_owner: None — when the placement stands (stashes drain home through the ordinary Complete path, and the owner pod converges back to serving). Only when neither exists (owner dead, nothing placeable) is the record deleted, which is safe fail-closed. A deletion event therefore has exactly one protocol meaning — cleanup after Complete — and routers treat it as inert. The cancellation log and a freeze_ack_missing_total{router} counter name the required routers whose acks never arrived, so a freeze wedge attributes itself to the non-participating router.
  • Chunked plan application: A plan is applied in transactions sized to etcd's --max-txn-ops (ETCD_MAX_TXN_OPS, default 128); an unchunked over-budget plan would be rejected outright and retried forever. Atomicity is per partition — each unit keeps its guards and writes in one transaction — so a conflicted unit stands down alone and a partially applied plan is placement drift the next replan heals. Stateright explores partial application (chunked_plans).
  • Phase-aware deadline: Time-in-phase is bounded, not total age — wedged is a property of a phase. Freezing and Draining wait only on acknowledgements and share a tight budget (COORDINATOR_HANDOFF_DEADLINE_SECS, default 120 s); Warming replays a changelog whose length scales with the partition and gets its own generous budget (COORDINATOR_WARMING_DEADLINE_SECS, default 1800 s, 0 disables). A total-age deadline would cancel a legitimately long warm and restart it from zero, forever. The reconcile tick detects both cancellation triggers and wakes the planning loop; the replacement itself is a planning decision.
  • Participant stall watchdog: A router's registration is lease-backed, and the lease keepalive is its own task — so a router whose handoff watch loop has stalled would stay registered (counted in freeze quorums) while never acking. The watch loop stamps a progress marker from a ticker arm inside its own select; a watchdog task fails the run when the stamp goes stale (ROUTER_PARTICIPANT_STALL_SECS, default 60 s), deregistering the router on the way down so quorums stop counting it and the process supervisor restarts a healthy participant. Pods deliberately get no such watchdog: their converge loop legitimately blocks for warm-duration, and a pod that stops acking is routed around by the phase deadline and replacement instead of being restarted mid-warm.
  • Level-triggered reconciliation everywhere: Watches are the latency path; every participant also has a truth path that re-derives its state from a fresh snapshot. The coordinator's tick re-evaluates handoffs; the router's reconcile pass (an arm of its watch loop, ROUTER_RECONCILE_SECS, default 5 s) re-asserts stashes and freeze acks, converges its table to the assignment authority (dropping routes at or above the live partition count), and drains any stash whose partition has no handoff: to its assignment owner, or with no owner when no assignment names it — which heals a missed Complete, a silently dead watch stream, or a deleted handoff; the pod's tick re-converges every involved partition, including locally-held state the durable world no longer mentions. Every reconcile action is idempotent against the event path. Reconcile failures are tolerated, not fatal: a failed pass leaves the participant exactly as stale as the previous tick (the watch-driven steady state), so brief etcd blips never restart the fleet — sustained outage self-fences through the lease keepalive instead. A consecutive-failure budget (ROUTER_RECONCILE_FAILURE_BUDGET, default 12 ticks; same default on pods) bounds the partial mode where snapshot reads fail while the lease stays healthy, and personhog_coordination_reconcile_failures_total{component} counts every failure.
  • Warm provenance: A pod records what each partition warm was installed for (serving, or a specific handoff id). Warming honors only a warm installed for the current handoff — one left over from an earlier ownership era is released and rebuilt, because its cache can hold values that predate writes the changelog accepted since, and a fresh warm replays only from the writer's committed offset. Without this, a returning owner whose Released convergence never ran could ack a warm it never performed and lose acknowledged writes — a counterexample the stateright model found and now pins.
  • Coordinator restart: On startup, the coordinator runs reconcile_pending_handoffs. For every existing handoff it applies cleanup if Complete, otherwise calls check_phase_advance to nudge the state machine. This handles the case where acks were written before the coordinator was up — the ack-watch only fires on Put events, so without reconcile those acks would be invisible.
  • Lost coordination keys: The coordinator watches its own election key and abdicates, then campaigns again, when the key is deleted or rewritten under another lease; it never deletes the new holder's key. A router watches its own registration and, when it is deleted, ends the attempt and registers again on a fresh lease. Each watcher keeps one watch open and re-reads the key only after a failed call or an ended watch.
  • Reconcile tick: Beyond the startup reconcile, the coordinator re-evaluates every in-flight handoff on a fixed interval (reconcile_interval, default 5 s). This is the liveness backstop for state changes that fire no watched event — e.g. a router departing (nothing watches router registrations) newly satisfying a freeze quorum. Everything the tick drives is idempotent: phase transitions use CAS and cleanup tolerates already-deleted records.
  • Lease loss (self-fencing): Connection trouble and lease loss are different things. A broken or silent keepalive stream is a fact about one connection — the lease lives in etcd's replicated keyspace and stays valid until its TTL passes without renewal — so ambiguous keepalive failures rebuild the stream and retry (the store moves to another member when its own keeps failing; see assignment_coordination::failover) for as long as the last confirmed renewal is recent; a routine single-member restart costs some retries (personhog_coordination_keepalive_retries_total{component}), not a fence. The keepalive declares loss on exactly two signals: etcd answering a round with TTL ≤ 0 (authoritative — the lease is revoked or expired), or the renewal margin running out (two thirds of the TTL without a confirmed renewal — the final third is reserved so the fence completes before the coordinator can possibly treat the lease as expired; retrying past it would gamble split-brain against availability). On loss the component ends the coordination attempt immediately, skipping the graceful drain: the coordinator already considers it dead and is reassigning its partitions via the dead-owner path, so a graceful drain would only race that. A pod self-fences locally before anything else: every held partition is drained and, once quiesced, released (dropping its cache and serving authority — re-acquisition always re-warms). A partition whose drain cannot quiesce is deliberately not released — release is what unfences admissions and drops the cache, and a leaseless pod must do neither over live in-flight work — so it stays fenced and held, in-place recovery is refused, and the process restart clears it by death. A zombie owner therefore never keeps serving partitions the protocol has reassigned beyond the window that restart closes. Lease loss is observed at every await the session makes while holding a lease — the attempt select, the backoff nap between failed attempts, and the graceful drain all race the keepalive — so once the keepalive declares loss, the fence is never deferred by supervisor state. Every other session end fences too — budget exhaustion and shutdown revoke the lease, which deletes the registration instantly, so the local fence runs first (a no-op after a completed graceful drain) and the dead-owner path never meets a still-serving owner. On lease loss the fence's drain budget is the runway the keepalive margin reserved (a third of the TTL), not the full graceful drain timeout; overshoot poisons recovery and the process restart clears stragglers by death.
  • etcd member failover: Each process keeps one connection to whichever etcd member it reached first. Three member-level failures in a row (Unavailable, DeadlineExceeded, a broken connection), or one call left unanswered for at least 2 s, start a switch: the store probes the other members from its cached member list with a linearizable read and moves new calls to the first that answers. Requests carry etcd's require-leader flag, so a member without a leader rejects them at once. Watches and keepalive streams already open stay on their connection until it closes or etcd cancels them. A renewal waits up to the heartbeat interval for an answer, so a member that stops answering without closing its connections still costs that pod one renewal margin, and with it the pod's partitions.
  • Supervised coordination runs (etcd resilience): A failed coordination attempt — a broken watch stream, a failed etcd write, an exhausted reconcile budget — no longer kills the process; the run supervisors tear the attempt down cleanly, back off exponentially, and rebuild through the same bootstrap that recovers a restarted process, while the data plane keeps serving. The two participant classes recover differently, on purpose. A router sheds its registration on every failed attempt (its registration is a freeze-quorum vote, and a registered but non-acking router would stall every quorum until the phase deadline); serving through its gap is safe because handoffs created while it is unregistered exclude it from their quorums and the leader-side fences reject its stale forwards. A pod does the opposite: its lease session (grant, register, heartbeat) lives above the attempt loop and survives attempt failures, so the coordinator keeps seeing a live pod and never runs the dead-owner path against a process that is still serving — the invariant is that serving authority is held exactly as long as the lease-backed registration behind it. This matters because the dead-owner path has no fence: its safety rests entirely on a deregistered owner being unable to produce, so a pod must never voluntarily deregister while it still serves. Only lease loss ends a pod session, self-fencing locally (every held partition released; re-acquisition re-warms) before a fresh session registers anew. A consecutive-failure budget (ROUTER_RUN_RETRY_BUDGET, default 10; same default on pods) keeps a genuinely wedged participant on the process-restart path. Failures a restart cannot fix back off like any other but spend nothing: etcd could not be reached (Unavailable, DeadlineExceeded, a broken connection, or a watch cancelled because its member has no leader), or the session's lease was lost, by expiring in etcd or by its renewal margin running out. A new attempt is already the remedy for these, through a fresh session and, for connection trouble, member failover; through an outage the participant stays up and rebuilds once etcd answers. An attempt that made real progress — a convergence applied, a reconcile pass completed — resets the count, and the reset is keyed on measured progress rather than elapsed time because a time threshold silently exempts every failure detector slower than it (the reconcile budget and the stall watchdog both take a minute to fire), and personhog_coordination_run_restarts_total{component} counts every rebuild.

Concurrency and atomicity

  • All phase advancement uses cas_handoff_phase, a CAS on the handoff key's mod_revision. Two watchers racing to advance the same handoff cannot both succeed; the loser observes false and leaves the work to the Put event from the winner. (mod_revision rather than per-key version: version resets to 1 when a key is deleted and recreated, so a stale guard could match a different incarnation of the handoff.)
  • Every snapshot read that seeds a watch is revision-anchored: the read returns the etcd revision it was served at, and the watch starts at revision + 1. An event landing between the snapshot and the watch attaching is replayed rather than lost. When two snapshot reads feed one watch (e.g. a pod's startup reconcile reads assignments and handoffs), the watch anchors to the lower of the two revisions so nothing between them is skipped.
  • Advancing to Complete uses an atomic etcd transaction that writes both phase=Complete and the new PartitionAssignment together. Routers can never observe a torn state where the phase has advanced but the assignment hasn't (or vice versa).
  • Routing changes flow through handoff Complete events; there is no separate assignment watch. An out-of-band write to assignments/{partition} is not invisible, though — the reconcile pass converges the table against the assignment keys every tick, so it flips routing within seconds, with no freeze, no drain, and no fence in front of it. That is exactly why nothing may mutate those keys outside the handoff protocol.

Known limitations

The double zombie is closed. A lease-expired router feeding a lease-expired pod could once ack a write the partition's new owner never saw. Epoch fencing rejects that pod's produce at the broker, the lease gate stops it answering reads, and acquisition is gated on the same claim so a waking zombie cannot take the epoch from the real owner on its way to noticing it is dead. Every build runs all three; the stateright model checks them together. Two consequences are worth knowing:

  • The read gate is only as prompt as the claim's lapse. Natural expiry lapses the stamp before etcd deletes anything, but an out-of-band revocation removes the registration while the stamp is still fresh, and only the best-effort registration watch turns that into an immediate surrender. With the watch dead the stamp stays valid for up to one heartbeat (prompt_detection_is_what_makes_the_read_gate_hold pins this). No production path revokes another pod's lease, so reaching the window takes an operator.
  • Writer progress is coupled to leader transaction liveness. read_committed means the writer cannot advance past an open window: a leader that stops mid-window holds every consumer of that partition at the last stable offset until the broker abandons the transaction, bounded by the derived broker transaction timeout (about 6s at the production lease). The price of aborted windows never reaching Postgres.

Rollout placement residuals (k8s_awareness_enabled, default off). During a Deployment rollout the coordinator holds old-generation Ready pods and caps incoming pods at their final share, keyed on departing-generation registrations rather than the transient k8s rollout flag. Placement-behavior residuals, none affecting write safety (the handoff protocol is unchanged):

  • Some partitions move twice. Orphans that overflow onto Hold members when no capped pod has room move again when capacity appears, and ceil-rounded caps can leave the final balance one leveling move away. The cost of every partition always having an owner.
  • Intent loss degrades to policy-free planning. A Deployment deleted mid-rollout, a watcher stream dying between evaluations (stale intent until the lazy rewatch reports), or the k8s API being down all fall back to plain balanced placement — availability over placement. A freshly elected coordinator bound-waits (FIRST_INTENT_WAIT) for its first intent per controller, so failover mid-rollout does not plan policy-free unless the API server is actually unreachable.
  • A capped pod's in-flight steal counts against its cap before it lands. Planning attributes pinned handoffs to their targets, so a capped member briefly under-fills while its steal is in flight; the next evaluation after completion levels it.

Assignment strategies are minimal: The two bundled strategies (JumpHashStrategy and StickyBalancedStrategy) are basic implementations meant to exercise the coordination layer. JumpHashStrategy is stateless and deterministic but ignores current assignments entirely, causing unnecessary partition movement on every rebalance. StickyBalancedStrategy minimizes movement but uses a naive greedy approach without considering locality, rack awareness, or weighted pods. Both are placeholders to be replaced with a production-grade strategy as requirements become clearer.

A late plan chunk can install an owner that just deregistered. Chunked replacement units (successors, reaffirms) are guarded on their records, not on pod liveness, so a unit applied after its plan's snapshot can complete toward a pod that died in between — routers cut over and fail closed until the next replan moves the partition; nothing is acked and lost. The window predates chunking (plan-to-apply was already seconds wide) and grows with plan size under it. The stateright chunked_plans mode covers chunked creations; extending it to replacement units is open.

Mid-stream watch reconnects don't replay missed events — bounded by reconciliation: The snapshot-to-watch-attach gap is closed (watches are revision-anchored to the snapshot that seeded them), but if an established etcd watch stream drops and the underlying client reconnects, events that fired during the disconnect are not replayed. Every participant now bounds the resulting staleness with its level-triggered truth path — the coordinator's tick (which also wakes the planner unconditionally), the router's reconcile pass, and the pod's converge-all tick — so a missed event costs at most one reconcile interval, not correctness.

Changelog migrations

A changelog resize is a new topic and a cutover, never in-place growth. config/changelog ({topic, partitions, epoch, writer_group, retire_groups, draining}) is the one live object every participant captures by a strong read at acquisition, and migrations/current is the saga only the coordinator creates and advances.

The control plane only writes a request, migrations/requested. The coordinator starts the saga from it only while the fleet is quiet: no handoff record, and every old partition assigned to a registered, Ready pod. It waits, holding nothing, until then, and creates migrations/current in a transaction that requires no handoff record to exist at commit. Creating the saga record moves the planner's plan basis, so no handoff can begin after the start either. The owner half of quiet is checked when the request is evaluated, so an owner that begins shutting down in between is held like any other and reaches the freeze's holder_died cancel if it deregisters. The freeze also cancels the saga if it ever finds an ordinary handoff, before anything that handoff touches could be stranded; the quiet start is what keeps that from happening, so a start does not end in a cancel. Admission (the live config, the target topic, no generation still draining, no rollout) runs at the same step, and a request still waiting five minutes after the coordinator first saw it is rejected, with what it was waiting on, so a saga only starts while whoever asked is plausibly watching.

A migration can also move the changelog to another Kafka cluster. The target, the live config and each draining generation carry an optional cluster endpoint (a name, bootstrap servers and TLS); unset is the process's own KAFKA_HOSTS. The control plane writes the endpoint from its own name map, so the personhog services need no cluster config. Topic names stay unique across clusters, so drain reports, dirty-index streams, transactional ids and history keep keying on the topic alone. The leader connects per generation: it seals the old generation's ids on the old cluster, produces on the new one, and warms, recovers and prunes each source on its own cluster. A writer or relay that has drained a generation whose successor lives on another cluster does not re-subscribe: it shuts down cleanly, and its restart subscribes on the new cluster, with nothing left to commit on the old one. The coordinator never reads Kafka, so it only carries the field. The control plane refuses a cross-cluster request until PERSONHOG_CROSS_CLUSTER_MIGRATIONS is on, because a router that predates the field drops it at the flip, and a leader, writer or relay that predates it uses its own KAFKA_HOSTS for the new topic.

Freezing holds every partition in place (a held self-handoff that never passes Draining, so routers stash and the holder keeps its cache). Draining waits on the holders. Sealing has every live leader claim and drop every lane id on the old topic for every partition the assignment table does not give to a registered peer, which covers its own partitions whether or not it has loaded them and every partition without a registered owner, so no stale producer survives; it is the point of no return. Two seconds into Sealing, every leader also seals whatever no ack reports sealed yet, so a registered owner that cannot seal does not hold the fleet; every hold is drained by then, so nothing still produces to those partitions. A leader acks each seal as it lands, so the grace measures progress rather than a whole pass, and its report keeps every partition it has sealed for the life of the process; a restarted leader reseals once.

The ordinary planner places only when no saga holds placement: no migration, or a Resuming one whose holds are all restarted. Every coordinator plan carries the record it was decided on, and its transactions require that record unchanged, so a stalled pass that wakes after a successor acted lands nothing. The planner's record is its plan basis: the changelog config and the migration record, read in one transaction and compared by the config's mod revision (a flip or retire moves it) and the migration's create revision (a new or archived saga moves it; the driver's own writes do not). A migration pass's record is the saga at the mod revision it read, which any write to it moves: a cancel, a phase change, a blocking-list write, the archive. A saga cancelled before its flip and archived leaves the planner's basis as it was, and a plan landing after it meets the world it was planned for.

A leader serving a partition holds in its cache every acked write for that partition's keys that Postgres does not have yet, so a restart warms from every source that can hold one: for each draining generation, the old partitions congruent to the new one modulo gcd(old, new), from the writer's committed offset to their sealed end and filtered to the keys that hash to the new partition, then the live partition itself. Marks from an old source keep their own stream (topic, source partition), so recovery fetches them from the old topic and the prune settles them against the writer's offsets there. A mark also names the cache partition it belongs to, and a release or prune of one partition only removes the marks and lifecycle fences homed there: after a flip, an old partition and a new one on the same pod share keys, and either may be released or pruned first. Nothing in the saga waits for the writer: it drains the oldest undrained generation first, flushes a sealed partition's tail as soon as it reaches the end, and re-subscribes in place once the group has drained the topic; its offsets carry their topic, since a resubscribe reuses the partition numbers. A generation retires from draining once the writer group and every group in retire_groups, such as the relay's, have read each partition to its end. Each consumer's follower proves that with the same probe it moves on, then writes a drain report under changelog/drained/{topic}/{group}; the coordinator retires the oldest generation every gated group has reported, in a transaction that requires the config unchanged and no saga, and deletes that generation's reports with it. It also deletes reports for any topic no longer draining, since a retired topic never returns. A migration request made while a generation drains waits, naming the groups that have not reported. The control plane's retire op does the same retire after its own probe, as a manual override. The config watch delivers a retire as it delivers a flip, and a leader settles the marks on a retired stream without the writer's offsets. Deleting the topic stays a Terraform change.

Routers park requests per key, so a key's requests keep one queue across the flip: a request goes straight to a leader only when its key has nothing parked or in flight and its partition is not stashing, and the live count decides only which partition's drain takes a key. A drain first reads the config from etcd, best effort, then takes the keys that map to its partition under the count the router has adopted; the leader bounces any key it took under an older one, and the reconcile pass drains every pending partition, with no owner when nothing names it; a write for a partition born in the flip parks until that partition's fresh handoff drains it. A router keeps a grow's floor past the retire until its own routing table has a route for every born partition, because its assignment and config watches are independent and the retire can arrive first. A leader answers a stale x-partition with FAILED_PRECONDITION, which the router treats as a bounce, so the write lands once the router's config watch delivers the flip; a router that has not adopted within its retries answers UNAVAILABLE, never an ack; a lifecycle batch with a member off the routed partition is refused with INVALID_ARGUMENT, which the identity service answers by retrying each member alone.

Modules

Module Responsibility
types Core data types: RegisteredPod, PartitionAssignment, HandoffState, RouterFreezeAck, PodDrainedAck, PodWarmedAck, etc.
store EtcdStore abstraction: typed CRUD, watches, leases, and transactions over etcd. Hosts cas_handoff_phase and complete_handoff (the atomic phase + assignment txn).
coordinator Leader election, assignment computation, handoff orchestration, ack quorum checking, reconcile-on-startup, periodic reconcile tick, stale-handoff cleanup.
pod PodHandle + HandoffHandler trait: pod registration, heartbeat, and the drain/warm/release/resume responses to handoff phases (state derived by desired_state from fresh reads and converged, not accumulated from event payloads).
routing_table RoutingTable + StashHandler trait: routing map maintenance and the begin-stash / drain-stash phase responses. Drains run off the watch loop in per-partition lanes (a newer drain supersedes and never overlaps the one in flight; observing a non-terminal phase pauses the lane, and an interrupted drain puts its in-flight requests back rather than failing them); the loop's progress feeds the stall watchdog.
strategy AssignmentStrategy trait with JumpHashStrategy and StickyBalancedStrategy implementations.
hash Jump consistent hash function.
util Shared helpers: lease keepalive with self-fencing on lease loss, handoff id generation.
error Error types.