:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(personhog): model-check and gate changelog migrations (#119813)
Zach Brown3e0edadd0af1
diff --git a/.github/workflows/ci-rust.yml b/.github/workflows/ci-rust.ymlindex 74efea4f5893..eeba6430a09b 100644--- a/.github/workflows/ci-rust.yml+++ b/.github/workflows/ci-rust.yml@@ -846,6 +846,28 @@ jobs:--routers 3 --leaders 3 --partitions 8 --persons 100 --concurrency 10 \--duration 18s --router-kill-after 4s --router-kill-fast false --shutdown-after 8s+ # A changelog migration grows 4 to 8 partitions while the+ # writer is paused, so a restarted partition must warm the old+ # topic's tail itself. Write-once persons are what expose a+ # warm that skips that tail: with a small person set every key+ # is rewritten after the flip and the loss is masked.+ - name: Run the e2e gate growing the changelog with the writer paused+ run: |+ ./target/debug/personhog-test-harness gate \+ --leaders 2 --partitions 4 --persons 5000 --concurrency 10 --duration 30s \+ --migrate-after 8s --migrate-to-partitions 8 \+ --migrate-pause-writer --writer-pause-duration 8s++ # A shrink dissolves the partitions past the new count, and a+ # holder killed at Sealing leaves its partitions for its peers to+ # seal. The slow coordinator keeps Sealing long enough to hit.+ - name: Run the e2e gate shrinking the changelog with a leader killed at Sealing+ run: |+ ./target/debug/personhog-test-harness gate \+ --leaders 3 --partitions 8 --persons 100 --concurrency 10 --duration 30s \+ --migrate-after 8s --migrate-to-partitions 4 \+ --migrate-kill-leader-at-sealing --migrate-slow-coordinator-ms 500+- name: Dump service logs on failureif: failure()run: |diff --git a/rust/Cargo.lock b/rust/Cargo.lockindex 505d632aa9fb..27bb72c4fd20 100644--- a/rust/Cargo.lock+++ b/rust/Cargo.lock@@ -9130,6 +9130,7 @@ dependencies = ["axum 0.7.9","chrono","clap",+ "common-kafka","common-metrics","futures","hdrhistogram",diff --git a/rust/personhog-stateright/README.md b/rust/personhog-stateright/README.mdindex 9104c4068684..f65e07140ca0 100644--- a/rust/personhog-stateright/README.md+++ b/rust/personhog-stateright/README.md@@ -28,22 +28,24 @@ web explorer.Configurations are deliberately small — state spaces growcombinatorially, and protocol bugs are structural, showing up atminimum viable scale or not at all.-The suite explores ~28.5M states across 28 runs.-Its long pole is the two-partition epoch-fenced double zombie, which is about half the wall clock on its own:+As of 2026-10-16 the suite explores ~19M states across 43 runs.+Its long pole is the two-partition epoch-fenced double zombie, about a quarter of the wall clock on its own:| Scenario | Unique states | Wall time ||---|---|---|-| `epoch_fenced_two_partitions_double_zombie_is_safe` | 13.1M | 19.1s |-| `rollout_quota_mode_is_safe_and_live` | 3.5M | 5.1s |-| `cancellation_with_live_owner_reaffirms_and_resumes` | 3.2M | 4.2s |-| `chunked_plan_application_is_safe_and_live` | 2.1M | 2.7s |-| `probe_chunked_pending_units_reach_both_fates_safely` | 2.1M | 3.2s |-| *everything else (23 runs)* | 4.4M | 6s |--Roughly: a second partition costs ~20x, a second failure in the budget ~10x, a third pod ~2x.+| `epoch_fenced_two_partitions_double_zombie_is_safe` | 4.9M | 8.7s |+| `cancellation_with_live_owner_reaffirms_and_resumes` | 1.9M | 3.1s |+| `chunked_plan_application_is_safe_and_live` | 1.4M | 2.2s |+| `probe_chunked_pending_units_reach_both_fates_safely` | 1.4M | 2.8s |+| `rollout_quota_mode_is_safe_and_live` | 1.2M | 2.0s |+| `a_two_to_four_grow_is_safe_and_live` | 1.1M | 2.8s |+| `a_chunked_plan_across_a_migration_is_safe_and_live` | 0.9M | 2.4s |+| *everything else (36 runs)* | 6.1M | 11s |++Roughly: a second partition costs ~20x, a second failure in the budget ~10x, a third pod ~2x. The migration scenarios run at one or two partitions with keyed traffic and cost 20k to 1.1M states each; the 2→4 grow is the largest.Times are release mode, one scenario at a time on 12 cores — a CI runner with 4 slower cores is several times that, so treat them as ratios rather than absolutes.-The test suite (twenty-odd scenarios; the two-partition double zombie dominates the runtime) runs the full+The test suite (41 scenarios; the two-partition double zombie dominates the runtime) runs the fullverdict matrix — the 1-partition scenarios, the 2-partition cases, the3-pod rejoin, and the reachability probes. Two partitions matter for the coordinator'scross-partition scheduling (rebalancing defers while any handoff is in@@ -106,6 +108,14 @@ checker verifies, automatically:| `protocol::freeze_quorum_met` / `drain_satisfied` / `warm_satisfied` | the phase-advancement rules (identity quorum, id-correlated acks, vacuous drain) — `Action::AdvancePhase` calls them || `protocol::plan_partial_rebalance` | the rebalance decision (placement via `StickyBalancedStrategy` + the move/fresh diff, old_owner from the current assignment, in-flight partitions pinned) — `Action::Rebalance` calls it, as does the coordinator || `types::HandoffPhase` | used directly as the model's phase enum, so adding a phase breaks the model's exhaustive matches at compile time |+| `protocol::migration::plan_freeze` | which held self-handoffs the freeze creates, and what cancels it (a stray ordinary handoff, a holder or owner that is gone) — `Action::MigrateDrive` at Freezing calls it |+| `protocol::migration::fleet_quiet` | when a requested saga may start: no handoff record, and every old partition assigned to a registered, Ready pod — `Action::MigrateStart` is enabled through it, as the coordinator's start step is |+| `protocol::migration::plan_resume` | what Resuming does with the holds: dissolve the shrunk ones, abandon a dead holder's, restart the rest over the new changelog, and place the partitions a grow created — `Action::MigrateDrive` at Resuming calls it |+| `protocol::migration::migration_unsealed` / `migration_unsettled` | the Sealing and Resuming gates (every old partition sealed; no held handoff left) — `Action::MigrateDrive` calls them |+| `protocol::migration::seal_targets` | which old partitions a pod seals: every one assigned to it, every one owned by no registered peer, and once the seal is overdue every one no ack reports sealed — `Action::SealTick` calls it |+| `protocol::migration::planner_may_place` | when the ordinary planner may place during a saga — `Action::Rebalance` and `RebalanceChunked` are enabled through it, as the coordinator's pass is |+| `changelog::warm_sources` | the partitions a warm replays: the congruent old partitions (mod gcd, key-filtered) then the live one — a pod's warm effect reads it |+| `types::MigrationPhase::may_transition_to` | the saga's legal phase moves, so a new phase or a new edge fails the model's matches at compile time |What remains model-side is the *environment and effect application* —what warming does to a cache, what leases and zombies mean, how stashes@@ -121,6 +131,13 @@ queue — mapped to named production behavior for review:| `Action::CrashRestartWithinTtl` | Process death + same-name restart before lease expiry: registration and assignments survive, memory wiped || `Action::LeaseExpire` / `SelfFence` | Lease loss with the bounded zombie window before the keepalive self-fences (fix 1); same pair for routers, where lease loss also drops them from the freeze quorum || `Changelog.epoch_holder` | Kafka transactional-producer fencing: warming = `init_transactions`, which takes the fence from whoever held it; a produce from a fenced-out producer is rejected before any client ack |+| `Action::MigrateRequest` / `MigrateStart` / `MigrateRequestExpires` | the control plane's request, the coordinator's start once the fleet is quiet, and a request rejected or expired before it starts |+| `Action::MigrateDrive` / `MigrateCancel` | the coordinator's migration saga: one guarded phase step per drive (freeze, drain, seal, flip, resume, complete); cancel from any pre-Seal phase, standing in for both the operator and the pre-flip deadline |+| `Action::SealTick(pod)` | the leader's seal duty: a registered pod seals its `seal_targets` on the old generation, no capture needed, plus every unsealed partition once the grace has lapsed |+| `Action::SealGraceElapses` / `BreakSeal(pod)` | the 2s Sealing grace lapsing, and a registered pod whose seal never succeeds (a hung Kafka client with a live lease); a new incarnation heals it |+| `Action::AdoptConfig(router)` | the router's config watch: adopt the flipped config. Parked requests stay where they are, and the new count only decides which drain takes them; a write for a partition born in the flip parks until its handoff drains |+| `Action::ClientWrite(router, key)` / `ClientStrongRead(router, key)` | keyed traffic: the router hashes the key under its own adopted modulus, so a stale router routes to the old partition until it adopts |+| `Changelog` keyed by `(epoch, partition)` | one changelog per generation; the old generation stays readable for the warm's tail replay and must take no produce after the flip |Full elimination of the second table would mean deterministicsimulation — a trait seam over `PersonhogStore` with an in-memory@@ -135,7 +152,12 @@ implementation so the model executes `converge` and| `no_split_write_acceptance` (safety) | No two pods are simultaneously capable of accepting writes for one partition **and** each reachable via a live, non-stashing router — the real split-brain condition || `drained_ack_is_final` (safety) | A pod that wrote a DrainedAck for the current handoff attempt cannot accept another write in that incarnation || `some_handoff_completes`, `some_write_accepted` (sanity) | The interesting states are actually reachable |-| `converges_to_stable` (liveness) | Every full run ends quiescent: no handoffs, every partition served warm and unfenced by its sticky target, all live routers agreeing, no stashed traffic |+| `no_overtaken_request` (safety) | A router never delivers a request while an earlier request for the same key is still parked |+| `no_mixed_modulus_acceptance` (safety) | Every acked write lands in the live generation's changelog, never a retired one |+| `holds_stop_at_draining` (safety) | The ordinary handoff loop never moves a migration hold past Draining |+| `old_generation_quiet_after_flip` (safety) | Once flipped, no pod can produce to the old generation |+| `migration_completes`, `migration_cancels` (sanity) | Both saga outcomes are reachable in a migration config |+| `converges_to_stable` (liveness) | Every full run ends quiescent: no handoffs, every partition served warm and unfenced by its sticky target, all live routers agreeing, no stashed traffic, and no saga in flight or left to request |## Results@@ -147,6 +169,16 @@ implementation so the model executes `converge` and| Single zombie pod | **holds** | **holds** | holds || Double zombie (router + pod) | holds | holds | holds || Strong reads + one failure | holds | holds | holds |+| Changelog migration: grow 1→2, shrink 2→1, grow 2→4 | holds | holds | holds |+| Migration with a stale router (adopts late) | holds | holds | holds |+| Migration with the holder crash-restarted mid-hold | holds | holds | holds |+| Migration, seal targets by residency (rejected rule) | holds | holds | wedges (liveness) |+| Migration, live-only warm (rejected rule) | holds | holds | **violated** |+| Migration, a registered holder that cannot seal, with the grace | holds | holds | holds |+| Migration, a registered holder that cannot seal, no grace (rejected rule) | holds | holds | wedges (liveness) |+| Migration racing a chunked bootstrap plan, with the plan basis | holds | holds | holds |+| Migration racing a chunked bootstrap plan, no plan basis (rejected rule) | **violated** (old generation written after the flip) | holds | holds |+| Migration started with a handoff in flight (rejected rule) | **violated** | **violated** | **violated** |Two results worth calling out:@@ -196,6 +228,8 @@ target rather than having it forced — the one axis a 2-pod worldunder-exercises; the per-partition safety relations themselves aretwo-party (old owner ↔ new owner, zombie ↔ successor).+**The migration saga is checked against the same production decision functions.** The model drives `plan_freeze`, `migration_unsealed`, `migration_unsettled`, `seal_targets` and `warm_sources` through a grow, a shrink and a 2→4 grow, with keyed traffic, a stale router, a crash-restarted holder, and the deadline cancel. Its first run on the branch found a real hole: the freeze held the partitions in the assignment table, but an ordinary handoff for a fresh placement (a partition with no assignment yet) kept running, warmed, and accepted a write the flip then stranded. The saga now starts only from a fleet with no handoff in flight (`MigrateStart` requires it), and `quiet_start: false` keeps the rejected start, which loses acked writes. Two rejected rules stay in the model as variants, each pinned by the counterexample that rejected it: sealing by residency (`SealRule::Residency`) wedges a holder that crash-restarted before sealing, because it no longer holds the data and nobody else is assigned, so the saga never leaves Sealing; a live-only warm (`WarmReplay::LiveOnly`) loses the old generation's tail, because a key's last write before the flip sits in an old partition the new one never replays. A third, `seal_grace: false`, pins why the seal grace exists: one registered holder whose seal never succeeds holds Sealing forever, since peers leave a registered owner's partitions alone and nothing cancels past Draining.+## Usage```sh@@ -241,6 +275,8 @@ owner to still claim its partitions, so an owner that refuses its ownreads while looking alive to the coordinator fails liveness rather thanpassing quietly.+Migration coverage: one migration per run with at most one draining generation, which admission also enforces, so the retire step and a second migration are not modeled; the key hash is `key % partitions`, which keeps the congruence structure of the real hash without its constant; the seal is instant (production claims and drops every lane's transactional id); the cancel is unbudgeted, standing in for the pre-flip deadline as well as the operator; the writer is not modeled, since its drain only gates retirement. Coordinator passes are atomic steps in the model, so a stalled pass that wakes after a successor acted cannot occur here; the rule that every plan requires the record it was decided on unchanged is pinned by the coordination store tests instead.+Known remaining abstractions: warming is instant and atomic (productionstreams from Kafka with retries — availability, not safety); stashdeadlines/bounds are not modeled (availability policies); thediff --git a/rust/personhog-stateright/src/main.rs b/rust/personhog-stateright/src/main.rsindex 859084d8d190..c4ec93a2dc71 100644--- a/rust/personhog-stateright/src/main.rs+++ b/rust/personhog-stateright/src/main.rs@@ -11,7 +11,7 @@use clap::{Parser, ValueEnum};-use personhog_stateright::model::{ClaimDetection, HandoffModel, WarmOrder};+use personhog_stateright::model::{ClaimDetection, HandoffModel, SealRule, WarmOrder, WarmReplay};use stateright::Model;#[derive(Parser)]@@ -58,6 +58,17 @@ fn main() {cancels: 0,chunked_plans: false,probes: false,+ migrations: 0,+ migrate_to: 1,+ keys: 1,+ seal_rule: SealRule::Assignment,+ warm_replay: WarmReplay::Sources,+ seal_breaks: 0,+ seal_grace: true,+ plan_basis_guard: true,+ quiet_start: true,+ freeze_checks_strays: true,+ park_behind_key: true,};println!("exploring {:?} at http://localhost:3000 …", args.scenario);model.checker().serve("localhost:3000");diff --git a/rust/personhog-stateright/src/model.rs b/rust/personhog-stateright/src/model.rsindex 21a8b4714ba2..57ff4d3699a2 100644--- a/rust/personhog-stateright/src/model.rs+++ b/rust/personhog-stateright/src/model.rs@@ -5,24 +5,36 @@//! reviewable line by line. The checker exhaustively interleaves every//! action from every reachable state and verifies the properties at each.-use std::collections::{BTreeMap, BTreeSet, HashMap};+mod migration;+use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};++use personhog_coordination::changelog::warm_sources;use personhog_coordination::pod::{desired_state, DesiredState};+use personhog_coordination::protocol::migration::{+ fleet_quiet, migration_unsealed, migration_unsettled, plan_freeze, plan_resume,+ planner_may_place, seal_targets,+};use personhog_coordination::protocol::{drain_satisfied, freeze_quorum_met, plan_partial_rebalance, warm_satisfied, PlannedHandoff,};use personhog_coordination::strategy::{AssignmentStrategy, Member, StickyBalancedStrategy};use personhog_coordination::types::{- AssignmentStatus, HandoffState, PartitionAssignment, PodDrainedAck, PodStatus, PodWarmedAck,- RegisteredPod, RegisteredRouter, RouterFreezeAck,+ AssignmentStatus, ChangelogConfig, Generation as ProductionGeneration, HandoffState,+ MigrationPhase, PartitionAssignment, PodDrainedAck, PodStatus, PodWarmedAck, RegisteredPod,+ RegisteredRouter, RouterFreezeAck, SealAck,};use stateright::{Model, Property};use crate::types::{- Action, Changelog, Handoff, HandoffId, Partition, PendingUnit, PendingWarm, Phase, Pod, PodId,- Router, RouterId, StashedRequest, SystemState, WarmState,+ Action, Changelog, Config, Epoch, Generation, Handoff, HandoffId, Key, Migration, Partition,+ PendingUnit, PendingWarm, Phase, Pod, PodId, Router, RouterId, StashedRequest, SystemState,+ WarmState,};+/// The one migration id the model ever runs.+const MIGRATION_ID: &str = "m";+/// Deterministic names bridging the model's compact u8 ids to the/// string-keyed production types.fn pod_name(x: PodId) -> String {@@ -34,6 +46,20 @@ fn pod_id(name: &str) -> PodId {fn router_name(r: RouterId) -> String {format!("r{r}")}+fn generation_topic(epoch: Epoch) -> String {+ format!("g{epoch}")+}+fn topic_epoch(topic: &str) -> Epoch {+ topic+ .trim_start_matches('g')+ .parse()+ .expect("generation topic")+}++/// The identity hash: a key's home is its remainder.+fn home(key: Key, partitions: u8) -> Partition {+ key % partitions+}/// Materialize the production `HandoffState` view of a model handoff, so/// shared production functions can be called on checker state.@@ -55,11 +81,97 @@ fn production_handoff(p: Partition, h: &Handoff) -> HandoffState {freeze_quorum_ref: None,created_at_ms: 0,phase_entered_at_ms: 0,- held_by: None,- config_epoch: 0,+ held_by: h.held.then(|| MIGRATION_ID.to_string()),+ config_epoch: h.config_epoch as u64,}}+fn production_config(config: &Config) -> ChangelogConfig {+ ChangelogConfig {+ topic: generation_topic(config.live.epoch),+ partitions: config.live.partitions as u32,+ epoch: config.live.epoch as u64,+ writer_group: String::new(),+ draining: config+ .draining+ .map(|g| ProductionGeneration {+ topic: generation_topic(g.epoch),+ partitions: g.partitions as u32,+ epoch: g.epoch as u64,+ })+ .into_iter()+ .collect(),+ }+}++fn production_assignments(state: &SystemState) -> Vec<PartitionAssignment> {+ state+ .assignments+ .iter()+ .map(|(p, owner)| PartitionAssignment {+ partition: *p as u32,+ owner: pod_name(*owner),+ advertise_address: None,+ status: AssignmentStatus::Active,+ })+ .collect()+}++fn production_registered(state: &SystemState) -> Vec<RegisteredPod> {+ state+ .pods+ .iter()+ .filter(|(_, pod)| pod.registered)+ .map(|(x, _)| RegisteredPod {+ pod_name: pod_name(*x),+ generation: String::new(),+ status: PodStatus::Ready,+ registered_at: 0,+ last_heartbeat: 0,+ controller: None,+ advertise_address: None,+ })+ .collect()+}++fn production_seal_acks(state: &SystemState) -> Vec<SealAck> {+ state+ .seal_acks+ .iter()+ .map(|(x, sealed)| SealAck {+ migration_id: MIGRATION_ID.to_string(),+ pod_name: pod_name(*x),+ sealed: sealed.iter().map(|p| *p as u32).collect(),+ })+ .collect()+}++/// The modulus a captured epoch hashes with; `None` once that generation is gone.+fn modulus_of(config: &Config, epoch: Epoch) -> Option<u8> {+ if config.live.epoch == epoch {+ Some(config.live.partitions)+ } else {+ config+ .draining+ .filter(|g| g.epoch == epoch)+ .map(|g| g.partitions)+ }+}++/// Which old partitions a pod seals; `Residency` is the rejected rule, kept as the record of why.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub enum SealRule {+ Assignment,+ Residency,+}++/// Where a restart's warm reads from; `LiveOnly` is the rejected rule.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub enum WarmReplay {+ Sources,+ LiveOnly,+}+/// The membership a model handoff was created with, in the form the/// production quorum predicates take. The model always captures a/// snapshot; the production `None` fallback is a serialization concern@@ -147,6 +259,30 @@ pub struct HandoffModel {/// than assume, which configurations actually reach them. Off in the/// verdict tests: an unreached probe would fail `assert_properties`.pub probes: bool,+ /// Migrations the control plane may start (at most one draining generation).+ pub migrations: u8,+ /// Partition count a migration moves to.+ pub migrate_to: u8,+ /// Distinct keys the workload writes and reads, at least one per starting+ /// partition so every partition takes writes.+ pub keys: u8,+ pub seal_rule: SealRule,+ pub warm_replay: WarmReplay,+ /// Budget of pods whose seal breaks (`Action::BreakSeal`).+ pub seal_breaks: u8,+ /// Whether peers seal overdue partitions; `false` keeps the rejected no-exit rule.+ pub seal_grace: bool,+ /// Whether a plan's later units stand down when its basis moved; `false`+ /// keeps the rejected unguarded planner.+ pub plan_basis_guard: bool,+ /// Whether a saga starts only once the fleet is quiet (`fleet_quiet`);+ /// `false` keeps the rejected start that freezes a fleet in motion.+ pub quiet_start: bool,+ /// Whether the freeze cancels on an ordinary handoff (`plan_freeze`'s stray).+ pub freeze_checks_strays: bool,+ /// Whether a request parks behind its key's parked requests; `false`+ /// keeps the rejected bucket rule, which lets a later request overtake.+ pub park_behind_key: bool,}/// Derive one pod's desired state by calling the production@@ -201,8 +337,43 @@ impl HandoffModel {fn router_ids(&self) -> impl Iterator<Item = RouterId> {0..(self.routers + self.late_routers)}+ /// Every partition number any generation can have.fn partition_ids(&self) -> impl Iterator<Item = Partition> {- 0..self.partitions+ 0..self.partitions.max(self.migrate_to)+ }+ fn key_ids(&self) -> impl Iterator<Item = Key> {+ 0..self.keys.max(self.partitions)+ }++ /// `fleet_quiet` over production views; the model's registered pods are all Ready.+ fn quiet(state: &SystemState) -> bool {+ let handoffs: Vec<HandoffState> = state+ .handoffs+ .iter()+ .map(|(p, h)| production_handoff(*p, h))+ .collect();+ let registered = Self::registered_names(state);+ let ready: HashSet<&str> = registered.iter().map(String::as_str).collect();+ fleet_quiet(+ state.config.live.partitions as u32,+ &production_assignments(state),+ &handoffs,+ &ready,+ )+ .is_empty()+ }++ /// Production `PlanBasis`, coarsened: whether a saga exists, and the live epoch.+ fn plan_basis(state: &SystemState) -> (bool, Epoch) {+ (state.migration.is_some(), state.config.live.epoch)+ }++ /// The planner stands down while a migration owns placement.+ fn planner_active(state: &SystemState) -> bool {+ planner_may_place(+ state.migration.map(|m| m.phase),+ state.handoffs.values().any(|h| h.held),+ )}/// The production `StickyBalancedStrategy`, called on@@ -215,7 +386,11 @@ impl HandoffModel {.map(|(p, owner)| (*p as u32, pod_name(*owner))).collect();let members = self.planner_members(state);- StickyBalancedStrategy.compute_assignments(¤t, &members, self.partitions as u32)+ StickyBalancedStrategy.compute_assignments(+ ¤t,+ &members,+ state.config.live.partitions as u32,+ )}/// Planner membership with the model's placement policies applied —@@ -239,7 +414,7 @@ impl HandoffModel {// fleet's *desired* size, not its live count, so a crashed// incoming pod does not inflate its peers' quota.let incoming = self.pods.saturating_sub(self.hold_pods).max(1);- let cap = (self.partitions as u32).div_ceil(incoming as u32);+ let cap = (state.config.live.partitions as u32).div_ceil(incoming as u32);ids.into_iter().map(|id| {if id < self.hold_pods {@@ -272,9 +447,9 @@ impl HandoffModel {if !pod.registered && pod.zombie_writes_left == 0 {return false;}- if !pod.warmed.contains_key(&partition) {+ let Some(warm) = pod.warmed.get(&partition) else {return false;- }+ };if pod.fenced.contains(&partition) {return false;}@@ -282,7 +457,37 @@ impl HandoffModel {// acquired the fence most recently. A warmed pod that is no// longer the holder would be producing under a fenced-out// producer, and the broker rejects it before any client ack.- state.changelogs[&partition].epoch_holder == Some(x)+ state.changelogs[&(warm.epoch, partition)].epoch_holder == Some(x)+ }++ /// The leader's `validate_partition`: the key's home under the captured modulus.+ fn validates(&self, state: &SystemState, x: PodId, partition: Partition, key: Key) -> bool {+ state.pods[&x]+ .warmed+ .get(&partition)+ .and_then(|warm| modulus_of(&state.config, warm.epoch))+ .is_some_and(|modulus| home(key, modulus) == partition)+ }++ /// The logs a warm of `partition` replays, each at its current length.+ fn warm_prefixes(+ &self,+ state: &SystemState,+ partition: Partition,+ ) -> BTreeMap<(Epoch, Partition), u8> {+ let live = (state.config.live.epoch, partition);+ let keys: Vec<(Epoch, Partition)> = match self.warm_replay {+ WarmReplay::Sources => {+ warm_sources(&production_config(&state.config), partition as u32)+ .into_iter()+ .map(|s| (topic_epoch(&s.topic), s.partition as Partition))+ .collect()+ }+ WarmReplay::LiveOnly => vec![live],+ };+ keys.into_iter()+ .map(|key| (key, state.changelogs.get(&key).map_or(0, Changelog::len)))+ .collect()}/// One step of `warm_partition` for pod `x` on `partition`. Under@@ -298,6 +503,9 @@ impl HandoffModel {partition: Partition,for_handoff: Option<HandoffId>,) {+ // The strong-read capture: the config as etcd holds it now.+ let epoch = state.config.live.epoch;+ let live = (epoch, partition);// Both orderings are two steps in production, but only one of// them is *observably* two steps. Acquiring the fence first bumps// the epoch before the read, and an append requires an installed@@ -317,16 +525,17 @@ impl HandoffModel {// warm), this collapse is the assumption to revisit first.let observable_gap = self.warm_order == WarmOrder::ReadFirst;if !observable_gap {- let warm = {- let log = state.changelogs.get_mut(&partition).unwrap();- log.epoch_holder = Some(x);+ let sources = self.warm_prefixes(state, partition);+ state.changelogs.get_mut(&live).unwrap().epoch_holder = Some(x);+ let pod = state.pods.get_mut(&x).unwrap();+ pod.warmed.insert(+ partition,WarmState {for_handoff,- visible: log.len,- }- };- let pod = state.pods.get_mut(&x).unwrap();- pod.warmed.insert(partition, warm);+ epoch,+ sources,+ },+ );pod.fenced.remove(&partition);return;}@@ -336,7 +545,7 @@ impl HandoffModel {// current HWM while the fence — and thus the rejection of// a zombie's write — does not exist yet.let pending = PendingWarm {- cutoff: state.changelogs[&partition].len,+ cutoff: state.changelogs[&live].len(),};state.pods@@ -348,55 +557,65 @@ impl HandoffModel {Some(PendingWarm { cutoff }) => {// The cutoff predates the fence: anything committed in// the gap sits beyond it, invisible forever.- let warm = {- let log = state.changelogs.get_mut(&partition).unwrap();- log.epoch_holder = Some(x);- WarmState {- for_handoff,- visible: cutoff,- }- };+ let mut sources = self.warm_prefixes(state, partition);+ sources.insert(live, cutoff);+ state.changelogs.get_mut(&live).unwrap().epoch_holder = Some(x);let pod = state.pods.get_mut(&x).unwrap();pod.pending_warm.remove(&partition);- pod.warmed.insert(partition, warm);+ pod.warmed.insert(+ partition,+ WarmState {+ for_handoff,+ epoch,+ sources,+ },+ );pod.fenced.remove(&partition);}}}- /// Whether the handoff's new owner has fixed the cutoff it will- /// serve with: a warm installed for *this* handoff, or a mid-warm- /// read already taken (a pending warm always belongs to the current- /// attempt — cancellation requires the target unregistered, which- /// wipes its process memory). An earlier-era warm does NOT freeze- /// the cutoff: the Acquiring convergence releases and rebuilds it.+ /// Whether the handoff's new owner has fixed its serving cutoff over `log`: a warm+ /// for this handoff, or a mid-warm read already taken. An earlier-era warm does not count.fn handoff_cutoff_frozen(&self,state: &SystemState,h: &Handoff,partition: Partition,+ log: (Epoch, Partition),) -> bool {let pod = &state.pods[&h.new_owner];match pod.warmed.get(&partition) {- Some(w) => w.for_handoff == Some(h.id),- None => pod.pending_warm.contains_key(&partition),+ Some(w) => w.for_handoff == Some(h.id) && w.sources.contains_key(&log),+ None => pod.pending_warm.contains_key(&partition) && log.1 == partition,}}- /// Whether the assignment owner has fixed its serving cutoff — the- /// Serving convergence honors any installed warm (an owner's warm- /// cannot go stale while ownership is continuous), so any warm or a- /// taken mid-warm read freezes it.- fn owner_cutoff_frozen(&self, state: &SystemState, x: PodId, partition: Partition) -> bool {+ /// Whether the assignment owner has fixed its serving cutoff over `log`: any warm,+ /// or a mid-warm read already taken.+ fn owner_cutoff_frozen(+ &self,+ state: &SystemState,+ x: PodId,+ partition: Partition,+ log: (Epoch, Partition),+ ) -> bool {let pod = &state.pods[&x];- pod.warmed.contains_key(&partition) || pod.pending_warm.contains_key(&partition)+ match pod.warmed.get(&partition) {+ Some(w) => w.sources.contains_key(&log),+ None => pod.pending_warm.contains_key(&partition) && log.1 == partition,+ }}- /// Serve a strong read at pod `x`, if it can (running, partition- /// warmed). Sets the staleness flag when the pod's visible prefix is- /// behind the changelog — the read returned state missing at least one- /// acked write. Returns whether the read was served.- fn serve_read(&self, state: &mut SystemState, x: PodId, partition: Partition) -> bool {+ /// Serve a strong read for `key` at pod `x` if it can, flagging a stale read when a+ /// log holds a newer record the pod cannot see. Returns whether it served.+ fn serve_read(+ &self,+ state: &mut SystemState,+ x: PodId,+ partition: Partition,+ key: Key,+ ) -> bool {let pod = &state.pods[&x];if !pod.running {return false;@@ -421,37 +640,50 @@ impl HandoffModel {let Some(warm) = pod.warmed.get(&partition) else {return false;};- if warm.visible < state.changelogs[&partition].len {+ if !self.validates(state, x, partition, key) {+ return false;+ }+ let stale = state.changelogs.iter().any(|(log, changelog)| {+ let visible = warm.sources.get(log).copied().unwrap_or(0) as usize;+ changelog.records.iter().skip(visible).any(|k| *k == key)+ });+ if stale {state.stale_strong_read = true;}state.read_served = true;true}- /// Append one acked write to the changelog, tracking the loss flag:- /// if the protocol has designated a *different* pod as the (incoming- /// or current) owner and that pod's warm cutoff is already fixed,- /// this write sits beyond the cutoff and is invisible to it forever.- fn accept_write(&self, state: &mut SystemState, x: PodId, partition: Partition) {- let designated_other = match state.handoffs.get(&partition) {- Some(h) if h.new_owner != x => self.handoff_cutoff_frozen(state, h, partition),+ /// Append one acked write to the pod's log, flagging a lost write when another pod+ /// that owns the key's home has already fixed its cutoff over this log.+ fn accept_write(&self, state: &mut SystemState, x: PodId, partition: Partition, key: Key) {+ let epoch = state.pods[&x].warmed[&partition].epoch;+ let log = (epoch, partition);+ let home_now = home(key, state.config.live.partitions);+ let designated_other = match state.handoffs.get(&home_now) {+ Some(h) if h.new_owner != x => self.handoff_cutoff_frozen(state, h, home_now, log),Some(_) => false,- None => match state.assignments.get(&partition) {- Some(owner) if *owner != x => self.owner_cutoff_frozen(state, *owner, partition),+ None => match state.assignments.get(&home_now) {+ Some(owner) if *owner != x => {+ self.owner_cutoff_frozen(state, *owner, home_now, log)+ }_ => false,},};if designated_other {state.lost_acked_write = true;}+ if epoch != state.config.live.epoch {+ state.mixed_modulus_acceptance = true;+ }- let log = state.changelogs.get_mut(&partition).unwrap();- log.len = log.len.saturating_add(1);+ state.changelogs.get_mut(&log).unwrap().records.push(key);let pod = state.pods.get_mut(&x).unwrap();if let Some(warm) = pod.warmed.get_mut(&partition) {// Its own write is immediately visible to it.- warm.visible = warm.visible.saturating_add(1);+ let visible = warm.sources.entry(log).or_insert(0);+ *visible = visible.saturating_add(1);}if !pod.registered {pod.zombie_writes_left = pod.zombie_writes_left.saturating_sub(1);@@ -479,19 +711,15 @@ impl HandoffModel {/// owner, fail-closed), not decided on the deletion event.fn cancel_by_replacement(&self, state: &mut SystemState, p: Partition) {if self.probes_stash_racing_cancellation()- && state- .routers- .values()- .any(|r| r.stash.get(&p).is_some_and(|q| !q.is_empty()))+ && state.routers.values().any(|r| Self::parked_at(r, p)){state.cancelled_while_stash_parked = true;}- state.freeze_acks.retain(|(fp, _), _| fp != &p);- state.drained_acks.retain(|(dp, _), _| dp != &p);- state.warmed_acks.retain(|(wp, _), _| wp != &p);+ Self::clear_acks(state, p);let owner = state.assignments.get(&p).copied();+ let epoch = state.config.live.epoch;if let Some(owner) = owner.filter(|o| state.pods[o].registered) {let id = state.next_handoff_id;state.next_handoff_id += 1;@@ -503,6 +731,8 @@ impl HandoffModel {new_owner: owner,phase: Phase::Complete,quorum: BTreeSet::new(),+ held: false,+ config_epoch: epoch,},);if self.probes_reaffirm() {@@ -513,10 +743,7 @@ impl HandoffModel {if let Some(target) = self.target_owner(state, p) {let id = state.next_handoff_id;state.next_handoff_id += 1;- let quorum: BTreeSet<RouterId> = self- .router_ids()- .filter(|r| state.routers[r].registered)- .collect();+ let quorum = Self::quorum_snapshot(state);state.handoffs.insert(p,Handoff {@@ -525,6 +752,8 @@ impl HandoffModel {new_owner: target,phase: Phase::Freezing,quorum,+ held: false,+ config_epoch: epoch,},);if self.probes_successor_replacement() {@@ -535,35 +764,112 @@ impl HandoffModel {state.handoffs.remove(&p);}- /// Route one write through router `r` exactly as the raw proxy does:- /// park it if the partition is stashing, otherwise forward to the- /// table entry and let the leader's admission decide.- fn route_write(&self, state: &mut SystemState, r: RouterId, partition: Partition) -> bool {+ fn clear_acks(state: &mut SystemState, p: Partition) {+ state.freeze_acks.retain(|(fp, _), _| fp != &p);+ state.drained_acks.retain(|(dp, _), _| dp != &p);+ state.warmed_acks.retain(|(wp, _), _| wp != &p);+ }++ /// The coordinator's snapshot read: the routers registered now.+ fn quorum_snapshot(state: &SystemState) -> BTreeSet<RouterId> {+ state+ .routers+ .iter()+ .filter(|(_, r)| r.registered)+ .map(|(r, _)| *r)+ .collect()+ }++ /// Whether router `r` holds a parked request whose key maps to `p` under its count.+ fn parked_at(router: &Router, p: Partition) -> bool {+ router+ .stash+ .iter()+ .any(|e| home(e.key(), router.adopted.partitions) == p)+ }++ /// Remove and return router `r`'s parked requests for `p`, in arrival order.+ fn take_parked(state: &mut SystemState, r: RouterId, p: Partition) -> Vec<StashedRequest> {+ let router = state.routers.get_mut(&r).unwrap();+ let partitions = router.adopted.partitions;+ let (taken, kept) = std::mem::take(&mut router.stash)+ .into_iter()+ .partition(|e| home(e.key(), partitions) == p);+ router.stash = kept;+ taken+ }++ /// Route one request as the raw proxy does: park it behind its key or a stashing+ /// partition, otherwise forward it and let the leader's admission decide.+ fn route(&self, state: &mut SystemState, r: RouterId, request: StashedRequest) -> bool {let router = &state.routers[&r];if !router.running {return false;}- if router.stashing.contains(&partition) {- state- .routers- .get_mut(&r)- .unwrap()- .stash- .entry(partition)- .or_default()- .push(StashedRequest::Write);+ let partition = home(request.key(), router.adopted.partitions);+ if router.stashing.contains(&partition)+ || (self.park_behind_key && router.stash.iter().any(|e| e.key() == request.key()))+ {+ state.routers.get_mut(&r).unwrap().stash.push(request);return true;}let Some(target) = router.table.get(&partition).copied() else {+ // A partition born in the flip parks until its handoff lands.+ let born = state+ .config+ .draining+ .is_some_and(|g| partition >= g.partitions);+ if born {+ let router = state.routers.get_mut(&r).unwrap();+ router.stashing.insert(partition);+ router.stash.push(request);+ state.born_partition_write_parked = true;+ return true;+ }// No route: the request is rejected; nothing changes.return false;};- if self.write_capable(state, target, partition) {- self.accept_write(state, target, partition);- true- } else {- // Rejected fail-closed at the leader; nothing changes.- false+ self.deliver(state, r, target, partition, request);+ true+ }++ /// Deliver one request from router `r` to `x` for `partition`.+ fn deliver(+ &self,+ state: &mut SystemState,+ r: RouterId,+ x: PodId,+ partition: Partition,+ entry: StashedRequest,+ ) {+ if state.routers[&r]+ .stash+ .iter()+ .any(|parked| parked.key() == entry.key())+ {+ state.overtaken_request = true;+ }+ match entry {+ StashedRequest::Write(key) => {+ if self.validates(state, x, partition, key)+ && self.write_capable(state, x, partition)+ {+ self.accept_write(state, x, partition, key);+ }+ // Rejected drains surface UNAVAILABLE to the client (never acked).+ }+ StashedRequest::StrongRead(key) => {+ self.serve_read(state, x, partition, key);+ }+ }+ }++ /// The router adopts the live config (`LiveChangelog::adopt`).+ fn adopt(state: &mut SystemState, r: RouterId) {+ let live = state.config.live;+ let router = state.routers.get_mut(&r).unwrap();+ if router.adopted.epoch < live.epoch {+ router.adopted = live;}}@@ -586,30 +892,89 @@ impl HandoffModel {match *action {// Decided in the arm: enablement is "the planner produced a// non-empty plan", which costs a full placement computation.- Action::Rebalance => true,+ Action::Rebalance => Self::planner_active(state),// One pending suffix at a time: chunks of one plan apply// sequentially in production; a competing plan is the// Rebalance interleavings.- Action::RebalanceChunked => self.chunked_plans && state.pending_plan.is_empty(),+ Action::RebalanceChunked => {+ self.chunked_plans && state.pending_plan.is_empty() && Self::planner_active(state)+ }Action::ApplyPendingUnit => !state.pending_plan.is_empty(),- Action::CancelDeadNewOwner(p) => matches!(- state.handoffs.get(&p),- Some(h) if h.phase != Phase::Complete && !state.pods[&h.new_owner].registered- ),+ // The planner's cancellations stand down with the planner and never touch a hold;+ // production's fallback deletes are guarded per record and not modeled.+ Action::CancelDeadNewOwner(p) => {+ Self::planner_active(state)+ && matches!(+ state.handoffs.get(&p),+ Some(h) if h.phase != Phase::Complete && !h.held && !state.pods[&h.new_owner].registered+ )+ }Action::Cancel(p) => {state.cancels_left > 0- && matches!(state.handoffs.get(&p), Some(h) if h.phase != Phase::Complete)+ && Self::planner_active(state)+ && matches!(state.handoffs.get(&p), Some(h) if h.phase != Phase::Complete && !h.held)}// The quorum/drain/warm predicates are the expensive half and- // stay in the arm; a handoff short of Complete must exist.+ // stay in the arm; a handoff short of Complete must exist, and+ // a hold stops at Draining.Action::AdvancePhase(p) => {- matches!(state.handoffs.get(&p), Some(h) if h.phase != Phase::Complete)+ matches!(+ state.handoffs.get(&p),+ Some(h) if h.phase != Phase::Complete && !(h.held && h.phase == Phase::Draining)+ )}Action::CleanupComplete(p) => {matches!(state.handoffs.get(&p), Some(h) if h.phase == Phase::Complete)}+ Action::MigrateRequest => {+ state.migrations_left > 0 && !state.requested && state.migration.is_none()+ }+ // Admission refuses a start mid-rollout or while a generation drains.+ Action::MigrateStart => {+ state.requested+ && state.migration.is_none()+ && state.config.draining.is_none()+ && self.hold_pods == 0+ && (!self.quiet_start || Self::quiet(state))+ }+ Action::MigrateRequestExpires => state.requested,+ Action::MigrateDrive => state.migration.is_some(),+ Action::MigrateCancel => state+ .migration+ .is_some_and(|m| m.phase.may_transition_to(MigrationPhase::Cancelling)),+ Action::SealTick(x) => {+ let pod = &state.pods[&x];+ pod.running+ && pod.registered+ && !pod.seal_broken+ && state+ .migration+ .is_some_and(|m| m.phase == MigrationPhase::Sealing)+ }+ Action::SealGraceElapses => {+ self.seal_grace+ && !state.seal_overdue+ && state+ .migration+ .is_some_and(|m| m.phase == MigrationPhase::Sealing)+ }+ Action::BreakSeal(x) => {+ let pod = &state.pods[&x];+ state.seal_breaks_left > 0+ && pod.running+ && pod.registered+ && !pod.seal_broken+ && state+ .migration+ .is_some_and(|m| m.phase == MigrationPhase::Sealing)+ }+ Action::AdoptConfig(r) => {+ let router = &state.routers[&r];+ router.running && router.adopted.epoch < state.config.live.epoch+ }+// Which transition a converging pod owes is `desired_state`'s// answer, derived in the arm; only a live, registered process// converges at all.@@ -717,7 +1082,7 @@ impl HandoffModel {¤t,&in_flight,&members,- self.partitions as u32,+ last.config.live.partitions as u32,);if plan.handoffs.is_empty() {return None;@@ -755,6 +1120,8 @@ impl HandoffModel {new_owner: pod_id(&planned.new_owner),phase: Phase::Freezing,quorum: quorum.clone(),+ held: false,+ config_epoch: state.config.live.epoch,},).is_some();@@ -793,6 +1160,7 @@ impl HandoffModel {.get(&partition).copied().unwrap_or(0),+ basis: Self::plan_basis(state),}}).collect();@@ -806,6 +1174,12 @@ impl HandoffModel {fn apply_pending_unit(&self, last: &SystemState) -> Option<SystemState> {let unit = last.pending_plan.first()?.clone();mutate(last, |state| {+ // `apply_plan_on`: a moved basis stands the rest of the plan down.+ if self.plan_basis_guard && unit.basis != Self::plan_basis(state) {+ state.pending_plan.clear();+ state.pending_unit_dropped = true;+ return;+ }state.pending_plan.remove(0);let version = state.assignment_versions@@ -820,6 +1194,8 @@ impl HandoffModel {}let id = state.next_handoff_id;state.next_handoff_id += 1;+ // Production stamps the epoch the plan was read under.+ let config_epoch = unit.basis.1;state.handoffs.insert(unit.partition,Handoff {@@ -828,6 +1204,8 @@ impl HandoffModel {new_owner: unit.new_owner,phase: Phase::Freezing,quorum: unit.quorum,+ held: false,+ config_epoch,},);state.pending_unit_applied = true;@@ -882,31 +1260,12 @@ impl HandoffModel {}}Phase::Draining => {- let registered: Vec<RegisteredPod> = self- .pod_ids()- .filter(|x| last.pods[x].registered)- .map(|x| RegisteredPod {- pod_name: pod_name(x),- generation: String::new(),- status: PodStatus::Ready,- registered_at: 0,- last_heartbeat: 0,- controller: None,- advertise_address: None,- })- .collect();- let acks: Vec<PodDrainedAck> = self- .pod_ids()- .filter_map(|x| {- last.drained_acks.get(&(p, x)).map(|id| PodDrainedAck {- pod_name: pod_name(x),- partition: p as u32,- acked_at: 0,- acked_at_ms: 0,- handoff_id: id.to_string(),- })- })- .collect();+ // A hold stops here; the migration driver replaces it.+ if h.held {+ return None;+ }+ let registered = production_registered(last);+ let acks = self.production_drained_acks(last, p);if !drain_satisfied(®istered, &acks, &production_handoff(p, h)) {return None;}@@ -960,23 +1319,21 @@ impl HandoffModel {match model_desired_state(x, last, p) {DesiredState::Serving => {let pod = &last.pods[&x];- if !pod.warmed.contains_key(&p) {+ match pod.warmed.get(&p) {// `warm_partition`, one step at a time (see// `warm_step` for the two orderings under// EpochFenced).- mutate(last, |state| self.warm_step(state, x, p, None))- } else if pod.fenced.contains(&p) {- mutate(last, |state| {- // `resume_partition`. The cancelled handoff's- // target may have taken the broker fence from- // this pod's producer; production re-acquires it- // before re-admitting writes, or every write- // would fail as fenced until the next handoff.- state.changelogs.get_mut(&p).unwrap().epoch_holder = Some(x);- state.pods.get_mut(&x).unwrap().fenced.remove(&p);- })- } else {- None+ None => mutate(last, |state| self.warm_step(state, x, p, None)),+ Some(w) if pod.fenced.contains(&p) => {+ let log = (w.epoch, p);+ mutate(last, |state| {+ // `resume_partition`: re-take the broker fence the cancelled+ // target may hold, or every write fails as fenced.+ state.changelogs.get_mut(&log).unwrap().epoch_holder = Some(x);+ state.pods.get_mut(&x).unwrap().fenced.remove(&p);+ })+ }+ Some(_) => None,}}DesiredState::Drained { ack } => {@@ -1005,6 +1362,10 @@ impl HandoffModel {}DesiredState::Acquiring => {let h_id = last.handoffs[&p].id;+ // The capture parks until the config reaches the handoff's epoch.+ if last.config.live.epoch < last.handoffs[&p].config_epoch {+ return None;+ }// Only a warm installed for *this* handoff satisfies its// warming; one from an earlier era is released and// rebuilt (its cutoff can predate writes accepted since —@@ -1066,7 +1427,7 @@ impl HandoffModel {/// router already agreeing with durable state costs no clone.fn observe(&self, last: &SystemState, r: RouterId, p: Partition) -> Option<SystemState> {let router = &last.routers[&r];- let parked = router.stash.get(&p).is_some_and(|q| !q.is_empty());+ let parked = Self::parked_at(router, p);match last.handoffs.get(&p) {Some(h) if h.phase != Phase::Complete => {// begin_stash on every non-terminal phase; FreezeAck only@@ -1086,8 +1447,7 @@ impl HandoffModel {})}Some(h) => {- // Complete: cutover the table, then drain the stash to- // the new owner in FIFO order.+ // Complete: cut over the table, then drain `p`'s requests to the new owner.let new_owner = h.new_owner;if router.table.get(&p) == Some(&new_owner)&& !router.stashing.contains(&p)@@ -1100,21 +1460,10 @@ impl HandoffModel {let router = state.routers.get_mut(&r).unwrap();router.table.insert(p, new_owner);router.stashing.remove(&p);- router.stash.remove(&p).unwrap_or_default()+ Self::take_parked(state, r, p)};for entry in parked {- match entry {- StashedRequest::Write => {- if self.write_capable(state, new_owner, p) {- self.accept_write(state, new_owner, p);- }- // Rejected drains surface UNAVAILABLE to- // the client (never acked).- }- StashedRequest::StrongRead => {- self.serve_read(state, new_owner, p);- }- }+ self.deliver(state, r, new_owner, p, entry);}})}@@ -1142,25 +1491,39 @@ impl HandoffModel {}}router.stashing.remove(&p);- router.stash.remove(&p).unwrap_or_default()+ Self::take_parked(state, r, p)};for entry in parked {let Some(owner) = assignment else { continue };- match entry {- StashedRequest::Write => {- if self.write_capable(state, owner, p) {- self.accept_write(state, owner, p);- }- }- StashedRequest::StrongRead => {- self.serve_read(state, owner, p);- }- }+ self.deliver(state, r, owner, p, entry);}})}}}++ fn production_drained_acks(&self, state: &SystemState, p: Partition) -> Vec<PodDrainedAck> {+ self.pod_ids()+ .filter_map(|x| {+ state.drained_acks.get(&(p, x)).map(|id| PodDrainedAck {+ pod_name: pod_name(x),+ partition: p as u32,+ acked_at: 0,+ acked_at_ms: 0,+ handoff_id: id.to_string(),+ })+ })+ .collect()+ }++ fn registered_names(state: &SystemState) -> HashSet<String> {+ state+ .pods+ .iter()+ .filter(|(_, pod)| pod.registered)+ .map(|(x, _)| pod_name(*x))+ .collect()+ }}impl Model for HandoffModel {@@ -1179,12 +1542,18 @@ impl Model for HandoffModel {warmed: BTreeMap::new(),fenced: BTreeSet::new(),pending_warm: BTreeMap::new(),+ sealed: BTreeSet::new(),+ seal_broken: false,zombie_writes_left: 0,claims_authority: true,},)}).collect();+ let live = Generation {+ partitions: self.partitions,+ epoch: 1,+ };let routers: BTreeMap<RouterId, Router> = self.router_ids().map(|id| {@@ -1196,16 +1565,16 @@ impl Model for HandoffModel {Router {registered: active,running: active,+ adopted: live,table: BTreeMap::new(),stashing: BTreeSet::new(),- stash: BTreeMap::new(),+ stash: Vec::new(),},)}).collect();- let changelogs: BTreeMap<Partition, Changelog> = self- .partition_ids()- .map(|p| (p, Changelog::default()))+ let changelogs: BTreeMap<(Epoch, Partition), Changelog> = (0..self.partitions)+ .map(|p| ((live.epoch, p), Changelog::default())).collect();vec![SystemState {@@ -1217,6 +1586,12 @@ impl Model for HandoffModel {next_handoff_id: 0,pending_plan: Vec::new(),assignment_versions: BTreeMap::new(),+ config: Config {+ live,+ draining: None,+ },+ migration: None,+ seal_acks: BTreeMap::new(),pods,routers,changelogs,@@ -1226,15 +1601,25 @@ impl Model for HandoffModel {rejoins_left: self.rejoins,router_joins_left: self.router_joins,cancels_left: self.cancels,+ migrations_left: self.migrations,read_served: false,lost_acked_write: false,double_planned_handoff: false,stale_strong_read: false,+ mixed_modulus_acceptance: false,+ overtaken_request: false,reaffirmed: false,replaced_with_successor: false,cancelled_while_stash_parked: false,pending_unit_applied: false,pending_unit_dropped: false,+ migration_completed: false,+ migration_cancelled: false,+ sealed_without_warm: false,+ born_partition_write_parked: false,+ seal_overdue: false,+ seal_breaks_left: self.seal_breaks,+ requested: false,}]}@@ -1252,6 +1637,11 @@ impl Model for HandoffModel {offer(Action::Rebalance);offer(Action::RebalanceChunked);offer(Action::ApplyPendingUnit);+ offer(Action::MigrateRequest);+ offer(Action::MigrateStart);+ offer(Action::MigrateRequestExpires);+ offer(Action::MigrateDrive);+ offer(Action::MigrateCancel);for p in self.partition_ids() {offer(Action::CancelDeadNewOwner(p));offer(Action::Cancel(p));@@ -1262,11 +1652,21 @@ impl Model for HandoffModel {}for r in self.router_ids() {offer(Action::Observe(r, p));- offer(Action::ClientWrite(r, p));- offer(Action::ClientStrongRead(r, p));}}+ for key in self.key_ids() {+ for r in self.router_ids() {+ offer(Action::ClientWrite(r, key));+ offer(Action::ClientStrongRead(r, key));+ }+ }+ for r in self.router_ids() {+ offer(Action::AdoptConfig(r));+ }+ offer(Action::SealGraceElapses);for pod in self.pod_ids() {+ offer(Action::SealTick(pod));+ offer(Action::BreakSeal(pod));offer(Action::CrashRestartWithinTtl(pod));offer(Action::LeaseExpire(pod));}@@ -1333,11 +1733,35 @@ impl Model for HandoffModel {// (guarded delete of the record and its acks).Action::CleanupComplete(p) => mutate(last, |state| {state.handoffs.remove(&p);- state.freeze_acks.retain(|(fp, _), _| fp != &p);- state.drained_acks.retain(|(dp, _), _| dp != &p);- state.warmed_acks.retain(|(wp, _), _| wp != &p);+ Self::clear_acks(state, p);}),+ // ── the migration saga ─────────────────────────────+ Action::MigrateRequest => mutate(last, |state| {+ state.migrations_left -= 1;+ state.requested = true;+ }),+ Action::MigrateRequestExpires => mutate(last, |state| state.requested = false),+ Action::MigrateStart => mutate(last, |state| {+ state.requested = false;+ state.migration = Some(Migration {+ from: state.config.live,+ to: self.migrate_to,+ phase: MigrationPhase::Freezing,+ });+ }),+ Action::MigrateDrive => self.migrate_drive(last),+ Action::MigrateCancel => mutate(last, |state| {+ Self::set_phase(state, MigrationPhase::Cancelling);+ }),+ Action::SealTick(x) => self.seal_tick(last, x),+ Action::SealGraceElapses => mutate(last, |state| state.seal_overdue = true),+ Action::BreakSeal(x) => mutate(last, |state| {+ state.seal_breaks_left -= 1;+ state.pods.get_mut(&x).unwrap().seal_broken = true;+ }),+ Action::AdoptConfig(r) => mutate(last, |state| Self::adopt(state, r)),+// ── pod converge ───────────────────────────────────Action::Converge(x, p) => self.converge(last, x, p),@@ -1345,37 +1769,19 @@ impl Model for HandoffModel {Action::Observe(r, p) => self.observe(last, r, p),// ── workload ───────────────────────────────────────- Action::ClientWrite(r, p) => mutate(last, |state| {+ Action::ClientWrite(r, key) => mutate(last, |state| {state.writes_left -= 1;// A rejected write still consumes its budget: the state// changed, so the transition is kept and rejected paths// are explored too.- self.route_write(state, r, p);+ self.route(state, r, StashedRequest::Write(key));}),- Action::ClientStrongRead(r, p) => mutate(last, |state| {+ Action::ClientStrongRead(r, key) => mutate(last, |state| {state.reads_left -= 1;- let router = &state.routers[&r];- if !router.running {- // Router process gone; request fails at the client.- } else if router.stashing.contains(&p) {- // Strong reads park in the same per-partition FIFO as- // writes while the partition is stashing (the shipped- // read-stashing design, #69456) — which is what keeps- // them complete across cutover.- state- .routers- .get_mut(&r)- .unwrap()- .stash- .entry(p)- .or_default()- .push(StashedRequest::StrongRead);- } else if let Some(target) = router.table.get(&p).copied() {- // Outside a handoff: forward to the table entry.- // Rejected reads fail closed at the leader.- self.serve_read(state, target, p);- }+ // Strong reads take the write path: parked while stashing+ // (#69456), else forwarded to the table entry.+ self.route(state, r, StashedRequest::StrongRead(key));}),// ── failures ───────────────────────────────────────@@ -1387,6 +1793,8 @@ impl Model for HandoffModel {pod.warmed.clear();pod.fenced.clear();pod.pending_warm.clear();+ pod.sealed.clear();+ pod.seal_broken = false;pod.zombie_writes_left = 0;// A new session is a new claim.pod.claims_authority = true;@@ -1397,6 +1805,8 @@ impl Model for HandoffModel {pod.warmed.clear();pod.fenced.clear();pod.pending_warm.clear();+ pod.sealed.clear();+ pod.seal_broken = false;}),Action::LeaseExpire(x) => mutate(last, |state| {state.crashes_left -= 1;@@ -1440,6 +1850,8 @@ impl Model for HandoffModel {pod.warmed.clear();pod.fenced.clear();pod.pending_warm.clear();+ pod.sealed.clear();+ pod.seal_broken = false;pod.zombie_writes_left = 0;pod.claims_authority = false;}),@@ -1527,12 +1939,39 @@ impl Model for HandoffModel {// model found the cutover race as a counterexample — the// machine validation that motivated shipping read stashing.Property::<Self>::always("strong_reads_complete", |_, s| !s.stale_strong_read),+ Property::<Self>::always("no_overtaken_request", |_, s| !s.overtaken_request),+ // An ack lands in the live generation, never a retired one.+ Property::<Self>::always("no_mixed_modulus_acceptance", |_, s| {+ !s.mixed_modulus_acceptance+ }),+ // The ordinary handoff loop never moves a hold past Draining.+ Property::<Self>::always("holds_stop_at_draining", |_, s| {+ s.handoffs+ .values()+ .all(|h| !h.held || matches!(h.phase, Phase::Freezing | Phase::Draining))+ }),+ // Once flipped, no pod can produce to the old generation.+ Property::<Self>::always("old_generation_quiet_after_flip", |m, s| {+ s.config.draining.is_none_or(|g| {+ !s.pods.iter().any(|(x, pod)| {+ pod.warmed+ .iter()+ .any(|(p, w)| w.epoch == g.epoch && m.write_capable(s, *x, *p))+ })+ })+ }),+ Property::<Self>::sometimes("migration_completes", |m, s| {+ m.migrations == 0 || s.migration_completed+ }),+ Property::<Self>::sometimes("migration_cancels", |m, s| {+ m.migrations == 0 || s.migration_cancelled+ }),// Sanity: the interesting states are actually reachable.Property::<Self>::sometimes("some_handoff_completes", |_, s| {s.handoffs.values().any(|h| h.phase == Phase::Complete)}),Property::<Self>::sometimes("some_write_accepted", |_, s| {- s.changelogs.values().any(|log| log.len > 0)+ s.changelogs.values().any(|log| !log.is_empty())}),Property::<Self>::sometimes("some_strong_read_served", |m, s| {// Vacuously discoverable in configs without reads, so@@ -1556,13 +1995,18 @@ impl Model for HandoffModel {// Liveness: every full run ends quiescent and converged —// no handoffs in flight, every assignment served by a warm,// unfenced, registered pod, all routers agreeing, nothing- // parked in a stash.+ // parked in a stash, and no migration left to request.Property::<Self>::eventually("converges_to_stable", |m, s| {// Ordered cheapest-first on purpose. This runs at every// explored state, and the placement computation below is// the one expensive part — it must stay behind the// conjuncts that reject most states outright.- if !s.handoffs.is_empty() || !s.pending_plan.is_empty() {+ if !s.handoffs.is_empty()+ || !s.pending_plan.is_empty()+ || s.migration.is_some()+ || s.requested+ || s.migrations_left > 0+ {return false;}let no_capacity = s.pods.values().all(|p| !p.registered);@@ -1573,7 +2017,7 @@ impl Model for HandoffModel {// rebuilds `String` maps and reruns the production// strategy on every call.let targets = m.target_assignments(s);- m.partition_ids().all(|p| {+ (0..s.config.live.partitions).all(|p| {let Some(target) = targets.get(&(p as u32)).map(|name| pod_id(name))else {return true;@@ -1602,14 +2046,25 @@ impl Model for HandoffModel {if !router.registered || !router.running {return true;}- router.stashing.is_empty()- && router.stash.values().all(|q| q.is_empty())+ router.adopted == s.config.live+ && router.stashing.is_empty()+ && router.stash.is_empty()&& s.assignments.iter().all(|(p, owner)| router.table.get(p) == Some(owner))})}),];+ if self.probes && self.migrations > 0 {+ props.push(Property::<Self>::sometimes(+ "sealed_without_warm",+ |_, s| s.sealed_without_warm,+ ));+ props.push(Property::<Self>::sometimes(+ "born_partition_write_parked",+ |_, s| s.born_partition_write_parked,+ ));+ }if self.probes {// The freeze-quorum snapshot doing its job: a handoff// advanced past Freezing while some registered router — adiff --git a/rust/personhog-stateright/src/model/migration.rs b/rust/personhog-stateright/src/model/migration.rsnew file mode 100644index 000000000000..438b739cd988--- /dev/null+++ b/rust/personhog-stateright/src/model/migration.rs@@ -0,0 +1,318 @@+//! The migration saga's actions: the coordinator's driver passes and the leaders' seal duty.++use super::*;++impl HandoffModel {+ fn held_handoffs(state: &SystemState) -> Vec<(Partition, Handoff)> {+ state+ .handoffs+ .iter()+ .filter(|(_, h)| h.held)+ .map(|(p, h)| (*p, h.clone()))+ .collect()+ }++ pub(super) fn set_phase(state: &mut SystemState, next: MigrationPhase) {+ let migration = state.migration.as_mut().expect("a migration is in flight");+ assert!(+ migration.phase.may_transition_to(next),+ "{:?} -> {next:?}",+ migration.phase+ );+ migration.phase = next;+ }++ fn insert_handoff(state: &mut SystemState, p: Partition, handoff: Handoff) {+ Self::clear_acks(state, p);+ state.handoffs.insert(p, handoff);+ }++ /// `migration::reconcile`, one pass.+ pub(super) fn migrate_drive(&self, last: &SystemState) -> Option<SystemState> {+ let migration = last.migration?;+ mutate(last, |state| match migration.phase {+ MigrationPhase::Freezing => self.drive_freeze(state, migration),+ MigrationPhase::Draining => self.drive_drain(state, migration),+ MigrationPhase::Sealing => Self::drive_seal(state, migration),+ MigrationPhase::Resuming => self.drive_resume(state, migration),+ MigrationPhase::Cancelling => Self::drive_unwind(state, migration),+ MigrationPhase::Complete | MigrationPhase::Cancelled => {+ state.migration = None;+ }+ })+ }++ /// `Pass::freeze`, through the production `plan_freeze`.+ fn drive_freeze(&self, state: &mut SystemState, migration: Migration) {+ let from = migration.from;+ let handoffs: Vec<HandoffState> = state+ .handoffs+ .iter()+ .map(|(p, h)| production_handoff(*p, h))+ .collect();+ let registered = Self::registered_names(state);+ let registered: HashSet<&str> = registered.iter().map(String::as_str).collect();+ let plan = plan_freeze(+ from.partitions as u32,+ &production_assignments(state),+ &handoffs,+ ®istered,+ );+ if self.freeze_checks_strays && plan.stray_handoff.is_some() {+ Self::set_phase(state, MigrationPhase::Cancelling);+ return;+ }+ if plan.holder_died.is_some() {+ Self::set_phase(state, MigrationPhase::Cancelling);+ return;+ }+ let quorum = Self::quorum_snapshot(state);+ for (p, owner) in plan.holds {+ let id = state.next_handoff_id;+ state.next_handoff_id += 1;+ let owner = pod_id(&owner);+ state.handoffs.insert(+ p as Partition,+ Handoff {+ id,+ old_owner: Some(owner),+ new_owner: owner,+ phase: Phase::Freezing,+ quorum: quorum.clone(),+ held: true,+ config_epoch: from.epoch,+ },+ );+ }+ if plan.blocking.is_empty() {+ Self::set_phase(state, MigrationPhase::Draining);+ }+ }++ /// `Pass::drain`: every hold drained, else a dead holder cancels.+ fn drive_drain(&self, state: &mut SystemState, _migration: Migration) {+ let registered = production_registered(state);+ let mut blocked = false;+ for (p, h) in Self::held_handoffs(state) {+ if !state.pods[&h.new_owner].registered {+ Self::set_phase(state, MigrationPhase::Cancelling);+ return;+ }+ if h.phase == Phase::Freezing {+ blocked = true;+ continue;+ }+ let acks = self.production_drained_acks(state, p);+ blocked |= !drain_satisfied(®istered, &acks, &production_handoff(p, &h));+ }+ if !blocked {+ Self::set_phase(state, MigrationPhase::Sealing);+ }+ }++ /// `Pass::seal`: the flip once every old partition is sealed, as one transaction.+ fn drive_seal(state: &mut SystemState, migration: Migration) {+ let acks = production_seal_acks(state);+ if !migration_unsealed(&acks, MIGRATION_ID, migration.from.partitions as u32).is_empty() {+ return;+ }+ let next = Generation {+ partitions: migration.to,+ epoch: state.config.live.epoch + 1,+ };+ for p in 0..next.partitions {+ state.changelogs.entry((next.epoch, p)).or_default();+ }+ state.config = Config {+ live: next,+ draining: Some(migration.from),+ };+ state.seal_overdue = false;+ Self::set_phase(state, MigrationPhase::Resuming);+ }++ /// `Pass::resume`: holds become restarts at Warming, shrunk partitions+ /// dissolve, grown ones are placed; then Complete once settled.+ fn drive_resume(&self, state: &mut SystemState, migration: Migration) {+ let to = migration.to;+ let epoch = state.config.live.epoch;+ let held = Self::held_handoffs(state);+ if !held.is_empty() {+ let holds: Vec<(u32, String)> = held+ .iter()+ .map(|(p, h)| (*p as u32, pod_name(h.new_owner)))+ .collect();+ let handoffs: Vec<HandoffState> = state+ .handoffs+ .iter()+ .map(|(p, h)| production_handoff(*p, h))+ .collect();+ let names = Self::registered_names(state);+ let registered: HashSet<&str> = names.iter().map(String::as_str).collect();+ let plan = plan_resume(+ &StickyBalancedStrategy,+ &self.planner_members(state),+ to as u32,+ &holds,+ &production_assignments(state),+ &handoffs,+ ®istered,+ );+ for p in plan.dissolve {+ let p = p as Partition;+ Self::clear_acks(state, p);+ state.handoffs.remove(&p);+ state.assignments.remove(&p);+ }+ for p in plan.abandon {+ let p = p as Partition;+ Self::clear_acks(state, p);+ state.handoffs.remove(&p);+ }+ for (p, holder) in plan.restart {+ let id = state.next_handoff_id;+ state.next_handoff_id += 1;+ let holder = pod_id(&holder);+ Self::insert_handoff(+ state,+ p as Partition,+ Handoff {+ id,+ old_owner: Some(holder),+ new_owner: holder,+ phase: Phase::Warming,+ quorum: BTreeSet::new(),+ held: false,+ config_epoch: epoch,+ },+ );+ }+ let quorum = Self::quorum_snapshot(state);+ for (p, owner) in plan.place {+ let id = state.next_handoff_id;+ state.next_handoff_id += 1;+ state.handoffs.insert(+ p as Partition,+ Handoff {+ id,+ old_owner: None,+ new_owner: pod_id(&owner),+ phase: Phase::Freezing,+ quorum: quorum.clone(),+ held: false,+ config_epoch: epoch,+ },+ );+ }+ return;+ }+ let handoffs: Vec<HandoffState> = state+ .handoffs+ .iter()+ .map(|(p, h)| production_handoff(*p, h))+ .collect();+ let registered = Self::registered_names(state);+ let registered: HashSet<&str> = registered.iter().map(String::as_str).collect();+ let unsettled = migration_unsettled(+ &production_assignments(state),+ &handoffs,+ ®istered,+ to as u32,+ );+ if unsettled.is_empty() {+ state.migration = None;+ state.seal_acks.clear();+ state.migration_completed = true;+ }+ }++ /// `Pass::unwind`: each hold becomes a Complete reaffirm, then the record is archived.+ fn drive_unwind(state: &mut SystemState, migration: Migration) {+ let held = Self::held_handoffs(state);+ if held.is_empty() {+ state.migration = None;+ state.seal_acks.clear();+ state.migration_cancelled = true;+ return;+ }+ for (p, h) in held {+ if !state.pods[&h.new_owner].registered {+ Self::clear_acks(state, p);+ state.handoffs.remove(&p);+ continue;+ }+ let id = state.next_handoff_id;+ state.next_handoff_id += 1;+ Self::insert_handoff(+ state,+ p,+ Handoff {+ id,+ old_owner: None,+ new_owner: h.new_owner,+ phase: Phase::Complete,+ quorum: BTreeSet::new(),+ held: false,+ config_epoch: migration.from.epoch,+ },+ );+ }+ }++ /// `SealDuty::tick`: seal every target not yet sealed, then rewrite the ack.+ pub(super) fn seal_tick(&self, last: &SystemState, x: PodId) -> Option<SystemState> {+ let migration = last.migration?;+ let from = migration.from;+ let targets: Vec<Partition> = match self.seal_rule {+ SealRule::Assignment => {+ let registered = Self::registered_names(last);+ let registered: HashSet<&str> = registered.iter().map(String::as_str).collect();+ seal_targets(+ &pod_name(x),+ MIGRATION_ID,+ from.partitions as u32,+ &production_assignments(last),+ ®istered,+ &production_seal_acks(last),+ last.seal_overdue,+ )+ .into_iter()+ .map(|p| p as Partition)+ .collect()+ }+ SealRule::Residency => (0..from.partitions)+ .filter(|p| {+ let orphaned = last+ .assignments+ .get(p)+ .is_none_or(|owner| !last.pods[owner].registered);+ last.pods[&x].warmed.contains_key(p) || orphaned+ })+ .collect(),+ };+ let pod = &last.pods[&x];+ let newly: Vec<Partition> = targets+ .iter()+ .copied()+ .filter(|p| !pod.sealed.contains(p))+ .collect();+ // `SealDuty::tick`: everything this pod sealed stays reported.+ let acked: BTreeSet<Partition> = pod.sealed.iter().chain(newly.iter()).copied().collect();+ let unloaded = newly.iter().any(|p| !pod.warmed.contains_key(p));+ mutate(last, |state| {+ for p in &newly {+ // `init_transactions` by the sealer, then its producer is dropped.+ state+ .changelogs+ .get_mut(&(from.epoch, *p))+ .unwrap()+ .epoch_holder = None;+ state.pods.get_mut(&x).unwrap().sealed.insert(*p);+ }+ if unloaded {+ state.sealed_without_warm = true;+ }+ state.seal_acks.insert(x, acked);+ })+ }+}diff --git a/rust/personhog-stateright/src/types.rs b/rust/personhog-stateright/src/types.rsindex 8e6bc803fb52..1bf679417026 100644--- a/rust/personhog-stateright/src/types.rs+++ b/rust/personhog-stateright/src/types.rs@@ -13,10 +13,17 @@ pub type Partition = u8;/// Identity of one handoff attempt — production `HandoffState.handoff_id`./// Acks echo it and quorum checks only count matching acks.pub type HandoffId = u8;+/// A changelog generation's epoch (production `ChangelogConfig::epoch`).+pub type Epoch = u8;+/// A person key. The model's hash is the identity: like production's+/// hash-then-mod, a key's homes under two counts agree modulo their gcd.+pub type Key = u8;/// The production phase enum, used directly: a new phase added to the/// protocol breaks the model's exhaustive matches at compile time.pub use personhog_coordination::types::HandoffPhase as Phase;+/// The production saga phases, used directly for the same reason.+pub use personhog_coordination::types::MigrationPhase;/// Production `HandoffState`, in compact model form (u8 ids; the model/// materializes production-typed views when calling shared logic).@@ -33,11 +40,37 @@ pub struct Handoff {/// With `RouterJoin` in the action space this diverges from the live/// registry, which is exactly what the checker is here to explore.pub quorum: BTreeSet<RouterId>,+ /// A migration's hold (production `held_by`): never advanced past Draining.+ pub held: bool,+ /// Production `config_epoch`: an acquisition parks until the config reaches it.+ pub config_epoch: Epoch,+}++/// One changelog generation (production: the live target, or a `draining` entry).+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]+pub struct Generation {+ pub partitions: u8,+ pub epoch: Epoch,+}++/// Production `ChangelogConfig`: the live generation and at most one draining.+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]+pub struct Config {+ pub live: Generation,+ pub draining: Option<Generation>,+}++/// Production `MigrationState`.+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]+pub struct Migration {+ pub from: Generation,+ pub to: u8,+ pub phase: MigrationPhase,}/// Per-partition warm state on a pod — everything the invariants need to/// know about one warmed partition.-#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]+#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]pub struct WarmState {/// What this warm was installed for (production:/// `WarmProvenance` on the pod handle). `None` is a serving-era warm@@ -48,17 +81,11 @@ pub struct WarmState {/// whose cache may predate writes accepted since, and is released/// and rebuilt.pub for_handoff: Option<HandoffId>,- /// How much of the changelog this pod's cache reflects: the HWM- /// captured at warm time, plus every write it has accepted since. A- /// strong read served while `changelog.len` exceeds this returns state- /// missing at least one acked write.- ///- /// The two halves are deliberately one number. Nothing reads them- /// apart — the cutoff is never consulted on its own, and the accept- /// count only ever advances the same total — so keeping them separate- /// would only split behaviorally identical states (warmed at 1 and- /// accepted nothing, versus warmed at 0 and accepted one).- pub visible: u8,+ /// The epoch captured at acquisition (production `Capture`).+ pub epoch: Epoch,+ /// Visible prefix per source log (production: marks keyed by stream);+ /// the live source advances with each accepted write.+ pub sources: BTreeMap<(Epoch, Partition), u8>,}/// A warm caught between its two steps, under the rejected read-first@@ -72,7 +99,7 @@ pub struct WarmState {/// warm atomically.#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]pub struct PendingWarm {- /// The changelog length captured before the fence existed.+ /// The live log's length captured before the fence existed.pub cutoff: u8,}@@ -94,6 +121,10 @@ pub struct Pod {/// Write-fenced partitions (production: `InflightTracker` fences +/// the pod handle's `fenced_partitions`).pub fenced: BTreeSet<Partition>,+ /// Old-topic partitions this incarnation sealed (the seal duty's memory).+ pub sealed: BTreeSet<Partition>,+ /// This incarnation's seal never succeeds.+ pub seal_broken: bool,/// Remaining writes this pod may accept after losing its lease before/// its keepalive notices and the process self-fences (production: the/// bounded zombie window; fix 1 bounds it to ~one heartbeat tick).@@ -122,42 +153,42 @@ pub struct Router {pub registered: bool,/// The process is running and forwards traffic.pub running: bool,+ /// The config this router hashes with (production `LiveChangelog`).+ pub adopted: Generation,/// partition → pod the router forwards to (production: the shared/// routing table, edge-updated from handoff Complete events).pub table: BTreeMap<Partition, PodId>,/// Partitions currently buffering leader-path requests (production:- /// the `StashTable`).+ /// the `StashTable`'s open freezes).pub stashing: BTreeSet<Partition>,- /// Parked leader-path requests in arrival order (production:- /// `StashedRequest` queues, which carry their gRPC method so writes- /// and strong reads share one per-partition FIFO).- pub stash: BTreeMap<Partition, Vec<StashedRequest>>,+ /// Parked leader-path requests in arrival order (production: the `StashTable`'s+ /// per-key queues). A drain for `p` takes those whose key maps to `p` under the router's count.+ pub stash: Vec<StashedRequest>,}/// One parked leader-path request.-///-/// Deliberately identity-free. A parked write is only ever drained or-/// dropped, never matched against a particular client, so carrying a write-/// id would add a dimension to the state space that nothing reads — two-/// otherwise-identical states would differ only in how many writes had ever-/// been parked. Order and count still matter, and the per-partition `Vec`-/// keeps both.#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]pub enum StashedRequest {- Write,- StrongRead,+ Write(Key),+ StrongRead(Key),+}++impl StashedRequest {+ pub fn key(self) -> Key {+ match self {+ StashedRequest::Write(key) | StashedRequest::StrongRead(key) => key,+ }+ }}-/// The Kafka changelog for one partition, reduced to what the safety-/// invariants need: an append counter (the HWM) and who currently holds-/// the broker's producer fence.+/// The Kafka changelog for one partition of one generation: the appended+/// keys (their count is the HWM) and the broker's producer fence.#[derive(Clone, Debug, Default, PartialEq, Eq, Hash, PartialOrd, Ord)]pub struct Changelog {- /// Number of records appended (the HWM).- pub len: u8,+ pub records: Vec<Key>,/// The pod whose transactional producer the broker currently accepts:/// the latest `init_transactions` wins and every earlier producer is- /// fenced out.+ /// fenced out. A seal initializes the id and drops the producer.////// This is an owner rather than an epoch number because nothing/// compares epochs for order — the single consumer asks whether *this*@@ -168,6 +199,16 @@ pub struct Changelog {pub epoch_holder: Option<PodId>,}+impl Changelog {+ pub fn len(&self) -> u8 {+ self.records.len() as u8+ }++ pub fn is_empty(&self) -> bool {+ self.records.is_empty()+ }+}+/// One planned-but-unapplied handoff from a chunked plan: the placement/// decision plus its plan-time guards (production: a later `apply_plan`/// chunk, applied without re-reading).@@ -181,6 +222,8 @@ pub struct PendingUnit {/// Assignment version at plan time (production: the mod_revision/// its `AssignmentPrecondition` compares; 0 = Absent).pub expected_assignment_version: u8,+ /// The plan's basis (production `PlanBasis`): a saga in flight, and the live epoch.+ pub basis: (bool, Epoch),}/// The entire distributed system. One value = one node in the explored@@ -209,13 +252,20 @@ pub struct SystemState {/// mod_revision an `AssignmentPrecondition` compares). Only/// maintained under `chunked_plans`.pub assignment_versions: BTreeMap<Partition, u8>,+ /// Production `config/changelog`.+ pub config: Config,+ /// Production `migrations/current`.+ pub migration: Option<Migration>,+ /// pod → old-topic partitions its seal ack reports sealed.+ pub seal_acks: BTreeMap<PodId, BTreeSet<Partition>>,// ── processes ──────────────────────────────────────────────pub pods: BTreeMap<PodId, Pod>,pub routers: BTreeMap<RouterId, Router>,// ── kafka ──────────────────────────────────────────────────- pub changelogs: BTreeMap<Partition, Changelog>,+ /// (generation epoch, partition) → its log.+ pub changelogs: BTreeMap<(Epoch, Partition), Changelog>,// ── failure/workload budgets (bound the state space) ───────pub writes_left: u8,@@ -227,6 +277,7 @@ pub struct SystemState {pub router_joins_left: u8,/// Deadline cancellations the checker may still inject.pub cancels_left: u8,+ pub migrations_left: u8,/// Whether any strong read was actually served (reachability evidence/// for the read properties). A flag rather than a count: nothing reads/// how many, and a count would split otherwise-identical states by@@ -244,10 +295,14 @@ pub struct SystemState {/// in-flight handoff, clobbering it — the overlap/// `plan_partial_rebalance`'s pinning must make unreachable.pub double_planned_handoff: bool,- /// Set when a strong read is served by a pod whose `WarmState::visible`- /// prefix is behind the changelog — the read returned state missing at- /// least one acked write.+ /// Set when a strong read is served by a pod whose warm prefixes are+ /// behind some log holding the key.pub stale_strong_read: bool,+ /// Set when a pod acks a write under a captured epoch that is not the live one.+ pub mixed_modulus_acceptance: bool,+ /// Set when a router delivers a request while an earlier request for+ /// the same key is still parked, so the later one lands first.+ pub overtaken_request: bool,// ── reachability flags (probe evidence, history encoding) ──/// A cancellation resolved as a reaffirm-Complete toward the live@@ -263,6 +318,17 @@ pub struct SystemState {pub pending_unit_applied: bool,/// A pending chunk unit dropped by a failed plan-time guard.pub pending_unit_dropped: bool,+ pub migration_completed: bool,+ pub migration_cancelled: bool,+ /// A pod sealed an old-topic partition it had not loaded.+ pub sealed_without_warm: bool,+ /// A write for a partition born in the flip parked before its handoff existed.+ pub born_partition_write_parked: bool,+ /// A migration request is waiting to start.+ pub requested: bool,+ /// The Sealing grace has lapsed for the current saga.+ pub seal_overdue: bool,+ pub seal_breaks_left: u8,}/// Everything that can happen, from every actor, including failures.@@ -307,6 +373,26 @@ pub enum Action {/// (guarded on the record still being the same attempt).CleanupComplete(Partition),+ // ── the migration saga ─────────────────────────────────────+ /// The control plane writes `migrations/requested`. Budgeted.+ MigrateRequest,+ /// The coordinator starts the saga from the request once no handoff is in flight.+ MigrateStart,+ /// A request is rejected or expires before it starts.+ MigrateRequestExpires,+ /// One pass of the migration driver (`migration::reconcile`).+ MigrateDrive,+ /// A cancel before Sealing: the operator's, or the pre-flip deadline's.+ MigrateCancel,+ /// One pass of a pod's seal duty while the saga is Sealing.+ SealTick(PodId),+ /// The seal grace lapses: every leader now seals what no ack reports sealed.+ SealGraceElapses,+ /// A pod's seal keeps failing while its lease stays alive (a hung Kafka client).+ BreakSeal(PodId),+ /// A router's config watch delivers the live config.+ AdoptConfig(RouterId),+// ── pods (the converge model) ──────────────────────────────/// One convergence step for (pod, partition): derive the desired/// state from durable state and apply the next transition toward it@@ -321,13 +407,10 @@ pub enum Action {Observe(RouterId, Partition),// ── workload ───────────────────────────────────────────────- /// A client write for (router, partition): stashes if the router is- /// stashing, otherwise forwards to the router's table entry.- ClientWrite(RouterId, Partition),- /// A strong read for (router, partition): parks with the writes in- /// the per-partition FIFO when the partition is stashing (the shipped- /// read-stashing design), otherwise forwards to the table entry.- ClientStrongRead(RouterId, Partition),+ /// A client write for (router, key), hashed with the router's adopted modulus.+ ClientWrite(RouterId, Key),+ /// A strong read for (router, key); parks with the writes while stashing.+ ClientStrongRead(RouterId, Key),// ── failures ───────────────────────────────────────────────/// The pod process dies and instantly restarts under the same namediff --git a/rust/personhog-stateright/tests/model_tests.rs b/rust/personhog-stateright/tests/model_tests.rsindex 1c4274a3ec1b..e907d317a84b 100644--- a/rust/personhog-stateright/tests/model_tests.rs+++ b/rust/personhog-stateright/tests/model_tests.rs@@ -4,7 +4,7 @@use std::time::Instant;-use personhog_stateright::model::{ClaimDetection, HandoffModel, WarmOrder};+use personhog_stateright::model::{ClaimDetection, HandoffModel, SealRule, WarmOrder, WarmReplay};use stateright::{Checker, HasDiscoveries, Model};/// Every checker explores in parallel: stateright defaults to a single@@ -109,6 +109,17 @@ fn base() -> HandoffModel {cancels: 0,chunked_plans: false,probes: false,+ migrations: 0,+ migrate_to: 1,+ keys: 1,+ seal_rule: SealRule::Assignment,+ warm_replay: WarmReplay::Sources,+ seal_breaks: 0,+ seal_grace: true,+ plan_basis_guard: true,+ quiet_start: true,+ freeze_checks_strays: true,+ park_behind_key: true,}}@@ -745,6 +756,184 @@ fn chunked_plan_application_is_safe_and_live() {.assert_properties();}+/// A migration: one router, two pods, two keys, writes and a read racing every phase.+fn migration(partitions: u8, migrate_to: u8) -> HandoffModel {+ HandoffModel {+ routers: 1,+ partitions,+ migrate_to,+ migrations: 1,+ keys: 2,+ writes: 2,+ reads: 1,+ ..base()+ }+}++/// A grow holds, seals, flips and restarts with every safety property intact;+/// a cancel at any point before the seal unwinds in place.+#[test]+fn a_grow_migration_is_safe_and_live() {+ migration(1, 2)+ .explore("a_grow_migration_is_safe_and_live")+ .assert_properties();+}++/// The rejected bucket rule: a request that parks only when its partition+/// is stashing overtakes a request for its key parked under the old count.+#[test]+fn without_parking_behind_its_key_a_request_overtakes() {+ let checker = HandoffModel {+ park_behind_key: false,+ ..migration(1, 2)+ }+ .explore_until(+ "without_parking_behind_its_key_a_request_overtakes",+ &["no_overtaken_request"],+ );+ assert!(+ checker.discovery("no_overtaken_request").is_some(),+ "the bucket rule must let a later request overtake its key's parked one"+ );+}++/// A shrink dissolves the partition past the new space and still converges.+#[test]+fn a_shrink_migration_is_safe_and_live() {+ migration(2, 1)+ .explore("a_shrink_migration_is_safe_and_live")+ .assert_properties();+}++/// Two routers adopt the flip independently, so one hashes with the old+/// modulus while the other already drains under the new one.+#[test]+fn a_grow_migration_with_a_stale_router_is_safe_and_live() {+ HandoffModel {+ routers: 2,+ writes: 1,+ ..migration(1, 2)+ }+ .explore("a_grow_migration_with_a_stale_router_is_safe_and_live")+ .assert_properties();+}++/// A doubling where each new partition inherits one old one and keys split.+#[test]+fn a_two_to_four_grow_is_safe_and_live() {+ HandoffModel {+ writes: 1,+ keys: 4,+ ..migration(2, 4)+ }+ .explore("a_two_to_four_grow_is_safe_and_live")+ .assert_properties();+}++/// A holder that crash-restarts under its own name seals the partitions it+/// never reloaded, so the saga still completes.+#[test]+fn a_holder_restarted_mid_sealing_still_seals_and_the_saga_completes() {+ let checker = HandoffModel {+ crashes: 1,+ probes: true,+ writes: 1,+ ..migration(1, 2)+ }+ .explore("a_holder_restarted_mid_sealing_still_seals_and_the_saga_completes");+ checker.assert_properties();+ assert!(+ checker.discovery("sealed_without_warm").is_some(),+ "the restarted holder must seal a partition it never loaded"+ );+}++/// The rejected seal rule: targets drawn from what a pod has loaded leave a+/// restarted holder's partitions to nobody, and the saga never finishes.+#[test]+fn seal_targets_by_residency_wedge_a_restarted_holder() {+ let checker = HandoffModel {+ crashes: 1,+ writes: 1,+ seal_rule: SealRule::Residency,+ seal_grace: false,+ ..migration(1, 2)+ }+ .explore_until(+ "seal_targets_by_residency_wedge_a_restarted_holder",+ &["converges_to_stable"],+ );+ assert!(+ checker.discovery("converges_to_stable").is_some(),+ "residency-based seal targets must leave a run that never converges"+ );+}++/// The rejected rule: a registered holder whose seal never succeeds holds Sealing forever.+#[test]+fn a_broken_seal_without_the_grace_wedges_sealing() {+ let checker = HandoffModel {+ writes: 1,+ seal_breaks: 1,+ seal_grace: false,+ ..migration(1, 2)+ }+ .explore_until(+ "a_broken_seal_without_the_grace_wedges_sealing",+ &["converges_to_stable"],+ );+ assert!(+ checker.discovery("converges_to_stable").is_some(),+ "a broken seal with no grace must leave a run that never converges"+ );+}++/// Past the grace, peers seal what a broken holder could not, and the saga completes safely.+#[test]+fn a_broken_seal_is_healed_by_peers_after_the_grace() {+ let checker = HandoffModel {+ writes: 1,+ seal_breaks: 1,+ ..migration(1, 2)+ }+ .explore("a_broken_seal_is_healed_by_peers_after_the_grace");+ checker.assert_properties();+}++/// The rejected warm: a restart that reads the live partition alone misses+/// the old tail and serves a stale strong read.+#[test]+fn a_live_only_warm_loses_the_old_tail() {+ let checker = HandoffModel {+ warm_replay: WarmReplay::LiveOnly,+ ..migration(1, 2)+ }+ .explore_until(+ "a_live_only_warm_loses_the_old_tail",+ &["strong_reads_complete"],+ );+ assert!(+ checker.discovery("strong_reads_complete").is_some(),+ "a live-only warm must leave a stale strong read reachable"+ );+}++/// A write for a partition born in the flip parks before its handoff exists+/// and is delivered once it lands.+#[test]+fn probe_a_born_partition_write_parks_and_drains() {+ let checker = HandoffModel {+ probes: true,+ ..migration(1, 2)+ }+ .explore("probe_a_born_partition_write_parks_and_drains");+ checker.assert_properties();+ assert!(+ checker.discovery("born_partition_write_parked").is_some(),+ "a write must be able to park for a partition born in the flip"+ );+}+/// Both fates of a pending unit are genuinely reachable — applied after/// other actions interleaved, and dropped by a failed plan-time guard —/// and no interleaving that reaches them loses an acked write or@@ -777,3 +966,76 @@ fn probe_chunked_pending_units_reach_both_fates_safely() {);}}++/// A chunked bootstrap plan racing a migration: later units land only on+/// the basis they were planned on, so every property holds.+#[test]+fn a_chunked_plan_across_a_migration_is_safe_and_live() {+ HandoffModel {+ chunked_plans: true,+ writes: 1,+ reads: 0,+ ..migration(2, 4)+ }+ .explore("a_chunked_plan_across_a_migration_is_safe_and_live")+ .assert_properties();+}++/// A planner that ignores the plan basis serves the old generation past the flip. The quiet+/// start is off, since this small config reaches a mid-plan start only without it.+#[test]+fn an_unguarded_plan_across_a_migration_serves_the_old_generation() {+ let checker = HandoffModel {+ chunked_plans: true,+ writes: 1,+ reads: 0,+ plan_basis_guard: false,+ quiet_start: false,+ ..migration(2, 4)+ }+ .explore_until(+ "an_unguarded_plan_across_a_migration_serves_the_old_generation",+ &[+ "old_generation_quiet_after_flip",+ "no_mixed_modulus_acceptance",+ ],+ );+ assert!(checker+ .discovery("old_generation_quiet_after_flip")+ .is_some());+ assert!(checker.discovery("no_mixed_modulus_acceptance").is_some());+}++/// The rejected design: a saga that begins mid-handoff, with a freeze that+/// does not notice, freezes a fleet with a partition in motion and loses an acked write.+#[test]+fn starting_mid_handoff_loses_acked_writes() {+ let checker = HandoffModel {+ quiet_start: false,+ freeze_checks_strays: false,+ ..migration(1, 2)+ }+ .explore_until(+ "starting_mid_handoff_loses_acked_writes",+ &["no_lost_acked_write"],+ );+ assert!(checker.discovery("no_lost_acked_write").is_some());+}++/// The quiet start and the freeze's stray check are independent layers: each+/// alone keeps every property, the stray check by cancelling instead.+#[test]+fn either_layer_alone_keeps_a_mid_handoff_start_safe() {+ for (name, quiet_start, freeze_checks_strays) in [+ ("only_the_stray_check", false, true),+ ("only_the_quiet_start", true, false),+ ] {+ HandoffModel {+ quiet_start,+ freeze_checks_strays,+ ..migration(1, 2)+ }+ .explore(name)+ .assert_properties();+ }+}diff --git a/rust/personhog-test-harness/Cargo.toml b/rust/personhog-test-harness/Cargo.tomlindex e61abbd1fd78..d9ab8e5d50a4 100644--- a/rust/personhog-test-harness/Cargo.toml+++ b/rust/personhog-test-harness/Cargo.toml@@ -8,6 +8,7 @@ publish = falsepersonhog-common = { path = "../personhog-common" }personhog-proto = { path = "../personhog-proto" }common-metrics = { path = "../common/metrics" }+common-kafka = { path = "../common/kafka" }personhog-coordination = { path = "../personhog-coordination" }assignment-coordination = { path = "../common/assignment-coordination" }diff --git a/rust/personhog-test-harness/src/cli.rs b/rust/personhog-test-harness/src/cli.rsindex 595c2f5df2ee..c42c832fbe1d 100644--- a/rust/personhog-test-harness/src/cli.rs+++ b/rust/personhog-test-harness/src/cli.rs@@ -153,6 +153,10 @@ pub struct GateArgs {#[arg(long = "leader-env", value_parser = parse_env_pair)]pub leader_env: Vec<(String, String)>,+ /// Extra KEY=VALUE environment for spawned routers. Repeatable.+ #[arg(long = "router-env", value_parser = parse_env_pair)]+ pub router_env: Vec<(String, String)>,+/// Target an already-running stack at this router URL instead of/// spawning one. When unset, the harness spawns its own isolated stack/// (replica, leaders, leader-mode router, writer) against the@@ -303,6 +307,54 @@ pub struct GateArgs {#[arg(long, default_value_t = false)]pub kill_handoff_target: bool,+ /// Migrate the changelog to a fresh topic this long into the traffic phase.+ #[arg(long, value_parser = humantime::parse_duration)]+ pub migrate_after: Option<Duration>,++ /// Partition count of the migration's target topic.+ #[arg(long)]+ pub migrate_to_partitions: Option<u32>,++ /// SIGKILL the coordinator router once the saga reaches Sealing. Requires --routers >= 3.+ #[arg(long, default_value_t = false)]+ pub migrate_kill_coordinator: bool,++ /// SIGKILL the busiest leader once the saga reaches Resuming.+ #[arg(long, default_value_t = false)]+ pub migrate_kill_leader: bool,++ /// SIGSTOP the busiest leader once it has drained, then SIGKILL it (lease revoked) at Sealing, so its peers seal its partitions.+ #[arg(long, default_value_t = false)]+ pub migrate_kill_leader_at_sealing: bool,++ /// Crash-restart the busiest leader under its own name as soon as the saga is holding; it must seal what it never reloaded.+ #[arg(long, default_value_t = false)]+ pub migrate_restart_leader: bool,++ /// SIGSTOP the busiest leader before the request, lease kept: its partitions never drain, so the saga must cancel at the 5s pre-flip deadline and unwind.+ #[arg(long, default_value_t = false)]+ pub migrate_stall_leader: bool,++ /// SIGSTOP the busiest leader once it has drained, lease kept: its peers must seal for it after the seal grace and the saga complete. Pair with --migrate-slow-coordinator-ms so the drained holder is observable before Sealing.+ #[arg(long, default_value_t = false)]+ pub migrate_stall_leader_at_sealing: bool,++ /// Debounce between the coordinator's migration passes, to widen the flip-to-restart window.+ #[arg(long)]+ pub migrate_slow_coordinator_ms: Option<u64>,++ /// Run a second migration this long into the traffic phase.+ #[arg(long, value_parser = humantime::parse_duration)]+ pub migrate_again_after: Option<Duration>,++ /// Partition count of the second migration's target topic.+ #[arg(long)]+ pub migrate_again_to_partitions: Option<u32>,++ /// SIGSTOP the writer for --writer-pause-duration as the migration starts; the saga must not wait for it.+ #[arg(long, default_value_t = false)]+ pub migrate_pause_writer: bool,+/// Leader per-partition cache budget in bytes (CACHE_MEMORY_CAPACITY_BYTES)./// Set below the seeded pool's footprint to put the cache under/// eviction pressure. Default matches the dev deployment (16 MiB).diff --git a/rust/personhog-test-harness/src/scenarios/blast.rs b/rust/personhog-test-harness/src/scenarios/blast.rsindex 59948db333fd..e596aece55bb 100644--- a/rust/personhog-test-harness/src/scenarios/blast.rs+++ b/rust/personhog-test-harness/src/scenarios/blast.rs@@ -277,7 +277,7 @@ pub async fn verify_strong(person_id,key: "__strong_read_failed".to_string(),expected: json!("readable"),- actual: json!(e.to_string()),+ actual: json!(format!("{e:#}")),});}}@@ -319,7 +319,7 @@ pub async fn verify_strong(person_id,key: "__strong_read_failed".to_string(),expected: json!("readable"),- actual: json!(e.to_string()),+ actual: json!(format!("{e:#}")),});}}diff --git a/rust/personhog-test-harness/src/scenarios/gate.rs b/rust/personhog-test-harness/src/scenarios/gate.rsindex 982212d57c92..7a5a63f1abe3 100644--- a/rust/personhog-test-harness/src/scenarios/gate.rs+++ b/rust/personhog-test-harness/src/scenarios/gate.rs@@ -8,6 +8,7 @@ use std::time::{Duration, Instant};use anyhow::{bail, Context, Result};use sqlx::postgres::PgPool;+use personhog_coordination::types::MigrationPhase;use personhog_proto::personhog::identity::v1::GetOrCreatePersonEntry;use crate::cli::{GateArgs, DEFAULT_KEYS_PER_PERSON};@@ -18,7 +19,7 @@ use crate::report::{print_report, ConsistencyViolation};use crate::scenarios::merge::{self, MergeLane, WideRole};use crate::scenarios::{blast, consistency};use crate::seed;-use crate::stack::{PublishConfig, Stack, StackConfig};+use crate::stack::{MigrationChaos, PublishConfig, Stack, StackConfig, StallAt};use crate::state::PersonState;use crate::stats::StatsCollector;use crate::verify::verify_postgres;@@ -44,6 +45,8 @@ enum ChaosEvent {ReleaseFences,PublisherCrash,IdentityCrash,+ Migrate,+ MigrateAgain,}impl fmt::Display for ChaosEvent {@@ -68,6 +71,8 @@ impl fmt::Display for ChaosEvent {ChaosEvent::RouterShutdown => write!(f, "coordinator router graceful shutdown"),ChaosEvent::PublisherCrash => write!(f, "publisher crash-restart"),ChaosEvent::IdentityCrash => write!(f, "identity crash-restart"),+ ChaosEvent::Migrate => write!(f, "changelog migration"),+ ChaosEvent::MigrateAgain => write!(f, "second changelog migration"),}}}@@ -113,6 +118,12 @@ fn chaos_timeline(args: &GateArgs) -> Vec<(Duration, ChaosEvent)> {if let Some(after) = args.router_shutdown_after {events.push((after, ChaosEvent::RouterShutdown));}+ if let Some(after) = args.migrate_after {+ events.push((after, ChaosEvent::Migrate));+ }+ if let Some(after) = args.migrate_again_after {+ events.push((after, ChaosEvent::MigrateAgain));+ }if let Some(after) = args.fence_after {events.push((after, ChaosEvent::FencePersons));events.push((@@ -218,6 +229,29 @@ pub async fn run(args: GateArgs) -> Result<()> {if args.fence_after.is_some() && args.fence_count == 0 {bail!("--fence-count must be at least 1 with --fence-after");}+ if args.migrate_after.is_some() && args.migrate_to_partitions.is_none_or(|n| n == 0) {+ bail!("--migrate-after needs --migrate-to-partitions >= 1");+ }+ if args.migrate_again_after.is_some() && args.migrate_again_to_partitions.is_none_or(|n| n == 0)+ {+ bail!("--migrate-again-after needs --migrate-again-to-partitions >= 1");+ }+ if args.migrate_kill_coordinator && args.routers < 3 {+ bail!("--migrate-kill-coordinator requires --routers >= 3");+ }+ if args.migrate_kill_leader && args.migrate_kill_leader_at_sealing {+ bail!("--migrate-kill-leader and --migrate-kill-leader-at-sealing are exclusive");+ }+ let mut router_env = args.router_env.clone();+ if let Some(ms) = args.migrate_slow_coordinator_ms {+ router_env.push(("COORDINATOR_MIGRATION_DEBOUNCE_MS".into(), ms.to_string()));+ }+ if args.migrate_stall_leader {+ router_env.push((+ "COORDINATOR_MIGRATION_PRE_FLIP_DEADLINE_SECS".into(),+ "5".into(),+ ));+ }if args.publish && args.external_router_url.is_some() {bail!("--publish requires a spawned stack; it cannot target --external-router-url");}@@ -291,6 +325,7 @@ pub async fn run(args: GateArgs) -> Result<()> {pg_target_table: args.pg_target_table.clone(),cache_memory_capacity: args.cache_capacity,extra_leader_env: args.leader_env.clone(),+ extra_router_env: router_env,recovery_pool_size: args.recovery_pool_size,leader_lease_ttl: args.leader_lease_ttl,spawn_identity: create_via_identity,@@ -703,6 +738,52 @@ async fn run_gate(stack.crash_restart_identity().await?;None}+ ChaosEvent::Migrate | ChaosEvent::MigrateAgain => {+ let chaos = match event {+ ChaosEvent::Migrate => MigrationChaos {+ to_partitions: args+ .migrate_to_partitions+ .expect("validated: --migrate-after needs --migrate-to-partitions"),+ kill_coordinator: args.migrate_kill_coordinator,+ kill_leader_at: if args.migrate_kill_leader {+ Some(MigrationPhase::Resuming)+ } else if args.migrate_kill_leader_at_sealing {+ Some(MigrationPhase::Sealing)+ } else {+ None+ },+ restart_leader: args.migrate_restart_leader,+ stall: if args.migrate_stall_leader {+ Some(StallAt::BeforeStart)+ } else if args.migrate_stall_leader_at_sealing {+ Some(StallAt::Sealing)+ } else {+ None+ },+ pause_writer: args+ .migrate_pause_writer+ .then_some(args.writer_pause_duration),+ },+ // The second migration runs clean; the first carries the chaos.+ _ => MigrationChaos {+ to_partitions: args.migrate_again_to_partitions.expect(+ "validated: --migrate-again-after needs --migrate-again-to-partitions",+ ),+ kill_coordinator: false,+ kill_leader_at: None,+ restart_leader: false,+ stall: None,+ pause_writer: None,+ },+ };+ let took = stack.migrate(chaos).await?;+ println!(+ "Chaos at {:.1}s: migration ended in {:.1}s",+ traffic_start.elapsed().as_secs_f64(),+ took.as_secs_f64()+ );+ None+ }ChaosEvent::FencePersons | ChaosEvent::ReleaseFences => {unreachable!("fence events are dispatched before the stack match")}@@ -731,6 +812,9 @@ async fn run_gate(traffic.await.context("traffic task panicked")??;let mut prober_violations = probers.await.context("prober task panicked")??;+ if let Some(stack) = stack.as_mut() {+ stack.retire_drained().await?;+ }let mut unresolved_merges = Vec::new();if let Some(merges) = merges {let result = merges.await.context("merge task panicked")??;diff --git a/rust/personhog-test-harness/src/scenarios/stack_cmd.rs b/rust/personhog-test-harness/src/scenarios/stack_cmd.rsindex ad3f052ec78d..f69757036994 100644--- a/rust/personhog-test-harness/src/scenarios/stack_cmd.rs+++ b/rust/personhog-test-harness/src/scenarios/stack_cmd.rs@@ -90,6 +90,7 @@ pub async fn run(args: StackArgs) -> Result<()> {pg_target_table: args.pg_target_table.clone(),cache_memory_capacity: 16_777_216,extra_leader_env: Vec::new(),+ extra_router_env: Vec::new(),recovery_pool_size: 16,leader_lease_ttl: 30,spawn_identity: true,diff --git a/rust/personhog-test-harness/src/stack/kafka.rs b/rust/personhog-test-harness/src/stack/kafka.rsindex 580db3b417b4..452a29775659 100644--- a/rust/personhog-test-harness/src/stack/kafka.rs+++ b/rust/personhog-test-harness/src/stack/kafka.rs@@ -11,6 +11,51 @@ use rdkafka::message::Message;use rdkafka::types::RDKafkaErrorCode;use rdkafka::{Offset, TopicPartitionList};+pub fn base_config(kafka_hosts: &str) -> ClientConfig {+ let mut cfg = ClientConfig::new();+ cfg.set("bootstrap.servers", kafka_hosts);+ cfg+}++/// Whether `group` has committed on any partition of `topic`.+pub async fn group_has_committed_offsets(+ kafka_hosts: &str,+ group: &str,+ topic: &str,+) -> Result<bool> {+ let (base, group, topic) = (+ base_config(kafka_hosts),+ group.to_string(),+ topic.to_string(),+ );+ tokio::task::spawn_blocking(move || -> Result<bool> {+ let timeout = Duration::from_secs(5);+ let consumer: BaseConsumer = base+ .clone()+ .set("group.id", &group)+ .set("enable.auto.commit", "false")+ .create()?;+ let metadata = consumer.fetch_metadata(Some(&topic), timeout)?;+ let partitions = metadata+ .topics()+ .iter()+ .find(|t| t.name() == topic)+ .map_or(0, |t| t.partitions().len());+ let mut tpl = TopicPartitionList::new();+ for partition in 0..partitions {+ tpl.add_partition(&topic, partition as i32);+ }+ let committed = consumer.committed_offsets(tpl, timeout)?;+ Ok(committed+ .elements()+ .iter()+ .any(|e| matches!(e.offset(), Offset::Offset(_))))+ })+ .await+ .context("offsets task panicked")?+ .context("fetching committed offsets")+}+fn admin_client(kafka_hosts: &str) -> Result<AdminClient<DefaultClientContext>> {ClientConfig::new().set("bootstrap.servers", kafka_hosts)diff --git a/rust/personhog-test-harness/src/stack/mod.rs b/rust/personhog-test-harness/src/stack/mod.rsindex 531bbfd6ecd1..4c27a0030a86 100644--- a/rust/personhog-test-harness/src/stack/mod.rs+++ b/rust/personhog-test-harness/src/stack/mod.rs@@ -3,9 +3,11 @@ use std::fs;use std::path::PathBuf;use std::time::{Duration, Instant};-use anyhow::{bail, Context, Result};+use anyhow::{bail, ensure, Context, Result};use personhog_coordination::store::PersonhogStore;-use personhog_coordination::types::HandoffPhase;+use personhog_coordination::types::{+ ChangelogTarget, HandoffPhase, MigrationPhase, MigrationRequest,+};mod etcd;pub(crate) mod kafka;@@ -49,6 +51,10 @@ const PUBLISH_TOPIC_PARTITIONS: u32 = 4;const ETCD_PREFIX: &str = "/personhog-test-harness/";const READY_DEADLINE: Duration = Duration::from_secs(90);+const MIGRATION_DEADLINE: Duration = Duration::from_secs(180);+const MIGRATION_POLL: Duration = Duration::from_millis(100);+const RETIREMENT_DEADLINE: Duration = Duration::from_secs(60);+const WRITER_MOVE_DEADLINE: Duration = Duration::from_secs(10);const SHUTDOWN_GRACE: Duration = Duration::from_secs(5);pub struct StackConfig {@@ -71,6 +77,7 @@ pub struct StackConfig {/// Extra environment for spawned leaders, appended after the/// standard set (so it can override any of it).pub extra_leader_env: Vec<(String, String)>,+ pub extra_router_env: Vec<(String, String)>,pub recovery_pool_size: usize,/// etcd lease TTL for leaders, in seconds. Bounds how long a crashed/// (unrevoked) leader stays the registered owner.@@ -133,6 +140,10 @@ pub struct Stack {next_leader_index: u32,store: PersonhogStore,topic: String,+ /// Topics a migration moved off; deleted with the live one at teardown.+ retired_topics: Vec<String>,+ /// Old generations awaiting the writer's drain and the retire op, with the saga's end.+ pending_retire: Vec<(String, Instant)>,pub router_url: String,/// Set when the stack spawned a personhog-identity service: the traffic/// router's URL, which proxies identity RPCs to it.@@ -287,7 +298,7 @@ impl Stack {}let name = format!("harness-router-{i}");let is_traffic_router = config.routers >= 2 && i == config.routers - 1;- let proc = ServiceProcess::spawn(+ let proc = ServiceProcess::spawn_with_extra(&name,&config.bin_dir.join("personhog-router"),&[@@ -316,6 +327,7 @@ impl Stack {(ROUTER_METRICS_BASE_PORT + i as u16).to_string(),),],+ &config.extra_router_env,&log_dir,)?;routers.push((name, proc));@@ -470,6 +482,8 @@ impl Stack {next_leader_index: 0,store,topic,+ retired_topics: Vec::new(),+ pending_retire: Vec::new(),router_url,identity_url,publish_topics,@@ -611,6 +625,57 @@ impl Stack {Ok(pod_name)}+ /// SIGSTOP the busiest leader with its lease intact: registered and+ /// owning, but unable to drain, ack or serve until `resume_stalled`.+ pub async fn stall_leader(&mut self) -> Result<String> {+ let victim = self.busiest_leader().await?;+ self.stall_named(&victim)+ }++ fn stall_named(&mut self, victim: &str) -> Result<String> {+ let (pod_name, proc) = self.remove_leader(victim)?;+ proc.sigstop();+ tracing::info!(pod = %pod_name, "SIGSTOPped leader (stalled, lease kept)");+ self.paused.push((pod_name.clone(), proc));+ Ok(pod_name)+ }++ /// SIGKILL the stalled leader and revoke its lease.+ async fn kill_stalled(&mut self) -> Result<String> {+ let (pod_name, mut proc) = self.paused.pop().context("no stalled leader to kill")?;+ proc.kill_now().await;+ etcd::revoke_pod_lease(&self.store, &pod_name).await?;+ tracing::info!(pod = %pod_name, "killed stalled leader");+ self.retired.push(proc);+ Ok(pod_name)+ }++ /// Whether every partition the saga holds on `pod` carries its drained ack.+ async fn holder_drained(&self, pod: &str) -> Result<bool> {+ for held in self.store.list_handoffs().await? {+ if held.held_by.is_none() || held.new_owner != pod {+ continue;+ }+ let acks = self.store.list_drained_acks(held.partition).await?;+ if !acks+ .iter()+ .any(|a| a.pod_name == pod && a.handoff_id == held.handoff_id)+ {+ return Ok(false);+ }+ }+ Ok(true)+ }++ /// SIGCONT the stalled leader; it resumes exactly where it was.+ pub fn resume_stalled(&mut self) -> Result<String> {+ let (pod_name, proc) = self.paused.pop().context("no stalled leader to resume")?;+ proc.sigcont();+ tracing::info!(pod = %pod_name, "SIGCONTed stalled leader");+ self.leaders.push((pod_name.clone(), proc));+ Ok(pod_name)+ }+/// SIGCONT the paused zombie. It wakes believing it still owns its/// partitions; the contract is that it detects the revoked lease,/// self-fences locally, and rejoins with a fresh session — at which@@ -996,7 +1061,8 @@ impl Stack {// Every topic is attempted even when one delete fails, so a failed// teardown leaves at most that one topic behind.- let mut topics = vec![self.topic.as_str()];+ let mut topics: Vec<&str> = self.retired_topics.iter().map(String::as_str).collect();+ topics.push(self.topic.as_str());if let Some(publish) = &self.publish_topics {topics.push(publish.distinct_ids.as_str());topics.push(publish.merge_events.as_str());@@ -1012,6 +1078,335 @@ impl Stack {None => Ok(()),}}++ /// Run a changelog migration to its end, with chaos keyed to its phases.+ /// Retirement of the old generation is deferred to `retire_drained`.+ pub async fn migrate(&mut self, chaos: MigrationChaos) -> Result<Duration> {+ // Admission refuses a migration while an earlier generation still drains.+ if !self.pending_retire.is_empty() {+ self.retire_drained().await?;+ }+ let from = self+ .store+ .get_changelog_config()+ .await?+ .context("config/changelog must be seeded")?;+ let generation = self.retired_topics.len() + 2;+ let to_topic = format!("{}_g{generation}", self.topic);+ kafka::create_topic(&self.config.kafka_hosts, &to_topic, chaos.to_partitions).await?;+ let migration = MigrationRequest {+ id: format!("harness-g{generation}"),+ from: from.target(),+ to: ChangelogTarget {+ topic: to_topic.clone(),+ partitions: chaos.to_partitions,+ },+ requested_at_ms: now_millis(),+ seen_at_ms: None,+ waiting: Vec::new(),+ };+ // A leader stalled before the request keeps its lease, so the fleet still+ // reads as quiet; the saga starts, holds its partitions and never drains them.+ let mut stalled: Option<String> = None;+ if chaos.stall == Some(StallAt::BeforeStart) {+ let name = self.stall_leader().await?;+ println!("Migration chaos: stalled leader {name} before the request");+ stalled = Some(name);+ }+ ensure!(+ self.store.create_migration_request(&migration).await?,+ "a migration is already requested or in flight"+ );+ // The coordinator starts it once the fleet is quiet.+ let requested = Instant::now();+ // The request is read before the saga: the start creates one and+ // deletes the other in a single transaction, so this order never sees neither.+ loop {+ let waiting = self+ .store+ .get_migration_request_with_mod_revision()+ .await?+ .is_some();+ if self.store.get_migration().await?.is_some() {+ break;+ }+ ensure!(+ waiting,+ "the request was rejected before starting: {:?}",+ self.store+ .list_migration_history()+ .await?+ .into_iter()+ .find(|m| m.id == migration.id)+ );+ ensure!(+ requested.elapsed() < MIGRATION_DEADLINE,+ "the request did not start within {MIGRATION_DEADLINE:?}"+ );+ tokio::time::sleep(MIGRATION_POLL).await;+ }+ if chaos.pause_writer.is_some() {+ self.pause_writer()?;+ }+ tracing::info!(+ from = %from.topic,+ to = %to_topic,+ partitions = chaos.to_partitions,+ waited_ms = requested.elapsed().as_millis() as u64,+ "changelog migration started"+ );++ let start = Instant::now();+ let mut last_phase = None;+ let mut writer_paused = chaos.pause_writer;+ let mut coordinator_killed = !chaos.kill_coordinator;+ let mut leader_killed = chaos.kill_leader_at.is_none();+ let mut leader_restarted = !chaos.restart_leader;+ let mut sealed_while_stalled = chaos.stall != Some(StallAt::Sealing);+ // A leader stopped at Sealing may already have sealed, which tests nothing, so the+ // victim is stopped once it has drained and must hold no seal ack when Sealing begins.+ let seal_victim_wanted = chaos.kill_leader_at == Some(MigrationPhase::Sealing)+ || chaos.stall == Some(StallAt::Sealing);+ let mut seal_victim: Option<String> = None;+ let mut seal_victim_checked = false;+ loop {+ if let Some(pause) = writer_paused {+ if start.elapsed() >= pause {+ self.resume_writer()?;+ writer_paused = None;+ }+ }+ let Some(current) = self.store.get_migration().await? else {+ break;+ };+ if last_phase != Some(current.phase) {+ println!(+ "Migration {:.1}s: {:?} {}",+ start.elapsed().as_secs_f64(),+ current.phase,+ current.blocking.join("; ")+ );+ last_phase = Some(current.phase);+ }+ let holding = matches!(+ current.phase,+ MigrationPhase::Freezing | MigrationPhase::Draining+ );+ if seal_victim_wanted+ && seal_victim.is_none()+ && current.phase == MigrationPhase::Draining+ {+ let name = self.busiest_leader().await?;+ if self.holder_drained(&name).await? {+ self.stall_named(&name)?;+ println!("Migration chaos: stalled leader {name} once drained");+ seal_victim = Some(name);+ }+ }+ if let Some(name) = seal_victim.clone().filter(|_| !seal_victim_checked) {+ if current.phase == MigrationPhase::Sealing {+ let sealed = self+ .store+ .list_seal_acks()+ .await?+ .into_iter()+ .any(|a| a.migration_id == current.id && a.pod_name == name);+ ensure!(!sealed, "leader {name} sealed before it stalled");+ seal_victim_checked = true;+ if chaos.kill_leader_at == Some(MigrationPhase::Sealing) {+ self.kill_stalled().await?;+ println!("Migration chaos: killed leader {name} at Sealing");+ leader_killed = true;+ } else {+ stalled = Some(name);+ }+ }+ }+ if chaos.stall == Some(StallAt::Sealing) {+ // Past Sealing, the seal grace let its peers seal for it; resume it so Resuming can restart it.+ if stalled.is_some()+ && matches!(+ current.phase,+ MigrationPhase::Resuming | MigrationPhase::Complete+ )+ {+ let name = stalled.take().expect("checked above");+ self.resume_stalled()?;+ println!(+ "Migration chaos: the saga flipped while {name} was stalled; resumed it"+ );+ sealed_while_stalled = true;+ }+ }+ if !leader_restarted && holding {+ let name = self.restart_leader().await?;+ println!("Migration chaos: crash-restarted leader {name}");+ leader_restarted = true;+ }+ if !coordinator_killed && current.phase == MigrationPhase::Sealing {+ let name = self.kill_coordinator_router(true).await?;+ println!("Migration chaos: killed coordinator router {name}");+ coordinator_killed = true;+ }+ if !leader_killed+ && chaos.kill_leader_at != Some(MigrationPhase::Sealing)+ && Some(current.phase) == chaos.kill_leader_at+ {+ let name = self.kill_leader(true).await?;+ println!("Migration chaos: killed leader {name}");+ leader_killed = true;+ }+ if start.elapsed() > MIGRATION_DEADLINE {+ bail!(+ "migration did not finish within {MIGRATION_DEADLINE:?}: {current:?}; {}",+ self.coordination_report().await+ );+ }+ tokio::time::sleep(MIGRATION_POLL).await;+ }+ let saga = start.elapsed();+ // A chaos step the harness never got to apply would let the gate pass vacuously.+ ensure!(+ coordinator_killed && leader_killed && leader_restarted && sealed_while_stalled,+ "a requested chaos step never ran (coordinator killed {coordinator_killed}, \+ leader killed {leader_killed}, leader restarted {leader_restarted}, \+ sealed while stalled {sealed_while_stalled}); slow the coordinator so the phase is observable"+ );+ if let Some(name) = stalled {+ self.resume_stalled()?;+ println!("Migration chaos: resumed stalled leader {name}");+ }+ // The saga never waits for the writer, so the lag it injects outlives the saga.+ if let Some(pause) = writer_paused {+ tokio::time::sleep(pause.saturating_sub(start.elapsed())).await;+ self.resume_writer()?;+ println!(+ "Migration chaos: resumed the writer {:.1}s after the saga started",+ start.elapsed().as_secs_f64()+ );+ }+ let archived = self+ .store+ .list_migration_history()+ .await?+ .into_iter()+ .find(|m| m.id == migration.id)+ .context("the migration record vanished without an archive")?;+ let expected = if chaos.stall == Some(StallAt::BeforeStart) {+ MigrationPhase::Cancelled+ } else {+ MigrationPhase::Complete+ };+ ensure!(+ archived.phase == expected,+ "migration ended in {:?}, expected {expected:?}: {}",+ archived.phase,+ archived.blocking.join("; ")+ );+ if expected == MigrationPhase::Cancelled {+ kafka::delete_topic(&self.config.kafka_hosts, &to_topic).await?;+ return Ok(saga);+ }+ self.config.partitions = chaos.to_partitions;+ let old_topic = std::mem::replace(&mut self.topic, to_topic);+ self.retired_topics.push(old_topic.clone());+ self.pending_retire.push((old_topic, Instant::now()));+ Ok(saga)+ }++ /// Wait for the writer to drain every old generation on its own, then retire each one.+ pub async fn retire_drained(&mut self) -> Result<()> {+ let pending = std::mem::take(&mut self.pending_retire);+ for (old_topic, saga_end) in pending {+ let drained_at = Instant::now();+ loop {+ let base = kafka::base_config(&self.config.kafka_hosts);+ let topic = old_topic.clone();+ let drained = tokio::task::spawn_blocking(move || {+ common_kafka::probe::group_drained(+ &base,+ WRITER_CONSUMER_GROUP,+ &topic,+ Duration::from_secs(5),+ )+ })+ .await??;+ if drained {+ break;+ }+ ensure!(+ drained_at.elapsed() < RETIREMENT_DEADLINE,+ "the writer did not drain {old_topic} within {RETIREMENT_DEADLINE:?}"+ );+ tokio::time::sleep(MIGRATION_POLL).await;+ }+ println!(+ "Migration: {old_topic} drained by the writer; checked {:.1}s after the saga, waited {:.1}s",+ saga_end.elapsed().as_secs_f64(),+ drained_at.elapsed().as_secs_f64()+ );+ ensure!(+ self.store.retire_changelog_topic(&old_topic).await?,+ "{old_topic} was not listed as draining"+ );+ }+ // The writer re-checks the drain every 2s and flushes every second.+ let moved_at = Instant::now();+ loop {+ if kafka::group_has_committed_offsets(+ &self.config.kafka_hosts,+ WRITER_CONSUMER_GROUP,+ &self.topic,+ )+ .await?+ {+ break;+ }+ ensure!(+ moved_at.elapsed() < WRITER_MOVE_DEADLINE,+ "the writer did not commit on {} within {WRITER_MOVE_DEADLINE:?}",+ self.topic+ );+ tokio::time::sleep(MIGRATION_POLL).await;+ }+ println!(+ "Migration: the writer committed on {} {:.1}s after the last drain",+ self.topic,+ moved_at.elapsed().as_secs_f64()+ );+ Ok(())+ }+}++/// When migration chaos stalls a leader.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum StallAt {+ /// Before the request: its partitions are held and never drain, so the+ /// pre-flip deadline cancels the saga.+ BeforeStart,+ /// Once drained, before Sealing: its peers seal for it after the seal grace, and the saga completes.+ Sealing,+}++pub struct MigrationChaos {+ pub to_partitions: u32,+ pub kill_coordinator: bool,+ /// SIGKILL the busiest leader, lease revoked, when the saga reaches this phase; for Sealing it+ /// is stalled once drained first.+ pub kill_leader_at: Option<MigrationPhase>,+ /// Crash-restart the busiest leader under its own name while the saga holds.+ pub restart_leader: bool,+ /// SIGSTOP the busiest leader with its lease kept.+ pub stall: Option<StallAt>,+ /// Pause the writer for this long as the migration starts.+ pub pause_writer: Option<Duration>,+}++fn now_millis() -> i64 {+ std::time::SystemTime::now()+ .duration_since(std::time::UNIX_EPOCH)+ .expect("clock before the epoch")+ .as_millis() as i64}async fn wait_tcp(addr: &str, deadline: Duration) -> Result<()> {