Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(personhog): model-check and gate changelog migrations (#119813)

Zach Brown3e0edadd0af1
diff --git a/.github/workflows/ci-rust.yml b/.github/workflows/ci-rust.yml
index 74efea4f5893..eeba6430a09b 100644
--- a/.github/workflows/ci-rust.yml
+++ b/.github/workflows/ci-rust.yml
@@ -846,6 +846,28 @@ jobs:
--routers 3 --leaders 3 --partitions 8 --persons 100 --concurrency 10 \
--duration 18s --router-kill-after 4s --router-kill-fast false --shutdown-after 8s
+ # A changelog migration grows 4 to 8 partitions while the
+ # writer is paused, so a restarted partition must warm the old
+ # topic's tail itself. Write-once persons are what expose a
+ # warm that skips that tail: with a small person set every key
+ # is rewritten after the flip and the loss is masked.
+ - name: Run the e2e gate growing the changelog with the writer paused
+ run: |
+ ./target/debug/personhog-test-harness gate \
+ --leaders 2 --partitions 4 --persons 5000 --concurrency 10 --duration 30s \
+ --migrate-after 8s --migrate-to-partitions 8 \
+ --migrate-pause-writer --writer-pause-duration 8s
+
+ # A shrink dissolves the partitions past the new count, and a
+ # holder killed at Sealing leaves its partitions for its peers to
+ # seal. The slow coordinator keeps Sealing long enough to hit.
+ - name: Run the e2e gate shrinking the changelog with a leader killed at Sealing
+ run: |
+ ./target/debug/personhog-test-harness gate \
+ --leaders 3 --partitions 8 --persons 100 --concurrency 10 --duration 30s \
+ --migrate-after 8s --migrate-to-partitions 4 \
+ --migrate-kill-leader-at-sealing --migrate-slow-coordinator-ms 500
+
- name: Dump service logs on failure
if: failure()
run: |
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index 505d632aa9fb..27bb72c4fd20 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -9130,6 +9130,7 @@ dependencies = [
"axum 0.7.9",
"chrono",
"clap",
+ "common-kafka",
"common-metrics",
"futures",
"hdrhistogram",
diff --git a/rust/personhog-stateright/README.md b/rust/personhog-stateright/README.md
index 9104c4068684..f65e07140ca0 100644
--- a/rust/personhog-stateright/README.md
+++ b/rust/personhog-stateright/README.md
@@ -28,22 +28,24 @@ web explorer.
Configurations are deliberately small — state spaces grow
combinatorially, and protocol bugs are structural, showing up at
minimum viable scale or not at all.
-The suite explores ~28.5M states across 28 runs.
-Its long pole is the two-partition epoch-fenced double zombie, which is about half the wall clock on its own:
+As of 2026-10-16 the suite explores ~19M states across 43 runs.
+Its long pole is the two-partition epoch-fenced double zombie, about a quarter of the wall clock on its own:
| Scenario | Unique states | Wall time |
|---|---|---|
-| `epoch_fenced_two_partitions_double_zombie_is_safe` | 13.1M | 19.1s |
-| `rollout_quota_mode_is_safe_and_live` | 3.5M | 5.1s |
-| `cancellation_with_live_owner_reaffirms_and_resumes` | 3.2M | 4.2s |
-| `chunked_plan_application_is_safe_and_live` | 2.1M | 2.7s |
-| `probe_chunked_pending_units_reach_both_fates_safely` | 2.1M | 3.2s |
-| *everything else (23 runs)* | 4.4M | 6s |
-
-Roughly: a second partition costs ~20x, a second failure in the budget ~10x, a third pod ~2x.
+| `epoch_fenced_two_partitions_double_zombie_is_safe` | 4.9M | 8.7s |
+| `cancellation_with_live_owner_reaffirms_and_resumes` | 1.9M | 3.1s |
+| `chunked_plan_application_is_safe_and_live` | 1.4M | 2.2s |
+| `probe_chunked_pending_units_reach_both_fates_safely` | 1.4M | 2.8s |
+| `rollout_quota_mode_is_safe_and_live` | 1.2M | 2.0s |
+| `a_two_to_four_grow_is_safe_and_live` | 1.1M | 2.8s |
+| `a_chunked_plan_across_a_migration_is_safe_and_live` | 0.9M | 2.4s |
+| *everything else (36 runs)* | 6.1M | 11s |
+
+Roughly: a second partition costs ~20x, a second failure in the budget ~10x, a third pod ~2x. The migration scenarios run at one or two partitions with keyed traffic and cost 20k to 1.1M states each; the 2→4 grow is the largest.
Times are release mode, one scenario at a time on 12 cores — a CI runner with 4 slower cores is several times that, so treat them as ratios rather than absolutes.
-The test suite (twenty-odd scenarios; the two-partition double zombie dominates the runtime) runs the full
+The test suite (41 scenarios; the two-partition double zombie dominates the runtime) runs the full
verdict matrix — the 1-partition scenarios, the 2-partition cases, the
3-pod rejoin, and the reachability probes. Two partitions matter for the coordinator's
cross-partition scheduling (rebalancing defers while any handoff is in
@@ -106,6 +108,14 @@ checker verifies, automatically:
| `protocol::freeze_quorum_met` / `drain_satisfied` / `warm_satisfied` | the phase-advancement rules (identity quorum, id-correlated acks, vacuous drain) — `Action::AdvancePhase` calls them |
| `protocol::plan_partial_rebalance` | the rebalance decision (placement via `StickyBalancedStrategy` + the move/fresh diff, old_owner from the current assignment, in-flight partitions pinned) — `Action::Rebalance` calls it, as does the coordinator |
| `types::HandoffPhase` | used directly as the model's phase enum, so adding a phase breaks the model's exhaustive matches at compile time |
+| `protocol::migration::plan_freeze` | which held self-handoffs the freeze creates, and what cancels it (a stray ordinary handoff, a holder or owner that is gone) — `Action::MigrateDrive` at Freezing calls it |
+| `protocol::migration::fleet_quiet` | when a requested saga may start: no handoff record, and every old partition assigned to a registered, Ready pod — `Action::MigrateStart` is enabled through it, as the coordinator's start step is |
+| `protocol::migration::plan_resume` | what Resuming does with the holds: dissolve the shrunk ones, abandon a dead holder's, restart the rest over the new changelog, and place the partitions a grow created — `Action::MigrateDrive` at Resuming calls it |
+| `protocol::migration::migration_unsealed` / `migration_unsettled` | the Sealing and Resuming gates (every old partition sealed; no held handoff left) — `Action::MigrateDrive` calls them |
+| `protocol::migration::seal_targets` | which old partitions a pod seals: every one assigned to it, every one owned by no registered peer, and once the seal is overdue every one no ack reports sealed — `Action::SealTick` calls it |
+| `protocol::migration::planner_may_place` | when the ordinary planner may place during a saga — `Action::Rebalance` and `RebalanceChunked` are enabled through it, as the coordinator's pass is |
+| `changelog::warm_sources` | the partitions a warm replays: the congruent old partitions (mod gcd, key-filtered) then the live one — a pod's warm effect reads it |
+| `types::MigrationPhase::may_transition_to` | the saga's legal phase moves, so a new phase or a new edge fails the model's matches at compile time |
What remains model-side is the *environment and effect application* —
what warming does to a cache, what leases and zombies mean, how stashes
@@ -121,6 +131,13 @@ queue — mapped to named production behavior for review:
| `Action::CrashRestartWithinTtl` | Process death + same-name restart before lease expiry: registration and assignments survive, memory wiped |
| `Action::LeaseExpire` / `SelfFence` | Lease loss with the bounded zombie window before the keepalive self-fences (fix 1); same pair for routers, where lease loss also drops them from the freeze quorum |
| `Changelog.epoch_holder` | Kafka transactional-producer fencing: warming = `init_transactions`, which takes the fence from whoever held it; a produce from a fenced-out producer is rejected before any client ack |
+| `Action::MigrateRequest` / `MigrateStart` / `MigrateRequestExpires` | the control plane's request, the coordinator's start once the fleet is quiet, and a request rejected or expired before it starts |
+| `Action::MigrateDrive` / `MigrateCancel` | the coordinator's migration saga: one guarded phase step per drive (freeze, drain, seal, flip, resume, complete); cancel from any pre-Seal phase, standing in for both the operator and the pre-flip deadline |
+| `Action::SealTick(pod)` | the leader's seal duty: a registered pod seals its `seal_targets` on the old generation, no capture needed, plus every unsealed partition once the grace has lapsed |
+| `Action::SealGraceElapses` / `BreakSeal(pod)` | the 2s Sealing grace lapsing, and a registered pod whose seal never succeeds (a hung Kafka client with a live lease); a new incarnation heals it |
+| `Action::AdoptConfig(router)` | the router's config watch: adopt the flipped config. Parked requests stay where they are, and the new count only decides which drain takes them; a write for a partition born in the flip parks until its handoff drains |
+| `Action::ClientWrite(router, key)` / `ClientStrongRead(router, key)` | keyed traffic: the router hashes the key under its own adopted modulus, so a stale router routes to the old partition until it adopts |
+| `Changelog` keyed by `(epoch, partition)` | one changelog per generation; the old generation stays readable for the warm's tail replay and must take no produce after the flip |
Full elimination of the second table would mean deterministic
simulation — a trait seam over `PersonhogStore` with an in-memory
@@ -135,7 +152,12 @@ implementation so the model executes `converge` and
| `no_split_write_acceptance` (safety) | No two pods are simultaneously capable of accepting writes for one partition **and** each reachable via a live, non-stashing router — the real split-brain condition |
| `drained_ack_is_final` (safety) | A pod that wrote a DrainedAck for the current handoff attempt cannot accept another write in that incarnation |
| `some_handoff_completes`, `some_write_accepted` (sanity) | The interesting states are actually reachable |
-| `converges_to_stable` (liveness) | Every full run ends quiescent: no handoffs, every partition served warm and unfenced by its sticky target, all live routers agreeing, no stashed traffic |
+| `no_overtaken_request` (safety) | A router never delivers a request while an earlier request for the same key is still parked |
+| `no_mixed_modulus_acceptance` (safety) | Every acked write lands in the live generation's changelog, never a retired one |
+| `holds_stop_at_draining` (safety) | The ordinary handoff loop never moves a migration hold past Draining |
+| `old_generation_quiet_after_flip` (safety) | Once flipped, no pod can produce to the old generation |
+| `migration_completes`, `migration_cancels` (sanity) | Both saga outcomes are reachable in a migration config |
+| `converges_to_stable` (liveness) | Every full run ends quiescent: no handoffs, every partition served warm and unfenced by its sticky target, all live routers agreeing, no stashed traffic, and no saga in flight or left to request |
## Results
@@ -147,6 +169,16 @@ implementation so the model executes `converge` and
| Single zombie pod | **holds** | **holds** | holds |
| Double zombie (router + pod) | holds | holds | holds |
| Strong reads + one failure | holds | holds | holds |
+| Changelog migration: grow 1→2, shrink 2→1, grow 2→4 | holds | holds | holds |
+| Migration with a stale router (adopts late) | holds | holds | holds |
+| Migration with the holder crash-restarted mid-hold | holds | holds | holds |
+| Migration, seal targets by residency (rejected rule) | holds | holds | wedges (liveness) |
+| Migration, live-only warm (rejected rule) | holds | holds | **violated** |
+| Migration, a registered holder that cannot seal, with the grace | holds | holds | holds |
+| Migration, a registered holder that cannot seal, no grace (rejected rule) | holds | holds | wedges (liveness) |
+| Migration racing a chunked bootstrap plan, with the plan basis | holds | holds | holds |
+| Migration racing a chunked bootstrap plan, no plan basis (rejected rule) | **violated** (old generation written after the flip) | holds | holds |
+| Migration started with a handoff in flight (rejected rule) | **violated** | **violated** | **violated** |
Two results worth calling out:
@@ -196,6 +228,8 @@ target rather than having it forced — the one axis a 2-pod world
under-exercises; the per-partition safety relations themselves are
two-party (old owner ↔ new owner, zombie ↔ successor).
+**The migration saga is checked against the same production decision functions.** The model drives `plan_freeze`, `migration_unsealed`, `migration_unsettled`, `seal_targets` and `warm_sources` through a grow, a shrink and a 2→4 grow, with keyed traffic, a stale router, a crash-restarted holder, and the deadline cancel. Its first run on the branch found a real hole: the freeze held the partitions in the assignment table, but an ordinary handoff for a fresh placement (a partition with no assignment yet) kept running, warmed, and accepted a write the flip then stranded. The saga now starts only from a fleet with no handoff in flight (`MigrateStart` requires it), and `quiet_start: false` keeps the rejected start, which loses acked writes. Two rejected rules stay in the model as variants, each pinned by the counterexample that rejected it: sealing by residency (`SealRule::Residency`) wedges a holder that crash-restarted before sealing, because it no longer holds the data and nobody else is assigned, so the saga never leaves Sealing; a live-only warm (`WarmReplay::LiveOnly`) loses the old generation's tail, because a key's last write before the flip sits in an old partition the new one never replays. A third, `seal_grace: false`, pins why the seal grace exists: one registered holder whose seal never succeeds holds Sealing forever, since peers leave a registered owner's partitions alone and nothing cancels past Draining.
+
## Usage
```sh
@@ -241,6 +275,8 @@ owner to still claim its partitions, so an owner that refuses its own
reads while looking alive to the coordinator fails liveness rather than
passing quietly.
+Migration coverage: one migration per run with at most one draining generation, which admission also enforces, so the retire step and a second migration are not modeled; the key hash is `key % partitions`, which keeps the congruence structure of the real hash without its constant; the seal is instant (production claims and drops every lane's transactional id); the cancel is unbudgeted, standing in for the pre-flip deadline as well as the operator; the writer is not modeled, since its drain only gates retirement. Coordinator passes are atomic steps in the model, so a stalled pass that wakes after a successor acted cannot occur here; the rule that every plan requires the record it was decided on unchanged is pinned by the coordination store tests instead.
+
Known remaining abstractions: warming is instant and atomic (production
streams from Kafka with retries — availability, not safety); stash
deadlines/bounds are not modeled (availability policies); the
diff --git a/rust/personhog-stateright/src/main.rs b/rust/personhog-stateright/src/main.rs
index 859084d8d190..c4ec93a2dc71 100644
--- a/rust/personhog-stateright/src/main.rs
+++ b/rust/personhog-stateright/src/main.rs
@@ -11,7 +11,7 @@
use clap::{Parser, ValueEnum};
-use personhog_stateright::model::{ClaimDetection, HandoffModel, WarmOrder};
+use personhog_stateright::model::{ClaimDetection, HandoffModel, SealRule, WarmOrder, WarmReplay};
use stateright::Model;
#[derive(Parser)]
@@ -58,6 +58,17 @@ fn main() {
cancels: 0,
chunked_plans: false,
probes: false,
+ migrations: 0,
+ migrate_to: 1,
+ keys: 1,
+ seal_rule: SealRule::Assignment,
+ warm_replay: WarmReplay::Sources,
+ seal_breaks: 0,
+ seal_grace: true,
+ plan_basis_guard: true,
+ quiet_start: true,
+ freeze_checks_strays: true,
+ park_behind_key: true,
};
println!("exploring {:?} at http://localhost:3000 …", args.scenario);
model.checker().serve("localhost:3000");
diff --git a/rust/personhog-stateright/src/model.rs b/rust/personhog-stateright/src/model.rs
index 21a8b4714ba2..57ff4d3699a2 100644
--- a/rust/personhog-stateright/src/model.rs
+++ b/rust/personhog-stateright/src/model.rs
@@ -5,24 +5,36 @@
//! reviewable line by line. The checker exhaustively interleaves every
//! action from every reachable state and verifies the properties at each.
-use std::collections::{BTreeMap, BTreeSet, HashMap};
+mod migration;
+use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
+
+use personhog_coordination::changelog::warm_sources;
use personhog_coordination::pod::{desired_state, DesiredState};
+use personhog_coordination::protocol::migration::{
+ fleet_quiet, migration_unsealed, migration_unsettled, plan_freeze, plan_resume,
+ planner_may_place, seal_targets,
+};
use personhog_coordination::protocol::{
drain_satisfied, freeze_quorum_met, plan_partial_rebalance, warm_satisfied, PlannedHandoff,
};
use personhog_coordination::strategy::{AssignmentStrategy, Member, StickyBalancedStrategy};
use personhog_coordination::types::{
- AssignmentStatus, HandoffState, PartitionAssignment, PodDrainedAck, PodStatus, PodWarmedAck,
- RegisteredPod, RegisteredRouter, RouterFreezeAck,
+ AssignmentStatus, ChangelogConfig, Generation as ProductionGeneration, HandoffState,
+ MigrationPhase, PartitionAssignment, PodDrainedAck, PodStatus, PodWarmedAck, RegisteredPod,
+ RegisteredRouter, RouterFreezeAck, SealAck,
};
use stateright::{Model, Property};
use crate::types::{
- Action, Changelog, Handoff, HandoffId, Partition, PendingUnit, PendingWarm, Phase, Pod, PodId,
- Router, RouterId, StashedRequest, SystemState, WarmState,
+ Action, Changelog, Config, Epoch, Generation, Handoff, HandoffId, Key, Migration, Partition,
+ PendingUnit, PendingWarm, Phase, Pod, PodId, Router, RouterId, StashedRequest, SystemState,
+ WarmState,
};
+/// The one migration id the model ever runs.
+const MIGRATION_ID: &str = "m";
+
/// Deterministic names bridging the model's compact u8 ids to the
/// string-keyed production types.
fn pod_name(x: PodId) -> String {
@@ -34,6 +46,20 @@ fn pod_id(name: &str) -> PodId {
fn router_name(r: RouterId) -> String {
format!("r{r}")
}
+fn generation_topic(epoch: Epoch) -> String {
+ format!("g{epoch}")
+}
+fn topic_epoch(topic: &str) -> Epoch {
+ topic
+ .trim_start_matches('g')
+ .parse()
+ .expect("generation topic")
+}
+
+/// The identity hash: a key's home is its remainder.
+fn home(key: Key, partitions: u8) -> Partition {
+ key % partitions
+}
/// Materialize the production `HandoffState` view of a model handoff, so
/// shared production functions can be called on checker state.
@@ -55,11 +81,97 @@ fn production_handoff(p: Partition, h: &Handoff) -> HandoffState {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
- held_by: None,
- config_epoch: 0,
+ held_by: h.held.then(|| MIGRATION_ID.to_string()),
+ config_epoch: h.config_epoch as u64,
}
}
+fn production_config(config: &Config) -> ChangelogConfig {
+ ChangelogConfig {
+ topic: generation_topic(config.live.epoch),
+ partitions: config.live.partitions as u32,
+ epoch: config.live.epoch as u64,
+ writer_group: String::new(),
+ draining: config
+ .draining
+ .map(|g| ProductionGeneration {
+ topic: generation_topic(g.epoch),
+ partitions: g.partitions as u32,
+ epoch: g.epoch as u64,
+ })
+ .into_iter()
+ .collect(),
+ }
+}
+
+fn production_assignments(state: &SystemState) -> Vec<PartitionAssignment> {
+ state
+ .assignments
+ .iter()
+ .map(|(p, owner)| PartitionAssignment {
+ partition: *p as u32,
+ owner: pod_name(*owner),
+ advertise_address: None,
+ status: AssignmentStatus::Active,
+ })
+ .collect()
+}
+
+fn production_registered(state: &SystemState) -> Vec<RegisteredPod> {
+ state
+ .pods
+ .iter()
+ .filter(|(_, pod)| pod.registered)
+ .map(|(x, _)| RegisteredPod {
+ pod_name: pod_name(*x),
+ generation: String::new(),
+ status: PodStatus::Ready,
+ registered_at: 0,
+ last_heartbeat: 0,
+ controller: None,
+ advertise_address: None,
+ })
+ .collect()
+}
+
+fn production_seal_acks(state: &SystemState) -> Vec<SealAck> {
+ state
+ .seal_acks
+ .iter()
+ .map(|(x, sealed)| SealAck {
+ migration_id: MIGRATION_ID.to_string(),
+ pod_name: pod_name(*x),
+ sealed: sealed.iter().map(|p| *p as u32).collect(),
+ })
+ .collect()
+}
+
+/// The modulus a captured epoch hashes with; `None` once that generation is gone.
+fn modulus_of(config: &Config, epoch: Epoch) -> Option<u8> {
+ if config.live.epoch == epoch {
+ Some(config.live.partitions)
+ } else {
+ config
+ .draining
+ .filter(|g| g.epoch == epoch)
+ .map(|g| g.partitions)
+ }
+}
+
+/// Which old partitions a pod seals; `Residency` is the rejected rule, kept as the record of why.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum SealRule {
+ Assignment,
+ Residency,
+}
+
+/// Where a restart's warm reads from; `LiveOnly` is the rejected rule.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum WarmReplay {
+ Sources,
+ LiveOnly,
+}
+
/// The membership a model handoff was created with, in the form the
/// production quorum predicates take. The model always captures a
/// snapshot; the production `None` fallback is a serialization concern
@@ -147,6 +259,30 @@ pub struct HandoffModel {
/// than assume, which configurations actually reach them. Off in the
/// verdict tests: an unreached probe would fail `assert_properties`.
pub probes: bool,
+ /// Migrations the control plane may start (at most one draining generation).
+ pub migrations: u8,
+ /// Partition count a migration moves to.
+ pub migrate_to: u8,
+ /// Distinct keys the workload writes and reads, at least one per starting
+ /// partition so every partition takes writes.
+ pub keys: u8,
+ pub seal_rule: SealRule,
+ pub warm_replay: WarmReplay,
+ /// Budget of pods whose seal breaks (`Action::BreakSeal`).
+ pub seal_breaks: u8,
+ /// Whether peers seal overdue partitions; `false` keeps the rejected no-exit rule.
+ pub seal_grace: bool,
+ /// Whether a plan's later units stand down when its basis moved; `false`
+ /// keeps the rejected unguarded planner.
+ pub plan_basis_guard: bool,
+ /// Whether a saga starts only once the fleet is quiet (`fleet_quiet`);
+ /// `false` keeps the rejected start that freezes a fleet in motion.
+ pub quiet_start: bool,
+ /// Whether the freeze cancels on an ordinary handoff (`plan_freeze`'s stray).
+ pub freeze_checks_strays: bool,
+ /// Whether a request parks behind its key's parked requests; `false`
+ /// keeps the rejected bucket rule, which lets a later request overtake.
+ pub park_behind_key: bool,
}
/// Derive one pod's desired state by calling the production
@@ -201,8 +337,43 @@ impl HandoffModel {
fn router_ids(&self) -> impl Iterator<Item = RouterId> {
0..(self.routers + self.late_routers)
}
+ /// Every partition number any generation can have.
fn partition_ids(&self) -> impl Iterator<Item = Partition> {
- 0..self.partitions
+ 0..self.partitions.max(self.migrate_to)
+ }
+ fn key_ids(&self) -> impl Iterator<Item = Key> {
+ 0..self.keys.max(self.partitions)
+ }
+
+ /// `fleet_quiet` over production views; the model's registered pods are all Ready.
+ fn quiet(state: &SystemState) -> bool {
+ let handoffs: Vec<HandoffState> = state
+ .handoffs
+ .iter()
+ .map(|(p, h)| production_handoff(*p, h))
+ .collect();
+ let registered = Self::registered_names(state);
+ let ready: HashSet<&str> = registered.iter().map(String::as_str).collect();
+ fleet_quiet(
+ state.config.live.partitions as u32,
+ &production_assignments(state),
+ &handoffs,
+ &ready,
+ )
+ .is_empty()
+ }
+
+ /// Production `PlanBasis`, coarsened: whether a saga exists, and the live epoch.
+ fn plan_basis(state: &SystemState) -> (bool, Epoch) {
+ (state.migration.is_some(), state.config.live.epoch)
+ }
+
+ /// The planner stands down while a migration owns placement.
+ fn planner_active(state: &SystemState) -> bool {
+ planner_may_place(
+ state.migration.map(|m| m.phase),
+ state.handoffs.values().any(|h| h.held),
+ )
}
/// The production `StickyBalancedStrategy`, called on
@@ -215,7 +386,11 @@ impl HandoffModel {
.map(|(p, owner)| (*p as u32, pod_name(*owner)))
.collect();
let members = self.planner_members(state);
- StickyBalancedStrategy.compute_assignments(&current, &members, self.partitions as u32)
+ StickyBalancedStrategy.compute_assignments(
+ &current,
+ &members,
+ state.config.live.partitions as u32,
+ )
}
/// Planner membership with the model's placement policies applied —
@@ -239,7 +414,7 @@ impl HandoffModel {
// fleet's *desired* size, not its live count, so a crashed
// incoming pod does not inflate its peers' quota.
let incoming = self.pods.saturating_sub(self.hold_pods).max(1);
- let cap = (self.partitions as u32).div_ceil(incoming as u32);
+ let cap = (state.config.live.partitions as u32).div_ceil(incoming as u32);
ids.into_iter()
.map(|id| {
if id < self.hold_pods {
@@ -272,9 +447,9 @@ impl HandoffModel {
if !pod.registered && pod.zombie_writes_left == 0 {
return false;
}
- if !pod.warmed.contains_key(&partition) {
+ let Some(warm) = pod.warmed.get(&partition) else {
return false;
- }
+ };
if pod.fenced.contains(&partition) {
return false;
}
@@ -282,7 +457,37 @@ impl HandoffModel {
// acquired the fence most recently. A warmed pod that is no
// longer the holder would be producing under a fenced-out
// producer, and the broker rejects it before any client ack.
- state.changelogs[&partition].epoch_holder == Some(x)
+ state.changelogs[&(warm.epoch, partition)].epoch_holder == Some(x)
+ }
+
+ /// The leader's `validate_partition`: the key's home under the captured modulus.
+ fn validates(&self, state: &SystemState, x: PodId, partition: Partition, key: Key) -> bool {
+ state.pods[&x]
+ .warmed
+ .get(&partition)
+ .and_then(|warm| modulus_of(&state.config, warm.epoch))
+ .is_some_and(|modulus| home(key, modulus) == partition)
+ }
+
+ /// The logs a warm of `partition` replays, each at its current length.
+ fn warm_prefixes(
+ &self,
+ state: &SystemState,
+ partition: Partition,
+ ) -> BTreeMap<(Epoch, Partition), u8> {
+ let live = (state.config.live.epoch, partition);
+ let keys: Vec<(Epoch, Partition)> = match self.warm_replay {
+ WarmReplay::Sources => {
+ warm_sources(&production_config(&state.config), partition as u32)
+ .into_iter()
+ .map(|s| (topic_epoch(&s.topic), s.partition as Partition))
+ .collect()
+ }
+ WarmReplay::LiveOnly => vec![live],
+ };
+ keys.into_iter()
+ .map(|key| (key, state.changelogs.get(&key).map_or(0, Changelog::len)))
+ .collect()
}
/// One step of `warm_partition` for pod `x` on `partition`. Under
@@ -298,6 +503,9 @@ impl HandoffModel {
partition: Partition,
for_handoff: Option<HandoffId>,
) {
+ // The strong-read capture: the config as etcd holds it now.
+ let epoch = state.config.live.epoch;
+ let live = (epoch, partition);
// Both orderings are two steps in production, but only one of
// them is *observably* two steps. Acquiring the fence first bumps
// the epoch before the read, and an append requires an installed
@@ -317,16 +525,17 @@ impl HandoffModel {
// warm), this collapse is the assumption to revisit first.
let observable_gap = self.warm_order == WarmOrder::ReadFirst;
if !observable_gap {
- let warm = {
- let log = state.changelogs.get_mut(&partition).unwrap();
- log.epoch_holder = Some(x);
+ let sources = self.warm_prefixes(state, partition);
+ state.changelogs.get_mut(&live).unwrap().epoch_holder = Some(x);
+ let pod = state.pods.get_mut(&x).unwrap();
+ pod.warmed.insert(
+ partition,
WarmState {
for_handoff,
- visible: log.len,
- }
- };
- let pod = state.pods.get_mut(&x).unwrap();
- pod.warmed.insert(partition, warm);
+ epoch,
+ sources,
+ },
+ );
pod.fenced.remove(&partition);
return;
}
@@ -336,7 +545,7 @@ impl HandoffModel {
// current HWM while the fence — and thus the rejection of
// a zombie's write — does not exist yet.
let pending = PendingWarm {
- cutoff: state.changelogs[&partition].len,
+ cutoff: state.changelogs[&live].len(),
};
state
.pods
@@ -348,55 +557,65 @@ impl HandoffModel {
Some(PendingWarm { cutoff }) => {
// The cutoff predates the fence: anything committed in
// the gap sits beyond it, invisible forever.
- let warm = {
- let log = state.changelogs.get_mut(&partition).unwrap();
- log.epoch_holder = Some(x);
- WarmState {
- for_handoff,
- visible: cutoff,
- }
- };
+ let mut sources = self.warm_prefixes(state, partition);
+ sources.insert(live, cutoff);
+ state.changelogs.get_mut(&live).unwrap().epoch_holder = Some(x);
let pod = state.pods.get_mut(&x).unwrap();
pod.pending_warm.remove(&partition);
- pod.warmed.insert(partition, warm);
+ pod.warmed.insert(
+ partition,
+ WarmState {
+ for_handoff,
+ epoch,
+ sources,
+ },
+ );
pod.fenced.remove(&partition);
}
}
}
- /// Whether the handoff's new owner has fixed the cutoff it will
- /// serve with: a warm installed for *this* handoff, or a mid-warm
- /// read already taken (a pending warm always belongs to the current
- /// attempt — cancellation requires the target unregistered, which
- /// wipes its process memory). An earlier-era warm does NOT freeze
- /// the cutoff: the Acquiring convergence releases and rebuilds it.
+ /// Whether the handoff's new owner has fixed its serving cutoff over `log`: a warm
+ /// for this handoff, or a mid-warm read already taken. An earlier-era warm does not count.
fn handoff_cutoff_frozen(
&self,
state: &SystemState,
h: &Handoff,
partition: Partition,
+ log: (Epoch, Partition),
) -> bool {
let pod = &state.pods[&h.new_owner];
match pod.warmed.get(&partition) {
- Some(w) => w.for_handoff == Some(h.id),
- None => pod.pending_warm.contains_key(&partition),
+ Some(w) => w.for_handoff == Some(h.id) && w.sources.contains_key(&log),
+ None => pod.pending_warm.contains_key(&partition) && log.1 == partition,
}
}
- /// Whether the assignment owner has fixed its serving cutoff — the
- /// Serving convergence honors any installed warm (an owner's warm
- /// cannot go stale while ownership is continuous), so any warm or a
- /// taken mid-warm read freezes it.
- fn owner_cutoff_frozen(&self, state: &SystemState, x: PodId, partition: Partition) -> bool {
+ /// Whether the assignment owner has fixed its serving cutoff over `log`: any warm,
+ /// or a mid-warm read already taken.
+ fn owner_cutoff_frozen(
+ &self,
+ state: &SystemState,
+ x: PodId,
+ partition: Partition,
+ log: (Epoch, Partition),
+ ) -> bool {
let pod = &state.pods[&x];
- pod.warmed.contains_key(&partition) || pod.pending_warm.contains_key(&partition)
+ match pod.warmed.get(&partition) {
+ Some(w) => w.sources.contains_key(&log),
+ None => pod.pending_warm.contains_key(&partition) && log.1 == partition,
+ }
}
- /// Serve a strong read at pod `x`, if it can (running, partition
- /// warmed). Sets the staleness flag when the pod's visible prefix is
- /// behind the changelog — the read returned state missing at least one
- /// acked write. Returns whether the read was served.
- fn serve_read(&self, state: &mut SystemState, x: PodId, partition: Partition) -> bool {
+ /// Serve a strong read for `key` at pod `x` if it can, flagging a stale read when a
+ /// log holds a newer record the pod cannot see. Returns whether it served.
+ fn serve_read(
+ &self,
+ state: &mut SystemState,
+ x: PodId,
+ partition: Partition,
+ key: Key,
+ ) -> bool {
let pod = &state.pods[&x];
if !pod.running {
return false;
@@ -421,37 +640,50 @@ impl HandoffModel {
let Some(warm) = pod.warmed.get(&partition) else {
return false;
};
- if warm.visible < state.changelogs[&partition].len {
+ if !self.validates(state, x, partition, key) {
+ return false;
+ }
+ let stale = state.changelogs.iter().any(|(log, changelog)| {
+ let visible = warm.sources.get(log).copied().unwrap_or(0) as usize;
+ changelog.records.iter().skip(visible).any(|k| *k == key)
+ });
+ if stale {
state.stale_strong_read = true;
}
state.read_served = true;
true
}
- /// Append one acked write to the changelog, tracking the loss flag:
- /// if the protocol has designated a *different* pod as the (incoming
- /// or current) owner and that pod's warm cutoff is already fixed,
- /// this write sits beyond the cutoff and is invisible to it forever.
- fn accept_write(&self, state: &mut SystemState, x: PodId, partition: Partition) {
- let designated_other = match state.handoffs.get(&partition) {
- Some(h) if h.new_owner != x => self.handoff_cutoff_frozen(state, h, partition),
+ /// Append one acked write to the pod's log, flagging a lost write when another pod
+ /// that owns the key's home has already fixed its cutoff over this log.
+ fn accept_write(&self, state: &mut SystemState, x: PodId, partition: Partition, key: Key) {
+ let epoch = state.pods[&x].warmed[&partition].epoch;
+ let log = (epoch, partition);
+ let home_now = home(key, state.config.live.partitions);
+ let designated_other = match state.handoffs.get(&home_now) {
+ Some(h) if h.new_owner != x => self.handoff_cutoff_frozen(state, h, home_now, log),
Some(_) => false,
- None => match state.assignments.get(&partition) {
- Some(owner) if *owner != x => self.owner_cutoff_frozen(state, *owner, partition),
+ None => match state.assignments.get(&home_now) {
+ Some(owner) if *owner != x => {
+ self.owner_cutoff_frozen(state, *owner, home_now, log)
+ }
_ => false,
},
};
if designated_other {
state.lost_acked_write = true;
}
+ if epoch != state.config.live.epoch {
+ state.mixed_modulus_acceptance = true;
+ }
- let log = state.changelogs.get_mut(&partition).unwrap();
- log.len = log.len.saturating_add(1);
+ state.changelogs.get_mut(&log).unwrap().records.push(key);
let pod = state.pods.get_mut(&x).unwrap();
if let Some(warm) = pod.warmed.get_mut(&partition) {
// Its own write is immediately visible to it.
- warm.visible = warm.visible.saturating_add(1);
+ let visible = warm.sources.entry(log).or_insert(0);
+ *visible = visible.saturating_add(1);
}
if !pod.registered {
pod.zombie_writes_left = pod.zombie_writes_left.saturating_sub(1);
@@ -479,19 +711,15 @@ impl HandoffModel {
/// owner, fail-closed), not decided on the deletion event.
fn cancel_by_replacement(&self, state: &mut SystemState, p: Partition) {
if self.probes_stash_racing_cancellation()
- && state
- .routers
- .values()
- .any(|r| r.stash.get(&p).is_some_and(|q| !q.is_empty()))
+ && state.routers.values().any(|r| Self::parked_at(r, p))
{
state.cancelled_while_stash_parked = true;
}
- state.freeze_acks.retain(|(fp, _), _| fp != &p);
- state.drained_acks.retain(|(dp, _), _| dp != &p);
- state.warmed_acks.retain(|(wp, _), _| wp != &p);
+ Self::clear_acks(state, p);
let owner = state.assignments.get(&p).copied();
+ let epoch = state.config.live.epoch;
if let Some(owner) = owner.filter(|o| state.pods[o].registered) {
let id = state.next_handoff_id;
state.next_handoff_id += 1;
@@ -503,6 +731,8 @@ impl HandoffModel {
new_owner: owner,
phase: Phase::Complete,
quorum: BTreeSet::new(),
+ held: false,
+ config_epoch: epoch,
},
);
if self.probes_reaffirm() {
@@ -513,10 +743,7 @@ impl HandoffModel {
if let Some(target) = self.target_owner(state, p) {
let id = state.next_handoff_id;
state.next_handoff_id += 1;
- let quorum: BTreeSet<RouterId> = self
- .router_ids()
- .filter(|r| state.routers[r].registered)
- .collect();
+ let quorum = Self::quorum_snapshot(state);
state.handoffs.insert(
p,
Handoff {
@@ -525,6 +752,8 @@ impl HandoffModel {
new_owner: target,
phase: Phase::Freezing,
quorum,
+ held: false,
+ config_epoch: epoch,
},
);
if self.probes_successor_replacement() {
@@ -535,35 +764,112 @@ impl HandoffModel {
state.handoffs.remove(&p);
}
- /// Route one write through router `r` exactly as the raw proxy does:
- /// park it if the partition is stashing, otherwise forward to the
- /// table entry and let the leader's admission decide.
- fn route_write(&self, state: &mut SystemState, r: RouterId, partition: Partition) -> bool {
+ fn clear_acks(state: &mut SystemState, p: Partition) {
+ state.freeze_acks.retain(|(fp, _), _| fp != &p);
+ state.drained_acks.retain(|(dp, _), _| dp != &p);
+ state.warmed_acks.retain(|(wp, _), _| wp != &p);
+ }
+
+ /// The coordinator's snapshot read: the routers registered now.
+ fn quorum_snapshot(state: &SystemState) -> BTreeSet<RouterId> {
+ state
+ .routers
+ .iter()
+ .filter(|(_, r)| r.registered)
+ .map(|(r, _)| *r)
+ .collect()
+ }
+
+ /// Whether router `r` holds a parked request whose key maps to `p` under its count.
+ fn parked_at(router: &Router, p: Partition) -> bool {
+ router
+ .stash
+ .iter()
+ .any(|e| home(e.key(), router.adopted.partitions) == p)
+ }
+
+ /// Remove and return router `r`'s parked requests for `p`, in arrival order.
+ fn take_parked(state: &mut SystemState, r: RouterId, p: Partition) -> Vec<StashedRequest> {
+ let router = state.routers.get_mut(&r).unwrap();
+ let partitions = router.adopted.partitions;
+ let (taken, kept) = std::mem::take(&mut router.stash)
+ .into_iter()
+ .partition(|e| home(e.key(), partitions) == p);
+ router.stash = kept;
+ taken
+ }
+
+ /// Route one request as the raw proxy does: park it behind its key or a stashing
+ /// partition, otherwise forward it and let the leader's admission decide.
+ fn route(&self, state: &mut SystemState, r: RouterId, request: StashedRequest) -> bool {
let router = &state.routers[&r];
if !router.running {
return false;
}
- if router.stashing.contains(&partition) {
- state
- .routers
- .get_mut(&r)
- .unwrap()
- .stash
- .entry(partition)
- .or_default()
- .push(StashedRequest::Write);
+ let partition = home(request.key(), router.adopted.partitions);
+ if router.stashing.contains(&partition)
+ || (self.park_behind_key && router.stash.iter().any(|e| e.key() == request.key()))
+ {
+ state.routers.get_mut(&r).unwrap().stash.push(request);
return true;
}
let Some(target) = router.table.get(&partition).copied() else {
+ // A partition born in the flip parks until its handoff lands.
+ let born = state
+ .config
+ .draining
+ .is_some_and(|g| partition >= g.partitions);
+ if born {
+ let router = state.routers.get_mut(&r).unwrap();
+ router.stashing.insert(partition);
+ router.stash.push(request);
+ state.born_partition_write_parked = true;
+ return true;
+ }
// No route: the request is rejected; nothing changes.
return false;
};
- if self.write_capable(state, target, partition) {
- self.accept_write(state, target, partition);
- true
- } else {
- // Rejected fail-closed at the leader; nothing changes.
- false
+ self.deliver(state, r, target, partition, request);
+ true
+ }
+
+ /// Deliver one request from router `r` to `x` for `partition`.
+ fn deliver(
+ &self,
+ state: &mut SystemState,
+ r: RouterId,
+ x: PodId,
+ partition: Partition,
+ entry: StashedRequest,
+ ) {
+ if state.routers[&r]
+ .stash
+ .iter()
+ .any(|parked| parked.key() == entry.key())
+ {
+ state.overtaken_request = true;
+ }
+ match entry {
+ StashedRequest::Write(key) => {
+ if self.validates(state, x, partition, key)
+ && self.write_capable(state, x, partition)
+ {
+ self.accept_write(state, x, partition, key);
+ }
+ // Rejected drains surface UNAVAILABLE to the client (never acked).
+ }
+ StashedRequest::StrongRead(key) => {
+ self.serve_read(state, x, partition, key);
+ }
+ }
+ }
+
+ /// The router adopts the live config (`LiveChangelog::adopt`).
+ fn adopt(state: &mut SystemState, r: RouterId) {
+ let live = state.config.live;
+ let router = state.routers.get_mut(&r).unwrap();
+ if router.adopted.epoch < live.epoch {
+ router.adopted = live;
}
}
@@ -586,30 +892,89 @@ impl HandoffModel {
match *action {
// Decided in the arm: enablement is "the planner produced a
// non-empty plan", which costs a full placement computation.
- Action::Rebalance => true,
+ Action::Rebalance => Self::planner_active(state),
// One pending suffix at a time: chunks of one plan apply
// sequentially in production; a competing plan is the
// Rebalance interleavings.
- Action::RebalanceChunked => self.chunked_plans && state.pending_plan.is_empty(),
+ Action::RebalanceChunked => {
+ self.chunked_plans && state.pending_plan.is_empty() && Self::planner_active(state)
+ }
Action::ApplyPendingUnit => !state.pending_plan.is_empty(),
- Action::CancelDeadNewOwner(p) => matches!(
- state.handoffs.get(&p),
- Some(h) if h.phase != Phase::Complete && !state.pods[&h.new_owner].registered
- ),
+ // The planner's cancellations stand down with the planner and never touch a hold;
+ // production's fallback deletes are guarded per record and not modeled.
+ Action::CancelDeadNewOwner(p) => {
+ Self::planner_active(state)
+ && matches!(
+ state.handoffs.get(&p),
+ Some(h) if h.phase != Phase::Complete && !h.held && !state.pods[&h.new_owner].registered
+ )
+ }
Action::Cancel(p) => {
state.cancels_left > 0
- && matches!(state.handoffs.get(&p), Some(h) if h.phase != Phase::Complete)
+ && Self::planner_active(state)
+ && matches!(state.handoffs.get(&p), Some(h) if h.phase != Phase::Complete && !h.held)
}
// The quorum/drain/warm predicates are the expensive half and
- // stay in the arm; a handoff short of Complete must exist.
+ // stay in the arm; a handoff short of Complete must exist, and
+ // a hold stops at Draining.
Action::AdvancePhase(p) => {
- matches!(state.handoffs.get(&p), Some(h) if h.phase != Phase::Complete)
+ matches!(
+ state.handoffs.get(&p),
+ Some(h) if h.phase != Phase::Complete && !(h.held && h.phase == Phase::Draining)
+ )
}
Action::CleanupComplete(p) => {
matches!(state.handoffs.get(&p), Some(h) if h.phase == Phase::Complete)
}
+ Action::MigrateRequest => {
+ state.migrations_left > 0 && !state.requested && state.migration.is_none()
+ }
+ // Admission refuses a start mid-rollout or while a generation drains.
+ Action::MigrateStart => {
+ state.requested
+ && state.migration.is_none()
+ && state.config.draining.is_none()
+ && self.hold_pods == 0
+ && (!self.quiet_start || Self::quiet(state))
+ }
+ Action::MigrateRequestExpires => state.requested,
+ Action::MigrateDrive => state.migration.is_some(),
+ Action::MigrateCancel => state
+ .migration
+ .is_some_and(|m| m.phase.may_transition_to(MigrationPhase::Cancelling)),
+ Action::SealTick(x) => {
+ let pod = &state.pods[&x];
+ pod.running
+ && pod.registered
+ && !pod.seal_broken
+ && state
+ .migration
+ .is_some_and(|m| m.phase == MigrationPhase::Sealing)
+ }
+ Action::SealGraceElapses => {
+ self.seal_grace
+ && !state.seal_overdue
+ && state
+ .migration
+ .is_some_and(|m| m.phase == MigrationPhase::Sealing)
+ }
+ Action::BreakSeal(x) => {
+ let pod = &state.pods[&x];
+ state.seal_breaks_left > 0
+ && pod.running
+ && pod.registered
+ && !pod.seal_broken
+ && state
+ .migration
+ .is_some_and(|m| m.phase == MigrationPhase::Sealing)
+ }
+ Action::AdoptConfig(r) => {
+ let router = &state.routers[&r];
+ router.running && router.adopted.epoch < state.config.live.epoch
+ }
+
// Which transition a converging pod owes is `desired_state`'s
// answer, derived in the arm; only a live, registered process
// converges at all.
@@ -717,7 +1082,7 @@ impl HandoffModel {
&current,
&in_flight,
&members,
- self.partitions as u32,
+ last.config.live.partitions as u32,
);
if plan.handoffs.is_empty() {
return None;
@@ -755,6 +1120,8 @@ impl HandoffModel {
new_owner: pod_id(&planned.new_owner),
phase: Phase::Freezing,
quorum: quorum.clone(),
+ held: false,
+ config_epoch: state.config.live.epoch,
},
)
.is_some();
@@ -793,6 +1160,7 @@ impl HandoffModel {
.get(&partition)
.copied()
.unwrap_or(0),
+ basis: Self::plan_basis(state),
}
})
.collect();
@@ -806,6 +1174,12 @@ impl HandoffModel {
fn apply_pending_unit(&self, last: &SystemState) -> Option<SystemState> {
let unit = last.pending_plan.first()?.clone();
mutate(last, |state| {
+ // `apply_plan_on`: a moved basis stands the rest of the plan down.
+ if self.plan_basis_guard && unit.basis != Self::plan_basis(state) {
+ state.pending_plan.clear();
+ state.pending_unit_dropped = true;
+ return;
+ }
state.pending_plan.remove(0);
let version = state
.assignment_versions
@@ -820,6 +1194,8 @@ impl HandoffModel {
}
let id = state.next_handoff_id;
state.next_handoff_id += 1;
+ // Production stamps the epoch the plan was read under.
+ let config_epoch = unit.basis.1;
state.handoffs.insert(
unit.partition,
Handoff {
@@ -828,6 +1204,8 @@ impl HandoffModel {
new_owner: unit.new_owner,
phase: Phase::Freezing,
quorum: unit.quorum,
+ held: false,
+ config_epoch,
},
);
state.pending_unit_applied = true;
@@ -882,31 +1260,12 @@ impl HandoffModel {
}
}
Phase::Draining => {
- let registered: Vec<RegisteredPod> = self
- .pod_ids()
- .filter(|x| last.pods[x].registered)
- .map(|x| RegisteredPod {
- pod_name: pod_name(x),
- generation: String::new(),
- status: PodStatus::Ready,
- registered_at: 0,
- last_heartbeat: 0,
- controller: None,
- advertise_address: None,
- })
- .collect();
- let acks: Vec<PodDrainedAck> = self
- .pod_ids()
- .filter_map(|x| {
- last.drained_acks.get(&(p, x)).map(|id| PodDrainedAck {
- pod_name: pod_name(x),
- partition: p as u32,
- acked_at: 0,
- acked_at_ms: 0,
- handoff_id: id.to_string(),
- })
- })
- .collect();
+ // A hold stops here; the migration driver replaces it.
+ if h.held {
+ return None;
+ }
+ let registered = production_registered(last);
+ let acks = self.production_drained_acks(last, p);
if !drain_satisfied(&registered, &acks, &production_handoff(p, h)) {
return None;
}
@@ -960,23 +1319,21 @@ impl HandoffModel {
match model_desired_state(x, last, p) {
DesiredState::Serving => {
let pod = &last.pods[&x];
- if !pod.warmed.contains_key(&p) {
+ match pod.warmed.get(&p) {
// `warm_partition`, one step at a time (see
// `warm_step` for the two orderings under
// EpochFenced).
- mutate(last, |state| self.warm_step(state, x, p, None))
- } else if pod.fenced.contains(&p) {
- mutate(last, |state| {
- // `resume_partition`. The cancelled handoff's
- // target may have taken the broker fence from
- // this pod's producer; production re-acquires it
- // before re-admitting writes, or every write
- // would fail as fenced until the next handoff.
- state.changelogs.get_mut(&p).unwrap().epoch_holder = Some(x);
- state.pods.get_mut(&x).unwrap().fenced.remove(&p);
- })
- } else {
- None
+ None => mutate(last, |state| self.warm_step(state, x, p, None)),
+ Some(w) if pod.fenced.contains(&p) => {
+ let log = (w.epoch, p);
+ mutate(last, |state| {
+ // `resume_partition`: re-take the broker fence the cancelled
+ // target may hold, or every write fails as fenced.
+ state.changelogs.get_mut(&log).unwrap().epoch_holder = Some(x);
+ state.pods.get_mut(&x).unwrap().fenced.remove(&p);
+ })
+ }
+ Some(_) => None,
}
}
DesiredState::Drained { ack } => {
@@ -1005,6 +1362,10 @@ impl HandoffModel {
}
DesiredState::Acquiring => {
let h_id = last.handoffs[&p].id;
+ // The capture parks until the config reaches the handoff's epoch.
+ if last.config.live.epoch < last.handoffs[&p].config_epoch {
+ return None;
+ }
// Only a warm installed for *this* handoff satisfies its
// warming; one from an earlier era is released and
// rebuilt (its cutoff can predate writes accepted since —
@@ -1066,7 +1427,7 @@ impl HandoffModel {
/// router already agreeing with durable state costs no clone.
fn observe(&self, last: &SystemState, r: RouterId, p: Partition) -> Option<SystemState> {
let router = &last.routers[&r];
- let parked = router.stash.get(&p).is_some_and(|q| !q.is_empty());
+ let parked = Self::parked_at(router, p);
match last.handoffs.get(&p) {
Some(h) if h.phase != Phase::Complete => {
// begin_stash on every non-terminal phase; FreezeAck only
@@ -1086,8 +1447,7 @@ impl HandoffModel {
})
}
Some(h) => {
- // Complete: cutover the table, then drain the stash to
- // the new owner in FIFO order.
+ // Complete: cut over the table, then drain `p`'s requests to the new owner.
let new_owner = h.new_owner;
if router.table.get(&p) == Some(&new_owner)
&& !router.stashing.contains(&p)
@@ -1100,21 +1460,10 @@ impl HandoffModel {
let router = state.routers.get_mut(&r).unwrap();
router.table.insert(p, new_owner);
router.stashing.remove(&p);
- router.stash.remove(&p).unwrap_or_default()
+ Self::take_parked(state, r, p)
};
for entry in parked {
- match entry {
- StashedRequest::Write => {
- if self.write_capable(state, new_owner, p) {
- self.accept_write(state, new_owner, p);
- }
- // Rejected drains surface UNAVAILABLE to
- // the client (never acked).
- }
- StashedRequest::StrongRead => {
- self.serve_read(state, new_owner, p);
- }
- }
+ self.deliver(state, r, new_owner, p, entry);
}
})
}
@@ -1142,25 +1491,39 @@ impl HandoffModel {
}
}
router.stashing.remove(&p);
- router.stash.remove(&p).unwrap_or_default()
+ Self::take_parked(state, r, p)
};
for entry in parked {
let Some(owner) = assignment else { continue };
- match entry {
- StashedRequest::Write => {
- if self.write_capable(state, owner, p) {
- self.accept_write(state, owner, p);
- }
- }
- StashedRequest::StrongRead => {
- self.serve_read(state, owner, p);
- }
- }
+ self.deliver(state, r, owner, p, entry);
}
})
}
}
}
+
+ fn production_drained_acks(&self, state: &SystemState, p: Partition) -> Vec<PodDrainedAck> {
+ self.pod_ids()
+ .filter_map(|x| {
+ state.drained_acks.get(&(p, x)).map(|id| PodDrainedAck {
+ pod_name: pod_name(x),
+ partition: p as u32,
+ acked_at: 0,
+ acked_at_ms: 0,
+ handoff_id: id.to_string(),
+ })
+ })
+ .collect()
+ }
+
+ fn registered_names(state: &SystemState) -> HashSet<String> {
+ state
+ .pods
+ .iter()
+ .filter(|(_, pod)| pod.registered)
+ .map(|(x, _)| pod_name(*x))
+ .collect()
+ }
}
impl Model for HandoffModel {
@@ -1179,12 +1542,18 @@ impl Model for HandoffModel {
warmed: BTreeMap::new(),
fenced: BTreeSet::new(),
pending_warm: BTreeMap::new(),
+ sealed: BTreeSet::new(),
+ seal_broken: false,
zombie_writes_left: 0,
claims_authority: true,
},
)
})
.collect();
+ let live = Generation {
+ partitions: self.partitions,
+ epoch: 1,
+ };
let routers: BTreeMap<RouterId, Router> = self
.router_ids()
.map(|id| {
@@ -1196,16 +1565,16 @@ impl Model for HandoffModel {
Router {
registered: active,
running: active,
+ adopted: live,
table: BTreeMap::new(),
stashing: BTreeSet::new(),
- stash: BTreeMap::new(),
+ stash: Vec::new(),
},
)
})
.collect();
- let changelogs: BTreeMap<Partition, Changelog> = self
- .partition_ids()
- .map(|p| (p, Changelog::default()))
+ let changelogs: BTreeMap<(Epoch, Partition), Changelog> = (0..self.partitions)
+ .map(|p| ((live.epoch, p), Changelog::default()))
.collect();
vec![SystemState {
@@ -1217,6 +1586,12 @@ impl Model for HandoffModel {
next_handoff_id: 0,
pending_plan: Vec::new(),
assignment_versions: BTreeMap::new(),
+ config: Config {
+ live,
+ draining: None,
+ },
+ migration: None,
+ seal_acks: BTreeMap::new(),
pods,
routers,
changelogs,
@@ -1226,15 +1601,25 @@ impl Model for HandoffModel {
rejoins_left: self.rejoins,
router_joins_left: self.router_joins,
cancels_left: self.cancels,
+ migrations_left: self.migrations,
read_served: false,
lost_acked_write: false,
double_planned_handoff: false,
stale_strong_read: false,
+ mixed_modulus_acceptance: false,
+ overtaken_request: false,
reaffirmed: false,
replaced_with_successor: false,
cancelled_while_stash_parked: false,
pending_unit_applied: false,
pending_unit_dropped: false,
+ migration_completed: false,
+ migration_cancelled: false,
+ sealed_without_warm: false,
+ born_partition_write_parked: false,
+ seal_overdue: false,
+ seal_breaks_left: self.seal_breaks,
+ requested: false,
}]
}
@@ -1252,6 +1637,11 @@ impl Model for HandoffModel {
offer(Action::Rebalance);
offer(Action::RebalanceChunked);
offer(Action::ApplyPendingUnit);
+ offer(Action::MigrateRequest);
+ offer(Action::MigrateStart);
+ offer(Action::MigrateRequestExpires);
+ offer(Action::MigrateDrive);
+ offer(Action::MigrateCancel);
for p in self.partition_ids() {
offer(Action::CancelDeadNewOwner(p));
offer(Action::Cancel(p));
@@ -1262,11 +1652,21 @@ impl Model for HandoffModel {
}
for r in self.router_ids() {
offer(Action::Observe(r, p));
- offer(Action::ClientWrite(r, p));
- offer(Action::ClientStrongRead(r, p));
}
}
+ for key in self.key_ids() {
+ for r in self.router_ids() {
+ offer(Action::ClientWrite(r, key));
+ offer(Action::ClientStrongRead(r, key));
+ }
+ }
+ for r in self.router_ids() {
+ offer(Action::AdoptConfig(r));
+ }
+ offer(Action::SealGraceElapses);
for pod in self.pod_ids() {
+ offer(Action::SealTick(pod));
+ offer(Action::BreakSeal(pod));
offer(Action::CrashRestartWithinTtl(pod));
offer(Action::LeaseExpire(pod));
}
@@ -1333,11 +1733,35 @@ impl Model for HandoffModel {
// (guarded delete of the record and its acks).
Action::CleanupComplete(p) => mutate(last, |state| {
state.handoffs.remove(&p);
- state.freeze_acks.retain(|(fp, _), _| fp != &p);
- state.drained_acks.retain(|(dp, _), _| dp != &p);
- state.warmed_acks.retain(|(wp, _), _| wp != &p);
+ Self::clear_acks(state, p);
}),
+ // ── the migration saga ─────────────────────────────
+ Action::MigrateRequest => mutate(last, |state| {
+ state.migrations_left -= 1;
+ state.requested = true;
+ }),
+ Action::MigrateRequestExpires => mutate(last, |state| state.requested = false),
+ Action::MigrateStart => mutate(last, |state| {
+ state.requested = false;
+ state.migration = Some(Migration {
+ from: state.config.live,
+ to: self.migrate_to,
+ phase: MigrationPhase::Freezing,
+ });
+ }),
+ Action::MigrateDrive => self.migrate_drive(last),
+ Action::MigrateCancel => mutate(last, |state| {
+ Self::set_phase(state, MigrationPhase::Cancelling);
+ }),
+ Action::SealTick(x) => self.seal_tick(last, x),
+ Action::SealGraceElapses => mutate(last, |state| state.seal_overdue = true),
+ Action::BreakSeal(x) => mutate(last, |state| {
+ state.seal_breaks_left -= 1;
+ state.pods.get_mut(&x).unwrap().seal_broken = true;
+ }),
+ Action::AdoptConfig(r) => mutate(last, |state| Self::adopt(state, r)),
+
// ── pod converge ───────────────────────────────────
Action::Converge(x, p) => self.converge(last, x, p),
@@ -1345,37 +1769,19 @@ impl Model for HandoffModel {
Action::Observe(r, p) => self.observe(last, r, p),
// ── workload ───────────────────────────────────────
- Action::ClientWrite(r, p) => mutate(last, |state| {
+ Action::ClientWrite(r, key) => mutate(last, |state| {
state.writes_left -= 1;
// A rejected write still consumes its budget: the state
// changed, so the transition is kept and rejected paths
// are explored too.
- self.route_write(state, r, p);
+ self.route(state, r, StashedRequest::Write(key));
}),
- Action::ClientStrongRead(r, p) => mutate(last, |state| {
+ Action::ClientStrongRead(r, key) => mutate(last, |state| {
state.reads_left -= 1;
- let router = &state.routers[&r];
- if !router.running {
- // Router process gone; request fails at the client.
- } else if router.stashing.contains(&p) {
- // Strong reads park in the same per-partition FIFO as
- // writes while the partition is stashing (the shipped
- // read-stashing design, #69456) — which is what keeps
- // them complete across cutover.
- state
- .routers
- .get_mut(&r)
- .unwrap()
- .stash
- .entry(p)
- .or_default()
- .push(StashedRequest::StrongRead);
- } else if let Some(target) = router.table.get(&p).copied() {
- // Outside a handoff: forward to the table entry.
- // Rejected reads fail closed at the leader.
- self.serve_read(state, target, p);
- }
+ // Strong reads take the write path: parked while stashing
+ // (#69456), else forwarded to the table entry.
+ self.route(state, r, StashedRequest::StrongRead(key));
}),
// ── failures ───────────────────────────────────────
@@ -1387,6 +1793,8 @@ impl Model for HandoffModel {
pod.warmed.clear();
pod.fenced.clear();
pod.pending_warm.clear();
+ pod.sealed.clear();
+ pod.seal_broken = false;
pod.zombie_writes_left = 0;
// A new session is a new claim.
pod.claims_authority = true;
@@ -1397,6 +1805,8 @@ impl Model for HandoffModel {
pod.warmed.clear();
pod.fenced.clear();
pod.pending_warm.clear();
+ pod.sealed.clear();
+ pod.seal_broken = false;
}),
Action::LeaseExpire(x) => mutate(last, |state| {
state.crashes_left -= 1;
@@ -1440,6 +1850,8 @@ impl Model for HandoffModel {
pod.warmed.clear();
pod.fenced.clear();
pod.pending_warm.clear();
+ pod.sealed.clear();
+ pod.seal_broken = false;
pod.zombie_writes_left = 0;
pod.claims_authority = false;
}),
@@ -1527,12 +1939,39 @@ impl Model for HandoffModel {
// model found the cutover race as a counterexample — the
// machine validation that motivated shipping read stashing.
Property::<Self>::always("strong_reads_complete", |_, s| !s.stale_strong_read),
+ Property::<Self>::always("no_overtaken_request", |_, s| !s.overtaken_request),
+ // An ack lands in the live generation, never a retired one.
+ Property::<Self>::always("no_mixed_modulus_acceptance", |_, s| {
+ !s.mixed_modulus_acceptance
+ }),
+ // The ordinary handoff loop never moves a hold past Draining.
+ Property::<Self>::always("holds_stop_at_draining", |_, s| {
+ s.handoffs
+ .values()
+ .all(|h| !h.held || matches!(h.phase, Phase::Freezing | Phase::Draining))
+ }),
+ // Once flipped, no pod can produce to the old generation.
+ Property::<Self>::always("old_generation_quiet_after_flip", |m, s| {
+ s.config.draining.is_none_or(|g| {
+ !s.pods.iter().any(|(x, pod)| {
+ pod.warmed
+ .iter()
+ .any(|(p, w)| w.epoch == g.epoch && m.write_capable(s, *x, *p))
+ })
+ })
+ }),
+ Property::<Self>::sometimes("migration_completes", |m, s| {
+ m.migrations == 0 || s.migration_completed
+ }),
+ Property::<Self>::sometimes("migration_cancels", |m, s| {
+ m.migrations == 0 || s.migration_cancelled
+ }),
// Sanity: the interesting states are actually reachable.
Property::<Self>::sometimes("some_handoff_completes", |_, s| {
s.handoffs.values().any(|h| h.phase == Phase::Complete)
}),
Property::<Self>::sometimes("some_write_accepted", |_, s| {
- s.changelogs.values().any(|log| log.len > 0)
+ s.changelogs.values().any(|log| !log.is_empty())
}),
Property::<Self>::sometimes("some_strong_read_served", |m, s| {
// Vacuously discoverable in configs without reads, so
@@ -1556,13 +1995,18 @@ impl Model for HandoffModel {
// Liveness: every full run ends quiescent and converged —
// no handoffs in flight, every assignment served by a warm,
// unfenced, registered pod, all routers agreeing, nothing
- // parked in a stash.
+ // parked in a stash, and no migration left to request.
Property::<Self>::eventually("converges_to_stable", |m, s| {
// Ordered cheapest-first on purpose. This runs at every
// explored state, and the placement computation below is
// the one expensive part — it must stay behind the
// conjuncts that reject most states outright.
- if !s.handoffs.is_empty() || !s.pending_plan.is_empty() {
+ if !s.handoffs.is_empty()
+ || !s.pending_plan.is_empty()
+ || s.migration.is_some()
+ || s.requested
+ || s.migrations_left > 0
+ {
return false;
}
let no_capacity = s.pods.values().all(|p| !p.registered);
@@ -1573,7 +2017,7 @@ impl Model for HandoffModel {
// rebuilds `String` maps and reruns the production
// strategy on every call.
let targets = m.target_assignments(s);
- m.partition_ids().all(|p| {
+ (0..s.config.live.partitions).all(|p| {
let Some(target) = targets.get(&(p as u32)).map(|name| pod_id(name))
else {
return true;
@@ -1602,14 +2046,25 @@ impl Model for HandoffModel {
if !router.registered || !router.running {
return true;
}
- router.stashing.is_empty()
- && router.stash.values().all(|q| q.is_empty())
+ router.adopted == s.config.live
+ && router.stashing.is_empty()
+ && router.stash.is_empty()
&& s.assignments
.iter()
.all(|(p, owner)| router.table.get(p) == Some(owner))
})
}),
];
+ if self.probes && self.migrations > 0 {
+ props.push(Property::<Self>::sometimes(
+ "sealed_without_warm",
+ |_, s| s.sealed_without_warm,
+ ));
+ props.push(Property::<Self>::sometimes(
+ "born_partition_write_parked",
+ |_, s| s.born_partition_write_parked,
+ ));
+ }
if self.probes {
// The freeze-quorum snapshot doing its job: a handoff
// advanced past Freezing while some registered router — a
diff --git a/rust/personhog-stateright/src/model/migration.rs b/rust/personhog-stateright/src/model/migration.rs
new file mode 100644
index 000000000000..438b739cd988
--- /dev/null
+++ b/rust/personhog-stateright/src/model/migration.rs
@@ -0,0 +1,318 @@
+//! The migration saga's actions: the coordinator's driver passes and the leaders' seal duty.
+
+use super::*;
+
+impl HandoffModel {
+ fn held_handoffs(state: &SystemState) -> Vec<(Partition, Handoff)> {
+ state
+ .handoffs
+ .iter()
+ .filter(|(_, h)| h.held)
+ .map(|(p, h)| (*p, h.clone()))
+ .collect()
+ }
+
+ pub(super) fn set_phase(state: &mut SystemState, next: MigrationPhase) {
+ let migration = state.migration.as_mut().expect("a migration is in flight");
+ assert!(
+ migration.phase.may_transition_to(next),
+ "{:?} -> {next:?}",
+ migration.phase
+ );
+ migration.phase = next;
+ }
+
+ fn insert_handoff(state: &mut SystemState, p: Partition, handoff: Handoff) {
+ Self::clear_acks(state, p);
+ state.handoffs.insert(p, handoff);
+ }
+
+ /// `migration::reconcile`, one pass.
+ pub(super) fn migrate_drive(&self, last: &SystemState) -> Option<SystemState> {
+ let migration = last.migration?;
+ mutate(last, |state| match migration.phase {
+ MigrationPhase::Freezing => self.drive_freeze(state, migration),
+ MigrationPhase::Draining => self.drive_drain(state, migration),
+ MigrationPhase::Sealing => Self::drive_seal(state, migration),
+ MigrationPhase::Resuming => self.drive_resume(state, migration),
+ MigrationPhase::Cancelling => Self::drive_unwind(state, migration),
+ MigrationPhase::Complete | MigrationPhase::Cancelled => {
+ state.migration = None;
+ }
+ })
+ }
+
+ /// `Pass::freeze`, through the production `plan_freeze`.
+ fn drive_freeze(&self, state: &mut SystemState, migration: Migration) {
+ let from = migration.from;
+ let handoffs: Vec<HandoffState> = state
+ .handoffs
+ .iter()
+ .map(|(p, h)| production_handoff(*p, h))
+ .collect();
+ let registered = Self::registered_names(state);
+ let registered: HashSet<&str> = registered.iter().map(String::as_str).collect();
+ let plan = plan_freeze(
+ from.partitions as u32,
+ &production_assignments(state),
+ &handoffs,
+ &registered,
+ );
+ if self.freeze_checks_strays && plan.stray_handoff.is_some() {
+ Self::set_phase(state, MigrationPhase::Cancelling);
+ return;
+ }
+ if plan.holder_died.is_some() {
+ Self::set_phase(state, MigrationPhase::Cancelling);
+ return;
+ }
+ let quorum = Self::quorum_snapshot(state);
+ for (p, owner) in plan.holds {
+ let id = state.next_handoff_id;
+ state.next_handoff_id += 1;
+ let owner = pod_id(&owner);
+ state.handoffs.insert(
+ p as Partition,
+ Handoff {
+ id,
+ old_owner: Some(owner),
+ new_owner: owner,
+ phase: Phase::Freezing,
+ quorum: quorum.clone(),
+ held: true,
+ config_epoch: from.epoch,
+ },
+ );
+ }
+ if plan.blocking.is_empty() {
+ Self::set_phase(state, MigrationPhase::Draining);
+ }
+ }
+
+ /// `Pass::drain`: every hold drained, else a dead holder cancels.
+ fn drive_drain(&self, state: &mut SystemState, _migration: Migration) {
+ let registered = production_registered(state);
+ let mut blocked = false;
+ for (p, h) in Self::held_handoffs(state) {
+ if !state.pods[&h.new_owner].registered {
+ Self::set_phase(state, MigrationPhase::Cancelling);
+ return;
+ }
+ if h.phase == Phase::Freezing {
+ blocked = true;
+ continue;
+ }
+ let acks = self.production_drained_acks(state, p);
+ blocked |= !drain_satisfied(&registered, &acks, &production_handoff(p, &h));
+ }
+ if !blocked {
+ Self::set_phase(state, MigrationPhase::Sealing);
+ }
+ }
+
+ /// `Pass::seal`: the flip once every old partition is sealed, as one transaction.
+ fn drive_seal(state: &mut SystemState, migration: Migration) {
+ let acks = production_seal_acks(state);
+ if !migration_unsealed(&acks, MIGRATION_ID, migration.from.partitions as u32).is_empty() {
+ return;
+ }
+ let next = Generation {
+ partitions: migration.to,
+ epoch: state.config.live.epoch + 1,
+ };
+ for p in 0..next.partitions {
+ state.changelogs.entry((next.epoch, p)).or_default();
+ }
+ state.config = Config {
+ live: next,
+ draining: Some(migration.from),
+ };
+ state.seal_overdue = false;
+ Self::set_phase(state, MigrationPhase::Resuming);
+ }
+
+ /// `Pass::resume`: holds become restarts at Warming, shrunk partitions
+ /// dissolve, grown ones are placed; then Complete once settled.
+ fn drive_resume(&self, state: &mut SystemState, migration: Migration) {
+ let to = migration.to;
+ let epoch = state.config.live.epoch;
+ let held = Self::held_handoffs(state);
+ if !held.is_empty() {
+ let holds: Vec<(u32, String)> = held
+ .iter()
+ .map(|(p, h)| (*p as u32, pod_name(h.new_owner)))
+ .collect();
+ let handoffs: Vec<HandoffState> = state
+ .handoffs
+ .iter()
+ .map(|(p, h)| production_handoff(*p, h))
+ .collect();
+ let names = Self::registered_names(state);
+ let registered: HashSet<&str> = names.iter().map(String::as_str).collect();
+ let plan = plan_resume(
+ &StickyBalancedStrategy,
+ &self.planner_members(state),
+ to as u32,
+ &holds,
+ &production_assignments(state),
+ &handoffs,
+ &registered,
+ );
+ for p in plan.dissolve {
+ let p = p as Partition;
+ Self::clear_acks(state, p);
+ state.handoffs.remove(&p);
+ state.assignments.remove(&p);
+ }
+ for p in plan.abandon {
+ let p = p as Partition;
+ Self::clear_acks(state, p);
+ state.handoffs.remove(&p);
+ }
+ for (p, holder) in plan.restart {
+ let id = state.next_handoff_id;
+ state.next_handoff_id += 1;
+ let holder = pod_id(&holder);
+ Self::insert_handoff(
+ state,
+ p as Partition,
+ Handoff {
+ id,
+ old_owner: Some(holder),
+ new_owner: holder,
+ phase: Phase::Warming,
+ quorum: BTreeSet::new(),
+ held: false,
+ config_epoch: epoch,
+ },
+ );
+ }
+ let quorum = Self::quorum_snapshot(state);
+ for (p, owner) in plan.place {
+ let id = state.next_handoff_id;
+ state.next_handoff_id += 1;
+ state.handoffs.insert(
+ p as Partition,
+ Handoff {
+ id,
+ old_owner: None,
+ new_owner: pod_id(&owner),
+ phase: Phase::Freezing,
+ quorum: quorum.clone(),
+ held: false,
+ config_epoch: epoch,
+ },
+ );
+ }
+ return;
+ }
+ let handoffs: Vec<HandoffState> = state
+ .handoffs
+ .iter()
+ .map(|(p, h)| production_handoff(*p, h))
+ .collect();
+ let registered = Self::registered_names(state);
+ let registered: HashSet<&str> = registered.iter().map(String::as_str).collect();
+ let unsettled = migration_unsettled(
+ &production_assignments(state),
+ &handoffs,
+ &registered,
+ to as u32,
+ );
+ if unsettled.is_empty() {
+ state.migration = None;
+ state.seal_acks.clear();
+ state.migration_completed = true;
+ }
+ }
+
+ /// `Pass::unwind`: each hold becomes a Complete reaffirm, then the record is archived.
+ fn drive_unwind(state: &mut SystemState, migration: Migration) {
+ let held = Self::held_handoffs(state);
+ if held.is_empty() {
+ state.migration = None;
+ state.seal_acks.clear();
+ state.migration_cancelled = true;
+ return;
+ }
+ for (p, h) in held {
+ if !state.pods[&h.new_owner].registered {
+ Self::clear_acks(state, p);
+ state.handoffs.remove(&p);
+ continue;
+ }
+ let id = state.next_handoff_id;
+ state.next_handoff_id += 1;
+ Self::insert_handoff(
+ state,
+ p,
+ Handoff {
+ id,
+ old_owner: None,
+ new_owner: h.new_owner,
+ phase: Phase::Complete,
+ quorum: BTreeSet::new(),
+ held: false,
+ config_epoch: migration.from.epoch,
+ },
+ );
+ }
+ }
+
+ /// `SealDuty::tick`: seal every target not yet sealed, then rewrite the ack.
+ pub(super) fn seal_tick(&self, last: &SystemState, x: PodId) -> Option<SystemState> {
+ let migration = last.migration?;
+ let from = migration.from;
+ let targets: Vec<Partition> = match self.seal_rule {
+ SealRule::Assignment => {
+ let registered = Self::registered_names(last);
+ let registered: HashSet<&str> = registered.iter().map(String::as_str).collect();
+ seal_targets(
+ &pod_name(x),
+ MIGRATION_ID,
+ from.partitions as u32,
+ &production_assignments(last),
+ &registered,
+ &production_seal_acks(last),
+ last.seal_overdue,
+ )
+ .into_iter()
+ .map(|p| p as Partition)
+ .collect()
+ }
+ SealRule::Residency => (0..from.partitions)
+ .filter(|p| {
+ let orphaned = last
+ .assignments
+ .get(p)
+ .is_none_or(|owner| !last.pods[owner].registered);
+ last.pods[&x].warmed.contains_key(p) || orphaned
+ })
+ .collect(),
+ };
+ let pod = &last.pods[&x];
+ let newly: Vec<Partition> = targets
+ .iter()
+ .copied()
+ .filter(|p| !pod.sealed.contains(p))
+ .collect();
+ // `SealDuty::tick`: everything this pod sealed stays reported.
+ let acked: BTreeSet<Partition> = pod.sealed.iter().chain(newly.iter()).copied().collect();
+ let unloaded = newly.iter().any(|p| !pod.warmed.contains_key(p));
+ mutate(last, |state| {
+ for p in &newly {
+ // `init_transactions` by the sealer, then its producer is dropped.
+ state
+ .changelogs
+ .get_mut(&(from.epoch, *p))
+ .unwrap()
+ .epoch_holder = None;
+ state.pods.get_mut(&x).unwrap().sealed.insert(*p);
+ }
+ if unloaded {
+ state.sealed_without_warm = true;
+ }
+ state.seal_acks.insert(x, acked);
+ })
+ }
+}
diff --git a/rust/personhog-stateright/src/types.rs b/rust/personhog-stateright/src/types.rs
index 8e6bc803fb52..1bf679417026 100644
--- a/rust/personhog-stateright/src/types.rs
+++ b/rust/personhog-stateright/src/types.rs
@@ -13,10 +13,17 @@ pub type Partition = u8;
/// Identity of one handoff attempt — production `HandoffState.handoff_id`.
/// Acks echo it and quorum checks only count matching acks.
pub type HandoffId = u8;
+/// A changelog generation's epoch (production `ChangelogConfig::epoch`).
+pub type Epoch = u8;
+/// A person key. The model's hash is the identity: like production's
+/// hash-then-mod, a key's homes under two counts agree modulo their gcd.
+pub type Key = u8;
/// The production phase enum, used directly: a new phase added to the
/// protocol breaks the model's exhaustive matches at compile time.
pub use personhog_coordination::types::HandoffPhase as Phase;
+/// The production saga phases, used directly for the same reason.
+pub use personhog_coordination::types::MigrationPhase;
/// Production `HandoffState`, in compact model form (u8 ids; the model
/// materializes production-typed views when calling shared logic).
@@ -33,11 +40,37 @@ pub struct Handoff {
/// With `RouterJoin` in the action space this diverges from the live
/// registry, which is exactly what the checker is here to explore.
pub quorum: BTreeSet<RouterId>,
+ /// A migration's hold (production `held_by`): never advanced past Draining.
+ pub held: bool,
+ /// Production `config_epoch`: an acquisition parks until the config reaches it.
+ pub config_epoch: Epoch,
+}
+
+/// One changelog generation (production: the live target, or a `draining` entry).
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct Generation {
+ pub partitions: u8,
+ pub epoch: Epoch,
+}
+
+/// Production `ChangelogConfig`: the live generation and at most one draining.
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct Config {
+ pub live: Generation,
+ pub draining: Option<Generation>,
+}
+
+/// Production `MigrationState`.
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct Migration {
+ pub from: Generation,
+ pub to: u8,
+ pub phase: MigrationPhase,
}
/// Per-partition warm state on a pod — everything the invariants need to
/// know about one warmed partition.
-#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
+#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct WarmState {
/// What this warm was installed for (production:
/// `WarmProvenance` on the pod handle). `None` is a serving-era warm
@@ -48,17 +81,11 @@ pub struct WarmState {
/// whose cache may predate writes accepted since, and is released
/// and rebuilt.
pub for_handoff: Option<HandoffId>,
- /// How much of the changelog this pod's cache reflects: the HWM
- /// captured at warm time, plus every write it has accepted since. A
- /// strong read served while `changelog.len` exceeds this returns state
- /// missing at least one acked write.
- ///
- /// The two halves are deliberately one number. Nothing reads them
- /// apart — the cutoff is never consulted on its own, and the accept
- /// count only ever advances the same total — so keeping them separate
- /// would only split behaviorally identical states (warmed at 1 and
- /// accepted nothing, versus warmed at 0 and accepted one).
- pub visible: u8,
+ /// The epoch captured at acquisition (production `Capture`).
+ pub epoch: Epoch,
+ /// Visible prefix per source log (production: marks keyed by stream);
+ /// the live source advances with each accepted write.
+ pub sources: BTreeMap<(Epoch, Partition), u8>,
}
/// A warm caught between its two steps, under the rejected read-first
@@ -72,7 +99,7 @@ pub struct WarmState {
/// warm atomically.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct PendingWarm {
- /// The changelog length captured before the fence existed.
+ /// The live log's length captured before the fence existed.
pub cutoff: u8,
}
@@ -94,6 +121,10 @@ pub struct Pod {
/// Write-fenced partitions (production: `InflightTracker` fences +
/// the pod handle's `fenced_partitions`).
pub fenced: BTreeSet<Partition>,
+ /// Old-topic partitions this incarnation sealed (the seal duty's memory).
+ pub sealed: BTreeSet<Partition>,
+ /// This incarnation's seal never succeeds.
+ pub seal_broken: bool,
/// Remaining writes this pod may accept after losing its lease before
/// its keepalive notices and the process self-fences (production: the
/// bounded zombie window; fix 1 bounds it to ~one heartbeat tick).
@@ -122,42 +153,42 @@ pub struct Router {
pub registered: bool,
/// The process is running and forwards traffic.
pub running: bool,
+ /// The config this router hashes with (production `LiveChangelog`).
+ pub adopted: Generation,
/// partition → pod the router forwards to (production: the shared
/// routing table, edge-updated from handoff Complete events).
pub table: BTreeMap<Partition, PodId>,
/// Partitions currently buffering leader-path requests (production:
- /// the `StashTable`).
+ /// the `StashTable`'s open freezes).
pub stashing: BTreeSet<Partition>,
- /// Parked leader-path requests in arrival order (production:
- /// `StashedRequest` queues, which carry their gRPC method so writes
- /// and strong reads share one per-partition FIFO).
- pub stash: BTreeMap<Partition, Vec<StashedRequest>>,
+ /// Parked leader-path requests in arrival order (production: the `StashTable`'s
+ /// per-key queues). A drain for `p` takes those whose key maps to `p` under the router's count.
+ pub stash: Vec<StashedRequest>,
}
/// One parked leader-path request.
-///
-/// Deliberately identity-free. A parked write is only ever drained or
-/// dropped, never matched against a particular client, so carrying a write
-/// id would add a dimension to the state space that nothing reads — two
-/// otherwise-identical states would differ only in how many writes had ever
-/// been parked. Order and count still matter, and the per-partition `Vec`
-/// keeps both.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub enum StashedRequest {
- Write,
- StrongRead,
+ Write(Key),
+ StrongRead(Key),
+}
+
+impl StashedRequest {
+ pub fn key(self) -> Key {
+ match self {
+ StashedRequest::Write(key) | StashedRequest::StrongRead(key) => key,
+ }
+ }
}
-/// The Kafka changelog for one partition, reduced to what the safety
-/// invariants need: an append counter (the HWM) and who currently holds
-/// the broker's producer fence.
+/// The Kafka changelog for one partition of one generation: the appended
+/// keys (their count is the HWM) and the broker's producer fence.
#[derive(Clone, Debug, Default, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct Changelog {
- /// Number of records appended (the HWM).
- pub len: u8,
+ pub records: Vec<Key>,
/// The pod whose transactional producer the broker currently accepts:
/// the latest `init_transactions` wins and every earlier producer is
- /// fenced out.
+ /// fenced out. A seal initializes the id and drops the producer.
///
/// This is an owner rather than an epoch number because nothing
/// compares epochs for order — the single consumer asks whether *this*
@@ -168,6 +199,16 @@ pub struct Changelog {
pub epoch_holder: Option<PodId>,
}
+impl Changelog {
+ pub fn len(&self) -> u8 {
+ self.records.len() as u8
+ }
+
+ pub fn is_empty(&self) -> bool {
+ self.records.is_empty()
+ }
+}
+
/// One planned-but-unapplied handoff from a chunked plan: the placement
/// decision plus its plan-time guards (production: a later `apply_plan`
/// chunk, applied without re-reading).
@@ -181,6 +222,8 @@ pub struct PendingUnit {
/// Assignment version at plan time (production: the mod_revision
/// its `AssignmentPrecondition` compares; 0 = Absent).
pub expected_assignment_version: u8,
+ /// The plan's basis (production `PlanBasis`): a saga in flight, and the live epoch.
+ pub basis: (bool, Epoch),
}
/// The entire distributed system. One value = one node in the explored
@@ -209,13 +252,20 @@ pub struct SystemState {
/// mod_revision an `AssignmentPrecondition` compares). Only
/// maintained under `chunked_plans`.
pub assignment_versions: BTreeMap<Partition, u8>,
+ /// Production `config/changelog`.
+ pub config: Config,
+ /// Production `migrations/current`.
+ pub migration: Option<Migration>,
+ /// pod → old-topic partitions its seal ack reports sealed.
+ pub seal_acks: BTreeMap<PodId, BTreeSet<Partition>>,
// ── processes ──────────────────────────────────────────────
pub pods: BTreeMap<PodId, Pod>,
pub routers: BTreeMap<RouterId, Router>,
// ── kafka ──────────────────────────────────────────────────
- pub changelogs: BTreeMap<Partition, Changelog>,
+ /// (generation epoch, partition) → its log.
+ pub changelogs: BTreeMap<(Epoch, Partition), Changelog>,
// ── failure/workload budgets (bound the state space) ───────
pub writes_left: u8,
@@ -227,6 +277,7 @@ pub struct SystemState {
pub router_joins_left: u8,
/// Deadline cancellations the checker may still inject.
pub cancels_left: u8,
+ pub migrations_left: u8,
/// Whether any strong read was actually served (reachability evidence
/// for the read properties). A flag rather than a count: nothing reads
/// how many, and a count would split otherwise-identical states by
@@ -244,10 +295,14 @@ pub struct SystemState {
/// in-flight handoff, clobbering it — the overlap
/// `plan_partial_rebalance`'s pinning must make unreachable.
pub double_planned_handoff: bool,
- /// Set when a strong read is served by a pod whose `WarmState::visible`
- /// prefix is behind the changelog — the read returned state missing at
- /// least one acked write.
+ /// Set when a strong read is served by a pod whose warm prefixes are
+ /// behind some log holding the key.
pub stale_strong_read: bool,
+ /// Set when a pod acks a write under a captured epoch that is not the live one.
+ pub mixed_modulus_acceptance: bool,
+ /// Set when a router delivers a request while an earlier request for
+ /// the same key is still parked, so the later one lands first.
+ pub overtaken_request: bool,
// ── reachability flags (probe evidence, history encoding) ──
/// A cancellation resolved as a reaffirm-Complete toward the live
@@ -263,6 +318,17 @@ pub struct SystemState {
pub pending_unit_applied: bool,
/// A pending chunk unit dropped by a failed plan-time guard.
pub pending_unit_dropped: bool,
+ pub migration_completed: bool,
+ pub migration_cancelled: bool,
+ /// A pod sealed an old-topic partition it had not loaded.
+ pub sealed_without_warm: bool,
+ /// A write for a partition born in the flip parked before its handoff existed.
+ pub born_partition_write_parked: bool,
+ /// A migration request is waiting to start.
+ pub requested: bool,
+ /// The Sealing grace has lapsed for the current saga.
+ pub seal_overdue: bool,
+ pub seal_breaks_left: u8,
}
/// Everything that can happen, from every actor, including failures.
@@ -307,6 +373,26 @@ pub enum Action {
/// (guarded on the record still being the same attempt).
CleanupComplete(Partition),
+ // ── the migration saga ─────────────────────────────────────
+ /// The control plane writes `migrations/requested`. Budgeted.
+ MigrateRequest,
+ /// The coordinator starts the saga from the request once no handoff is in flight.
+ MigrateStart,
+ /// A request is rejected or expires before it starts.
+ MigrateRequestExpires,
+ /// One pass of the migration driver (`migration::reconcile`).
+ MigrateDrive,
+ /// A cancel before Sealing: the operator's, or the pre-flip deadline's.
+ MigrateCancel,
+ /// One pass of a pod's seal duty while the saga is Sealing.
+ SealTick(PodId),
+ /// The seal grace lapses: every leader now seals what no ack reports sealed.
+ SealGraceElapses,
+ /// A pod's seal keeps failing while its lease stays alive (a hung Kafka client).
+ BreakSeal(PodId),
+ /// A router's config watch delivers the live config.
+ AdoptConfig(RouterId),
+
// ── pods (the converge model) ──────────────────────────────
/// One convergence step for (pod, partition): derive the desired
/// state from durable state and apply the next transition toward it
@@ -321,13 +407,10 @@ pub enum Action {
Observe(RouterId, Partition),
// ── workload ───────────────────────────────────────────────
- /// A client write for (router, partition): stashes if the router is
- /// stashing, otherwise forwards to the router's table entry.
- ClientWrite(RouterId, Partition),
- /// A strong read for (router, partition): parks with the writes in
- /// the per-partition FIFO when the partition is stashing (the shipped
- /// read-stashing design), otherwise forwards to the table entry.
- ClientStrongRead(RouterId, Partition),
+ /// A client write for (router, key), hashed with the router's adopted modulus.
+ ClientWrite(RouterId, Key),
+ /// A strong read for (router, key); parks with the writes while stashing.
+ ClientStrongRead(RouterId, Key),
// ── failures ───────────────────────────────────────────────
/// The pod process dies and instantly restarts under the same name
diff --git a/rust/personhog-stateright/tests/model_tests.rs b/rust/personhog-stateright/tests/model_tests.rs
index 1c4274a3ec1b..e907d317a84b 100644
--- a/rust/personhog-stateright/tests/model_tests.rs
+++ b/rust/personhog-stateright/tests/model_tests.rs
@@ -4,7 +4,7 @@
use std::time::Instant;
-use personhog_stateright::model::{ClaimDetection, HandoffModel, WarmOrder};
+use personhog_stateright::model::{ClaimDetection, HandoffModel, SealRule, WarmOrder, WarmReplay};
use stateright::{Checker, HasDiscoveries, Model};
/// Every checker explores in parallel: stateright defaults to a single
@@ -109,6 +109,17 @@ fn base() -> HandoffModel {
cancels: 0,
chunked_plans: false,
probes: false,
+ migrations: 0,
+ migrate_to: 1,
+ keys: 1,
+ seal_rule: SealRule::Assignment,
+ warm_replay: WarmReplay::Sources,
+ seal_breaks: 0,
+ seal_grace: true,
+ plan_basis_guard: true,
+ quiet_start: true,
+ freeze_checks_strays: true,
+ park_behind_key: true,
}
}
@@ -745,6 +756,184 @@ fn chunked_plan_application_is_safe_and_live() {
.assert_properties();
}
+/// A migration: one router, two pods, two keys, writes and a read racing every phase.
+fn migration(partitions: u8, migrate_to: u8) -> HandoffModel {
+ HandoffModel {
+ routers: 1,
+ partitions,
+ migrate_to,
+ migrations: 1,
+ keys: 2,
+ writes: 2,
+ reads: 1,
+ ..base()
+ }
+}
+
+/// A grow holds, seals, flips and restarts with every safety property intact;
+/// a cancel at any point before the seal unwinds in place.
+#[test]
+fn a_grow_migration_is_safe_and_live() {
+ migration(1, 2)
+ .explore("a_grow_migration_is_safe_and_live")
+ .assert_properties();
+}
+
+/// The rejected bucket rule: a request that parks only when its partition
+/// is stashing overtakes a request for its key parked under the old count.
+#[test]
+fn without_parking_behind_its_key_a_request_overtakes() {
+ let checker = HandoffModel {
+ park_behind_key: false,
+ ..migration(1, 2)
+ }
+ .explore_until(
+ "without_parking_behind_its_key_a_request_overtakes",
+ &["no_overtaken_request"],
+ );
+ assert!(
+ checker.discovery("no_overtaken_request").is_some(),
+ "the bucket rule must let a later request overtake its key's parked one"
+ );
+}
+
+/// A shrink dissolves the partition past the new space and still converges.
+#[test]
+fn a_shrink_migration_is_safe_and_live() {
+ migration(2, 1)
+ .explore("a_shrink_migration_is_safe_and_live")
+ .assert_properties();
+}
+
+/// Two routers adopt the flip independently, so one hashes with the old
+/// modulus while the other already drains under the new one.
+#[test]
+fn a_grow_migration_with_a_stale_router_is_safe_and_live() {
+ HandoffModel {
+ routers: 2,
+ writes: 1,
+ ..migration(1, 2)
+ }
+ .explore("a_grow_migration_with_a_stale_router_is_safe_and_live")
+ .assert_properties();
+}
+
+/// A doubling where each new partition inherits one old one and keys split.
+#[test]
+fn a_two_to_four_grow_is_safe_and_live() {
+ HandoffModel {
+ writes: 1,
+ keys: 4,
+ ..migration(2, 4)
+ }
+ .explore("a_two_to_four_grow_is_safe_and_live")
+ .assert_properties();
+}
+
+/// A holder that crash-restarts under its own name seals the partitions it
+/// never reloaded, so the saga still completes.
+#[test]
+fn a_holder_restarted_mid_sealing_still_seals_and_the_saga_completes() {
+ let checker = HandoffModel {
+ crashes: 1,
+ probes: true,
+ writes: 1,
+ ..migration(1, 2)
+ }
+ .explore("a_holder_restarted_mid_sealing_still_seals_and_the_saga_completes");
+ checker.assert_properties();
+ assert!(
+ checker.discovery("sealed_without_warm").is_some(),
+ "the restarted holder must seal a partition it never loaded"
+ );
+}
+
+/// The rejected seal rule: targets drawn from what a pod has loaded leave a
+/// restarted holder's partitions to nobody, and the saga never finishes.
+#[test]
+fn seal_targets_by_residency_wedge_a_restarted_holder() {
+ let checker = HandoffModel {
+ crashes: 1,
+ writes: 1,
+ seal_rule: SealRule::Residency,
+ seal_grace: false,
+ ..migration(1, 2)
+ }
+ .explore_until(
+ "seal_targets_by_residency_wedge_a_restarted_holder",
+ &["converges_to_stable"],
+ );
+ assert!(
+ checker.discovery("converges_to_stable").is_some(),
+ "residency-based seal targets must leave a run that never converges"
+ );
+}
+
+/// The rejected rule: a registered holder whose seal never succeeds holds Sealing forever.
+#[test]
+fn a_broken_seal_without_the_grace_wedges_sealing() {
+ let checker = HandoffModel {
+ writes: 1,
+ seal_breaks: 1,
+ seal_grace: false,
+ ..migration(1, 2)
+ }
+ .explore_until(
+ "a_broken_seal_without_the_grace_wedges_sealing",
+ &["converges_to_stable"],
+ );
+ assert!(
+ checker.discovery("converges_to_stable").is_some(),
+ "a broken seal with no grace must leave a run that never converges"
+ );
+}
+
+/// Past the grace, peers seal what a broken holder could not, and the saga completes safely.
+#[test]
+fn a_broken_seal_is_healed_by_peers_after_the_grace() {
+ let checker = HandoffModel {
+ writes: 1,
+ seal_breaks: 1,
+ ..migration(1, 2)
+ }
+ .explore("a_broken_seal_is_healed_by_peers_after_the_grace");
+ checker.assert_properties();
+}
+
+/// The rejected warm: a restart that reads the live partition alone misses
+/// the old tail and serves a stale strong read.
+#[test]
+fn a_live_only_warm_loses_the_old_tail() {
+ let checker = HandoffModel {
+ warm_replay: WarmReplay::LiveOnly,
+ ..migration(1, 2)
+ }
+ .explore_until(
+ "a_live_only_warm_loses_the_old_tail",
+ &["strong_reads_complete"],
+ );
+ assert!(
+ checker.discovery("strong_reads_complete").is_some(),
+ "a live-only warm must leave a stale strong read reachable"
+ );
+}
+
+/// A write for a partition born in the flip parks before its handoff exists
+/// and is delivered once it lands.
+#[test]
+fn probe_a_born_partition_write_parks_and_drains() {
+ let checker = HandoffModel {
+ probes: true,
+ ..migration(1, 2)
+ }
+ .explore("probe_a_born_partition_write_parks_and_drains");
+ checker.assert_properties();
+ assert!(
+ checker.discovery("born_partition_write_parked").is_some(),
+ "a write must be able to park for a partition born in the flip"
+ );
+}
+
/// Both fates of a pending unit are genuinely reachable — applied after
/// other actions interleaved, and dropped by a failed plan-time guard —
/// and no interleaving that reaches them loses an acked write or
@@ -777,3 +966,76 @@ fn probe_chunked_pending_units_reach_both_fates_safely() {
);
}
}
+
+/// A chunked bootstrap plan racing a migration: later units land only on
+/// the basis they were planned on, so every property holds.
+#[test]
+fn a_chunked_plan_across_a_migration_is_safe_and_live() {
+ HandoffModel {
+ chunked_plans: true,
+ writes: 1,
+ reads: 0,
+ ..migration(2, 4)
+ }
+ .explore("a_chunked_plan_across_a_migration_is_safe_and_live")
+ .assert_properties();
+}
+
+/// A planner that ignores the plan basis serves the old generation past the flip. The quiet
+/// start is off, since this small config reaches a mid-plan start only without it.
+#[test]
+fn an_unguarded_plan_across_a_migration_serves_the_old_generation() {
+ let checker = HandoffModel {
+ chunked_plans: true,
+ writes: 1,
+ reads: 0,
+ plan_basis_guard: false,
+ quiet_start: false,
+ ..migration(2, 4)
+ }
+ .explore_until(
+ "an_unguarded_plan_across_a_migration_serves_the_old_generation",
+ &[
+ "old_generation_quiet_after_flip",
+ "no_mixed_modulus_acceptance",
+ ],
+ );
+ assert!(checker
+ .discovery("old_generation_quiet_after_flip")
+ .is_some());
+ assert!(checker.discovery("no_mixed_modulus_acceptance").is_some());
+}
+
+/// The rejected design: a saga that begins mid-handoff, with a freeze that
+/// does not notice, freezes a fleet with a partition in motion and loses an acked write.
+#[test]
+fn starting_mid_handoff_loses_acked_writes() {
+ let checker = HandoffModel {
+ quiet_start: false,
+ freeze_checks_strays: false,
+ ..migration(1, 2)
+ }
+ .explore_until(
+ "starting_mid_handoff_loses_acked_writes",
+ &["no_lost_acked_write"],
+ );
+ assert!(checker.discovery("no_lost_acked_write").is_some());
+}
+
+/// The quiet start and the freeze's stray check are independent layers: each
+/// alone keeps every property, the stray check by cancelling instead.
+#[test]
+fn either_layer_alone_keeps_a_mid_handoff_start_safe() {
+ for (name, quiet_start, freeze_checks_strays) in [
+ ("only_the_stray_check", false, true),
+ ("only_the_quiet_start", true, false),
+ ] {
+ HandoffModel {
+ quiet_start,
+ freeze_checks_strays,
+ ..migration(1, 2)
+ }
+ .explore(name)
+ .assert_properties();
+ }
+}
diff --git a/rust/personhog-test-harness/Cargo.toml b/rust/personhog-test-harness/Cargo.toml
index e61abbd1fd78..d9ab8e5d50a4 100644
--- a/rust/personhog-test-harness/Cargo.toml
+++ b/rust/personhog-test-harness/Cargo.toml
@@ -8,6 +8,7 @@ publish = false
personhog-common = { path = "../personhog-common" }
personhog-proto = { path = "../personhog-proto" }
common-metrics = { path = "../common/metrics" }
+common-kafka = { path = "../common/kafka" }
personhog-coordination = { path = "../personhog-coordination" }
assignment-coordination = { path = "../common/assignment-coordination" }
diff --git a/rust/personhog-test-harness/src/cli.rs b/rust/personhog-test-harness/src/cli.rs
index 595c2f5df2ee..c42c832fbe1d 100644
--- a/rust/personhog-test-harness/src/cli.rs
+++ b/rust/personhog-test-harness/src/cli.rs
@@ -153,6 +153,10 @@ pub struct GateArgs {
#[arg(long = "leader-env", value_parser = parse_env_pair)]
pub leader_env: Vec<(String, String)>,
+ /// Extra KEY=VALUE environment for spawned routers. Repeatable.
+ #[arg(long = "router-env", value_parser = parse_env_pair)]
+ pub router_env: Vec<(String, String)>,
+
/// Target an already-running stack at this router URL instead of
/// spawning one. When unset, the harness spawns its own isolated stack
/// (replica, leaders, leader-mode router, writer) against the
@@ -303,6 +307,54 @@ pub struct GateArgs {
#[arg(long, default_value_t = false)]
pub kill_handoff_target: bool,
+ /// Migrate the changelog to a fresh topic this long into the traffic phase.
+ #[arg(long, value_parser = humantime::parse_duration)]
+ pub migrate_after: Option<Duration>,
+
+ /// Partition count of the migration's target topic.
+ #[arg(long)]
+ pub migrate_to_partitions: Option<u32>,
+
+ /// SIGKILL the coordinator router once the saga reaches Sealing. Requires --routers >= 3.
+ #[arg(long, default_value_t = false)]
+ pub migrate_kill_coordinator: bool,
+
+ /// SIGKILL the busiest leader once the saga reaches Resuming.
+ #[arg(long, default_value_t = false)]
+ pub migrate_kill_leader: bool,
+
+ /// SIGSTOP the busiest leader once it has drained, then SIGKILL it (lease revoked) at Sealing, so its peers seal its partitions.
+ #[arg(long, default_value_t = false)]
+ pub migrate_kill_leader_at_sealing: bool,
+
+ /// Crash-restart the busiest leader under its own name as soon as the saga is holding; it must seal what it never reloaded.
+ #[arg(long, default_value_t = false)]
+ pub migrate_restart_leader: bool,
+
+ /// SIGSTOP the busiest leader before the request, lease kept: its partitions never drain, so the saga must cancel at the 5s pre-flip deadline and unwind.
+ #[arg(long, default_value_t = false)]
+ pub migrate_stall_leader: bool,
+
+ /// SIGSTOP the busiest leader once it has drained, lease kept: its peers must seal for it after the seal grace and the saga complete. Pair with --migrate-slow-coordinator-ms so the drained holder is observable before Sealing.
+ #[arg(long, default_value_t = false)]
+ pub migrate_stall_leader_at_sealing: bool,
+
+ /// Debounce between the coordinator's migration passes, to widen the flip-to-restart window.
+ #[arg(long)]
+ pub migrate_slow_coordinator_ms: Option<u64>,
+
+ /// Run a second migration this long into the traffic phase.
+ #[arg(long, value_parser = humantime::parse_duration)]
+ pub migrate_again_after: Option<Duration>,
+
+ /// Partition count of the second migration's target topic.
+ #[arg(long)]
+ pub migrate_again_to_partitions: Option<u32>,
+
+ /// SIGSTOP the writer for --writer-pause-duration as the migration starts; the saga must not wait for it.
+ #[arg(long, default_value_t = false)]
+ pub migrate_pause_writer: bool,
+
/// Leader per-partition cache budget in bytes (CACHE_MEMORY_CAPACITY_BYTES).
/// Set below the seeded pool's footprint to put the cache under
/// eviction pressure. Default matches the dev deployment (16 MiB).
diff --git a/rust/personhog-test-harness/src/scenarios/blast.rs b/rust/personhog-test-harness/src/scenarios/blast.rs
index 59948db333fd..e596aece55bb 100644
--- a/rust/personhog-test-harness/src/scenarios/blast.rs
+++ b/rust/personhog-test-harness/src/scenarios/blast.rs
@@ -277,7 +277,7 @@ pub async fn verify_strong(
person_id,
key: "__strong_read_failed".to_string(),
expected: json!("readable"),
- actual: json!(e.to_string()),
+ actual: json!(format!("{e:#}")),
});
}
}
@@ -319,7 +319,7 @@ pub async fn verify_strong(
person_id,
key: "__strong_read_failed".to_string(),
expected: json!("readable"),
- actual: json!(e.to_string()),
+ actual: json!(format!("{e:#}")),
});
}
}
diff --git a/rust/personhog-test-harness/src/scenarios/gate.rs b/rust/personhog-test-harness/src/scenarios/gate.rs
index 982212d57c92..7a5a63f1abe3 100644
--- a/rust/personhog-test-harness/src/scenarios/gate.rs
+++ b/rust/personhog-test-harness/src/scenarios/gate.rs
@@ -8,6 +8,7 @@ use std::time::{Duration, Instant};
use anyhow::{bail, Context, Result};
use sqlx::postgres::PgPool;
+use personhog_coordination::types::MigrationPhase;
use personhog_proto::personhog::identity::v1::GetOrCreatePersonEntry;
use crate::cli::{GateArgs, DEFAULT_KEYS_PER_PERSON};
@@ -18,7 +19,7 @@ use crate::report::{print_report, ConsistencyViolation};
use crate::scenarios::merge::{self, MergeLane, WideRole};
use crate::scenarios::{blast, consistency};
use crate::seed;
-use crate::stack::{PublishConfig, Stack, StackConfig};
+use crate::stack::{MigrationChaos, PublishConfig, Stack, StackConfig, StallAt};
use crate::state::PersonState;
use crate::stats::StatsCollector;
use crate::verify::verify_postgres;
@@ -44,6 +45,8 @@ enum ChaosEvent {
ReleaseFences,
PublisherCrash,
IdentityCrash,
+ Migrate,
+ MigrateAgain,
}
impl fmt::Display for ChaosEvent {
@@ -68,6 +71,8 @@ impl fmt::Display for ChaosEvent {
ChaosEvent::RouterShutdown => write!(f, "coordinator router graceful shutdown"),
ChaosEvent::PublisherCrash => write!(f, "publisher crash-restart"),
ChaosEvent::IdentityCrash => write!(f, "identity crash-restart"),
+ ChaosEvent::Migrate => write!(f, "changelog migration"),
+ ChaosEvent::MigrateAgain => write!(f, "second changelog migration"),
}
}
}
@@ -113,6 +118,12 @@ fn chaos_timeline(args: &GateArgs) -> Vec<(Duration, ChaosEvent)> {
if let Some(after) = args.router_shutdown_after {
events.push((after, ChaosEvent::RouterShutdown));
}
+ if let Some(after) = args.migrate_after {
+ events.push((after, ChaosEvent::Migrate));
+ }
+ if let Some(after) = args.migrate_again_after {
+ events.push((after, ChaosEvent::MigrateAgain));
+ }
if let Some(after) = args.fence_after {
events.push((after, ChaosEvent::FencePersons));
events.push((
@@ -218,6 +229,29 @@ pub async fn run(args: GateArgs) -> Result<()> {
if args.fence_after.is_some() && args.fence_count == 0 {
bail!("--fence-count must be at least 1 with --fence-after");
}
+ if args.migrate_after.is_some() && args.migrate_to_partitions.is_none_or(|n| n == 0) {
+ bail!("--migrate-after needs --migrate-to-partitions >= 1");
+ }
+ if args.migrate_again_after.is_some() && args.migrate_again_to_partitions.is_none_or(|n| n == 0)
+ {
+ bail!("--migrate-again-after needs --migrate-again-to-partitions >= 1");
+ }
+ if args.migrate_kill_coordinator && args.routers < 3 {
+ bail!("--migrate-kill-coordinator requires --routers >= 3");
+ }
+ if args.migrate_kill_leader && args.migrate_kill_leader_at_sealing {
+ bail!("--migrate-kill-leader and --migrate-kill-leader-at-sealing are exclusive");
+ }
+ let mut router_env = args.router_env.clone();
+ if let Some(ms) = args.migrate_slow_coordinator_ms {
+ router_env.push(("COORDINATOR_MIGRATION_DEBOUNCE_MS".into(), ms.to_string()));
+ }
+ if args.migrate_stall_leader {
+ router_env.push((
+ "COORDINATOR_MIGRATION_PRE_FLIP_DEADLINE_SECS".into(),
+ "5".into(),
+ ));
+ }
if args.publish && args.external_router_url.is_some() {
bail!("--publish requires a spawned stack; it cannot target --external-router-url");
}
@@ -291,6 +325,7 @@ pub async fn run(args: GateArgs) -> Result<()> {
pg_target_table: args.pg_target_table.clone(),
cache_memory_capacity: args.cache_capacity,
extra_leader_env: args.leader_env.clone(),
+ extra_router_env: router_env,
recovery_pool_size: args.recovery_pool_size,
leader_lease_ttl: args.leader_lease_ttl,
spawn_identity: create_via_identity,
@@ -703,6 +738,52 @@ async fn run_gate(
stack.crash_restart_identity().await?;
None
}
+ ChaosEvent::Migrate | ChaosEvent::MigrateAgain => {
+ let chaos = match event {
+ ChaosEvent::Migrate => MigrationChaos {
+ to_partitions: args
+ .migrate_to_partitions
+ .expect("validated: --migrate-after needs --migrate-to-partitions"),
+ kill_coordinator: args.migrate_kill_coordinator,
+ kill_leader_at: if args.migrate_kill_leader {
+ Some(MigrationPhase::Resuming)
+ } else if args.migrate_kill_leader_at_sealing {
+ Some(MigrationPhase::Sealing)
+ } else {
+ None
+ },
+ restart_leader: args.migrate_restart_leader,
+ stall: if args.migrate_stall_leader {
+ Some(StallAt::BeforeStart)
+ } else if args.migrate_stall_leader_at_sealing {
+ Some(StallAt::Sealing)
+ } else {
+ None
+ },
+ pause_writer: args
+ .migrate_pause_writer
+ .then_some(args.writer_pause_duration),
+ },
+ // The second migration runs clean; the first carries the chaos.
+ _ => MigrationChaos {
+ to_partitions: args.migrate_again_to_partitions.expect(
+ "validated: --migrate-again-after needs --migrate-again-to-partitions",
+ ),
+ kill_coordinator: false,
+ kill_leader_at: None,
+ restart_leader: false,
+ stall: None,
+ pause_writer: None,
+ },
+ };
+ let took = stack.migrate(chaos).await?;
+ println!(
+ "Chaos at {:.1}s: migration ended in {:.1}s",
+ traffic_start.elapsed().as_secs_f64(),
+ took.as_secs_f64()
+ );
+ None
+ }
ChaosEvent::FencePersons | ChaosEvent::ReleaseFences => {
unreachable!("fence events are dispatched before the stack match")
}
@@ -731,6 +812,9 @@ async fn run_gate(
traffic.await.context("traffic task panicked")??;
let mut prober_violations = probers.await.context("prober task panicked")??;
+ if let Some(stack) = stack.as_mut() {
+ stack.retire_drained().await?;
+ }
let mut unresolved_merges = Vec::new();
if let Some(merges) = merges {
let result = merges.await.context("merge task panicked")??;
diff --git a/rust/personhog-test-harness/src/scenarios/stack_cmd.rs b/rust/personhog-test-harness/src/scenarios/stack_cmd.rs
index ad3f052ec78d..f69757036994 100644
--- a/rust/personhog-test-harness/src/scenarios/stack_cmd.rs
+++ b/rust/personhog-test-harness/src/scenarios/stack_cmd.rs
@@ -90,6 +90,7 @@ pub async fn run(args: StackArgs) -> Result<()> {
pg_target_table: args.pg_target_table.clone(),
cache_memory_capacity: 16_777_216,
extra_leader_env: Vec::new(),
+ extra_router_env: Vec::new(),
recovery_pool_size: 16,
leader_lease_ttl: 30,
spawn_identity: true,
diff --git a/rust/personhog-test-harness/src/stack/kafka.rs b/rust/personhog-test-harness/src/stack/kafka.rs
index 580db3b417b4..452a29775659 100644
--- a/rust/personhog-test-harness/src/stack/kafka.rs
+++ b/rust/personhog-test-harness/src/stack/kafka.rs
@@ -11,6 +11,51 @@ use rdkafka::message::Message;
use rdkafka::types::RDKafkaErrorCode;
use rdkafka::{Offset, TopicPartitionList};
+pub fn base_config(kafka_hosts: &str) -> ClientConfig {
+ let mut cfg = ClientConfig::new();
+ cfg.set("bootstrap.servers", kafka_hosts);
+ cfg
+}
+
+/// Whether `group` has committed on any partition of `topic`.
+pub async fn group_has_committed_offsets(
+ kafka_hosts: &str,
+ group: &str,
+ topic: &str,
+) -> Result<bool> {
+ let (base, group, topic) = (
+ base_config(kafka_hosts),
+ group.to_string(),
+ topic.to_string(),
+ );
+ tokio::task::spawn_blocking(move || -> Result<bool> {
+ let timeout = Duration::from_secs(5);
+ let consumer: BaseConsumer = base
+ .clone()
+ .set("group.id", &group)
+ .set("enable.auto.commit", "false")
+ .create()?;
+ let metadata = consumer.fetch_metadata(Some(&topic), timeout)?;
+ let partitions = metadata
+ .topics()
+ .iter()
+ .find(|t| t.name() == topic)
+ .map_or(0, |t| t.partitions().len());
+ let mut tpl = TopicPartitionList::new();
+ for partition in 0..partitions {
+ tpl.add_partition(&topic, partition as i32);
+ }
+ let committed = consumer.committed_offsets(tpl, timeout)?;
+ Ok(committed
+ .elements()
+ .iter()
+ .any(|e| matches!(e.offset(), Offset::Offset(_))))
+ })
+ .await
+ .context("offsets task panicked")?
+ .context("fetching committed offsets")
+}
+
fn admin_client(kafka_hosts: &str) -> Result<AdminClient<DefaultClientContext>> {
ClientConfig::new()
.set("bootstrap.servers", kafka_hosts)
diff --git a/rust/personhog-test-harness/src/stack/mod.rs b/rust/personhog-test-harness/src/stack/mod.rs
index 531bbfd6ecd1..4c27a0030a86 100644
--- a/rust/personhog-test-harness/src/stack/mod.rs
+++ b/rust/personhog-test-harness/src/stack/mod.rs
@@ -3,9 +3,11 @@ use std::fs;
use std::path::PathBuf;
use std::time::{Duration, Instant};
-use anyhow::{bail, Context, Result};
+use anyhow::{bail, ensure, Context, Result};
use personhog_coordination::store::PersonhogStore;
-use personhog_coordination::types::HandoffPhase;
+use personhog_coordination::types::{
+ ChangelogTarget, HandoffPhase, MigrationPhase, MigrationRequest,
+};
mod etcd;
pub(crate) mod kafka;
@@ -49,6 +51,10 @@ const PUBLISH_TOPIC_PARTITIONS: u32 = 4;
const ETCD_PREFIX: &str = "/personhog-test-harness/";
const READY_DEADLINE: Duration = Duration::from_secs(90);
+const MIGRATION_DEADLINE: Duration = Duration::from_secs(180);
+const MIGRATION_POLL: Duration = Duration::from_millis(100);
+const RETIREMENT_DEADLINE: Duration = Duration::from_secs(60);
+const WRITER_MOVE_DEADLINE: Duration = Duration::from_secs(10);
const SHUTDOWN_GRACE: Duration = Duration::from_secs(5);
pub struct StackConfig {
@@ -71,6 +77,7 @@ pub struct StackConfig {
/// Extra environment for spawned leaders, appended after the
/// standard set (so it can override any of it).
pub extra_leader_env: Vec<(String, String)>,
+ pub extra_router_env: Vec<(String, String)>,
pub recovery_pool_size: usize,
/// etcd lease TTL for leaders, in seconds. Bounds how long a crashed
/// (unrevoked) leader stays the registered owner.
@@ -133,6 +140,10 @@ pub struct Stack {
next_leader_index: u32,
store: PersonhogStore,
topic: String,
+ /// Topics a migration moved off; deleted with the live one at teardown.
+ retired_topics: Vec<String>,
+ /// Old generations awaiting the writer's drain and the retire op, with the saga's end.
+ pending_retire: Vec<(String, Instant)>,
pub router_url: String,
/// Set when the stack spawned a personhog-identity service: the traffic
/// router's URL, which proxies identity RPCs to it.
@@ -287,7 +298,7 @@ impl Stack {
}
let name = format!("harness-router-{i}");
let is_traffic_router = config.routers >= 2 && i == config.routers - 1;
- let proc = ServiceProcess::spawn(
+ let proc = ServiceProcess::spawn_with_extra(
&name,
&config.bin_dir.join("personhog-router"),
&[
@@ -316,6 +327,7 @@ impl Stack {
(ROUTER_METRICS_BASE_PORT + i as u16).to_string(),
),
],
+ &config.extra_router_env,
&log_dir,
)?;
routers.push((name, proc));
@@ -470,6 +482,8 @@ impl Stack {
next_leader_index: 0,
store,
topic,
+ retired_topics: Vec::new(),
+ pending_retire: Vec::new(),
router_url,
identity_url,
publish_topics,
@@ -611,6 +625,57 @@ impl Stack {
Ok(pod_name)
}
+ /// SIGSTOP the busiest leader with its lease intact: registered and
+ /// owning, but unable to drain, ack or serve until `resume_stalled`.
+ pub async fn stall_leader(&mut self) -> Result<String> {
+ let victim = self.busiest_leader().await?;
+ self.stall_named(&victim)
+ }
+
+ fn stall_named(&mut self, victim: &str) -> Result<String> {
+ let (pod_name, proc) = self.remove_leader(victim)?;
+ proc.sigstop();
+ tracing::info!(pod = %pod_name, "SIGSTOPped leader (stalled, lease kept)");
+ self.paused.push((pod_name.clone(), proc));
+ Ok(pod_name)
+ }
+
+ /// SIGKILL the stalled leader and revoke its lease.
+ async fn kill_stalled(&mut self) -> Result<String> {
+ let (pod_name, mut proc) = self.paused.pop().context("no stalled leader to kill")?;
+ proc.kill_now().await;
+ etcd::revoke_pod_lease(&self.store, &pod_name).await?;
+ tracing::info!(pod = %pod_name, "killed stalled leader");
+ self.retired.push(proc);
+ Ok(pod_name)
+ }
+
+ /// Whether every partition the saga holds on `pod` carries its drained ack.
+ async fn holder_drained(&self, pod: &str) -> Result<bool> {
+ for held in self.store.list_handoffs().await? {
+ if held.held_by.is_none() || held.new_owner != pod {
+ continue;
+ }
+ let acks = self.store.list_drained_acks(held.partition).await?;
+ if !acks
+ .iter()
+ .any(|a| a.pod_name == pod && a.handoff_id == held.handoff_id)
+ {
+ return Ok(false);
+ }
+ }
+ Ok(true)
+ }
+
+ /// SIGCONT the stalled leader; it resumes exactly where it was.
+ pub fn resume_stalled(&mut self) -> Result<String> {
+ let (pod_name, proc) = self.paused.pop().context("no stalled leader to resume")?;
+ proc.sigcont();
+ tracing::info!(pod = %pod_name, "SIGCONTed stalled leader");
+ self.leaders.push((pod_name.clone(), proc));
+ Ok(pod_name)
+ }
+
/// SIGCONT the paused zombie. It wakes believing it still owns its
/// partitions; the contract is that it detects the revoked lease,
/// self-fences locally, and rejoins with a fresh session — at which
@@ -996,7 +1061,8 @@ impl Stack {
// Every topic is attempted even when one delete fails, so a failed
// teardown leaves at most that one topic behind.
- let mut topics = vec![self.topic.as_str()];
+ let mut topics: Vec<&str> = self.retired_topics.iter().map(String::as_str).collect();
+ topics.push(self.topic.as_str());
if let Some(publish) = &self.publish_topics {
topics.push(publish.distinct_ids.as_str());
topics.push(publish.merge_events.as_str());
@@ -1012,6 +1078,335 @@ impl Stack {
None => Ok(()),
}
}
+
+ /// Run a changelog migration to its end, with chaos keyed to its phases.
+ /// Retirement of the old generation is deferred to `retire_drained`.
+ pub async fn migrate(&mut self, chaos: MigrationChaos) -> Result<Duration> {
+ // Admission refuses a migration while an earlier generation still drains.
+ if !self.pending_retire.is_empty() {
+ self.retire_drained().await?;
+ }
+ let from = self
+ .store
+ .get_changelog_config()
+ .await?
+ .context("config/changelog must be seeded")?;
+ let generation = self.retired_topics.len() + 2;
+ let to_topic = format!("{}_g{generation}", self.topic);
+ kafka::create_topic(&self.config.kafka_hosts, &to_topic, chaos.to_partitions).await?;
+ let migration = MigrationRequest {
+ id: format!("harness-g{generation}"),
+ from: from.target(),
+ to: ChangelogTarget {
+ topic: to_topic.clone(),
+ partitions: chaos.to_partitions,
+ },
+ requested_at_ms: now_millis(),
+ seen_at_ms: None,
+ waiting: Vec::new(),
+ };
+ // A leader stalled before the request keeps its lease, so the fleet still
+ // reads as quiet; the saga starts, holds its partitions and never drains them.
+ let mut stalled: Option<String> = None;
+ if chaos.stall == Some(StallAt::BeforeStart) {
+ let name = self.stall_leader().await?;
+ println!("Migration chaos: stalled leader {name} before the request");
+ stalled = Some(name);
+ }
+ ensure!(
+ self.store.create_migration_request(&migration).await?,
+ "a migration is already requested or in flight"
+ );
+ // The coordinator starts it once the fleet is quiet.
+ let requested = Instant::now();
+ // The request is read before the saga: the start creates one and
+ // deletes the other in a single transaction, so this order never sees neither.
+ loop {
+ let waiting = self
+ .store
+ .get_migration_request_with_mod_revision()
+ .await?
+ .is_some();
+ if self.store.get_migration().await?.is_some() {
+ break;
+ }
+ ensure!(
+ waiting,
+ "the request was rejected before starting: {:?}",
+ self.store
+ .list_migration_history()
+ .await?
+ .into_iter()
+ .find(|m| m.id == migration.id)
+ );
+ ensure!(
+ requested.elapsed() < MIGRATION_DEADLINE,
+ "the request did not start within {MIGRATION_DEADLINE:?}"
+ );
+ tokio::time::sleep(MIGRATION_POLL).await;
+ }
+ if chaos.pause_writer.is_some() {
+ self.pause_writer()?;
+ }
+ tracing::info!(
+ from = %from.topic,
+ to = %to_topic,
+ partitions = chaos.to_partitions,
+ waited_ms = requested.elapsed().as_millis() as u64,
+ "changelog migration started"
+ );
+
+ let start = Instant::now();
+ let mut last_phase = None;
+ let mut writer_paused = chaos.pause_writer;
+ let mut coordinator_killed = !chaos.kill_coordinator;
+ let mut leader_killed = chaos.kill_leader_at.is_none();
+ let mut leader_restarted = !chaos.restart_leader;
+ let mut sealed_while_stalled = chaos.stall != Some(StallAt::Sealing);
+ // A leader stopped at Sealing may already have sealed, which tests nothing, so the
+ // victim is stopped once it has drained and must hold no seal ack when Sealing begins.
+ let seal_victim_wanted = chaos.kill_leader_at == Some(MigrationPhase::Sealing)
+ || chaos.stall == Some(StallAt::Sealing);
+ let mut seal_victim: Option<String> = None;
+ let mut seal_victim_checked = false;
+ loop {
+ if let Some(pause) = writer_paused {
+ if start.elapsed() >= pause {
+ self.resume_writer()?;
+ writer_paused = None;
+ }
+ }
+ let Some(current) = self.store.get_migration().await? else {
+ break;
+ };
+ if last_phase != Some(current.phase) {
+ println!(
+ "Migration {:.1}s: {:?} {}",
+ start.elapsed().as_secs_f64(),
+ current.phase,
+ current.blocking.join("; ")
+ );
+ last_phase = Some(current.phase);
+ }
+ let holding = matches!(
+ current.phase,
+ MigrationPhase::Freezing | MigrationPhase::Draining
+ );
+ if seal_victim_wanted
+ && seal_victim.is_none()
+ && current.phase == MigrationPhase::Draining
+ {
+ let name = self.busiest_leader().await?;
+ if self.holder_drained(&name).await? {
+ self.stall_named(&name)?;
+ println!("Migration chaos: stalled leader {name} once drained");
+ seal_victim = Some(name);
+ }
+ }
+ if let Some(name) = seal_victim.clone().filter(|_| !seal_victim_checked) {
+ if current.phase == MigrationPhase::Sealing {
+ let sealed = self
+ .store
+ .list_seal_acks()
+ .await?
+ .into_iter()
+ .any(|a| a.migration_id == current.id && a.pod_name == name);
+ ensure!(!sealed, "leader {name} sealed before it stalled");
+ seal_victim_checked = true;
+ if chaos.kill_leader_at == Some(MigrationPhase::Sealing) {
+ self.kill_stalled().await?;
+ println!("Migration chaos: killed leader {name} at Sealing");
+ leader_killed = true;
+ } else {
+ stalled = Some(name);
+ }
+ }
+ }
+ if chaos.stall == Some(StallAt::Sealing) {
+ // Past Sealing, the seal grace let its peers seal for it; resume it so Resuming can restart it.
+ if stalled.is_some()
+ && matches!(
+ current.phase,
+ MigrationPhase::Resuming | MigrationPhase::Complete
+ )
+ {
+ let name = stalled.take().expect("checked above");
+ self.resume_stalled()?;
+ println!(
+ "Migration chaos: the saga flipped while {name} was stalled; resumed it"
+ );
+ sealed_while_stalled = true;
+ }
+ }
+ if !leader_restarted && holding {
+ let name = self.restart_leader().await?;
+ println!("Migration chaos: crash-restarted leader {name}");
+ leader_restarted = true;
+ }
+ if !coordinator_killed && current.phase == MigrationPhase::Sealing {
+ let name = self.kill_coordinator_router(true).await?;
+ println!("Migration chaos: killed coordinator router {name}");
+ coordinator_killed = true;
+ }
+ if !leader_killed
+ && chaos.kill_leader_at != Some(MigrationPhase::Sealing)
+ && Some(current.phase) == chaos.kill_leader_at
+ {
+ let name = self.kill_leader(true).await?;
+ println!("Migration chaos: killed leader {name}");
+ leader_killed = true;
+ }
+ if start.elapsed() > MIGRATION_DEADLINE {
+ bail!(
+ "migration did not finish within {MIGRATION_DEADLINE:?}: {current:?}; {}",
+ self.coordination_report().await
+ );
+ }
+ tokio::time::sleep(MIGRATION_POLL).await;
+ }
+ let saga = start.elapsed();
+ // A chaos step the harness never got to apply would let the gate pass vacuously.
+ ensure!(
+ coordinator_killed && leader_killed && leader_restarted && sealed_while_stalled,
+ "a requested chaos step never ran (coordinator killed {coordinator_killed}, \
+ leader killed {leader_killed}, leader restarted {leader_restarted}, \
+ sealed while stalled {sealed_while_stalled}); slow the coordinator so the phase is observable"
+ );
+ if let Some(name) = stalled {
+ self.resume_stalled()?;
+ println!("Migration chaos: resumed stalled leader {name}");
+ }
+ // The saga never waits for the writer, so the lag it injects outlives the saga.
+ if let Some(pause) = writer_paused {
+ tokio::time::sleep(pause.saturating_sub(start.elapsed())).await;
+ self.resume_writer()?;
+ println!(
+ "Migration chaos: resumed the writer {:.1}s after the saga started",
+ start.elapsed().as_secs_f64()
+ );
+ }
+ let archived = self
+ .store
+ .list_migration_history()
+ .await?
+ .into_iter()
+ .find(|m| m.id == migration.id)
+ .context("the migration record vanished without an archive")?;
+ let expected = if chaos.stall == Some(StallAt::BeforeStart) {
+ MigrationPhase::Cancelled
+ } else {
+ MigrationPhase::Complete
+ };
+ ensure!(
+ archived.phase == expected,
+ "migration ended in {:?}, expected {expected:?}: {}",
+ archived.phase,
+ archived.blocking.join("; ")
+ );
+ if expected == MigrationPhase::Cancelled {
+ kafka::delete_topic(&self.config.kafka_hosts, &to_topic).await?;
+ return Ok(saga);
+ }
+ self.config.partitions = chaos.to_partitions;
+ let old_topic = std::mem::replace(&mut self.topic, to_topic);
+ self.retired_topics.push(old_topic.clone());
+ self.pending_retire.push((old_topic, Instant::now()));
+ Ok(saga)
+ }
+
+ /// Wait for the writer to drain every old generation on its own, then retire each one.
+ pub async fn retire_drained(&mut self) -> Result<()> {
+ let pending = std::mem::take(&mut self.pending_retire);
+ for (old_topic, saga_end) in pending {
+ let drained_at = Instant::now();
+ loop {
+ let base = kafka::base_config(&self.config.kafka_hosts);
+ let topic = old_topic.clone();
+ let drained = tokio::task::spawn_blocking(move || {
+ common_kafka::probe::group_drained(
+ &base,
+ WRITER_CONSUMER_GROUP,
+ &topic,
+ Duration::from_secs(5),
+ )
+ })
+ .await??;
+ if drained {
+ break;
+ }
+ ensure!(
+ drained_at.elapsed() < RETIREMENT_DEADLINE,
+ "the writer did not drain {old_topic} within {RETIREMENT_DEADLINE:?}"
+ );
+ tokio::time::sleep(MIGRATION_POLL).await;
+ }
+ println!(
+ "Migration: {old_topic} drained by the writer; checked {:.1}s after the saga, waited {:.1}s",
+ saga_end.elapsed().as_secs_f64(),
+ drained_at.elapsed().as_secs_f64()
+ );
+ ensure!(
+ self.store.retire_changelog_topic(&old_topic).await?,
+ "{old_topic} was not listed as draining"
+ );
+ }
+ // The writer re-checks the drain every 2s and flushes every second.
+ let moved_at = Instant::now();
+ loop {
+ if kafka::group_has_committed_offsets(
+ &self.config.kafka_hosts,
+ WRITER_CONSUMER_GROUP,
+ &self.topic,
+ )
+ .await?
+ {
+ break;
+ }
+ ensure!(
+ moved_at.elapsed() < WRITER_MOVE_DEADLINE,
+ "the writer did not commit on {} within {WRITER_MOVE_DEADLINE:?}",
+ self.topic
+ );
+ tokio::time::sleep(MIGRATION_POLL).await;
+ }
+ println!(
+ "Migration: the writer committed on {} {:.1}s after the last drain",
+ self.topic,
+ moved_at.elapsed().as_secs_f64()
+ );
+ Ok(())
+ }
+}
+
+/// When migration chaos stalls a leader.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum StallAt {
+ /// Before the request: its partitions are held and never drain, so the
+ /// pre-flip deadline cancels the saga.
+ BeforeStart,
+ /// Once drained, before Sealing: its peers seal for it after the seal grace, and the saga completes.
+ Sealing,
+}
+
+pub struct MigrationChaos {
+ pub to_partitions: u32,
+ pub kill_coordinator: bool,
+ /// SIGKILL the busiest leader, lease revoked, when the saga reaches this phase; for Sealing it
+ /// is stalled once drained first.
+ pub kill_leader_at: Option<MigrationPhase>,
+ /// Crash-restart the busiest leader under its own name while the saga holds.
+ pub restart_leader: bool,
+ /// SIGSTOP the busiest leader with its lease kept.
+ pub stall: Option<StallAt>,
+ /// Pause the writer for this long as the migration starts.
+ pub pause_writer: Option<Duration>,
+}
+
+fn now_millis() -> i64 {
+ std::time::SystemTime::now()
+ .duration_since(std::time::UNIX_EPOCH)
+ .expect("clock before the epoch")
+ .as_millis() as i64
}
async fn wait_tcp(addr: &str, deadline: Duration) -> Result<()> {