Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(ingestion-control-plane): move the personhog changelog across clusters (#129707)

Zach Browndaf799f25cd3
diff --git a/rust/ingestion-control-plane/README.md b/rust/ingestion-control-plane/README.md
index ccc023e8370d..3aae5d113584 100644
--- a/rust/ingestion-control-plane/README.md
+++ b/rust/ingestion-control-plane/README.md
@@ -75,6 +75,8 @@ Both etcd tools are disabled (their APIs return 503) unless `ETCD_ENDPOINTS` is
| `PERSONHOG_KAFKA_HOSTS` | `KAFKA_HOSTS` | Cluster the personhog changelog lives on, for the changelog panel |
| `CHANGELOG_BACKUP_BUCKET` | empty | Versioned S3 bucket for `config/changelog` backups; empty disables backups and restore. Credentials and region come from the standard AWS environment |
| `PERSONHOG_KAFKA_TLS` | `KAFKA_TLS` | TLS for `PERSONHOG_KAFKA_HOSTS` |
+| `CHANGELOG_KAFKA_CLUSTERS` | empty | Clusters the changelog can live on, by name, as JSON `{"name": {"hosts": "...", "tls": bool}}`; seeding and a migration write the named endpoint into the changelog config |
+| `PERSONHOG_CROSS_CLUSTER_MIGRATIONS` | `false` | Allows a migration onto another cluster; enable only once every personhog leader, writer, relay and router runs a build that reads the cluster, because an older router's coordinator drops the cluster at the flip and an older leader, writer or relay uses its own `KAFKA_HOSTS` for the new topic |
| `PERSONHOG_HANDOFF_DEADLINE_SECS` | `120` | Per-phase deadline for flagging stuck handoffs; must match `COORDINATOR_HANDOFF_DEADLINE_SECS` |
| `PERSONHOG_WARMING_DEADLINE_SECS` | `1800` | Warming-phase deadline; must match `COORDINATOR_WARMING_DEADLINE_SECS` |
diff --git a/rust/ingestion-control-plane/src/api.rs b/rust/ingestion-control-plane/src/api.rs
index 95dddaa08429..867e26a6b50b 100644
--- a/rust/ingestion-control-plane/src/api.rs
+++ b/rust/ingestion-control-plane/src/api.rs
@@ -448,6 +448,8 @@ struct SeedChangelogRequest {
topic: String,
/// The writer's consumer group, as its `KAFKA_CONSUMER_GROUP` names it.
writer_group: String,
+ #[serde(default)]
+ cluster: Option<String>,
}
async fn personhog_seed_changelog(
@@ -465,12 +467,24 @@ async fn personhog_seed_changelog(
return Err(ApiError::bad_request("writer_group must not be empty"));
}
let client = etcd_client(&state).await?;
- let seeded = changelog::seed(&state.config, &client, &prefix, topic, writer_group)
- .await
- .map_err(|e| ApiError::bad_request(format!("seed failed: {e:#}")))?;
+ let cluster = trimmed(request.cluster.as_deref());
+ let seeded = changelog::seed(
+ &state.config,
+ &client,
+ &prefix,
+ topic,
+ writer_group,
+ cluster,
+ )
+ .await
+ .map_err(|e| ApiError::bad_request(format!("seed failed: {e:#}")))?;
Ok(Json(seeded))
}
+fn trimmed(value: Option<&str>) -> Option<&str> {
+ value.map(str::trim).filter(|v| !v.is_empty())
+}
+
fn backup_store(state: &AppState) -> Result<Arc<dyn crate::backup::BackupStore>, ApiError> {
state.backups.clone().ok_or_else(|| {
ApiError::unavailable("changelog backups are disabled (CHANGELOG_BACKUP_BUCKET not set)")
@@ -526,6 +540,8 @@ async fn personhog_restore_changelog(
#[derive(Deserialize)]
struct StartMigrationRequest {
topic: String,
+ #[serde(default)]
+ cluster: Option<String>,
}
async fn personhog_start_migration(
@@ -539,7 +555,8 @@ async fn personhog_start_migration(
return Err(ApiError::bad_request("topic must not be empty"));
}
let client = etcd_client(&state).await?;
- let migration = changelog::start_migration(&state.config, &client, &prefix, topic)
+ let cluster = trimmed(request.cluster.as_deref());
+ let migration = changelog::start_migration(&state.config, &client, &prefix, topic, cluster)
.await
.map_err(|e| ApiError::bad_request(format!("migration refused: {e:#}")))?;
Ok(Json(migration))
diff --git a/rust/ingestion-control-plane/src/changelog.rs b/rust/ingestion-control-plane/src/changelog.rs
index 1534ac4fe646..aa42b0884905 100644
--- a/rust/ingestion-control-plane/src/changelog.rs
+++ b/rust/ingestion-control-plane/src/changelog.rs
@@ -5,7 +5,8 @@ use assignment_coordination::store::{EtcdStore, StoreConfig};
use etcd_client::Client;
use personhog_coordination::store::PersonhogStore;
use personhog_coordination::types::{
- ChangelogConfig, ChangelogTarget, MigrationPhase, MigrationRequest, MigrationState,
+ ChangelogConfig, ChangelogTarget, ClusterEndpoint, MigrationPhase, MigrationRequest,
+ MigrationState,
};
use rdkafka::consumer::Consumer;
use serde::Serialize;
@@ -58,21 +59,41 @@ pub async fn seed(
prefix: &str,
topic: &str,
writer_group: &str,
+ cluster: Option<&str>,
) -> anyhow::Result<ChangelogConfig> {
- let config = &config.personhog_kafka();
- let partitions = topic_partition_count(config, topic).await?;
+ let cluster = match cluster {
+ Some(name) => Some(config.changelog_cluster(name)?),
+ None => config.current_changelog_cluster()?,
+ };
+ let partitions =
+ topic_partition_count(&config.changelog_kafka(cluster.as_ref()), topic).await?;
let created = store(client, prefix)
- .seed_changelog_config(topic, partitions, writer_group)
+ .seed_changelog_config_on(topic, partitions, writer_group, cluster.clone())
.await?;
if !created {
bail!("config/changelog is already seeded; only a migration may move it");
}
Ok(ChangelogConfig {
writer_group: writer_group.to_string(),
+ cluster,
..ChangelogConfig::new(topic, partitions)
})
}
+pub async fn record_cluster(
+ client: &Client,
+ prefix: &str,
+ cluster: &ClusterEndpoint,
+) -> anyhow::Result<bool> {
+ let store = store(client, prefix);
+ let Some((live, mod_revision)) = store.get_changelog_config_with_mod_revision().await? else {
+ return Ok(false);
+ };
+ Ok(store
+ .set_changelog_cluster(&live, mod_revision, cluster)
+ .await?)
+}
+
pub async fn backups(
backups: &dyn BackupStore,
prefix: &str,
@@ -111,7 +132,11 @@ pub async fn restore(
backup::unwrap_config(&object.body).map_err(|e| RestoreError::Refused(format!("{e:#}")))?;
let restored: ChangelogConfig = serde_json::from_slice(&value)
.map_err(|e| RestoreError::Refused(format!("the backup is not a changelog config: {e}")))?;
- let live_partitions = topic_partition_count(&config.personhog_kafka(), &restored.topic).await?;
+ let live_partitions = topic_partition_count(
+ &config.changelog_kafka(restored.cluster.as_ref()),
+ &restored.topic,
+ )
+ .await?;
apply_restore(&store(client, prefix), &value, restored, live_partitions).await
}
@@ -249,9 +274,10 @@ pub async fn retire(
.get_changelog_config_with_mod_revision()
.await?
.context("config/changelog is not seeded")?;
- if !live.draining.iter().any(|g| g.topic == topic) {
+ let Some(generation) = live.draining.iter().find(|g| g.topic == topic) else {
bail!("{topic} is not a draining generation");
- }
+ };
+ let config = &config.changelog_kafka(generation.cluster.as_ref());
if let Some(migration) = store.get_migration().await? {
bail!(
"migration {} is {:?}; retire once it completes",
@@ -293,8 +319,8 @@ pub async fn start_migration(
client: &Client,
prefix: &str,
topic: &str,
+ cluster: Option<&str>,
) -> anyhow::Result<MigrationRequest> {
- let config = &config.personhog_kafka();
let store = store(client, prefix);
let live = store
.get_changelog_config()
@@ -303,13 +329,22 @@ pub async fn start_migration(
if live.topic == topic {
bail!("{topic} is already the live changelog");
}
- let partitions = topic_partition_count(config, topic).await?;
+ let requested = cluster
+ .map(|name| config.changelog_cluster(name))
+ .transpose()?;
+ let cluster = target_cluster(
+ live.cluster.as_ref(),
+ requested,
+ config.personhog_cross_cluster_migrations,
+ )?;
+ let partitions =
+ topic_partition_count(&config.changelog_kafka(cluster.as_ref()), topic).await?;
let now_ms = assignment_coordination::util::now_millis();
let request = MigrationRequest {
id: format!("m-{now_ms}"),
from: live.target(),
to: ChangelogTarget {
- cluster: live.cluster.clone(),
+ cluster,
topic: topic.to_string(),
partitions,
},
@@ -323,6 +358,27 @@ pub async fn start_migration(
Ok(request)
}
+/// Another cluster needs the switch, because an older router drops the cluster at the flip
+/// and an older leader, writer or relay ignores it.
+fn target_cluster(
+ live: Option<&ClusterEndpoint>,
+ requested: Option<ClusterEndpoint>,
+ cross_cluster_allowed: bool,
+) -> anyhow::Result<Option<ClusterEndpoint>> {
+ let Some(requested) = requested else {
+ return Ok(live.cloned());
+ };
+ let Some(live) = live else {
+ bail!("the live changelog records no cluster; list the cluster PERSONHOG_KAFKA_HOSTS names in CHANGELOG_KAFKA_CLUSTERS and restart the control plane");
+ };
+ if requested != *live && !cross_cluster_allowed {
+ bail!(
+ "moving the changelog to another cluster is off; enable PERSONHOG_CROSS_CLUSTER_MIGRATIONS once every personhog leader, writer, relay and router reads the cluster"
+ );
+ }
+ Ok(Some(requested))
+}
+
const WITHDRAW_ATTEMPTS: usize = 5;
/// Withdraw a waiting request, or cancel a running saga before Sealing.
@@ -389,7 +445,6 @@ pub async fn cancel_migration(
}
pub async fn view(config: &Config, client: &Client, prefix: &str) -> anyhow::Result<ChangelogView> {
- let config = &config.personhog_kafka();
let store = store(client, prefix);
let changelog = store.get_changelog_config().await?;
let migration = store.get_migration().await?;
@@ -402,40 +457,40 @@ pub async fn view(config: &Config, client: &Client, prefix: &str) -> anyhow::Res
let mut draining = Vec::new();
if let Some(changelog) = &changelog {
let reports = store.list_drain_reports().await?;
- let drain = |group: &str, topic: &str| {
+ let drain = |kafka: &Config, group: &str, topic: &str| {
let reported = reports.iter().any(|r| r.group == group && r.topic == topic);
- let (group, topic) = (group.to_string(), topic.to_string());
+ let (kafka, group, topic) = (kafka.clone(), group.to_string(), topic.to_string());
async move {
GroupDrain {
reported,
- ..group_drain(config, &group, &topic).await
+ ..group_drain(&kafka, &group, &topic).await
}
}
};
for generation in &changelog.draining {
+ let kafka = config.changelog_kafka(generation.cluster.as_ref());
draining.push(DrainingView {
topic: generation.topic.clone(),
partitions: generation.partitions,
epoch: generation.epoch,
- writer: drain(&changelog.writer_group, &generation.topic).await,
+ writer: drain(&kafka, &changelog.writer_group, &generation.topic).await,
retire_groups: {
let mut drains = Vec::new();
for group in &changelog.retire_groups {
- drains.push(drain(group, &generation.topic).await);
+ drains.push(drain(&kafka, group, &generation.topic).await);
}
drains
},
});
}
}
- let writer = match &changelog {
- Some(changelog) => Some(group_lag(config, &changelog.writer_group, &changelog.topic).await),
- None => None,
- };
+ let mut writer = None;
let mut retire_groups = Vec::new();
if let Some(changelog) = &changelog {
+ let kafka = config.changelog_kafka(changelog.cluster.as_ref());
+ writer = Some(group_lag(&kafka, &changelog.writer_group, &changelog.topic).await);
for group in &changelog.retire_groups {
- retire_groups.push(group_lag(config, group, &changelog.topic).await);
+ retire_groups.push(group_lag(&kafka, group, &changelog.topic).await);
}
}
Ok(ChangelogView {
@@ -479,6 +534,27 @@ mod tests {
use super::*;
use crate::backup::tests::test_etcd;
+ #[test]
+ fn a_migration_keeps_the_live_cluster_unless_another_is_allowed() {
+ let endpoint = |name: &str| ClusterEndpoint {
+ name: name.to_string(),
+ hosts: format!("{name}:9092"),
+ tls: false,
+ };
+ let live = endpoint("ingestion");
+ assert_eq!(
+ target_cluster(Some(&live), None, false).unwrap(),
+ Some(live.clone())
+ );
+ assert_eq!(target_cluster(None, None, false).unwrap(), None);
+ assert!(target_cluster(Some(&live), Some(endpoint("dedicated")), false).is_err());
+ assert_eq!(
+ target_cluster(Some(&live), Some(endpoint("dedicated")), true).unwrap(),
+ Some(endpoint("dedicated"))
+ );
+ assert!(target_cluster(None, Some(endpoint("dedicated")), true).is_err());
+ }
+
async fn empty_store(name: &str) -> PersonhogStore {
let (etcd, prefix) = test_etcd(name).await;
store(&etcd.client().await.unwrap(), &prefix)
diff --git a/rust/ingestion-control-plane/src/config.rs b/rust/ingestion-control-plane/src/config.rs
index 2ce5e7da4438..cb4ecee28409 100644
--- a/rust/ingestion-control-plane/src/config.rs
+++ b/rust/ingestion-control-plane/src/config.rs
@@ -1,6 +1,10 @@
+use std::collections::HashMap;
use std::str::FromStr;
+use anyhow::Context;
use envconfig::Envconfig;
+use personhog_coordination::types::ClusterEndpoint;
+use serde::Deserialize;
/// How consumer pods are discovered, mirroring the ingestion-consumer's
/// typed `DiscoveryMode`.
@@ -178,6 +182,15 @@ pub struct Config {
#[envconfig(from = "PERSONHOG_KAFKA_TLS")]
pub personhog_kafka_tls: Option<bool>,
+ /// Clusters the changelog can live on, as JSON `{"name": {"hosts": "...", "tls": bool}}`.
+ #[envconfig(from = "CHANGELOG_KAFKA_CLUSTERS", default = "")]
+ pub changelog_kafka_clusters: String,
+
+ /// Allows migrations onto another cluster. An older router's coordinator drops the
+ /// cluster at the flip, and an older leader, writer or relay ignores it.
+ #[envconfig(from = "PERSONHOG_CROSS_CLUSTER_MIGRATIONS", default = "false")]
+ pub personhog_cross_cluster_migrations: bool,
+
/// Per-phase handoff deadline used to flag stuck handoffs; must match
/// the coordinator's `COORDINATOR_HANDOFF_DEADLINE_SECS` to agree with
/// its cancellation decisions.
@@ -189,6 +202,25 @@ pub struct Config {
pub personhog_warming_deadline_secs: u64,
}
+/// One cluster in `CHANGELOG_KAFKA_CLUSTERS`; `tls` is required and unknown keys are
+/// refused, so neither an omission nor a typo can drop TLS.
+#[derive(Deserialize)]
+#[serde(deny_unknown_fields)]
+struct ClusterSpec {
+ hosts: String,
+ tls: bool,
+}
+
+impl ClusterSpec {
+ fn endpoint(self, name: String) -> ClusterEndpoint {
+ ClusterEndpoint {
+ name,
+ hosts: self.hosts,
+ tls: self.tls,
+ }
+ }
+}
+
impl Config {
pub fn init_with_defaults() -> Result<Self, envconfig::Error> {
Self::init_from_env()
@@ -210,6 +242,40 @@ impl Config {
config
}
+ fn changelog_clusters(&self) -> anyhow::Result<HashMap<String, ClusterSpec>> {
+ match self.changelog_kafka_clusters.trim() {
+ "" => Ok(HashMap::new()),
+ spec => serde_json::from_str(spec)
+ .context("CHANGELOG_KAFKA_CLUSTERS is not a valid cluster map"),
+ }
+ }
+
+ pub fn changelog_cluster(&self, name: &str) -> anyhow::Result<ClusterEndpoint> {
+ let spec = self
+ .changelog_clusters()?
+ .remove(name)
+ .with_context(|| format!("cluster {name:?} is not in CHANGELOG_KAFKA_CLUSTERS"))?;
+ Ok(spec.endpoint(name.to_string()))
+ }
+
+ pub fn current_changelog_cluster(&self) -> anyhow::Result<Option<ClusterEndpoint>> {
+ let current = self.personhog_kafka();
+ Ok(self
+ .changelog_clusters()?
+ .into_iter()
+ .find(|(_, spec)| spec.hosts == current.kafka_hosts && spec.tls == current.kafka_tls)
+ .map(|(name, spec)| spec.endpoint(name)))
+ }
+
+ pub fn changelog_kafka(&self, cluster: Option<&ClusterEndpoint>) -> Config {
+ let mut config = self.personhog_kafka();
+ if let Some(cluster) = cluster {
+ config.kafka_hosts = cluster.hosts.clone();
+ config.kafka_tls = cluster.tls;
+ }
+ config
+ }
+
pub fn etcd_endpoint_list(&self) -> Vec<String> {
self.etcd_endpoints
.split(',')
@@ -292,6 +358,40 @@ mod tests {
(set.kafka_hosts.as_str(), set.kafka_tls),
("personhog:9094", true)
);
+
+ config.changelog_kafka_clusters =
+ r#"{"dedicated": {"hosts": "d:9092", "tls": false}}"#.to_string();
+ let dedicated = config.changelog_cluster("dedicated").unwrap();
+ let moved = config.changelog_kafka(Some(&dedicated));
+ assert_eq!(
+ (moved.kafka_hosts.as_str(), moved.kafka_tls),
+ ("d:9092", false)
+ );
+ assert!(config.changelog_cluster("unknown").is_err());
+ config.changelog_kafka_clusters =
+ r#"{"dedicated": {"hosts": "d:9092", "tsl": true}}"#.to_string();
+ assert!(config.changelog_cluster("dedicated").is_err());
+ config.changelog_kafka_clusters = r#"{"dedicated": {"hosts": "d:9092"}}"#.to_string();
+ assert!(config.changelog_cluster("dedicated").is_err());
+ }
+
+ #[test]
+ fn the_current_cluster_is_the_entry_matching_the_personhog_hosts() {
+ let mut config = Config::init_with_defaults().expect("config defaults are valid");
+ config.personhog_kafka_hosts = Some("ingestion:9092".to_string());
+ config.personhog_kafka_tls = Some(true);
+ config.changelog_kafka_clusters = r#"{
+ "ingestion": {"hosts": "ingestion:9092", "tls": true},
+ "dedicated": {"hosts": "dedicated:9092", "tls": true}
+ }"#
+ .to_string();
+ assert_eq!(
+ config.current_changelog_cluster().unwrap(),
+ Some(config.changelog_cluster("ingestion").unwrap())
+ );
+
+ config.personhog_kafka_tls = Some(false);
+ assert_eq!(config.current_changelog_cluster().unwrap(), None);
}
#[test]
diff --git a/rust/ingestion-control-plane/src/main.rs b/rust/ingestion-control-plane/src/main.rs
index 437a4567c455..845ffefb880c 100644
--- a/rust/ingestion-control-plane/src/main.rs
+++ b/rust/ingestion-control-plane/src/main.rs
@@ -7,6 +7,7 @@ use common_metrics::setup_metrics_routes;
use health::{readiness_handler, HealthRegistry};
use ingestion_control_plane::api;
use ingestion_control_plane::backup::{self, BackupStore, S3BackupStore};
+use ingestion_control_plane::changelog;
use ingestion_control_plane::config::Config;
use ingestion_control_plane::state::AppState;
use tokio::signal;
@@ -67,6 +68,7 @@ async fn main() {
let mut state = AppState::new(config).expect("failed to initialize app state");
start_changelog_backups(&mut state).await;
+ record_changelog_cluster(&state);
let router = api::router(state)
.route("/_readiness", get(readiness_handler))
@@ -87,6 +89,37 @@ async fn main() {
.expect("server failed");
}
+fn record_changelog_cluster(state: &AppState) {
+ let Some(etcd) = state.etcd.clone() else {
+ return;
+ };
+ let cluster = match state.config.current_changelog_cluster() {
+ Ok(Some(cluster)) => cluster,
+ Ok(None) => {
+ info!("no CHANGELOG_KAFKA_CLUSTERS entry matches PERSONHOG_KAFKA_HOSTS; the changelog's cluster stays unrecorded");
+ return;
+ }
+ Err(e) => {
+ warn!(error = %format!("{e:#}"), "could not read CHANGELOG_KAFKA_CLUSTERS; the changelog's cluster stays unrecorded");
+ return;
+ }
+ };
+ let prefix = state.config.personhog_etcd_prefix.clone();
+ tokio::spawn(async move {
+ let recorded = async {
+ let client = etcd.client().await?;
+ changelog::record_cluster(&client, &prefix, &cluster).await
+ };
+ match recorded.await {
+ Ok(true) => info!(cluster = %cluster.name, "recorded the live changelog's cluster"),
+ Ok(false) => info!("the live changelog's cluster was not recorded: it is unseeded, already recorded, or a migration is requested"),
+ Err(e) => {
+ warn!(error = %format!("{e:#}"), "could not record the live changelog's cluster; the next start retries")
+ }
+ }
+ });
+}
+
async fn start_changelog_backups(state: &mut AppState) {
let bucket = state.config.changelog_backup_bucket.clone();
if bucket.is_empty() {
diff --git a/rust/personhog-coordination/README.md b/rust/personhog-coordination/README.md
index fcb7dc4dd7a1..a7e5895d4cde 100644
--- a/rust/personhog-coordination/README.md
+++ b/rust/personhog-coordination/README.md
@@ -131,6 +131,8 @@ A changelog resize is a new topic and a cutover, never in-place growth. `config/
The control plane only writes a request, `migrations/requested`. The coordinator starts the saga from it only while the fleet is quiet: no handoff record, and every old partition assigned to a registered, Ready pod. It waits, holding nothing, until then, and creates `migrations/current` in a transaction that requires no handoff record to exist at commit. Creating the saga record moves the planner's plan basis, so no handoff can begin after the start either. The owner half of quiet is checked when the request is evaluated, so an owner that begins shutting down in between is held like any other and reaches the freeze's `holder_died` cancel if it deregisters. The freeze also cancels the saga if it ever finds an ordinary handoff, before anything that handoff touches could be stranded; the quiet start is what keeps that from happening, so a start does not end in a cancel. Admission (the live config, the target topic, no generation still draining, no rollout) runs at the same step, and a request still waiting five minutes after the coordinator first saw it is rejected, with what it was waiting on, so a saga only starts while whoever asked is plausibly watching.
+A migration can also move the changelog to another Kafka cluster. The target, the live config and each draining generation carry an optional `cluster` endpoint (a name, bootstrap servers and TLS); unset is the process's own `KAFKA_HOSTS`. The control plane writes the endpoint from its own name map, so the personhog services need no cluster config. Topic names stay unique across clusters, so drain reports, dirty-index streams, transactional ids and history keep keying on the topic alone. The leader connects per generation: it seals the old generation's ids on the old cluster, produces on the new one, and warms, recovers and prunes each source on its own cluster. A writer or relay that has drained a generation whose successor lives on another cluster does not re-subscribe: it shuts down cleanly, and its restart subscribes on the new cluster, with nothing left to commit on the old one. The coordinator never reads Kafka, so it only carries the field. The control plane refuses a cross-cluster request until `PERSONHOG_CROSS_CLUSTER_MIGRATIONS` is on, because a router that predates the field drops it at the flip, and a leader, writer or relay that predates it uses its own `KAFKA_HOSTS` for the new topic.
+
Freezing holds every partition in place (a held self-handoff that never passes Draining, so routers stash and the holder keeps its cache). Draining waits on the holders. Sealing has every live leader claim and drop every lane id on the old topic for every partition the assignment table does not give to a registered peer, which covers its own partitions whether or not it has loaded them and every partition without a registered owner, so no stale producer survives; it is the point of no return. Two seconds into Sealing, every leader also seals whatever no ack reports sealed yet, so a registered owner that cannot seal does not hold the fleet; every hold is drained by then, so nothing still produces to those partitions. A leader acks each seal as it lands, so the grace measures progress rather than a whole pass, and its report keeps every partition it has sealed for the life of the process; a restarted leader reseals once.
The ordinary planner places only when no saga holds placement: no migration, or a Resuming one whose holds are all restarted. Every coordinator plan carries the record it was decided on, and its transactions require that record unchanged, so a stalled pass that wakes after a successor acted lands nothing. The planner's record is its plan basis: the changelog config and the migration record, read in one transaction and compared by the config's mod revision (a flip or retire moves it) and the migration's create revision (a new or archived saga moves it; the driver's own writes do not). A migration pass's record is the saga at the mod revision it read, which any write to it moves: a cancel, a phase change, a blocking-list write, the archive. A saga cancelled before its flip and archived leaves the planner's basis as it was, and a plan landing after it meets the world it was planned for.