Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(personhog): serve partitions across changelog generations (#119811)

Zach Brown4e6e703a7bb3
diff --git a/rust/personhog-coordination/README.md b/rust/personhog-coordination/README.md
index 20dd2ff94351..4ab0cbdd1e7e 100644
--- a/rust/personhog-coordination/README.md
+++ b/rust/personhog-coordination/README.md
@@ -123,11 +123,13 @@ Several failure modes are handled explicitly:
**Mid-stream watch reconnects don't replay missed events — bounded by reconciliation**: The snapshot-to-watch-attach gap is closed (watches are revision-anchored to the snapshot that seeded them), but if an *established* etcd watch stream drops and the underlying client reconnects, events that fired during the disconnect are not replayed. Every participant now bounds the resulting staleness with its level-triggered truth path — the coordinator's tick (which also wakes the planner unconditionally), the router's reconcile pass, and the pod's converge-all tick — so a missed event costs at most one reconcile interval, not correctness.
-## Changelog config
+## Changelog generations
-`config/changelog` (`{topic, partitions, epoch, writer_group, draining}`) is the one live object every participant reads. Every service loads it at boot; the router and the identity service also follow it through a watch, so the partition count they hash under is the live one.
+`config/changelog` (`{topic, partitions, epoch, writer_group, draining}`) is the one live object every participant reads. A pod captures it by a strong read when it acquires a partition and serves the partition under that capture until it releases it.
-Routers park requests per key, so a key's requests keep one queue when the partition count changes: a request goes straight to a leader only when its key has nothing parked or in flight and its partition is not stashing, and the live count decides only which partition's drain takes a key. A drain takes the keys that map to its partition under the count the router has adopted.
+A leader serving a partition holds in its cache every acked write for that partition's keys that Postgres does not have yet, so a warm reads every source that can hold one: for each draining generation, the old partitions congruent to the new one modulo `gcd(old, new)`, from the writer's committed offset to their end and filtered to the keys that hash to the new partition, then the live partition itself. Marks from an old source keep their own stream (`topic`, source partition), so recovery fetches them from the old topic and the prune settles them against the writer's offsets there. A mark also names the cache partition it belongs to, and a release or prune of one partition only removes the marks and lifecycle fences homed there: an old partition and a new one on the same pod share keys, and either may be released or pruned first.
+
+Routers park requests per key, so a key's requests keep one queue when the partition count changes: a request goes straight to a leader only when its key has nothing parked or in flight and its partition is not stashing, and the live count decides only which partition's drain takes a key. A drain takes the keys that map to its partition under the count the router has adopted, and the leader bounces any key it took under an older one. A leader answers a stale `x-partition` with `FAILED_PRECONDITION`, which the router treats as a bounce, so the write lands once the router's config watch delivers the new count; a router that has not adopted within its retries answers `UNAVAILABLE`, never an ack; a lifecycle batch with a member off the routed partition is refused with `INVALID_ARGUMENT`, which the identity service answers by retrying each member alone.
## Modules
diff --git a/rust/personhog-coordination/src/changelog.rs b/rust/personhog-coordination/src/changelog.rs
index da0a1cdf0dad..7d73ae115a0b 100644
--- a/rust/personhog-coordination/src/changelog.rs
+++ b/rust/personhog-coordination/src/changelog.rs
@@ -122,6 +122,63 @@ pub async fn watch_changelog_config_loop<F>(
}
}
+/// Keys a source contributes: those that hash to `target` under `modulus`.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct KeyFilter {
+ pub modulus: u32,
+ pub target: u32,
+}
+
+/// One changelog partition a warm replays from the writer's committed offset.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct WarmSource {
+ pub topic: Arc<str>,
+ pub partition: u32,
+ pub keep: Option<KeyFilter>,
+}
+
+impl WarmSource {
+ pub fn live(topic: &str, partition: u32) -> Self {
+ Self {
+ topic: Arc::from(topic),
+ partition,
+ keep: None,
+ }
+ }
+}
+
+fn gcd(a: u32, b: u32) -> u32 {
+ if b == 0 {
+ a
+ } else {
+ gcd(b, a % b)
+ }
+}
+
+/// What a warm of `partition` replays: each draining generation's partitions
+/// that can hold its keys (a key's old and new partitions agree mod the gcd),
+/// oldest first, then the live partition.
+pub fn warm_sources(config: &ChangelogConfig, partition: u32) -> Vec<WarmSource> {
+ let mut draining = config.draining.clone();
+ draining.sort_by_key(|g| g.epoch);
+ let mut sources = Vec::new();
+ for generation in &draining {
+ let g = gcd(generation.partitions, config.partitions);
+ for q in (0..generation.partitions).filter(|q| q % g == partition % g) {
+ sources.push(WarmSource {
+ topic: Arc::from(generation.topic.as_str()),
+ partition: q,
+ keep: Some(KeyFilter {
+ modulus: config.partitions,
+ target: partition,
+ }),
+ });
+ }
+ }
+ sources.push(WarmSource::live(&config.topic, partition));
+ sources
+}
+
/// Bounds the staleness a dropped watch stream can hide.
pub const CHANGELOG_RECONCILE_INTERVAL: Duration = Duration::from_secs(30);
pub const CHANGELOG_BOOT_POLL_INTERVAL: Duration = Duration::from_secs(1);
@@ -219,3 +276,70 @@ where
);
on_config(config)
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::types::Generation;
+
+ fn config(partitions: u32, draining: Vec<(u32, u64)>) -> ChangelogConfig {
+ ChangelogConfig {
+ topic: "new".to_string(),
+ partitions,
+ epoch: 9,
+ writer_group: String::new(),
+ draining: draining
+ .into_iter()
+ .map(|(partitions, epoch)| Generation {
+ topic: format!("old{epoch}"),
+ partitions,
+ epoch,
+ })
+ .collect(),
+ }
+ }
+
+ fn old_partitions(
+ partitions: u32,
+ draining: Vec<(u32, u64)>,
+ partition: u32,
+ ) -> Vec<(String, u32)> {
+ warm_sources(&config(partitions, draining), partition)
+ .into_iter()
+ .filter(|s| s.keep.is_some())
+ .map(|s| (s.topic.to_string(), s.partition))
+ .collect()
+ }
+
+ #[test]
+ fn a_warm_replays_the_congruent_old_partitions_then_the_live_one() {
+ // Doubling: one old partition feeds each new one.
+ assert_eq!(
+ old_partitions(128, vec![(64, 1)], 65),
+ vec![("old1".to_string(), 1)]
+ );
+ // Dev's shrink: four old partitions feed each new one.
+ assert_eq!(
+ old_partitions(64, vec![(256, 1)], 5),
+ vec![
+ ("old1".to_string(), 5),
+ ("old1".to_string(), 69),
+ ("old1".to_string(), 133),
+ ("old1".to_string(), 197)
+ ]
+ );
+ // Coprime counts: every old partition may hold the keys.
+ assert_eq!(old_partitions(4, vec![(3, 1)], 1).len(), 3);
+ // Two generations replay oldest first, whatever the list order.
+ let sources = warm_sources(&config(8, vec![(4, 2), (2, 1)]), 1);
+ let topics: Vec<String> = sources.iter().map(|s| s.topic.to_string()).collect();
+ assert_eq!(topics, vec!["old1", "old2", "new"]);
+ let live = sources.last().unwrap();
+ assert!(live.keep.is_none());
+ assert_eq!(live.partition, 1);
+ assert!(sources[0]
+ .keep
+ .as_ref()
+ .is_some_and(|k| k.modulus == 8 && k.target == 1));
+ }
+}
diff --git a/rust/personhog-coordination/src/coordinator.rs b/rust/personhog-coordination/src/coordinator.rs
index 3669cfeb24cc..a2a424f85aa8 100644
--- a/rust/personhog-coordination/src/coordinator.rs
+++ b/rust/personhog-coordination/src/coordinator.rs
@@ -1285,7 +1285,7 @@ impl Coordinator {
tracing::debug!("changelog config not seeded, skipping assignment");
return Ok(());
};
- let total_partitions = changelog.partitions;
+ let (total_partitions, config_epoch) = (changelog.partitions, changelog.epoch);
// K8s-aware placement policies for smarter rebalancing
let members = match k8s_awareness {
@@ -1377,6 +1377,7 @@ impl Coordinator {
freeze_quorum_ref: Some(freeze_quorum_id.clone()),
created_at_ms: now_ms,
phase_entered_at_ms: now_ms,
+ config_epoch,
})
.collect();
@@ -1465,6 +1466,7 @@ impl Coordinator {
freeze_quorum_ref: None,
created_at_ms: now_ms,
phase_entered_at_ms: now_ms,
+ config_epoch,
},
expected_mod_revision: mod_revision,
});
diff --git a/rust/personhog-coordination/src/pod.rs b/rust/personhog-coordination/src/pod.rs
index 2629d623ddd8..b30995516cb7 100644
--- a/rust/personhog-coordination/src/pod.rs
+++ b/rust/personhog-coordination/src/pod.rs
@@ -21,6 +21,7 @@ use k8s_awareness::{DepartureReason, K8sAwareness};
use crate::authority::AuthorityClock;
use crate::error::{Error, Result};
use crate::store::{self, PersonhogStore};
+use crate::types::ChangelogConfig;
use crate::types::{
HandoffPhase, HandoffState, PartitionAssignment, PodDrainedAck, PodStatus, PodWarmedAck,
RegisteredPod,
@@ -157,6 +158,9 @@ pub trait HandoffHandler: Send + Sync {
/// must be idempotent and must not block: spawn and return.
async fn prepare_acquire(&self, _partition: u32) {}
+ /// The config a strong read captured just before the `warm_partition` that follows.
+ async fn capture_changelog(&self, _partition: u32, _config: &ChangelogConfig) {}
+
/// Old owner: release the partition from this pod's local state (drop cache,
/// close consumers, etc.).
///
@@ -351,6 +355,44 @@ impl PodHandle {
}
}
+ /// A strong read, never the adopted copy: a lagging watch would warm from a retired topic.
+ async fn capture_changelog(
+ &self,
+ partition: u32,
+ required_epoch: u64,
+ ) -> Result<Option<ChangelogConfig>> {
+ let Some(config) = self.store.get_changelog_config().await? else {
+ counter!("personhog_coordination_acquisitions_parked_total").increment(1);
+ tracing::warn!(
+ partition,
+ "config/changelog is not seeded; parking the acquisition"
+ );
+ return Ok(None);
+ };
+ if config.epoch < required_epoch {
+ counter!("personhog_coordination_acquisitions_parked_total").increment(1);
+ tracing::warn!(
+ partition,
+ read_epoch = config.epoch,
+ required_epoch,
+ "config/changelog is older than the handoff's epoch; parking the acquisition"
+ );
+ return Ok(None);
+ }
+ self.handler.capture_changelog(partition, &config).await;
+ Ok(Some(config))
+ }
+
+ /// Release the partition from the handler and forget its warm here.
+ async fn release_warm(&self, partition: u32) -> Result<()> {
+ self.handler.release_partition(partition).await?;
+ self.warmed_partitions
+ .lock()
+ .expect("warmed partitions lock poisoned")
+ .remove(&partition);
+ Ok(())
+ }
+
/// Run a reconcile pass whenever `nudge` fires, in addition to the
/// periodic tick. The nudging end announces breakage the protocol
/// has no event for — the leader's condemned changelog producer —
@@ -895,7 +937,7 @@ impl PodHandle {
if !quiesced.contains(&partition) {
continue;
}
- if let Err(e) = self.handler.release_partition(partition).await {
+ if let Err(e) = self.release_warm(partition).await {
failures.push(format!("release of partition {partition}: {e}"));
// Still held: the handler may retain the cache and the
// serving authority, so forgetting it here would make the
@@ -903,10 +945,6 @@ impl PodHandle {
// not. The run is about to end poisoned either way.
continue;
}
- self.warmed_partitions
- .lock()
- .expect("warmed partitions lock poisoned")
- .remove(&partition);
self.fenced_partitions
.lock()
.expect("fenced partitions lock poisoned")
@@ -1239,6 +1277,9 @@ impl PodHandle {
{
tracing::info!(pod, partition, "converging to Serving: warming");
let _warm_slot = self.acquire_warm_slot().await?;
+ if self.capture_changelog(partition, 0).await?.is_none() {
+ return Ok(did_work);
+ }
let start = Instant::now();
self.handler.warm_partition(partition).await?;
histogram!("personhog_coordination_partition_warm_ms", "trigger" => "restart")
@@ -1364,14 +1405,17 @@ impl PodHandle {
partition,
"converging to Acquiring: releasing a warm from an earlier era"
);
- self.handler.release_partition(partition).await?;
- self.warmed_partitions
- .lock()
- .expect("warmed partitions lock poisoned")
- .remove(&partition);
+ self.release_warm(partition).await?;
}
tracing::info!(pod, partition, "converging to Acquiring: warming");
let _warm_slot = self.acquire_warm_slot().await?;
+ if self
+ .capture_changelog(partition, handoff.config_epoch)
+ .await?
+ .is_none()
+ {
+ return Ok(did_work);
+ }
let start = Instant::now();
self.handler.warm_partition(partition).await?;
histogram!("personhog_coordination_partition_warm_ms", "trigger" => "handoff")
@@ -1424,11 +1468,7 @@ impl PodHandle {
.contains(&partition);
if was_warmed || was_fenced {
tracing::info!(pod, partition, "converging to Released: releasing");
- self.handler.release_partition(partition).await?;
- self.warmed_partitions
- .lock()
- .expect("warmed partitions lock poisoned")
- .remove(&partition);
+ self.release_warm(partition).await?;
self.fenced_partitions
.lock()
.expect("fenced partitions lock poisoned")
@@ -1993,6 +2033,7 @@ mod tests {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
}
}
diff --git a/rust/personhog-coordination/src/protocol.rs b/rust/personhog-coordination/src/protocol.rs
index ad54f576f9b9..6336cee7909e 100644
--- a/rust/personhog-coordination/src/protocol.rs
+++ b/rust/personhog-coordination/src/protocol.rs
@@ -293,6 +293,7 @@ mod tests {
created_at_ms: 0,
phase_entered_at_ms: 0,
new_owner_address: None,
+ config_epoch: 0,
}
}
diff --git a/rust/personhog-coordination/src/types.rs b/rust/personhog-coordination/src/types.rs
index 6e4aa94bde8a..25f7a3d1d6dd 100644
--- a/rust/personhog-coordination/src/types.rs
+++ b/rust/personhog-coordination/src/types.rs
@@ -188,6 +188,9 @@ pub struct HandoffState {
/// Zero means the record predates the field.
#[serde(default)]
pub phase_entered_at_ms: i64,
+ /// Changelog epoch the record was planned under.
+ #[serde(default)]
+ pub config_epoch: u64,
}
/// State machine for partition handoffs:
@@ -336,6 +339,13 @@ impl ChangelogConfig {
}
}
+ pub fn target(&self) -> ChangelogTarget {
+ ChangelogTarget {
+ topic: self.topic.clone(),
+ partitions: self.partitions,
+ }
+ }
+
/// A flip raises the epoch. Within an epoch only a retire changes the
/// object, and a retire only shrinks `draining`, so a same-epoch object
/// with more generations is an older one delivered late.
@@ -345,6 +355,12 @@ impl ChangelogConfig {
}
}
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+pub struct ChangelogTarget {
+ pub topic: String,
+ pub partitions: u32,
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -368,6 +384,7 @@ mod tests {
assert!(h.freeze_quorum.is_none());
assert_eq!(h.created_at_ms, 0);
assert_eq!(h.phase_entered_at_ms, 0);
+ assert_eq!(h.config_epoch, 0);
}
#[test]
@@ -469,6 +486,7 @@ mod tests {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
let json = serde_json::to_string(&handoff).unwrap();
@@ -489,6 +507,7 @@ mod tests {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
let json = serde_json::to_string(&handoff).unwrap();
diff --git a/rust/personhog-coordination/tests/integration.rs b/rust/personhog-coordination/tests/integration.rs
index f01555bb4f1f..d615e95426d2 100644
--- a/rust/personhog-coordination/tests/integration.rs
+++ b/rust/personhog-coordination/tests/integration.rs
@@ -1471,6 +1471,7 @@ async fn a_dead_new_owner_handoff_is_reaffirmed_to_the_current_owner() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&stuck_handoff).await.unwrap();
@@ -1605,6 +1606,7 @@ async fn stale_handoff_address_never_overwrites_a_fresher_registration() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
})
.await
.unwrap();
@@ -1836,6 +1838,7 @@ async fn late_joining_router_during_warming_begins_stash() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&warming_handoff).await.unwrap();
@@ -1907,6 +1910,7 @@ async fn dead_old_owner_in_freezing_advances_to_completion() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&stale).await.unwrap();
@@ -1963,6 +1967,7 @@ async fn late_joining_router_during_freezing_acks_and_stashes() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&freezing_handoff).await.unwrap();
@@ -2049,6 +2054,7 @@ async fn a_dead_new_owner_handoff_in_warming_is_reaffirmed() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&stuck).await.unwrap();
@@ -2154,6 +2160,7 @@ async fn reconcile_advances_warming_with_pre_staged_warmed_ack() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&warming).await.unwrap();
@@ -2238,6 +2245,7 @@ async fn draining_old_owner_blocks_phase_advance() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&handoff).await.unwrap();
@@ -2347,6 +2355,7 @@ async fn draining_old_owner_does_not_trigger_cleanup() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&handoff).await.unwrap();
@@ -2461,6 +2470,7 @@ async fn a_handoff_that_can_never_reach_quorum_is_cancelled() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
})
.await
.unwrap();
@@ -2550,6 +2560,7 @@ async fn freezing_blocks_until_routers_ack_before_draining() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&handoff).await.unwrap();
@@ -2654,6 +2665,7 @@ async fn initial_assignment_skips_draining_phase() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&handoff).await.unwrap();
@@ -2720,6 +2732,7 @@ async fn dead_old_owner_in_draining_recovers() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&handoff).await.unwrap();
@@ -2826,6 +2839,7 @@ async fn reconcile_advances_draining_with_pre_staged_drained_ack() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&handoff).await.unwrap();
@@ -2890,6 +2904,7 @@ async fn late_joining_router_during_draining_begins_stash_no_ack() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&draining_handoff).await.unwrap();
@@ -2981,6 +2996,7 @@ async fn a_dead_new_owner_handoff_in_draining_is_reaffirmed() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&stuck).await.unwrap();
@@ -3234,6 +3250,7 @@ async fn conflicting_plan_cannot_replace_an_in_flight_handoff() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
assert!(store
@@ -3254,6 +3271,7 @@ async fn conflicting_plan_cannot_replace_an_in_flight_handoff() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
let stable = PartitionAssignment {
@@ -3308,6 +3326,7 @@ async fn stale_plan_is_rejected_when_the_assignment_moved() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
@@ -3391,6 +3410,7 @@ async fn fresh_plan_is_rejected_when_an_assignment_appeared() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
diff --git a/rust/personhog-coordination/tests/protocol_hardening.rs b/rust/personhog-coordination/tests/protocol_hardening.rs
index b1ce656f5d8a..c99eeb2c28bb 100644
--- a/rust/personhog-coordination/tests/protocol_hardening.rs
+++ b/rust/personhog-coordination/tests/protocol_hardening.rs
@@ -62,6 +62,7 @@ async fn put_handoff(
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
// Raw put on purpose: fixtures force arbitrary handoff states,
@@ -99,6 +100,10 @@ async fn wait_for_event(events: &Arc<Mutex<Vec<HandoffEvent>>>, expected: Handof
#[tokio::test]
async fn pod_self_fences_locally_and_rejoins_after_lease_loss() {
let (store, prefix) = test_store_with_prefix("pod-self-fence").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
// lease_ttl 5 → 1s heartbeat interval, so the keepalive observes the
@@ -305,6 +310,10 @@ async fn coordinator_abdicates_and_recampaigns_when_election_lease_revoked() {
#[tokio::test]
async fn pod_resumes_partition_when_cancelled_handoff_leaves_it_assigned() {
let store = test_store("handoff-cancel-resume").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let pod = start_pod(Arc::clone(&store), "resume-pod-a", cancel.clone());
@@ -368,6 +377,10 @@ async fn pod_resumes_partition_when_cancelled_handoff_leaves_it_assigned() {
#[tokio::test]
async fn pod_releases_partition_when_cancelled_handoff_leaves_it_unassigned() {
let store = test_store("handoff-cancel-release").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let pod = start_pod(Arc::clone(&store), "release-pod-a", cancel.clone());
@@ -407,6 +420,10 @@ async fn pod_releases_partition_when_cancelled_handoff_leaves_it_unassigned() {
#[tokio::test]
async fn handoff_cleanup_after_complete_does_not_resume() {
let store = test_store("handoff-cleanup-no-resume").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let pod = start_pod(Arc::clone(&store), "cleanup-pod-a", cancel.clone());
@@ -467,6 +484,10 @@ async fn handoff_cleanup_after_complete_does_not_resume() {
#[tokio::test]
async fn restarted_pod_rewarms_assigned_partitions() {
let store = test_store("restart-rewarm").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
store
@@ -513,6 +534,10 @@ async fn restarted_pod_rewarms_assigned_partitions() {
#[tokio::test]
async fn restarted_old_owner_serves_again_after_handoff_cancelled() {
let store = test_store("restart-cancel-serve").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
store
@@ -563,6 +588,10 @@ async fn restarted_old_owner_serves_again_after_handoff_cancelled() {
#[tokio::test]
async fn restarted_old_owner_refences_when_handoff_in_warming() {
let store = test_store("restart-refence").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
store
@@ -1312,6 +1341,10 @@ async fn freezing_handoff_advances_when_unacked_router_departs() {
#[tokio::test]
async fn pod_catches_up_on_existing_warming_handoff_at_startup() {
let store = test_store("pod-startup-warm-catchup").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
// The handoff exists before the pod starts — as after a fast restart.
@@ -1344,6 +1377,10 @@ async fn pod_catches_up_on_existing_warming_handoff_at_startup() {
#[tokio::test]
async fn pod_catches_up_on_existing_draining_handoff_at_startup() {
let store = test_store("pod-startup-drain-catchup").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
put_handoff(
@@ -2125,6 +2162,10 @@ impl personhog_coordination::pod::HandoffHandler for FlakyHandoffHandler {
#[tokio::test]
async fn a_pod_attempt_failure_preserves_registration_and_partitions() {
let store = test_store("pod-attempt-preserves").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let events: Arc<Mutex<Vec<HandoffEvent>>> = Arc::new(Mutex::new(Vec::new()));
@@ -2610,6 +2651,7 @@ async fn put_handoff_with_id(
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
new_owner_address: None,
};
store.put_handoff(&handoff).await.expect("write handoff");
@@ -2639,6 +2681,10 @@ async fn wait_for_warmed_ack(store: &Arc<PersonhogStore>, partition: u32, pod: &
#[tokio::test]
async fn concurrent_inbound_handoffs_warm_in_parallel() {
let store = test_store("pod-parallel-warm").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let pod = start_pod_gated(Arc::clone(&store), "parallel-warm-pod", 4, cancel.clone());
@@ -2670,6 +2716,10 @@ async fn concurrent_inbound_handoffs_warm_in_parallel() {
#[tokio::test]
async fn convergences_for_one_partition_never_interleave() {
let store = test_store("pod-single-flight").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let pod = start_pod_gated(Arc::clone(&store), "single-flight-pod", 4, cancel.clone());
@@ -2737,6 +2787,10 @@ async fn convergences_for_one_partition_never_interleave() {
#[tokio::test]
async fn concurrent_warms_are_bounded_by_warm_concurrency() {
let store = test_store("pod-warm-bound").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let pod = start_pod_gated(Arc::clone(&store), "warm-bound-pod", 2, cancel.clone());
@@ -2795,6 +2849,10 @@ async fn concurrent_warms_are_bounded_by_warm_concurrency() {
#[tokio::test]
async fn lease_loss_during_attempt_backoff_self_fences_promptly() {
let (store, prefix) = test_store_with_prefix("pod-backoff-fence").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let events: Arc<Mutex<Vec<HandoffEvent>>> = Arc::new(Mutex::new(Vec::new()));
@@ -2858,6 +2916,10 @@ async fn lease_loss_during_attempt_backoff_self_fences_promptly() {
#[tokio::test]
async fn lease_loss_during_graceful_drain_self_fences_promptly() {
let (store, prefix) = test_store_with_prefix("pod-drain-fence").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let (handler, events) = MockHandoffHandler::new();
@@ -2943,6 +3005,10 @@ async fn a_connection_blip_does_not_fence_the_pod() {
let prefix = format!("/test-keepalive-blip-{}/", uuid::Uuid::new_v4());
let pod_store = store_at(&proxy.endpoint, &prefix).await;
let direct_store = store_at(ETCD_ENDPOINT, &prefix).await;
+ assert!(direct_store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let (handler, events) = MockHandoffHandler::new();
@@ -3003,6 +3069,10 @@ async fn a_sustained_outage_fences_at_the_renewal_margin() {
let prefix = format!("/test-keepalive-margin-{}/", uuid::Uuid::new_v4());
let pod_store = store_at(&proxy.endpoint, &prefix).await;
let direct_store = store_at(ETCD_ENDPOINT, &prefix).await;
+ assert!(direct_store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let (handler, events) = MockHandoffHandler::new();
@@ -3060,6 +3130,10 @@ async fn a_sustained_outage_fences_at_the_renewal_margin() {
#[tokio::test]
async fn progress_between_failures_keeps_the_run_alive() {
let store = test_store("pod-progress-reset").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let events: Arc<Mutex<Vec<HandoffEvent>>> = Arc::new(Mutex::new(Vec::new()));
@@ -3124,6 +3198,10 @@ async fn progress_between_failures_keeps_the_run_alive() {
#[tokio::test]
async fn budget_exhaustion_fences_before_deregistering() {
let store = test_store("pod-fatal-fence").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let events: Arc<Mutex<Vec<HandoffEvent>>> = Arc::new(Mutex::new(Vec::new()));
@@ -3181,6 +3259,10 @@ async fn budget_exhaustion_fences_before_deregistering() {
#[tokio::test]
async fn pod_retries_resume_after_a_failed_attempt() {
let store = test_store("handoff-cancel-resume-retry").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let pod = start_pod_with_flaky_resume(Arc::clone(&store), "resume-flaky-a", 1, cancel.clone());
@@ -3262,6 +3344,11 @@ async fn authority_lapses_when_renewals_stop() {
tokio::spawn(async move { pod.run(token).await });
let store = store_at(ETCD_ENDPOINT, &prefix).await;
+
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
put_handoff(&store, 0, None, "lapse-pod", HandoffPhase::Warming).await;
wait_for_event(&events, HandoffEvent::Warmed(0)).await;
assert!(
@@ -3327,6 +3414,10 @@ async fn authority_lapses_when_renewals_stop() {
async fn authority_is_surrendered_when_the_registration_is_deleted() {
let prefix = format!("/test-registration-delete-{}/", uuid::Uuid::new_v4());
let store = store_at(ETCD_ENDPOINT, &prefix).await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let (handler, events) = MockHandoffHandler::new();
@@ -3381,6 +3472,10 @@ async fn authority_is_surrendered_when_the_registration_is_deleted() {
async fn a_deleted_registration_starts_a_new_session() {
let prefix = format!("/test-registration-resession-{}/", uuid::Uuid::new_v4());
let store = store_at(ETCD_ENDPOINT, &prefix).await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let (handler, events) = MockHandoffHandler::new();
@@ -3436,6 +3531,10 @@ async fn a_deleted_registration_starts_a_new_session() {
async fn a_foreign_deletion_under_the_pods_prefix_does_not_cost_the_session() {
let prefix = format!("/test-registration-foreign-{}/", uuid::Uuid::new_v4());
let store = store_at(ETCD_ENDPOINT, &prefix).await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let (handler, events) = MockHandoffHandler::new();
@@ -3509,6 +3608,10 @@ async fn a_foreign_deletion_under_the_pods_prefix_does_not_cost_the_session() {
async fn an_operator_delete_of_a_live_registration_revokes_its_lease() {
let prefix = format!("/test-registration-operator-del-{}/", uuid::Uuid::new_v4());
let store = store_at(ETCD_ENDPOINT, &prefix).await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let (handler, events) = MockHandoffHandler::new();
@@ -3590,6 +3693,10 @@ async fn an_operator_delete_of_a_live_registration_revokes_its_lease() {
#[tokio::test]
async fn a_stuck_drain_does_not_strand_the_other_partitions_on_lease_loss() {
let (store, prefix) = test_store_with_prefix("stuck-drain-fence").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let pod = start_pod_with_stuck_drain(Arc::clone(&store), "stuck-pod-0", 1, 5, cancel.clone());
@@ -3681,6 +3788,10 @@ async fn a_stuck_drain_does_not_strand_the_other_partitions_on_lease_loss() {
#[tokio::test]
async fn a_drain_that_times_out_is_a_failure_not_a_quiesce() {
let (store, prefix) = test_store_with_prefix("hung-drain-fence").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let mut pod =
start_pod_with_hanging_drain(Arc::clone(&store), "hung-pod-0", 1, 5, cancel.clone());
@@ -3758,6 +3869,10 @@ async fn a_drain_that_times_out_is_a_failure_not_a_quiesce() {
#[tokio::test]
async fn a_failed_drain_still_leaves_a_partition_that_can_be_resumed() {
let (store, _prefix) = test_store_with_prefix("failed-drain-resume").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let pod = start_pod_with_stuck_drain(Arc::clone(&store), "stuck-pod-1", 0, 30, cancel.clone());
@@ -3840,6 +3955,10 @@ async fn a_failed_drain_still_leaves_a_partition_that_can_be_resumed() {
#[tokio::test]
async fn a_failing_release_does_not_stop_the_pod_giving_up_the_rest() {
let (store, prefix) = test_store_with_prefix("failing-release-fence").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let (pod, attempts) =
start_pod_with_failing_release(Arc::clone(&store), "failing-rel-0", 5, cancel.clone());
@@ -3915,6 +4034,10 @@ async fn a_failing_release_does_not_stop_the_pod_giving_up_the_rest() {
#[tokio::test]
async fn a_self_fence_that_could_not_finish_refuses_in_place_recovery() {
let (store, prefix) = test_store_with_prefix("poisoned-self-fence").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let mut pod =
start_pod_with_stuck_drain(Arc::clone(&store), "poisoned-pod-0", 0, 5, cancel.clone());
@@ -3963,6 +4086,10 @@ async fn a_self_fence_that_could_not_finish_refuses_in_place_recovery() {
#[tokio::test]
async fn a_failed_release_is_retried_rather_than_forgotten() {
let (store, _prefix) = test_store_with_prefix("flaky-release").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let pod =
start_pod_with_flaky_release(Arc::clone(&store), "flaky-release-pod", 1, cancel.clone());
@@ -4158,6 +4285,10 @@ impl personhog_coordination::pod::HandoffHandler for RepairProbeHandler {
#[tokio::test]
async fn a_repair_nudge_converges_without_waiting_for_reconcile() {
let store = test_store("repair-request-converges").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
store
@@ -4288,6 +4419,10 @@ impl personhog_coordination::pod::HandoffHandler for PrepareProbeHandler {
#[tokio::test]
async fn a_pending_new_owner_is_hinted_but_not_warmed() {
let store = test_store("pending-owner-hint").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
store
@@ -4477,6 +4612,10 @@ async fn a_standby_waits_on_the_leader_key_rather_than_campaigning() {
#[tokio::test]
async fn a_replaced_handoff_reaches_the_old_owner_it_no_longer_names() {
let store = test_store("handoff-replaced-old-owner").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let pod = start_pod(Arc::clone(&store), "replaced-pod-a", cancel.clone());
@@ -4677,6 +4816,7 @@ async fn a_freeze_quorum_reference_that_is_gone_requires_every_live_router() {
freeze_quorum_ref: Some("never-written".to_string()),
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
};
let quorum = store
@@ -4739,6 +4879,10 @@ async fn a_freeze_quorum_reference_that_is_gone_requires_every_live_router() {
#[tokio::test]
async fn a_handoff_cancelled_mid_warm_reaches_the_pod_still_warming() {
let store = test_store("cancel-mid-warm").await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let pod = start_pod_gated(Arc::clone(&store), "mid-warm-pod", 4, cancel.clone());
@@ -5175,6 +5319,10 @@ async fn a_pod_asked_to_stop_while_etcd_hangs_exits_inside_its_bounds() {
let proxy = FlakyProxy::start("127.0.0.1:2379").await;
let prefix = format!("/test-pod-teardown-{}/", uuid::Uuid::new_v4());
let store = store_at(&proxy.endpoint, &prefix).await;
+ assert!(store
+ .seed_changelog_config("personhog_updates", 4, "personhog-writer")
+ .await
+ .unwrap());
let cancel = CancellationToken::new();
let mut pod = start_pod_with_lease_ttl(Arc::clone(&store), "teardown-pod", 30, cancel.clone());
@@ -5342,6 +5490,7 @@ async fn a_recorded_membership_advances_past_a_router_outside_it() {
freeze_quorum_ref: Some("narrow-membership".to_string()),
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
};
store.put_handoff(&handoff).await.expect("write handoff");
store
@@ -5489,6 +5638,7 @@ async fn a_cached_absent_membership_still_requires_every_live_router() {
freeze_quorum_ref: Some("never-written".to_string()),
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
};
let routers = [
@@ -5609,6 +5759,7 @@ async fn a_membership_already_read_is_answered_without_reading_again() {
freeze_quorum_ref: Some(id.to_string()),
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
};
assert_eq!(
@@ -5681,6 +5832,7 @@ async fn a_plan_past_the_server_txn_budget_still_applies() {
freeze_quorum_ref: Some(quorum_id.to_string()),
created_at_ms: now * 1000,
phase_entered_at_ms: now * 1000,
+ config_epoch: 0,
})
.collect();
let preconditions: Vec<AssignmentPrecondition> = (0..total)
@@ -5736,6 +5888,7 @@ async fn a_conflicted_partition_stands_down_alone() {
freeze_quorum_ref: None,
created_at_ms: 1_000_000,
phase_entered_at_ms: 1_000_000,
+ config_epoch: 0,
};
// Partition 40 already has a handoff — a concurrent planner won it.
@@ -5800,6 +5953,7 @@ async fn a_swept_quorum_delete_loses_to_a_concurrent_re_put() {
freeze_quorum_ref: Some(quorum_id.to_string()),
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
};
let listed_revision = |store: Arc<personhog_coordination::store::PersonhogStore>| async move {
store
@@ -5883,6 +6037,7 @@ async fn a_missing_membership_is_resolved_again_once_written() {
freeze_quorum_ref: Some(quorum_id.to_string()),
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
};
assert!(store
@@ -5933,6 +6088,7 @@ async fn an_over_server_budget_chunk_degrades_to_per_unit_transactions() {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
})
.collect();
let preconditions: Vec<AssignmentPrecondition> = (0..total)
diff --git a/rust/personhog-leader/README.md b/rust/personhog-leader/README.md
index 35b1dbff1877..0f65bd797646 100644
--- a/rust/personhog-leader/README.md
+++ b/rust/personhog-leader/README.md
@@ -165,10 +165,9 @@ and warming can consume to a known endpoint without racing producers.
(`O_pg_writer(P)`) via a short-lived OffsetFetch consumer. Anything
at or after this offset still needs to be in cache; anything below
is already durable in PG.
-2. Resolve the start offset: `committed_offset - lookback_offsets`,
- clamped to the partition's earliest available offset. The lookback
- is a configurable safety margin against momentary races between
- the writer's commit and our read of it.
+2. Resolve the start offset: the committed offset, clamped to the
+ partition's earliest available offset. Every record from there is
+ marked dirty as it warms.
3. `assign()` (not `subscribe()`) the warming consumer to the
partition at the resolved start offset and consume until HWM.
4. Buffer decoded records locally; only commit them to the cache
@@ -183,8 +182,6 @@ and warming can consume to a known endpoint without racing producers.
- `WRITER_CONSUMER_GROUP` — group whose committed offset bounds the
warming range.
-- `WARM_LOOKBACK_OFFSETS` — safety margin to rewind past the writer's
- commit.
- `WARM_COMMITTED_OFFSETS_TIMEOUT_SECS`, `WARM_FETCH_WATERMARKS_TIMEOUT_SECS`,
`WARM_RECV_TIMEOUT_SECS` — Kafka call timeouts.
- `WARM_RETRY_MAX_ATTEMPTS`, `WARM_RETRY_INITIAL_BACKOFF_MS`,
diff --git a/rust/personhog-leader/src/cache/dirty_index.rs b/rust/personhog-leader/src/cache/dirty_index.rs
index 431530fb090d..ab96e2e18e3d 100644
--- a/rust/personhog-leader/src/cache/dirty_index.rs
+++ b/rust/personhog-leader/src/cache/dirty_index.rs
@@ -7,9 +7,16 @@ use dashmap::DashMap;
use super::PersonCacheKey;
+/// The changelog partition a mark's record lives in.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct Stream {
+ pub topic: Arc<str>,
+ pub partition: u32,
+}
+
/// A mark for a person whose latest acked state may not yet be applied to
/// Postgres by the writer.
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DirtyMark {
/// Version of the latest acked changelog record for this person.
/// Recovery cross-checks it against the record it fetches: the two
@@ -27,6 +34,7 @@ pub struct DirtyMark {
/// The person's routing partition, denormalized so pruning can work
/// per partition without rehashing keys.
pub partition: u32,
+ pub stream: Stream,
/// Whether the marked record is a death document. The prune returns
/// only these pairs, so the settle never touches live persons' keys.
pub is_deleted: bool,
@@ -39,10 +47,12 @@ pub struct PrunedMarks {
pub death_marks: Vec<(PersonCacheKey, DirtyMark)>,
}
-/// A partition's reclaim queue: `(offset, key)` entries in rough offset
-/// order, popped from the head as the writer's committed offset passes them.
+/// `(offset, key)` entries in rough offset order, popped as the writer's commit passes them.
type ReclaimQueue = Arc<Mutex<VecDeque<(i64, PersonCacheKey)>>>;
+/// One reclaim queue per cache partition and stream feeding it.
+type QueueKey = (u32, Stream);
+
/// Maximum queue pops per prune lock hold (see `prune_chunk`).
pub const PRUNE_CHUNK: usize = 4_096;
@@ -73,15 +83,10 @@ const SHRINK_CAPACITY_FLOOR: usize = 4 * PRUNE_CHUNK;
/// proportional to the marks actually reclaimed — never the index size —
/// which is what lets the prune loop run every second.
///
-/// The queue is a hint; the map is the truth. Every marked key has exactly
-/// one live queue entry, carrying some offset at or below its map offset:
-/// `mark` enqueues only brand-new keys, so a re-marked key's queue entry
-/// goes stale, and when the prune pops it and finds the map ahead of the
-/// committed offset it re-enqueues the entry at the map's offset instead
-/// of removing the mark. Task scheduling can also interleave near-
-/// simultaneous marks slightly out of offset order; the prune stops at the
-/// first unapplied head, so an inversion only delays the entries behind it
-/// until a later tick.
+/// The map is the truth and the queues are a hint. Each marked key has a queue
+/// entry under its mark's home (partition and stream), at or below its offset.
+/// The prune re-queues an entry whose mark moved ahead, and drops one whose mark
+/// moved home. Out-of-order marks only delay the entries behind them.
///
/// The index is soft-bounded by `max_entries`: under a sustained writer
/// outage it would otherwise grow one mark per unique person written —
@@ -101,12 +106,9 @@ pub struct DirtyIndex {
/// queue gone instead of racing the drain. Entries popped before the
/// detach have already had their map marks removed; everything else
/// is still in the queue for the drain — no entry escapes a release.
- queues: DashMap<u32, ReclaimQueue>,
- /// Highest offset ever marked per partition since its last clear.
- /// Map offsets only grow and pruning removes low offsets first, so
- /// while any mark lives this equals the largest live mark — an O(1)
- /// read for the writer-lag gauge.
- max_marked: DashMap<u32, i64>,
+ queues: DashMap<QueueKey, ReclaimQueue>,
+ /// Highest offset marked per queue since its last clear, for the writer-lag gauge.
+ max_marked: DashMap<QueueKey, i64>,
/// Entry count maintained alongside the map: `DashMap::len()` takes a
/// read lock on every shard, which is too expensive for the admission
/// check on the write hot path. Kept in step by `mark` (insert only)
@@ -137,21 +139,21 @@ impl DirtyIndex {
self.size.load(Ordering::Relaxed) < self.max_entries || self.map.contains_key(key)
}
- /// Record the latest acked offset for a person. Writers hold the
- /// per-key lock, so marks normally arrive in offset order; the guard
- /// against regressing keeps a concurrent warm-seeding pass from
- /// clobbering a newer mark with an older replayed one. The map is
- /// updated before the queue so a concurrent prune popping the entry
- /// always sees the mark.
+ /// Record the latest acked offset for a person. A replayed mark never moves one
+ /// backwards within a home; a mark that moves home replaces the old one. The map
+ /// updates before the queue, so a concurrent prune always sees the mark.
pub fn mark(&self, key: PersonCacheKey, mark: DirtyMark) {
- let partition = mark.partition;
+ let queue_key = (mark.partition, mark.stream.clone());
let offset = mark.offset;
- let is_new = match self.map.entry(key.clone()) {
+ let enqueue = match self.map.entry(key.clone()) {
Entry::Occupied(mut entry) => {
- if mark.offset > entry.get().offset {
+ // A newer generation's record outranks any offset of an older one.
+ let current = entry.get();
+ let moved = mark.partition != current.partition || mark.stream != current.stream;
+ if moved || mark.offset > current.offset {
*entry.get_mut() = mark;
}
- false
+ moved
}
Entry::Vacant(entry) => {
entry.insert(mark);
@@ -160,13 +162,13 @@ impl DirtyIndex {
}
};
self.max_marked
- .entry(partition)
+ .entry(queue_key.clone())
.and_modify(|max| *max = (*max).max(offset))
.or_insert(offset);
- if is_new {
+ if enqueue {
let queue = Arc::clone(
self.queues
- .entry(partition)
+ .entry(queue_key)
.or_insert_with(|| Arc::new(Mutex::new(VecDeque::new())))
.value(),
);
@@ -178,7 +180,7 @@ impl DirtyIndex {
}
pub fn get(&self, key: &PersonCacheKey) -> Option<DirtyMark> {
- self.map.get(key).map(|entry| *entry.value())
+ self.map.get(key).map(|entry| entry.value().clone())
}
/// Return the map's spare capacity to the allocator. Removal never
@@ -205,14 +207,19 @@ impl DirtyIndex {
/// is empty, or it was detached by a concurrent `clear_partition` —
/// the re-fetch is what lets a release end an in-flight pass instead
/// of racing its drain.
- pub fn prune_chunk(&self, partition: u32, committed: i64) -> (PrunedMarks, bool) {
+ pub fn prune_chunk(
+ &self,
+ partition: u32,
+ stream: &Stream,
+ committed: i64,
+ ) -> (PrunedMarks, bool) {
let mut pruned = PrunedMarks {
removed: 0,
death_marks: Vec::new(),
};
let Some(queue) = self
.queues
- .get(&partition)
+ .get(&(partition, stream.clone()))
.map(|entry| Arc::clone(entry.value()))
else {
return (pruned, true);
@@ -232,28 +239,24 @@ impl DirtyIndex {
}
budget -= 1;
let (_, key) = queue.pop_front().expect("front was just observed");
- // The removal re-checks the map's offset under the shard
- // lock: between the pop and here the key may have been
+ // The removal re-checks the map's home and offset under the
+ // shard lock: between the pop and here the key may have been
// re-marked, and removing the newer mark would silently
// reopen the stale-fallback hole.
- if let Some(entry) = self.map.remove_if(&key, |_, mark| mark.offset < committed) {
+ if let Some(entry) = self.map.remove_if(&key, |_, mark| {
+ mark.partition == partition && mark.stream == *stream && mark.offset < committed
+ }) {
removed += 1;
pruned.removed += 1;
if entry.1.is_deleted {
pruned.death_marks.push(entry);
}
} else if let Some(mark) = self.map.get(&key) {
- // Superseded: the live mark is ahead of the committed
- // offset, and the entry just popped was this key's only
- // queue presence (`mark` enqueues keys once). Dropping
- // it would leave the mark unreclaimable — pops are the
- // only reclaim path — so re-enqueue at the map's
- // offset. The tail lands past the committed threshold,
- // so the head check terminates the pass before
- // re-processing it; the resulting disorder only delays
- // that key's reclaim, since removal is always re-proved
- // against the map.
- queue.push_back((mark.offset, key.clone()));
+ // The live mark is newer: re-queue it at its offset. An entry whose mark
+ // moved home is stale here and dropped.
+ if mark.partition == partition && mark.stream == *stream {
+ queue.push_back((mark.offset, key.clone()));
+ }
}
}
self.size.fetch_sub(removed, Ordering::Relaxed);
@@ -270,18 +273,30 @@ impl DirtyIndex {
(pruned, exhausted)
}
- /// Drop every mark on `partition` — used when the partition is released
- /// to another owner, whose warming rebuilds its own marks.
+ /// Drop every mark homed on `partition`, across every stream feeding it.
+ /// A queued key whose live mark moved to another home is left to that home.
pub fn clear_partition(&self, partition: u32) -> usize {
- let Some((_, queue)) = self.queues.remove(&partition) else {
- return 0;
- };
- self.max_marked.remove(&partition);
- let mut queue = queue.lock().expect("dirty index queue lock poisoned");
+ let keys: Vec<QueueKey> = self
+ .queues
+ .iter()
+ .filter(|entry| entry.key().0 == partition)
+ .map(|entry| entry.key().clone())
+ .collect();
let mut removed = 0;
- while let Some((_, key)) = queue.pop_front() {
- if self.map.remove(&key).is_some() {
- removed += 1;
+ for queue_key in keys {
+ let Some((_, queue)) = self.queues.remove(&queue_key) else {
+ continue;
+ };
+ self.max_marked.remove(&queue_key);
+ let mut queue = queue.lock().expect("dirty index queue lock poisoned");
+ while let Some((_, key)) = queue.pop_front() {
+ if self
+ .map
+ .remove_if(&key, |_, mark| mark.partition == partition)
+ .is_some()
+ {
+ removed += 1;
+ }
}
}
self.size.fetch_sub(removed, Ordering::Relaxed);
@@ -293,9 +308,8 @@ impl DirtyIndex {
removed
}
- /// Partitions that currently hold marks — the prune loop's targeting
- /// set for the writer committed-offset fetch.
- pub fn partitions_with_marks(&self) -> Vec<u32> {
+ /// Cache partitions and the streams on which they hold marks.
+ pub fn streams_with_marks(&self) -> Vec<(u32, Stream)> {
self.queues
.iter()
.filter(|entry| {
@@ -305,14 +319,14 @@ impl DirtyIndex {
.expect("dirty index queue lock poisoned")
.is_empty()
})
- .map(|entry| *entry.key())
+ .map(|entry| entry.key().clone())
.collect()
}
- /// Highest live marked offset for a partition, if any — feeds the
- /// writer-lag gauge.
- pub fn max_offset(&self, partition: u32) -> Option<i64> {
- let has_marks = self.queues.get(&partition).is_some_and(|queue| {
+ /// Highest live marked offset on a stream, for the writer-lag gauge.
+ pub fn max_offset(&self, partition: u32, stream: &Stream) -> Option<i64> {
+ let queue_key = (partition, stream.clone());
+ let has_marks = self.queues.get(&queue_key).is_some_and(|queue| {
!queue
.value()
.lock()
@@ -322,7 +336,7 @@ impl DirtyIndex {
if !has_marks {
return None;
}
- self.max_marked.get(&partition).map(|max| *max.value())
+ self.max_marked.get(&queue_key).map(|max| *max.value())
}
pub fn len(&self) -> usize {
@@ -351,11 +365,19 @@ mod tests {
}
}
+ fn stream(partition: u32) -> Stream {
+ Stream {
+ topic: Arc::from("personhog_updates"),
+ partition,
+ }
+ }
+
fn mark(offset: i64, partition: u32) -> DirtyMark {
DirtyMark {
version: offset,
offset,
partition,
+ stream: stream(partition),
is_deleted: false,
}
}
@@ -373,7 +395,7 @@ mod tests {
death_marks: Vec::new(),
};
loop {
- let (pruned, exhausted) = index.prune_chunk(partition, committed);
+ let (pruned, exhausted) = index.prune_chunk(partition, &stream(partition), committed);
total.removed += pruned.removed;
total.death_marks.extend(pruned.death_marks);
if exhausted {
@@ -494,7 +516,7 @@ mod tests {
// The first chunk of an in-flight pass reclaims its bounded slice
// and reports the pass unfinished.
- let (removed, exhausted) = index.prune_chunk(0, total);
+ let (removed, exhausted) = index.prune_chunk(0, &stream(0), total);
assert_eq!(removed.removed, PRUNE_CHUNK);
assert!(!exhausted);
@@ -505,7 +527,7 @@ mod tests {
// The paused pass re-fetches, observes the detachment, and ends
// instead of racing the drain it would otherwise double-count.
- let (removed, exhausted) = index.prune_chunk(0, total);
+ let (removed, exhausted) = index.prune_chunk(0, &stream(0), total);
assert_eq!(removed.removed, 0);
assert!(exhausted);
@@ -524,11 +546,11 @@ mod tests {
index.mark(key(offset), death_mark(offset, 0));
}
- let (first, exhausted) = index.prune_chunk(0, total);
+ let (first, exhausted) = index.prune_chunk(0, &stream(0), total);
assert_eq!(first.death_marks.len(), PRUNE_CHUNK);
assert!(!exhausted);
- let (rest, exhausted) = index.prune_chunk(0, total);
+ let (rest, exhausted) = index.prune_chunk(0, &stream(0), total);
assert_eq!(rest.death_marks.len(), 100);
assert!(exhausted);
assert!(index.is_empty());
@@ -555,16 +577,144 @@ mod tests {
index.mark(key(2), mark(9, 0));
index.mark(key(3), mark(2, 7));
- let mut partitions = index.partitions_with_marks();
+ let mut partitions: Vec<u32> = index
+ .streams_with_marks()
+ .into_iter()
+ .map(|(p, _)| p)
+ .collect();
partitions.sort_unstable();
assert_eq!(partitions, vec![0, 7]);
- assert_eq!(index.max_offset(0), Some(9));
- assert_eq!(index.max_offset(7), Some(2));
- assert_eq!(index.max_offset(3), None);
+ assert_eq!(index.max_offset(0, &stream(0)), Some(9));
+ assert_eq!(index.max_offset(7, &stream(7)), Some(2));
+ assert_eq!(index.max_offset(3, &stream(3)), None);
prune_all(&index, 0, 100);
- assert_eq!(index.max_offset(0), None);
- assert_eq!(index.partitions_with_marks(), vec![7]);
+ assert_eq!(index.max_offset(0, &stream(0)), None);
+ let left: Vec<u32> = index
+ .streams_with_marks()
+ .into_iter()
+ .map(|(p, _)| p)
+ .collect();
+ assert_eq!(left, vec![7]);
+ }
+
+ /// Two streams under one partition prune apart and clear together.
+ #[test]
+ fn streams_under_one_partition_prune_apart_and_clear_together() {
+ let index = DirtyIndex::new(1_000);
+ let old = Stream {
+ topic: Arc::from("personhog_updates_old"),
+ partition: 1,
+ };
+ index.mark(key(1), mark(4, 65));
+ index.mark(
+ key(2),
+ DirtyMark {
+ stream: old.clone(),
+ ..mark(900, 65)
+ },
+ );
+
+ let (pruned, _) = index.prune_chunk(65, &stream(65), 100);
+ assert_eq!(pruned.removed, 1);
+ assert!(index.get(&key(1)).is_none());
+ assert!(index.get(&key(2)).is_some());
+ assert_eq!(index.max_offset(65, &old), Some(900));
+
+ index.mark(key(2), mark(5, 65));
+ assert_eq!(index.get(&key(2)).unwrap().stream, stream(65));
+ assert_eq!(index.clear_partition(65), 1);
+ assert!(index.streams_with_marks().is_empty());
+ }
+
+ #[test]
+ fn pruning_a_stream_terminates_when_a_key_was_re_marked_on_a_newer_stream() {
+ let old = Stream {
+ topic: Arc::from("personhog_updates_old"),
+ partition: 0,
+ };
+ for committed in [1_457, i64::MAX] {
+ let index = DirtyIndex::new(1_000);
+ index.mark(
+ key(1),
+ DirtyMark {
+ stream: old.clone(),
+ ..mark(900, 0)
+ },
+ );
+ index.mark(key(1), mark(12, 0));
+ let mut chunks = 0;
+ loop {
+ let (_, exhausted) = index.prune_chunk(0, &old, committed);
+ chunks += 1;
+ if exhausted {
+ break;
+ }
+ assert!(
+ chunks < 64,
+ "the old stream never exhausts at committed {committed}"
+ );
+ }
+ assert_eq!(index.get(&key(1)).unwrap().stream, stream(0));
+ // Refiled under its own stream, the mark is reclaimable there.
+ assert_eq!(index.max_offset(0, &old), None);
+ assert_eq!(prune_all(&index, 0, 13).removed, 1);
+ assert!(index.get(&key(1)).is_none());
+ }
+ }
+
+ #[test]
+ fn a_key_that_moved_home_survives_the_old_homes_clear_and_prune() {
+ // Shrink 8 -> 4: old partition 5 and new partition 1 share keys.
+ let old5 = Stream {
+ topic: Arc::from("personhog_updates_old"),
+ partition: 5,
+ };
+ let index = DirtyIndex::new(1_000);
+ index.mark(
+ key(1),
+ DirtyMark {
+ partition: 5,
+ stream: old5.clone(),
+ ..mark(900, 5)
+ },
+ );
+ // The new home's warm replays the same record: equal offset, new partition.
+ index.mark(
+ key(1),
+ DirtyMark {
+ partition: 1,
+ stream: old5.clone(),
+ ..mark(900, 1)
+ },
+ );
+ assert_eq!(index.get(&key(1)).unwrap().partition, 1);
+
+ // Old-5's last prune and its release both leave the moved mark alone.
+ let (pruned, exhausted) = index.prune_chunk(5, &old5, 1_457);
+ assert_eq!((pruned.removed, exhausted), (0, true));
+ assert_eq!(index.clear_partition(5), 0);
+ assert_eq!(index.get(&key(1)).unwrap().partition, 1);
+
+ // And the new home reclaims it once the writer applies the old record.
+ let (pruned, _) = index.prune_chunk(1, &old5, 901);
+ assert_eq!(pruned.removed, 1);
+ assert!(index.get(&key(1)).is_none());
+
+ // The release may also come first, before any prune touched old-5.
+ let index = DirtyIndex::new(1_000);
+ for partition in [5, 1] {
+ index.mark(
+ key(1),
+ DirtyMark {
+ partition,
+ stream: old5.clone(),
+ ..mark(900, partition)
+ },
+ );
+ }
+ assert_eq!(index.clear_partition(5), 0);
+ assert_eq!(index.get(&key(1)).unwrap().partition, 1);
}
#[test]
diff --git a/rust/personhog-leader/src/cache/mod.rs b/rust/personhog-leader/src/cache/mod.rs
index 76c605f8169b..a872807dc829 100644
--- a/rust/personhog-leader/src/cache/mod.rs
+++ b/rust/personhog-leader/src/cache/mod.rs
@@ -4,6 +4,6 @@ mod partitioned;
mod persons;
mod stored;
-pub use dirty_index::{DirtyIndex, DirtyMark, PRUNE_CHUNK};
+pub use dirty_index::{DirtyIndex, DirtyMark, Stream, PRUNE_CHUNK};
pub use partitioned::{CacheLookup, PartitionedCache};
pub use persons::{CachedPerson, CachedVersion, PersonCache, PersonCacheKey};
diff --git a/rust/personhog-leader/src/cache/partitioned.rs b/rust/personhog-leader/src/cache/partitioned.rs
index c8c4090e700c..2f47f0e70645 100644
--- a/rust/personhog-leader/src/cache/partitioned.rs
+++ b/rust/personhog-leader/src/cache/partitioned.rs
@@ -101,15 +101,6 @@ impl PartitionedCache {
.put(key, person);
}
- /// Remove a key from a partition cache under construction. An applied
- /// death record erases its predecessors from the build this way.
- pub fn warm_remove(&self, partition: u32, key: &PersonCacheKey) {
- self.warming
- .get(&partition)
- .expect("warm_remove before begin_warm_partition")
- .remove(key);
- }
-
/// Whether the partition is published (serving reads). The prune
/// only settles published partitions' marks.
pub fn is_published(&self, partition: u32) -> bool {
diff --git a/rust/personhog-leader/src/config.rs b/rust/personhog-leader/src/config.rs
index 6a778dba4d2e..d1cf1d9d0bb5 100644
--- a/rust/personhog-leader/src/config.rs
+++ b/rust/personhog-leader/src/config.rs
@@ -149,14 +149,6 @@ pub struct Config {
#[envconfig(default = "personhog-writer")]
pub writer_consumer_group: String,
- /// How many offsets to rewind past the writer's committed offset when
- /// warming. Pure safety margin — any non-negative value is correct, but
- /// a larger value is more forgiving of momentary races between the
- /// writer's commit and our observation of it. Bounded above by Kafka's
- /// earliest available offset.
- #[envconfig(default = "1000")]
- pub warm_lookback_offsets: i64,
-
/// Timeout for the OffsetFetch round-trip that asks the writer's
/// consumer group for its committed offset.
#[envconfig(default = "5")]
@@ -451,54 +443,6 @@ impl Config {
(self.fencing_budget() / (FENCING_TXN_CALLS + 1)).max(MIN_TXN_TIMEOUT)
}
- /// The producer queue each fenced producer gets, in MiB.
- ///
- /// The shared producer's queue is sized for one client; the fenced
- /// path creates one producer *per owned partition*, so inheriting
- /// that figure multiplies it by the partition count. A lone survivor
- /// owning every partition would be entitled to the whole product,
- /// which for the deployed shape is several gigabytes against a pod
- /// sized for one — a client spreading large property updates across
- /// partitions while Kafka is slow could fill the independent queues
- /// and exhaust the leader.
- ///
- /// So the budget is the aggregate, divided. The floor keeps a
- /// high-partition-count deployment from starving any single producer
- /// below a workable depth; it trades the guarantee for a bound that
- /// is still far under the un-divided figure. Each share must still
- /// hold one window of records; raise the aggregate if it cannot.
- pub fn fencing_queue_mib(&self, partitions: u32) -> u32 {
- // The floor cannot be unconditional: above roughly fifty
- // partitions it would start multiplying again, and the aggregate
- // this exists to bound would scale with partition count exactly
- // as it did before. So it applies only while it stays inside the
- // aggregate, and past that point the division wins and each
- // producer gets a shallow queue — which is the honest answer for
- // a deployment with more partitions than a producer's worth of
- // memory to give them.
- // No floor. One was tried and was dead code: inside a guard that
- // only admits it when `MIN * partitions <= aggregate`, the
- // division already yields at least `MIN`, so the `max` never
- // fired. Applying it *outside* such a guard is worse — it
- // multiplies back up, which is the defect this division exists to
- // remove. The honest bound is the aggregate, and one MiB is the
- // smallest queue librdkafka will take.
- (self.kafka.kafka_producer_queue_mib / self.fenced_producers(partitions)).max(1)
- }
-
- /// The same division for the message-count limit, which bounds the
- /// queue independently of record size.
- pub fn fencing_queue_messages(&self, partitions: u32) -> u32 {
- (self.kafka.kafka_producer_queue_messages / self.fenced_producers(partitions)).max(1)
- }
-
- /// Producers the aggregate queue budget is divided among: one per
- /// lane per partition.
- fn fenced_producers(&self, partitions: u32) -> u32 {
- let lanes = u32::try_from(self.fencing_lanes.max(1)).unwrap_or(u32::MAX);
- partitions.max(1).saturating_mul(lanes)
- }
-
/// How long fence acquisition may take.
///
/// Deliberately not the transaction timeout. That one is sized so
@@ -1104,8 +1048,8 @@ mod fencing_timescale_tests {
let shared_messages = config.kafka.kafka_producer_queue_messages;
for partitions in [1, 8, 16, 50, 64, 256, 1024, 4096] {
- let per_partition_mib = config.fencing_queue_mib(partitions);
- let per_partition_messages = config.fencing_queue_messages(partitions);
+ let per_partition_mib = queue_mib(&config, partitions);
+ let per_partition_messages = queue_messages(&config, partitions);
assert!(
per_partition_mib <= shared_mib,
"partitions={partitions}: a single fenced producer must never be entitled \
@@ -1217,18 +1161,28 @@ mod fencing_timescale_tests {
}
}
+ fn queue_mib(config: &Config, partitions: u32) -> u32 {
+ let producers = crate::fencing::fenced_producer_count(partitions, config.fencing_lanes);
+ crate::fencing::queue_share(config.kafka.kafka_producer_queue_mib, producers)
+ }
+
+ fn queue_messages(config: &Config, partitions: u32) -> u32 {
+ let producers = crate::fencing::fenced_producer_count(partitions, config.fencing_lanes);
+ crate::fencing::queue_share(config.kafka.kafka_producer_queue_messages, producers)
+ }
+
#[test]
fn a_high_partition_count_still_leaves_a_workable_queue() {
let config = fenced(30);
// The budget is divided among every fenced producer, four lanes
// per partition at the default, and never rounds to a value
// librdkafka would reject.
- assert_eq!(config.fencing_queue_mib(16), 6);
- assert_eq!(config.fencing_queue_messages(16), 156_250);
- assert!(config.fencing_queue_mib(1024) >= 1);
- assert!(config.fencing_queue_messages(1024) >= 1);
+ assert_eq!(queue_mib(&config, 16), 6);
+ assert_eq!(queue_messages(&config, 16), 156_250);
+ assert!(queue_mib(&config, 1024) >= 1);
+ assert!(queue_messages(&config, 1024) >= 1);
assert!(
- config.fencing_queue_mib(0) >= 1,
+ queue_mib(&config, 0) >= 1,
"partitions=0 must not divide by zero"
);
}
diff --git a/rust/personhog-leader/src/coordination/mod.rs b/rust/personhog-leader/src/coordination/mod.rs
index f551de223726..4b76808cf369 100644
--- a/rust/personhog-leader/src/coordination/mod.rs
+++ b/rust/personhog-leader/src/coordination/mod.rs
@@ -2,10 +2,14 @@ use std::sync::Arc;
use std::time::Duration;
use async_trait::async_trait;
+use dashmap::DashMap;
use metrics::counter;
+use personhog_common::partitioning::partition_for_person;
use personhog_coordination::authority::AuthorityClock;
+use personhog_coordination::changelog::{warm_sources, LiveChangelog};
use personhog_coordination::error::{Error, Result};
use personhog_coordination::pod::HandoffHandler;
+use personhog_coordination::types::ChangelogConfig;
use tracing::{error, info};
use crate::cache::{DirtyIndex, PartitionedCache};
@@ -13,10 +17,39 @@ use crate::emitted::EmittedVersions;
use crate::fence::{drop_partition_fences, rebuild_partition_fences, FenceMap};
#[cfg(test)]
use crate::fencing::FencedProducerConfig;
-use crate::fencing::{heal_fence, FenceGuard, FencedChangelogProducers, HealOutcome};
+use crate::fencing::{
+ heal_fence, preregister_partition_fencing_metrics, FenceGuard, FencedChangelogProducers,
+ HealOutcome,
+};
use crate::inflight::InflightTracker;
use crate::pg::PgFallback;
-use crate::warming::{warm_from_kafka, WarmClientPools, WarmingConfig};
+use crate::warming::{warm_from_sources, WarmClientPools, WarmingConfig};
+
+/// The config a partition was acquired under.
+pub struct Capture {
+ pub config: ChangelogConfig,
+ /// The config's topic, shared so the write path names it without allocating.
+ pub topic: Arc<str>,
+}
+
+/// Per-partition captures, shared by the handler and the service.
+pub type ChangelogCaptures = Arc<DashMap<u32, Capture>>;
+
+/// Drops the partition's capture unless the warm that took it completes, so a
+/// failed or abandoned warm leaves no capture for a partition this pod does not serve.
+struct CaptureGuard<'a> {
+ captures: &'a ChangelogCaptures,
+ partition: u32,
+ armed: bool,
+}
+
+impl Drop for CaptureGuard<'_> {
+ fn drop(&mut self) {
+ if self.armed {
+ self.captures.remove(&self.partition);
+ }
+ }
+}
const DRAIN_POLL_INTERVAL: Duration = Duration::from_millis(50);
@@ -59,7 +92,6 @@ pub struct LeaderHandoffHandler {
/// The takeover scan's source. Without it (dev fixtures) fences are
/// not rebuilt on takeover and only FencePerson calls fill the map.
fence_scan: Option<PgFallback>,
- num_partitions: u32,
pools: Arc<WarmClientPools>,
/// Acquiring a partition initializes its transactional producer,
/// fencing every predecessor; releasing it drops the producer.
@@ -80,6 +112,8 @@ pub struct LeaderHandoffHandler {
/// under every write admitted in between — the pod fencing its own
/// live window.
freshly_fenced: Arc<dashmap::DashSet<u32>>,
+ captures: ChangelogCaptures,
+ changelog: LiveChangelog,
}
impl LeaderHandoffHandler {
@@ -91,7 +125,8 @@ impl LeaderHandoffHandler {
warming: WarmingConfig,
fences: FenceMap,
fence_scan: Option<PgFallback>,
- num_partitions: u32,
+ changelog: LiveChangelog,
+ captures: ChangelogCaptures,
pools: Arc<WarmClientPools>,
fenced: Arc<FencedChangelogProducers>,
authority: Arc<AuthorityClock>,
@@ -104,12 +139,21 @@ impl LeaderHandoffHandler {
warming,
fences,
fence_scan,
- num_partitions,
pools,
fenced,
authority,
emitted_versions,
freshly_fenced: Arc::new(dashmap::DashSet::new()),
+ captures,
+ changelog,
+ }
+ }
+
+ /// The capture, else the live config: a preconnect runs before the capture.
+ fn captured(&self, partition: u32) -> ChangelogConfig {
+ match self.captures.get(&partition) {
+ Some(captured) => captured.config.clone(),
+ None => self.changelog.current(),
}
}
@@ -201,7 +245,19 @@ impl HandoffHandler for LeaderHandoffHandler {
// a cancelled inbound handoff leaves no convergence behind, so
// the sweep is the only path that reaches its leftovers.
let fenced = Arc::clone(&self.fenced);
- tokio::spawn(async move { fenced.preconnect(partition).await });
+ let target = self.captured(partition).target();
+ tokio::spawn(async move { fenced.preconnect(partition, &target).await });
+ }
+
+ async fn capture_changelog(&self, partition: u32, config: &ChangelogConfig) {
+ preregister_partition_fencing_metrics(partition);
+ self.captures.insert(
+ partition,
+ Capture {
+ topic: Arc::from(config.topic.as_str()),
+ config: config.clone(),
+ },
+ );
}
async fn drain_partition_inflight(&self, partition: u32) -> Result<()> {
@@ -238,15 +294,21 @@ impl HandoffHandler for LeaderHandoffHandler {
async fn warm_partition(&self, partition: u32) -> Result<()> {
// The takeover scan: rebuild the partition's lifecycle fences from
// the live marks BEFORE warming — the partition becomes servable
- // the moment `warm_from_kafka` installs it in the cache, and a
+ // the moment the warm installs it in the cache, and a
// fenced person must never be writable in that gap. A mark
// committed after this read arrives as a FencePerson call to this
// (now current) owner, so the two sources cover every mark; a
// fence installed for a partition that is not yet serving is
// harmless.
+ let config = self.captured(partition);
+ let mut capture_guard = CaptureGuard {
+ captures: &self.captures,
+ partition,
+ armed: true,
+ };
if let Some(fallback) = &self.fence_scan {
let installed =
- rebuild_partition_fences(fallback, &self.fences, partition, self.num_partitions)
+ rebuild_partition_fences(fallback, &self.fences, partition, config.partitions)
.await
.map_err(|e| personhog_coordination::error::Error::HandoffFailed {
partition,
@@ -276,20 +338,22 @@ impl HandoffHandler for LeaderHandoffHandler {
// which is what lets the read complete; the admitted writes fail
// as fenced and their versions stay spent.
self.fenced
- .acquire(partition)
+ .acquire(partition, &config.target())
.await
.map_err(Error::invalid_state)?;
self.check_authority_after_acquire(partition, "warm")?;
- info!(partition, "changelog fence acquired");
+ info!(partition, topic = %config.topic, "changelog fence acquired");
// The guard gives the epoch back if the warm fails or never
// returns, rather than holding a partition this pod lost.
let fence_guard = FenceGuard::new(Arc::clone(&self.fenced), partition, "warm");
- warm_from_kafka(
+ let sources = warm_sources(&config, partition);
+ warm_from_sources(
&self.warming,
&self.pools,
&self.cache,
&self.dirty_index,
partition,
+ &sources,
)
.await?;
// This pod may still carry a fence from a previous ownership of
@@ -297,6 +361,7 @@ impl HandoffHandler for LeaderHandoffHandler {
// ownership through a fresh warm re-admits writes.
self.inflight.unfence(partition);
fence_guard.keep();
+ capture_guard.armed = false;
self.freshly_fenced.insert(partition);
info!(partition, "partition warmed");
Ok(())
@@ -316,7 +381,15 @@ impl HandoffHandler for LeaderHandoffHandler {
/// applied work, so a repairing pod's budgets reset like any other
/// progress.
async fn verify_serving(&self, partition: u32) -> Result<bool> {
- match heal_fence(&self.fenced, &self.inflight, &self.authority, partition).await {
+ match heal_fence(
+ &self.fenced,
+ &self.inflight,
+ &self.authority,
+ partition,
+ &self.captured(partition).target(),
+ )
+ .await
+ {
Ok(HealOutcome::Healed) => {
// The epoch just moved. Mark it like every other
// acquisition site, so the resume step of this same
@@ -342,8 +415,17 @@ impl HandoffHandler for LeaderHandoffHandler {
// would only pin memory for a partition this pod no longer serves.
self.dirty_index.clear_partition(partition);
// Same for the lifecycle fences: the new owner's takeover scan
- // rebuilds its own.
- drop_partition_fences(&self.fences, partition, self.num_partitions);
+ // rebuilds its own. A key homed on another capture under that
+ // capture's modulus belongs to that partition, not this one.
+ let modulus = self.captured(partition).partitions;
+ drop_partition_fences(&self.fences, partition, modulus, |key| {
+ self.captures.iter().any(|capture| {
+ *capture.key() != partition
+ && partition_for_person(key.team_id, key.person_id, capture.config.partitions)
+ == *capture.key()
+ })
+ });
+ self.captures.remove(&partition);
// The incoming owner derives versions from the changelog, which
// is the authority these floors stood in for; carrying them would
// only constrain a partition this pod no longer serves.
@@ -377,7 +459,7 @@ impl HandoffHandler for LeaderHandoffHandler {
return Ok(());
}
self.fenced
- .acquire(partition)
+ .acquire(partition, &self.captured(partition).target())
.await
.map_err(Error::invalid_state)?;
self.check_authority_after_acquire(partition, "resume")?;
@@ -431,10 +513,8 @@ mod tests {
Arc::new(DirtyIndex::new(1_000_000)),
WarmingConfig {
kafka,
- topic: "personhog_updates".to_string(),
pod_name: "test".to_string(),
writer_consumer_group: "personhog-writer".to_string(),
- lookback_offsets: 0,
committed_offsets_timeout: Duration::from_secs(5),
fetch_watermarks_timeout: Duration::from_secs(5),
recv_timeout: Duration::from_secs(10),
@@ -446,13 +526,13 @@ mod tests {
},
Arc::new(dashmap::DashMap::new()),
None,
- 4,
+ LiveChangelog::new(ChangelogConfig::new("personhog_updates", 4)),
+ Arc::new(DashMap::new()),
pools,
// Lazily built: these tests never acquire a partition, so
// nothing here reaches a broker.
Arc::new(FencedChangelogProducers::new(FencedProducerConfig {
kafka: fencing_kafka,
- topic: "personhog_updates".to_string(),
init_timeout: Duration::from_secs(1),
commit_timeout: Duration::from_secs(1),
broker_txn_timeout: Duration::from_secs(10),
@@ -508,6 +588,86 @@ mod tests {
);
}
+ /// A warm that does not complete leaves no capture behind.
+ #[tokio::test]
+ async fn a_failed_warm_drops_its_capture() {
+ let mut handler = handler();
+ // Unclaimed: the warm refuses before it touches the broker.
+ handler.authority = Arc::new(AuthorityClock::unclaimed());
+ handler
+ .capture_changelog(0, &ChangelogConfig::new("personhog_updates", 4))
+ .await;
+ assert!(handler.warm_partition(0).await.is_err());
+ assert!(!handler.captures.contains_key(&0));
+ }
+
+ /// After a shrink, old-5 and new-1 share keys on this pod, and the
+ /// release of old-5 may run after new-1 warmed.
+ #[tokio::test]
+ async fn releasing_an_old_home_keeps_the_new_homes_marks_and_fences() {
+ use crate::cache::{DirtyMark, PersonCacheKey, Stream};
+ use crate::fence::FenceState;
+ use personhog_proto::personhog::types::v1::LifecycleOpType;
+
+ let handler = handler();
+ let config = |topic: &str, partitions: u32, epoch: u64| ChangelogConfig {
+ topic: topic.to_string(),
+ partitions,
+ epoch,
+ writer_group: String::new(),
+ draining: Vec::new(),
+ };
+ for (partition, cfg) in [(5, config("old", 8, 1)), (1, config("new", 4, 2))] {
+ handler.cache.create_partition(partition);
+ handler.captures.insert(
+ partition,
+ Capture {
+ topic: Arc::from(cfg.topic.as_str()),
+ config: cfg,
+ },
+ );
+ }
+ let key = (1..)
+ .map(|person_id| PersonCacheKey {
+ team_id: 1,
+ person_id,
+ })
+ .find(|k| partition_for_person(k.team_id, k.person_id, 8) == 5)
+ .unwrap();
+ let old5 = Stream {
+ topic: Arc::from("old"),
+ partition: 5,
+ };
+ for partition in [5, 1] {
+ handler.dirty_index.mark(
+ key.clone(),
+ DirtyMark {
+ version: 1,
+ offset: 900,
+ partition,
+ stream: old5.clone(),
+ is_deleted: false,
+ },
+ );
+ }
+ handler.fences.insert(
+ key.clone(),
+ FenceState {
+ op_id: uuid::Uuid::new_v4(),
+ op_type: LifecycleOpType::from_op_type_str("delete"),
+ sealed_at: None,
+ },
+ );
+
+ handler.release_partition(5).await.unwrap();
+
+ assert_eq!(handler.dirty_index.get(&key).map(|m| m.partition), Some(1));
+ assert!(
+ handler.fences.contains_key(&key),
+ "new-1's fence must survive old-5's release"
+ );
+ }
+
#[tokio::test]
async fn release_partition_is_idempotent_for_unknown_partition() {
let handler = handler();
diff --git a/rust/personhog-leader/src/fence.rs b/rust/personhog-leader/src/fence.rs
index 7167fef2e589..d7173b218ae8 100644
--- a/rust/personhog-leader/src/fence.rs
+++ b/rust/personhog-leader/src/fence.rs
@@ -172,7 +172,7 @@ pub async fn rebuild_partition_fences(
// Converge rather than accumulate: this partition's fences are
// exactly what the marks say, not that plus whatever a previous warm
// left behind.
- drop_partition_fences(fences, partition, num_partitions);
+ drop_partition_fences(fences, partition, num_partitions, |_| false);
let mut installed = 0usize;
for row in rows {
@@ -200,15 +200,17 @@ pub async fn rebuild_partition_fences(
Ok(installed)
}
-/// Drop every fence belonging to `partition` — the counterpart of
-/// [`rebuild_partition_fences`] for partition release. The new owner
-/// rebuilds its own; stale entries here would only pin memory for persons
-/// this pod no longer serves (misrouted requests are already rejected by
-/// partition validation).
-pub fn drop_partition_fences(fences: &FenceMap, partition: u32, num_partitions: u32) -> usize {
+/// Drop the fences homed on `partition`, except keys `keep` claims for another
+/// partition this pod serves. The new owner rebuilds its own.
+pub fn drop_partition_fences(
+ fences: &FenceMap,
+ partition: u32,
+ num_partitions: u32,
+ keep: impl Fn(&PersonCacheKey) -> bool,
+) -> usize {
let before = fences.len();
fences.retain(|key, _| {
- partition_for_person(key.team_id, key.person_id, num_partitions) != partition
+ partition_for_person(key.team_id, key.person_id, num_partitions) != partition || keep(key)
});
gauge!("personhog_leader_fences_active").set(fences.len() as f64);
before - fences.len()
@@ -711,3 +713,46 @@ mod mark_verifier_tests {
);
}
}
+
+#[cfg(test)]
+mod partition_fence_tests {
+ use super::*;
+
+ #[test]
+ fn a_drop_spares_the_keys_another_home_still_claims() {
+ let fences: FenceMap = Arc::new(DashMap::new());
+ let homed_on = |partition: u32| {
+ (1..)
+ .map(|person_id| PersonCacheKey {
+ team_id: 1,
+ person_id,
+ })
+ .filter(move |k| partition_for_person(k.team_id, k.person_id, 8) == partition)
+ };
+ let [kept, dropped, elsewhere] = {
+ let mut on_five = homed_on(5);
+ [
+ on_five.next().unwrap(),
+ on_five.next().unwrap(),
+ homed_on(6).next().unwrap(),
+ ]
+ };
+ for key in [&kept, &dropped, &elsewhere] {
+ fences.insert(
+ key.clone(),
+ FenceState {
+ op_id: Uuid::now_v7(),
+ op_type: LifecycleOpType::Delete,
+ sealed_at: None,
+ },
+ );
+ }
+
+ let removed = drop_partition_fences(&fences, 5, 8, |key| *key == kept);
+
+ assert_eq!(removed, 1);
+ assert!(fences.contains_key(&kept));
+ assert!(!fences.contains_key(&dropped));
+ assert!(fences.contains_key(&elsewhere));
+ }
+}
diff --git a/rust/personhog-leader/src/fencing.rs b/rust/personhog-leader/src/fencing.rs
index cfb8b08faa3f..83a0f394cab7 100644
--- a/rust/personhog-leader/src/fencing.rs
+++ b/rust/personhog-leader/src/fencing.rs
@@ -38,6 +38,7 @@ use std::{fmt, mem};
use common_kafka::config::KafkaConfig;
use common_kafka::transaction::{ConnectedTransactionalProducer, TransactionalProducer};
+use personhog_coordination::types::ChangelogTarget;
use crate::producer_stats::FencedProducerContext;
@@ -164,6 +165,17 @@ impl Gate {
}
}
+/// Producers a queue budget is divided among: one per lane per partition.
+pub fn fenced_producer_count(partitions: u32, lanes: usize) -> u32 {
+ let lanes = u32::try_from(lanes.max(1)).unwrap_or(u32::MAX);
+ partitions.max(1).saturating_mul(lanes)
+}
+
+/// One producer's share of an aggregate queue limit, never below 1.
+pub fn queue_share(aggregate: u32, producers: u32) -> u32 {
+ (aggregate / producers.max(1)).max(1)
+}
+
struct Dial {
kafka: KafkaConfig,
topic: String,
@@ -361,6 +373,7 @@ impl PartitionFence {
/// producer's next transaction for a while after it commits, so a write
/// takes the lane least likely to be held.
struct PartitionLanes {
+ topic: String,
lanes: Vec<Arc<PartitionFence>>,
/// Serializes lane decisions: a writer holds it from reading the gates
/// to moving one, and a committer while it clears `committing`, so a
@@ -662,15 +675,18 @@ enum Prepared {
Connecting { token: u64, since: Instant },
/// A finished dial, one connection per lane, parked for the next
/// acquire to consume.
- Ready(Vec<ConnectedFencedProducer>),
+ Ready {
+ topic: String,
+ connected: Vec<ConnectedFencedProducer>,
+ },
}
/// Per-partition fenced producers for the changelog. Constructed once
/// and shared; partitions are acquired at warm completion and released
/// with ownership.
pub struct FencedChangelogProducers {
+ /// Queue limits here are the aggregate every producer shares (see `queue_share`).
kafka: KafkaConfig,
- topic: String,
init_timeout: Duration,
commit_timeout: Duration,
/// How long the broker keeps one of our windows open before
@@ -733,7 +749,6 @@ pub struct FencedChangelogProducers {
/// every fenced write.
pub struct FencedProducerConfig {
pub kafka: KafkaConfig,
- pub topic: String,
pub init_timeout: Duration,
pub commit_timeout: Duration,
pub broker_txn_timeout: Duration,
@@ -747,7 +762,6 @@ impl FencedChangelogProducers {
pub fn new(config: FencedProducerConfig) -> Self {
let FencedProducerConfig {
kafka,
- topic,
init_timeout,
commit_timeout,
broker_txn_timeout,
@@ -762,7 +776,6 @@ impl FencedChangelogProducers {
let produce_bound = (window + commit_timeout * (COMMIT_RETRIES as u32 + 2)) * 2;
Self {
kafka,
- topic,
init_timeout,
commit_timeout,
broker_txn_timeout,
@@ -795,24 +808,25 @@ impl FencedChangelogProducers {
/// every previous owner of them. The claims run as one blocking task
/// and their producers travel back in a guard, so a cancelled acquire
/// never destroys a producer on a runtime worker.
- async fn acquire_installed(&self, partition: u32) -> Result<Arc<PartitionLanes>, String> {
+ async fn acquire_installed(
+ &self,
+ partition: u32,
+ target: &ChangelogTarget,
+ ) -> Result<Arc<PartitionLanes>, String> {
+ let topic = target.topic.as_str();
let start = Instant::now();
// The removal also clears a Connecting claim: acquisition is
// happening now, so a dial still in flight is too late to help,
// and losing its claim makes it discard on completion instead
// of parking a connection nothing will consume.
- let mut parked: Vec<Option<ConnectedFencedProducer>> =
- match self.prepared.remove(&partition) {
- Some((_, Prepared::Ready(parked))) => parked.into_iter().map(Some).collect(),
- _ => Vec::new(),
- };
+ let mut parked = self.take_prepared(partition, topic);
let path = if parked.is_empty() {
"cold"
} else {
"prepared"
};
parked.resize_with(self.lanes, || None);
- let dial = self.dial(partition);
+ let dial = self.dial(partition, target);
let claimed = spawn_blocking(move || {
let mut producers = Vec::with_capacity(parked.len());
let mut failure = None;
@@ -885,7 +899,11 @@ impl FencedChangelogProducers {
})
})
.collect();
- let installed = Arc::new(PartitionLanes { lanes, turn });
+ let installed = Arc::new(PartitionLanes {
+ topic: topic.to_string(),
+ lanes,
+ turn,
+ });
if let Some(replaced) = self.partitions.insert(partition, Arc::clone(&installed)) {
// The heal path installs over a still-present condemned
// fence, and by then the commit task has usually dropped its
@@ -897,10 +915,22 @@ impl FencedChangelogProducers {
Ok(installed)
}
- fn dial(&self, partition: u32) -> Dial {
+ /// Each producer's queue is its share of the aggregate under the generation's partition count.
+ fn dial(&self, partition: u32, target: &ChangelogTarget) -> Dial {
+ let producers = fenced_producer_count(target.partitions, self.lanes);
Dial {
- kafka: self.kafka.clone(),
- topic: self.topic.clone(),
+ kafka: KafkaConfig {
+ kafka_producer_queue_mib: queue_share(
+ self.kafka.kafka_producer_queue_mib,
+ producers,
+ ),
+ kafka_producer_queue_messages: queue_share(
+ self.kafka.kafka_producer_queue_messages,
+ producers,
+ ),
+ ..self.kafka.clone()
+ },
+ topic: target.topic.clone(),
partition,
timeout: self.init_timeout,
broker_txn_timeout: self.broker_txn_timeout,
@@ -909,10 +939,32 @@ impl FencedChangelogProducers {
}
}
+ /// The partition's parked connections for `topic`, removing its slot; one
+ /// parked for another topic is discarded.
+ fn take_prepared(&self, partition: u32, topic: &str) -> Vec<Option<ConnectedFencedProducer>> {
+ match self.prepared.remove(&partition) {
+ Some((
+ _,
+ Prepared::Ready {
+ topic: parked_topic,
+ connected,
+ },
+ )) if parked_topic == topic => connected.into_iter().map(Some).collect(),
+ Some((_, Prepared::Ready { connected, .. })) => {
+ counter!("personhog_leader_fence_preconnect_total", "outcome" => "stale_topic")
+ .increment(1);
+ spawn_blocking(move || drop(connected));
+ Vec::new()
+ }
+ _ => Vec::new(),
+ }
+ }
+
/// Take the partition's fence, discarding the handle. The caller
- /// relies on the map rather than on holding the fence itself.
- pub async fn acquire(&self, partition: u32) -> Result<(), String> {
- self.acquire_installed(partition).await.map(|_| ())
+ /// relies on the map rather than on holding the fence itself. Ids are
+ /// topic-scoped: a new generation fences nothing on the old one.
+ pub async fn acquire(&self, partition: u32, target: &ChangelogTarget) -> Result<(), String> {
+ self.acquire_installed(partition, target).await.map(|_| ())
}
/// Mark the installed lanes as about to be fenced by this pod's own
@@ -953,7 +1005,7 @@ impl FencedChangelogProducers {
/// Single-flight per partition: the slot is claimed before the dial
/// begins, so however often convergence fires this through a drain
/// window, one dial runs and the rest return as coalesced.
- pub async fn preconnect(&self, partition: u32) {
+ pub async fn preconnect(&self, partition: u32, target: &ChangelogTarget) {
// The claim must precede the dial, and the entry guard must not
// be held across it: the guard holds a shard lock.
let token = self.dial_token.fetch_add(1, Ordering::Relaxed);
@@ -973,7 +1025,7 @@ impl FencedChangelogProducers {
}
}
let start = Instant::now();
- let dial = self.dial(partition);
+ let dial = self.dial(partition, target);
let lanes = self.lanes;
let dialed = spawn_blocking(move || {
// All or nothing: each connection carries its own lane's id.
@@ -990,7 +1042,7 @@ impl FencedChangelogProducers {
counter!("personhog_leader_fence_preconnect_total", "outcome" => "ok").increment(1);
histogram!("personhog_leader_fence_preconnect_ms")
.record(start.elapsed().as_secs_f64() * 1000.0);
- self.park(partition, token, connected);
+ self.park(partition, token, &target.topic, connected);
}
Err(e) => {
counter!("personhog_leader_fence_preconnect_total", "outcome" => "error")
@@ -1014,11 +1066,20 @@ impl FencedChangelogProducers {
/// finding a replacement dial's claim here must not usurp it, or
/// the replacement's own park would discard the newer connection
/// while later convergence passes see whatever raced in.
- fn park(&self, partition: u32, token: u64, connected: Vec<ConnectedFencedProducer>) {
+ fn park(
+ &self,
+ partition: u32,
+ token: u64,
+ topic: &str,
+ connected: Vec<ConnectedFencedProducer>,
+ ) {
match self.prepared.entry(partition) {
Entry::Occupied(mut slot) if matches!(slot.get(), Prepared::Connecting { token: t, .. } if *t == token) =>
{
- slot.insert(Prepared::Ready(connected));
+ slot.insert(Prepared::Ready {
+ topic: topic.to_string(),
+ connected,
+ });
}
_ => {
counter!("personhog_leader_fence_preconnect_total", "outcome" => "discarded")
@@ -1033,7 +1094,7 @@ impl FencedChangelogProducers {
/// claim is removed without a drop: its dial discards on
/// completion once the claim is gone.
fn discard_prepared(&self, partition: u32) {
- if let Some((_, Prepared::Ready(connected))) = self.prepared.remove(&partition) {
+ if let Some((_, Prepared::Ready { connected, .. })) = self.prepared.remove(&partition) {
spawn_blocking(move || drop(connected));
}
}
@@ -1061,12 +1122,12 @@ impl FencedChangelogProducers {
// dial that parks between the scan and this removal is not
// swept as a stale claim.
match self.prepared.remove_if(&partition, |_, slot| match slot {
- Prepared::Ready(_) => true,
+ Prepared::Ready { .. } => true,
Prepared::Connecting { since, .. } => {
now.duration_since(*since) > stale_claim_bound
}
}) {
- Some((_, Prepared::Ready(connected))) => {
+ Some((_, Prepared::Ready { connected, .. })) => {
counter!("personhog_leader_fence_preconnect_total", "outcome" => "swept")
.increment(1);
warn!(
@@ -1090,7 +1151,7 @@ impl FencedChangelogProducers {
pub fn has_prepared(&self, partition: u32) -> bool {
self.prepared
.get(&partition)
- .is_some_and(|slot| matches!(*slot, Prepared::Ready(_)))
+ .is_some_and(|slot| matches!(*slot, Prepared::Ready { .. }))
}
/// How many dials have been attempted.
@@ -1547,7 +1608,7 @@ impl FencedChangelogProducers {
if let Some(fill) = opened {
let fence_for_commit = Arc::clone(&fence);
let window = self.window;
- let topic = self.topic.clone();
+ let topic = lanes.topic.clone();
let part = partition;
tokio::spawn(async move {
commit_window_after(fence_for_commit, window, fill, topic, part).await;
@@ -1563,7 +1624,7 @@ impl FencedChangelogProducers {
// fencing rollout. Recorded before the send so a payload rejected
// by the broker (message.max.bytes) still shows up.
histogram!("personhog_leader_kafka_produce_bytes").record(payload.len() as f64);
- let record = FutureRecord::to(&self.topic)
+ let record = FutureRecord::to(&lanes.topic)
.partition(partition as i32)
.key(&key)
.payload(&payload);
@@ -2120,7 +2181,7 @@ async fn commit_window_after(
/// first-increment lazy registration — fence acquisition in particular
/// fires exactly during deploys. A touched histogram renders with zero
/// count until its first sample.
-pub fn preregister_fencing_metrics(partitions: u32) {
+pub fn preregister_fencing_metrics() {
for outcome in ["ok", "error"] {
counter!("personhog_leader_fence_init_total", "outcome" => outcome).increment(0);
}
@@ -2166,38 +2227,39 @@ pub fn preregister_fencing_metrics(partitions: u32) {
counter!("personhog_coordination_repair_passes_total", "outcome" => outcome).increment(0);
}
counter!("personhog_leader_kafka_produce_errors_total").increment(0);
- for partition in 0..partitions {
- let p = partition.to_string();
- counter!("personhog_leader_fence_aborts_total", "partition" => p.clone()).increment(0);
- counter!(
- "personhog_leader_fenced_partition_drops_total",
- "partition" => p.clone()
- )
- .increment(0);
- counter!(
- "personhog_leader_fence_heal_failures_total",
- "partition" => p.clone()
- )
- .increment(0);
+ counter!("personhog_leader_fence_produce_bound_exceeded_total").increment(0);
+ // Histograms are deliberately absent: the Prometheus exporter
+ // renders one only once it has a sample, so there is nothing a
+ // startup call can materialize.
+}
+/// Registers one partition's fencing series when it is captured.
+pub fn preregister_partition_fencing_metrics(partition: u32) {
+ let p = partition.to_string();
+ counter!("personhog_leader_fence_aborts_total", "partition" => p.clone()).increment(0);
+ counter!(
+ "personhog_leader_fenced_partition_drops_total",
+ "partition" => p.clone()
+ )
+ .increment(0);
+ counter!(
+ "personhog_leader_fence_heal_failures_total",
+ "partition" => p.clone()
+ )
+ .increment(0);
+ counter!(
+ "personhog_leader_fence_commit_indeterminate_total",
+ "partition" => p.clone()
+ )
+ .increment(0);
+ for cause in ["takeover", "heal"] {
counter!(
- "personhog_leader_fence_commit_indeterminate_total",
- "partition" => p.clone()
+ "personhog_leader_produce_fenced_total",
+ "partition" => p.clone(),
+ "cause" => cause
)
.increment(0);
- for cause in ["takeover", "heal"] {
- counter!(
- "personhog_leader_produce_fenced_total",
- "partition" => p.clone(),
- "cause" => cause
- )
- .increment(0);
- }
}
- counter!("personhog_leader_fence_produce_bound_exceeded_total").increment(0);
- // Histograms are deliberately absent: the Prometheus exporter
- // renders one only once it has a sample, so there is nothing a
- // startup call can materialize.
}
/// `FencedProduceError` holds a String and cannot derive Clone cheaply
@@ -2256,13 +2318,14 @@ pub async fn heal_fence(
inflight: &InflightTracker,
authority: &AuthorityClock,
partition: u32,
+ target: &ChangelogTarget,
) -> Result<HealOutcome, String> {
let lost_standing = !authority.is_valid();
if lost_standing || fenced.holds(partition) || inflight.is_fenced(partition) {
return Ok(HealOutcome::Intact);
}
fenced.expect_self_fence(partition);
- let taken = match fenced.acquire_installed(partition).await {
+ let taken = match fenced.acquire_installed(partition, target).await {
Ok(taken) => taken,
Err(e) => {
// Partition-labeled: a wedged partition must be attributable
@@ -2318,6 +2381,39 @@ pub async fn heal_fence(
#[cfg(test)]
mod tests {
use super::*;
+ use envconfig::Envconfig;
+
+ #[test]
+ fn a_producer_queue_is_its_share_under_the_generation_it_dials() {
+ let mut kafka = KafkaConfig::init_from_hashmap(&std::collections::HashMap::new()).unwrap();
+ kafka.kafka_producer_queue_mib = 1024;
+ kafka.kafka_producer_queue_messages = 64_000;
+ let producers = FencedChangelogProducers::new(FencedProducerConfig {
+ kafka,
+ init_timeout: Duration::from_secs(1),
+ commit_timeout: Duration::from_secs(1),
+ broker_txn_timeout: Duration::from_secs(10),
+ window: Duration::from_millis(5),
+ window_max_writes: 32,
+ settle_budget: Duration::from_secs(1),
+ lanes: 2,
+ });
+ for (partitions, mib, messages) in [(8, 64, 4_000), (16, 32, 2_000)] {
+ let target = ChangelogTarget {
+ topic: format!("g{partitions}"),
+ partitions,
+ };
+ let dial = producers.dial(0, &target);
+ assert_eq!(
+ (
+ dial.kafka.kafka_producer_queue_mib,
+ dial.kafka.kafka_producer_queue_messages
+ ),
+ (mib, messages),
+ "{partitions} partitions"
+ );
+ }
+ }
/// The verdict the caller acts on, as a truth table over the two
/// independent facts it combines.
diff --git a/rust/personhog-leader/src/main.rs b/rust/personhog-leader/src/main.rs
index 34a7d66222e9..2a36dbfac708 100644
--- a/rust/personhog-leader/src/main.rs
+++ b/rust/personhog-leader/src/main.rs
@@ -1,3 +1,4 @@
+use std::collections::HashMap;
use std::sync::Arc;
use std::time::Duration;
@@ -14,7 +15,9 @@ use personhog_common::async_gzip::{AsyncGzipConfig, AsyncGzipLayer};
use personhog_common::grpc::{tracked_tcp_incoming, GrpcLoadShedLayer, GrpcMetricsLayer};
use personhog_common::metrics::{WARM_LATENCY_BUCKETS_MS, WRITE_PATH_LATENCY_BUCKETS_MS};
use personhog_coordination::authority::AuthorityClock;
-use personhog_coordination::changelog::{load_changelog_config, CHANGELOG_BOOT_POLL_INTERVAL};
+use personhog_coordination::changelog::{
+ keep_changelog_live, load_changelog_config, LiveChangelog, CHANGELOG_BOOT_POLL_INTERVAL,
+};
use personhog_coordination::pod::{PodConfig, PodHandle};
use personhog_coordination::store::PersonhogStore;
use personhog_proto::personhog::leader::v1::person_hog_leader_server::PersonHogLeaderServer;
@@ -32,7 +35,7 @@ use tracing_subscriber::EnvFilter;
use metrics::{counter, gauge};
use personhog_leader::cache::{DirtyIndex, PartitionedCache, PersonCacheKey};
use personhog_leader::config::Config;
-use personhog_leader::coordination::LeaderHandoffHandler;
+use personhog_leader::coordination::{ChangelogCaptures, LeaderHandoffHandler};
use personhog_leader::fencing::{
preregister_fencing_metrics, FencedChangelogProducers, FencedProducerConfig,
};
@@ -143,6 +146,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
"authority-metrics",
ComponentOptions::new().is_observability(true),
);
+ let changelog_handle = manager.register("changelog-config", ComponentOptions::new());
let readiness = manager.readiness_handler();
let liveness = manager.liveness_handler();
@@ -377,8 +381,22 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
epoch = loaded.epoch,
"loaded changelog config"
);
+ let changelog = LiveChangelog::new(loaded);
+ {
+ let store = Arc::clone(&store);
+ let changelog = changelog.clone();
+ tokio::spawn(async move {
+ let _guard = changelog_handle.process_scope();
+ let cancel = CancellationToken::new();
+ tokio::select! {
+ _ = changelog_handle.shutdown_signal() => cancel.cancel(),
+ _ = keep_changelog_live(store, changelog, cancel.clone()) => {}
+ }
+ });
+ }
let locks = Arc::new(DashMap::new());
+ let captures: ChangelogCaptures = Arc::new(DashMap::new());
let fences: personhog_leader::fence::FenceMap = Arc::new(DashMap::new());
let inflight = Arc::new(InflightTracker::new());
let dirty_index = Arc::new(DirtyIndex::new(config.dirty_index_max_entries));
@@ -392,7 +410,6 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
let recovery = Arc::new(
ChangelogRecovery::new(RecoveryConfig {
kafka: config.kafka.clone(),
- topic: changelog_topic.clone(),
pod_name: config.pod_name.clone(),
recv_timeout: Duration::from_secs(config.recovery_recv_timeout_secs),
pool_size: config.recovery_pool_size,
@@ -414,7 +431,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
lease_runway_ms = config.lease_fence_runway().as_millis(),
"producing the changelog through per-partition transactional producers"
);
- preregister_fencing_metrics(num_partitions);
+ preregister_fencing_metrics();
// A condemned producer's repair otherwise waits for the next
// reconcile tick; this nudge lets the condemnation itself
// trigger the repair pass that heals it.
@@ -423,17 +440,11 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
// shared one: its writes must resolve inside the lease runway.
let fencing_kafka = common_kafka::config::KafkaConfig {
kafka_message_timeout_ms: config.fencing_message_timeout().as_millis() as u32,
- // One producer per lane per owned partition, so the shared
- // producer's queue limits are an aggregate to divide rather
- // than a per-producer figure to copy.
- kafka_producer_queue_mib: config.fencing_queue_mib(num_partitions),
- kafka_producer_queue_messages: config.fencing_queue_messages(num_partitions),
..config.kafka.clone()
};
let fenced = Arc::new(
FencedChangelogProducers::new(FencedProducerConfig {
kafka: fencing_kafka,
- topic: changelog_topic.clone(),
init_timeout: config.fencing_init_timeout(),
commit_timeout: config.fencing_txn_timeout(),
broker_txn_timeout: config.fencing_broker_txn_timeout(),
@@ -497,7 +508,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
fallback,
Arc::clone(&locks),
Arc::clone(&inflight),
- num_partitions,
+ Arc::clone(&captures),
Arc::clone(&dirty_index),
Arc::clone(&recovery),
PropertySizeLimits::new(
@@ -523,10 +534,8 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
Arc::clone(&dirty_index),
WarmingConfig {
kafka: config.kafka.clone(),
- topic: changelog_topic.clone(),
pod_name: config.pod_name.clone(),
writer_consumer_group: config.writer_consumer_group.clone(),
- lookback_offsets: config.warm_lookback_offsets,
committed_offsets_timeout: Duration::from_secs(
config.warm_committed_offsets_timeout_secs,
),
@@ -542,7 +551,8 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
},
Arc::clone(&fences),
fence_scan,
- num_partitions,
+ changelog.clone(),
+ Arc::clone(&captures),
Arc::clone(&warm_pools),
Arc::clone(&fenced),
Arc::clone(&authority),
@@ -670,7 +680,6 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
Arc::clone(&cache),
Arc::clone(&locks),
Arc::clone(&warm_pools),
- changelog_topic.clone(),
Duration::from_secs(config.warm_committed_offsets_timeout_secs),
Duration::from_secs(config.dirty_index_prune_interval_secs.max(1)),
));
@@ -768,48 +777,34 @@ async fn run_dirty_index_prune_loop(
cache: Arc<PartitionedCache>,
locks: Arc<DashMap<PersonCacheKey, Arc<tokio::sync::Mutex<()>>>>,
pools: Arc<WarmClientPools>,
- topic: String,
offsets_timeout: Duration,
prune_interval: Duration,
) {
let mut interval = tokio::time::interval(prune_interval);
loop {
interval.tick().await;
- // One batched OffsetFetch, then queue pops proportional to the
- // marks actually reclaimed — a tick never scans the index, which
- // is what makes the 1s interval affordable even when a lagging
- // writer has made the index large.
gauge!("personhog_leader_dirty_index_size").set(dirty_index.len() as f64);
gauge!("personhog_leader_dirty_index_max_entries").set(dirty_index.max_entries() as f64);
gauge!("personhog_leader_cache_weight_bytes").set(cache.usage_bytes() as f64);
- let Some((partitions, committed_offsets)) = prune_and_settle_tick(
+ let lags = prune_and_settle_tick(
&dirty_index,
&cache,
&locks,
&pools.offsets,
- &topic,
offsets_timeout,
)
- .await
- else {
- continue;
- };
- if partitions.is_empty() {
- continue;
- }
- // A partition absent from the committed offsets has no writer
- // commit yet: nothing is applied, every mark stays, and its lag
- // is the entire marked backlog. A partition the prune fully
- // reclaimed has no live marks left — the writer caught up, so its
- // lag reads zero.
- for partition in &partitions {
- let lag = match dirty_index.max_offset(*partition) {
- Some(max_offset) => {
- let committed = committed_offsets.get(partition).copied().unwrap_or(0);
- (max_offset + 1 - committed).max(0)
- }
+ .await;
+ // A partition's lag sums the streams feeding it; a stream the writer never
+ // committed on lags by its whole marked backlog, and a reclaimed one by zero.
+ let mut per_partition: HashMap<u32, i64> = HashMap::new();
+ for lag in &lags {
+ let outstanding = match dirty_index.max_offset(lag.partition, &lag.stream) {
+ Some(max_offset) => (max_offset + 1 - lag.committed.unwrap_or(0)).max(0),
None => 0,
};
+ *per_partition.entry(lag.partition).or_default() += outstanding;
+ }
+ for (partition, lag) in per_partition {
gauge!(
"personhog_leader_writer_uncommitted_offsets",
"partition" => partition.to_string()
diff --git a/rust/personhog-leader/src/recovery.rs b/rust/personhog-leader/src/recovery.rs
index 46bc26afe477..486f28dbec7d 100644
--- a/rust/personhog-leader/src/recovery.rs
+++ b/rust/personhog-leader/src/recovery.rs
@@ -23,7 +23,6 @@ const RETRY_BACKOFF: Duration = Duration::from_millis(100);
#[derive(Clone)]
pub struct RecoveryConfig {
pub kafka: KafkaConfig,
- pub topic: String,
pub pod_name: String,
/// Overall deadline for one recovery, including transient-failure
/// retries.
@@ -73,7 +72,6 @@ enum FetchError {
/// consumer while freeing its permit, eventually leaving permits with an
/// empty pool.
pub struct ChangelogRecovery {
- topic: String,
recv_timeout: Duration,
idle: Mutex<VecDeque<StreamConsumer>>,
permits: Semaphore,
@@ -93,7 +91,6 @@ impl ChangelogRecovery {
);
}
Ok(Self {
- topic: cfg.topic,
recv_timeout: cfg.recv_timeout,
idle: Mutex::new(idle),
permits: Semaphore::new(cfg.pool_size),
@@ -139,7 +136,8 @@ impl ChangelogRecovery {
mark: &DirtyMark,
expected: &PersonCacheKey,
) -> Result<CachedPerson, String> {
- let partition = mark.partition;
+ let topic = &mark.stream.topic;
+ let partition = mark.stream.partition;
let offset = mark.offset;
let partition_i32 = i32::try_from(partition)
.map_err(|_| format!("partition {partition} exceeds i32::MAX"))?;
@@ -154,7 +152,7 @@ impl ChangelogRecovery {
let deadline = Instant::now() + self.recv_timeout;
loop {
let attempt = self
- .attempt_fetch(&consumer, partition_i32, mark, expected, deadline)
+ .attempt_fetch(&consumer, topic, partition_i32, mark, expected, deadline)
.await;
match attempt {
Ok(person) => return Ok(person),
@@ -180,13 +178,14 @@ impl ChangelogRecovery {
async fn attempt_fetch(
&self,
consumer: &StreamConsumer,
+ topic: &str,
partition_i32: i32,
mark: &DirtyMark,
expected: &PersonCacheKey,
deadline: Instant,
) -> Result<CachedPerson, FetchError> {
let mut tpl = TopicPartitionList::new();
- tpl.add_partition_offset(&self.topic, partition_i32, Offset::Offset(mark.offset))
+ tpl.add_partition_offset(topic, partition_i32, Offset::Offset(mark.offset))
.map_err(|e| FetchError::Permanent(format!("tpl add_partition_offset: {e}")))?;
consumer
.assign(&tpl)
@@ -201,7 +200,7 @@ impl ChangelogRecovery {
Err(_) => {
return Err(FetchError::Transient(format!(
"recovery timed out after {:?} seeking partition {} offset {offset}",
- self.recv_timeout, mark.partition
+ self.recv_timeout, mark.stream.partition
)));
}
};
diff --git a/rust/personhog-leader/src/service.rs b/rust/personhog-leader/src/service.rs
index bc9b2297ed6e..5a75fbd03539 100644
--- a/rust/personhog-leader/src/service.rs
+++ b/rust/personhog-leader/src/service.rs
@@ -19,8 +19,9 @@ use personhog_common::partitioning::partition_for_person;
use personhog_coordination::authority::AuthorityClock;
use crate::cache::{
- CacheLookup, CachedPerson, DirtyIndex, DirtyMark, PartitionedCache, PersonCacheKey,
+ CacheLookup, CachedPerson, DirtyIndex, DirtyMark, PartitionedCache, PersonCacheKey, Stream,
};
+use crate::coordination::ChangelogCaptures;
use crate::emitted::{EmittedVersionGuard, EmittedVersions};
use crate::fence::{
fenced_status, semantic_refusal, target_mark_status, FenceHealer, FenceMap, FenceState,
@@ -82,6 +83,12 @@ struct Committer {
dirty_index: Arc<DirtyIndex>,
emitted_versions: Arc<EmittedVersions>,
fenced: Arc<FencedChangelogProducers>,
+ /// Shared with the service; a mark names the partition's captured topic.
+ captures: ChangelogCaptures,
+}
+
+fn not_served(partition: u32) -> Status {
+ Status::failed_precondition(format!("partition {partition} is not served by this pod"))
}
/// A committed document, with the locks the handler took for it, so the
@@ -103,10 +110,8 @@ pub struct PersonHogLeaderService {
fallback: Option<PgFallback>,
/// Per-partition inflight counter used to drive the handoff drain phase.
inflight: Arc<InflightTracker>,
- /// Total changelog partition count, read from etcd at startup (the same
- /// source the router uses). Used to validate the router's routing
- /// decision against each request's key.
- num_partitions: u32,
+ /// What each served partition was acquired under; shared with the handler.
+ captures: ChangelogCaptures,
/// Persons whose latest acked state the writer may not have applied to
/// PG yet. Consulted on every cache miss: marked persons recover from
/// the changelog, unmarked persons' PG rows are known current.
@@ -208,7 +213,7 @@ impl PersonHogLeaderService {
fallback: Option<PgFallback>,
locks: Arc<DashMap<PersonCacheKey, Arc<Mutex<()>>>>,
inflight: Arc<InflightTracker>,
- num_partitions: u32,
+ captures: ChangelogCaptures,
dirty_index: Arc<DirtyIndex>,
recovery: Arc<ChangelogRecovery>,
size_limits: PropertySizeLimits,
@@ -223,6 +228,7 @@ impl PersonHogLeaderService {
dirty_index: Arc::clone(&dirty_index),
emitted_versions: Arc::clone(&emitted_versions),
fenced,
+ captures: Arc::clone(&captures),
});
Self {
committer,
@@ -234,7 +240,7 @@ impl PersonHogLeaderService {
mark_verifier: fallback.as_ref().map(MarkVerifier::new),
fallback,
inflight,
- num_partitions,
+ captures,
dirty_index,
recovery,
size_limits,
@@ -267,10 +273,16 @@ impl PersonHogLeaderService {
team_id: i64,
person_id: i64,
) -> Result<(), Status> {
- let expected = partition_for_person(team_id, person_id, self.num_partitions);
+ let modulus = self
+ .captures
+ .get(&partition)
+ .map(|c| c.config.partitions)
+ .ok_or_else(|| not_served(partition))?;
+ let expected = partition_for_person(team_id, person_id, modulus);
if partition != expected {
counter!("personhog_leader_partition_mismatch_total").increment(1);
- return Err(Status::invalid_argument(format!(
+ // A precondition, so the router bounces and retries, landing once it adopts the new count.
+ return Err(Status::failed_precondition(format!(
"x-partition {partition} does not match partition {expected} \
derived from team_id={team_id} person_id={person_id}"
)));
@@ -288,9 +300,7 @@ impl PersonHogLeaderService {
#[allow(clippy::result_large_err)]
fn validate_ownership(&self, partition: u32) -> Result<(), Status> {
if !self.cache.has_partition(partition) {
- return Err(Status::failed_precondition(format!(
- "partition {partition} is not served by this pod"
- )));
+ return Err(not_served(partition));
}
Ok(())
}
@@ -561,6 +571,17 @@ impl PersonHogLeaderService {
}
impl Committer {
+ /// The producer was acquired on the capture's topic, so its records land there.
+ #[allow(clippy::result_large_err)]
+ fn stream_of(&self, partition: u32) -> Result<Stream, Status> {
+ let topic = self
+ .captures
+ .get(&partition)
+ .map(|c| Arc::clone(&c.topic))
+ .ok_or_else(|| not_served(partition))?;
+ Ok(Stream { topic, partition })
+ }
+
async fn commit(
&self,
partition: u32,
@@ -570,6 +591,7 @@ impl Committer {
lock: OwnedMutexGuard<()>,
inflight: InflightGuard,
) -> Result<Committed, Status> {
+ let stream = self.stream_of(partition)?;
// The guard is what makes the version un-reusable if this task
// never reports.
let mut emitted = EmittedVersionGuard::new(
@@ -689,6 +711,7 @@ impl Committer {
version: person.version,
offset,
partition,
+ stream,
is_deleted: person.is_deleted,
},
);
@@ -1802,12 +1825,21 @@ mod tests {
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- 1,
+ // Partition 0 served under one partition.
+ Arc::new(DashMap::from_iter([(
+ 0,
+ crate::coordination::Capture {
+ topic: Arc::from("personhog_updates"),
+ config: personhog_coordination::types::ChangelogConfig::new(
+ "personhog_updates",
+ 1,
+ ),
+ },
+ )])),
Arc::new(DirtyIndex::new(16)),
Arc::new(
ChangelogRecovery::new(RecoveryConfig {
kafka,
- topic: "personhog_updates".to_string(),
pod_name: "test".to_string(),
recv_timeout: Duration::from_millis(10),
pool_size: 1,
@@ -1821,7 +1853,6 @@ mod tests {
// a partition, which these tests never do.
Arc::new(FencedChangelogProducers::new(FencedProducerConfig {
kafka: KafkaConfig::init_from_hashmap(&HashMap::new()).unwrap(),
- topic: "personhog_updates".to_string(),
init_timeout: Duration::from_secs(1),
commit_timeout: Duration::from_secs(1),
broker_txn_timeout: Duration::from_secs(10),
@@ -2177,7 +2208,7 @@ mod tests {
// The handoff completes while the release is parked.
service.cache.drop_partition(0);
- crate::fence::drop_partition_fences(&service.fences, 0, 1);
+ crate::fence::drop_partition_fences(&service.fences, 0, 1, |_| false);
drop(held);
let err = releasing
diff --git a/rust/personhog-leader/src/service/lifecycle.rs b/rust/personhog-leader/src/service/lifecycle.rs
index e4d69967c68f..579e74353634 100644
--- a/rust/personhog-leader/src/service/lifecycle.rs
+++ b/rust/personhog-leader/src/service/lifecycle.rs
@@ -440,6 +440,19 @@ impl PersonHogLeaderService {
)
}
+ /// A member off the routed partition is a final answer, not a routing bounce:
+ /// the caller regroups and retries each member alone.
+ #[allow(clippy::result_large_err)]
+ fn validate_batch_member(
+ &self,
+ partition: u32,
+ team_id: i64,
+ person_id: i64,
+ ) -> Result<(), Status> {
+ self.validate_partition(partition, team_id, person_id)
+ .map_err(|status| Status::invalid_argument(status.message()))
+ }
+
pub(super) async fn fence_persons_rpc(
&self,
request: Request<FencePersonsRequest>,
@@ -453,11 +466,13 @@ impl PersonHogLeaderService {
)));
}
let team_id = req.team_id;
+ // Ownership first: an unserved partition is a routing bounce, not a bad batch.
+ self.validate_ownership(partition)?;
// The caller grouped its batch by partition and routed it by one
// member, so every member must hash to the routed partition; a
// stale partition count refuses the whole batch before any fence.
for person_id in &req.person_ids {
- self.validate_partition(partition, team_id, *person_id)?;
+ self.validate_batch_member(partition, team_id, *person_id)?;
}
self.check_authority(partition)?;
let op_id = Uuid::parse_str(&req.op_id)
@@ -466,7 +481,6 @@ impl PersonHogLeaderService {
if op_type == LifecycleOpType::Unspecified {
return Err(Status::invalid_argument("op_type must be specified"));
}
- self.validate_ownership(partition)?;
let Some(_inflight_guard) = self.inflight.try_begin(partition) else {
return Err(Status::failed_precondition(format!(
"partition {partition} is fenced for handoff; writes are rejected"
@@ -573,15 +587,15 @@ impl PersonHogLeaderService {
)));
}
let team_id = req.team_id;
+ self.validate_ownership(partition)?;
// Same strict routing check as a batched fence: every member hashes
// to the routed partition or the whole batch is refused untouched.
for person in &req.persons {
- self.validate_partition(partition, team_id, person.person_id)?;
+ self.validate_batch_member(partition, team_id, person.person_id)?;
}
// The aborted arm would otherwise ack under a lapsed lease; the
// committed arm's produce and durable-fact acks need no cover.
self.check_authority(partition)?;
- self.validate_ownership(partition)?;
let op_id = Uuid::parse_str(&req.op_id)
.map_err(|_| Status::invalid_argument("op_id must be a valid UUID"))?;
histogram!("personhog_leader_release_batch_size").record(req.persons.len() as f64);
diff --git a/rust/personhog-leader/src/settle.rs b/rust/personhog-leader/src/settle.rs
index 4b42225b050e..4c32df6c7e84 100644
--- a/rust/personhog-leader/src/settle.rs
+++ b/rust/personhog-leader/src/settle.rs
@@ -9,54 +9,77 @@ use dashmap::DashMap;
use metrics::{counter, histogram};
use tokio::sync::Mutex;
-use crate::cache::{CachedVersion, DirtyIndex, DirtyMark, PartitionedCache, PersonCacheKey};
+use crate::cache::{
+ CachedVersion, DirtyIndex, DirtyMark, PartitionedCache, PersonCacheKey, Stream,
+};
use crate::warming::{fetch_writer_committed_offsets, ConsumerPool};
-/// One prune pass: fetch committed offsets, prune applied marks of the
-/// published partitions only (the settle cannot see a warming build),
-/// and settle the death documents those marks covered. Returns marked
-/// partitions and offsets for lag gauges; None if the fetch failed.
+/// One prune pass over the published partitions: fetch committed offsets per
+/// topic, prune the applied marks and settle their death documents. A failed
+/// fetch leaves its topic's marks waiting.
pub async fn prune_and_settle_tick(
dirty_index: &DirtyIndex,
cache: &PartitionedCache,
locks: &DashMap<PersonCacheKey, Arc<Mutex<()>>>,
offsets_pool: &ConsumerPool,
- topic: &str,
offsets_timeout: Duration,
-) -> Option<(Vec<u32>, HashMap<u32, i64>)> {
- let partitions = dirty_index.partitions_with_marks();
- if partitions.is_empty() {
- return Some((partitions, HashMap::new()));
+) -> Vec<StreamLag> {
+ let targets = dirty_index.streams_with_marks();
+ if targets.is_empty() {
+ return Vec::new();
}
- let committed_offsets =
- match fetch_writer_committed_offsets(offsets_pool, topic, &partitions, offsets_timeout)
+ // One OffsetFetch per topic a live mark lives on.
+ let mut by_topic: HashMap<Arc<str>, Vec<u32>> = HashMap::new();
+ for (_, stream) in &targets {
+ let partitions = by_topic.entry(Arc::clone(&stream.topic)).or_default();
+ if !partitions.contains(&stream.partition) {
+ partitions.push(stream.partition);
+ }
+ }
+ let mut committed: HashMap<Arc<str>, HashMap<u32, i64>> = HashMap::new();
+ for (topic, partitions) in by_topic {
+ match fetch_writer_committed_offsets(offsets_pool, &topic, &partitions, offsets_timeout)
.await
{
- Ok(offsets) => offsets,
+ Ok(offsets) => {
+ committed.insert(topic, offsets);
+ }
Err(e) => {
- tracing::warn!(error = %e, "dirty-index prune offset fetch failed");
- return None;
+ counter!("personhog_leader_prune_offset_fetch_failures_total").increment(1);
+ tracing::warn!(error = %e, topic = %topic, "dirty-index prune offset fetch failed; its marks wait");
}
- };
+ }
+ }
// Prune chunk by chunk, settling each chunk's death pairs before the
// next, so a catch-up never materializes more than one chunk.
let started = Instant::now();
let mut removed_total = 0u64;
- for (partition, committed) in &committed_offsets {
- if !cache.is_published(*partition) {
- continue;
- }
- loop {
- let (pruned, exhausted) = dirty_index.prune_chunk(*partition, *committed);
- removed_total += pruned.removed as u64;
- if !pruned.death_marks.is_empty() {
- drop_settled_death_documents(cache, locks, &pruned.death_marks).await;
- }
- if exhausted {
- break;
+ let mut lags = Vec::with_capacity(targets.len());
+ for (partition, stream) in targets {
+ let offset = committed
+ .get(&stream.topic)
+ .and_then(|offsets| offsets.get(&stream.partition))
+ .copied();
+ if let Some(offset) = offset {
+ if cache.is_published(partition) {
+ loop {
+ let (pruned, exhausted) = dirty_index.prune_chunk(partition, &stream, offset);
+ removed_total += pruned.removed as u64;
+ if !pruned.death_marks.is_empty() {
+ drop_settled_death_documents(cache, locks, &pruned.death_marks).await;
+ }
+ if exhausted {
+ break;
+ }
+ }
}
}
+ lags.push(StreamLag {
+ partition,
+ stream,
+ committed: offset,
+ });
}
if removed_total > 0 {
counter!("personhog_leader_dirty_index_pruned_total").increment(removed_total);
@@ -65,7 +88,14 @@ pub async fn prune_and_settle_tick(
// pass.
histogram!("personhog_leader_prune_tick_duration_ms")
.record(started.elapsed().as_secs_f64() * 1000.0);
- Some((partitions, committed_offsets))
+ lags
+}
+
+/// The writer's committed offset on a stream, if it has one.
+pub struct StreamLag {
+ pub partition: u32,
+ pub stream: Stream,
+ pub committed: Option<i64>,
}
/// Drop death documents whose marks the writer settled: past the mark,
@@ -140,6 +170,10 @@ mod tests {
version: 5,
offset: 0,
partition: 0,
+ stream: Stream {
+ topic: Arc::from("personhog_updates"),
+ partition: 0,
+ },
is_deleted: true,
},
)];
diff --git a/rust/personhog-leader/src/warming.rs b/rust/personhog-leader/src/warming.rs
index 55c73a14c5a9..2267faede44d 100644
--- a/rust/personhog-leader/src/warming.rs
+++ b/rust/personhog-leader/src/warming.rs
@@ -12,10 +12,12 @@ use rdkafka::message::Message;
use rdkafka::{ClientConfig, Offset, TopicPartitionList};
use tokio::time::{sleep, timeout};
+use personhog_common::partitioning::partition_for_person;
+use personhog_coordination::changelog::WarmSource;
use personhog_coordination::error::{Error as CoordError, Result as CoordResult};
use personhog_proto::personhog::types::v1::Person;
-use crate::cache::{CachedPerson, DirtyIndex, DirtyMark, PartitionedCache, PersonCacheKey};
+use crate::cache::{CachedPerson, DirtyIndex, DirtyMark, PartitionedCache, PersonCacheKey, Stream};
/// Retry policy for transient warming-metadata failures.
#[derive(Clone, Copy)]
@@ -89,19 +91,15 @@ where
}
/// Configuration for the warming pipeline. Owned by the leader handoff
-/// handler and borrowed at each `warm_from_kafka` call.
+/// handler and borrowed at each warm.
#[derive(Clone)]
pub struct WarmingConfig {
pub kafka: KafkaConfig,
- pub topic: String,
pub pod_name: String,
/// Writer's consumer group id. We query this group's committed offset
/// for the partition to find "everything at or after this offset has
/// not yet been written to PG."
pub writer_consumer_group: String,
- /// Number of offsets to rewind past the writer's committed offset as a
- /// safety margin. Bounded by Kafka's earliest-available offset.
- pub lookback_offsets: i64,
/// Timeout for the OffsetFetch metadata call.
pub committed_offsets_timeout: Duration,
/// Timeout for the per-partition `fetch_watermarks` call.
@@ -168,7 +166,9 @@ pub(crate) fn make_consumer(
// Only committed transactions: with fencing on, aborted windows
// and zombie leftovers must never reach the cache. Identical
// behavior on a topic without transactional records.
- .set("isolation.level", "read_committed");
+ .set("isolation.level", "read_committed")
+ // A warm learns it reached the end from an empty fetch, held 500ms by default.
+ .set("fetch.wait.max.ms", "10");
if kafka.kafka_tls {
cfg.set("security.protocol", "ssl")
.set("enable.ssl.certificate.verification", "false");
@@ -395,23 +395,10 @@ pub async fn fetch_writer_committed_offsets(
result
}
-/// Decide where to start consuming for a partition.
-///
-/// The writer's committed offset is the authoritative "everything up to
-/// here is durable in PG" marker. Messages at or after it need to be warmed
-/// into the leader's cache — otherwise a cache miss for a key in that
-/// range would fall through to PG and return a stale value (PG hasn't seen
-/// the update yet).
-///
-/// We rewind an extra `lookback` offsets as a safety margin against races
-/// between the writer's commit and our read of it, and clamp to the
-/// earliest available offset so we don't seek past Kafka's retention.
-fn resolve_start_offset(committed: Option<i64>, earliest: i64, lookback: i64) -> i64 {
- let lookback = lookback.max(0);
- match committed {
- Some(c) => (c - lookback).max(earliest),
- None => earliest,
- }
+/// The writer commits an offset only after PG holds every record below it,
+/// so a warm replays from the commit, clamped to Kafka's retention.
+fn resolve_start_offset(committed: Option<i64>, earliest: i64) -> i64 {
+ committed.map_or(earliest, |c| c.max(earliest))
}
/// One warm sub-span sample. The spans share the warm bucket ladder and
@@ -458,21 +445,26 @@ impl Drop for WarmCleanup<'_> {
/// produce path awaits the Kafka delivery future before returning,
/// "no in-flight" implies "every acked write is durable in Kafka."
///
-/// Together those mean: by the time `warm_from_kafka` runs, no producer
+/// Together those mean: by the time a warm runs, no producer
/// can append to this partition's Kafka log. The HWM we snapshot here
/// is therefore stable, and we can consume to a known endpoint without
/// racing producers.
-pub async fn warm_from_kafka(
+async fn warm_source(
cfg: &WarmingConfig,
pools: &WarmClientPools,
cache: &PartitionedCache,
dirty_index: &DirtyIndex,
partition: u32,
-) -> CoordResult<()> {
- let start = Instant::now();
- let partition_i32 = i32::try_from(partition).map_err(|_| {
- CoordError::invalid_state(format!("partition {partition} exceeds i32::MAX"))
+ source: &WarmSource,
+ committed_offset: Option<i64>,
+) -> CoordResult<u64> {
+ let partition_i32 = i32::try_from(source.partition).map_err(|_| {
+ CoordError::invalid_state(format!("partition {} exceeds i32::MAX", source.partition))
})?;
+ let stream = Stream {
+ topic: Arc::clone(&source.topic),
+ partition: source.partition,
+ };
// Arc because the watermark retry closure needs its own handle for
// the blocking pool; sole ownership returns once the retries finish.
@@ -483,25 +475,11 @@ pub async fn warm_from_kafka(
let consumer = Arc::new(pools.warming.checkout()?);
record_warm_span("checkout", span_start);
- // The two offset queries are independent — the writer's committed
- // position comes from the offsets pool, the watermarks from this
- // consumer — so they run concurrently; each keeps its own retry
- // policy. The committed query uses a separate, short-lived client to
- // keep the long-lived warming consumer isolated from the writer's
- // consumer group. `fetch_watermarks` is synchronous in rdkafka and
- // may block for the full timeout, so it runs on the blocking pool.
- let committed_fut = with_warm_retry("committed_offset", partition, cfg.retry, || async {
- fetch_writer_committed_offset(
- &pools.offsets,
- &cfg.topic,
- partition,
- cfg.committed_offsets_timeout,
- )
- .await
- });
+ // `fetch_watermarks` is synchronous in rdkafka and may block for the
+ // full timeout, so it runs on the blocking pool.
let watermarks_fut = with_warm_retry("fetch_watermarks", partition, cfg.retry, || {
let consumer = Arc::clone(&consumer);
- let topic = cfg.topic.clone();
+ let topic = Arc::clone(&source.topic);
let timeout = cfg.fetch_watermarks_timeout;
async move {
tokio::task::spawn_blocking(move || {
@@ -514,90 +492,43 @@ pub async fn warm_from_kafka(
}
});
let span_start = Instant::now();
- let (committed_res, watermarks_res) = tokio::join!(committed_fut, watermarks_fut);
+ // A failed watermark call leaves this consumer doubtful, so the pool's
+ // drop-doubtful-clients contract applies: it is not given back.
+ let (low, hwm) = watermarks_fut.await?;
record_warm_span("metadata", span_start);
- let (low, hwm) = match watermarks_res {
- Ok(marks) => marks,
- // The watermark call failed on this consumer, so the pool's
- // drop-doubtful-clients contract applies: fall through without
- // giving it back.
- Err(e) => return Err(e),
- };
- let committed_offset = match committed_res {
- Ok(offset) => offset,
- Err(e) => {
- // The committed-offset query runs on the offsets pool's
- // client; this consumer did nothing and is sound. Returning
- // it keeps reconcile-driven warm retries from rebuilding a
- // Kafka client per attempt.
- if let Ok(consumer) = Arc::try_unwrap(consumer) {
- pools.warming.give_back(consumer);
- }
- return Err(e);
- }
- };
- let start_offset = resolve_start_offset(committed_offset, low, cfg.lookback_offsets);
+ let start_offset = resolve_start_offset(committed_offset, low);
tracing::info!(
partition,
+ topic = %source.topic,
+ source_partition = source.partition,
writer_group = cfg.writer_consumer_group,
committed = ?committed_offset,
earliest = low,
hwm,
- lookback = cfg.lookback_offsets,
start_offset,
"computed warming range"
);
if hwm <= start_offset {
- // Empty range — install an empty partition cache. Use the
- // atomic install path so this matches the populated path's
- // publication semantics (the partition becomes observable in
- // a single dashmap insert). The consumer never got assigned, so
- // it goes straight back to the pool.
- cache.install_warmed_partition(partition, std::iter::empty());
- tracing::info!(partition, hwm, start_offset, "no messages to warm in range");
+ tracing::info!(partition, topic = %source.topic, source_partition = source.partition, hwm, start_offset, "no messages to warm in range");
if let Ok(consumer) = Arc::try_unwrap(consumer) {
pools.warming.give_back(consumer);
}
- return Ok(());
+ return Ok(0);
}
let span_start = Instant::now();
let mut assign_tpl = TopicPartitionList::new();
assign_tpl
- .add_partition_offset(&cfg.topic, partition_i32, Offset::Offset(start_offset))
+ .add_partition_offset(&source.topic, partition_i32, Offset::Offset(start_offset))
.map_err(|e| CoordError::invalid_state(format!("tpl add_partition_offset: {e}")))?;
consumer
.assign(&assign_tpl)
.map_err(|e| CoordError::invalid_state(format!("consumer assign: {e}")))?;
- // Stream records straight into an unpublished partition cache: the
- // build evicts under the same per-partition byte budget as a serving
- // cache, so the warm's peak memory is bounded no matter how large
- // the range is. Each record writes its dirty mark BEFORE its insert,
- // so at every instant an evicted unapplied person is already
- // recoverable from the changelog — the same miss path a serving
- // partition relies on. Nothing is observable until the publish at
- // the end.
- cache.begin_warm_partition(partition);
- // Every non-published exit — an error, a panic, or the coordination
- // loop dropping this future on lease loss or shutdown (the
- // cancellation `HandoffHandler::warm_partition`'s contract names) —
- // must leave no trace: the guard aborts the build and clears the
- // partition's seeded marks on drop. A stale mark surviving into a
- // later acquisition would redirect a miss to a superseded changelog
- // offset; the re-warm only overwrites marks for records past the
- // writer's *new* committed offset, so it cannot heal one.
- let mut cleanup = WarmCleanup {
- cache,
- dirty_index,
- partition,
- armed: true,
- };
let mut consumed: u64 = 0;
- let mut seeded = 0u64;
let mut last_offset: i64 = -1;
// A transactionally-produced range can end in control records
@@ -677,7 +608,7 @@ pub async fn warm_from_kafka(
let position_reached = consumer
.position()
.map_err(|e| CoordError::invalid_state(format!("warm position: {e}")))?
- .find_partition(&cfg.topic, partition_i32)
+ .find_partition(&source.topic, partition_i32)
.is_some_and(|elem| matches!(elem.offset(), Offset::Offset(p) if p >= hwm));
if position_reached {
break;
@@ -708,31 +639,29 @@ pub async fn warm_from_kafka(
team_id: cached.team_id,
person_id: cached.id,
};
- consumed += 1;
- // Records at or past the writer's committed offset are not
- // yet in PG: mark before insert, so if the build evicts this
- // person a later miss recovers from the changelog instead of
- // trusting a stale PG row.
- if committed_offset.is_none_or(|committed| offset >= committed) {
- dirty_index.mark(
- key.clone(),
- DirtyMark {
- version: cached.version,
- offset,
- partition,
- is_deleted: cached.is_deleted,
- },
- );
- seeded += 1;
- cache.warm_put(partition, key, cached);
- } else if cached.is_deleted {
- // An applied death record leaves no residue: unmarked it
- // would answer forever, and its predecessor in the build
- // would resurrect the destroyed person's data.
- cache.warm_remove(partition, &key);
- } else {
- cache.warm_put(partition, key, cached);
+ // An old partition also holds keys that now belong elsewhere.
+ if let Some(keep) = &source.keep {
+ if partition_for_person(cached.team_id, cached.id, keep.modulus) != keep.target {
+ if offset + 1 >= hwm {
+ break;
+ }
+ continue;
+ }
}
+ consumed += 1;
+ // PG may not hold a replayed record yet: mark before insert, so a
+ // miss after an eviction recovers from the changelog, not PG.
+ dirty_index.mark(
+ key.clone(),
+ DirtyMark {
+ version: cached.version,
+ offset,
+ partition,
+ stream: stream.clone(),
+ is_deleted: cached.is_deleted,
+ },
+ );
+ cache.warm_put(partition, key, cached);
} else {
// The writer never produces null-payload (tombstone) records
// to `personhog_updates` today. If one ever appears it would
@@ -760,19 +689,58 @@ pub async fn warm_from_kafka(
record_warm_span("consume", span_start);
- // Atomic publish: one `DashMap` insert flips the fully-built cache
- // from invisible to observable. The previous pattern
- // (`create_partition` + per-record `put` loop) created a window
- // where readers could observe `has_partition == true` while the
- // cache was still being populated, and then fall through to PG —
- // potentially returning stale values for records the writer hasn't
- // yet persisted. Publishing at the end removes the dependency on the
- // protocol invariant ("no reads during Warming") for correctness.
+ // Every error path above dropped the consumer: a client that just failed is not pool material.
+ if let Ok(consumer) = Arc::try_unwrap(consumer) {
+ pools.warming.give_back(consumer);
+ }
+
+ Ok(consumed)
+}
+
+/// Every source is sealed or fenced, so each replay runs to a known end.
+pub async fn warm_from_sources(
+ cfg: &WarmingConfig,
+ pools: &WarmClientPools,
+ cache: &PartitionedCache,
+ dirty_index: &DirtyIndex,
+ partition: u32,
+ sources: &[WarmSource],
+) -> CoordResult<()> {
+ let start = Instant::now();
+ // Every writer position is read before any replay: a live position read
+ // after the writer left an old generation would skip records newer than
+ // that generation's marked tail.
+ let mut committed = Vec::with_capacity(sources.len());
+ for source in sources {
+ committed.push(
+ with_warm_retry("committed_offset", partition, cfg.retry, || async {
+ fetch_writer_committed_offset(
+ &pools.offsets,
+ &source.topic,
+ source.partition,
+ cfg.committed_offsets_timeout,
+ )
+ .await
+ })
+ .await?,
+ );
+ }
+ // The build is invisible and evicts under the serving budget until the publish.
+ cache.begin_warm_partition(partition);
+ let mut cleanup = WarmCleanup {
+ cache,
+ dirty_index,
+ partition,
+ armed: true,
+ };
+ let mut consumed = 0u64;
+ for (source, committed) in sources.iter().zip(committed) {
+ consumed +=
+ warm_source(cfg, pools, cache, dirty_index, partition, source, committed).await?;
+ }
+
let resident_bytes = cache.warm_usage_bytes(partition) as u64;
- // Disarm before the publish: from here the marks and the cache are
- // the partition's serving state, not warm residue. No await sits
- // between the disarm and the publish, so no cancellation can land
- // in between.
+ // One insert publishes the whole build, and no await sits between the disarm and it.
cleanup.armed = false;
cache.publish_warmed_partition(partition);
@@ -780,24 +748,14 @@ pub async fn warm_from_kafka(
tracing::info!(
pod = cfg.pod_name,
partition,
+ sources = sources.len(),
messages = consumed,
resident_bytes,
- dirty_seeded = seeded,
- hwm,
- start_offset,
elapsed_ms = elapsed.as_millis() as u64,
"warmed partition from kafka"
);
histogram!("personhog_leader_warm_duration_ms").record(elapsed.as_secs_f64() * 1000.0);
counter!("personhog_leader_warmed_messages_total").increment(consumed);
-
- // Every error path above dropped the consumer instead of returning
- // it — a client that just failed is not pool material. Failing to
- // unwrap the Arc would only mean the same disposition.
- if let Ok(consumer) = Arc::try_unwrap(consumer) {
- pools.warming.give_back(consumer);
- }
-
Ok(())
}
@@ -857,6 +815,10 @@ mod tests {
version: 1,
offset: 7,
partition: 0,
+ stream: Stream {
+ topic: Arc::from("personhog_updates"),
+ partition: 0,
+ },
is_deleted: false,
},
);
@@ -901,6 +863,10 @@ mod tests {
version: 1,
offset: 7,
partition: 0,
+ stream: Stream {
+ topic: Arc::from("personhog_updates"),
+ partition: 0,
+ },
is_deleted: false,
},
);
@@ -921,29 +887,10 @@ mod tests {
}
#[test]
- fn resolve_uses_committed_minus_lookback() {
- assert_eq!(resolve_start_offset(Some(500), 0, 100), 400);
- }
-
- #[test]
- fn resolve_clamps_to_earliest() {
- // Lookback would take us to 50, but earliest is 200.
- assert_eq!(resolve_start_offset(Some(100), 200, 50), 200);
- }
-
- #[test]
- fn resolve_falls_back_to_earliest_when_no_commit() {
- assert_eq!(resolve_start_offset(None, 42, 100), 42);
- }
-
- #[test]
- fn resolve_treats_negative_lookback_as_zero() {
- assert_eq!(resolve_start_offset(Some(500), 0, -10), 500);
- }
-
- #[test]
- fn resolve_handles_zero_lookback() {
- assert_eq!(resolve_start_offset(Some(500), 0, 0), 500);
+ fn resolve_starts_at_the_commit_clamped_to_earliest() {
+ assert_eq!(resolve_start_offset(Some(500), 0), 500);
+ assert_eq!(resolve_start_offset(Some(100), 200), 200);
+ assert_eq!(resolve_start_offset(None, 42), 42);
}
#[tokio::test(flavor = "current_thread", start_paused = true)]
diff --git a/rust/personhog-leader/tests/commit_integration.rs b/rust/personhog-leader/tests/commit_integration.rs
index 2a6852d2e36d..9a20f319c479 100644
--- a/rust/personhog-leader/tests/commit_integration.rs
+++ b/rust/personhog-leader/tests/commit_integration.rs
@@ -6,8 +6,8 @@ use std::sync::Arc;
use std::time::{Duration, Instant};
use common::{
- create_local_kafka_producer, seed_person, test_cached_person, test_kafka_config, test_recovery,
- unique_team_id, KAFKA_BOOTSTRAP,
+ create_local_kafka_producer, seed_person, served_captures, target, test_cached_person,
+ test_kafka_config, test_recovery, unique_team_id, KAFKA_BOOTSTRAP,
};
use dashmap::DashMap;
use personhog_common::partitioning::partition_for_person;
@@ -55,7 +55,6 @@ async fn start_harness() -> Harness {
kafka.kafka_hosts = KAFKA_BOOTSTRAP.to_string();
let fenced = Arc::new(FencedChangelogProducers::new(FencedProducerConfig {
kafka,
- topic: topic.clone(),
init_timeout: Duration::from_secs(10),
commit_timeout: Duration::from_secs(10),
broker_txn_timeout: BROKER_TXN_TIMEOUT,
@@ -72,7 +71,10 @@ async fn start_harness() -> Harness {
person_id: seed.id,
};
let partition = partition_for_person(key.team_id, key.person_id, NUM_PARTITIONS);
- fenced.acquire(partition).await.expect("acquire the fence");
+ fenced
+ .acquire(partition, &target(&topic))
+ .await
+ .expect("acquire the fence");
let cache = Arc::new(PartitionedCache::new(1 << 20));
cache.create_partition(partition);
@@ -84,9 +86,9 @@ async fn start_harness() -> Harness {
None,
Arc::clone(&locks),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&topic, NUM_PARTITIONS),
Arc::new(DirtyIndex::new(1_000_000)),
- test_recovery(&topic, KAFKA_BOOTSTRAP),
+ test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(DashMap::new()),
diff --git a/rust/personhog-leader/tests/common/mod.rs b/rust/personhog-leader/tests/common/mod.rs
index e76e2b139f01..5c8e2d6fbe8e 100644
--- a/rust/personhog-leader/tests/common/mod.rs
+++ b/rust/personhog-leader/tests/common/mod.rs
@@ -15,12 +15,15 @@ use common_kafka::config::KafkaConfig;
use common_kafka::kafka_producer::{create_kafka_producer, KafkaContext};
use health::HealthRegistry;
use personhog_coordination::authority::AuthorityClock;
+use personhog_coordination::changelog::LiveChangelog;
use personhog_coordination::coordinator::{Coordinator, CoordinatorConfig};
use personhog_coordination::error::Result;
use personhog_coordination::pod::{PodConfig, PodHandle};
use personhog_coordination::routing_table::{RoutingTable, RoutingTableConfig, StashHandler};
use personhog_coordination::store::PersonhogStore;
use personhog_coordination::strategy::AssignmentStrategy;
+use personhog_coordination::types::{ChangelogConfig, ChangelogTarget};
+use personhog_leader::coordination::{Capture, ChangelogCaptures};
use rdkafka::admin::{AdminClient, AdminOptions, NewTopic, TopicReplication};
use rdkafka::client::DefaultClientContext;
use rdkafka::mocking::MockCluster;
@@ -201,8 +204,16 @@ pub const CHANGELOG_TOPIC: &str = "personhog_updates";
pub struct LeaderPodHandles {
pub cache: Arc<PartitionedCache>,
pub leader_addr: SocketAddr,
- /// Kept alive so the topic outlives the test that reads it.
- pub topic: TestTopic,
+}
+
+/// Seeds `config/changelog` with a topic of this test's own; pods read it from there.
+pub async fn seed_test_changelog(store: &PersonhogStore) -> TestTopic {
+ let topic = owned_broker_topic(NUM_PARTITIONS as i32).await;
+ assert!(store
+ .seed_changelog_config(&topic.topic, NUM_PARTITIONS, "personhog-writer")
+ .await
+ .unwrap());
+ topic
}
/// Kafka config pointing at local kafka for e2e tests. Used for both the
@@ -238,17 +249,14 @@ pub fn test_kafka_config() -> KafkaConfig {
/// with real local Kafka, `KAFKA_BOOTSTRAP`.
pub fn test_warming_config(
pod_name: &str,
- topic: &str,
kafka_bootstrap: &str,
) -> personhog_leader::warming::WarmingConfig {
let mut kafka = test_kafka_config();
kafka.kafka_hosts = kafka_bootstrap.to_string();
personhog_leader::warming::WarmingConfig {
kafka,
- topic: topic.to_string(),
pod_name: pod_name.to_string(),
writer_consumer_group: "personhog-writer".to_string(),
- lookback_offsets: 0,
committed_offsets_timeout: Duration::from_secs(5),
fetch_watermarks_timeout: Duration::from_secs(5),
recv_timeout: Duration::from_secs(10),
@@ -262,13 +270,12 @@ pub fn test_warming_config(
/// Recovery pointed at the given broker. A short receive timeout keeps
/// tests that exercise failed recoveries fast.
-pub fn test_recovery(topic: &str, kafka_bootstrap: &str) -> Arc<ChangelogRecovery> {
+pub fn test_recovery(kafka_bootstrap: &str) -> Arc<ChangelogRecovery> {
let mut kafka = test_kafka_config();
kafka.kafka_hosts = kafka_bootstrap.to_string();
Arc::new(
ChangelogRecovery::new(RecoveryConfig {
kafka,
- topic: topic.to_string(),
pod_name: "test-pod".to_string(),
recv_timeout: Duration::from_secs(2),
pool_size: 2,
@@ -429,12 +436,17 @@ pub async fn fenced_producers_acquired(
topic: &str,
partitions: i32,
) -> Arc<personhog_leader::fencing::FencedChangelogProducers> {
- let fenced = Arc::new(fenced_producers_for(topic));
+ let fenced = Arc::new(test_fenced_producers());
+ let target = ChangelogTarget {
+ topic: topic.to_string(),
+ partitions: partitions as u32,
+ };
let acquires = (0..partitions as u32).map(|partition| {
let fenced = Arc::clone(&fenced);
+ let target = target.clone();
async move {
fenced
- .acquire(partition)
+ .acquire(partition, &target)
.await
.expect("acquiring the changelog fence")
}
@@ -452,7 +464,13 @@ pub async fn start_leader_pod(
cancel: CancellationToken,
) -> LeaderPodHandles {
let cache = Arc::new(PartitionedCache::new(cache_capacity));
- let (mock_cluster, kafka_producer) = create_test_kafka().await;
+ let topic = store
+ .get_changelog_config()
+ .await
+ .unwrap()
+ .expect("seed config/changelog before starting a pod")
+ .topic;
+ let kafka_producer = create_local_kafka_producer().await;
// Pod with real handoff handler. Warming consumer reads from the same
// mock broker the producer is publishing to so the topic actually
@@ -467,8 +485,8 @@ pub async fn start_leader_pod(
// while production drops them.
let emitted_versions = Arc::new(personhog_leader::emitted::EmittedVersions::new(1_000_000));
// Recovery must read the broker the service produces to.
- let recovery = test_recovery(&mock_cluster.topic, &mock_cluster.bootstrap_servers());
- let warming = test_warming_config(name, &mock_cluster.topic, &mock_cluster.bootstrap_servers());
+ let recovery = test_recovery(KAFKA_BOOTSTRAP);
+ let warming = test_warming_config(name, KAFKA_BOOTSTRAP);
let pools = Arc::new(WarmClientPools::new(
&warming.kafka,
name,
@@ -479,7 +497,9 @@ pub async fn start_leader_pod(
// cannot gate anything. Left unacquired, because warming is what
// takes each partition's epoch here, as it does in production.
let authority = Arc::new(AuthorityClock::unclaimed());
- let fenced = Arc::new(fenced_producers_for(&mock_cluster.topic));
+ let fenced = Arc::new(test_fenced_producers());
+ let changelog = live_changelog(&topic);
+ let captures: ChangelogCaptures = Arc::new(DashMap::new());
let handler = LeaderHandoffHandler::new(
Arc::clone(&cache),
Arc::clone(&inflight),
@@ -487,7 +507,8 @@ pub async fn start_leader_pod(
warming,
Arc::new(dashmap::DashMap::new()),
None,
- NUM_PARTITIONS,
+ changelog.clone(),
+ Arc::clone(&captures),
pools,
Arc::clone(&fenced),
Arc::clone(&authority),
@@ -515,7 +536,7 @@ pub async fn start_leader_pod(
None,
Arc::new(DashMap::new()),
Arc::clone(&inflight),
- NUM_PARTITIONS,
+ captures,
Arc::clone(&dirty_index),
recovery,
PropertySizeLimits::new(655360, 524288),
@@ -542,11 +563,7 @@ pub async fn start_leader_pod(
tokio::time::sleep(Duration::from_millis(10)).await;
- LeaderPodHandles {
- cache,
- leader_addr,
- topic: mock_cluster,
- }
+ LeaderPodHandles { cache, leader_addr }
}
pub async fn start_leader_pod_with_lease_ttl(
@@ -557,7 +574,13 @@ pub async fn start_leader_pod_with_lease_ttl(
cancel: CancellationToken,
) -> LeaderPodHandles {
let cache = Arc::new(PartitionedCache::new(cache_capacity));
- let (mock_cluster, kafka_producer) = create_test_kafka().await;
+ let topic = store
+ .get_changelog_config()
+ .await
+ .unwrap()
+ .expect("seed config/changelog before starting a pod")
+ .topic;
+ let kafka_producer = create_local_kafka_producer().await;
let heartbeat_secs = (lease_ttl as u64 / 3).max(1);
let inflight = Arc::new(InflightTracker::new());
@@ -567,8 +590,8 @@ pub async fn start_leader_pod_with_lease_ttl(
// service never reads, and the floors survive a handoff in tests
// while production drops them.
let emitted_versions = Arc::new(personhog_leader::emitted::EmittedVersions::new(1_000_000));
- let recovery = test_recovery(&mock_cluster.topic, &mock_cluster.bootstrap_servers());
- let warming = test_warming_config(name, &mock_cluster.topic, &mock_cluster.bootstrap_servers());
+ let recovery = test_recovery(KAFKA_BOOTSTRAP);
+ let warming = test_warming_config(name, KAFKA_BOOTSTRAP);
let pools = Arc::new(WarmClientPools::new(
&warming.kafka,
name,
@@ -579,7 +602,9 @@ pub async fn start_leader_pod_with_lease_ttl(
// cannot gate anything. Left unacquired, because warming is what
// takes each partition's epoch here, as it does in production.
let authority = Arc::new(AuthorityClock::unclaimed());
- let fenced = Arc::new(fenced_producers_for(&mock_cluster.topic));
+ let fenced = Arc::new(test_fenced_producers());
+ let changelog = live_changelog(&topic);
+ let captures: ChangelogCaptures = Arc::new(DashMap::new());
let handler = LeaderHandoffHandler::new(
Arc::clone(&cache),
Arc::clone(&inflight),
@@ -587,7 +612,8 @@ pub async fn start_leader_pod_with_lease_ttl(
warming,
Arc::new(dashmap::DashMap::new()),
None,
- NUM_PARTITIONS,
+ changelog.clone(),
+ Arc::clone(&captures),
pools,
Arc::clone(&fenced),
Arc::clone(&authority),
@@ -616,7 +642,7 @@ pub async fn start_leader_pod_with_lease_ttl(
None,
Arc::new(DashMap::new()),
Arc::clone(&inflight),
- NUM_PARTITIONS,
+ captures,
Arc::clone(&dirty_index),
recovery,
PropertySizeLimits::new(655360, 524288),
@@ -643,11 +669,7 @@ pub async fn start_leader_pod_with_lease_ttl(
tokio::time::sleep(Duration::from_millis(10)).await;
- LeaderPodHandles {
- cache,
- leader_addr,
- topic: mock_cluster,
- }
+ LeaderPodHandles { cache, leader_addr }
}
// ── Helpers ─────────────────────────────────────────────────
@@ -717,9 +739,9 @@ pub async fn start_leader_with_pg_fallback(
}),
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&mock_cluster.topic, NUM_PARTITIONS),
Arc::new(DirtyIndex::new(1_000_000)),
- test_recovery(&mock_cluster.topic, &mock_cluster.bootstrap_servers()),
+ test_recovery(&mock_cluster.bootstrap_servers()),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(dashmap::DashMap::new()),
@@ -758,13 +780,12 @@ pub const BROKER_TXN_TIMEOUT: Duration = Duration::from_secs(30);
/// fencing on, its entire fenced write arm — the version settlement, the
/// cache eviction, the four-way error mapping — is unreachable from any
/// test.
-pub fn fenced_producers_for(topic: &str) -> personhog_leader::fencing::FencedChangelogProducers {
+pub fn test_fenced_producers() -> personhog_leader::fencing::FencedChangelogProducers {
let mut kafka = test_kafka_config();
kafka.kafka_hosts = KAFKA_BOOTSTRAP.to_string();
personhog_leader::fencing::FencedChangelogProducers::new(
personhog_leader::fencing::FencedProducerConfig {
kafka,
- topic: topic.to_string(),
init_timeout: Duration::from_secs(10),
commit_timeout: Duration::from_secs(10),
broker_txn_timeout: BROKER_TXN_TIMEOUT,
@@ -841,8 +862,7 @@ fn handoff_handler_with(
inflight: Arc<personhog_leader::inflight::InflightTracker>,
authority: Arc<AuthorityClock>,
) -> personhog_leader::coordination::LeaderHandoffHandler {
- let mut warming = test_warming_config("test", topic, KAFKA_BOOTSTRAP);
- warming.topic = topic.to_string();
+ let warming = test_warming_config("test", KAFKA_BOOTSTRAP);
personhog_leader::coordination::LeaderHandoffHandler::new(
Arc::new(PartitionedCache::new(1 << 20)),
inflight,
@@ -850,7 +870,8 @@ fn handoff_handler_with(
warming,
Arc::new(dashmap::DashMap::new()),
None,
- NUM_PARTITIONS,
+ live_changelog(topic),
+ Arc::new(DashMap::new()),
Arc::new(personhog_leader::warming::WarmClientPools::new(
&test_kafka_config(),
"test",
@@ -877,3 +898,30 @@ pub fn live_authority() -> Arc<AuthorityClock> {
clock.begin_session(Duration::from_secs(3600), std::time::Instant::now());
clock
}
+
+pub fn live_changelog(topic: &str) -> LiveChangelog {
+ LiveChangelog::new(ChangelogConfig::new(topic, NUM_PARTITIONS))
+}
+
+pub fn target(topic: &str) -> ChangelogTarget {
+ ChangelogTarget {
+ topic: topic.to_string(),
+ partitions: NUM_PARTITIONS,
+ }
+}
+
+/// Every partition captured on `topic`, as a pod that serves them all holds it.
+pub fn served_captures(topic: &str, partitions: u32) -> ChangelogCaptures {
+ let config = ChangelogConfig::new(topic, partitions);
+ Arc::new(
+ (0..partitions)
+ .map(|partition| {
+ let capture = Capture {
+ topic: Arc::from(topic),
+ config: config.clone(),
+ };
+ (partition, capture)
+ })
+ .collect(),
+ )
+}
diff --git a/rust/personhog-leader/tests/fence_integration.rs b/rust/personhog-leader/tests/fence_integration.rs
index eedc7d682717..53cdcd130d9a 100644
--- a/rust/personhog-leader/tests/fence_integration.rs
+++ b/rust/personhog-leader/tests/fence_integration.rs
@@ -10,8 +10,8 @@ use std::time::Duration;
use dashmap::DashMap;
use common::{
- create_leader_client, create_test_kafka, seed_person, test_cached_person, test_recovery,
- unique_team_id, NUM_PARTITIONS,
+ create_leader_client, create_test_kafka, seed_person, served_captures, test_cached_person,
+ test_recovery, unique_team_id, NUM_PARTITIONS,
};
use personhog_common::partitioning::partition_for_person;
use personhog_leader::cache::{
@@ -89,6 +89,7 @@ struct FenceHarness {
dirty_index: Arc<DirtyIndex>,
locks: Arc<DashMap<PersonCacheKey, Arc<tokio::sync::Mutex<()>>>>,
fences: FenceMap,
+ captures: personhog_leader::coordination::ChangelogCaptures,
_cancel: CancellationToken,
/// Kept alive so the topic outlives the test that reads it.
topic: common::TestTopic,
@@ -114,14 +115,15 @@ async fn start_fence_harness(mut seed: CachedPerson, fallback: Option<PgFallback
let fences: FenceMap = Arc::new(DashMap::new());
// Recovery consumes the same topic so a post-death cache miss can
// recover the death document.
+ let captures = served_captures(&mock_cluster.topic, NUM_PARTITIONS);
let service = PersonHogLeaderService::new(
Arc::clone(&cache),
fallback,
Arc::clone(&locks),
Arc::clone(&inflight),
- NUM_PARTITIONS,
+ Arc::clone(&captures),
Arc::clone(&dirty_index),
- test_recovery(&mock_cluster.topic, &bootstrap),
+ test_recovery(&bootstrap),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::clone(&fences),
@@ -161,6 +163,7 @@ async fn start_fence_harness(mut seed: CachedPerson, fallback: Option<PgFallback
dirty_index,
locks,
fences,
+ captures,
_cancel: cancel,
topic: mock_cluster,
}
@@ -594,7 +597,10 @@ async fn a_revival_is_served_once_the_death_documents_mark_settles() {
.dirty_index
.get(&key)
.expect("the release marks the death record");
- let (pruned, exhausted) = harness.dirty_index.prune_chunk(partition, mark.offset + 1);
+ let (pruned, exhausted) =
+ harness
+ .dirty_index
+ .prune_chunk(partition, &mark.stream, mark.offset + 1);
assert!(exhausted);
assert_eq!(pruned.death_marks.len(), 1);
drop_settled_death_documents(&harness.cache, &harness.locks, &pruned.death_marks).await;
@@ -1416,7 +1422,7 @@ async fn the_takeover_scan_rebuilds_exactly_the_partitions_live_fences() {
);
}
- let dropped = drop_partition_fences(&fences, partition_a, NUM_PARTITIONS);
+ let dropped = drop_partition_fences(&fences, partition_a, NUM_PARTITIONS, |_| false);
assert!(dropped >= 1);
assert!(
fences.get(&key(fenced_a)).is_none(),
@@ -1446,9 +1452,9 @@ async fn at_capacity_new_fences_shed_but_reseals_succeed() {
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&_mock_cluster.topic, NUM_PARTITIONS),
Arc::new(DirtyIndex::new(1_000_000)),
- test_recovery(&_mock_cluster.topic, &_mock_cluster.bootstrap_servers()),
+ test_recovery(&_mock_cluster.bootstrap_servers()),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(DashMap::new()),
@@ -2859,6 +2865,7 @@ async fn a_batched_fence_with_a_person_off_the_routed_partition_is_refused_whole
))
.await
.expect_err("a batch with a person off the routed partition is refused");
+ // A final answer, not a routing bounce: the caller falls back to single calls.
assert_eq!(status.code(), Code::InvalidArgument);
assert!(status.metadata().get(FENCED_METADATA_KEY).is_none());
@@ -2882,6 +2889,30 @@ async fn a_batched_fence_with_a_person_off_the_routed_partition_is_refused_whole
.expect("the foreign person carries no fence from the refused batch");
}
+/// A batch routed to a partition this pod no longer serves is a routing bounce,
+/// not a bad request, so the router retries it toward the owner.
+#[tokio::test]
+async fn a_batch_for_a_released_partition_is_a_routing_bounce() {
+ let mut harness = start_fence_harness(test_cached_person(), None).await;
+ let (foreign_id, foreign_partition) = person_on_other_partition(&harness);
+ harness.captures.remove(&foreign_partition);
+
+ let status = harness
+ .client
+ .fence_persons(with_partition(
+ FencePersonsRequest {
+ team_id: harness.team_id,
+ op_id: Uuid::now_v7().to_string(),
+ op_type: LifecycleOpType::Delete.into(),
+ person_ids: vec![foreign_id],
+ },
+ foreign_partition,
+ ))
+ .await
+ .expect_err("this pod does not serve the partition");
+ assert_eq!(status.code(), Code::FailedPrecondition);
+}
+
/// One `ReleaseFences` call closes every fence of an op on the routed
/// partition, producing each death document exactly as a single release
/// would.
diff --git a/rust/personhog-leader/tests/fencing_integration.rs b/rust/personhog-leader/tests/fencing_integration.rs
index dcc250f4c6cc..1c16932126eb 100644
--- a/rust/personhog-leader/tests/fencing_integration.rs
+++ b/rust/personhog-leader/tests/fencing_integration.rs
@@ -21,7 +21,7 @@ use personhog_proto::personhog::types::v1::Person;
use tokio::sync::Notify;
use tokio::time::sleep;
-use common::{test_kafka_config, KAFKA_BOOTSTRAP};
+use common::{target, test_kafka_config, KAFKA_BOOTSTRAP};
fn test_person(version: i64) -> Person {
Person {
@@ -74,7 +74,6 @@ async fn read_committed_count(topic: &str) -> usize {
const BROKER_TXN_TIMEOUT: Duration = Duration::from_secs(30);
fn fenced_producers_with(
- topic: &str,
window: Duration,
window_max_writes: usize,
lanes: usize,
@@ -83,7 +82,6 @@ fn fenced_producers_with(
kafka.kafka_hosts = KAFKA_BOOTSTRAP.to_string();
FencedChangelogProducers::new(FencedProducerConfig {
kafka,
- topic: topic.to_string(),
init_timeout: Duration::from_secs(10),
commit_timeout: Duration::from_secs(10),
broker_txn_timeout: BROKER_TXN_TIMEOUT,
@@ -94,27 +92,70 @@ fn fenced_producers_with(
})
}
-fn fenced_producers(topic: &str, lanes: usize) -> FencedChangelogProducers {
- fenced_producers_with(topic, Duration::from_millis(5), 32, lanes)
+fn fenced_producers(lanes: usize) -> FencedChangelogProducers {
+ fenced_producers_with(Duration::from_millis(5), 32, lanes)
}
/// The prepared path must fence exactly as the cold path does: a
/// connection built ahead of acquisition carries no broker transactional
/// state, so the epoch bump happens at `acquire` — and the previous
/// owner must find itself fenced by it.
+/// Ids are topic-scoped: a new-topic acquisition fences nothing on the old topic.
+#[tokio::test]
+async fn acquiring_on_another_topic_produces_there_and_discards_a_stale_preconnect() {
+ let old = format!("fence_gen_old_{}", uuid::Uuid::new_v4().simple());
+ let new = format!("fence_gen_new_{}", uuid::Uuid::new_v4().simple());
+
+ let holder = fenced_producers(1);
+ holder
+ .acquire(0, &target(&old))
+ .await
+ .expect("holder acquires the old topic");
+ holder
+ .produce(0, &test_person(1))
+ .await
+ .expect("holder produces on the old topic");
+
+ let successor = fenced_producers(1);
+ successor.preconnect(0, &target(&old)).await;
+ assert!(successor.has_prepared(0));
+ successor
+ .acquire(0, &target(&new))
+ .await
+ .expect("successor acquires on the new topic");
+ assert!(
+ !successor.has_prepared(0),
+ "a preconnect for another topic is discarded, not claimed"
+ );
+ successor
+ .produce(0, &test_person(2))
+ .await
+ .expect("successor produces on the new topic");
+ holder
+ .produce(0, &test_person(3))
+ .await
+ .expect("the old topic's holder is not fenced by a new-topic acquisition");
+
+ assert_eq!(read_committed_count(&old).await, 2);
+ assert_eq!(read_committed_count(&new).await, 1);
+}
+
#[tokio::test]
async fn a_prepared_connection_still_fences_the_previous_owner() {
let topic = format!("fence_prepared_{}", uuid::Uuid::new_v4().simple());
- let first = fenced_producers(&topic, 1);
- first.acquire(0).await.expect("first owner acquires");
+ let first = fenced_producers(1);
+ first
+ .acquire(0, &target(&topic))
+ .await
+ .expect("first owner acquires");
first
.produce(0, &test_person(1))
.await
.expect("first owner produces while unfenced");
- let second = fenced_producers(&topic, 1);
- second.preconnect(0).await;
+ let second = fenced_producers(1);
+ second.preconnect(0, &target(&topic)).await;
assert!(second.has_prepared(0), "preconnect parks a connection");
// The property the phase split exists for: the parked connection has
// touched no broker transactional state, so the serving owner is
@@ -123,7 +164,10 @@ async fn a_prepared_connection_still_fences_the_previous_owner() {
.produce(0, &test_person(10))
.await
.expect("a parked connection must not fence the serving owner");
- second.acquire(0).await.expect("second owner acquires");
+ second
+ .acquire(0, &target(&topic))
+ .await
+ .expect("second owner acquires");
assert!(
!second.has_prepared(0),
"acquisition consumes the parked connection"
@@ -145,8 +189,8 @@ async fn a_prepared_connection_still_fences_the_previous_owner() {
#[tokio::test]
async fn the_sweep_discards_parked_connections() {
let topic = format!("fence_prepared_{}", uuid::Uuid::new_v4().simple());
- let producers = fenced_producers(&topic, 1);
- producers.preconnect(6).await;
+ let producers = fenced_producers(1);
+ producers.preconnect(6, &target(&topic)).await;
assert!(producers.has_prepared(6), "preconnect parks a connection");
producers.sweep_prepared();
@@ -163,8 +207,8 @@ async fn the_sweep_discards_parked_connections() {
#[tokio::test]
async fn a_released_partition_discards_its_prepared_connection() {
let topic = format!("fence_prepared_{}", uuid::Uuid::new_v4().simple());
- let producers = fenced_producers(&topic, 1);
- producers.preconnect(5).await;
+ let producers = fenced_producers(1);
+ producers.preconnect(5, &target(&topic)).await;
assert!(producers.has_prepared(5), "preconnect parks a connection");
producers.release(5);
assert!(
@@ -181,12 +225,13 @@ async fn a_released_partition_discards_its_prepared_connection() {
#[tokio::test]
async fn concurrent_preconnects_collapse_to_one_dial() {
let topic = format!("fence_prepared_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
+ let producers = Arc::new(fenced_producers(1));
let calls: Vec<_> = (0..20)
.map(|_| {
let p = Arc::clone(&producers);
- tokio::spawn(async move { p.preconnect(3).await })
+ let t = target(&topic);
+ tokio::spawn(async move { p.preconnect(3, &t).await })
})
.collect();
for call in calls {
@@ -209,12 +254,12 @@ async fn concurrent_preconnects_collapse_to_one_dial() {
/// after a broker hiccup.
#[tokio::test]
async fn a_failed_preconnect_releases_its_claim() {
+ let topic = "fence_unroutable";
let mut kafka = test_kafka_config();
// An unroutable broker fails the dial at its bounded metadata ping.
kafka.kafka_hosts = "127.0.0.1:1".to_string();
let producers = FencedChangelogProducers::new(FencedProducerConfig {
kafka,
- topic: "fence_unroutable".to_string(),
init_timeout: Duration::from_secs(2),
commit_timeout: Duration::from_secs(2),
broker_txn_timeout: BROKER_TXN_TIMEOUT,
@@ -224,10 +269,10 @@ async fn a_failed_preconnect_releases_its_claim() {
settle_budget: Duration::from_secs(5),
});
- producers.preconnect(0).await;
+ producers.preconnect(0, &target(topic)).await;
assert!(!producers.has_prepared(0), "a failed dial parks nothing");
- producers.preconnect(0).await;
+ producers.preconnect(0, &target(topic)).await;
assert_eq!(
producers.connect_attempts_for_test(),
2,
@@ -241,13 +286,13 @@ async fn a_failed_preconnect_releases_its_claim() {
/// single-flight exists to stop.
#[tokio::test]
async fn a_stale_dials_failure_leaves_the_replacements_claim() {
+ let topic = "fence_unroutable";
let mut kafka = test_kafka_config();
// An unroutable broker holds the dial open until its 2s bound, so
// the mid-flight claim churn below happens while it is in flight.
kafka.kafka_hosts = "127.0.0.1:1".to_string();
let producers = Arc::new(FencedChangelogProducers::new(FencedProducerConfig {
kafka,
- topic: "fence_unroutable".to_string(),
init_timeout: Duration::from_secs(2),
commit_timeout: Duration::from_secs(2),
broker_txn_timeout: BROKER_TXN_TIMEOUT,
@@ -259,7 +304,7 @@ async fn a_stale_dials_failure_leaves_the_replacements_claim() {
let dial = {
let p = Arc::clone(&producers);
- tokio::spawn(async move { p.preconnect(0).await })
+ tokio::spawn(async move { p.preconnect(0, &target(topic)).await })
};
while !producers.has_connecting_claim_for_test(0) {
tokio::task::yield_now().await;
@@ -284,18 +329,18 @@ async fn a_stale_dials_failure_leaves_the_replacements_claim() {
#[tokio::test]
async fn the_sweep_clears_orphaned_claims_but_not_live_ones() {
let topic = format!("fence_prepared_{}", uuid::Uuid::new_v4().simple());
- let producers = fenced_producers(&topic, 1);
+ let producers = fenced_producers(1);
producers.stage_connecting_for_test(1, Duration::from_secs(60));
producers.stage_connecting_for_test(2, Duration::from_secs(0));
producers.sweep_prepared();
- producers.preconnect(1).await;
+ producers.preconnect(1, &target(&topic)).await;
assert!(
producers.has_prepared(1),
"the orphaned claim must clear so preconnect works again"
);
- producers.preconnect(2).await;
+ producers.preconnect(2, &target(&topic)).await;
assert_eq!(
producers.connect_attempts_for_test(),
1,
@@ -315,15 +360,21 @@ async fn second_acquisition_fences_the_first_producer() {
tokio::time::timeout(Duration::from_secs(60), async {
let topic = format!("fence_test_{}", uuid::Uuid::new_v4().simple());
- let first = fenced_producers(&topic, 1);
- first.acquire(0).await.expect("first owner acquires");
+ let first = fenced_producers(1);
+ first
+ .acquire(0, &target(&topic))
+ .await
+ .expect("first owner acquires");
first
.produce(0, &test_person(1))
.await
.expect("first owner produces while unfenced");
- let second = fenced_producers(&topic, 1);
- second.acquire(0).await.expect("second owner acquires");
+ let second = fenced_producers(1);
+ second
+ .acquire(0, &target(&topic))
+ .await
+ .expect("second owner acquires");
second
.produce(0, &test_person(2))
.await
@@ -345,8 +396,11 @@ async fn second_acquisition_fences_the_first_producer() {
async fn successive_writes_rotate_across_lanes() {
tokio::time::timeout(Duration::from_secs(60), async {
let topic = format!("fence_test_{}", uuid::Uuid::new_v4().simple());
- let producers = fenced_producers(&topic, 2);
- producers.acquire(0).await.expect("acquires");
+ let producers = fenced_producers(2);
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquires");
producers
.produce(0, &test_person(1))
.await
@@ -374,15 +428,21 @@ async fn a_successor_fences_every_lane_of_the_predecessor() {
tokio::time::timeout(Duration::from_secs(60), async {
let topic = format!("fence_test_{}", uuid::Uuid::new_v4().simple());
- let first = fenced_producers(&topic, 2);
- first.acquire(0).await.expect("first owner acquires");
+ let first = fenced_producers(2);
+ first
+ .acquire(0, &target(&topic))
+ .await
+ .expect("first owner acquires");
first
.produce_on_lane_for_test(0, 1, &test_person(1))
.await
.expect("first owner produces on lane 1 while unfenced");
- let second = fenced_producers(&topic, 2);
- second.acquire(0).await.expect("second owner acquires");
+ let second = fenced_producers(2);
+ second
+ .acquire(0, &target(&topic))
+ .await
+ .expect("second owner acquires");
match first.produce_on_lane_for_test(0, 1, &test_person(2)).await {
Err(FencedProduceError::Fenced)
@@ -405,8 +465,11 @@ async fn a_successor_fences_every_lane_of_the_predecessor() {
#[tokio::test]
async fn a_write_parks_behind_a_commit_on_another_lane() {
let topic = format!("fence_serial_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 2));
- producers.acquire(0).await.expect("acquire the fence");
+ let producers = Arc::new(fenced_producers(2));
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
producers.begin_committing_for_test(0, 0);
let write = {
let p = Arc::clone(&producers);
@@ -443,8 +506,11 @@ async fn a_write_parks_behind_a_commit_on_another_lane() {
#[tokio::test]
async fn a_parked_writer_picks_again_when_it_wakes() {
let topic = format!("fence_repick_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 2));
- producers.acquire(0).await.expect("acquire the fence");
+ let producers = Arc::new(fenced_producers(2));
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
producers.begin_committing_for_test(0, 0);
producers.begin_committing_for_test(0, 1);
@@ -484,9 +550,9 @@ async fn a_parked_writer_picks_again_when_it_wakes() {
#[tokio::test]
async fn preconnect_dials_every_lane() {
let topic = format!("fence_preconnect_lanes_{}", uuid::Uuid::new_v4().simple());
- let producers = fenced_producers(&topic, 2);
+ let producers = fenced_producers(2);
- producers.preconnect(0).await;
+ producers.preconnect(0, &target(&topic)).await;
assert!(
producers.has_prepared(0),
"preconnect parks the connections"
@@ -497,7 +563,10 @@ async fn preconnect_dials_every_lane() {
"one dial per lane"
);
- producers.acquire(0).await.expect("acquire the fence");
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
assert_eq!(
producers.connect_attempts_for_test(),
2,
@@ -511,8 +580,11 @@ async fn preconnect_dials_every_lane() {
#[tokio::test]
async fn a_poisoned_gate_gives_up_the_partition() {
let topic = format!("fence_poisoned_gate_{}", uuid::Uuid::new_v4().simple());
- let producers = fenced_producers(&topic, 2);
- producers.acquire(0).await.expect("acquire the fence");
+ let producers = fenced_producers(2);
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
producers.poison_gate_for_test(0, 1);
@@ -535,7 +607,6 @@ async fn a_write_parked_forever_is_bounded() {
kafka.kafka_hosts = KAFKA_BOOTSTRAP.to_string();
let producers = FencedChangelogProducers::new(FencedProducerConfig {
kafka,
- topic,
init_timeout: Duration::from_secs(10),
// Sizes the produce bound well under the test's own patience.
commit_timeout: Duration::from_millis(100),
@@ -545,7 +616,10 @@ async fn a_write_parked_forever_is_bounded() {
settle_budget: Duration::from_secs(5),
lanes: 1,
});
- producers.acquire(0).await.expect("acquire the fence");
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
producers.begin_committing_for_test(0, 0);
let outcome = tokio::time::timeout(
@@ -566,8 +640,11 @@ async fn a_write_parked_forever_is_bounded() {
#[tokio::test]
async fn a_condemned_lane_above_zero_gives_up_the_partition() {
let topic = format!("fence_condemned_lane_{}", uuid::Uuid::new_v4().simple());
- let producers = fenced_producers(&topic, 2);
- producers.acquire(0).await.expect("acquire the fence");
+ let producers = fenced_producers(2);
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
producers.condemn_for_test(0, 1);
@@ -587,8 +664,11 @@ async fn a_condemned_lane_above_zero_gives_up_the_partition() {
#[tokio::test]
async fn concurrent_writes_share_a_window() {
let topic = format!("fence_test_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
- producers.acquire(0).await.expect("acquire");
+ let producers = Arc::new(fenced_producers(1));
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire");
let a = {
let p = Arc::clone(&producers);
@@ -609,8 +689,11 @@ async fn concurrent_writes_share_a_window() {
#[tokio::test]
async fn a_filled_window_commits_before_its_timer() {
let topic = format!("fence_test_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers_with(&topic, Duration::from_secs(60), 3, 1));
- producers.acquire(0).await.expect("acquire");
+ let producers = Arc::new(fenced_producers_with(Duration::from_secs(60), 3, 1));
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire");
let writes: Vec<_> = (1..=3i64)
.map(|v| {
@@ -640,8 +723,11 @@ async fn sustained_writes_across_window_boundaries() {
// failure.
tokio::time::timeout(Duration::from_secs(60), async {
let topic = format!("fence_test_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
- producers.acquire(0).await.expect("acquire");
+ let producers = Arc::new(fenced_producers(1));
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire");
let writes: Vec<_> = (0..200i64)
.map(|k| {
@@ -677,13 +763,11 @@ async fn a_committer_that_vanishes_leaves_the_outcome_in_doubt() {
let topic = format!("fence_orphan_{}", uuid::Uuid::new_v4().simple());
// A long window keeps the committer asleep, so the write is parked on
// an outcome that is still nobody's to report.
- let producers = Arc::new(fenced_producers_with(
- &topic,
- Duration::from_secs(30),
- 32,
- 1,
- ));
- producers.acquire(0).await.expect("acquire the fence");
+ let producers = Arc::new(fenced_producers_with(Duration::from_secs(30), 32, 1));
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
let writing = {
let p = Arc::clone(&producers);
@@ -715,8 +799,11 @@ async fn a_committer_that_vanishes_leaves_the_outcome_in_doubt() {
#[tokio::test]
async fn a_cancelled_produce_does_not_wedge_the_partition() {
let topic = format!("fence_cancel_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
- producers.acquire(0).await.expect("acquire");
+ let producers = Arc::new(fenced_producers(1));
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire");
{
let p = Arc::clone(&producers);
@@ -757,8 +844,11 @@ async fn a_successors_init_aborts_the_predecessors_open_window() {
// has taken the epoch, because "invisible while uncommitted" proves
// nothing. What has to be shown is that the record can never become
// visible once the successor owns the partition.
- let first = Arc::new(fenced_producers_with(&topic, Duration::from_secs(1), 32, 1));
- first.acquire(0).await.expect("first owner acquires");
+ let first = Arc::new(fenced_producers_with(Duration::from_secs(1), 32, 1));
+ first
+ .acquire(0, &target(&topic))
+ .await
+ .expect("first owner acquires");
{
let p = Arc::clone(&first);
let mut inflight = Box::pin(async move { p.produce(0, &test_person(1)).await });
@@ -771,8 +861,11 @@ async fn a_successors_init_aborts_the_predecessors_open_window() {
// Successor takes the partition before that window closes, as a
// warming new owner does.
- let second = fenced_producers(&topic, 1);
- second.acquire(0).await.expect("successor acquires");
+ let second = fenced_producers(1);
+ second
+ .acquire(0, &target(&topic))
+ .await
+ .expect("successor acquires");
// Now let the predecessor's committer run. This is the moment the
// drain's wait exists to prevent: an abandoned record committing
@@ -792,13 +885,10 @@ async fn a_successors_init_aborts_the_predecessors_open_window() {
// record arriving. Otherwise this test passes just as well when the
// send never left the client.
let control_topic = format!("fence_abort_control_{}", uuid::Uuid::new_v4().simple());
- let lone = Arc::new(fenced_producers_with(
- &control_topic,
- Duration::from_secs(1),
- 32,
- 1,
- ));
- lone.acquire(0).await.expect("control owner acquires");
+ let lone = Arc::new(fenced_producers_with(Duration::from_secs(1), 32, 1));
+ lone.acquire(0, &target(&control_topic))
+ .await
+ .expect("control owner acquires");
{
let p = Arc::clone(&lone);
let mut inflight = Box::pin(async move { p.produce(0, &test_person(1)).await });
@@ -826,10 +916,13 @@ async fn a_successors_init_aborts_the_predecessors_open_window() {
#[tokio::test]
async fn a_fence_taken_for_an_unfinished_warm_is_given_back() {
let topic = format!("fence_guard_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
+ let producers = Arc::new(fenced_producers(1));
{
- producers.acquire(0).await.expect("acquire");
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire");
let _guard = FenceGuard::new(Arc::clone(&producers), 0, "warm");
// The warm ends here without returning, as a torn-down attempt
// does.
@@ -845,9 +938,12 @@ async fn a_fence_taken_for_an_unfinished_warm_is_given_back() {
#[tokio::test]
async fn a_completed_warm_keeps_its_fence() {
let topic = format!("fence_guard_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
+ let producers = Arc::new(fenced_producers(1));
- producers.acquire(0).await.expect("acquire");
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire");
FenceGuard::new(Arc::clone(&producers), 0, "warm").keep();
producers
@@ -865,8 +961,11 @@ async fn a_completed_warm_keeps_its_fence() {
async fn a_condemnation_nudges_repair_once() {
let topic = format!("fence_repair_{}", uuid::Uuid::new_v4().simple());
let nudge = Arc::new(Notify::new());
- let producers = fenced_producers(&topic, 1).with_repair_nudge(Arc::clone(&nudge));
- producers.acquire(3).await.expect("acquire the fence");
+ let producers = fenced_producers(1).with_repair_nudge(Arc::clone(&nudge));
+ producers
+ .acquire(3, &target(&topic))
+ .await
+ .expect("acquire the fence");
producers.condemn_for_test(3, 0);
tokio::time::timeout(Duration::from_secs(5), nudge.notified())
@@ -897,8 +996,11 @@ async fn a_condemnation_nudges_repair_once() {
#[tokio::test]
async fn a_condemned_producer_stops_claiming_the_partition() {
let topic = format!("fence_condemned_{}", uuid::Uuid::new_v4().simple());
- let producers = fenced_producers(&topic, 1);
- producers.acquire(0).await.expect("acquire the fence");
+ let producers = fenced_producers(1);
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
producers
.produce(0, &test_person(1))
.await
@@ -915,7 +1017,10 @@ async fn a_condemned_producer_stops_claiming_the_partition() {
// re-acquisition is the only thing that makes the partition writable
// again, and it can only run against a partition this pod no longer
// claims to fence.
- producers.acquire(0).await.expect("re-acquire the fence");
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("re-acquire the fence");
producers
.produce(0, &test_person(3))
.await
@@ -930,8 +1035,11 @@ async fn a_condemned_producer_stops_claiming_the_partition() {
#[tokio::test]
async fn an_abandoned_guard_does_not_evict_its_replacement() {
let topic = format!("fence_guard_id_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
- producers.acquire(0).await.expect("first acquire");
+ let producers = Arc::new(fenced_producers(1));
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("first acquire");
// A warm takes the fence, then never finishes.
let stale = FenceGuard::new(Arc::clone(&producers), 0, "warm");
@@ -939,7 +1047,10 @@ async fn an_abandoned_guard_does_not_evict_its_replacement() {
// Meanwhile the partition is released and taken again, so what is
// installed is no longer what the guard is answerable for.
producers.release(0);
- producers.acquire(0).await.expect("re-acquire");
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("re-acquire");
drop(stale);
@@ -957,12 +1068,12 @@ async fn an_abandoned_guard_does_not_evict_its_replacement() {
#[tokio::test]
async fn healing_retakes_a_fence_for_a_served_partition() {
let topic = format!("fence_heal_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
+ let producers = Arc::new(fenced_producers(1));
let inflight = InflightTracker::new();
let clock = AuthorityClock::unclaimed();
clock.begin_session(Duration::from_secs(30), std::time::Instant::now());
- let outcome = heal_fence(&producers, &inflight, &clock, 0).await;
+ let outcome = heal_fence(&producers, &inflight, &clock, 0, &target(&topic)).await;
assert_eq!(
outcome,
Ok(HealOutcome::Healed),
@@ -991,19 +1102,22 @@ async fn healing_without_standing_does_not_steal_the_epoch() {
let inflight = InflightTracker::new();
// The partition's real owner, holding a working fence.
- let owner = Arc::new(fenced_producers(&topic, 1));
- owner.acquire(0).await.expect("the owner takes its fence");
+ let owner = Arc::new(fenced_producers(1));
+ owner
+ .acquire(0, &target(&topic))
+ .await
+ .expect("the owner takes its fence");
owner
.produce(0, &test_person(1))
.await
.expect("the owner can write");
// A pod whose claim is gone tries to heal the same partition.
- let zombie = Arc::new(fenced_producers(&topic, 1));
+ let zombie = Arc::new(fenced_producers(1));
let lapsed = AuthorityClock::unclaimed();
lapsed.begin_session(Duration::from_secs(30), std::time::Instant::now());
lapsed.surrender();
- let outcome = heal_fence(&zombie, &inflight, &lapsed, 0).await;
+ let outcome = heal_fence(&zombie, &inflight, &lapsed, 0, &target(&topic)).await;
assert_eq!(outcome, Ok(HealOutcome::Intact));
owner
@@ -1019,14 +1133,17 @@ async fn healing_skips_a_partition_under_handoff() {
let topic = format!("fence_heal_{}", uuid::Uuid::new_v4().simple());
let inflight = InflightTracker::new();
- let owner = Arc::new(fenced_producers(&topic, 1));
- owner.acquire(0).await.expect("the owner takes its fence");
+ let owner = Arc::new(fenced_producers(1));
+ owner
+ .acquire(0, &target(&topic))
+ .await
+ .expect("the owner takes its fence");
- let other = Arc::new(fenced_producers(&topic, 1));
+ let other = Arc::new(fenced_producers(1));
let valid = AuthorityClock::unclaimed();
valid.begin_session(Duration::from_secs(30), std::time::Instant::now());
inflight.fence(0);
- let outcome = heal_fence(&other, &inflight, &valid, 0).await;
+ let outcome = heal_fence(&other, &inflight, &valid, 0, &target(&topic)).await;
assert_eq!(outcome, Ok(HealOutcome::Intact));
owner
@@ -1043,7 +1160,7 @@ async fn healing_skips_a_partition_under_handoff() {
#[tokio::test]
async fn healing_gives_back_a_fence_it_lost_standing_for() {
let topic = format!("fence_heal_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
+ let producers = Arc::new(fenced_producers(1));
let inflight = InflightTracker::new();
let clock = Arc::new(AuthorityClock::unclaimed());
@@ -1054,7 +1171,7 @@ async fn healing_gives_back_a_fence_it_lost_standing_for() {
losing.surrender();
});
- let outcome = heal_fence(&producers, &inflight, &clock, 0).await;
+ let outcome = heal_fence(&producers, &inflight, &clock, 0, &target(&topic)).await;
assert_ne!(
outcome,
Ok(HealOutcome::Healed),
@@ -1077,8 +1194,11 @@ async fn healing_gives_back_a_fence_it_lost_standing_for() {
#[tokio::test]
async fn a_writer_woken_onto_a_condemned_producer_is_bounced() {
let topic = format!("fence_woken_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
- producers.acquire(0).await.expect("acquire the fence");
+ let producers = Arc::new(fenced_producers(1));
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
// Stage the gate exactly as a commit in flight leaves it, so the
// write below parks rather than opening its own window.
@@ -1119,17 +1239,15 @@ async fn a_writer_woken_onto_a_condemned_producer_is_bounced() {
#[tokio::test]
async fn healing_leaves_a_fence_it_already_holds_alone() {
let topic = format!("fence_heal_noop_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers_with(
- &topic,
- Duration::from_millis(600),
- 32,
- 1,
- ));
+ let producers = Arc::new(fenced_producers_with(Duration::from_millis(600), 32, 1));
let inflight = InflightTracker::new();
let clock = AuthorityClock::unclaimed();
clock.begin_session(Duration::from_secs(30), std::time::Instant::now());
- producers.acquire(0).await.expect("take the fence");
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("take the fence");
// In flight: the window is open and its commit has not fired.
let writing = {
@@ -1139,7 +1257,7 @@ async fn healing_leaves_a_fence_it_already_holds_alone() {
tokio::time::sleep(Duration::from_millis(100)).await;
// A reconcile tick with everything healthy.
- let outcome = heal_fence(&producers, &inflight, &clock, 0).await;
+ let outcome = heal_fence(&producers, &inflight, &clock, 0, &target(&topic)).await;
assert_eq!(outcome, Ok(HealOutcome::Intact));
let result = writing.await.expect("the write task must not panic");
@@ -1161,8 +1279,11 @@ async fn healing_leaves_a_fence_it_already_holds_alone() {
#[tokio::test]
async fn a_committer_that_never_reports_condemns_its_producer() {
let topic = format!("fence_lost_commit_{}", uuid::Uuid::new_v4().simple());
- let producers = fenced_producers(&topic, 1);
- producers.acquire(0).await.expect("acquire the fence");
+ let producers = fenced_producers(1);
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
producers.panic_next_commit_for_test(0);
@@ -1198,13 +1319,11 @@ async fn a_committer_that_never_reports_condemns_its_producer() {
async fn settling_commits_an_abandoned_record_before_the_handoff_advances() {
let topic = format!("fence_settle_{}", uuid::Uuid::new_v4().simple());
- let first = Arc::new(fenced_producers_with(
- &topic,
- Duration::from_secs(15),
- 32,
- 1,
- ));
- first.acquire(0).await.expect("first owner acquires");
+ let first = Arc::new(fenced_producers_with(Duration::from_secs(15), 32, 1));
+ first
+ .acquire(0, &target(&topic))
+ .await
+ .expect("first owner acquires");
{
let p = Arc::clone(&first);
let mut inflight = Box::pin(async move { p.produce(0, &test_person(1)).await });
@@ -1238,8 +1357,11 @@ async fn settling_commits_an_abandoned_record_before_the_handoff_advances() {
// And it settled before the successor existed, which is what makes
// the ack a boundary rather than the start of a race.
- let second = fenced_producers(&topic, 1);
- second.acquire(0).await.expect("successor acquires");
+ let second = fenced_producers(1);
+ second
+ .acquire(0, &target(&topic))
+ .await
+ .expect("successor acquires");
assert_eq!(
read_committed_count(&topic).await,
1,
@@ -1267,18 +1389,16 @@ async fn a_resume_does_not_fence_the_window_its_own_warm_admitted() {
// Long enough that the write is still sitting in an open window when
// the resume runs — a window that closed first would be committed
// already and could not be fenced.
- let producers = Arc::new(fenced_producers_with(
- &topic,
- Duration::from_secs(10),
- 32,
- 1,
- ));
+ let producers = Arc::new(fenced_producers_with(Duration::from_secs(10), 32, 1));
let handler = common::test_handoff_handler(&topic, Arc::clone(&producers));
// A record for the warm to read, which also brings the topic into
// existence — `fetch_watermarks` has nothing to answer for a topic
// no one has produced to.
- producers.acquire(0).await.expect("seed the topic");
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("seed the topic");
producers
.produce(0, &test_person(1))
.await
@@ -1316,8 +1436,11 @@ async fn a_resume_does_not_fence_the_window_its_own_warm_admitted() {
#[tokio::test]
async fn a_poisoned_window_is_aborted_rather_than_committed() {
let topic = format!("fence_poison_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers_with(&topic, Duration::from_secs(1), 32, 1));
- producers.acquire(0).await.expect("acquire the fence");
+ let producers = Arc::new(fenced_producers_with(Duration::from_secs(1), 32, 1));
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
let writing = {
let p = Arc::clone(&producers);
@@ -1360,8 +1483,11 @@ async fn a_poisoned_window_is_aborted_rather_than_committed() {
#[tokio::test]
async fn a_window_that_cannot_settle_still_lets_the_handoff_proceed() {
let topic = format!("fence_unsettled_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
- producers.acquire(0).await.expect("acquire the fence");
+ let producers = Arc::new(fenced_producers(1));
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
producers
.produce(0, &test_person(1))
.await
@@ -1388,15 +1514,13 @@ async fn a_window_that_cannot_settle_still_lets_the_handoff_proceed() {
#[tokio::test]
async fn a_repeated_resume_does_not_fence_the_window_the_first_one_admitted() {
let topic = format!("fence_resume_twice_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers_with(
- &topic,
- Duration::from_secs(10),
- 32,
- 1,
- ));
+ let producers = Arc::new(fenced_producers_with(Duration::from_secs(10), 32, 1));
let handler = common::test_handoff_handler(&topic, Arc::clone(&producers));
- producers.acquire(0).await.expect("seed the topic");
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("seed the topic");
producers
.produce(0, &test_person(1))
.await
@@ -1433,8 +1557,11 @@ async fn a_repeated_resume_does_not_fence_the_window_the_first_one_admitted() {
#[tokio::test]
async fn a_drain_waits_for_a_commit_that_is_already_running() {
let topic = format!("fence_settle_mid_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 2));
- producers.acquire(0).await.expect("acquire the fence");
+ let producers = Arc::new(fenced_producers(2));
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
// The gate a commit in flight leaves behind, on the second lane.
producers.begin_committing_for_test(0, 1);
@@ -1466,15 +1593,13 @@ async fn a_drain_waits_for_a_commit_that_is_already_running() {
#[tokio::test]
async fn a_resume_that_took_the_fence_itself_does_not_take_it_again() {
let topic = format!("fence_resume_own_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers_with(
- &topic,
- Duration::from_secs(10),
- 32,
- 1,
- ));
+ let producers = Arc::new(fenced_producers_with(Duration::from_secs(10), 32, 1));
let handler = common::test_handoff_handler(&topic, Arc::clone(&producers));
- producers.acquire(0).await.expect("seed the topic");
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("seed the topic");
// Handed away, then cancelled: the drain retires the mark and the
// resume has to take the fence — and record it — on its own.
handler
@@ -1507,8 +1632,11 @@ async fn a_resume_that_took_the_fence_itself_does_not_take_it_again() {
#[tokio::test]
async fn warming_without_a_confirmed_lease_does_not_take_the_epoch() {
let topic = format!("fence_warm_auth_{}", uuid::Uuid::new_v4().simple());
- let owner = Arc::new(fenced_producers(&topic, 1));
- owner.acquire(0).await.expect("the owner takes its fence");
+ let owner = Arc::new(fenced_producers(1));
+ owner
+ .acquire(0, &target(&topic))
+ .await
+ .expect("the owner takes its fence");
owner
.produce(0, &test_person(1))
.await
@@ -1516,7 +1644,7 @@ async fn warming_without_a_confirmed_lease_does_not_take_the_epoch() {
let lapsed = common::live_authority();
lapsed.surrender();
- let zombie = Arc::new(fenced_producers(&topic, 1));
+ let zombie = Arc::new(fenced_producers(1));
let handler = common::test_handoff_handler_with_authority(&topic, Arc::clone(&zombie), lapsed);
handler
.warm_partition(0)
@@ -1534,8 +1662,11 @@ async fn warming_without_a_confirmed_lease_does_not_take_the_epoch() {
#[tokio::test]
async fn a_resume_without_a_confirmed_lease_does_not_take_the_epoch() {
let topic = format!("fence_resume_auth_{}", uuid::Uuid::new_v4().simple());
- let owner = Arc::new(fenced_producers(&topic, 1));
- owner.acquire(0).await.expect("the owner takes its fence");
+ let owner = Arc::new(fenced_producers(1));
+ owner
+ .acquire(0, &target(&topic))
+ .await
+ .expect("the owner takes its fence");
owner
.produce(0, &test_person(1))
.await
@@ -1543,7 +1674,7 @@ async fn a_resume_without_a_confirmed_lease_does_not_take_the_epoch() {
let lapsed = common::live_authority();
lapsed.surrender();
- let zombie = Arc::new(fenced_producers(&topic, 1));
+ let zombie = Arc::new(fenced_producers(1));
let handler = common::test_handoff_handler_with_authority(&topic, Arc::clone(&zombie), lapsed);
handler
.resume_partition(0)
@@ -1567,11 +1698,13 @@ async fn a_warm_that_loses_its_claim_mid_acquire_gives_the_fence_back() {
// Seeded through a separate owner so the warm below is a real one: a
// warm that fails for want of a topic would leave the fence behind
// through its guard and prove nothing about this branch.
- let seed = fenced_producers(&topic, 1);
- seed.acquire(0).await.expect("seed the topic");
+ let seed = fenced_producers(1);
+ seed.acquire(0, &target(&topic))
+ .await
+ .expect("seed the topic");
seed.produce(0, &test_person(1)).await.expect("seed record");
- let producers = Arc::new(fenced_producers(&topic, 1));
+ let producers = Arc::new(fenced_producers(1));
let clock = common::live_authority();
let handler = common::test_handoff_handler_with_authority(
&topic,
@@ -1615,7 +1748,7 @@ async fn a_warm_that_loses_its_claim_mid_acquire_gives_the_fence_back() {
#[tokio::test]
async fn a_resume_that_loses_its_claim_mid_acquire_gives_the_fence_back() {
let topic = format!("fence_resume_midauth_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
+ let producers = Arc::new(fenced_producers(1));
let clock = common::live_authority();
let handler = common::test_handoff_handler_with_authority(
&topic,
@@ -1660,9 +1793,12 @@ async fn a_resume_that_loses_its_claim_mid_acquire_gives_the_fence_back() {
#[tokio::test]
async fn the_drain_refuses_new_writes_while_it_is_still_waiting() {
let topic = format!("fence_drain_order_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
+ let producers = Arc::new(fenced_producers(1));
let inflight = Arc::new(InflightTracker::new());
- producers.acquire(0).await.expect("acquire the fence");
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
let handler = common::test_handoff_handler_with_inflight(
&topic,
@@ -1696,10 +1832,13 @@ async fn the_drain_refuses_new_writes_while_it_is_still_waiting() {
#[tokio::test]
async fn releasing_a_partition_retires_its_fresh_fence_mark() {
let topic = format!("fence_release_mark_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
+ let producers = Arc::new(fenced_producers(1));
let handler = common::test_handoff_handler(&topic, Arc::clone(&producers));
- producers.acquire(0).await.expect("seed the topic");
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("seed the topic");
producers
.produce(0, &test_person(1))
.await
@@ -1726,9 +1865,12 @@ async fn releasing_a_partition_retires_its_fresh_fence_mark() {
#[tokio::test]
async fn verifying_a_served_partition_retakes_a_condemned_fence() {
let topic = format!("fence_verify_heal_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
+ let producers = Arc::new(fenced_producers(1));
let handler = common::test_handoff_handler(&topic, Arc::clone(&producers));
- producers.acquire(0).await.expect("acquire the fence");
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
// The state a failed abort or an unknown commit leaves behind.
producers.condemn_for_test(0, 0);
@@ -1766,7 +1908,6 @@ async fn a_heal_that_cannot_acquire_does_not_fail_the_run() {
kafka.kafka_hosts = "127.0.0.1:1".to_string();
let producers = Arc::new(FencedChangelogProducers::new(FencedProducerConfig {
kafka,
- topic: topic.clone(),
init_timeout: Duration::from_secs(2),
commit_timeout: Duration::from_secs(2),
broker_txn_timeout: BROKER_TXN_TIMEOUT,
@@ -1795,12 +1936,7 @@ async fn a_heal_that_cannot_acquire_does_not_fail_the_run() {
#[tokio::test]
async fn a_healed_fence_is_not_reacquired_by_the_same_convergences_resume() {
let topic = format!("fence_heal_mark_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers_with(
- &topic,
- Duration::from_millis(600),
- 32,
- 1,
- ));
+ let producers = Arc::new(fenced_producers_with(Duration::from_millis(600), 32, 1));
let handler = common::test_handoff_handler(&topic, Arc::clone(&producers));
// A served partition with no usable fence — the state heal repairs.
@@ -1841,8 +1977,11 @@ async fn a_healed_fence_is_not_reacquired_by_the_same_convergences_resume() {
#[tokio::test]
async fn an_unwound_committer_condemns_rather_than_stranding_its_waiters() {
let topic = format!("fence_orphan_{}", uuid::Uuid::new_v4().simple());
- let producers = Arc::new(fenced_producers(&topic, 1));
- producers.acquire(0).await.expect("acquire the fence");
+ let producers = Arc::new(fenced_producers(1));
+ producers
+ .acquire(0, &target(&topic))
+ .await
+ .expect("acquire the fence");
producers
.produce(0, &test_person(1))
.await
@@ -1887,7 +2026,6 @@ async fn the_derived_production_timescales_compose_against_a_real_broker() {
kafka.kafka_hosts = KAFKA_BOOTSTRAP.to_string();
let producers = Arc::new(FencedChangelogProducers::new(FencedProducerConfig {
kafka,
- topic: topic.clone(),
init_timeout: config.fencing_init_timeout(),
commit_timeout: config.fencing_txn_timeout(),
broker_txn_timeout: config.fencing_broker_txn_timeout(),
@@ -1897,7 +2035,7 @@ async fn the_derived_production_timescales_compose_against_a_real_broker() {
lanes: config.fencing_lanes,
}));
producers
- .acquire(0)
+ .acquire(0, &target(&topic))
.await
.expect("acquire at derived init timeout");
diff --git a/rust/personhog-leader/tests/integration.rs b/rust/personhog-leader/tests/integration.rs
index 33ebef66a429..235280ecb36d 100644
--- a/rust/personhog-leader/tests/integration.rs
+++ b/rust/personhog-leader/tests/integration.rs
@@ -10,14 +10,16 @@ use dashmap::DashMap;
use common::{
create_leader_client, create_local_kafka_producer, create_test_kafka,
- create_test_kafka_with_partitions, person_id_for_partition, seed_person, start_coordinator,
- start_leader_pod, start_leader_pod_with_lease_ttl, start_leader_with_pg_fallback, start_router,
- test_cached_person, test_recovery, test_store, test_warming_config, wait_for_condition,
- KAFKA_BOOTSTRAP, NUM_PARTITIONS, POLL_INTERVAL, WAIT_TIMEOUT,
+ create_test_kafka_with_partitions, person_id_for_partition, seed_person, seed_test_changelog,
+ served_captures, start_coordinator, start_leader_pod, start_leader_pod_with_lease_ttl,
+ start_leader_with_pg_fallback, start_router, target, test_cached_person, test_recovery,
+ test_store, test_warming_config, wait_for_condition, KAFKA_BOOTSTRAP, NUM_PARTITIONS,
+ POLL_INTERVAL, WAIT_TIMEOUT,
};
use personhog_common::partitioning::partition_for_person;
use personhog_coordination::pod::HandoffHandler;
use personhog_coordination::strategy::StickyBalancedStrategy;
+use personhog_coordination::types::ChangelogConfig;
use personhog_leader::cache::{
CacheLookup, CachedPerson, DirtyIndex, DirtyMark, PartitionedCache, PersonCacheKey,
};
@@ -71,10 +73,7 @@ async fn service_accepts_requests_after_coordination_warmup() {
let store = test_store("leader-warmup").await;
let cancel = CancellationToken::new();
- assert!(store
- .seed_changelog_config(common::CHANGELOG_TOPIC, NUM_PARTITIONS, "personhog-writer")
- .await
- .unwrap());
+ let _topic = seed_test_changelog(&store).await;
let _coord = start_coordinator(
Arc::clone(&store),
@@ -187,9 +186,9 @@ async fn unowned_partition_returns_failed_precondition() {
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&_mock_cluster.topic, NUM_PARTITIONS),
Arc::new(DirtyIndex::new(1_000_000)),
- test_recovery(&_mock_cluster.topic, KAFKA_BOOTSTRAP),
+ test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(DashMap::new()),
@@ -273,9 +272,9 @@ async fn missing_partition_metadata_returns_invalid_argument() {
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&_mock_cluster.topic, NUM_PARTITIONS),
Arc::new(DirtyIndex::new(1_000_000)),
- test_recovery(&_mock_cluster.topic, KAFKA_BOOTSTRAP),
+ test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(DashMap::new()),
@@ -351,7 +350,7 @@ async fn missing_partition_metadata_returns_invalid_argument() {
/// write through the wrong partition's cache, so it must be rejected even
/// when the named partition is warm and the person exists there.
#[tokio::test]
-async fn mismatched_partition_metadata_returns_invalid_argument() {
+async fn mismatched_partition_metadata_returns_failed_precondition() {
let cache = Arc::new(PartitionedCache::new(1 << 20));
let (_mock_cluster, kafka_producer) = create_test_kafka().await;
// One set for both: a second would fence the first at the broker.
@@ -362,9 +361,9 @@ async fn mismatched_partition_metadata_returns_invalid_argument() {
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&_mock_cluster.topic, NUM_PARTITIONS),
Arc::new(DirtyIndex::new(1_000_000)),
- test_recovery(&_mock_cluster.topic, KAFKA_BOOTSTRAP),
+ test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(DashMap::new()),
@@ -403,7 +402,7 @@ async fn mismatched_partition_metadata_returns_invalid_argument() {
.get_person(leader_get_request(1, 42, wrong_partition))
.await;
let status = result.expect_err("read with mismatched x-partition must fail closed");
- assert_eq!(status.code(), tonic::Code::InvalidArgument);
+ assert_eq!(status.code(), tonic::Code::FailedPrecondition);
assert!(status.message().contains("does not match"));
let result = client
@@ -423,7 +422,7 @@ async fn mismatched_partition_metadata_returns_invalid_argument() {
))
.await;
let status = result.expect_err("write with mismatched x-partition must fail closed");
- assert_eq!(status.code(), tonic::Code::InvalidArgument);
+ assert_eq!(status.code(), tonic::Code::FailedPrecondition);
assert!(status.message().contains("does not match"));
cancel.cancel();
@@ -448,16 +447,17 @@ async fn writes_fenced_after_drain_reads_still_served() {
let (_mock_cluster, kafka_producer) = create_test_kafka().await;
let inflight = Arc::new(InflightTracker::new());
let dirty_index = Arc::new(DirtyIndex::new(1_000_000));
- let recovery = test_recovery(&_mock_cluster.topic, KAFKA_BOOTSTRAP);
+ let recovery = test_recovery(KAFKA_BOOTSTRAP);
// One set for both: a second would fence the first at the broker.
let fenced =
common::fenced_producers_acquired(&_mock_cluster.topic, NUM_PARTITIONS as i32).await;
+ let captures = served_captures(&_mock_cluster.topic, NUM_PARTITIONS);
let service = PersonHogLeaderService::new(
Arc::clone(&cache),
None,
Arc::new(DashMap::new()),
Arc::clone(&inflight),
- NUM_PARTITIONS,
+ Arc::clone(&captures),
Arc::clone(&dirty_index),
Arc::clone(&recovery),
PropertySizeLimits::new(655360, 524288),
@@ -469,12 +469,13 @@ async fn writes_fenced_after_drain_reads_still_served() {
);
// The handler shares the cache, inflight tracker, dirty index, and
// recovery pool with the service, exactly as main.rs wires them.
- let warming = test_warming_config("fence-pod", &_mock_cluster.topic, KAFKA_BOOTSTRAP);
+ let warming = test_warming_config("fence-pod", KAFKA_BOOTSTRAP);
let pools = Arc::new(WarmClientPools::new(
&warming.kafka,
"fence-pod",
&warming.writer_consumer_group,
));
+ let changelog = common::live_changelog(&_mock_cluster.topic);
let handler = LeaderHandoffHandler::new(
Arc::clone(&cache),
Arc::clone(&inflight),
@@ -482,7 +483,8 @@ async fn writes_fenced_after_drain_reads_still_served() {
warming,
Arc::new(DashMap::new()),
None,
- NUM_PARTITIONS,
+ changelog,
+ Arc::clone(&captures),
pools,
Arc::clone(&fenced),
common::live_authority(),
@@ -558,10 +560,16 @@ async fn writes_fenced_after_drain_reads_still_served() {
// stood in for, epoch included, since warming needs a changelog this
// test never writes.
handler.release_partition(0).await.unwrap();
+ handler
+ .capture_changelog(
+ 0,
+ &ChangelogConfig::new(&_mock_cluster.topic, NUM_PARTITIONS),
+ )
+ .await;
cache.create_partition(0);
seed_person(&cache, 0, test_cached_person());
fenced
- .acquire(0)
+ .acquire(0, &target(&_mock_cluster.topic))
.await
.expect("the re-warm takes the epoch");
let response = client
@@ -585,16 +593,17 @@ async fn drain_fences_before_waiting_on_inflight() {
let (_mock_cluster, kafka_producer) = create_test_kafka().await;
let inflight = Arc::new(InflightTracker::new());
let dirty_index = Arc::new(DirtyIndex::new(1_000_000));
- let recovery = test_recovery(&_mock_cluster.topic, KAFKA_BOOTSTRAP);
+ let recovery = test_recovery(KAFKA_BOOTSTRAP);
// One set for both: a second would fence the first at the broker.
let fenced =
common::fenced_producers_acquired(&_mock_cluster.topic, NUM_PARTITIONS as i32).await;
+ let captures = served_captures(&_mock_cluster.topic, NUM_PARTITIONS);
let service = PersonHogLeaderService::new(
Arc::clone(&cache),
None,
Arc::new(DashMap::new()),
Arc::clone(&inflight),
- NUM_PARTITIONS,
+ Arc::clone(&captures),
Arc::clone(&dirty_index),
Arc::clone(&recovery),
PropertySizeLimits::new(655360, 524288),
@@ -604,12 +613,13 @@ async fn drain_fences_before_waiting_on_inflight() {
common::live_authority(),
std::sync::Arc::new(personhog_leader::emitted::EmittedVersions::new(1_000_000)),
);
- let warming = test_warming_config("fence-race-pod", &_mock_cluster.topic, KAFKA_BOOTSTRAP);
+ let warming = test_warming_config("fence-race-pod", KAFKA_BOOTSTRAP);
let pools = Arc::new(WarmClientPools::new(
&warming.kafka,
"fence-race-pod",
&warming.writer_consumer_group,
));
+ let changelog = common::live_changelog(&_mock_cluster.topic);
let handler = Arc::new(LeaderHandoffHandler::new(
Arc::clone(&cache),
Arc::clone(&inflight),
@@ -617,7 +627,8 @@ async fn drain_fences_before_waiting_on_inflight() {
warming,
Arc::new(DashMap::new()),
None,
- NUM_PARTITIONS,
+ changelog,
+ Arc::clone(&captures),
pools,
Arc::clone(&fenced),
common::live_authority(),
@@ -703,10 +714,7 @@ async fn release_partition_stops_serving() {
let store = test_store("leader-release").await;
let cancel = CancellationToken::new();
- assert!(store
- .seed_changelog_config(common::CHANGELOG_TOPIC, NUM_PARTITIONS, "personhog-writer")
- .await
- .unwrap());
+ let _topic = seed_test_changelog(&store).await;
let _coord = start_coordinator(
Arc::clone(&store),
@@ -824,10 +832,7 @@ async fn rewarm_after_pod_crash() {
let store = test_store("leader-rewarm").await;
let cancel = CancellationToken::new();
- assert!(store
- .seed_changelog_config(common::CHANGELOG_TOPIC, NUM_PARTITIONS, "personhog-writer")
- .await
- .unwrap());
+ let _topic = seed_test_changelog(&store).await;
let _coord = start_coordinator(
Arc::clone(&store),
@@ -927,9 +932,9 @@ async fn update_produces_person_state_to_kafka() {
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&mock_cluster.topic, NUM_PARTITIONS),
Arc::new(DirtyIndex::new(1_000_000)),
- test_recovery(&mock_cluster.topic, KAFKA_BOOTSTRAP),
+ test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(DashMap::new()),
@@ -1044,9 +1049,9 @@ async fn kafka_produce_failure_leaves_cache_unchanged() {
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&mock_cluster.topic, NUM_PARTITIONS),
Arc::new(DirtyIndex::new(1_000_000)),
- test_recovery(&mock_cluster.topic, KAFKA_BOOTSTRAP),
+ test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(DashMap::new()),
@@ -1128,7 +1133,7 @@ async fn kafka_produce_failure_leaves_cache_unchanged() {
drop(successor);
fenced.expect_self_fence(0);
fenced
- .acquire(0)
+ .acquire(0, &target(&mock_cluster.topic))
.await
.expect("this pod re-takes the fence");
@@ -1181,9 +1186,9 @@ async fn e2e_update_produces_to_local_kafka() {
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&topic, NUM_PARTITIONS),
Arc::new(DirtyIndex::new(1_000_000)),
- test_recovery(&topic, KAFKA_BOOTSTRAP),
+ test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(DashMap::new()),
@@ -1512,13 +1517,13 @@ async fn evicted_dirty_person_recovers_from_changelog() {
// Recovery reads from the same mock broker the update produces to.
// Deliberately no PG pool: a recovery path that (wrongly) fell back to
// PG would return NotFound and fail the final assertion.
- let recovery = test_recovery(&mock_cluster.topic, &mock_cluster.bootstrap_servers());
+ let recovery = test_recovery(&mock_cluster.bootstrap_servers());
let service = PersonHogLeaderService::new(
Arc::clone(&cache),
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&mock_cluster.topic, NUM_PARTITIONS),
Arc::clone(&dirty_index),
Arc::clone(&recovery),
PropertySizeLimits::new(655360, 524288),
@@ -1615,13 +1620,13 @@ async fn dirty_person_with_failed_recovery_is_unavailable_not_stale() {
let cache = Arc::new(PartitionedCache::new(1 << 20));
let dirty_index = Arc::new(DirtyIndex::new(1_000_000));
- let recovery = test_recovery(&mock_cluster.topic, &mock_cluster.bootstrap_servers());
+ let recovery = test_recovery(&mock_cluster.bootstrap_servers());
let service = PersonHogLeaderService::new(
Arc::clone(&cache),
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&mock_cluster.topic, NUM_PARTITIONS),
Arc::clone(&dirty_index),
Arc::clone(&recovery),
PropertySizeLimits::new(655360, 524288),
@@ -1688,6 +1693,7 @@ async fn dirty_person_with_failed_recovery_is_unavailable_not_stale() {
version: mark.version,
offset: mark.offset + 1_000,
partition: mark.partition,
+ stream: mark.stream.clone(),
is_deleted: mark.is_deleted,
},
);
@@ -1731,9 +1737,9 @@ async fn writes_shed_when_dirty_index_is_full() {
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&mock_cluster.topic, NUM_PARTITIONS),
Arc::clone(&dirty_index),
- test_recovery(&mock_cluster.topic, &mock_cluster.bootstrap_servers()),
+ test_recovery(&mock_cluster.bootstrap_servers()),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(DashMap::new()),
@@ -1825,13 +1831,13 @@ async fn recovery_fails_when_record_version_disagrees_with_the_mark() {
let cache = Arc::new(PartitionedCache::new(1 << 20));
let dirty_index = Arc::new(DirtyIndex::new(1_000_000));
- let recovery = test_recovery(&mock_cluster.topic, &mock_cluster.bootstrap_servers());
+ let recovery = test_recovery(&mock_cluster.bootstrap_servers());
let service = PersonHogLeaderService::new(
Arc::clone(&cache),
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&mock_cluster.topic, NUM_PARTITIONS),
Arc::clone(&dirty_index),
Arc::clone(&recovery),
PropertySizeLimits::new(655360, 524288),
@@ -1935,13 +1941,13 @@ async fn recovery_reuses_the_partition_consumer_across_fetches() {
let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;
let cache = Arc::new(PartitionedCache::new(1 << 20));
let dirty_index = Arc::new(DirtyIndex::new(1_000_000));
- let recovery = test_recovery(&mock_cluster.topic, &mock_cluster.bootstrap_servers());
+ let recovery = test_recovery(&mock_cluster.bootstrap_servers());
let service = PersonHogLeaderService::new(
Arc::clone(&cache),
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&mock_cluster.topic, NUM_PARTITIONS),
Arc::clone(&dirty_index),
Arc::clone(&recovery),
PropertySizeLimits::new(655360, 524288),
@@ -2063,7 +2069,6 @@ async fn cancelled_recovery_returns_its_consumer_to_the_pool() {
// pool_size 1: a single leaked consumer exhausts the pool.
let recovery = ChangelogRecovery::new(RecoveryConfig {
kafka,
- topic: mock_cluster.topic.clone(),
pod_name: "cancel-pod".to_string(),
recv_timeout: Duration::from_secs(2),
pool_size: 1,
@@ -2078,6 +2083,10 @@ async fn cancelled_recovery_returns_its_consumer_to_the_pool() {
version: 1,
offset: 0,
partition: 0,
+ stream: personhog_leader::cache::Stream {
+ topic: std::sync::Arc::from(mock_cluster.topic.as_str()),
+ partition: 0,
+ },
is_deleted: false,
};
@@ -2149,13 +2158,13 @@ async fn oversize_updates_are_rejected_and_oversized_rows_remediated() {
let cache = Arc::new(PartitionedCache::new(1 << 20));
let dirty_index = Arc::new(DirtyIndex::new(1_000_000));
- let recovery = test_recovery(&mock_cluster.topic, &mock_cluster.bootstrap_servers());
+ let recovery = test_recovery(&mock_cluster.bootstrap_servers());
let service = PersonHogLeaderService::new(
Arc::clone(&cache),
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&mock_cluster.topic, NUM_PARTITIONS),
Arc::clone(&dirty_index),
recovery,
PropertySizeLimits::new(655360, 524288),
@@ -2620,9 +2629,9 @@ async fn an_unresolved_version_is_never_reused() {
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&_mock_cluster.topic, NUM_PARTITIONS),
Arc::new(DirtyIndex::new(1_000_000)),
- test_recovery(&_mock_cluster.topic, KAFKA_BOOTSTRAP),
+ test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(DashMap::new()),
@@ -2694,16 +2703,16 @@ async fn a_fenced_write_that_bounces_does_not_hand_its_version_back() {
let topic = format!("fenced_svc_{}", uuid::Uuid::new_v4().simple());
let cache = Arc::new(PartitionedCache::new(1 << 20));
let (_mock_cluster, kafka_producer) = create_test_kafka().await;
- let fenced = Arc::new(common::fenced_producers_for(&topic));
+ let fenced = Arc::new(common::test_fenced_producers());
let service = PersonHogLeaderService::new(
Arc::clone(&cache),
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&topic, NUM_PARTITIONS),
Arc::new(DirtyIndex::new(1_000_000)),
- test_recovery(&_mock_cluster.topic, KAFKA_BOOTSTRAP),
+ test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(DashMap::new()),
@@ -2716,7 +2725,10 @@ async fn a_fenced_write_that_bounces_does_not_hand_its_version_back() {
let person = test_cached_person();
let base_version = person.version;
seed_person(&cache, 0, person);
- fenced.acquire(0).await.expect("take the partition's fence");
+ fenced
+ .acquire(0, &target(&topic))
+ .await
+ .expect("take the partition's fence");
// Each call must change something, or the handler takes its
// no-change fast path and never reaches the produce at all.
@@ -2761,7 +2773,10 @@ async fn a_fenced_write_that_bounces_does_not_hand_its_version_back() {
// The bounced write never reached the broker, so its version is free
// to derive again — the next successful write must land exactly one
// past the first, not two.
- fenced.acquire(0).await.expect("re-take the fence");
+ fenced
+ .acquire(0, &target(&topic))
+ .await
+ .expect("re-take the fence");
let third = service
.update_person_properties(update(3))
.await
@@ -2812,7 +2827,7 @@ async fn an_unknown_outcome_keeps_its_version(
let topic = format!("fenced_uncertain_{}", uuid::Uuid::new_v4().simple());
let cache = Arc::new(PartitionedCache::new(1 << 20));
let (_mock_cluster, kafka_producer) = create_test_kafka().await;
- let fenced = Arc::new(common::fenced_producers_for(&topic));
+ let fenced = Arc::new(common::test_fenced_producers());
let emitted_versions = Arc::new(personhog_leader::emitted::EmittedVersions::new(1_000_000));
let service = PersonHogLeaderService::new(
@@ -2820,9 +2835,9 @@ async fn an_unknown_outcome_keeps_its_version(
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&topic, NUM_PARTITIONS),
Arc::new(DirtyIndex::new(1_000_000)),
- test_recovery(&_mock_cluster.topic, KAFKA_BOOTSTRAP),
+ test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(DashMap::new()),
@@ -2835,7 +2850,10 @@ async fn an_unknown_outcome_keeps_its_version(
let person = test_cached_person();
let base_version = person.version;
seed_person(&cache, 0, person);
- fenced.acquire(0).await.expect("take the partition's fence");
+ fenced
+ .acquire(0, &target(&topic))
+ .await
+ .expect("take the partition's fence");
let cache_key = PersonCacheKey {
team_id: 1,
@@ -2908,9 +2926,9 @@ async fn scalar_fields_merge_rather_than_assign() {
None,
Arc::new(DashMap::new()),
Arc::new(InflightTracker::new()),
- NUM_PARTITIONS,
+ served_captures(&topic, NUM_PARTITIONS),
Arc::new(DirtyIndex::new(1_000_000)),
- test_recovery(&_mock_cluster.topic, KAFKA_BOOTSTRAP),
+ test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
Arc::new(DashMap::new()),
diff --git a/rust/personhog-leader/tests/settle_integration.rs b/rust/personhog-leader/tests/settle_integration.rs
index 98536e97f8cf..fb93edd9d559 100644
--- a/rust/personhog-leader/tests/settle_integration.rs
+++ b/rust/personhog-leader/tests/settle_integration.rs
@@ -47,6 +47,10 @@ fn pruned(key: &PersonCacheKey, version: i64) -> Vec<(PersonCacheKey, DirtyMark)
version,
offset: 9,
partition: PARTITION,
+ stream: personhog_leader::cache::Stream {
+ topic: std::sync::Arc::from("personhog_updates"),
+ partition: PARTITION,
+ },
is_deleted: true,
},
)]
diff --git a/rust/personhog-leader/tests/warming_integration.rs b/rust/personhog-leader/tests/warming_integration.rs
index 9536e172cd0c..55bfb5358d6e 100644
--- a/rust/personhog-leader/tests/warming_integration.rs
+++ b/rust/personhog-leader/tests/warming_integration.rs
@@ -1,5 +1,5 @@
//! Integration tests for the Kafka warming pipeline. Drives the real
-//! `warm_from_kafka` function against a mock Kafka cluster, producing
+//! warm against a mock Kafka cluster, producing
//! known Person messages and asserting the resulting cache state matches.
mod common;
@@ -8,8 +8,9 @@ use std::sync::Arc;
use std::time::Duration;
use common::{create_mock_kafka, test_warming_config, CHANGELOG_TOPIC, NUM_PARTITIONS};
+use personhog_coordination::changelog::WarmSource;
use personhog_leader::cache::{CacheLookup, DirtyIndex, PartitionedCache, PersonCacheKey};
-use personhog_leader::warming::{warm_from_kafka, WarmClientPools, WarmingConfig};
+use personhog_leader::warming::{warm_from_sources, WarmClientPools, WarmingConfig};
use personhog_proto::personhog::types::v1::Person;
use prost::Message as ProtoMessage;
use rdkafka::config::ClientConfig;
@@ -85,11 +86,23 @@ fn warming_config_for(
pod_name: &str,
cluster: &MockCluster<'static, DefaultProducerContext>,
) -> (WarmingConfig, WarmClientPools) {
- let cfg = test_warming_config(pod_name, CHANGELOG_TOPIC, &cluster.bootstrap_servers());
+ let cfg = test_warming_config(pod_name, &cluster.bootstrap_servers());
let pools = WarmClientPools::new(&cfg.kafka, pod_name, &cfg.writer_consumer_group);
(cfg, pools)
}
+/// A warm from the live topic alone.
+async fn warm_live(
+ cfg: &WarmingConfig,
+ pools: &WarmClientPools,
+ cache: &PartitionedCache,
+ dirty_index: &DirtyIndex,
+ partition: u32,
+) -> personhog_coordination::error::Result<()> {
+ let sources = [WarmSource::live(CHANGELOG_TOPIC, partition)];
+ warm_from_sources(cfg, pools, cache, dirty_index, partition, &sources).await
+}
+
/// Happy path: produce N messages to a partition, warm, assert every
/// record landed in the cache. Also asserts FIFO ordering by relying on
/// the cache's `put` semantics — later puts overwrite earlier ones, so
@@ -107,7 +120,7 @@ async fn warming_populates_cache_from_kafka() {
let cache = PartitionedCache::new(1 << 20);
let (cfg, pools) = warming_config_for("warmer-0", &cluster);
- warm_from_kafka(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0)
+ warm_live(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0)
.await
.expect("warming should succeed");
@@ -139,7 +152,7 @@ async fn warming_handles_empty_partition() {
let cache = PartitionedCache::new(1 << 20);
let (cfg, pools) = warming_config_for("warmer-empty", &cluster);
- warm_from_kafka(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0)
+ warm_live(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0)
.await
.expect("warming an empty partition should succeed");
@@ -160,7 +173,7 @@ async fn warming_handles_empty_partition() {
// a second empty warm reuses the same client instead of creating one.
let created_after_first = pools.warming.created_count();
cache.drop_partition(0);
- warm_from_kafka(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0)
+ warm_live(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0)
.await
.expect("second empty warm should succeed");
assert_eq!(
@@ -184,7 +197,7 @@ async fn warming_only_populates_target_partition() {
let cache = PartitionedCache::new(1 << 20);
let (cfg, pools) = warming_config_for("warmer-iso", &cluster);
- warm_from_kafka(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0)
+ warm_live(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0)
.await
.expect("warming partition 0 should succeed");
@@ -220,7 +233,7 @@ async fn warming_fails_loudly_on_decode_error_and_leaves_cache_clean() {
let cache = PartitionedCache::new(1 << 20);
let (cfg, pools) = warming_config_for("warmer-decode", &cluster);
- let result = warm_from_kafka(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0).await;
+ let result = warm_live(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0).await;
assert!(
result.is_err(),
"a malformed message must fail the entire warm"
@@ -250,7 +263,7 @@ async fn warming_works_across_all_partitions() {
let (cfg, pools) = warming_config_for("warmer-all", &cluster);
for partition in 0..NUM_PARTITIONS {
- warm_from_kafka(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), partition)
+ warm_live(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), partition)
.await
.unwrap_or_else(|e| panic!("warming partition {partition} failed: {e}"));
}
@@ -296,8 +309,8 @@ fn commit_writer_offset_at(
/// Warming must consult the writer's committed offset and skip records
/// the writer has already durably persisted to PG. Produce 8 records;
/// commit the writer's offset at 5 (meaning records 0..4 are durable,
-/// records 5..7 still need to be in cache); warm with `lookback=0` and
-/// assert only records with offsets ≥ 5 land in the cache.
+/// records 5..7 still need to be in cache), and assert only records with
+/// offsets ≥ 5 land in the cache, each marked dirty.
#[tokio::test]
async fn warming_starts_from_writer_committed_offset() {
let (cluster, producer) = create_mock_kafka().await;
@@ -315,15 +328,13 @@ async fn warming_starts_from_writer_committed_offset() {
commit_writer_offset_at(&cluster, "personhog-writer", 0, 5);
let cache = PartitionedCache::new(1 << 20);
- let (mut cfg, pools) = warming_config_for("warmer-committed", &cluster);
- // Set lookback to 0 so the start offset is exactly the committed
- // value; otherwise it would rewind further and include earlier
- // records, defeating the point of this assertion.
- cfg.lookback_offsets = 0;
+ let dirty_index = DirtyIndex::new(1_000_000);
+ let (cfg, pools) = warming_config_for("warmer-committed", &cluster);
- warm_from_kafka(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0)
+ warm_live(&cfg, &pools, &cache, &dirty_index, 0)
.await
.expect("warming should succeed");
+ assert_eq!(dirty_index.len(), 3);
// Records 5..7 (person_ids 6, 7, 8) must be in cache.
for person_id in 6..=8 {
@@ -335,6 +346,10 @@ async fn warming_starts_from_writer_committed_offset() {
matches!(cache.get(0, &key), CacheLookup::Found(_)),
"record after committed offset (person_id={person_id}) must be warmed"
);
+ let mark = dirty_index
+ .get(&key)
+ .expect("an unapplied record is marked");
+ assert_eq!(mark.offset, person_id - 1);
}
// Records 0..4 (person_ids 1..5) must NOT be in cache — they're
@@ -386,7 +401,7 @@ async fn warming_fails_loudly_on_properties_json_error() {
let (cfg, pools) = warming_config_for("warmer-bad-props", &cluster);
let dirty_index = DirtyIndex::new(1_000_000);
- let result = warm_from_kafka(&cfg, &pools, &cache, &dirty_index, 0).await;
+ let result = warm_live(&cfg, &pools, &cache, &dirty_index, 0).await;
assert!(
result.is_err(),
"invalid JSON in properties must fail the entire warm"
@@ -440,7 +455,7 @@ async fn warming_preserves_last_write_for_same_key() {
let (cfg, pools) = warming_config_for("warmer-lww", &cluster);
let dirty_index = DirtyIndex::new(1_000_000);
- warm_from_kafka(&cfg, &pools, &cache, &dirty_index, 0)
+ warm_live(&cfg, &pools, &cache, &dirty_index, 0)
.await
.expect("warming should succeed");
@@ -469,62 +484,8 @@ async fn warming_preserves_last_write_for_same_key() {
assert_eq!(mark.version, 3, "dirty mark must carry the latest version");
}
-/// When the warm range spans the writer's committed offset, only the
-/// records at or past it may be marked: marking applied ones would send
-/// their misses to the changelog for state PG already has, and skipping
-/// unapplied ones would reopen the stale-fallback hole.
-#[tokio::test]
-async fn warming_seeds_dirty_index_only_at_or_past_the_committed_offset() {
- let (cluster, producer) = create_mock_kafka().await;
-
- // Offset N → person_id N+1.
- for i in 0..8 {
- let person = make_person(1, i + 1);
- produce_person_to_partition(&producer, 0, &person).await;
- }
-
- // Records at offsets 0..4 are applied to PG; 5..7 are not.
- commit_writer_offset_at(&cluster, "personhog-writer", 0, 5);
-
- let cache = PartitionedCache::new(1 << 20);
- let dirty_index = DirtyIndex::new(1_000_000);
- let (mut cfg, pools) = warming_config_for("warmer-seed-committed", &cluster);
- // Rewind the warm range below the committed offset so it spans the
- // boundary and both seeding arms are exercised.
- cfg.lookback_offsets = 8;
-
- warm_from_kafka(&cfg, &pools, &cache, &dirty_index, 0)
- .await
- .expect("warming should succeed");
-
- assert_eq!(dirty_index.len(), 3);
- for person_id in 1..=8 {
- let key = PersonCacheKey {
- team_id: 1,
- person_id,
- };
- assert!(matches!(cache.get(0, &key), CacheLookup::Found(_)));
- match dirty_index.get(&key) {
- None => assert!(
- person_id <= 5,
- "person {person_id} is unapplied, must be marked"
- ),
- Some(mark) => {
- assert!(
- person_id > 5,
- "person {person_id} is applied, must not be marked"
- );
- assert_eq!(mark.offset, person_id - 1);
- }
- }
- }
-}
-
-/// An applied death record must leave no residue in the warmed cache:
-/// not the death document (unmarked, it would answer not-found forever
-/// while PG may already hold a revival), and not the same key's earlier
-/// live record (it would resurrect the destroyed person's data). A live
-/// applied record of another key still warms.
+/// Applied records, deaths included, leave no residue in the warmed cache;
+/// an unapplied death record warms in, marked.
#[tokio::test]
async fn warming_skips_applied_death_records() {
let (cluster, producer) = create_mock_kafka().await;
@@ -556,10 +517,9 @@ async fn warming_skips_applied_death_records() {
let cache = PartitionedCache::new(1 << 20);
let dirty_index = DirtyIndex::new(1_000_000);
- let (mut cfg, pools) = warming_config_for("warmer-skip-dead", &cluster);
- cfg.lookback_offsets = 7;
+ let (cfg, pools) = warming_config_for("warmer-skip-dead", &cluster);
- warm_from_kafka(&cfg, &pools, &cache, &dirty_index, 0)
+ warm_live(&cfg, &pools, &cache, &dirty_index, 0)
.await
.expect("warming should succeed");
@@ -571,30 +531,15 @@ async fn warming_skips_applied_death_records() {
team_id: 1,
person_id: 2,
};
- assert!(
- matches!(cache.get(0, &dead_key), CacheLookup::PersonNotFound),
- "an applied death record must leave no residue: neither itself nor its live predecessor"
- );
- assert!(
- matches!(cache.get(0, &live_key), CacheLookup::Found(_)),
- "an applied live record still warms"
- );
let revived_key = PersonCacheKey {
team_id: 1,
person_id: 3,
};
- match cache.get(0, &revived_key) {
- CacheLookup::Found(entry) => {
- assert_eq!(
- entry.version, 3,
- "the revival record warms after its death record"
- );
- assert!(!entry.is_deleted);
- }
- other => panic!(
- "a revived key must warm, got {:?}",
- std::mem::discriminant(&other)
- ),
+ for key in [&dead_key, &live_key, &revived_key] {
+ assert!(
+ matches!(cache.get(0, key), CacheLookup::PersonNotFound),
+ "an applied record must leave no residue"
+ );
}
assert!(dirty_index.get(&dead_key).is_none());
assert!(dirty_index.get(&live_key).is_none());
@@ -635,7 +580,7 @@ async fn warming_seeds_dirty_index_when_writer_has_no_commits() {
let dirty_index = DirtyIndex::new(1_000_000);
let (cfg, pools) = warming_config_for("warmer-seed", &cluster);
- warm_from_kafka(&cfg, &pools, &cache, &dirty_index, 0)
+ warm_live(&cfg, &pools, &cache, &dirty_index, 0)
.await
.expect("warming should succeed");
@@ -645,7 +590,11 @@ async fn warming_seeds_dirty_index_when_writer_has_no_commits() {
"every warmed record must be marked dirty when the writer has no commits"
);
// Marks carry the record offsets: 3 records at offsets 0..=2.
- assert_eq!(dirty_index.max_offset(0), Some(2));
+ let stream = personhog_leader::cache::Stream {
+ topic: std::sync::Arc::from(CHANGELOG_TOPIC),
+ partition: 0,
+ };
+ assert_eq!(dirty_index.max_offset(0, &stream), Some(2));
}
/// Sequential warms reuse pooled clients instead of rebuilding them:
@@ -668,7 +617,7 @@ async fn sequential_warms_reuse_pooled_clients() {
for partition in [0u32, 1, 0, 1] {
// A re-warm follows a release in production; drop to match.
cache.drop_partition(partition);
- warm_from_kafka(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), partition)
+ warm_live(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), partition)
.await
.expect("warming should succeed");
}
@@ -731,7 +680,7 @@ async fn warming_tolerates_a_transient_consumer_error() {
let cache = PartitionedCache::new(1 << 20);
let (cfg, pools) = warming_config_for("warmer-transient", &cluster);
- warm_from_kafka(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0)
+ warm_live(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0)
.await
.expect("a non-fatal consumer error must not fail the warm");
@@ -774,7 +723,7 @@ async fn warming_still_fails_on_a_persistent_stop_class_error() {
let (mut cfg, pools) = warming_config_for("warmer-stop-class", &cluster);
cfg.recv_timeout = Duration::from_secs(2);
- let result = warm_from_kafka(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0).await;
+ let result = warm_live(&cfg, &pools, &cache, &DirtyIndex::new(1_000_000), 0).await;
let err = result.expect_err("a persistent stop-class error must end the warm");
// The stall path, not an immediate abort: riding the error out means
// the warm ends only when the budget expires with no records read.
@@ -818,7 +767,7 @@ async fn warming_a_range_larger_than_the_budget_marks_every_person() {
let (cfg, pools) = warming_config_for("warmer-bounded", &cluster);
let dirty_index = DirtyIndex::new(1_000_000);
- warm_from_kafka(&cfg, &pools, &cache, &dirty_index, 0)
+ warm_live(&cfg, &pools, &cache, &dirty_index, 0)
.await
.expect("a range larger than the budget must still warm");
@@ -865,6 +814,10 @@ async fn the_prune_tick_settles_published_partitions_only() {
version: 2,
offset: 0,
partition,
+ stream: personhog_leader::cache::Stream {
+ topic: std::sync::Arc::from("personhog_updates"),
+ partition,
+ },
is_deleted: true,
};
let published_key = PersonCacheKey {
@@ -892,11 +845,9 @@ async fn the_prune_tick_settles_published_partitions_only() {
&cache,
&locks,
&pools.offsets,
- CHANGELOG_TOPIC,
Duration::from_secs(5),
)
- .await
- .expect("offset fetch succeeds");
+ .await;
assert!(dirty_index.get(&published_key).is_none());
assert!(
@@ -915,11 +866,9 @@ async fn the_prune_tick_settles_published_partitions_only() {
&cache,
&locks,
&pools.offsets,
- CHANGELOG_TOPIC,
Duration::from_secs(5),
)
- .await
- .expect("offset fetch succeeds");
+ .await;
assert!(dirty_index.get(&warming_key).is_none());
assert!(cache.peek(1, &warming_key).is_none());
}
@@ -956,6 +905,10 @@ async fn the_prune_tick_drains_a_multi_chunk_death_backlog() {
version: 2,
offset,
partition: 0,
+ stream: personhog_leader::cache::Stream {
+ topic: std::sync::Arc::from("personhog_updates"),
+ partition: 0,
+ },
is_deleted: true,
},
);
@@ -967,11 +920,9 @@ async fn the_prune_tick_drains_a_multi_chunk_death_backlog() {
&cache,
&locks,
&pools.offsets,
- CHANGELOG_TOPIC,
Duration::from_secs(5),
)
- .await
- .expect("offset fetch succeeds");
+ .await;
assert!(dirty_index.is_empty(), "every mark settles in one tick");
for person_id in [0, total / 2, total - 1] {
@@ -985,3 +936,215 @@ async fn the_prune_tick_drains_a_multi_chunk_death_backlog() {
);
}
}
+
+/// Produce a Person to a partition of any topic on the local broker.
+async fn produce_to(
+ producer: &FutureProducer<common_kafka::kafka_producer::KafkaContext>,
+ topic: &str,
+ partition: i32,
+ person: &Person,
+) {
+ let key = format!("{}:{}", person.team_id, person.id);
+ let payload = person.encode_to_vec();
+ let record = FutureRecord::to(topic)
+ .key(&key)
+ .partition(partition)
+ .payload(&payload);
+ producer
+ .send(record, Duration::from_secs(5))
+ .await
+ .expect("produce should succeed");
+}
+
+fn commit_group_offset(bootstrap: &str, group: &str, topic: &str, partition: i32, offset: i64) {
+ let consumer: BaseConsumer = ClientConfig::new()
+ .set("bootstrap.servers", bootstrap)
+ .set("group.id", group)
+ .set("enable.auto.commit", "false")
+ .create()
+ .expect("failed to create writer-group consumer");
+ let mut tpl = TopicPartitionList::new();
+ tpl.add_partition_offset(topic, partition, Offset::Offset(offset))
+ .expect("tpl add_partition_offset");
+ consumer.assign(&tpl).expect("assign");
+ consumer
+ .commit(&tpl, CommitMode::Sync)
+ .expect("writer-group commit");
+}
+
+/// After a 2 → 4 flip, new partition 1 inherits half of old partition 1's tail.
+#[tokio::test]
+async fn a_restart_warms_the_old_generations_congruent_tail_for_its_keys() {
+ use personhog_common::partitioning::partition_for_person;
+ use personhog_coordination::changelog::KeyFilter;
+ use personhog_leader::cache::Stream;
+
+ let (new_topic, producer) = common::create_test_kafka_with_partitions(4).await;
+ let old_topic = common::owned_broker_topic(2).await;
+ let bootstrap = new_topic.bootstrap_servers();
+
+ // Keys whose home under 4 partitions is 1 or 3 all live in old partition 1.
+ let mut inherited = Vec::new();
+ let mut strangers = Vec::new();
+ for person_id in 1..=200 {
+ match partition_for_person(1, person_id, 4) {
+ 1 => inherited.push(person_id),
+ 3 => strangers.push(person_id),
+ _ => {}
+ }
+ if inherited.len() >= 5 && strangers.len() >= 5 {
+ break;
+ }
+ }
+ for person_id in inherited.iter().chain(&strangers) {
+ assert_eq!(partition_for_person(1, *person_id, 2), 1);
+ produce_to(&producer, &old_topic.topic, 1, &make_person(1, *person_id)).await;
+ }
+ // A group of this test's own: the shared broker may host live writers.
+ let writer_group = format!("restart-writer-{}", uuid::Uuid::now_v7());
+ // One record already applied by the writer: below its commit, no mark.
+ commit_group_offset(&bootstrap, &writer_group, &old_topic.topic, 1, 1);
+
+ let mut cfg = common::test_warming_config("restarter", &bootstrap);
+ cfg.writer_consumer_group = writer_group.clone();
+ let pools = WarmClientPools::new(&cfg.kafka, "restarter", &cfg.writer_consumer_group);
+ let cache = PartitionedCache::new(1 << 20);
+ let dirty_index = DirtyIndex::new(1_000_000);
+ let sources = [
+ WarmSource {
+ topic: Arc::from(old_topic.topic.as_str()),
+ partition: 1,
+ keep: Some(KeyFilter {
+ modulus: 4,
+ target: 1,
+ }),
+ },
+ WarmSource::live(&new_topic.topic, 1),
+ ];
+ warm_from_sources(&cfg, &pools, &cache, &dirty_index, 1, &sources)
+ .await
+ .expect("warm from the old tail");
+
+ let old_stream = Stream {
+ topic: Arc::from(old_topic.topic.as_str()),
+ partition: 1,
+ };
+ // The first record is below the writer's commit: applied, no mark.
+ let applied = PersonCacheKey {
+ team_id: 1,
+ person_id: inherited[0],
+ };
+ assert!(dirty_index.get(&applied).is_none());
+ for person_id in &inherited[1..] {
+ let key = PersonCacheKey {
+ team_id: 1,
+ person_id: *person_id,
+ };
+ assert!(
+ matches!(cache.get(1, &key), CacheLookup::Found(_)),
+ "inherited person {person_id} must be warm"
+ );
+ assert!(
+ dirty_index.get(&key).is_some(),
+ "person {person_id} must be marked"
+ );
+ }
+ for person_id in &strangers {
+ let key = PersonCacheKey {
+ team_id: 1,
+ person_id: *person_id,
+ };
+ assert!(
+ matches!(cache.get(1, &key), CacheLookup::PersonNotFound),
+ "person {person_id} belongs to partition 3, not 1"
+ );
+ }
+ let marked: Vec<_> = inherited
+ .iter()
+ .filter_map(|id| {
+ dirty_index.get(&PersonCacheKey {
+ team_id: 1,
+ person_id: *id,
+ })
+ })
+ .collect();
+ assert_eq!(marked.len(), inherited.len() - 1);
+ assert!(marked
+ .iter()
+ .all(|m| m.stream == old_stream && m.partition == 1));
+ assert_eq!(
+ dirty_index.streams_with_marks(),
+ vec![(1, old_stream.clone())]
+ );
+
+ // The writer finishes the old partition: its marks settle.
+ commit_group_offset(&bootstrap, &writer_group, &old_topic.topic, 1, 1_000);
+ let locks = dashmap::DashMap::new();
+ personhog_leader::settle::prune_and_settle_tick(
+ &dirty_index,
+ &cache,
+ &locks,
+ &pools.offsets,
+ Duration::from_secs(5),
+ )
+ .await;
+ assert!(dirty_index.is_empty(), "every old-stream mark settles");
+}
+
+/// An applied old-generation record never warms, so it cannot shadow the
+/// same person's newer record that the writer already applied on the live
+/// topic, which a warm from the live commit does not replay.
+#[tokio::test]
+async fn an_applied_old_record_never_shadows_a_newer_applied_live_one() {
+ use personhog_common::partitioning::partition_for_person;
+ use personhog_coordination::changelog::KeyFilter;
+
+ let (new_topic, producer) = common::create_test_kafka_with_partitions(2).await;
+ let old_topic = common::owned_broker_topic(2).await;
+ let bootstrap = new_topic.bootstrap_servers();
+
+ let person_id = (1..)
+ .find(|id| partition_for_person(1, *id, 2) == 0)
+ .expect("a person on partition 0");
+ produce_to(&producer, &old_topic.topic, 0, &make_person(1, person_id)).await;
+ let mut newer = make_person(1, person_id);
+ newer.version = 2;
+ produce_to(&producer, &new_topic.topic, 0, &newer).await;
+ for filler in 1..=3 {
+ produce_to(&producer, &new_topic.topic, 0, &make_person(2, filler)).await;
+ }
+ // The writer drained the old partition and applied the live one's records.
+ let writer_group = format!("shadow-writer-{}", uuid::Uuid::now_v7());
+ commit_group_offset(&bootstrap, &writer_group, &old_topic.topic, 0, 1);
+ commit_group_offset(&bootstrap, &writer_group, &new_topic.topic, 0, 4);
+
+ let mut cfg = common::test_warming_config("shadow-warmer", &bootstrap);
+ cfg.writer_consumer_group = writer_group.clone();
+ let pools = WarmClientPools::new(&cfg.kafka, "shadow-warmer", &cfg.writer_consumer_group);
+ let cache = PartitionedCache::new(1 << 20);
+ let dirty_index = DirtyIndex::new(1_000_000);
+ let sources = [
+ WarmSource {
+ topic: Arc::from(old_topic.topic.as_str()),
+ partition: 0,
+ keep: Some(KeyFilter {
+ modulus: 2,
+ target: 0,
+ }),
+ },
+ WarmSource::live(&new_topic.topic, 0),
+ ];
+ warm_from_sources(&cfg, &pools, &cache, &dirty_index, 0, &sources)
+ .await
+ .expect("warm across generations");
+
+ let key = PersonCacheKey {
+ team_id: 1,
+ person_id,
+ };
+ assert!(
+ matches!(cache.get(0, &key), CacheLookup::PersonNotFound),
+ "a miss must load the newer row from PG"
+ );
+ assert!(dirty_index.get(&key).is_none());
+}
diff --git a/rust/personhog-stateright/src/model.rs b/rust/personhog-stateright/src/model.rs
index 25cb1a6b123f..9e3a857ed3e7 100644
--- a/rust/personhog-stateright/src/model.rs
+++ b/rust/personhog-stateright/src/model.rs
@@ -55,6 +55,7 @@ fn production_handoff(p: Partition, h: &Handoff) -> HandoffState {
freeze_quorum_ref: None,
created_at_ms: 0,
phase_entered_at_ms: 0,
+ config_epoch: 0,
}
}