Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(cohorts): fence partition ownership with offset commit stamps (#118890)

Matheus Batista4745d2c49a94
diff --git a/docs/internal/realtime-cohorts/README.md b/docs/internal/realtime-cohorts/README.md
index 2cbef2e41dbc..d612e207262a 100644
--- a/docs/internal/realtime-cohorts/README.md
+++ b/docs/internal/realtime-cohorts/README.md
@@ -91,7 +91,7 @@ When all partitions have reported, Django stamps the cohort ready, and feature f
[Processor runtime](processor-runtime.md#delivery-semantics) lists each path's guarantee.
- **The processor runs as a single pod.**
Each pod owns a fixed share of the partitions, and keeps their state in a local RocksDB store that never moves between pods.
- A second replica refuses to start unless `COHORT_POD_COUNT` is raised and every store is wiped, and then every slice starts without its history.
+ A second replica refuses to start unless `COHORT_POD_COUNT` is raised, every store is wiped and the old split's stamps on the broker are cleared, and then every slice starts without its history.
- **Most of the pipeline is off by default.**
The processor writes to a shadow output topic unless configured otherwise, and seed apply, reconcile, cascades, durable restore, the seeder's completion driver and Django's finalizer each have their own switch.
The pages name the switch where each behavior is described.
diff --git a/docs/internal/realtime-cohorts/processor-runtime.md b/docs/internal/realtime-cohorts/processor-runtime.md
index e630e5faa61d..8a2fe43ec5df 100644
--- a/docs/internal/realtime-cohorts/processor-runtime.md
+++ b/docs/internal/realtime-cohorts/processor-runtime.md
@@ -56,7 +56,7 @@ Cascades and seeds have one when those features are enabled.
No consumer subscribes to its group.
At startup each one is assigned the owned partitions, resuming at its group's committed offsets.
So the pod owns partition N on every input topic at once, which is what [partition affinity](event-routing.md#partition-affinity) needs, and no rebalance moves a partition while the pod runs.
-The consumer groups have no members and only hold the committed offsets.
+The consumer groups have no members and only hold the committed offsets and their [stamps](#ownership-and-the-single-pod-deployment).
A broker can expire the offsets of a group without members, so every consumer resets to `earliest`: a partition that lost its offset replays instead of skipping to the tail.
A commit from a client outside the group is refused while the group has a live member, such as the static member an older build left behind until its session expires.
librdkafka logs each refused commit as a `COMMITFAIL` warning, and the next commit retries it.
@@ -188,10 +188,33 @@ A store from before that record takes the configured split as its own the first
A store written under another split refuses to open, so the pod crash-loops instead of reading slices that another pod may have changed since.
With durable restore enabled, boot deletes the slices of partitions the pod does not own, so a single pod deletes nothing.
+The store record cannot stop a pod on an empty volume, so the **ownership fence** keeps a second record on the broker.
+Every offset commit on every input group carries the committer's split as Kafka commit metadata, for example `cohort-split/1:2:0:64` for pod 0 of 2 over 64 partitions.
+The pod reads these stamps three times:
+
+1. Before the store opens, it reads the last commit of every group on each partition it owns.
+ A stamp from another split, or metadata it cannot read, holds the boot.
+ The pod retries after 30 seconds, doubling the wait up to 5 minutes, and meanwhile `/_ready` reads not ready and `ownership_fence_blocked{reason}` is 1.
+ A commit without a stamp passes, because older builds and standard Kafka tools, such as an offset reset, write none.
+ A checkpoint restore waits for the same check again before it commits the follower positions.
+2. After the store opens, it checks the stamps again, commits every owned partition on every group with its own stamp, and reads the stamps back.
+ Another split's stamp, metadata it cannot read, or a stamp that reads back empty stops the pod with an error.
+ An empty read-back means the broker does not keep commit metadata, so the fence cannot work on it.
+3. Every 30 seconds, an audit reads the stamps on the owned partitions.
+ Another split's stamp, or metadata it cannot read, increments `ownership_fence_violations_total{topic}`, makes every later commit of the pod fail, and stops the pod.
+ So the final commits of that stop fail and leave the foreign stamp in place, and the next boot holds at step 1.
+
+Two processes of the same split, such as two pods with the same ordinal, write the same stamp, so the fence cannot tell them apart.
+Builds from before stamps commit without one, so the fence cannot see them either.
+Change `COHORT_POD_COUNT` only after every pod runs a build with stamps.
+
So changing `COHORT_POD_COUNT` requires wiping every store, including the volume of an ordinal that a scale-down removed.
That volume keeps its old record, so when a later change brings that split back, it opens with slices another pod has moved past.
With checkpoints enabled, also delete every pod's checkpoints, local and in S3.
A checkpoint from the old split restores and then refuses to open, so every boot fails the same way.
+The old split's stamps also hold every pod of the new split at boot.
+An offset reset of each input group to its current offsets clears them, because it commits without a stamp.
+Run it only after every pod of the old split has stopped, or the fence admits the new split while the old one still commits.
Each slice then begins again behind the coverage fence and holds no history until a backfill runs.
Nothing moves a slice between pods yet, so one pod that owns every partition is the supported deployment.
@@ -203,10 +226,12 @@ At boot the processor:
2. binds the health server, so `/_health` answers from here on and `/_ready` reads not ready,
3. loads the catalog once,
4. checks the partition counts of the co-partitioned topics against broker metadata,
-5. decides where the store comes from: with durable restore enabled, the live store on disk or a checkpoint, otherwise an empty store.
+5. waits until the [ownership fence](#ownership-and-the-single-pod-deployment) admits its split,
+6. decides where the store comes from: with durable restore enabled, the live store on disk or a checkpoint, otherwise an empty store.
A checkpoint restore that cannot finish blocks here and retries, as [a blocked restore](state-store-and-durability.md#a-blocked-restore) describes.
A store written under another split of the partitions refuses to open, as [ownership](#ownership-and-the-single-pod-deployment) explains,
-6. starts the timer loops and the events consumer, and starts the followers and the checkpoint loop once the catalog has loaded and boot recovery has ended.
+7. stamps every owned partition on every input group, and reads the stamps back,
+8. starts the timer loops and the events consumer, and starts the followers and the checkpoint loop once the catalog has loaded and boot recovery has ended.
By default the store is wiped at boot, so there is nothing to recover, and the events consumer dispatches from its first poll.
diff --git a/docs/internal/realtime-cohorts/state-store-and-durability.md b/docs/internal/realtime-cohorts/state-store-and-durability.md
index 13f8c732aac8..a789345f9678 100644
--- a/docs/internal/realtime-cohorts/state-store-and-durability.md
+++ b/docs/internal/realtime-cohorts/state-store-and-durability.md
@@ -216,6 +216,7 @@ The window bounds only the listing: a restore already published resumes at any a
A published restore stays pending until the events group commits the restored positions:
1. Before the store opens, the restore commits each follower group's positions on the partitions it keeps.
+ It first waits until the [ownership fence](processor-runtime.md#ownership-and-the-single-pod-deployment) admits the pod's split again, because another split can claim the partitions while the restore runs.
A failure blocks the boot.
2. The store opens, and the resets apply.
3. The events consumer seeks each owned partition to the lower of its restored position and the first event it polled, commits that seek list, then deletes the marker.
diff --git a/rust/cohort-stream-processor/clippy.toml b/rust/cohort-stream-processor/clippy.toml
index fa89894851e6..5de12f755f8c 100644
--- a/rust/cohort-stream-processor/clippy.toml
+++ b/rust/cohort-stream-processor/clippy.toml
@@ -32,4 +32,7 @@ disallowed-methods = [
{ path = "cohort_stream_processor::store::rocks::CohortStore::reset_slices", reason = "async code must not write the store on a runtime thread; sync contexts need a module-level allow — see store/handle.rs" },
{ path = "cohort_stream_processor::store::rocks::CohortStore::create_checkpoint", reason = "async code must go through the checkpoint sweeper's own offload; sync contexts need a module-level allow — see store/handle.rs" },
{ path = "cohort_stream_processor::store::rocks::CohortStore::stats_snapshot", reason = "async code must use StoreHandle::stats_snapshot; sync contexts need a module-level allow — see store/handle.rs" },
+ { path = "rdkafka::consumer::Consumer::commit", reason = "every offset commit must carry this pod's stamp; commit a StampedList — see partitions/stamp.rs" },
+ { path = "rdkafka::consumer::Consumer::commit_consumer_state", reason = "every offset commit must carry this pod's stamp; commit a StampedList — see partitions/stamp.rs" },
+ { path = "rdkafka::consumer::Consumer::commit_message", reason = "every offset commit must carry this pod's stamp; commit a StampedList — see partitions/stamp.rs" },
]
diff --git a/rust/cohort-stream-processor/src/consumers/boot.rs b/rust/cohort-stream-processor/src/consumers/boot.rs
index 518c8d1a8007..6c465d1d7e50 100644
--- a/rust/cohort-stream-processor/src/consumers/boot.rs
+++ b/rust/cohort-stream-processor/src/consumers/boot.rs
@@ -10,10 +10,9 @@
use std::collections::{BTreeMap, HashSet};
use rdkafka::error::KafkaResult;
-use rdkafka::{Offset, TopicPartitionList};
use crate::consumers::events::ConsumedEvent;
-use crate::partitions::ResumeOffset;
+use crate::partitions::{CommitStamp, ResumeOffset, StampedList};
use crate::store::durability::PendingRestore;
/// The lowest offset per events partition that boot polled and did not fold. A seek there fetches
@@ -38,20 +37,25 @@ impl ResumePoints {
}
}
- /// The seek list for the partitions still owned, or `None` when nothing needs a seek. Fails only
- /// on a negative offset, which a polled message never carries.
+ /// The seek list for the partitions still owned, stamped for the restore's commit, or `None`
+ /// when nothing needs a seek. Fails only on a negative offset, which a polled message never
+ /// carries.
pub(crate) fn seek_list(
&self,
+ stamp: &CommitStamp,
topic: &str,
owned: &HashSet<i32>,
- ) -> KafkaResult<Option<TopicPartitionList>> {
- let mut tpl = TopicPartitionList::new();
- for (&partition, &offset) in &self.0 {
- if owned.contains(&partition) {
- tpl.add_partition_offset(topic, partition, Offset::Offset(offset))?;
- }
+ ) -> KafkaResult<Option<StampedList>> {
+ let positions: Vec<(i32, i64)> = self
+ .0
+ .iter()
+ .filter(|(partition, _)| owned.contains(partition))
+ .map(|(&partition, &offset)| (partition, offset))
+ .collect();
+ if positions.is_empty() {
+ return Ok(None);
}
- Ok((tpl.count() > 0).then_some(tpl))
+ stamp.list(topic, positions).map(Some)
}
fn lower(&mut self, partition: i32, offset: i64) {
@@ -79,7 +83,10 @@ pub(crate) enum BootPhase {
#[cfg(test)]
mod tests {
use super::*;
+ use rdkafka::Offset;
+
use crate::consumers::events::CohortStreamEvent;
+ use crate::partitions::PartitionOwnership;
const TOPIC: &str = "cohort_stream_events";
@@ -106,9 +113,10 @@ mod tests {
}
}
- fn sought(seek_list: Option<TopicPartitionList>) -> Option<BTreeMap<i32, i64>> {
- seek_list.map(|tpl| {
- tpl.elements_for_topic(TOPIC)
+ fn sought(seek_list: Option<StampedList>) -> Option<BTreeMap<i32, i64>> {
+ seek_list.map(|list| {
+ list.as_list()
+ .elements_for_topic(TOPIC)
.iter()
.map(|elem| match elem.offset() {
Offset::Offset(offset) => (elem.partition(), offset),
@@ -186,8 +194,9 @@ mod tests {
resume.rewind_to(&positions);
}
+ let stamp = CommitStamp::for_test(PartitionOwnership::single(4));
assert_eq!(
- sought(resume.seek_list(TOPIC, &owned).unwrap()),
+ sought(resume.seek_list(&stamp, TOPIC, &owned).unwrap()),
case.expected
.map(|offsets| offsets.iter().copied().collect()),
"{}",
diff --git a/rust/cohort-stream-processor/src/consumers/events.rs b/rust/cohort-stream-processor/src/consumers/events.rs
index 42fbffd7eab1..bac733757bda 100644
--- a/rust/cohort-stream-processor/src/consumers/events.rs
+++ b/rust/cohort-stream-processor/src/consumers/events.rs
@@ -21,7 +21,7 @@ use metrics::{counter, gauge, histogram};
use rdkafka::consumer::{CommitMode, Consumer, ConsumerContext, StreamConsumer};
use rdkafka::error::KafkaResult;
use rdkafka::message::Message;
-use rdkafka::{Offset, TopicPartitionList};
+use rdkafka::Offset;
use tracing::{debug, error, info, warn};
pub use cohort_core::events::CohortStreamEvent;
@@ -51,7 +51,7 @@ use crate::partitions::pause::{ConsumerPauser, PartitionPauser};
use crate::partitions::rebalance::CohortConsumerContext;
use crate::partitions::router::{PartitionRouter, SeedRefusal, SeedSendOutcome, SendOutcome};
use crate::partitions::shuffle_message::ShuffleMessage;
-use crate::partitions::{list_offsets, InputTopic, PartitionOwnership};
+use crate::partitions::{list_offsets, ClaimedSplit, CommitStamp, InputTopic, PartitionOwnership};
use crate::producer::MembershipSink;
use crate::store::durability::{FinishError, PendingRestore};
use crate::store::StoreHandle;
@@ -881,15 +881,15 @@ pub struct CohortStreamEventsConsumer {
recv_batch_size: usize,
recv_batch_timeout: Duration,
offset_commit_interval: Duration,
- ownership: PartitionOwnership,
+ claimed: ClaimedSplit,
/// A checkpoint restore boot must finish. Boot rewinds every owned partition the restore keeps
/// to its restored position, commits, and finishes it before anything folds.
restore: Option<PendingRestore>,
}
impl CohortStreamEventsConsumer {
- /// Takes the partitions `ownership` owns for the life of the process: the consumer is
- /// assigned them at its group's committed offsets, and the dispatcher owns them.
+ /// Takes the partitions `claimed` owns for the life of the process: the consumer is assigned
+ /// them at its group's committed offsets, and the dispatcher owns them.
#[allow(clippy::too_many_arguments)]
pub fn new(
consumer: Arc<StreamConsumer<CohortConsumerContext>>,
@@ -899,9 +899,10 @@ impl CohortStreamEventsConsumer {
recv_batch_size: usize,
recv_batch_timeout: Duration,
offset_commit_interval: Duration,
- ownership: PartitionOwnership,
+ claimed: &ClaimedSplit,
restore: Option<PendingRestore>,
) -> KafkaResult<Self> {
+ let ownership = claimed.ownership();
consumer.assign(&ownership.assignment(&topic)?)?;
for partition in ownership.partitions() {
dispatcher.assign_partition(i32::from(partition));
@@ -923,7 +924,7 @@ impl CohortStreamEventsConsumer {
recv_batch_size,
recv_batch_timeout,
offset_commit_interval,
- ownership,
+ claimed: claimed.clone(),
restore,
})
}
@@ -944,6 +945,7 @@ impl CohortStreamEventsConsumer {
let commit_task = tokio::spawn(run_commit_loop(
self.consumer.clone(),
self.dispatcher.clone(),
+ self.claimed.stamp(),
self.topic.clone(),
self.offset_commit_interval,
self.handle.clone(),
@@ -971,7 +973,7 @@ impl CohortStreamEventsConsumer {
self.recover().await;
let mut resume = ResumePoints::default();
let topic = InputTopic::new(self.topic.as_str());
- resume.rewind_to(&restore.plan().positions(&topic, &self.ownership));
+ resume.rewind_to(&restore.plan().positions(&topic, &self.claimed.ownership()));
BootPhase::Rewinding {
resume,
restore: Box::new(restore),
@@ -1033,6 +1035,7 @@ impl CohortStreamEventsConsumer {
fsync_then_commit(
self.dispatcher.handle(),
&self.consumer,
+ &self.claimed.stamp(),
offsets,
&self.topic,
CommitMode::Sync,
@@ -1045,7 +1048,7 @@ impl CohortStreamEventsConsumer {
/// the partitions this pod does not own, and re-produce the merge outbox of those it owns.
async fn recover(&self) {
self.dispatcher
- .reclaim_unowned_slices(&self.ownership)
+ .reclaim_unowned_slices(&self.claimed.ownership())
.await;
self.dispatcher
.eager_redrive_pending_transfers_on_boot(&self.dispatcher.owned_set())
@@ -1066,7 +1069,7 @@ impl CohortStreamEventsConsumer {
boot_started: Instant,
) -> BootPhase {
let owned = self.dispatcher.owned_set();
- let seek_list = match resume.seek_list(&self.topic, &owned) {
+ let seek_list = match resume.seek_list(&self.claimed.stamp(), &self.topic, &owned) {
Ok(seek_list) => seek_list,
Err(err) => {
warn!(
@@ -1077,12 +1080,14 @@ impl CohortStreamEventsConsumer {
return BootPhase::Rewinding { resume, restore };
}
};
- let sought = seek_list.as_ref().map_or(0, TopicPartitionList::count);
+ let sought = seek_list
+ .as_ref()
+ .map_or(0, |seek_list| seek_list.as_list().count());
if let Some(seek_list) = &seek_list {
let failed: Vec<i32> = match self
.consumer
- .seek_partitions(seek_list.clone(), BOOT_SEEK_TIMEOUT)
+ .seek_partitions(seek_list.as_list().clone(), BOOT_SEEK_TIMEOUT)
{
Ok(result) => result
.elements_for_topic(&self.topic)
@@ -1114,7 +1119,7 @@ impl CohortStreamEventsConsumer {
list_offsets(self.consumer.as_ref(), partitions, Offset::Beginning)
};
let commit_events = || match &seek_list {
- Some(seek_list) => self.consumer.commit(seek_list, CommitMode::Sync),
+ Some(seek_list) => seek_list.commit(self.consumer.as_ref(), CommitMode::Sync),
None => Ok(()),
};
// A sync commit waits until the group coordinator answers. `block_in_place` hands this
@@ -1274,18 +1279,9 @@ struct ConsumeOutcome {
transport_error: bool,
}
-pub(crate) fn build_commit_tpl(topic: &str, offsets: &HashMap<i32, i64>) -> TopicPartitionList {
- let mut tpl = TopicPartitionList::new();
- for (&partition, &next_offset) in offsets {
- if let Err(err) = tpl.add_partition_offset(topic, partition, Offset::Offset(next_offset)) {
- warn!(topic, partition, next_offset, error = %err, "skipping partition in commit list");
- }
- }
- tpl
-}
-
pub(crate) fn commit_offsets<C: ConsumerContext>(
consumer: &StreamConsumer<C>,
+ stamp: &CommitStamp,
offsets: HashMap<i32, i64>,
topic: &str,
mode: CommitMode,
@@ -1293,8 +1289,10 @@ pub(crate) fn commit_offsets<C: ConsumerContext>(
if offsets.is_empty() {
return;
}
- let tpl = build_commit_tpl(topic, &offsets);
- match consumer.commit(&tpl, mode) {
+ match stamp
+ .list(topic, offsets)
+ .and_then(|list| list.commit(consumer, mode))
+ {
Ok(()) => counter!(COHORT_STREAM_OFFSET_COMMITS).increment(1),
Err(err) => {
counter!(COHORT_STREAM_OFFSET_COMMIT_ERRORS).increment(1);
@@ -1312,6 +1310,7 @@ pub(crate) fn commit_offsets<C: ConsumerContext>(
pub(crate) async fn fsync_then_commit<C: ConsumerContext>(
handle: &StoreHandle,
consumer: &StreamConsumer<C>,
+ stamp: &CommitStamp,
offsets: HashMap<i32, i64>,
topic: &str,
mode: CommitMode,
@@ -1322,7 +1321,7 @@ pub(crate) async fn fsync_then_commit<C: ConsumerContext>(
if handle.flush_wal_sync().await.is_err() {
return; // store counted the error; skip commit so `committed` never outruns `durable`
}
- commit_offsets(consumer, offsets, topic, mode);
+ commit_offsets(consumer, stamp, offsets, topic, mode);
}
/// Flushes the store's WAL and commits the owned committable offsets on a fixed interval, independent
@@ -1330,6 +1329,7 @@ pub(crate) async fn fsync_then_commit<C: ConsumerContext>(
async fn run_commit_loop(
consumer: Arc<StreamConsumer<CohortConsumerContext>>,
dispatcher: Arc<EventDispatcher>,
+ stamp: CommitStamp,
topic: String,
interval: Duration,
handle: Handle,
@@ -1344,6 +1344,7 @@ async fn run_commit_loop(
fsync_then_commit(
dispatcher.handle(),
&consumer,
+ &stamp,
dispatcher.owned_committable_offsets(),
&topic,
CommitMode::Async,
@@ -1650,35 +1651,6 @@ mod tests {
assert!(event.person_properties.is_none());
}
- #[test]
- fn build_commit_tpl_maps_each_partition_to_its_next_offset() {
- let mut offsets = HashMap::new();
- offsets.insert(3, 100);
- offsets.insert(7, 250);
-
- let tpl = build_commit_tpl("cohort_stream_events", &offsets);
-
- assert_eq!(tpl.count(), 2);
- assert_eq!(
- tpl.find_partition("cohort_stream_events", 3)
- .unwrap()
- .offset(),
- Offset::Offset(100),
- );
- assert_eq!(
- tpl.find_partition("cohort_stream_events", 7)
- .unwrap()
- .offset(),
- Offset::Offset(250),
- );
- }
-
- #[test]
- fn build_commit_tpl_for_no_offsets_is_empty() {
- let tpl = build_commit_tpl("cohort_stream_events", &HashMap::new());
- assert_eq!(tpl.count(), 0);
- }
-
fn temp_store() -> (TempDir, CohortStore) {
let dir = TempDir::new().unwrap();
let config = StoreConfig {
diff --git a/rust/cohort-stream-processor/src/consumers/merges.rs b/rust/cohort-stream-processor/src/consumers/merges.rs
index 629b42b199fc..d068e7cec626 100644
--- a/rust/cohort-stream-processor/src/consumers/merges.rs
+++ b/rust/cohort-stream-processor/src/consumers/merges.rs
@@ -27,6 +27,7 @@ use crate::observability::metrics::{
COHORT_STREAM_TRANSFERS_CONSUME_BATCH_SIZE, COHORT_STREAM_TRANSFER_DESERIALIZE_ERRORS,
};
use crate::partitions::offset_tracker::OffsetTracker;
+use crate::partitions::stamp::CommitStamp;
use crate::workers::MergeWorkerDeps;
/// One merge trigger consumed from `person_merge_events`, paired with its commit coordinates.
@@ -178,6 +179,7 @@ impl FollowerRoute for CascadeRoute {
pub struct FollowerConsumer<R: FollowerRoute> {
consumer: Arc<StreamConsumer>,
topic: String,
+ stamp: CommitStamp,
dispatcher: Arc<EventDispatcher>,
handle: Handle,
recv_batch_size: usize,
@@ -187,9 +189,11 @@ pub struct FollowerConsumer<R: FollowerRoute> {
}
impl<R: FollowerRoute> FollowerConsumer<R> {
+ #[allow(clippy::too_many_arguments)]
pub fn new(
consumer: Arc<StreamConsumer>,
topic: String,
+ stamp: CommitStamp,
dispatcher: Arc<EventDispatcher>,
handle: Handle,
recv_batch_size: usize,
@@ -199,6 +203,7 @@ impl<R: FollowerRoute> FollowerConsumer<R> {
Self {
consumer,
topic,
+ stamp,
dispatcher,
handle,
recv_batch_size,
@@ -232,6 +237,7 @@ impl<R: FollowerRoute> FollowerConsumer<R> {
fsync_then_commit(
self.dispatcher.handle(),
&self.consumer,
+ &self.stamp,
self.owned_committable_offsets(),
&self.topic,
CommitMode::Async,
@@ -253,6 +259,7 @@ impl<R: FollowerRoute> FollowerConsumer<R> {
fsync_then_commit(
self.dispatcher.handle(),
&self.consumer,
+ &self.stamp,
offsets,
&self.topic,
CommitMode::Sync,
diff --git a/rust/cohort-stream-processor/src/consumers/seeds.rs b/rust/cohort-stream-processor/src/consumers/seeds.rs
index 575839e62c7a..205ec0ae5b23 100644
--- a/rust/cohort-stream-processor/src/consumers/seeds.rs
+++ b/rust/cohort-stream-processor/src/consumers/seeds.rs
@@ -41,6 +41,7 @@ use crate::partitions::pacing::{
};
use crate::partitions::pause::PartitionPauser;
use crate::partitions::rebalance::CohortConsumerContext;
+use crate::partitions::stamp::CommitStamp;
use crate::partitions::watermarks::WatermarkMs;
/// Back-off after a Kafka transport error, mirroring the sibling consume loops.
@@ -454,6 +455,7 @@ fn run_admission_cycle(
pub struct SeedFollowerConsumer {
consumer: Arc<StreamConsumer>,
topic: String,
+ stamp: CommitStamp,
/// The events group's consumer, shared for the idle probe's watermark + boot-committed reads.
events_consumer: Arc<StreamConsumer<CohortConsumerContext>>,
events_topic: String,
@@ -474,6 +476,7 @@ impl SeedFollowerConsumer {
pub fn new(
consumer: Arc<StreamConsumer>,
topic: String,
+ stamp: CommitStamp,
events_consumer: Arc<StreamConsumer<CohortConsumerContext>>,
events_topic: String,
dispatcher: Arc<EventDispatcher>,
@@ -490,6 +493,7 @@ impl SeedFollowerConsumer {
Self {
consumer,
topic,
+ stamp,
events_consumer,
events_topic,
dispatcher,
@@ -541,6 +545,7 @@ impl SeedFollowerConsumer {
fsync_then_commit(
self.dispatcher.handle(),
&self.consumer,
+ &self.stamp,
self.owned_committable_offsets(),
&self.topic,
CommitMode::Async,
@@ -570,6 +575,7 @@ impl SeedFollowerConsumer {
fsync_then_commit(
self.dispatcher.handle(),
&self.consumer,
+ &self.stamp,
offsets,
&self.topic,
CommitMode::Sync,
diff --git a/rust/cohort-stream-processor/src/main.rs b/rust/cohort-stream-processor/src/main.rs
index eb4af5072710..5a24f17b36f3 100644
--- a/rust/cohort-stream-processor/src/main.rs
+++ b/rust/cohort-stream-processor/src/main.rs
@@ -26,8 +26,9 @@ use cohort_stream_processor::observability::disk::SharedDiskUtilization;
use cohort_stream_processor::observability::store_stats::{DiskProbe, StoreStatsSweeper};
use cohort_stream_processor::observability::tokio_monitor::TokioRuntimeMonitor;
use cohort_stream_processor::partitions::{
- CohortConsumerContext, ConsumerPauser, Follower, FollowerSet, InputGroups, LiveWatermarks,
- OffsetTracker, PartitionPauser, PartitionRouter,
+ AdmittedSplit, CohortConsumerContext, ConsumerPauser, Follower, FollowerSet, InputGroups,
+ LiveWatermarks, OffsetTracker, OwnershipAudit, PartitionPauser, PartitionRouter,
+ AUDIT_INTERVAL,
};
use cohort_stream_processor::producer::{
CascadeSink, KafkaCascadeSink, KafkaMembershipSink, KafkaReconcileMarkerSink,
@@ -320,9 +321,17 @@ async fn async_main(config: Config) -> Result<()> {
)
.context("checking the checkpoint directory")?;
}
- let (store, restore) = open_store(&config, ownership, lineage, groups.clone())
+ let admitted = AdmittedSplit::admit(ownership, groups.clone()).await;
+ let (store, restore) = open_store(&config, &admitted, lineage, groups.clone())
.await
.context("opening RocksDB state store")?;
+ let claimed = {
+ let groups = groups.clone();
+ tokio::task::spawn_blocking(move || admitted.claim(&groups))
+ .await
+ .context("the ownership claim panicked")?
+ .context("claiming this pod's partitions")?
+ };
let router = PartitionRouter::with_intake_cap(
config.partition_channel_buffer,
config.partition_intake_max_events,
@@ -474,7 +483,7 @@ async fn async_main(config: Config) -> Result<()> {
));
}
FollowerSet::new(followers)
- .assign(&ownership)
+ .assign(&claimed)
.context("assigning the follower consumers this pod's partitions")?;
let guard = manager.monitor_background();
@@ -555,6 +564,13 @@ async fn async_main(config: Config) -> Result<()> {
.start_monitoring(tokio_monitor_handle),
);
+ tokio::spawn(run_sweep_loop(
+ OwnershipAudit::new(groups.clone(), claimed.clone(), consumer_handle.clone()),
+ AUDIT_INTERVAL,
+ "ownership_audit",
+ consumer_handle.shutdown_token(),
+ ));
+
// Whole-DB checkpoints to the local volume and S3, from boot to the final one on a graceful stop.
// The loop waits for boot: a capture before the events rewind would record the broker's old
// offsets of a pending restore.
@@ -589,6 +605,7 @@ async fn async_main(config: Config) -> Result<()> {
let merge_follower = FollowerConsumer::<MergeRoute>::new(
merges_follower_consumer,
config.person_merge_events_topic.clone(),
+ claimed.stamp(),
dispatcher.clone(),
merge_follower_handle.clone(),
config.recv_batch_size,
@@ -604,6 +621,7 @@ async fn async_main(config: Config) -> Result<()> {
let transfer_follower = FollowerConsumer::<TransferRoute>::new(
transfers_follower_consumer,
config.cohort_merge_state_transfer_topic.clone(),
+ claimed.stamp(),
dispatcher.clone(),
transfer_follower_handle.clone(),
config.recv_batch_size,
@@ -622,6 +640,7 @@ async fn async_main(config: Config) -> Result<()> {
let cascade_follower = FollowerConsumer::<CascadeRoute>::new(
cascade_consumer,
config.cohort_cascade_events_topic.clone(),
+ claimed.stamp(),
dispatcher.clone(),
cascade_handle.clone(),
config.recv_batch_size,
@@ -641,6 +660,7 @@ async fn async_main(config: Config) -> Result<()> {
let seed_follower = SeedFollowerConsumer::new(
seed_consumer,
seed_topic,
+ claimed.stamp(),
stream_consumer.clone(),
config.cohort_stream_events_topic.clone(),
dispatcher.clone(),
@@ -672,7 +692,7 @@ async fn async_main(config: Config) -> Result<()> {
config.recv_batch_size,
config.recv_batch_timeout(),
config.offset_commit_interval(),
- ownership,
+ &claimed,
// A checkpoint restore boot finishes before anything folds; `None` when the store reopened
// or was created.
restore,
diff --git a/rust/cohort-stream-processor/src/observability/metrics.rs b/rust/cohort-stream-processor/src/observability/metrics.rs
index 66f137c18308..bab45a934bd3 100644
--- a/rust/cohort-stream-processor/src/observability/metrics.rs
+++ b/rust/cohort-stream-processor/src/observability/metrics.rs
@@ -307,6 +307,12 @@ pub const PARTITIONS_ASSIGNED_TOTAL: &str = "partitions_assigned_total";
pub const OWNED_PARTITIONS: &str = "owned_partitions";
/// This pod's place in the partition split, labelled by `pods` and `ordinal`. Always 1 (gauge).
pub const PARTITION_OWNERSHIP_INFO: &str = "partition_ownership_info";
+/// 1 while the ownership fence holds the boot, labelled by `reason` (`foreign_split`|`unrecognized`),
+/// else 0 (gauge). **Alert on it**: the pod reads not ready until the logged cause clears.
+pub const OWNERSHIP_FENCE_BLOCKED: &str = "ownership_fence_blocked";
+/// Commits by another split, or with unreadable metadata, that the run-time audit found on an owned
+/// partition, labelled by `topic` (counter). Each one stops the pod.
+pub const OWNERSHIP_FENCE_VIOLATIONS_TOTAL: &str = "ownership_fence_violations_total";
/// Events fully processed (counter).
pub const STAGE1_EVENTS_PROCESSED: &str = "stage1_events_processed_total";
@@ -859,6 +865,11 @@ mod tests {
"durable_restore_pending_transfers_recovered_partitions_total",
);
// Alerts select on these.
+ assert_eq!(OWNERSHIP_FENCE_BLOCKED, "ownership_fence_blocked");
+ assert_eq!(
+ OWNERSHIP_FENCE_VIOLATIONS_TOTAL,
+ "ownership_fence_violations_total"
+ );
assert_eq!(SLICES_ADOPTED_TOTAL, "cohort_slices_adopted_total");
assert_eq!(SLICES_BEGUN_TOTAL, "cohort_slices_begun_total");
assert_eq!(
diff --git a/rust/cohort-stream-processor/src/partitions/fence.rs b/rust/cohort-stream-processor/src/partitions/fence.rs
new file mode 100644
index 000000000000..f4cb3181d5b2
--- /dev/null
+++ b/rust/cohort-stream-processor/src/partitions/fence.rs
@@ -0,0 +1,477 @@
+//! The ownership fence finds a partition that two splits commit, across pods and restarts. Every
+//! commit on the input groups carries its committer's split (see [`super::stamp`]), and the fence
+//! reads those stamps three times.
+//!
+//! 1. **Admission**, before the store opens. A stamp from another split on an owned partition holds
+//! the boot.
+//! 2. **The claim**, after the store opens. The pod checks the stamps again, stamps every owned
+//! partition on every group and reads the stamps back, so a broker that drops commit metadata
+//! stops the pod.
+//! 3. **The audit**, while the pod runs. Another split's commit on an owned partition revokes this
+//! pod's commits and stops the pod, so the next boot holds in admission.
+//!
+//! The fence detects an overlap but cannot prevent one, because Kafka commits are last-writer-wins
+//! and have no compare-and-set. So a change of split relies on every pod of the old split stopping
+//! first. Two processes with one ordinal are not fenced: they carry the same stamp.
+
+use std::collections::{BTreeMap, BTreeSet};
+use std::sync::Arc;
+use std::time::Duration;
+
+use anyhow::bail;
+use async_trait::async_trait;
+use lifecycle::Handle;
+use metrics::{counter, gauge};
+use tracing::{error, info, warn};
+
+use super::positions::{InputGroups, InputTopic, PositionsError, ResumeOffset};
+use super::stamp::{CommitStamp, Stamp};
+use super::PartitionOwnership;
+use crate::observability::metrics::{OWNERSHIP_FENCE_BLOCKED, OWNERSHIP_FENCE_VIOLATIONS_TOTAL};
+use crate::store::durability::recovery::{RETRY_MAX, RETRY_START};
+use crate::sweep::Sweeper;
+
+pub const AUDIT_INTERVAL: Duration = Duration::from_secs(30);
+
+/// The last commit of one group on one partition.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum Committed {
+ Never,
+ At { offset: ResumeOffset, stamp: Stamp },
+}
+
+impl Committed {
+ pub fn stamp(&self) -> Option<&Stamp> {
+ match self {
+ Self::Never => None,
+ Self::At { stamp, .. } => Some(stamp),
+ }
+ }
+}
+
+/// One input group's last commits.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct InputCommits {
+ topic: InputTopic,
+ commits: BTreeMap<u16, Committed>,
+}
+
+impl InputCommits {
+ pub(super) fn new(topic: InputTopic, commits: BTreeMap<u16, Committed>) -> Self {
+ Self { topic, commits }
+ }
+
+ pub fn topic(&self) -> &InputTopic {
+ &self.topic
+ }
+
+ pub fn commits(&self) -> &BTreeMap<u16, Committed> {
+ &self.commits
+ }
+}
+
+/// The last commits on a set of partitions, on every enabled group.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CommitLedger {
+ events: InputCommits,
+ followers: Vec<InputCommits>,
+}
+
+impl CommitLedger {
+ pub(super) fn new(events: InputCommits, followers: Vec<InputCommits>) -> Self {
+ Self { events, followers }
+ }
+
+ /// Every group's commits, the events group first.
+ pub fn inputs(&self) -> impl Iterator<Item = &InputCommits> {
+ std::iter::once(&self.events).chain(&self.followers)
+ }
+
+ fn stamps(&self) -> impl Iterator<Item = (&InputTopic, u16, &Stamp)> {
+ self.inputs().flat_map(|input| {
+ input
+ .commits
+ .iter()
+ .filter_map(move |(&partition, committed)| {
+ Some((&input.topic, partition, committed.stamp()?))
+ })
+ })
+ }
+
+ /// The fence decision over a ledger that holds `ownership`'s partitions. An owned partition may
+ /// carry no stamp, from a build before stamps or a standard Kafka tool, or this pod's split.
+ fn admit(&self, ownership: PartitionOwnership) -> Result<(), FenceBlock> {
+ for (topic, partition, stamp) in self.stamps() {
+ if !ownership.owns(partition) {
+ continue;
+ }
+ match stamp {
+ Stamp::Unstamped => {}
+ Stamp::Split(found) if *found == ownership => {}
+ Stamp::Split(found) => {
+ return Err(FenceBlock::ForeignSplit {
+ topic: topic.clone(),
+ partition,
+ found: *found,
+ })
+ }
+ Stamp::Unrecognized(metadata) => {
+ return Err(FenceBlock::Unrecognized {
+ topic: topic.clone(),
+ partition,
+ metadata: metadata.clone(),
+ })
+ }
+ }
+ }
+ Ok(())
+ }
+
+ /// The claim's read-back: every partition of the ledger carries `ownership`'s stamp.
+ fn confirm(&self, ownership: PartitionOwnership) -> anyhow::Result<()> {
+ let ours = Stamp::Split(ownership);
+ for input in self.inputs() {
+ let topic = input.topic();
+ for (&partition, committed) in input.commits() {
+ match committed.stamp() {
+ Some(found) if *found == ours => {}
+ Some(found @ (Stamp::Split(_) | Stamp::Unrecognized(_))) => bail!(
+ "{topic} partition {partition} read back {found} after this pod stamped \
+ it: another process committed it during the claim"
+ ),
+ Some(Stamp::Unstamped) | None => bail!(
+ "the broker does not keep offset commit metadata, so the ownership fence \
+ cannot work: {topic} partition {partition} read back no stamp after this \
+ pod stamped it"
+ ),
+ }
+ }
+ }
+ Ok(())
+ }
+}
+
+/// Why the fence holds the boot.
+#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
+pub enum FenceBlock {
+ #[error(
+ "{found} committed {topic} partition {partition}: its pods may still run, or the pod count \
+ changed"
+ )]
+ ForeignSplit {
+ topic: InputTopic,
+ partition: u16,
+ found: PartitionOwnership,
+ },
+ #[error("{topic} partition {partition} carries commit metadata this build cannot read: {metadata:?}")]
+ Unrecognized {
+ topic: InputTopic,
+ partition: u16,
+ metadata: String,
+ },
+}
+
+impl FenceBlock {
+ const REASONS: [&'static str; 2] = ["foreign_split", "unrecognized"];
+
+ fn reason(&self) -> &'static str {
+ match self {
+ Self::ForeignSplit { .. } => Self::REASONS[0],
+ Self::Unrecognized { .. } => Self::REASONS[1],
+ }
+ }
+
+ fn record(blocked: Option<&Self>) {
+ let reason = blocked.map(Self::reason);
+ for label in Self::REASONS {
+ gauge!(OWNERSHIP_FENCE_BLOCKED, "reason" => label).set(if reason == Some(label) {
+ 1.0
+ } else {
+ 0.0
+ });
+ }
+ }
+}
+
+#[derive(Debug, thiserror::Error)]
+pub enum FenceError {
+ #[error(transparent)]
+ Blocked(#[from] FenceBlock),
+ #[error("reading the commit ledger failed")]
+ Ledger(#[from] PositionsError),
+}
+
+/// A split the fence admitted: no other split commits its partitions. Only the admission builds
+/// one. The claim hands its stamp on, so the audit's revoke reaches every stamp the pod uses.
+#[derive(Debug, Clone)]
+pub struct AdmittedSplit {
+ stamp: CommitStamp,
+}
+
+impl AdmittedSplit {
+ /// Retries [`Self::try_admit`] with backoff until the fence admits `ownership`. Each read runs
+ /// on a blocking thread, so the health server keeps answering, and the pod reads not ready.
+ pub async fn admit(ownership: PartitionOwnership, groups: Arc<InputGroups>) -> Self {
+ let mut backoff = RETRY_START;
+ loop {
+ let reader = groups.clone();
+ let attempt = match tokio::task::spawn_blocking(move || {
+ Self::try_admit(ownership, &reader)
+ })
+ .await
+ {
+ Ok(attempt) => attempt,
+ Err(err) => std::panic::resume_unwind(err.into_panic()),
+ };
+ match attempt {
+ Ok(admitted) => {
+ FenceBlock::record(None);
+ info!(%ownership, "the ownership fence admitted this pod");
+ return admitted;
+ }
+ Err(FenceError::Blocked(block)) => {
+ FenceBlock::record(Some(&block));
+ warn!(
+ %ownership,
+ %block,
+ retry_in_secs = backoff.as_secs(),
+ "the ownership fence holds the boot",
+ );
+ }
+ Err(FenceError::Ledger(err)) => warn!(
+ %ownership,
+ error = %err,
+ retry_in_secs = backoff.as_secs(),
+ "reading the commit ledger failed; retrying",
+ ),
+ }
+ tokio::time::sleep(backoff).await;
+ backoff = (backoff * 2).min(RETRY_MAX);
+ }
+ }
+
+ /// One fence read. Blocks on Kafka.
+ pub fn try_admit(
+ ownership: PartitionOwnership,
+ groups: &InputGroups,
+ ) -> Result<Self, FenceError> {
+ let partitions: BTreeSet<u16> = ownership.partitions().collect();
+ groups.ledger(&partitions)?.admit(ownership)?;
+ Ok(Self {
+ stamp: CommitStamp::new(ownership),
+ })
+ }
+
+ pub fn ownership(&self) -> PartitionOwnership {
+ self.stamp.split()
+ }
+
+ pub fn stamp(&self) -> CommitStamp {
+ self.stamp.clone()
+ }
+
+ /// Checks the fence again, then commits every owned partition on every group again with this
+ /// pod's stamp, at the group's commit or at the low watermark where the group never committed,
+ /// and reads the stamps back. Blocks on Kafka.
+ pub fn claim(self, groups: &InputGroups) -> anyhow::Result<ClaimedSplit> {
+ let ownership = self.ownership();
+ let partitions: BTreeSet<u16> = ownership.partitions().collect();
+ // Another split can claim while this pod opens its store, so the claim checks the fence
+ // again first. Commits have no compare-and-set: a commit between this read and the
+ // restamp is overwritten, and only the audit can catch it.
+ let ledger = groups.ledger(&partitions)?;
+ ledger.admit(ownership)?;
+ groups.restamp(&self.stamp, &ledger)?;
+ groups.ledger(&partitions)?.confirm(ownership)?;
+ info!(%ownership, "claimed every owned partition; the stamps read back");
+ Ok(ClaimedSplit(self.stamp))
+ }
+}
+
+/// A split whose every owned partition carries its stamp on every group. The consumers and the
+/// followers take it as proof, and commit with its stamp until the audit revokes it.
+#[derive(Debug, Clone)]
+pub struct ClaimedSplit(CommitStamp);
+
+impl ClaimedSplit {
+ pub fn ownership(&self) -> PartitionOwnership {
+ self.0.split()
+ }
+
+ pub fn stamp(&self) -> CommitStamp {
+ self.0.clone()
+ }
+}
+
+/// Stops the pod when another split commits one of its partitions. A commit without a stamp passes:
+/// an operator's offset reset writes none, and the claim proved that the broker keeps stamps.
+pub struct OwnershipAudit {
+ groups: Arc<InputGroups>,
+ claimed: ClaimedSplit,
+ handle: Handle,
+}
+
+impl OwnershipAudit {
+ pub fn new(groups: Arc<InputGroups>, claimed: ClaimedSplit, handle: Handle) -> Self {
+ Self {
+ groups,
+ claimed,
+ handle,
+ }
+ }
+}
+
+#[async_trait]
+impl Sweeper for OwnershipAudit {
+ async fn run_once(&self) {
+ let groups = self.groups.clone();
+ let partitions: BTreeSet<u16> = self.claimed.ownership().partitions().collect();
+ let ledger = match tokio::task::spawn_blocking(move || groups.ledger(&partitions)).await {
+ Ok(Ok(ledger)) => ledger,
+ Ok(Err(err)) => {
+ warn!(error = %err, "ownership audit: reading the commit ledger failed; skipping");
+ return;
+ }
+ Err(err) => {
+ error!(error = %err, "ownership audit: reading the commit ledger panicked; skipping");
+ return;
+ }
+ };
+ let ownership = self.claimed.ownership();
+ let mut violated = false;
+ for (topic, partition, stamp) in ledger.stamps() {
+ if stamp.is_foreign_to(&ownership) {
+ let label: Arc<str> = Arc::from(topic.as_str());
+ counter!(OWNERSHIP_FENCE_VIOLATIONS_TOTAL, "topic" => label).increment(1);
+ error!(
+ %topic,
+ partition,
+ %stamp,
+ %ownership,
+ "another split committed a partition this pod owns",
+ );
+ violated = true;
+ }
+ }
+ if violated {
+ // The graceful stop commits this pod's offsets with its stamp over the foreign one, and
+ // the next boot would then pass the fence.
+ self.claimed.0.revoke();
+ self.handle
+ .signal_failure("another split committed a partition this pod owns");
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use std::num::NonZeroU16;
+
+ use super::*;
+ use crate::store::durability::PodOrdinal;
+
+ const PARTITIONS: u16 = 4;
+ const EVENTS: &str = "cohort_stream_events";
+ const MERGES: &str = "person_merge_events";
+
+ fn split(pods: u16, ordinal: u16) -> PartitionOwnership {
+ PartitionOwnership::new(
+ NonZeroU16::new(pods).unwrap(),
+ PodOrdinal::from(ordinal),
+ PARTITIONS,
+ )
+ .unwrap()
+ }
+
+ /// `None` is a partition its group never committed.
+ fn commits(topic: &str, stamps: [Option<Stamp>; PARTITIONS as usize]) -> InputCommits {
+ let commits = (0..PARTITIONS)
+ .zip(stamps)
+ .map(|(partition, stamp)| {
+ let committed = match stamp {
+ None => Committed::Never,
+ Some(stamp) => Committed::At {
+ offset: ResumeOffset::try_from(7).unwrap(),
+ stamp,
+ },
+ };
+ (partition, committed)
+ })
+ .collect();
+ InputCommits::new(InputTopic::new(topic), commits)
+ }
+
+ fn by(pods: u16, ordinal: u16) -> Option<Stamp> {
+ Some(Stamp::Split(split(pods, ordinal)))
+ }
+
+ #[test]
+ fn the_fence_admits_a_split_only_over_its_own_or_no_stamps() {
+ let unstamped = || Some(Stamp::Unstamped);
+ let unreadable = || Some(Stamp::Unrecognized("other-tool".to_owned()));
+ type Row = (
+ &'static str,
+ PartitionOwnership,
+ [Option<Stamp>; 4],
+ [Option<Stamp>; 4],
+ Result<(), FenceBlock>,
+ );
+ let rows: Vec<Row> = vec![
+ (
+ "unreadable metadata blocks",
+ split(1, 0),
+ [by(1, 0), by(1, 0), unreadable(), by(1, 0)],
+ [by(1, 0), by(1, 0), by(1, 0), by(1, 0)],
+ Err(FenceBlock::Unrecognized {
+ topic: InputTopic::new(EVENTS),
+ partition: 2,
+ metadata: "other-tool".to_owned(),
+ }),
+ ),
+ (
+ "its own stamps, none, and partitions never committed",
+ split(2, 1),
+ [by(2, 0), by(2, 1), by(2, 0), unstamped()],
+ [None, by(2, 1), None, None],
+ Ok(()),
+ ),
+ (
+ "another split: the pod count changed",
+ split(2, 1),
+ [by(2, 0), by(1, 0), by(2, 0), by(1, 0)],
+ [by(2, 0), by(1, 0), by(2, 0), by(1, 0)],
+ Err(FenceBlock::ForeignSplit {
+ topic: InputTopic::new(EVENTS),
+ partition: 1,
+ found: split(1, 0),
+ }),
+ ),
+ ];
+ for (name, ownership, events, merges, expected) in rows {
+ let ledger = CommitLedger::new(commits(EVENTS, events), vec![commits(MERGES, merges)]);
+ assert_eq!(ledger.admit(ownership), expected, "{name}");
+ }
+ }
+
+ #[test]
+ fn the_claim_confirms_only_its_own_stamp_on_every_partition() {
+ let ours = || by(1, 0);
+ let rows = [
+ ("every partition reads back its stamp", ours(), true),
+ (
+ "the broker drops commit metadata",
+ Some(Stamp::Unstamped),
+ false,
+ ),
+ ("the restamp left a partition uncommitted", None, false),
+ ("another split committed during the claim", by(2, 0), false),
+ ];
+ for (name, read_back, confirmed) in rows {
+ let ledger = CommitLedger::new(
+ commits(EVENTS, [ours(), ours(), ours(), ours()]),
+ vec![commits(MERGES, [ours(), ours(), read_back, ours()])],
+ );
+ assert_eq!(ledger.confirm(split(1, 0)).is_ok(), confirmed, "{name}");
+ }
+ }
+}
diff --git a/rust/cohort-stream-processor/src/partitions/follower.rs b/rust/cohort-stream-processor/src/partitions/follower.rs
index 7f60d05338eb..e119831f81f5 100644
--- a/rust/cohort-stream-processor/src/partitions/follower.rs
+++ b/rust/cohort-stream-processor/src/partitions/follower.rs
@@ -9,7 +9,7 @@ use std::sync::Arc;
use rdkafka::consumer::{Consumer, StreamConsumer};
use rdkafka::error::KafkaResult;
-use crate::partitions::PartitionOwnership;
+use crate::partitions::ClaimedSplit;
/// One follower consumer and the topic its assignment is built against.
pub struct Follower {
@@ -36,9 +36,10 @@ impl FollowerSet {
}
}
- /// Assigns every follower the partitions `ownership` owns, each at its group's committed
+ /// Assigns every follower the partitions `claimed` owns, each at its group's committed
/// offset. A follower that fails would never consume its topic, so the first failure stops.
- pub fn assign(&self, ownership: &PartitionOwnership) -> KafkaResult<()> {
+ pub fn assign(&self, claimed: &ClaimedSplit) -> KafkaResult<()> {
+ let ownership = claimed.ownership();
for follower in &self.followers {
follower
.consumer
diff --git a/rust/cohort-stream-processor/src/partitions/mod.rs b/rust/cohort-stream-processor/src/partitions/mod.rs
index 75a7be778e12..cb6c75043589 100644
--- a/rust/cohort-stream-processor/src/partitions/mod.rs
+++ b/rust/cohort-stream-processor/src/partitions/mod.rs
@@ -6,6 +6,7 @@
//! runs the Stage 1 loop lives in [`crate::workers`].
pub mod backpressure;
+pub mod fence;
pub mod follower;
pub mod intake;
pub mod offset_tracker;
@@ -16,11 +17,16 @@ pub mod positions;
pub mod rebalance;
pub mod router;
pub mod shuffle_message;
+pub mod stamp;
pub mod watermarks;
pub use cohort_core::partitioner;
pub use backpressure::{Backpressure, PartitionHoldover};
+pub use fence::{
+ AdmittedSplit, ClaimedSplit, CommitLedger, Committed, FenceBlock, FenceError, InputCommits,
+ OwnershipAudit, AUDIT_INTERVAL,
+};
pub use follower::{Follower, FollowerSet};
pub use intake::{Admission, MeteredReceiver, PartitionIntake};
pub use offset_tracker::{DeferredOffset, MarkOutcome, OffsetTracker};
@@ -41,4 +47,5 @@ pub use router::{
WorkerInbox,
};
pub use shuffle_message::ShuffleMessage;
+pub use stamp::{CommitStamp, Stamp, StampedList};
pub use watermarks::{LiveWatermarks, WatermarkMs};
diff --git a/rust/cohort-stream-processor/src/partitions/positions.rs b/rust/cohort-stream-processor/src/partitions/positions.rs
index 7a699482a854..e6e17a421003 100644
--- a/rust/cohort-stream-processor/src/partitions/positions.rs
+++ b/rust/cohort-stream-processor/src/partitions/positions.rs
@@ -1,5 +1,5 @@
-//! Where each input topic resumes, read from its consumer group on the broker. A checkpoint records
-//! these positions, and a restore commits them back.
+//! Where each input topic resumes, and who committed it, read from its consumer group on the
+//! broker. A checkpoint records these positions, and a restore commits them back.
//!
//! Every commit the processor makes follows a WAL fsync, so a group's committed offset is at or
//! below the durable state, and a checkpoint taken after reading it contains at least that state.
@@ -16,6 +16,8 @@ use rdkafka::types::RDKafkaErrorCode;
use rdkafka::{ClientConfig, Offset, TopicPartitionList};
use serde::{Deserialize, Serialize};
+use super::fence::{CommitLedger, Committed, InputCommits};
+use super::stamp::{CommitStamp, Stamp};
use crate::config::Config;
const FETCH_TIMEOUT: Duration = Duration::from_secs(10);
@@ -163,8 +165,8 @@ pub enum PositionsError {
Negative(#[from] NegativeOffset),
}
-/// Reads one input's consumer-group positions without joining the group. Every method blocks, so
-/// async callers go through `spawn_blocking`.
+/// Reads and commits one input's consumer-group positions without joining the group. Every method
+/// blocks, so async callers go through `spawn_blocking`.
struct GroupReader {
topic: InputTopic,
config: ClientConfig,
@@ -190,38 +192,90 @@ impl GroupReader {
&self.topic
}
- /// The group's committed offset per partition, or the low watermark where the group has none.
- /// A consumer without a commit starts at or after the low watermark, so replay from there
- /// misses nothing.
- fn resume_positions(
+ /// The group's last commit on each of `partitions`, from one `OffsetFetch`.
+ fn commits(
&self,
+ client: &BaseConsumer,
partitions: &BTreeSet<u16>,
- ) -> Result<InputPositions, PositionsError> {
+ ) -> Result<InputCommits, PositionsError> {
let topic = self.topic.as_str();
let mut request = TopicPartitionList::new();
for &partition in partitions {
request.add_partition(topic, i32::from(partition));
}
- let client = self.client()?;
- let committed = client.committed_offsets(request, FETCH_TIMEOUT)?;
+ let answered = client.committed_offsets(request, FETCH_TIMEOUT)?;
+ let mut commits = BTreeMap::new();
+ for &partition in partitions {
+ let committed = match answered.find_partition(topic, i32::from(partition)) {
+ Some(elem) => {
+ elem.error()?;
+ match elem.offset() {
+ Offset::Offset(next) => Committed::At {
+ offset: ResumeOffset::try_from(next)?,
+ stamp: Stamp::parse(elem.metadata()),
+ },
+ _ => Committed::Never,
+ }
+ }
+ None => Committed::Never,
+ };
+ commits.insert(partition, committed);
+ }
+ Ok(InputCommits::new(self.topic.clone(), commits))
+ }
- let mut offsets = BTreeMap::new();
+ /// Where the group resumes on each partition of `commits`: its committed offset, or the low
+ /// watermark where it has none. Every consumer here starts at `earliest`, so replay from the low
+ /// watermark misses nothing.
+ fn resume_positions(
+ &self,
+ client: &BaseConsumer,
+ commits: &InputCommits,
+ ) -> Result<BTreeMap<u16, ResumeOffset>, PositionsError> {
+ let mut positions = BTreeMap::new();
let mut uncommitted = Vec::new();
- for &partition in partitions {
- let commit = committed.find_partition(topic, i32::from(partition));
- match commit.as_ref().map(|elem| (elem.error(), elem.offset())) {
- Some((Err(err), _)) => return Err(err.into()),
- Some((Ok(()), Offset::Offset(next))) => {
- offsets.insert(partition, ResumeOffset::try_from(next)?);
+ for (&partition, committed) in commits.commits() {
+ match committed {
+ Committed::At { offset, .. } => {
+ positions.insert(partition, *offset);
}
- _ => uncommitted.push((&self.topic, partition)),
+ Committed::Never => uncommitted.push((&self.topic, partition)),
}
}
- let lows = list_offsets(&client, &uncommitted, Offset::Beginning)?;
+ let lows = list_offsets(client, &uncommitted, Offset::Beginning)?;
for (&(_, partition), low) in uncommitted.iter().zip(lows) {
- offsets.insert(partition, ResumeOffset::try_from(low)?);
+ positions.insert(partition, ResumeOffset::try_from(low)?);
}
- Ok(InputPositions::new(self.topic.clone(), offsets))
+ Ok(positions)
+ }
+
+ fn positions(&self, partitions: &BTreeSet<u16>) -> Result<InputPositions, PositionsError> {
+ let client = self.client()?;
+ let commits = self.commits(&client, partitions)?;
+ let positions = self.resume_positions(&client, &commits)?;
+ Ok(InputPositions::new(self.topic.clone(), positions))
+ }
+
+ /// Commits each partition of `commits` again at its resume position, with `stamp`.
+ fn restamp(&self, stamp: &CommitStamp, commits: &InputCommits) -> Result<(), PositionsError> {
+ let positions = self.resume_positions(&self.client()?, commits)?;
+ Ok(self.commit(stamp, &positions)?)
+ }
+
+ fn commit(
+ &self,
+ stamp: &CommitStamp,
+ positions: &BTreeMap<u16, ResumeOffset>,
+ ) -> KafkaResult<()> {
+ if positions.is_empty() {
+ return Ok(());
+ }
+ let offsets = positions
+ .iter()
+ .map(|(&partition, &offset)| (i32::from(partition), offset.get()));
+ stamp
+ .list(self.topic.as_str(), offsets)?
+ .commit(&self.client()?, CommitMode::Sync)
}
fn watermarks(
@@ -232,8 +286,8 @@ impl GroupReader {
}
}
-/// A follower's group has no members, so a commit from any client needs no generation. Only
-/// followers get one: the events group commits through its member consumer.
+/// A follower's group has no members, so a commit from any client needs no generation. The
+/// restore commits follower positions through it before any consumer runs.
pub struct FollowerGroup(GroupReader);
impl FollowerGroup {
@@ -241,19 +295,12 @@ impl FollowerGroup {
self.0.topic()
}
- pub fn commit(&self, positions: &BTreeMap<u16, ResumeOffset>) -> KafkaResult<()> {
- if positions.is_empty() {
- return Ok(());
- }
- let mut tpl = TopicPartitionList::new();
- for (&partition, &offset) in positions {
- tpl.add_partition_offset(
- self.topic().as_str(),
- i32::from(partition),
- Offset::Offset(offset.get()),
- )?;
- }
- self.0.client()?.commit(&tpl, CommitMode::Sync)
+ pub fn commit(
+ &self,
+ stamp: &CommitStamp,
+ positions: &BTreeMap<u16, ResumeOffset>,
+ ) -> KafkaResult<()> {
+ self.0.commit(stamp, positions)
}
}
@@ -317,13 +364,40 @@ impl InputGroups {
&self.followers
}
+ /// The last commit on `partitions`, on every enabled group.
+ pub fn ledger(&self, partitions: &BTreeSet<u16>) -> Result<CommitLedger, PositionsError> {
+ let read = |reader: &GroupReader| reader.commits(&reader.client()?, partitions);
+ let events = read(&self.events)?;
+ let followers = self
+ .followers
+ .iter()
+ .map(|follower| read(&follower.0))
+ .collect::<Result<_, _>>()?;
+ Ok(CommitLedger::new(events, followers))
+ }
+
+ /// Commits the partitions of `ledger`, read by [`Self::ledger`], again with `stamp` on every
+ /// enabled group.
+ pub(super) fn restamp(
+ &self,
+ stamp: &CommitStamp,
+ ledger: &CommitLedger,
+ ) -> Result<(), PositionsError> {
+ self.readers()
+ .zip(ledger.inputs())
+ .try_for_each(|(reader, commits)| {
+ debug_assert_eq!(reader.topic(), commits.topic());
+ reader.restamp(stamp, commits)
+ })
+ }
+
/// Every input's resume positions on `partitions`.
pub fn resume_positions(
&self,
partitions: &BTreeSet<u16>,
) -> Result<Vec<InputPositions>, PositionsError> {
self.readers()
- .map(|reader| reader.resume_positions(partitions))
+ .map(|reader| reader.positions(partitions))
.collect()
}
diff --git a/rust/cohort-stream-processor/src/partitions/stamp.rs b/rust/cohort-stream-processor/src/partitions/stamp.rs
new file mode 100644
index 000000000000..c330cdb1d97e
--- /dev/null
+++ b/rust/cohort-stream-processor/src/partitions/stamp.rs
@@ -0,0 +1,232 @@
+//! Commit stamps. Every offset commit on the input groups carries the committer's split as Kafka
+//! commit metadata, so a pod can tell who last committed each of its partitions (see
+//! [`super::fence`]). [`StampedList::commit`] is the only offset commit in the crate, and only a
+//! [`CommitStamp`] builds a [`StampedList`].
+
+use std::fmt;
+use std::num::NonZeroU16;
+
+use rdkafka::consumer::{CommitMode, Consumer, ConsumerContext};
+use rdkafka::error::{KafkaError, KafkaResult};
+use rdkafka::types::RDKafkaErrorCode;
+use rdkafka::{Offset, TopicPartitionList};
+use tokio_util::sync::CancellationToken;
+
+use super::PartitionOwnership;
+use crate::store::durability::PodOrdinal;
+
+/// The format tag. A later format gets a new tag, which this build reads as unrecognized.
+const PREFIX: &str = "cohort-split/1:";
+
+/// What a commit's metadata says about who made it.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum Stamp {
+ /// Empty: a build from before stamps, or a standard Kafka tool such as an offset reset.
+ Unstamped,
+ Split(PartitionOwnership),
+ /// Metadata this build cannot read. The fence treats it like another split.
+ Unrecognized(String),
+}
+
+impl Stamp {
+ pub fn parse(metadata: &str) -> Self {
+ if metadata.is_empty() {
+ return Self::Unstamped;
+ }
+ Self::decode(metadata).map_or_else(|| Self::Unrecognized(metadata.to_owned()), Self::Split)
+ }
+
+ fn decode(metadata: &str) -> Option<PartitionOwnership> {
+ let mut fields = metadata.strip_prefix(PREFIX)?.split(':');
+ let (Some(pods), Some(ordinal), Some(partitions), None) =
+ (fields.next(), fields.next(), fields.next(), fields.next())
+ else {
+ return None;
+ };
+ PartitionOwnership::new(
+ pods.parse::<NonZeroU16>().ok()?,
+ PodOrdinal::from(ordinal.parse::<u16>().ok()?),
+ partitions.parse().ok()?,
+ )
+ .ok()
+ }
+
+ /// True when a split other than `split` made the commit, or when the metadata cannot say.
+ pub fn is_foreign_to(&self, split: &PartitionOwnership) -> bool {
+ match self {
+ Self::Unstamped => false,
+ Self::Split(found) => found != split,
+ Self::Unrecognized(_) => true,
+ }
+ }
+}
+
+impl fmt::Display for Stamp {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ match self {
+ Self::Unstamped => f.write_str("no stamp"),
+ Self::Split(split) => write!(f, "{split}"),
+ Self::Unrecognized(metadata) => write!(f, "unreadable metadata {metadata:?}"),
+ }
+ }
+}
+
+/// This pod's stamp. Only an admitted split builds one. The ownership audit revokes it when another
+/// split commits an owned partition, and from then on no list built from it or a clone commits.
+#[derive(Debug, Clone)]
+pub struct CommitStamp {
+ split: PartitionOwnership,
+ revoked: CancellationToken,
+}
+
+impl CommitStamp {
+ pub(super) fn new(split: PartitionOwnership) -> Self {
+ Self {
+ split,
+ revoked: CancellationToken::new(),
+ }
+ }
+
+ #[cfg(test)]
+ pub(crate) fn for_test(split: PartitionOwnership) -> Self {
+ Self::new(split)
+ }
+
+ pub(super) fn split(&self) -> PartitionOwnership {
+ self.split
+ }
+
+ pub(super) fn revoke(&self) {
+ self.revoked.cancel();
+ }
+
+ /// `offsets` on `topic` as a commit list, each element stamped. Fails on a negative offset.
+ pub fn list(
+ &self,
+ topic: &str,
+ offsets: impl IntoIterator<Item = (i32, i64)>,
+ ) -> KafkaResult<StampedList> {
+ let metadata = self.to_string();
+ let mut list = TopicPartitionList::new();
+ for (partition, offset) in offsets {
+ let mut elem = list.add_partition(topic, partition);
+ elem.set_offset(Offset::Offset(offset))?;
+ elem.set_metadata(&metadata);
+ }
+ Ok(StampedList {
+ list,
+ revoked: self.revoked.clone(),
+ })
+ }
+}
+
+impl fmt::Display for CommitStamp {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ write!(
+ f,
+ "{PREFIX}{}:{}:{}",
+ self.split.pods(),
+ self.split.ordinal(),
+ self.split.partition_count()
+ )
+ }
+}
+
+/// A commit list where every element carries this pod's stamp.
+#[derive(Debug)]
+pub struct StampedList {
+ list: TopicPartitionList,
+ revoked: CancellationToken,
+}
+
+impl StampedList {
+ /// The list without its role as a commit, for a seek, which ignores metadata.
+ pub fn as_list(&self) -> &TopicPartitionList {
+ &self.list
+ }
+
+ /// Fails with `Fenced` once the stamp is revoked.
+ #[allow(clippy::disallowed_methods)]
+ pub fn commit<C, X>(&self, consumer: &C, mode: CommitMode) -> KafkaResult<()>
+ where
+ C: Consumer<X>,
+ X: ConsumerContext,
+ {
+ if self.revoked.is_cancelled() {
+ return Err(KafkaError::ConsumerCommit(RDKafkaErrorCode::Fenced));
+ }
+ consumer.commit(&self.list, mode)
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn every_split_of_the_partitions_reads_back_from_its_stamp() {
+ for pods in [1, 2, 4, 8, 16, 32, 64] {
+ let pods = NonZeroU16::new(pods).unwrap();
+ for ordinal in 0..pods.get() {
+ let split = PartitionOwnership::new(pods, PodOrdinal::from(ordinal), 64).unwrap();
+ let stamp = CommitStamp::new(split).to_string();
+ assert_eq!(Stamp::parse(&stamp), Stamp::Split(split), "{stamp}");
+ }
+ }
+ assert_eq!(
+ CommitStamp::new(PartitionOwnership::single(64)).to_string(),
+ "cohort-split/1:1:0:64"
+ );
+ }
+
+ #[test]
+ fn empty_metadata_is_unstamped_and_anything_else_unreadable_is_unrecognized() {
+ let own = PartitionOwnership::single(64);
+ assert_eq!(Stamp::parse(""), Stamp::Unstamped);
+ assert!(!Stamp::Unstamped.is_foreign_to(&own));
+ assert!(Stamp::parse("cohort-split/1:1:0:32").is_foreign_to(&own));
+ for metadata in [
+ "cohort-split/2:2:0:64",
+ "other-tool",
+ "cohort-split/1:2:0",
+ "cohort-split/1:2:0:64:1",
+ "cohort-split/1:0:0:64",
+ "cohort-split/1:3:0:64",
+ "cohort-split/1:2:2:64",
+ "cohort-split/1:2:x:64",
+ ] {
+ let stamp = Stamp::parse(metadata);
+ assert_eq!(
+ stamp,
+ Stamp::Unrecognized(metadata.to_owned()),
+ "{metadata}"
+ );
+ assert!(stamp.is_foreign_to(&own), "the audit stops on {metadata}");
+ }
+ }
+
+ #[test]
+ fn a_stamped_list_carries_the_stamp_on_every_offset() {
+ let split =
+ PartitionOwnership::new(NonZeroU16::new(2).unwrap(), PodOrdinal::from(1), 4).unwrap();
+ let list = CommitStamp::new(split)
+ .list("cohort_stream_events", [(1, 10), (3, 0)])
+ .unwrap();
+ let elements: Vec<(i32, Offset, String)> = list
+ .as_list()
+ .elements()
+ .iter()
+ .map(|elem| (elem.partition(), elem.offset(), elem.metadata().to_owned()))
+ .collect();
+ assert_eq!(
+ elements,
+ [
+ (1, Offset::Offset(10), "cohort-split/1:2:1:4".to_owned()),
+ (3, Offset::Offset(0), "cohort-split/1:2:1:4".to_owned()),
+ ],
+ );
+ assert!(CommitStamp::new(split)
+ .list("cohort_stream_events", [(1, -1)])
+ .is_err());
+ }
+}
diff --git a/rust/cohort-stream-processor/src/store/durability/recovery.rs b/rust/cohort-stream-processor/src/store/durability/recovery.rs
index 0087abc9b089..0d99a94657b5 100644
--- a/rust/cohort-stream-processor/src/store/durability/recovery.rs
+++ b/rust/cohort-stream-processor/src/store/durability/recovery.rs
@@ -42,11 +42,11 @@ use crate::observability::metrics::{
CHECKPOINT_RESTORE_BLOCKED, CHECKPOINT_RESTORE_CANDIDATES_TOTAL,
CHECKPOINT_RESTORE_DURATION_SECONDS, CHECKPOINT_RESTORE_TOTAL,
};
-use crate::partitions::{InputGroups, InputTopic, PartitionOwnership};
+use crate::partitions::{AdmittedSplit, InputGroups, InputTopic};
use crate::store::{CohortStore, StoreError, STORE_SCHEMA_VERSION};
-const RETRY_START: Duration = Duration::from_secs(30);
-const RETRY_MAX: Duration = Duration::from_secs(300);
+pub(crate) const RETRY_START: Duration = Duration::from_secs(30);
+pub(crate) const RETRY_MAX: Duration = Duration::from_secs(300);
/// Where the live store came from.
#[derive(Debug)]
@@ -198,17 +198,17 @@ impl Search {
///
/// A restore that cannot finish blocks here, retrying with backoff, until the operator fixes the
/// cause or turns checkpoints off. `lineage` is `None` when checkpoints are disabled. The store
-/// refuses to open when it was written under another `ownership`.
+/// refuses to open when it was written under another split than the one `admitted`.
pub async fn open_store(
config: &Config,
- ownership: PartitionOwnership,
+ admitted: &AdmittedSplit,
lineage: Option<CheckpointLineage>,
groups: Arc<InputGroups>,
) -> Result<(CohortStore, Option<PendingRestore>), StoreError> {
- let boot = restore_until_ready(config.clone(), ownership, lineage, groups).await;
+ let boot = restore_until_ready(config.clone(), admitted.clone(), lineage, groups).await;
// A published restore leaves a store at the path, so `effective_wipe_on_start` keeps it.
- let store_config = config.store_config(ownership);
+ let store_config = config.store_config(admitted.ownership());
info!(
durable_restore_enabled = config.durable_restore_enabled,
wipe_store_on_start = config.wipe_store_on_start,
@@ -228,15 +228,16 @@ pub async fn open_store(
Ok((store, restore))
}
-/// Retries [`prepare_store`] with backoff until the boot has a store, and positions the follower
-/// groups of a restore on the partitions `ownership` owns before returning it.
+/// Retries [`prepare_store`] with backoff until the boot has a store. For a restore, it waits until
+/// the fence admits `admitted`'s split again, then positions the follower groups on the partitions
+/// the split owns before returning it.
///
/// The restore runs on a blocking thread: staging, validation and the Kafka position calls block,
/// and the health server must keep answering meanwhile. A SIGTERM here kills the process; staging
/// survives a kill.
async fn restore_until_ready(
config: Config,
- ownership: PartitionOwnership,
+ admitted: AdmittedSplit,
lineage: Option<CheckpointLineage>,
groups: Arc<InputGroups>,
) -> BootStore {
@@ -260,10 +261,16 @@ async fn restore_until_ready(
)
.await
{
- Ok((BootStore::Restored(pending), source)) => pending
- .position_followers(&groups, &ownership)
- .map(|()| (BootStore::Restored(pending), source))
- .map_err(RestoreBlocked::Positions),
+ Ok((BootStore::Restored(pending), source)) => {
+ // Another split can claim while the restore runs, so the boot's admission
+ // no longer covers the follower commits.
+ let admitted =
+ AdmittedSplit::admit(admitted.ownership(), groups.clone()).await;
+ pending
+ .position_followers(&groups, &admitted)
+ .map(|()| (BootStore::Restored(pending), source))
+ .map_err(RestoreBlocked::Positions)
+ }
other => other,
};
match attempt {
diff --git a/rust/cohort-stream-processor/src/store/durability/stage.rs b/rust/cohort-stream-processor/src/store/durability/stage.rs
index 18b9bb4b0bd1..83cf87a84f5a 100644
--- a/rust/cohort-stream-processor/src/store/durability/stage.rs
+++ b/rust/cohort-stream-processor/src/store/durability/stage.rs
@@ -21,7 +21,7 @@ use super::manifest::OffsetManifest;
use super::metadata::CheckpointMetadata;
use super::restore_plan::{ReplayWindow, RestorePlan};
use crate::observability::metrics::CHECKPOINT_RESTORE_SLICES_RESET_TOTAL;
-use crate::partitions::{InputGroups, InputTopic, PartitionOwnership, ResumeOffset};
+use crate::partitions::{AdmittedSplit, InputGroups, InputTopic, ResumeOffset};
use crate::store::{CohortStore, StoreError};
/// The marker a published restore carries inside the store directory. RocksDB ignores files it
@@ -205,16 +205,19 @@ impl PendingRestore {
&self.record.source
}
- /// Commits each follower's planned positions on the partitions `ownership` owns. The other
- /// partitions belong to other pods, whose followers keep their own commits. Safe to repeat:
- /// nothing consumes a follower before the restore finishes.
+ /// Commits each follower's planned positions on the partitions `admitted` owns, with its stamp.
+ /// The other partitions belong to other pods, whose followers keep their own commits. Safe to
+ /// repeat: nothing consumes a follower before the restore finishes.
pub(crate) fn position_followers(
&self,
groups: &InputGroups,
- ownership: &PartitionOwnership,
+ admitted: &AdmittedSplit,
) -> KafkaResult<()> {
for follower in groups.followers() {
- follower.commit(&self.plan.positions(follower.topic(), ownership))?;
+ follower.commit(
+ &admitted.stamp(),
+ &self.plan.positions(follower.topic(), &admitted.ownership()),
+ )?;
}
Ok(())
}
diff --git a/rust/cohort-stream-processor/tests/cascade_consumer.rs b/rust/cohort-stream-processor/tests/cascade_consumer.rs
index f38dadb13850..33c95e71ae51 100644
--- a/rust/cohort-stream-processor/tests/cascade_consumer.rs
+++ b/rust/cohort-stream-processor/tests/cascade_consumer.rs
@@ -13,7 +13,7 @@
//!
//! Each test deletes its topics on the way out via `with_topics_cleanup`.
-use std::collections::HashSet;
+use std::collections::{HashMap, HashSet};
use std::future::Future;
use std::panic::AssertUnwindSafe;
use std::sync::Arc;
@@ -21,6 +21,7 @@ use std::time::{Duration, Instant};
use chrono_tz::UTC;
use cohort_stream_processor::cascade::{first_cascade, CascadeMessage};
+use cohort_stream_processor::config::Config;
use cohort_stream_processor::consumers::{
CascadeRoute, CohortStreamEventsConsumer, EventDispatcher, FollowerConsumer, MergeRoute,
TransferRoute,
@@ -29,8 +30,9 @@ use cohort_stream_processor::filters::{
CatalogHandle, CohortId, FilterCatalog, TeamFiltersBuilder, TeamId,
};
use cohort_stream_processor::partitions::{
- merge_partition_key, partition_of, CohortConsumerContext, Follower, FollowerSet, OffsetTracker,
- PartitionOwnership, PartitionRouter, COHORT_PARTITION_COUNT,
+ merge_partition_key, partition_of, AdmittedSplit, ClaimedSplit, CohortConsumerContext,
+ Follower, FollowerSet, InputGroups, OffsetTracker, PartitionOwnership, PartitionRouter,
+ COHORT_PARTITION_COUNT,
};
use cohort_stream_processor::producer::{
CascadeSink, CohortMembershipChange, KafkaCascadeSink, KafkaMembershipSink,
@@ -44,6 +46,7 @@ use cohort_stream_processor::workers::{
CascadeConfig, MergeWorkerDeps, TransferRetryPolicy, DEFAULT_MERGE_GC_SCAN_LIMIT,
};
use common_kafka::config::KafkaConfig;
+use envconfig::Envconfig;
use futures::FutureExt;
use lifecycle::{ComponentOptions, Handle, Manager};
use rdkafka::admin::{AdminClient, AdminOptions, NewTopic, TopicReplication};
@@ -380,6 +383,36 @@ impl Groups {
cascade: format!("cohort-stream-cascade-{suffix}"),
}
}
+
+ /// The processor config that reads `topics` through these groups.
+ fn config(&self, topics: &Topics) -> Config {
+ let mut config = Config::init_from_hashmap(&HashMap::new()).expect("default config");
+ config.kafka_hosts = bootstrap_servers();
+ config.cohort_stream_events_topic = topics.events.clone();
+ config.kafka_consumer_group = self.events.clone();
+ config.person_merge_events_topic = topics.merges.clone();
+ config.kafka_merge_consumer_group = self.merges.clone();
+ config.cohort_merge_state_transfer_topic = topics.transfers.clone();
+ config.kafka_merge_apply_consumer_group = self.transfers.clone();
+ config.cohort_cascade_enabled = true;
+ config.cohort_cascade_events_topic = topics.cascade.clone();
+ config.kafka_cascade_consumer_group = self.cascade.clone();
+ config
+ }
+}
+
+/// Admits and claims `ownership` through the ownership fence on the groups `config` names, as the
+/// processor does at boot.
+async fn claim(config: Config, ownership: PartitionOwnership) -> ClaimedSplit {
+ tokio::task::spawn_blocking(move || {
+ let groups = InputGroups::new(&config);
+ AdmittedSplit::try_admit(ownership, &groups)
+ .expect("the fence admits the split")
+ .claim(&groups)
+ .expect("claim the owned partitions")
+ })
+ .await
+ .expect("the claim panicked")
}
fn register_instance(manager: &mut Manager) -> [Handle; 4] {
@@ -499,13 +532,17 @@ async fn spawn_instance(
.create()
.expect("create cascade follower consumer"),
);
- let ownership = PartitionOwnership::single(NUM_PARTITIONS as u16);
+ let claimed = claim(
+ groups.config(topics),
+ PartitionOwnership::single(NUM_PARTITIONS as u16),
+ )
+ .await;
FollowerSet::new([
Follower::new(merges_consumer.clone(), topics.merges.clone()),
Follower::new(transfers_consumer.clone(), topics.transfers.clone()),
Follower::new(cascade_consumer.clone(), topics.cascade.clone()),
])
- .assign(&ownership)
+ .assign(&claimed)
.expect("assign the followers");
let events_client: StreamConsumer<CohortConsumerContext> = events_client_config(&groups.events)
@@ -517,6 +554,7 @@ async fn spawn_instance(
let merge_follower = FollowerConsumer::<MergeRoute>::new(
merges_consumer,
topics.merges.clone(),
+ claimed.stamp(),
dispatcher.clone(),
merge_handle,
100,
@@ -528,6 +566,7 @@ async fn spawn_instance(
let transfer_follower = FollowerConsumer::<TransferRoute>::new(
transfers_consumer,
topics.transfers.clone(),
+ claimed.stamp(),
dispatcher.clone(),
transfer_handle,
100,
@@ -539,6 +578,7 @@ async fn spawn_instance(
let cascade_follower = FollowerConsumer::<CascadeRoute>::new(
cascade_consumer,
topics.cascade.clone(),
+ claimed.stamp(),
dispatcher.clone(),
cascade_handle,
100,
@@ -555,7 +595,7 @@ async fn spawn_instance(
100,
RECV_TIMEOUT,
COMMIT_INTERVAL,
- ownership,
+ &claimed,
None,
)
.expect("assign cohort_stream_events");
diff --git a/rust/cohort-stream-processor/tests/events_consumer.rs b/rust/cohort-stream-processor/tests/events_consumer.rs
index 6f7c7f61a1ab..c53a72882198 100644
--- a/rust/cohort-stream-processor/tests/events_consumer.rs
+++ b/rust/cohort-stream-processor/tests/events_consumer.rs
@@ -43,8 +43,8 @@ use cohort_stream_processor::filters::{
CatalogHandle, CohortId, FilterCatalog, TeamFiltersBuilder, TeamId,
};
use cohort_stream_processor::partitions::{
- CohortConsumerContext, InputGroups, InputTopic, OffsetTracker, PartitionOwnership,
- PartitionRouter, ShuffleMessage, WorkerInbox,
+ AdmittedSplit, ClaimedSplit, CohortConsumerContext, InputGroups, InputTopic, OffsetTracker,
+ PartitionOwnership, PartitionRouter, ShuffleMessage, WorkerInbox,
};
use cohort_stream_processor::producer::{
CaptureSink, CohortMembershipChange, KafkaMembershipSink, MembershipSink, MembershipStatus,
@@ -240,7 +240,43 @@ async fn produce_new_persons(topic: &str, count: u128) -> usize {
/// The single pod's ownership of the test topics.
const SINGLE_POD: PartitionOwnership = PartitionOwnership::single(NUM_PARTITIONS as u16);
-fn build_consumer(
+/// Admits `ownership` through the ownership fence, as the processor does before it opens the
+/// store.
+async fn admit(ownership: PartitionOwnership, groups: &Arc<InputGroups>) -> AdmittedSplit {
+ let groups = groups.clone();
+ tokio::task::spawn_blocking(move || {
+ AdmittedSplit::try_admit(ownership, &groups).expect("the fence admits the split")
+ })
+ .await
+ .expect("the admission panicked")
+}
+
+/// The claim the processor makes once the store is open.
+async fn claim(admitted: AdmittedSplit, groups: &Arc<InputGroups>) -> ClaimedSplit {
+ let groups = groups.clone();
+ tokio::task::spawn_blocking(move || {
+ admitted.claim(&groups).expect("claim the owned partitions")
+ })
+ .await
+ .expect("the claim panicked")
+}
+
+/// Admits and claims `ownership` of `topic` for `group`. These tests read only the events topic,
+/// so the follower groups, which the fence claims too, read it as well.
+async fn claim_events(topic: &str, group: &str, ownership: PartitionOwnership) -> ClaimedSplit {
+ let mut config = Config::init_from_hashmap(&HashMap::new()).expect("default config");
+ config.kafka_hosts = bootstrap_servers();
+ config.cohort_stream_events_topic = topic.to_owned();
+ config.kafka_consumer_group = group.to_owned();
+ config.person_merge_events_topic = topic.to_owned();
+ config.kafka_merge_consumer_group = format!("{group}-merges");
+ config.cohort_merge_state_transfer_topic = topic.to_owned();
+ config.kafka_merge_apply_consumer_group = format!("{group}-transfers");
+ let groups = Arc::new(InputGroups::new(&config));
+ claim(admit(ownership, &groups).await, &groups).await
+}
+
+async fn build_consumer(
topic: &str,
group: &str,
store: CohortStore,
@@ -249,6 +285,7 @@ fn build_consumer(
sink: Arc<dyn MembershipSink>,
offset_commit_interval: Duration,
) -> CohortStreamEventsConsumer {
+ let claimed = claim_events(topic, group, SINGLE_POD).await;
let (consumer, _dispatcher) = build_consumer_with_restore(
topic,
group,
@@ -257,7 +294,7 @@ fn build_consumer(
handle,
sink,
offset_commit_interval,
- SINGLE_POD,
+ &claimed,
false,
100,
);
@@ -273,7 +310,7 @@ fn build_consumer_with_restore(
handle: lifecycle::Handle,
sink: Arc<dyn MembershipSink>,
offset_commit_interval: Duration,
- ownership: PartitionOwnership,
+ claimed: &ClaimedSplit,
durable_restore: bool,
recv_batch_size: usize,
) -> (CohortStreamEventsConsumer, Arc<EventDispatcher>) {
@@ -297,7 +334,7 @@ fn build_consumer_with_restore(
recv_batch_size,
Duration::from_millis(200),
offset_commit_interval,
- ownership,
+ claimed,
None,
)
.expect("assign the owned partitions");
@@ -456,7 +493,8 @@ async fn consumes_routes_and_commits_end_to_end() {
handle,
Arc::new(CaptureSink::new()),
Duration::from_millis(250),
- );
+ )
+ .await;
let task = tokio::spawn(consumer.process());
let deadline = Duration::from_secs(60);
@@ -540,7 +578,8 @@ async fn produces_membership_changes_and_commits_offsets() {
handle,
sink,
Duration::from_millis(250),
- );
+ )
+ .await;
let task = tokio::spawn(consumer.process());
let start = Instant::now();
@@ -670,7 +709,8 @@ async fn trickled_events_are_never_silently_dropped() {
handle,
Arc::new(CaptureSink::new()),
Duration::from_millis(300),
- );
+ )
+ .await;
let task = tokio::spawn(consumer.process());
let total = produce_events_trickle(&topic, N, Duration::from_millis(120)).await;
@@ -813,7 +853,8 @@ async fn does_not_commit_past_a_blocked_produce() {
handle,
sink.clone(),
Duration::from_millis(100),
- );
+ )
+ .await;
let task = tokio::spawn(consumer.process());
tokio::time::timeout(Duration::from_secs(30), entered.notified())
@@ -932,6 +973,7 @@ async fn two_pods_split_the_events_by_residue_class_and_commit_every_offset() {
..StoreConfig::default()
})
.expect("open store");
+ let claimed = claim_events(&topic, &group, ownership).await;
let (consumer, dispatcher) = build_consumer_with_restore(
&topic,
&group,
@@ -940,7 +982,7 @@ async fn two_pods_split_the_events_by_residue_class_and_commit_every_offset() {
handle.clone(),
Arc::new(CaptureSink::new()),
Duration::from_millis(250),
- ownership,
+ &claimed,
false,
100,
);
@@ -1065,6 +1107,7 @@ async fn durable_restart_reopens_live_state_and_fires_a_dormant_left() {
);
let shutdown_handle = handle.clone();
let _monitor = manager.monitor_background();
+ let claimed = claim_events(&topic, &group, SINGLE_POD).await;
let (consumer, _dispatcher) = build_consumer_with_restore(
&topic,
&group,
@@ -1073,7 +1116,7 @@ async fn durable_restart_reopens_live_state_and_fires_a_dormant_left() {
handle,
Arc::new(CaptureSink::new()),
Duration::from_millis(250),
- SINGLE_POD,
+ &claimed,
true,
100,
);
@@ -1122,6 +1165,7 @@ async fn durable_restart_reopens_live_state_and_fires_a_dormant_left() {
);
let shutdown_handle = handle.clone();
let _monitor = manager.monitor_background();
+ let claimed = claim_events(&topic, &group, SINGLE_POD).await;
let (consumer, _dispatcher) = build_consumer_with_restore(
&topic,
&group,
@@ -1130,7 +1174,7 @@ async fn durable_restart_reopens_live_state_and_fires_a_dormant_left() {
handle,
Arc::new(CaptureSink::new()),
Duration::from_millis(250),
- SINGLE_POD,
+ &claimed,
true,
10,
);
@@ -1273,6 +1317,7 @@ fn build_consumer_finishing_restore(
sink: Arc<dyn MembershipSink>,
offset_commit_interval: Duration,
restore: PendingRestore,
+ claimed: &ClaimedSplit,
) -> CohortStreamEventsConsumer {
let dispatcher = Arc::new(EventDispatcher::new(
PartitionRouter::new(64),
@@ -1292,7 +1337,7 @@ fn build_consumer_finishing_restore(
100,
Duration::from_millis(200),
offset_commit_interval,
- SINGLE_POD,
+ claimed,
Some(restore),
)
.expect("assign the owned partitions")
@@ -1402,8 +1447,14 @@ async fn s3_restore_replays_past_the_checkpoint_after_the_broker_moved_on() {
// --- Tenure 1: fold, checkpoint through the sweeper, then fold and commit past it. ---
{
- let store =
- CohortStore::open(&config.store_config(SINGLE_POD)).expect("open tenure-1 store");
+ let admitted = admit(
+ config.partition_ownership().expect("a standalone split"),
+ &groups,
+ )
+ .await;
+ let store = CohortStore::open(&config.store_config(admitted.ownership()))
+ .expect("open tenure-1 store");
+ let claimed = claim(admitted, &groups).await;
let mut manager = Manager::builder("s3restore-itest-1")
.with_trap_signals(false)
.build();
@@ -1421,7 +1472,7 @@ async fn s3_restore_replays_past_the_checkpoint_after_the_broker_moved_on() {
handle,
Arc::new(CaptureSink::new()),
Duration::from_millis(250),
- SINGLE_POD,
+ &claimed,
true,
100,
);
@@ -1492,7 +1543,12 @@ async fn s3_restore_replays_past_the_checkpoint_after_the_broker_moved_on() {
.expect("remove checkpoint_local_dir (simulate PVC loss)");
// --- Tenure 2: the boot restores from S3, then the consumer finishes the restore. ---
- let (store2, restore) = open_store(&config, SINGLE_POD, Some(lineage), groups.clone())
+ let admitted = admit(
+ config.partition_ownership().expect("a standalone split"),
+ &groups,
+ )
+ .await;
+ let (store2, restore) = open_store(&config, &admitted, Some(lineage), groups.clone())
.await
.expect("open the restored store");
let restore = restore
@@ -1541,6 +1597,7 @@ async fn s3_restore_replays_past_the_checkpoint_after_the_broker_moved_on() {
);
let shutdown_handle = handle.clone();
let _monitor = manager.monitor_background();
+ let claimed = claim(admitted, &groups).await;
let consumer = build_consumer_finishing_restore(
&topic,
&group,
@@ -1550,6 +1607,7 @@ async fn s3_restore_replays_past_the_checkpoint_after_the_broker_moved_on() {
Arc::new(CaptureSink::new()),
Duration::from_millis(250),
restore,
+ &claimed,
);
let task = tokio::spawn(consumer.process());
diff --git a/rust/cohort-stream-processor/tests/merge_consumer.rs b/rust/cohort-stream-processor/tests/merge_consumer.rs
index 0ce9e39a33fa..6f2368770d29 100644
--- a/rust/cohort-stream-processor/tests/merge_consumer.rs
+++ b/rust/cohort-stream-processor/tests/merge_consumer.rs
@@ -23,13 +23,14 @@
// Tests seed and assert through `CohortStore` directly — the sanctioned direct-store test surface.
#![allow(clippy::disallowed_methods)]
-use std::collections::HashSet;
+use std::collections::{HashMap, HashSet};
use std::future::Future;
use std::panic::AssertUnwindSafe;
use std::sync::Arc;
use std::time::{Duration, Instant};
use chrono_tz::UTC;
+use cohort_stream_processor::config::Config;
use cohort_stream_processor::consumers::{
CohortStreamEvent, CohortStreamEventsConsumer, EventDispatcher, FollowerConsumer, MergeRoute,
TransferRoute,
@@ -41,8 +42,9 @@ use cohort_stream_processor::merge::transfer::{
MergeStateTransfer, PersonMergeEvent, Tombstone, MERGE_EVENT_SCHEMA_VERSION,
};
use cohort_stream_processor::partitions::{
- merge_partition_key, partition_of, CohortConsumerContext, Follower, FollowerSet, OffsetTracker,
- PartitionOwnership, PartitionRouter, COHORT_PARTITION_COUNT,
+ merge_partition_key, partition_of, AdmittedSplit, ClaimedSplit, CohortConsumerContext,
+ Follower, FollowerSet, InputGroups, OffsetTracker, PartitionOwnership, PartitionRouter,
+ COHORT_PARTITION_COUNT,
};
use cohort_stream_processor::producer::{
CaptureCascadeSink, CaptureSink, CohortMembershipChange, KafkaMembershipSink,
@@ -59,6 +61,7 @@ use cohort_stream_processor::workers::{
CascadeConfig, MergeWorkerDeps, TransferRetryPolicy, DEFAULT_MERGE_GC_SCAN_LIMIT,
};
use common_kafka::config::KafkaConfig;
+use envconfig::Envconfig;
use futures::FutureExt;
use lifecycle::{ComponentOptions, Handle, Manager};
use rdkafka::admin::{AdminClient, AdminOptions, NewTopic, TopicReplication};
@@ -544,6 +547,33 @@ impl Groups {
transfers: format!("cohort-stream-merge-apply-{suffix}"),
}
}
+
+ /// The processor config that reads `topics` through these groups.
+ fn config(&self, topics: &Topics) -> Config {
+ let mut config = Config::init_from_hashmap(&HashMap::new()).expect("default config");
+ config.kafka_hosts = bootstrap_servers();
+ config.cohort_stream_events_topic = topics.events.clone();
+ config.kafka_consumer_group = self.events.clone();
+ config.person_merge_events_topic = topics.merges.clone();
+ config.kafka_merge_consumer_group = self.merges.clone();
+ config.cohort_merge_state_transfer_topic = topics.transfers.clone();
+ config.kafka_merge_apply_consumer_group = self.transfers.clone();
+ config
+ }
+}
+
+/// Admits and claims `ownership` through the ownership fence on the groups `config` names, as the
+/// processor does at boot.
+async fn claim(config: Config, ownership: PartitionOwnership) -> ClaimedSplit {
+ tokio::task::spawn_blocking(move || {
+ let groups = InputGroups::new(&config);
+ AdmittedSplit::try_admit(ownership, &groups)
+ .expect("the fence admits the split")
+ .claim(&groups)
+ .expect("claim the owned partitions")
+ })
+ .await
+ .expect("the claim panicked")
}
fn register_instance(manager: &mut Manager, name: &str) -> [Handle; 3] {
@@ -650,11 +680,12 @@ async fn spawn_instance(
.create()
.expect("create transfers follower consumer"),
);
+ let claimed = claim(groups.config(topics), ownership).await;
FollowerSet::new([
Follower::new(merges_consumer.clone(), topics.merges.clone()),
Follower::new(transfers_consumer.clone(), topics.transfers.clone()),
])
- .assign(&ownership)
+ .assign(&claimed)
.expect("assign the followers");
let events_client: StreamConsumer<CohortConsumerContext> = events_client_config(&groups.events)
@@ -666,6 +697,7 @@ async fn spawn_instance(
let merge_follower = FollowerConsumer::<MergeRoute>::new(
merges_consumer,
topics.merges.clone(),
+ claimed.stamp(),
dispatcher.clone(),
merge_handle,
100,
@@ -677,6 +709,7 @@ async fn spawn_instance(
let transfer_follower = FollowerConsumer::<TransferRoute>::new(
transfers_consumer,
topics.transfers.clone(),
+ claimed.stamp(),
dispatcher.clone(),
transfer_handle,
100,
@@ -693,7 +726,7 @@ async fn spawn_instance(
100,
RECV_TIMEOUT,
COMMIT_INTERVAL,
- ownership,
+ &claimed,
None,
)
.expect("assign cohort_stream_events");
diff --git a/rust/cohort-stream-processor/tests/merge_durability.rs b/rust/cohort-stream-processor/tests/merge_durability.rs
index 1f309991bbb0..16be7ec08b39 100644
--- a/rust/cohort-stream-processor/tests/merge_durability.rs
+++ b/rust/cohort-stream-processor/tests/merge_durability.rs
@@ -39,8 +39,9 @@ use cohort_stream_processor::merge::transfer::{
PersonMergeEvent, Tombstone, MERGE_EVENT_SCHEMA_VERSION,
};
use cohort_stream_processor::partitions::{
- merge_partition_key, partition_of, CohortConsumerContext, Follower, FollowerSet, InputGroups,
- InputTopic, OffsetTracker, PartitionOwnership, PartitionRouter, COHORT_PARTITION_COUNT,
+ merge_partition_key, partition_of, AdmittedSplit, ClaimedSplit, CohortConsumerContext,
+ Follower, FollowerSet, InputGroups, InputTopic, OffsetTracker, PartitionRouter,
+ COHORT_PARTITION_COUNT,
};
use cohort_stream_processor::producer::{
CohortMembershipChange, KafkaCascadeSink, KafkaMembershipSink, KafkaStreamEventSink,
@@ -632,6 +633,30 @@ impl Instance {
}
}
+/// The ownership fence's admission of the split `config` names, which the processor runs before it
+/// opens the store.
+async fn admit(config: &Config, groups: &Arc<InputGroups>) -> AdmittedSplit {
+ let ownership = config
+ .partition_ownership()
+ .expect("the pod name ends in an ordinal");
+ let groups = groups.clone();
+ tokio::task::spawn_blocking(move || {
+ AdmittedSplit::try_admit(ownership, &groups).expect("the fence admits the split")
+ })
+ .await
+ .expect("the admission panicked")
+}
+
+/// The claim the processor makes once the store is open.
+async fn claim(admitted: AdmittedSplit, groups: &Arc<InputGroups>) -> ClaimedSplit {
+ let groups = groups.clone();
+ tokio::task::spawn_blocking(move || {
+ admitted.claim(&groups).expect("claim the owned partitions")
+ })
+ .await
+ .expect("the claim panicked")
+}
+
/// An optional pending `restore` is finished by the events consumer's boot. Its follower positions
/// must already be committed, so each `incremental_assign(Offset::Stored)` resolves to them.
#[allow(clippy::too_many_arguments)]
@@ -643,6 +668,7 @@ async fn spawn_instance(
handles: [Handle; 4],
durable_restore: bool,
restore: Option<PendingRestore>,
+ claimed: &ClaimedSplit,
) -> Instance {
let [events_handle, merge_handle, transfer_handle, cascade_handle] = handles;
let kafka_config = producer_kafka_config();
@@ -730,13 +756,12 @@ async fn spawn_instance(
.expect("create cascade follower consumer"),
);
- let ownership = PartitionOwnership::single(NUM_PARTITIONS as u16);
FollowerSet::new([
Follower::new(merges_consumer.clone(), topics.merges.clone()),
Follower::new(transfers_consumer.clone(), topics.transfers.clone()),
Follower::new(cascade_consumer.clone(), topics.cascade.clone()),
])
- .assign(&ownership)
+ .assign(claimed)
.expect("assign the followers");
let events_client: StreamConsumer<CohortConsumerContext> = events_client_config(&groups.events)
@@ -748,6 +773,7 @@ async fn spawn_instance(
let merge_follower = FollowerConsumer::<MergeRoute>::new(
merges_consumer,
topics.merges.clone(),
+ claimed.stamp(),
dispatcher.clone(),
merge_handle,
100,
@@ -757,6 +783,7 @@ async fn spawn_instance(
let transfer_follower = FollowerConsumer::<TransferRoute>::new(
transfers_consumer,
topics.transfers.clone(),
+ claimed.stamp(),
dispatcher.clone(),
transfer_handle,
100,
@@ -766,6 +793,7 @@ async fn spawn_instance(
let cascade_follower = FollowerConsumer::<CascadeRoute>::new(
cascade_consumer,
topics.cascade.clone(),
+ claimed.stamp(),
dispatcher.clone(),
cascade_handle,
100,
@@ -791,7 +819,7 @@ async fn spawn_instance(
100,
RECV_TIMEOUT,
COMMIT_INTERVAL,
- ownership,
+ claimed,
restore,
)
.expect("assign cohort_stream_events");
@@ -983,9 +1011,7 @@ async fn merge_and_cascade_state_survive_an_s3_disaster_restore() {
let store_path = root.path().join("db");
let checkpoint_dir = root.path().join("checkpoints");
let config = merge_durability_config(&store_path, &checkpoint_dir, &prefix, &topics, &groups);
- let ownership = config
- .partition_ownership()
- .expect("the pod name ends in an ordinal");
+ let input_groups = Arc::new(InputGroups::new(&config));
let names = topics.names();
let config_for_cleanup = config.clone();
@@ -1003,7 +1029,10 @@ async fn merge_and_cascade_state_survive_an_s3_disaster_restore() {
let shutdown = handles[0].clone();
let _monitor = manager.monitor_background();
- let store1 = CohortStore::open(&config.store_config(ownership)).expect("open tenure-1 store");
+ let admitted = admit(&config, &input_groups).await;
+ let store1 = CohortStore::open(&config.store_config(admitted.ownership()))
+ .expect("open tenure-1 store");
+ let claimed = claim(admitted, &input_groups).await;
let instance = spawn_instance(
&topics,
&groups,
@@ -1012,6 +1041,7 @@ async fn merge_and_cascade_state_survive_an_s3_disaster_restore() {
handles,
config.durable_restore_enabled,
None,
+ &claimed,
)
.await;
wait_for(
@@ -1102,7 +1132,6 @@ async fn merge_and_cascade_state_survive_an_s3_disaster_restore() {
NUM_PARTITIONS as usize,
"the dispatcher must still own all 64 partitions at checkpoint time",
);
- let input_groups = Arc::new(InputGroups::new(&config));
checkpoint_to_s3(&instance, &config, &input_groups).await;
shutdown.request_shutdown();
@@ -1114,9 +1143,10 @@ async fn merge_and_cascade_state_survive_an_s3_disaster_restore() {
.expect("remove checkpoint_local_dir (simulate PVC loss)");
// Tenure 2: the boot restores from S3 and positions the follower groups; verify the state.
+ let admitted = admit(&config, &input_groups).await;
let (store2, restore) = open_store(
&config,
- ownership,
+ &admitted,
Some(config.checkpoint_lineage().expect("ordinal pod name")),
input_groups.clone(),
)
@@ -1129,10 +1159,10 @@ async fn merge_and_cascade_state_survive_an_s3_disaster_restore() {
// The followers resume past the merge on P_old's partition and the transfer on P_new's.
let merge_resume = restore
.plan()
- .positions(&InputTopic::new(topics.merges.as_str()), &ownership);
+ .positions(&InputTopic::new(topics.merges.as_str()), &admitted.ownership());
let transfer_resume = restore
.plan()
- .positions(&InputTopic::new(topics.transfers.as_str()), &ownership);
+ .positions(&InputTopic::new(topics.transfers.as_str()), &admitted.ownership());
assert!(
merge_resume.get(&part(p_old)).map(|offset| offset.get()) >= Some(1),
"the merge follower resumes past the merge on P_old's partition: {merge_resume:?}",
@@ -1182,6 +1212,7 @@ async fn merge_and_cascade_state_survive_an_s3_disaster_restore() {
let shutdown2 = handles2[0].clone();
let _monitor2 = manager2.monitor_background();
+ let claimed = claim(admitted, &input_groups).await;
let instance2 = spawn_instance(
&topics,
&groups,
@@ -1190,6 +1221,7 @@ async fn merge_and_cascade_state_survive_an_s3_disaster_restore() {
handles2,
config.durable_restore_enabled,
Some(restore),
+ &claimed,
)
.await;
wait_for(
diff --git a/rust/cohort-stream-processor/tests/ownership_fence.rs b/rust/cohort-stream-processor/tests/ownership_fence.rs
new file mode 100644
index 000000000000..a137c41415f6
--- /dev/null
+++ b/rust/cohort-stream-processor/tests/ownership_fence.rs
@@ -0,0 +1,258 @@
+//! The ownership fence against a **real** Kafka broker: commit stamps round-trip on every input
+//! group, and the run-time audit stops a pod when another split commits one of its partitions.
+//!
+//! `#[ignore]`d by default: the fence reads Kafka commit metadata, which the in-process
+//! `MockCluster` does not keep. Run against a local stack with:
+//!
+//! ```sh
+//! cargo test -p cohort-stream-processor --test ownership_fence -- --ignored
+//! ```
+
+// Writes commits as an older build, an offset reset or another split would, outside the stamp.
+#![allow(clippy::disallowed_methods)]
+
+use std::collections::{BTreeSet, HashMap};
+use std::num::NonZeroU16;
+use std::sync::Arc;
+use std::time::Duration;
+
+use cohort_stream_processor::config::Config;
+use cohort_stream_processor::partitions::{
+ AdmittedSplit, Committed, InputGroups, OwnershipAudit, PartitionOwnership, ResumeOffset, Stamp,
+};
+use cohort_stream_processor::store::durability::PodOrdinal;
+use cohort_stream_processor::sweep::Sweeper;
+use envconfig::Envconfig;
+use lifecycle::{ComponentOptions, LifecycleError, Manager};
+use rdkafka::admin::{AdminClient, AdminOptions, NewTopic, TopicReplication};
+use rdkafka::client::DefaultClientContext;
+use rdkafka::consumer::{BaseConsumer, CommitMode, Consumer};
+use rdkafka::{ClientConfig, Offset, TopicPartitionList};
+use uuid::Uuid;
+
+const PARTITIONS: u16 = 4;
+
+fn bootstrap_servers() -> String {
+ std::env::var("KAFKA_HOSTS").unwrap_or_else(|_| "localhost:9092".to_string())
+}
+
+fn split(pods: u16, ordinal: u16) -> PartitionOwnership {
+ PartitionOwnership::new(
+ NonZeroU16::new(pods).unwrap(),
+ PodOrdinal::from(ordinal),
+ PARTITIONS,
+ )
+ .unwrap()
+}
+
+/// The test's own events, merges and transfers topics, each read through its own group.
+struct Inputs {
+ events: String,
+ merges: String,
+ transfers: String,
+ group: String,
+}
+
+impl Inputs {
+ async fn create() -> Self {
+ let suffix = Uuid::new_v4();
+ let inputs = Self {
+ events: format!("cohort_stream_events_fence_{suffix}"),
+ merges: format!("person_merge_events_fence_{suffix}"),
+ transfers: format!("cohort_merge_state_transfer_fence_{suffix}"),
+ group: format!("cohort-stream-processor-fence-{suffix}"),
+ };
+ let admin = admin();
+ let topics: Vec<NewTopic<'_>> = inputs
+ .names()
+ .iter()
+ .map(|name| NewTopic::new(name, i32::from(PARTITIONS), TopicReplication::Fixed(1)))
+ .collect();
+ for result in admin
+ .create_topics(&topics, &admin_options())
+ .await
+ .expect("create_topics")
+ {
+ result.expect("create a test topic");
+ }
+ inputs
+ }
+
+ fn names(&self) -> [&str; 3] {
+ [&self.events, &self.merges, &self.transfers]
+ }
+
+ fn merges_group(&self) -> String {
+ format!("{}-merges", self.group)
+ }
+
+ fn groups(&self) -> Arc<InputGroups> {
+ let mut config = Config::init_from_hashmap(&HashMap::new()).expect("default config");
+ config.kafka_hosts = bootstrap_servers();
+ config.cohort_stream_events_topic = self.events.clone();
+ config.kafka_consumer_group = self.group.clone();
+ config.person_merge_events_topic = self.merges.clone();
+ config.kafka_merge_consumer_group = self.merges_group();
+ config.cohort_merge_state_transfer_topic = self.transfers.clone();
+ config.kafka_merge_apply_consumer_group = format!("{}-transfers", self.group);
+ Arc::new(InputGroups::new(&config))
+ }
+
+ async fn delete(&self) {
+ let _result = admin().delete_topics(&self.names(), &admin_options()).await;
+ }
+}
+
+fn admin() -> AdminClient<DefaultClientContext> {
+ ClientConfig::new()
+ .set("bootstrap.servers", bootstrap_servers())
+ .create()
+ .expect("create admin client")
+}
+
+fn admin_options() -> AdminOptions {
+ AdminOptions::new().operation_timeout(Some(Duration::from_secs(10)))
+}
+
+fn committer(group: &str) -> BaseConsumer {
+ ClientConfig::new()
+ .set("bootstrap.servers", bootstrap_servers())
+ .set("group.id", group)
+ .create()
+ .expect("create committer")
+}
+
+/// Commits `offset` on `partition` for `group` with `metadata`, as a build from before stamps
+/// (empty), an offset reset (empty) or another split would.
+fn commit_as(group: &str, topic: &str, partition: i32, offset: i64, metadata: &str) {
+ let mut list = TopicPartitionList::new();
+ let mut elem = list.add_partition(topic, partition);
+ elem.set_offset(Offset::Offset(offset)).unwrap();
+ elem.set_metadata(metadata);
+ committer(group)
+ .commit(&list, CommitMode::Sync)
+ .expect("commit");
+}
+
+fn every_partition() -> BTreeSet<u16> {
+ (0..PARTITIONS).collect()
+}
+
+/// The broker keeps a commit's metadata, on every group: the claim stamps each owned partition at
+/// its commit, or at the low watermark where the group never committed, and reads every stamp back.
+/// A commit from a build before stamps reads back unstamped and admits. A claim admitted before
+/// another split claimed refuses to commit over that split's stamps. If the broker stops keeping
+/// metadata, this test fails first.
+#[tokio::test]
+#[ignore = "requires a running Kafka broker (KAFKA_HOSTS); run with --ignored against a local stack"]
+async fn commit_stamps_round_trip_on_every_group() {
+ let inputs = Inputs::create().await;
+ let groups = inputs.groups();
+ commit_as(&inputs.group, &inputs.events, 1, 5, "");
+
+ let before = groups.ledger(&every_partition()).unwrap();
+ let events = before.inputs().next().unwrap();
+ assert_eq!(
+ events.commits()[&1],
+ Committed::At {
+ offset: ResumeOffset::try_from(5).unwrap(),
+ stamp: Stamp::Unstamped,
+ },
+ );
+ assert_eq!(events.commits()[&0], Committed::Never);
+
+ let admitted =
+ AdmittedSplit::try_admit(split(1, 0), &groups).expect("a commit without a stamp admits");
+ let stale = AdmittedSplit::try_admit(split(2, 0), &groups).expect("nothing is stamped yet");
+ admitted
+ .claim(&groups)
+ .expect("the claim reads its stamps back");
+ assert!(
+ stale.claim(&groups).is_err(),
+ "a stale admission must not claim over another split's stamps",
+ );
+
+ let after = groups.ledger(&every_partition()).unwrap();
+ let ours = Stamp::Split(split(1, 0));
+ for input in after.inputs() {
+ for (&partition, committed) in input.commits() {
+ let resumes = if input.topic().as_str() == inputs.events && partition == 1 {
+ 5
+ } else {
+ 0
+ };
+ assert_eq!(
+ *committed,
+ Committed::At {
+ offset: ResumeOffset::try_from(resumes).unwrap(),
+ stamp: ours.clone(),
+ },
+ "{} partition {partition}",
+ input.topic(),
+ );
+ }
+ }
+ assert_eq!(after.inputs().count(), 3, "the claim stamps every group");
+
+ inputs.delete().await;
+}
+
+/// The audit lets a commit without a stamp pass, as an operator's offset reset writes one, and stops
+/// the pod when another split commits one of its partitions. It also revokes the pod's commits, so
+/// the final commits of the stop cannot cover the foreign stamp, and the next boot holds.
+#[tokio::test]
+#[ignore = "requires a running Kafka broker (KAFKA_HOSTS); run with --ignored against a local stack"]
+async fn the_audit_stops_a_pod_whose_partition_another_split_commits() {
+ let inputs = Inputs::create().await;
+ let groups = inputs.groups();
+ let claimed = AdmittedSplit::try_admit(split(1, 0), &groups)
+ .unwrap()
+ .claim(&groups)
+ .unwrap();
+
+ let audit_once = |name: &str| {
+ let mut manager = Manager::builder(name).with_trap_signals(false).build();
+ let handle = manager.register(
+ "consumer",
+ ComponentOptions::new().with_graceful_shutdown(Duration::from_secs(5)),
+ );
+ let monitor = manager.monitor_background();
+ let audit = OwnershipAudit::new(groups.clone(), claimed.clone(), handle.clone());
+ async move {
+ audit.run_once().await;
+ handle.work_completed();
+ handle.request_shutdown();
+ monitor.wait().await
+ }
+ };
+
+ commit_as(&inputs.group, &inputs.events, 0, 3, "");
+ audit_once("audit-reset")
+ .await
+ .expect("an unstamped commit passes");
+
+ commit_as(
+ &inputs.merges_group(),
+ &inputs.merges,
+ 3,
+ 0,
+ "cohort-split/1:2:1:4",
+ );
+ let stopped = audit_once("audit-foreign").await;
+ assert!(
+ matches!(stopped, Err(LifecycleError::ComponentFailure { ref reason, .. }) if reason.contains("another split")),
+ "{stopped:?}",
+ );
+ let final_commit = claimed
+ .stamp()
+ .list(&inputs.merges, [(3, 1)])
+ .unwrap()
+ .commit(&committer(&inputs.merges_group()), CommitMode::Sync);
+ assert!(final_commit.is_err(), "the audit revokes the pod's commits");
+ assert!(
+ AdmittedSplit::try_admit(split(1, 0), &groups).is_err(),
+ "the next boot holds in the fence",
+ );
+
+ inputs.delete().await;
+}
diff --git a/rust/cohort-stream-processor/tests/seed_consumer.rs b/rust/cohort-stream-processor/tests/seed_consumer.rs
index 99947e92282a..e9feb66f115b 100644
--- a/rust/cohort-stream-processor/tests/seed_consumer.rs
+++ b/rust/cohort-stream-processor/tests/seed_consumer.rs
@@ -26,6 +26,7 @@ use cohort_core::seed::{
BehavioralShapeHash, ClaimEpoch, ConditionHash, CoverageStartMs, ReconcileMarker,
ReconcileScope, ReconcileTile, RunBoundaryMs, RunId, SChunkMs, SeedTile, WithheldReason,
};
+use cohort_stream_processor::config::Config;
use cohort_stream_processor::consumers::{
CascadeRoute, CohortStreamEventsConsumer, EventDispatcher, FollowerConsumer, MergeRoute,
SeedFollowerConsumer, TransferRoute,
@@ -35,9 +36,10 @@ use cohort_stream_processor::filters::{
};
use cohort_stream_processor::observability::disk::SharedDiskUtilization;
use cohort_stream_processor::partitions::{
- merge_partition_key, partition_for, partition_of, CohortConsumerContext, ConsumerPauser,
- Follower, FollowerSet, LiveWatermarks, OffsetTracker, PartitionOwnership, PartitionPauser,
- PartitionRouter, SeedPacingConfig, COHORT_PARTITION_COUNT,
+ merge_partition_key, partition_for, partition_of, AdmittedSplit, ClaimedSplit,
+ CohortConsumerContext, ConsumerPauser, Follower, FollowerSet, InputGroups, LiveWatermarks,
+ OffsetTracker, PartitionOwnership, PartitionPauser, PartitionRouter, SeedPacingConfig,
+ COHORT_PARTITION_COUNT,
};
use cohort_stream_processor::producer::{
CascadeSink, ChangeOrigin, CohortMembershipChange, KafkaCascadeSink, KafkaMembershipSink,
@@ -57,6 +59,7 @@ use cohort_stream_processor::workers::{
DEFAULT_MERGE_GC_SCAN_LIMIT,
};
use common_kafka::config::KafkaConfig;
+use envconfig::Envconfig;
use futures::FutureExt;
use lifecycle::{ComponentOptions, Handle, Manager};
use rdkafka::admin::{AdminClient, AdminOptions, NewTopic, TopicReplication};
@@ -527,6 +530,39 @@ impl Groups {
seeds: format!("cohort-stream-seeds-{suffix}"),
}
}
+
+ /// The processor config that reads `topics` through these groups.
+ fn config(&self, topics: &Topics) -> Config {
+ let mut config = Config::init_from_hashmap(&HashMap::new()).expect("default config");
+ config.kafka_hosts = bootstrap_servers();
+ config.cohort_stream_events_topic = topics.events.clone();
+ config.kafka_consumer_group = self.events.clone();
+ config.person_merge_events_topic = topics.merges.clone();
+ config.kafka_merge_consumer_group = self.merges.clone();
+ config.cohort_merge_state_transfer_topic = topics.transfers.clone();
+ config.kafka_merge_apply_consumer_group = self.transfers.clone();
+ config.cohort_cascade_enabled = true;
+ config.cohort_cascade_events_topic = topics.cascade.clone();
+ config.kafka_cascade_consumer_group = self.cascade.clone();
+ config.cohort_seed_consumer_enabled = true;
+ config.cohort_stream_seed_events_topic = topics.seeds.clone();
+ config.kafka_seed_consumer_group = self.seeds.clone();
+ config
+ }
+}
+
+/// Admits and claims `ownership` through the ownership fence on the groups `config` names, as the
+/// processor does at boot.
+async fn claim(config: Config, ownership: PartitionOwnership) -> ClaimedSplit {
+ tokio::task::spawn_blocking(move || {
+ let groups = InputGroups::new(&config);
+ AdmittedSplit::try_admit(ownership, &groups)
+ .expect("the fence admits the split")
+ .claim(&groups)
+ .expect("claim the owned partitions")
+ })
+ .await
+ .expect("the claim panicked")
}
fn register_instance(manager: &mut Manager) -> [Handle; 5] {
@@ -739,13 +775,14 @@ async fn spawn_instance(
.create()
.expect("create seed follower consumer"),
);
+ let claimed = claim(groups.config(topics), ownership).await;
FollowerSet::new([
Follower::new(merges_consumer.clone(), topics.merges.clone()),
Follower::new(transfers_consumer.clone(), topics.transfers.clone()),
Follower::new(cascade_consumer.clone(), topics.cascade.clone()),
Follower::new(seeds_consumer.clone(), topics.seeds.clone()),
])
- .assign(&ownership)
+ .assign(&claimed)
.expect("assign the followers");
let follower_assignments = [
assigned(&merges_consumer),
@@ -764,6 +801,7 @@ async fn spawn_instance(
let merge_follower = FollowerConsumer::<MergeRoute>::new(
merges_consumer,
topics.merges.clone(),
+ claimed.stamp(),
dispatcher.clone(),
merge_handle,
100,
@@ -775,6 +813,7 @@ async fn spawn_instance(
let transfer_follower = FollowerConsumer::<TransferRoute>::new(
transfers_consumer,
topics.transfers.clone(),
+ claimed.stamp(),
dispatcher.clone(),
transfer_handle,
100,
@@ -786,6 +825,7 @@ async fn spawn_instance(
let cascade_follower = FollowerConsumer::<CascadeRoute>::new(
cascade_consumer,
topics.cascade.clone(),
+ claimed.stamp(),
dispatcher.clone(),
cascade_handle,
100,
@@ -801,6 +841,7 @@ async fn spawn_instance(
let seed_follower = SeedFollowerConsumer::new(
seeds_consumer.clone(),
topics.seeds.clone(),
+ claimed.stamp(),
events_client.clone(),
topics.events.clone(),
dispatcher.clone(),
@@ -826,7 +867,7 @@ async fn spawn_instance(
100,
RECV_TIMEOUT,
COMMIT_INTERVAL,
- ownership,
+ &claimed,
None,
)
.expect("assign cohort_stream_events");
@@ -1309,10 +1350,11 @@ async fn reconcile_snapshot_repairs_stale_state_and_commits_after_markers() {
|| instance.reconcile_backlog.len() == i64::from(NUM_PARTITIONS),
)
.await;
+ // The claim committed every partition at its start, offset zero.
let before_first_tick = seed_group_committed_offsets(&groups.seeds, &topics.seeds);
assert!(
- before_first_tick.is_empty(),
- "no reconcile control commits before draining starts",
+ (0..NUM_PARTITIONS).all(|partition| before_first_tick.get(&partition) == Some(&0)),
+ "no reconcile control commits before draining starts: {before_first_tick:?}",
);
assert!((0..NUM_PARTITIONS)
.all(|partition| { instance.seed_committable(partition as u16).is_none() }));
@@ -1629,8 +1671,8 @@ async fn fence_holds_a_fresh_tile_until_live_consumption_passes_its_scan_point()
);
assert_eq!(
seed_group_committed(&groups.seeds, &topics.seeds, part(alice) as i32),
- None,
- "a held tile's offset must never commit",
+ Some(0),
+ "a held tile's offset must never commit: the commit stays where the claim put it",
);
assert!(
instance.reconcile_backlog.is_empty(),