Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(cohorts): reshard the stream processor by doubling the pod count (#118893)

Matheus Batista44e1a0888afe
diff --git a/docs/internal/realtime-cohorts/README.md b/docs/internal/realtime-cohorts/README.md
index d612e207262a..812dd7c7ec73 100644
--- a/docs/internal/realtime-cohorts/README.md
+++ b/docs/internal/realtime-cohorts/README.md
@@ -90,8 +90,9 @@ When all partitions have reported, Django stamps the cohort ready, and feature f
Reconcile re-emits from the state the processor holds, so it repairs lost output, not lost state.
[Processor runtime](processor-runtime.md#delivery-semantics) lists each path's guarantee.
- **The processor runs as a single pod.**
- Each pod owns a fixed share of the partitions, and keeps their state in a local RocksDB store that never moves between pods.
- A second replica refuses to start unless `COHORT_POD_COUNT` is raised, every store is wiped and the old split's stamps on the broker are cleared, and then every slice starts without its history.
+ Each pod owns a fixed share of the partitions, and keeps their state in a local RocksDB store.
+ More pods need a reshard, as [ownership](processor-runtime.md#ownership-and-the-single-pod-deployment) describes.
+ With checkpoints enabled, a reshard keeps the history of every slice.
- **Most of the pipeline is off by default.**
The processor writes to a shadow output topic unless configured otherwise, and seed apply, reconcile, cascades, durable restore, the seeder's completion driver and Django's finalizer each have their own switch.
The pages name the switch where each behavior is described.
diff --git a/docs/internal/realtime-cohorts/processor-runtime.md b/docs/internal/realtime-cohorts/processor-runtime.md
index 0f19693ff0c3..fd23e6ea8478 100644
--- a/docs/internal/realtime-cohorts/processor-runtime.md
+++ b/docs/internal/realtime-cohorts/processor-runtime.md
@@ -181,21 +181,22 @@ A wiped store does the same.
## Ownership and the single-pod deployment
-State never moves between pods.
-A pod's store holds the slices of the partitions it owns, and nothing hands a slice to another pod.
+A pod's store holds the slices of the partitions it owns, and only a reshard hands a slice to another pod.
The store records the split it was written under in `cf_meta[pod_partitions]`: the pod count, the ordinal and the partition count.
A store from before that record takes the configured split as its own the first time it opens.
A split covers another when it owns every partition the other owns.
-A store written under a split that covers the pod's opens and records the pod's split.
-A store written under any other split refuses to open, so the pod crash-loops instead of reading slices that another pod may have changed since.
+A store opens when its recorded split covers the pod's split, and then records the pod's split.
+Any other store refuses to open, so the pod crash-loops instead of reading slices that another pod may have changed since.
With durable restore enabled, boot deletes the slices of partitions the pod does not own, so a single pod deletes nothing.
The store record cannot stop a pod on an empty volume, so the **ownership fence** keeps a second record on the broker.
-Every offset commit on every input group carries the committer's split as Kafka commit metadata, for example `cohort-split/1:2:0:64` for pod 0 of 2 over 64 partitions.
+Every offset commit on every input group carries the committer's split as Kafka commit metadata, its stamp, for example `cohort-split/1:2:0:64` for pod 0 of 2 over 64 partitions.
The pod reads these stamps three times:
1. Before the store opens, it reads the last commit of every group on each partition it owns.
- A stamp from another split, or metadata it cannot read, holds the boot.
+ A stamp from another split, or metadata it cannot read, holds the boot, because the pods of that split may still run.
+ During [a reshard](#raising-the-pod-count), the parent's stamp passes too.
+ On a follower group, the stamp of a split that covers the pod's also passes, because a pod of that split that still runs also commits the events group.
The pod retries after 30 seconds, doubling the wait up to 5 minutes, and meanwhile `/_ready` reads not ready and `ownership_fence_blocked{reason}` is 1.
A commit without a stamp passes, because older builds and standard Kafka tools, such as an offset reset, write none.
A checkpoint restore waits for the same check again before it commits the follower positions.
@@ -210,14 +211,42 @@ Two processes of the same split, such as two pods with the same ordinal, write t
Builds from before stamps commit without one, so the fence cannot see them either.
Change `COHORT_POD_COUNT` only after every pod runs a build with stamps.
-So changing `COHORT_POD_COUNT` requires wiping every store, including the volume of an ordinal that a scale-down removed.
+### Raising the pod count
+
+`COHORT_RESHARD_FROM_POD_COUNT` marks a reshard and names the old pod count.
+`COHORT_POD_COUNT` must be a larger whole multiple of it, or the pod refuses to start.
+With durable restore enabled, the pod also refuses to start without checkpoints, because each parent deletes the slices it gives away and its child would start without them.
+
+- A pod whose ordinal is below the old count is a parent.
+ It reopens its store, and boot deletes the slices it gave away.
+- Every other pod is a child, and takes its partitions over from the parent at its ordinal modulo the old count.
+ It waits, with the `parent_running` reason, until the parent has stamped a partition under the new split.
+ It then deletes a live store that holds no pending restore, and restores the newest checkpoint in the parent's S3 lineage that covers it, counted in `checkpoint_restore_total{source="parent_s3"}`.
+ With no usable checkpoint, it creates its store behind the coverage fence.
+- Once one of a child's events partitions carries the child's stamp, the takeover is over, and later boots open its store as usual.
+
+To reshard:
+
+1. Scale the processor to 0, and wait until every pod has stopped after its final checkpoint.
+2. Set `COHORT_POD_COUNT` to the new count, `COHORT_RESHARD_FROM_POD_COUNT` to the old count, and the replicas to the new count.
+3. Check `owned_partitions` on each pod, `ownership_fence_blocked` at 0, and one `parent_s3` restore for each child.
+4. Remove `COHORT_RESHARD_FROM_POD_COUNT` in a later change, and before any offset reset of the events group.
+ While it is set, a reset that clears the stamp from every events partition of a child makes the child take its partitions over again and delete its store.
+
+If step 1 is skipped, each child waits until its parent restarts, and the old split keeps every partition until then.
+Deleting the parent pods moves the reshard on.
+After its takeover, a child searches only its own lineage.
+So a child that loses its volume before its first upload finishes creates its store behind the coverage fence, even though its parent's checkpoint still covers it.
+
+### Lowering the pod count
+
+Nothing merges two stores, so lowering `COHORT_POD_COUNT` requires wiping every store, including the volume of an ordinal that a scale-down removed.
That volume keeps its old record, so when a later change brings back that split or one it covers, it opens with slices another pod has moved past.
-With checkpoints enabled, keep the checkpoints: a restore passes over one whose split does not cover the pod.
-The old split's stamps also hold every pod of the new split at boot.
-An offset reset of each input group to its current offsets clears them, because it commits without a stamp.
-Run it only after every pod of the old split has stopped, or the fence admits the new split while the old one still commits.
-Each slice without a covering checkpoint then begins again behind the coverage fence and holds no history until a backfill runs.
-Nothing moves a slice between pods yet, so one pod that owns every partition is the supported deployment.
+With checkpoints enabled, also delete every pod's checkpoints, local and in S3.
+While no pod runs, reset every group's offsets to their current positions, because the stamps of the larger split hold every boot and a standard offset reset writes no stamp.
+Each slice then begins again behind the coverage fence and holds no history until a backfill runs.
+
+More than one pod needs chart changes that are not done yet, so one pod that owns every partition is the supported deployment.
## Startup
diff --git a/docs/internal/realtime-cohorts/state-store-and-durability.md b/docs/internal/realtime-cohorts/state-store-and-durability.md
index a558e54c952e..52ef9fa82f2a 100644
--- a/docs/internal/realtime-cohorts/state-store-and-durability.md
+++ b/docs/internal/realtime-cohorts/state-store-and-durability.md
@@ -142,9 +142,9 @@ Nothing is dispatched to a worker, from the events consumer or a follower, until
[Processor runtime](processor-runtime.md#startup) lists them in order.
A pod owns its partitions for as long as it runs, so no slice changes owner while the pod runs.
-State never moves between pods, which is why one pod that owns every partition is the supported deployment.
-The store records the split of the partitions it was written under, and refuses to open under a split that the recorded one does not cover.
-[Processor runtime](processor-runtime.md#ownership-and-the-single-pod-deployment) explains that constraint.
+State moves between pods only in a reshard, which raises the pod count.
+The store records the split of the partitions it was written under, and opens only under a split that the recorded one covers.
+[Processor runtime](processor-runtime.md#ownership-and-the-single-pod-deployment) explains that constraint and the reshard.
### Choosing a checkpoint
@@ -241,7 +241,9 @@ Checkpoints require durable restore, a bucket, and an absolute checkpoint direct
- **One lineage per pod.**
A pod's checkpoints live under `cohort_stream_state/<ordinal>`, locally and in object storage, where the ordinal is the number that ends the StatefulSet pod name in `POD_NAME`.
Ordinal 0 keeps the single pod's original layout.
- Two pods never read or write each other's checkpoints.
+ A pod writes only its own lineage.
+ A reshard child reads its parent's lineage only to take its partitions over.
+ Its first upload is full, so its lineage never names objects in its parent's lineage.
- **Positions from the broker.**
Each checkpoint carries `offsets.json`: for every owned slice, the committed offset of every enabled input's consumer group, or the low watermark where the group has none, so idle partitions are recorded too.
The positions are read after a write-ahead-log flush and before the checkpoint.
@@ -355,7 +357,8 @@ A lint rule denies direct calls to most store I/O methods, and the synchronous s
With checkpoints enabled, the boot deletes it and restores.
- A store directory that holds `restore.json` is a checkpoint restore in progress, not a live store.
An image from before that marker ignores it and reopens the store at the broker's old offsets, so do not roll back while a restore is pending.
-- Changing `COHORT_POD_COUNT` makes every store whose split does not cover its pod's new one refuse to open, because each records the split it was written under.
+- Changing `COHORT_POD_COUNT` without `COHORT_RESHARD_FROM_POD_COUNT` holds every boot in the ownership fence.
+ Lowering it also makes every store refuse to open, because each records the split it was written under.
[Ownership](processor-runtime.md#ownership-and-the-single-pod-deployment) lists what to wipe.
Wiping a store deletes every slice, including merge transfers not yet sent, whose merge offsets may already be committed.
- Person records grow with every person seen on a team with person-property conditions, including those that match nothing, unless a time-to-live is configured.
diff --git a/rust/cohort-stream-processor/src/config.rs b/rust/cohort-stream-processor/src/config.rs
index 2c4d836d20a8..bed6e0db83a7 100644
--- a/rust/cohort-stream-processor/src/config.rs
+++ b/rust/cohort-stream-processor/src/config.rs
@@ -13,7 +13,7 @@ use rdkafka::ClientConfig;
use tracing::warn;
use crate::partitions::pacing::{AgeMs, Hysteresis, SeedPacingConfig, UsedPct};
-use crate::partitions::PartitionOwnership;
+use crate::partitions::{PartitionOwnership, SplitClaim};
use crate::store::durability::stage::staging_path;
use crate::store::durability::{CheckpointLineage, DurabilityConfig, NotAnOrdinalPod, PodOrdinal};
use crate::store::{OffloadConfig, OffloadMode, StoreConfig};
@@ -332,6 +332,12 @@ pub struct Config {
#[envconfig(from = "COHORT_POD_COUNT", default = "1")]
pub cohort_pod_count: NonZeroU16,
+ /// The pod count before a reshard, which must divide `COHORT_POD_COUNT`. While it is set, each
+ /// pod whose ordinal is below it keeps its store, and every other pod takes its partitions over
+ /// from the pod that owned them. Remove it once the reshard is done.
+ #[envconfig(from = "COHORT_RESHARD_FROM_POD_COUNT")]
+ pub cohort_reshard_from_pod_count: Option<NonZeroU16>,
+
#[envconfig(from = "HOSTNAME")]
pub pod_hostname: Option<String>,
@@ -1043,10 +1049,28 @@ impl Config {
))
}
- /// The partitions this pod owns: pod `ordinal` of `COHORT_POD_COUNT`, over
- /// `COHORT_PARTITION_COUNT` partitions. Only a single pod may run without `POD_NAME`: every
- /// pod without it would take ordinal 0, and they would all own the same partitions.
- pub fn partition_ownership(&self) -> anyhow::Result<PartitionOwnership> {
+ /// This pod's split and its place in a reshard: pod `ordinal` of `COHORT_POD_COUNT`, over
+ /// `COHORT_PARTITION_COUNT` partitions, resharded from `COHORT_RESHARD_FROM_POD_COUNT` pods
+ /// when that is set.
+ pub fn split_claim(&self) -> anyhow::Result<SplitClaim> {
+ ensure!(
+ self.cohort_reshard_from_pod_count.is_none()
+ || self.checkpoint_enabled
+ || !self.durable_restore_enabled,
+ "COHORT_RESHARD_FROM_POD_COUNT with DURABLE_RESTORE_ENABLED requires CHECKPOINT_ENABLED: \
+ without checkpoints each child starts empty while its parent deletes the partitions it \
+ gave away",
+ );
+ SplitClaim::new(
+ self.partition_ownership()?,
+ self.cohort_reshard_from_pod_count,
+ )
+ .context("COHORT_RESHARD_FROM_POD_COUNT must divide COHORT_POD_COUNT and be below it")
+ }
+
+ /// The partitions this pod owns. Only a single pod may run without `POD_NAME`: every pod
+ /// without it would take ordinal 0, and they would all own the same partitions.
+ fn partition_ownership(&self) -> anyhow::Result<PartitionOwnership> {
let ordinal = match self
.pod_ordinal()
.context("POD_NAME must end in a StatefulSet ordinal, or be unset outside Kubernetes")?
@@ -1286,6 +1310,7 @@ mod tests {
kafka_cascade_consumer_group: "cohort-stream-cascade".to_string(),
pod_name: None,
cohort_pod_count: NonZeroU16::MIN,
+ cohort_reshard_from_pod_count: None,
pod_hostname: None,
cohort_membership_changed_topic: "cohort_membership_changed_shadow".to_string(),
cohort_reconcile_markers_topic: "cohort_reconcile_markers".to_string(),
@@ -1669,9 +1694,9 @@ mod tests {
}
#[test]
- fn partition_ownership_comes_from_the_pod_count_and_the_pod_name_ordinal() {
+ fn the_split_claim_comes_from_the_pod_count_the_pod_name_ordinal_and_the_reshard_marker() {
type Env = &'static [(&'static str, &'static str)];
- let cases: [(Env, Result<&str, &str>); 6] = [
+ let cases: [(Env, Result<&str, &str>); 9] = [
(&[], Ok("pod 0 of 1 over 64 partitions")),
(
&[
@@ -1696,19 +1721,44 @@ mod tests {
Err("not below"),
),
(&[("POD_NAME", "cohort-stream-processor")], Err("POD_NAME")),
+ (
+ &[
+ ("COHORT_POD_COUNT", "2"),
+ ("COHORT_RESHARD_FROM_POD_COUNT", "1"),
+ ("POD_NAME", "cohort-stream-processor-1"),
+ ],
+ Ok("pod 1 of 2 over 64 partitions, resharded from pod 0 of 1 over 64 partitions"),
+ ),
+ (
+ &[
+ ("COHORT_POD_COUNT", "2"),
+ ("COHORT_RESHARD_FROM_POD_COUNT", "1"),
+ ("DURABLE_RESTORE_ENABLED", "true"),
+ ("POD_NAME", "cohort-stream-processor-1"),
+ ],
+ Err("requires CHECKPOINT_ENABLED"),
+ ),
+ (
+ &[
+ ("COHORT_POD_COUNT", "2"),
+ ("COHORT_RESHARD_FROM_POD_COUNT", "4"),
+ ("POD_NAME", "cohort-stream-processor-1"),
+ ],
+ Err("must raise the pod count"),
+ ),
];
for (env, expected) in cases {
let env = env
.iter()
.map(|&(key, value)| (key.to_string(), value.to_string()))
.collect();
- let ownership = Config::init_from_hashmap(&env)
+ let claim = Config::init_from_hashmap(&env)
.unwrap()
- .partition_ownership()
- .map(|ownership| ownership.to_string())
+ .split_claim()
+ .map(|claim| claim.to_string())
.map_err(|err| format!("{err:#}"));
- match (ownership, expected) {
- (Ok(ownership), Ok(expected)) => assert_eq!(ownership, expected, "{env:?}"),
+ match (claim, expected) {
+ (Ok(claim), Ok(expected)) => assert_eq!(claim, expected, "{env:?}"),
(Err(err), Err(expected)) => assert!(err.contains(expected), "{env:?}: {err}"),
(actual, expected) => panic!("{env:?}: got {actual:?}, expected {expected:?}"),
}
diff --git a/rust/cohort-stream-processor/src/consumers/events.rs b/rust/cohort-stream-processor/src/consumers/events.rs
index bac733757bda..548e91cef338 100644
--- a/rust/cohort-stream-processor/src/consumers/events.rs
+++ b/rust/cohort-stream-processor/src/consumers/events.rs
@@ -13,6 +13,7 @@ use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, OnceLock};
use std::time::{Duration, Instant};
+use chrono::Utc;
use cohort_core::seed::CoverageStartMs;
use dashmap::mapref::entry::Entry;
use dashmap::DashMap;
@@ -971,6 +972,7 @@ impl CohortStreamEventsConsumer {
}
(_, Some(restore)) => {
self.recover().await;
+ self.wait_for_capture(&restore).await;
let mut resume = ResumePoints::default();
let topic = InputTopic::new(self.topic.as_str());
resume.rewind_to(&restore.plan().positions(&topic, &self.claimed.ownership()));
@@ -1055,6 +1057,27 @@ impl CohortStreamEventsConsumer {
.await;
}
+ /// Nothing polls or folds while this host's clock reads before the checkpoint's capture, so no
+ /// fold emits a version below the outputs that made the restored state. The source pod can emit
+ /// more after the capture, and only the wall clocks of the two hosts order those outputs before
+ /// this pod's. A shutdown ends the wait.
+ async fn wait_for_capture(&self, restore: &PendingRestore) {
+ let captured_at = restore.captured_at();
+ while let Ok(wait) = (captured_at - Utc::now()).to_std() {
+ warn!(
+ %captured_at,
+ wait_secs = wait.as_secs(),
+ "this host's clock reads before the restored checkpoint's capture; holding off \
+ dispatch until it passes",
+ );
+ tokio::select! {
+ biased;
+ _ = self.handle.shutdown_recv() => return,
+ _ = tokio::time::sleep(wait) => {}
+ }
+ }
+ }
+
/// Seek every owned partition back to its resume point, finish the restore, then go live. Any
/// failure keeps boot in `Rewinding`, so the next poll retries and nothing folds from the
/// broker offset. `Rewinding` reports no liveness, so a persistent failure restarts the process.
diff --git a/rust/cohort-stream-processor/src/main.rs b/rust/cohort-stream-processor/src/main.rs
index 45e266151ead..38bf909fdcde 100644
--- a/rust/cohort-stream-processor/src/main.rs
+++ b/rust/cohort-stream-processor/src/main.rs
@@ -73,7 +73,7 @@ async fn async_main(config: Config) -> Result<()> {
log_startup(&config);
config.validate_startup()?;
- let ownership = config.partition_ownership()?;
+ let claim = config.split_claim()?;
let lineage = config
.checkpoint_enabled
@@ -321,7 +321,7 @@ async fn async_main(config: Config) -> Result<()> {
)
.context("checking the checkpoint directory")?;
}
- let admitted = AdmittedSplit::admit(ownership, groups.clone()).await;
+ let admitted = AdmittedSplit::admit(claim, groups.clone()).await;
let (store, restore) = open_store(&config, &admitted, lineage, groups.clone())
.await
.context("opening RocksDB state store")?;
@@ -765,6 +765,7 @@ fn log_startup(config: &Config) {
merge_apply_consumer_group = %config.kafka_merge_apply_consumer_group,
pod_identity = config.pod_identity().unwrap_or("<none>"),
pod_count = config.cohort_pod_count,
+ reshard_from_pod_count = ?config.cohort_reshard_from_pod_count,
recv_batch_size = config.recv_batch_size,
partition_channel_buffer = config.partition_channel_buffer,
store_path = %config.store_path,
diff --git a/rust/cohort-stream-processor/src/observability/metrics.rs b/rust/cohort-stream-processor/src/observability/metrics.rs
index a15ad35ef605..6da376469a19 100644
--- a/rust/cohort-stream-processor/src/observability/metrics.rs
+++ b/rust/cohort-stream-processor/src/observability/metrics.rs
@@ -139,7 +139,8 @@ pub const CHECKPOINT_FILE_FETCH_STORE_DURATION_SECONDS: &str =
"checkpoint_file_fetch_store_duration_seconds";
/// Latency of listing a checkpoint's recent attempt folders from S3 (histogram, seconds).
pub const CHECKPOINT_LIST_DURATION_SECONDS: &str = "checkpoint_list_duration_seconds";
-/// Boot store choices, labelled by `source` (`reopen_live`|`pending`|`local`|`s3`|`cold`) (counter).
+/// Boot store choices, labelled by `source` (`reopen_live`|`pending`|`local`|`s3`|`parent_s3`|`cold`)
+/// (counter). `parent_s3` is a reshard child restoring its parent's lineage.
pub const CHECKPOINT_RESTORE_TOTAL: &str = "checkpoint_restore_total";
/// Checkpoint candidates a boot restore tried, labelled by `verdict`
/// (`published`|`unusable`|`failed`) (counter).
@@ -307,8 +308,9 @@ pub const PARTITIONS_ASSIGNED_TOTAL: &str = "partitions_assigned_total";
pub const OWNED_PARTITIONS: &str = "owned_partitions";
/// This pod's place in the partition split, labelled by `pods` and `ordinal`. Always 1 (gauge).
pub const PARTITION_OWNERSHIP_INFO: &str = "partition_ownership_info";
-/// 1 while the ownership fence holds the boot, labelled by `reason` (`foreign_split`|`unrecognized`),
-/// else 0 (gauge). **Alert on it**: the pod reads not ready until the logged cause clears.
+/// 1 while the ownership fence holds the boot, labelled by `reason`
+/// (`foreign_split`|`unrecognized`|`parent_running`), else 0 (gauge). **Alert on it**: the pod reads
+/// not ready until the logged cause clears.
pub const OWNERSHIP_FENCE_BLOCKED: &str = "ownership_fence_blocked";
/// Commits by another split, or with unreadable metadata, that the run-time audit found on an owned
/// partition, labelled by `topic` (counter). Each one stops the pod.
diff --git a/rust/cohort-stream-processor/src/partitions/fence.rs b/rust/cohort-stream-processor/src/partitions/fence.rs
index f4cb3181d5b2..9bee59936e56 100644
--- a/rust/cohort-stream-processor/src/partitions/fence.rs
+++ b/rust/cohort-stream-processor/src/partitions/fence.rs
@@ -3,7 +3,8 @@
//! reads those stamps three times.
//!
//! 1. **Admission**, before the store opens. A stamp from another split on an owned partition holds
-//! the boot.
+//! the boot, unless a reshard names that split as this pod's parent. A reshard child also waits
+//! until its parent restarts under the new split, then takes its partitions over.
//! 2. **The claim**, after the store opens. The pod checks the stamps again, stamps every owned
//! partition on every group and reads the stamps back, so a broker that drops commit metadata
//! stops the pod.
@@ -25,6 +26,7 @@ use metrics::{counter, gauge};
use tracing::{error, info, warn};
use super::positions::{InputGroups, InputTopic, PositionsError, ResumeOffset};
+use super::split::{Role, SplitClaim};
use super::stamp::{CommitStamp, Stamp};
use super::PartitionOwnership;
use crate::observability::metrics::{OWNERSHIP_FENCE_BLOCKED, OWNERSHIP_FENCE_VIOLATIONS_TOTAL};
@@ -68,11 +70,18 @@ impl InputCommits {
pub fn commits(&self) -> &BTreeMap<u16, Committed> {
&self.commits
}
+
+ fn carries(&self, partition: u16, split: PartitionOwnership) -> bool {
+ self.commits.get(&partition).and_then(Committed::stamp) == Some(&Stamp::Split(split))
+ }
}
/// The last commits on a set of partitions, on every enabled group.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CommitLedger {
+ /// Every pod reads the events group, so it alone says whether another split still runs and
+ /// whether a takeover finished. A follower group can be enabled after a reshard, with stamps
+ /// that an older split left.
events: InputCommits,
followers: Vec<InputCommits>,
}
@@ -98,16 +107,57 @@ impl CommitLedger {
})
}
- /// The fence decision over a ledger that holds `ownership`'s partitions. An owned partition may
- /// carry no stamp, from a build before stamps or a standard Kafka tool, or this pod's split.
- fn admit(&self, ownership: PartitionOwnership) -> Result<(), FenceBlock> {
+ /// The fence decision, over a ledger that holds `claim`'s partitions and, for a child, its
+ /// parent's partitions in the new split.
+ ///
+ /// - Every stamp on an owned partition passes [`Self::check_stamps`].
+ /// - A child takes over only once its parent has stamped one of its own partitions under the
+ /// new split. A StatefulSet runs one pod per ordinal, so no process runs the old split at the
+ /// parent's ordinal after that.
+ /// - A child with its own split on any events partition finished its takeover before, because
+ /// only its claim and its later commits write that stamp. An offset reset removes the stamp
+ /// from a partition, so the rule cannot require it on every partition.
+ fn admit(&self, claim: &SplitClaim) -> Result<StoreSource, FenceBlock> {
+ self.check_stamps(claim)?;
+ let ownership = claim.ownership();
+ let Role::Child { parent, parent_now } = claim.role() else {
+ return Ok(StoreSource::Own);
+ };
+ if ownership
+ .partitions()
+ .any(|partition| self.events.carries(partition, ownership))
+ {
+ Ok(StoreSource::Own)
+ } else if parent_now
+ .partitions()
+ .any(|partition| self.events.carries(partition, parent_now))
+ {
+ Ok(StoreSource::TakeOver { parent })
+ } else {
+ Err(FenceBlock::ParentRunning { parent })
+ }
+ }
+
+ /// Refuses a stamp on an owned partition that `claim` does not accept. An owned partition may
+ /// carry no stamp, this pod's split, or during a reshard the parent's. An unstamped commit
+ /// passes in every role, because older builds and standard Kafka tools write none. A follower
+ /// partition may also carry any split that covers this pod, because a pod of that split that
+ /// still runs also commits the events group.
+ fn check_stamps(&self, claim: &SplitClaim) -> Result<(), FenceBlock> {
+ let ownership = claim.ownership();
+ let parent = match claim.role() {
+ Role::Steady => None,
+ Role::Parent { parent } | Role::Child { parent, .. } => Some(parent),
+ };
+ let events = self.events.topic();
for (topic, partition, stamp) in self.stamps() {
if !ownership.owns(partition) {
continue;
}
match stamp {
Stamp::Unstamped => {}
- Stamp::Split(found) if *found == ownership => {}
+ Stamp::Split(found) if *found == ownership || Some(*found) == parent => {}
+ Stamp::Split(found) if topic != events && found.covers(&ownership) => {}
Stamp::Split(found) => {
return Err(FenceBlock::ForeignSplit {
topic: topic.clone(),
@@ -151,12 +201,23 @@ impl CommitLedger {
}
}
+/// Where an admitted pod's store comes from.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum StoreSource {
+ /// The live store, or this pod's own checkpoint lineage.
+ Own,
+ /// A reshard child takes its partitions over from `parent`, which advanced them. A live store
+ /// without a pending restore is discarded, then `parent`'s checkpoint lineage is restored, or
+ /// the store is created.
+ TakeOver { parent: PartitionOwnership },
+}
+
/// Why the fence holds the boot.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum FenceBlock {
#[error(
"{found} committed {topic} partition {partition}: its pods may still run, or the pod count \
- changed"
+ changed without COHORT_RESHARD_FROM_POD_COUNT"
)]
ForeignSplit {
topic: InputTopic,
@@ -169,15 +230,18 @@ pub enum FenceBlock {
partition: u16,
metadata: String,
},
+ #[error("waiting for the pod that ran {parent} to restart under the new split")]
+ ParentRunning { parent: PartitionOwnership },
}
impl FenceBlock {
- const REASONS: [&'static str; 2] = ["foreign_split", "unrecognized"];
+ const REASONS: [&'static str; 3] = ["foreign_split", "unrecognized", "parent_running"];
fn reason(&self) -> &'static str {
match self {
Self::ForeignSplit { .. } => Self::REASONS[0],
Self::Unrecognized { .. } => Self::REASONS[1],
+ Self::ParentRunning { .. } => Self::REASONS[2],
}
}
@@ -205,41 +269,40 @@ pub enum FenceError {
/// one. The claim hands its stamp on, so the audit's revoke reaches every stamp the pod uses.
#[derive(Debug, Clone)]
pub struct AdmittedSplit {
+ claim: SplitClaim,
+ source: StoreSource,
stamp: CommitStamp,
}
impl AdmittedSplit {
- /// Retries [`Self::try_admit`] with backoff until the fence admits `ownership`. Each read runs
- /// on a blocking thread, so the health server keeps answering, and the pod reads not ready.
- pub async fn admit(ownership: PartitionOwnership, groups: Arc<InputGroups>) -> Self {
+ /// Retries [`Self::try_admit`] with backoff until the fence admits `claim`. Each read runs on a
+ /// blocking thread, so the health server keeps answering, and the pod reads not ready.
+ pub async fn admit(claim: SplitClaim, groups: Arc<InputGroups>) -> Self {
let mut backoff = RETRY_START;
loop {
let reader = groups.clone();
- let attempt = match tokio::task::spawn_blocking(move || {
- Self::try_admit(ownership, &reader)
- })
- .await
- {
- Ok(attempt) => attempt,
- Err(err) => std::panic::resume_unwind(err.into_panic()),
- };
+ let attempt =
+ match tokio::task::spawn_blocking(move || Self::try_admit(claim, &reader)).await {
+ Ok(attempt) => attempt,
+ Err(err) => std::panic::resume_unwind(err.into_panic()),
+ };
match attempt {
Ok(admitted) => {
FenceBlock::record(None);
- info!(%ownership, "the ownership fence admitted this pod");
+ info!(%claim, source = ?admitted.source, "the ownership fence admitted this pod");
return admitted;
}
Err(FenceError::Blocked(block)) => {
FenceBlock::record(Some(&block));
warn!(
- %ownership,
+ %claim,
%block,
retry_in_secs = backoff.as_secs(),
"the ownership fence holds the boot",
);
}
Err(FenceError::Ledger(err)) => warn!(
- %ownership,
+ %claim,
error = %err,
retry_in_secs = backoff.as_secs(),
"reading the commit ledger failed; retrying",
@@ -251,28 +314,46 @@ impl AdmittedSplit {
}
/// One fence read. Blocks on Kafka.
- pub fn try_admit(
- ownership: PartitionOwnership,
- groups: &InputGroups,
- ) -> Result<Self, FenceError> {
- let partitions: BTreeSet<u16> = ownership.partitions().collect();
- groups.ledger(&partitions)?.admit(ownership)?;
+ pub fn try_admit(claim: SplitClaim, groups: &InputGroups) -> Result<Self, FenceError> {
+ let parent_now = match claim.role() {
+ Role::Child { parent_now, .. } => Some(parent_now),
+ Role::Steady | Role::Parent { .. } => None,
+ };
+ let partitions: BTreeSet<u16> = claim
+ .ownership()
+ .partitions()
+ .chain(parent_now.iter().flat_map(PartitionOwnership::partitions))
+ .collect();
+ let source = groups.ledger(&partitions)?.admit(&claim)?;
Ok(Self {
- stamp: CommitStamp::new(ownership),
+ claim,
+ source,
+ stamp: CommitStamp::new(claim.ownership()),
})
}
pub fn ownership(&self) -> PartitionOwnership {
- self.stamp.split()
+ self.claim.ownership()
+ }
+
+ /// The split the fence admitted, with its place in a reshard.
+ pub fn split_claim(&self) -> SplitClaim {
+ self.claim
+ }
+
+ pub fn source(&self) -> StoreSource {
+ self.source
}
pub fn stamp(&self) -> CommitStamp {
self.stamp.clone()
}
- /// Checks the fence again, then commits every owned partition on every group again with this
- /// pod's stamp, at the group's commit or at the low watermark where the group never committed,
- /// and reads the stamps back. Blocks on Kafka.
+ /// Checks the stamps on the owned partitions again, then commits every owned partition on every
+ /// group again with this pod's stamp, at the group's commit or at the low watermark where the
+ /// group never committed, and reads the stamps back. Run it after the store opens: a child that
+ /// stops before its claim still reads as its parent's, so the next boot takes over again.
+ /// Blocks on Kafka.
pub fn claim(self, groups: &InputGroups) -> anyhow::Result<ClaimedSplit> {
let ownership = self.ownership();
let partitions: BTreeSet<u16> = ownership.partitions().collect();
@@ -280,7 +361,7 @@ impl AdmittedSplit {
// again first. Commits have no compare-and-set: a commit between this read and the
// restamp is overwritten, and only the audit can catch it.
let ledger = groups.ledger(&partitions)?;
- ledger.admit(ownership)?;
+ ledger.check_stamps(&self.claim)?;
groups.restamp(&self.stamp, &ledger)?;
groups.ledger(&partitions)?.confirm(ownership)?;
info!(%ownership, "claimed every owned partition; the stamps read back");
@@ -383,6 +464,14 @@ mod tests {
.unwrap()
}
+ fn claim(pods: u16, ordinal: u16, reshard_from: Option<u16>) -> SplitClaim {
+ SplitClaim::new(
+ split(pods, ordinal),
+ reshard_from.map(|from| NonZeroU16::new(from).unwrap()),
+ )
+ .unwrap()
+ }
+
/// `None` is a partition its group never committed.
fn commits(topic: &str, stamps: [Option<Stamp>; PARTITIONS as usize]) -> InputCommits {
let commits = (0..PARTITIONS)
@@ -406,20 +495,20 @@ mod tests {
}
#[test]
- fn the_fence_admits_a_split_only_over_its_own_or_no_stamps() {
+ fn the_fence_admits_a_split_only_over_its_own_its_ancestors_or_no_stamps() {
let unstamped = || Some(Stamp::Unstamped);
let unreadable = || Some(Stamp::Unrecognized("other-tool".to_owned()));
type Row = (
&'static str,
- PartitionOwnership,
+ SplitClaim,
[Option<Stamp>; 4],
[Option<Stamp>; 4],
- Result<(), FenceBlock>,
+ Result<StoreSource, FenceBlock>,
);
let rows: Vec<Row> = vec![
(
- "unreadable metadata blocks",
- split(1, 0),
+ "unreadable metadata blocks any role",
+ claim(1, 0, None),
[by(1, 0), by(1, 0), unreadable(), by(1, 0)],
[by(1, 0), by(1, 0), by(1, 0), by(1, 0)],
Err(FenceBlock::Unrecognized {
@@ -429,15 +518,15 @@ mod tests {
}),
),
(
- "its own stamps, none, and partitions never committed",
- split(2, 1),
+ "steady over its own stamps, none, and partitions never committed",
+ claim(2, 1, None),
[by(2, 0), by(2, 1), by(2, 0), unstamped()],
[None, by(2, 1), None, None],
- Ok(()),
+ Ok(StoreSource::Own),
),
(
- "another split: the pod count changed",
- split(2, 1),
+ "steady over another split: the pod count changed without a reshard",
+ claim(2, 1, None),
[by(2, 0), by(1, 0), by(2, 0), by(1, 0)],
[by(2, 0), by(1, 0), by(2, 0), by(1, 0)],
Err(FenceBlock::ForeignSplit {
@@ -446,10 +535,72 @@ mod tests {
found: split(1, 0),
}),
),
+ (
+ "steady over a follower that was off through a reshard",
+ claim(2, 1, None),
+ [by(2, 0), by(2, 1), by(2, 0), by(2, 1)],
+ [by(1, 0), by(1, 0), by(1, 0), by(1, 0)],
+ Ok(StoreSource::Own),
+ ),
+ (
+ "parent over its own, its old split's and no stamps",
+ claim(2, 0, Some(1)),
+ [by(2, 0), by(1, 0), unstamped(), by(1, 0)],
+ [by(1, 0), by(1, 0), by(1, 0), by(1, 0)],
+ Ok(StoreSource::Own),
+ ),
+ (
+ "parent over another split",
+ claim(2, 0, Some(1)),
+ [by(4, 0), by(1, 0), by(2, 0), by(1, 0)],
+ [by(1, 0), by(1, 0), by(1, 0), by(1, 0)],
+ Err(FenceBlock::ForeignSplit {
+ topic: InputTopic::new(EVENTS),
+ partition: 0,
+ found: split(4, 0),
+ }),
+ ),
+ (
+ "child with its split on an events partition finished its takeover, though an \
+ offset reset removed it from another",
+ claim(2, 1, Some(1)),
+ [by(2, 0), by(2, 1), by(2, 0), unstamped()],
+ [by(1, 0), by(1, 0), by(1, 0), unstamped()],
+ Ok(StoreSource::Own),
+ ),
+ (
+ "child takes over once its parent stamped under the new split",
+ claim(2, 1, Some(1)),
+ [by(2, 0), unstamped(), by(1, 0), by(1, 0)],
+ [by(2, 0), by(1, 0), by(1, 0), by(1, 0)],
+ Ok(StoreSource::TakeOver {
+ parent: split(1, 0),
+ }),
+ ),
+ (
+ "child waits while its parent has not restarted",
+ claim(2, 1, Some(1)),
+ [by(1, 0), unstamped(), by(1, 0), None],
+ [by(1, 0), by(1, 0), by(2, 0), by(1, 0)],
+ Err(FenceBlock::ParentRunning {
+ parent: split(1, 0),
+ }),
+ ),
+ (
+ "child over another split",
+ claim(2, 1, Some(1)),
+ [by(2, 0), by(1, 0), by(2, 0), by(1, 0)],
+ [by(2, 0), by(1, 0), by(2, 0), by(4, 3)],
+ Err(FenceBlock::ForeignSplit {
+ topic: InputTopic::new(MERGES),
+ partition: 3,
+ found: split(4, 3),
+ }),
+ ),
];
- for (name, ownership, events, merges, expected) in rows {
+ for (name, claim, events, merges, expected) in rows {
let ledger = CommitLedger::new(commits(EVENTS, events), vec![commits(MERGES, merges)]);
- assert_eq!(ledger.admit(ownership), expected, "{name}");
+ assert_eq!(ledger.admit(&claim), expected, "{name}");
}
}
diff --git a/rust/cohort-stream-processor/src/partitions/mod.rs b/rust/cohort-stream-processor/src/partitions/mod.rs
index bd1c2124e40f..3922e5c9ae5c 100644
--- a/rust/cohort-stream-processor/src/partitions/mod.rs
+++ b/rust/cohort-stream-processor/src/partitions/mod.rs
@@ -17,6 +17,7 @@ pub mod positions;
pub mod rebalance;
pub mod router;
pub mod shuffle_message;
+pub mod split;
pub mod stamp;
pub mod watermarks;
@@ -25,7 +26,7 @@ pub use cohort_core::partitioner;
pub use backpressure::{Backpressure, PartitionHoldover};
pub use fence::{
AdmittedSplit, ClaimedSplit, CommitLedger, Committed, FenceBlock, FenceError, InputCommits,
- OwnershipAudit, AUDIT_INTERVAL,
+ OwnershipAudit, StoreSource, AUDIT_INTERVAL,
};
pub use follower::{Follower, FollowerSet};
pub use intake::{Admission, MeteredReceiver, PartitionIntake};
@@ -48,5 +49,6 @@ pub use router::{
WorkerInbox,
};
pub use shuffle_message::ShuffleMessage;
+pub use split::{Role, SplitClaim, SplitError};
pub use stamp::{CommitStamp, Stamp, StampedList};
pub use watermarks::{LiveWatermarks, WatermarkMs};
diff --git a/rust/cohort-stream-processor/src/partitions/split.rs b/rust/cohort-stream-processor/src/partitions/split.rs
new file mode 100644
index 000000000000..f4a217b402cc
--- /dev/null
+++ b/rust/cohort-stream-processor/src/partitions/split.rs
@@ -0,0 +1,221 @@
+//! The split this pod runs under, and its place in a reshard. A reshard multiplies the pod count by
+//! a whole factor. A pod whose ordinal is below the old count keeps its store and drops the
+//! partitions it gives away. Every other pod takes its partitions over from the pod of the old split
+//! that owned them.
+
+use std::fmt;
+use std::num::NonZeroU16;
+
+use super::PartitionOwnership;
+use crate::store::durability::PodOrdinal;
+
+/// This pod's split, and its place in a reshard. Only [`Self::new`] builds one, so a reshard always
+/// raises the pod count by a whole factor, and the parent always covers this pod.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct SplitClaim {
+ ownership: PartitionOwnership,
+ role: Role,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum Role {
+ /// No reshard is under way.
+ Steady,
+ /// The ordinal is below the old pod count, so this pod's store already holds its partitions. It
+ /// wrote them under `parent`.
+ Parent { parent: PartitionOwnership },
+ /// The ordinal is at or above the old pod count. Its partitions belong to `parent` until it
+ /// takes them over. `parent_now` is the parent's place in the new split.
+ Child {
+ parent: PartitionOwnership,
+ parent_now: PartitionOwnership,
+ },
+}
+
+#[derive(Debug, thiserror::Error)]
+pub enum SplitError {
+ #[error("a reshard from {from} to {pods} pods must raise the pod count")]
+ NotAGrowth { from: NonZeroU16, pods: NonZeroU16 },
+ #[error("a reshard from {from} to {pods} pods must multiply the pod count by a whole number")]
+ NotAMultiple { from: NonZeroU16, pods: NonZeroU16 },
+}
+
+impl SplitClaim {
+ /// `reshard_from` is the pod count before the reshard, or `None` outside one. No path merges
+ /// two stores, so a reshard that lowers the pod count is refused.
+ pub fn new(
+ ownership: PartitionOwnership,
+ reshard_from: Option<NonZeroU16>,
+ ) -> Result<Self, SplitError> {
+ let Some(from) = reshard_from else {
+ return Ok(Self {
+ ownership,
+ role: Role::Steady,
+ });
+ };
+ let pods = ownership.pods();
+ if from >= pods {
+ return Err(SplitError::NotAGrowth { from, pods });
+ }
+ if !pods.get().is_multiple_of(from.get()) {
+ return Err(SplitError::NotAMultiple { from, pods });
+ }
+ let ordinal = ownership.ordinal().get();
+ let place = |pods| {
+ PartitionOwnership::new(
+ pods,
+ PodOrdinal::from(ordinal % from.get()),
+ ownership.partition_count(),
+ )
+ .expect("a divisor of the pod count splits the partitions evenly")
+ };
+ let parent = place(from);
+ let role = if ordinal < from.get() {
+ Role::Parent { parent }
+ } else {
+ Role::Child {
+ parent,
+ parent_now: place(pods),
+ }
+ };
+ Ok(Self { ownership, role })
+ }
+
+ pub fn ownership(&self) -> PartitionOwnership {
+ self.ownership
+ }
+
+ pub fn role(&self) -> Role {
+ self.role
+ }
+}
+
+impl fmt::Display for SplitClaim {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ match self.role {
+ Role::Steady => write!(f, "{}", self.ownership),
+ Role::Parent { parent } | Role::Child { parent, .. } => {
+ write!(f, "{}, resharded from {parent}", self.ownership)
+ }
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ const PARTITIONS: u16 = 64;
+
+ fn split(pods: u16, ordinal: u16) -> PartitionOwnership {
+ PartitionOwnership::new(
+ NonZeroU16::new(pods).unwrap(),
+ PodOrdinal::from(ordinal),
+ PARTITIONS,
+ )
+ .unwrap()
+ }
+
+ #[test]
+ fn a_reshard_gives_every_ordinal_its_role_and_parent() {
+ let cases = [
+ (
+ 1,
+ 2,
+ 0,
+ Role::Parent {
+ parent: split(1, 0),
+ },
+ ),
+ (
+ 1,
+ 2,
+ 1,
+ Role::Child {
+ parent: split(1, 0),
+ parent_now: split(2, 0),
+ },
+ ),
+ (
+ 2,
+ 4,
+ 0,
+ Role::Parent {
+ parent: split(2, 0),
+ },
+ ),
+ (
+ 2,
+ 4,
+ 1,
+ Role::Parent {
+ parent: split(2, 1),
+ },
+ ),
+ (
+ 2,
+ 4,
+ 2,
+ Role::Child {
+ parent: split(2, 0),
+ parent_now: split(4, 0),
+ },
+ ),
+ (
+ 2,
+ 4,
+ 3,
+ Role::Child {
+ parent: split(2, 1),
+ parent_now: split(4, 1),
+ },
+ ),
+ (
+ 1,
+ 4,
+ 0,
+ Role::Parent {
+ parent: split(1, 0),
+ },
+ ),
+ (
+ 1,
+ 4,
+ 3,
+ Role::Child {
+ parent: split(1, 0),
+ parent_now: split(4, 0),
+ },
+ ),
+ ];
+ for (from, pods, ordinal, role) in cases {
+ let claim = SplitClaim::new(split(pods, ordinal), NonZeroU16::new(from)).unwrap();
+ assert_eq!(claim.role(), role, "pod {ordinal} of {from} -> {pods}");
+ assert_eq!(claim.ownership(), split(pods, ordinal));
+ }
+
+ assert_eq!(
+ SplitClaim::new(split(2, 1), None).unwrap().role(),
+ Role::Steady
+ );
+ }
+
+ #[test]
+ fn a_reshard_must_raise_the_pod_count_by_a_whole_factor() {
+ let two = NonZeroU16::new(2).unwrap();
+ assert!(matches!(
+ SplitClaim::new(split(1, 0), Some(two)),
+ Err(SplitError::NotAGrowth { .. })
+ ));
+ assert!(matches!(
+ SplitClaim::new(split(2, 1), Some(two)),
+ Err(SplitError::NotAGrowth { .. })
+ ));
+ let three_of_twelve =
+ PartitionOwnership::new(NonZeroU16::new(3).unwrap(), PodOrdinal::from(2), 12).unwrap();
+ assert!(matches!(
+ SplitClaim::new(three_of_twelve, Some(two)),
+ Err(SplitError::NotAMultiple { .. })
+ ));
+ }
+}
diff --git a/rust/cohort-stream-processor/src/store/durability/checkpoint.rs b/rust/cohort-stream-processor/src/store/durability/checkpoint.rs
index 1fbf61bfebc4..f6d797162cbd 100644
--- a/rust/cohort-stream-processor/src/store/durability/checkpoint.rs
+++ b/rust/cohort-stream-processor/src/store/durability/checkpoint.rs
@@ -182,6 +182,10 @@ enum UploadKind<'a> {
#[derive(Debug, PartialEq)]
enum FullReason {
NoBaseline,
+ /// The baseline went up under another lineage: a reshard child restored its parent's
+ /// checkpoint. An incremental chain would name objects in the parent's prefix, and a lineage
+ /// must hold every object its own restores need.
+ OtherLineage,
/// A created store restarts SST numbering, so a file name from another DB names other data.
OtherDb,
/// The chain is older than the full-upload interval, which bounds the age of what a restore
@@ -208,12 +212,16 @@ impl UploadKind<'_> {
fn upload_kind<'a>(
baseline: Option<&'a CheckpointMetadata>,
db: &DbIdentity,
+ lineage: CheckpointLineage,
now: DateTime<Utc>,
full_max_age: Duration,
) -> UploadKind<'a> {
let Some(baseline) = baseline else {
return UploadKind::Full(FullReason::NoBaseline);
};
+ if baseline.ordinal != lineage.ordinal() {
+ return UploadKind::Full(FullReason::OtherLineage);
+ }
if &baseline.db_identity != db {
return UploadKind::Full(FullReason::OtherDb);
}
@@ -244,7 +252,7 @@ pub struct CheckpointSweeper {
impl CheckpointSweeper {
/// The first baseline is the lineage's `uploaded.json` when it went to the configured bucket.
- /// [`upload_kind`] uses it only for the same DB.
+ /// [`upload_kind`] uses it only for the same lineage and DB.
pub fn new(
store: CohortStore,
dispatcher: Arc<EventDispatcher>,
@@ -428,6 +436,7 @@ impl CheckpointSweeper {
let kind = upload_kind(
baseline.as_ref(),
&db,
+ self.lineage,
attempt_timestamp,
self.config.full_upload_interval,
);
@@ -715,10 +724,12 @@ mod tests {
}
#[test]
- fn an_upload_builds_on_its_baseline_only_for_the_same_db_and_bucket_and_a_young_chain() {
+ fn an_upload_builds_on_its_baseline_only_for_the_same_lineage_db_and_bucket_and_a_young_chain()
+ {
let now = Utc::now();
let day = Duration::from_secs(86_400);
let db = DbIdentity::for_test("db-a");
+ let lineage = CheckpointLineage::new(PodOrdinal::STANDALONE);
let chain_started = |age: chrono::Duration, db: &str| {
let mut baseline = CheckpointMetadata::new(
PodOrdinal::STANDALONE,
@@ -734,23 +745,34 @@ mod tests {
let other_db = chain_started(chrono::Duration::hours(1), "db-b");
assert_eq!(
- upload_kind(None, &db, now, day),
+ upload_kind(None, &db, lineage, now, day),
UploadKind::Full(FullReason::NoBaseline)
);
+ // A reshard child's restored store keeps its parent's DB id.
+ assert_eq!(
+ upload_kind(
+ Some(&young),
+ &db,
+ CheckpointLineage::new(PodOrdinal::from(1)),
+ now,
+ day
+ ),
+ UploadKind::Full(FullReason::OtherLineage)
+ );
assert_eq!(
- upload_kind(Some(&other_db), &db, now, day),
+ upload_kind(Some(&other_db), &db, lineage, now, day),
UploadKind::Full(FullReason::OtherDb)
);
assert_eq!(
- upload_kind(Some(&expired), &db, now, day),
+ upload_kind(Some(&expired), &db, lineage, now, day),
UploadKind::Full(FullReason::ChainExpired)
);
assert_eq!(
- upload_kind(Some(&at_limit), &db, now, day),
+ upload_kind(Some(&at_limit), &db, lineage, now, day),
UploadKind::Incremental(&at_limit)
);
assert_eq!(
- upload_kind(Some(&young), &db, now, day),
+ upload_kind(Some(&young), &db, lineage, now, day),
UploadKind::Incremental(&young)
);
diff --git a/rust/cohort-stream-processor/src/store/durability/manifest.rs b/rust/cohort-stream-processor/src/store/durability/manifest.rs
index 46ea2a454a2a..f4c733e6490d 100644
--- a/rust/cohort-stream-processor/src/store/durability/manifest.rs
+++ b/rust/cohort-stream-processor/src/store/durability/manifest.rs
@@ -11,7 +11,7 @@
//! stamped.
//!
//! The manifest records the split it was captured under. A restore accepts it on any pod whose
-//! partitions that split covers.
+//! partitions that split covers: its own lineage, or during a reshard its parent's.
use std::collections::{BTreeMap, BTreeSet};
use std::path::Path;
diff --git a/rust/cohort-stream-processor/src/store/durability/recovery.rs b/rust/cohort-stream-processor/src/store/durability/recovery.rs
index a27e7d4263b8..c527b180455b 100644
--- a/rust/cohort-stream-processor/src/store/durability/recovery.rs
+++ b/rust/cohort-stream-processor/src/store/durability/recovery.rs
@@ -4,10 +4,12 @@
//!
//! 1. A leftover staging directory from a killed restore is deleted.
//! 2. **Reopened.** An intact live store reopens. If it carries a restore marker, the restore it
-//! published is still pending and resumes instead.
+//! published is still pending and resumes instead. A reshard child that is taking its
+//! partitions over discards a live store without a marker, because its parent advanced them.
//! 3. With checkpoints enabled, candidates are tried newest first, the freshest local checkpoint
-//! before S3. A candidate restores on any pod whose partitions its split covers. The first one
-//! that publishes wins.
+//! before S3. A reshard child that is taking over searches only its parent's S3 lineage. A
+//! candidate restores on any pod whose partitions its split covers. The first one that
+//! publishes wins.
//! 4. **Created.** Nothing is restorable: no candidate, or only unusable ones. Every slice then
//! begins when its worker spawns, behind the coverage fence.
//!
@@ -43,7 +45,7 @@ use crate::observability::metrics::{
CHECKPOINT_RESTORE_BLOCKED, CHECKPOINT_RESTORE_CANDIDATES_TOTAL,
CHECKPOINT_RESTORE_DURATION_SECONDS, CHECKPOINT_RESTORE_TOTAL,
};
-use crate::partitions::{AdmittedSplit, InputGroups, InputTopic, PartitionOwnership};
+use crate::partitions::{AdmittedSplit, InputGroups, InputTopic, PartitionOwnership, StoreSource};
use crate::store::{CohortStore, StoreError, STORE_SCHEMA_VERSION};
pub(crate) const RETRY_START: Duration = Duration::from_secs(30);
@@ -200,8 +202,9 @@ impl Search {
/// reset. A restore comes back with it, for the events consumer to finish before anything folds.
///
/// A restore that cannot finish blocks here, retrying with backoff, until the operator fixes the
-/// cause or turns checkpoints off. `lineage` is `None` when checkpoints are disabled. The store
-/// refuses to open when it was written under a split that does not cover the one `admitted`.
+/// cause or turns checkpoints off. `lineage` is this pod's own, and `None` when checkpoints are
+/// disabled. The store refuses to open when it was written under a split that does not cover the
+/// one `admitted`.
pub async fn open_store(
config: &Config,
admitted: &AdmittedSplit,
@@ -257,6 +260,7 @@ async fn restore_until_ready(
let started = Instant::now();
let attempt = match prepare_store(
&config,
+ admitted.source(),
admitted.ownership(),
lineage,
groups.as_ref(),
@@ -269,7 +273,7 @@ async fn restore_until_ready(
// Another split can claim while the restore runs, so the boot's admission
// no longer covers the follower commits.
let admitted =
- AdmittedSplit::admit(admitted.ownership(), groups.clone()).await;
+ AdmittedSplit::admit(admitted.split_claim(), groups.clone()).await;
pending
.position_followers(&groups, &admitted)
.map(|()| (BootStore::Restored(pending), source))
@@ -311,6 +315,7 @@ async fn restore_until_ready(
/// Blocks on file and Kafka I/O; see [`restore_until_ready`].
async fn prepare_store(
config: &Config,
+ source: StoreSource,
ownership: PartitionOwnership,
lineage: Option<CheckpointLineage>,
window: &impl ReplayWindow,
@@ -324,7 +329,18 @@ async fn prepare_store(
if !config.effective_wipe_on_start() && live_store_is_intact(&live) {
match PendingRestore::resume(&live, window, config.partition_count()) {
Ok(Some(pending)) => return Ok((BootStore::Restored(Box::new(pending)), "pending")),
- Ok(None) => return Ok((BootStore::Reopened, "reopen_live")),
+ Ok(None) => match source {
+ StoreSource::Own => return Ok((BootStore::Reopened, "reopen_live")),
+ StoreSource::TakeOver { parent } => {
+ warn!(
+ %parent,
+ store = %live.display(),
+ "taking the partitions over from the parent, which advanced them since this \
+ store; deleting the store",
+ );
+ discard_live_store(&live, &stage).map_err(RestoreBlocked::Local)?;
+ }
+ },
Err(ResumeError::Undecodable(err)) => {
warn!(
error = %err,
@@ -332,11 +348,7 @@ async fn prepare_store(
"the pending restore's marker is unreadable, so its positions are unknown; \
deleting the store and restoring again",
);
- // Renamed first, because a deletion cut short could leave `CURRENT` without the
- // marker, which would reopen at the broker's old offsets. The next boot deletes a
- // leftover stage.
- std::fs::rename(&live, &stage).map_err(RestoreBlocked::Local)?;
- remove_dir_if_present(&stage).map_err(RestoreBlocked::Local)?;
+ discard_live_store(&live, &stage).map_err(RestoreBlocked::Local)?;
}
Err(ResumeError::Window(err)) => return Err(RestoreBlocked::Positions(err)),
Err(ResumeError::Io(err)) => return Err(RestoreBlocked::Local(err)),
@@ -365,11 +377,21 @@ async fn prepare_store(
};
let mut search = Search::default();
- let local = newest_fresh_local_checkpoint(
- &local_dir,
- durability.local_checkpoint_max_staleness,
- searched_at,
- );
+ let (local, searched, s3_label) = match source {
+ StoreSource::Own => (
+ newest_fresh_local_checkpoint(
+ &local_dir,
+ durability.local_checkpoint_max_staleness,
+ searched_at,
+ ),
+ lineage,
+ "s3",
+ ),
+ // The local directory holds only this pod's own lineage.
+ StoreSource::TakeOver { parent } => {
+ (None, CheckpointLineage::new(parent.ordinal()), "parent_s3")
+ }
+ };
if let Some(attempt) = local {
let verdict = match attempt {
Ok(attempt) => restore.local(attempt)?,
@@ -380,7 +402,7 @@ async fn prepare_store(
}
}
- let downloader = S3Downloader::new(&durability, lineage)
+ let downloader = S3Downloader::new(&durability, searched)
.await
.map_err(RestoreBlocked::Listing)?;
let importer =
@@ -394,7 +416,7 @@ async fn prepare_store(
)
.await?
{
- return Ok((BootStore::Restored(pending), "s3"));
+ return Ok((BootStore::Restored(pending), s3_label));
}
search.conclude().map(|store| (store, "cold"))
}
@@ -471,8 +493,8 @@ impl<W: ReplayWindow> Restore<'_, W> {
/// Tries the S3 candidates newest first, up to `depth` of them. An unfinished upload does not
/// count toward the depth, so a run of failed uploads cannot hide a complete one behind it.
- /// Neither does a candidate whose split does not cover this pod, so checkpoints of another
- /// split cannot hide a covering one behind them.
+ /// Neither does a candidate whose split does not cover this pod: a reshard child that starts
+ /// late passes over its parent's checkpoints from after the reshard.
async fn search_s3(
&mut self,
importer: &CheckpointImporter,
@@ -662,6 +684,14 @@ impl<W: ReplayWindow> Restore<'_, W> {
}
}
+/// Renames the live store to the stage before deleting it, because a deletion cut short could leave
+/// `CURRENT` without the marker, which would reopen at the broker's old offsets. The next boot
+/// deletes a leftover stage.
+fn discard_live_store(live: &Path, stage: &Path) -> io::Result<()> {
+ std::fs::rename(live, stage)?;
+ remove_dir_if_present(stage)
+}
+
/// True when `path` is an opened RocksDB store: the directory holds `CURRENT`, RocksDB's manifest
/// pointer. A directory without it is torn and never reopens.
fn live_store_is_intact(path: &Path) -> bool {
@@ -1009,10 +1039,25 @@ mod tests {
}
async fn prepare(&self, window: &impl ReplayWindow) -> Result<BootStore, RestoreBlocked> {
+ self.prepare_from(
+ StoreSource::Own,
+ Some(CheckpointLineage::new(PodOrdinal::STANDALONE)),
+ window,
+ )
+ .await
+ }
+
+ async fn prepare_from(
+ &self,
+ source: StoreSource,
+ lineage: Option<CheckpointLineage>,
+ window: &impl ReplayWindow,
+ ) -> Result<BootStore, RestoreBlocked> {
prepare_store(
&self.config,
+ source,
self.ownership,
- Some(CheckpointLineage::new(PodOrdinal::STANDALONE)),
+ lineage,
window,
Utc::now(),
&mut HashSet::new(),
@@ -1117,8 +1162,8 @@ mod tests {
#[tokio::test]
async fn a_candidate_of_a_split_that_does_not_cover_this_pod_is_passed_over_outside_the_depth()
{
- // The newest checkpoint is of pod 0 of 2, which does not cover pod 1 of 2, and the one
- // before is of pod 0 of 1, which does.
+ // Pod 1 of 2 restores its parent's lineage: the newest checkpoint is from after the
+ // reshard, under the parent's new split, and the one before is from the old split.
let boot = Boot::of(split(2, 1));
let mut objects = HashMap::new();
let now = Utc::now();
@@ -1152,6 +1197,35 @@ mod tests {
assert!(search.failed.is_empty(), "{:?}", search.failed);
}
+ #[tokio::test]
+ async fn a_takeover_deletes_a_live_store_but_resumes_a_pending_restore() {
+ let takeover = StoreSource::TakeOver {
+ parent: PartitionOwnership::single(1),
+ };
+
+ let boot = Boot::new();
+ drop(CohortStore::open(&StoreConfig {
+ path: boot.live(),
+ ..StoreConfig::default()
+ }));
+ assert!(matches!(
+ boot.prepare_from(takeover, None, &window()).await,
+ Ok(BootStore::Created)
+ ));
+ assert!(!boot.live().exists(), "the parent advanced what it holds");
+
+ let boot = Boot::new();
+ boot.local_checkpoint();
+ let Ok(BootStore::Restored(published)) = boot.prepare(&window()).await else {
+ panic!("a fresh local checkpoint restores");
+ };
+ let Ok(BootStore::Restored(resumed)) = boot.prepare_from(takeover, None, &window()).await
+ else {
+ panic!("a restore the takeover published resumes");
+ };
+ assert_eq!(resumed.source(), published.source());
+ }
+
#[tokio::test(start_paused = true)]
async fn a_download_timeout_ends_the_search_because_older_candidates_are_as_large() {
let boot = Boot::new();
diff --git a/rust/cohort-stream-processor/src/store/durability/stage.rs b/rust/cohort-stream-processor/src/store/durability/stage.rs
index 5c622b2de5c4..9afd9b2b8a1c 100644
--- a/rust/cohort-stream-processor/src/store/durability/stage.rs
+++ b/rust/cohort-stream-processor/src/store/durability/stage.rs
@@ -12,6 +12,7 @@ use std::fs::File;
use std::io;
use std::path::{Path, PathBuf};
+use chrono::{DateTime, Utc};
use metrics::counter;
use rdkafka::error::{KafkaError, KafkaResult};
use serde::{Deserialize, Serialize};
@@ -205,6 +206,11 @@ impl PendingRestore {
&self.record.source
}
+ /// When the restored checkpoint read its positions.
+ pub fn captured_at(&self) -> DateTime<Utc> {
+ self.record.manifest.captured_at()
+ }
+
/// Commits each follower's planned positions on the partitions `admitted` owns, with its stamp.
/// The other partitions belong to other pods, whose followers keep their own commits. Safe to
/// repeat: nothing consumes a follower before the restore finishes.
diff --git a/rust/cohort-stream-processor/src/store/rocks.rs b/rust/cohort-stream-processor/src/store/rocks.rs
index 621f9e3f5550..cbbc00119371 100644
--- a/rust/cohort-stream-processor/src/store/rocks.rs
+++ b/rust/cohort-stream-processor/src/store/rocks.rs
@@ -518,7 +518,9 @@ impl CohortStore {
}
/// Records `ownership` in a store written under a split that covers it, and refuses any other
- /// store. A store from before ownership records adopts the configured one.
+ /// store. A covering split is this pod's own, or in a reshard the split that wrote the
+ /// parent's store or checkpoint. A store from before ownership records adopts the configured
+ /// one.
fn claim_ownership(&self, ownership: PartitionOwnership) -> Result<(), StoreError> {
if let Some(recorded) = self.get(Meta::CF, &Meta::encode(&META_POD_PARTITIONS))? {
match PartitionOwnership::decode(&recorded) {
diff --git a/rust/cohort-stream-processor/tests/cascade_consumer.rs b/rust/cohort-stream-processor/tests/cascade_consumer.rs
index 33c95e71ae51..d6ecb49c0222 100644
--- a/rust/cohort-stream-processor/tests/cascade_consumer.rs
+++ b/rust/cohort-stream-processor/tests/cascade_consumer.rs
@@ -32,7 +32,7 @@ use cohort_stream_processor::filters::{
use cohort_stream_processor::partitions::{
merge_partition_key, partition_of, AdmittedSplit, ClaimedSplit, CohortConsumerContext,
Follower, FollowerSet, InputGroups, OffsetTracker, PartitionOwnership, PartitionRouter,
- COHORT_PARTITION_COUNT,
+ SplitClaim, COHORT_PARTITION_COUNT,
};
use cohort_stream_processor::producer::{
CascadeSink, CohortMembershipChange, KafkaCascadeSink, KafkaMembershipSink,
@@ -404,9 +404,10 @@ impl Groups {
/// Admits and claims `ownership` through the ownership fence on the groups `config` names, as the
/// processor does at boot.
async fn claim(config: Config, ownership: PartitionOwnership) -> ClaimedSplit {
+ let claim = SplitClaim::new(ownership, None).expect("a split outside a reshard");
tokio::task::spawn_blocking(move || {
let groups = InputGroups::new(&config);
- AdmittedSplit::try_admit(ownership, &groups)
+ AdmittedSplit::try_admit(claim, &groups)
.expect("the fence admits the split")
.claim(&groups)
.expect("claim the owned partitions")
diff --git a/rust/cohort-stream-processor/tests/events_consumer.rs b/rust/cohort-stream-processor/tests/events_consumer.rs
index 724eee7a1563..9b19ca4b96d9 100644
--- a/rust/cohort-stream-processor/tests/events_consumer.rs
+++ b/rust/cohort-stream-processor/tests/events_consumer.rs
@@ -28,7 +28,8 @@
// Tests seed and assert through `CohortStore` directly — the sanctioned direct-store test surface.
#![allow(clippy::disallowed_methods)]
-use std::collections::HashMap;
+use std::collections::{BTreeMap, BTreeSet, HashMap};
+use std::num::NonZeroU16;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
@@ -44,14 +45,15 @@ use cohort_stream_processor::filters::{
};
use cohort_stream_processor::partitions::{
AdmittedSplit, ClaimedSplit, CohortConsumerContext, InputGroups, InputTopic, OffsetTracker,
- PartitionOwnership, PartitionRouter, ShuffleMessage, WorkerInbox,
+ PartitionOwnership, PartitionRouter, ShuffleMessage, SplitClaim, Stamp, StoreSource,
+ WorkerInbox,
};
use cohort_stream_processor::producer::{
CaptureSink, CohortMembershipChange, KafkaMembershipSink, MembershipSink, MembershipStatus,
};
use cohort_stream_processor::stage1::{Stage1State, StatefulRecord};
use cohort_stream_processor::store::durability::{
- open_store, CheckpointLineage, CheckpointSweeper, PendingRestore, PodOrdinal, MARKER_FILENAME,
+ open_store, CheckpointSweeper, PendingRestore, PodOrdinal, MARKER_FILENAME,
};
use cohort_stream_processor::store::{
BehavioralKey, CohortStore, LeafStateKey, OffloadConfig, OffloadMode, StoreConfig, StoreHandle,
@@ -240,12 +242,11 @@ async fn produce_new_persons(topic: &str, count: u128) -> usize {
/// The single pod's ownership of the test topics.
const SINGLE_POD: PartitionOwnership = PartitionOwnership::single(NUM_PARTITIONS as u16);
-/// Admits `ownership` through the ownership fence, as the processor does before it opens the
-/// store.
-async fn admit(ownership: PartitionOwnership, groups: &Arc<InputGroups>) -> AdmittedSplit {
+/// Admits `claim` through the ownership fence, as the processor does before it opens the store.
+async fn admit(claim: SplitClaim, groups: &Arc<InputGroups>) -> AdmittedSplit {
let groups = groups.clone();
tokio::task::spawn_blocking(move || {
- AdmittedSplit::try_admit(ownership, &groups).expect("the fence admits the split")
+ AdmittedSplit::try_admit(claim, &groups).expect("the fence admits the split")
})
.await
.expect("the admission panicked")
@@ -273,7 +274,8 @@ async fn claim_events(topic: &str, group: &str, ownership: PartitionOwnership) -
config.cohort_merge_state_transfer_topic = topic.to_owned();
config.kafka_merge_apply_consumer_group = format!("{group}-transfers");
let groups = Arc::new(InputGroups::new(&config));
- claim(admit(ownership, &groups).await, &groups).await
+ let split = SplitClaim::new(ownership, None).expect("a split outside a reshard");
+ claim(admit(split, &groups).await, &groups).await
}
async fn build_consumer(
@@ -1318,7 +1320,7 @@ fn build_consumer_finishing_restore(
offset_commit_interval: Duration,
restore: PendingRestore,
claimed: &ClaimedSplit,
-) -> CohortStreamEventsConsumer {
+) -> (CohortStreamEventsConsumer, Arc<EventDispatcher>) {
let dispatcher = Arc::new(EventDispatcher::new(
PartitionRouter::new(64),
Arc::new(OffsetTracker::new()),
@@ -1329,10 +1331,10 @@ fn build_consumer_finishing_restore(
));
dispatcher.enable_durable_restore();
- CohortStreamEventsConsumer::new(
+ let consumer = CohortStreamEventsConsumer::new(
Arc::new(events_client(group)),
topic.to_string(),
- dispatcher,
+ dispatcher.clone(),
handle,
100,
Duration::from_millis(200),
@@ -1340,7 +1342,8 @@ fn build_consumer_finishing_restore(
claimed,
Some(restore),
)
- .expect("assign the owned partitions")
+ .expect("assign the owned partitions");
+ (consumer, dispatcher)
}
/// Each partition's high watermark.
@@ -1355,9 +1358,10 @@ fn high_watermarks(consumer: &StreamConsumer, topic: &str) -> HashMap<i32, i64>
.collect()
}
-/// Delete every S3 object under the configured prefix on test exit. Best-effort: builds a throwaway
-/// `object_store` client from the durability config and sweeps the per-test prefix. Any error
-/// building, listing, or deleting is ignored so cleanup never fails a passing test.
+/// Delete every S3 object of the config's lineage under the configured prefix on test exit.
+/// Best-effort: builds a throwaway `object_store` client from the durability config and sweeps the
+/// per-test prefix. Any error building, listing, or deleting is ignored so cleanup never fails a
+/// passing test.
async fn delete_s3_prefix(config: &Config) {
use futures::StreamExt;
use object_store::aws::AmazonS3Builder;
@@ -1390,7 +1394,9 @@ async fn delete_s3_prefix(config: &Config) {
Ok(store) => store,
Err(_) => return,
};
- let lineage = CheckpointLineage::new(PodOrdinal::STANDALONE);
+ let Ok(lineage) = config.checkpoint_lineage() else {
+ return;
+ };
let prefix = ObjPath::from(lineage.remote_dir(&d.s3_key_prefix).as_str());
let mut stream = store.list(Some(&prefix));
while let Some(entry) = stream.next().await {
@@ -1447,11 +1453,7 @@ async fn s3_restore_replays_past_the_checkpoint_after_the_broker_moved_on() {
// --- Tenure 1: fold, checkpoint through the sweeper, then fold and commit past it. ---
{
- let admitted = admit(
- config.partition_ownership().expect("a standalone split"),
- &groups,
- )
- .await;
+ let admitted = admit(config.split_claim().expect("a standalone split"), &groups).await;
let store = CohortStore::open(&config.store_config(admitted.ownership()))
.expect("open tenure-1 store");
let claimed = claim(admitted, &groups).await;
@@ -1543,11 +1545,7 @@ async fn s3_restore_replays_past_the_checkpoint_after_the_broker_moved_on() {
.expect("remove checkpoint_local_dir (simulate PVC loss)");
// --- Tenure 2: the boot restores from S3, then the consumer finishes the restore. ---
- let admitted = admit(
- config.partition_ownership().expect("a standalone split"),
- &groups,
- )
- .await;
+ let admitted = admit(config.split_claim().expect("a standalone split"), &groups).await;
let (store2, restore) = open_store(&config, &admitted, Some(lineage), groups.clone())
.await
.expect("open the restored store");
@@ -1598,7 +1596,7 @@ async fn s3_restore_replays_past_the_checkpoint_after_the_broker_moved_on() {
let shutdown_handle = handle.clone();
let _monitor = manager.monitor_background();
let claimed = claim(admitted, &groups).await;
- let consumer = build_consumer_finishing_restore(
+ let (consumer, _dispatcher) = build_consumer_finishing_restore(
&topic,
&group,
store2.clone(),
@@ -1678,3 +1676,337 @@ async fn s3_restore_replays_past_the_checkpoint_after_the_broker_moved_on() {
}
delete_s3_prefix(&config).await;
}
+
+/// One matching `$pageview` for each of `persons`, at source offsets from `first_offset`. Returns
+/// the total produced.
+async fn produce_persons(
+ topic: &str,
+ persons: std::ops::RangeInclusive<u128>,
+ first_offset: i64,
+) -> usize {
+ let producer: FutureProducer = ClientConfig::new()
+ .set("bootstrap.servers", bootstrap_servers())
+ .set("message.timeout.ms", "10000")
+ .create()
+ .expect("create producer");
+ let mut total = 0;
+ for (n, offset) in persons.zip(first_offset..) {
+ let p = person(n);
+ let key = format!("{TEAM}:{p}");
+ producer
+ .send(
+ FutureRecord::to(topic)
+ .key(&key)
+ .payload(&envelope(p, 0, offset)),
+ Timeout::After(Duration::from_secs(10)),
+ )
+ .await
+ .expect("produce event");
+ total += 1;
+ }
+ total
+}
+
+/// Each of the first `persons` whose state on one of `partitions` entered the cohort, with that
+/// partition.
+fn entered_by_partition(
+ store: &CohortStore,
+ partitions: impl Iterator<Item = u16> + Clone,
+ lsk: LeafStateKey,
+ persons: u128,
+) -> BTreeMap<u128, u16> {
+ (1..=persons)
+ .filter_map(|n| {
+ partitions.clone().find_map(|partition| {
+ let key = BehavioralKey::new(partition, TEAM as u64, person(n), lsk);
+ let state = store
+ .get_behavioral(&key)
+ .unwrap()
+ .map(|bytes| StatefulRecord::decode(&bytes).unwrap().state);
+ matches!(
+ state,
+ Some(Stage1State::BehavioralSingle {
+ has_match: true,
+ ..
+ })
+ )
+ .then_some((n, partition))
+ })
+ })
+ .collect()
+}
+
+/// Pod `ordinal` of two, resharded from one pod, with its own volume under `root`.
+fn reshard_config(
+ root: &std::path::Path,
+ ordinal: u16,
+ prefix: &str,
+ topics: &RestoreTopics,
+) -> Config {
+ // Kubernetes mounts the pod's volume before the process starts.
+ let volume = root.join(format!("pod-{ordinal}"));
+ std::fs::create_dir_all(&volume).expect("create the pod's volume");
+ let mut config = s3_restore_config(
+ &volume.join("db"),
+ &volume.join("checkpoints"),
+ prefix,
+ topics,
+ );
+ config.pod_name = Some(format!("cohort-stream-processor-{ordinal}"));
+ config.cohort_pod_count = NonZeroU16::new(2).unwrap();
+ config.cohort_reshard_from_pod_count = NonZeroU16::new(1);
+ config
+}
+
+/// The split procedure through a real broker and S3. One pod folds every person and takes a
+/// checkpoint, then stops. Two pods start with the reshard marker. Pod 0 reopens its store and drops
+/// the odd partitions. Pod 1 waits for pod 0's claim, restores pod 0's checkpoint, and drops the even
+/// ones. Every person the single pod held stays entered on the one pod that owns its partition, the
+/// events after the split land on both, every commit carries its owner's stamp, and pod 1 uploads in
+/// full under its own lineage.
+// Multi-threaded: finishing the restore commits with `block_in_place`.
+#[tokio::test(flavor = "multi_thread")]
+#[ignore = "requires a running Kafka broker (KAFKA_HOSTS) AND an S3-compatible store (CHECKPOINT_S3_*); see module docs"]
+async fn resharding_one_pod_into_two_keeps_every_member() {
+ const BEFORE: u128 = PERSONS + NEW_PERSONS;
+ const AFTER: u128 = BEFORE + 20;
+ let topics = RestoreTopics::unique();
+ let topic = topics.events.clone();
+ let group = topics.group.clone();
+ let prefix = format!("cohort-stream-checkpoints-itest/{}", Uuid::new_v4());
+ for name in topics.all() {
+ create_topic(name).await;
+ }
+ let produced = produce_events(&topic).await + produce_new_persons(&topic, NEW_PERSONS).await;
+
+ let root = TempDir::new().unwrap();
+ let single = {
+ let volume = root.path().join("pod-0");
+ s3_restore_config(
+ &volume.join("db"),
+ &volume.join("checkpoints"),
+ &prefix,
+ &topics,
+ )
+ };
+ let groups = Arc::new(InputGroups::new(&single));
+ let lsk = behavioral_lsk(&behavioral_catalog());
+ let verifier: StreamConsumer = ClientConfig::new()
+ .set("bootstrap.servers", bootstrap_servers())
+ .set("group.id", &group)
+ .set("enable.auto.commit", "false")
+ .create()
+ .expect("create verifier consumer");
+ let every_partition = || 0..NUM_PARTITIONS as u16;
+
+ // --- Tenure 1, one pod: fold every person, checkpoint to S3, stop. ---
+ let control = {
+ let admitted = admit(single.split_claim().unwrap(), &groups).await;
+ let store = CohortStore::open(&single.store_config(admitted.ownership()))
+ .expect("open the single pod's store");
+ let claimed = claim(admitted, &groups).await;
+ let mut manager = Manager::builder("reshard-itest-1")
+ .with_trap_signals(false)
+ .build();
+ let handle = manager.register(
+ "consumer",
+ ComponentOptions::new().with_graceful_shutdown(Duration::from_secs(15)),
+ );
+ let shutdown = handle.clone();
+ let _monitor = manager.monitor_background();
+ let (consumer, dispatcher) = build_consumer_with_restore(
+ &topic,
+ &group,
+ store.clone(),
+ behavioral_catalog(),
+ handle,
+ Arc::new(CaptureSink::new()),
+ Duration::from_millis(250),
+ &claimed,
+ true,
+ 100,
+ );
+ let task = tokio::spawn(consumer.process());
+ wait_for_committed(&verifier, &topic, produced as i64, Duration::from_secs(60)).await;
+ CheckpointSweeper::new(
+ store.clone(),
+ dispatcher,
+ groups.clone(),
+ claimed,
+ single.durability_config(),
+ 1,
+ tokio_util::sync::CancellationToken::new(),
+ )
+ .run_once()
+ .await;
+ shutdown.request_shutdown();
+ task.await.expect("the single pod panicked");
+ entered_by_partition(&store, every_partition(), lsk, BEFORE)
+ };
+ assert_eq!(
+ control.len(),
+ BEFORE as usize,
+ "the single pod entered every person"
+ );
+
+ // --- Tenure 2, two pods. The parent admits over its own stamps and reopens its store. ---
+ let parent_config = reshard_config(root.path(), 0, &prefix, &topics);
+ let child_config = reshard_config(root.path(), 1, &prefix, &topics);
+ let admitted = admit(parent_config.split_claim().unwrap(), &groups).await;
+ assert_eq!(admitted.source(), StoreSource::Own);
+ let (parent_store, restore) = open_store(
+ &parent_config,
+ &admitted,
+ Some(parent_config.checkpoint_lineage().unwrap()),
+ groups.clone(),
+ )
+ .await
+ .expect("the parent reopens its store");
+ assert!(restore.is_none(), "the parent keeps its live store");
+ let parent_claimed = claim(admitted, &groups).await;
+
+ // The child takes over once the parent has claimed under the new split.
+ let admitted = admit(child_config.split_claim().unwrap(), &groups).await;
+ assert_eq!(
+ admitted.source(),
+ StoreSource::TakeOver {
+ parent: PartitionOwnership::single(NUM_PARTITIONS as u16),
+ }
+ );
+ let child_lineage = child_config.checkpoint_lineage().unwrap();
+ let (child_store, restore) = open_store(
+ &child_config,
+ &admitted,
+ Some(child_lineage),
+ groups.clone(),
+ )
+ .await
+ .expect("the child restores its parent's checkpoint");
+ let restore = restore.expect("the child restores from the parent's lineage");
+ let child_claimed = claim(admitted, &groups).await;
+
+ let mut manager = Manager::builder("reshard-itest-2")
+ .with_trap_signals(false)
+ .build();
+ let parent_handle = manager.register(
+ "parent",
+ ComponentOptions::new().with_graceful_shutdown(Duration::from_secs(15)),
+ );
+ let child_handle = manager.register(
+ "child",
+ ComponentOptions::new().with_graceful_shutdown(Duration::from_secs(15)),
+ );
+ let shutdown = parent_handle.clone();
+ let _monitor = manager.monitor_background();
+ let (parent, _) = build_consumer_with_restore(
+ &topic,
+ &group,
+ parent_store.clone(),
+ behavioral_catalog(),
+ parent_handle,
+ Arc::new(CaptureSink::new()),
+ Duration::from_millis(250),
+ &parent_claimed,
+ true,
+ 100,
+ );
+ let (child, child_dispatcher) = build_consumer_finishing_restore(
+ &topic,
+ &group,
+ child_store.clone(),
+ behavioral_catalog(),
+ child_handle,
+ Arc::new(CaptureSink::new()),
+ Duration::from_millis(250),
+ restore,
+ &child_claimed,
+ );
+ let tasks = [
+ tokio::spawn(parent.process()),
+ tokio::spawn(child.process()),
+ ];
+
+ let produced = produced + produce_persons(&topic, BEFORE + 1..=AFTER, 1_000).await;
+ wait_for_committed(&verifier, &topic, produced as i64, Duration::from_secs(60)).await;
+ CheckpointSweeper::new(
+ child_store.clone(),
+ child_dispatcher,
+ groups.clone(),
+ child_claimed,
+ child_config.durability_config(),
+ 1,
+ tokio_util::sync::CancellationToken::new(),
+ )
+ .run_once()
+ .await;
+ shutdown.request_shutdown();
+ for task in tasks {
+ task.await.expect("a pod panicked");
+ }
+
+ let owned = |ordinal: u16| every_partition().filter(move |p| p % 2 == ordinal);
+ for (ordinal, store) in [(0, &parent_store), (1, &child_store)] {
+ let given_away: Vec<u16> = owned(1 - ordinal)
+ .filter(|&p| partition_has_state(store, i32::from(p), lsk, AFTER))
+ .collect();
+ assert!(
+ given_away.is_empty(),
+ "pod {ordinal} still holds the partitions it gave away: {given_away:?}",
+ );
+ }
+ let mut entered = entered_by_partition(&parent_store, owned(0), lsk, AFTER);
+ entered.extend(entered_by_partition(&child_store, owned(1), lsk, AFTER));
+ let kept: BTreeMap<u128, u16> = entered
+ .iter()
+ .filter(|&(&n, _)| n <= BEFORE)
+ .map(|(&n, &p)| (n, p))
+ .collect();
+ assert_eq!(kept, control, "every member stays on the partition it had");
+ assert_eq!(
+ entered.keys().copied().collect::<BTreeSet<u128>>(),
+ (1..=AFTER).collect(),
+ "the events after the split entered on both pods",
+ );
+ assert!(entered.values().any(|&p| p % 2 == 1) && entered.values().any(|&p| p % 2 == 0));
+
+ let ledger = groups
+ .ledger(&every_partition().collect())
+ .expect("read the ledger");
+ for input in ledger.inputs() {
+ for (&partition, committed) in input.commits() {
+ let owner = PartitionOwnership::new(
+ NonZeroU16::new(2).unwrap(),
+ PodOrdinal::from(partition % 2),
+ NUM_PARTITIONS as u16,
+ )
+ .unwrap();
+ assert_eq!(
+ committed.stamp(),
+ Some(&Stamp::Split(owner)),
+ "{} partition {partition}",
+ input.topic(),
+ );
+ }
+ }
+
+ let uploaded: serde_json::Value = serde_json::from_slice(
+ &std::fs::read(
+ child_lineage
+ .local_dir(std::path::Path::new(&child_config.checkpoint_local_dir))
+ .join("uploaded.json"),
+ )
+ .expect("the child uploads a checkpoint"),
+ )
+ .unwrap();
+ assert_eq!(uploaded["metadata"]["ordinal"], 1, "under its own lineage");
+ assert_eq!(
+ uploaded["metadata"]["full_upload_at"], uploaded["metadata"]["attempt_timestamp"],
+ "the child's first upload is full, so its chain names nothing in the parent's lineage",
+ );
+
+ for name in topics.all() {
+ delete_topic(name).await;
+ }
+ delete_s3_prefix(&single).await;
+ delete_s3_prefix(&child_config).await;
+}
diff --git a/rust/cohort-stream-processor/tests/merge_consumer.rs b/rust/cohort-stream-processor/tests/merge_consumer.rs
index 6f2368770d29..86ee2faf92e2 100644
--- a/rust/cohort-stream-processor/tests/merge_consumer.rs
+++ b/rust/cohort-stream-processor/tests/merge_consumer.rs
@@ -44,7 +44,7 @@ use cohort_stream_processor::merge::transfer::{
use cohort_stream_processor::partitions::{
merge_partition_key, partition_of, AdmittedSplit, ClaimedSplit, CohortConsumerContext,
Follower, FollowerSet, InputGroups, OffsetTracker, PartitionOwnership, PartitionRouter,
- COHORT_PARTITION_COUNT,
+ SplitClaim, COHORT_PARTITION_COUNT,
};
use cohort_stream_processor::producer::{
CaptureCascadeSink, CaptureSink, CohortMembershipChange, KafkaMembershipSink,
@@ -565,9 +565,10 @@ impl Groups {
/// Admits and claims `ownership` through the ownership fence on the groups `config` names, as the
/// processor does at boot.
async fn claim(config: Config, ownership: PartitionOwnership) -> ClaimedSplit {
+ let claim = SplitClaim::new(ownership, None).expect("a split outside a reshard");
tokio::task::spawn_blocking(move || {
let groups = InputGroups::new(&config);
- AdmittedSplit::try_admit(ownership, &groups)
+ AdmittedSplit::try_admit(claim, &groups)
.expect("the fence admits the split")
.claim(&groups)
.expect("claim the owned partitions")
diff --git a/rust/cohort-stream-processor/tests/merge_durability.rs b/rust/cohort-stream-processor/tests/merge_durability.rs
index b18b795ac376..682e4ff70685 100644
--- a/rust/cohort-stream-processor/tests/merge_durability.rs
+++ b/rust/cohort-stream-processor/tests/merge_durability.rs
@@ -636,12 +636,12 @@ impl Instance {
/// The ownership fence's admission of the split `config` names, which the processor runs before it
/// opens the store.
async fn admit(config: &Config, groups: &Arc<InputGroups>) -> AdmittedSplit {
- let ownership = config
- .partition_ownership()
+ let claim = config
+ .split_claim()
.expect("the pod name ends in an ordinal");
let groups = groups.clone();
tokio::task::spawn_blocking(move || {
- AdmittedSplit::try_admit(ownership, &groups).expect("the fence admits the split")
+ AdmittedSplit::try_admit(claim, &groups).expect("the fence admits the split")
})
.await
.expect("the admission panicked")
diff --git a/rust/cohort-stream-processor/tests/ownership_fence.rs b/rust/cohort-stream-processor/tests/ownership_fence.rs
index a9000f3811ae..05df26816175 100644
--- a/rust/cohort-stream-processor/tests/ownership_fence.rs
+++ b/rust/cohort-stream-processor/tests/ownership_fence.rs
@@ -1,5 +1,6 @@
//! The ownership fence against a **real** Kafka broker: commit stamps round-trip on every input
-//! group, and the run-time audit stops a pod when another split commits one of its partitions.
+//! group, a pod of another split waits until the split is consistent, and the run-time audit stops
+//! a pod when another split commits one of its partitions.
//!
//! `#[ignore]`d by default: the fence reads Kafka commit metadata, which the in-process
//! `MockCluster` does not keep. Run against a local stack with:
@@ -18,8 +19,8 @@ use std::time::Duration;
use cohort_stream_processor::config::Config;
use cohort_stream_processor::partitions::{
- AdmittedSplit, Committed, InputGroups, OwnershipAudit, PartitionOwnership, PositionsError,
- ResumeOffset, Stamp,
+ AdmittedSplit, Committed, FenceBlock, FenceError, InputGroups, OwnershipAudit,
+ PartitionOwnership, PositionsError, ResumeOffset, SplitClaim, Stamp, StoreSource,
};
use cohort_stream_processor::store::durability::PodOrdinal;
use cohort_stream_processor::sweep::Sweeper;
@@ -46,6 +47,10 @@ fn split(pods: u16, ordinal: u16) -> PartitionOwnership {
.unwrap()
}
+fn claim(pods: u16, ordinal: u16, reshard_from: Option<u16>) -> SplitClaim {
+ SplitClaim::new(split(pods, ordinal), reshard_from.and_then(NonZeroU16::new)).unwrap()
+}
+
/// The test's own events, merges and transfers topics, each read through its own group.
struct Inputs {
events: String,
@@ -162,9 +167,11 @@ async fn commit_stamps_round_trip_on_every_group() {
);
assert_eq!(events.commits()[&0], Committed::Never);
- let admitted =
- AdmittedSplit::try_admit(split(1, 0), &groups).expect("a commit without a stamp admits");
- let stale = AdmittedSplit::try_admit(split(2, 0), &groups).expect("nothing is stamped yet");
+ let admitted = AdmittedSplit::try_admit(claim(1, 0, None), &groups)
+ .expect("a commit without a stamp admits");
+ assert_eq!(admitted.source(), StoreSource::Own);
+ let stale =
+ AdmittedSplit::try_admit(claim(2, 0, None), &groups).expect("nothing is stamped yet");
admitted
.claim(&groups)
.expect("the claim reads its stamps back");
@@ -198,6 +205,50 @@ async fn commit_stamps_round_trip_on_every_group() {
inputs.delete().await;
}
+/// A pod of a new split waits while the old split's stamps say its pods may still run. Without a
+/// reshard marker it never admits. With one, it waits until its parent restarts under the new split,
+/// then takes the partitions over, and after its own claim it keeps what it took over.
+#[tokio::test]
+#[ignore = "requires a running Kafka broker (KAFKA_HOSTS); run with --ignored against a local stack"]
+async fn a_pod_of_another_split_waits_until_the_split_is_consistent() {
+ let inputs = Inputs::create().await;
+ let groups = inputs.groups();
+ let admit = |claim| AdmittedSplit::try_admit(claim, &groups);
+ admit(claim(1, 0, None))
+ .unwrap()
+ .claim(&groups)
+ .expect("the single pod claims every partition");
+
+ assert!(matches!(
+ admit(claim(2, 1, None)),
+ Err(FenceError::Blocked(FenceBlock::ForeignSplit { found, .. })) if found == split(1, 0)
+ ));
+ assert!(matches!(
+ admit(claim(2, 1, Some(1))),
+ Err(FenceError::Blocked(FenceBlock::ParentRunning { parent })) if parent == split(1, 0)
+ ));
+
+ let parent = admit(claim(2, 0, Some(1))).expect("the parent admits over its own stamps");
+ assert_eq!(parent.source(), StoreSource::Own);
+ parent.claim(&groups).unwrap();
+
+ let child = admit(claim(2, 1, Some(1))).expect("the parent restarted under the new split");
+ assert_eq!(
+ child.source(),
+ StoreSource::TakeOver {
+ parent: split(1, 0)
+ }
+ );
+ child.claim(&groups).unwrap();
+ assert_eq!(
+ admit(claim(2, 1, Some(1))).unwrap().source(),
+ StoreSource::Own,
+ "a child that restarts after its claim finished its takeover",
+ );
+
+ inputs.delete().await;
+}
+
/// The audit lets a commit without a stamp pass, as an operator's offset reset writes one, and stops
/// the pod when another split commits one of its partitions. It also revokes the pod's commits, so
/// the final commits of the stop cannot cover the foreign stamp, and the next boot holds. A
@@ -207,7 +258,7 @@ async fn commit_stamps_round_trip_on_every_group() {
async fn the_audit_stops_a_pod_whose_partition_another_split_commits() {
let inputs = Inputs::create().await;
let groups = inputs.groups();
- let claimed = AdmittedSplit::try_admit(split(1, 0), &groups)
+ let claimed = AdmittedSplit::try_admit(claim(1, 0, None), &groups)
.unwrap()
.claim(&groups)
.unwrap();
@@ -252,7 +303,7 @@ async fn the_audit_stops_a_pod_whose_partition_another_split_commits() {
.commit(&committer(&inputs.merges_group()), CommitMode::Sync);
assert!(final_commit.is_err(), "the audit revokes the pod's commits");
assert!(
- AdmittedSplit::try_admit(split(1, 0), &groups).is_err(),
+ AdmittedSplit::try_admit(claim(1, 0, None), &groups).is_err(),
"the next boot holds in the fence",
);
let captured = groups.owned_positions(&claimed, &every_partition());
diff --git a/rust/cohort-stream-processor/tests/seed_consumer.rs b/rust/cohort-stream-processor/tests/seed_consumer.rs
index e9feb66f115b..9193c5e2211c 100644
--- a/rust/cohort-stream-processor/tests/seed_consumer.rs
+++ b/rust/cohort-stream-processor/tests/seed_consumer.rs
@@ -39,7 +39,7 @@ use cohort_stream_processor::partitions::{
merge_partition_key, partition_for, partition_of, AdmittedSplit, ClaimedSplit,
CohortConsumerContext, ConsumerPauser, Follower, FollowerSet, InputGroups, LiveWatermarks,
OffsetTracker, PartitionOwnership, PartitionPauser, PartitionRouter, SeedPacingConfig,
- COHORT_PARTITION_COUNT,
+ SplitClaim, COHORT_PARTITION_COUNT,
};
use cohort_stream_processor::producer::{
CascadeSink, ChangeOrigin, CohortMembershipChange, KafkaCascadeSink, KafkaMembershipSink,
@@ -554,9 +554,10 @@ impl Groups {
/// Admits and claims `ownership` through the ownership fence on the groups `config` names, as the
/// processor does at boot.
async fn claim(config: Config, ownership: PartitionOwnership) -> ClaimedSplit {
+ let claim = SplitClaim::new(ownership, None).expect("a split outside a reshard");
tokio::task::spawn_blocking(move || {
let groups = InputGroups::new(&config);
- AdmittedSplit::try_admit(ownership, &groups)
+ AdmittedSplit::try_admit(claim, &groups)
.expect("the fence admits the split")
.claim(&groups)
.expect("claim the owned partitions")