Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(cohorts): own processor partitions statically by pod ordinal (#118195)

Matheus Batista5af0887004ee
diff --git a/docs/internal/realtime-cohorts/README.md b/docs/internal/realtime-cohorts/README.md
index c873708960d0..2cbef2e41dbc 100644
--- a/docs/internal/realtime-cohorts/README.md
+++ b/docs/internal/realtime-cohorts/README.md
@@ -90,7 +90,8 @@ When all partitions have reported, Django stamps the cohort ready, and feature f
Reconcile re-emits from the state the processor holds, so it repairs lost output, not lost state.
[Processor runtime](processor-runtime.md#delivery-semantics) lists each path's guarantee.
- **The processor runs as a single pod.**
- Its state lives in a local RocksDB store that never moves between pods, so a second replica would corrupt state without any error.
+ Each pod owns a fixed share of the partitions, and keeps their state in a local RocksDB store that never moves between pods.
+ A second replica refuses to start unless `COHORT_POD_COUNT` is raised and every store is wiped, and then every slice starts without its history.
- **Most of the pipeline is off by default.**
The processor writes to a shadow output topic unless configured otherwise, and seed apply, reconcile, cascades, durable restore, the seeder's completion driver and Django's finalizer each have their own switch.
The pages name the switch where each behavior is described.
@@ -151,7 +152,7 @@ Each page states the setup its example needs, and the details differ where a pag
| Sweep | The periodic pass that applies eviction deadlines |
| Tombstone | A forwarding address from a merged-away person to the survivor |
| Cascade | Recomposing cohorts that reference a cohort whose membership flipped |
-| Tenure | The time one worker owns a partition, from assignment to the next revoke or restart |
+| Tenure | The time one worker owns a partition, from boot to the next restart |
| Run | One backfill of one kind, for one cohort or a whole team |
| Participation | One cohort's membership in a run, with its pinned definition and outcome |
| Pinning | Freezing the definition a run replays |
diff --git a/docs/internal/realtime-cohorts/invariants.md b/docs/internal/realtime-cohorts/invariants.md
index 62734f8e5f4e..ee1a2a0c69f8 100644
--- a/docs/internal/realtime-cohorts/invariants.md
+++ b/docs/internal/realtime-cohorts/invariants.md
@@ -58,10 +58,7 @@ Most of these are not enforced by a single test, and a violation usually shows u
- Why: reads must see the worker's own previous writes, with no locks or snapshots.
- Kept by: every event, merge, cascade, sweep, backfill and reconcile step for the partition runs on its worker.
- A revoke wipes a slice after its worker exits, and a partition that moves in after boot is wiped just before its worker spawns.
-- Limit: with durable restore on, nothing makes boot recovery finish before workers start.
- The events consumer dispatches polls that arrive before its assignment settles, and the followers start once the catalog loads, so a worker can run while the boot redrive clears outbox rows.
- This is a gap in the ordering, not a reproduced failure.
+ Boot recovery wipes unowned slices and redrives the outbox before the first poll, and workers spawn only after it.
See [processor runtime](processor-runtime.md#startup).
**8. A committed offset never runs ahead of state on disk.**
@@ -258,7 +255,7 @@ Most of these are not enforced by a single test, and a violation usually shows u
| Merge membership output, composed sweep output, first-hop cascades from the live, merge and sweep paths | At most once | Reconcile |
| Single-leaf sweep output | At least once while the worker runs, and across a restart only with durable restore on | Retries within the path |
| Cascade messages consumed | At least once | Holds and replays |
-| Merge state and its transfer | At least once while the partition stays assigned. A revoke deletes a pending transfer | The outbox and its redrive |
+| Merge state and its transfer | At least once while the slice survives. A store wipe deletes a pending transfer | The outbox and its redrive |
| Seeds and reconcile, including their first-hop cascades | At least once | Replays after holds |
| Membership consumer | At least once, applied idempotently | The version guard, when version writes are on |
diff --git a/docs/internal/realtime-cohorts/merges-and-cascades.md b/docs/internal/realtime-cohorts/merges-and-cascades.md
index fb45209d6822..b82159844b53 100644
--- a/docs/internal/realtime-cohorts/merges-and-cascades.md
+++ b/docs/internal/realtime-cohorts/merges-and-cascades.md
@@ -128,7 +128,7 @@ Both are settings sized to the topic retentions, and nothing checks that they st
The outbox is never garbage-collected.
Until the transfer is acknowledged, it is the only copy of the old person's state.
-A partition revoke still deletes it with the rest of the slice.
+A store wipe still deletes it with the rest of the slice.
### Worked example
@@ -199,7 +199,7 @@ That answer is wrong for anyone who really is in the untracked cohort.
4. For each referrer that flips, it produces the membership change and, if the chain may continue, a further cascade message.
5. Only after both produces are acknowledged does it write the referrers' new Stage 2 bits and mark the offset.
A failure pins the partition's cascade commit at that message.
- Kafka redelivers it after the next restart or rebalance, and the redelivered message recomputes the same flip.
+ Kafka redelivers it after the next restart, and the redelivered message recomputes the same flip.
Until then, cascade commits for that partition stall.
Three limits bound the work:
diff --git a/docs/internal/realtime-cohorts/processor-runtime.md b/docs/internal/realtime-cohorts/processor-runtime.md
index 987adc01e395..e630e5faa61d 100644
--- a/docs/internal/realtime-cohorts/processor-runtime.md
+++ b/docs/internal/realtime-cohorts/processor-runtime.md
@@ -37,28 +37,39 @@ flowchart TB
Timers feed the same workers.
The eviction sweep, the pending-transfer redrive, merge garbage collection and the reconcile drain all send messages through the router instead of touching the store themselves.
So while a worker runs, it is the only writer of its partition, and nothing needs a lock.
-Boot recovery and revoke cleanup are the only code that touches a slice from outside its worker.
-Revoke cleanup waits for the worker to exit.
-Boot recovery finishes before any worker starts, as [startup](#startup) explains.
+Boot recovery is the only code that touches a slice from outside its worker.
+It finishes before any worker starts, as [startup](#startup) explains.
## Consumers and ownership
-The **events consumer** is the only consumer that joins a Kafka consumer group with a real assignment.
-It reads `cohort_stream_events` with cooperative-sticky assignment and static membership, so a quick restart reclaims the same partitions without a rebalance.
+Each pod owns a fixed set of partitions for as long as it runs.
+`COHORT_POD_COUNT` sets how many pods split the partitions, and the ordinal at the end of the StatefulSet pod name in `POD_NAME` says which pod this is.
+Pod `i` owns every partition `p` with `p mod COHORT_POD_COUNT = i`.
+The pod refuses to start unless `COHORT_POD_COUNT` divides `COHORT_PARTITION_COUNT` and the ordinal is below `COHORT_POD_COUNT`.
+The default is one pod, which owns every partition.
+Only a single pod may run without `POD_NAME`, and it counts as ordinal 0.
+The **events consumer** reads `cohort_stream_events`.
**Follower consumers** read the other co-partitioned topics.
Merges and merge transfers always have one.
Cascades and seeds have one when those features are enabled.
-Followers never subscribe.
-Whenever the events consumer is assigned or revoked a partition, the followers are assigned or unassigned the same partition number, resuming at their group's committed offsets.
-One rebalance decides ownership of partition N on every input topic at once, which is what [partition affinity](event-routing.md#partition-affinity) needs.
+No consumer subscribes to its group.
+At startup each one is assigned the owned partitions, resuming at its group's committed offsets.
+So the pod owns partition N on every input topic at once, which is what [partition affinity](event-routing.md#partition-affinity) needs, and no rebalance moves a partition while the pod runs.
+The consumer groups have no members and only hold the committed offsets.
+A broker can expire the offsets of a group without members, so every consumer resets to `earliest`: a partition that lost its offset replays instead of skipping to the tail.
+A commit from a client outside the group is refused while the group has a live member, such as the static member an older build left behind until its session expires.
+librdkafka logs each refused commit as a `COMMITFAIL` warning, and the next commit retries it.
+`rebalances_total` counts rebalance callbacks on the events consumer, which fire only if the consumer subscribes.
+`owned_partitions` and `partition_ownership_info`, labelled by `pods` and `ordinal`, show the pod's share.
+No pod can see the StatefulSet's replica count, so fewer replicas than `COHORT_POD_COUNT` leave the partitions of the missing ordinals without a consumer.
+Only their group lag and a `sum(owned_partitions)` below `COHORT_PARTITION_COUNT` show it, so alert on that sum.
The merge, transfer, cascade and seed followers start only after the first catalog load and the end of boot recovery.
The events consumer does not wait for the catalog.
-Workers spawn lazily, on the first message for an owned partition.
-With durable restore enabled, the end of boot recovery also spawns a worker for every owned partition.
-A **tenure** is the time a worker owns a partition, from an assignment to the next revoke or restart.
+The end of boot spawns a worker for every owned partition.
+A **tenure** is the time a worker owns a partition, from boot to the next restart.
## From a Kafka batch to a worker
@@ -143,19 +154,19 @@ Hot-path writes do not wait for the disk, so this one flush per commit is what m
Different paths make different promises.
This table is worth knowing before you debug a missing or duplicated membership change.
-| Path | Guarantee | Why |
-| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Events into Stage 1 state | At least once across restarts, applied once. Within a tenure, an event whose fold fails on a store error, or whose re-key to a survivor's partition fails, is skipped and later committed past | Offsets commit after the work. A redelivered event is recognized by its replay marks. Only the merge, cascade and seed paths hold a failed offset |
-| Event to membership change | At most once | State commits inside the fold, before the produce. If the produce fails, the next clean sub-batch marks past it, and a replay finds no transition to emit |
-| First-hop cascade messages | At most once from the live, merge and sweep paths. At least once from seed apply and reconcile | The live, merge and sweep paths produce them after the flip is committed. Seed apply and reconcile produce them before writing the flip, and retry on failure |
-| Sweep expiry of a single-leaf cohort | At least once while the worker runs | The sweep produces before it writes, and reschedules the keys on failure. The retry lives in the in-memory queue, so a restart keeps it only with durable restore on |
-| Sweep expiry that recomposes a cohort | At most once | The composed result is written before it is produced |
-| Cascade messages consumed | At least once | Produce before state, and a failure holds the offset |
-| Merge state | At least once, applied once | Drain and apply markers keyed by the original merge message |
-| Merge state transfer | At least once while the partition stays assigned | An outbox row survives until the transfer is acknowledged, and a timer redrives it. A revoke deletes the row with the slice, and later merges may already have committed past its merge offset |
-| Merge membership output | At most once | State commits before the produce, and a failed produce is dropped |
-| Backfill seed runs | At least once | A failure holds the run's first offset, and the seed replays on the next tenure. A redelivered seed re-emits a change that was lost |
-| Reconcile | At least once | A failed page retries on the next tick, and a deferred floor keeps the request uncommitted until the job completes |
+| Path | Guarantee | Why |
+| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Events into Stage 1 state | At least once across restarts, applied once. Within a tenure, an event whose fold fails on a store error, or whose re-key to a survivor's partition fails, is skipped and later committed past | Offsets commit after the work. A redelivered event is recognized by its replay marks. Only the merge, cascade and seed paths hold a failed offset |
+| Event to membership change | At most once | State commits inside the fold, before the produce. If the produce fails, the next clean sub-batch marks past it, and a replay finds no transition to emit |
+| First-hop cascade messages | At most once from the live, merge and sweep paths. At least once from seed apply and reconcile | The live, merge and sweep paths produce them after the flip is committed. Seed apply and reconcile produce them before writing the flip, and retry on failure |
+| Sweep expiry of a single-leaf cohort | At least once while the worker runs | The sweep produces before it writes, and reschedules the keys on failure. The retry lives in the in-memory queue, so a restart keeps it only with durable restore on |
+| Sweep expiry that recomposes a cohort | At most once | The composed result is written before it is produced |
+| Cascade messages consumed | At least once | Produce before state, and a failure holds the offset |
+| Merge state | At least once, applied once | Drain and apply markers keyed by the original merge message |
+| Merge state transfer | At least once while the slice survives | An outbox row survives until the transfer is acknowledged, and a timer redrives it. A wiped slice loses the row, and later merges may already have committed past its merge offset |
+| Merge membership output | At most once | State commits before the produce, and a failed produce is dropped |
+| Backfill seed runs | At least once | A failure holds the run's first offset, and the seed replays on the next tenure. A redelivered seed re-emits a change that was lost |
+| Reconcile | At least once | A failed page retries on the next tick, and a deferred floor keeps the request uncommitted until the job completes |
So live, merge, sweep-recompose and their first-hop cascade output can all lose a membership change, and nothing on those paths puts it back.
The pipeline accepts that and relies on reconcile, run as part of every backfill, to re-emit the full membership of a cohort.
@@ -168,50 +179,47 @@ Writes reach the disk only at the next commit's flush, while output is acknowled
A host crash that loses unflushed writes rolls state back to the last flush, and the replayed inputs can emit the same changes again.
A wiped store does the same.
-## Rebalance and the single-pod constraint
+## Ownership and the single-pod deployment
-When the events consumer loses a partition, the processor:
+State never moves between pods.
+A pod's store holds the slices of the partitions it owns, and nothing hands a slice to another pod.
+The store records the split it was written under in `cf_meta[pod_partitions]`: the pod count, the ordinal and the partition count.
+A store from before that record takes the configured split as its own the first time it opens.
+A store written under another split refuses to open, so the pod crash-loops instead of reading slices that another pod may have changed since.
+With durable restore enabled, boot deletes the slices of partitions the pod does not own, so a single pod deletes nothing.
-1. stops routing to the partition and unassigns the followers,
-2. lets the worker drain both lanes and exit,
-3. forgets every tracker's progress for the partition without committing it,
-4. deletes the partition's slice of the store.
-
-If a cooperative rebalance hands the partition straight back, the cleanup is skipped and the slice is kept.
-
-State never moves with a partition.
-Whoever owns the partition next, even the same pod later, starts from an empty slice at the last committed offset.
-It rebuilds no history, so every cohort on that partition is wrong until a backfill runs.
-The slice's coverage record goes with it, so the new slice withholds every reconcile whose run boundary is earlier than its start.
-That is why the processor runs as a single pod that owns every partition.
-Nothing moves state between pods, so more than one replica is unsupported, and running two would corrupt state without any error.
+So changing `COHORT_POD_COUNT` requires wiping every store, including the volume of an ordinal that a scale-down removed.
+That volume keeps its old record, so when a later change brings that split back, it opens with slices another pod has moved past.
+With checkpoints enabled, also delete every pod's checkpoints, local and in S3.
+A checkpoint from the old split restores and then refuses to open, so every boot fails the same way.
+Each slice then begins again behind the coverage fence and holds no history until a backfill runs.
+Nothing moves a slice between pods yet, so one pod that owns every partition is the supported deployment.
## Startup
At boot the processor:
-1. validates its configuration and refuses unsafe combinations,
+1. validates its configuration, refuses unsafe combinations, and works out the partitions it owns,
2. binds the health server, so `/_health` answers from here on and `/_ready` reads not ready,
3. loads the catalog once,
-4. decides where the store comes from: with durable restore enabled, the live store on disk or a checkpoint, otherwise an empty store.
- A checkpoint restore that cannot finish blocks here and retries, as [a blocked restore](state-store-and-durability.md#a-blocked-restore) describes,
-5. checks the partition counts of the co-partitioned topics against broker metadata,
+4. checks the partition counts of the co-partitioned topics against broker metadata,
+5. decides where the store comes from: with durable restore enabled, the live store on disk or a checkpoint, otherwise an empty store.
+ A checkpoint restore that cannot finish blocks here and retries, as [a blocked restore](state-store-and-durability.md#a-blocked-restore) describes.
+ A store written under another split of the partitions refuses to open, as [ownership](#ownership-and-the-single-pod-deployment) explains,
6. starts the timer loops and the events consumer, and starts the followers and the checkpoint loop once the catalog has loaded and boot recovery has ended.
By default the store is wiped at boot, so there is nothing to recover, and the events consumer dispatches from its first poll.
With durable restore enabled, a restart reopens the same local store instead, and the events consumer runs boot recovery before it dispatches anything:
-1. It polls until two consecutive polls report the same non-empty assignment.
- It dispatches nothing these polls return, and records the lowest offset it polled on each partition.
-2. It deletes the slices of partitions it does not own, and re-produces every transfer waiting in the merge outbox, clearing each outbox row from outside the worker.
-3. After a checkpoint restore, it lowers each owned partition's resume point to the position the restore kept for it.
-4. It seeks every owned partition back to its resume point, so every event it polled and did not dispatch is fetched again.
- After a checkpoint restore, it then commits those positions and deletes the restore marker.
+1. Before its first poll, it deletes the slices of partitions it does not own, and re-produces every transfer waiting in the merge outbox, clearing each outbox row from outside the worker.
+2. After a checkpoint restore, it seeks every owned partition back to the position the restore kept for it, or to the first event it polled and did not dispatch if that is lower.
+ It dispatches nothing it polls before that seek succeeds.
+ It then commits those positions and deletes the restore marker.
A failed seek, commit or delete is retried on the next poll, and nothing is dispatched until all of them succeed.
The consumer reports no liveness while one of them fails, so a failure that persists restarts the process.
A restored position the broker expired since the restore checked it stops the process instead, so the next boot resets that slice.
-5. It spawns a worker for every owned partition.
+3. It spawns a worker for every owned partition.
Each worker rebuilds its in-memory eviction queue by scanning its slice of behavioral state, so a dormant member on a partition with no traffic still leaves on time.
The followers start only after these steps, and the timers reach only partitions that have a worker.
@@ -223,7 +231,6 @@ So no worker runs while boot recovery writes to the store.
`/_ready` returns 200 only once the catalog has loaded, boot recovery has ended, and no worker is still rebuilding its eviction queue.
Until then it returns 503, and the body names the reason.
The startup and liveness probes use `/_health`, so a slow rebuild never restarts the pod.
-A partition that moves in after boot spawns a worker that rebuilds too, and the pod reads not ready until that rebuild ends.
## Shutdown
@@ -232,7 +239,7 @@ When the last worker exits, the events consumer records the partitions it owned
The merge, transfer, cascade and seed followers wait for that drain before their final commit.
So follower work that finishes during the drain is committed, not replayed on the next boot.
-They commit only for the partitions recorded at the drain, because closing the events consumer revokes every partition from the live ownership.
+They commit only for the partitions recorded at the drain.
A follower that waits longer than 35 seconds commits for the partitions it owned when shutdown began.
With checkpoints enabled, a final checkpoint follows once every consumer and follower has finished.
@@ -272,7 +279,7 @@ Durable restore is enabled.
Processed becomes 1204568.
5. On the next commit tick, the task flushes the write-ahead log and commits 1204568.
-Three variations show the edges.
+Two variations show the edges.
- **Backpressure.** If partition 26 already held more than 1,016 events, the default budget of 1,024 would refuse these 8.
The sub-batch would go to the holdover and fetching for the partition would pause.
@@ -282,8 +289,6 @@ Three variations show the edges.
When the next sub-batch succeeds, processed jumps past 1204567, and the `entered` is never emitted on the live path.
A restart before that would replay 1204567 against the reopened store, see it as a duplicate, and still emit nothing.
Only reconcile repairs it.
-- **Revoke.** If partition 26 were revoked between steps 4 and 5, the commit would never happen.
- The slice would be deleted, and the next owner would start empty and replay from 1204501.
## Optimizations
@@ -309,13 +314,12 @@ Three variations show the edges.
- The events consumer does not wait for the catalog.
If the first catalog load fails, events are consumed, skipped for lack of definitions, and committed.
The pod reads not ready meanwhile.
-- A revoke does not commit.
- Anything processed since the last commit replays on the next owner.
-- A held offset stays held for the rest of the partition's tenure.
- With one pod, that means until the next restart, and the symptom is growing lag on that topic and partition.
+- A held offset stays held for the rest of the partition's tenure, which means until the next restart.
+ The symptom is growing lag on that topic and partition.
- A created store withholds every reconcile whose run boundary predates its slices, because each slice begins when its partition is assigned.
In an environment that wipes the store at every start, that is every run created before the last boot, and `cohort_slices_begun_total` rises at every boot.
[Slice coverage](state-store-and-durability.md#slice-coverage) covers the records and their metrics.
-- A timer message does not spawn a worker.
- Without durable restore that costs nothing, because the store starts empty.
- With durable restore, boot spawns a worker for every owned partition, so overdue evictions run from boot.
+- Nothing fences two processes that run with the same ordinal.
+ Both consume and commit the same partitions, and both produce.
+ A StatefulSet runs one pod per ordinal, so this takes a force-deleted pod that still runs on a node the cluster cannot reach.
+ Do not force-delete processor pods.
diff --git a/docs/internal/realtime-cohorts/seed-apply-and-reconcile.md b/docs/internal/realtime-cohorts/seed-apply-and-reconcile.md
index d8554ca3fa17..0ee3b8b1780a 100644
--- a/docs/internal/realtime-cohorts/seed-apply-and-reconcile.md
+++ b/docs/internal/realtime-cohorts/seed-apply-and-reconcile.md
@@ -137,7 +137,7 @@ Both kinds finish the same way, in an order the code enforces with types.
7. **Mark** the run's offsets, published once the whole quantum has applied, or **hold** the first offset at once if any step failed.
This order is what makes seed output self-healing.
-If the produce in step 5 fails, the offset is held, and the seed is redelivered at the next rebalance or restart.
+If the produce in step 5 fails, the offset is held, and the seed is redelivered at the next restart.
On redelivery the fold changes nothing: for a tile, `max` of the same count is the same count, and for a person seed, its own earlier write makes the verdict a skip.
But the single-leaf Stage 2 rows still hold the replaced value, and the composed cohorts' Stage 2 bits were never written, so both disagree with the truth and the change is emitted again.
Live output has no such repair, which is why live is at most once and seeds are at least once.
@@ -216,9 +216,9 @@ The committable offset is the lowest of the processed offset, any held offset, a
| Failure | Result |
| -------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| A store read or write fails during a run | The run's first offset is held for the rest of the partition's tenure. Later runs still apply, but the commit stays at the hold until a restart or rebalance replays from it |
+| A store read or write fails during a run | The run's first offset is held for the rest of the partition's tenure. Later runs still apply, but the commit stays at the hold until a restart replays from it |
| A run fails before its Stage 1 commit, and a reconcile request for the same cohort follows it on the partition | The reconcile still completes and produces its marker, so the partition is certified without that tile. The tile applies, and emits its change, only when the next tenure replays from the hold |
-| A membership produce fails | Hold. At the next rebalance or restart the redelivered seed emits the change again, through the pre-written row for single-leaf cohorts and the unwritten Stage 2 bit for composed ones |
+| A membership produce fails | Hold. At the next restart the redelivered seed emits the change again, through the pre-written row for single-leaf cohorts and the unwritten Stage 2 bit for composed ones |
| The processor crashes mid-run | Nothing was committed past the run, so it replays. Every step is idempotent |
| A tile is redelivered | `max` of the same count changes nothing |
| The cohort is edited mid-run | Old tiles whose hash still maps to a leaf are max-merged into the current leaves, harmlessly. Old reconcile requests are withheld if the edit changed the shape hash for their kind. Otherwise they walk the current definition |
diff --git a/docs/internal/realtime-cohorts/state-store-and-durability.md b/docs/internal/realtime-cohorts/state-store-and-durability.md
index 402629e16960..13f8c732aac8 100644
--- a/docs/internal/realtime-cohorts/state-store-and-durability.md
+++ b/docs/internal/realtime-cohorts/state-store-and-durability.md
@@ -75,9 +75,7 @@ If the lost write would have been the person's first row, reconcile never sees t
While a partition's worker runs, it is the only writer of its key range.
Merges, cascades, sweeps, garbage collection, backfill and reconcile all run on that worker, so reads always see the worker's own previous writes.
Partition wipes and the boot redrive of the transfer outbox run from outside the worker.
-A revoke wipes a slice after its worker exits, and a partition that moves in after boot is wiped just before its worker spawns.
-The boot redrive is meant to run before workers exist, but nothing enforces that, so a worker can already be running while it clears outbox rows.
-[Processor runtime](processor-runtime.md#startup) describes the gap.
+Both run before the first poll, and workers spawn only after them, as [processor runtime](processor-runtime.md#startup) describes.
The processor needs no locks and no RocksDB snapshots.
## The durability invariant
@@ -143,9 +141,10 @@ With durable restore on, the processor also:
Nothing is dispatched to a worker, from the events consumer or a follower, until these boot steps end.
[Processor runtime](processor-runtime.md#startup) lists them in order.
-When a partition is revoked, its worker drains and exits, and its slice is deleted with its coverage record.
-State never moves between pods, which is why the processor runs as a single pod.
-[Processor runtime](processor-runtime.md#rebalance-and-the-single-pod-constraint) explains that constraint.
+A pod owns its partitions for as long as it runs, so no slice changes owner while the pod runs.
+State never moves between pods, which is why one pod that owns every partition is the supported deployment.
+The store records the split of the partitions it was written under and refuses to open under another.
+[Processor runtime](processor-runtime.md#ownership-and-the-single-pod-deployment) explains that constraint.
### Choosing a checkpoint
@@ -207,7 +206,7 @@ An input whose gate turned on after the capture has no position, so it resumes f
An input that is no longer enabled is ignored.
The check holds at boot, but the consumer still has to read past the oldest retained segment before retention deletes it.
-A consumer whose position falls below the low watermark resets to `latest` for the events topic and `earliest` for the followers, without a trace.
+A consumer whose position falls below the low watermark resets to `earliest` without a trace, so the events that retention deleted are lost.
So keep `CHECKPOINT_IMPORT_WINDOW_HOURS` below the events topic's retention by more than a restore's catch-up time.
A pod down longer than the window then lists no candidate and creates its store behind the coverage fence, which is the safe direction.
The window bounds only the listing: a restore already published resumes at any age, and the checks before and after the download, and when the restore finishes, still catch a position that expired.
@@ -287,7 +286,7 @@ It lives in `cf_person_records` under a 3-byte key, `[partition][0xFE]`, which n
| The store is created: none on disk, a wipe at start, a wipe on schema mismatch, or nothing to restore | The open stamps `cf_meta[slice_coverage]`, with no records. Each slice begins when its partition is assigned |
| A store with the stamp reopens | Each slice resumes its own record |
| A store without the stamp reopens | It predates coverage records. The open writes `complete` for every partition and the stamp in one batch, so it adopts once |
-| A revoke, the boot deletion of an unowned partition, or a move-in wipe | `delete_partition` deletes the record with the slice, and the next worker begins the slice anew from its assignment |
+| The boot deletion of an unowned partition | `delete_partition` deletes the record with the slice, and the next worker begins the slice anew from its assignment |
| A checkpoint restore | The record and the stamp travel inside the checkpoint. A slice the restore cannot replay is reset with its record and begins anew |
A reconcile certifies its partition only when the slice holds the run's history: the slice is complete, or it began at or before the run's boundary.
@@ -354,8 +353,8 @@ A lint rule denies direct calls to most store I/O methods, and the synchronous s
With checkpoints enabled, the boot deletes it and restores.
- A store directory that holds `restore.json` is a checkpoint restore in progress, not a live store.
An image from before that marker ignores it and reopens the store at the broker's old offsets, so do not roll back while a restore is pending.
-- Static group membership keeps a restart from causing a revoke.
- A second group member, or a pod that loses its session or stops polling, still causes one.
- Each revoke deletes the whole slice, including merge transfers not yet sent, whose merge offsets may already be committed.
+- Changing `COHORT_POD_COUNT` makes every store written under another split refuse to open, because each records the split it was written under.
+ [Ownership](processor-runtime.md#ownership-and-the-single-pod-deployment) lists what to wipe.
+ Wiping a store deletes every slice, including merge transfers not yet sent, whose merge offsets may already be committed.
- Person records grow with every person seen on a team with person-property conditions, including those that match nothing, unless a time-to-live is configured.
- The person record time-to-live is judged against event time, so importing old events can write records that are already eligible for expiry.
diff --git a/rust/cohort-stream-processor/Cargo.toml b/rust/cohort-stream-processor/Cargo.toml
index 651929a8df61..e2829220a5fc 100644
--- a/rust/cohort-stream-processor/Cargo.toml
+++ b/rust/cohort-stream-processor/Cargo.toml
@@ -55,8 +55,8 @@ sha2 = { workspace = true }
sqlx = { workspace = true }
thiserror = { workspace = true }
tokio = { workspace = true }
-# `CancellationToken` for the rebalance worker's shutdown signal (from `lifecycle::Handle`), the
-# same primitive the sibling consumers use for cooperative cancellation.
+# `CancellationToken` for the sweep loops' and the checkpoint's shutdown signal (from
+# `lifecycle::Handle`), the same primitive the sibling consumers use for cooperative cancellation.
tokio-util = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true, features = ["env-filter", "json"] }
diff --git a/rust/cohort-stream-processor/clippy.toml b/rust/cohort-stream-processor/clippy.toml
index 7d6c30f97333..fa89894851e6 100644
--- a/rust/cohort-stream-processor/clippy.toml
+++ b/rust/cohort-stream-processor/clippy.toml
@@ -24,7 +24,7 @@ disallowed-methods = [
{ path = "cohort_stream_processor::store::rocks::CohortStore::scan_merge_cf", reason = "async code must scan through StoreHandle or run_section; sync contexts need a module-level allow — see store/handle.rs" },
{ path = "cohort_stream_processor::store::rocks::CohortStore::scan_behavioral", reason = "async code must scan through StoreHandle; sync contexts need a module-level allow — see store/handle.rs" },
{ path = "cohort_stream_processor::store::rocks::CohortStore::scan_behavioral_prefix", reason = "async code must scan through StoreHandle; sync contexts need a module-level allow — see store/handle.rs" },
- { path = "cohort_stream_processor::store::rocks::CohortStore::delete_partition", reason = "async code must use StoreHandle::delete_partition (or delete_partition_blocking under a no-await guard) — see store/handle.rs" },
+ { path = "cohort_stream_processor::store::rocks::CohortStore::delete_partition", reason = "async code must use StoreHandle::delete_partition — see store/handle.rs" },
{ path = "cohort_stream_processor::store::rocks::CohortStore::slice_coverage", reason = "async code must not read the store on a runtime thread; sync contexts need a module-level allow — see store/handle.rs" },
{ path = "cohort_stream_processor::store::rocks::CohortStore::resume_or_begin_slice", reason = "async code must use StoreHandle::resume_or_begin_slice — see store/handle.rs" },
{ path = "cohort_stream_processor::store::rocks::CohortStore::flush", reason = "async code must go through StoreHandle; sync contexts need a module-level allow — see store/handle.rs" },
diff --git a/rust/cohort-stream-processor/src/config.rs b/rust/cohort-stream-processor/src/config.rs
index f3d39ece6956..2c4d836d20a8 100644
--- a/rust/cohort-stream-processor/src/config.rs
+++ b/rust/cohort-stream-processor/src/config.rs
@@ -1,6 +1,6 @@
//! Service configuration, loaded from environment variables via `envconfig`.
-use std::num::NonZeroUsize;
+use std::num::{NonZeroU16, NonZeroUsize};
use std::path::{Component, Path, PathBuf};
use std::time::Duration;
@@ -13,6 +13,7 @@ use rdkafka::ClientConfig;
use tracing::warn;
use crate::partitions::pacing::{AgeMs, Hysteresis, SeedPacingConfig, UsedPct};
+use crate::partitions::PartitionOwnership;
use crate::store::durability::stage::staging_path;
use crate::store::durability::{CheckpointLineage, DurabilityConfig, NotAnOrdinalPod, PodOrdinal};
use crate::store::{OffloadConfig, OffloadMode, StoreConfig};
@@ -91,7 +92,7 @@ pub struct Config {
/// the slots it takes at the start of a seed turn and the dispatcher refills them while the
/// turn runs, which is wanted — the next turn is already full when this one ends.
///
- /// Additive with the event cap, so a revoke or shutdown drain now covers up to this many seeds
+ /// Additive with the event cap, so a shutdown drain covers up to this many seeds
/// **plus** `PARTITION_INTAKE_MAX_EVENTS` events per partition. At the default that is about
/// four quanta of seeds, seconds of drain at the rates the apply path sustains.
#[envconfig(from = "PARTITION_INTAKE_MAX_SEEDS", default = "1024")]
@@ -116,17 +117,8 @@ pub struct Config {
#[envconfig(default = "cohort-stream-processor")]
pub kafka_consumer_group: String,
- /// Start at the tail, not the topic's retention.
- #[envconfig(default = "latest")]
- pub kafka_consumer_offset_reset: String,
-
- /// How long the broker waits for heartbeats before declaring this consumer dead. With static
- /// membership, a restart within this window reclaims partitions with no rebalance.
- #[envconfig(default = "60000")]
- pub kafka_session_timeout_ms: u64,
-
- /// `queued.max.messages.kbytes` (KB). For a `subscribe()` consumer this is an *aggregate* cap
- /// across all partitions — the ceiling on librdkafka's fetch buffer. 128 MB.
+ /// `queued.max.messages.kbytes` (KB). For the events consumer this is an *aggregate* cap
+ /// across all its partitions — the ceiling on librdkafka's fetch buffer. 128 MB.
#[envconfig(from = "COHORT_KAFKA_QUEUED_MAX_MESSAGES_KBYTES", default = "131072")]
pub kafka_queued_max_messages_kbytes: u32,
@@ -329,11 +321,17 @@ pub struct Config {
#[envconfig(from = "COHORT_SEED_DISK_RESUME_PCT", default = "55")]
pub cohort_seed_disk_resume_pct: f64,
- /// Stable per-pod identity for `group.instance.id` + `client.id`, enabling static membership.
- /// Read from `POD_NAME`, else `HOSTNAME`. Absent means no static membership.
+ /// The StatefulSet pod name. Its ordinal suffix picks the partitions this pod owns and its
+ /// checkpoint lineage, and it names the Kafka clients. Absent outside Kubernetes.
#[envconfig(from = "POD_NAME")]
pub pod_name: Option<String>,
+ /// How many processor pods split the partitions. The pod with ordinal `i` owns every partition
+ /// `p` with `p mod COHORT_POD_COUNT = i`, on every input. It must divide
+ /// `COHORT_PARTITION_COUNT`. Set it with the StatefulSet's replica count.
+ #[envconfig(from = "COHORT_POD_COUNT", default = "1")]
+ pub cohort_pod_count: NonZeroU16,
+
#[envconfig(from = "HOSTNAME")]
pub pod_hostname: Option<String>,
@@ -491,20 +489,10 @@ pub struct Config {
/// When on, reopen the existing store on restart instead of wiping it: recent Stage 1 state is
/// restored and only the gap since the last committed offset is replayed (idempotent via per-key
- /// `AppliedOffsets`). Refused alongside `cohort_cascade_enabled` — merge column families are not
- /// restored yet. See [`Self::effective_wipe_on_start`].
+ /// `AppliedOffsets`). See [`Self::effective_wipe_on_start`].
#[envconfig(from = "DURABLE_RESTORE_ENABLED", default = "false")]
pub durable_restore_enabled: bool,
- /// Operator assertion that this is a single-pod, static-membership shadow deploy. Default off.
- ///
- /// Required to run `DURABLE_RESTORE_ENABLED` together with `COHORT_CASCADE_ENABLED`: a single pod
- /// with static membership reclaims its own partitions on a restart and never triggers a live
- /// rebalance. `pod_identity()` cannot gate this because it is set on every k8s pod (single or
- /// multi), so the single-vs-multi assertion needs its own flag.
- #[envconfig(from = "DURABLE_RESTORE_SINGLE_POD", default = "false")]
- pub durable_restore_single_pod: bool,
-
/// Master gate for the whole-DB checkpoint + S3 backup/restore layer. Default off.
#[envconfig(from = "CHECKPOINT_ENABLED", default = "false")]
pub checkpoint_enabled: bool,
@@ -588,7 +576,8 @@ pub struct Config {
/// Hours prior to now the S3 importer searches for checkpoint attempts in a PVC-lost recovery.
/// Keep it below the `cohort_stream_events` retention by more than a restore's catch-up time: a
/// restore resumes about this far back, and a consumer that falls below the low watermark while
- /// it catches up resets to `latest` without a trace.
+ /// it catches up resets to `earliest` without a trace, so the events that retention deleted are
+ /// lost.
#[envconfig(default = "20")]
pub checkpoint_import_window_hours: u32,
@@ -839,8 +828,6 @@ impl Config {
/// on open, silently discarding the restore. It also requires a bucket, non-zero knobs, an
/// absolute checkpoint directory apart from the store, and a pod name with an ordinal, so a
/// misconfigured pod fails at start instead of running without checkpoints.
- /// - `durable_restore_enabled` + `cohort_cascade_enabled` requires `durable_restore_single_pod`
- /// and a pod identity: `pod_identity()` alone is not a single-pod signal (set on every k8s pod).
pub fn validate_startup(&self) -> anyhow::Result<()> {
ensure!(
self.cohort_seed_reconcile_scan_page > 0,
@@ -932,27 +919,6 @@ impl Config {
self.validate_checkpoints()?;
}
- if self.durable_restore_enabled && self.cohort_cascade_enabled {
- let single_pod_static =
- self.durable_restore_single_pod && self.pod_identity().is_some();
- ensure!(
- single_pod_static,
- "DURABLE_RESTORE_ENABLED=true is unsafe with COHORT_CASCADE_ENABLED=true: merge \
- column families are not restored yet. Disable one of them, or set \
- DURABLE_RESTORE_SINGLE_POD=true on a single-pod static-membership (POD_NAME/HOSTNAME \
- set) shadow deploy.",
- );
- warn!(
- durable_restore_single_pod = self.durable_restore_single_pod,
- pod_identity = self.pod_identity().unwrap_or("<none>"),
- "DURABLE_RESTORE_ENABLED + COHORT_CASCADE_ENABLED allowed on a single-pod \
- static-membership shadow. RESIDUALS deferred to Slice 3: multi-pod rebalance \
- F3-on-revoke and the post-join↔delete REVOKE race are NOT covered (single-pod \
- membership sidesteps them); merge column-family *content* resume after an S3 restore \
- is validated only on this shadow, NOT production multi-pod merge durability.",
- );
- }
-
// Not a refusal (dev runs with durability off), but a crash then loses applied tiles
// that never replay.
if self.cohort_seed_consumer_enabled && !self.durable_restore_enabled {
@@ -1058,15 +1024,51 @@ impl Config {
Ok(())
}
- /// The checkpoint lineage of this pod, from the ordinal at the end of `POD_NAME`. A process
- /// without `POD_NAME` (a local run or a test) uses the single pod's lineage. `HOSTNAME` is not
- /// consulted: outside Kubernetes it names the machine, not a StatefulSet pod.
+ /// The ordinal at the end of `POD_NAME`, or `None` for a process without `POD_NAME` (a local
+ /// run or a test). `HOSTNAME` is not consulted: outside Kubernetes it names the machine, not a
+ /// StatefulSet pod.
+ fn pod_ordinal(&self) -> Result<Option<PodOrdinal>, NotAnOrdinalPod> {
+ self.pod_name
+ .as_deref()
+ .filter(|name| !name.is_empty())
+ .map(PodOrdinal::from_pod_name)
+ .transpose()
+ }
+
+ /// The checkpoint lineage of this pod, from its ordinal. A process without `POD_NAME` uses the
+ /// single pod's lineage.
pub fn checkpoint_lineage(&self) -> Result<CheckpointLineage, NotAnOrdinalPod> {
- let ordinal = match self.pod_name.as_deref().filter(|name| !name.is_empty()) {
- Some(name) => PodOrdinal::from_pod_name(name)?,
- None => PodOrdinal::STANDALONE,
+ Ok(CheckpointLineage::new(
+ self.pod_ordinal()?.unwrap_or(PodOrdinal::STANDALONE),
+ ))
+ }
+
+ /// The partitions this pod owns: pod `ordinal` of `COHORT_POD_COUNT`, over
+ /// `COHORT_PARTITION_COUNT` partitions. Only a single pod may run without `POD_NAME`: every
+ /// pod without it would take ordinal 0, and they would all own the same partitions.
+ pub fn partition_ownership(&self) -> anyhow::Result<PartitionOwnership> {
+ let ordinal = match self
+ .pod_ordinal()
+ .context("POD_NAME must end in a StatefulSet ordinal, or be unset outside Kubernetes")?
+ {
+ Some(ordinal) => ordinal,
+ None => {
+ ensure!(
+ self.cohort_pod_count == NonZeroU16::MIN,
+ "COHORT_POD_COUNT={} requires POD_NAME, so that each pod owns its own partitions",
+ self.cohort_pod_count,
+ );
+ PodOrdinal::STANDALONE
+ }
};
- Ok(CheckpointLineage::new(ordinal))
+ PartitionOwnership::new(self.cohort_pod_count, ordinal, self.partition_count())
+ .with_context(|| {
+ format!(
+ "COHORT_POD_COUNT={} must divide COHORT_PARTITION_COUNT={}, and the ordinal of \
+ POD_NAME must be below it",
+ self.cohort_pod_count, self.cohort_partition_count,
+ )
+ })
}
/// `COHORT_PARTITION_COUNT` as a slice id range. A count past `u16::MAX` fails the topic checks
@@ -1083,7 +1085,7 @@ impl Config {
self.wipe_store_on_start && !(self.durable_restore_enabled && db_dir_exists)
}
- pub fn store_config(&self) -> StoreConfig {
+ pub fn store_config(&self, ownership: PartitionOwnership) -> StoreConfig {
if self.person_record_ttl_below_safe_floor() {
warn!(
cohort_person_record_ttl_days = self.cohort_person_record_ttl_days,
@@ -1105,7 +1107,7 @@ impl Config {
periodic_compaction_seconds: self.cohort_periodic_compaction_seconds,
max_background_jobs: self.cohort_max_background_jobs,
person_record_ttl_days: self.cohort_person_record_ttl_days,
- partition_count: self.partition_count(),
+ ownership,
..StoreConfig::default()
}
}
@@ -1127,8 +1129,7 @@ impl Config {
}
}
- /// Stable per-pod identity for static group membership, `POD_NAME` preferred over `HOSTNAME`.
- /// `None` leaves static membership off.
+ /// Stable per-pod name for the Kafka `client.id`, `POD_NAME` preferred over `HOSTNAME`.
pub fn pod_identity(&self) -> Option<&str> {
[self.pod_name.as_deref(), self.pod_hostname.as_deref()]
.into_iter()
@@ -1136,28 +1137,10 @@ impl Config {
.find(|id| !id.is_empty())
}
- /// Build the `rdkafka` client config for the `cohort_stream_events` group consumer.
- ///
- /// Auto-commit and auto-offset-store are off: the consume loop marks offsets only after a
- /// sub-batch is routed and produced. `cooperative-sticky` + static membership ensure a
- /// membership change revokes only the partitions that move.
+ /// Build the `rdkafka` client config for the `cohort_stream_events` consumer: the follower
+ /// config on the events group, with a bounded fetch buffer.
pub fn consumer_client_config(&self) -> ClientConfig {
- let mut config = ClientConfig::new();
- config
- .set("bootstrap.servers", &self.kafka_hosts)
- .set("group.id", &self.kafka_consumer_group)
- .set("enable.auto.commit", "false")
- .set("enable.auto.offset.store", "false")
- .set("auto.offset.reset", &self.kafka_consumer_offset_reset)
- .set("partition.assignment.strategy", "cooperative-sticky")
- .set("socket.timeout.ms", "10000")
- .set(
- "session.timeout.ms",
- self.kafka_session_timeout_ms.to_string(),
- )
- .set("heartbeat.interval.ms", "5000")
- .set("max.poll.interval.ms", "300000");
-
+ let mut config = self.follower_client_config(&self.kafka_consumer_group);
// Bound librdkafka's fetch buffer: an aggregate byte ceiling plus a per-partition prefetch floor.
let fetch = self.fetch_queue_config();
config
@@ -1166,33 +1149,19 @@ impl Config {
fetch.queued_max_messages_kbytes.to_string(),
)
.set("queued.min.messages", fetch.queued_min_messages.to_string());
-
- // Static membership; an explicit `kafka_client_id` overrides `client.id` below.
- if let Some(id) = self.pod_identity() {
- config.set("group.instance.id", id).set("client.id", id);
- }
- if !self.kafka_client_id.is_empty() {
- config.set("client.id", &self.kafka_client_id);
- }
- if !self.kafka_client_rack.is_empty() {
- config.set("client.rack", &self.kafka_client_rack);
- }
- if self.kafka_tls {
- config
- .set("security.protocol", "ssl")
- .set("enable.ssl.certificate.verification", "false");
- }
config
}
- /// Build the `rdkafka` client config for a merge-protocol follower consumer.
+ /// Build the `rdkafka` client config for a consumer of `group`: the events consumer or a
+ /// follower.
///
- /// Followers never `subscribe()` — the events group's rebalance mirrors assignments onto them —
- /// so there is no assignment strategy and no static membership; `group.id` exists only so
- /// commits land on an observable group. `auto.offset.reset` is hard-coded to `earliest`: a
- /// never-subscribing group is `Empty` to the broker, so its committed offsets are pruned after
- /// `offsets.retention.minutes`, and a tail reset after pruning would silently skip the merge
- /// backlog.
+ /// Auto-commit and auto-offset-store are off: a consume loop marks offsets only after its
+ /// messages are processed. Every consumer assigns itself the partitions this pod owns and never
+ /// joins its group, so `group.id` only names where its commits land. `auto.offset.reset` is
+ /// hard-coded to `earliest`: a group without members is `Empty` to the broker, which can expire
+ /// its committed offsets, and a tail reset would then silently skip the messages in between. It
+ /// sets no `group.instance.id`: librdkafka sends that id with every offset commit, and a broker
+ /// can refuse a commit that names a static member from a client outside the group.
pub fn follower_client_config(&self, group: &str) -> ClientConfig {
let mut config = ClientConfig::new();
config
@@ -1203,7 +1172,6 @@ impl Config {
.set("auto.offset.reset", "earliest")
.set("socket.timeout.ms", "10000");
- // `client.id` for observability only; no `group.instance.id` (no membership to make static).
if let Some(id) = self.pod_identity() {
config.set("client.id", id);
}
@@ -1295,7 +1263,6 @@ mod tests {
kafka_client_rack: String::new(),
cohort_stream_events_topic: "cohort_stream_events".to_string(),
kafka_consumer_group: "cohort-stream-processor".to_string(),
- kafka_consumer_offset_reset: "latest".to_string(),
kafka_queued_max_messages_kbytes: 131_072,
kafka_queued_min_messages: 2000,
person_merge_events_topic: "person_merge_events".to_string(),
@@ -1317,8 +1284,8 @@ mod tests {
cohort_cascade_enabled: false,
cohort_cascade_events_topic: "cohort_cascade_events".to_string(),
kafka_cascade_consumer_group: "cohort-stream-cascade".to_string(),
- kafka_session_timeout_ms: 60000,
pod_name: None,
+ cohort_pod_count: NonZeroU16::MIN,
pod_hostname: None,
cohort_membership_changed_topic: "cohort_membership_changed_shadow".to_string(),
cohort_reconcile_markers_topic: "cohort_reconcile_markers".to_string(),
@@ -1349,7 +1316,6 @@ mod tests {
cohort_store_event_read_permits: 16,
cohort_store_maintenance_permits: 6,
durable_restore_enabled: false,
- durable_restore_single_pod: false,
checkpoint_enabled: false,
checkpoint_interval_ms: 300_000,
checkpoint_s3_upload_interval_ms: 900_000,
@@ -1644,22 +1610,10 @@ mod tests {
assert_eq!(client.get("enable.auto.commit"), Some("false"));
assert_eq!(client.get("enable.auto.offset.store"), Some("false"));
assert_eq!(client.get("group.id"), Some("cohort-stream-processor"));
- assert_eq!(client.get("auto.offset.reset"), Some("latest"));
+ assert_eq!(client.get("auto.offset.reset"), Some("earliest"));
assert_eq!(client.get("bootstrap.servers"), Some("localhost:9092"));
}
- #[test]
- fn consumer_config_uses_cooperative_sticky_and_the_configured_session_timeout() {
- let mut config = test_config();
- config.kafka_session_timeout_ms = 45000;
- let client = config.consumer_client_config();
- assert_eq!(
- client.get("partition.assignment.strategy"),
- Some("cooperative-sticky"),
- );
- assert_eq!(client.get("session.timeout.ms"), Some("45000"));
- }
-
#[test]
fn consumer_config_sets_the_fetch_queue_bounds() {
let config = test_config();
@@ -1706,22 +1660,64 @@ mod tests {
}
#[test]
- fn consumer_config_sets_static_membership_only_when_pod_identity_is_present() {
+ fn consumer_config_names_the_client_after_the_pod_without_a_static_member_id() {
let mut config = test_config();
- assert_eq!(
- config.consumer_client_config().get("group.instance.id"),
- None,
- );
-
- config.pod_hostname = Some("cohort-stream-processor-2".to_string());
+ config.pod_name = Some("cohort-stream-processor-2".to_string());
let client = config.consumer_client_config();
- assert_eq!(
- client.get("group.instance.id"),
- Some("cohort-stream-processor-2"),
- );
+ assert_eq!(client.get("group.instance.id"), None);
assert_eq!(client.get("client.id"), Some("cohort-stream-processor-2"));
}
+ #[test]
+ fn partition_ownership_comes_from_the_pod_count_and_the_pod_name_ordinal() {
+ type Env = &'static [(&'static str, &'static str)];
+ let cases: [(Env, Result<&str, &str>); 6] = [
+ (&[], Ok("pod 0 of 1 over 64 partitions")),
+ (
+ &[
+ ("COHORT_POD_COUNT", "2"),
+ ("POD_NAME", "cohort-stream-processor-1"),
+ ],
+ Ok("pod 1 of 2 over 64 partitions"),
+ ),
+ (&[("COHORT_POD_COUNT", "4")], Err("requires POD_NAME")),
+ (
+ &[
+ ("COHORT_POD_COUNT", "3"),
+ ("POD_NAME", "cohort-stream-processor-0"),
+ ],
+ Err("evenly"),
+ ),
+ (
+ &[
+ ("COHORT_POD_COUNT", "2"),
+ ("POD_NAME", "cohort-stream-processor-2"),
+ ],
+ Err("not below"),
+ ),
+ (&[("POD_NAME", "cohort-stream-processor")], Err("POD_NAME")),
+ ];
+ for (env, expected) in cases {
+ let env = env
+ .iter()
+ .map(|&(key, value)| (key.to_string(), value.to_string()))
+ .collect();
+ let ownership = Config::init_from_hashmap(&env)
+ .unwrap()
+ .partition_ownership()
+ .map(|ownership| ownership.to_string())
+ .map_err(|err| format!("{err:#}"));
+ match (ownership, expected) {
+ (Ok(ownership), Ok(expected)) => assert_eq!(ownership, expected, "{env:?}"),
+ (Err(err), Err(expected)) => assert!(err.contains(expected), "{env:?}: {err}"),
+ (actual, expected) => panic!("{env:?}: got {actual:?}, expected {expected:?}"),
+ }
+ }
+
+ let zero = [("COHORT_POD_COUNT".to_string(), "0".to_string())].into();
+ assert!(Config::init_from_hashmap(&zero).is_err());
+ }
+
#[test]
fn pod_identity_prefers_pod_name_and_ignores_blanks() {
let mut config = test_config();
@@ -1742,18 +1738,17 @@ mod tests {
config.pod_hostname = Some("hostname".to_string());
config.kafka_client_id = "explicit-client".to_string();
let client = config.consumer_client_config();
- // Static membership still keyed on the pod identity, but client.id is the explicit override.
- assert_eq!(client.get("group.instance.id"), Some("hostname"));
assert_eq!(client.get("client.id"), Some("explicit-client"));
}
#[test]
fn store_config_threads_the_wipe_on_start_flag() {
let mut config = test_config();
+ let ownership = PartitionOwnership::single(64);
config.wipe_store_on_start = true;
- assert!(config.store_config().wipe_on_start);
+ assert!(config.store_config(ownership).wipe_on_start);
config.wipe_store_on_start = false;
- assert!(!config.store_config().wipe_on_start);
+ assert!(!config.store_config(ownership).wipe_on_start);
}
#[test]
@@ -1779,14 +1774,22 @@ mod tests {
!defaults.cohort_wipe_on_schema_mismatch,
"a schema mismatch must fail fast unless the operator opts in",
);
- assert!(!defaults.store_config().wipe_on_schema_mismatch);
+ assert!(
+ !defaults
+ .store_config(PartitionOwnership::single(64))
+ .wipe_on_schema_mismatch
+ );
let enabled = Config::init_from_hashmap(&std::collections::HashMap::from([(
"COHORT_WIPE_ON_SCHEMA_MISMATCH".to_owned(),
"true".to_owned(),
)]))
.unwrap();
- assert!(enabled.store_config().wipe_on_schema_mismatch);
+ assert!(
+ enabled
+ .store_config(PartitionOwnership::single(64))
+ .wipe_on_schema_mismatch
+ );
}
#[test]
@@ -1796,7 +1799,12 @@ mod tests {
defaults.cohort_person_record_ttl_days, 0,
"the person-record TTL defaults off (no compaction filter)",
);
- assert_eq!(defaults.store_config().person_record_ttl_days, 0);
+ assert_eq!(
+ defaults
+ .store_config(PartitionOwnership::single(64))
+ .person_record_ttl_days,
+ 0
+ );
let enabled = Config::init_from_hashmap(&std::collections::HashMap::from([(
"COHORT_PERSON_RECORD_TTL_DAYS".to_owned(),
@@ -1805,7 +1813,9 @@ mod tests {
.unwrap();
assert_eq!(enabled.cohort_person_record_ttl_days, 30);
assert_eq!(
- enabled.store_config().person_record_ttl_days,
+ enabled
+ .store_config(PartitionOwnership::single(64))
+ .person_record_ttl_days,
30,
"the TTL reaches StoreConfig",
);
@@ -1836,10 +1846,19 @@ mod tests {
fn stats_knobs_default_on_and_thread_into_store_config() {
let defaults = Config::init_from_hashmap(&std::collections::HashMap::new()).unwrap();
assert!(defaults.store_statistics_enabled);
- assert!(defaults.store_config().statistics_enabled);
+ assert!(
+ defaults
+ .store_config(PartitionOwnership::single(64))
+ .statistics_enabled
+ );
assert_eq!(defaults.stats_publish_interval(), Duration::from_secs(15));
assert_eq!(defaults.store_read_sample_ratio, 64);
- assert_eq!(defaults.store_config().read_sample_ratio, 64);
+ assert_eq!(
+ defaults
+ .store_config(PartitionOwnership::single(64))
+ .read_sample_ratio,
+ 64
+ );
let env: std::collections::HashMap<String, String> = [
("COHORT_STORE_STATISTICS_ENABLED", "false"),
@@ -1852,13 +1871,17 @@ mod tests {
let config = Config::init_from_hashmap(&env).unwrap();
assert!(!config.store_statistics_enabled);
assert!(
- !config.store_config().statistics_enabled,
+ !config
+ .store_config(PartitionOwnership::single(64))
+ .statistics_enabled,
"the flag reaches StoreConfig",
);
assert_eq!(config.stats_publish_interval(), Duration::from_secs(30));
assert_eq!(config.store_read_sample_ratio, 8);
assert_eq!(
- config.store_config().read_sample_ratio,
+ config
+ .store_config(PartitionOwnership::single(64))
+ .read_sample_ratio,
8,
"the sample ratio reaches StoreConfig",
);
@@ -1975,75 +1998,17 @@ mod tests {
assert!(config.durable_restore_enabled);
}
- #[test]
- fn durable_restore_single_pod_defaults_off_and_overrides_from_env() {
- let defaults = Config::init_from_hashmap(&std::collections::HashMap::new()).unwrap();
- assert!(
- !defaults.durable_restore_single_pod,
- "the single-pod assertion defaults off (inert)",
- );
-
- let env: std::collections::HashMap<String, String> =
- [("DURABLE_RESTORE_SINGLE_POD", "true")]
- .into_iter()
- .map(|(key, value)| (key.to_string(), value.to_string()))
- .collect();
- let config = Config::init_from_hashmap(&env).unwrap();
- assert!(config.durable_restore_single_pod);
- }
-
#[test]
fn durability_startup_guard_passes_for_the_default_config() {
assert!(test_config().validate_startup().is_ok());
}
#[test]
- fn durability_startup_guard_passes_for_a_plain_durable_restore() {
- let mut config = test_config();
- config.durable_restore_enabled = true;
- assert!(config.validate_startup().is_ok());
- }
-
- #[test]
- fn durability_startup_guard_refuses_durable_plus_cascade_without_the_opt_in() {
+ fn durability_startup_guard_passes_for_durable_restore_with_cascades() {
let mut config = test_config();
config.durable_restore_enabled = true;
config.cohort_cascade_enabled = true;
- config.pod_name = Some("pod-0".to_string());
- let err = config
- .validate_startup()
- .expect_err("durable + cascade without the opt-in must be refused");
- assert!(
- err.to_string().contains("DURABLE_RESTORE_SINGLE_POD"),
- "the refusal points the operator at the single-pod opt-in: {err}",
- );
- }
-
- #[test]
- fn durability_startup_guard_refuses_durable_plus_cascade_with_opt_in_but_no_pod_identity() {
- let mut config = test_config();
- config.durable_restore_enabled = true;
- config.cohort_cascade_enabled = true;
- config.durable_restore_single_pod = true;
- config.pod_name = None;
- config.pod_hostname = None;
- assert!(
- config.validate_startup().is_err(),
- "single-pod opt-in without a pod identity must still refuse the combo",
- );
- }
-
- #[test]
- fn durability_startup_guard_allows_durable_plus_cascade_with_opt_in_and_pod_identity() {
- let mut config = test_config();
- config.durable_restore_enabled = true;
- config.cohort_cascade_enabled = true;
- config.durable_restore_single_pod = true;
- config.pod_name = Some("cohort-stream-processor-0".to_string());
- assert!(
- config.validate_startup().is_ok(),
- "durable + cascade is allowed on a single-pod static-membership deploy",
- );
+ assert!(config.validate_startup().is_ok());
}
#[test]
@@ -2304,7 +2269,9 @@ mod tests {
};
assert_eq!(config.effective_wipe_on_start(), expected, "{why}");
assert_eq!(
- config.store_config().wipe_on_start,
+ config
+ .store_config(PartitionOwnership::single(64))
+ .wipe_on_start,
expected,
"store_config must thread the effective wipe: {why}",
);
@@ -2316,7 +2283,7 @@ mod tests {
let mut config = test_config();
config.store_path = "/var/lib/cohort/state".to_string();
assert_eq!(
- config.store_config().path,
+ config.store_config(PartitionOwnership::single(64)).path,
std::path::PathBuf::from("/var/lib/cohort/state"),
);
}
@@ -2536,9 +2503,8 @@ mod tests {
}
#[test]
- fn follower_config_hard_codes_earliest_regardless_of_the_events_reset() {
- let mut config = test_config();
- config.kafka_consumer_offset_reset = "latest".to_string();
+ fn follower_config_hard_codes_earliest() {
+ let config = test_config();
let client = config.follower_client_config("cohort-stream-merges");
assert_eq!(client.get("auto.offset.reset"), Some("earliest"));
assert_eq!(client.get("group.id"), Some("cohort-stream-merges"));
@@ -2626,7 +2592,7 @@ mod tests {
config.cohort_periodic_compaction_seconds = 3600;
config.cohort_max_background_jobs = 2;
- let store = config.store_config();
+ let store = config.store_config(PartitionOwnership::single(64));
assert_eq!(store.block_cache_bytes, 3_221_225_472);
assert!(!store.tuned_block_options);
assert!(!store.compact_on_deletion);
diff --git a/rust/cohort-stream-processor/src/consumers/boot.rs b/rust/cohort-stream-processor/src/consumers/boot.rs
index fb6a479ab060..518c8d1a8007 100644
--- a/rust/cohort-stream-processor/src/consumers/boot.rs
+++ b/rust/cohort-stream-processor/src/consumers/boot.rs
@@ -1,8 +1,9 @@
//! Boot recovery for the events consumer.
//!
//! With durable restore, the store reopens with state the broker's committed offsets may not
-//! match. Nothing may fold until the consumer has reclaimed unowned slices, redriven the merge
-//! outbox and sought every owned partition back to the first event it polled and did not fold. A
+//! match. The consumer reclaims unowned slices and redrives the merge outbox before its first
+//! poll. After a checkpoint restore, nothing may fold until it has also sought every owned
+//! partition back to its restored position, or to the first event it polled and did not fold. A
//! fold before that would set per-key replay marks above offsets that are fetched again later, so
//! the second fetch would skip those events as replays.
@@ -61,52 +62,20 @@ impl ResumePoints {
}
}
-/// Where the events consumer's boot stands. Nothing folds before [`BootPhase::Live`]. A pending
-/// checkpoint restore rides along until the rewind commits its positions and finishes it.
+/// Where the events consumer's boot stands. Nothing folds before [`BootPhase::Live`].
pub(crate) enum BootPhase {
- /// Waiting for two consecutive polls to report the same non-empty assignment.
- Settling {
- previous: Option<HashSet<i32>>,
- resume: ResumePoints,
- restore: Option<PendingRestore>,
- },
- /// Recovery ran. The owned partitions must seek before anything folds.
+ /// Recovery ran on a restored store. The owned partitions must seek, and the rewind must commit
+ /// the restore's positions and finish it, before anything folds.
Rewinding {
resume: ResumePoints,
- restore: Option<PendingRestore>,
+ restore: Box<PendingRestore>,
},
- /// A restored position expired during boot and the process is stopping. Nothing seeks, settles
- /// or folds, and the restore marker keeps the next boot on the restore path.
+ /// A restored position expired during boot and the process is stopping. Nothing seeks or folds,
+ /// and the restore marker keeps the next boot on the restore path.
Halted,
Live,
}
-impl BootPhase {
- pub(crate) fn settling(restore: Option<PendingRestore>) -> Self {
- Self::Settling {
- previous: None,
- resume: ResumePoints::default(),
- restore,
- }
- }
-}
-
-/// Returns `true` once the same non-empty assignment is seen on two consecutive polls. An empty or
-/// changing assignment is not yet settled; a cooperative-incremental assign re-baselines until stable.
-pub(crate) fn boot_assignment_settled(
- assignment: &HashSet<i32>,
- prev: &mut Option<HashSet<i32>>,
-) -> bool {
- if assignment.is_empty() {
- return false;
- }
- if prev.as_ref() != Some(assignment) {
- *prev = Some(assignment.clone());
- return false;
- }
- true
-}
-
#[cfg(test)]
mod tests {
use super::*;
@@ -175,7 +144,7 @@ mod tests {
expected: Some(&[(0, 4), (1, 2), (2, 9)]),
},
Case {
- name: "a partition revoked during boot is not sought",
+ name: "a partition another pod owns is not sought",
polls: &[&[(0, 3), (1, 7)]],
restored: None,
owned: &[1],
@@ -226,22 +195,4 @@ mod tests {
);
}
}
-
- #[test]
- fn boot_assignment_settled_requires_a_stable_non_empty_assignment() {
- let mut prev: Option<HashSet<i32>> = None;
-
- assert!(!boot_assignment_settled(&HashSet::new(), &mut prev));
- assert_eq!(prev, None);
-
- let first: HashSet<i32> = [0].into_iter().collect();
- assert!(!boot_assignment_settled(&first, &mut prev));
- assert_eq!(prev.as_ref(), Some(&first));
-
- let second: HashSet<i32> = [0, 1].into_iter().collect();
- assert!(!boot_assignment_settled(&second, &mut prev));
- assert_eq!(prev.as_ref(), Some(&second));
-
- assert!(boot_assignment_settled(&second, &mut prev));
- }
}
diff --git a/rust/cohort-stream-processor/src/consumers/events.rs b/rust/cohort-stream-processor/src/consumers/events.rs
index d72f13d4b0c9..42fbffd7eab1 100644
--- a/rust/cohort-stream-processor/src/consumers/events.rs
+++ b/rust/cohort-stream-processor/src/consumers/events.rs
@@ -19,42 +19,39 @@ use dashmap::DashMap;
use lifecycle::Handle;
use metrics::{counter, gauge, histogram};
use rdkafka::consumer::{CommitMode, Consumer, ConsumerContext, StreamConsumer};
+use rdkafka::error::KafkaResult;
use rdkafka::message::Message;
use rdkafka::{Offset, TopicPartitionList};
use tracing::{debug, error, info, warn};
pub use cohort_core::events::CohortStreamEvent;
-use crate::consumers::boot::{boot_assignment_settled, BootPhase, ResumePoints};
+use crate::consumers::boot::{BootPhase, ResumePoints};
use crate::consumers::merges::{ConsumedCascade, ConsumedMerge, ConsumedTransfer};
use crate::consumers::readiness::BootReadiness;
use crate::consumers::seeds::ConsumedSeed;
use crate::filters::manager::CatalogHandle;
use crate::merge::transfer::PendingTransfer;
use crate::observability::metrics::{
- CASCADE_HELD_OFFSET_GAUGE, CHECKPOINT_RESTORE_BLOCKED,
- COHORT_STREAM_CASCADES_SKIPPED_NOT_OWNED, COHORT_STREAM_CONSUME_BATCH_SIZE,
- COHORT_STREAM_DESERIALIZE_ERRORS, COHORT_STREAM_EMPTY_PAYLOAD, COHORT_STREAM_EVENTS_CONSUMED,
- COHORT_STREAM_EVENTS_DISPATCHED, COHORT_STREAM_EVENTS_SKIPPED_NOT_OWNED,
- COHORT_STREAM_KAFKA_RECV_ERRORS, COHORT_STREAM_MERGES_SKIPPED_NOT_OWNED,
- COHORT_STREAM_OFFSET_COMMITS, COHORT_STREAM_OFFSET_COMMIT_ERRORS, COHORT_STREAM_ROUTE_ERRORS,
+ CHECKPOINT_RESTORE_BLOCKED, COHORT_STREAM_CASCADES_SKIPPED_NOT_OWNED,
+ COHORT_STREAM_CONSUME_BATCH_SIZE, COHORT_STREAM_DESERIALIZE_ERRORS,
+ COHORT_STREAM_EMPTY_PAYLOAD, COHORT_STREAM_EVENTS_CONSUMED, COHORT_STREAM_EVENTS_DISPATCHED,
+ COHORT_STREAM_EVENTS_SKIPPED_NOT_OWNED, COHORT_STREAM_KAFKA_RECV_ERRORS,
+ COHORT_STREAM_MERGES_SKIPPED_NOT_OWNED, COHORT_STREAM_OFFSET_COMMITS,
+ COHORT_STREAM_OFFSET_COMMIT_ERRORS, COHORT_STREAM_ROUTE_ERRORS,
COHORT_STREAM_SEEDS_SKIPPED_NOT_OWNED, COHORT_STREAM_TRANSFERS_SKIPPED_NOT_OWNED,
COHORT_STREAM_WORKERS_SPAWNED, DURABLE_RESTORE_PARTITIONS_KEPT_TOTAL,
DURABLE_RESTORE_PARTITIONS_WIPED_STALE_TOTAL,
- DURABLE_RESTORE_PENDING_TRANSFERS_RECOVERED_PARTITIONS_TOTAL, LIVE_WATERMARK_AGE_MS,
- MERGE_HELD_OFFSET_GAUGE, MERGE_PENDING_TRANSFERS_GAUGE, PARTITIONS_ASSIGNED_TOTAL,
- PARTITIONS_PAUSED, PARTITIONS_REVOKED_TOTAL, PARTITION_STATE_DELETED_TOTAL,
- PENDING_HELD_EVENTS, REBALANCE_CLEANUP_SKIPPED_TOTAL, REVOKE_DRAIN_DURATION_SECONDS,
- SEED_HELD_OFFSET_GAUGE, SEED_OLDEST_HELD_AGE_MS, SEED_PAUSE_AGE_MS,
- SLICE_COVERED_SINCE_SECONDS,
+ DURABLE_RESTORE_PENDING_TRANSFERS_RECOVERED_PARTITIONS_TOTAL, OWNED_PARTITIONS,
+ PARTITIONS_ASSIGNED_TOTAL, PARTITIONS_PAUSED, PARTITION_OWNERSHIP_INFO, PENDING_HELD_EVENTS,
};
use crate::partitions::backpressure::Backpressure;
use crate::partitions::offset_tracker::OffsetTracker;
use crate::partitions::pause::{ConsumerPauser, PartitionPauser};
-use crate::partitions::rebalance::{CohortConsumerContext, ConsumerCommandReceiver};
+use crate::partitions::rebalance::CohortConsumerContext;
use crate::partitions::router::{PartitionRouter, SeedRefusal, SeedSendOutcome, SendOutcome};
use crate::partitions::shuffle_message::ShuffleMessage;
-use crate::partitions::{list_offsets, InputTopic};
+use crate::partitions::{list_offsets, InputTopic, PartitionOwnership};
use crate::producer::MembershipSink;
use crate::store::durability::{FinishError, PendingRestore};
use crate::store::StoreHandle;
@@ -87,9 +84,8 @@ pub struct ConsumedEvent {
pub broker_ts_ms: Option<i64>,
}
-/// The partitions this pod owned when its workers finished draining. Closing the events consumer
-/// revokes them from the live set, so followers commit against this snapshot instead. Only
-/// [`EventDispatcher::shutdown`] builds one.
+/// The partitions this pod owned when its workers finished draining. Followers make their final
+/// commits for this snapshot. Only [`EventDispatcher::shutdown`] builds one.
#[derive(Debug, Clone)]
pub(crate) struct DrainedOwnership(HashSet<i32>);
@@ -109,21 +105,14 @@ pub(crate) enum DrainWait {
TimedOut,
}
-/// A partition entry remains present while its old worker drains. That sentinel serializes worker
-/// replacement with tracker-tenure reset during a rapid revoke/reassign.
-enum WorkerSlot {
- Running(Stage1Worker),
- Draining,
-}
-
/// The Kafka-free routing core: dispatches consumed batches to per-partition workers, tracks
-/// processed offsets, and owns the partition lifecycle the rebalance handler drives.
+/// processed offsets, and records which partitions this pod owns.
pub struct EventDispatcher {
router: Arc<PartitionRouter>,
tracker: Arc<OffsetTracker>,
- workers: Arc<DashMap<i32, WorkerSlot>>,
- /// Partitions currently assigned to this consumer, with when each was assigned. A slice
- /// without a coverage record begins at that instant, not at its worker's spawn.
+ workers: Arc<DashMap<i32, Stage1Worker>>,
+ /// Partitions this pod owns, with when each was assigned. A slice without a coverage record
+ /// begins at that instant, not at its worker's spawn.
owned: Arc<DashMap<i32, CoverageStartMs>>,
handle: StoreHandle,
catalog: Arc<CatalogHandle>,
@@ -132,13 +121,9 @@ pub struct EventDispatcher {
/// Prevents post-shutdown worker registration that would hang the join. The structural guarantee
/// is the router's terminal closed state, which refuses registration after `clear()`.
draining: AtomicBool,
- /// When set, workers rebuild their `EvictionQueue` on spawn and boot/assign paths reclaim stale
- /// on-disk slices. Set once at startup before any worker spawns.
+ /// When set, workers rebuild their `EvictionQueue` on spawn and boot reclaims the on-disk
+ /// slices of partitions this pod does not own. Set once at startup before any worker spawns.
durable_restore: AtomicBool,
- /// Settled boot assignment, recorded once. The assign path reclaims only partitions that move in
- /// after boot. `OnceLock` is `Sync`; the only race (get sees `None` before set) resolves to
- /// "skip", so a boot partition is never wiped.
- boot_assignment: OnceLock<HashSet<i32>>,
/// Event-name fan-out gating for spawned workers, set once at startup. Unset → disabled.
event_name_gating: OnceLock<EventNameGating>,
/// `/_ready`'s gate. Every rebuilding worker this dispatcher spawns holds it closed.
@@ -168,7 +153,6 @@ impl EventDispatcher {
merge,
draining: AtomicBool::new(false),
durable_restore: AtomicBool::new(false),
- boot_assignment: OnceLock::new(),
event_name_gating: OnceLock::new(),
drained: tokio::sync::SetOnce::new(),
}
@@ -524,14 +508,7 @@ impl EventDispatcher {
/// partition is unowned. The ownership check and insert are atomic under the DashMap shard guard.
fn ensure_worker(&self, partition: i32) {
match self.workers.entry(partition) {
- Entry::Occupied(slot) => {
- if matches!(slot.get(), WorkerSlot::Draining) {
- debug!(
- partition,
- "worker replacement waits for the revoked worker to finish draining"
- );
- }
- }
+ Entry::Occupied(_) => {}
Entry::Vacant(slot) => {
// No `.await` in this arm — the DashMap shard guard is held.
let Some(assigned_at) = self.owned.get(&partition).map(|entry| *entry) else {
@@ -540,14 +517,6 @@ impl EventDispatcher {
if self.draining.load(Ordering::SeqCst) {
return;
}
- // Durable restart only: wipe a post-boot move-in's stale on-disk slice (left by a
- // revoke-delete that failed before the partition returned to this pod) before the
- // worker rebuilds its eviction queue from it. Runs under the shard guard, so the wipe
- // strictly precedes the spawn. `reclaim_stale_slice` must never touch `self.workers` —
- // a `contains_key`/`entry` on the same shard while this write guard is held deadlocks.
- if self.durable_restore_enabled() {
- self.reclaim_stale_slice(partition);
- }
match self.router.add_partition(partition) {
Some(inbox) => {
let restore = if self.durable_restore_enabled() {
@@ -567,7 +536,7 @@ impl EventDispatcher {
self.event_name_gating(),
assigned_at,
);
- slot.insert(WorkerSlot::Running(worker));
+ slot.insert(worker);
counter!(COHORT_STREAM_WORKERS_SPAWNED).increment(1);
info!(
partition,
@@ -614,9 +583,9 @@ impl EventDispatcher {
&self.merge
}
- /// Route a sweep tick to each owned partition that has a live worker. Never spawns, so a revoked
- /// partition is not resurrected; a worker-less owned partition has no in-memory state to evict, so
- /// it is skipped rather than producing a `no_worker` drop.
+ /// Route a sweep tick to each owned partition that has a live worker. Never spawns: a worker-less
+ /// owned partition has no in-memory state to evict, so it is skipped rather than producing a
+ /// `no_worker` drop.
///
/// Routes nothing before the first catalog load. Every team is absent from the unloaded catalog,
/// and a sweep batch drops the keys it claims for an absent team without rescheduling them, so a
@@ -652,9 +621,9 @@ impl EventDispatcher {
}
/// Owned partitions that currently have a live worker channel — the only partitions a maintenance
- /// tick can reach. An owned partition that never received an event has no worker, so a tick to it
- /// is a guaranteed `no_worker` no-op (no in-memory state to evict, redrive, or GC); skipping it
- /// keeps the idle fan-out silent instead of dropping one message per worker-less partition.
+ /// tick can reach. No owned partition has a worker before boot ends, so a tick to one then is a
+ /// guaranteed `no_worker` no-op (no in-memory state to evict, redrive, or GC); skipping it keeps
+ /// the fan-out silent instead of dropping one message per worker-less partition.
fn tickable_partitions(&self) -> Vec<i32> {
self.owned_partitions()
.into_iter()
@@ -688,195 +657,37 @@ impl EventDispatcher {
counter!(PARTITIONS_ASSIGNED_TOTAL).increment(1);
}
- /// Synchronous half of a revoke: mark the partition un-owned. The worker channel is left intact
- /// for rapid revoke-then-reassign; teardown is decided in the async drain.
- pub fn revoke_partition_sync(&self, partition: i32) {
- self.owned.remove(&partition);
- counter!(PARTITIONS_REVOKED_TOTAL).increment(1);
- }
-
- /// Asynchronous half of a revoke: reclaim the partition unless a reassign re-acquired it.
- /// Re-checks ownership at entry and after the worker join to handle rapid revoke-then-assign.
- pub async fn revoke_partition_drain(&self, partition: i32) {
- if self.owned.contains_key(&partition) {
- counter!(REBALANCE_CLEANUP_SKIPPED_TOTAL, "phase" => "entry").increment(1);
- debug!(
- partition,
- "skipping revoke cleanup: partition re-assigned before cleanup ran"
- );
- return;
- }
-
- let worker = match self.workers.entry(partition) {
- Entry::Occupied(mut slot) => {
- if matches!(slot.get(), WorkerSlot::Draining) {
- debug!(partition, "revoke cleanup is already draining this worker");
- return;
- }
- match slot.insert(WorkerSlot::Draining) {
- WorkerSlot::Running(worker) => Some(worker),
- WorkerSlot::Draining => unreachable!("the occupied slot was checked above"),
- }
- }
- Entry::Vacant(slot) => {
- slot.insert(WorkerSlot::Draining);
- None
- }
- };
- self.router.remove_partition(partition);
- if let Some(worker) = worker {
- let started = Instant::now();
- if let Err(err) = worker.join().await {
- warn!(partition, error = %err, "stage 1 worker panicked during revoke drain");
- }
- histogram!(REVOKE_DRAIN_DURATION_SECONDS).record(started.elapsed().as_secs_f64());
- }
-
- if self.owned.contains_key(&partition) {
- // The draining sentinel prevents an in-flight follower batch from spawning a replacement
- // worker before the old in-memory reconcile queue is gone and its tracker tenure resets.
- // The rebalance worker then rewinds the follower, so anything dropped against the
- // sentinel is replayed into the fresh tenure.
- self.merge.seed_tracker.forget_partition(partition);
- self.finish_worker_drain(partition);
- counter!(REBALANCE_CLEANUP_SKIPPED_TOTAL, "phase" => "post_join").increment(1);
- debug!(
- partition,
- "skipping revoke cleanup post-join: partition re-acquired during the worker drain"
- );
- return;
- }
-
- self.tracker.forget_partition(partition);
- self.merge.merge_tracker.forget_partition(partition);
- self.merge.transfer_tracker.forget_partition(partition);
- self.merge.cascade_tracker.forget_partition(partition);
- self.merge.seed_tracker.forget_partition(partition);
- // Fail-closed on re-acquire: the next tenure re-derives its own watermark.
- self.merge.live_watermarks.forget_partition(partition);
-
- // Reset per-partition gauges. Held-offset gauges in particular are alerted on a sustained
- // non-zero level — without this, a hold that cleared on revoke would keep the alert firing.
- gauge!(MERGE_PENDING_TRANSFERS_GAUGE, "partition" => partition.to_string()).set(0.0);
- gauge!(MERGE_HELD_OFFSET_GAUGE, "partition" => partition.to_string()).set(0.0);
- gauge!(CASCADE_HELD_OFFSET_GAUGE, "partition" => partition.to_string()).set(0.0);
- gauge!(SEED_HELD_OFFSET_GAUGE, "partition" => partition.to_string()).set(0.0);
- gauge!(LIVE_WATERMARK_AGE_MS, "partition" => partition.to_string()).set(0.0);
- gauge!(SEED_PAUSE_AGE_MS, "partition" => partition.to_string()).set(0.0);
- gauge!(SEED_OLDEST_HELD_AGE_MS, "partition" => partition.to_string()).set(0.0);
- gauge!(SLICE_COVERED_SINCE_SECONDS, "partition" => partition.to_string()).set(0.0);
-
- let Some(partition_id) = partition_to_store_id(partition) else {
- warn!(
- partition,
- "revoked partition out of u16 range; skipping state delete"
- );
- self.finish_worker_drain(partition);
- return;
- };
- match self.handle.delete_partition(partition_id).await {
- Ok(()) => counter!(PARTITION_STATE_DELETED_TOTAL).increment(1),
- Err(err) => {
- warn!(partition, error = %err, "failed to delete revoked partition state")
- }
- }
- self.finish_worker_drain(partition);
- }
-
- fn finish_worker_drain(&self, partition: i32) {
- self.workers
- .remove_if(&partition, |_, slot| matches!(slot, WorkerSlot::Draining));
- }
-
- /// Delete a moving-in partition's stale on-disk slice so the worker cold-rebuilds from the
- /// committed offset. Called from [`Self::ensure_worker`] under the partition's shard guard, right
- /// before the worker spawns, so the wipe deterministically precedes the eviction-queue rebuild.
- ///
- /// Boot-snapshot partitions are kept (the boot sweep is the authority and reopen-live is intact),
- /// and pre-boot calls (snapshot not yet recorded) are kept too — only genuine post-boot move-ins
- /// are reclaimed. Must not touch `self.workers`: the caller holds a DashMap shard write guard for
- /// `partition`, so a `contains_key`/`entry` on the same shard would deadlock.
- fn reclaim_stale_slice(&self, partition: i32) {
- match self.boot_assignment.get() {
- // Pre-boot: the sweep is the authority — never wipe.
- None => return,
- Some(boot) if boot.contains(&partition) => return,
- Some(_) => {}
- }
- let Some(partition_id) = partition_to_store_id(partition) else {
- warn!(
- partition,
- "assigned partition out of u16 range; skipping stale-slice reclaim"
- );
- return;
- };
- // Sync escape hatch: runs under the partition's DashMap shard guard, so no `.await`. The
- // delete is one range tombstone per CF and rare (a post-boot move-in).
- match self.handle.delete_partition_blocking(partition_id) {
- Ok(()) => counter!(DURABLE_RESTORE_PARTITIONS_WIPED_STALE_TOTAL).increment(1),
- Err(err) => warn!(
- partition,
- error = %err, "failed to reclaim stale assigned-partition state"
- ),
- }
- }
-
- /// Delete on-disk slices for partitions in `0..partition_count` not in `assignment`: they
- /// moved away while this pod was down. `assignment` must be the settled (non-empty) assignment
- /// — an empty set would delete all slices.
- pub(crate) async fn reclaim_unassigned_partitions_on_boot(
- &self,
- assignment: &std::collections::HashSet<i32>,
- partition_count: usize,
- ) {
- let mut kept = 0u64;
+ /// Delete the on-disk slices of every partition `ownership` does not own. The store opens only
+ /// under the ownership it was written under, or adopts it, so at one pod this deletes nothing.
+ pub(crate) async fn reclaim_unowned_slices(&self, ownership: &PartitionOwnership) {
let mut wiped = 0u64;
- for partition in 0..partition_count as i32 {
- if assignment.contains(&partition) {
- kept += 1;
- continue;
- }
- if let Some(partition_id) = partition_to_store_id(partition) {
- match self.handle.delete_partition(partition_id).await {
- Ok(()) => wiped += 1,
- Err(err) => warn!(
- partition,
- error = %err, "failed to reclaim unassigned-partition state at boot"
- ),
- }
+ for partition in (0..ownership.partition_count()).filter(|&p| !ownership.owns(p)) {
+ match self.handle.delete_partition(partition).await {
+ Ok(()) => wiped += 1,
+ Err(err) => warn!(
+ partition,
+ error = %err, "failed to reclaim unowned-partition state at boot"
+ ),
}
}
+ let kept = ownership.partitions().count() as u64;
counter!(DURABLE_RESTORE_PARTITIONS_KEPT_TOTAL).increment(kept);
counter!(DURABLE_RESTORE_PARTITIONS_WIPED_STALE_TOTAL).increment(wiped);
info!(
kept,
- wiped, partition_count, "durable restore: boot staleness sweep complete"
+ wiped,
+ partition_count = ownership.partition_count(),
+ "durable restore: boot staleness sweep complete"
);
}
- /// Record the boot assignment snapshot, then sweep stale on-disk slices. Recording first makes
- /// the assign path defer to the boot decision and reclaim only post-boot move-ins.
- pub(crate) async fn reconcile_boot_assignment(
- &self,
- assignment: &HashSet<i32>,
- partition_count: usize,
- ) {
- if self.boot_assignment.set(assignment.clone()).is_err() {
- debug!("boot assignment snapshot already recorded; keeping the first");
- }
- self.reclaim_unassigned_partitions_on_boot(assignment, partition_count)
- .await;
- }
-
- /// Re-produce every staged `cf_pending_transfers` entry for the owned partitions at boot,
- /// without spawning workers. An idle partition never spawns a worker — and the periodic redrive
- /// routes through `route_to_owned`, which skips worker-less partitions, so it never fires for one
- /// either; this paginates the whole outbox directly and re-produces every entry regardless of
- /// backlog size.
+ /// Re-produce every staged `cf_pending_transfers` entry for the owned partitions at boot, before
+ /// any worker spawns. This paginates the whole outbox directly and re-produces every entry
+ /// regardless of backlog size.
///
/// Produce + clear only — no `mark_processed`: a fresh tenure starts at `dispatched_offset == 0`
/// so marking would clamp to 0 and trip `CappedAheadOfDispatch`. A per-entry produce failure
- /// leaves that entry for the periodic redrive. Must run after the boot staleness reconcile.
+ /// leaves that entry for the periodic redrive. Must run after `reclaim_unowned_slices`.
pub async fn eager_redrive_pending_transfers_on_boot(&self, assignment: &HashSet<i32>) {
for &partition in assignment {
let Some(store_partition) = partition_to_store_id(partition) else {
@@ -1013,7 +824,7 @@ impl EventDispatcher {
let partitions: Vec<i32> = self.workers.iter().map(|entry| *entry.key()).collect();
let mut workers = 0usize;
for partition in partitions {
- if let Some((_, WorkerSlot::Running(worker))) = self.workers.remove(&partition) {
+ if let Some((_, worker)) = self.workers.remove(&partition) {
workers += 1;
if let Err(err) = worker.join().await {
warn!(partition, error = %err, "stage 1 worker panicked during shutdown drain");
@@ -1054,7 +865,7 @@ fn partition_to_store_id(partition: i32) -> Option<u16> {
u16::try_from(partition).ok()
}
-/// The `cohort_stream_events` group consumer: consume, route, commit.
+/// The `cohort_stream_events` consumer: consume, route, commit.
///
/// Uses a raw `StreamConsumer` with manual commit because the per-partition `OffsetTracker`
/// commits a `TopicPartitionList` that the `common-kafka` wrapper can't express.
@@ -1070,16 +881,15 @@ pub struct CohortStreamEventsConsumer {
recv_batch_size: usize,
recv_batch_timeout: Duration,
offset_commit_interval: Duration,
- /// Partition count of `cohort_stream_events`, used by the boot staleness sweep.
- events_partitions: usize,
- #[allow(dead_code)]
- consumer_command_rx: ConsumerCommandReceiver,
+ ownership: PartitionOwnership,
/// A checkpoint restore boot must finish. Boot rewinds every owned partition the restore keeps
/// to its restored position, commits, and finishes it before anything folds.
restore: Option<PendingRestore>,
}
impl CohortStreamEventsConsumer {
+ /// Takes the partitions `ownership` owns for the life of the process: the consumer is
+ /// assigned them at its group's committed offsets, and the dispatcher owns them.
#[allow(clippy::too_many_arguments)]
pub fn new(
consumer: Arc<StreamConsumer<CohortConsumerContext>>,
@@ -1089,13 +899,22 @@ impl CohortStreamEventsConsumer {
recv_batch_size: usize,
recv_batch_timeout: Duration,
offset_commit_interval: Duration,
- events_partitions: usize,
- consumer_command_rx: ConsumerCommandReceiver,
+ ownership: PartitionOwnership,
restore: Option<PendingRestore>,
- ) -> Self {
+ ) -> KafkaResult<Self> {
+ consumer.assign(&ownership.assignment(&topic)?)?;
+ for partition in ownership.partitions() {
+ dispatcher.assign_partition(i32::from(partition));
+ }
+ let pods: Arc<str> = Arc::from(ownership.pods().to_string());
+ let ordinal: Arc<str> = Arc::from(ownership.ordinal().to_string());
+ gauge!(PARTITION_OWNERSHIP_INFO, "pods" => pods, "ordinal" => ordinal).set(1.0);
+ gauge!(OWNED_PARTITIONS).set(ownership.partitions().count() as f64);
+ info!(%ownership, topic = %topic, "assigned the owned partitions");
+
let pauser: Arc<dyn PartitionPauser> =
Arc::new(ConsumerPauser::new(consumer.clone(), topic.clone()));
- Self {
+ Ok(Self {
consumer,
topic,
dispatcher,
@@ -1104,10 +923,9 @@ impl CohortStreamEventsConsumer {
recv_batch_size,
recv_batch_timeout,
offset_commit_interval,
- events_partitions,
- consumer_command_rx,
+ ownership,
restore,
- }
+ })
}
/// Run until the lifecycle handle signals shutdown.
@@ -1119,6 +937,9 @@ impl CohortStreamEventsConsumer {
pub async fn process(mut self) {
let _guard = self.handle.process_scope();
info!(topic = %self.topic, "cohort_stream_events consume loop starting");
+ // Lifecycle ignores a component until its first report, so this starts the liveness clock
+ // before boot recovery: a recovery or a rewind that never finishes then restarts the process.
+ self.handle.report_healthy();
let commit_task = tokio::spawn(run_commit_loop(
self.consumer.clone(),
@@ -1142,7 +963,20 @@ impl CohortStreamEventsConsumer {
self.restore.take(),
) {
(false, None) => self.go_live(boot_started, 0),
- (_, restore) => BootPhase::settling(restore),
+ (true, None) => {
+ self.recover().await;
+ self.go_live(boot_started, 0)
+ }
+ (_, Some(restore)) => {
+ self.recover().await;
+ let mut resume = ResumePoints::default();
+ let topic = InputTopic::new(self.topic.as_str());
+ resume.rewind_to(&restore.plan().positions(&topic, &self.ownership));
+ BootPhase::Rewinding {
+ resume,
+ restore: Box::new(restore),
+ }
+ }
};
// Untouched until boot recovery completes, so nothing pauses during boot.
let mut backpressure = Backpressure::new();
@@ -1169,26 +1003,6 @@ impl CohortStreamEventsConsumer {
);
BootPhase::Live
}
- BootPhase::Settling { mut previous, mut resume, restore } => {
- resume.hold_back(&outcome.events);
- let assignment = self.assigned_partitions();
- if boot_assignment_settled(&assignment, &mut previous) {
- self.dispatcher
- .reconcile_boot_assignment(&assignment, self.events_partitions)
- .await;
- let owned = self.dispatcher.owned_set();
- self.dispatcher
- .eager_redrive_pending_transfers_on_boot(&owned)
- .await;
- if let Some(restore) = &restore {
- let topic = InputTopic::new(self.topic.as_str());
- resume.rewind_to(&restore.plan().positions(&topic));
- }
- self.rewind(resume, restore, boot_started)
- } else {
- BootPhase::Settling { previous, resume, restore }
- }
- }
BootPhase::Rewinding { mut resume, restore } => {
resume.hold_back(&outcome.events);
self.rewind(resume, restore, boot_started)
@@ -1227,17 +1041,28 @@ impl CohortStreamEventsConsumer {
info!(topic = %self.topic, "cohort_stream_events consume loop stopped");
}
- /// Seek every owned partition back to its resume point, finish a pending restore, then go live.
- /// Any failure keeps boot in `Rewinding`, so the next poll retries and nothing folds from the
+ /// Boot recovery of a reopened or restored store, before the first poll: delete the slices of
+ /// the partitions this pod does not own, and re-produce the merge outbox of those it owns.
+ async fn recover(&self) {
+ self.dispatcher
+ .reclaim_unowned_slices(&self.ownership)
+ .await;
+ self.dispatcher
+ .eager_redrive_pending_transfers_on_boot(&self.dispatcher.owned_set())
+ .await;
+ }
+
+ /// Seek every owned partition back to its resume point, finish the restore, then go live. Any
+ /// failure keeps boot in `Rewinding`, so the next poll retries and nothing folds from the
/// broker offset. `Rewinding` reports no liveness, so a persistent failure restarts the process.
///
- /// A restore commits the seek list before it finishes. Finishing deletes the restore marker, so
- /// without the commit a crash right after would reopen the restored store at the broker's old,
- /// higher offsets and skip the events between.
+ /// The restore commits the seek list before it finishes. Finishing deletes the restore marker,
+ /// so without the commit a crash right after would reopen the restored store at the broker's
+ /// old, higher offsets and skip the events between.
fn rewind(
&self,
resume: ResumePoints,
- restore: Option<PendingRestore>,
+ restore: Box<PendingRestore>,
boot_started: Instant,
) -> BootPhase {
let owned = self.dispatcher.owned_set();
@@ -1285,48 +1110,45 @@ impl CohortStreamEventsConsumer {
}
}
- if let Some(pending) = &restore {
- let low_watermarks = |partitions: &[(&InputTopic, u16)]| {
- list_offsets(self.consumer.as_ref(), partitions, Offset::Beginning)
- };
- let commit_events = || match &seek_list {
- Some(seek_list) => self.consumer.commit(seek_list, CommitMode::Sync),
- None => Ok(()),
- };
- // A sync commit waits until the group coordinator answers. `block_in_place` hands this
- // worker's other tasks to another thread meanwhile, so a runtime with one worker still
- // serves the health server.
- match tokio::task::block_in_place(|| {
- pending.finish(&owned, low_watermarks, commit_events)
- }) {
- Ok(()) => {
- gauge!(CHECKPOINT_RESTORE_BLOCKED).set(0.0);
- info!(
- topic = %self.topic,
- partitions = sought,
- "checkpoint restore finished: the events group committed the restored positions",
- );
- }
- Err(FinishError::Expired(expired)) => {
- // The marker stays, so the next boot's window check resets these slices before
- // anything reads them. Halted never finishes, even if a revoke drops one of them.
- error!(
- ?expired,
- "the broker expired restored positions during boot; stopping so the next boot resets those slices",
- );
- self.handle
- .signal_failure("checkpoint restore positions expired during boot");
- return BootPhase::Halted;
- }
- Err(err) => {
- gauge!(CHECKPOINT_RESTORE_BLOCKED).set(1.0);
- warn!(
- topic = %self.topic,
- error = %format!("{:#}", anyhow::Error::new(err)),
- "checkpoint restore could not finish; holding off dispatch and retrying",
- );
- return BootPhase::Rewinding { resume, restore };
- }
+ let low_watermarks = |partitions: &[(&InputTopic, u16)]| {
+ list_offsets(self.consumer.as_ref(), partitions, Offset::Beginning)
+ };
+ let commit_events = || match &seek_list {
+ Some(seek_list) => self.consumer.commit(seek_list, CommitMode::Sync),
+ None => Ok(()),
+ };
+ // A sync commit waits until the group coordinator answers. `block_in_place` hands this
+ // worker's other tasks to another thread meanwhile, so a runtime with one worker still
+ // serves the health server.
+ match tokio::task::block_in_place(|| restore.finish(&owned, low_watermarks, commit_events))
+ {
+ Ok(()) => {
+ gauge!(CHECKPOINT_RESTORE_BLOCKED).set(0.0);
+ info!(
+ topic = %self.topic,
+ partitions = sought,
+ "checkpoint restore finished: the events group committed the restored positions",
+ );
+ }
+ Err(FinishError::Expired(expired)) => {
+ // The marker stays, so the next boot's window check resets these slices before
+ // anything reads them.
+ error!(
+ ?expired,
+ "the broker expired restored positions during boot; stopping so the next boot resets those slices",
+ );
+ self.handle
+ .signal_failure("checkpoint restore positions expired during boot");
+ return BootPhase::Halted;
+ }
+ Err(err) => {
+ gauge!(CHECKPOINT_RESTORE_BLOCKED).set(1.0);
+ warn!(
+ topic = %self.topic,
+ error = %format!("{:#}", anyhow::Error::new(err)),
+ "checkpoint restore could not finish; holding off dispatch and retrying",
+ );
+ return BootPhase::Rewinding { resume, restore };
}
}
self.go_live(boot_started, sought)
@@ -1341,20 +1163,6 @@ impl CohortStreamEventsConsumer {
BootPhase::Live
}
- fn assigned_partitions(&self) -> HashSet<i32> {
- match self.consumer.assignment() {
- Ok(tpl) => tpl
- .elements_for_topic(&self.topic)
- .iter()
- .map(|elem| elem.partition())
- .collect(),
- Err(err) => {
- warn!(error = %err, "failed to read consumer assignment for boot recovery");
- HashSet::new()
- }
- }
- }
-
/// One steady-state cycle: prune revoked holdover, retry-flush held partitions, dispatch the polled
/// batch, reconcile the paused set and hand its deltas to the pauser task. Every step is
/// non-blocking, so the loop's heartbeat fires each iteration regardless of downstream drain.
@@ -1389,8 +1197,8 @@ impl CohortStreamEventsConsumer {
// Hand the pauser task the paused target (the still-held partitions); it re-asserts pause
// idempotently, so no librdkafka call can delay the heartbeat below. Send every cycle the
- // target is non-empty (re-asserting heals a swallowed pause error or a rebalance-reset toppar),
- // plus the one cycle it clears (to resume); stay silent otherwise.
+ // target is non-empty (re-asserting heals a swallowed pause error), plus the one cycle it
+ // clears (to resume); stay silent otherwise.
let target = backpressure.held_partitions();
if !target.is_empty() || !prev_paused_target.is_empty() {
if pause_tx.send(target.clone()).is_err() {
@@ -1548,8 +1356,8 @@ async fn run_commit_loop(
/// Applies the paused-partition target off the consume loop, so blocking librdkafka `pause`/`resume`
/// never delays the heartbeat. Re-asserts the full target every update — idempotent in librdkafka — so
-/// a swallowed pause error or a rebalance-reset toppar self-heals. `applied` tracks only which
-/// partitions to resume (those that left the target). Exits when the sender is dropped.
+/// a swallowed pause error self-heals. `applied` tracks only which partitions to resume (those that
+/// left the target). Exits when the sender is dropped.
pub(crate) async fn run_pauser_loop(
pauser: Arc<dyn PartitionPauser>,
mut rx: tokio::sync::mpsc::UnboundedReceiver<HashSet<i32>>,
@@ -1584,7 +1392,7 @@ pub(crate) async fn run_pauser_loop(
// facade.
#[allow(clippy::disallowed_methods)]
mod tests {
- use std::num::NonZeroUsize;
+ use std::num::{NonZeroU16, NonZeroUsize};
use super::*;
use chrono_tz::UTC;
@@ -1594,12 +1402,9 @@ mod tests {
use tempfile::TempDir;
use uuid::Uuid;
- use cohort_core::seed::{
- BehavioralShapeHash, CoverageStartMs, ReconcileScope, ReconcileTile, RunBoundaryMs, RunId,
- };
+ use cohort_core::seed::CoverageStartMs;
use crate::consumers::readiness::NotReady;
- use crate::consumers::seeds::SeedWork;
use crate::filters::{CohortId, FilterCatalog, TeamFiltersBuilder, TeamId};
use crate::merge::transfer::{
MergeStateTransfer, PendingTransfer, PersonMergeEvent, Tombstone, TransferLeaf,
@@ -1613,6 +1418,7 @@ mod tests {
};
use crate::stage1::state::AppliedOffsets;
use crate::stage1::{Stage1State, StatefulRecord};
+ use crate::store::durability::PodOrdinal;
use crate::store::{
Behavioral, BehavioralKey, CohortStore, LeafStateKey, MergeAppliedKey, MergeDrainKey,
OffloadConfig, OffloadMode, PendingTransferKey, SliceCoverage, SliceCoverageKey,
@@ -1712,7 +1518,7 @@ mod tests {
let task = tokio::spawn(run_pauser_loop(pauser.clone(), rx));
// Stable target over two cycles: re-paused each time. That idempotent re-assertion is what
- // heals a swallowed pause error or a rebalance-reset toppar.
+ // heals a swallowed pause error.
tx.send(HashSet::from([5])).unwrap();
tx.send(HashSet::from([5])).unwrap();
// Partition 5 drains and 7 becomes held: 5 leaves the target (resumed), 7 is paused.
@@ -2045,133 +1851,25 @@ mod tests {
}
#[tokio::test]
- async fn reclaim_stale_slice_wipes_move_in_keeps_boot_partition() {
- let (_dir, store) = temp_store();
- let catalog = behavioral_catalog();
- let lsk = behavioral_lsk(&catalog);
- let dispatcher = dispatcher_with(&store, catalog);
- seed_slice(&store, 5, lsk);
- seed_slice(&store, 6, lsk);
-
- // Boot snapshot includes 6 (reopen-live) but not 5 (post-boot move-in); partition_count 5
- // keeps the boot sweep off both 5 and 6.
- dispatcher
- .reconcile_boot_assignment(&[6].into_iter().collect(), 5)
- .await;
-
- dispatcher.reclaim_stale_slice(5);
- assert!(
- !slice_present(&store, 5, lsk),
- "a post-boot move-in has its stale slice wiped",
- );
-
- dispatcher.reclaim_stale_slice(6);
- assert!(
- slice_present(&store, 6, lsk),
- "a boot-snapshot partition keeps its slice (the boot sweep is the authority)",
- );
- }
-
- #[tokio::test]
- async fn ensure_worker_wipes_stale_move_in_slice_before_spawn() {
- let (_dir, store) = temp_store();
- let catalog = behavioral_catalog();
- let lsk = behavioral_lsk(&catalog);
- let dispatcher = dispatcher_with(&store, catalog);
- dispatcher.enable_durable_restore();
-
- seed_slice(&store, 5, lsk);
- // Boot excludes 5 (a post-boot move-in); partition_count 5 keeps the boot sweep off 5.
- dispatcher
- .reconcile_boot_assignment(&[6].into_iter().collect(), 5)
- .await;
- let before_move_in = CoverageStartMs::now();
- dispatcher.assign_partition(5);
- store
- .write_batch(|b| {
- b.put::<SliceCoverages>(&SliceCoverageKey(5), &SliceCoverage::Complete.encode())
- })
- .unwrap();
-
- dispatcher.ensure_worker(5);
- assert!(
- dispatcher.workers.contains_key(&5),
- "the moved-in partition spawned a worker",
- );
- assert!(
- !slice_present(&store, 5, lsk),
- "the stale slice was wiped before the worker read it (cold rebuild)",
- );
-
- // A second delivery finds the worker already live: the Entry::Occupied arm is a no-op, so a
- // freshly-written (current, not stale) slice is never re-wiped.
- seed_slice(&store, 5, lsk);
- dispatcher.ensure_worker(5);
- assert!(
- slice_present(&store, 5, lsk),
- "an existing worker keeps its current slice (ensure_worker does not re-reclaim)",
- );
-
- dispatcher.shutdown().await;
- assert!(
- matches!(
- store.slice_coverage(5).unwrap(),
- Some(SliceCoverage::Since(since)) if since >= before_move_in
- ),
- "the wiped slice's complete record went with it, so the moved-in slice began again",
- );
- }
-
- #[tokio::test]
- async fn ensure_worker_with_durable_restore_off_keeps_slice() {
- let (_dir, store) = temp_store();
- let catalog = behavioral_catalog();
- let lsk = behavioral_lsk(&catalog);
- let dispatcher = dispatcher_with(&store, catalog);
- // Durable restore left off: ensure_worker must never reclaim — wipe-on-start handles staleness.
-
- seed_slice(&store, 5, lsk);
- dispatcher
- .reconcile_boot_assignment(&[6].into_iter().collect(), 5)
- .await;
- dispatcher.assign_partition(5);
-
- dispatcher.ensure_worker(5);
- assert!(dispatcher.workers.contains_key(&5), "the worker spawned");
- assert!(
- slice_present(&store, 5, lsk),
- "with durable restore off, ensure_worker leaves the on-disk slice untouched",
- );
-
- dispatcher.shutdown().await;
- }
-
- #[tokio::test]
- async fn reclaim_unassigned_partitions_on_boot_wipes_only_unassigned_slices() {
+ async fn the_boot_sweep_wipes_only_the_slices_another_pod_owns() {
let (_dir, store) = temp_store();
let catalog = behavioral_catalog();
let lsk = behavioral_lsk(&catalog);
let dispatcher = dispatcher_with(&store, catalog);
- for partition in 0..3u16 {
+ for partition in 0..4u16 {
seed_slice(&store, partition, lsk);
}
+ let pod_zero_of_two =
+ PartitionOwnership::new(NonZeroU16::new(2).unwrap(), PodOrdinal::STANDALONE, 4)
+ .unwrap();
- let assignment: HashSet<i32> = [0, 2].into_iter().collect();
- dispatcher
- .reclaim_unassigned_partitions_on_boot(&assignment, 3)
- .await;
+ dispatcher.reclaim_unowned_slices(&pod_zero_of_two).await;
- assert!(
- slice_present(&store, 0, lsk),
- "assigned 0 is kept (reopen-live)",
- );
- assert!(
- !slice_present(&store, 1, lsk),
- "unassigned 1 is wiped (stale slice from a previous tenure)",
- );
- assert!(
- slice_present(&store, 2, lsk),
- "assigned 2 is kept (reopen-live)",
+ assert_eq!(
+ (0..4u16)
+ .map(|partition| slice_present(&store, partition, lsk))
+ .collect::<Vec<_>>(),
+ [true, false, true, false],
);
}
@@ -2613,35 +2311,6 @@ mod tests {
);
}
- #[tokio::test]
- async fn revoke_partition_drains_produces_evicts_and_deletes_state() {
- let (_dir, store) = temp_store();
- let catalog = behavioral_catalog();
- let lsk = behavioral_lsk(&catalog);
- let (dispatcher, sink) = dispatcher_and_sink(&store, catalog);
-
- dispatcher.assign_partition(0);
- dispatcher.dispatch(vec![consumed(person(1), 0, 10)]).await;
-
- dispatcher.revoke_partition_sync(0);
- assert!(!dispatcher.owns(0));
- assert_eq!(dispatcher.router.partition_count(), 1);
-
- dispatcher.revoke_partition_drain(0).await;
-
- let changes = sink.changes();
- assert_eq!(changes.len(), 1);
- assert_eq!(changes[0].status, MembershipStatus::Entered);
- assert_eq!(changes[0].person_id, person(1).to_string());
-
- assert_eq!(dispatcher.workers.len(), 0, "worker evicted");
- assert_eq!(dispatcher.router.partition_count(), 0, "sender removed");
- assert!(
- behavioral_state(&store, 0, person(1), lsk).is_none(),
- "the partition's state slice was deleted",
- );
- }
-
/// A worker resolves its slice coverage first, so a processed offset proves it resolved.
async fn processed_through(dispatcher: &EventDispatcher, partition: i32, next: i64) {
let start = Instant::now();
@@ -2655,7 +2324,7 @@ mod tests {
}
#[tokio::test]
- async fn reassign_after_a_full_revoke_respawns_and_advances_offsets() {
+ async fn a_spawn_resumes_recorded_coverage_and_begins_a_new_slice_at_its_assignment() {
let (_dir, store) = temp_store();
let dispatcher = dispatcher_with(&store, behavioral_catalog());
// Recorded by an earlier tenure.
@@ -2672,274 +2341,30 @@ mod tests {
Some(earlier),
"a spawn resumes the recorded coverage instead of beginning the slice again",
);
- dispatcher.revoke_partition_sync(0);
- dispatcher.revoke_partition_drain(0).await;
- assert_eq!(dispatcher.workers.len(), 0);
- assert_eq!(
- store.slice_coverage(0).unwrap(),
- None,
- "the revoke deleted it"
- );
- let before_respawn = CoverageStartMs::now();
- dispatcher.assign_partition(0);
- let reassigned = CoverageStartMs::now();
+ let before_assignment = CoverageStartMs::now();
+ dispatcher.assign_partition(1);
+ let assigned = CoverageStartMs::now();
// Spawn in a later millisecond, so a slice begun at spawn fails the bound below.
- while CoverageStartMs::now() <= reassigned {
+ while CoverageStartMs::now() <= assigned {
tokio::task::yield_now().await;
}
- dispatcher.dispatch(vec![consumed(person(2), 0, 20)]).await;
- assert_eq!(
- dispatcher.workers.len(),
- 1,
- "a reassigned partition respawns"
- );
- processed_through(&dispatcher, 0, 21).await;
+ dispatcher.dispatch(vec![consumed(person(2), 1, 20)]).await;
+ processed_through(&dispatcher, 1, 21).await;
assert!(
matches!(
- store.slice_coverage(0).unwrap(),
+ store.slice_coverage(1).unwrap(),
Some(SliceCoverage::Since(since))
- if before_respawn <= since && since <= reassigned
- ),
- "a revoked slice comes back without its history, never complete, and begins at its \
- reassignment rather than its first delivery",
- );
-
- let tracker = dispatcher.shutdown().await;
- assert_eq!(tracker.committable_offsets().get(&0), Some(&21));
- }
-
- #[tokio::test]
- async fn rapid_revoke_then_assign_skips_cleanup_and_keeps_the_live_worker() {
- let (_dir, store) = temp_store();
- let catalog = behavioral_catalog();
- let lsk = behavioral_lsk(&catalog);
- let dispatcher = dispatcher_with(&store, catalog);
-
- dispatcher.assign_partition(0);
- dispatcher.dispatch(vec![consumed(person(1), 0, 10)]).await;
-
- dispatcher.revoke_partition_sync(0);
- dispatcher.assign_partition(0);
- dispatcher.revoke_partition_drain(0).await;
-
- assert!(dispatcher.owns(0), "re-owned partition stays owned");
- assert_eq!(dispatcher.workers.len(), 1, "worker preserved");
- assert_eq!(dispatcher.router.partition_count(), 1, "sender preserved");
-
- dispatcher.dispatch(vec![consumed(person(2), 0, 11)]).await;
- let tracker = dispatcher.shutdown().await;
- assert_eq!(tracker.committable_offsets().get(&0), Some(&12));
- assert!(
- behavioral_state(&store, 0, person(1), lsk).is_some(),
- "state survived the skipped cleanup",
- );
- }
-
- #[tokio::test]
- async fn post_join_reacquire_starts_a_fresh_seed_tenure_for_reconcile_replay() {
- let (_dir, store) = temp_store();
- let sink = BlockingMembershipSink::new();
- let reconcile = crate::workers::ReconcileDeps {
- enabled: true,
- ..crate::workers::ReconcileDeps::default()
- };
- let merge = Arc::new(MergeWorkerDeps {
- transfer_sink: Arc::new(CaptureTransferSink::new()),
- stream_event_sink: Arc::new(CaptureStreamEventSink::new()),
- merge_tracker: Arc::new(OffsetTracker::new()),
- transfer_tracker: Arc::new(OffsetTracker::new()),
- retry: TransferRetryPolicy::default(),
- gc_scan_limit: crate::workers::DEFAULT_MERGE_GC_SCAN_LIMIT,
- stage2_orphan_gc_enabled: true,
- cascade_sink: Arc::new(crate::producer::CaptureCascadeSink::new()),
- cascade_tracker: Arc::new(OffsetTracker::new()),
- cascade: crate::workers::CascadeConfig::default(),
- partition_count: COHORT_PARTITION_COUNT,
- seed_tile_sink: Arc::new(crate::producer::CaptureSeedTileSink::new()),
- seed_tracker: Arc::new(OffsetTracker::new()),
- live_watermarks: Arc::new(crate::partitions::watermarks::LiveWatermarks::new()),
- register_transfer_enabled: false,
- reconcile,
- person_seed: crate::workers::PersonSeedDeps::default(),
- seed_budget: crate::workers::seed_run::RunBudget::default(),
- });
- let dispatcher = Arc::new(EventDispatcher::new(
- PartitionRouter::new(64),
- Arc::new(OffsetTracker::new()),
- test_handle(&store),
- behavioral_catalog(),
- sink.clone(),
- merge.clone(),
- ));
- dispatcher.assign_partition(0);
-
- let tile = ReconcileTile::new(
- TeamId(TEAM),
- CohortId(1),
- ReconcileScope::Behavioral(BehavioralShapeHash::parse("0123456789abcdef").unwrap()),
- RunId(Uuid::from_u128(1)),
- RunBoundaryMs(0),
- );
- let held = dispatcher.dispatch_seeds(vec![ConsumedSeed {
- work: SeedWork::Reconcile(tile.clone()),
- partition: 0,
- offset: 5,
- broker_ts_ms: None,
- }]);
- assert!(held.is_empty());
- for _ in 0..10_000 {
- if merge.reconcile.backlog.len() == 1 {
- break;
- }
- tokio::task::yield_now().await;
- }
- assert_eq!(merge.reconcile.backlog.len(), 1, "reconcile was admitted");
-
- dispatcher.dispatch(vec![consumed(person(1), 0, 6)]).await;
- sink.entered.notified().await;
-
- dispatcher.revoke_partition_sync(0);
- let drain = {
- let dispatcher = dispatcher.clone();
- tokio::spawn(async move { dispatcher.revoke_partition_drain(0).await })
- };
- for _ in 0..10_000 {
- if dispatcher
- .workers
- .get(&0)
- .is_some_and(|slot| matches!(slot.value(), WorkerSlot::Draining))
- {
- break;
- }
- tokio::task::yield_now().await;
- }
- assert!(
- dispatcher
- .workers
- .get(&0)
- .is_some_and(|slot| matches!(slot.value(), WorkerSlot::Draining)),
- "the old worker was replaced by a draining sentinel before its blocked join",
- );
-
- dispatcher.assign_partition(0);
- let held_events = dispatcher
- .dispatch_events_nonblocking(vec![consumed(person(2), 0, 7)], &HashSet::new());
- assert_eq!(
- held_offsets(
- held_events
- .get(&0)
- .expect("a primary event is held behind the draining worker"),
+ if before_assignment <= since && since <= assigned
),
- vec![7],
- );
- let mut held = dispatcher.dispatch_seeds(vec![ConsumedSeed {
- work: SeedWork::Reconcile(tile.clone()),
- partition: 0,
- offset: 5,
- broker_ts_ms: None,
- }]);
- assert_eq!(
- held_seed_offsets(held.get(&0).expect("the draining route is held")),
- vec![5],
- );
- assert!(
- dispatcher
- .workers
- .get(&0)
- .is_some_and(|slot| matches!(slot.value(), WorkerSlot::Draining)),
- "an in-flight follower batch cannot replace the draining worker",
- );
-
- sink.release.notify_one();
- drain.await.unwrap();
-
- assert!(dispatcher.owns(0));
- assert!(merge.reconcile.backlog.is_empty());
- assert!(dispatcher.workers.is_empty(), "the sentinel was released");
-
- let still_held = dispatcher.redispatch_held(held_events.into_iter().collect());
- assert!(
- still_held.is_empty(),
- "the primary event routes after the draining sentinel is released",
- );
- let held = dispatcher.dispatch_seeds(
- held.remove(&0)
- .expect("the held follower batch is retried after the drain"),
- );
- assert!(held.is_empty());
- for _ in 0..10_000 {
- if merge.reconcile.backlog.len() == 1 {
- break;
- }
- tokio::task::yield_now().await;
- }
- assert_eq!(merge.reconcile.backlog.len(), 1, "replay was re-admitted");
- assert_eq!(
- merge.seed_tracker.mark_processed(0, 6),
- MarkOutcome::WithinDispatch,
- "worker receipt raises the fresh tenure's seed ceiling before admission",
- );
- assert_eq!(
- merge.seed_tracker.committable_offsets().get(&0),
- Some(&5),
- "the replayed reconcile owns the fresh tenure's commit floor",
- );
- for _ in 0..10_000 {
- if dispatcher.tracker().committable_offsets().get(&0) == Some(&8) {
- break;
- }
- tokio::task::yield_now().await;
- }
- assert_eq!(
- dispatcher.tracker().committable_offsets().get(&0),
- Some(&8),
- "the held primary event is processed after worker replacement",
+ "a slice without a record begins at its assignment rather than its first delivery",
);
dispatcher.shutdown().await;
}
#[tokio::test]
- async fn revoke_then_shutdown_drain_each_worker_exactly_once() {
- let (_dir, store) = temp_store();
- let (dispatcher, sink) = dispatcher_and_sink(&store, behavioral_catalog());
-
- dispatcher.assign_partition(0);
- dispatcher.assign_partition(1);
- dispatcher
- .dispatch(vec![consumed(person(1), 0, 10), consumed(person(2), 1, 20)])
- .await;
- assert_eq!(dispatcher.workers.len(), 2);
-
- dispatcher.revoke_partition_sync(0);
- dispatcher.revoke_partition_drain(0).await;
- assert_eq!(dispatcher.workers.len(), 1);
-
- let tracker = dispatcher.shutdown().await;
- assert_eq!(dispatcher.workers.len(), 0);
-
- assert_eq!(
- sink.changes()
- .iter()
- .filter(|change| change.person_id == person(1).to_string())
- .count(),
- 1,
- );
- assert_eq!(
- tracker.committable_offsets().get(&0),
- None,
- "the revoked partition's offset was forgotten on drain, not left to re-commit",
- );
- assert_eq!(
- tracker.committable_offsets().get(&1),
- Some(&21),
- "the survivor drained on shutdown",
- );
- }
-
- #[tokio::test]
- async fn the_drain_signal_waits_for_every_worker_and_keeps_the_ownership_it_drained_under() {
+ async fn the_drain_signal_waits_for_every_worker_and_records_the_ownership_it_drained_under() {
let (_dir, store) = temp_store();
let sink = BlockingMembershipSink::new();
let dispatcher = Arc::new(EventDispatcher::new(
@@ -2968,9 +2393,6 @@ mod tests {
sink.release.notify_one();
drain.await.unwrap();
- // Closing the events consumer revokes every partition after the drain.
- dispatcher.revoke_partition_sync(0);
- dispatcher.revoke_partition_sync(1);
let DrainWait::Drained(drained) = dispatcher
.wait_for_worker_drain(Duration::from_secs(10))
.await
@@ -3003,7 +2425,7 @@ mod tests {
)
.await,
HashMap::from([(0, 6), (1, 6)]),
- "a follower commits for the partitions drained, not the emptied live set",
+ "a follower commits for the partitions drained, not the set it saw at shutdown",
);
}
@@ -3036,83 +2458,21 @@ mod tests {
dispatcher.shutdown().await;
}
- #[tokio::test]
- async fn revoke_partition_drain_forgets_the_offset_so_it_is_not_recommitted() {
- let (_dir, store) = temp_store();
- let (dispatcher, sink) = dispatcher_and_sink(&store, behavioral_catalog());
-
- dispatcher.assign_partition(0);
- dispatcher.assign_partition(1);
- dispatcher
- .dispatch(vec![consumed(person(1), 0, 10), consumed(person(2), 1, 20)])
- .await;
-
- dispatcher.revoke_partition_sync(0);
- dispatcher.revoke_partition_drain(0).await;
-
- assert_eq!(
- sink.changes()
- .iter()
- .filter(|change| change.person_id == person(1).to_string())
- .count(),
- 1,
- );
- assert_eq!(dispatcher.tracker().committable_offsets().get(&0), None);
- assert!(!dispatcher.owned_committable_offsets().contains_key(&0));
-
- let tracker = dispatcher.shutdown().await;
- assert_eq!(tracker.committable_offsets().get(&1), Some(&21));
- }
-
#[tokio::test]
async fn owned_committable_offsets_excludes_unowned_partitions() {
let (_dir, store) = temp_store();
let dispatcher = dispatcher_with(&store, behavioral_catalog());
+ dispatcher.assign_partition(1);
for (partition, next) in [(0, 11), (1, 21)] {
- dispatcher.assign_partition(partition);
dispatcher.tracker().mark_dispatched(partition, next);
let _ = dispatcher.tracker().mark_processed(partition, next);
}
- assert_eq!(dispatcher.owned_committable_offsets().get(&0), Some(&11));
-
- dispatcher.revoke_partition_sync(0);
assert_eq!(
- dispatcher.tracker().committable_offsets().get(&0),
- Some(&11),
- "the entry lingers in the tracker until the async drain forgets it",
+ dispatcher.owned_committable_offsets(),
+ HashMap::from([(1, 21)])
);
- assert!(!dispatcher.owned_committable_offsets().contains_key(&0));
- assert_eq!(dispatcher.owned_committable_offsets().get(&1), Some(&21));
- }
-
- #[tokio::test]
- async fn assign_and_revoke_track_ownership() {
- let (_dir, store) = temp_store();
- let dispatcher = dispatcher_with(&store, behavioral_catalog());
-
- assert!(!dispatcher.owns(3));
- dispatcher.assign_partition(3);
- assert!(dispatcher.owns(3));
- dispatcher.revoke_partition_sync(3);
- assert!(!dispatcher.owns(3));
- }
-
- #[tokio::test]
- async fn owned_partitions_reflects_assign_and_revoke() {
- let (_dir, store) = temp_store();
- let dispatcher = dispatcher_with(&store, behavioral_catalog());
-
- assert!(dispatcher.owned_partitions().is_empty());
- dispatcher.assign_partition(3);
- dispatcher.assign_partition(7);
- let mut owned = dispatcher.owned_partitions();
- owned.sort_unstable();
- assert_eq!(owned, vec![3, 7]);
-
- dispatcher.revoke_partition_sync(3);
- assert_eq!(dispatcher.owned_partitions(), vec![7]);
}
#[tokio::test]
@@ -3175,6 +2535,7 @@ mod tests {
let (_dir, store) = temp_store();
let dispatcher = dispatcher_with(&store, behavioral_catalog());
+ dispatcher.route_sweep(123).await;
dispatcher.assign_partition(0);
dispatcher.route_sweep(123).await;
assert_eq!(dispatcher.workers.len(), 0, "a sweep never spawns a worker");
@@ -3183,9 +2544,6 @@ mod tests {
0,
"no channel registered"
);
-
- dispatcher.revoke_partition_sync(0);
- dispatcher.route_sweep(123).await;
}
#[tokio::test]
@@ -3227,47 +2585,6 @@ mod tests {
}
}
- #[tokio::test]
- async fn dispatch_after_full_revoke_does_not_resurrect_partition() {
- let (_dir, store) = temp_store();
- let catalog = behavioral_catalog();
- let lsk = behavioral_lsk(&catalog);
- let dispatcher = dispatcher_with(&store, catalog);
-
- dispatcher.assign_partition(0);
- dispatcher.dispatch(vec![consumed(person(1), 0, 10)]).await;
- dispatcher.revoke_partition_sync(0);
- dispatcher.revoke_partition_drain(0).await;
-
- assert_eq!(dispatcher.workers.len(), 0);
- assert_eq!(dispatcher.router.partition_count(), 0);
- assert!(behavioral_state(&store, 0, person(1), lsk).is_none());
-
- dispatcher.dispatch(vec![consumed(person(2), 0, 11)]).await;
-
- assert_eq!(
- dispatcher.workers.len(),
- 0,
- "no worker respawned for the unowned partition",
- );
- assert_eq!(
- dispatcher.router.partition_count(),
- 0,
- "no sender re-registered",
- );
- assert!(
- behavioral_state(&store, 0, person(2), lsk).is_none(),
- "no fresh slice written for the reclaimed partition",
- );
- assert_eq!(
- dispatcher.tracker().partition_count(),
- 0,
- "the dropped event raised no dispatch ceiling",
- );
- assert!(!dispatcher.tracker().committable_offsets().contains_key(&0));
- assert!(!dispatcher.owned_committable_offsets().contains_key(&0));
- }
-
#[tokio::test]
async fn ensure_worker_does_not_spawn_for_an_unowned_partition() {
let (_dir, store) = temp_store();
@@ -3294,16 +2611,13 @@ mod tests {
}
#[tokio::test]
- async fn dispatch_routes_owned_and_drops_revoked_without_raising_its_ceiling() {
+ async fn dispatch_routes_owned_and_drops_unowned_without_raising_its_ceiling() {
let (_dir, store) = temp_store();
let catalog = behavioral_catalog();
let lsk = behavioral_lsk(&catalog);
let dispatcher = dispatcher_with(&store, catalog);
dispatcher.assign_partition(0);
- dispatcher.assign_partition(1);
- dispatcher.revoke_partition_sync(1);
- dispatcher.revoke_partition_drain(1).await;
dispatcher
.dispatch(vec![consumed(person(1), 0, 10), consumed(person(2), 1, 20)])
@@ -3327,25 +2641,10 @@ mod tests {
assert!(behavioral_state(&store, 0, person(1), lsk).is_some());
assert!(
behavioral_state(&store, 1, person(2), lsk).is_none(),
- "the revoked partition wrote no state",
+ "the unowned partition wrote no state",
);
}
- #[tokio::test]
- async fn revoke_drain_of_a_never_spawned_partition_is_a_clean_noop() {
- let (_dir, store) = temp_store();
- let dispatcher = dispatcher_with(&store, behavioral_catalog());
-
- dispatcher.assign_partition(3);
- dispatcher.revoke_partition_sync(3);
- dispatcher.revoke_partition_drain(3).await;
-
- assert_eq!(dispatcher.workers.len(), 0);
- assert_eq!(dispatcher.router.partition_count(), 0);
- assert!(!dispatcher.owns(3));
- assert_eq!(dispatcher.tracker().partition_count(), 0);
- }
-
#[tokio::test]
async fn shutdown_drains_a_late_dispatch_without_stranding_its_worker() {
let (_dir, store) = temp_store();
@@ -3374,54 +2673,6 @@ mod tests {
assert_eq!(sink.changes().len(), 1, "the worker flushed before exiting");
}
- #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
- async fn concurrent_revoke_and_dispatch_never_resurrects_an_unowned_partition() {
- let (_dir, store) = temp_store();
- let catalog = behavioral_catalog();
- let lsk = behavioral_lsk(&catalog);
- let dispatcher = Arc::new(dispatcher_with(&store, catalog));
-
- const P: i32 = 0;
- for round in 0..200u128 {
- dispatcher.assign_partition(P);
-
- let dispatch = {
- let dispatcher = dispatcher.clone();
- tokio::spawn(async move {
- dispatcher
- .dispatch(vec![consumed(person(round + 1), P, round as i64)])
- .await;
- })
- };
-
- dispatcher.revoke_partition_sync(P);
- let drain = {
- let dispatcher = dispatcher.clone();
- tokio::spawn(async move {
- dispatcher.revoke_partition_drain(P).await;
- })
- };
-
- dispatch.await.unwrap();
- drain.await.unwrap();
-
- assert!(!dispatcher.owns(P));
- assert!(
- !dispatcher.workers.contains_key(&P),
- "round {round}: a worker survived an unowned partition",
- );
- assert_eq!(
- dispatcher.router.partition_count(),
- 0,
- "round {round}: a sender survived an unowned partition",
- );
- assert!(
- behavioral_state(&store, P as u16, person(round + 1), lsk).is_none(),
- "round {round}: a state slice survived an unowned partition",
- );
- }
- }
-
const MERGED_AT: i64 = 1_716_800_000_000;
fn composable_catalog() -> Arc<CatalogHandle> {
@@ -3948,6 +3199,7 @@ mod tests {
let (_dir, store) = temp_store();
let dispatcher = dispatcher_with(&store, behavioral_catalog());
+ dispatcher.route_merge_gc(1_000, 500).await;
dispatcher.assign_partition(0);
dispatcher.route_merge_gc(1_000, 500).await;
assert_eq!(dispatcher.workers.len(), 0, "GC never spawns a worker");
@@ -3956,9 +3208,6 @@ mod tests {
0,
"no channel registered"
);
-
- dispatcher.revoke_partition_sync(0);
- dispatcher.route_merge_gc(1_000, 500).await;
}
#[tokio::test]
diff --git a/rust/cohort-stream-processor/src/consumers/merges.rs b/rust/cohort-stream-processor/src/consumers/merges.rs
index 0749d7957cee..629b42b199fc 100644
--- a/rust/cohort-stream-processor/src/consumers/merges.rs
+++ b/rust/cohort-stream-processor/src/consumers/merges.rs
@@ -1,8 +1,8 @@
//! Merge-protocol follower consumers: `person_merge_events` (drain trigger, keyed by P_old) and
//! `cohort_merge_state_transfer` (packaged state, keyed by P_new).
//!
-//! Both use [`FollowerConsumer`] generic over a [`FollowerRoute`]. Followers never `subscribe()` —
-//! the events group's rebalance mirrors partition ownership onto them.
+//! Both use [`FollowerConsumer`] generic over a [`FollowerRoute`]. Followers never `subscribe()`:
+//! each is assigned the partitions this pod owns, like the events consumer.
use std::collections::{HashMap, HashSet};
use std::marker::PhantomData;
diff --git a/rust/cohort-stream-processor/src/consumers/seeds.rs b/rust/cohort-stream-processor/src/consumers/seeds.rs
index 5be48adf4f29..575839e62c7a 100644
--- a/rust/cohort-stream-processor/src/consumers/seeds.rs
+++ b/rust/cohort-stream-processor/src/consumers/seeds.rs
@@ -1,10 +1,10 @@
//! The `cohort_stream_seed_events` follower consumer — the backfill day-tile input.
//!
-//! Assignment arrives via the events group's rebalance mirror. Four admission gates share one
-//! holdover + pause mechanism: the apply fence (a tile dispatches only once its partition's live
-//! watermark clears `s_chunk + margin`), a full seed lane, live-priority (the partition's live
-//! watermark age crossed the pause threshold — live traffic always wins), and pod-wide disk
-//! pressure. An un-dispatched tile was never `mark_dispatched`ed, so its offset cannot commit.
+//! It is assigned the partitions this pod owns, like the events consumer. Four admission gates
+//! share one holdover + pause mechanism: the apply fence (a tile dispatches only once its
+//! partition's live watermark clears `s_chunk + margin`), a full seed lane, live-priority (the
+//! partition's live watermark age crossed the pause threshold — live traffic always wins), and
+//! pod-wide disk pressure. An un-dispatched tile was never `mark_dispatched`ed, so its offset cannot commit.
//! The live-lag gate remains an admission backstop because an admitted seed run still uses the
//! worker and its CPU and I/O even though separate lanes keep queued seeds behind live traffic.
//! Consume-side skips ride the seed lane so their offsets mark in order; they never close the
diff --git a/rust/cohort-stream-processor/src/main.rs b/rust/cohort-stream-processor/src/main.rs
index 21be8a2bcb11..eb4af5072710 100644
--- a/rust/cohort-stream-processor/src/main.rs
+++ b/rust/cohort-stream-processor/src/main.rs
@@ -8,7 +8,6 @@ use envconfig::Envconfig;
use lifecycle::{ComponentOptions, Handle, Manager};
use rdkafka::consumer::{Consumer, ConsumerContext, StreamConsumer};
use tokio::net::TcpListener;
-use tokio::sync::mpsc;
use tracing::{info, warn};
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
@@ -27,8 +26,8 @@ use cohort_stream_processor::observability::disk::SharedDiskUtilization;
use cohort_stream_processor::observability::store_stats::{DiskProbe, StoreStatsSweeper};
use cohort_stream_processor::observability::tokio_monitor::TokioRuntimeMonitor;
use cohort_stream_processor::partitions::{
- run_rebalance_worker, CohortConsumerContext, ConsumerPauser, Follower, FollowerSet,
- InputGroups, LiveWatermarks, OffsetTracker, PartitionPauser, PartitionRouter,
+ CohortConsumerContext, ConsumerPauser, Follower, FollowerSet, InputGroups, LiveWatermarks,
+ OffsetTracker, PartitionPauser, PartitionRouter,
};
use cohort_stream_processor::producer::{
CascadeSink, KafkaCascadeSink, KafkaMembershipSink, KafkaReconcileMarkerSink,
@@ -73,6 +72,7 @@ async fn async_main(config: Config) -> Result<()> {
log_startup(&config);
config.validate_startup()?;
+ let ownership = config.partition_ownership()?;
let lineage = config
.checkpoint_enabled
@@ -192,6 +192,126 @@ async fn async_main(config: Config) -> Result<()> {
),
}
+ let stream_consumer: StreamConsumer<CohortConsumerContext> = config
+ .consumer_client_config()
+ .create_with_context(CohortConsumerContext)
+ .context("creating cohort_stream_events consumer")?;
+ // Shared with the seed consumer's idle probe.
+ let stream_consumer = Arc::new(stream_consumer);
+
+ let merges_follower_consumer: Arc<StreamConsumer> = Arc::new(
+ config
+ .follower_client_config(&config.kafka_merge_consumer_group)
+ .create()
+ .context("creating person_merge_events follower consumer")?,
+ );
+ let transfers_follower_consumer: Arc<StreamConsumer> = Arc::new(
+ config
+ .follower_client_config(&config.kafka_merge_apply_consumer_group)
+ .create()
+ .context("creating cohort_merge_state_transfer follower consumer")?,
+ );
+ // Built only when the gate is on, so a dormant deploy needs no `cohort_cascade_events` topic.
+ let cascade_follower_consumer: Option<Arc<StreamConsumer>> = if config.cohort_cascade_enabled {
+ Some(Arc::new(
+ config
+ .follower_client_config(&config.kafka_cascade_consumer_group)
+ .create()
+ .context("creating cohort_cascade_events follower consumer")?,
+ ))
+ } else {
+ None
+ };
+ // Gated: a gate-off deploy is safe without the seed topic existing at all.
+ let seed_follower_consumer: Option<Arc<StreamConsumer>> = if config.cohort_seed_consumer_enabled
+ {
+ Some(Arc::new(
+ config
+ .follower_client_config(&config.kafka_seed_consumer_group)
+ .create()
+ .context("creating cohort_stream_seed_events follower consumer")?,
+ ))
+ } else {
+ None
+ };
+
+ let events_partitions =
+ fetch_partition_count(&stream_consumer, &config.cohort_stream_events_topic)?;
+ let merge_partitions =
+ fetch_partition_count(&merges_follower_consumer, &config.person_merge_events_topic)?;
+ let transfer_partitions = fetch_partition_count(
+ &transfers_follower_consumer,
+ &config.cohort_merge_state_transfer_topic,
+ )?;
+ // Checked before the store opens, because the store records the partition count it opens under.
+ // The merge math hashes `(team, person)` against `config.cohort_partition_count`, so the topics
+ // must not only be co-partitioned with each other but partitioned at exactly that count — a
+ // deploy/lane at N != the configured count silently misroutes every merge.
+ anyhow::ensure!(
+ events_partitions as u32 == config.cohort_partition_count,
+ "{} is partitioned at {} but COHORT_PARTITION_COUNT is {}: the merge partition arithmetic \
+ would misroute. Re-partition the topic to {} or set COHORT_PARTITION_COUNT to {}.",
+ config.cohort_stream_events_topic,
+ events_partitions,
+ config.cohort_partition_count,
+ config.cohort_partition_count,
+ events_partitions,
+ );
+ anyhow::ensure!(
+ merge_partitions == events_partitions && transfer_partitions == events_partitions,
+ "merge topics must be co-partitioned with {} ({} partitions): {} has {}, {} has {}",
+ config.cohort_stream_events_topic,
+ events_partitions,
+ config.person_merge_events_topic,
+ merge_partitions,
+ config.cohort_merge_state_transfer_topic,
+ transfer_partitions,
+ );
+ // A cascade for (team, person) must land on the partition owning that person's `cf_stage2` — the
+ // same partition number as the events topic — so refuse to start co-partitioned at a different
+ // count. Skipped when the gate is off (no cascade topic required).
+ if let Some(cascade_consumer) = &cascade_follower_consumer {
+ let cascade_partitions =
+ fetch_partition_count(cascade_consumer, &config.cohort_cascade_events_topic)?;
+ anyhow::ensure!(
+ cascade_partitions as u32 == config.cohort_partition_count,
+ "cohort_cascade_events must be co-partitioned with {} at COHORT_PARTITION_COUNT={}: {} has {}",
+ config.cohort_stream_events_topic,
+ config.cohort_partition_count,
+ config.cohort_cascade_events_topic,
+ cascade_partitions,
+ );
+ }
+ // A tile must land on the partition owning its person's state slice.
+ if let Some(seed_consumer) = &seed_follower_consumer {
+ let seed_partitions =
+ fetch_partition_count(seed_consumer, &config.cohort_stream_seed_events_topic)?;
+ anyhow::ensure!(
+ seed_partitions as u32 == config.cohort_partition_count,
+ "cohort_stream_seed_events must be co-partitioned with {} at COHORT_PARTITION_COUNT={}: {} has {}",
+ config.cohort_stream_events_topic,
+ config.cohort_partition_count,
+ config.cohort_stream_seed_events_topic,
+ seed_partitions,
+ );
+ // Nothing co-partitions with the marker topic, so only its existence matters. Failing here
+ // beats the alternative: a marker produce retrying forever while it holds the seed offset.
+ // Nested under the seed consumer on purpose: reconcile jobs are admitted only from seed
+ // tiles, so without it no marker can be produced and there is nothing to prove.
+ if config.cohort_seed_reconcile_enabled {
+ fetch_partition_count(seed_consumer, &config.cohort_reconcile_markers_topic).with_context(
+ || {
+ format!(
+ "{} must exist before a processor with COHORT_SEED_RECONCILE_ENABLED starts. \
+ Provision the topic, or set COHORT_SEED_RECONCILE_ENABLED=false to start \
+ without the reconcile path.",
+ config.cohort_reconcile_markers_topic,
+ )
+ },
+ )?;
+ }
+ }
+
let groups = Arc::new(InputGroups::new(&config));
if lineage.is_some() {
ensure_one_filesystem(
@@ -200,7 +320,7 @@ async fn async_main(config: Config) -> Result<()> {
)
.context("checking the checkpoint directory")?;
}
- let (store, restore) = open_store(&config, lineage, groups.clone())
+ let (store, restore) = open_store(&config, ownership, lineage, groups.clone())
.await
.context("opening RocksDB state store")?;
let router = PartitionRouter::with_intake_cap(
@@ -331,130 +451,7 @@ async fn async_main(config: Config) -> Result<()> {
// Event-name fan-out gating, likewise set before any worker spawns.
dispatcher.set_event_name_gating(config.event_name_gating());
- let (context, rebalance_rx) = CohortConsumerContext::new(dispatcher.clone());
- let stream_consumer: StreamConsumer<CohortConsumerContext> = config
- .consumer_client_config()
- .create_with_context(context)
- .context("creating cohort_stream_events consumer")?;
- stream_consumer
- .subscribe(&[config.cohort_stream_events_topic.as_str()])
- .context("subscribing to cohort_stream_events")?;
- // Shared with the seed consumer's idle probe.
- let stream_consumer = Arc::new(stream_consumer);
-
- let merges_follower_consumer: Arc<StreamConsumer> = Arc::new(
- config
- .follower_client_config(&config.kafka_merge_consumer_group)
- .create()
- .context("creating person_merge_events follower consumer")?,
- );
- let transfers_follower_consumer: Arc<StreamConsumer> = Arc::new(
- config
- .follower_client_config(&config.kafka_merge_apply_consumer_group)
- .create()
- .context("creating cohort_merge_state_transfer follower consumer")?,
- );
- // Built only when the gate is on, so a dormant deploy needs no `cohort_cascade_events` topic.
- let cascade_follower_consumer: Option<Arc<StreamConsumer>> = if config.cohort_cascade_enabled {
- Some(Arc::new(
- config
- .follower_client_config(&config.kafka_cascade_consumer_group)
- .create()
- .context("creating cohort_cascade_events follower consumer")?,
- ))
- } else {
- None
- };
- // Gated: a gate-off deploy is safe without the seed topic existing at all.
- let seed_follower_consumer: Option<Arc<StreamConsumer>> = if config.cohort_seed_consumer_enabled
- {
- Some(Arc::new(
- config
- .follower_client_config(&config.kafka_seed_consumer_group)
- .create()
- .context("creating cohort_stream_seed_events follower consumer")?,
- ))
- } else {
- None
- };
-
- let events_partitions =
- fetch_partition_count(&stream_consumer, &config.cohort_stream_events_topic)?;
- let merge_partitions =
- fetch_partition_count(&merges_follower_consumer, &config.person_merge_events_topic)?;
- let transfer_partitions = fetch_partition_count(
- &transfers_follower_consumer,
- &config.cohort_merge_state_transfer_topic,
- )?;
- // The merge math hashes `(team, person)` against `config.cohort_partition_count`, so the topics
- // must not only be co-partitioned with each other but partitioned at exactly that count — a
- // deploy/lane at N != the configured count silently misroutes every merge.
- anyhow::ensure!(
- events_partitions as u32 == config.cohort_partition_count,
- "{} is partitioned at {} but COHORT_PARTITION_COUNT is {}: the merge partition arithmetic \
- would misroute. Re-partition the topic to {} or set COHORT_PARTITION_COUNT to {}.",
- config.cohort_stream_events_topic,
- events_partitions,
- config.cohort_partition_count,
- config.cohort_partition_count,
- events_partitions,
- );
- anyhow::ensure!(
- merge_partitions == events_partitions && transfer_partitions == events_partitions,
- "merge topics must be co-partitioned with {} ({} partitions): {} has {}, {} has {}",
- config.cohort_stream_events_topic,
- events_partitions,
- config.person_merge_events_topic,
- merge_partitions,
- config.cohort_merge_state_transfer_topic,
- transfer_partitions,
- );
- // A cascade for (team, person) must land on the partition owning that person's `cf_stage2` — the
- // same partition number as the events topic — so refuse to start co-partitioned at a different
- // count. Skipped when the gate is off (no cascade topic required).
- if let Some(cascade_consumer) = &cascade_follower_consumer {
- let cascade_partitions =
- fetch_partition_count(cascade_consumer, &config.cohort_cascade_events_topic)?;
- anyhow::ensure!(
- cascade_partitions as u32 == config.cohort_partition_count,
- "cohort_cascade_events must be co-partitioned with {} at COHORT_PARTITION_COUNT={}: {} has {}",
- config.cohort_stream_events_topic,
- config.cohort_partition_count,
- config.cohort_cascade_events_topic,
- cascade_partitions,
- );
- }
- // A tile must land on the partition owning its person's state slice.
- if let Some(seed_consumer) = &seed_follower_consumer {
- let seed_partitions =
- fetch_partition_count(seed_consumer, &config.cohort_stream_seed_events_topic)?;
- anyhow::ensure!(
- seed_partitions as u32 == config.cohort_partition_count,
- "cohort_stream_seed_events must be co-partitioned with {} at COHORT_PARTITION_COUNT={}: {} has {}",
- config.cohort_stream_events_topic,
- config.cohort_partition_count,
- config.cohort_stream_seed_events_topic,
- seed_partitions,
- );
- // Nothing co-partitions with the marker topic, so only its existence matters. Failing here
- // beats the alternative: a marker produce retrying forever while it holds the seed offset.
- // Nested under the seed consumer on purpose: reconcile jobs are admitted only from seed
- // tiles, so without it no marker can be produced and there is nothing to prove.
- if config.cohort_seed_reconcile_enabled {
- fetch_partition_count(seed_consumer, &config.cohort_reconcile_markers_topic).with_context(
- || {
- format!(
- "{} must exist before a processor with COHORT_SEED_RECONCILE_ENABLED starts. \
- Provision the topic, or set COHORT_SEED_RECONCILE_ENABLED=false to start \
- without the reconcile path.",
- config.cohort_reconcile_markers_topic,
- )
- },
- )?;
- }
- }
-
- let mut follower_mirrors = vec![
+ let mut followers = vec![
Follower::new(
merges_follower_consumer.clone(),
config.person_merge_events_topic.clone(),
@@ -465,20 +462,20 @@ async fn async_main(config: Config) -> Result<()> {
),
];
if let Some(cascade_consumer) = &cascade_follower_consumer {
- follower_mirrors.push(Follower::new(
+ followers.push(Follower::new(
cascade_consumer.clone(),
config.cohort_cascade_events_topic.clone(),
));
}
if let Some(seed_consumer) = &seed_follower_consumer {
- follower_mirrors.push(Follower::new(
+ followers.push(Follower::new(
seed_consumer.clone(),
config.cohort_stream_seed_events_topic.clone(),
));
}
- let followers = Arc::new(FollowerSet::new(follower_mirrors));
-
- let (consumer_command_tx, consumer_command_rx) = mpsc::unbounded_channel();
+ FollowerSet::new(followers)
+ .assign(&ownership)
+ .context("assigning the follower consumers this pod's partitions")?;
let guard = manager.monitor_background();
@@ -497,14 +494,6 @@ async fn async_main(config: Config) -> Result<()> {
.await;
});
- let rebalance_worker = tokio::spawn(run_rebalance_worker(
- rebalance_rx,
- dispatcher.clone(),
- followers,
- consumer_command_tx,
- consumer_handle.shutdown_token(),
- ));
-
// Hold the first eviction pass past the boot window so its read burst doesn't stack on backlog
// catch-up; only the eviction sweep is delayed.
tokio::spawn(run_sweep_loop_delayed(
@@ -683,12 +672,12 @@ async fn async_main(config: Config) -> Result<()> {
config.recv_batch_size,
config.recv_batch_timeout(),
config.offset_commit_interval(),
- events_partitions,
- consumer_command_rx,
+ ownership,
// A checkpoint restore boot finishes before anything folds; `None` when the store reopened
- // or was created, where boot rewinds only what it polled.
+ // or was created.
restore,
- );
+ )
+ .context("assigning cohort_stream_events this pod's partitions")?;
tokio::spawn(events_consumer.process());
health_server
@@ -699,10 +688,6 @@ async fn async_main(config: Config) -> Result<()> {
guard.wait().await?;
- if let Err(err) = rebalance_worker.await {
- warn!(error = %err, "rebalance worker task did not exit cleanly");
- }
-
info!(service = SERVICE_NAME, "service stopped");
Ok(())
}
@@ -754,13 +739,12 @@ fn log_startup(config: &Config) {
output_topic = %config.cohort_membership_changed_topic,
reconcile_markers_topic = %config.cohort_reconcile_markers_topic,
consumer_group = %config.kafka_consumer_group,
- offset_reset = %config.kafka_consumer_offset_reset,
merge_topic = %config.person_merge_events_topic,
transfer_topic = %config.cohort_merge_state_transfer_topic,
merge_consumer_group = %config.kafka_merge_consumer_group,
merge_apply_consumer_group = %config.kafka_merge_apply_consumer_group,
- session_timeout_ms = config.kafka_session_timeout_ms,
- pod_identity = config.pod_identity().unwrap_or("<dynamic>"),
+ pod_identity = config.pod_identity().unwrap_or("<none>"),
+ pod_count = config.cohort_pod_count,
recv_batch_size = config.recv_batch_size,
partition_channel_buffer = config.partition_channel_buffer,
store_path = %config.store_path,
diff --git a/rust/cohort-stream-processor/src/observability/metrics.rs b/rust/cohort-stream-processor/src/observability/metrics.rs
index c43026adad28..66f137c18308 100644
--- a/rust/cohort-stream-processor/src/observability/metrics.rs
+++ b/rust/cohort-stream-processor/src/observability/metrics.rs
@@ -297,20 +297,16 @@ pub const PARTITIONS_PAUSED: &str = "partitions_paused";
/// paused partition stops fetching — so a climbing value flags a stuck worker.
pub const PENDING_HELD_EVENTS: &str = "pending_held_events";
-/// Non-empty rebalance callbacks, labelled by `event_type` (`assign`|`revoke`) (counter).
+/// Rebalance callbacks on the events consumer, labelled by `event_type`
+/// (`assign`|`revoke`|`error`) (counter). The consumer never subscribes, so any increase means a
+/// code change made it subscribe (see `partitions::rebalance`).
pub const REBALANCES_TOTAL: &str = "rebalances_total";
-/// Partitions assigned to this consumer across all rebalances (counter).
+/// Partitions this pod took ownership of at boot (counter).
pub const PARTITIONS_ASSIGNED_TOTAL: &str = "partitions_assigned_total";
-/// Partitions revoked from this consumer across all rebalances (counter).
-pub const PARTITIONS_REVOKED_TOTAL: &str = "partitions_revoked_total";
-/// Empty rebalance callbacks short-circuited, labelled by `event_type` (counter).
-pub const REBALANCE_EMPTY_SKIPPED_TOTAL: &str = "rebalance_empty_skipped_total";
-/// Per-partition revoke drain latency (histogram, seconds).
-pub const REVOKE_DRAIN_DURATION_SECONDS: &str = "revoke_drain_duration_seconds";
-/// Per-partition RocksDB state slices reclaimed on revoke (counter).
-pub const PARTITION_STATE_DELETED_TOTAL: &str = "partition_state_deleted_total";
-/// Revoke cleanups skipped because the partition was re-acquired, labelled by `phase` (counter).
-pub const REBALANCE_CLEANUP_SKIPPED_TOTAL: &str = "rebalance_cleanup_skipped_total";
+/// Partitions this pod owns on every input (gauge).
+pub const OWNED_PARTITIONS: &str = "owned_partitions";
+/// This pod's place in the partition split, labelled by `pods` and `ordinal`. Always 1 (gauge).
+pub const PARTITION_OWNERSHIP_INFO: &str = "partition_ownership_info";
/// Events fully processed (counter).
pub const STAGE1_EVENTS_PROCESSED: &str = "stage1_events_processed_total";
diff --git a/rust/cohort-stream-processor/src/partitions/follower.rs b/rust/cohort-stream-processor/src/partitions/follower.rs
index 7cb0a229df24..7f60d05338eb 100644
--- a/rust/cohort-stream-processor/src/partitions/follower.rs
+++ b/rust/cohort-stream-processor/src/partitions/follower.rs
@@ -1,35 +1,17 @@
-//! Assignment mirroring for the internal follower consumers.
-//!
-//! The follower consumers (`person_merge_events`, `cohort_merge_state_transfer`, and — when the
-//! cascade gate is on — `cohort_cascade_events`) never `subscribe()`: the `cohort_stream_events`
-//! group's rebalance is the single source of partition ownership, and the rebalance worker mirrors
-//! every Assign/Revoke onto them through [`PartitionMirror`] — `incremental_assign` at
-//! [`Offset::Stored`] and `incremental_unassign`, both unconditional. The topics are co-partitioned,
-//! so the events topic's partition numbers index every follower.
-//!
-//! Why unconditional, including the rapid revoke→assign path: between the sync revoke and the async
-//! unassign the followers keep fetching, and the dispatcher's owned-gate drops those messages while
-//! their fetch position advances in-session. If a re-acquire skipped re-assignment, a later
-//! monotonic-max mark would commit past the dropped window — a permanently lost merge, not lag.
-//! Re-assigning at `Offset::Stored` rewinds to the last commit and redelivers the window; the
-//! drain/apply source-coords markers absorb the replays.
+//! The follower consumers: `person_merge_events`, `cohort_merge_state_transfer`, and, when their
+//! gates are on, `cohort_cascade_events` and `cohort_stream_seed_events`. They never subscribe.
+//! Each is assigned the partitions this pod owns, like the events consumer, so the pod owns
+//! partition N on every input at once. The topics are co-partitioned, so the events topic's
+//! partition numbers index every follower.
use std::sync::Arc;
use rdkafka::consumer::{Consumer, StreamConsumer};
-use rdkafka::{Offset, TopicPartitionList};
-use tracing::warn;
+use rdkafka::error::KafkaResult;
-/// Mirror an events-group (un)assignment onto the follower consumers.
-/// Trait-based so the rebalance worker is unit-testable; production impl is [`FollowerSet`].
-pub trait PartitionMirror: Send + Sync {
- /// Mirror newly-assigned partitions onto every follower.
- fn assign(&self, partitions: &[i32]);
- /// Mirror revoked partitions off every follower.
- fn unassign(&self, partitions: &[i32]);
-}
+use crate::partitions::PartitionOwnership;
-/// One follower consumer and the topic its assignments are built against.
+/// One follower consumer and the topic its assignment is built against.
pub struct Follower {
consumer: Arc<StreamConsumer>,
topic: String,
@@ -41,104 +23,27 @@ impl Follower {
}
}
-/// Production [`PartitionMirror`]: drives `incremental_assign`/`incremental_unassign` on a set of
-/// follower consumers (`person_merge_events`, `cohort_merge_state_transfer`, and — when the gate is
-/// on — `cohort_cascade_events`). Errors are warned and skipped: the rebalance worker must keep
-/// draining. A missed assign degrades to visible group lag; a missed unassign is practically
-/// unreachable (either "partition wasn't assigned" or a fatally-dead consumer).
+/// The follower consumers of one pod. Their topics must all be co-partitioned with
+/// `cohort_stream_events`.
pub struct FollowerSet {
followers: Vec<Follower>,
}
impl FollowerSet {
- /// The followers' topics must all be co-partitioned with `cohort_stream_events`.
pub fn new(followers: impl IntoIterator<Item = Follower>) -> Self {
Self {
followers: followers.into_iter().collect(),
}
}
-}
-
-impl PartitionMirror for FollowerSet {
- fn assign(&self, partitions: &[i32]) {
- for follower in &self.followers {
- let tpl = stored_tpl(&follower.topic, partitions);
- if let Err(err) = follower.consumer.incremental_assign(&tpl) {
- warn!(
- topic = %follower.topic,
- ?partitions,
- error = %err,
- "follower incremental_assign failed; surfaces as lag on its consumer group",
- );
- }
- }
- }
- fn unassign(&self, partitions: &[i32]) {
+ /// Assigns every follower the partitions `ownership` owns, each at its group's committed
+ /// offset. A follower that fails would never consume its topic, so the first failure stops.
+ pub fn assign(&self, ownership: &PartitionOwnership) -> KafkaResult<()> {
for follower in &self.followers {
- let tpl = bare_tpl(&follower.topic, partitions);
- if let Err(err) = follower.consumer.incremental_unassign(&tpl) {
- warn!(
- topic = %follower.topic,
- ?partitions,
- error = %err,
- "follower incremental_unassign failed; owned-gate still drops its messages",
- );
- }
- }
- }
-}
-
-pub(crate) fn stored_tpl(topic: &str, partitions: &[i32]) -> TopicPartitionList {
- let mut tpl = TopicPartitionList::new();
- for &partition in partitions {
- if let Err(err) = tpl.add_partition_offset(topic, partition, Offset::Stored) {
- warn!(topic, partition, error = %err, "skipping partition in follower assign list");
- }
- }
- tpl
-}
-
-pub(crate) fn bare_tpl(topic: &str, partitions: &[i32]) -> TopicPartitionList {
- let mut tpl = TopicPartitionList::new();
- for &partition in partitions {
- tpl.add_partition(topic, partition);
- }
- tpl
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
-
- #[test]
- fn stored_tpl_maps_each_partition_to_offset_stored() {
- let tpl = stored_tpl("person_merge_events", &[3, 7, 0]);
-
- assert_eq!(tpl.count(), 3);
- let mut partitions = Vec::new();
- for elem in tpl.elements() {
- assert_eq!(elem.topic(), "person_merge_events");
- assert_eq!(elem.offset(), Offset::Stored);
- partitions.push(elem.partition());
+ follower
+ .consumer
+ .assign(&ownership.assignment(&follower.topic)?)?;
}
- partitions.sort_unstable();
- assert_eq!(partitions, vec![0, 3, 7]);
- }
-
- #[test]
- fn stored_tpl_for_no_partitions_is_empty() {
- assert_eq!(stored_tpl("cohort_merge_state_transfer", &[]).count(), 0);
- }
-
- #[test]
- fn bare_tpl_carries_partitions_without_a_real_offset() {
- let tpl = bare_tpl("cohort_merge_state_transfer", &[5]);
-
- assert_eq!(tpl.count(), 1);
- let elems = tpl.elements();
- assert_eq!(elems[0].topic(), "cohort_merge_state_transfer");
- assert_eq!(elems[0].partition(), 5);
- assert_eq!(elems[0].offset(), Offset::Invalid);
+ Ok(())
}
}
diff --git a/rust/cohort-stream-processor/src/partitions/mod.rs b/rust/cohort-stream-processor/src/partitions/mod.rs
index 24acad661f90..75a7be778e12 100644
--- a/rust/cohort-stream-processor/src/partitions/mod.rs
+++ b/rust/cohort-stream-processor/src/partitions/mod.rs
@@ -9,6 +9,7 @@ pub mod backpressure;
pub mod follower;
pub mod intake;
pub mod offset_tracker;
+pub mod ownership;
pub mod pacing;
pub mod pause;
pub mod positions;
@@ -20,9 +21,10 @@ pub mod watermarks;
pub use cohort_core::partitioner;
pub use backpressure::{Backpressure, PartitionHoldover};
-pub use follower::{Follower, FollowerSet, PartitionMirror};
+pub use follower::{Follower, FollowerSet};
pub use intake::{Admission, MeteredReceiver, PartitionIntake};
pub use offset_tracker::{DeferredOffset, MarkOutcome, OffsetTracker};
+pub use ownership::{OwnershipError, PartitionOwnership};
pub use pacing::{
AgeMs, CauseSet, Hysteresis, PauseCause, PauseLedger, SeedPacing, SeedPacingConfig, UsedPct,
};
@@ -33,10 +35,7 @@ pub use pause::{ConsumerPauser, PartitionPauser};
pub use positions::{
list_offsets, FollowerGroup, InputGroups, InputPositions, InputTopic, ResumeOffset,
};
-pub use rebalance::{
- run_rebalance_worker, CohortConsumerContext, ConsumerCommand, ConsumerCommandReceiver,
- ConsumerCommandSender, RebalanceEvent, RebalanceEventReceiver,
-};
+pub use rebalance::CohortConsumerContext;
pub use router::{
PartitionRouter, RouteError, SeedReceiver, SeedRefusal, SeedSendOutcome, SendOutcome,
WorkerInbox,
diff --git a/rust/cohort-stream-processor/src/partitions/ownership.rs b/rust/cohort-stream-processor/src/partitions/ownership.rs
new file mode 100644
index 000000000000..e74d3c79d721
--- /dev/null
+++ b/rust/cohort-stream-processor/src/partitions/ownership.rs
@@ -0,0 +1,215 @@
+//! Static partition ownership. Pod `ordinal` of `pods` owns every partition `p` with
+//! `p % pods == ordinal` on all five inputs, for as long as it runs. The consumers assign
+//! themselves these partitions instead of joining a group rebalance, so no partition moves between
+//! running pods. The store records the ownership it was written under.
+
+use std::fmt;
+use std::num::NonZeroU16;
+
+use rdkafka::error::KafkaResult;
+use rdkafka::{Offset, TopicPartitionList};
+
+use crate::store::durability::PodOrdinal;
+
+/// The partitions one pod owns. Only [`Self::new`] splits the partitions between more than one
+/// pod, so `pods` divides the partition count and the ordinal is below `pods`.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PartitionOwnership {
+ pods: NonZeroU16,
+ ordinal: PodOrdinal,
+ partition_count: u16,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
+pub enum OwnershipError {
+ #[error("{pods} pods cannot split {partition_count} partitions evenly")]
+ UnevenSplit {
+ pods: NonZeroU16,
+ partition_count: u16,
+ },
+ #[error("pod ordinal {ordinal} is not below the pod count {pods}")]
+ OrdinalOutOfRange {
+ ordinal: PodOrdinal,
+ pods: NonZeroU16,
+ },
+}
+
+const ENCODED_LEN: usize = 6;
+
+impl PartitionOwnership {
+ /// One pod owns every partition.
+ pub const fn single(partition_count: u16) -> Self {
+ Self {
+ pods: NonZeroU16::MIN,
+ ordinal: PodOrdinal::STANDALONE,
+ partition_count,
+ }
+ }
+
+ /// Every pod must own the same number of partitions, so `pods` must divide `partition_count`.
+ pub fn new(
+ pods: NonZeroU16,
+ ordinal: PodOrdinal,
+ partition_count: u16,
+ ) -> Result<Self, OwnershipError> {
+ if !partition_count.is_multiple_of(pods.get()) {
+ return Err(OwnershipError::UnevenSplit {
+ pods,
+ partition_count,
+ });
+ }
+ if ordinal.get() >= pods.get() {
+ return Err(OwnershipError::OrdinalOutOfRange { ordinal, pods });
+ }
+ Ok(Self {
+ pods,
+ ordinal,
+ partition_count,
+ })
+ }
+
+ pub fn pods(&self) -> NonZeroU16 {
+ self.pods
+ }
+
+ pub fn ordinal(&self) -> PodOrdinal {
+ self.ordinal
+ }
+
+ pub fn partition_count(&self) -> u16 {
+ self.partition_count
+ }
+
+ pub fn owns(&self, partition: u16) -> bool {
+ partition < self.partition_count && partition % self.pods.get() == self.ordinal.get()
+ }
+
+ /// The owned partitions in ascending order.
+ pub fn partitions(&self) -> impl Iterator<Item = u16> {
+ (self.ordinal.get()..self.partition_count).step_by(usize::from(self.pods.get()))
+ }
+
+ /// `topic`'s owned partitions, each at its group's committed offset, for `Consumer::assign`.
+ pub fn assignment(&self, topic: &str) -> KafkaResult<TopicPartitionList> {
+ let mut assignment = TopicPartitionList::new();
+ for partition in self.partitions() {
+ assignment.add_partition_offset(topic, i32::from(partition), Offset::Stored)?;
+ }
+ Ok(assignment)
+ }
+
+ /// The `cf_meta` record: the pod count, the ordinal and the partition count, each a
+ /// big-endian `u16`.
+ pub(crate) fn encode(&self) -> [u8; ENCODED_LEN] {
+ let mut bytes = [0; ENCODED_LEN];
+ bytes[0..2].copy_from_slice(&self.pods.get().to_be_bytes());
+ bytes[2..4].copy_from_slice(&self.ordinal.get().to_be_bytes());
+ bytes[4..6].copy_from_slice(&self.partition_count.to_be_bytes());
+ bytes
+ }
+
+ /// `None` for bytes that [`Self::encode`] did not write.
+ pub(crate) fn decode(bytes: &[u8]) -> Option<Self> {
+ let [p0, p1, o0, o1, c0, c1] = <[u8; ENCODED_LEN]>::try_from(bytes).ok()?;
+ let pods = NonZeroU16::new(u16::from_be_bytes([p0, p1]))?;
+ let ordinal = PodOrdinal::from(u16::from_be_bytes([o0, o1]));
+ Self::new(pods, ordinal, u16::from_be_bytes([c0, c1])).ok()
+ }
+}
+
+impl fmt::Display for PartitionOwnership {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ write!(
+ f,
+ "pod {} of {} over {} partitions",
+ self.ordinal, self.pods, self.partition_count
+ )
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn every_split_of_the_partitions_gives_each_partition_exactly_one_owner() {
+ const PARTITIONS: u16 = 64;
+ for pods in [1, 2, 4, 8, 16, 32, 64] {
+ let pods = NonZeroU16::new(pods).unwrap();
+ let split: Vec<PartitionOwnership> = (0..pods.get())
+ .map(|ordinal| {
+ PartitionOwnership::new(pods, PodOrdinal::from(ordinal), PARTITIONS).unwrap()
+ })
+ .collect();
+ let mut owners = [0usize; PARTITIONS as usize];
+ for ownership in &split {
+ let partitions: Vec<u16> = ownership.partitions().collect();
+ assert_eq!(
+ partitions.len(),
+ usize::from(PARTITIONS / pods.get()),
+ "{ownership}"
+ );
+ for partition in 0..PARTITIONS {
+ assert_eq!(
+ ownership.owns(partition),
+ partitions.contains(&partition),
+ "{ownership}: partition {partition}",
+ );
+ }
+ for partition in partitions {
+ owners[usize::from(partition)] += 1;
+ }
+ }
+ assert!(
+ owners.iter().all(|&count| count == 1),
+ "{pods} pods: {owners:?}"
+ );
+ }
+
+ let pod_one_of_four =
+ PartitionOwnership::new(NonZeroU16::new(4).unwrap(), PodOrdinal::from(1), 16).unwrap();
+ assert_eq!(
+ pod_one_of_four.partitions().collect::<Vec<_>>(),
+ [1, 5, 9, 13]
+ );
+ assert!(!pod_one_of_four.owns(17), "past the partition count");
+ }
+
+ #[test]
+ fn the_ownership_record_keeps_its_layout_and_rejects_bytes_encode_never_writes() {
+ let pod_one_of_two =
+ PartitionOwnership::new(NonZeroU16::new(2).unwrap(), PodOrdinal::from(1), 64).unwrap();
+ assert_eq!(pod_one_of_two.encode(), [0, 2, 0, 1, 0, 64]);
+ assert_eq!(
+ PartitionOwnership::decode(&[0, 2, 0, 1, 0, 64]),
+ Some(pod_one_of_two)
+ );
+ for bytes in [
+ &[0, 2, 0, 1, 0][..],
+ &[0, 0, 0, 0, 0, 64],
+ &[0, 2, 0, 2, 0, 64],
+ &[0, 3, 0, 0, 0, 64],
+ ] {
+ assert_eq!(PartitionOwnership::decode(bytes), None, "{bytes:?}");
+ }
+ }
+
+ #[test]
+ fn the_assignment_resumes_each_owned_partition_at_its_committed_offset() {
+ let ownership =
+ PartitionOwnership::new(NonZeroU16::new(2).unwrap(), PodOrdinal::from(1), 4).unwrap();
+ let assignment = ownership.assignment("cohort_stream_events").unwrap();
+ let assigned: Vec<(String, i32, Offset)> = assignment
+ .elements()
+ .iter()
+ .map(|elem| (elem.topic().to_owned(), elem.partition(), elem.offset()))
+ .collect();
+ assert_eq!(
+ assigned,
+ [
+ ("cohort_stream_events".to_owned(), 1, Offset::Stored),
+ ("cohort_stream_events".to_owned(), 3, Offset::Stored),
+ ],
+ );
+ }
+}
diff --git a/rust/cohort-stream-processor/src/partitions/rebalance.rs b/rust/cohort-stream-processor/src/partitions/rebalance.rs
index f7f3c771bbf8..830f5feba256 100644
--- a/rust/cohort-stream-processor/src/partitions/rebalance.rs
+++ b/rust/cohort-stream-processor/src/partitions/rebalance.rs
@@ -1,557 +1,30 @@
-//! The rebalance bridge: a [`ConsumerContext`] that turns Kafka's rebalance callbacks into the
-//! partition lifecycle a stateful, partition-affined consumer needs.
-//!
-//! Correctness rests on one invariant: all of a `(team_id, person_id)`'s state lives on exactly one
-//! partition → one pod → one store slice. So ownership must *move* — on revoke, reclaim the slice; on
-//! assign, record ownership.
-//!
-//! librdkafka fires `pre_rebalance`/`post_rebalance` synchronously on the poll thread, where blocking
-//! or panicking is forbidden (a panic across the FFI boundary is UB). The callbacks therefore do only
-//! fast sync work and ship the slow part (draining a worker is async I/O) to [`run_rebalance_worker`]
-//! over an mpsc channel.
-//!
-//! Cooperative-sticky can revoke a partition and immediately re-assign it to the same pod. The async
-//! cleanup re-checks ownership against current state (not the queued snapshot) and skips when the
-//! partition was re-acquired, leaving the live worker and its slice untouched — see
-//! [`EventDispatcher::revoke_partition_drain`].
-
-use std::sync::Arc;
+//! The events consumer's [`ConsumerContext`]. Every consumer assigns itself the partitions this pod
+//! owns (see [`super::ownership`]) and never joins its group, so a rebalance callback means this
+//! consumer subscribed after all, which static ownership rules out. It is a tripwire for a code
+//! change, not for another client.
use metrics::counter;
use rdkafka::consumer::{BaseConsumer, ConsumerContext, Rebalance};
-use rdkafka::{ClientContext, TopicPartitionList};
-use tokio::sync::mpsc;
-use tokio_util::sync::CancellationToken;
-use tracing::{debug, error, info};
-
-use crate::consumers::events::EventDispatcher;
-use crate::observability::metrics::{REBALANCES_TOTAL, REBALANCE_EMPTY_SKIPPED_TOTAL};
-use crate::partitions::follower::PartitionMirror;
+use rdkafka::ClientContext;
+use tracing::error;
-/// Work shipped from the synchronous rebalance callbacks to the async [`run_rebalance_worker`].
-#[derive(Debug, Clone)]
-pub enum RebalanceEvent {
- /// Reclaim these revoked partitions: drain each worker, then delete its state slice.
- Revoke(Vec<i32>),
- /// Partitions newly assigned.
- Assign(Vec<i32>),
-}
+use crate::observability::metrics::REBALANCES_TOTAL;
-pub type RebalanceEventReceiver = mpsc::UnboundedReceiver<RebalanceEvent>;
-
-/// Commands from the rebalance worker to the consume loop. Currently unused.
-#[derive(Debug)]
-pub enum ConsumerCommand {
- /// Resume consumption for these partitions.
- Resume(TopicPartitionList),
- /// Seek these partitions to specific offsets.
- SeekPartitions(TopicPartitionList),
-}
-
-pub type ConsumerCommandSender = mpsc::UnboundedSender<ConsumerCommand>;
-pub type ConsumerCommandReceiver = mpsc::UnboundedReceiver<ConsumerCommand>;
-
-/// The rdkafka [`ConsumerContext`] for `cohort_stream_events`. Rebalance callbacks drive the
-/// [`EventDispatcher`]'s partition lifecycle and hand slow cleanup to the async worker.
-pub struct CohortConsumerContext {
- dispatcher: Arc<EventDispatcher>,
- rebalance_tx: mpsc::UnboundedSender<RebalanceEvent>,
-}
-
-impl CohortConsumerContext {
- /// Build the context and the receiver half of its rebalance channel.
- pub fn new(dispatcher: Arc<EventDispatcher>) -> (Self, RebalanceEventReceiver) {
- let (rebalance_tx, rebalance_rx) = mpsc::unbounded_channel();
- (
- Self {
- dispatcher,
- rebalance_tx,
- },
- rebalance_rx,
- )
- }
-
- /// Sync, on the poll thread: mark each revoked partition un-owned and queue async cleanup.
- fn on_revoke(&self, partitions: &TopicPartitionList) {
- if partitions.count() == 0 {
- counter!(REBALANCE_EMPTY_SKIPPED_TOTAL, "event_type" => "revoke").increment(1);
- debug!("skipping empty revoke");
- return;
- }
- let partitions = partition_numbers(partitions);
- counter!(REBALANCES_TOTAL, "event_type" => "revoke").increment(1);
- info!(count = partitions.len(), "revoking partitions");
- for &partition in &partitions {
- self.dispatcher.revoke_partition_sync(partition);
- }
- if let Err(err) = self.rebalance_tx.send(RebalanceEvent::Revoke(partitions)) {
- error!(error = %err, "failed to queue revoke cleanup");
- }
- }
-
- /// Sync, on the poll thread: record ownership for each assigned partition.
- fn on_assign(&self, partitions: &TopicPartitionList) {
- if partitions.count() == 0 {
- counter!(REBALANCE_EMPTY_SKIPPED_TOTAL, "event_type" => "assign").increment(1);
- debug!("skipping empty assign");
- return;
- }
- let partitions = partition_numbers(partitions);
- counter!(REBALANCES_TOTAL, "event_type" => "assign").increment(1);
- info!(count = partitions.len(), "assigned partitions");
- for &partition in &partitions {
- self.dispatcher.assign_partition(partition);
- }
- if let Err(err) = self.rebalance_tx.send(RebalanceEvent::Assign(partitions)) {
- error!(error = %err, "failed to queue assign setup");
- }
- }
-}
+pub struct CohortConsumerContext;
impl ClientContext for CohortConsumerContext {}
impl ConsumerContext for CohortConsumerContext {
fn pre_rebalance(&self, _base: &BaseConsumer<Self>, rebalance: &Rebalance<'_>) {
- match rebalance {
- Rebalance::Revoke(partitions) => self.on_revoke(partitions),
- Rebalance::Assign(_) => {}
- Rebalance::Error(err) => error!(error = %err, "pre-rebalance error"),
- }
- }
-
- fn post_rebalance(&self, _base: &BaseConsumer<Self>, rebalance: &Rebalance<'_>) {
- match rebalance {
- Rebalance::Assign(partitions) => self.on_assign(partitions),
- Rebalance::Revoke(_) => {}
- Rebalance::Error(err) => error!(error = %err, "post-rebalance error"),
- }
- }
-}
-
-/// Async rebalance worker: mirrors (un)assignments onto follower consumers and drains revoked
-/// partitions off the poll thread. Runs until shutdown or the sender is dropped.
-///
-/// Mirror calls are unconditional — even on rapid revoke→assign where the drain skips — because
-/// the followers' in-session fetch position advances past owned-gate-dropped messages and only a
-/// re-assign at `Offset::Stored` rewinds over that gap.
-pub async fn run_rebalance_worker(
- mut events: RebalanceEventReceiver,
- dispatcher: Arc<EventDispatcher>,
- mirror: Arc<dyn PartitionMirror>,
- // Currently unused; held so the command channel's receiver stays open.
- _consumer_command_tx: ConsumerCommandSender,
- shutdown: CancellationToken,
-) {
- info!("rebalance worker started");
- loop {
- tokio::select! {
- biased;
- _ = shutdown.cancelled() => {
- info!("rebalance worker stopping on shutdown signal");
- break;
- }
- event = events.recv() => {
- let Some(event) = event else {
- info!("rebalance worker channel closed; stopping");
- break;
- };
- match event {
- RebalanceEvent::Revoke(partitions) => {
- // Unassign followers before drain: during drain, their fetch position
- // would advance past owned-gate-dropped messages.
- mirror.unassign(&partitions);
- for partition in partitions {
- dispatcher.revoke_partition_drain(partition).await;
- }
- }
- RebalanceEvent::Assign(partitions) => {
- // Stale-slice reclamation for post-boot move-ins now happens at worker-spawn
- // time (`EventDispatcher::ensure_worker`), under the partition's shard guard,
- // so the wipe deterministically precedes the eviction-queue rebuild. The assign
- // arm only mirrors followers.
- mirror.assign(&partitions);
- }
- }
- }
- }
- }
-}
-
-fn partition_numbers(partitions: &TopicPartitionList) -> Vec<i32> {
- partitions
- .elements()
- .iter()
- .map(|elem| elem.partition())
- .collect()
-}
-
-#[cfg(test)]
-// Tests seed and probe partition slices against `CohortStore` directly while the dispatcher holds
-// the `StoreHandle` facade.
-#[allow(clippy::disallowed_methods)]
-mod tests {
- use super::*;
- use std::sync::Mutex;
- use tempfile::TempDir;
-
- use crate::filters::{CatalogHandle, FilterCatalog};
- use crate::partitions::{MarkOutcome, OffsetTracker, PartitionRouter};
- use crate::producer::{CaptureSink, MembershipSink};
- use crate::store::{
- Behavioral, BehavioralKey, CohortStore, LeafStateKey, OffloadConfig, OffloadMode,
- StoreConfig, StoreHandle,
- };
- use crate::workers::MergeWorkerDeps;
- use uuid::Uuid;
-
- const TOPIC: &str = "cohort_stream_events";
-
- fn tpl(partitions: &[i32]) -> TopicPartitionList {
- let mut list = TopicPartitionList::new();
- for &partition in partitions {
- list.add_partition(TOPIC, partition);
- }
- list
- }
-
- /// Returns the raw store alongside the dispatcher (which wraps a clone of it) so tests can
- /// seed/probe slices directly; the dispatcher itself only exposes the async facade.
- fn test_dispatcher() -> (
- TempDir,
- Arc<EventDispatcher>,
- Arc<OffsetTracker>,
- CohortStore,
- ) {
- let dir = TempDir::new().unwrap();
- let store = CohortStore::open(&StoreConfig {
- path: dir.path().join("db"),
- ..StoreConfig::default()
- })
- .unwrap();
- let handle = StoreHandle::new(
- store.clone(),
- OffloadConfig {
- mode: OffloadMode::All,
- event_read_permits: 16,
- maintenance_permits: 6,
- },
- );
- let catalog = Arc::new(CatalogHandle::from_catalog(FilterCatalog::from_teams([])));
- let sink: Arc<dyn MembershipSink> = Arc::new(CaptureSink::new());
- let tracker = Arc::new(OffsetTracker::new());
- let dispatcher = EventDispatcher::new(
- PartitionRouter::new(64),
- tracker.clone(),
- handle,
- catalog,
- sink,
- MergeWorkerDeps::capture(),
- );
- (dir, Arc::new(dispatcher), tracker, store)
- }
-
- #[derive(Debug, Clone, PartialEq, Eq)]
- enum MirrorCall {
- Assign(Vec<i32>),
- Unassign(Vec<i32>),
- }
-
- /// Records mirror calls. At unassign time, snapshots whether the tracker still held the
- /// partition's entry (proving unassign ran before drain).
- struct RecordingMirror {
- tracker: Arc<OffsetTracker>,
- calls: Mutex<Vec<MirrorCall>>,
- unassign_saw_tracker_entry: Mutex<Vec<bool>>,
- }
-
- impl RecordingMirror {
- fn new(tracker: Arc<OffsetTracker>) -> Arc<Self> {
- Arc::new(Self {
- tracker,
- calls: Mutex::new(Vec::new()),
- unassign_saw_tracker_entry: Mutex::new(Vec::new()),
- })
- }
-
- fn calls(&self) -> Vec<MirrorCall> {
- self.calls.lock().unwrap().clone()
- }
- }
-
- impl PartitionMirror for RecordingMirror {
- fn assign(&self, partitions: &[i32]) {
- self.calls
- .lock()
- .unwrap()
- .push(MirrorCall::Assign(partitions.to_vec()));
- }
-
- fn unassign(&self, partitions: &[i32]) {
- let offsets = self.tracker.committable_offsets();
- let saw_all = partitions
- .iter()
- .all(|partition| offsets.contains_key(partition));
- self.unassign_saw_tracker_entry
- .lock()
- .unwrap()
- .push(saw_all);
- self.calls
- .lock()
- .unwrap()
- .push(MirrorCall::Unassign(partitions.to_vec()));
- }
- }
-
- async fn run_worker_to_completion(
- events: RebalanceEventReceiver,
- dispatcher: Arc<EventDispatcher>,
- mirror: Arc<RecordingMirror>,
- ) {
- let (cmd_tx, _cmd_rx) = mpsc::unbounded_channel();
- run_rebalance_worker(events, dispatcher, mirror, cmd_tx, CancellationToken::new()).await;
- }
-
- #[test]
- fn partition_numbers_extracts_each_partition() {
- let mut numbers = partition_numbers(&tpl(&[3, 7, 1]));
- numbers.sort_unstable();
- assert_eq!(numbers, vec![1, 3, 7]);
- }
-
- #[test]
- fn empty_tpl_has_zero_count() {
- assert_eq!(TopicPartitionList::new().count(), 0);
- assert_eq!(tpl(&[0]).count(), 1);
- }
-
- #[tokio::test]
- async fn on_assign_records_ownership_and_queues_an_assign_event() {
- let (_dir, dispatcher, _tracker, _store) = test_dispatcher();
- let (ctx, mut rx) = CohortConsumerContext::new(dispatcher.clone());
-
- ctx.on_assign(&tpl(&[0, 1, 4]));
-
- assert!(dispatcher.owns(0) && dispatcher.owns(1) && dispatcher.owns(4));
- assert!(!dispatcher.owns(2));
- match rx.try_recv().expect("an assign event was queued") {
- RebalanceEvent::Assign(mut partitions) => {
- partitions.sort_unstable();
- assert_eq!(partitions, vec![0, 1, 4]);
- }
- other => panic!("expected Assign, got {other:?}"),
- }
- }
-
- #[tokio::test]
- async fn on_revoke_clears_ownership_and_queues_a_revoke_event() {
- let (_dir, dispatcher, _tracker, _store) = test_dispatcher();
- let (ctx, mut rx) = CohortConsumerContext::new(dispatcher.clone());
-
- ctx.on_assign(&tpl(&[0, 1]));
- assert!(
- matches!(rx.try_recv(), Ok(RebalanceEvent::Assign(_))),
- "assign is queued first",
- );
- ctx.on_revoke(&tpl(&[1]));
-
- assert!(dispatcher.owns(0), "unrevoked partition stays owned");
- assert!(!dispatcher.owns(1), "revoked partition is un-owned");
- match rx.try_recv().expect("a revoke event was queued") {
- RebalanceEvent::Revoke(partitions) => assert_eq!(partitions, vec![1]),
- other => panic!("expected Revoke, got {other:?}"),
- }
- }
-
- #[tokio::test]
- async fn empty_callbacks_are_noops_and_queue_nothing() {
- let (_dir, dispatcher, _tracker, _store) = test_dispatcher();
- let (ctx, mut rx) = CohortConsumerContext::new(dispatcher.clone());
-
- ctx.on_assign(&tpl(&[]));
- ctx.on_revoke(&tpl(&[]));
-
- assert!(!dispatcher.owns(0));
- assert!(
- rx.try_recv().is_err(),
- "an empty callback must not queue work",
- );
- }
-
- #[tokio::test]
- async fn assign_event_mirrors_the_followers() {
- let (_dir, dispatcher, tracker, _store) = test_dispatcher();
- let (ctx, rx) = CohortConsumerContext::new(dispatcher.clone());
- let mirror = RecordingMirror::new(tracker);
-
- ctx.on_assign(&tpl(&[0, 4]));
- drop(ctx);
- run_worker_to_completion(rx, dispatcher, mirror.clone()).await;
-
- assert_eq!(mirror.calls(), vec![MirrorCall::Assign(vec![0, 4])]);
- }
-
- #[tokio::test]
- async fn revoke_unassigns_the_followers_before_the_drain() {
- let (_dir, dispatcher, tracker, _store) = test_dispatcher();
- let (ctx, rx) = CohortConsumerContext::new(dispatcher.clone());
- let mirror = RecordingMirror::new(tracker.clone());
-
- // Seed a committable entry the drain will forget — the order probe.
- tracker.mark_dispatched(3, 10);
- assert_eq!(tracker.mark_processed(3, 10), MarkOutcome::WithinDispatch);
-
- ctx.on_assign(&tpl(&[3]));
- ctx.on_revoke(&tpl(&[3]));
- drop(ctx);
- run_worker_to_completion(rx, dispatcher.clone(), mirror.clone()).await;
-
- assert_eq!(
- mirror.calls(),
- vec![MirrorCall::Assign(vec![3]), MirrorCall::Unassign(vec![3])],
- );
- assert_eq!(
- *mirror.unassign_saw_tracker_entry.lock().unwrap(),
- vec![true],
- "the tracker entry was still live at unassign time, so the unassign ran before the drain",
- );
- assert!(
- !tracker.committable_offsets().contains_key(&3),
- "the drain then forgot the revoked partition's entry",
- );
- assert!(!dispatcher.owns(3));
- }
-
- fn slice_key(partition: u16) -> BehavioralKey {
- BehavioralKey::new(partition, 7, Uuid::from_u128(1), LeafStateKey([0xAB; 16]))
- }
-
- fn seed_slice(store: &CohortStore, partition: u16) {
- store
- .write_batch(|b| b.put::<Behavioral>(&slice_key(partition), b"state"))
- .unwrap();
- }
-
- /// Probe whether `partition`'s slice is present without seeding it, so the caller controls seed
- /// ordering relative to a boot reconcile.
- fn present_probe(store: &CohortStore, partition: u16) -> impl Fn() -> bool + '_ {
- move || {
- store
- .get_behavioral(&slice_key(partition))
- .unwrap()
- .is_some()
- }
- }
-
- fn seed_and_present(store: &CohortStore, partition: u16) -> impl Fn() -> bool + '_ {
- seed_slice(store, partition);
- present_probe(store, partition)
- }
-
- #[tokio::test]
- async fn assign_arm_does_not_reclaim_slices_reclamation_is_deferred_to_worker_spawn() {
- // The rebalance Assign arm only mirrors followers; it never wipes a slice. A post-boot move-in's
- // stale slice is reclaimed at `ensure_worker` spawn time instead (under the shard guard, before
- // the eviction-queue rebuild), so the slice survives the assign path until the first event —
- // regardless of the durable-restore gate.
- for durable_on in [false, true] {
- let (_dir, dispatcher, tracker, store) = test_dispatcher();
- if durable_on {
- dispatcher.enable_durable_restore();
- }
- // Snapshot excludes 3; seeding *after* reconcile keeps the boot wipe from deleting it, so
- // only the (now inert) assign path could have wiped it.
- dispatcher
- .reconcile_boot_assignment(&[0].into_iter().collect(), 64)
- .await;
- let present = seed_and_present(&store, 3);
- let (ctx, rx) = CohortConsumerContext::new(dispatcher.clone());
- let mirror = RecordingMirror::new(tracker);
- ctx.on_assign(&tpl(&[3]));
- drop(ctx);
- run_worker_to_completion(rx, dispatcher.clone(), mirror).await;
- assert!(
- present(),
- "the assign arm leaves the slice in place (durable_on={durable_on}); reclamation is deferred to spawn",
- );
- }
- }
-
- #[tokio::test]
- async fn durable_boot_assign_keeps_restored_slices() {
- // The pod's own initial assignment routes through the assign path; the restored slices for the
- // assigned partitions must survive it (the assign arm never wipes; spawn-time reclaim keeps
- // boot-snapshot partitions).
- let (_dir, dispatcher, tracker, store) = test_dispatcher();
- dispatcher.enable_durable_restore();
- seed_slice(&store, 0);
- seed_slice(&store, 1);
- let present_0 = present_probe(&store, 0);
- let present_1 = present_probe(&store, 1);
- dispatcher
- .reconcile_boot_assignment(&[0, 1].into_iter().collect(), 64)
- .await;
-
- let (ctx, rx) = CohortConsumerContext::new(dispatcher.clone());
- let mirror = RecordingMirror::new(tracker);
- ctx.on_assign(&tpl(&[0, 1]));
- drop(ctx);
- run_worker_to_completion(rx, dispatcher.clone(), mirror).await;
-
- assert!(present_0() && present_1(), "boot-assigned slices survive");
- }
-
- #[tokio::test]
- async fn durable_assign_before_boot_reconcile_keeps_slices() {
- // The rebalance worker can process the initial Assign before the consume loop records the
- // snapshot; the assign path never wipes, so the slices survive until they are either reopened
- // live or reclaimed at spawn time.
- let (_dir, dispatcher, tracker, store) = test_dispatcher();
- dispatcher.enable_durable_restore();
- seed_slice(&store, 0);
- seed_slice(&store, 1);
- let present_0 = present_probe(&store, 0);
- let present_1 = present_probe(&store, 1);
-
- let (ctx, rx) = CohortConsumerContext::new(dispatcher.clone());
- let mirror = RecordingMirror::new(tracker);
- ctx.on_assign(&tpl(&[0, 1]));
- drop(ctx);
- run_worker_to_completion(rx, dispatcher.clone(), mirror).await;
-
- assert!(
- present_0() && present_1(),
- "with no snapshot yet, the assign path skips and the slices survive",
- );
- }
-
- #[tokio::test]
- async fn rapid_revoke_assign_mirrors_both_calls_unconditionally() {
- let (_dir, dispatcher, tracker, _store) = test_dispatcher();
- let (ctx, rx) = CohortConsumerContext::new(dispatcher.clone());
- let mirror = RecordingMirror::new(tracker.clone());
-
- tracker.mark_dispatched(3, 10);
- assert_eq!(tracker.mark_processed(3, 10), MarkOutcome::WithinDispatch);
-
- ctx.on_assign(&tpl(&[3]));
- ctx.on_revoke(&tpl(&[3]));
- ctx.on_assign(&tpl(&[3]));
- drop(ctx);
- run_worker_to_completion(rx, dispatcher.clone(), mirror.clone()).await;
-
- assert_eq!(
- mirror.calls(),
- vec![
- MirrorCall::Assign(vec![3]),
- MirrorCall::Unassign(vec![3]),
- MirrorCall::Assign(vec![3]),
- ],
- );
- assert!(dispatcher.owns(3), "the rapid re-assign restored ownership");
- assert_eq!(
- tracker.committable_offsets().get(&3),
- Some(&10),
- "drain skipped for re-acquired partition, so tracker entry survives",
+ let event_type = match rebalance {
+ Rebalance::Assign(_) => "assign",
+ Rebalance::Revoke(_) => "revoke",
+ Rebalance::Error(_) => "error",
+ };
+ counter!(REBALANCES_TOTAL, "event_type" => event_type).increment(1);
+ error!(
+ ?rebalance,
+ "the events consumer rebalanced, which static partition ownership rules out",
);
}
}
diff --git a/rust/cohort-stream-processor/src/store/durability/lineage.rs b/rust/cohort-stream-processor/src/store/durability/lineage.rs
index 86b34f61c72e..5dd269e8f042 100644
--- a/rust/cohort-stream-processor/src/store/durability/lineage.rs
+++ b/rust/cohort-stream-processor/src/store/durability/lineage.rs
@@ -33,6 +33,16 @@ impl PodOrdinal {
.map(Self)
.ok_or_else(|| NotAnOrdinalPod(name.to_owned()))
}
+
+ pub fn get(self) -> u16 {
+ self.0
+ }
+}
+
+impl From<u16> for PodOrdinal {
+ fn from(ordinal: u16) -> Self {
+ Self(ordinal)
+ }
}
impl fmt::Display for PodOrdinal {
diff --git a/rust/cohort-stream-processor/src/store/durability/recovery.rs b/rust/cohort-stream-processor/src/store/durability/recovery.rs
index 2742c4f00a96..0087abc9b089 100644
--- a/rust/cohort-stream-processor/src/store/durability/recovery.rs
+++ b/rust/cohort-stream-processor/src/store/durability/recovery.rs
@@ -42,7 +42,7 @@ use crate::observability::metrics::{
CHECKPOINT_RESTORE_BLOCKED, CHECKPOINT_RESTORE_CANDIDATES_TOTAL,
CHECKPOINT_RESTORE_DURATION_SECONDS, CHECKPOINT_RESTORE_TOTAL,
};
-use crate::partitions::{InputGroups, InputTopic};
+use crate::partitions::{InputGroups, InputTopic, PartitionOwnership};
use crate::store::{CohortStore, StoreError, STORE_SCHEMA_VERSION};
const RETRY_START: Duration = Duration::from_secs(30);
@@ -197,16 +197,18 @@ impl Search {
/// reset. A restore comes back with it, for the events consumer to finish before anything folds.
///
/// A restore that cannot finish blocks here, retrying with backoff, until the operator fixes the
-/// cause or turns checkpoints off. `lineage` is `None` when checkpoints are disabled.
+/// cause or turns checkpoints off. `lineage` is `None` when checkpoints are disabled. The store
+/// refuses to open when it was written under another `ownership`.
pub async fn open_store(
config: &Config,
+ ownership: PartitionOwnership,
lineage: Option<CheckpointLineage>,
groups: Arc<InputGroups>,
) -> Result<(CohortStore, Option<PendingRestore>), StoreError> {
- let boot = restore_until_ready(config.clone(), lineage, groups).await;
+ let boot = restore_until_ready(config.clone(), ownership, lineage, groups).await;
// A published restore leaves a store at the path, so `effective_wipe_on_start` keeps it.
- let store_config = config.store_config();
+ let store_config = config.store_config(ownership);
info!(
durable_restore_enabled = config.durable_restore_enabled,
wipe_store_on_start = config.wipe_store_on_start,
@@ -227,13 +229,14 @@ pub async fn open_store(
}
/// Retries [`prepare_store`] with backoff until the boot has a store, and positions the follower
-/// groups of a restore before returning it.
+/// groups of a restore on the partitions `ownership` owns before returning it.
///
/// The restore runs on a blocking thread: staging, validation and the Kafka position calls block,
/// and the health server must keep answering meanwhile. A SIGTERM here kills the process; staging
/// survives a kill.
async fn restore_until_ready(
config: Config,
+ ownership: PartitionOwnership,
lineage: Option<CheckpointLineage>,
groups: Arc<InputGroups>,
) -> BootStore {
@@ -258,7 +261,7 @@ async fn restore_until_ready(
.await
{
Ok((BootStore::Restored(pending), source)) => pending
- .position_followers(&groups)
+ .position_followers(&groups, &ownership)
.map(|()| (BootStore::Restored(pending), source))
.map_err(RestoreBlocked::Positions),
other => other,
diff --git a/rust/cohort-stream-processor/src/store/durability/restore_plan.rs b/rust/cohort-stream-processor/src/store/durability/restore_plan.rs
index 7639ca0ff78a..67e29ebade1f 100644
--- a/rust/cohort-stream-processor/src/store/durability/restore_plan.rs
+++ b/rust/cohort-stream-processor/src/store/durability/restore_plan.rs
@@ -7,7 +7,7 @@ use rdkafka::error::{KafkaError, KafkaResult};
use rdkafka::types::RDKafkaErrorCode;
use super::manifest::OffsetManifest;
-use crate::partitions::{InputGroups, InputTopic, ResumeOffset};
+use crate::partitions::{InputGroups, InputTopic, PartitionOwnership, ResumeOffset};
/// Watermarks of the enabled inputs.
pub trait ReplayWindow {
@@ -143,10 +143,15 @@ impl RestorePlan {
})
}
- /// Where `topic` resumes on every slice the plan keeps.
- pub fn positions(&self, topic: &InputTopic) -> BTreeMap<u16, ResumeOffset> {
+ /// Where `topic` resumes on every slice the plan keeps that `ownership` owns.
+ pub fn positions(
+ &self,
+ topic: &InputTopic,
+ ownership: &PartitionOwnership,
+ ) -> BTreeMap<u16, ResumeOffset> {
self.0
.iter()
+ .filter(|&(&partition, _)| ownership.owns(partition))
.filter_map(|(&partition, slice)| match slice {
SliceRestore::Resume(positions) => Some((partition, *positions.get(topic)?)),
SliceRestore::Reset(_) => None,
@@ -158,6 +163,8 @@ impl RestorePlan {
#[cfg(test)]
mod tests {
use super::*;
+ use std::num::NonZeroU16;
+
use crate::partitions::InputPositions;
use crate::store::durability::lineage::PodOrdinal;
@@ -293,9 +300,20 @@ mod tests {
);
assert!(plan.resumes_any());
assert_eq!(
- plan.positions(&InputTopic::new(EVENTS)),
+ plan.positions(&InputTopic::new(EVENTS), &PartitionOwnership::single(5)),
BTreeMap::from([(0, offset(10)), (4, offset(10))]),
);
+ let pod_of_eight = |ordinal| {
+ PartitionOwnership::new(NonZeroU16::new(8).unwrap(), PodOrdinal::from(ordinal), 8)
+ .unwrap()
+ };
+ for (ordinal, kept) in [(0, 0), (4, 4)] {
+ assert_eq!(
+ plan.positions(&InputTopic::new(MERGES), &pod_of_eight(ordinal)),
+ BTreeMap::from([(kept, offset(5))]),
+ "pod {ordinal} positions only its own kept slice",
+ );
+ }
}
#[test]
@@ -313,7 +331,9 @@ mod tests {
SliceRestore::Resume(BTreeMap::from([(InputTopic::new(EVENTS), offset(10))])),
)]),
);
- assert!(plan.positions(&InputTopic::new(SEEDS)).is_empty());
+ assert!(plan
+ .positions(&InputTopic::new(SEEDS), &PartitionOwnership::single(1))
+ .is_empty());
}
#[test]
diff --git a/rust/cohort-stream-processor/src/store/durability/stage.rs b/rust/cohort-stream-processor/src/store/durability/stage.rs
index 83619263a110..18b9bb4b0bd1 100644
--- a/rust/cohort-stream-processor/src/store/durability/stage.rs
+++ b/rust/cohort-stream-processor/src/store/durability/stage.rs
@@ -21,7 +21,7 @@ use super::manifest::OffsetManifest;
use super::metadata::CheckpointMetadata;
use super::restore_plan::{ReplayWindow, RestorePlan};
use crate::observability::metrics::CHECKPOINT_RESTORE_SLICES_RESET_TOTAL;
-use crate::partitions::{InputGroups, InputTopic, ResumeOffset};
+use crate::partitions::{InputGroups, InputTopic, PartitionOwnership, ResumeOffset};
use crate::store::{CohortStore, StoreError};
/// The marker a published restore carries inside the store directory. RocksDB ignores files it
@@ -205,11 +205,16 @@ impl PendingRestore {
&self.record.source
}
- /// Commits each follower's planned positions. Safe to repeat: nothing consumes a follower
- /// before the restore finishes.
- pub(crate) fn position_followers(&self, groups: &InputGroups) -> KafkaResult<()> {
+ /// Commits each follower's planned positions on the partitions `ownership` owns. The other
+ /// partitions belong to other pods, whose followers keep their own commits. Safe to repeat:
+ /// nothing consumes a follower before the restore finishes.
+ pub(crate) fn position_followers(
+ &self,
+ groups: &InputGroups,
+ ownership: &PartitionOwnership,
+ ) -> KafkaResult<()> {
for follower in groups.followers() {
- follower.commit(&self.plan.positions(follower.topic()))?;
+ follower.commit(&self.plan.positions(follower.topic(), ownership))?;
}
Ok(())
}
diff --git a/rust/cohort-stream-processor/src/store/handle.rs b/rust/cohort-stream-processor/src/store/handle.rs
index 1a27fb06e537..fb43cd35da43 100644
--- a/rust/cohort-stream-processor/src/store/handle.rs
+++ b/rust/cohort-stream-processor/src/store/handle.rs
@@ -490,7 +490,7 @@ impl StoreHandle {
.await
}
- /// Reclaim all state for one partition on rebalance. No permit.
+ /// Reclaim all state for one partition. No permit.
pub async fn delete_partition(&self, partition_id: u16) -> Result<(), StoreError> {
self.write("delete_partition", false, move |store| {
store.delete_partition(partition_id)
@@ -546,17 +546,6 @@ impl StoreHandle {
})
.await
}
-
- /// SYNCHRONOUS escape hatch: delete one partition's state on the caller's thread, bypassing the
- /// offload path entirely.
- ///
- /// The single sanctioned caller is `EventDispatcher::reclaim_stale_slice`, which runs under a
- /// DashMap shard guard where no `.await` is possible. It is rare (a post-boot partition move-in)
- /// and fast (one range tombstone per CF). Any new caller must justify the same no-await
- /// constraint — prefer the async [`Self::delete_partition`] everywhere else.
- pub fn delete_partition_blocking(&self, partition_id: u16) -> Result<(), StoreError> {
- self.store.delete_partition(partition_id)
- }
}
/// RAII in-flight gauge guard, incremented on construction and decremented on drop, both on the
diff --git a/rust/cohort-stream-processor/src/store/keyspace.rs b/rust/cohort-stream-processor/src/store/keyspace.rs
index 604fcf625bf5..06a641e8d80e 100644
--- a/rust/cohort-stream-processor/src/store/keyspace.rs
+++ b/rust/cohort-stream-processor/src/store/keyspace.rs
@@ -179,8 +179,13 @@ pub const META_SCHEMA_VERSION: MetaKey = MetaKey(b"schema_version");
/// Present once the store records slice coverage. The value is empty.
pub const META_SLICE_COVERAGE: MetaKey = MetaKey(b"slice_coverage");
+/// `cf_meta[b"pod_partitions"]` → the [`PartitionOwnership`](crate::partitions::PartitionOwnership)
+/// the store was written under.
+pub const META_POD_PARTITIONS: MetaKey = MetaKey(b"pod_partitions");
+
// `Meta::decode` assumes one key length.
const _: () = assert!(META_SLICE_COVERAGE.0.len() == META_SCHEMA_VERSION.0.len());
+const _: () = assert!(META_POD_PARTITIONS.0.len() == META_SCHEMA_VERSION.0.len());
mod sealed {
/// Sealed so only this module's keyspaces implement [`super::Keyspace`]; downstream code cannot
@@ -283,9 +288,13 @@ impl Keyspace for Meta {
// `cf_meta` keys are a closed set of literals; match them rather than fabricate a `&'static`
// from runtime bytes. A wrong length is a length error; a right-length key that matches no
// literal is an unknown key — conflating the two would report "expected N bytes, got N".
- if let Some(key) = [META_SCHEMA_VERSION, META_SLICE_COVERAGE]
- .into_iter()
- .find(|key| key.0 == bytes)
+ if let Some(key) = [
+ META_SCHEMA_VERSION,
+ META_SLICE_COVERAGE,
+ META_POD_PARTITIONS,
+ ]
+ .into_iter()
+ .find(|key| key.0 == bytes)
{
Ok(key)
} else if bytes.len() != META_SCHEMA_VERSION.0.len() {
diff --git a/rust/cohort-stream-processor/src/store/rocks.rs b/rust/cohort-stream-processor/src/store/rocks.rs
index 01d19c4d4a3e..e4bb7c44f5f0 100644
--- a/rust/cohort-stream-processor/src/store/rocks.rs
+++ b/rust/cohort-stream-processor/src/store/rocks.rs
@@ -37,7 +37,7 @@ use super::keys::{
};
use super::keyspace::{
BehavioralKey, Keyspace, Meta, PersonPrefix, PersonRecordKey, SliceCoverageKey, SliceCoverages,
- META_SCHEMA_VERSION, META_SLICE_COVERAGE,
+ META_POD_PARTITIONS, META_SCHEMA_VERSION, META_SLICE_COVERAGE,
};
use super::staged::{StagedBatch, StagedOp};
use crate::observability::metrics::{
@@ -46,6 +46,7 @@ use crate::observability::metrics::{
STORE_SCHEMA_MISMATCH_WIPES_TOTAL, STORE_WRITE_BATCH_TOTAL, STORE_WRITE_DURATION_SECONDS,
WAL_FSYNC_DURATION_SECONDS, WAL_FSYNC_ERRORS_TOTAL,
};
+use crate::partitions::PartitionOwnership;
/// On-disk store schema version, stamped into `cf_meta` at first open and checked on every reopen.
/// A layout revision (key codec, value shape) that keeps the same CF set MUST bump this so an older
@@ -120,8 +121,10 @@ pub struct StoreConfig {
/// `cf_person_records` **only** — never `cf_behavioral`, whose eviction deadlines are the sweep's
/// contract. See [`super::ttl_filter`].
pub person_record_ttl_days: u32,
- /// Partitions adopted as complete when the store predates coverage records.
- pub partition_count: u16,
+ /// The partitions this pod owns. The store records it at the first open and refuses to open
+ /// under another. A store that predates coverage records adopts all of its partitions as
+ /// complete.
+ pub ownership: PartitionOwnership,
}
impl Default for StoreConfig {
@@ -144,7 +147,7 @@ impl Default for StoreConfig {
periodic_compaction_seconds: 0,
max_background_jobs: 0,
person_record_ttl_days: 0,
- partition_count: COHORT_PARTITION_COUNT as u16,
+ ownership: PartitionOwnership::single(COHORT_PARTITION_COUNT as u16),
}
}
}
@@ -192,6 +195,19 @@ pub enum StoreError {
)]
SchemaMismatch { found: Option<u32>, expected: u32 },
+ /// The store's slices belong to another split of the partitions, so its slices for this pod's
+ /// partitions may be missing or another pod's. `None` when the record is unreadable.
+ #[error(
+ "store ownership mismatch: the store was written as {}, but this is {expected}; refusing \
+ to open (wipe the store, and with checkpoints enabled delete this pod's local and S3 \
+ checkpoints too, to start its slices over)",
+ .found.map_or_else(|| "an unreadable ownership record".to_owned(), |found| found.to_string())
+ )]
+ OwnershipMismatch {
+ found: Option<PartitionOwnership>,
+ expected: PartitionOwnership,
+ },
+
/// A batched read answered fewer slots than it was asked keys. RocksDB answers one slot per
/// key, so this is a backend contract break rather than a data problem, and pairing a slot
/// with the wrong key would attribute one person's stored state to another.
@@ -357,8 +373,9 @@ impl CohortStore {
let store = Self::open_inner(config, &db_opts)?;
match store.check_schema(path_existed, config.wipe_on_schema_mismatch)? {
SchemaCheck::Ok => {
+ store.claim_ownership(config.ownership)?;
if path_existed {
- store.settle_coverage(config.partition_count)?;
+ store.settle_coverage(config.ownership.partition_count())?;
}
return Ok(store);
}
@@ -501,6 +518,27 @@ impl CohortStore {
})
}
+ /// Records `ownership` in a store without an ownership record, and refuses a store written
+ /// under another. A store from before ownership records adopts the configured one.
+ fn claim_ownership(&self, ownership: PartitionOwnership) -> Result<(), StoreError> {
+ match self.get(Meta::CF, &Meta::encode(&META_POD_PARTITIONS))? {
+ None => {
+ self.write_batch(|batch| {
+ batch.put::<Meta>(&META_POD_PARTITIONS, &ownership.encode());
+ })?;
+ info!(%ownership, "recorded the store's partition ownership");
+ Ok(())
+ }
+ Some(recorded) => match PartitionOwnership::decode(&recorded) {
+ Some(found) if found == ownership => Ok(()),
+ found => Err(StoreError::OwnershipMismatch {
+ found,
+ expected: ownership,
+ }),
+ },
+ }
+ }
+
/// Adopt every partition as complete in a store without the coverage stamp. The stamp rides the
/// same batch, so adoption happens once.
fn settle_coverage(&self, partition_count: u16) -> Result<(), StoreError> {
@@ -1623,6 +1661,7 @@ mod tests {
use super::super::keyspace::{Behavioral, PersonRecords};
use crate::stage1::key::LeafStateKey;
+ use crate::store::durability::PodOrdinal;
fn behavioral_key() -> BehavioralKey {
BehavioralKey::new(3, 7, Uuid::from_u128(1), LeafStateKey([0xAB; 16]))
@@ -2795,7 +2834,7 @@ mod tests {
let created = TempDir::new().unwrap();
let config = StoreConfig {
path: created.path().join("db"),
- partition_count: 4,
+ ownership: PartitionOwnership::single(4),
..StoreConfig::default()
};
{
@@ -2862,6 +2901,52 @@ mod tests {
);
}
+ #[test]
+ fn a_store_opens_only_under_the_ownership_it_recorded() {
+ let pods = std::num::NonZeroU16::new(2).unwrap();
+ let pod_zero = PartitionOwnership::new(pods, PodOrdinal::STANDALONE, 4).unwrap();
+ let pod_one = PartitionOwnership::new(pods, PodOrdinal::from(1), 4).unwrap();
+ let dir = TempDir::new().unwrap();
+ let under = |ownership| StoreConfig {
+ path: dir.path().join("db"),
+ ownership,
+ ..StoreConfig::default()
+ };
+
+ drop(CohortStore::open(&under(pod_one)).unwrap());
+ CohortStore::open(&under(pod_one)).expect("the recorded ownership reopens");
+ for other in [pod_zero, PartitionOwnership::single(4)] {
+ assert!(
+ matches!(
+ CohortStore::open(&under(other)),
+ Err(StoreError::OwnershipMismatch { found: Some(found), expected })
+ if found == pod_one && expected == other
+ ),
+ "{other}",
+ );
+ }
+
+ // A store from before ownership records adopts the ownership it opens under.
+ CohortStore::open(&under(pod_one))
+ .unwrap()
+ .write_batch(|batch| batch.delete::<Meta>(&META_POD_PARTITIONS))
+ .unwrap();
+ drop(CohortStore::open(&under(pod_zero)).unwrap());
+ assert!(matches!(
+ CohortStore::open(&under(pod_one)),
+ Err(StoreError::OwnershipMismatch { .. })
+ ));
+
+ CohortStore::open(&under(pod_zero))
+ .unwrap()
+ .write_batch(|batch| batch.put::<Meta>(&META_POD_PARTITIONS, &[0xFF]))
+ .unwrap();
+ assert!(matches!(
+ CohortStore::open(&under(pod_zero)),
+ Err(StoreError::OwnershipMismatch { found: None, .. })
+ ));
+ }
+
#[test]
fn delete_partition_wipes_partitioned_cfs_but_preserves_the_schema_stamp() {
let dir = TempDir::new().unwrap();
diff --git a/rust/cohort-stream-processor/tests/cascade_consumer.rs b/rust/cohort-stream-processor/tests/cascade_consumer.rs
index 7071ca7aa353..f38dadb13850 100644
--- a/rust/cohort-stream-processor/tests/cascade_consumer.rs
+++ b/rust/cohort-stream-processor/tests/cascade_consumer.rs
@@ -29,8 +29,8 @@ use cohort_stream_processor::filters::{
CatalogHandle, CohortId, FilterCatalog, TeamFiltersBuilder, TeamId,
};
use cohort_stream_processor::partitions::{
- merge_partition_key, partition_of, run_rebalance_worker, CohortConsumerContext, Follower,
- FollowerSet, OffsetTracker, PartitionRouter, COHORT_PARTITION_COUNT,
+ merge_partition_key, partition_of, CohortConsumerContext, Follower, FollowerSet, OffsetTracker,
+ PartitionOwnership, PartitionRouter, COHORT_PARTITION_COUNT,
};
use cohort_stream_processor::producer::{
CascadeSink, CohortMembershipChange, KafkaCascadeSink, KafkaMembershipSink,
@@ -55,7 +55,6 @@ use rdkafka::util::Timeout;
use rdkafka::{ClientConfig, Offset, TopicPartitionList};
use serde_json::json;
use tempfile::TempDir;
-use tokio::sync::mpsc;
use tokio::task::JoinHandle;
use uuid::Uuid;
@@ -206,9 +205,7 @@ fn events_client_config(group: &str) -> ClientConfig {
.set("group.id", group)
.set("enable.auto.commit", "false")
.set("enable.auto.offset.store", "false")
- .set("auto.offset.reset", "earliest")
- .set("partition.assignment.strategy", "cooperative-sticky")
- .set("session.timeout.ms", "6000");
+ .set("auto.offset.reset", "earliest");
config
}
@@ -502,28 +499,20 @@ async fn spawn_instance(
.create()
.expect("create cascade follower consumer"),
);
- let followers = Arc::new(FollowerSet::new([
+ let ownership = PartitionOwnership::single(NUM_PARTITIONS as u16);
+ FollowerSet::new([
Follower::new(merges_consumer.clone(), topics.merges.clone()),
Follower::new(transfers_consumer.clone(), topics.transfers.clone()),
Follower::new(cascade_consumer.clone(), topics.cascade.clone()),
- ]));
+ ])
+ .assign(&ownership)
+ .expect("assign the followers");
- let (context, rebalance_rx) = CohortConsumerContext::new(dispatcher.clone());
let events_client: StreamConsumer<CohortConsumerContext> = events_client_config(&groups.events)
- .create_with_context(context)
+ .create_with_context(CohortConsumerContext)
.expect("create events consumer");
- events_client
- .subscribe(&[topics.events.as_str()])
- .expect("subscribe events");
- let (command_tx, command_rx) = mpsc::unbounded_channel();
- let mut tasks = vec![tokio::spawn(run_rebalance_worker(
- rebalance_rx,
- dispatcher.clone(),
- followers,
- command_tx,
- events_handle.shutdown_token(),
- ))];
+ let mut tasks = Vec::new();
let merge_follower = FollowerConsumer::<MergeRoute>::new(
merges_consumer,
@@ -566,10 +555,10 @@ async fn spawn_instance(
100,
RECV_TIMEOUT,
COMMIT_INTERVAL,
- NUM_PARTITIONS as usize,
- command_rx,
+ ownership,
None,
- );
+ )
+ .expect("assign cohort_stream_events");
tasks.push(tokio::spawn(events_consumer.process()));
Instance { dispatcher, tasks }
diff --git a/rust/cohort-stream-processor/tests/events_consumer.rs b/rust/cohort-stream-processor/tests/events_consumer.rs
index 27c5b38cae0a..6f7c7f61a1ab 100644
--- a/rust/cohort-stream-processor/tests/events_consumer.rs
+++ b/rust/cohort-stream-processor/tests/events_consumer.rs
@@ -1,8 +1,8 @@
//! End-to-end tests for the `cohort_stream_events` consumer against a **real** Kafka broker.
//!
-//! `#[ignore]`d by default: the consumer joins a consumer group, commits a `TopicPartitionList`,
-//! and reads its own committed offsets back, none of which the in-process `MockCluster` exercises
-//! faithfully. Run against a local stack with:
+//! `#[ignore]`d by default: the consumer commits a `TopicPartitionList` to its group and reads its
+//! own committed offsets back, which the in-process `MockCluster` does not exercise faithfully. Run
+//! against a local stack with:
//!
//! ```sh
//! cargo test -p cohort-stream-processor --test events_consumer -- --ignored
@@ -43,8 +43,8 @@ use cohort_stream_processor::filters::{
CatalogHandle, CohortId, FilterCatalog, TeamFiltersBuilder, TeamId,
};
use cohort_stream_processor::partitions::{
- run_rebalance_worker, CohortConsumerContext, InputGroups, InputTopic, OffsetTracker,
- PartitionMirror, PartitionRouter, ShuffleMessage, WorkerInbox,
+ CohortConsumerContext, InputGroups, InputTopic, OffsetTracker, PartitionOwnership,
+ PartitionRouter, ShuffleMessage, WorkerInbox,
};
use cohort_stream_processor::producer::{
CaptureSink, CohortMembershipChange, KafkaMembershipSink, MembershipSink, MembershipStatus,
@@ -237,12 +237,8 @@ async fn produce_new_persons(topic: &str, count: u128) -> usize {
count as usize
}
-struct NoopMirror;
-
-impl PartitionMirror for NoopMirror {
- fn assign(&self, _partitions: &[i32]) {}
- fn unassign(&self, _partitions: &[i32]) {}
-}
+/// The single pod's ownership of the test topics.
+const SINGLE_POD: PartitionOwnership = PartitionOwnership::single(NUM_PARTITIONS as u16);
fn build_consumer(
topic: &str,
@@ -261,6 +257,7 @@ fn build_consumer(
handle,
sink,
offset_commit_interval,
+ SINGLE_POD,
false,
100,
);
@@ -276,6 +273,7 @@ fn build_consumer_with_restore(
handle: lifecycle::Handle,
sink: Arc<dyn MembershipSink>,
offset_commit_interval: Duration,
+ ownership: PartitionOwnership,
durable_restore: bool,
recv_batch_size: usize,
) -> (CohortStreamEventsConsumer, Arc<EventDispatcher>) {
@@ -291,43 +289,32 @@ fn build_consumer_with_restore(
dispatcher.enable_durable_restore();
}
- let (context, rebalance_rx) = CohortConsumerContext::new(dispatcher.clone());
- let consumer: StreamConsumer<CohortConsumerContext> = ClientConfig::new()
- .set("bootstrap.servers", bootstrap_servers())
- .set("group.id", group)
- .set("enable.auto.commit", "false")
- .set("enable.auto.offset.store", "false")
- .set("auto.offset.reset", "earliest")
- .set("partition.assignment.strategy", "cooperative-sticky")
- .set("session.timeout.ms", "6000")
- .create_with_context(context)
- .expect("create consumer");
- consumer.subscribe(&[topic]).expect("subscribe");
-
- let (consumer_command_tx, consumer_command_rx) = mpsc::unbounded_channel();
- tokio::spawn(run_rebalance_worker(
- rebalance_rx,
- dispatcher.clone(),
- Arc::new(NoopMirror),
- consumer_command_tx,
- handle.shutdown_token(),
- ));
-
let events_consumer = CohortStreamEventsConsumer::new(
- Arc::new(consumer),
+ Arc::new(events_client(group)),
topic.to_string(),
dispatcher.clone(),
handle,
recv_batch_size,
Duration::from_millis(200),
offset_commit_interval,
- NUM_PARTITIONS as usize,
- consumer_command_rx,
+ ownership,
None,
- );
+ )
+ .expect("assign the owned partitions");
(events_consumer, dispatcher)
}
+fn events_client(group: &str) -> StreamConsumer<CohortConsumerContext> {
+ ClientConfig::new()
+ .set("bootstrap.servers", bootstrap_servers())
+ .set("group.id", group)
+ .set("enable.auto.commit", "false")
+ .set("enable.auto.offset.store", "false")
+ .set("auto.offset.reset", "earliest")
+ .create_with_context(CohortConsumerContext)
+ .expect("create consumer")
+}
+
/// The `murmur2_random` partitioner must match production for keyed co-partitioning.
fn shadow_kafka_config() -> KafkaConfig {
KafkaConfig {
@@ -869,23 +856,19 @@ async fn does_not_commit_past_a_blocked_produce() {
task.await.expect("consumer task panicked");
}
-/// Whether any produced person has behavioral state under `partition` in `store`.
-fn partition_has_state(store: &CohortStore, partition: i32, lsk: LeafStateKey) -> bool {
- (1..=PERSONS).any(|n| {
+/// Whether any of the first `persons` persons has behavioral state under `partition` in `store`.
+fn partition_has_state(
+ store: &CohortStore,
+ partition: i32,
+ lsk: LeafStateKey,
+ persons: u128,
+) -> bool {
+ (1..=persons).any(|n| {
let key = BehavioralKey::new(partition as u16, TEAM as u64, person(n), lsk);
store.get_behavioral(&key).unwrap().is_some()
})
}
-fn distinct_entered(changes: &[CohortMembershipChange]) -> usize {
- changes
- .iter()
- .filter(|change| change.status == MembershipStatus::Entered)
- .map(|change| change.person_id.clone())
- .collect::<std::collections::HashSet<_>>()
- .len()
-}
-
/// Poll the group's committed-offset sum until it reaches `target` or the deadline elapses.
async fn wait_for_committed(
consumer: &StreamConsumer,
@@ -909,31 +892,13 @@ async fn wait_for_committed(
#[tokio::test]
#[ignore = "requires a running Kafka broker (KAFKA_HOSTS); run with --ignored against a local stack"]
-async fn cooperative_sticky_migration_preserves_offsets_and_partition_colocation() {
+async fn two_pods_split_the_events_by_residue_class_and_commit_every_offset() {
let suffix = Uuid::new_v4();
- let topic = format!("cohort_stream_events_migrate_{suffix}");
- let group = format!("cohort-stream-processor-migrate-{suffix}");
+ let topic = format!("cohort_stream_events_split_{suffix}");
+ let group = format!("cohort-stream-processor-split-{suffix}");
create_topic(&topic).await;
-
- let catalog_a = behavioral_catalog();
- let catalog_b = behavioral_catalog();
- let lsk = behavioral_lsk(&catalog_a);
-
- let dir_a = TempDir::new().unwrap();
- let store_a = CohortStore::open(&StoreConfig {
- path: dir_a.path().join("db"),
- ..StoreConfig::default()
- })
- .expect("open store A");
- let dir_b = TempDir::new().unwrap();
- let store_b = CohortStore::open(&StoreConfig {
- path: dir_b.path().join("db"),
- ..StoreConfig::default()
- })
- .expect("open store B");
-
- let sink_a = Arc::new(CaptureSink::new());
- let sink_b = Arc::new(CaptureSink::new());
+ let total = produce_events(&topic).await + produce_new_persons(&topic, NEW_PERSONS).await;
+ let persons = PERSONS + NEW_PERSONS;
let verifier: StreamConsumer = ClientConfig::new()
.set("bootstrap.servers", bootstrap_servers())
@@ -942,111 +907,94 @@ async fn cooperative_sticky_migration_preserves_offsets_and_partition_colocation
.create()
.expect("create verifier");
- let mut manager = Manager::builder("migrate-itest")
+ let mut manager = Manager::builder("split-itest")
.with_trap_signals(false)
.build();
- let handle_a = manager.register(
- "consumer-a",
- ComponentOptions::new().with_graceful_shutdown(Duration::from_secs(15)),
- );
- let handle_b = manager.register(
- "consumer-b",
- ComponentOptions::new().with_graceful_shutdown(Duration::from_secs(15)),
- );
- let shutdown = handle_a.clone();
+ let pods = std::num::NonZeroU16::new(2).unwrap();
+ let handles: Vec<lifecycle::Handle> = (0..pods.get())
+ .map(|ordinal| {
+ manager.register(
+ &format!("consumer-{ordinal}"),
+ ComponentOptions::new().with_graceful_shutdown(Duration::from_secs(15)),
+ )
+ })
+ .collect();
let _monitor = manager.monitor_background();
-
- // Batch 1: only A is up.
- let batch1 = produce_events(&topic).await;
- let (consumer_a, dispatcher_a) = build_consumer_with_restore(
- &topic,
- &group,
- store_a.clone(),
- catalog_a,
- handle_a,
- sink_a.clone(),
- Duration::from_millis(250),
- false,
- 100,
- );
- let task_a = tokio::spawn(consumer_a.process());
- wait_for_committed(&verifier, &topic, batch1 as i64, Duration::from_secs(60)).await;
-
- // B joins → cooperative-sticky moves ~half the partitions off A.
- let (consumer_b, dispatcher_b) = build_consumer_with_restore(
- &topic,
- &group,
- store_b.clone(),
- catalog_b,
- handle_b,
- sink_b.clone(),
- Duration::from_millis(250),
- false,
- 100,
- );
- let task_b = tokio::spawn(consumer_b.process());
-
- // A revoke deletes the slice and hands nothing over. If A folds and commits a batch-2 event
- // before its partition moves to B, neither pod holds that person afterwards. So batch 2 waits
- // for a stable, disjoint, full split, and each event lands on its final owner.
- let start = Instant::now();
- loop {
- let a = dispatcher_a.owned_set();
- let b = dispatcher_b.owned_set();
- if !b.is_empty() && a.is_disjoint(&b) && a.len() + b.len() == NUM_PARTITIONS as usize {
- tokio::time::sleep(Duration::from_millis(750)).await;
- if dispatcher_a.owned_set() == a && dispatcher_b.owned_set() == b {
- break;
- }
- }
- assert!(
- start.elapsed() < Duration::from_secs(60),
- "timed out waiting for the cooperative split to settle (a={}, b={})",
- a.len(),
- b.len(),
+ let mut split = Vec::new();
+ for (ordinal, handle) in (0..pods.get()).zip(handles) {
+ let ownership =
+ PartitionOwnership::new(pods, PodOrdinal::from(ordinal), NUM_PARTITIONS as u16)
+ .unwrap();
+ let dir = TempDir::new().unwrap();
+ let store = CohortStore::open(&StoreConfig {
+ path: dir.path().join("db"),
+ ownership,
+ ..StoreConfig::default()
+ })
+ .expect("open store");
+ let (consumer, dispatcher) = build_consumer_with_restore(
+ &topic,
+ &group,
+ store.clone(),
+ behavioral_catalog(),
+ handle.clone(),
+ Arc::new(CaptureSink::new()),
+ Duration::from_millis(250),
+ ownership,
+ false,
+ 100,
);
- tokio::time::sleep(Duration::from_millis(250)).await;
+ let task = tokio::spawn(consumer.process());
+ split.push((dir, store, dispatcher, handle, task));
}
- // Batch 2: split across both pods.
- let batch2 = produce_events(&topic).await;
- let total = (batch1 + batch2) as i64;
- wait_for_committed(&verifier, &topic, total, Duration::from_secs(60)).await;
- tokio::time::sleep(Duration::from_secs(2)).await;
-
- shutdown.request_shutdown();
- task_a.await.expect("consumer A panicked");
- task_b.await.expect("consumer B panicked");
+ wait_for_committed(&verifier, &topic, total as i64, Duration::from_secs(60)).await;
+ let lsk = behavioral_lsk(&behavioral_catalog());
+ for (_, _, _, handle, _) in &split {
+ handle.request_shutdown();
+ }
+ let mut owned = Vec::new();
+ let mut stores = Vec::new();
+ for (dir, store, dispatcher, _, task) in split {
+ task.await.expect("consumer panicked");
+ let mut partitions: Vec<i32> = dispatcher.owned_partitions();
+ partitions.sort_unstable();
+ owned.push(partitions);
+ stores.push((dir, store));
+ }
+ assert_eq!(
+ owned,
+ [vec![0, 2], vec![1, 3]],
+ "each pod owns its residue class"
+ );
assert_eq!(
committed_sum(&verifier, &topic),
- total,
- "committed offsets cover every produced event",
+ total as i64,
+ "two member-less consumers commit every offset of the one group between them",
);
-
- // Co-location: no partition's state lives in both pods at once.
for partition in 0..NUM_PARTITIONS {
+ let holders: Vec<usize> = stores
+ .iter()
+ .enumerate()
+ .filter(|(_, (_, store))| partition_has_state(store, partition, lsk, persons))
+ .map(|(pod, _)| pod)
+ .collect();
assert!(
- !(partition_has_state(&store_a, partition, lsk)
- && partition_has_state(&store_b, partition, lsk)),
- "partition {partition} has state in both pods — affinity invariant violated",
+ holders.iter().all(|&pod| pod == (partition % 2) as usize),
+ "partition {partition} has state on pods {holders:?}, but only its owner may hold it",
);
}
- let entered_a = entered_persons(&store_a, lsk);
- let entered_b = entered_persons(&store_b, lsk);
assert_eq!(
- entered_a + entered_b,
- PERSONS as usize,
- "every person has state in exactly one pod (a={entered_a}, b={entered_b})",
+ stores
+ .iter()
+ .map(|(_, store)| entered_persons_range(store, lsk, persons as usize))
+ .sum::<usize>(),
+ persons as usize,
+ "every person entered on exactly one pod",
);
- let mut changes = sink_a.changes();
- changes.extend(sink_b.changes());
- assert_eq!(
- distinct_entered(&changes),
- PERSONS as usize,
- "every person entered exactly once (idempotent re-produce aside)",
- );
+ delete_topic(&topic).await;
}
/// Reopen the store at `path` without wiping (durable restore), retrying briefly while the previous
@@ -1125,6 +1073,7 @@ async fn durable_restart_reopens_live_state_and_fires_a_dormant_left() {
handle,
Arc::new(CaptureSink::new()),
Duration::from_millis(250),
+ SINGLE_POD,
true,
100,
);
@@ -1181,6 +1130,7 @@ async fn durable_restart_reopens_live_state_and_fires_a_dormant_left() {
handle,
Arc::new(CaptureSink::new()),
Duration::from_millis(250),
+ SINGLE_POD,
true,
10,
);
@@ -1334,40 +1284,18 @@ fn build_consumer_finishing_restore(
));
dispatcher.enable_durable_restore();
- let (context, rebalance_rx) = CohortConsumerContext::new(dispatcher.clone());
- let consumer: StreamConsumer<CohortConsumerContext> = ClientConfig::new()
- .set("bootstrap.servers", bootstrap_servers())
- .set("group.id", group)
- .set("enable.auto.commit", "false")
- .set("enable.auto.offset.store", "false")
- .set("auto.offset.reset", "earliest")
- .set("partition.assignment.strategy", "cooperative-sticky")
- .set("session.timeout.ms", "6000")
- .create_with_context(context)
- .expect("create consumer");
- consumer.subscribe(&[topic]).expect("subscribe");
-
- let (consumer_command_tx, consumer_command_rx) = mpsc::unbounded_channel();
- tokio::spawn(run_rebalance_worker(
- rebalance_rx,
- dispatcher.clone(),
- Arc::new(NoopMirror),
- consumer_command_tx,
- handle.shutdown_token(),
- ));
-
CohortStreamEventsConsumer::new(
- Arc::new(consumer),
+ Arc::new(events_client(group)),
topic.to_string(),
dispatcher,
handle,
100,
Duration::from_millis(200),
offset_commit_interval,
- NUM_PARTITIONS as usize,
- consumer_command_rx,
+ SINGLE_POD,
Some(restore),
)
+ .expect("assign the owned partitions")
}
/// Each partition's high watermark.
@@ -1474,7 +1402,8 @@ async fn s3_restore_replays_past_the_checkpoint_after_the_broker_moved_on() {
// --- Tenure 1: fold, checkpoint through the sweeper, then fold and commit past it. ---
{
- let store = CohortStore::open(&config.store_config()).expect("open tenure-1 store");
+ let store =
+ CohortStore::open(&config.store_config(SINGLE_POD)).expect("open tenure-1 store");
let mut manager = Manager::builder("s3restore-itest-1")
.with_trap_signals(false)
.build();
@@ -1492,6 +1421,7 @@ async fn s3_restore_replays_past_the_checkpoint_after_the_broker_moved_on() {
handle,
Arc::new(CaptureSink::new()),
Duration::from_millis(250),
+ SINGLE_POD,
true,
100,
);
@@ -1562,13 +1492,15 @@ async fn s3_restore_replays_past_the_checkpoint_after_the_broker_moved_on() {
.expect("remove checkpoint_local_dir (simulate PVC loss)");
// --- Tenure 2: the boot restores from S3, then the consumer finishes the restore. ---
- let (store2, restore) = open_store(&config, Some(lineage), groups.clone())
+ let (store2, restore) = open_store(&config, SINGLE_POD, Some(lineage), groups.clone())
.await
.expect("open the restored store");
let restore = restore
.expect("with the live store and local checkpoint gone, the boot restores a checkpoint");
assert_eq!(restore.source().label(), "s3");
- let resumed = restore.plan().positions(&InputTopic::new(topic.as_str()));
+ let resumed = restore
+ .plan()
+ .positions(&InputTopic::new(topic.as_str()), &SINGLE_POD);
assert_eq!(
resumed
.get(&(idle_partition as u16))
@@ -1583,7 +1515,7 @@ async fn s3_restore_replays_past_the_checkpoint_after_the_broker_moved_on() {
assert_eq!(
restore
.plan()
- .positions(&InputTopic::new(topics.merges.as_str()))
+ .positions(&InputTopic::new(topics.merges.as_str()), &SINGLE_POD)
.get(&(merges_with_events as u16))
.map(|offset| offset.get()),
Some(0),
diff --git a/rust/cohort-stream-processor/tests/merge_consumer.rs b/rust/cohort-stream-processor/tests/merge_consumer.rs
index a37704b0ebd0..0ce9e39a33fa 100644
--- a/rust/cohort-stream-processor/tests/merge_consumer.rs
+++ b/rust/cohort-stream-processor/tests/merge_consumer.rs
@@ -41,8 +41,8 @@ use cohort_stream_processor::merge::transfer::{
MergeStateTransfer, PersonMergeEvent, Tombstone, MERGE_EVENT_SCHEMA_VERSION,
};
use cohort_stream_processor::partitions::{
- merge_partition_key, partition_of, run_rebalance_worker, CohortConsumerContext, Follower,
- FollowerSet, OffsetTracker, PartitionRouter, COHORT_PARTITION_COUNT,
+ merge_partition_key, partition_of, CohortConsumerContext, Follower, FollowerSet, OffsetTracker,
+ PartitionOwnership, PartitionRouter, COHORT_PARTITION_COUNT,
};
use cohort_stream_processor::producer::{
CaptureCascadeSink, CaptureSink, CohortMembershipChange, KafkaMembershipSink,
@@ -50,6 +50,7 @@ use cohort_stream_processor::producer::{
TransferSink,
};
use cohort_stream_processor::stage1::{Stage1State, StateVariant, StatefulRecord};
+use cohort_stream_processor::store::durability::PodOrdinal;
use cohort_stream_processor::store::{
BehavioralKey, CohortStore, LeafStateKey, OffloadConfig, OffloadMode, StoreConfig, StoreHandle,
TombstoneKey,
@@ -69,7 +70,6 @@ use rdkafka::util::Timeout;
use rdkafka::{ClientConfig, Offset, TopicPartitionList};
use serde_json::json;
use tempfile::TempDir;
-use tokio::sync::mpsc;
use tokio::task::JoinHandle;
use uuid::Uuid;
@@ -244,9 +244,7 @@ fn events_client_config(group: &str) -> ClientConfig {
.set("group.id", group)
.set("enable.auto.commit", "false")
.set("enable.auto.offset.store", "false")
- .set("auto.offset.reset", "earliest")
- .set("partition.assignment.strategy", "cooperative-sticky")
- .set("session.timeout.ms", "6000");
+ .set("auto.offset.reset", "earliest");
config
}
@@ -453,8 +451,13 @@ async fn produce_merge(producer: &FutureProducer, topic: &str, event: &PersonMer
}
fn open_store(dir: &TempDir) -> CohortStore {
+ open_store_as(dir, PartitionOwnership::single(NUM_PARTITIONS as u16))
+}
+
+fn open_store_as(dir: &TempDir, ownership: PartitionOwnership) -> CohortStore {
CohortStore::open(&StoreConfig {
path: dir.path().join("db"),
+ ownership,
..StoreConfig::default()
})
.expect("open store")
@@ -588,6 +591,7 @@ async fn spawn_instance(
catalog: CatalogHandle,
membership_sink: Arc<dyn MembershipSink>,
handles: [Handle; 3],
+ ownership: PartitionOwnership,
) -> Instance {
let [events_handle, merge_handle, transfer_handle] = handles;
let kafka_config = producer_kafka_config();
@@ -646,27 +650,18 @@ async fn spawn_instance(
.create()
.expect("create transfers follower consumer"),
);
- let followers = Arc::new(FollowerSet::new([
+ FollowerSet::new([
Follower::new(merges_consumer.clone(), topics.merges.clone()),
Follower::new(transfers_consumer.clone(), topics.transfers.clone()),
- ]));
+ ])
+ .assign(&ownership)
+ .expect("assign the followers");
- let (context, rebalance_rx) = CohortConsumerContext::new(dispatcher.clone());
let events_client: StreamConsumer<CohortConsumerContext> = events_client_config(&groups.events)
- .create_with_context(context)
+ .create_with_context(CohortConsumerContext)
.expect("create events consumer");
- events_client
- .subscribe(&[topics.events.as_str()])
- .expect("subscribe events");
- let (command_tx, command_rx) = mpsc::unbounded_channel();
- let mut tasks = vec![tokio::spawn(run_rebalance_worker(
- rebalance_rx,
- dispatcher.clone(),
- followers,
- command_tx,
- events_handle.shutdown_token(),
- ))];
+ let mut tasks = Vec::new();
let merge_follower = FollowerConsumer::<MergeRoute>::new(
merges_consumer,
@@ -698,10 +693,10 @@ async fn spawn_instance(
100,
RECV_TIMEOUT,
COMMIT_INTERVAL,
- NUM_PARTITIONS as usize,
- command_rx,
+ ownership,
None,
- );
+ )
+ .expect("assign cohort_stream_events");
tasks.push(tokio::spawn(events_consumer.process()));
Instance {
@@ -870,6 +865,7 @@ async fn cross_partition_merge_completes_end_to_end_on_the_wire() {
catalog,
membership,
handles,
+ PartitionOwnership::single(NUM_PARTITIONS as u16),
)
.await;
wait_for(
@@ -1008,6 +1004,7 @@ async fn duplicate_merge_message_produces_exactly_one_transfer() {
catalog,
Arc::new(CaptureSink::new()),
handles,
+ PartitionOwnership::single(NUM_PARTITIONS as u16),
)
.await;
wait_for(
@@ -1092,6 +1089,7 @@ async fn post_merge_straggler_re_keys_to_p_new_and_folds() {
catalog,
Arc::new(CaptureSink::new()),
handles,
+ PartitionOwnership::single(NUM_PARTITIONS as u16),
)
.await;
wait_for(
@@ -1274,20 +1272,20 @@ async fn follower_re_establishes_consumption_from_the_committed_offset() {
.await;
}
+/// Two pods at N=2: each owns one residue class on every input, and a merge whose old and new
+/// persons sit on different pods drains on one and applies on the other, in both directions.
#[tokio::test]
#[ignore = "requires a running Kafka broker (KAFKA_HOSTS); run with --ignored against a local stack"]
-async fn cooperative_migration_mid_merge_stream_loses_no_merge() {
+async fn two_pods_complete_merges_across_their_residue_classes() {
let suffix = Uuid::new_v4();
let topics = Topics::unique(&suffix);
let groups = Groups::unique(&suffix);
with_topics_cleanup(&topics.names(), async {
topics.create().await;
- let catalog_a = merge_catalog();
- let catalog_b = merge_catalog();
- let (_, daily_lsk) = behavioral_lsks(&catalog_a);
+ let (_, daily_lsk) = behavioral_lsks(&merge_catalog());
- let mut manager = Manager::builder("merge-migrate-itest")
+ let mut manager = Manager::builder("merge-split-itest")
.with_trap_signals(false)
.build();
let handles_a = register_instance(&mut manager, "a");
@@ -1295,164 +1293,88 @@ async fn cooperative_migration_mid_merge_stream_loses_no_merge() {
let shutdown = handles_a[0].clone();
let _monitor = manager.monitor_background();
+ let pods = std::num::NonZeroU16::new(2).unwrap();
+ let pod = |ordinal| {
+ PartitionOwnership::new(pods, PodOrdinal::from(ordinal), NUM_PARTITIONS as u16).unwrap()
+ };
let dir_a = TempDir::new().unwrap();
let dir_b = TempDir::new().unwrap();
let sink_a = Arc::new(CaptureSink::new());
let sink_b = Arc::new(CaptureSink::new());
-
let a = spawn_instance(
&topics,
&groups,
- open_store(&dir_a),
- catalog_a,
+ open_store_as(&dir_a, pod(0)),
+ merge_catalog(),
sink_a.clone(),
handles_a,
+ pod(0),
)
.await;
- wait_for(
- "instance A to own every partition",
- Duration::from_secs(30),
- || a.owned().len() == NUM_PARTITIONS as usize,
- )
- .await;
-
- let producer = murmur2_producer();
- let mut alloc = PersonAlloc::new();
- let mut source_offset = 0i64;
- let mut seed = |producer: &FutureProducer, topic: &str, person: Uuid| {
- let offsets = (source_offset, source_offset + 1);
- source_offset += 2;
- let topic = topic.to_string();
- let producer = producer.clone();
- async move {
- produce_event(&producer, &topic, person, TS, offsets.0).await;
- produce_event(&producer, &topic, person, TS, offsets.1).await;
- }
- };
-
- // Batch 1: seed and trigger against A alone, B joins while merges are in flight.
- let batch1: Vec<(Uuid, Uuid)> = (0..4).map(|_| alloc.cross_partition_pair()).collect();
- for (p_old, _) in &batch1 {
- seed(&producer, &topics.events, *p_old).await;
- }
- wait_for("batch 1 seed state", Duration::from_secs(60), || {
- batch1
- .iter()
- .all(|(p_old, _)| daily_total(&a.store, daily_lsk, *p_old) == Some(2))
- })
- .await;
- for (p_old, p_new) in &batch1 {
- produce_merge(&producer, &topics.merges, &merge_event(*p_old, *p_new)).await;
- }
-
let b = spawn_instance(
&topics,
&groups,
- open_store(&dir_b),
- catalog_b,
+ open_store_as(&dir_b, pod(1)),
+ merge_catalog(),
sink_b.clone(),
handles_b,
+ pod(1),
)
.await;
+ let evens: HashSet<i32> = (0..NUM_PARTITIONS).step_by(2).collect();
+ let odds: HashSet<i32> = (1..NUM_PARTITIONS).step_by(2).collect();
+ assert_eq!(a.owned(), evens);
+ assert_eq!(b.owned(), odds);
- // Wait for a stable, disjoint full split.
- let (a_owned, b_owned) = {
- let deadline = Duration::from_secs(60);
- let start = Instant::now();
- loop {
- let a1 = a.owned();
- let b1 = b.owned();
- if !b1.is_empty()
- && a1.is_disjoint(&b1)
- && a1.len() + b1.len() == NUM_PARTITIONS as usize
- {
- tokio::time::sleep(Duration::from_millis(750)).await;
- if a.owned() == a1 && b.owned() == b1 {
- break (a1, b1);
- }
- }
- assert!(
- start.elapsed() < deadline,
- "timed out waiting for the cooperative split to settle (a={}, b={})",
- a.owned().len(),
- b.owned().len(),
- );
- tokio::time::sleep(Duration::from_millis(250)).await;
- }
- };
-
- // Batch 2: directed pairs (A→B, B→A) plus generic pairs to cover both transfer directions.
- let pair_ab = (
- alloc.next_on(|p| a_owned.contains(&(p as i32))),
- alloc.next_on(|p| b_owned.contains(&(p as i32))),
- );
- let pair_ba = (
- alloc.next_on(|p| b_owned.contains(&(p as i32))),
- alloc.next_on(|p| a_owned.contains(&(p as i32))),
- );
- let mut batch2 = vec![pair_ab, pair_ba];
- batch2.extend((0..4).map(|_| alloc.cross_partition_pair()));
-
- let in_either = |stores: [&CohortStore; 2], person: Uuid| {
- stores
- .iter()
- .filter_map(|store| daily_total(store, daily_lsk, person))
- .next()
- };
-
- for (p_old, _) in &batch2 {
- seed(&producer, &topics.events, *p_old).await;
+ let producer = murmur2_producer();
+ let mut alloc = PersonAlloc::new();
+ let on_a = |p: u16| p.is_multiple_of(2);
+ let on_b = |p: u16| p % 2 == 1;
+ let mut pairs = vec![
+ (alloc.next_on(on_a), alloc.next_on(on_b)),
+ (alloc.next_on(on_b), alloc.next_on(on_a)),
+ ];
+ pairs.extend((0..4).map(|_| alloc.cross_partition_pair()));
+ let owner = |person: Uuid| if on_a(part(person)) { &a } else { &b };
+
+ for (n, (p_old, _)) in pairs.iter().enumerate() {
+ let offset = 2 * n as i64;
+ produce_event(&producer, &topics.events, *p_old, TS, offset).await;
+ produce_event(&producer, &topics.events, *p_old, TS, offset + 1).await;
}
- wait_for("batch 2 seed state", Duration::from_secs(60), || {
- batch2
- .iter()
- .all(|(p_old, _)| in_either([&a.store, &b.store], *p_old) == Some(2))
- })
+ wait_for(
+ "seed state on each old person's pod",
+ Duration::from_secs(60),
+ || {
+ pairs.iter().all(|(p_old, _)| {
+ daily_total(&owner(*p_old).store, daily_lsk, *p_old) == Some(2)
+ })
+ },
+ )
.await;
- for (p_old, p_new) in &batch2 {
+ for (p_old, p_new) in &pairs {
produce_merge(&producer, &topics.merges, &merge_event(*p_old, *p_new)).await;
}
-
wait_for(
- "every batch 2 merge to complete",
+ "every merge to apply on the new person's pod",
Duration::from_secs(90),
|| {
- batch2
- .iter()
- .all(|(_, p_new)| in_either([&a.store, &b.store], *p_new) == Some(2))
+ pairs.iter().all(|(_, p_new)| {
+ daily_total(&owner(*p_new).store, daily_lsk, *p_new) == Some(2)
+ })
},
)
.await;
- let total_merges = (batch1.len() + batch2.len()) as i64;
let merge_verifier = group_verifier(&groups.merges);
let transfer_verifier = group_verifier(&groups.transfers);
wait_for(
"merge commits to cover every merge",
Duration::from_secs(90),
- || committed_sum(&merge_verifier, &topics.merges) == total_merges,
+ || committed_sum(&merge_verifier, &topics.merges) == pairs.len() as i64,
)
.await;
- // Batch 1 may produce empty re-drains after migration; batch 2 transfers are deterministic.
let total_transfers = topic_message_count(&topics.transfers);
- let transfer_pairs: HashSet<(Uuid, Uuid)> = consume_all(
- &topics.transfers,
- total_transfers as usize,
- Duration::from_secs(30),
- )
- .await
- .iter()
- .map(|(_, payload)| {
- let transfer = MergeStateTransfer::decode(payload).expect("decode transfer");
- (transfer.old_person_uuid, transfer.new_person_uuid)
- })
- .collect();
- for (p_old, p_new) in &batch2 {
- assert!(
- transfer_pairs.contains(&(*p_old, *p_new)),
- "no transfer rode the wire for batch-2 merge {p_old} → {p_new}",
- );
- }
wait_for(
"transfer commits to cover every transfer",
Duration::from_secs(90),
@@ -1460,59 +1382,29 @@ async fn cooperative_migration_mid_merge_stream_loses_no_merge() {
)
.await;
- for (p_old, p_new) in &batch2 {
- let totals = [
- daily_total(&a.store, daily_lsk, *p_new),
- daily_total(&b.store, daily_lsk, *p_new),
- ];
- assert!(
- totals.contains(&Some(2)) && totals.contains(&None),
- "P_new {p_new} must hold the merged count in exactly one store, got {totals:?}",
- );
- assert_eq!(
- daily_total(&a.store, daily_lsk, *p_old),
- None,
- "P_old {p_old} drained in A"
- );
+ for (p_old, p_new) in &pairs {
+ for instance in [&a, &b] {
+ assert_eq!(
+ daily_total(&instance.store, daily_lsk, *p_old),
+ None,
+ "P_old {p_old} drained",
+ );
+ }
+ let other = if std::ptr::eq(owner(*p_new), &a) {
+ &b
+ } else {
+ &a
+ };
assert_eq!(
- daily_total(&b.store, daily_lsk, *p_old),
+ daily_total(&other.store, daily_lsk, *p_new),
None,
- "P_old {p_old} drained in B"
+ "P_new {p_new} holds the merged count only on its own pod",
);
-
- let owner = if a_owned.contains(&(part(*p_old) as i32)) {
- &a.store
- } else {
- &b.store
- };
- let tombstone = tombstone_for(owner, *p_old)
+ let tombstone = tombstone_for(&owner(*p_old).store, *p_old)
.unwrap_or_else(|| panic!("tombstone for {p_old} on its owner's slice"));
assert_eq!(tombstone.new_person, *p_new);
}
- assert_eq!(
- daily_total(&b.store, daily_lsk, pair_ab.1),
- Some(2),
- "the A-drained transfer applied on B",
- );
- assert_eq!(
- daily_total(&a.store, daily_lsk, pair_ba.1),
- Some(2),
- "the B-drained transfer applied on A",
- );
-
- // Batch 1 raced the migration; any surviving count must be exact (no double-apply).
- for (_, p_new) in &batch1 {
- for store in [&a.store, &b.store] {
- if let Some(total) = daily_total(store, daily_lsk, *p_new) {
- assert_eq!(
- total, 2,
- "batch 1 P_new {p_new} was double-applied despite the dedup",
- );
- }
- }
- }
-
let entered: HashSet<String> = sink_a
.changes()
.into_iter()
@@ -1520,7 +1412,7 @@ async fn cooperative_migration_mid_merge_stream_loses_no_merge() {
.filter(|change| change.status == MembershipStatus::Entered)
.map(|change| change.person_id)
.collect();
- for (_, p_new) in &batch2 {
+ for (_, p_new) in &pairs {
assert!(
entered.contains(&p_new.to_string()),
"P_new {p_new} entered on the shadow output",
diff --git a/rust/cohort-stream-processor/tests/merge_durability.rs b/rust/cohort-stream-processor/tests/merge_durability.rs
index 1ee02585663d..1f309991bbb0 100644
--- a/rust/cohort-stream-processor/tests/merge_durability.rs
+++ b/rust/cohort-stream-processor/tests/merge_durability.rs
@@ -39,8 +39,8 @@ use cohort_stream_processor::merge::transfer::{
PersonMergeEvent, Tombstone, MERGE_EVENT_SCHEMA_VERSION,
};
use cohort_stream_processor::partitions::{
- merge_partition_key, partition_of, run_rebalance_worker, CohortConsumerContext, Follower,
- FollowerSet, InputGroups, InputTopic, OffsetTracker, PartitionRouter, COHORT_PARTITION_COUNT,
+ merge_partition_key, partition_of, CohortConsumerContext, Follower, FollowerSet, InputGroups,
+ InputTopic, OffsetTracker, PartitionOwnership, PartitionRouter, COHORT_PARTITION_COUNT,
};
use cohort_stream_processor::producer::{
CohortMembershipChange, KafkaCascadeSink, KafkaMembershipSink, KafkaStreamEventSink,
@@ -69,7 +69,6 @@ use rdkafka::producer::{FutureProducer, FutureRecord};
use rdkafka::util::Timeout;
use rdkafka::{ClientConfig, Offset, TopicPartitionList};
use serde_json::json;
-use tokio::sync::mpsc;
use tokio::task::JoinHandle;
use uuid::Uuid;
@@ -257,9 +256,7 @@ fn events_client_config(group: &str) -> ClientConfig {
.set("group.id", group)
.set("enable.auto.commit", "false")
.set("enable.auto.offset.store", "false")
- .set("auto.offset.reset", "earliest")
- .set("partition.assignment.strategy", "cooperative-sticky")
- .set("session.timeout.ms", "6000");
+ .set("auto.offset.reset", "earliest");
config
}
@@ -733,28 +730,20 @@ async fn spawn_instance(
.expect("create cascade follower consumer"),
);
- let followers = Arc::new(FollowerSet::new([
+ let ownership = PartitionOwnership::single(NUM_PARTITIONS as u16);
+ FollowerSet::new([
Follower::new(merges_consumer.clone(), topics.merges.clone()),
Follower::new(transfers_consumer.clone(), topics.transfers.clone()),
Follower::new(cascade_consumer.clone(), topics.cascade.clone()),
- ]));
+ ])
+ .assign(&ownership)
+ .expect("assign the followers");
- let (context, rebalance_rx) = CohortConsumerContext::new(dispatcher.clone());
let events_client: StreamConsumer<CohortConsumerContext> = events_client_config(&groups.events)
- .create_with_context(context)
+ .create_with_context(CohortConsumerContext)
.expect("create events consumer");
- events_client
- .subscribe(&[topics.events.as_str()])
- .expect("subscribe events");
- let (command_tx, command_rx) = mpsc::unbounded_channel();
- let mut tasks = vec![tokio::spawn(run_rebalance_worker(
- rebalance_rx,
- dispatcher.clone(),
- followers,
- command_tx,
- events_handle.shutdown_token(),
- ))];
+ let mut tasks = Vec::new();
let merge_follower = FollowerConsumer::<MergeRoute>::new(
merges_consumer,
@@ -802,10 +791,10 @@ async fn spawn_instance(
100,
RECV_TIMEOUT,
COMMIT_INTERVAL,
- NUM_PARTITIONS as usize,
- command_rx,
+ ownership,
restore,
- );
+ )
+ .expect("assign cohort_stream_events");
tasks.push(tokio::spawn(events_consumer.process()));
Instance {
@@ -815,8 +804,6 @@ async fn spawn_instance(
}
}
-/// `durable_restore_enabled && cohort_cascade_enabled` requires `durable_restore_single_pod &&
-/// pod_identity().is_some()` to pass `validate_startup`, so those knobs are set too.
#[allow(clippy::too_many_arguments)]
fn merge_durability_config(
store_path: &Path,
@@ -829,7 +816,6 @@ fn merge_durability_config(
env.insert("CHECKPOINT_ENABLED".into(), "true".into());
env.insert("DURABLE_RESTORE_ENABLED".into(), "true".into());
env.insert("COHORT_CASCADE_ENABLED".into(), "true".into());
- env.insert("DURABLE_RESTORE_SINGLE_POD".into(), "true".into());
// Ends in a StatefulSet ordinal, which checkpoints require.
env.insert(
"POD_NAME".into(),
@@ -997,6 +983,9 @@ async fn merge_and_cascade_state_survive_an_s3_disaster_restore() {
let store_path = root.path().join("db");
let checkpoint_dir = root.path().join("checkpoints");
let config = merge_durability_config(&store_path, &checkpoint_dir, &prefix, &topics, &groups);
+ let ownership = config
+ .partition_ownership()
+ .expect("the pod name ends in an ordinal");
let names = topics.names();
let config_for_cleanup = config.clone();
@@ -1014,7 +1003,7 @@ async fn merge_and_cascade_state_survive_an_s3_disaster_restore() {
let shutdown = handles[0].clone();
let _monitor = manager.monitor_background();
- let store1 = CohortStore::open(&config.store_config()).expect("open tenure-1 store");
+ let store1 = CohortStore::open(&config.store_config(ownership)).expect("open tenure-1 store");
let instance = spawn_instance(
&topics,
&groups,
@@ -1127,6 +1116,7 @@ async fn merge_and_cascade_state_survive_an_s3_disaster_restore() {
// Tenure 2: the boot restores from S3 and positions the follower groups; verify the state.
let (store2, restore) = open_store(
&config,
+ ownership,
Some(config.checkpoint_lineage().expect("ordinal pod name")),
input_groups.clone(),
)
@@ -1137,10 +1127,12 @@ async fn merge_and_cascade_state_survive_an_s3_disaster_restore() {
assert_eq!(restore.source().label(), "s3");
// The followers resume past the merge on P_old's partition and the transfer on P_new's.
- let merge_resume = restore.plan().positions(&InputTopic::new(topics.merges.as_str()));
+ let merge_resume = restore
+ .plan()
+ .positions(&InputTopic::new(topics.merges.as_str()), &ownership);
let transfer_resume = restore
.plan()
- .positions(&InputTopic::new(topics.transfers.as_str()));
+ .positions(&InputTopic::new(topics.transfers.as_str()), &ownership);
assert!(
merge_resume.get(&part(p_old)).map(|offset| offset.get()) >= Some(1),
"the merge follower resumes past the merge on P_old's partition: {merge_resume:?}",
diff --git a/rust/cohort-stream-processor/tests/seed_consumer.rs b/rust/cohort-stream-processor/tests/seed_consumer.rs
index 05a2d5ad2b7d..99947e92282a 100644
--- a/rust/cohort-stream-processor/tests/seed_consumer.rs
+++ b/rust/cohort-stream-processor/tests/seed_consumer.rs
@@ -14,7 +14,7 @@
use std::collections::{HashMap, HashSet};
use std::future::Future;
-use std::num::NonZeroU32;
+use std::num::{NonZeroU16, NonZeroU32};
use std::panic::AssertUnwindSafe;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
@@ -35,8 +35,8 @@ use cohort_stream_processor::filters::{
};
use cohort_stream_processor::observability::disk::SharedDiskUtilization;
use cohort_stream_processor::partitions::{
- merge_partition_key, partition_for, partition_of, run_rebalance_worker, CohortConsumerContext,
- ConsumerPauser, Follower, FollowerSet, LiveWatermarks, OffsetTracker, PartitionPauser,
+ merge_partition_key, partition_for, partition_of, CohortConsumerContext, ConsumerPauser,
+ Follower, FollowerSet, LiveWatermarks, OffsetTracker, PartitionOwnership, PartitionPauser,
PartitionRouter, SeedPacingConfig, COHORT_PARTITION_COUNT,
};
use cohort_stream_processor::producer::{
@@ -47,6 +47,7 @@ use cohort_stream_processor::producer::{
};
use cohort_stream_processor::stage1::bucket_tz::day_idx_in_tz;
use cohort_stream_processor::stage2::state::Stage2State;
+use cohort_stream_processor::store::durability::PodOrdinal;
use cohort_stream_processor::store::{
CohortStore, OffloadConfig, OffloadMode, ReadLane, Stage2Key, StagedBatch, StoreConfig,
StoreHandle,
@@ -60,14 +61,13 @@ use futures::FutureExt;
use lifecycle::{ComponentOptions, Handle, Manager};
use rdkafka::admin::{AdminClient, AdminOptions, NewTopic, TopicReplication};
use rdkafka::client::DefaultClientContext;
-use rdkafka::consumer::{Consumer, StreamConsumer};
+use rdkafka::consumer::{Consumer, ConsumerContext, StreamConsumer};
use rdkafka::message::Message;
use rdkafka::producer::{FutureProducer, FutureRecord};
use rdkafka::util::Timeout;
use rdkafka::{ClientConfig, Offset, TopicPartitionList};
use serde_json::json;
use tempfile::TempDir;
-use tokio::sync::mpsc;
use tokio::task::JoinHandle;
use uuid::Uuid;
@@ -214,9 +214,7 @@ fn events_client_config(group: &str) -> ClientConfig {
.set("group.id", group)
.set("enable.auto.commit", "false")
.set("enable.auto.offset.store", "false")
- .set("auto.offset.reset", "earliest")
- .set("partition.assignment.strategy", "cooperative-sticky")
- .set("session.timeout.ms", "6000");
+ .set("auto.offset.reset", "earliest");
config
}
@@ -457,8 +455,13 @@ async fn produce_reconcile(
}
fn open_store(dir: &TempDir) -> CohortStore {
+ open_store_as(dir, PartitionOwnership::single(NUM_PARTITIONS as u16))
+}
+
+fn open_store_as(dir: &TempDir, ownership: PartitionOwnership) -> CohortStore {
CohortStore::open(&StoreConfig {
path: dir.path().join("db"),
+ ownership,
..StoreConfig::default()
})
.expect("open store")
@@ -544,9 +547,24 @@ struct Instance {
seed_tracker: Arc<OffsetTracker>,
seed_consumer: Arc<StreamConsumer>,
live_watermarks: Arc<LiveWatermarks>,
+ /// The partitions each input's consumer is assigned: events, merges, transfers, cascades and
+ /// seeds, in that order.
+ assignments: [Vec<i32>; 5],
tasks: Vec<JoinHandle<()>>,
}
+fn assigned<C: ConsumerContext>(consumer: &StreamConsumer<C>) -> Vec<i32> {
+ let mut partitions: Vec<i32> = consumer
+ .assignment()
+ .expect("read the consumer's assignment")
+ .elements()
+ .iter()
+ .map(|elem| elem.partition())
+ .collect();
+ partitions.sort_unstable();
+ partitions
+}
+
impl Instance {
fn owned(&self) -> HashSet<i32> {
self.dispatcher.owned_partitions().into_iter().collect()
@@ -615,6 +633,7 @@ async fn spawn_instance(
store: CohortStore,
catalog: CatalogHandle,
handles: [Handle; 5],
+ ownership: PartitionOwnership,
fence_margin_ms: i64,
membership: Option<Arc<dyn MembershipSink>>,
) -> Instance {
@@ -720,30 +739,27 @@ async fn spawn_instance(
.create()
.expect("create seed follower consumer"),
);
- let followers = Arc::new(FollowerSet::new([
+ FollowerSet::new([
Follower::new(merges_consumer.clone(), topics.merges.clone()),
Follower::new(transfers_consumer.clone(), topics.transfers.clone()),
Follower::new(cascade_consumer.clone(), topics.cascade.clone()),
Follower::new(seeds_consumer.clone(), topics.seeds.clone()),
- ]));
+ ])
+ .assign(&ownership)
+ .expect("assign the followers");
+ let follower_assignments = [
+ assigned(&merges_consumer),
+ assigned(&transfers_consumer),
+ assigned(&cascade_consumer),
+ assigned(&seeds_consumer),
+ ];
- let (context, rebalance_rx) = CohortConsumerContext::new(dispatcher.clone());
let events_client: StreamConsumer<CohortConsumerContext> = events_client_config(&groups.events)
- .create_with_context(context)
+ .create_with_context(CohortConsumerContext)
.expect("create events consumer");
- events_client
- .subscribe(&[topics.events.as_str()])
- .expect("subscribe events");
let events_client = Arc::new(events_client);
- let (command_tx, command_rx) = mpsc::unbounded_channel();
- let mut tasks = vec![tokio::spawn(run_rebalance_worker(
- rebalance_rx,
- dispatcher.clone(),
- followers,
- command_tx,
- events_handle.shutdown_token(),
- ))];
+ let mut tasks = Vec::new();
let merge_follower = FollowerConsumer::<MergeRoute>::new(
merges_consumer,
@@ -801,6 +817,7 @@ async fn spawn_instance(
);
tasks.push(tokio::spawn(seed_follower.process()));
+ let events_probe = events_client.clone();
let events_consumer = CohortStreamEventsConsumer::new(
events_client,
topics.events.clone(),
@@ -809,11 +826,12 @@ async fn spawn_instance(
100,
RECV_TIMEOUT,
COMMIT_INTERVAL,
- NUM_PARTITIONS as usize,
- command_rx,
+ ownership,
None,
- );
+ )
+ .expect("assign cohort_stream_events");
tasks.push(tokio::spawn(events_consumer.process()));
+ let [merges, transfers, cascades, seeds] = follower_assignments;
Instance {
dispatcher,
@@ -822,10 +840,71 @@ async fn spawn_instance(
seed_tracker,
seed_consumer: seeds_consumer,
live_watermarks,
+ assignments: [assigned(&events_probe), merges, transfers, cascades, seeds],
tasks,
}
}
+/// Two pods at N=2 each own one residue class of every input, so no partition of any input belongs
+/// to both, and the dispatcher owns the same class the consumers fetch.
+#[tokio::test]
+#[ignore = "requires a running Kafka broker (KAFKA_HOSTS); run with --ignored against a local stack"]
+async fn two_pods_own_disjoint_residue_classes_on_all_five_inputs() {
+ let suffix = Uuid::new_v4();
+ let topics = Topics::unique(&suffix);
+ let groups = Groups::unique(&suffix);
+ with_topics_cleanup(&topics.names(), async {
+ topics.create().await;
+
+ let pods = NonZeroU16::new(2).unwrap();
+ let mut pods_running = Vec::new();
+ for ordinal in 0..pods.get() {
+ let mut manager = Manager::builder(format!("seed-split-itest-{ordinal}"))
+ .with_trap_signals(false)
+ .build();
+ let handles = register_instance(&mut manager);
+ let shutdown = handles[0].clone();
+ let monitor = manager.monitor_background();
+ let dir = TempDir::new().unwrap();
+ let ownership =
+ PartitionOwnership::new(pods, PodOrdinal::from(ordinal), NUM_PARTITIONS as u16)
+ .unwrap();
+ let instance = spawn_instance(
+ &topics,
+ &groups,
+ open_store_as(&dir, ownership),
+ seed_catalog(),
+ handles,
+ ownership,
+ 2_000,
+ None,
+ )
+ .await;
+ pods_running.push((dir, monitor, shutdown, instance));
+ }
+
+ for (ordinal, (_, _, _, instance)) in pods_running.iter().enumerate() {
+ let class: Vec<i32> = (0..NUM_PARTITIONS)
+ .filter(|partition| partition % 2 == ordinal as i32)
+ .collect();
+ assert_eq!(
+ instance.assignments,
+ [(); 5].map(|()| class.clone()),
+ "pod {ordinal}"
+ );
+ let mut owned: Vec<i32> = instance.owned().into_iter().collect();
+ owned.sort_unstable();
+ assert_eq!(owned, class, "pod {ordinal}'s dispatcher");
+ }
+
+ for (_, _, shutdown, instance) in pods_running {
+ shutdown.request_shutdown();
+ instance.join().await;
+ }
+ })
+ .await;
+}
+
/// A tile lands on its owning worker via the 5-topic co-assignment, applies behind an open
/// fence, emits a tagged change, and the seed group's commits reach the produced high-water mark.
#[tokio::test]
@@ -851,6 +930,7 @@ async fn seed_tile_applies_on_the_owning_worker_and_commits_to_the_hwm() {
open_store(&dir),
seed_catalog(),
handles,
+ PartitionOwnership::single(NUM_PARTITIONS as u16),
2_000,
None,
)
@@ -1030,6 +1110,7 @@ async fn a_failed_seed_produce_is_re_emitted_after_a_restart() {
open_store(&dir),
seed_catalog(),
handles,
+ PartitionOwnership::single(NUM_PARTITIONS as u16),
2_000,
Some(failing.clone()),
)
@@ -1095,6 +1176,7 @@ async fn a_failed_seed_produce_is_re_emitted_after_a_restart() {
reopen_store_live(&dir).await,
seed_catalog(),
handles,
+ PartitionOwnership::single(NUM_PARTITIONS as u16),
2_000,
None,
)
@@ -1171,6 +1253,7 @@ async fn reconcile_snapshot_repairs_stale_state_and_commits_after_markers() {
open_store(&dir),
seed_catalog(),
handles,
+ PartitionOwnership::single(NUM_PARTITIONS as u16),
2_000,
None,
)
@@ -1518,6 +1601,7 @@ async fn fence_holds_a_fresh_tile_until_live_consumption_passes_its_scan_point()
open_store(&dir),
seed_catalog(),
handles,
+ PartitionOwnership::single(NUM_PARTITIONS as u16),
fence_margin_ms,
None,
)
diff --git a/rust/cohort-stream-processor/tests/sweep_worker.rs b/rust/cohort-stream-processor/tests/sweep_worker.rs
index 9d567fc65224..d8e6da4ed652 100644
--- a/rust/cohort-stream-processor/tests/sweep_worker.rs
+++ b/rust/cohort-stream-processor/tests/sweep_worker.rs
@@ -2264,11 +2264,17 @@ async fn dispatch_sweeper_routes_an_end_to_end_eviction() {
let sweeper = DispatchSweeper::with_clock(dispatcher.clone(), margin_ms, Arc::new(move || now));
sweeper.run_once().await;
- // Drain the worker through a revoke so it processes the queued event then the routed sweep before
- // we assert. The `Left` can only come from the sweep evicting the member (a revoke alone never
- // emits one), so its presence proves the routed `Sweep` reached the worker and fired.
- dispatcher.revoke_partition_sync(0);
- dispatcher.revoke_partition_drain(0).await;
+ // The `Left` can only come from the sweep evicting the member, so its arrival proves the routed
+ // `Sweep` reached the worker and fired.
+ let start = std::time::Instant::now();
+ while sink.changes().len() < 2 {
+ assert!(
+ start.elapsed() < std::time::Duration::from_secs(10),
+ "the routed sweep never evicted the member: {:?}",
+ sink.changes(),
+ );
+ tokio::time::sleep(std::time::Duration::from_millis(10)).await;
+ }
let changes = sink.changes();
assert_eq!(