:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
perf(cohorts): pipeline live produce acks and batch cascade settles (#126619)
Matheus Batistaf62351e72452
diff --git a/docs/internal/realtime-cohorts/invariants.md b/docs/internal/realtime-cohorts/invariants.mdindex 9c2a545a2d0c..b5eb1f5fab83 100644--- a/docs/internal/realtime-cohorts/invariants.md+++ b/docs/internal/realtime-cohorts/invariants.md@@ -232,6 +232,7 @@ Most of these are not enforced by a single test, and a violation usually shows uWith it off, which is the default, the last message in Kafka order wins.- Limit: across a restart or a partition move, version order follows the wall clock.A change with no usable version bypasses the ordering.+ With version writes off, a producer retry can land an older change after a newer one.- See [membership output and readers](membership-output-and-readers.md).**29. Membership rows, snapshot bookkeeping and consumer progress commit together.**diff --git a/docs/internal/realtime-cohorts/live-evaluation.md b/docs/internal/realtime-cohorts/live-evaluation.mdindex 95b964eda9d8..a6b18b4cbe8a 100644--- a/docs/internal/realtime-cohorts/live-evaluation.md+++ b/docs/internal/realtime-cohorts/live-evaluation.md@@ -89,7 +89,8 @@ For one event the steps are:The event's changes join the sub-batch's output buffer.Each behavioral row with a finite deadline is placed in the worker's eviction queue at its new deadline.10. **Produce at the end of the sub-batch.**- The worker produces the buffered changes and, with cascades enabled, one cascade message per change, then marks the offsets.+ The worker hands the buffered changes and, with cascades enabled, one cascade message per change to the producer.+ It then marks the offsets without waiting for the acknowledgments.[Processor runtime](processor-runtime.md#the-worker-loop) describes this step.## Behavioral leaves@@ -223,7 +224,7 @@ Each change is a small JSON message on the membership output topic, keyed by `peLive output is at most once.State commits before the change is produced.-If the produce fails, the next successful batch commits past it, and a replay finds nothing left to emit.+If the produce fails, the batch's mark has already passed it, and a replay finds nothing left to emit.The pipeline accepts this and relies on reconcile to repair the downstream table.[Processor runtime](processor-runtime.md#delivery-semantics) lists every path's guarantee.diff --git a/docs/internal/realtime-cohorts/merges-and-cascades.md b/docs/internal/realtime-cohorts/merges-and-cascades.mdindex 70df9c7988dc..6c18565ceb76 100644--- a/docs/internal/realtime-cohorts/merges-and-cascades.md+++ b/docs/internal/realtime-cohorts/merges-and-cascades.md@@ -194,13 +194,20 @@ That answer is wrong for anyone who really is in the untracked cohort.The sweep and reconcile send them only for recomposed cohorts.The message carries the flipped cohort, a depth, and the chain of cohorts visited so far.2. The topic is co-partitioned with the event stream, so the message comes back to the same worker, the one that owns the person's state.-3. The worker finds every cohort that references the flipped cohort and recomposes each referrer that composes.+3. The worker handles a sub-batch of cascade messages as one group.+ For each message it finds every cohort that references the flipped cohort and recomposes each referrer that composes.It reads a single-leaf referenced cohort from its leaf state, a composed one from its Stage 2 row, and an untracked one as not a member.-4. For each referrer that flips, it produces the membership change and, if the chain may continue, a further cascade message.-5. Only after both produces are acknowledged does it write the referrers' new Stage 2 bits and mark the offset.- A failure pins the partition's cascade commit at that message.- Kafka redelivers it after the next restart, and the redelivered message recomputes the same flip.+ If a message's person already has an unwritten Stage 2 bit in the group, the group settles first, so the message reads the written bit.+ A group that holds 1,000 Stage 2 writes also settles before its next message, which bounds the output it holds in memory.+4. For each referrer that flips, the group collects the membership change and, if the chain may continue, a further cascade message.+ The group produces both topics for all of its messages at once, in one round trip.+5. A message's new Stage 2 bits are written only after all of its produces are acknowledged.+ The group writes the bits of every such message in one batch, and the worker marks the sub-batch's last offset.+ A message with a failed produce pins the partition's cascade commit at its own offset, and the rest of the sub-batch goes on.+ Kafka redelivers it after the next restart, and the redelivered message recomputes the same flips.Until then, cascade commits for that partition stall.+ The flips of the failed message that did reach the topic also have no Stage 2 bit behind them until then.+ So a later recompute for that person can miss a change, and only reconcile repairs it.Three limits bound the work:diff --git a/docs/internal/realtime-cohorts/processor-runtime.md b/docs/internal/realtime-cohorts/processor-runtime.mdindex 0a4cb46dd74e..59af0328160e 100644--- a/docs/internal/realtime-cohorts/processor-runtime.md+++ b/docs/internal/realtime-cohorts/processor-runtime.md@@ -115,19 +115,29 @@ The sweep alternates with live batches, so a large wave of expirations at midnig[Time and eviction](time-and-eviction.md) explains the sweep.Each live sub-batch holds one topic's messages, or one timer tick, in offset order.-As a guard, before any merge, transfer, cascade, sweep or reconcile message, the worker first produces the membership changes it has buffered, so the order of output matches the order of state changes.+As a guard, before any merge, transfer, cascade, sweep or reconcile message, the worker first hands the membership changes it has buffered to the producer, so output is submitted in the order of state changes.It yields to the async runtime every few milliseconds, so a CPU-heavy backlog cannot starve other tasks.At the end of an events sub-batch the worker:-1. produces its membership changes and waits for every acknowledgment,-2. produces first-hop cascade messages, when cascades are enabled,-3. re-keys events for merged persons to their survivor's partition,-4. marks the events tracker at the highest event offset of the sub-batch,-5. advances the partition's live watermark to the newest broker timestamp it folded.--If any produce fails, the worker skips the mark.-Merge, transfer and cascade messages settle their own trackers as each one is handled.+1. hands its membership changes, and their first-hop cascade messages when cascades are enabled, to the partition's produce window,+2. re-keys events for merged persons to their survivor's partition and waits for those acknowledgments,+3. marks the events tracker at the highest event offset of the sub-batch,+4. advances the partition's live watermark to the newest broker timestamp it folded.++The mark does not wait for the membership and cascade acknowledgments.+Live output is at most once and its state is already committed, so waiting would only hold the partition's live lane for a broker round trip.+The window holds at most four unacknowledged batches per partition.+When it is full, the worker waits for the oldest one, so a slow broker shows as lag on that partition, and `output_window_waits_total` counts each wait.+The worker enqueues each batch's records before it moves on and collects only the acknowledgments later, so records are enqueued in the order of state changes.+A producer retry can still reorder batches that are in flight together.+With version writes on, the membership consumer's version check puts them back in order ([invariant 28](invariants.md#membership-output)).+With them off, an older change can win until the next change for that cohort and person.+Before a worker stops, it waits for every batch in its window.++If the re-key produce fails, the worker skips the mark.+Merge and transfer messages settle their own trackers as each one is handled.+The cascade messages of a sub-batch settle in groups, as [merges and cascades](merges-and-cascades.md#how-a-cascade-flows) describes.## Offset tracking@@ -157,7 +167,7 @@ This table is worth knowing before you debug a missing or duplicated membership| Path | Guarantee | Why || ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- || Events into Stage 1 state | At least once across restarts, applied once. Within a tenure, an event whose fold fails on a store error, or whose re-key to a survivor's partition fails, is skipped and later committed past | Offsets commit after the work. A redelivered event is recognized by its replay marks. Only the merge, cascade and seed paths hold a failed offset |-| Event to membership change | At most once | State commits inside the fold, before the produce. If the produce fails, the next clean sub-batch marks past it, and a replay finds no transition to emit |+| Event to membership change | At most once | State commits inside the fold, before the produce. The mark does not wait for the produce, and a replay finds no transition to emit || First-hop cascade messages | At most once from the live, merge and sweep paths. At least once from seed apply and reconcile | The live, merge and sweep paths produce them after the flip is committed. Seed apply and reconcile produce them before writing the flip, and retry on failure || Sweep expiry of a single-leaf cohort | At least once while the worker runs | The sweep produces before it writes, and reschedules the keys on failure. The retry lives in the in-memory queue, so a restart keeps it only with durable restore on || Sweep expiry that recomposes a cohort | At most once | The composed result is written before it is produced |@@ -335,7 +345,7 @@ Durable restore is enabled.The last one, 1204567, is a `$pageview` from p-1.Its fold commits the new leaf state to RocksDB without waiting for the disk.The leaf flips, so cohort 42 gets an `entered` change for p-1.-4. At the end of the sub-batch, the worker produces the change and waits for the acknowledgment.+4. At the end of the sub-batch, the worker hands the change to the producer and marks the offset without waiting for the acknowledgment.Processed becomes 1204568.5. On the next commit tick, the task flushes the write-ahead log and commits 1204568.@@ -344,10 +354,9 @@ Two variations show the edges.- **Backpressure.** If partition 26 already held more than 1,016 events, the default budget of 1,024 would refuse these 8.The sub-batch would go to the holdover and fetching for the partition would pause.Dispatched would stay at 1204560 until the worker caught up.-- **Produce failure.** If the produce in step 4 failed, processed would not move.- But the leaf state is already committed.- When the next sub-batch succeeds, processed jumps past 1204567, and the `entered` is never emitted on the live path.- A restart before that would replay 1204567 against the reopened store, see it as a duplicate, and still emit nothing.+- **Produce failure.** If the produce in step 4 failed, processed would still become 1204568, because the mark does not wait for it.+ The leaf state is already committed, so the `entered` is never emitted on the live path.+ A restart that replays 1204567 against the reopened store sees it as a duplicate and still emits nothing.Only reconcile repairs it.## Optimizationsdiff --git a/rust/cohort-stream-processor/src/consumers/events.rs b/rust/cohort-stream-processor/src/consumers/events.rsindex 3f789d628de6..bd077488fb56 100644--- a/rust/cohort-stream-processor/src/consumers/events.rs+++ b/rust/cohort-stream-processor/src/consumers/events.rs@@ -3273,7 +3273,7 @@ mod tests {}#[tokio::test]- async fn mixed_batch_marks_each_tracker_and_membership_produce_failure_holds_only_events() {+ async fn mixed_batch_marks_each_tracker_and_a_membership_produce_failure_holds_none() {let (_dir, store) = temp_store();let (dispatcher, sink, _transfer_sink, merge) = dispatcher_full(&store,@@ -3304,12 +3304,12 @@ mod tests {assert!(sink.changes().is_empty(),- "the failed flush recorded nothing"+ "the failed produce recorded nothing");assert_eq!(tracker.committable_offsets().get(&partition),- None,- "the membership produce failure holds the events offset",+ Some(&11),+ "live output is at most once, so the failed produce does not hold the events offset",);assert_eq!(merge.merge_tracker.committable_offsets().get(&partition),diff --git a/rust/cohort-stream-processor/src/observability/metrics.rs b/rust/cohort-stream-processor/src/observability/metrics.rsindex effe1945d184..9b543b034c32 100644--- a/rust/cohort-stream-processor/src/observability/metrics.rs+++ b/rust/cohort-stream-processor/src/observability/metrics.rs@@ -45,9 +45,14 @@ pub const CASCADE_CYCLE_DETECTED_RUNTIME_TOTAL: &str = "cascade_cycle_detected_r/// Referrer re-evaluations dropped past `cohort_cascade_fanout_cap` (counter). Cohort ids are/// logged, not labelled, to keep cardinality bounded.pub const CASCADE_FANOUT_CAPPED_TOTAL: &str = "cascade_fanout_capped_total";-/// First-hop or onward cascade produces to `cohort_cascade_events` that failed (counter). On the-/// event path a failure holds the batch; on the sweep/merge paths it drops (at-most-once).+/// First-hop or onward cascade produces to `cohort_cascade_events` that failed (counter). The+/// live, sweep and merge paths drop the cascade (at-most-once); the cascade consumer holds its+/// offset.pub const CASCADE_PRODUCE_ERRORS_TOTAL: &str = "cascade_produce_errors_total";+/// Cascade messages settled as one group, which share one produce round trip and one `cf_stage2`+/// write (histogram). Only messages that write a bit count. A person who repeats in a sub-batch, or a+/// group that fills, splits the sub-batch into groups, so small values mean little batching.+pub const CASCADE_BATCH_MESSAGES: &str = "cascade_batch_messages";/// The cascade-consumer commit floor pinned by a sticky offset hold, labelled by `partition` (gauge)./// Non-zero means a held cascade re-evaluation is stalling the commit. **Alert on a sustained/// non-zero level.**@@ -437,9 +442,12 @@ pub const OUTPUT_MEMBERSHIP_CHANGES_EMITTED: &str = "output_membership_changes_e/// Leaf transitions that mapped to zero output cohorts, labelled by `reason` (counter).pub const OUTPUT_TRANSITIONS_UNMAPPED: &str = "output_transitions_unmapped_total";/// Produce failures on the membership output topic (counter). Not every failure loses data: most-/// paths hold or reschedule the offset for replay. Sweep stage 2 and merge drop the change,+/// paths hold or reschedule the offset for replay. Live, sweep stage 2 and merge drop the change,/// because their state is already committed.pub const OUTPUT_PRODUCE_ERRORS: &str = "output_produce_errors_total";+/// Live output batches a worker waited on because its produce window was full (counter). Non-zero+/// means the broker acks slower than the partition's live output arrives.+pub const OUTPUT_WINDOW_WAITS_TOTAL: &str = "output_window_waits_total";/// Sweep cycles that fired, labelled by `loop`/// (`eviction`|`redrive`|`merge_gc`|`checkpoint`|`store_stats`) (counter).diff --git a/rust/cohort-stream-processor/src/producer/batcher.rs b/rust/cohort-stream-processor/src/producer/batcher.rsindex 368c7cfcb271..c5119ac02d0c 100644--- a/rust/cohort-stream-processor/src/producer/batcher.rs+++ b/rust/cohort-stream-processor/src/producer/batcher.rs@@ -20,12 +20,12 @@ impl OutputBuffer {self.changes.extend(changes);}- pub fn is_empty(&self) -> bool {- self.changes.is_empty()- }-- pub fn take(&mut self) -> Vec<CohortMembershipChange> {- std::mem::take(&mut self.changes)+ /// `None` when nothing is buffered, so a caller never produces an empty batch.+ pub fn take(&mut self) -> Option<Vec<CohortMembershipChange>> {+ if self.changes.is_empty() {+ return None;+ }+ Some(std::mem::take(&mut self.changes))}}@@ -47,19 +47,18 @@ mod tests {}#[test]- fn push_extend_take_round_trip() {+ fn take_drains_in_order_and_yields_nothing_when_empty() {let mut buffer = OutputBuffer::new();- assert!(buffer.is_empty());+ assert!(buffer.take().is_none(), "an empty buffer has no batch");buffer.push(change(1));buffer.extend([change(2), change(3)]);- assert!(!buffer.is_empty());- let taken = buffer.take();+ let taken = buffer.take().expect("three changes buffered");assert_eq!(taken.iter().map(|c| c.cohort_id).collect::<Vec<_>>(),vec![1, 2, 3]);- assert!(buffer.is_empty(), "take leaves the buffer empty");+ assert!(buffer.take().is_none(), "take left the buffer empty");}}diff --git a/rust/cohort-stream-processor/src/producer/cascade.rs b/rust/cohort-stream-processor/src/producer/cascade.rsindex dd8ca9daf1df..7a878249a34d 100644--- a/rust/cohort-stream-processor/src/producer/cascade.rs+++ b/rust/cohort-stream-processor/src/producer/cascade.rs@@ -19,6 +19,8 @@ use crate::partitions::partitioner::merge_partition_key;use crate::producer::kafka::AlwaysHealthy;use crate::producer::merge::Capture;+/// `produce` must enqueue every message before its future first returns `Pending`, for the reason+/// [`MembershipSink`](crate::producer::MembershipSink) gives.#[async_trait]pub trait CascadeSink: Send + Sync {async fn produce(&self, messages: Vec<CascadeMessage>) -> Vec<Result<(), KafkaProduceError>>;diff --git a/rust/cohort-stream-processor/src/producer/mod.rs b/rust/cohort-stream-processor/src/producer/mod.rsindex 6327cd2128de..6c8f08cf9a90 100644--- a/rust/cohort-stream-processor/src/producer/mod.rs+++ b/rust/cohort-stream-processor/src/producer/mod.rs@@ -161,6 +161,9 @@ fn format_last_updated(timestamp_micros: i64) -> String {.to_string()}+/// `produce` must hand every change to the producer before its future first returns `Pending`, and+/// await only the acks after that. The worker's produce window polls a live batch's produce once+/// and then moves on, so a sink that awaits before it enqueues lets later output overtake the batch.#[async_trait]pub trait MembershipSink: Send + Sync {async fn produce(diff --git a/rust/cohort-stream-processor/src/workers/cascade_path.rs b/rust/cohort-stream-processor/src/workers/cascade_path.rsindex 7fce6d46f0d0..d602aee7d5fb 100644--- a/rust/cohort-stream-processor/src/workers/cascade_path.rs+++ b/rust/cohort-stream-processor/src/workers/cascade_path.rs@@ -1,20 +1,29 @@//! The cascade re-evaluation handler.//!-//! [`handle_cascade`] re-evaluates every cohort that references the flipped cohort+//! [`handle_cascades`] re-evaluates every cohort that references a flipped cohort//! ([`TeamFilters::cohorts_referencing`]), emits each referrer's flip to `cohort_membership_changed`,//! and continues the chain to `cohort_cascade_events`, depth- and cycle-bounded by [`should_emit`].//! `cohort_cascade_events` is co-partitioned with `cohort_stream_events` so the worker that owns//! the person's `cf_stage2` always handles it.//!-//! **Produce-before-state.** A referrer's new `cf_stage2` bit is committed only after both produces-//! ack; a failure holds without writing, so replay re-detects the still-old bit and re-emits (the-//! downstream UPSERT is idempotent). The cascade input is a fixed message that recomputes the same-//! flip on replay — unlike Stage 2 composition, which dedupes by `cf_behavioral`'s applied-offset and-//! writes state-before-produce (at-most-once).-+//! **One group per sub-batch.** The messages of a worker sub-batch share one produce round trip and+//! one `cf_stage2` write batch. A message whose person already has an unwritten bit in the group+//! settles the group first, so the message reads the written bit instead of the stale one. A group+//! that holds [`CASCADE_GROUP_WRITES`] writes also settles before its next message.+//!+//! **Produce-before-state, per message.** A message's new `cf_stage2` bits are committed only after+//! all of its records ack; a message with a failed record holds its own offset without writing, so+//! replay re-detects the still-old bits and re-emits (the downstream UPSERT is idempotent). The rest+//! of its group still commits, because a message reads and writes only its own person's rows. The+//! cascade input is a fixed message that recomputes the same flip on replay — unlike Stage 2+//! composition, which dedupes by `cf_behavioral`'s applied-offset and writes state-before-produce+//! (at-most-once).++use std::collections::HashSet;use std::sync::Arc;use chrono::NaiveDateTime;+use futures::future::join_all;use metrics::{counter, gauge, histogram};use tracing::{debug, warn};use uuid::Uuid;@@ -22,211 +31,313 @@ use uuid::Uuid;use crate::cascade::{should_emit, CascadeDecision, CascadeMessage, DropReason};use crate::filters::manager::CatalogHandle;use crate::filters::reverse_index::TeamFilters;-use crate::filters::{CohortId, TeamId};+use crate::filters::{CohortId, FilterCatalog, TeamId};use crate::observability::metrics::{- CASCADE_CYCLE_DETECTED_RUNTIME_TOTAL, CASCADE_DEPTH_EXCEEDED_TOTAL, CASCADE_DEPTH_OBSERVED,- CASCADE_FANOUT_CAPPED_TOTAL, CASCADE_HELD_OFFSET_GAUGE, COHORT_STREAM_OFFSET_AHEAD_OF_DISPATCH,+ CASCADE_BATCH_MESSAGES, CASCADE_CYCLE_DETECTED_RUNTIME_TOTAL, CASCADE_DEPTH_EXCEEDED_TOTAL,+ CASCADE_DEPTH_OBSERVED, CASCADE_FANOUT_CAPPED_TOTAL, CASCADE_HELD_OFFSET_GAUGE,+ COHORT_STREAM_OFFSET_AHEAD_OF_DISPATCH,};use crate::partitions::offset_tracker::{MarkOutcome, OffsetTracker};-use crate::producer::{CohortMembershipChange, MembershipSink};+use crate::producer::{CohortMembershipChange, LastUpdatedClock, MembershipSink};use crate::stage2::state::Stage2State;use crate::store::{ReadLane, Stage2Key, StagedBatch, StoreHandle};use crate::workers::merge_path::MergeWorkerDeps;use crate::workers::stage2_path::recompute_and_diff;use crate::workers::worker::{produce_cascades, produce_membership};-/// Re-evaluate every referrer of the flipped cohort carried by one `cohort_cascade_events` message,-/// emit their flips, continue the chain, and settle the cascade offset.+/// The `cf_stage2` writes at which a cascade group settles before it takes another message. One+/// message adds at most `fanout_cap` writes, so this bounds the output a group holds in memory,+/// which the sub-batch length times the fan-out cap would not.+const CASCADE_GROUP_WRITES: usize = 1_000;++/// Re-evaluate every referrer of the cohorts flipped by one worker sub-batch of+/// `cohort_cascade_events` messages, emit their flips, continue the chains, and settle the cascade+/// offsets. `batch` is in offset order.#[allow(clippy::too_many_arguments)]#[hotpath::measure]-pub(crate) async fn handle_cascade(+pub(crate) async fn handle_cascades(partition_id: u16,handle: &StoreHandle,catalog: &CatalogHandle,sink: &Arc<dyn MembershipSink>,merge: &MergeWorkerDeps,- last_updated: &str,- message: &CascadeMessage,- offset: i64,+ clock: &mut LastUpdatedClock,+ batch: Vec<(Box<CascadeMessage>, i64)>,) {- // A config flip can leave a cascade queued after the gate closed; drain it rather than wedge.- if !merge.cascade.enabled {- mark_processed(&merge.cascade_tracker, partition_id, offset);- return;- }-- histogram!(CASCADE_DEPTH_OBSERVED).record(f64::from(message.depth));-- let snapshot = catalog.load();- let Some(filters) = snapshot.team(TeamId(message.change.team_id)) else {- mark_processed(&merge.cascade_tracker, partition_id, offset);- return;- };- let filters: &TeamFilters = filters;-- let Ok(person_id) = Uuid::parse_str(&message.change.person_id) else {- mark_processed(&merge.cascade_tracker, partition_id, offset);+ let Some(&(_, last)) = batch.last() else {return;};+ // A config flip can leave cascades queued after the gate closed; drain them rather than wedge.+ if merge.cascade.enabled {+ let snapshot = catalog.load();+ let mut group = CascadeGroup::open(clock);+ for (message, offset) in batch {+ histogram!(CASCADE_DEPTH_OBSERVED).record(f64::from(message.depth));+ let Some((filters, person_id, referrers)) =+ referrers_to_reevaluate(&snapshot, &message, merge.cascade.fanout_cap)+ else {+ continue;+ };+ if group.touched.contains(&person_id) || group.writes >= CASCADE_GROUP_WRITES {+ group.settle(partition_id, handle, sink, merge).await;+ group = CascadeGroup::open(clock);+ }+ group+ .reevaluate(+ partition_id,+ handle,+ filters,+ merge,+ &message,+ offset,+ person_id,+ referrers,+ )+ .await;+ }+ group.settle(partition_id, handle, sink, merge).await;+ }+ // A message whose read, produce or write failed is held, and the hold caps this mark.+ mark_processed(&merge.cascade_tracker, partition_id, last);+}- let flipped = CohortId(message.change.cohort_id);- let referrers = filters.cohorts_referencing(flipped);+/// The team filters, person and referrers one cascade message re-evaluates. `None` when there is+/// nothing to re-evaluate: an unknown team, an unparseable person, or no referrer.+fn referrers_to_reevaluate<'a>(+ catalog: &'a FilterCatalog,+ message: &CascadeMessage,+ fanout_cap: usize,+) -> Option<(&'a TeamFilters, Uuid, &'a [CohortId])> {+ let filters: &TeamFilters = catalog.team(TeamId(message.change.team_id))?;+ let person_id = Uuid::parse_str(&message.change.person_id).ok()?;+ let referrers = filters.cohorts_referencing(CohortId(message.change.cohort_id));if referrers.is_empty() {- mark_processed(&merge.cascade_tracker, partition_id, offset);- return;+ return None;}-// The capped remainder is recovered only when each dropped referrer is re-evaluated on its next// event; the sweep does not re-evaluate cohort-ref shapes with no behavioral leaf.- let referrers: &[CohortId] = if referrers.len() > merge.cascade.fanout_cap {- let dropped = (referrers.len() - merge.cascade.fanout_cap) as u64;+ if referrers.len() > fanout_cap {+ let dropped = (referrers.len() - fanout_cap) as u64;counter!(CASCADE_FANOUT_CAPPED_TOTAL).increment(dropped);debug!(upstream_cohort_id = message.change.cohort_id,dropped, "cascade fan-out capped",);- &referrers[..merge.cascade.fanout_cap]- } else {- referrers- };+ return Some((filters, person_id, &referrers[..fanout_cap]));+ }+ Some((filters, person_id, referrers))+}- let evaluated_at_ms = clickhouse_millis(last_updated);- let mut writes: Vec<(Stage2Key, Stage2State)> = Vec::new();- let mut changes: Vec<CohortMembershipChange> = Vec::new();- let mut outgoing: Vec<CascadeMessage> = Vec::new();- for &referrer in referrers {- // The reverse index includes excluded referrers (cycle members, unresolved refs); only- // re-evaluate cohorts maintained in `cf_stage2`, else a cycle member would re-enter the chain.- let composes = filters- .eligibility- .get(&referrer)- .is_some_and(|elig| elig.writes_cf_stage2());- if !composes {- continue;+/// Cascade messages of one sub-batch whose flips share one produce round trip and whose bits one+/// write batch commits.+struct CascadeGroup {+ /// One stamp for every change in the group. A group holds at most one message per person, so no+ /// two of its changes share a cohort and person.+ last_updated: String,+ evaluated_at_ms: i64,+ /// The persons of `messages`, whose `cf_stage2` writes are not committed yet.+ touched: HashSet<Uuid>,+ /// The `cf_stage2` writes across `messages`.+ writes: usize,+ messages: Vec<Reevaluated>,+}++/// One message's output. Its bits commit only if all of its records ack.+struct Reevaluated {+ offset: i64,+ writes: Vec<(Stage2Key, Stage2State)>,+ changes: Vec<CohortMembershipChange>,+ outgoing: Vec<CascadeMessage>,+}++impl CascadeGroup {+ fn open(clock: &mut LastUpdatedClock) -> Self {+ let last_updated = clock.next();+ Self {+ evaluated_at_ms: clickhouse_millis(&last_updated),+ last_updated,+ touched: HashSet::new(),+ writes: 0,+ messages: Vec::new(),}- let Some(tree) = filters.cohorts.get(&referrer) else {- continue;+ }++ /// Recompose each referrer for `person_id` and collect its bit, its flip and its onward hop. The+ /// message joins the group only if it writes a bit, and every flip writes one.+ #[allow(clippy::too_many_arguments)]+ async fn reevaluate(+ &mut self,+ partition_id: u16,+ handle: &StoreHandle,+ filters: &TeamFilters,+ merge: &MergeWorkerDeps,+ message: &CascadeMessage,+ offset: i64,+ person_id: Uuid,+ referrers: &[CohortId],+ ) {+ let mut reevaluated = Reevaluated {+ offset,+ writes: Vec::new(),+ changes: Vec::new(),+ outgoing: Vec::new(),};- let diff = match recompute_and_diff(- partition_id,- person_id,- tree,- filters,- handle,- ReadLane::Event,- )- .await- {- Ok(diff) => diff,- Err(error) => {- warn!(- partition_id,- team_id = message.change.team_id,- referrer = referrer.0,- error = %error,- "cascade recompute read failed; holding the cascade offset for redelivery",- );- hold(&merge.cascade_tracker, partition_id, offset);- return;+ for &referrer in referrers {+ // The reverse index includes excluded referrers (cycle members, unresolved refs); only+ // re-evaluate cohorts maintained in `cf_stage2`, else a cycle member would re-enter the+ // chain.+ let composes = filters+ .eligibility+ .get(&referrer)+ .is_some_and(|elig| elig.writes_cf_stage2());+ if !composes {+ continue;}- };- if diff.requires_write() {- writes.push((- diff.stage2_key,- Stage2State {- in_cohort: diff.new_bit,- last_evaluated_at_ms: evaluated_at_ms,- },- ));- }- if !diff.flipped() {- continue;- }- let status = diff.status();- changes.push(CohortMembershipChange {- team_id: message.change.team_id,- cohort_id: referrer.0,- person_id: message.change.person_id.clone(),- last_updated: last_updated.to_string(),- status,- origin: message.change.origin,- run_id: None,- });- // The external flip above is unconditional; only the onward hop is depth/cycle bounded.- match should_emit(- message,- referrer,- status,- last_updated,- merge.cascade.depth_cap,- ) {- CascadeDecision::Emit { outgoing: hop } => outgoing.push(hop),- CascadeDecision::Drop {- reason: DropReason::DepthExceeded,- } => {- counter!(CASCADE_DEPTH_EXCEEDED_TOTAL).increment(1);- debug!(- originating_cohort_id = message.originating_cohort_id,- "cascade depth cap reached",- );+ let Some(tree) = filters.cohorts.get(&referrer) else {+ continue;+ };+ let diff = match recompute_and_diff(+ partition_id,+ person_id,+ tree,+ filters,+ handle,+ ReadLane::Event,+ )+ .await+ {+ Ok(diff) => diff,+ Err(error) => {+ warn!(+ partition_id,+ offset,+ team_id = message.change.team_id,+ referrer = referrer.0,+ error = %error,+ "cascade recompute read failed; holding the cascade offset for redelivery",+ );+ hold(&merge.cascade_tracker, partition_id, offset);+ return;+ }+ };+ if diff.requires_write() {+ reevaluated.writes.push((+ diff.stage2_key,+ Stage2State {+ in_cohort: diff.new_bit,+ last_evaluated_at_ms: self.evaluated_at_ms,+ },+ ));}- CascadeDecision::Drop {- reason: DropReason::CycleDetectedRuntime,- } => {- counter!(CASCADE_CYCLE_DETECTED_RUNTIME_TOTAL).increment(1);- debug!(- originating_cohort_id = message.originating_cohort_id,- cycle_cohort_id = referrer.0,- "cascade cycle detected at runtime",- );+ if !diff.flipped() {+ continue;+ }+ let status = diff.status();+ reevaluated.changes.push(CohortMembershipChange {+ team_id: message.change.team_id,+ cohort_id: referrer.0,+ person_id: message.change.person_id.clone(),+ last_updated: self.last_updated.clone(),+ status,+ origin: message.change.origin,+ run_id: None,+ });+ // The external flip above is unconditional; only the onward hop is depth/cycle bounded.+ match should_emit(+ message,+ referrer,+ status,+ &self.last_updated,+ merge.cascade.depth_cap,+ ) {+ CascadeDecision::Emit { outgoing: hop } => reevaluated.outgoing.push(hop),+ CascadeDecision::Drop {+ reason: DropReason::DepthExceeded,+ } => {+ counter!(CASCADE_DEPTH_EXCEEDED_TOTAL).increment(1);+ debug!(+ originating_cohort_id = message.originating_cohort_id,+ "cascade depth cap reached",+ );+ }+ CascadeDecision::Drop {+ reason: DropReason::CycleDetectedRuntime,+ } => {+ counter!(CASCADE_CYCLE_DETECTED_RUNTIME_TOTAL).increment(1);+ debug!(+ originating_cohort_id = message.originating_cohort_id,+ cycle_cohort_id = referrer.0,+ "cascade cycle detected at runtime",+ );+ }}}+ if !reevaluated.writes.is_empty() {+ self.touched.insert(person_id);+ self.writes += reevaluated.writes.len();+ self.messages.push(reevaluated);+ }}- if changes.is_empty() && writes.is_empty() {- mark_processed(&merge.cascade_tracker, partition_id, offset);- return;- }-- if !changes.is_empty() {- // Produce-before-state: both legs must ack before flipped bits commit (see the module doc).- let membership_errors = produce_membership(sink, changes).await;- if membership_errors > 0 {- warn!(- partition_id,- errors = membership_errors,- "cascade membership produce failed; holding the cascade offset for redelivery",+ /// Produce the group's flips and onward hops, then commit the bits of every message whose records+ /// all acked. A message with a failed record holds its own offset.+ async fn settle(+ self,+ partition_id: u16,+ handle: &StoreHandle,+ sink: &Arc<dyn MembershipSink>,+ merge: &MergeWorkerDeps,+ ) {+ let Some(first) = self.messages.first().map(|message| message.offset) else {+ return;+ };+ histogram!(CASCADE_BATCH_MESSAGES).record(self.messages.len() as f64);+ // One produce per message and topic, so each message learns whether its own records acked.+ // Every produce is in flight at once, so the group still costs one round trip.+ let produced = join_all(self.messages.into_iter().map(|message| async move {+ // Produce-before-state: both legs must ack before the message's bits commit (see the+ // module doc).+ let (membership_errors, cascade_errors) = tokio::join!(+ produce_membership(sink, message.changes),+ produce_cascades(merge, message.outgoing),);- hold(&merge.cascade_tracker, partition_id, offset);+ (+ message.offset,+ message.writes,+ membership_errors,+ cascade_errors,+ )+ }))+ .await;+ let mut staged = StagedBatch::default();+ for (offset, writes, membership_errors, cascade_errors) in produced {+ if membership_errors + cascade_errors > 0 {+ warn!(+ partition_id,+ offset,+ membership_errors,+ cascade_errors,+ "cascade produce failed; holding the cascade offset for redelivery",+ );+ hold(&merge.cascade_tracker, partition_id, offset);+ continue;+ }+ for (key, state) in &writes {+ staged.put_stage2(key, &state.encode());+ }+ }+ if staged.is_empty() {return;}- let cascade_errors = produce_cascades(merge, outgoing).await;- if cascade_errors > 0 {+ if let Err(error) = handle.commit(staged).await {warn!(partition_id,- errors = cascade_errors,- "cascade onward produce failed; holding the cascade offset for redelivery",+ error = %error,+ "cascade cf_stage2 write failed; holding the cascade group for redelivery",);- hold(&merge.cascade_tracker, partition_id, offset);- return;+ hold(&merge.cascade_tracker, partition_id, first);}}-- let mut staged = StagedBatch::default();- for (key, state) in &writes {- staged.put_stage2(key, &state.encode());- }- if let Err(error) = handle.commit(staged).await {- warn!(- partition_id,- error = %error,- "cascade cf_stage2 write failed; holding the cascade offset for redelivery",- );- hold(&merge.cascade_tracker, partition_id, offset);- return;- }-- mark_processed(&merge.cascade_tracker, partition_id, offset);}/// Advance the cascade tracker past `offset`. A mark beyond the dispatch ceiling is capped and counted.@@ -302,7 +413,8 @@ mod tests {(dir, store)}- /// Wraps the store so `handle_cascade` exercises the same blocking-pool transport as production.+ /// Wraps the store so `handle_cascades` exercises the same blocking-pool transport as+ /// production.fn handle(store: &CohortStore) -> StoreHandle {StoreHandle::new(store.clone(),@@ -453,12 +565,37 @@ mod tests {person_seed: crate::workers::PersonSeedDeps::default(),seed_budget: crate::workers::seed_run::RunBudget::default(),};- // The cascade was dispatched this tenure, so its ceiling is raised — mirrors the dispatcher.- deps.cascade_tracker- .mark_dispatched(PARTITION as i32, OFFSET + 1);(sink, deps)}+ /// Handle `messages` as one worker sub-batch at offsets `OFFSET`, `OFFSET + 1`, and so on.+ async fn run(+ store: &CohortStore,+ catalog: &CatalogHandle,+ sink: &Arc<dyn MembershipSink>,+ deps: &MergeWorkerDeps,+ messages: Vec<CascadeMessage>,+ ) {+ let batch: Vec<(Box<CascadeMessage>, i64)> = (OFFSET..)+ .zip(messages)+ .map(|(offset, message)| (Box::new(message), offset))+ .collect();+ // The cascades were dispatched this tenure, so their ceiling is raised, as the dispatcher+ // does.+ deps.cascade_tracker+ .mark_dispatched(PARTITION as i32, OFFSET + batch.len() as i64);+ handle_cascades(+ PARTITION,+ &handle(store),+ catalog,+ sink,+ deps,+ &mut LastUpdatedClock::default(),+ batch,+ )+ .await;+ }+fn incoming(flipped: i32, who: Uuid, depth: u8, chain: Vec<i32>) -> CascadeMessage {CascadeMessage {change: CohortMembershipChange {@@ -508,17 +645,7 @@ mod tests {flipped.run_id = Some(RunId(Uuid::nil()));let msg = first_cascade(flipped, 99);- handle_cascade(- PARTITION,- &handle(&store),- &catalog,- &sink,- &deps,- TS,- &msg,- OFFSET,- )- .await;+ run(&store, &catalog, &sink, &deps, vec![msg]).await;let changes = membership.changes();assert_eq!(changes.len(), 1, "B's external flip");@@ -567,17 +694,7 @@ mod tests {let (sink, deps) = deps(&membership, &cascade, true, 1000);let msg = first_cascade(incoming(2, alice, 1, vec![2]).change, 99);- handle_cascade(- PARTITION,- &handle(&store),- &catalog,- &sink,- &deps,- TS,- &msg,- OFFSET,- )- .await;+ run(&store, &catalog, &sink, &deps, vec![msg]).await;assert!(membership.changes().is_empty(), "no flip, no membership");assert!(cascade.messages().is_empty(), "no flip, no onward cascade");@@ -621,17 +738,7 @@ mod tests {let (sink, deps) = deps(&membership, &cascade, true, 1000);let msg = first_cascade(incoming(2, alice, 1, vec![2]).change, 99);- handle_cascade(- PARTITION,- &handle(&store),- &catalog,- &sink,- &deps,- TS,- &msg,- OFFSET,- )- .await;+ run(&store, &catalog, &sink, &deps, vec![msg]).await;let bytes = store.get_stage2(&key).unwrap().unwrap();let (state, ownership) = Stage2State::decode_with_ownership(&bytes).unwrap();@@ -661,17 +768,7 @@ mod tests {let (sink, deps) = deps(&membership, &cascade, true, 1000);let msg = first_cascade(incoming(2, alice, 1, vec![2]).change, 99);- handle_cascade(- PARTITION,- &handle(&store),- &catalog,- &sink,- &deps,- TS,- &msg,- OFFSET,- )- .await;+ run(&store, &catalog, &sink, &deps, vec![msg]).await;assert!(membership.changes().is_empty());assert!(cascade.messages().is_empty());@@ -697,17 +794,7 @@ mod tests {let (sink, deps) = deps(&membership, &cascade, false, 1000); // gate OFFlet msg = first_cascade(incoming(2, alice, 1, vec![2]).change, 99);- handle_cascade(- PARTITION,- &handle(&store),- &catalog,- &sink,- &deps,- TS,- &msg,- OFFSET,- )- .await;+ run(&store, &catalog, &sink, &deps, vec![msg]).await;assert!(membership.changes().is_empty(), "gate off: no re-eval");assert!(cascade.messages().is_empty());@@ -735,17 +822,7 @@ mod tests {// Incoming depth == cap (8): the external flip still emits, the onward cascade drops.let msg = incoming(2, alice, 8, vec![2, 90, 91, 92, 93, 94, 95, 96]);- handle_cascade(- PARTITION,- &handle(&store),- &catalog,- &sink,- &deps,- TS,- &msg,- OFFSET,- )- .await;+ run(&store, &catalog, &sink, &deps, vec![msg]).await;assert_eq!(membership.changes().len(),@@ -784,17 +861,7 @@ mod tests {// B (cohort 1) is already in the chain: the onward hop is a runtime cycle.let msg = incoming(2, alice, 2, vec![2, 1]);- handle_cascade(- PARTITION,- &handle(&store),- &catalog,- &sink,- &deps,- TS,- &msg,- OFFSET,- )- .await;+ run(&store, &catalog, &sink, &deps, vec![msg]).await;assert_eq!(membership.changes().len(),@@ -831,17 +898,7 @@ mod tests {let (sink, deps) = deps(&membership, &cascade, true, 2); // cap 2 of 4 referrerslet msg = first_cascade(incoming(100, alice, 1, vec![100]).change, 99);- handle_cascade(- PARTITION,- &handle(&store),- &catalog,- &sink,- &deps,- TS,- &msg,- OFFSET,- )- .await;+ run(&store, &catalog, &sink, &deps, vec![msg]).await;let cohorts: Vec<i32> = membership.changes().iter().map(|c| c.cohort_id).collect();assert_eq!(@@ -875,30 +932,10 @@ mod tests {let (sink, deps) = deps(&membership, &cascade, true, 1000);let msg = first_cascade(incoming(2, alice, 1, vec![2]).change, 99);- handle_cascade(- PARTITION,- &handle(&store),- &catalog,- &sink,- &deps,- TS,- &msg,- OFFSET,- )- .await;+ run(&store, &catalog, &sink, &deps, vec![msg.clone()]).await;assert_eq!(membership.changes().len(), 1, "first pass emits");- handle_cascade(- PARTITION,- &handle(&store),- &catalog,- &sink,- &deps,- TS,- &msg,- OFFSET,- )- .await;+ run(&store, &catalog, &sink, &deps, vec![msg]).await;assert_eq!(membership.changes().len(),1,@@ -906,8 +943,25 @@ mod tests {);}+ /// Alice, Bob and Carol each enter cohort 2, which cohort 1 references.+ fn three_flips(+ store: &CohortStore,+ catalog: &CatalogHandle,+ ) -> (Vec<Uuid>, Vec<CascadeMessage>) {+ let people = vec![person(1), person(2), person(3)];+ let messages = people+ .iter()+ .map(|&who| {+ write_behavioral(store, behavioral_lsk(catalog), who, behavioral_match());+ write_person_record(store, who, &[PERSON_HASH]);+ first_cascade(incoming(2, who, 1, vec![2]).change, 99)+ })+ .collect();+ (people, messages)+ }+#[tokio::test]- async fn membership_produce_failure_holds_without_writing_state() {+ async fn a_batch_of_flips_commits_every_bit() {let (_dir, store) = temp_store();let catalog = catalog(vec![@@ -916,47 +970,41 @@ mod tests {],true,);- let alice = person(1);- write_behavioral(&store, behavioral_lsk(&catalog), alice, behavioral_match());- write_person_record(&store, alice, &[PERSON_HASH]);+ let (people, messages) = three_flips(&store, &catalog);- let membership = CaptureSink::failing_first(1);+ let membership = CaptureSink::new();let cascade = CaptureCascadeSink::new();let (sink, deps) = deps(&membership, &cascade, true, 1000);- let msg = first_cascade(incoming(2, alice, 1, vec![2]).change, 99);-- handle_cascade(- PARTITION,- &handle(&store),- &catalog,- &sink,- &deps,- TS,- &msg,- OFFSET,- )- .await;+ run(&store, &catalog, &sink, &deps, messages).await;+ let changes = membership.changes();+ assert_eq!(changes.len(), 3);assert!(- membership.changes().is_empty(),- "the failed produce recorded nothing"- );- assert!(- cascade.messages().is_empty(),- "no onward cascade after a membership failure"+ changes+ .iter()+ .all(|change| change.last_updated == changes[0].last_updated),+ "the three flips settle as one group, which stamps them once",);- assert_eq!(read_stage2(&store, 1, alice), None, "no bit on a held flip");- assert_eq!(committed(&deps), None, "the offset is held for redelivery");+ assert_eq!(cascade.messages().len(), 3);+ for who in people {+ assert_eq!(+ read_stage2(&store, 1, who),+ Some(true),+ "{who}'s bit committed"+ );+ }+ assert_eq!(committed(&deps), Some(OFFSET + 3), "the last offset marked");}- /// The membership leg acks but the onward cascade fails, so the bit is not committed and the- /// offset holds.+ /// C references B, which references A. The second message is B's flip coming back for the same+ /// person, so recomposing C must read the bit the first message writes for B.#[tokio::test]- async fn cascade_produce_failure_holds_without_writing_state() {+ async fn a_second_cascade_for_a_touched_person_reads_the_committed_bit() {let (_dir, store) = temp_store();let catalog = catalog(vec![- (2, vec![behavioral_leaf()]),+ (3, vec![behavioral_leaf()]),+ (2, vec![person_leaf(), cohort_ref(3)]),(1, vec![person_leaf(), cohort_ref(2)]),],true,@@ -966,33 +1014,76 @@ mod tests {write_person_record(&store, alice, &[PERSON_HASH]);let membership = CaptureSink::new();- let cascade = CaptureCascadeSink::failing_always();+ let cascade = CaptureCascadeSink::new();let (sink, deps) = deps(&membership, &cascade, true, 1000);- let msg = first_cascade(incoming(2, alice, 1, vec![2]).change, 99);-- handle_cascade(- PARTITION,- &handle(&store),+ run(+ &store,&catalog,&sink,&deps,- TS,- &msg,- OFFSET,+ vec![+ first_cascade(incoming(3, alice, 1, vec![3]).change, 99),+ incoming(2, alice, 2, vec![3, 2]),+ ],).await;+ let cohorts: Vec<i32> = membership.changes().iter().map(|c| c.cohort_id).collect();assert_eq!(- membership.changes().len(),- 1,- "the membership leg already acked"- );- assert!(cascade.messages().is_empty(), "the onward cascade failed");- assert_eq!(- read_stage2(&store, 1, alice),- None,- "produce-before-state: the bit is not committed on a held flip, so replay recovers it",+ cohorts,+ vec![2, 1],+ "B enters, then C enters on B's written bit");- assert_eq!(committed(&deps), None, "the offset is held for redelivery");+ assert_eq!(read_stage2(&store, 2, alice), Some(true));+ assert_eq!(read_stage2(&store, 1, alice), Some(true));+ assert_eq!(committed(&deps), Some(OFFSET + 2));+ }++ /// Alice's records fail and Bob's and Carol's ack in the same round trip.+ #[tokio::test]+ async fn a_failed_produce_leg_holds_only_its_own_message() {+ for (failing_leg, membership, cascade) in [+ (+ "membership",+ CaptureSink::failing_first(1),+ CaptureCascadeSink::new(),+ ),+ (+ "cascade",+ CaptureSink::new(),+ CaptureCascadeSink::failing_first(1),+ ),+ ] {+ let (_dir, store) = temp_store();+ let catalog = catalog(+ vec![+ (2, vec![behavioral_leaf()]),+ (1, vec![person_leaf(), cohort_ref(2)]),+ ],+ true,+ );+ let (people, messages) = three_flips(&store, &catalog);+ let (sink, deps) = deps(&membership, &cascade, true, 1000);++ run(&store, &catalog, &sink, &deps, messages).await;++ assert_eq!(+ read_stage2(&store, 1, people[0]),+ None,+ "{failing_leg} leg failed: produce-before-state leaves alice's bit for the replay",+ );+ for who in &people[1..] {+ assert_eq!(+ read_stage2(&store, 1, *who),+ Some(true),+ "{failing_leg} leg failed for alice only: {who}'s acked flip keeps its bit",+ );+ }+ assert_eq!(+ committed(&deps),+ Some(OFFSET),+ "{failing_leg} leg failed: the floor pins alice's offset under the run's mark",+ );+ }}}diff --git a/rust/cohort-stream-processor/src/workers/worker.rs b/rust/cohort-stream-processor/src/workers/worker.rsindex 783ca05201aa..af9fd149ccfb 100644--- a/rust/cohort-stream-processor/src/workers/worker.rs+++ b/rust/cohort-stream-processor/src/workers/worker.rs@@ -1,12 +1,13 @@//! Per-partition Stage 1 worker.//!//! [`Stage1Worker::spawn`] drains one partition's two lanes on a dedicated tokio task. Per live-//! sub-batch it produces membership changes and straggler re-keys, awaits all acks, then marks the-//! offset processed. A produce failure holds the offset; a store error is logged and the event is-//! skipped. Between live sub-batches it applies at most one seed run: a quantum of seeds leaves-//! the lane only once the previous quantum has fully applied, and its runs go one per round, so-//! live latency is at most one run and an admitted seed backlog never queues in front of live-//! traffic.+//! sub-batch it hands its membership changes to a bounded `ProduceWindow`, produces straggler+//! re-keys and awaits their acks, then marks the offset processed. Live output is at most once, so+//! the mark does not wait for the window's acks. A failed re-key skips the mark; a store error is+//! logged and the event is skipped. Between live sub-batches it applies at most one seed run: a+//! quantum of seeds leaves the lane only once the previous quantum has fully applied, and its runs+//! go one per round, so live latency is at most one run and an admitted seed backlog never queues+//! in front of live traffic.//!//! A `Sweep` message records an eviction request rather than running one. The pass then advances one//! bounded batch per round, alternating with live batches and running ahead of seeds@@ -19,9 +20,10 @@ use std::sync::Arc;use std::time::{Duration, Instant};use cohort_core::seed::CoverageStartMs;+use futures::FutureExt;use metrics::{counter, gauge, histogram};use tokio::task::JoinHandle;-use tracing::{info, warn};+use tracing::{error, info, warn};use uuid::Uuid;use crate::cascade::{first_cascade, CascadeMessage};@@ -36,9 +38,9 @@ use crate::observability::metrics::{CASCADE_PRODUCE_ERRORS_TOTAL, COHORT_STREAM_OFFSET_AHEAD_OF_DISPATCH,EVICTION_QUEUE_REBUILT_KEYS_TOTAL, MERGE_REDIRECT_HOP_CAPPED_TOTAL,MERGE_REKEY_PRODUCE_FAILURE_TOTAL, OUTPUT_MEMBERSHIP_CHANGES_EMITTED, OUTPUT_PRODUCE_ERRORS,- SLICES_BEGUN_TOTAL, SLICE_COVERED_SINCE_SECONDS, STAGE1_EVENTS_PROCESSED,- STAGE1_EVENTS_SKIPPED, STAGE1_EVENT_PROCESS_DURATION, STAGE1_STATE_DECODE_ERROR,- STAGE1_TRANSITIONS,+ OUTPUT_WINDOW_WAITS_TOTAL, SLICES_BEGUN_TOTAL, SLICE_COVERED_SINCE_SECONDS,+ STAGE1_EVENTS_PROCESSED, STAGE1_EVENTS_SKIPPED, STAGE1_EVENT_PROCESS_DURATION,+ STAGE1_STATE_DECODE_ERROR, STAGE1_TRANSITIONS,};use crate::partitions::offset_tracker::{MarkOutcome, OffsetTracker};use crate::partitions::router::WorkerInbox;@@ -52,7 +54,7 @@ use crate::stage1::state::{StateVariant, StatefulRecord};use crate::stage1::transition::{LeafTransition, TransitionKind};use crate::store::{BehavioralKey, ReadLane, SliceCoverage, SliceTenure, StoreHandle};use crate::sweep::EvictionQueue;-use crate::workers::cascade_path::handle_cascade;+use crate::workers::cascade_path::handle_cascades;use crate::workers::event_path::{process_event_offloaded, schedule_deadline, EventNameGating, SkipReason,};@@ -72,6 +74,10 @@ const REBUILD_SCAN_PAGE: usize = 10_000;/// wall-clock interval adapts to per-event cost across catalogs of different sizes.const WORKER_YIELD_INTERVAL: Duration = Duration::from_millis(5);+/// Unacked live output batches a worker allows before it waits. Sized for a few broker round trips+/// of a partition's peak live output; `output_window_waits_total` says whether it binds.+const PRODUCE_WINDOW: usize = 4;+pub struct Stage1Worker {partition_id: u16,handle: JoinHandle<()>,@@ -172,6 +178,117 @@ struct PendingRuns {marks: BatchMarks,}+/// Live output batches handed to the producer whose acks the worker has not collected yet.+///+/// Live output is at most once and its state is already committed, so the offset mark does not+/// wait for the ack. The bound is for a broker stall: with [`PRODUCE_WINDOW`] batches unacked the+/// worker waits, so the stall shows as lag on this partition instead of as output held in memory+/// until the message timeout drops it.+///+/// [`submit`](Self::submit) enqueues a batch's records before it returns, and only the acks are+/// collected on a spawned task. So batches are enqueued in submit order, ahead of anything the+/// worker produces next. A producer retry can still reorder batches that are in flight together.+/// With version writes on, the membership consumer's version check puts them back in order. With+/// them off, an older change can win until the next change for that cohort and person.+struct ProduceWindow {+ partition_id: u16,+ sink: Arc<dyn MembershipSink>,+ merge: Arc<MergeWorkerDeps>,+ in_flight: VecDeque<JoinHandle<()>>,+}++impl ProduceWindow {+ fn new(partition_id: u16, sink: Arc<dyn MembershipSink>, merge: Arc<MergeWorkerDeps>) -> Self {+ Self {+ partition_id,+ sink,+ merge,+ in_flight: VecDeque::new(),+ }+ }++ /// Hand the buffered changes, if any, to the producer. Waits only while the window is full.+ ///+ /// The worker submits before each merge, transfer, cascade, sweep or reconcile message and at+ /// the end of each live sub-batch, so batches enter the window in state-commit order.+ async fn submit(&mut self, buffer: &mut OutputBuffer, max_offset: Option<i64>) {+ let Some(changes) = buffer.take() else {+ return;+ };+ while self.in_flight.front().is_some_and(JoinHandle::is_finished) {+ self.settle_oldest().await;+ }+ if self.in_flight.len() >= PRODUCE_WINDOW {+ counter!(OUTPUT_WINDOW_WAITS_TOTAL).increment(1);+ self.settle_oldest().await;+ }+ let mut produce = Box::pin(produce_live(+ self.partition_id,+ Arc::clone(&self.sink),+ Arc::clone(&self.merge),+ changes,+ max_offset.unwrap_or(0),+ ));+ // The first poll enqueues every record, because the producer's enqueue is synchronous and+ // only the acks are awaited. A spawned task first runs when the worker yields, which can be+ // after the worker's next inline produce. Enqueueing here keeps Kafka order equal to+ // state-commit order except across a producer retry, and Kafka order is what the membership+ // consumer applies when its version writes are off.+ if produce.as_mut().now_or_never().is_none() {+ self.in_flight.push_back(tokio::spawn(produce));+ }+ }++ /// Collect every outstanding produce, so a stopped worker leaves no output of its tenure in+ /// flight.+ async fn drain(mut self) {+ while !self.in_flight.is_empty() {+ self.settle_oldest().await;+ }+ }++ async fn settle_oldest(&mut self) {+ let Some(produce) = self.in_flight.pop_front() else {+ return;+ };+ if let Err(error) = produce.await {+ error!(+ partition_id = self.partition_id,+ error = %error,+ "live produce task failed; its output is lost",+ );+ }+ }+}++/// Produce one live batch's membership changes and their first-hop cascades. A failed leg is+/// counted and logged but never retried, because the batch's state is already committed and a+/// replay finds nothing left to emit.+async fn produce_live(+ partition_id: u16,+ sink: Arc<dyn MembershipSink>,+ merge: Arc<MergeWorkerDeps>,+ changes: Vec<CohortMembershipChange>,+ source_offset: i64,+) {+ let produced = changes.len();+ // Built from a borrow before `changes` moves: the cascade payload embeds the change.+ let cascades = first_cascades(&merge, &changes, source_offset);+ let (membership_errors, cascade_errors) = tokio::join!(+ produce_membership(&sink, changes),+ produce_cascades(&merge, cascades),+ );+ if membership_errors + cascade_errors > 0 {+ warn!(+ partition_id,+ changes = produced,+ membership_errors,+ cascade_errors,+ "live produce failed; the changes are lost, and only a reconcile re-emits them",+ );+ }+}+#[allow(clippy::too_many_arguments)]async fn run_worker(partition_id: u16,@@ -214,6 +331,7 @@ async fn run_worker(// Persists across batches so a stream of buffered batches still yields on the wall-clock interval.let mut last_yield = Instant::now();let mut last_updated_clock = LastUpdatedClock::default();+ let mut window = ProduceWindow::new(partition_id, Arc::clone(&sink), Arc::clone(&merge));// Two locals, not `inbox.live`/`inbox.seeds`: both branch futures live at once inside the// macro, so they must borrow disjoint places.let WorkerInbox {@@ -323,9 +441,9 @@ async fn run_worker(continue;}};- // Claimed before the batch runs, not after: every exit path below (held offset, produce- // failure, clean mark) must still yield the next turn, or a partition that keeps failing to- // produce would starve eviction.+ // Claimed before the batch runs, not after: every exit path below (re-key failure, clean+ // mark) must still yield the next turn, or a partition that keeps failing to re-key would+ // starve eviction.sweep.live_batch_done();let mut buffer = OutputBuffer::new();let mut re_keys: Vec<CohortStreamEvent> = Vec::new();@@ -333,10 +451,9 @@ async fn run_worker(// Observed into the live watermarks only post-mark, so a held batch never advances the// seed fence.let mut max_broker_ts: Option<i64> = None;- // Set when a pre-arm flush fails: holds the whole batch's offset so Kafka replays it.- let mut held = false;- for message in batch {+ let mut messages = batch.into_iter().peekable();+ while let Some(message) = messages.next() {let last_updated = last_updated_clock.next();match message {ShuffleMessage::Event {@@ -364,36 +481,13 @@ async fn run_worker(re_keys.extend(effects.re_keys);}ShuffleMessage::Sweep { due_before_ms } => {- // The eviction runs on a later turn, but the buffer still flushes here, so this- // batch's own `Entered` output precedes the request that may retract it. A- // later live batch whose end-of-batch produce fails is not covered: its output- // is dropped for replay while the sweep keeps its turn, so a `Left` can reach- // the wire first. That converges, because the replayed `Entered` stamps newer- // than the sweep's `Left`, and holding the sweep behind a failing lane instead- // would starve eviction (see `live_batch_done`).- if flush_event_changes_before_inline(- &sink,- &mut buffer,- partition_id,- &mut held,- )- .await- {- break;- }+ // The eviction runs on a later turn, but the buffer is still submitted here,+ // ahead of the request that may retract this batch's own `Entered` output.+ window.submit(&mut buffer, max_offset).await;sweep.request(due_before_ms, &queue);}ShuffleMessage::Merge { event, offset } => {- if flush_event_changes_before_inline(- &sink,- &mut buffer,- partition_id,- &mut held,- )- .await- {- break;- }+ window.submit(&mut buffer, max_offset).await;handle_merge(partition_id,&handle,@@ -408,16 +502,7 @@ async fn run_worker(.await;}ShuffleMessage::Transfer { transfer, offset } => {- if flush_event_changes_before_inline(- &sink,- &mut buffer,- partition_id,- &mut held,- )- .await- {- break;- }+ window.submit(&mut buffer, max_offset).await;handle_apply(partition_id,&handle,@@ -432,25 +517,22 @@ async fn run_worker(.await;}ShuffleMessage::Cascade { message, offset } => {- if flush_event_changes_before_inline(- &sink,- &mut buffer,- partition_id,- &mut held,- )- .await+ // A sub-batch carries one topic, so the run is normally the whole sub-batch.+ let mut run = vec![(message, offset)];+ while let Some(ShuffleMessage::Cascade { message, offset }) =+ messages.next_if(|next| matches!(next, ShuffleMessage::Cascade { .. })){- break;+ run.push((message, offset));}- handle_cascade(+ window.submit(&mut buffer, max_offset).await;+ handle_cascades(partition_id,&handle,&catalog,&sink,&merge,- &last_updated,- &message,- offset,+ &mut last_updated_clock,+ run,).await;}@@ -499,16 +581,7 @@ async fn run_worker(}}ShuffleMessage::ReconcileDrain => {- if flush_event_changes_before_inline(- &sink,- &mut buffer,- partition_id,- &mut held,- )- .await- {- break;- }+ window.submit(&mut buffer, max_offset).await;handle_reconcile_drain(partition_id,&handle,@@ -527,32 +600,7 @@ async fn run_worker(}}- if held {- continue;- }-- if !buffer.is_empty() {- let changes = buffer.take();- // Build cascades from a borrow before `changes` is moved into produce; gate-off allocates nothing.- let cascades = first_cascades(&merge, &changes, max_offset.unwrap_or(0));- let errors = produce_membership(&sink, changes).await;- if errors > 0 {- warn!(- partition_id,- errors, "produce to the membership topic failed; holding offset for replay",- );- continue;- }- let cascade_errors = produce_cascades(&merge, cascades).await;- if cascade_errors > 0 {- warn!(- partition_id,- errors = cascade_errors,- "produce to cohort_cascade_events failed; holding offset for replay",- );- continue;- }- }+ window.submit(&mut buffer, max_offset).await;if !re_keys.is_empty() {let produced = re_keys.len() as u64;@@ -591,51 +639,14 @@ async fn run_worker(}}+ window.drain().await;info!(partition_id, "stage 1 worker stopped");}-/// Flush buffered event-path changes so they land before an inline (merge/transfer) produce,-/// preserving produce order == state-commit order. No-op when the buffer is empty.-/// Returns the failed-ack count (`0` = fully acked / empty).-async fn flush_membership_buffer(- sink: &Arc<dyn MembershipSink>,- buffer: &mut OutputBuffer,-) -> usize {- if buffer.is_empty() {- return 0;- }- produce_membership(sink, buffer.take()).await-}--/// Flush buffered event-path changes ahead of an inline arm (merge, transfer, cascade, or reconcile-/// drain). Returns `true` when the flush failed: sets `held` so the caller `break`s and-/// `mark_processed` is skipped, causing Kafka to replay. Returns `false` (empty or all acked) to-/// run the arm normally.-///-/// The `Sweep` arm produces nothing inline, but flushes here so the batch's own `Entered` output-/// precedes the request that may retract it. Its call site says what that does not cover.-async fn flush_event_changes_before_inline(- sink: &Arc<dyn MembershipSink>,- buffer: &mut OutputBuffer,- partition_id: u16,- held: &mut bool,-) -> bool {- let errors = flush_membership_buffer(sink, buffer).await;- if errors > 0 {- *held = true;- warn!(- partition_id,- errors, "produce to the membership topic failed; holding offset for replay",- );- return true;- }- false-}-/// Split one quantum into the runs it will apply, one per worker round.///-/// No live-buffer flush is needed first: the live `OutputBuffer` is produced (or its batch held) at-/// the end of every live sub-batch, so it is always empty when a seed run starts.+/// No live-buffer submit is needed first: the live `OutputBuffer` is submitted at the end of every+/// live sub-batch, so it is always empty when a seed run starts.fn group_seed_turn(partition_id: u16,catalog: &CatalogHandle,@@ -751,8 +762,8 @@ pub(crate) fn first_cascades(/// Produce cascade messages and await one ack per message. Returns the failed or/// missing/extra-ack count (`0` when empty or fully acked). The caller owns the recovery posture:-/// the event path holds the offset; the sweep/merge paths drop (at-most-once). Shared by the-/// first-hop leg and onward-hop consumer legs.+/// the live, sweep and merge paths drop (at-most-once); the cascade consumer and seed apply hold+/// the offset. Shared by the first-hop leg and onward-hop consumer legs.#[hotpath::measure]pub(crate) async fn produce_cascades(merge: &MergeWorkerDeps,@@ -1525,7 +1536,7 @@ mod tombstone_redirect_tests {}#[tokio::test]- async fn cascade_produce_failure_holds_the_events_batch() {+ async fn cascade_produce_failure_does_not_hold_the_events_batch() {let (_dir, store) = temp_store();let membership = CaptureSink::new();let cascade = CaptureCascadeSink::failing_always();@@ -1540,13 +1551,210 @@ mod tombstone_redirect_tests {assert_eq!(membership.changes().len(),1,- "membership is the first leg and acked before the cascade leg",+ "the membership leg acks on its own");assert!(cascade.messages().is_empty(), "the cascade produce failed");assert_eq!(tracker.committable_offsets().get(&(partition_id as i32)),- None,- "a failed cascade produce holds the events offset for replay",+ Some(&1),+ "live output is at most once, so a failed leg does not hold the events offset",+ );+ }++ /// A membership sink that records changes when it is called, as a broker enqueue does, and+ /// holds each ack until a [`GatedSink::release`], oldest first.+ #[derive(Clone)]+ struct GatedSink {+ enqueued: CaptureSink,+ acked: Arc<std::sync::atomic::AtomicUsize>,+ gate: Arc<tokio::sync::Semaphore>,+ }++ impl GatedSink {+ fn new() -> Self {+ Self {+ enqueued: CaptureSink::new(),+ acked: Arc::default(),+ gate: Arc::new(tokio::sync::Semaphore::new(0)),+ }+ }++ fn release(&self, produces: usize) {+ self.gate.add_permits(produces);+ }++ fn enqueued_cohorts(&self) -> Vec<i32> {+ self.enqueued+ .changes()+ .iter()+ .map(|change| change.cohort_id)+ .collect()+ }++ fn acked(&self) -> usize {+ self.acked.load(std::sync::atomic::Ordering::SeqCst)+ }+ }++ #[async_trait::async_trait]+ impl MembershipSink for GatedSink {+ async fn produce(+ &self,+ changes: Vec<CohortMembershipChange>,+ ) -> Vec<Result<(), common_kafka::kafka_producer::KafkaProduceError>> {+ let acks = self.enqueued.produce(changes).await;+ self.gate+ .acquire()+ .await+ .expect("the gate is never closed")+ .forget();+ self.acked.fetch_add(1, std::sync::atomic::Ordering::SeqCst);+ acks+ }+ }++ fn entered(cohort_id: i32) -> CohortMembershipChange {+ CohortMembershipChange {+ team_id: TEAM,+ cohort_id,+ person_id: Uuid::from_u128(1).to_string(),+ last_updated: "2026-05-26 12:34:56.789000".to_string(),+ status: MembershipStatus::Entered,+ origin: None,+ run_id: None,+ }+ }++ #[tokio::test]+ async fn produce_window_enqueues_in_submit_order_and_waits_at_capacity() {+ let sink = GatedSink::new();+ let mut window = ProduceWindow::new(+ 0,+ Arc::new(sink.clone()),+ merge_deps_with(CaptureStreamEventSink::new()),+ );+ let mut buffer = OutputBuffer::new();+ for cohort_id in 0..PRODUCE_WINDOW as i32 {+ buffer.push(entered(cohort_id));+ assert!(+ window.submit(&mut buffer, None).now_or_never().is_some(),+ "a window with room hands batch {cohort_id} over without waiting",+ );+ }+ assert_eq!(+ sink.enqueued_cohorts(),+ (0..PRODUCE_WINDOW as i32).collect::<Vec<_>>(),+ "each submit enqueues its batch before it returns",+ );++ buffer.push(entered(PRODUCE_WINDOW as i32));+ {+ let mut full = std::pin::pin!(window.submit(&mut buffer, None));+ assert!(+ futures::poll!(&mut full).is_pending(),+ "a full window waits for its oldest ack",+ );+ assert_eq!(+ sink.enqueued_cohorts().len(),+ PRODUCE_WINDOW,+ "the waiting submit enqueues its batch only after the oldest acks",+ );+ sink.release(1);+ full.await;+ }++ sink.release(PRODUCE_WINDOW);+ window.drain().await;+ assert_eq!(+ sink.enqueued_cohorts(),+ (0..=PRODUCE_WINDOW as i32).collect::<Vec<_>>()+ );+ assert_eq!(sink.acked(), PRODUCE_WINDOW + 1, "drain collects every ack");+ }++ #[tokio::test]+ async fn the_offset_marks_while_the_membership_produce_is_unacked() {+ let (_dir, store) = temp_store();+ let membership = GatedSink::new();+ let tracker = Arc::new(OffsetTracker::new());+ let person = Uuid::from_u128(0x0AC4);+ let partition_id = partition_of(TeamId(TEAM), &person, COHORT_PARTITION_COUNT) as u16;+ let (live_tx, live_rx) = mpsc::channel(4);+ let (seed_tx, seed_rx) = mpsc::channel(4);+ drop(seed_tx);+ let worker = Stage1Worker::spawn(+ partition_id,+ WorkerInbox::unmetered(live_rx, seed_rx),+ test_handle(&store),+ person_catalog(),+ Arc::new(membership.clone()),+ tracker.clone(),+ merge_deps_with(CaptureStreamEventSink::new()),+ EvictionRestore::Skip,+ );+ tracker.mark_dispatched(partition_id as i32, 1);+ live_tx+ .send(vec![ShuffleMessage::Event {+ event: Box::new(person_event(person, "[email protected]", 5, 0)),+ cse_offset: 0,+ broker_ts_ms: None,+ }])+ .await+ .unwrap();++ let deadline = Instant::now() + Duration::from_secs(10);+ while tracker.committable_offsets().get(&(partition_id as i32)) != Some(&1) {+ assert!(+ Instant::now() < deadline,+ "the offset never marked while the produce waited for its ack",+ );+ tokio::time::sleep(Duration::from_millis(5)).await;+ }+ assert_eq!(+ membership.enqueued.changes().len(),+ 1,+ "the flip was enqueued"+ );+ assert_eq!(membership.acked(), 0, "and is still unacked");++ membership.release(1);+ drop(live_tx);+ worker.join().await.unwrap();+ }++ #[tokio::test]+ async fn a_run_of_cascade_messages_marks_every_offset() {+ let (_dir, store) = temp_store();+ let merge = merge_deps_cascade(CaptureCascadeSink::new(), true);+ merge.cascade_tracker.mark_dispatched(0, 3);+ let cascade = |person: u128, offset: i64| ShuffleMessage::Cascade {+ message: Box::new(first_cascade(+ CohortMembershipChange {+ person_id: Uuid::from_u128(person).to_string(),+ ..entered(1)+ },+ 0,+ )),+ offset,+ };++ run_batch(+ 0,+ &store,+ person_catalog(),+ &CaptureSink::new(),+ &Arc::new(OffsetTracker::new()),+ merge.clone(),+ 0,+ vec![cascade(1, 0), cascade(2, 1), cascade(3, 2)],+ vec![],+ )+ .await;++ assert_eq!(+ merge.cascade_tracker.committable_offsets().get(&0),+ Some(&3),+ "every message the arm gathers into the run is settled",);}@@ -1608,13 +1816,15 @@ mod tombstone_redirect_tests {vec![ShuffleMessage::Event {event: Box::new(event),cse_offset: 0,- broker_ts_ms: None,+ broker_ts_ms: Some(STRAGGLER_BROKER_TS_MS),}],vec![],).await;}+ const STRAGGLER_BROKER_TS_MS: i64 = 1_750_000_000_000;+fn record_key(partition_id: u16, person: Uuid) -> PersonRecordKey {PersonRecordKey::new(partition_id, TEAM as u64, person)}@@ -1774,13 +1984,14 @@ mod tombstone_redirect_tests {let tracker = Arc::new(OffsetTracker::new());let failing = CaptureStreamEventSink::failing_always();+ let failing_deps = merge_deps_with(failing.clone());run_one_straggler(partition_id,&store,catalog.clone(),&membership,&tracker,- merge_deps_with(failing.clone()),+ failing_deps.clone(),person_event(p_old, "[email protected]", 5, 9),).await;@@ -1793,18 +2004,29 @@ mod tombstone_redirect_tests {None,"the failed produce holds the events offset",);+ assert_eq!(+ failing_deps.live_watermarks.get(partition_id as i32),+ None,+ "an unmarked batch never advances the seed fence",+ );let succeeding = CaptureStreamEventSink::new();+ let succeeding_deps = merge_deps_with(succeeding.clone());run_one_straggler(partition_id,&store,catalog,&membership,&tracker,- merge_deps_with(succeeding.clone()),+ succeeding_deps.clone(),person_event(p_old, "[email protected]", 5, 9),).await;+ assert_eq!(+ succeeding_deps.live_watermarks.get(partition_id as i32),+ Some(crate::partitions::WatermarkMs(STRAGGLER_BROKER_TS_MS)),+ "the marked redelivery advances it",+ );let produced = succeeding.events();assert_eq!(produced.len(),@@ -1825,7 +2047,8 @@ mod tombstone_redirect_tests {}#[tokio::test]- async fn membership_produce_failure_in_a_mixed_batch_withholds_the_re_key_and_the_mark() {+ async fn membership_produce_failure_in_a_mixed_batch_drops_the_change_and_still_re_keys_and_marks(+ ) {let (_dir, store) = temp_store();let catalog = person_catalog();let (p_old, partition_id, p_new) = cross_partition_pair();@@ -1834,51 +2057,11 @@ mod tombstone_redirect_tests {.map(Uuid::from_u128).find(|p| partition_of(TeamId(TEAM), p, COHORT_PARTITION_COUNT) as u16 == partition_id).expect("some uuid hashes onto p_old's partition");- let batch = || {- vec![- ShuffleMessage::Event {- event: Box::new(person_event(alice, "[email protected]", 5, 0)),- cse_offset: 0,- broker_ts_ms: None,- },- ShuffleMessage::Event {- event: Box::new(person_event(p_old, "[email protected]", 5, 9)),- cse_offset: 1,- broker_ts_ms: None,- },- ]- };let membership = CaptureSink::failing_first(1);let stream_sink = CaptureStreamEventSink::new();let tracker = Arc::new(OffsetTracker::new());- run_batch(- partition_id,- &store,- catalog.clone(),- &membership,- &tracker,- merge_deps_with(stream_sink.clone()),- 2,- batch(),- vec![],- )- .await;- assert!(- membership.changes().is_empty(),- "the failed flush recorded nothing"- );- assert!(- stream_sink.events().is_empty(),- "the membership failure withholds the straggler's re-key produce",- );- assert_eq!(- tracker.committable_offsets().get(&(partition_id as i32)),- None,- "the membership failure holds the whole sub-batch's offset",- );-run_batch(partition_id,&store,@@ -1887,16 +2070,27 @@ mod tombstone_redirect_tests {&tracker,merge_deps_with(stream_sink.clone()),2,- batch(),+ vec![+ ShuffleMessage::Event {+ event: Box::new(person_event(alice, "[email protected]", 5, 0)),+ cse_offset: 0,+ broker_ts_ms: None,+ },+ ShuffleMessage::Event {+ event: Box::new(person_event(p_old, "[email protected]", 5, 9)),+ cse_offset: 1,+ broker_ts_ms: None,+ },+ ],vec![],).await;assert!(membership.changes().is_empty(),- "the replay emits no duplicate membership change (the failed flip stays dropped, at-most-once)",+ "the failed produce dropped alice's flip (at-most-once)",);let produced = stream_sink.events();- assert_eq!(produced.len(), 1, "the replay produces the re-key once");+ assert_eq!(produced.len(), 1, "the straggler's re-key still produces");assert_eq!(produced[0].person_id, p_new.to_string());assert_eq!((produced[0].source_partition, produced[0].source_offset),@@ -1906,7 +2100,7 @@ mod tombstone_redirect_tests {assert_eq!(tracker.committable_offsets().get(&(partition_id as i32)),Some(&2),- "the acked replay releases the offset past both events",+ "the mark passes both events",);}diff --git a/rust/cohort-stream-processor/tests/events_consumer.rs b/rust/cohort-stream-processor/tests/events_consumer.rsindex 9766f8f60247..07da8f75355f 100644--- a/rust/cohort-stream-processor/tests/events_consumer.rs+++ b/rust/cohort-stream-processor/tests/events_consumer.rs@@ -745,7 +745,7 @@ async fn trickled_events_are_never_silently_dropped() {);}-/// Blocks its first flush until released, preventing the worker from marking its offset.+/// Holds its first produce until released, as a broker that has not acked yet would.struct BarrierSink {entered: Arc<tokio::sync::Notify>,release: Arc<tokio::sync::Notify>,@@ -796,7 +796,7 @@ impl MembershipSink for BarrierSink {#[tokio::test]#[ignore = "requires a running Kafka broker (KAFKA_HOSTS); run with --ignored against a local stack"]-async fn does_not_commit_past_a_blocked_produce() {+async fn commits_past_a_blocked_produce() {let suffix = Uuid::new_v4();let topic = format!("cohort_stream_events_barrier_{suffix}");let group = format!("cohort-stream-processor-barrier-{suffix}");@@ -863,40 +863,26 @@ async fn does_not_commit_past_a_blocked_produce() {.await.expect("worker reached the produce barrier");- // Several commit ticks fire while the produce is blocked.- tokio::time::sleep(Duration::from_millis(500)).await;- assert_eq!(- committed_sum(&verifier, &topic),- 0,- "no offset may be committed before its produce is acked",- );- assert_eq!(- sink.recorded_len(),- 0,- "the blocked flush has recorded nothing yet",- );-- release.notify_one();-+ // Live output is at most once and its state is already committed, so the commit does not+ // wait for the ack.let start = Instant::now();- loop {- if committed_sum(&verifier, &topic) == 1 {- break;- }+ while committed_sum(&verifier, &topic) != 1 {assert!(start.elapsed() < Duration::from_secs(30),- "offset did not advance after releasing the produce",+ "the offset did not commit while the produce was blocked",);tokio::time::sleep(Duration::from_millis(100)).await;}+ assert_eq!(sink.recorded_len(), 0, "the produce is still blocked");++ release.notify_one();+ shutdown_handle.request_shutdown();+ task.await.expect("consumer task panicked");assert_eq!(sink.recorded_len(),1,- "the released flush recorded its change",+ "the worker collected the released produce before it stopped",);-- shutdown_handle.request_shutdown();- task.await.expect("consumer task panicked");}/// Whether any of the first `persons` persons has behavioral state under `partition` in `store`.diff --git a/rust/cohort-stream-processor/tests/stage1_worker.rs b/rust/cohort-stream-processor/tests/stage1_worker.rsindex 57dd4df2b789..a5a894b8bda6 100644--- a/rust/cohort-stream-processor/tests/stage1_worker.rs+++ b/rust/cohort-stream-processor/tests/stage1_worker.rs@@ -1485,7 +1485,7 @@ async fn worker_advances_offset_on_empty_transition_subbatch() {}#[tokio::test]-async fn worker_holds_offset_when_the_only_flush_fails() {+async fn worker_marks_past_its_only_failed_produce() {let (_dir, store) = temp_store();let sink = CaptureSink::failing_first(1);let tracker = Arc::new(OffsetTracker::new());@@ -1509,10 +1509,10 @@ async fn worker_holds_offset_when_the_only_flush_fails() {assert_eq!(tracker.committable_offsets().get(&(PARTITION_ID as i32)),- None,- "a failed produce holds the offset back for replay",+ Some(&11),+ "live output is at most once: the state is committed, so the mark does not wait on the produce",);- assert!(sink.changes().is_empty(), "a failed flush emits nothing");+ assert!(sink.changes().is_empty(), "a failed produce emits nothing");}#[tokio::test]@@ -3231,67 +3231,3 @@ async fn seed_produce_failure_holds_only_the_seed_tracker_and_the_redelivery_re_"a seed-only batch never advances the live watermark",);}--/// A held batch must not advance the watermark; the successful redelivery does.-#[tokio::test]-async fn watermark_advances_only_after_a_successful_mark() {- let (_dir, store) = temp_store();- let filters = build_team_filters(vec![(CohortId(1), cohort(vec![behavioral_leaf(7)]))]);- let alice = person(1);-- let catalog = catalog_of(filters);- let sink = CaptureSink::failing_first(1);- let tracker = Arc::new(OffsetTracker::new());- let deps = MergeWorkerDeps::capture();-- let broker_ts = 1_750_000_000_000_i64;- let run = |batch: Vec<ShuffleMessage>| {- let catalog = catalog.clone();- let sink = sink.clone();- let tracker = tracker.clone();- let deps = deps.clone();- let store = store.clone();- async move {- let (tx, rx) = mpsc::channel(16);- let rx = WorkerInbox::live_only(rx);- let worker = Stage1Worker::spawn(- PARTITION_ID,- rx,- test_handle(&store),- catalog,- Arc::new(sink),- tracker.clone(),- deps,- EvictionRestore::Skip,- );- tracker.mark_dispatched(PARTITION_ID as i32, 1);- tx.send(batch).await.unwrap();- drop(tx);- worker.join().await.unwrap();- }- };-- run(vec![ShuffleMessage::Event {- event: Box::new(event(alice, 5, 0)),- cse_offset: 0,- broker_ts_ms: Some(broker_ts),- }])- .await;- assert_eq!(- deps.live_watermarks.get(PARTITION_ID as i32),- None,- "a held (unmarked) batch never advances the watermark",- );-- run(vec![ShuffleMessage::Event {- event: Box::new(event(alice, 5, 0)),- cse_offset: 0,- broker_ts_ms: Some(broker_ts),- }])- .await;- assert_eq!(- deps.live_watermarks.get(PARTITION_ID as i32),- Some(cohort_stream_processor::partitions::WatermarkMs(broker_ts)),- "the successful redelivery advances it",- );-}diff --git a/rust/cohort-stream-processor/tests/sweep_worker.rs b/rust/cohort-stream-processor/tests/sweep_worker.rsindex d8e6da4ed652..cc63e86ed645 100644--- a/rust/cohort-stream-processor/tests/sweep_worker.rs+++ b/rust/cohort-stream-processor/tests/sweep_worker.rs@@ -24,11 +24,13 @@ use cohort_stream_processor::consumers::{BootReadiness, CohortStreamEvent, Eventuse cohort_stream_processor::filters::{CatalogHandle, CohortId, FilterCatalog, TeamFilters, TeamFiltersBuilder, TeamId,};+use cohort_stream_processor::merge::transfer::Tombstone;use cohort_stream_processor::partitions::{- OffsetTracker, PartitionRouter, ShuffleMessage, WorkerInbox,+ partition_of, OffsetTracker, PartitionRouter, ShuffleMessage, WorkerInbox,+ COHORT_PARTITION_COUNT,};use cohort_stream_processor::producer::{- CaptureSink, CohortMembershipChange, MembershipSink, MembershipStatus,+ CaptureSink, CaptureStreamEventSink, CohortMembershipChange, MembershipSink, MembershipStatus,};use cohort_stream_processor::stage1::bucket_tz::{day_idx_in_tz, start_of_day_ms_in_tz};use cohort_stream_processor::stage1::{@@ -37,7 +39,7 @@ use cohort_stream_processor::stage1::{use cohort_stream_processor::stage2::Stage2State;use cohort_stream_processor::store::{BehavioralKey, CohortStore, LeafStateKey, OffloadConfig, OffloadMode, PersonRecordKey,- Stage2Key, StoreConfig, StoreHandle,+ Stage2Key, StoreConfig, StoreHandle, TombstoneKey,};use cohort_stream_processor::workers::{process_event, EvictionRestore, MergeWorkerDeps, Stage1Worker,@@ -364,6 +366,7 @@ fn spawn_worker_prefilled(catalog: Arc<CatalogHandle>,sink: Arc<dyn MembershipSink>,tracker: Arc<OffsetTracker>,+ merge: Arc<MergeWorkerDeps>,batches: Vec<Vec<ShuffleMessage>>,) -> Stage1Worker {let (tx, rx) = mpsc::channel(batches.len().max(1));@@ -379,7 +382,7 @@ fn spawn_worker_prefilled(catalog,sink,tracker,- MergeWorkerDeps::capture(),+ merge,EvictionRestore::Skip,)}@@ -419,15 +422,44 @@ fn status_runs(changes: &[CohortMembershipChange], status: MembershipStatus) ->}/// Drive a due wave larger than one sweep batch against a live lane that is already full, producing-/// into `sink`, and return once the worker has drained both.+/// into `sink`, and return once the worker has drained both. With `failing_re_key`, each live batch+/// after the wave also carries an event of a merged person whose re-key fails, which skips that+/// batch's mark.async fn wave_against_a_full_live_lane(wave: usize,live_after: usize,sink: Arc<dyn MembershipSink>,+ failing_re_key: bool,) {let (_dir, store) = temp_store();let filters = build_team_filters(vec![behavioral_leaf(7)]);let tracker = Arc::new(OffsetTracker::new());+ let mut merge = Arc::try_unwrap(MergeWorkerDeps::capture())+ .unwrap_or_else(|_| panic!("the test owns the only dependency Arc"));+ let merged = person(u128::MAX);+ if failing_re_key {+ let survivor = (1u128 << 64..)+ .map(person)+ .find(|p| partition_of(TeamId(TEAM), p, COHORT_PARTITION_COUNT) as u16 != PARTITION_ID)+ .expect("some person hashes off the worker's partition");+ store+ .write_batch(|batch| {+ batch.put_tombstone(+ &TombstoneKey {+ partition_id: PARTITION_ID,+ team_id: TEAM as u64,+ person: merged,+ },+ &Tombstone {+ new_person: survivor,+ merged_at_ms: 1,+ }+ .encode(),+ )+ })+ .unwrap();+ merge.stream_event_sink = Arc::new(CaptureStreamEventSink::failing_always());+ }let ts = "2026-05-20 10:00:00.000000";let event_ms = clickhouse_timestamp_to_millis(ts).unwrap();@@ -438,14 +470,29 @@ async fn wave_against_a_full_live_lane(// Distinct persons, so these enter on their own rather than reviving a key the pass selected.batches.extend((0..live_after).map(|i| {let offset = (wave + i) as i64;- vec![ShuffleMessage::Event {+ let mut batch = vec![ShuffleMessage::Event {event: Box::new(event_at(person(offset as u128 + 1), ts, offset)),cse_offset: offset,broker_ts_ms: None,- }]+ }];+ if failing_re_key {+ batch.push(ShuffleMessage::Event {+ event: Box::new(event_at(merged, ts, offset)),+ cse_offset: offset,+ broker_ts_ms: None,+ });+ }+ batch}));- let worker = spawn_worker_prefilled(&store, catalog_of(filters), sink, tracker, batches);+ let worker = spawn_worker_prefilled(+ &store,+ catalog_of(filters),+ sink,+ tracker,+ Arc::new(merge),+ batches,+ );worker.join().await.unwrap();}@@ -826,41 +873,47 @@ async fn sweep_caps_evictions_per_pass_and_drains_the_remainder_next_tick() {async fn a_due_wave_drains_in_batches_that_alternate_with_live_traffic() {// A wave used to drain in one pass, holding the partition for its whole duration and allocating// every state value at once. Each batch now claims at most the target, and a live batch takes the- // turn in between, so a tz-midnight wave costs live traffic one batch at a time.- let sink = CaptureSink::new();- wave_against_a_full_live_lane(- INTERLEAVED_WAVE,- INTERLEAVED_LIVE_AFTER,- Arc::new(sink.clone()),- )- .await;- let changes = sink.changes();+ // turn in between, so a tz-midnight wave costs live traffic one batch at a time. The turn is+ // claimed before a live batch runs, so a batch whose re-key fails and skips the rest of the loop+ // still owes it. Claiming after the batch instead passes the case whose re-keys succeed, while a+ // lane that keeps failing holds a due wave until it closes.+ for failing_re_key in [false, true] {+ let sink = CaptureSink::new();+ wave_against_a_full_live_lane(+ INTERLEAVED_WAVE,+ INTERLEAVED_LIVE_AFTER,+ Arc::new(sink.clone()),+ failing_re_key,+ )+ .await;+ let changes = sink.changes();- assert_eq!(- status_runs(&changes, MembershipStatus::Left),- vec![- SWEEP_BATCH_KEYS,- SWEEP_BATCH_KEYS,- INTERLEAVED_WAVE - 2 * SWEEP_BATCH_KEYS,- ],- "three bounded batches, each separated by the live output of the batch that ran between",- );- assert_eq!(- changes- .iter()- .filter(|change| change.status == MembershipStatus::Left)- .count(),- INTERLEAVED_WAVE,- "interleaving does not cost the sweep any of its wave",- );- assert_eq!(- changes- .iter()- .filter(|change| change.status == MembershipStatus::Entered)- .count(),- INTERLEAVED_WAVE + INTERLEAVED_LIVE_AFTER,- "every queued live event entered while the sweep was draining",- );+ assert_eq!(+ status_runs(&changes, MembershipStatus::Left),+ vec![+ SWEEP_BATCH_KEYS,+ SWEEP_BATCH_KEYS,+ INTERLEAVED_WAVE - 2 * SWEEP_BATCH_KEYS,+ ],+ "failing_re_key={failing_re_key}: three bounded batches, each separated by the live output of the batch that ran between",+ );+ assert_eq!(+ changes+ .iter()+ .filter(|change| change.status == MembershipStatus::Left)+ .count(),+ INTERLEAVED_WAVE,+ "failing_re_key={failing_re_key}: interleaving does not cost the sweep any of its wave",+ );+ assert_eq!(+ changes+ .iter()+ .filter(|change| change.status == MembershipStatus::Entered)+ .count(),+ INTERLEAVED_WAVE + INTERLEAVED_LIVE_AFTER,+ "failing_re_key={failing_re_key}: every queued live event entered while the sweep was draining",+ );+ }}#[tokio::test]@@ -873,6 +926,7 @@ async fn a_resumed_sweep_stamps_newer_than_the_live_output_it_yielded_to() {INTERLEAVED_WAVE,INTERLEAVED_LIVE_AFTER,Arc::new(sink.clone()),+ false,).await;let changes = sink.changes();@@ -923,6 +977,7 @@ async fn an_event_between_batches_saves_a_key_the_pass_had_already_selected() {catalog_of(filters),Arc::new(sink.clone()),tracker,+ MergeWorkerDeps::capture(),vec![wave_then_sweep(WAVE, ts, cutoff),// Two days later, so this person's new deadline lands past the pass's cutoff.@@ -980,6 +1035,7 @@ async fn a_later_batchs_produce_failure_leaves_the_earlier_batches_settled() {catalog_of(filters),Arc::new(sink.clone()),tracker,+ MergeWorkerDeps::capture(),vec![wave_then_sweep(WAVE, ts, cutoff),vec![ShuffleMessage::Sweep {@@ -1022,41 +1078,6 @@ async fn a_later_batchs_produce_failure_leaves_the_earlier_batches_settled() {}}-#[tokio::test]-async fn a_live_lane_that_keeps_failing_its_produce_still_yields_the_sweep_its_turns() {- // The sweep's turn is claimed before a live batch runs, so a batch that fails its produce and- // skips the rest of the loop body still owes it. Claiming after the batch instead passes every- // test whose live lane succeeds, while a lane that keeps failing holds a due wave until it closes.- // Call 1 is the wave's own Entered; every live produce after it fails.- let sink = FailingLiveSink::new(2);- wave_against_a_full_live_lane(- INTERLEAVED_WAVE,- INTERLEAVED_LIVE_AFTER,- Arc::new(sink.clone()),- )- .await;-- // `None` is a failed live produce. Each sweep batch after the first runs on the turn the failed- // live batch before it owed, rather than all of them once the lane has closed.- assert_eq!(- sink.outcomes(),- [- vec![- Some(MembershipStatus::Entered),- Some(MembershipStatus::Left)- ],- vec![- None,- Some(MembershipStatus::Left),- None,- Some(MembershipStatus::Left)- ],- vec![None; INTERLEAVED_LIVE_AFTER - 2],- ]- .concat(),- );-}-#[tokio::test]async fn single_does_not_evict_before_its_calendar_midnight_and_does_after() {let lsk = behavioral_lsk(&build_team_filters_tz(vec![behavioral_leaf(7)], Kolkata));@@ -1585,51 +1606,6 @@ impl MembershipSink for FailNthSink {}}-/// A sink that fails every live produce from `fail_from` (1-based) on while letting sweep batches-/// through, telling them apart by their changes being all `Left`. Records each call as the status of-/// its first change, or `None` when it failed.-#[derive(Clone)]-struct FailingLiveSink {- outcomes: Arc<Mutex<Vec<Option<MembershipStatus>>>>,- fail_from: usize,-}--impl FailingLiveSink {- fn new(fail_from: usize) -> Self {- Self {- outcomes: Arc::default(),- fail_from,- }- }-- fn outcomes(&self) -> Vec<Option<MembershipStatus>> {- self.outcomes.lock().unwrap().clone()- }-}--#[async_trait]-impl MembershipSink for FailingLiveSink {- async fn produce(- &self,- changes: Vec<CohortMembershipChange>,- ) -> Vec<Result<(), KafkaProduceError>> {- let mut outcomes = self.outcomes.lock().unwrap();- let call = outcomes.len() + 1;- let is_sweep = changes- .iter()- .all(|change| change.status == MembershipStatus::Left);- if call >= self.fail_from && !is_sweep {- outcomes.push(None);- return changes- .iter()- .map(|_| Err(KafkaProduceError::KafkaProduceCanceled))- .collect();- }- outcomes.push(changes.first().map(|change| change.status));- changes.iter().map(|_| Ok(())).collect()- }-}-#[tokio::test]async fn sweep_produce_failure_reschedules_and_a_later_sweep_retries() {let (_dir, store) = temp_store();