:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(cohorts): recompose composed cohorts whose stage 2 row is missing on seed apply (#121092)
Harley Alexander4ecd79bd56c6
diff --git a/docs/internal/realtime-cohorts/invariants.md b/docs/internal/realtime-cohorts/invariants.mdindex ee1a2a0c69f8..b020d1c0d28b 100644--- a/docs/internal/realtime-cohorts/invariants.md+++ b/docs/internal/realtime-cohorts/invariants.md@@ -265,6 +265,7 @@ It does not rule out duplicates: a store that loses unflushed writes, or is wipeReconcile repairs lost output, not lost state.It re-tells the rows the store holds.An event that Stage 1 skipped needs a later seed that covers it, and a person whose first Stage 2 row was never written needs a seed or a leaf flip that touches them.+A person seed that changes nothing still recomposes a composed cohort with no row for the person, so a cohort added over already-seeded leaves gets its rows from its own run.The system is built on one bargain: live paths favor speed and may lose a change, and every backfill ends with a reconcile that re-tells everything the store holds.Anything that stops reconcile from running, or from covering a cohort, leaves downstream wrong with no signal.diff --git a/docs/internal/realtime-cohorts/seed-apply-and-reconcile.md b/docs/internal/realtime-cohorts/seed-apply-and-reconcile.mdindex 0ee3b8b1780a..b6ffbc92b028 100644--- a/docs/internal/realtime-cohorts/seed-apply-and-reconcile.md+++ b/docs/internal/realtime-cohorts/seed-apply-and-reconcile.md@@ -131,6 +131,11 @@ Both kinds finish the same way, in an order the code enforces with types.One write batch holds the new leaf rows or person records and, for each single-leaf cohort about to be emitted, a Stage 2 row set to the value being **replaced**.3. **Schedule** eviction deadlines for the changed leaf rows.4. **Recompose** composed cohorts, reading each person's state once for all their cohorts, and decide their emissions.+ A person is recomposed when the fold changed their state, or when a single-leaf row for them lagged in step 1.+ For a person seed, a composed cohort over one of their evaluated leaves that has no Stage 2 row for them also recomposes, on its own, without the other cohorts on its leaves.+ That case covers a cohort added over leaves an earlier run already seeded: the seed changes nothing, so without it the cohort never gets a row, and its reconcile certifies it with no members.+ A non-member's row stays absent, so a cohort the run's own leaf answers already make false is skipped without a read, and only the cohort recomposes, not its leaves.+ The seed path also rewrites an unreadable Stage 2 row with a readable bit when it recomposes the cohort, as the single-leaf diff does.5. **Produce**, in parallel: the membership changes tagged `origin: seed` with the run id, first-hop cascades if enabled, and re-keyed seeds for merged persons.6. **Stage 2 commit.**Write the new truth into the Stage 2 rows.@@ -198,7 +203,7 @@ Reconcile re-tells what the store holds, and it cannot bring back what the storeAn event that Stage 1 skipped on a store error is missing from the leaf state, so the walk recomputes the same wrong answer.A person whose first Stage 2 row for the cohort was never written, for example after a crash between a Stage 1 batch and its Stage 2 batch, is not in the walk at all.Seeds from a later run that cover the missed event repair the first case.-A seed or a leaf flip that touches the person repairs the second.+A seed or a leaf flip that touches the person repairs the second: a person seed recomposes a composed cohort that has no row for the person even when it changes nothing.### Why every row is emitteddiff --git a/rust/cohort-stream-processor/src/observability/metrics.rs b/rust/cohort-stream-processor/src/observability/metrics.rsindex 8ec201630f3c..a006b3092ebd 100644--- a/rust/cohort-stream-processor/src/observability/metrics.rs+++ b/rust/cohort-stream-processor/src/observability/metrics.rs@@ -627,6 +627,12 @@ pub const PERSON_SEED_REKEY_PRODUCE_FAILURE_TOTAL: &str =/// produce-failure counters before concluding produces are failing.**pub const SEED_REGISTER_REPAIRS_TOTAL: &str = "cohort_seed_register_repairs_total";+/// Composed-cohort membership changes a person seed apply emitted for a cohort it recomposed only+/// because the cohort had no row for the person, labelled by `kind` (`entered`|`left`) (counter).+/// Counted when the writes commit, like [`SEED_REGISTER_REPAIRS_TOTAL`]. Expect these in bulk on the+/// first run of a composed cohort whose leaves earlier runs already seeded, and rarely after.+pub const SEED_COMPOSED_ROW_REPAIRS_TOTAL: &str = "cohort_seed_composed_row_repairs_total";+/// Persons whose Stage 2 inputs a seed apply read through shared store sections (counter)./// Attempt-based: a held run counts its persons, and so does the redelivery that replays it./// **Do not divide [`STAGE2_COHORTS_EVALUATED`] by this**, because that counter is settled-based and@@ -1100,6 +1106,10 @@ mod tests {SEED_REGISTER_REPAIRS_TOTAL,"cohort_seed_register_repairs_total");+ assert_eq!(+ SEED_COMPOSED_ROW_REPAIRS_TOTAL,+ "cohort_seed_composed_row_repairs_total"+ );assert_eq!(SEED_RECOMPUTE_PERSONS_TOTAL,"cohort_seed_recompute_persons_total"diff --git a/rust/cohort-stream-processor/src/stage2/evaluator.rs b/rust/cohort-stream-processor/src/stage2/evaluator.rsindex 1b01ced82f97..78db13867250 100644--- a/rust/cohort-stream-processor/src/stage2/evaluator.rs+++ b/rust/cohort-stream-processor/src/stage2/evaluator.rs@@ -84,6 +84,32 @@ pub fn evaluate_tree(}}+/// The cohort's membership as far as `known` decides it, or `None` when the answer depends on a leaf+/// `known` does not hold or on a cohort reference. Agrees with [`evaluate_tree`] wherever it returns+/// `Some`, since that reads a leaf `known` lacks as whatever the store holds.+pub fn evaluate_known(node: &FilterNode, known: &HashMap<LeafStateKey, bool>) -> Option<bool> {+ match node {+ FilterNode::Group { op, children } => {+ let mut undecided = false;+ for child in children {+ match (op, evaluate_known(child, known)) {+ (BoolOp::And, Some(false)) => return Some(false),+ (BoolOp::Or, Some(true)) => return Some(true),+ (_, None) => undecided = true,+ _ => {}+ }+ }+ (!undecided).then_some(*op == BoolOp::And)+ }+ FilterNode::Leaf(CohortLeaf::CohortRef(_)) => None,+ FilterNode::Leaf(leaf) => match leaf.leaf_state_key() {+ Some(lsk) => known.get(&lsk).map(|bit| bit ^ leaf.negated()),+ // [`evaluate_tree`] reads a leaf with no state key as `false` whatever is stored.+ None => Some(leaf.negated()),+ },+ }+}+#[cfg(test)]mod tests {use super::*;@@ -127,6 +153,53 @@ mod tests {FilterNode::Group { op, children }}+ #[test]+ fn evaluate_known_decides_only_what_the_known_leaves_force() {+ let known = HashMap::from([(lsk(1), true), (lsk(2), false)]);+ for (why, tree, want) in [+ (+ "a false leaf decides an AND",+ group(BoolOp::And, vec![person_leaf(lsk(2)), person_leaf(lsk(3))]),+ Some(false),+ ),+ (+ "an unknown leaf leaves an AND open",+ group(BoolOp::And, vec![person_leaf(lsk(1)), person_leaf(lsk(3))]),+ None,+ ),+ (+ "a true leaf decides an OR",+ group(BoolOp::Or, vec![person_leaf(lsk(1)), person_leaf(lsk(3))]),+ Some(true),+ ),+ (+ "an unknown leaf leaves an OR open",+ group(BoolOp::Or, vec![person_leaf(lsk(2)), person_leaf(lsk(3))]),+ None,+ ),+ (+ "negation applies to a known leaf",+ group(BoolOp::And, vec![person_leaf_neg(lsk(1), true)]),+ Some(false),+ ),+ (+ "a cohort reference is never known",+ group(BoolOp::And, vec![person_leaf(lsk(1)), cohort_ref_leaf()]),+ None,+ ),+ (+ "all known leaves decide",+ group(+ BoolOp::And,+ vec![person_leaf(lsk(1)), person_leaf_neg(lsk(2), true)],+ ),+ Some(true),+ ),+ ] {+ assert_eq!(evaluate_known(&tree, &known), want, "{why}");+ }+ }+fn classifier_excludes_top_level_negation(root: &FilterNode) -> bool {fn state_keyed_leaf_count(node: &FilterNode) -> u32 {match node {diff --git a/rust/cohort-stream-processor/src/workers/person_seed_path.rs b/rust/cohort-stream-processor/src/workers/person_seed_path.rsindex a1da25c29878..dd2b22a5842e 100644--- a/rust/cohort-stream-processor/src/workers/person_seed_path.rs+++ b/rust/cohort-stream-processor/src/workers/person_seed_path.rs@@ -322,6 +322,7 @@ fn fold_person_seeds(let mut records = StagedBatch::default();let mut leaves: BTreeMap<TeamId, Vec<FoldedLeaf>> = BTreeMap::new();+ let mut composed_row_candidates: BTreeMap<TeamId, Vec<FoldedLeaf>> = BTreeMap::new();for (key, touch) in &touches {let slot = overlay.slot(key)@@ -354,13 +355,21 @@ fn fold_person_seeds(.before().is_some_and(|record| record.matched.contains(hash));let minted_transition = before != in_cohort;- leaves.entry(touch.team_id).or_default().push(FoldedLeaf {- leaf_state_key: LeafStateKey::for_person_property(hash),+ let leaf_state_key = LeafStateKey::for_person_property(hash);+ let folded = FoldedLeaf {+ leaf_state_key,person_id: touch.person,in_cohort,minted_transition,run_id,- });+ };+ leaves.entry(touch.team_id).or_default().push(folded);+ if !touch.recompose.contains(&leaf_state_key) {+ composed_row_candidates+ .entry(touch.team_id)+ .or_default()+ .push(folded);+ }if minted_transition {let kind = if in_cohort {TransitionKind::Entered@@ -385,6 +394,7 @@ fn fold_person_seeds(records,leaves,recompose,+ composed_row_candidates,// Nothing to schedule: person-property membership has no window, so the sweep never owns// these leaves.schedules: Vec::new(),@@ -416,10 +426,12 @@ impl RecordUpdate {/// Whether stage 2 has to be recomposed for this seed.///-/// A merge that changed nothing leaves stage 2 already consistent with stage 1, and skipping the-/// recompose keeps a non-matching dormant person to a single point read instead of a composition-/// over every cohort its hashes touch — the dominant message shape of a scan that emits-/// non-matchers. Two outcomes still have to recompose:+/// A merge that changed nothing leaves stage 2 consistent with stage 1 for every cohort that already+/// has a row; a composed cohort without one is the register diff's to admit, see+/// [`crate::workers::stage2_path::composed_rows_missing`]. Skipping the recompose keeps a+/// non-matching dormant person to a single point read instead of a composition over every cohort its+/// hashes touch — the dominant message shape of a scan that emits non-matchers. Two outcomes still+/// have to recompose:////// - `SkipLiveFresh` may be the redelivery of an attempt that committed stage 1 and then failed its/// produce. The replay merges to `Unchanged` and mints no transition, and a live event that@@ -844,6 +856,18 @@ mod tests {.unwrap();}+ fn put_raw_stage2(&self, partition_id: u16, person: Uuid, cohort_id: u64, bytes: &[u8]) {+ let key = Stage2Key {+ partition_id,+ team_id: TEAM.0 as u64,+ cohort_id,+ person_id: person,+ };+ self.store+ .write_batch(|b| b.put_stage2(&key, bytes))+ .unwrap();+ }+fn stage2(&self, partition_id: u16, person: Uuid, cohort_id: u64) -> Option<Stage2State> {self.store.get_stage2(&Stage2Key {@@ -1843,6 +1867,82 @@ mod tests {assert_eq!(shell.committable(partition_id), Some(2));}+ /// The composed half of the same case. The seed asserts what the record already holds and no+ /// single-leaf register lags, so only the missing composed row can admit the recompose; without+ /// it the cohort's backfill completes with no members.+ #[tokio::test]+ async fn a_composed_cohort_added_over_an_existing_record_composes_on_the_next_unchanged_seed() {+ // (member, row before, change, row after)+ for (member, corrupt_row, expected_change, expected_row) in [+ (true, false, Some(MembershipStatus::Entered), Some(true)),+ // A non-member's row stays absent, so the next run reads it again and writes nothing.+ (false, false, None, None),+ // An unreadable row is rewritten with a readable bit even when nothing flips.+ (false, true, None, Some(false)),+ ] {+ let (person, partition_id) = dormant_person();+ let single_leaf = vec![+ (1, wrap(vec![person_leaf(PERSON_HASH)])),+ (4, wrap(vec![behavioral_leaf()])),+ ];+ let mut shell = Shell::new(single_leaf.clone());+ if member {+ shell.live_pageview(partition_id, person, None);+ }+ // An earlier run's result, stamped old enough that this seed applies as an ordinary+ // Unchanged merge rather than a live-fresh skip.+ shell.put_record(+ partition_id,+ person,+ &PersonRecord {+ last_seen_ms: now_ms(),+ stamp: Stamp::new(now_ms() - 2 * MARGIN_MS, 0),+ props_fingerprint: PropsFingerprint::of(r#"{"email":"[email protected]"}"#),+ catalog_fingerprint: shell.filters.catalog_fingerprint,+ matched: MatchedSet::from_iter([hash(PERSON_HASH).as_bytes()]),+ applied_offsets: AppliedOffsets::default(),+ redirect_dedup: Default::default(),+ },+ );+ shell.put_register(partition_id, person, 1, true);+ if corrupt_row {+ shell.put_raw_stage2(partition_id, person, 2, b"not a stage 2 row");+ }++ // Cohort 2 composes both leaves, neither of them new to the store.+ let mut cohorts = single_leaf;+ cohorts.push((2, wrap(vec![person_leaf(PERSON_HASH), behavioral_leaf()])));+ shell.catalog = Arc::new(CatalogHandle::from_catalog(FilterCatalog::from_teams([(+ TEAM,+ build_filters(&cohorts),+ )])));+ let seed = seed_for(person, &[PERSON_HASH], &[PERSON_HASH], now_ms());+ shell.run(partition_id, &seed, 0).await;++ let case = format!("member={member} corrupt_row={corrupt_row}");+ assert_eq!(+ shell+ .sink+ .changes()+ .iter()+ .map(|change| (change.cohort_id, change.status))+ .collect::<Vec<_>>(),+ expected_change+ .map(|status| vec![(2, status)])+ .unwrap_or_default(),+ "{case}",+ );+ assert_eq!(+ shell+ .stage2(partition_id, person, 2)+ .map(|state| state.in_cohort),+ expected_row,+ "{case}",+ );+ assert_eq!(shell.committable(partition_id), Some(1), "{case}");+ }+ }+#[tokio::test]async fn a_seeded_dormant_person_reaches_the_same_membership_as_a_live_evaluated_one() {let live_person = Uuid::from_u128(0xA11CE);diff --git a/rust/cohort-stream-processor/src/workers/seed_apply.rs b/rust/cohort-stream-processor/src/workers/seed_apply.rsindex 71883dce9196..47642d515dbf 100644--- a/rust/cohort-stream-processor/src/workers/seed_apply.rs+++ b/rust/cohort-stream-processor/src/workers/seed_apply.rs@@ -26,7 +26,7 @@ use tracing::warn;use uuid::Uuid;use crate::filters::reverse_index::TeamFilters;-use crate::filters::{FilterCatalog, TeamId};+use crate::filters::{CohortId, FilterCatalog, TeamId};use crate::merge::tombstone_redirect::{self, Resolution};use crate::observability::metrics::{COHORT_STREAM_OFFSET_AHEAD_OF_DISPATCH, OUTPUT_TRANSITIONS_UNMAPPED,@@ -49,9 +49,10 @@ use crate::workers::merge_path::MergeWorkerDeps;use crate::workers::reconcile::{ReconcileQueue, SupersedeOutcome};use crate::workers::seed_run::{Admitted, OffsetSpan, SeedGroup, SeedKind, SeedOffset, SeedRun};use crate::workers::stage2_path::{- commit_stage2_writes, diff_single_leaf_registers, FoldedLeaf, RegisterDiff, Stage2Recompute,+ commit_stage2_writes, composed_rows_missing, diff_single_leaf_registers, FoldedLeaf,+ RegisterDiff, Stage2Recompute,};-use crate::workers::stage2_person_inputs::recompute_stage2_by_person;+use crate::workers::stage2_person_inputs::{recompute_stage2_by_person, RecomposeGroup};use crate::workers::worker::{first_cascades, produce_cascades, produce_membership, transition_metric_label,};@@ -447,6 +448,8 @@ pub(crate) struct TouchedPersons {struct PersonTouch {run: RunId,leaves: BTreeSet<LeafStateKey>,+ /// Composed cohorts to recompose on their own, without the other cohorts on their leaves.+ cohorts: BTreeSet<CohortId>,}impl TouchedPersons {@@ -456,6 +459,7 @@ impl TouchedPersons {.or_insert_with(|| PersonTouch {run,leaves: BTreeSet::new(),+ cohorts: BTreeSet::new(),})}@@ -479,17 +483,34 @@ impl TouchedPersons {self.entry(team_id, person, run).leaves.insert(leaf);}+ /// Admit one composed cohort whose row is missing. Like a lagging leaf, its run counts only for+ /// a person the fold never admitted.+ pub(crate) fn touch_missing(+ &mut self,+ team_id: TeamId,+ person: Uuid,+ run: RunId,+ cohort: CohortId,+ ) {+ self.entry(team_id, person, run).cohorts.insert(cohort);+ }+fn is_empty(&self) -> bool {self.per_person.is_empty()}/// One recompute call per `(team, run)`: the team owns the catalog that composes the leaves,/// the run owns their provenance tag.- fn groups(self) -> BTreeMap<(TeamId, RunId), Vec<(LeafStateKey, Uuid)>> {- let mut groups: BTreeMap<(TeamId, RunId), Vec<(LeafStateKey, Uuid)>> = BTreeMap::new();+ fn groups(self) -> BTreeMap<(TeamId, RunId), RecomposeGroup> {+ let mut groups: BTreeMap<(TeamId, RunId), RecomposeGroup> = BTreeMap::new();for ((team_id, person), touch) in self.per_person {let group = groups.entry((team_id, touch.run)).or_default();- group.extend(touch.leaves.into_iter().map(|leaf| (leaf, person)));+ group+ .leaves+ .extend(touch.leaves.into_iter().map(|leaf| (leaf, person)));+ group+ .cohorts+ .extend(touch.cohorts.into_iter().map(|cohort| (cohort, person)));}groups}@@ -685,6 +706,10 @@ pub(crate) struct Folded {pub leaves: BTreeMap<TeamId, Vec<FoldedLeaf>>,/// Persons whose composable cohorts must recompose, with the run that last touched them.pub recompose: TouchedPersons,+ /// Folded leaves the fold did not recompose, whose composed cohorts may still lack a row for the+ /// person; see [`composed_rows_missing`]. Only person runs fill it: a tile run recomposes every+ /// tile it folds.+ pub composed_row_candidates: BTreeMap<TeamId, Vec<FoldedLeaf>>,/// Final eviction deadline per touched behavioral key. Empty for person seeds.pub schedules: Vec<(BehavioralKey, i64)>,pub re_keys: ReKeys,@@ -700,6 +725,7 @@ impl Folded {records: StagedBatch::default(),leaves: BTreeMap::new(),recompose: TouchedPersons::default(),+ composed_row_candidates: BTreeMap::new(),schedules: Vec::new(),re_keys,tally,@@ -708,7 +734,8 @@ impl Folded {/// One [`diff_single_leaf_registers`] per team. Every person the diff emits a change or a write/// for joins `recompose`: a lagging register is its own reason to recompose, and nothing the- /// fold saw would admit it — the redelivery of a failed produce mints no transition.+ /// fold saw would admit it — the redelivery of a failed produce mints no transition. A composed+ /// cohort with no row for a person the run evaluated joins too, see [`composed_rows_missing`].async fn diff_registers(mut self,deps: ApplyDeps<'_>,@@ -750,6 +777,21 @@ impl Folded {);}}+ if let Some(candidates) = self.composed_row_candidates.get(&team_id) {+ let missing = composed_rows_missing(+ deps.partition_id,+ deps.handle,+ filters,+ candidates,+ leaves,+ ReadLane::Maintenance,+ )+ .await+ .map_err(SeedHold::store(ApplyStage::RegisterDiff))?;+ for ((person, cohort), run) in missing {+ self.recompose.touch_missing(team_id, person, run, cohort);+ }+ }diff.extend(team_diff);}@@ -851,7 +893,7 @@ impl Scheduled {stamp: &RunStamp,) -> Result<Recomputed, SeedHold> {if !self.recompose.is_empty() {- for ((team_id, run), leaves) in self.recompose.groups() {+ for ((team_id, run), group) in self.recompose.groups() {let Some(filters) = deps.team(team_id) else {continue;};@@ -860,7 +902,7 @@ impl Scheduled {deps.handle,team_id,filters,- &leaves,+ &group,stamp.now_ms,&stamp.last_updated,)@@ -1364,11 +1406,12 @@ mod tests {touched.touch(team, alice, first, LeafStateKey([1; 16]));touched.touch(team, alice, last, LeafStateKey([2; 16]));touched.touch_lagging(team, alice, first, LeafStateKey([3; 16]));+ touched.touch_missing(team, alice, first, CohortId(9));let groups = touched.groups();assert_eq!(groups.len(), 1, "one call, not one per seed");assert_eq!(- groups[&(team, last)],+ groups[&(team, last)].leaves,vec![(LeafStateKey([1; 16]), alice),(LeafStateKey([2; 16]), alice),@@ -1376,5 +1419,10 @@ mod tests {],"every leaf rides the last seed's run; a lagging leaf joins without moving it",);+ assert_eq!(+ groups[&(team, last)].cohorts,+ vec![(CohortId(9), alice)],+ "a cohort with a missing row joins the same call, without its leaves",+ );}}diff --git a/rust/cohort-stream-processor/src/workers/seed_path.rs b/rust/cohort-stream-processor/src/workers/seed_path.rsindex 44d9837a09fe..03e227dee8d3 100644--- a/rust/cohort-stream-processor/src/workers/seed_path.rs+++ b/rust/cohort-stream-processor/src/workers/seed_path.rs@@ -798,6 +798,7 @@ fn fold_tiles(records,leaves,recompose,+ composed_row_candidates: BTreeMap::new(),schedules,re_keys: ReKeys::Tiles(re_keys),tally,diff --git a/rust/cohort-stream-processor/src/workers/stage2_path.rs b/rust/cohort-stream-processor/src/workers/stage2_path.rsindex a50c82f7a615..7297379c4354 100644--- a/rust/cohort-stream-processor/src/workers/stage2_path.rs+++ b/rust/cohort-stream-processor/src/workers/stage2_path.rs@@ -15,14 +15,14 @@ use crate::filters::reverse_index::TeamFilters;use crate::filters::tree::{CohortLeaf, CohortTree, FilterNode};use crate::filters::CohortId;use crate::observability::metrics::{- SEED_REGISTER_REPAIRS_TOTAL, STAGE2_COHORTS_EVALUATED, STAGE2_STATE_DECODE_ERROR,- STAGE2_TRANSITIONS,+ SEED_COMPOSED_ROW_REPAIRS_TOTAL, SEED_REGISTER_REPAIRS_TOTAL, STAGE2_COHORTS_EVALUATED,+ STAGE2_STATE_DECODE_ERROR, STAGE2_TRANSITIONS,};use crate::producer::{ChangeOrigin, CohortMembershipChange, MembershipStatus};use crate::stage1::key::LeafStateKey;use crate::stage1::person_record::PersonRecord;use crate::stage1::state::{Stage1State, StateVariant, StatefulRecord};-use crate::stage2::evaluator::{evaluate_tree, leaf_membership};+use crate::stage2::evaluator::{evaluate_known, evaluate_tree, leaf_membership};use crate::stage2::state::{Stage2Ownership, Stage2State};use crate::store::{BehavioralKey, PersonRecordKey, ReadLane, Stage2Key, StagedBatch, StoreError, StoreHandle,@@ -71,6 +71,8 @@ pub(crate) struct Stage2Recompute {composed: StatusCounts,/// Single-leaf changes a register diff derived with no stage-1 transition behind them.repairs: RepairCounts,+ /// Composed flips of cohorts recomposed only because their row was missing.+ composed_repairs: StatusCounts,}impl Stage2Recompute {@@ -79,6 +81,8 @@ impl Stage2Recompute {counter!(STAGE2_COHORTS_EVALUATED).increment(self.evaluated);self.composed.record(STAGE2_TRANSITIONS);self.repairs.record(SEED_REGISTER_REPAIRS_TOTAL);+ self.composed_repairs+ .record(SEED_COMPOSED_ROW_REPAIRS_TOTAL);}/// Fold a sibling recompute in, so one produce and one commit cover both halves.@@ -88,6 +92,15 @@ impl Stage2Recompute {self.evaluated += other.evaluated;self.composed.add(other.composed);self.repairs.add(other.repairs);+ self.composed_repairs.add(other.composed_repairs);+ }++ /// Count a flip of a cohort recomposed only because its row was missing, see+ /// [`composed_rows_missing`]. Its own call, because the pair already went through `record_pair`.+ pub(super) fn record_composed_repair(&mut self, diff: &RecomputeDiff) {+ if diff.flipped() {+ self.composed_repairs.count(diff.status());+ }}/// Both recompute orders call this, so what a composed evaluation emits cannot depend on@@ -444,6 +457,79 @@ pub(crate) async fn diff_single_leaf_registers(})}+/// The composed cohorts keyed on `candidates`' leaves that hold no readable stage-2 row for the+/// person, each with the run of the first leaf that reached it.+///+/// A composed cohort recomposes only when a seed touches one of its leaves, and a person seed that+/// asserts what the record already holds touches nothing. A cohort added over leaves an earlier run+/// already seeded would then never get a row, and reconcile, which walks only existing rows, would+/// certify it with no members. So a missing row is its own reason to recompose that cohort, as a+/// lagging register is for the single-leaf half.+///+/// `candidates` are the leaves the fold did not recompose. `folded` is every leaf the run decided,+/// whose truths make a cohort definitely false without a read: an absent row is already right for+/// it, and a never-member's row stays absent, so recomposing it would repeat on every later run.+///+/// One batched `cf_stage2` read on `lane`. Leaves with no composed cohort keyed on them cost nothing.+pub(crate) async fn composed_rows_missing(+ partition_id: u16,+ handle: &StoreHandle,+ filters: &TeamFilters,+ candidates: &[FoldedLeaf],+ folded: &[FoldedLeaf],+ lane: ReadLane,+) -> Result<BTreeMap<(Uuid, CohortId), RunId>, StoreError> {+ let mut known: HashMap<Uuid, HashMap<LeafStateKey, bool>> = HashMap::new();+ for leaf in folded {+ known+ .entry(leaf.person_id)+ .or_default()+ .insert(leaf.leaf_state_key, leaf.in_cohort);+ }++ let mut wanted: BTreeMap<Stage2Key, (Uuid, CohortId, RunId)> = BTreeMap::new();+ for leaf in candidates {+ let Some(cohort_ids) = filters+ .by_lsk_to_composable_cohorts+ .get(&leaf.leaf_state_key)+ else {+ continue;+ };+ for cohort_id in cohort_ids {+ let Some(tree) = filters.cohorts.get(cohort_id) else {+ continue;+ };+ let truths = known.get(&leaf.person_id);+ if truths.is_some_and(|truths| evaluate_known(&tree.root, truths) == Some(false)) {+ continue;+ }+ let key = Stage2Key {+ partition_id,+ team_id: tree.team_id.0 as u64,+ cohort_id: cohort_id.0 as u64,+ person_id: leaf.person_id,+ };+ wanted+ .entry(key)+ .or_insert((leaf.person_id, *cohort_id, leaf.run_id));+ }+ }+ if wanted.is_empty() {+ return Ok(BTreeMap::new());+ }++ let keys: Vec<Stage2Key> = wanted.keys().copied().collect();+ let stored = handle.multi_get_stage2(keys, lane).await?;++ let mut missing = BTreeMap::new();+ for ((person_id, cohort_id, run_id), bytes) in wanted.into_values().zip(stored) {+ if !matches!(read_register(bytes), StoredRegister::Present(_)) {+ missing.insert((person_id, cohort_id), run_id);+ }+ }+ Ok(missing)+}+/// One `(single-leaf cohort, person)` the fold decided, before its stored row is read.#[derive(Clone, Copy, PartialEq, Eq)]struct WantedRegister {@@ -481,6 +567,7 @@ fn read_register(bytes: Option<Vec<u8>>) -> StoredRegister {Ok((state, ownership)) => StoredRegister::Present(PriorStage2State {in_cohort: state.in_cohort,ownership,+ corrupt: false,}),Err(_) => {counter!(STAGE2_STATE_DECODE_ERROR).increment(1);@@ -512,6 +599,8 @@ pub(crate) struct RecomputeDiff {pub prior_bit: bool,pub stage2_key: Stage2Key,settles_transfer_fallback: bool,+ prior_corrupt: bool,+ rewrites_corrupt: bool,}impl RecomputeDiff {@@ -521,9 +610,19 @@ impl RecomputeDiff {prior_bit: prior.in_cohort,stage2_key,settles_transfer_fallback: prior.ownership == Stage2Ownership::TransferredFallback,+ prior_corrupt: prior.corrupt,+ rewrites_corrupt: false,}}+ /// Rewrite an undecodable prior row even when the bit does not flip, so later reads stop failing+ /// on it. The seed path opts in, as its single-leaf diff does; reconcile does not, so its walk+ /// writes only what it must.+ pub(super) fn rewriting_corrupt(mut self) -> Self {+ self.rewrites_corrupt = self.prior_corrupt;+ self+ }+pub fn flipped(&self) -> bool {self.new_bit != self.prior_bit}@@ -539,7 +638,7 @@ impl RecomputeDiff {/// A receiver evaluation must rewrite a transferred fallback even when the logical bit is/// unchanged, making source provenance observably stale without touching ordinary no-flip rows.pub fn requires_write(&self) -> bool {- self.flipped() || self.settles_transfer_fallback+ self.flipped() || self.settles_transfer_fallback || self.rewrites_corrupt}}@@ -808,6 +907,8 @@ pub(super) fn decode_stage1_state(bytes: Option<Vec<u8>>) -> Option<Stage1State>pub(super) struct PriorStage2State {pub in_cohort: bool,ownership: Stage2Ownership,+ /// The row exists but did not decode.+ corrupt: bool,}/// Decode both the logical prior bit and its ownership. Missing or corrupt rows keep the existing@@ -820,10 +921,14 @@ pub(super) fn read_prior_stage2(bytes: Option<Vec<u8>>) -> PriorStage2State {Ok((state, ownership)) => PriorStage2State {in_cohort: state.in_cohort,ownership,+ corrupt: false,},Err(_) => {counter!(STAGE2_STATE_DECODE_ERROR).increment(1);- PriorStage2State::default()+ PriorStage2State {+ corrupt: true,+ ..PriorStage2State::default()+ }}}}@@ -2056,7 +2161,9 @@ mod tests {// Every test below holds `recompute_stage2_by_person` to the cohort-ordered path as an oracle// over the same store, then asserts what the shared read had to get right for the two to agree.- use crate::workers::stage2_person_inputs::{recompute_stage2_by_person, READ_CHUNK_KEYS};+ use crate::workers::stage2_person_inputs::{+ recompute_stage2_by_person, RecomposeGroup, READ_CHUNK_KEYS,+ };/// Run both recompute orders over the same store and return the agreed result. Neither commits,/// so running them back to back reads the same durable state twice.@@ -2083,7 +2190,10 @@ mod tests {&handle,TeamId(team_id),filters,- affected,+ &RecomposeGroup {+ leaves: affected.to_vec(),+ cohorts: Vec::new(),+ },EVENT_MS,TS,)@@ -2660,13 +2770,17 @@ mod tests {ReadLane::Maintenance,)};+ let group = RecomposeGroup {+ leaves: affected.clone(),+ cohorts: Vec::new(),+ };let run_by_person = || {recompute_stage2_by_person(PARTITION,&handle,TeamId(TEAM as i32),&filters,- &affected,+ &group,EVENT_MS,TS,)diff --git a/rust/cohort-stream-processor/src/workers/stage2_person_inputs.rs b/rust/cohort-stream-processor/src/workers/stage2_person_inputs.rsindex 1b0b5bf84f52..fa01c00321b2 100644--- a/rust/cohort-stream-processor/src/workers/stage2_person_inputs.rs+++ b/rust/cohort-stream-processor/src/workers/stage2_person_inputs.rs@@ -38,6 +38,14 @@ pub(super) const READ_CHUNK_KEYS: usize = 1024;const SECTION_OP: &str = "stage2_person_inputs";+/// One recompute call's input: the leaves whose every composed cohort recomposes, and single+/// composed cohorts admitted without their leaves.+#[derive(Debug, Default, PartialEq, Eq)]+pub(crate) struct RecomposeGroup {+ pub leaves: Vec<(LeafStateKey, Uuid)>,+ pub cohorts: Vec<(CohortId, Uuid)>,+}+/// Recompute one seed group's composable cohorts, sharing each person's reads across their cohorts.////// Seed-only by construction: the reads run through [`StoreHandle::run_section`], which draws the@@ -49,6 +57,9 @@ const SECTION_OP: &str = "stage2_person_inputs";/// `team_id` is the key `filters` sit under in the frozen catalog, so every cohort here composes/// from that team's keyspace.///+/// `group.leaves` recompose every composed cohort on each leaf; `group.cohorts` recompose only the+/// named cohort, for a cohort whose row was missing. A cohort reached both ways evaluates once.+////// Changes and writes come back in `(cohort, person)` order, matching the cohort-ordered path.////// A read failure returns before any output exists, so the held run publishes nothing partial.@@ -57,12 +68,12 @@ pub(crate) async fn recompute_stage2_by_person(handle: &StoreHandle,team_id: TeamId,filters: &TeamFilters,- affected_leaves: &[(LeafStateKey, Uuid)],+ group: &RecomposeGroup,event_ms: i64,last_updated: &str,) -> Result<Stage2Recompute, StoreError> {let mut affected: BTreeMap<Uuid, BTreeSet<CohortId>> = BTreeMap::new();- for &(leaf_state_key, person_id) in affected_leaves {+ for &(leaf_state_key, person_id) in &group.leaves {if let Some(cohorts) = filters.by_lsk_to_composable_cohorts.get(&leaf_state_key) {affected.entry(person_id)@@ -70,6 +81,13 @@ pub(crate) async fn recompute_stage2_by_person(.extend(cohorts.iter().copied());}}+ // Only the cohorts no leaf reached are repairs; one a leaf reached recomposes for that reason.+ let mut repairs: BTreeSet<(CohortId, Uuid)> = BTreeSet::new();+ for &(cohort_id, person_id) in &group.cohorts {+ if affected.entry(person_id).or_default().insert(cohort_id) {+ repairs.insert((cohort_id, person_id));+ }+ }// A map rather than pushes. No consumer needs cohort order (changes are keyed by person and// writes by row), but the differential tests compare this path with the cohort-ordered one@@ -120,7 +138,8 @@ pub(crate) async fn recompute_stage2_by_person(new_bit,inputs.prior_stage2(cohort_id),scope.stage2_key(cohort_id),- ),+ )+ .rewriting_corrupt(),);}}@@ -137,6 +156,9 @@ pub(crate) async fn recompute_stage2_by_person(event_ms,last_updated,);+ if repairs.contains(&(cohort_id, person_id)) {+ recompute.record_composed_repair(&diff);+ }}Ok(recompute)}