Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(cohorts): recompose composed cohorts whose stage 2 row is missing on seed apply (#121092)

Harley Alexander4ecd79bd56c6
diff --git a/docs/internal/realtime-cohorts/invariants.md b/docs/internal/realtime-cohorts/invariants.md
index ee1a2a0c69f8..b020d1c0d28b 100644
--- a/docs/internal/realtime-cohorts/invariants.md
+++ b/docs/internal/realtime-cohorts/invariants.md
@@ -265,6 +265,7 @@ It does not rule out duplicates: a store that loses unflushed writes, or is wipe
Reconcile repairs lost output, not lost state.
It re-tells the rows the store holds.
An event that Stage 1 skipped needs a later seed that covers it, and a person whose first Stage 2 row was never written needs a seed or a leaf flip that touches them.
+A person seed that changes nothing still recomposes a composed cohort with no row for the person, so a cohort added over already-seeded leaves gets its rows from its own run.
The system is built on one bargain: live paths favor speed and may lose a change, and every backfill ends with a reconcile that re-tells everything the store holds.
Anything that stops reconcile from running, or from covering a cohort, leaves downstream wrong with no signal.
diff --git a/docs/internal/realtime-cohorts/seed-apply-and-reconcile.md b/docs/internal/realtime-cohorts/seed-apply-and-reconcile.md
index 0ee3b8b1780a..b6ffbc92b028 100644
--- a/docs/internal/realtime-cohorts/seed-apply-and-reconcile.md
+++ b/docs/internal/realtime-cohorts/seed-apply-and-reconcile.md
@@ -131,6 +131,11 @@ Both kinds finish the same way, in an order the code enforces with types.
One write batch holds the new leaf rows or person records and, for each single-leaf cohort about to be emitted, a Stage 2 row set to the value being **replaced**.
3. **Schedule** eviction deadlines for the changed leaf rows.
4. **Recompose** composed cohorts, reading each person's state once for all their cohorts, and decide their emissions.
+ A person is recomposed when the fold changed their state, or when a single-leaf row for them lagged in step 1.
+ For a person seed, a composed cohort over one of their evaluated leaves that has no Stage 2 row for them also recomposes, on its own, without the other cohorts on its leaves.
+ That case covers a cohort added over leaves an earlier run already seeded: the seed changes nothing, so without it the cohort never gets a row, and its reconcile certifies it with no members.
+ A non-member's row stays absent, so a cohort the run's own leaf answers already make false is skipped without a read, and only the cohort recomposes, not its leaves.
+ The seed path also rewrites an unreadable Stage 2 row with a readable bit when it recomposes the cohort, as the single-leaf diff does.
5. **Produce**, in parallel: the membership changes tagged `origin: seed` with the run id, first-hop cascades if enabled, and re-keyed seeds for merged persons.
6. **Stage 2 commit.**
Write the new truth into the Stage 2 rows.
@@ -198,7 +203,7 @@ Reconcile re-tells what the store holds, and it cannot bring back what the store
An event that Stage 1 skipped on a store error is missing from the leaf state, so the walk recomputes the same wrong answer.
A person whose first Stage 2 row for the cohort was never written, for example after a crash between a Stage 1 batch and its Stage 2 batch, is not in the walk at all.
Seeds from a later run that cover the missed event repair the first case.
-A seed or a leaf flip that touches the person repairs the second.
+A seed or a leaf flip that touches the person repairs the second: a person seed recomposes a composed cohort that has no row for the person even when it changes nothing.
### Why every row is emitted
diff --git a/rust/cohort-stream-processor/src/observability/metrics.rs b/rust/cohort-stream-processor/src/observability/metrics.rs
index 8ec201630f3c..a006b3092ebd 100644
--- a/rust/cohort-stream-processor/src/observability/metrics.rs
+++ b/rust/cohort-stream-processor/src/observability/metrics.rs
@@ -627,6 +627,12 @@ pub const PERSON_SEED_REKEY_PRODUCE_FAILURE_TOTAL: &str =
/// produce-failure counters before concluding produces are failing.**
pub const SEED_REGISTER_REPAIRS_TOTAL: &str = "cohort_seed_register_repairs_total";
+/// Composed-cohort membership changes a person seed apply emitted for a cohort it recomposed only
+/// because the cohort had no row for the person, labelled by `kind` (`entered`|`left`) (counter).
+/// Counted when the writes commit, like [`SEED_REGISTER_REPAIRS_TOTAL`]. Expect these in bulk on the
+/// first run of a composed cohort whose leaves earlier runs already seeded, and rarely after.
+pub const SEED_COMPOSED_ROW_REPAIRS_TOTAL: &str = "cohort_seed_composed_row_repairs_total";
+
/// Persons whose Stage 2 inputs a seed apply read through shared store sections (counter).
/// Attempt-based: a held run counts its persons, and so does the redelivery that replays it.
/// **Do not divide [`STAGE2_COHORTS_EVALUATED`] by this**, because that counter is settled-based and
@@ -1100,6 +1106,10 @@ mod tests {
SEED_REGISTER_REPAIRS_TOTAL,
"cohort_seed_register_repairs_total"
);
+ assert_eq!(
+ SEED_COMPOSED_ROW_REPAIRS_TOTAL,
+ "cohort_seed_composed_row_repairs_total"
+ );
assert_eq!(
SEED_RECOMPUTE_PERSONS_TOTAL,
"cohort_seed_recompute_persons_total"
diff --git a/rust/cohort-stream-processor/src/stage2/evaluator.rs b/rust/cohort-stream-processor/src/stage2/evaluator.rs
index 1b01ced82f97..78db13867250 100644
--- a/rust/cohort-stream-processor/src/stage2/evaluator.rs
+++ b/rust/cohort-stream-processor/src/stage2/evaluator.rs
@@ -84,6 +84,32 @@ pub fn evaluate_tree(
}
}
+/// The cohort's membership as far as `known` decides it, or `None` when the answer depends on a leaf
+/// `known` does not hold or on a cohort reference. Agrees with [`evaluate_tree`] wherever it returns
+/// `Some`, since that reads a leaf `known` lacks as whatever the store holds.
+pub fn evaluate_known(node: &FilterNode, known: &HashMap<LeafStateKey, bool>) -> Option<bool> {
+ match node {
+ FilterNode::Group { op, children } => {
+ let mut undecided = false;
+ for child in children {
+ match (op, evaluate_known(child, known)) {
+ (BoolOp::And, Some(false)) => return Some(false),
+ (BoolOp::Or, Some(true)) => return Some(true),
+ (_, None) => undecided = true,
+ _ => {}
+ }
+ }
+ (!undecided).then_some(*op == BoolOp::And)
+ }
+ FilterNode::Leaf(CohortLeaf::CohortRef(_)) => None,
+ FilterNode::Leaf(leaf) => match leaf.leaf_state_key() {
+ Some(lsk) => known.get(&lsk).map(|bit| bit ^ leaf.negated()),
+ // [`evaluate_tree`] reads a leaf with no state key as `false` whatever is stored.
+ None => Some(leaf.negated()),
+ },
+ }
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -127,6 +153,53 @@ mod tests {
FilterNode::Group { op, children }
}
+ #[test]
+ fn evaluate_known_decides_only_what_the_known_leaves_force() {
+ let known = HashMap::from([(lsk(1), true), (lsk(2), false)]);
+ for (why, tree, want) in [
+ (
+ "a false leaf decides an AND",
+ group(BoolOp::And, vec![person_leaf(lsk(2)), person_leaf(lsk(3))]),
+ Some(false),
+ ),
+ (
+ "an unknown leaf leaves an AND open",
+ group(BoolOp::And, vec![person_leaf(lsk(1)), person_leaf(lsk(3))]),
+ None,
+ ),
+ (
+ "a true leaf decides an OR",
+ group(BoolOp::Or, vec![person_leaf(lsk(1)), person_leaf(lsk(3))]),
+ Some(true),
+ ),
+ (
+ "an unknown leaf leaves an OR open",
+ group(BoolOp::Or, vec![person_leaf(lsk(2)), person_leaf(lsk(3))]),
+ None,
+ ),
+ (
+ "negation applies to a known leaf",
+ group(BoolOp::And, vec![person_leaf_neg(lsk(1), true)]),
+ Some(false),
+ ),
+ (
+ "a cohort reference is never known",
+ group(BoolOp::And, vec![person_leaf(lsk(1)), cohort_ref_leaf()]),
+ None,
+ ),
+ (
+ "all known leaves decide",
+ group(
+ BoolOp::And,
+ vec![person_leaf(lsk(1)), person_leaf_neg(lsk(2), true)],
+ ),
+ Some(true),
+ ),
+ ] {
+ assert_eq!(evaluate_known(&tree, &known), want, "{why}");
+ }
+ }
+
fn classifier_excludes_top_level_negation(root: &FilterNode) -> bool {
fn state_keyed_leaf_count(node: &FilterNode) -> u32 {
match node {
diff --git a/rust/cohort-stream-processor/src/workers/person_seed_path.rs b/rust/cohort-stream-processor/src/workers/person_seed_path.rs
index a1da25c29878..dd2b22a5842e 100644
--- a/rust/cohort-stream-processor/src/workers/person_seed_path.rs
+++ b/rust/cohort-stream-processor/src/workers/person_seed_path.rs
@@ -322,6 +322,7 @@ fn fold_person_seeds(
let mut records = StagedBatch::default();
let mut leaves: BTreeMap<TeamId, Vec<FoldedLeaf>> = BTreeMap::new();
+ let mut composed_row_candidates: BTreeMap<TeamId, Vec<FoldedLeaf>> = BTreeMap::new();
for (key, touch) in &touches {
let slot = overlay
.slot(key)
@@ -354,13 +355,21 @@ fn fold_person_seeds(
.before()
.is_some_and(|record| record.matched.contains(hash));
let minted_transition = before != in_cohort;
- leaves.entry(touch.team_id).or_default().push(FoldedLeaf {
- leaf_state_key: LeafStateKey::for_person_property(hash),
+ let leaf_state_key = LeafStateKey::for_person_property(hash);
+ let folded = FoldedLeaf {
+ leaf_state_key,
person_id: touch.person,
in_cohort,
minted_transition,
run_id,
- });
+ };
+ leaves.entry(touch.team_id).or_default().push(folded);
+ if !touch.recompose.contains(&leaf_state_key) {
+ composed_row_candidates
+ .entry(touch.team_id)
+ .or_default()
+ .push(folded);
+ }
if minted_transition {
let kind = if in_cohort {
TransitionKind::Entered
@@ -385,6 +394,7 @@ fn fold_person_seeds(
records,
leaves,
recompose,
+ composed_row_candidates,
// Nothing to schedule: person-property membership has no window, so the sweep never owns
// these leaves.
schedules: Vec::new(),
@@ -416,10 +426,12 @@ impl RecordUpdate {
/// Whether stage 2 has to be recomposed for this seed.
///
-/// A merge that changed nothing leaves stage 2 already consistent with stage 1, and skipping the
-/// recompose keeps a non-matching dormant person to a single point read instead of a composition
-/// over every cohort its hashes touch — the dominant message shape of a scan that emits
-/// non-matchers. Two outcomes still have to recompose:
+/// A merge that changed nothing leaves stage 2 consistent with stage 1 for every cohort that already
+/// has a row; a composed cohort without one is the register diff's to admit, see
+/// [`crate::workers::stage2_path::composed_rows_missing`]. Skipping the recompose keeps a
+/// non-matching dormant person to a single point read instead of a composition over every cohort its
+/// hashes touch — the dominant message shape of a scan that emits non-matchers. Two outcomes still
+/// have to recompose:
///
/// - `SkipLiveFresh` may be the redelivery of an attempt that committed stage 1 and then failed its
/// produce. The replay merges to `Unchanged` and mints no transition, and a live event that
@@ -844,6 +856,18 @@ mod tests {
.unwrap();
}
+ fn put_raw_stage2(&self, partition_id: u16, person: Uuid, cohort_id: u64, bytes: &[u8]) {
+ let key = Stage2Key {
+ partition_id,
+ team_id: TEAM.0 as u64,
+ cohort_id,
+ person_id: person,
+ };
+ self.store
+ .write_batch(|b| b.put_stage2(&key, bytes))
+ .unwrap();
+ }
+
fn stage2(&self, partition_id: u16, person: Uuid, cohort_id: u64) -> Option<Stage2State> {
self.store
.get_stage2(&Stage2Key {
@@ -1843,6 +1867,82 @@ mod tests {
assert_eq!(shell.committable(partition_id), Some(2));
}
+ /// The composed half of the same case. The seed asserts what the record already holds and no
+ /// single-leaf register lags, so only the missing composed row can admit the recompose; without
+ /// it the cohort's backfill completes with no members.
+ #[tokio::test]
+ async fn a_composed_cohort_added_over_an_existing_record_composes_on_the_next_unchanged_seed() {
+ // (member, row before, change, row after)
+ for (member, corrupt_row, expected_change, expected_row) in [
+ (true, false, Some(MembershipStatus::Entered), Some(true)),
+ // A non-member's row stays absent, so the next run reads it again and writes nothing.
+ (false, false, None, None),
+ // An unreadable row is rewritten with a readable bit even when nothing flips.
+ (false, true, None, Some(false)),
+ ] {
+ let (person, partition_id) = dormant_person();
+ let single_leaf = vec![
+ (1, wrap(vec![person_leaf(PERSON_HASH)])),
+ (4, wrap(vec![behavioral_leaf()])),
+ ];
+ let mut shell = Shell::new(single_leaf.clone());
+ if member {
+ shell.live_pageview(partition_id, person, None);
+ }
+ // An earlier run's result, stamped old enough that this seed applies as an ordinary
+ // Unchanged merge rather than a live-fresh skip.
+ shell.put_record(
+ partition_id,
+ person,
+ &PersonRecord {
+ last_seen_ms: now_ms(),
+ stamp: Stamp::new(now_ms() - 2 * MARGIN_MS, 0),
+ props_fingerprint: PropsFingerprint::of(r#"{"email":"[email protected]"}"#),
+ catalog_fingerprint: shell.filters.catalog_fingerprint,
+ matched: MatchedSet::from_iter([hash(PERSON_HASH).as_bytes()]),
+ applied_offsets: AppliedOffsets::default(),
+ redirect_dedup: Default::default(),
+ },
+ );
+ shell.put_register(partition_id, person, 1, true);
+ if corrupt_row {
+ shell.put_raw_stage2(partition_id, person, 2, b"not a stage 2 row");
+ }
+
+ // Cohort 2 composes both leaves, neither of them new to the store.
+ let mut cohorts = single_leaf;
+ cohorts.push((2, wrap(vec![person_leaf(PERSON_HASH), behavioral_leaf()])));
+ shell.catalog = Arc::new(CatalogHandle::from_catalog(FilterCatalog::from_teams([(
+ TEAM,
+ build_filters(&cohorts),
+ )])));
+ let seed = seed_for(person, &[PERSON_HASH], &[PERSON_HASH], now_ms());
+ shell.run(partition_id, &seed, 0).await;
+
+ let case = format!("member={member} corrupt_row={corrupt_row}");
+ assert_eq!(
+ shell
+ .sink
+ .changes()
+ .iter()
+ .map(|change| (change.cohort_id, change.status))
+ .collect::<Vec<_>>(),
+ expected_change
+ .map(|status| vec![(2, status)])
+ .unwrap_or_default(),
+ "{case}",
+ );
+ assert_eq!(
+ shell
+ .stage2(partition_id, person, 2)
+ .map(|state| state.in_cohort),
+ expected_row,
+ "{case}",
+ );
+ assert_eq!(shell.committable(partition_id), Some(1), "{case}");
+ }
+ }
+
#[tokio::test]
async fn a_seeded_dormant_person_reaches_the_same_membership_as_a_live_evaluated_one() {
let live_person = Uuid::from_u128(0xA11CE);
diff --git a/rust/cohort-stream-processor/src/workers/seed_apply.rs b/rust/cohort-stream-processor/src/workers/seed_apply.rs
index 71883dce9196..47642d515dbf 100644
--- a/rust/cohort-stream-processor/src/workers/seed_apply.rs
+++ b/rust/cohort-stream-processor/src/workers/seed_apply.rs
@@ -26,7 +26,7 @@ use tracing::warn;
use uuid::Uuid;
use crate::filters::reverse_index::TeamFilters;
-use crate::filters::{FilterCatalog, TeamId};
+use crate::filters::{CohortId, FilterCatalog, TeamId};
use crate::merge::tombstone_redirect::{self, Resolution};
use crate::observability::metrics::{
COHORT_STREAM_OFFSET_AHEAD_OF_DISPATCH, OUTPUT_TRANSITIONS_UNMAPPED,
@@ -49,9 +49,10 @@ use crate::workers::merge_path::MergeWorkerDeps;
use crate::workers::reconcile::{ReconcileQueue, SupersedeOutcome};
use crate::workers::seed_run::{Admitted, OffsetSpan, SeedGroup, SeedKind, SeedOffset, SeedRun};
use crate::workers::stage2_path::{
- commit_stage2_writes, diff_single_leaf_registers, FoldedLeaf, RegisterDiff, Stage2Recompute,
+ commit_stage2_writes, composed_rows_missing, diff_single_leaf_registers, FoldedLeaf,
+ RegisterDiff, Stage2Recompute,
};
-use crate::workers::stage2_person_inputs::recompute_stage2_by_person;
+use crate::workers::stage2_person_inputs::{recompute_stage2_by_person, RecomposeGroup};
use crate::workers::worker::{
first_cascades, produce_cascades, produce_membership, transition_metric_label,
};
@@ -447,6 +448,8 @@ pub(crate) struct TouchedPersons {
struct PersonTouch {
run: RunId,
leaves: BTreeSet<LeafStateKey>,
+ /// Composed cohorts to recompose on their own, without the other cohorts on their leaves.
+ cohorts: BTreeSet<CohortId>,
}
impl TouchedPersons {
@@ -456,6 +459,7 @@ impl TouchedPersons {
.or_insert_with(|| PersonTouch {
run,
leaves: BTreeSet::new(),
+ cohorts: BTreeSet::new(),
})
}
@@ -479,17 +483,34 @@ impl TouchedPersons {
self.entry(team_id, person, run).leaves.insert(leaf);
}
+ /// Admit one composed cohort whose row is missing. Like a lagging leaf, its run counts only for
+ /// a person the fold never admitted.
+ pub(crate) fn touch_missing(
+ &mut self,
+ team_id: TeamId,
+ person: Uuid,
+ run: RunId,
+ cohort: CohortId,
+ ) {
+ self.entry(team_id, person, run).cohorts.insert(cohort);
+ }
+
fn is_empty(&self) -> bool {
self.per_person.is_empty()
}
/// One recompute call per `(team, run)`: the team owns the catalog that composes the leaves,
/// the run owns their provenance tag.
- fn groups(self) -> BTreeMap<(TeamId, RunId), Vec<(LeafStateKey, Uuid)>> {
- let mut groups: BTreeMap<(TeamId, RunId), Vec<(LeafStateKey, Uuid)>> = BTreeMap::new();
+ fn groups(self) -> BTreeMap<(TeamId, RunId), RecomposeGroup> {
+ let mut groups: BTreeMap<(TeamId, RunId), RecomposeGroup> = BTreeMap::new();
for ((team_id, person), touch) in self.per_person {
let group = groups.entry((team_id, touch.run)).or_default();
- group.extend(touch.leaves.into_iter().map(|leaf| (leaf, person)));
+ group
+ .leaves
+ .extend(touch.leaves.into_iter().map(|leaf| (leaf, person)));
+ group
+ .cohorts
+ .extend(touch.cohorts.into_iter().map(|cohort| (cohort, person)));
}
groups
}
@@ -685,6 +706,10 @@ pub(crate) struct Folded {
pub leaves: BTreeMap<TeamId, Vec<FoldedLeaf>>,
/// Persons whose composable cohorts must recompose, with the run that last touched them.
pub recompose: TouchedPersons,
+ /// Folded leaves the fold did not recompose, whose composed cohorts may still lack a row for the
+ /// person; see [`composed_rows_missing`]. Only person runs fill it: a tile run recomposes every
+ /// tile it folds.
+ pub composed_row_candidates: BTreeMap<TeamId, Vec<FoldedLeaf>>,
/// Final eviction deadline per touched behavioral key. Empty for person seeds.
pub schedules: Vec<(BehavioralKey, i64)>,
pub re_keys: ReKeys,
@@ -700,6 +725,7 @@ impl Folded {
records: StagedBatch::default(),
leaves: BTreeMap::new(),
recompose: TouchedPersons::default(),
+ composed_row_candidates: BTreeMap::new(),
schedules: Vec::new(),
re_keys,
tally,
@@ -708,7 +734,8 @@ impl Folded {
/// One [`diff_single_leaf_registers`] per team. Every person the diff emits a change or a write
/// for joins `recompose`: a lagging register is its own reason to recompose, and nothing the
- /// fold saw would admit it — the redelivery of a failed produce mints no transition.
+ /// fold saw would admit it — the redelivery of a failed produce mints no transition. A composed
+ /// cohort with no row for a person the run evaluated joins too, see [`composed_rows_missing`].
async fn diff_registers(
mut self,
deps: ApplyDeps<'_>,
@@ -750,6 +777,21 @@ impl Folded {
);
}
}
+ if let Some(candidates) = self.composed_row_candidates.get(&team_id) {
+ let missing = composed_rows_missing(
+ deps.partition_id,
+ deps.handle,
+ filters,
+ candidates,
+ leaves,
+ ReadLane::Maintenance,
+ )
+ .await
+ .map_err(SeedHold::store(ApplyStage::RegisterDiff))?;
+ for ((person, cohort), run) in missing {
+ self.recompose.touch_missing(team_id, person, run, cohort);
+ }
+ }
diff.extend(team_diff);
}
@@ -851,7 +893,7 @@ impl Scheduled {
stamp: &RunStamp,
) -> Result<Recomputed, SeedHold> {
if !self.recompose.is_empty() {
- for ((team_id, run), leaves) in self.recompose.groups() {
+ for ((team_id, run), group) in self.recompose.groups() {
let Some(filters) = deps.team(team_id) else {
continue;
};
@@ -860,7 +902,7 @@ impl Scheduled {
deps.handle,
team_id,
filters,
- &leaves,
+ &group,
stamp.now_ms,
&stamp.last_updated,
)
@@ -1364,11 +1406,12 @@ mod tests {
touched.touch(team, alice, first, LeafStateKey([1; 16]));
touched.touch(team, alice, last, LeafStateKey([2; 16]));
touched.touch_lagging(team, alice, first, LeafStateKey([3; 16]));
+ touched.touch_missing(team, alice, first, CohortId(9));
let groups = touched.groups();
assert_eq!(groups.len(), 1, "one call, not one per seed");
assert_eq!(
- groups[&(team, last)],
+ groups[&(team, last)].leaves,
vec![
(LeafStateKey([1; 16]), alice),
(LeafStateKey([2; 16]), alice),
@@ -1376,5 +1419,10 @@ mod tests {
],
"every leaf rides the last seed's run; a lagging leaf joins without moving it",
);
+ assert_eq!(
+ groups[&(team, last)].cohorts,
+ vec![(CohortId(9), alice)],
+ "a cohort with a missing row joins the same call, without its leaves",
+ );
}
}
diff --git a/rust/cohort-stream-processor/src/workers/seed_path.rs b/rust/cohort-stream-processor/src/workers/seed_path.rs
index 44d9837a09fe..03e227dee8d3 100644
--- a/rust/cohort-stream-processor/src/workers/seed_path.rs
+++ b/rust/cohort-stream-processor/src/workers/seed_path.rs
@@ -798,6 +798,7 @@ fn fold_tiles(
records,
leaves,
recompose,
+ composed_row_candidates: BTreeMap::new(),
schedules,
re_keys: ReKeys::Tiles(re_keys),
tally,
diff --git a/rust/cohort-stream-processor/src/workers/stage2_path.rs b/rust/cohort-stream-processor/src/workers/stage2_path.rs
index a50c82f7a615..7297379c4354 100644
--- a/rust/cohort-stream-processor/src/workers/stage2_path.rs
+++ b/rust/cohort-stream-processor/src/workers/stage2_path.rs
@@ -15,14 +15,14 @@ use crate::filters::reverse_index::TeamFilters;
use crate::filters::tree::{CohortLeaf, CohortTree, FilterNode};
use crate::filters::CohortId;
use crate::observability::metrics::{
- SEED_REGISTER_REPAIRS_TOTAL, STAGE2_COHORTS_EVALUATED, STAGE2_STATE_DECODE_ERROR,
- STAGE2_TRANSITIONS,
+ SEED_COMPOSED_ROW_REPAIRS_TOTAL, SEED_REGISTER_REPAIRS_TOTAL, STAGE2_COHORTS_EVALUATED,
+ STAGE2_STATE_DECODE_ERROR, STAGE2_TRANSITIONS,
};
use crate::producer::{ChangeOrigin, CohortMembershipChange, MembershipStatus};
use crate::stage1::key::LeafStateKey;
use crate::stage1::person_record::PersonRecord;
use crate::stage1::state::{Stage1State, StateVariant, StatefulRecord};
-use crate::stage2::evaluator::{evaluate_tree, leaf_membership};
+use crate::stage2::evaluator::{evaluate_known, evaluate_tree, leaf_membership};
use crate::stage2::state::{Stage2Ownership, Stage2State};
use crate::store::{
BehavioralKey, PersonRecordKey, ReadLane, Stage2Key, StagedBatch, StoreError, StoreHandle,
@@ -71,6 +71,8 @@ pub(crate) struct Stage2Recompute {
composed: StatusCounts,
/// Single-leaf changes a register diff derived with no stage-1 transition behind them.
repairs: RepairCounts,
+ /// Composed flips of cohorts recomposed only because their row was missing.
+ composed_repairs: StatusCounts,
}
impl Stage2Recompute {
@@ -79,6 +81,8 @@ impl Stage2Recompute {
counter!(STAGE2_COHORTS_EVALUATED).increment(self.evaluated);
self.composed.record(STAGE2_TRANSITIONS);
self.repairs.record(SEED_REGISTER_REPAIRS_TOTAL);
+ self.composed_repairs
+ .record(SEED_COMPOSED_ROW_REPAIRS_TOTAL);
}
/// Fold a sibling recompute in, so one produce and one commit cover both halves.
@@ -88,6 +92,15 @@ impl Stage2Recompute {
self.evaluated += other.evaluated;
self.composed.add(other.composed);
self.repairs.add(other.repairs);
+ self.composed_repairs.add(other.composed_repairs);
+ }
+
+ /// Count a flip of a cohort recomposed only because its row was missing, see
+ /// [`composed_rows_missing`]. Its own call, because the pair already went through `record_pair`.
+ pub(super) fn record_composed_repair(&mut self, diff: &RecomputeDiff) {
+ if diff.flipped() {
+ self.composed_repairs.count(diff.status());
+ }
}
/// Both recompute orders call this, so what a composed evaluation emits cannot depend on
@@ -444,6 +457,79 @@ pub(crate) async fn diff_single_leaf_registers(
})
}
+/// The composed cohorts keyed on `candidates`' leaves that hold no readable stage-2 row for the
+/// person, each with the run of the first leaf that reached it.
+///
+/// A composed cohort recomposes only when a seed touches one of its leaves, and a person seed that
+/// asserts what the record already holds touches nothing. A cohort added over leaves an earlier run
+/// already seeded would then never get a row, and reconcile, which walks only existing rows, would
+/// certify it with no members. So a missing row is its own reason to recompose that cohort, as a
+/// lagging register is for the single-leaf half.
+///
+/// `candidates` are the leaves the fold did not recompose. `folded` is every leaf the run decided,
+/// whose truths make a cohort definitely false without a read: an absent row is already right for
+/// it, and a never-member's row stays absent, so recomposing it would repeat on every later run.
+///
+/// One batched `cf_stage2` read on `lane`. Leaves with no composed cohort keyed on them cost nothing.
+pub(crate) async fn composed_rows_missing(
+ partition_id: u16,
+ handle: &StoreHandle,
+ filters: &TeamFilters,
+ candidates: &[FoldedLeaf],
+ folded: &[FoldedLeaf],
+ lane: ReadLane,
+) -> Result<BTreeMap<(Uuid, CohortId), RunId>, StoreError> {
+ let mut known: HashMap<Uuid, HashMap<LeafStateKey, bool>> = HashMap::new();
+ for leaf in folded {
+ known
+ .entry(leaf.person_id)
+ .or_default()
+ .insert(leaf.leaf_state_key, leaf.in_cohort);
+ }
+
+ let mut wanted: BTreeMap<Stage2Key, (Uuid, CohortId, RunId)> = BTreeMap::new();
+ for leaf in candidates {
+ let Some(cohort_ids) = filters
+ .by_lsk_to_composable_cohorts
+ .get(&leaf.leaf_state_key)
+ else {
+ continue;
+ };
+ for cohort_id in cohort_ids {
+ let Some(tree) = filters.cohorts.get(cohort_id) else {
+ continue;
+ };
+ let truths = known.get(&leaf.person_id);
+ if truths.is_some_and(|truths| evaluate_known(&tree.root, truths) == Some(false)) {
+ continue;
+ }
+ let key = Stage2Key {
+ partition_id,
+ team_id: tree.team_id.0 as u64,
+ cohort_id: cohort_id.0 as u64,
+ person_id: leaf.person_id,
+ };
+ wanted
+ .entry(key)
+ .or_insert((leaf.person_id, *cohort_id, leaf.run_id));
+ }
+ }
+ if wanted.is_empty() {
+ return Ok(BTreeMap::new());
+ }
+
+ let keys: Vec<Stage2Key> = wanted.keys().copied().collect();
+ let stored = handle.multi_get_stage2(keys, lane).await?;
+
+ let mut missing = BTreeMap::new();
+ for ((person_id, cohort_id, run_id), bytes) in wanted.into_values().zip(stored) {
+ if !matches!(read_register(bytes), StoredRegister::Present(_)) {
+ missing.insert((person_id, cohort_id), run_id);
+ }
+ }
+ Ok(missing)
+}
+
/// One `(single-leaf cohort, person)` the fold decided, before its stored row is read.
#[derive(Clone, Copy, PartialEq, Eq)]
struct WantedRegister {
@@ -481,6 +567,7 @@ fn read_register(bytes: Option<Vec<u8>>) -> StoredRegister {
Ok((state, ownership)) => StoredRegister::Present(PriorStage2State {
in_cohort: state.in_cohort,
ownership,
+ corrupt: false,
}),
Err(_) => {
counter!(STAGE2_STATE_DECODE_ERROR).increment(1);
@@ -512,6 +599,8 @@ pub(crate) struct RecomputeDiff {
pub prior_bit: bool,
pub stage2_key: Stage2Key,
settles_transfer_fallback: bool,
+ prior_corrupt: bool,
+ rewrites_corrupt: bool,
}
impl RecomputeDiff {
@@ -521,9 +610,19 @@ impl RecomputeDiff {
prior_bit: prior.in_cohort,
stage2_key,
settles_transfer_fallback: prior.ownership == Stage2Ownership::TransferredFallback,
+ prior_corrupt: prior.corrupt,
+ rewrites_corrupt: false,
}
}
+ /// Rewrite an undecodable prior row even when the bit does not flip, so later reads stop failing
+ /// on it. The seed path opts in, as its single-leaf diff does; reconcile does not, so its walk
+ /// writes only what it must.
+ pub(super) fn rewriting_corrupt(mut self) -> Self {
+ self.rewrites_corrupt = self.prior_corrupt;
+ self
+ }
+
pub fn flipped(&self) -> bool {
self.new_bit != self.prior_bit
}
@@ -539,7 +638,7 @@ impl RecomputeDiff {
/// A receiver evaluation must rewrite a transferred fallback even when the logical bit is
/// unchanged, making source provenance observably stale without touching ordinary no-flip rows.
pub fn requires_write(&self) -> bool {
- self.flipped() || self.settles_transfer_fallback
+ self.flipped() || self.settles_transfer_fallback || self.rewrites_corrupt
}
}
@@ -808,6 +907,8 @@ pub(super) fn decode_stage1_state(bytes: Option<Vec<u8>>) -> Option<Stage1State>
pub(super) struct PriorStage2State {
pub in_cohort: bool,
ownership: Stage2Ownership,
+ /// The row exists but did not decode.
+ corrupt: bool,
}
/// Decode both the logical prior bit and its ownership. Missing or corrupt rows keep the existing
@@ -820,10 +921,14 @@ pub(super) fn read_prior_stage2(bytes: Option<Vec<u8>>) -> PriorStage2State {
Ok((state, ownership)) => PriorStage2State {
in_cohort: state.in_cohort,
ownership,
+ corrupt: false,
},
Err(_) => {
counter!(STAGE2_STATE_DECODE_ERROR).increment(1);
- PriorStage2State::default()
+ PriorStage2State {
+ corrupt: true,
+ ..PriorStage2State::default()
+ }
}
}
}
@@ -2056,7 +2161,9 @@ mod tests {
// Every test below holds `recompute_stage2_by_person` to the cohort-ordered path as an oracle
// over the same store, then asserts what the shared read had to get right for the two to agree.
- use crate::workers::stage2_person_inputs::{recompute_stage2_by_person, READ_CHUNK_KEYS};
+ use crate::workers::stage2_person_inputs::{
+ recompute_stage2_by_person, RecomposeGroup, READ_CHUNK_KEYS,
+ };
/// Run both recompute orders over the same store and return the agreed result. Neither commits,
/// so running them back to back reads the same durable state twice.
@@ -2083,7 +2190,10 @@ mod tests {
&handle,
TeamId(team_id),
filters,
- affected,
+ &RecomposeGroup {
+ leaves: affected.to_vec(),
+ cohorts: Vec::new(),
+ },
EVENT_MS,
TS,
)
@@ -2660,13 +2770,17 @@ mod tests {
ReadLane::Maintenance,
)
};
+ let group = RecomposeGroup {
+ leaves: affected.clone(),
+ cohorts: Vec::new(),
+ };
let run_by_person = || {
recompute_stage2_by_person(
PARTITION,
&handle,
TeamId(TEAM as i32),
&filters,
- &affected,
+ &group,
EVENT_MS,
TS,
)
diff --git a/rust/cohort-stream-processor/src/workers/stage2_person_inputs.rs b/rust/cohort-stream-processor/src/workers/stage2_person_inputs.rs
index 1b0b5bf84f52..fa01c00321b2 100644
--- a/rust/cohort-stream-processor/src/workers/stage2_person_inputs.rs
+++ b/rust/cohort-stream-processor/src/workers/stage2_person_inputs.rs
@@ -38,6 +38,14 @@ pub(super) const READ_CHUNK_KEYS: usize = 1024;
const SECTION_OP: &str = "stage2_person_inputs";
+/// One recompute call's input: the leaves whose every composed cohort recomposes, and single
+/// composed cohorts admitted without their leaves.
+#[derive(Debug, Default, PartialEq, Eq)]
+pub(crate) struct RecomposeGroup {
+ pub leaves: Vec<(LeafStateKey, Uuid)>,
+ pub cohorts: Vec<(CohortId, Uuid)>,
+}
+
/// Recompute one seed group's composable cohorts, sharing each person's reads across their cohorts.
///
/// Seed-only by construction: the reads run through [`StoreHandle::run_section`], which draws the
@@ -49,6 +57,9 @@ const SECTION_OP: &str = "stage2_person_inputs";
/// `team_id` is the key `filters` sit under in the frozen catalog, so every cohort here composes
/// from that team's keyspace.
///
+/// `group.leaves` recompose every composed cohort on each leaf; `group.cohorts` recompose only the
+/// named cohort, for a cohort whose row was missing. A cohort reached both ways evaluates once.
+///
/// Changes and writes come back in `(cohort, person)` order, matching the cohort-ordered path.
///
/// A read failure returns before any output exists, so the held run publishes nothing partial.
@@ -57,12 +68,12 @@ pub(crate) async fn recompute_stage2_by_person(
handle: &StoreHandle,
team_id: TeamId,
filters: &TeamFilters,
- affected_leaves: &[(LeafStateKey, Uuid)],
+ group: &RecomposeGroup,
event_ms: i64,
last_updated: &str,
) -> Result<Stage2Recompute, StoreError> {
let mut affected: BTreeMap<Uuid, BTreeSet<CohortId>> = BTreeMap::new();
- for &(leaf_state_key, person_id) in affected_leaves {
+ for &(leaf_state_key, person_id) in &group.leaves {
if let Some(cohorts) = filters.by_lsk_to_composable_cohorts.get(&leaf_state_key) {
affected
.entry(person_id)
@@ -70,6 +81,13 @@ pub(crate) async fn recompute_stage2_by_person(
.extend(cohorts.iter().copied());
}
}
+ // Only the cohorts no leaf reached are repairs; one a leaf reached recomposes for that reason.
+ let mut repairs: BTreeSet<(CohortId, Uuid)> = BTreeSet::new();
+ for &(cohort_id, person_id) in &group.cohorts {
+ if affected.entry(person_id).or_default().insert(cohort_id) {
+ repairs.insert((cohort_id, person_id));
+ }
+ }
// A map rather than pushes. No consumer needs cohort order (changes are keyed by person and
// writes by row), but the differential tests compare this path with the cohort-ordered one
@@ -120,7 +138,8 @@ pub(crate) async fn recompute_stage2_by_person(
new_bit,
inputs.prior_stage2(cohort_id),
scope.stage2_key(cohort_id),
- ),
+ )
+ .rewriting_corrupt(),
);
}
}
@@ -137,6 +156,9 @@ pub(crate) async fn recompute_stage2_by_person(
event_ms,
last_updated,
);
+ if repairs.contains(&(cohort_id, person_id)) {
+ recompute.record_composed_repair(&diff);
+ }
}
Ok(recompute)
}