Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(cohorts): drop merges of teams the processor does not process (#128860)

Matheus Batista2ef551a0ba63
diff --git a/docs/internal/realtime-cohorts/backfill-coordination.md b/docs/internal/realtime-cohorts/backfill-coordination.md
index 124827c89e94..2f15c32e00f2 100644
--- a/docs/internal/realtime-cohorts/backfill-coordination.md
+++ b/docs/internal/realtime-cohorts/backfill-coordination.md
@@ -165,14 +165,14 @@ The membership consumer does count those markers toward its sweep, see [membersh
Several independent gates decide whether anything happens.
Missing one of them fails quietly, so they are worth knowing.
-| Gate | Effect |
-| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `REALTIME_COHORT_TEAM_ALLOWLIST` | Teams whose cohorts get hash maintenance, stamp invalidation, supersession and any run at all. Unset means no teams in Django, but the shuffler, processor and seeder default to team 2, so set it explicitly on every service. Set but empty means **all** teams. Django puts a listed team at the `serving` level, and adds the teams that have a team access row. The shufflers, the processor and the seeder add those teams too, the seeder 120 seconds after the grant. Never list a team granted `processing` here, because that makes it `serving` |
-| Team access rows | `cohort_realtime_team_access`, written only by `realtime_cohort_team_access grant\|revoke\|list`. Django, the shufflers, the processor and the seeder read it as a union with their `REALTIME_COHORT_TEAM_ALLOWLIST`. Node ingestion does not read the table, and emits person merges only for the teams in its `PERSON_MERGE_EVENTS_TEAM_ALLOWLIST`, so add a granted team there too. The seeder works on a granted team's runs only once `processing_since` is 120 seconds old. After a revoke the seeder stops working on the team's runs, unless its own env list names the team, and each run keeps its cohort's run slot. The processor drops a revoked team from its catalog, and stage 2 GC then deletes the team's state without emitting `left`, so the team's `cohort_membership` rows stay `in_cohort=true`. A shuffler whose env list does not name the team stops forwarding its events. A row at `processing` gets the Django behavior above, but the team's users see no realtime state: no readiness, no realtime batch audience, and no backfilled cohort in flags or workflows, even one stamped before the grant, so a workflow that already uses one fails its next save. A row at `serving` also shows them. `revoke` sets `off`, which hides the stamps the same way, and a revoked team cannot be granted again. `grant` and `revoke` refuse a team on the env list |
-| `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST` | Teams whose saves create runs automatically. Unset or empty means **no** teams. A granted row with `save_triggers_backfill` adds its team, so `resave_cohorts --no-backfill-triggers`, not this setting, is what stops a resave from queueing runs |
-| Operator attestations | Settings an operator sets to declare that prerequisites hold. They are declarations, not live checks. Behavioral runs need `BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED` and `BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED`. Person runs also need the person TTL and sizing attestations and a positive seed-bytes budget |
-| Person sizing budget | A cohort-scoped person run is refused when the team's active person-run estimates plus its own would exceed the budget. The check reads the active estimates before the run exists and reserves nothing, so two cohorts sized at the same time can together exceed the budget. A team run checks only its own estimate. Hitting the size estimate's scan cap, or the cap on pinned conditions, also refuses |
-| Behavioral scan limit | `create_cohort_backfill_run` refuses a behavioral team run when its pinned event names had more events than `BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY` on the busiest of the last 7 complete UTC days. `--max-scan-events-per-day` overrides it for one invocation, and 0 skips the count and the check. The save path does not check it |
+| Gate | Effect |
+| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `REALTIME_COHORT_TEAM_ALLOWLIST` | Teams whose cohorts get hash maintenance, stamp invalidation, supersession and any run at all. Unset means no teams in Django, but the shuffler, processor and seeder default to team 2, so set it explicitly on every service. Set but empty means **all** teams. Django puts a listed team at the `serving` level, and adds the teams that have a team access row. The shufflers, the processor and the seeder add those teams too, the seeder 120 seconds after the grant. Never list a team granted `processing` here, because that makes it `serving` |
+| Team access rows | `cohort_realtime_team_access`, written only by `realtime_cohort_team_access grant\|revoke\|list`. Django, the shufflers, the processor and the seeder read it as a union with their `REALTIME_COHORT_TEAM_ALLOWLIST`. Node ingestion does not read the table, and emits person merges only for the teams in its `PERSON_MERGE_EVENTS_TEAM_ALLOWLIST`. Set that list to `*`, because the processor drops the merges of teams it does not process. The seeder works on a granted team's runs only once `processing_since` is 120 seconds old. After a revoke the seeder stops working on the team's runs, unless its own env list names the team, and each run keeps its cohort's run slot. The processor drops a revoked team from its catalog, and stage 2 GC then deletes the team's state without emitting `left`, so the team's `cohort_membership` rows stay `in_cohort=true`. A shuffler whose env list does not name the team stops forwarding its events. A row at `processing` gets the Django behavior above, but the team's users see no realtime state: no readiness, no realtime batch audience, and no backfilled cohort in flags or workflows, even one stamped before the grant, so a workflow that already uses one fails its next save. A row at `serving` also shows them. `revoke` sets `off`, which hides the stamps the same way, and a revoked team cannot be granted again. `grant` and `revoke` refuse a team on the env list |
+| `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST` | Teams whose saves create runs automatically. Unset or empty means **no** teams. A granted row with `save_triggers_backfill` adds its team, so `resave_cohorts --no-backfill-triggers`, not this setting, is what stops a resave from queueing runs |
+| Operator attestations | Settings an operator sets to declare that prerequisites hold. They are declarations, not live checks. Behavioral runs need `BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED` and `BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED`. Person runs also need the person TTL and sizing attestations and a positive seed-bytes budget |
+| Person sizing budget | A cohort-scoped person run is refused when the team's active person-run estimates plus its own would exceed the budget. The check reads the active estimates before the run exists and reserves nothing, so two cohorts sized at the same time can together exceed the budget. A team run checks only its own estimate. Hitting the size estimate's scan cap, or the cap on pinned conditions, also refuses |
+| Behavioral scan limit | `create_cohort_backfill_run` refuses a behavioral team run when its pinned event names had more events than `BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY` on the busiest of the last 7 complete UTC days. `--max-scan-events-per-day` overrides it for one invocation, and 0 skips the count and the check. The save path does not check it |
The Rust services have their own switches, and Django cannot see them.
The processor must hold a team before the seeder works on the team's runs.
diff --git a/docs/internal/realtime-cohorts/event-routing.md b/docs/internal/realtime-cohorts/event-routing.md
index 1e3664bbf460..d96d47af1339 100644
--- a/docs/internal/realtime-cohorts/event-routing.md
+++ b/docs/internal/realtime-cohorts/event-routing.md
@@ -244,7 +244,8 @@ The processor sees firehose offset 9870001 from partition 3 for p-1 a second tim
- The shuffler and the processor each read their own copy of `REALTIME_COHORT_TEAM_ALLOWLIST` at startup, and each polls Postgres on its own schedule.
Both also let through the teams that a `cohort_realtime_team_access` row grants, and each reads that table about every 15 seconds.
Only teams that both let through are processed, so a newly granted team can lose about half a minute of events until both have read the grant.
- Merges also need the team in Node's merge allowlist.
+ Node ingestion produces merges only for the teams in its own `PERSON_MERGE_EVENTS_TEAM_ALLOWLIST`.
+ Set that list to `*`, because the processor drops the merges of teams it does not process.
- The processor recognizes applied events with a high-water mark per source partition, not a set of offsets.
The shuffler's producer does not guarantee ordering on retries, so an event that is overtaken by a later event from the same source partition is skipped as if it were a replay.
- The broker timestamp on `cohort_stream_events` is the shuffler's send time, not the event's time.
diff --git a/docs/internal/realtime-cohorts/invariants.md b/docs/internal/realtime-cohorts/invariants.md
index b5eb1f5fab83..f924db4d1189 100644
--- a/docs/internal/realtime-cohorts/invariants.md
+++ b/docs/internal/realtime-cohorts/invariants.md
@@ -132,6 +132,7 @@ Most of these are not enforced by a single test, and a violation usually shows u
- Why: merge messages and transfers are redelivered, redriven and retried.
- Kept by: drain and apply markers keyed by the original merge message, and an outbox row that is cleared only after the transfer is acknowledged.
- Limit: markers are local to a partition and expire after their retention, and a revoke or a wipe deletes the outbox with the rest of the slice.
+ A pod whose catalog does not process the team drops the team's merges and transfers, after a team access revoke and also before the pod has read a grant.
- See [merges and cascades](merges-and-cascades.md).
**17. Cascades are bounded.**
@@ -253,17 +254,17 @@ Most of these are not enforced by a single test, and a violation usually shows u
## Delivery, in one table
-| Path | Guarantee | What repairs a loss |
-| ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | --------------------------------------------- |
-| Firehose to stream topic | At least once, with counted abandons | Backfill |
-| Node to person merge events | At most once, uncounted | Backfill, for the survivor's counts |
-| Live event to membership change | At most once | Reconcile |
-| Merge membership output, composed sweep output, first-hop cascades from the live, merge and sweep paths | At most once | Reconcile |
-| Single-leaf sweep output | At least once while the worker runs, and across a restart only with durable restore on | Retries within the path |
-| Cascade messages consumed | At least once | Holds and replays |
-| Merge state and its transfer | At least once while the slice survives. A store wipe deletes a pending transfer | The outbox and its redrive |
-| Seeds and reconcile, including their first-hop cascades | At least once | Replays after holds |
-| Membership consumer | At least once, applied idempotently | The version guard, when version writes are on |
+| Path | Guarantee | What repairs a loss |
+| ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
+| Firehose to stream topic | At least once, with counted abandons | Backfill |
+| Node to person merge events | At most once, uncounted | Backfill, for the survivor's counts |
+| Live event to membership change | At most once | Reconcile |
+| Merge membership output, composed sweep output, first-hop cascades from the live, merge and sweep paths | At most once | Reconcile |
+| Single-leaf sweep output | At least once while the worker runs, and across a restart only with durable restore on | Retries within the path |
+| Cascade messages consumed | At least once | Holds and replays |
+| Merge state and its transfer | At least once while the slice survives. A store wipe deletes a pending transfer, and a pod that does not process the team drops it | The outbox and its redrive, and backfill for a dropped transfer |
+| Seeds and reconcile, including their first-hop cascades | At least once | Replays after holds |
+| Membership consumer | At least once, applied idempotently | The version guard, when version writes are on |
"At most once" means the path never retries a failed produce.
It does not rule out duplicates: a store that loses unflushed writes, or is wiped, replays its inputs against older state and can emit a change a second time.
diff --git a/docs/internal/realtime-cohorts/merges-and-cascades.md b/docs/internal/realtime-cohorts/merges-and-cascades.md
index 6c18565ceb76..7483ce501747 100644
--- a/docs/internal/realtime-cohorts/merges-and-cascades.md
+++ b/docs/internal/realtime-cohorts/merges-and-cascades.md
@@ -30,9 +30,13 @@ In this example `p-2` and `p-1` stand for person UUIDs:
```
The key is `"{team_id}:{old_person_uuid}"`, so the message lands on the old person's partition.
-The producer is gated per team and best effort.
+Node ingestion produces one for every team in its `PERSON_MERGE_EVENTS_TEAM_ALLOWLIST`, best effort.
A failed send is logged and dropped.
+The processor drops a merge, and a transfer, of a team it does not process before it reads the store.
+It commits past the message and counts it in `merge_dropped_total`.
+So ingestion can produce merges for every team, and the processor writes nothing for the teams it does not process.
+
### The protocol
A merge runs in two phases, one on each person's partition.
@@ -74,6 +78,8 @@ sequenceDiagram
If the produce keeps failing, the outbox entry stays and a periodic redrive retries it.
Stage 2 rows of cohorts no longer in the catalog are left for Stage 2 garbage collection.
+A processed team with no realtime cohort still drains and gets a tombstone.
+So a cohort that the team creates later finds the old person's late events under the survivor.
**Apply, on the survivor's worker.**
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index 4c45c73a6630..450e8c200271 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -2752,6 +2752,7 @@ version = "0.1.0"
dependencies = [
"chrono",
"chrono-tz",
+ "common-types",
"dashmap 6.1.0",
"hogvm",
"metrics",
diff --git a/rust/cohort-core/Cargo.toml b/rust/cohort-core/Cargo.toml
index 7758ed75aa13..8e4c9ff2c469 100644
--- a/rust/cohort-core/Cargo.toml
+++ b/rust/cohort-core/Cargo.toml
@@ -9,6 +9,7 @@ workspace = true
[dependencies]
chrono = { workspace = true }
chrono-tz = { workspace = true }
+common-types = { path = "../common/types" }
dashmap = { workspace = true }
hogvm = { path = "../common/hogvm" }
metrics = { workspace = true }
diff --git a/rust/cohort-core/src/filters/catalog.rs b/rust/cohort-core/src/filters/catalog.rs
index 52b534368ff4..7a76404483f3 100644
--- a/rust/cohort-core/src/filters/catalog.rs
+++ b/rust/cohort-core/src/filters/catalog.rs
@@ -2,6 +2,8 @@ use std::collections::HashMap;
use std::hash::{Hash, Hasher};
use std::sync::Arc;
+use common_types::cohort::TeamAllowlist;
+
use crate::filters::reverse_index::TeamFilters;
use crate::filters::TeamId;
@@ -18,21 +20,52 @@ impl Generation {
}
}
-#[derive(Debug, Default)]
+#[derive(Debug)]
pub struct FilterCatalog {
teams: HashMap<TeamId, Arc<TeamFilters>>,
+ /// The teams the catalog was built for. A superset of `teams`, because a processed team with no
+ /// parseable realtime cohort has no filters.
+ processing: TeamAllowlist,
generation: Generation,
}
impl FilterCatalog {
+ /// The empty pre-load catalog, which processes no team.
pub fn new() -> Self {
+ Self::from_teams([])
+ }
+
+ /// A catalog for the teams in `processing`, with the filters of `teams`. A team outside
+ /// `processing` gets no filters.
+ pub fn for_processing(
+ processing: TeamAllowlist,
+ teams: impl IntoIterator<Item = (TeamId, TeamFilters)>,
+ ) -> Self {
Self {
- teams: HashMap::new(),
+ teams: teams
+ .into_iter()
+ .filter(|(team, _)| processing.includes(team.0))
+ .map(|(team, filters)| (team, Arc::new(filters)))
+ .collect(),
+ processing,
generation: Generation::INITIAL,
}
}
- /// The frozen filters for a team, or `None` if it has no realtime cohorts.
+ /// A catalog that processes exactly `teams`.
+ pub fn from_teams(teams: impl IntoIterator<Item = (TeamId, TeamFilters)>) -> Self {
+ let teams: Vec<(TeamId, TeamFilters)> = teams.into_iter().collect();
+ let processing = TeamAllowlist::Only(teams.iter().map(|(team, _)| team.0).collect());
+ Self::for_processing(processing, teams)
+ }
+
+ /// Whether the catalog was built for `team_id`. True for a processed team with no realtime
+ /// cohort, which [`Self::team`] has no filters for.
+ pub fn processes(&self, team_id: TeamId) -> bool {
+ self.processing.includes(team_id.0)
+ }
+
+ /// The frozen filters for a team, or `None` if it has no realtime cohorts or is not processed.
pub fn team(&self, team_id: TeamId) -> Option<&Arc<TeamFilters>> {
self.teams.get(&team_id)
}
@@ -59,22 +92,19 @@ impl FilterCatalog {
.map(|team| team.unique_condition_hashes.len())
.sum()
}
+}
- pub fn from_teams(teams: impl IntoIterator<Item = (TeamId, TeamFilters)>) -> Self {
- Self {
- teams: teams
- .into_iter()
- .map(|(team, filters)| (team, Arc::new(filters)))
- .collect(),
- generation: Generation::INITIAL,
- }
+impl Default for FilterCatalog {
+ fn default() -> Self {
+ Self::new()
}
}
/// Hashes only the per-team condition sets (order-independent via the sorts), deliberately excluding
-/// `generation`. This backs the processor's `catalog_signature` memo invalidation: two catalogs with
-/// the same conditions must hash equal so a no-op refresh reuses the generation and keeps memo entries
-/// valid. `manager.rs`'s equivalence test guards the contract.
+/// `generation` and which teams the catalog processes, because neither changes a condition's
+/// evaluation. This backs the processor's `catalog_signature` memo invalidation: two catalogs with
+/// the same conditions must hash equal so a no-op refresh reuses the generation and keeps memo
+/// entries valid. `manager.rs`'s equivalence test guards the contract.
impl Hash for FilterCatalog {
fn hash<H: Hasher>(&self, state: &mut H) {
let mut teams: Vec<(&TeamId, &Arc<TeamFilters>)> = self.teams.iter().collect();
diff --git a/rust/cohort-core/src/filters/loader.rs b/rust/cohort-core/src/filters/loader.rs
index 455bde140107..8aacdac0c4c3 100644
--- a/rust/cohort-core/src/filters/loader.rs
+++ b/rust/cohort-core/src/filters/loader.rs
@@ -3,6 +3,7 @@
use std::collections::HashMap;
use chrono_tz::{Tz, UTC};
+use common_types::cohort::TeamAllowlist;
use metrics::counter;
use serde_json::Value;
use sqlx::PgPool;
@@ -66,10 +67,21 @@ pub async fn load_realtime_cohorts_for_teams(
Ok(rows)
}
-/// Group rows by team into a catalog. A cohort that fails to parse is counted, warned, and skipped
-/// rather than poisoning the rest of the catalog.
-pub fn build_catalog_from_rows(rows: Vec<CohortRow>, cascade_enabled: bool) -> FilterCatalog {
- build_catalog_within(rows, cascade_enabled, &mut catalog_analysis_budget())
+/// Group rows by team into a catalog for the teams in `processing`. The catalog keeps the scope, so
+/// a reader can tell a team it does not process from a processed team with no realtime cohort. A
+/// cohort that fails to parse is counted, warned, and skipped rather than poisoning the rest of the
+/// catalog.
+pub fn build_catalog_from_rows(
+ rows: Vec<CohortRow>,
+ processing: TeamAllowlist,
+ cascade_enabled: bool,
+) -> FilterCatalog {
+ build_catalog_within(
+ rows,
+ processing,
+ cascade_enabled,
+ &mut catalog_analysis_budget(),
+ )
}
/// [`build_catalog_from_rows`] analyzing every team's conditions against one `budget`. Teams
@@ -77,6 +89,7 @@ pub fn build_catalog_from_rows(rows: Vec<CohortRow>, cascade_enabled: bool) -> F
/// pure function of the rows.
fn build_catalog_within(
rows: Vec<CohortRow>,
+ processing: TeamAllowlist,
cascade_enabled: bool,
budget: &mut AnalysisBudget,
) -> FilterCatalog {
@@ -130,13 +143,14 @@ fn build_catalog_within(
// The team the budget ran out on, so an operator chasing a widened catalog has a place to start:
// it and every team after it in id order carry the conditions the budget refused.
let mut spent_on: Option<TeamId> = None;
- let catalog = FilterCatalog::from_teams(teams.into_iter().map(|(team, (builder, tz))| {
+ let filters = teams.into_iter().map(|(team, (builder, tz))| {
let filters = builder.freeze_within(tz, cascade_enabled, budget);
if spent_on.is_none() && (budget.steps == 0 || budget.cells == 0) {
spent_on = Some(team);
}
(team, filters)
- }));
+ });
+ let catalog = FilterCatalog::for_processing(processing, filters);
if let Some(team_id) = spent_on {
warn!(
team_id = team_id.0,
@@ -192,6 +206,8 @@ fn resolve_team_tz(raw: &str, team_id: TeamId) -> Tz {
#[cfg(test)]
mod tests {
+ use std::collections::HashSet;
+
use super::*;
use serde_json::json;
@@ -241,9 +257,27 @@ mod tests {
}
#[test]
- fn empty_rows_build_an_empty_catalog() {
- let catalog = build_catalog_from_rows(vec![], false);
- assert_eq!(catalog.team_count(), 0);
+ fn a_catalog_processes_exactly_its_scope() {
+ let scoped = build_catalog_from_rows(
+ vec![row(1, 9, behavioral_cohort())],
+ TeamAllowlist::Only(HashSet::from([8])),
+ false,
+ );
+ assert!(
+ scoped.processes(TeamId(8)),
+ "a team in scope with no cohort"
+ );
+ assert!(
+ !scoped.processes(TeamId(9)) && scoped.team(TeamId(9)).is_none(),
+ "a row outside the scope neither processes its team nor gives it filters",
+ );
+ assert_eq!(scoped.team_count(), 0);
+
+ let unscoped = build_catalog_from_rows(vec![], TeamAllowlist::All, false);
+ assert!(
+ unscoped.processes(TeamId(8)),
+ "every team, with or without a cohort"
+ );
}
#[test]
@@ -264,6 +298,7 @@ mod tests {
row_with_guards(6, Some(SHAPE_HASH), None),
row_with_guards(7, None, Some(PERSON_SHAPE_HASH)),
],
+ TeamAllowlist::All,
false,
);
@@ -303,7 +338,7 @@ mod tests {
row_with_tz(1, 7, behavioral_cohort(), "America/New_York"),
row_with_tz(2, 9, behavioral_cohort(), "not a real zone"),
];
- let catalog = build_catalog_from_rows(rows, false);
+ let catalog = build_catalog_from_rows(rows, TeamAllowlist::All, false);
assert_eq!(catalog.team(TeamId(7)).expect("team 7").timezone, New_York);
assert_eq!(
catalog.team(TeamId(9)).expect("team 9").timezone,
@@ -318,7 +353,7 @@ mod tests {
row(1, 7, json!({ "bogus": true })),
row(2, 7, behavioral_cohort()),
];
- let catalog = build_catalog_from_rows(rows, false);
+ let catalog = build_catalog_from_rows(rows, TeamAllowlist::All, false);
let team = catalog.team(TeamId(7)).expect("team present");
assert!(team.cohorts.contains_key(&CohortId(2)));
@@ -340,7 +375,7 @@ mod tests {
steps: 5,
cells: usize::MAX,
};
- let catalog = build_catalog_within(rows, false, &mut budget);
+ let catalog = build_catalog_within(rows, TeamAllowlist::All, false, &mut budget);
let plan = |team: i32| {
catalog
@@ -372,14 +407,14 @@ mod tests {
});
let rows = || vec![row(2, 7, behavioral_cohort()), row(1, 7, referrer.clone())];
- let off = build_catalog_from_rows(rows(), false);
+ let off = build_catalog_from_rows(rows(), TeamAllowlist::All, false);
assert_eq!(
off.team(TeamId(7)).unwrap().eligibility[&CohortId(1)],
CohortEligibility::Excluded(ExcludedReason::HasCohortRef),
"gate off keeps the ref cohort excluded",
);
- let on = build_catalog_from_rows(rows(), true);
+ let on = build_catalog_from_rows(rows(), TeamAllowlist::All, true);
assert_eq!(
on.team(TeamId(7)).unwrap().eligibility[&CohortId(1)],
CohortEligibility::Stage2ComposableRef,
diff --git a/rust/cohort-stream-processor/src/consumers/events.rs b/rust/cohort-stream-processor/src/consumers/events.rs
index bd077488fb56..094ac3bb9888 100644
--- a/rust/cohort-stream-processor/src/consumers/events.rs
+++ b/rust/cohort-stream-processor/src/consumers/events.rs
@@ -1420,11 +1420,13 @@ pub(crate) async fn run_pauser_loop(
// facade.
#[allow(clippy::disallowed_methods)]
mod tests {
+ use std::collections::HashSet;
use std::num::{NonZeroU16, NonZeroUsize};
use super::*;
use chrono_tz::UTC;
use common_kafka::kafka_producer::KafkaProduceError;
+ use common_types::cohort::TeamAllowlist;
use futures::FutureExt;
use serde_json::json;
use tempfile::TempDir;
@@ -2708,8 +2710,11 @@ mod tests {
)])))
}
- fn absent_team_catalog() -> Arc<CatalogHandle> {
- Arc::new(CatalogHandle::from_catalog(FilterCatalog::from_teams([])))
+ fn team_without_cohorts_catalog() -> Arc<CatalogHandle> {
+ Arc::new(CatalogHandle::from_catalog(FilterCatalog::for_processing(
+ TeamAllowlist::Only(HashSet::from([TEAM])),
+ [],
+ )))
}
fn person_hashing_to(partition: u16, start: u128) -> Uuid {
@@ -3421,11 +3426,11 @@ mod tests {
}
#[tokio::test]
- async fn absent_team_merge_still_drains_tombstone_and_marker_and_marks() {
+ async fn team_without_cohorts_merge_still_drains_tombstone_and_marker_and_marks() {
let (_dir, store) = temp_store();
let (dispatcher, sink, transfer_sink, merge) = dispatcher_full(
&store,
- absent_team_catalog(),
+ team_without_cohorts_catalog(),
CaptureSink::new(),
CaptureTransferSink::new(),
);
@@ -3477,16 +3482,16 @@ mod tests {
assert_eq!(
merge.merge_tracker.committable_offsets().get(&partition),
Some(&10),
- "an absent team never holds the offset",
+ "a team without cohorts never holds the offset",
);
}
#[tokio::test]
- async fn absent_team_transfer_applies_dropping_leaves_as_drift_and_marks() {
+ async fn team_without_cohorts_transfer_applies_dropping_leaves_as_drift_and_marks() {
let (_dir, store) = temp_store();
let (dispatcher, sink, _transfer_sink, merge) = dispatcher_full(
&store,
- absent_team_catalog(),
+ team_without_cohorts_catalog(),
CaptureSink::new(),
CaptureTransferSink::new(),
);
diff --git a/rust/cohort-stream-processor/src/filters/manager.rs b/rust/cohort-stream-processor/src/filters/manager.rs
index e53b729626b7..4a52a141cf97 100644
--- a/rust/cohort-stream-processor/src/filters/manager.rs
+++ b/rust/cohort-stream-processor/src/filters/manager.rs
@@ -277,11 +277,6 @@ async fn load_processing(
}
}
-/// Backstop for the load query's team filter.
-fn retain_processing(rows: &mut Vec<CohortRow>, teams: &ProcessingTeams) {
- rows.retain(|row| teams.scope().includes(row.team_id));
-}
-
/// A refresh that has begun and not yet published (test seam).
#[cfg(test)]
#[must_use = "a pending refresh publishes nothing until `publish` is called"]
@@ -409,19 +404,13 @@ impl CatalogRefresher {
teams: &ProcessingTeams,
load: impl AsyncFnOnce(&ProcessingTeams) -> Result<Vec<CohortRow>, FilterError>,
) -> Result<CatalogStats, CatalogRefreshError> {
- let mut rows = tokio::time::timeout(LOAD_TIMEOUT, load(teams)).await??;
- let fetched_rows = rows.len();
- retain_processing(&mut rows, teams);
- if rows.len() != fetched_rows {
- debug!(
- fetched_rows,
- kept_rows = rows.len(),
- "filter catalog dropped cohort rows of teams it does not process",
- );
- }
+ let rows = tokio::time::timeout(LOAD_TIMEOUT, load(teams)).await??;
+ let processing = teams.scope().clone();
let cascade_enabled = self.cascade_enabled;
self.catalog
- .build_and_store(seq, move || build_catalog_from_rows(rows, cascade_enabled))
+ .build_and_store(seq, move || {
+ build_catalog_from_rows(rows, processing, cascade_enabled)
+ })
.await
}
@@ -532,17 +521,6 @@ mod tests {
use crate::filters::reverse_index::TeamFiltersBuilder;
use crate::filters::{CohortId, TeamFilters, TeamId};
- fn cohort_row(id: i32, team_id: i32) -> CohortRow {
- CohortRow {
- id,
- team_id,
- filters: serde_json::Value::Null,
- behavioral_filters_shape_hash: None,
- person_filters_shape_hash: None,
- timezone: "UTC".to_string(),
- }
- }
-
fn processing(env: TeamAllowlist, granted: &[i32]) -> Arc<ProcessingTeams> {
TeamAccess::with_granted(env, granted.iter().copied())
.processing_teams()
@@ -703,6 +681,14 @@ mod tests {
"a job admitted during a read must not count the refresh as begun after it",
);
}
+ assert!(
+ catalog.load().processes(TeamId(5)),
+ "a granted team with no realtime cohort is processed, so its merges still drain",
+ );
+ assert!(
+ !catalog.load().processes(TeamId(9)),
+ "a team outside the read's scope is not processed, so its merges drop",
+ );
}
#[tokio::test]
@@ -871,15 +857,20 @@ mod tests {
let gen1 = handle.load().generation();
// Same condition set → same signature → generation unchanged (the memo survives the refresh).
- handle.publish(FilterCatalog::from_teams([(
- TeamId(7),
- team_with_one_behavioral(),
- )]));
+ // The scope is outside the signature, so a grant of a team with no cohort is such a refresh.
+ handle.publish(FilterCatalog::for_processing(
+ TeamAllowlist::Only(HashSet::from([7, 8])),
+ [(TeamId(7), team_with_one_behavioral())],
+ ));
assert_eq!(
handle.load().generation(),
gen1,
"a no-op refresh reuses the generation",
);
+ assert!(
+ handle.load().processes(TeamId(8)),
+ "a refresh with the same signature still publishes its scope",
+ );
// A different condition hash → signature changes → generation advances (memo invalidates).
handle.publish(FilterCatalog::from_teams([(
@@ -906,20 +897,6 @@ mod tests {
assert_eq!(catalog_signature(&a), catalog_signature(&b));
}
- #[test]
- fn retain_processing_keeps_the_rows_of_the_env_list_and_the_granted_teams() {
- for (env, granted, kept) in [
- (TeamAllowlist::Only(HashSet::from([2])), vec![5], vec![2, 5]),
- (TeamAllowlist::All, vec![], vec![2, 5, 7]),
- (TeamAllowlist::Only(HashSet::new()), vec![], vec![]),
- ] {
- let mut rows = vec![cohort_row(1, 2), cohort_row(2, 5), cohort_row(3, 7)];
- retain_processing(&mut rows, &processing(env.clone(), &granted));
- let kept_teams: Vec<i32> = rows.iter().map(|row| row.team_id).collect();
- assert_eq!(kept_teams, kept, "env {env:?}, granted {granted:?}");
- }
- }
-
#[test]
fn next_interval_without_jitter_is_exact() {
let interval = Duration::from_secs(300);
@@ -970,7 +947,8 @@ mod tests {
assert!(handle.is_loaded());
assert_eq!(handle.load().team_count(), stats.teams);
- // Bypass `retain_processing`, which would hide a missing SQL filter.
+ // Read the rows directly, because the catalog leaves out other teams' filters, which would
+ // hide a missing SQL filter.
let keys = |rows: &[CohortRow]| -> BTreeSet<(i32, i32)> {
rows.iter().map(|row| (row.team_id, row.id)).collect()
};
diff --git a/rust/cohort-stream-processor/src/observability/metrics.rs b/rust/cohort-stream-processor/src/observability/metrics.rs
index 400096a08df2..a6ecd060aab0 100644
--- a/rust/cohort-stream-processor/src/observability/metrics.rs
+++ b/rust/cohort-stream-processor/src/observability/metrics.rs
@@ -486,6 +486,9 @@ pub const SWEEP_READ_CHUNK_BYTES: &str = "sweep_read_chunk_bytes";
pub const SWEEP_QUEUE_LAG_SECONDS: &str = "sweep_queue_lag_seconds";
/// Person merges handled, labelled by `path` (`same_partition`|`cross_partition`) (counter).
pub const MERGE_HANDLED_TOTAL: &str = "merge_handled_total";
+/// Merge and transfer messages of teams the processor does not process, dropped before they touch
+/// the store, labelled by `path` (`drain`|`apply`) (counter).
+pub const MERGE_DROPPED_TOTAL: &str = "merge_dropped_total";
/// Drain messages short-circuited by a `cf_merge_drains_applied` hit (counter).
pub const MERGE_DRAINS_SKIPPED_REPLAY_TOTAL: &str = "merge_drains_skipped_replay_total";
/// Transfer messages short-circuited by a `cf_merge_applied` hit (counter).
diff --git a/rust/cohort-stream-processor/src/workers/merge_path.rs b/rust/cohort-stream-processor/src/workers/merge_path.rs
index 4fca02cf7cb2..ea7c355c93fa 100644
--- a/rust/cohort-stream-processor/src/workers/merge_path.rs
+++ b/rust/cohort-stream-processor/src/workers/merge_path.rs
@@ -14,7 +14,7 @@ use tracing::{debug, warn};
use crate::filters::manager::CatalogHandle;
use crate::filters::reverse_index::TeamFilters;
-use crate::filters::{TeamFiltersBuilder, TeamId};
+use crate::filters::{FilterCatalog, TeamFiltersBuilder, TeamId};
use crate::merge::apply_handler::{handle_transfer, ApplyOutcome};
use crate::merge::drain_handler::{
handle_merge_event_with_transfer_mode, DrainOutcome, MembershipRegisterTransferMode,
@@ -22,9 +22,10 @@ use crate::merge::drain_handler::{
use crate::merge::transfer::{MergeStateTransfer, PendingTransfer, PersonMergeEvent};
use crate::observability::metrics::{
COHORT_STREAM_OFFSET_AHEAD_OF_DISPATCH, MERGE_APPLY_DURATION_SECONDS,
- MERGE_DRAIN_DURATION_SECONDS, MERGE_HELD_OFFSET_GAUGE, MERGE_OUTBOX_CLEAR_FAILURE_TOTAL,
- MERGE_PENDING_TRANSFERS_GAUGE, MERGE_TRANSFERS_SKIPPED_EMPTY_TOTAL,
- MERGE_TRANSFER_FORWARDS_TOTAL, MERGE_TRANSFER_PRODUCE_FAILURE_TOTAL, STAGE1_TRANSITIONS,
+ MERGE_DRAIN_DURATION_SECONDS, MERGE_DROPPED_TOTAL, MERGE_HELD_OFFSET_GAUGE,
+ MERGE_OUTBOX_CLEAR_FAILURE_TOTAL, MERGE_PENDING_TRANSFERS_GAUGE,
+ MERGE_TRANSFERS_SKIPPED_EMPTY_TOTAL, MERGE_TRANSFER_FORWARDS_TOTAL,
+ MERGE_TRANSFER_PRODUCE_FAILURE_TOTAL, STAGE1_TRANSITIONS,
};
use crate::partitions::offset_tracker::{MarkOutcome, OffsetTracker};
use crate::partitions::partitioner::COHORT_PARTITION_COUNT;
@@ -184,34 +185,28 @@ pub(crate) async fn handle_merge(
) {
let msg_coords = (partition_id as i32, offset);
- // One snapshot serves both resolutions below; `filters` falls back to empty when the team is
- // absent, so the drain never wedges on an unknown team.
- let snapshot = catalog.load_full();
+ let Some(filters) = merge_filters(&catalog.load_full(), TeamId(event.team_id)) else {
+ counter!(MERGE_DROPPED_TOTAL, "path" => "drain").increment(1);
+ mark_processed(&merge.merge_tracker, partition_id, offset);
+ return;
+ };
// The whole drain state machine runs as one sync section on the blocking pool with owned inputs.
let section = {
- let snapshot = snapshot.clone();
+ let filters = filters.clone();
let event_owned = event.clone();
let partition_count = merge.partition_count;
let register_transfer_mode =
MembershipRegisterTransferMode::from_enabled(merge.register_transfer_enabled);
handle
.run_section("merge_drain", move |store| {
- let fallback: TeamFilters;
- let filters: &TeamFilters = match snapshot.team(TeamId(event_owned.team_id)) {
- Some(team) => team,
- None => {
- fallback = empty_team_filters();
- &fallback
- }
- };
// Timed inside the section so the histogram keeps measuring drain execution only;
// permit and pool-queue waits are on `store_offload_*{op="merge_drain"}`.
let started = Instant::now();
let outcome = handle_merge_event_with_transfer_mode(
partition_id,
store,
- filters,
+ &filters,
&event_owned,
msg_coords,
partition_count,
@@ -223,15 +218,6 @@ pub(crate) async fn handle_merge(
.await
};
- let fallback: TeamFilters;
- let filters: &TeamFilters = match snapshot.team(TeamId(event.team_id)) {
- Some(team) => team,
- None => {
- fallback = empty_team_filters();
- &fallback
- }
- };
-
// The section is only `Err` on teardown cancellation; hold the offset so the next tenure's
// redelivery resolves the merge.
let outcome = match section {
@@ -264,7 +250,7 @@ pub(crate) async fn handle_merge(
handle,
sink,
merge,
- filters,
+ &filters,
&transitions,
event.merged_at_ms,
last_updated,
@@ -359,30 +345,26 @@ pub(crate) async fn handle_apply(
) {
let transfer_coords = (partition_id as i32, offset);
- let snapshot = catalog.load_full();
+ let Some(filters) = merge_filters(&catalog.load_full(), TeamId(transfer.team_id)) else {
+ counter!(MERGE_DROPPED_TOTAL, "path" => "apply").increment(1);
+ mark_processed(&merge.transfer_tracker, partition_id, offset);
+ return;
+ };
// The whole apply state machine runs as one sync section on the blocking pool with owned inputs.
let section = {
- let snapshot = snapshot.clone();
+ let filters = filters.clone();
let transfer_owned = transfer.clone();
let partition_count = merge.partition_count;
handle
.run_section("merge_apply", move |store| {
- let fallback: TeamFilters;
- let filters: &TeamFilters = match snapshot.team(TeamId(transfer_owned.team_id)) {
- Some(team) => team,
- None => {
- fallback = empty_team_filters();
- &fallback
- }
- };
// Timed inside the section so the histogram keeps measuring apply execution only;
// permit and pool-queue waits are on `store_offload_*{op="merge_apply"}`.
let started = Instant::now();
let outcome = handle_transfer(
partition_id,
store,
- filters,
+ &filters,
&transfer_owned,
transfer_coords,
partition_count,
@@ -393,15 +375,6 @@ pub(crate) async fn handle_apply(
.await
};
- let fallback: TeamFilters;
- let filters: &TeamFilters = match snapshot.team(TeamId(transfer.team_id)) {
- Some(team) => team,
- None => {
- fallback = empty_team_filters();
- &fallback
- }
- };
-
let outcome = match section {
Ok(outcome) => outcome,
Err(_cancelled) => {
@@ -427,7 +400,7 @@ pub(crate) async fn handle_apply(
handle,
sink,
merge,
- filters,
+ &filters,
&transitions,
transfer.merged_at_ms,
last_updated,
@@ -736,9 +709,22 @@ fn hold(tracker: &OffsetTracker, partition_id: u16, offset: i64) {
gauge!(MERGE_HELD_OFFSET_GAUGE, "partition" => partition_id.to_string()).set(floor as f64);
}
-/// Fallback for a team absent from the catalog: no cohorts, UTC timezone.
-fn empty_team_filters() -> TeamFilters {
- TeamFiltersBuilder::default().freeze(UTC)
+/// The filters a merge or a transfer of `team_id` runs with, or `None` when the catalog does not
+/// process the team. Even with empty filters, a drain writes a tombstone and a drain marker and an
+/// apply writes an apply marker, and the processor must hold no state for a team it does not
+/// process. A processed team with no realtime cohort gets empty filters, so its tombstones still
+/// redirect stragglers to the survivor.
+///
+/// The empty boot catalog processes no team, but it drops nothing, because
+/// `spawn_follower_after_boot` starts the merge and transfer followers only after the first catalog
+/// load.
+fn merge_filters(catalog: &FilterCatalog, team_id: TeamId) -> Option<Arc<TeamFilters>> {
+ catalog.processes(team_id).then(|| {
+ catalog
+ .team(team_id)
+ .cloned()
+ .unwrap_or_else(|| Arc::new(TeamFiltersBuilder::default().freeze(UTC)))
+ })
}
#[cfg(test)]
@@ -746,7 +732,10 @@ fn empty_team_filters() -> TeamFilters {
// `StoreHandle` facade.
#[allow(clippy::disallowed_methods)]
mod tests {
+ use std::collections::HashSet;
+
use super::*;
+ use common_types::cohort::TeamAllowlist;
use envconfig::Envconfig;
use serde_json::json;
use tempfile::TempDir;
@@ -759,7 +748,10 @@ mod tests {
use crate::stage1::key::LeafStateKey;
use crate::stage1::state::{AppliedOffsets, Stage1State, StatefulRecord};
use crate::stage2::Stage2State;
- use crate::store::{CohortStore, OffloadConfig, OffloadMode, Stage2Key, StoreConfig};
+ use crate::store::{
+ CohortStore, MergeAppliedKey, OffloadConfig, OffloadMode, Stage2Key, StoreConfig,
+ TombstoneKey,
+ };
/// Wrap a test store in the `All` operating point so `handle_redrive` runs on the blocking pool.
fn handle(store: &CohortStore) -> StoreHandle {
@@ -1298,4 +1290,111 @@ mod tests {
"P_old's register row is reclaimed by the drain, like the pre-register pipeline",
);
}
+
+ /// The team-without-cohorts tests in `consumers::events` pin that a processed team still drains.
+ #[tokio::test]
+ async fn a_merge_and_its_transfer_of_a_team_the_catalog_does_not_process_write_nothing() {
+ const TEAM: i32 = 8;
+ const MERGE_OFFSET: i64 = 50;
+ const TRANSFER_OFFSET: i64 = 80;
+
+ let catalog = CatalogHandle::from_catalog(FilterCatalog::for_processing(
+ TeamAllowlist::Only(HashSet::from([7])),
+ [],
+ ));
+ let old_person = Uuid::from_u128(1);
+ let new_person = Uuid::from_u128(2);
+ let old_partition = partition_of(TeamId(TEAM), &old_person, COHORT_PARTITION_COUNT) as u16;
+ let new_partition = partition_of(TeamId(TEAM), &new_person, COHORT_PARTITION_COUNT) as u16;
+
+ let (_dir, store) = temp_store();
+ let deps = capture_deps(CaptureTransferSink::new());
+ deps.merge_tracker
+ .mark_dispatched(old_partition as i32, MERGE_OFFSET + 1);
+ deps.transfer_tracker
+ .mark_dispatched(new_partition as i32, TRANSFER_OFFSET + 1);
+ let sink: Arc<dyn MembershipSink> = Arc::new(CaptureSink::new());
+ let mut queue = EvictionQueue::new();
+
+ handle_merge(
+ old_partition,
+ &handle(&store),
+ &catalog,
+ &sink,
+ &deps,
+ &mut queue,
+ "1",
+ &PersonMergeEvent {
+ team_id: TEAM,
+ old_person_uuid: old_person,
+ new_person_uuid: new_person,
+ merged_at_ms: 2,
+ schema_version: MERGE_EVENT_SCHEMA_VERSION,
+ },
+ MERGE_OFFSET,
+ )
+ .await;
+ handle_apply(
+ new_partition,
+ &handle(&store),
+ &catalog,
+ &sink,
+ &deps,
+ &mut queue,
+ "1",
+ &MergeStateTransfer {
+ team_id: TEAM,
+ old_person_uuid: old_person,
+ new_person_uuid: new_person,
+ merged_at_ms: 2,
+ source_partition: old_partition as i32,
+ source_offset: MERGE_OFFSET,
+ leaves: vec![],
+ membership_registers: vec![],
+ forward_hops: 0,
+ person_dedup: None,
+ },
+ TRANSFER_OFFSET,
+ )
+ .await;
+
+ assert!(
+ store
+ .get_tombstone(&TombstoneKey {
+ partition_id: old_partition,
+ team_id: TEAM as u64,
+ person: old_person,
+ })
+ .unwrap()
+ .is_none(),
+ "the dropped merge wrote no tombstone",
+ );
+ assert!(
+ store
+ .get_merge_applied(&MergeAppliedKey {
+ partition_id: new_partition,
+ team_id: TEAM as u64,
+ new_person,
+ source_partition: old_partition as i32,
+ source_offset: MERGE_OFFSET,
+ })
+ .unwrap()
+ .is_none(),
+ "the dropped transfer wrote no apply marker",
+ );
+ assert_eq!(
+ deps.merge_tracker
+ .committable_offsets()
+ .get(&(old_partition as i32)),
+ Some(&(MERGE_OFFSET + 1)),
+ "the dropped merge commits past its offset",
+ );
+ assert_eq!(
+ deps.transfer_tracker
+ .committable_offsets()
+ .get(&(new_partition as i32)),
+ Some(&(TRANSFER_OFFSET + 1)),
+ "the dropped transfer commits past its offset",
+ );
+ }
}
diff --git a/rust/cohort-stream-processor/tests/filter_catalog.rs b/rust/cohort-stream-processor/tests/filter_catalog.rs
index 5dc4a6f09fe9..655c4710b094 100644
--- a/rust/cohort-stream-processor/tests/filter_catalog.rs
+++ b/rust/cohort-stream-processor/tests/filter_catalog.rs
@@ -8,10 +8,11 @@ use cohort_stream_processor::filters::{
build_catalog_from_rows, CohortId, CohortLeaf, CohortRow, FilterCatalog, FilterNode, TeamId,
};
use cohort_stream_processor::stage2::{CohortEligibility, ExcludedReason};
+use common_types::cohort::TeamAllowlist;
use serde_json::{json, Value};
fn build_catalog(rows: Vec<CohortRow>) -> FilterCatalog {
- build_catalog_from_rows(rows, false)
+ build_catalog_from_rows(rows, TeamAllowlist::All, false)
}
const BEHAVIORAL_HASH: [u8; 16] = *b"0123456789abcdef";