Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(cohorts): read the team access record in the seeder (#128768)

Matheus Batistad454feec4a44
diff --git a/docs/internal/realtime-cohorts/backfill-coordination.md b/docs/internal/realtime-cohorts/backfill-coordination.md
index 983fbae9b83a..8e7db16bef51 100644
--- a/docs/internal/realtime-cohorts/backfill-coordination.md
+++ b/docs/internal/realtime-cohorts/backfill-coordination.md
@@ -152,21 +152,28 @@ The membership consumer does count those markers toward its sweep, see [membersh
Several independent gates decide whether anything happens.
Missing one of them fails quietly, so they are worth knowing.
-| Gate | Effect |
-| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `REALTIME_COHORT_TEAM_ALLOWLIST` | Teams whose cohorts get hash maintenance, stamp invalidation, supersession and any run at all. Unset means no teams in Django, but the shuffler, processor and seeder default to team 2, so set it explicitly on every service. Set but empty means **all** teams. Django puts a listed team at the `serving` level, and adds the teams that have a team access row. The shufflers and the processor add those teams too, but the seeder's list must name both. Never list a team granted `processing` here, because that makes it `serving` |
-| Team access rows | `cohort_realtime_team_access`, written only by `realtime_cohort_team_access grant\|revoke\|list`. Django, the shufflers and the processor read it as a union with their `REALTIME_COHORT_TEAM_ALLOWLIST`; the seeder reads only its env list. Until it reads the table, keep its env list in step with the rows. Node ingestion does not read the table, and emits person merges only for the teams in its `PERSON_MERGE_EVENTS_TEAM_ALLOWLIST`, so add a granted team there too. A team granted only here gets runs nobody claims, each holding its cohort's run slot. A team revoked only here loses Django's edit-time supersede while the seeder still processes it, so its readiness stamps outlive later edits. The processor drops a revoked team from its catalog, and stage 2 GC then deletes the team's state without emitting `left`, so the team's `cohort_membership` rows stay `in_cohort=true`. A shuffler whose env list does not name the team stops forwarding its events. A row at `processing` gets the Django behavior above, but the team's users see no realtime state: no readiness, no realtime batch audience, and no backfilled cohort in flags or workflows, even one stamped before the grant, so a workflow that already uses one fails its next save. A row at `serving` also shows them. `revoke` sets `off`, which hides the stamps the same way, and a revoked team cannot be granted again. `grant` and `revoke` refuse a team on the env list |
-| `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST` | Teams whose saves create runs automatically. Unset or empty means **no** teams. A granted row with `save_triggers_backfill` adds its team, so `resave_cohorts --no-backfill-triggers`, not this setting, is what stops a resave from queueing runs |
-| Operator attestations | Settings an operator sets to declare that prerequisites hold. They are declarations, not live checks. Behavioral runs need `BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED` and `BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED`. Person runs also need the person TTL and sizing attestations and a positive seed-bytes budget |
-| Person sizing budget | A cohort-scoped person run is refused when the team's active person-run estimates plus its own would exceed the budget. The check reads the active estimates before the run exists and reserves nothing, so two cohorts sized at the same time can together exceed the budget. A team run checks only its own estimate. Hitting the size estimate's scan cap, or the cap on pinned conditions, also refuses |
-| Behavioral scan limit | `create_cohort_backfill_run` refuses a behavioral team run when its pinned event names had more events than `BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY` on the busiest of the last 7 complete UTC days. `--max-scan-events-per-day` overrides it for one invocation, and 0 skips the count and the check. The save path does not check it |
+| Gate | Effect |
+| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `REALTIME_COHORT_TEAM_ALLOWLIST` | Teams whose cohorts get hash maintenance, stamp invalidation, supersession and any run at all. Unset means no teams in Django, but the shuffler, processor and seeder default to team 2, so set it explicitly on every service. Set but empty means **all** teams. Django puts a listed team at the `serving` level, and adds the teams that have a team access row. The shufflers, the processor and the seeder add those teams too, the seeder 120 seconds after the grant. Never list a team granted `processing` here, because that makes it `serving` |
+| Team access rows | `cohort_realtime_team_access`, written only by `realtime_cohort_team_access grant\|revoke\|list`. Django, the shufflers, the processor and the seeder read it as a union with their `REALTIME_COHORT_TEAM_ALLOWLIST`. Node ingestion does not read the table, and emits person merges only for the teams in its `PERSON_MERGE_EVENTS_TEAM_ALLOWLIST`, so add a granted team there too. The seeder works on a granted team's runs only once `processing_since` is 120 seconds old. After a revoke the seeder stops working on the team's runs, unless its own env list names the team, and each run keeps its cohort's run slot. The processor drops a revoked team from its catalog, and stage 2 GC then deletes the team's state without emitting `left`, so the team's `cohort_membership` rows stay `in_cohort=true`. A shuffler whose env list does not name the team stops forwarding its events. A row at `processing` gets the Django behavior above, but the team's users see no realtime state: no readiness, no realtime batch audience, and no backfilled cohort in flags or workflows, even one stamped before the grant, so a workflow that already uses one fails its next save. A row at `serving` also shows them. `revoke` sets `off`, which hides the stamps the same way, and a revoked team cannot be granted again. `grant` and `revoke` refuse a team on the env list |
+| `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST` | Teams whose saves create runs automatically. Unset or empty means **no** teams. A granted row with `save_triggers_backfill` adds its team, so `resave_cohorts --no-backfill-triggers`, not this setting, is what stops a resave from queueing runs |
+| Operator attestations | Settings an operator sets to declare that prerequisites hold. They are declarations, not live checks. Behavioral runs need `BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED` and `BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED`. Person runs also need the person TTL and sizing attestations and a positive seed-bytes budget |
+| Person sizing budget | A cohort-scoped person run is refused when the team's active person-run estimates plus its own would exceed the budget. The check reads the active estimates before the run exists and reserves nothing, so two cohorts sized at the same time can together exceed the budget. A team run checks only its own estimate. Hitting the size estimate's scan cap, or the cap on pinned conditions, also refuses |
+| Behavioral scan limit | `create_cohort_backfill_run` refuses a behavioral team run when its pinned event names had more events than `BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY` on the busiest of the last 7 complete UTC days. `--max-scan-events-per-day` overrides it for one invocation, and 0 skips the count and the check. The save path does not check it |
The Rust services have their own switches, and Django cannot see them.
-The processor must hold a team before the seeder's copy of `REALTIME_COHORT_TEAM_ALLOWLIST` names it.
-The processor holds a team that its own copy names, or about 15 seconds after a team access row grants it.
+The processor must hold a team before the seeder works on the team's runs.
+The processor holds a team that its own copy of `REALTIME_COHORT_TEAM_ALLOWLIST` names, or about 15 seconds after a team access row grants it.
+For a team access grant, the seeder waits until the row's `processing_since` is 120 seconds old.
+That covers a processor that reads team access and loads the catalog within that time, but nothing checks that every pod did.
+For env lists, the processor's copy must name a team before the seeder's copy does.
A processor that does not hold the team withholds every reconcile request of its runs with `team_absent`, and the seeder supersedes those participations.
No new run is owed afterwards, so the team never becomes ready until an operator creates one.
`seeder_reconcile_cohorts_withheld_total{reason="team_absent"}` shows that this happened.
+The processor also drops the team's person seeds, which skip the apply fence, and counts them in `cohort_person_seeds_dropped_total{reason="team_absent"}`.
+The membership of each dropped person stays missing until a live event for that person arrives.
+The run can still complete and be stamped ready over that missing membership.
+After a grant, check that counter and each processor pod's `posthog_cohort_team_access_last_success_timestamp_seconds`.
Two of the switches gate person runs, and they must open in order.
The processor's `COHORT_SEED_PERSON_APPLY_ENABLED` must be on everywhere before the seeder's `SEEDER_PERSON_SEEDS_ENABLED`, or the processor skips and commits the seeds.
The seeder's switch must be on before a team with person-leaf cohorts joins the trigger allowlist.
diff --git a/docs/internal/realtime-cohorts/event-routing.md b/docs/internal/realtime-cohorts/event-routing.md
index f8e54821deb8..1e3664bbf460 100644
--- a/docs/internal/realtime-cohorts/event-routing.md
+++ b/docs/internal/realtime-cohorts/event-routing.md
@@ -244,7 +244,7 @@ The processor sees firehose offset 9870001 from partition 3 for p-1 a second tim
- The shuffler and the processor each read their own copy of `REALTIME_COHORT_TEAM_ALLOWLIST` at startup, and each polls Postgres on its own schedule.
Both also let through the teams that a `cohort_realtime_team_access` row grants, and each reads that table about every 15 seconds.
Only teams that both let through are processed, so a newly granted team can lose about half a minute of events until both have read the grant.
- Merges also need the team in Node's merge allowlist, and backfills need it in the seeder's copy.
+ Merges also need the team in Node's merge allowlist.
- The processor recognizes applied events with a high-water mark per source partition, not a set of offsets.
The shuffler's producer does not guarantee ordering on retries, so an event that is overtaken by a later event from the same source partition is skipped as if it were a replay.
- The broker timestamp on `cohort_stream_events` is the shuffler's send time, not the event's time.
diff --git a/docs/internal/realtime-cohorts/seeder.md b/docs/internal/realtime-cohorts/seeder.md
index bd915703d14a..884a6b5486b1 100644
--- a/docs/internal/realtime-cohorts/seeder.md
+++ b/docs/internal/realtime-cohorts/seeder.md
@@ -14,7 +14,7 @@ It never touches the processor's state directly.
The seeder is a set of replicas polling Postgres, every 15 seconds by default.
Each tick it:
-1. discovers runs in `awaiting_boundary` or `seeding` for allowlisted teams and enabled kinds, where person runs need `SEEDER_PERSON_SEEDS_ENABLED`,
+1. discovers runs in `awaiting_boundary` or `seeding` for teams in its env list or granted in the team access table for at least 120 seconds, and for enabled kinds, where person runs need `SEEDER_PERSON_SEEDS_ENABLED`,
2. sets the boundary on newly discovered runs,
3. validates each run's pinned payload and plans its chunks,
4. fails chunks abandoned at the attempt cap, and fails runs that hold an exhausted chunk,
diff --git a/rust/cohort-seeder/src/observability/metrics.rs b/rust/cohort-seeder/src/observability/metrics.rs
index a9d362d1607e..5d36c8ba72b4 100644
--- a/rust/cohort-seeder/src/observability/metrics.rs
+++ b/rust/cohort-seeder/src/observability/metrics.rs
@@ -31,9 +31,9 @@ pub const LOOKBACK_TRUNCATED: &str = "seeder_lookback_truncated_total";
/// replica counts every run, because the once-per-run gate is process-local. The paired `info!` line
/// carries one run's exact numbers.
///
-/// `team_id` names a team only while `REALTIME_COHORT_TEAM_ALLOWLIST` names few enough of them, and
-/// collapses to `other` otherwise. It is here because the question these counters answer is about
-/// one team's catalog; blended across teams they answer nobody's.
+/// `team_id` names a team only while `REALTIME_COHORT_TEAM_ALLOWLIST` names it and few enough other
+/// teams, and collapses to `other` otherwise. It is here because the question these counters answer
+/// is about one team's catalog; blended across teams they answer nobody's.
pub const CONDITIONS_CLASSIFIED: &str = "seeder_conditions_classified_total";
/// Conditions the static read could not narrow, labelled by a closed `reason`, the `op` that
/// stopped it (`none` when no opcode did), and `team_id` (counter). Counted on the same
@@ -241,15 +241,17 @@ const UNLABELLED_TEAM: &str = "other";
/// The team a metric belongs to, as a label value.
///
-/// A team gets its own series only while `REALTIME_COHORT_TEAM_ALLOWLIST` names few enough of them.
-/// The allowlist does not bound this on its own: it accepts `all` (which a set-but-empty variable
-/// also parses to) and ranges of up to 100,000 ids, and the recorder never evicts a series, so a
-/// wide rollout would retain one per team that ever seeded. The label exists because these series
-/// answer questions about one team's catalog and scans, which a blended counter cannot; the paired
-/// log lines carry the exact team either way, so collapsing costs the dashboard, not the answer.
+/// A team gets its own series only while `REALTIME_COHORT_TEAM_ALLOWLIST` names it and few enough
+/// other teams. The allowlist does not bound this on its own: it accepts `all` (which a set-but-empty
+/// variable also parses to) and ranges of up to 100,000 ids, and the recorder never evicts a series,
+/// so a wide rollout would retain one per team that ever seeded. Teams that only the access record
+/// grants share the collapsed label too, because grants have no such bound. The label exists
+/// because these series answer questions about one team's catalog and scans, which a blended
+/// counter cannot; the paired log lines carry the exact team either way, so collapsing costs the
+/// dashboard, not the answer.
pub fn team_label(allowlist: &TeamAllowlist, team_id: TeamId) -> Arc<str> {
match allowlist {
- TeamAllowlist::Only(ids) if ids.len() <= MAX_LABELLED_TEAMS => {
+ TeamAllowlist::Only(ids) if ids.len() <= MAX_LABELLED_TEAMS && ids.contains(&team_id.0) => {
Arc::from(team_id.0.to_string().as_str())
}
TeamAllowlist::Only(_) | TeamAllowlist::All => Arc::from(UNLABELLED_TEAM),
@@ -531,6 +533,8 @@ mod tests {
fn only_a_narrow_allowlist_names_teams_in_a_per_team_label() {
let narrow = TeamAllowlist::Only(HashSet::from([2, 7]));
assert_eq!(&*team_label(&narrow, TeamId(2)), "2");
+ // A team outside the env list, which only the access record can have granted.
+ assert_eq!(&*team_label(&narrow, TeamId(3)), UNLABELLED_TEAM);
let wide = TeamAllowlist::Only((0..=MAX_LABELLED_TEAMS as i32).collect());
assert_eq!(&*team_label(&wide, TeamId(2)), UNLABELLED_TEAM);
diff --git a/rust/cohort-seeder/src/store/completion.rs b/rust/cohort-seeder/src/store/completion.rs
index b555f3119590..339562b36c53 100644
--- a/rust/cohort-seeder/src/store/completion.rs
+++ b/rust/cohort-seeder/src/store/completion.rs
@@ -106,7 +106,8 @@ const DISCOVER_COMPLETION_ONLY: &str = concat!(
reconciling_status!(),
")",
"\n AND backfill_kind = ANY($2)",
- "\n AND team_id = ANY($1)",
+ "\n AND ",
+ processing_team!(),
"\n AND (status = ",
seeding_status!(),
" OR reconcile_observed_at IS NULL)",
@@ -132,7 +133,8 @@ const COUNT_RECONCILING_ONLY: &str = concat!(
"\n WHERE status = ",
reconciling_status!(),
"\n AND backfill_kind = ANY($2)",
- "\n AND team_id = ANY($1)",
+ "\n AND ",
+ processing_team!(),
"\n GROUP BY backfill_kind\n"
);
@@ -882,6 +884,8 @@ pub struct DiscoveredCompletion {
/// classify each. Rows whose status or kind is outside the vocabulary the query already bounds are
/// skipped defensively. An observed run is the driver's exit condition, so it is filtered in SQL
/// rather than hydrated and then matched away — [`count_reconciling_by_kind`] still sees it.
+/// `allowlist` is the env list. `Only` also admits the grants that `processing_team!` reads from the
+/// access record.
pub async fn discover_completions(
pool: &PgPool,
allowlist: &TeamAllowlist,
@@ -916,6 +920,8 @@ pub async fn discover_completions(
/// is the whole point: while the person readiness gate is shut, that backlog is the only thing the
/// `seeder_runs_reconciling` gauge has to report, and it is exactly what the driver cannot act on.
/// One aggregate, so the gauge costs a count rather than a hydration of every parked row.
+/// `allowlist` is the env list. `Only` also admits the grants that `processing_team!` reads from the
+/// access record.
pub async fn count_reconciling_by_kind(
pool: &PgPool,
allowlist: &TeamAllowlist,
diff --git a/rust/cohort-seeder/src/store/mod.rs b/rust/cohort-seeder/src/store/mod.rs
index 6e53bd935661..2a81001636fe 100644
--- a/rust/cohort-seeder/src/store/mod.rs
+++ b/rust/cohort-seeder/src/store/mod.rs
@@ -4,6 +4,26 @@
//! establishment + pinned load), `lease` (the heartbeat handle). Depends on `domain` (typed ids and
//! rows) and the `observability` metric constants — nothing else in the crate.
+// The team half of every `*_ONLY` statement, with `$1` bound to the env list's team ids. A team that
+// only the access record grants counts 120 s after `processing_since`. Person seeds skip the
+// processor's apply fence, and a processor that does not hold the team yet drops them as
+// `team_absent` for good. The processor picks up a grant on its next access poll, every 15 ± 3 s
+// (`common-cohort-team-access`), and then loads the team's cohorts. The wait cannot see a processor
+// pod whose polls or loads keep failing for longer than 120 s. Django creates runs only in
+// `awaiting_boundary` or `blocked`, so every run that completion and the reconciling count read
+// passed discovery first, and the wait delays nothing there. The levels are listed instead of
+// `<> 'off'`, so a level that Django adds later stalls the team's runs until the seeder knows it.
+// Defined before the submodules, because a `macro_rules!` is in scope only after its definition.
+macro_rules! processing_team {
+ () => {
+ "(team_id = ANY($1) OR EXISTS (
+ SELECT 1 FROM cohort_realtime_team_access access
+ WHERE access.team_id = cohort_backfill_runs.team_id
+ AND access.level IN ('processing', 'serving')
+ AND access.processing_since <= now() - interval '120 seconds'))"
+ };
+}
+
pub mod chunks;
pub mod completion;
pub mod lease;
diff --git a/rust/cohort-seeder/src/store/runs.rs b/rust/cohort-seeder/src/store/runs.rs
index 0dfd071cb90d..8bd4431b47ed 100644
--- a/rust/cohort-seeder/src/store/runs.rs
+++ b/rust/cohort-seeder/src/store/runs.rs
@@ -63,7 +63,8 @@ const DISCOVER_ONLY: &str = concat!(
discoverable_run!(),
")",
"\n AND backfill_kind = ANY($2)",
- "\n AND team_id = ANY($1)",
+ "\n AND ",
+ processing_team!(),
"\n ORDER BY created_at\n"
);
@@ -644,6 +645,8 @@ struct BoundaryRow {
boundary_at: DateTime<Utc>,
}
+/// `allowlist` is the env list. `Only` also admits the grants that `processing_team!` reads from the
+/// access record.
pub async fn discover_runs(
pool: &PgPool,
allowlist: &TeamAllowlist,
diff --git a/rust/cohort-seeder/tests/fixtures/cohort_backfill_0015.sql b/rust/cohort-seeder/tests/fixtures/cohort_backfill_0017.sql
similarity index 81%
rename from rust/cohort-seeder/tests/fixtures/cohort_backfill_0015.sql
rename to rust/cohort-seeder/tests/fixtures/cohort_backfill_0017.sql
index 04ae85ba2438..360763653d80 100644
--- a/rust/cohort-seeder/tests/fixtures/cohort_backfill_0015.sql
+++ b/rust/cohort-seeder/tests/fixtures/cohort_backfill_0017.sql
@@ -1,17 +1,16 @@
--- Snapshot pinned to
--- products/cohorts/backend/migrations/0015_cohortbackfillchunk_claimable_after_and_more.py, the last
--- migration that changed this DDL. 0010 alters only `marker_watch`'s help_text, which emits no SQL,
--- 0011 touches posthog_cohort columns this projection does not carry, and 0013 and 0014 emit none
--- for these tables.
+-- Snapshot pinned to products/cohorts/backend/migrations/0017_realtime_cohort_team_access.py, the
+-- last migration that changed this DDL. 0010 alters only `marker_watch`'s help_text, which emits no
+-- SQL, 0011 and 0016 touch posthog_cohort columns this projection does not carry, and 0013 and 0014
+-- emit none for these tables.
-- External Team/Cohort foreign keys are omitted so the contract test stays schema-local.
--
-- Nothing enforces this snapshot against Django. No test diffs it with `sqlmigrate`, and the
-- filename appears at one call site, so a later migration that changes these tables and forgets
-- this file passes CI and diverges from production. Re-derive it by hand when you touch the
--- `cohort_backfill_*` DDL: `python manage.py sqlmigrate cohorts <number>` prints the statements the
--- migration emits (CI runs the same command to post migration SQL on a PR). Fold them in, drop the
--- external Team/Cohort foreign key lines to keep the projection schema-local, and rename this file
--- and its `include_str!` to the new number.
+-- `cohort_backfill_*` or `cohort_realtime_team_access` DDL: `python manage.py sqlmigrate cohorts
+-- <number>` prints the statements the migration emits (CI runs the same command to post migration
+-- SQL on a PR). Fold them in, drop the external Team/Cohort foreign key lines to keep the
+-- projection schema-local, and rename this file and its `include_str!` to the new number.
CREATE TABLE cohort_backfill_runs (
id uuid PRIMARY KEY,
@@ -107,6 +106,17 @@ CREATE TABLE cohort_backfill_run_cohorts (
CONSTRAINT cohort_bfrc_run_cohort_uq UNIQUE (run_id, cohort_id)
);
+-- 0017. The model sets `db_constraint=False`, so the table has no Team foreign key to drop.
+CREATE TABLE cohort_realtime_team_access (
+ team_id integer PRIMARY KEY,
+ level varchar(16) NOT NULL,
+ purpose varchar(16) NOT NULL,
+ save_triggers_backfill boolean NOT NULL,
+ processing_since timestamptz,
+ off_since timestamptz,
+ changed_at timestamptz NOT NULL
+);
+
-- A minimal projection of posthog_cohort, present only so load_current_behavioral_hashes can read a
-- cohort's current behavioral shape hash when attributing a shortfall. Not part of the cohorts app's
-- backfill migrations.
diff --git a/rust/cohort-seeder/tests/pg_chunks.rs b/rust/cohort-seeder/tests/pg_chunks.rs
index 266c7f9db62d..5d183629907d 100644
--- a/rust/cohort-seeder/tests/pg_chunks.rs
+++ b/rust/cohort-seeder/tests/pg_chunks.rs
@@ -37,9 +37,9 @@ use uuid::Uuid;
mod support;
use support::{
- behavioral_filter, empty_pinned, ensure_lease_lost, historical, insert_participation,
- insert_person_run, insert_run, person_filter, person_pinned, pinned_condition, planned_count,
- trailing, with_db, ACTIVE_HASH, SUPERSEDED_HASH,
+ behavioral_filter, empty_pinned, ensure_lease_lost, historical, insert_access,
+ insert_participation, insert_person_run, insert_run, person_filter, person_pinned,
+ pinned_condition, planned_count, trailing, with_db, ACTIVE_HASH, SUPERSEDED_HASH,
};
const ONE_BAND: NonZeroU16 = NonZeroU16::MIN;
@@ -59,8 +59,8 @@ fn idle(run_ids: &[RunId]) -> ClaimCandidates {
ClaimCandidates::new(candidates).expect("every scenario claims from at least one run")
}
-/// Discovery honors an `Only` allowlist (self team only) and `All` admits every eligible run
-/// regardless of team, trigger, or already-seeding status.
+/// Discovery honors an `Only` allowlist plus the teams the access record has granted for 120 s, and
+/// `All` admits every eligible run regardless of team, trigger, or already-seeding status.
#[tokio::test]
async fn discovery_scopes_to_the_allowlist_and_all_admits_every_run() -> Result<()> {
with_db(|pool| async move {
@@ -93,11 +93,33 @@ async fn discovery_scopes_to_the_allowlist_and_all_admits_every_run() -> Result<
.await?;
let cross_team_run =
insert_run(&pool, 5, "team_enablement", "seeding", true, empty_pinned()).await?;
+ // Grant ages sit 30 s either side of the 120 s wait, so a changed wait fails this test.
+ for (team_id, level, granted_ago) in [
+ (6, "processing", Duration::from_secs(150)),
+ (7, "serving", Duration::from_secs(150)),
+ (8, "processing", Duration::from_secs(90)),
+ (9, "off", Duration::from_secs(150)),
+ ] {
+ insert_access(&pool, team_id, level, granted_ago).await?;
+ insert_run(
+ &pool,
+ team_id,
+ "team_enablement",
+ "awaiting_boundary",
+ false,
+ empty_pinned(),
+ )
+ .await?;
+ }
let only_team_two: TeamAllowlist = "2".parse().map_err(anyhow::Error::msg)?;
let discovered = discover_runs(&pool, &only_team_two, &[RunKind::Behavioral]).await?;
- ensure!(discovered.len() == 1);
- ensure!(discovered[0].run_id == self_run);
+ let mut discovered_teams = discovered
+ .iter()
+ .map(|run| run.team_id.0)
+ .collect::<Vec<_>>();
+ discovered_teams.sort_unstable();
+ ensure!(discovered_teams == [2, 6, 7], "{discovered_teams:?}");
let all_runs = discover_runs(&pool, &TeamAllowlist::All, &[RunKind::Behavioral]).await?;
ensure!(all_runs.iter().any(|run| run.run_id == self_run));
diff --git a/rust/cohort-seeder/tests/pg_completion.rs b/rust/cohort-seeder/tests/pg_completion.rs
index 2a9232f782cf..d470c7a27d16 100644
--- a/rust/cohort-seeder/tests/pg_completion.rs
+++ b/rust/cohort-seeder/tests/pg_completion.rs
@@ -39,9 +39,9 @@ use sqlx::PgPool;
mod support;
use support::{
- empty_pinned, ensure_fence_lost, historical, insert_cohort, insert_participation,
- insert_person_participation, insert_person_run, insert_reconciling_run, insert_run,
- person_pinned, set_marker_bits, trailing, with_db,
+ empty_pinned, ensure_fence_lost, historical, insert_access, insert_cohort,
+ insert_participation, insert_person_participation, insert_person_run, insert_reconciling_run,
+ insert_run, person_pinned, set_marker_bits, trailing, with_db,
};
const ONE_BAND: NonZeroU16 = NonZeroU16::MIN;
@@ -1082,12 +1082,17 @@ async fn discovery_classifies_each_phase_and_drops_observed_runs() -> Result<()>
ensure!(!counts.contains_key(&RunKind::PersonProperty));
// Both statements have a team-scoped variant, and dev — where this rollout lands first —
- // is the environment that runs allowlisted.
+ // is the environment that runs allowlisted. Both also admit a team the access record grants.
+ let granted = insert_reconciling_run(&pool, 8).await?;
+ insert_access(&pool, 8, "processing", Duration::from_secs(600)).await?;
let allowlisted = TeamAllowlist::Only([4, 6].into_iter().collect());
let scoped = discover_completions(&pool, &allowlisted, BOTH_KINDS, MARKER_TOPIC).await?;
- ensure!(scoped.iter().map(|row| row.run_id).eq([dispatched]));
+ ensure!(scoped
+ .iter()
+ .map(|row| row.run_id)
+ .eq([dispatched, granted]));
let scoped_counts = count_reconciling_by_kind(&pool, &allowlisted, BOTH_KINDS).await?;
- ensure!(scoped_counts.get(&RunKind::Behavioral) == Some(&2));
+ ensure!(scoped_counts.get(&RunKind::Behavioral) == Some(&3));
Ok(())
})
.await
diff --git a/rust/cohort-seeder/tests/support/mod.rs b/rust/cohort-seeder/tests/support/mod.rs
index 438f2d29c3e8..84db3263c56b 100644
--- a/rust/cohort-seeder/tests/support/mod.rs
+++ b/rust/cohort-seeder/tests/support/mod.rs
@@ -13,6 +13,7 @@
use std::future::Future;
use std::str::FromStr;
+use std::time::Duration;
use anyhow::{bail, Context, Result};
use chrono::{DateTime, Utc};
@@ -25,9 +26,9 @@ use sqlx::types::Json;
use sqlx::{Connection, PgConnection, PgPool};
use uuid::Uuid;
-/// The `cohort_backfill_*` DDL (plus a schema-local `posthog_cohort` projection), pinned to Django
-/// migration 0015, applied fresh into each test schema.
-pub const DDL: &str = include_str!("../fixtures/cohort_backfill_0015.sql");
+/// The `cohort_backfill_*` and `cohort_realtime_team_access` DDL (plus a schema-local
+/// `posthog_cohort` projection), pinned to Django migration 0017, applied fresh into each test schema.
+pub const DDL: &str = include_str!("../fixtures/cohort_backfill_0017.sql");
/// A live cohort condition hash used by the superseded-load fixtures.
pub const ACTIVE_HASH: &str = "active0000000000";
/// A superseded cohort condition hash used by the superseded-load fixtures.
@@ -178,6 +179,27 @@ pub async fn insert_cohort(
Ok(())
}
+/// Write a team access row at `level`, granted `granted_ago` before the Postgres clock that the
+/// seeder's team predicate compares `processing_since` against.
+pub async fn insert_access(
+ pool: &PgPool,
+ team_id: i32,
+ level: &str,
+ granted_ago: Duration,
+) -> Result<()> {
+ sqlx::query(
+ "INSERT INTO cohort_realtime_team_access \
+ (team_id, level, purpose, save_triggers_backfill, processing_since, changed_at) \
+ VALUES ($1, $2, 'pilot', false, now() - $3, now())",
+ )
+ .bind(team_id)
+ .bind(level)
+ .bind(granted_ago)
+ .execute(pool)
+ .await?;
+ Ok(())
+}
+
/// Unwrap the inserted-chunk count, failing if the run was unexpectedly not seeding or planned.
pub fn planned_count(outcome: PlanOutcome) -> Result<u64> {
match outcome {