Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(cohorts): read the team access record in the stream processor (#127899)

Matheus Batistab1db2d9aabe9
diff --git a/docs/internal/realtime-cohorts/backfill-coordination.md b/docs/internal/realtime-cohorts/backfill-coordination.md
index f75e1a71fcd5..983fbae9b83a 100644
--- a/docs/internal/realtime-cohorts/backfill-coordination.md
+++ b/docs/internal/realtime-cohorts/backfill-coordination.md
@@ -152,17 +152,18 @@ The membership consumer does count those markers toward its sweep, see [membersh
Several independent gates decide whether anything happens.
Missing one of them fails quietly, so they are worth knowing.
-| Gate | Effect |
-| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `REALTIME_COHORT_TEAM_ALLOWLIST` | Teams whose cohorts get hash maintenance, stamp invalidation, supersession and any run at all. Unset means no teams in Django, but the shuffler, processor and seeder default to team 2, so set it explicitly on every service. Set but empty means **all** teams. Django puts a listed team at the `serving` level, and adds the teams that have a team access row. The shufflers add those teams too, but the processor's and seeder's lists must name both. Never list a team granted `processing` here, because that makes it `serving` |
-| Team access rows | `cohort_realtime_team_access`, written only by `realtime_cohort_team_access grant\|revoke\|list`. Django and the shufflers read it as a union with their `REALTIME_COHORT_TEAM_ALLOWLIST`; the processor and seeder read only their env lists. Until they read it, keep their env lists in step with the rows. A team granted only here gets runs nobody claims, each holding its cohort's run slot. A team revoked only here loses Django's edit-time supersede while the processor and seeder still process it, so its readiness stamps outlive later edits. A shuffler whose env list does not name the team stops forwarding its events. A row at `processing` gets the Django behavior above, but the team's users see no realtime state: no readiness, no realtime batch audience, and no backfilled cohort in flags or workflows, even one stamped before the grant, so a workflow that already uses one fails its next save. A row at `serving` also shows them. `revoke` sets `off`, which hides the stamps the same way, and a revoked team cannot be granted again. `grant` and `revoke` refuse a team on the env list |
-| `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST` | Teams whose saves create runs automatically. Unset or empty means **no** teams. A granted row with `save_triggers_backfill` adds its team, so `resave_cohorts --no-backfill-triggers`, not this setting, is what stops a resave from queueing runs |
-| Operator attestations | Settings an operator sets to declare that prerequisites hold. They are declarations, not live checks. Behavioral runs need `BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED` and `BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED`. Person runs also need the person TTL and sizing attestations and a positive seed-bytes budget |
-| Person sizing budget | A cohort-scoped person run is refused when the team's active person-run estimates plus its own would exceed the budget. The check reads the active estimates before the run exists and reserves nothing, so two cohorts sized at the same time can together exceed the budget. A team run checks only its own estimate. Hitting the size estimate's scan cap, or the cap on pinned conditions, also refuses |
-| Behavioral scan limit | `create_cohort_backfill_run` refuses a behavioral team run when its pinned event names had more events than `BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY` on the busiest of the last 7 complete UTC days. `--max-scan-events-per-day` overrides it for one invocation, and 0 skips the count and the check. The save path does not check it |
+| Gate | Effect |
+| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `REALTIME_COHORT_TEAM_ALLOWLIST` | Teams whose cohorts get hash maintenance, stamp invalidation, supersession and any run at all. Unset means no teams in Django, but the shuffler, processor and seeder default to team 2, so set it explicitly on every service. Set but empty means **all** teams. Django puts a listed team at the `serving` level, and adds the teams that have a team access row. The shufflers and the processor add those teams too, but the seeder's list must name both. Never list a team granted `processing` here, because that makes it `serving` |
+| Team access rows | `cohort_realtime_team_access`, written only by `realtime_cohort_team_access grant\|revoke\|list`. Django, the shufflers and the processor read it as a union with their `REALTIME_COHORT_TEAM_ALLOWLIST`; the seeder reads only its env list. Until it reads the table, keep its env list in step with the rows. Node ingestion does not read the table, and emits person merges only for the teams in its `PERSON_MERGE_EVENTS_TEAM_ALLOWLIST`, so add a granted team there too. A team granted only here gets runs nobody claims, each holding its cohort's run slot. A team revoked only here loses Django's edit-time supersede while the seeder still processes it, so its readiness stamps outlive later edits. The processor drops a revoked team from its catalog, and stage 2 GC then deletes the team's state without emitting `left`, so the team's `cohort_membership` rows stay `in_cohort=true`. A shuffler whose env list does not name the team stops forwarding its events. A row at `processing` gets the Django behavior above, but the team's users see no realtime state: no readiness, no realtime batch audience, and no backfilled cohort in flags or workflows, even one stamped before the grant, so a workflow that already uses one fails its next save. A row at `serving` also shows them. `revoke` sets `off`, which hides the stamps the same way, and a revoked team cannot be granted again. `grant` and `revoke` refuse a team on the env list |
+| `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST` | Teams whose saves create runs automatically. Unset or empty means **no** teams. A granted row with `save_triggers_backfill` adds its team, so `resave_cohorts --no-backfill-triggers`, not this setting, is what stops a resave from queueing runs |
+| Operator attestations | Settings an operator sets to declare that prerequisites hold. They are declarations, not live checks. Behavioral runs need `BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED` and `BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED`. Person runs also need the person TTL and sizing attestations and a positive seed-bytes budget |
+| Person sizing budget | A cohort-scoped person run is refused when the team's active person-run estimates plus its own would exceed the budget. The check reads the active estimates before the run exists and reserves nothing, so two cohorts sized at the same time can together exceed the budget. A team run checks only its own estimate. Hitting the size estimate's scan cap, or the cap on pinned conditions, also refuses |
+| Behavioral scan limit | `create_cohort_backfill_run` refuses a behavioral team run when its pinned event names had more events than `BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY` on the busiest of the last 7 complete UTC days. `--max-scan-events-per-day` overrides it for one invocation, and 0 skips the count and the check. The save path does not check it |
The Rust services have their own switches, and Django cannot see them.
-A team must reach the processor's copy of `REALTIME_COHORT_TEAM_ALLOWLIST` before the seeder's.
+The processor must hold a team before the seeder's copy of `REALTIME_COHORT_TEAM_ALLOWLIST` names it.
+The processor holds a team that its own copy names, or about 15 seconds after a team access row grants it.
A processor that does not hold the team withholds every reconcile request of its runs with `team_absent`, and the seeder supersedes those participations.
No new run is owed afterwards, so the team never becomes ready until an operator creates one.
`seeder_reconcile_cohorts_withheld_total{reason="team_absent"}` shows that this happened.
diff --git a/docs/internal/realtime-cohorts/definitions-and-eligibility.md b/docs/internal/realtime-cohorts/definitions-and-eligibility.md
index b66dacf2d83d..5b53f3661d3d 100644
--- a/docs/internal/realtime-cohorts/definitions-and-eligibility.md
+++ b/docs/internal/realtime-cohorts/definitions-and-eligibility.md
@@ -183,8 +183,9 @@ Every Rust service that needs definitions builds a **catalog**: an immutable, pe
The shared `cohort-core` crate owns the parsing, classification and indexing, so the processor and the seeder interpret a definition with the same code.
- The **processor** loads every row with `cohort_type = 'realtime' AND deleted = false AND filters IS NOT NULL`, joined to the team's timezone.
- It keeps only teams in its own copy of `REALTIME_COHORT_TEAM_ALLOWLIST`, builds the catalog off the async runtime, and swaps it in atomically.
- It refreshes every few minutes with jitter.
+ It keeps only the teams in its own copy of `REALTIME_COHORT_TEAM_ALLOWLIST` and the teams that a `cohort_realtime_team_access` row grants, builds the catalog off the async runtime, and swaps it in atomically.
+ A refresh whose read of that table fails builds nothing, so the catalog never misses a team that the table granted before the refresh began, as long as the database it reads is current.
+ It refreshes every few minutes with jitter, and within about 15 seconds of a grant or a revoke.
A failed refresh keeps the previous catalog, so a database outage freezes definitions instead of emptying them.
- The **seeder** never loads the live definitions.
It builds a catalog from the filters Django pinned into each backfill run, so a run's seeds replay exactly the definition it was created for.
diff --git a/docs/internal/realtime-cohorts/event-routing.md b/docs/internal/realtime-cohorts/event-routing.md
index 67daae28aa9c..f8e54821deb8 100644
--- a/docs/internal/realtime-cohorts/event-routing.md
+++ b/docs/internal/realtime-cohorts/event-routing.md
@@ -51,7 +51,7 @@ An event passes the gate when it has a person id and its team passes two checks.
So when a read lacks a row the shuffler holds, or grants one of its `off` rows again, the shuffler keeps its previous row for that team, because that read comes from a lagging replica or the wrong database.
The read's other rows still take effect, so a revoke always gets through.
After a failed read, it keeps all the previous rows.
- A row deleted or granted again by hand keeps its old state in every running shuffler, so restart both shuffler deployments after one.
+ A row deleted or granted again by hand keeps its old state in every running shuffler and processor, so restart both shuffler deployments and the processor after one.
Apart from the person id check, the gate looks at the team only.
It cannot filter by event name: person-property criteria are re-evaluated from every event that carries person properties, whatever the event is called.
@@ -242,8 +242,8 @@ The processor sees firehose offset 9870001 from partition 3 for p-1 a second tim
- The processor's event-name gate only skips behavioral evaluation, after the event is already on `cohort_stream_events`.
- The shuffler and the processor each read their own copy of `REALTIME_COHORT_TEAM_ALLOWLIST` at startup, and each polls Postgres on its own schedule.
- The shuffler also lets through the teams that a `cohort_realtime_team_access` row grants, but the processor reads only its list.
- Only teams that both let through are processed, and a newly enabled team can lose a few minutes of events until both have refreshed.
+ Both also let through the teams that a `cohort_realtime_team_access` row grants, and each reads that table about every 15 seconds.
+ Only teams that both let through are processed, so a newly granted team can lose about half a minute of events until both have read the grant.
Merges also need the team in Node's merge allowlist, and backfills need it in the seeder's copy.
- The processor recognizes applied events with a high-water mark per source partition, not a set of offsets.
The shuffler's producer does not guarantee ordering on retries, so an event that is overtaken by a later event from the same source partition is skipped as if it were a replay.
diff --git a/docs/internal/realtime-cohorts/processor-runtime.md b/docs/internal/realtime-cohorts/processor-runtime.md
index 59af0328160e..cc85879b3c6f 100644
--- a/docs/internal/realtime-cohorts/processor-runtime.md
+++ b/docs/internal/realtime-cohorts/processor-runtime.md
@@ -268,7 +268,7 @@ At boot the processor:
1. validates its configuration, refuses unsafe combinations, and works out the partitions it owns,
2. binds the health server, so `/_health` answers from here on and `/_ready` reads not ready,
-3. loads the catalog once,
+3. reads team access and loads the catalog once,
4. checks the partition counts of the co-partitioned topics against broker metadata,
5. waits until the [ownership fence](#ownership-and-the-single-pod-deployment) admits its split,
6. decides where the store comes from: with durable restore enabled, the live store on disk or a checkpoint, otherwise an empty store.
@@ -381,8 +381,11 @@ Two variations show the edges.
On the event path it is not.
Only the merge, cascade and seed paths hold.
- The events consumer does not wait for the catalog.
- If the first catalog load fails, events are consumed, skipped for lack of definitions, and committed.
+ If the first catalog load fails, which includes a failed read of team access, events are consumed, skipped for lack of definitions, and committed.
The pod reads not ready meanwhile.
+- A processor image that does not read team access builds its catalog from its env list alone.
+ A rollback to such an image drops every team that only a team access row grants, and stage 2 GC then deletes the state of those teams without emitting `left`.
+ Before such a rollback, add those teams to the processor's `REALTIME_COHORT_TEAM_ALLOWLIST`.
- A held offset stays held for the rest of the partition's tenure, which means until the next restart.
The symptom is growing lag on that topic and partition.
- A created store withholds every reconcile whose run boundary predates its slices, because each slice begins when its partition is assigned.
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index 440405416836..90a9275f3396 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -2853,6 +2853,7 @@ dependencies = [
"chrono-tz",
"cohort-core",
"common-alloc",
+ "common-cohort-team-access",
"common-database",
"common-kafka",
"common-liveness",
diff --git a/rust/cohort-stream-processor/Cargo.toml b/rust/cohort-stream-processor/Cargo.toml
index 90cbef16ff32..bc8864a51a0a 100644
--- a/rust/cohort-stream-processor/Cargo.toml
+++ b/rust/cohort-stream-processor/Cargo.toml
@@ -19,6 +19,7 @@ chrono = { workspace = true }
chrono-tz = { workspace = true }
cohort-core = { path = "../cohort-core" }
common-alloc = { path = "../common/alloc" }
+common-cohort-team-access = { path = "../common/cohort-team-access" }
common-database = { path = "../common/database" }
common-types = { path = "../common/types" }
# Shadow-topic output producer: `create_kafka_producer` + `send_keyed_iter_to_kafka_*`
@@ -79,6 +80,7 @@ hotpath-prometheus = ["hotpath", "hotpath/hotpath-prometheus"]
# Integration tests in `tests/` build `TeamFilters` via `freeze(UTC)`, so they name `chrono_tz::UTC`
# directly (a normal dependency is not in scope for the separate integration-test crates).
chrono-tz = { workspace = true }
+common-cohort-team-access = { path = "../common/cohort-team-access", features = ["test-util"] }
# Convergence/idempotency property tests for the seed-tile apply core (`workers::seed_path`).
proptest = "1"
serde_json = { workspace = true }
diff --git a/rust/cohort-stream-processor/src/config.rs b/rust/cohort-stream-processor/src/config.rs
index a991464dcf3b..8dc67ab4e154 100644
--- a/rust/cohort-stream-processor/src/config.rs
+++ b/rust/cohort-stream-processor/src/config.rs
@@ -64,7 +64,8 @@ pub struct Config {
#[envconfig(default = "60")]
pub filter_catalog_refresh_jitter_secs: u64,
- /// Teams the filter catalog is scoped to. Set `all` to disable the gate. See [`TeamAllowlist`].
+ /// Teams the filter catalog covers, on top of the teams that a row of
+ /// `cohort_realtime_team_access` grants. Set `all` to disable the gate. See [`TeamAllowlist`].
#[envconfig(from = "REALTIME_COHORT_TEAM_ALLOWLIST", default = "2")]
pub team_allowlist: TeamAllowlist,
diff --git a/rust/cohort-stream-processor/src/filters/manager.rs b/rust/cohort-stream-processor/src/filters/manager.rs
index 99d70df2aa4d..e53b729626b7 100644
--- a/rust/cohort-stream-processor/src/filters/manager.rs
+++ b/rust/cohort-stream-processor/src/filters/manager.rs
@@ -1,13 +1,16 @@
-//! `FilterCatalog` + atomic swap + jittered periodic refresh loop.
+//! `FilterCatalog` + atomic swap + the refresher that rebuilds it on a team access change or a
+//! jittered period.
use std::collections::hash_map::DefaultHasher;
-use std::future::Future;
use std::hash::{Hash, Hasher};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use arc_swap::{ArcSwap, Guard};
+use common_cohort_team_access::{
+ next_poll_delay, PollError, ProcessingTeams, TeamAccess, TeamAccessPoller,
+};
use common_types::cohort::TeamAllowlist;
use lifecycle::Handle;
use metrics::{counter, gauge, histogram};
@@ -28,7 +31,7 @@ use crate::observability::metrics::{
FILTER_CATALOG_REFRESH_TOTAL, FILTER_CATALOG_TEAMS, FILTER_CATALOG_UNIQUE_CONDITIONS,
};
-/// Snapshot counts returned by [`CatalogHandle::refresh`] for logging.
+/// Snapshot counts returned by [`CatalogRefresher::refresh`] for logging.
#[derive(Debug, Clone, Copy)]
pub struct CatalogStats {
pub teams: usize,
@@ -37,8 +40,13 @@ pub struct CatalogStats {
#[derive(Debug, Error)]
pub enum CatalogRefreshError {
+ /// The refresh has no access read that began after it did, so it reads no cohort rows.
+ #[error("reading team access failed: {0}")]
+ Access(#[from] PollError),
#[error(transparent)]
Load(#[from] FilterError),
+ #[error("loading posthog_cohort timed out")]
+ LoadTimedOut(#[from] tokio::time::error::Elapsed),
#[error("filter catalog build worker failed: {0}")]
Build(#[from] JoinError),
}
@@ -64,8 +72,8 @@ pub(crate) enum CatalogView {
/// Lock-free, atomically-swapped catalog handle. Starts empty and unloaded; the pipeline fails
/// closed until the first successful refresh.
///
-/// The refresh loop is its only writer after boot, and the boot refresh ends before the loop
-/// starts, so refreshes never overlap.
+/// Its [`CatalogRefresher`] is its only writer, and a refresh takes the refresher by `&mut`, so
+/// refreshes never overlap.
pub struct CatalogHandle {
catalog: ArcSwap<FilterCatalog>,
/// Refreshes started. Refresh `n` takes `n` when it starts.
@@ -73,34 +81,50 @@ pub struct CatalogHandle {
/// Sequence of the published snapshot, 0 before the first store.
published: AtomicU64,
loaded_notify: Notify,
- /// Cohorts for teams outside this allowlist never enter the catalog.
- allowlist: TeamAllowlist,
- cascade_enabled: bool,
- /// Last stamped generation, advanced by [`Self::store`]. Single-writer (the refresh loop),
- /// so `Relaxed` suffices.
+ /// Last stamped generation, advanced by [`Self::store`]. Single-writer (the refresher), so
+ /// `Relaxed` suffices.
current_generation: AtomicU64,
/// Signature of the last stored catalog, to detect a no-op refresh.
current_signature: AtomicU64,
}
impl CatalogHandle {
+ /// An unloaded handle that nothing refreshes from Postgres.
pub fn new() -> Self {
- Self::with_allowlist(TeamAllowlist::All, false)
- }
-
- pub fn with_allowlist(allowlist: TeamAllowlist, cascade_enabled: bool) -> Self {
Self {
catalog: ArcSwap::from_pointee(FilterCatalog::new()),
begun: AtomicU64::new(0),
published: AtomicU64::new(0),
loaded_notify: Notify::new(),
- allowlist,
- cascade_enabled,
current_generation: AtomicU64::new(0),
current_signature: AtomicU64::new(0),
}
}
+ /// An unloaded handle and the one refresher that publishes to it, for the teams in `env` and the
+ /// teams that a `cohort_realtime_team_access` row grants.
+ pub fn with_refresher(
+ env: TeamAllowlist,
+ pool: PgPool,
+ interval: Duration,
+ jitter: Duration,
+ cascade_enabled: bool,
+ ) -> (Arc<Self>, CatalogRefresher) {
+ let catalog = Arc::new(Self::new());
+ let (access, poller) = TeamAccess::new(env);
+ let refresher = CatalogRefresher {
+ catalog: catalog.clone(),
+ access,
+ poller,
+ pool,
+ interval,
+ jitter,
+ cascade_enabled,
+ last: None,
+ };
+ (catalog, refresher)
+ }
+
/// Load the current snapshot for a wait-free hot-path read.
pub fn load(&self) -> Guard<Arc<FilterCatalog>> {
self.catalog.load()
@@ -204,54 +228,6 @@ impl CatalogHandle {
self.loaded_notify.notify_waiters();
}
- /// Query `posthog_cohort` for the allowlisted teams, rebuild the catalog, and swap it in.
- ///
- /// The staleness metrics are stamped here rather than in [`run_refresh_loop`] so the boot load
- /// in `main` counts too — otherwise a pod that booted fine and then lost its refresh loop would
- /// look identical to one that never loaded.
- #[hotpath::measure(impl_type = "CatalogHandle")]
- pub async fn refresh(&self, pool: &PgPool) -> Result<CatalogStats, CatalogRefreshError> {
- match self
- .refresh_from(|| load_allowlisted(pool, &self.allowlist))
- .await
- {
- Ok(stats) => {
- gauge!(FILTER_CATALOG_LAST_SUCCESS_TIMESTAMP_SECONDS).set(now_unix_seconds());
- counter!(FILTER_CATALOG_REFRESH_TOTAL, "result" => "success").increment(1);
- Ok(stats)
- }
- Err(err) => {
- counter!(FILTER_CATALOG_REFRESH_TOTAL, "result" => "error").increment(1);
- Err(err)
- }
- }
- }
-
- /// `load` reads the cohort rows. The refresh begins before it runs, so every write committed
- /// before the read is in what it returns.
- async fn refresh_from<Rows>(
- &self,
- load: impl FnOnce() -> Rows,
- ) -> Result<CatalogStats, CatalogRefreshError>
- where
- Rows: Future<Output = Result<Vec<CohortRow>, FilterError>>,
- {
- let seq = self.begin();
- let mut rows = load().await?;
- let fetched_rows = rows.len();
- retain_allowlisted(&mut rows, &self.allowlist);
- if rows.len() != fetched_rows {
- debug!(
- fetched_rows,
- kept_rows = rows.len(),
- "filter catalog dropped cohort rows outside REALTIME_COHORT_TEAM_ALLOWLIST",
- );
- }
- let cascade_enabled = self.cascade_enabled;
- self.build_and_store(seq, move || build_catalog_from_rows(rows, cascade_enabled))
- .await
- }
-
async fn build_and_store(
&self,
seq: RefreshSeq,
@@ -282,11 +258,17 @@ fn now_unix_seconds() -> f64 {
.unwrap_or(0.0)
}
-async fn load_allowlisted(
+/// Bounds the cohort load on the client. The statement timeout stops only the server, so a socket
+/// that dies after the pool hands it out would otherwise hang the refresher, and with it every grant
+/// and revoke. Keep it longer than `PG_ACQUIRE_TIMEOUT_SECS` plus `PG_STATEMENT_TIMEOUT_MS`, so the
+/// pool and the server end a slow load first.
+const LOAD_TIMEOUT: Duration = Duration::from_secs(30);
+
+async fn load_processing(
pool: &PgPool,
- allowlist: &TeamAllowlist,
+ teams: &ProcessingTeams,
) -> Result<Vec<CohortRow>, FilterError> {
- match allowlist {
+ match teams.scope() {
TeamAllowlist::All => load_realtime_cohorts(pool).await,
TeamAllowlist::Only(team_ids) => {
let team_ids: Vec<i32> = team_ids.iter().copied().collect();
@@ -296,8 +278,8 @@ async fn load_allowlisted(
}
/// Backstop for the load query's team filter.
-fn retain_allowlisted(rows: &mut Vec<CohortRow>, allowlist: &TeamAllowlist) {
- rows.retain(|row| allowlist.includes(row.team_id));
+fn retain_processing(rows: &mut Vec<CohortRow>, teams: &ProcessingTeams) {
+ rows.retain(|row| teams.scope().includes(row.team_id));
}
/// A refresh that has begun and not yet published (test seam).
@@ -330,31 +312,199 @@ fn catalog_signature(catalog: &FilterCatalog) -> u64 {
hasher.finish()
}
-/// Periodic catalog refresh task. On failure, keeps the previous snapshot and retries next tick.
-pub async fn run_refresh_loop(
+/// Why a tick of [`CatalogRefresher::run`] refreshes the catalog.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum RefreshReason {
+ /// No refresh has published, so a failed boot load retries on every access poll.
+ NeverPublished,
+ /// Team access lets through other teams than the published catalog covers.
+ AccessChanged,
+ /// `interval ± jitter` has passed since the last refresh.
+ Due,
+}
+
+/// The teams of the published catalog, and when the next periodic refresh is due.
+#[derive(Debug)]
+struct Published {
+ teams: Arc<ProcessingTeams>,
+ due: Instant,
+}
+
+/// The only writer of its [`CatalogHandle`] outside tests. It owns the team access poller, and each
+/// refresh builds a catalog only for the teams of an access read that the refresh made itself. A
+/// catalog built for the env list alone would lack the granted teams, so stage 2 GC would delete
+/// their state.
+pub struct CatalogRefresher {
catalog: Arc<CatalogHandle>,
+ access: Arc<TeamAccess>,
+ poller: TeamAccessPoller,
pool: PgPool,
interval: Duration,
jitter: Duration,
- handle: Handle,
-) {
- let _guard = handle.process_scope();
-
- loop {
- let sleep_for = next_interval(interval, jitter);
- tokio::select! {
- _ = handle.shutdown_recv() => break,
- _ = tokio::time::sleep(sleep_for) => match catalog.refresh(&pool).await {
- Ok(stats) => info!(
- teams = stats.teams,
- unique_conditions = stats.unique_conditions,
- "filter catalog refreshed",
- ),
- Err(err) => warn!(
- error = %err,
- "filter catalog refresh failed; keeping previous snapshot",
- ),
- },
+ cascade_enabled: bool,
+ /// `None` until a refresh publishes. A failed refresh after that moves only `due`. So a cohort
+ /// load that keeps failing waits for `due`, but a grant or a revoke that no refresh has published
+ /// retries on every access poll.
+ last: Option<Published>,
+}
+
+impl CatalogRefresher {
+ /// Read team access and `posthog_cohort`, rebuild the catalog for the teams that team access
+ /// lets through, and swap it in.
+ ///
+ /// The staleness metrics are stamped here rather than in [`Self::run`] so the boot load in
+ /// `main` counts too. Otherwise a pod that booted fine and then lost its refresh loop would look
+ /// identical to one that never loaded.
+ ///
+ /// # Errors
+ ///
+ /// Fails before it reads a cohort row when the access read fails, so also before the first
+ /// accepted read. Also fails when the cohort load or the build fails. The previous catalog then
+ /// stays.
+ #[hotpath::measure(impl_type = "CatalogRefresher")]
+ pub async fn refresh(&mut self) -> Result<CatalogStats, CatalogRefreshError> {
+ let pool = self.pool.clone();
+ match self
+ .refresh_from(
+ async |poller| poller.poll(&pool).await,
+ async |teams| load_processing(&pool, teams).await,
+ )
+ .await
+ {
+ Ok(stats) => {
+ gauge!(FILTER_CATALOG_LAST_SUCCESS_TIMESTAMP_SECONDS).set(now_unix_seconds());
+ counter!(FILTER_CATALOG_REFRESH_TOTAL, "result" => "success").increment(1);
+ Ok(stats)
+ }
+ Err(err) => {
+ counter!(FILTER_CATALOG_REFRESH_TOTAL, "result" => "error").increment(1);
+ Err(err)
+ }
+ }
+ }
+
+ /// `poll` reads team access, and `load` reads the cohort rows of its teams. The refresh begins
+ /// before both reads, so every write committed before it began is in what it uses.
+ async fn refresh_from(
+ &mut self,
+ poll: impl AsyncFnOnce(&mut TeamAccessPoller) -> Result<usize, PollError>,
+ load: impl AsyncFnOnce(&ProcessingTeams) -> Result<Vec<CohortRow>, FilterError>,
+ ) -> Result<CatalogStats, CatalogRefreshError> {
+ let seq = self.catalog.begin();
+ let polled = poll(&mut self.poller).await;
+ let teams = self.current_teams(polled)?;
+ let refreshed = self.load_and_store(seq, &teams, load).await;
+ let due = Instant::now() + next_interval(self.interval, self.jitter);
+ if refreshed.is_ok() {
+ self.last = Some(Published { teams, due });
+ } else if let Some(last) = &mut self.last {
+ last.due = due;
+ }
+ refreshed
+ }
+
+ async fn load_and_store(
+ &self,
+ seq: RefreshSeq,
+ teams: &ProcessingTeams,
+ load: impl AsyncFnOnce(&ProcessingTeams) -> Result<Vec<CohortRow>, FilterError>,
+ ) -> Result<CatalogStats, CatalogRefreshError> {
+ let mut rows = tokio::time::timeout(LOAD_TIMEOUT, load(teams)).await??;
+ let fetched_rows = rows.len();
+ retain_processing(&mut rows, teams);
+ if rows.len() != fetched_rows {
+ debug!(
+ fetched_rows,
+ kept_rows = rows.len(),
+ "filter catalog dropped cohort rows of teams it does not process",
+ );
+ }
+ let cascade_enabled = self.cascade_enabled;
+ self.catalog
+ .build_and_store(seq, move || build_catalog_from_rows(rows, cascade_enabled))
+ .await
+ }
+
+ /// The teams of the read that `polled` reports. A failed read leaves an earlier read in place,
+ /// which can predate the refresh's [`RefreshSeq`], so this returns its error instead. A regressed
+ /// read is current, except that the teams it regressed keep their previous rows.
+ fn current_teams(
+ &self,
+ polled: Result<usize, PollError>,
+ ) -> Result<Arc<ProcessingTeams>, PollError> {
+ match polled {
+ Ok(granted) => debug!(granted_teams = granted, "team access polled"),
+ Err(err @ PollError::RegressedRows(_)) => {
+ warn!(error = %err, "team access poll kept regressed rows")
+ }
+ Err(
+ err @ (PollError::Query(_)
+ | PollError::TimedOut(_)
+ | PollError::UnknownLevel { .. }),
+ ) => return Err(err),
+ }
+ Ok(self
+ .access
+ .processing_teams()
+ .expect("a read that does not fail stores a snapshot"))
+ }
+
+ fn refresh_reason(&self, current: &ProcessingTeams, now: Instant) -> Option<RefreshReason> {
+ match &self.last {
+ None => Some(RefreshReason::NeverPublished),
+ Some(last) if *last.teams != *current => Some(RefreshReason::AccessChanged),
+ Some(last) => (now >= last.due).then_some(RefreshReason::Due),
+ }
+ }
+
+ async fn tick(&mut self) {
+ let polled = self.poller.poll(&self.pool).await;
+ let current = match self.current_teams(polled) {
+ Ok(current) => current,
+ Err(err) => {
+ warn!(error = %err, "team access poll failed; the catalog waits for the next poll");
+ return;
+ }
+ };
+ let Some(reason) = self.refresh_reason(&current, Instant::now()) else {
+ return;
+ };
+ // `refresh` polls again after it begins, so the access read it uses follows its
+ // `RefreshSeq`.
+ match self.refresh().await {
+ Ok(stats) => info!(
+ ?reason,
+ teams = stats.teams,
+ unique_conditions = stats.unique_conditions,
+ "filter catalog refreshed",
+ ),
+ Err(err) => warn!(
+ ?reason,
+ error = %err,
+ "filter catalog refresh failed; keeping previous snapshot",
+ ),
+ }
+ }
+
+ /// Polls team access every [`next_poll_delay`] until shutdown. After each poll whose read does
+ /// not fail, refreshes until a refresh publishes, and after that when team access changes or
+ /// `interval ± jitter` has passed. Shutdown also cancels a poll or a refresh in flight, which
+ /// then publishes nothing, so the task stays inside a short graceful window.
+ ///
+ /// Register `handle` without a liveness deadline: a database outage must not stop a processor
+ /// that can run on its last catalog.
+ pub async fn run(mut self, handle: Handle) {
+ let _guard = handle.process_scope();
+
+ loop {
+ let tick = async {
+ tokio::time::sleep(next_poll_delay()).await;
+ self.tick().await;
+ };
+ tokio::select! {
+ _ = handle.shutdown_recv() => break,
+ () = tick => {}
+ }
}
}
}
@@ -372,6 +522,7 @@ fn next_interval(interval: Duration, jitter: Duration) -> Duration {
#[cfg(test)]
mod tests {
+ use std::cell::Cell;
use std::collections::{BTreeSet, HashSet};
use super::*;
@@ -392,6 +543,27 @@ mod tests {
}
}
+ fn processing(env: TeamAllowlist, granted: &[i32]) -> Arc<ProcessingTeams> {
+ TeamAccess::with_granted(env, granted.iter().copied())
+ .processing_teams()
+ .expect("with_granted accepts a read")
+ }
+
+ /// The tests read team access and cohort rows through the closures of `refresh_from`, so
+ /// nothing reads this pool.
+ fn refresher(env: TeamAllowlist) -> (Arc<CatalogHandle>, CatalogRefresher) {
+ let pool = PgPool::connect_lazy("postgres://localhost/unused").expect("a valid url");
+ CatalogHandle::with_refresher(env, pool, Duration::from_secs(300), Duration::ZERO, false)
+ }
+
+ fn access_read_failed() -> Result<usize, PollError> {
+ Err(PollError::Query(sqlx::Error::PoolClosed))
+ }
+
+ fn cohort_load_failed() -> Result<Vec<CohortRow>, FilterError> {
+ Err(FilterError::Query(sqlx::Error::PoolClosed))
+ }
+
fn team_with_one_behavioral() -> TeamFilters {
let mut builder = TeamFiltersBuilder::default();
let filters = json!({
@@ -503,22 +675,170 @@ mod tests {
}
#[tokio::test]
- async fn a_refresh_begins_before_it_reads_the_cohort_rows() {
- let handle = CatalogHandle::new();
- let before_refresh = handle.last_begun();
- let begun_at_read = std::cell::Cell::new(before_refresh);
-
- handle
- .refresh_from(|| async {
- begun_at_read.set(handle.last_begun());
- Ok(Vec::new())
- })
+ async fn a_refresh_begins_before_it_reads_team_access_and_the_cohort_rows_of_the_granted_teams()
+ {
+ let (catalog, mut refresher) = refresher(TeamAllowlist::Only(HashSet::from([2])));
+ let before_refresh = catalog.last_begun();
+ let begun_at_access_read = Cell::new(before_refresh);
+ let begun_at_cohort_read = Cell::new(before_refresh);
+
+ refresher
+ .refresh_from(
+ async |poller| {
+ begun_at_access_read.set(catalog.last_begun());
+ poller.accept_granted([5])
+ },
+ async |teams| {
+ assert_eq!(teams.scope(), &TeamAllowlist::Only(HashSet::from([2, 5])));
+ begun_at_cohort_read.set(catalog.last_begun());
+ Ok(Vec::new())
+ },
+ )
.await
.unwrap();
+ for begun_at_read in [begun_at_access_read, begun_at_cohort_read] {
+ assert!(
+ begun_at_read.get() > before_refresh,
+ "a job admitted during a read must not count the refresh as begun after it",
+ );
+ }
+ }
+
+ #[tokio::test]
+ async fn a_refresh_reads_no_cohort_rows_only_when_its_access_read_fails() {
+ let (catalog, mut refresher) = refresher(TeamAllowlist::Only(HashSet::from([2])));
+ let cohorts_read = Cell::new(false);
+ let read_cohorts = async |_: &ProcessingTeams| {
+ cohorts_read.set(true);
+ Ok(Vec::new())
+ };
+
+ let before_first_read = refresher
+ .refresh_from(async |_| access_read_failed(), read_cohorts)
+ .await;
+ assert!(matches!(
+ before_first_read,
+ Err(CatalogRefreshError::Access(PollError::Query(_)))
+ ));
assert!(
- begun_at_read.get() > before_refresh,
- "a job admitted during the read must not count the refresh as begun after it",
+ !cohorts_read.get(),
+ "a catalog of the env list alone lets stage 2 GC delete the granted teams' state",
+ );
+ assert!(!catalog.is_loaded());
+
+ refresher
+ .refresh_from(
+ async |poller| poller.accept_granted([5]),
+ async |_| Ok(Vec::new()),
+ )
+ .await
+ .unwrap();
+ let after_publish = refresher
+ .refresh_from(async |_| access_read_failed(), read_cohorts)
+ .await;
+ assert!(after_publish.is_err());
+ assert!(
+ !cohorts_read.get(),
+ "the rows of an earlier read can lack a grant that a job admitted after it relies on",
+ );
+
+ refresher
+ .refresh_from(
+ async |poller| poller.accept_granted([]),
+ async |teams| {
+ assert_eq!(teams.scope(), &TeamAllowlist::Only(HashSet::from([2, 5])));
+ Ok(Vec::new())
+ },
+ )
+ .await
+ .expect("a deleted access row must not fail every refresh until a restart");
+ }
+
+ #[tokio::test(start_paused = true)]
+ async fn a_refresh_is_due_until_one_publishes_then_on_an_access_change_or_after_the_period() {
+ let (_catalog, mut refresher) = refresher(TeamAllowlist::Only(HashSet::from([2])));
+ let now = Instant::now();
+ let past_due = now + Duration::from_secs(3600);
+
+ let polled = refresher.poller.accept_granted([5]);
+ let granted = refresher.current_teams(polled).unwrap();
+ assert_eq!(
+ refresher.refresh_reason(&granted, now),
+ Some(RefreshReason::NeverPublished)
+ );
+ let hung_boot_load = refresher
+ .refresh_from(
+ async |poller| poller.accept_granted([5]),
+ async |_| {
+ tokio::time::sleep(2 * LOAD_TIMEOUT).await;
+ Ok(Vec::new())
+ },
+ )
+ .await;
+ assert!(matches!(
+ hung_boot_load,
+ Err(CatalogRefreshError::LoadTimedOut(_))
+ ));
+ assert_eq!(
+ refresher.refresh_reason(&granted, now),
+ Some(RefreshReason::NeverPublished),
+ "a boot load that times out retries on the next tick",
+ );
+
+ refresher
+ .refresh_from(
+ async |poller| poller.accept_granted([5]),
+ async |_| Ok(Vec::new()),
+ )
+ .await
+ .unwrap();
+ assert_eq!(refresher.refresh_reason(&granted, now), None);
+ assert_eq!(
+ refresher.refresh_reason(&granted, past_due),
+ Some(RefreshReason::Due)
+ );
+ // A failed refresh that moves `due` puts it after `past_due`.
+ refresher.interval = Duration::from_secs(7200);
+ refresher
+ .refresh_from(
+ async |poller| poller.accept_granted([5]),
+ async |_| cohort_load_failed(),
+ )
+ .await
+ .unwrap_err();
+ assert_eq!(
+ refresher.refresh_reason(&granted, past_due),
+ None,
+ "a cohort load that keeps failing waits for the period, not for the next tick",
+ );
+
+ let polled = refresher.poller.accept_granted([5, 6]);
+ let regranted = refresher.current_teams(polled).unwrap();
+ assert_eq!(
+ refresher.refresh_reason(&regranted, now),
+ Some(RefreshReason::AccessChanged)
+ );
+ refresher
+ .refresh_from(async |_| access_read_failed(), async |_| Ok(Vec::new()))
+ .await
+ .unwrap_err();
+ assert_eq!(
+ refresher.refresh_reason(&regranted, now),
+ Some(RefreshReason::AccessChanged),
+ "a refresh whose access read failed retries on the next tick",
+ );
+ refresher
+ .refresh_from(
+ async |poller| poller.accept_granted([5, 6]),
+ async |_| cohort_load_failed(),
+ )
+ .await
+ .unwrap_err();
+ assert_eq!(
+ refresher.refresh_reason(&regranted, now),
+ Some(RefreshReason::AccessChanged),
+ "a grant that a failed cohort load did not publish retries on the next tick",
);
}
@@ -587,25 +907,17 @@ mod tests {
}
#[test]
- fn retain_allowlisted_keeps_only_in_scope_rows() {
- let mut rows = vec![cohort_row(1, 2), cohort_row(2, 7)];
- retain_allowlisted(&mut rows, &TeamAllowlist::Only(HashSet::from([2])));
- assert_eq!(rows.len(), 1);
- assert_eq!(rows[0].team_id, 2);
- }
-
- #[test]
- fn retain_allowlisted_all_keeps_everything() {
- let mut rows = vec![cohort_row(1, 2), cohort_row(2, 7)];
- retain_allowlisted(&mut rows, &TeamAllowlist::All);
- assert_eq!(rows.len(), 2);
- }
-
- #[test]
- fn retain_allowlisted_empty_scope_drops_everything() {
- let mut rows = vec![cohort_row(1, 2)];
- retain_allowlisted(&mut rows, &TeamAllowlist::Only(HashSet::new()));
- assert!(rows.is_empty());
+ fn retain_processing_keeps_the_rows_of_the_env_list_and_the_granted_teams() {
+ for (env, granted, kept) in [
+ (TeamAllowlist::Only(HashSet::from([2])), vec![5], vec![2, 5]),
+ (TeamAllowlist::All, vec![], vec![2, 5, 7]),
+ (TeamAllowlist::Only(HashSet::new()), vec![], vec![]),
+ ] {
+ let mut rows = vec![cohort_row(1, 2), cohort_row(2, 5), cohort_row(3, 7)];
+ retain_processing(&mut rows, &processing(env.clone(), &granted));
+ let kept_teams: Vec<i32> = rows.iter().map(|row| row.team_id).collect();
+ assert_eq!(kept_teams, kept, "env {env:?}, granted {granted:?}");
+ }
}
#[test]
@@ -641,26 +953,33 @@ mod tests {
common_database::get_pool_with_config(&url, common_database::PoolConfig::default())
.expect("build posthog_cohort pool");
- let handle = CatalogHandle::new();
+ let (handle, mut refresher) = CatalogHandle::with_refresher(
+ TeamAllowlist::All,
+ pool.clone(),
+ Duration::from_secs(300),
+ Duration::ZERO,
+ false,
+ );
assert!(!handle.is_loaded());
- let stats = handle
- .refresh(&pool)
+ let stats = refresher
+ .refresh()
.await
- .expect("refresh against live posthog_cohort");
+ .expect("refresh against live posthog_cohort and cohort_realtime_team_access");
assert!(handle.is_loaded());
assert_eq!(handle.load().team_count(), stats.teams);
- // Bypass `retain_allowlisted`, which would hide a missing SQL filter.
+ // Bypass `retain_processing`, which would hide a missing SQL filter.
let keys = |rows: &[CohortRow]| -> BTreeSet<(i32, i32)> {
rows.iter().map(|row| (row.team_id, row.id)).collect()
};
- let unscoped = load_allowlisted(&pool, &TeamAllowlist::All)
+ let unscoped = load_processing(&pool, &processing(TeamAllowlist::All, &[]))
.await
.expect("unscoped load against live posthog_cohort");
if let Some(team_id) = unscoped.first().map(|row| row.team_id) {
- let scoped = load_allowlisted(&pool, &TeamAllowlist::Only(HashSet::from([team_id])))
+ let only_team = processing(TeamAllowlist::Only(HashSet::from([team_id])), &[]);
+ let scoped = load_processing(&pool, &only_team)
.await
.expect("team-scoped load against live posthog_cohort");
let expected: BTreeSet<_> = keys(&unscoped)
@@ -669,7 +988,7 @@ mod tests {
.collect();
assert_eq!(keys(&scoped), expected);
}
- let none = load_allowlisted(&pool, &TeamAllowlist::Only(HashSet::new()))
+ let none = load_processing(&pool, &processing(TeamAllowlist::Only(HashSet::new()), &[]))
.await
.expect("empty-scope load against live posthog_cohort");
assert!(none.is_empty());
diff --git a/rust/cohort-stream-processor/src/filters/mod.rs b/rust/cohort-stream-processor/src/filters/mod.rs
index 699b01bdde35..897944f01dbd 100644
--- a/rust/cohort-stream-processor/src/filters/mod.rs
+++ b/rust/cohort-stream-processor/src/filters/mod.rs
@@ -14,4 +14,4 @@ pub use cohort_core::filters::{
pub use cohort_core::filters::{FilterCatalog, Generation};
pub use cohort_core::filters::{TeamFilters, TeamFiltersBuilder};
-pub use manager::{run_refresh_loop, CatalogHandle, CatalogRefreshError, CatalogStats};
+pub use manager::{CatalogHandle, CatalogRefreshError, CatalogRefresher, CatalogStats};
diff --git a/rust/cohort-stream-processor/src/main.rs b/rust/cohort-stream-processor/src/main.rs
index 8c2fed874e45..a6a5c4cb95e0 100644
--- a/rust/cohort-stream-processor/src/main.rs
+++ b/rust/cohort-stream-processor/src/main.rs
@@ -19,7 +19,7 @@ use cohort_stream_processor::consumers::{
BootReadiness, CascadeRoute, CohortStreamEventsConsumer, EventDispatcher, FollowerConsumer,
FollowerRoute, MergeRoute, SeedFollowerConsumer, TransferRoute,
};
-use cohort_stream_processor::filters::{run_refresh_loop, CatalogHandle};
+use cohort_stream_processor::filters::CatalogHandle;
use cohort_stream_processor::merge::gc::MergeGcSweeper;
use cohort_stream_processor::merge::redrive::RedriveSweeper;
use cohort_stream_processor::observability;
@@ -164,10 +164,16 @@ async fn async_main(config: Config) -> Result<()> {
None
};
- let catalog = Arc::new(CatalogHandle::with_allowlist(
+ let pool = get_pool_with_config(&config.database_url, config.pool_config())
+ .context("creating posthog_cohort database pool")?;
+
+ let (catalog, mut catalog_refresher) = CatalogHandle::with_refresher(
config.team_allowlist.clone(),
+ pool,
+ config.filter_catalog_refresh_interval(),
+ config.filter_catalog_refresh_jitter(),
config.cohort_cascade_enabled,
- ));
+ );
let boot_readiness = BootReadiness::new(catalog.clone());
// Bound before the restore, which can outlast the startup probe. `/_health` answers throughout;
@@ -191,10 +197,7 @@ async fn async_main(config: Config) -> Result<()> {
.await
});
- let pool = get_pool_with_config(&config.database_url, config.pool_config())
- .context("creating posthog_cohort database pool")?;
-
- match catalog.refresh(&pool).await {
+ match catalog_refresher.refresh().await {
Ok(stats) => info!(
teams = stats.teams,
unique_conditions = stats.unique_conditions,
@@ -501,20 +504,7 @@ async fn async_main(config: Config) -> Result<()> {
let guard = manager.monitor_background();
- let refresh_catalog = catalog.clone();
- let refresh_pool = pool.clone();
- let refresh_interval = config.filter_catalog_refresh_interval();
- let refresh_jitter = config.filter_catalog_refresh_jitter();
- tokio::spawn(async move {
- run_refresh_loop(
- refresh_catalog,
- refresh_pool,
- refresh_interval,
- refresh_jitter,
- catalog_handle_lifecycle,
- )
- .await;
- });
+ tokio::spawn(catalog_refresher.run(catalog_handle_lifecycle));
// Hold the first eviction pass past the boot window so its read burst doesn't stack on backlog
// catch-up; only the eviction sweep is delayed.
diff --git a/rust/cohort-stream-processor/src/observability/metrics.rs b/rust/cohort-stream-processor/src/observability/metrics.rs
index 587ac3ac8cc7..400096a08df2 100644
--- a/rust/cohort-stream-processor/src/observability/metrics.rs
+++ b/rust/cohort-stream-processor/src/observability/metrics.rs
@@ -15,7 +15,7 @@ pub const FILTER_CATALOG_TEAMS: &str = "filter_catalog_teams";
/// Distinct `conditionHash`es across all teams in the current snapshot (gauge).
pub const FILTER_CATALOG_UNIQUE_CONDITIONS: &str = "filter_catalog_unique_conditions";
/// Unix timestamp of the last *successful* catalog refresh (gauge, seconds). Stamped inside
-/// `CatalogHandle::refresh`, so the boot load and the periodic loop both advance it. **Alert on
+/// `CatalogRefresher::refresh`, so the boot load and the periodic loop both advance it. **Alert on
/// staleness** via `time() - gauge`: a failed refresh keeps serving the previous snapshot silently,
/// so cohort edits go invisible with nothing else moving. A gauge written only on success would go
/// flat rather than climb, and the refresh loop is a detached task — a per-tick age gauge would
diff --git a/rust/common/cohort-team-access/Cargo.toml b/rust/common/cohort-team-access/Cargo.toml
index 5f3f0bcc39d1..99243afe5f7d 100644
--- a/rust/common/cohort-team-access/Cargo.toml
+++ b/rust/common/cohort-team-access/Cargo.toml
@@ -7,7 +7,8 @@ edition = "2021"
workspace = true
[features]
-# Exposes `TeamAccess::with_granted`, a loaded reader for tests that have no database.
+# Exposes `TeamAccess::with_granted` and `TeamAccessPoller::accept_granted`, which load a reader
+# for tests that have no database.
test-util = []
[dependencies]
diff --git a/rust/common/cohort-team-access/src/lib.rs b/rust/common/cohort-team-access/src/lib.rs
index ee464560c645..a2b50f7ed1e9 100644
--- a/rust/common/cohort-team-access/src/lib.rs
+++ b/rust/common/cohort-team-access/src/lib.rs
@@ -104,12 +104,11 @@ impl AccessRows {
.map(Self)
}
- fn processes(&self, team_id: i32) -> bool {
- self.0.get(&team_id).is_some_and(|level| level.processes())
- }
-
- fn granted(&self) -> usize {
- self.0.values().filter(|level| level.processes()).count()
+ fn granted(&self) -> impl Iterator<Item = i32> + '_ {
+ self.0
+ .iter()
+ .filter(|(_, level)| level.processes())
+ .map(|(&team_id, _)| team_id)
}
/// Puts back the `previous` row of each team whose row here is a change that
@@ -127,13 +126,49 @@ impl AccessRows {
}
}
-/// Answers whether a team processes, from the rows the poller stored last.
+/// The teams the pipeline processes after an accepted read: the env list united with the teams
+/// whose row is at `processing` or `serving`. Only an accepted read builds one, so a reader that
+/// holds one never runs on the env list alone.
+#[derive(Debug, PartialEq, Eq)]
+pub struct ProcessingTeams(TeamAllowlist);
+
+impl ProcessingTeams {
+ /// An `off` row never removes a team that the env list names.
+ fn new(env: &TeamAllowlist, rows: &AccessRows) -> Self {
+ Self(match env {
+ TeamAllowlist::All => TeamAllowlist::All,
+ TeamAllowlist::Only(listed) => {
+ TeamAllowlist::Only(listed.iter().copied().chain(rows.granted()).collect())
+ }
+ })
+ }
+
+ pub fn scope(&self) -> &TeamAllowlist {
+ &self.0
+ }
+}
+
+/// One accepted read. The poller checks the next read against `rows`, and readers ask `processing`.
+#[derive(Debug)]
+struct Snapshot {
+ rows: AccessRows,
+ processing: Arc<ProcessingTeams>,
+}
+
+impl Snapshot {
+ fn new(env: &TeamAllowlist, rows: AccessRows) -> Self {
+ let processing = Arc::new(ProcessingTeams::new(env, &rows));
+ Self { rows, processing }
+ }
+}
+
+/// Answers whether a team processes, from the read the poller accepted last.
#[derive(Debug)]
pub struct TeamAccess {
env: TeamAllowlist,
/// `None` until the first accepted read. Until then no team processes, not even one the env list
/// names: a reader that ran on the env list alone would treat every granted team as revoked.
- rows: ArcSwapOption<AccessRows>,
+ snapshot: ArcSwapOption<Snapshot>,
}
impl TeamAccess {
@@ -141,7 +176,7 @@ impl TeamAccess {
gauge!(LAST_SUCCESS).set(0.0);
let access = Arc::new(Self {
env,
- rows: ArcSwapOption::empty(),
+ snapshot: ArcSwapOption::empty(),
});
let poller = TeamAccessPoller {
access: access.clone(),
@@ -152,25 +187,30 @@ impl TeamAccess {
/// A loaded reader whose one read granted `granted` at `processing`.
#[cfg(feature = "test-util")]
pub fn with_granted(env: TeamAllowlist, granted: impl IntoIterator<Item = i32>) -> Arc<Self> {
- let rows = granted
- .into_iter()
- .map(|team_id| (team_id, Level::Processing))
- .collect();
- Arc::new(Self {
- env,
- rows: ArcSwapOption::from_pointee(AccessRows(rows)),
- })
+ let (access, mut poller) = Self::new(env);
+ poller
+ .accept_granted(granted)
+ .expect("a first read has no previous rows to regress");
+ access
}
pub fn processes(&self, team_id: i32) -> bool {
- self.rows
+ self.snapshot
+ .load()
+ .as_ref()
+ .is_some_and(|snapshot| snapshot.processing.scope().includes(team_id))
+ }
+
+ /// `None` until the first accepted read.
+ pub fn processing_teams(&self) -> Option<Arc<ProcessingTeams>> {
+ self.snapshot
.load()
.as_ref()
- .is_some_and(|rows| self.env.includes(team_id) || rows.processes(team_id))
+ .map(|snapshot| snapshot.processing.clone())
}
pub fn is_loaded(&self) -> bool {
- self.rows.load().is_some()
+ self.snapshot.load().is_some()
}
}
@@ -208,12 +248,13 @@ impl TeamAccessPoller {
}
fn accept(&mut self, mut read: AccessRows) -> Result<usize, PollError> {
- let regressed = match self.access.rows.load().as_deref() {
- Some(previous) => read.restore_regressed(previous),
+ let regressed = match self.access.snapshot.load().as_deref() {
+ Some(previous) => read.restore_regressed(&previous.rows),
None => Vec::new(),
};
- let granted = read.granted();
- self.access.rows.store(Some(Arc::new(read)));
+ let granted = read.granted().count();
+ let snapshot = Snapshot::new(&self.access.env, read);
+ self.access.snapshot.store(Some(Arc::new(snapshot)));
if !regressed.is_empty() {
return Err(PollError::RegressedRows(regressed));
}
@@ -232,7 +273,7 @@ impl TeamAccessPoller {
loop {
let tick = async {
- tokio::time::sleep(next_interval()).await;
+ tokio::time::sleep(next_poll_delay()).await;
self.poll(&pool).await
};
tokio::select! {
@@ -246,8 +287,24 @@ impl TeamAccessPoller {
}
}
-/// The jitter spreads the polls of many pods over time.
-fn next_interval() -> Duration {
+#[cfg(feature = "test-util")]
+impl TeamAccessPoller {
+ /// Accepts a read whose rows grant `granted` at `processing`, as a poll would.
+ pub fn accept_granted(
+ &mut self,
+ granted: impl IntoIterator<Item = i32>,
+ ) -> Result<usize, PollError> {
+ let rows = granted
+ .into_iter()
+ .map(|team_id| (team_id, Level::Processing))
+ .collect();
+ self.accept(AccessRows(rows))
+ }
+}
+
+/// `POLL_INTERVAL ± POLL_JITTER`. Every reader polls at this cadence, and the jitter spreads the
+/// polls of many pods over time.
+pub fn next_poll_delay() -> Duration {
let max_jitter_ms = 2 * POLL_JITTER.as_millis() as u64;
let jitter = Duration::from_millis(rand::thread_rng().gen_range(0..=max_jitter_ms));
POLL_INTERVAL - POLL_JITTER + jitter
@@ -263,6 +320,8 @@ fn now_unix_seconds() -> f64 {
#[cfg(test)]
mod tests {
+ use std::collections::HashSet;
+
use super::*;
use metrics_util::debugging::DebuggingRecorder;
@@ -297,29 +356,37 @@ mod tests {
assert!(!access.is_loaded());
assert!(!access.processes(2));
+ assert!(access.processing_teams().is_none());
}
#[test]
fn processing_is_the_env_list_united_with_the_granted_rows() {
- let (access, mut poller) = TeamAccess::new(env_only(&[2, 3]));
- poller
- .accept(rows(&[
- (3, "off"),
- (5, "processing"),
- (6, "serving"),
- (7, "off"),
- ]))
- .unwrap();
-
- for (team_id, expected) in [
- (2, true),
- (3, true),
- (5, true),
- (6, true),
- (7, false),
- (8, false),
+ let read = [(3, "off"), (5, "processing"), (6, "serving"), (7, "off")];
+ for (env, expected_scope) in [
+ (
+ env_only(&[2, 3]),
+ TeamAllowlist::Only(HashSet::from([2, 3, 5, 6])),
+ ),
+ (TeamAllowlist::All, TeamAllowlist::All),
] {
- assert_eq!(access.processes(team_id), expected, "team {team_id}");
+ let (access, mut poller) = TeamAccess::new(env.clone());
+ poller.accept(rows(&read)).unwrap();
+ let processing = access.processing_teams().unwrap();
+
+ assert_eq!(processing.scope(), &expected_scope, "env {env:?}");
+ for team_id in [2, 3, 5, 6, 7, 8] {
+ let expected = expected_scope.includes(team_id);
+ assert_eq!(
+ access.processes(team_id),
+ expected,
+ "env {env:?}, team {team_id}"
+ );
+ assert_eq!(
+ processing.scope().includes(team_id),
+ expected,
+ "env {env:?}, team {team_id}"
+ );
+ }
}
}