Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(cohorts): read the team access record in the cohort shufflers (#126563)

Matheus Batistaeb4e11fe3e81
diff --git a/.github/scripts/rust_test_shards.py b/.github/scripts/rust_test_shards.py
index e47f4476c920..93507e5b8206 100644
--- a/.github/scripts/rust_test_shards.py
+++ b/.github/scripts/rust_test_shards.py
@@ -20,6 +20,7 @@ PACKAGE_WEIGHTS: dict[str, int] = {
"cohort-stream-processor": 285,
"common-alloc": 5,
"common-cache": 6,
+ "common-cohort-team-access": 5,
"common-compression": 5,
"common-continuous-profiling": 9,
"common-cookieless": 103,
diff --git a/.github/workflows/ci-rust.yml b/.github/workflows/ci-rust.yml
index 7326daa94771..382a0facfe9c 100644
--- a/.github/workflows/ci-rust.yml
+++ b/.github/workflows/ci-rust.yml
@@ -97,6 +97,8 @@ jobs:
- 'posthog/models/feature_flag/flags_cache.py'
- 'posthog/models/feature_flag/feature_flag.py'
- 'rust/feature-flags/tests/fixtures/hypercache_contract.json'
+ # common-cohort-team-access tests its query against the table these migrations build
+ - 'products/cohorts/backend/migrations/**'
# replay-core-v4's tests read the encryption vectors the Node writers pin
- 'nodejs/src/ingestion/pipelines/sessionreplay/ml-mirror/keys/encryption-vector*.json'
- 'nodejs/src/ingestion/pipelines/sessionreplay/ai-research/v4-keys/encryption-vector*.json'
diff --git a/docs/internal/realtime-cohorts/backfill-coordination.md b/docs/internal/realtime-cohorts/backfill-coordination.md
index 57a3ed82fc85..f75e1a71fcd5 100644
--- a/docs/internal/realtime-cohorts/backfill-coordination.md
+++ b/docs/internal/realtime-cohorts/backfill-coordination.md
@@ -152,14 +152,14 @@ The membership consumer does count those markers toward its sweep, see [membersh
Several independent gates decide whether anything happens.
Missing one of them fails quietly, so they are worth knowing.
-| Gate | Effect |
-| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `REALTIME_COHORT_TEAM_ALLOWLIST` | Teams whose cohorts get hash maintenance, stamp invalidation, supersession and any run at all. Unset means no teams in Django, but the shuffler, processor and seeder default to team 2, so set it explicitly on every service. Set but empty means **all** teams. Django puts a listed team at the `serving` level, and adds the teams that have a team access row. The Rust services' lists must name both, but never list a team granted `processing` here, because that makes it `serving` |
-| Team access rows | `cohort_realtime_team_access`, written only by `realtime_cohort_team_access grant\|revoke\|list`. Django reads it as a union with `REALTIME_COHORT_TEAM_ALLOWLIST`; the shuffler, processor and seeder read only their env lists, so a row changes Django alone. Until they read it, keep their env lists in step with the rows. A team granted only here gets runs nobody claims, each holding its cohort's run slot. A team revoked only here loses Django's edit-time supersede while Rust still processes it, so its readiness stamps outlive later edits. A row at `processing` gets the Django behavior above, but the team's users see no realtime state: no readiness, no realtime batch audience, and no backfilled cohort in flags or workflows, even one stamped before the grant, so a workflow that already uses one fails its next save. A row at `serving` also shows them. `revoke` sets `off`, which hides the stamps the same way, and a revoked team cannot be granted again. `grant` and `revoke` refuse a team on the env list |
-| `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST` | Teams whose saves create runs automatically. Unset or empty means **no** teams. A granted row with `save_triggers_backfill` adds its team, so `resave_cohorts --no-backfill-triggers`, not this setting, is what stops a resave from queueing runs |
-| Operator attestations | Settings an operator sets to declare that prerequisites hold. They are declarations, not live checks. Behavioral runs need `BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED` and `BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED`. Person runs also need the person TTL and sizing attestations and a positive seed-bytes budget |
-| Person sizing budget | A cohort-scoped person run is refused when the team's active person-run estimates plus its own would exceed the budget. The check reads the active estimates before the run exists and reserves nothing, so two cohorts sized at the same time can together exceed the budget. A team run checks only its own estimate. Hitting the size estimate's scan cap, or the cap on pinned conditions, also refuses |
-| Behavioral scan limit | `create_cohort_backfill_run` refuses a behavioral team run when its pinned event names had more events than `BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY` on the busiest of the last 7 complete UTC days. `--max-scan-events-per-day` overrides it for one invocation, and 0 skips the count and the check. The save path does not check it |
+| Gate | Effect |
+| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `REALTIME_COHORT_TEAM_ALLOWLIST` | Teams whose cohorts get hash maintenance, stamp invalidation, supersession and any run at all. Unset means no teams in Django, but the shuffler, processor and seeder default to team 2, so set it explicitly on every service. Set but empty means **all** teams. Django puts a listed team at the `serving` level, and adds the teams that have a team access row. The shufflers add those teams too, but the processor's and seeder's lists must name both. Never list a team granted `processing` here, because that makes it `serving` |
+| Team access rows | `cohort_realtime_team_access`, written only by `realtime_cohort_team_access grant\|revoke\|list`. Django and the shufflers read it as a union with their `REALTIME_COHORT_TEAM_ALLOWLIST`; the processor and seeder read only their env lists. Until they read it, keep their env lists in step with the rows. A team granted only here gets runs nobody claims, each holding its cohort's run slot. A team revoked only here loses Django's edit-time supersede while the processor and seeder still process it, so its readiness stamps outlive later edits. A shuffler whose env list does not name the team stops forwarding its events. A row at `processing` gets the Django behavior above, but the team's users see no realtime state: no readiness, no realtime batch audience, and no backfilled cohort in flags or workflows, even one stamped before the grant, so a workflow that already uses one fails its next save. A row at `serving` also shows them. `revoke` sets `off`, which hides the stamps the same way, and a revoked team cannot be granted again. `grant` and `revoke` refuse a team on the env list |
+| `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST` | Teams whose saves create runs automatically. Unset or empty means **no** teams. A granted row with `save_triggers_backfill` adds its team, so `resave_cohorts --no-backfill-triggers`, not this setting, is what stops a resave from queueing runs |
+| Operator attestations | Settings an operator sets to declare that prerequisites hold. They are declarations, not live checks. Behavioral runs need `BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED` and `BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED`. Person runs also need the person TTL and sizing attestations and a positive seed-bytes budget |
+| Person sizing budget | A cohort-scoped person run is refused when the team's active person-run estimates plus its own would exceed the budget. The check reads the active estimates before the run exists and reserves nothing, so two cohorts sized at the same time can together exceed the budget. A team run checks only its own estimate. Hitting the size estimate's scan cap, or the cap on pinned conditions, also refuses |
+| Behavioral scan limit | `create_cohort_backfill_run` refuses a behavioral team run when its pinned event names had more events than `BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY` on the busiest of the last 7 complete UTC days. `--max-scan-events-per-day` overrides it for one invocation, and 0 skips the count and the check. The save path does not check it |
The Rust services have their own switches, and Django cannot see them.
A team must reach the processor's copy of `REALTIME_COHORT_TEAM_ALLOWLIST` before the seeder's.
diff --git a/docs/internal/realtime-cohorts/event-routing.md b/docs/internal/realtime-cohorts/event-routing.md
index 782f95a72bb5..67daae28aa9c 100644
--- a/docs/internal/realtime-cohorts/event-routing.md
+++ b/docs/internal/realtime-cohorts/event-routing.md
@@ -39,15 +39,25 @@ It parses every message in two phases.
2. **A full parse for survivors.**
Only events that pass the gate are parsed completely and turned into the output envelope.
-An event passes the gate when it has a person id and its team is in the **team index**.
-The team index is the set of teams that have at least one realtime cohort, intersected with `REALTIME_COHORT_TEAM_ALLOWLIST`.
-The shuffler rebuilds it from Postgres every few minutes and swaps it in atomically.
+An event passes the gate when it has a person id and its team passes two checks.
+
+1. **The team index.**
+ The team has at least one realtime cohort.
+ The shuffler rebuilds the index from Postgres every few minutes and swaps it in atomically.
+2. **Team access.**
+ `REALTIME_COHORT_TEAM_ALLOWLIST` names the team, or the team's row in `cohort_realtime_team_access` is at `processing` or `serving`.
+ The shuffler reads the whole table about every 15 seconds, so a grant or a revoke takes effect without a deploy.
+ Django never deletes a row, and it never grants an `off` row again.
+ So when a read lacks a row the shuffler holds, or grants one of its `off` rows again, the shuffler keeps its previous row for that team, because that read comes from a lagging replica or the wrong database.
+ The read's other rows still take effect, so a revoke always gets through.
+ After a failed read, it keeps all the previous rows.
+ A row deleted or granted again by hand keeps its old state in every running shuffler, so restart both shuffler deployments after one.
Apart from the person id check, the gate looks at the team only.
It cannot filter by event name: person-property criteria are re-evaluated from every event that carries person properties, whatever the event is called.
So every event from an enabled team with a person id is forwarded.
-The shuffler consumes nothing until the team index has loaded once.
+The shuffler consumes nothing until the team index and the team access rows have each loaded once.
If Postgres is down when it boots, the shuffler lags and retries on its refresh interval.
It loses nothing as long as its consumer group has a committed offset that is still inside the firehose's retention.
A group with no usable offset starts at the tail when consumption begins, as described below, so it skips what arrived during the wait.
@@ -224,15 +234,16 @@ The processor sees firehose offset 9870001 from partition 3 for p-1 a second tim
Consuming, producing and committing run independently.
Thousands of forwards can be in flight at once, so slow broker acknowledgments do not throttle consumption.
For typical event sizes, the in-flight cap sits below the producer's byte queue, so the shuffler pauses cleanly instead of churning on a full queue.
-- **Lock-free team index.**
- The refresh builds a new set and swaps a pointer.
- The hot path never waits on it.
+- **Lock-free team gate.**
+ The index refresh and the access poll each build a new snapshot and swap a pointer.
+ The hot path never waits on them.
## Things that surprise people
- The processor's event-name gate only skips behavioral evaluation, after the event is already on `cohort_stream_events`.
- The shuffler and the processor each read their own copy of `REALTIME_COHORT_TEAM_ALLOWLIST` at startup, and each polls Postgres on its own schedule.
- Only teams in both are processed, and a newly enabled team can lose a few minutes of events until both have refreshed.
+ The shuffler also lets through the teams that a `cohort_realtime_team_access` row grants, but the processor reads only its list.
+ Only teams that both let through are processed, and a newly enabled team can lose a few minutes of events until both have refreshed.
Merges also need the team in Node's merge allowlist, and backfills need it in the seeder's copy.
- The processor recognizes applied events with a high-water mark per source partition, not a set of offsets.
The shuffler's producer does not guarantee ordering on retries, so an event that is overtaken by a later event from the same source partition is skipped as if it were a replay.
diff --git a/products/cohorts/backend/management/commands/realtime_cohort_team_access.py b/products/cohorts/backend/management/commands/realtime_cohort_team_access.py
index 50652787e974..cd4fd4b9922a 100644
--- a/products/cohorts/backend/management/commands/realtime_cohort_team_access.py
+++ b/products/cohorts/backend/management/commands/realtime_cohort_team_access.py
@@ -1,7 +1,7 @@
"""Grant, revoke and list team access to the realtime cohort pipeline, meant for a toolbox pod.
-Only Django reads these rows, as a union with its `REALTIME_COHORT_TEAM_ALLOWLIST`. The shuffler,
-processor and seeder read their own env lists, so every grant or revoke needs the same change there.
+Django and the shufflers read these rows, each as a union with its own `REALTIME_COHORT_TEAM_ALLOWLIST`.
+The processor and seeder read only their env lists, so every grant or revoke needs the same change there.
A grant at `processing` gives the team Django's side of the pipeline while its users see none of it,
which is what a load test needs. `serving` also shows them realtime cohort state. `revoke --purpose
diff --git a/products/cohorts/backend/models/team_access.py b/products/cohorts/backend/models/team_access.py
index c49bc5f25b8f..bd8c754f134f 100644
--- a/products/cohorts/backend/models/team_access.py
+++ b/products/cohorts/backend/models/team_access.py
@@ -1,6 +1,8 @@
from django.db import models
+# The Rust readers in `rust/common/cohort-team-access` refuse a read that holds a level they do not
+# know, and keep their previous rows. Add a new level there, and deploy it, before Django writes it.
class RealtimeCohortLevel(models.TextChoices):
OFF = "off", "Off"
# The pipeline maintains membership for the team, but nothing the team's users see reads it.
@@ -17,13 +19,15 @@ class RealtimeCohortAccessPurpose(models.TextChoices):
class RealtimeCohortTeamAccess(models.Model):
"""A team's access to the realtime cohort pipeline, granted on top of `REALTIME_COHORT_TEAM_ALLOWLIST`.
- Staff write it only through the `realtime_cohort_team_access` command. Only Django reads it, so the
- Rust services' env lists must change with it. A missing row means `off`. Never read it through
- `get_or_create_team_extension`, because that creates the row.
+ Staff write it only through the `realtime_cohort_team_access` command. Django and the shufflers read
+ it. The processor and the seeder do not, so their env lists must change with it. A missing row means
+ `off`. Never read it through `get_or_create_team_extension`, because that creates the row.
Rows are never deleted: a revoke sets `off`, and a team deletion leaves the row (`DO_NOTHING`).
So when a team's row appears on one read and is missing on a later one, the later read failed.
- A reader treats that as an error, not as a revoke.
+ A reader treats that as an error, not as a revoke. The Rust readers also treat `off` as final.
+ So after a row is deleted, or granted again after `off`, by hand, they keep that team's old row
+ until they restart.
"""
# db_constraint=False: a real FK constraint would take a SHARE ROW EXCLUSIVE lock on
diff --git a/products/cohorts/backend/realtime_teams.py b/products/cohorts/backend/realtime_teams.py
index 7e4585535a9c..1bbe5c740dcd 100644
--- a/products/cohorts/backend/realtime_teams.py
+++ b/products/cohorts/backend/realtime_teams.py
@@ -195,6 +195,8 @@ def grant_realtime_cohort_access(
row.team_id: row
for row in RealtimeCohortTeamAccess.objects.select_for_update().filter(team_id__in=requested)
}
+ # The Rust readers in rust/common/cohort-team-access treat `off` as final, so allowing this
+ # needs a change there first.
_refuse(
"Revoked, and its pipeline state was never purged",
[team_id for team_id, row in rows.items() if row.level == RealtimeCohortLevel.OFF],
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index 2335705ed671..fa5af945a268 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -2774,6 +2774,7 @@ dependencies = [
"arc-swap",
"axum 0.7.9",
"common-alloc",
+ "common-cohort-team-access",
"common-database",
"common-kafka",
"common-liveness",
@@ -2931,6 +2932,22 @@ dependencies = [
"tracing",
]
+[[package]]
+name = "common-cohort-team-access"
+version = "0.1.0"
+dependencies = [
+ "arc-swap",
+ "common-types",
+ "lifecycle",
+ "metrics",
+ "metrics-util",
+ "rand 0.8.5",
+ "sqlx",
+ "thiserror 2.0.18",
+ "tokio",
+ "tracing",
+]
+
[[package]]
name = "common-compression"
version = "0.1.0"
diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index b8bfc96d2357..40967d48b4e2 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -14,6 +14,7 @@ members = [
"cohort-stream-processor",
"common/alloc",
"common/assignment-coordination",
+ "common/cohort-team-access",
"common/k8s-awareness",
"common/keyed-stash",
"common/cache",
diff --git a/rust/affected-services/determinator-rules.toml b/rust/affected-services/determinator-rules.toml
index e18ef1f9d0e6..f52a84c93751 100644
--- a/rust/affected-services/determinator-rules.toml
+++ b/rust/affected-services/determinator-rules.toml
@@ -52,6 +52,12 @@ globs = [
]
mark-changed = ["feature-flags", "hypercache-server", "common-hypercache"]
+# common-cohort-team-access tests its query against the table that these migrations build.
+# Must come BEFORE the catch-all rule below.
+[[path-rule]]
+globs = ["../products/cohorts/backend/migrations/**"]
+mark-changed = ["common-cohort-team-access"]
+
# replay-core-v4's tests read the encryption vectors the Node writers pin.
# Must come BEFORE the catch-all rule below.
[[path-rule]]
diff --git a/rust/cohort-event-shuffler/Cargo.toml b/rust/cohort-event-shuffler/Cargo.toml
index a4a8461843e1..3d89549e4f67 100644
--- a/rust/cohort-event-shuffler/Cargo.toml
+++ b/rust/cohort-event-shuffler/Cargo.toml
@@ -11,6 +11,7 @@ anyhow = { workspace = true }
arc-swap = { workspace = true }
axum = { workspace = true }
common-alloc = { path = "../common/alloc" }
+common-cohort-team-access = { path = "../common/cohort-team-access" }
common-database = { path = "../common/database" }
common-kafka = { path = "../common/kafka" }
common-liveness = { path = "../common/liveness" }
@@ -32,5 +33,6 @@ tracing-subscriber = { workspace = true, features = ["env-filter", "json"] }
uuid = { workspace = true }
[dev-dependencies]
+common-cohort-team-access = { path = "../common/cohort-team-access", features = ["test-util"] }
proptest = "1"
tokio-util = { workspace = true }
diff --git a/rust/cohort-event-shuffler/src/config.rs b/rust/cohort-event-shuffler/src/config.rs
index 63e1559164a3..1b6ee87d5f5a 100644
--- a/rust/cohort-event-shuffler/src/config.rs
+++ b/rust/cohort-event-shuffler/src/config.rs
@@ -79,8 +79,9 @@ pub struct Config {
#[envconfig(default = "60")]
pub team_index_refresh_jitter_secs: u64,
- /// Teams whose events are forwarded into the shadow pipeline. Defaults to team 2 (the parity
- /// baseline's gate); set `all` to disable the gate. See [`TeamAllowlist`].
+ /// Teams whose events are forwarded into the shadow pipeline, on top of the teams that a row of
+ /// `cohort_realtime_team_access` grants. Defaults to team 2 (the parity baseline's gate); set
+ /// `all` to disable the gate. See [`TeamAllowlist`].
#[envconfig(from = "REALTIME_COHORT_TEAM_ALLOWLIST", default = "2")]
pub team_allowlist: TeamAllowlist,
diff --git a/rust/cohort-event-shuffler/src/consumer.rs b/rust/cohort-event-shuffler/src/consumer.rs
index 61f4dc7c751a..b057146dac19 100644
--- a/rust/cohort-event-shuffler/src/consumer.rs
+++ b/rust/cohort-event-shuffler/src/consumer.rs
@@ -10,7 +10,6 @@
use std::future::Future;
use std::pin::Pin;
-use std::sync::Arc;
use std::time::{Duration, Instant};
use common_kafka::kafka_consumer::{Offset, RecvErr, SingleTopicConsumer};
@@ -25,15 +24,15 @@ use tokio::task::JoinError;
use tracing::{debug, error, info, warn};
use crate::event::{personless_mode, CohortStreamEvent, SourcePartitionRange};
-use crate::filter_team_index::TeamIndex;
+use crate::filter_team_index::TeamGate;
use crate::ledger::{
DeliveryOutcome, Ledger, NextOffset, Observation, Resolution, SourceOffset, SourcePartition,
};
use crate::observability::metrics::{
- COMMITS, COMMIT_ERRORS, EVENTS_ABANDONED, EVENTS_CONSUMED, EVENTS_DROPPED_NO_PERSON_ID,
- EVENTS_FORWARDED, EVENTS_PERSON_PAYLOAD_DROPPED, EVENTS_SKIPPED_TEAM_GATE, EVENTS_UNPARSEABLE,
- FORWARDS_ENQUEUED, FORWARDS_INFLIGHT, LEDGER_PARTITIONS, PRODUCE_ACK_SECONDS, PRODUCE_ERRORS,
- PRODUCE_QUEUE_FULL, UNCOMMITTED_EVENTS,
+ ACTIVE_TEAMS, COMMITS, COMMIT_ERRORS, EVENTS_ABANDONED, EVENTS_CONSUMED,
+ EVENTS_DROPPED_NO_PERSON_ID, EVENTS_FORWARDED, EVENTS_PERSON_PAYLOAD_DROPPED,
+ EVENTS_SKIPPED_TEAM_GATE, EVENTS_UNPARSEABLE, FORWARDS_ENQUEUED, FORWARDS_INFLIGHT,
+ LEDGER_PARTITIONS, PRODUCE_ACK_SECONDS, PRODUCE_ERRORS, PRODUCE_QUEUE_FULL, UNCOMMITTED_EVENTS,
};
use crate::producer::{CohortStreamProducer, EnqueueError};
@@ -79,11 +78,11 @@ enum GateOutcome {
/// `person_id` is checked first: a missing routing key is attributed to `DropNoPersonId` even when
/// the team would also be skipped.
-fn classify_gate(gate: &GateFields, team_index: &TeamIndex) -> GateOutcome {
+fn classify_gate(gate: &GateFields, team_gate: &TeamGate) -> GateOutcome {
if gate.person_id.is_none() {
return GateOutcome::DropNoPersonId;
}
- if !team_index.contains(gate.team_id) {
+ if !team_gate.passes(gate.team_id) {
return GateOutcome::SkipTeamGate;
}
GateOutcome::Forward
@@ -110,11 +109,11 @@ enum Decoded {
/// Cheap gate parse for every event; a full `ClickHouseEvent` parse only for survivors, over the
/// same bytes so forwarded envelopes are byte-identical to a single-parse path.
-fn decode_gated(payload: &[u8], team_index: &TeamIndex) -> Decoded {
+fn decode_gated(payload: &[u8], team_gate: &TeamGate) -> Decoded {
let Ok(gate) = serde_json::from_slice::<GateFields>(payload) else {
return Decoded::Unparseable;
};
- match classify_gate(&gate, team_index) {
+ match classify_gate(&gate, team_gate) {
GateOutcome::DropNoPersonId => Decoded::DropNoPersonId,
GateOutcome::SkipTeamGate => Decoded::SkipTeamGate,
GateOutcome::Forward => {
@@ -206,7 +205,7 @@ impl Intake {
pub struct EventShuffler {
consumer: SingleTopicConsumer,
producer: CohortStreamProducer,
- team_index: Arc<TeamIndex>,
+ team_gate: TeamGate,
handle: Handle,
settings: ShufflerSettings,
}
@@ -215,14 +214,14 @@ impl EventShuffler {
pub fn new(
consumer: SingleTopicConsumer,
producer: CohortStreamProducer,
- team_index: Arc<TeamIndex>,
+ team_gate: TeamGate,
handle: Handle,
settings: ShufflerSettings,
) -> Self {
Self {
consumer,
producer,
- team_index,
+ team_gate,
handle,
settings,
}
@@ -283,14 +282,14 @@ impl EventShuffler {
};
}
- // Fail-closed until the team index loads once: don't consume — and advance
+ // Fail-closed until the team gate loads once: don't consume — and advance
// offsets past — events while the gate would forward nothing.
recv = tokio::time::timeout(
RECV_TIMEOUT,
- self.consumer.recv_with(|payload| Ok(decode_gated(payload, &self.team_index))),
+ self.consumer.recv_with(|payload| Ok(decode_gated(payload, &self.team_gate))),
), if intake.is_open()
&& ledger.in_flight() < self.settings.max_inflight_forwards
- && self.team_index.is_loaded() =>
+ && self.team_gate.is_loaded() =>
{
match recv {
Ok(Ok((decoded, offset))) => {
@@ -446,8 +445,8 @@ impl EventShuffler {
}
/// Plans a commit from the current assignment snapshot and spawns the blocking Sync commit
- /// into the single in-flight slot. Emits the ledger gauges even when there is nothing to
- /// commit.
+ /// into the single in-flight slot. Emits the ledger and team gauges even when there is nothing
+ /// to commit.
fn start_commit(&self, ledger: &mut Ledger) -> Option<CommitFuture> {
let assigned = match self.consumer.assigned_partitions() {
Ok(partitions) => partitions.into_iter().map(SourcePartition).collect(),
@@ -465,6 +464,7 @@ impl EventShuffler {
gauge!(FORWARDS_INFLIGHT).set(stats.in_flight as f64);
gauge!(UNCOMMITTED_EVENTS).set(stats.uncommitted_events as f64);
gauge!(LEDGER_PARTITIONS).set(stats.partitions as f64);
+ gauge!(ACTIVE_TEAMS).set(self.team_gate.passing_teams() as f64);
if plan.offsets.is_empty() {
return None;
@@ -631,11 +631,20 @@ fn should_heartbeat(commit_stale: bool, has_committable: bool) -> bool {
#[cfg(test)]
mod tests {
use super::*;
+ use std::sync::Arc;
+
use crate::event::sample_clickhouse_event;
+ use crate::filter_team_index::TeamIndex;
+ use common_cohort_team_access::TeamAccess;
+ use common_types::cohort::TeamAllowlist;
use serde_json::{json, Value};
- fn index_with(teams: &[i32]) -> TeamIndex {
- TeamIndex::from_teams(teams.iter().copied())
+ /// Every team processes, so only the index decides.
+ fn gate_with(teams: &[i32]) -> TeamGate {
+ TeamGate::new(
+ Arc::new(TeamIndex::from_teams(teams.iter().copied())),
+ TeamAccess::with_granted(TeamAllowlist::All, []),
+ )
}
fn gate_fields(team_id: i32, person_id: Option<IgnoredAny>) -> GateFields {
@@ -689,9 +698,9 @@ mod tests {
];
for (name, person_id, team_id, teams, expected) in cases {
- let index = index_with(&teams);
+ let gate = gate_with(&teams);
assert_eq!(
- classify_gate(&gate_fields(team_id, person_id), &index),
+ classify_gate(&gate_fields(team_id, person_id), &gate),
expected,
"case: {name}"
);
@@ -700,9 +709,12 @@ mod tests {
#[test]
fn unloaded_index_forwards_nothing() {
- let index = TeamIndex::new();
+ let gate = TeamGate::new(
+ Arc::new(TeamIndex::new()),
+ TeamAccess::with_granted(TeamAllowlist::All, []),
+ );
assert_eq!(
- classify_gate(&gate_fields(2, Some(IgnoredAny)), &index),
+ classify_gate(&gate_fields(2, Some(IgnoredAny)), &gate),
GateOutcome::SkipTeamGate
);
}
@@ -718,11 +730,11 @@ mod tests {
let payload = serde_json::to_vec(&value).unwrap();
assert!(matches!(
- decode_gated(&payload, &index_with(&[2])),
+ decode_gated(&payload, &gate_with(&[2])),
Decoded::SkipTeamGate
));
assert!(matches!(
- decode_gated(&payload, &index_with(&[99])),
+ decode_gated(&payload, &gate_with(&[99])),
Decoded::Unparseable
));
}
@@ -730,7 +742,7 @@ mod tests {
#[test]
fn garbage_payload_is_unparseable_not_an_error() {
assert!(matches!(
- decode_gated(b"not json at all", &index_with(&[2])),
+ decode_gated(b"not json at all", &gate_with(&[2])),
Decoded::Unparseable
));
}
@@ -744,17 +756,17 @@ mod tests {
let payload = serde_json::to_vec(&value).unwrap();
assert!(matches!(
- decode_gated(&payload, &index_with(&[2])),
+ decode_gated(&payload, &gate_with(&[2])),
Decoded::SkipTeamGate
));
}
#[test]
fn person_id_presence_matches_full_parse_semantics() {
- let index = index_with(&[2]);
+ let gate = gate_with(&[2]);
let base: Value =
serde_json::to_value(sample_clickhouse_event(2, Some("ignored"))).unwrap();
- let decode = |value: Value| decode_gated(&serde_json::to_vec(&value).unwrap(), &index);
+ let decode = |value: Value| decode_gated(&serde_json::to_vec(&value).unwrap(), &gate);
// Absent: drop (person check runs before the team gate).
let mut absent = base.clone();
@@ -795,7 +807,7 @@ mod tests {
#[test]
fn survivor_fully_materializes_the_forward_envelope() {
let payload = serde_json::to_vec(&sample_clickhouse_event(2, Some("p"))).unwrap();
- match decode_gated(&payload, &index_with(&[2])) {
+ match decode_gated(&payload, &gate_with(&[2])) {
Decoded::Forward { event, person_id } => {
assert_eq!(person_id, "p");
assert_eq!(event.team_id, 2);
diff --git a/rust/cohort-event-shuffler/src/filter_team_index.rs b/rust/cohort-event-shuffler/src/filter_team_index.rs
index f01dc9a2541d..9443b84bbc08 100644
--- a/rust/cohort-event-shuffler/src/filter_team_index.rs
+++ b/rust/cohort-event-shuffler/src/filter_team_index.rs
@@ -1,6 +1,7 @@
-//! In-memory index of teams with ≥1 realtime-supported cohort, consulted by the consumer's team
-//! gate. Refreshed on a `refresh_secs ± jitter` poll and swapped atomically via
-//! [`arc_swap::ArcSwap`] so hot-path reads are lock-free and a refresh never blocks the consumer.
+//! In-memory index of teams with ≥1 realtime-supported cohort, and the consumer's [`TeamGate`]
+//! that joins it with the team access record. Refreshed on a `refresh_secs ± jitter` poll and
+//! swapped atomically via [`arc_swap::ArcSwap`] so hot-path reads are lock-free and a refresh never
+//! blocks the consumer.
//!
//! Staleness is safe: the gate is defense-in-depth and the downstream processor re-checks
//! membership, so a stale snapshot only over-forwards briefly. The index starts empty and unloaded
@@ -14,14 +15,11 @@ use std::time::Duration;
use anyhow::{Context, Result};
use arc_swap::ArcSwap;
-use common_types::cohort::TeamAllowlist;
+use common_cohort_team_access::TeamAccess;
use lifecycle::Handle;
-use metrics::gauge;
use rand::Rng;
use sqlx::PgPool;
-use tracing::{debug, info, warn};
-
-use crate::observability::metrics::ACTIVE_TEAMS;
+use tracing::{info, warn};
/// Must match the Node filter manager's predicate (`cohort_type='realtime'`, not deleted, non-null
/// `filters`); projected to `DISTINCT team_id` since the shuffler only needs the gate.
@@ -32,22 +30,13 @@ const TEAMS_WITH_REALTIME_COHORTS_SQL: &str = "SELECT DISTINCT team_id \
pub struct TeamIndex {
teams: ArcSwap<HashSet<i32>>,
loaded: AtomicBool,
- /// Applied at [`refresh`](Self::refresh) time: teams the DB reports as realtime but outside this
- /// allowlist never enter the snapshot, so the hot-path gate and the `ACTIVE_TEAMS` gauge both
- /// reflect the scoped set for free.
- allowlist: TeamAllowlist,
}
impl TeamIndex {
pub fn new() -> Self {
- Self::with_allowlist(TeamAllowlist::All)
- }
-
- pub fn with_allowlist(allowlist: TeamAllowlist) -> Self {
Self {
teams: ArcSwap::from_pointee(HashSet::new()),
loaded: AtomicBool::new(false),
- allowlist,
}
}
@@ -76,47 +65,61 @@ impl TeamIndex {
}
fn store(&self, teams: HashSet<i32>) {
- gauge!(ACTIVE_TEAMS).set(teams.len() as f64);
self.teams.store(Arc::new(teams));
self.loaded.store(true, Ordering::Release);
}
pub async fn refresh(&self, pool: &PgPool) -> Result<usize> {
- let fetched: Vec<i32> = sqlx::query_scalar(TEAMS_WITH_REALTIME_COHORTS_SQL)
+ let fetched: HashSet<i32> = sqlx::query_scalar(TEAMS_WITH_REALTIME_COHORTS_SQL)
.fetch_all(pool)
.await
- .context("querying posthog_cohort for realtime teams")?;
- let fetched_count = fetched.len();
- let kept = filter_by_allowlist(fetched, &self.allowlist);
- if kept.len() != fetched_count {
- debug!(
- fetched = fetched_count,
- kept = kept.len(),
- excluded = fetched_count - kept.len(),
- "team index filtered realtime teams by the allowlist",
- );
- }
- let count = kept.len();
- self.store(kept);
+ .context("querying posthog_cohort for realtime teams")?
+ .into_iter()
+ .collect();
+ let count = fetched.len();
+ self.store(fetched);
Ok(count)
}
}
-/// Keep only the teams in scope. Separated from [`TeamIndex::refresh`] so the allowlist filter is
-/// testable without a database.
-fn filter_by_allowlist(fetched: Vec<i32>, allowlist: &TeamAllowlist) -> HashSet<i32> {
- fetched
- .into_iter()
- .filter(|team_id| allowlist.includes(*team_id))
- .collect()
-}
-
impl Default for TeamIndex {
fn default() -> Self {
Self::new()
}
}
+/// Passes an event when its team processes and has a realtime cohort. The index and the access
+/// record refresh on their own polls. Neither rebuilds the other's snapshot, because two writers
+/// to one snapshot can lose an update and bring back a revoked team.
+pub struct TeamGate {
+ index: Arc<TeamIndex>,
+ access: Arc<TeamAccess>,
+}
+
+impl TeamGate {
+ pub fn new(index: Arc<TeamIndex>, access: Arc<TeamAccess>) -> Self {
+ Self { index, access }
+ }
+
+ pub fn passes(&self, team_id: i32) -> bool {
+ self.index.contains(team_id) && self.access.processes(team_id)
+ }
+
+ /// False until both halves have loaded once.
+ pub fn is_loaded(&self) -> bool {
+ self.index.is_loaded() && self.access.is_loaded()
+ }
+
+ pub fn passing_teams(&self) -> usize {
+ self.index
+ .teams
+ .load()
+ .iter()
+ .filter(|&&team_id| self.access.processes(team_id))
+ .count()
+ }
+}
+
/// Sleeps `interval ± jitter` between refreshes to spread DB load across pods. On failure it keeps
/// serving the previous snapshot — staleness is safe — and retries next tick.
///
@@ -137,13 +140,8 @@ pub async fn run_refresh_loop(
tokio::select! {
_ = handle.shutdown_recv() => break,
_ = tokio::time::sleep(sleep_for) => match index.refresh(&pool).await {
- Ok(0) => {
- warn!(
- "team index has zero in-scope teams (no realtime cohorts, or all filtered \
- by REALTIME_COHORT_TEAM_ALLOWLIST); forwarding nothing",
- );
- }
- Ok(count) => info!(active_teams = count, "team index refreshed"),
+ Ok(0) => warn!("team index has no team with a realtime cohort; forwarding nothing"),
+ Ok(count) => info!(realtime_teams = count, "team index refreshed"),
Err(err) => warn!(
error = %err,
"team index refresh failed; keeping previous snapshot",
@@ -167,6 +165,7 @@ fn next_interval(interval: Duration, jitter: Duration) -> Duration {
#[cfg(test)]
mod tests {
use super::*;
+ use common_types::cohort::TeamAllowlist;
#[test]
fn new_index_is_unloaded_and_empty() {
@@ -200,21 +199,31 @@ mod tests {
}
#[test]
- fn filter_by_allowlist_all_keeps_everything() {
- let kept = filter_by_allowlist(vec![1, 2, 99], &TeamAllowlist::All);
- assert_eq!(kept, HashSet::from([1, 2, 99]));
- }
-
- #[test]
- fn filter_by_allowlist_only_drops_out_of_scope_teams() {
- let kept = filter_by_allowlist(vec![1, 2, 99], &TeamAllowlist::Only(HashSet::from([2])));
- assert_eq!(kept, HashSet::from([2]));
+ fn gate_passes_teams_that_process_and_have_a_realtime_cohort() {
+ let gate = TeamGate::new(
+ Arc::new(TeamIndex::from_teams([2, 5, 6])),
+ TeamAccess::with_granted(TeamAllowlist::Only(HashSet::from([2, 3])), [5]),
+ );
+
+ for (team_id, expected) in [(2, true), (3, false), (5, true), (6, false), (7, false)] {
+ assert_eq!(gate.passes(team_id), expected, "team {team_id}");
+ }
+ assert_eq!(gate.passing_teams(), 2);
}
#[test]
- fn filter_by_allowlist_empty_scope_keeps_nothing() {
- let kept = filter_by_allowlist(vec![1, 2, 99], &TeamAllowlist::Only(HashSet::new()));
- assert!(kept.is_empty());
+ fn gate_is_loaded_only_once_both_halves_are() {
+ let (unloaded_access, _poller) = TeamAccess::new(TeamAllowlist::All);
+ let loaded_access = TeamAccess::with_granted(TeamAllowlist::All, []);
+
+ let index_only = TeamGate::new(Arc::new(TeamIndex::from_teams([2])), unloaded_access);
+ let access_only = TeamGate::new(Arc::new(TeamIndex::new()), loaded_access.clone());
+ let both = TeamGate::new(Arc::new(TeamIndex::from_teams([2])), loaded_access);
+
+ assert!(!index_only.is_loaded());
+ assert!(!index_only.passes(2));
+ assert!(!access_only.is_loaded());
+ assert!(both.is_loaded());
}
#[test]
diff --git a/rust/cohort-event-shuffler/src/main.rs b/rust/cohort-event-shuffler/src/main.rs
index 0a4b85bb6599..642785e00f58 100644
--- a/rust/cohort-event-shuffler/src/main.rs
+++ b/rust/cohort-event-shuffler/src/main.rs
@@ -2,6 +2,7 @@ use std::sync::Arc;
use std::time::Duration;
use anyhow::{Context, Result};
+use common_cohort_team_access::TeamAccess;
use common_database::get_pool_with_config;
use common_kafka::kafka_consumer::SingleTopicConsumer;
use envconfig::Envconfig;
@@ -14,7 +15,7 @@ use tracing_subscriber::{fmt, EnvFilter, Layer};
use cohort_event_shuffler::config::Config;
use cohort_event_shuffler::consumer::EventShuffler;
-use cohort_event_shuffler::filter_team_index::{run_refresh_loop, TeamIndex};
+use cohort_event_shuffler::filter_team_index::{run_refresh_loop, TeamGate, TeamIndex};
use cohort_event_shuffler::observability;
use cohort_event_shuffler::producer::CohortStreamProducer;
@@ -55,12 +56,16 @@ async fn async_main(config: Config) -> Result<()> {
.with_liveness_deadline(Duration::from_secs(60))
.with_stall_threshold(3),
);
- // No liveness deadline: a refresh outage must not kill the service (staleness is safe). An
- // unexpected exit still signals the manager via the process-scope guard.
+ // No liveness deadline on either refresh: a refresh outage must not kill the service
+ // (staleness is safe). An unexpected exit still signals the manager via the process-scope guard.
let team_index_handle = manager.register(
"team-index",
ComponentOptions::new().with_graceful_shutdown(Duration::from_secs(5)),
);
+ let team_access_handle = manager.register(
+ "team-access",
+ ComponentOptions::new().with_graceful_shutdown(Duration::from_secs(5)),
+ );
let readiness = manager.readiness_handler();
let liveness = manager.liveness_handler();
@@ -76,15 +81,24 @@ async fn async_main(config: Config) -> Result<()> {
let pool = get_pool_with_config(&config.database_url, config.pool_config())
.context("creating posthog_cohort database pool")?;
- let team_index = Arc::new(TeamIndex::with_allowlist(config.team_allowlist.clone()));
+ let team_index = Arc::new(TeamIndex::new());
match team_index.refresh(&pool).await {
- Ok(count) => info!(active_teams = count, "initial team index loaded"),
+ Ok(count) => info!(realtime_teams = count, "initial team index loaded"),
Err(err) => warn!(
error = %err,
"initial team index load failed; consumer forwards nothing until the refresh task succeeds",
),
}
+ let (team_access, mut team_access_poller) = TeamAccess::new(config.team_allowlist.clone());
+ match team_access_poller.poll(&pool).await {
+ Ok(granted) => info!(granted_teams = granted, "initial team access loaded"),
+ Err(err) => warn!(
+ error = %err,
+ "initial team access load failed; consumer forwards nothing until a poll succeeds",
+ ),
+ }
+
let kafka_config = config.build_kafka_config();
let producer = CohortStreamProducer::new(&kafka_config, config.output_topic.clone())
.await
@@ -109,7 +123,10 @@ async fn async_main(config: Config) -> Result<()> {
.await;
});
- let shuffler = EventShuffler::new(consumer, producer, team_index, consumer_handle, settings);
+ tokio::spawn(team_access_poller.run(pool, team_access_handle));
+
+ let team_gate = TeamGate::new(team_index, team_access);
+ let shuffler = EventShuffler::new(consumer, producer, team_gate, consumer_handle, settings);
tokio::spawn(async move {
shuffler.process().await;
});
diff --git a/rust/cohort-event-shuffler/src/observability/metrics.rs b/rust/cohort-event-shuffler/src/observability/metrics.rs
index e8ad5819d338..c062e9161af1 100644
--- a/rust/cohort-event-shuffler/src/observability/metrics.rs
+++ b/rust/cohort-event-shuffler/src/observability/metrics.rs
@@ -18,6 +18,7 @@ pub const EVENTS_UNPARSEABLE: &str = "shuffler_events_unparseable_total";
pub const EVENTS_PERSON_PAYLOAD_DROPPED: &str = "shuffler_events_person_payload_dropped_total";
/// Delivery failures + fatal enqueue errors.
pub const PRODUCE_ERRORS: &str = "shuffler_produce_errors_total";
+/// Gauge, sampled at each commit tick: teams whose events pass the team gate.
pub const ACTIVE_TEAMS: &str = "shuffler_active_teams";
pub const FORWARDS_ENQUEUED: &str = "shuffler_forwards_enqueued_total";
diff --git a/rust/cohort-event-shuffler/tests/integration.rs b/rust/cohort-event-shuffler/tests/integration.rs
index 3d7fd3f79135..b85b2b17a60c 100644
--- a/rust/cohort-event-shuffler/tests/integration.rs
+++ b/rust/cohort-event-shuffler/tests/integration.rs
@@ -9,8 +9,9 @@ use std::time::{Duration, Instant};
use cohort_event_shuffler::config::Config;
use cohort_event_shuffler::consumer::EventShuffler;
use cohort_event_shuffler::event::CohortStreamEvent;
-use cohort_event_shuffler::filter_team_index::TeamIndex;
+use cohort_event_shuffler::filter_team_index::{TeamGate, TeamIndex};
use cohort_event_shuffler::producer::CohortStreamProducer;
+use common_cohort_team_access::TeamAccess;
use common_kafka::kafka_consumer::SingleTopicConsumer;
use common_types::cohort::TeamAllowlist;
use common_types::{ClickHouseEvent, PersonMode};
@@ -196,7 +197,11 @@ async fn shuffler_forwards_only_gated_events_with_stable_partitioning() {
..test_config(&bootstrap)
};
- let team_index = Arc::new(TeamIndex::from_teams([2, 7])); // team 99 is absent (no realtime cohorts)
+ // Team 99 is absent (no realtime cohorts).
+ let team_gate = TeamGate::new(
+ Arc::new(TeamIndex::from_teams([2, 7])),
+ TeamAccess::with_granted(TeamAllowlist::All, []),
+ );
let inputs = [
InputSpec {
@@ -306,7 +311,7 @@ async fn shuffler_forwards_only_gated_events_with_stable_partitioning() {
let shuffler = EventShuffler::new(
consumer,
producer,
- team_index,
+ team_gate,
handle,
config.shuffler_settings().unwrap(),
);
@@ -409,7 +414,10 @@ async fn shuffler_commits_via_drain_on_graceful_shutdown() {
// can carry the committed offset to the full input count.
config.commit_interval_ms = 3_600_000;
- let team_index = Arc::new(TeamIndex::from_teams([2, 7]));
+ let team_gate = TeamGate::new(
+ Arc::new(TeamIndex::from_teams([2, 7])),
+ TeamAccess::with_granted(TeamAllowlist::All, []),
+ );
// Forward, drop, skip, then forward last: seeing every forward in the output proves intake
// consumed the whole input (including the poison pill) before we trigger shutdown.
@@ -492,7 +500,7 @@ async fn shuffler_commits_via_drain_on_graceful_shutdown() {
let shuffler = EventShuffler::new(
consumer,
producer,
- team_index,
+ team_gate,
handle,
config.shuffler_settings().unwrap(),
);
@@ -539,7 +547,10 @@ async fn shuffler_retries_forwards_parked_on_queue_full() {
// whole burst through the RetryForward path.
config.kafka_producer_queue_messages = 1;
- let team_index = Arc::new(TeamIndex::from_teams([2, 7]));
+ let team_gate = TeamGate::new(
+ Arc::new(TeamIndex::from_teams([2, 7])),
+ TeamAccess::with_granted(TeamAllowlist::All, []),
+ );
// A drop and a skip, then a burst of forwards (all re-keyed to one partition), so both settled
// and acked offsets are committed under queue pressure.
@@ -605,7 +616,7 @@ async fn shuffler_retries_forwards_parked_on_queue_full() {
let shuffler = EventShuffler::new(
consumer,
producer,
- team_index,
+ team_gate,
handle,
config.shuffler_settings().unwrap(),
);
diff --git a/rust/common/cohort-team-access/Cargo.toml b/rust/common/cohort-team-access/Cargo.toml
new file mode 100644
index 000000000000..5f3f0bcc39d1
--- /dev/null
+++ b/rust/common/cohort-team-access/Cargo.toml
@@ -0,0 +1,25 @@
+[package]
+name = "common-cohort-team-access"
+version = "0.1.0"
+edition = "2021"
+
+[lints]
+workspace = true
+
+[features]
+# Exposes `TeamAccess::with_granted`, a loaded reader for tests that have no database.
+test-util = []
+
+[dependencies]
+arc-swap = { workspace = true }
+common-types = { path = "../types" }
+lifecycle = { path = "../lifecycle" }
+metrics = { workspace = true }
+rand = { workspace = true }
+sqlx = { workspace = true }
+thiserror = { workspace = true }
+tokio = { workspace = true }
+tracing = { workspace = true }
+
+[dev-dependencies]
+metrics-util = { workspace = true }
diff --git a/rust/common/cohort-team-access/src/lib.rs b/rust/common/cohort-team-access/src/lib.rs
new file mode 100644
index 000000000000..ee464560c645
--- /dev/null
+++ b/rust/common/cohort-team-access/src/lib.rs
@@ -0,0 +1,372 @@
+//! Which teams the realtime cohort pipeline processes.
+//!
+//! A team processes when `REALTIME_COHORT_TEAM_ALLOWLIST` names it, or when its row in Django's
+//! `cohort_realtime_team_access` table is at `processing` or `serving`. [`TeamAccessPoller`] reads
+//! the whole table on a short poll and swaps the rows into [`TeamAccess`], so a grant or a revoke
+//! reaches every reader within one poll, and the per-event check never waits on a lock.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+
+use arc_swap::ArcSwapOption;
+use common_types::cohort::TeamAllowlist;
+use lifecycle::Handle;
+use metrics::gauge;
+use rand::Rng;
+use sqlx::PgPool;
+use thiserror::Error;
+use tracing::{debug, warn};
+
+/// A transaction-mode PgBouncer does not keep the session `statement_timeout` that
+/// `common-database` sets after connect. So each read sets its own timeout inside its transaction,
+/// and the server cancels a blocked read instead of holding a pool connection.
+const QUERY_TIMEOUT_SQL: &str = "SET LOCAL statement_timeout = '5s'";
+const ACCESS_ROWS_SQL: &str = "SELECT team_id, level FROM cohort_realtime_team_access";
+
+/// Bounds the whole poll on the client. The statement timeout stops only the server, so a socket
+/// that dies after the pool hands it out would otherwise hang the poll and freeze every grant and
+/// revoke. Longer than the statement timeout, so the server cancels a slow read first.
+const POLL_TIMEOUT: Duration = Duration::from_secs(10);
+
+const POLL_INTERVAL: Duration = Duration::from_secs(15);
+const POLL_JITTER: Duration = Duration::from_secs(3);
+
+/// Zero until the first poll succeeds, then set only when a poll succeeds. So a staleness alert
+/// covers a reader that never loaded, and failed polls, including those that keep a regressed row.
+/// The charts allowlist of metrics that alerts can read matches the `posthog_cohort_` prefix.
+const LAST_SUCCESS: &str = "posthog_cohort_team_access_last_success_timestamp_seconds";
+
+#[derive(Debug, Error)]
+pub enum PollError {
+ #[error("querying cohort_realtime_team_access failed: {0}")]
+ Query(#[from] sqlx::Error),
+ #[error("reading cohort_realtime_team_access timed out")]
+ TimedOut(#[from] tokio::time::error::Elapsed),
+ #[error("team {team_id} has the unknown level {level:?}")]
+ UnknownLevel { team_id: i32, level: String },
+ /// Such a read is wrong: an empty table, another database, or a replica that lags behind the
+ /// previous read. See [`Level::may_become`]. These teams keep their previous rows, and the
+ /// read's other rows still take effect, so a revoke reaches the reader even then.
+ #[error("the read lacks the rows of teams {0:?}, or grants them again after `off`, so they keep their previous rows")]
+ RegressedRows(Vec<i32>),
+}
+
+/// The levels of `RealtimeCohortLevel` in `products/cohorts/backend/models/team_access.py`.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+enum Level {
+ Off,
+ Processing,
+ Serving,
+}
+
+impl Level {
+ fn parse(level: &str) -> Option<Self> {
+ match level {
+ "off" => Some(Self::Off),
+ "processing" => Some(Self::Processing),
+ "serving" => Some(Self::Serving),
+ _ => None,
+ }
+ }
+
+ fn processes(self) -> bool {
+ match self {
+ Self::Off => false,
+ Self::Processing | Self::Serving => true,
+ }
+ }
+
+ /// Whether a later read may show this row at `next`. Django never deletes a row, and it refuses
+ /// to grant an `off` row again, so only a wrong read shows either change.
+ fn may_become(self, next: Option<Self>) -> bool {
+ match (self, next) {
+ (_, None) => false,
+ (Self::Off, Some(Self::Off)) => true,
+ (Self::Off, Some(Self::Processing | Self::Serving)) => false,
+ (Self::Processing | Self::Serving, Some(_)) => true,
+ }
+ }
+}
+
+/// Every row of one read, `off` rows included, because the poller checks the next read against them.
+#[derive(Debug)]
+struct AccessRows(HashMap<i32, Level>);
+
+impl AccessRows {
+ fn parse(rows: Vec<(i32, String)>) -> Result<Self, PollError> {
+ rows.into_iter()
+ .map(|(team_id, level)| match Level::parse(&level) {
+ Some(parsed) => Ok((team_id, parsed)),
+ None => Err(PollError::UnknownLevel { team_id, level }),
+ })
+ .collect::<Result<_, _>>()
+ .map(Self)
+ }
+
+ fn processes(&self, team_id: i32) -> bool {
+ self.0.get(&team_id).is_some_and(|level| level.processes())
+ }
+
+ fn granted(&self) -> usize {
+ self.0.values().filter(|level| level.processes()).count()
+ }
+
+ /// Puts back the `previous` row of each team whose row here is a change that
+ /// [`Level::may_become`] rules out, and returns those teams.
+ fn restore_regressed(&mut self, previous: &Self) -> Vec<i32> {
+ let mut regressed = Vec::new();
+ for (&team_id, &level) in &previous.0 {
+ if !level.may_become(self.0.get(&team_id).copied()) {
+ self.0.insert(team_id, level);
+ regressed.push(team_id);
+ }
+ }
+ regressed.sort_unstable();
+ regressed
+ }
+}
+
+/// Answers whether a team processes, from the rows the poller stored last.
+#[derive(Debug)]
+pub struct TeamAccess {
+ env: TeamAllowlist,
+ /// `None` until the first accepted read. Until then no team processes, not even one the env list
+ /// names: a reader that ran on the env list alone would treat every granted team as revoked.
+ rows: ArcSwapOption<AccessRows>,
+}
+
+impl TeamAccess {
+ pub fn new(env: TeamAllowlist) -> (Arc<Self>, TeamAccessPoller) {
+ gauge!(LAST_SUCCESS).set(0.0);
+ let access = Arc::new(Self {
+ env,
+ rows: ArcSwapOption::empty(),
+ });
+ let poller = TeamAccessPoller {
+ access: access.clone(),
+ };
+ (access, poller)
+ }
+
+ /// A loaded reader whose one read granted `granted` at `processing`.
+ #[cfg(feature = "test-util")]
+ pub fn with_granted(env: TeamAllowlist, granted: impl IntoIterator<Item = i32>) -> Arc<Self> {
+ let rows = granted
+ .into_iter()
+ .map(|team_id| (team_id, Level::Processing))
+ .collect();
+ Arc::new(Self {
+ env,
+ rows: ArcSwapOption::from_pointee(AccessRows(rows)),
+ })
+ }
+
+ pub fn processes(&self, team_id: i32) -> bool {
+ self.rows
+ .load()
+ .as_ref()
+ .is_some_and(|rows| self.env.includes(team_id) || rows.processes(team_id))
+ }
+
+ pub fn is_loaded(&self) -> bool {
+ self.rows.load().is_some()
+ }
+}
+
+/// The only writer of its [`TeamAccess`]. It checks each read against the rows it stored last,
+/// which is sound only while one poller swaps the rows, so it is not `Clone`.
+#[derive(Debug)]
+pub struct TeamAccessPoller {
+ access: Arc<TeamAccess>,
+}
+
+impl TeamAccessPoller {
+ /// Reads the table and swaps the rows in. Returns how many rows grant processing.
+ ///
+ /// # Errors
+ ///
+ /// Fails when the query fails or times out, or when a row has a level this crate does not know.
+ /// The previous rows then stay in place. Also fails when the read lacks a row that the reader
+ /// holds, or grants one of its `off` rows again. Only those teams keep their previous rows.
+ /// So a row deleted or granted again by hand keeps its old state, and fails every later poll,
+ /// until the service restarts.
+ pub async fn poll(&mut self, pool: &PgPool) -> Result<usize, PollError> {
+ let read = async {
+ let mut transaction = pool.begin().await?;
+ sqlx::query(QUERY_TIMEOUT_SQL)
+ .execute(&mut *transaction)
+ .await?;
+ let fetched = sqlx::query_as::<_, (i32, String)>(ACCESS_ROWS_SQL)
+ .fetch_all(&mut *transaction)
+ .await?;
+ transaction.commit().await?;
+ Ok::<_, sqlx::Error>(fetched)
+ };
+ let fetched = tokio::time::timeout(POLL_TIMEOUT, read).await??;
+ self.accept(AccessRows::parse(fetched)?)
+ }
+
+ fn accept(&mut self, mut read: AccessRows) -> Result<usize, PollError> {
+ let regressed = match self.access.rows.load().as_deref() {
+ Some(previous) => read.restore_regressed(previous),
+ None => Vec::new(),
+ };
+ let granted = read.granted();
+ self.access.rows.store(Some(Arc::new(read)));
+ if !regressed.is_empty() {
+ return Err(PollError::RegressedRows(regressed));
+ }
+ gauge!(LAST_SUCCESS).set(now_unix_seconds());
+ Ok(granted)
+ }
+
+ /// Polls every `POLL_INTERVAL ± POLL_JITTER` until shutdown. After a failed poll it waits for
+ /// the next tick. Shutdown also cancels a poll in flight, because a poll can last longer than
+ /// the graceful shutdown window of the service. A cancelled poll changes no rows.
+ ///
+ /// Register `handle` without a liveness deadline: a database outage must not stop a service that
+ /// can keep working on the rows it has.
+ pub async fn run(mut self, pool: PgPool, handle: Handle) {
+ let _guard = handle.process_scope();
+
+ loop {
+ let tick = async {
+ tokio::time::sleep(next_interval()).await;
+ self.poll(&pool).await
+ };
+ tokio::select! {
+ _ = handle.shutdown_recv() => break,
+ polled = tick => match polled {
+ Ok(granted) => debug!(granted_teams = granted, "team access polled"),
+ Err(err) => warn!(error = %err, "team access poll failed"),
+ },
+ }
+ }
+ }
+}
+
+/// The jitter spreads the polls of many pods over time.
+fn next_interval() -> Duration {
+ let max_jitter_ms = 2 * POLL_JITTER.as_millis() as u64;
+ let jitter = Duration::from_millis(rand::thread_rng().gen_range(0..=max_jitter_ms));
+ POLL_INTERVAL - POLL_JITTER + jitter
+}
+
+/// Seconds since the Unix epoch, saturating at 0 on a clock set before it.
+fn now_unix_seconds() -> f64 {
+ SystemTime::now()
+ .duration_since(UNIX_EPOCH)
+ .map(|since| since.as_secs_f64())
+ .unwrap_or(0.0)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use metrics_util::debugging::DebuggingRecorder;
+
+ fn rows(rows: &[(i32, &str)]) -> AccessRows {
+ AccessRows::parse(
+ rows.iter()
+ .map(|&(team_id, level)| (team_id, level.to_string()))
+ .collect(),
+ )
+ .unwrap()
+ }
+
+ fn env_only(teams: &[i32]) -> TeamAllowlist {
+ TeamAllowlist::Only(teams.iter().copied().collect())
+ }
+
+ fn sets_last_success<T>(f: impl FnOnce() -> T) -> (T, bool) {
+ let recorder = DebuggingRecorder::new();
+ let snapshotter = recorder.snapshotter();
+ let out = metrics::with_local_recorder(&recorder, f);
+ let set = snapshotter
+ .snapshot()
+ .into_vec()
+ .iter()
+ .any(|(key, ..)| key.key().name() == LAST_SUCCESS);
+ (out, set)
+ }
+
+ #[test]
+ fn no_team_processes_before_the_first_accepted_read() {
+ let (access, _poller) = TeamAccess::new(TeamAllowlist::All);
+
+ assert!(!access.is_loaded());
+ assert!(!access.processes(2));
+ }
+
+ #[test]
+ fn processing_is_the_env_list_united_with_the_granted_rows() {
+ let (access, mut poller) = TeamAccess::new(env_only(&[2, 3]));
+ poller
+ .accept(rows(&[
+ (3, "off"),
+ (5, "processing"),
+ (6, "serving"),
+ (7, "off"),
+ ]))
+ .unwrap();
+
+ for (team_id, expected) in [
+ (2, true),
+ (3, true),
+ (5, true),
+ (6, true),
+ (7, false),
+ (8, false),
+ ] {
+ assert_eq!(access.processes(team_id), expected, "team {team_id}");
+ }
+ }
+
+ #[test]
+ fn a_read_that_lost_a_row_or_regranted_an_off_row_keeps_those_rows_and_applies_the_others() {
+ let (access, mut poller) = TeamAccess::new(env_only(&[]));
+ assert_eq!(poller.accept(rows(&[])).unwrap(), 0);
+ assert!(access.is_loaded());
+ let (accepted, set) = sets_last_success(|| {
+ poller.accept(rows(&[(5, "off"), (6, "processing"), (7, "processing")]))
+ });
+ accepted.unwrap();
+ assert!(set);
+
+ let (regressed, set) = sets_last_success(|| {
+ poller.accept(rows(&[(5, "processing"), (6, "off"), (8, "processing")]))
+ });
+ assert!(matches!(regressed, Err(PollError::RegressedRows(teams)) if teams == [5, 7]));
+ assert!(!set, "a reader that keeps a regressed row must look stale");
+ for (team_id, expected) in [(5, false), (6, false), (7, true), (8, true)] {
+ assert_eq!(access.processes(team_id), expected, "team {team_id}");
+ }
+ }
+
+ #[test]
+ fn a_read_that_changes_granted_levels_or_turns_every_row_off_is_accepted() {
+ let (access, mut poller) = TeamAccess::new(env_only(&[]));
+ poller
+ .accept(rows(&[(5, "processing"), (6, "serving")]))
+ .unwrap();
+ poller
+ .accept(rows(&[(5, "serving"), (6, "processing")]))
+ .unwrap();
+
+ assert_eq!(poller.accept(rows(&[(5, "off"), (6, "off")])).unwrap(), 0);
+ assert!(!access.processes(5));
+ assert!(!access.processes(6));
+ }
+
+ #[test]
+ fn an_unknown_level_fails_the_read() {
+ let parsed =
+ AccessRows::parse(vec![(5, "processing".to_string()), (6, "beta".to_string())]);
+
+ assert!(matches!(
+ parsed,
+ Err(PollError::UnknownLevel { team_id: 6, level }) if level == "beta"
+ ));
+ }
+}
diff --git a/rust/common/cohort-team-access/tests/postgres.rs b/rust/common/cohort-team-access/tests/postgres.rs
new file mode 100644
index 000000000000..b8e9a03733da
--- /dev/null
+++ b/rust/common/cohort-team-access/tests/postgres.rs
@@ -0,0 +1,65 @@
+//! Polls the table that Django creates. It fails instead of skipping without a database, because CI
+//! builds `test_posthog` before the Rust tests.
+
+use std::collections::HashSet;
+
+use common_cohort_team_access::{PollError, TeamAccess};
+use common_types::cohort::TeamAllowlist;
+use rand::Rng;
+use sqlx::PgPool;
+
+const DEFAULT_TEST_DATABASE_URL: &str = "postgres://posthog:posthog@localhost:5432/test_posthog";
+
+#[tokio::test]
+async fn poll_reads_every_level_django_writes_and_times_out_on_the_server() {
+ let url =
+ std::env::var("DATABASE_URL").unwrap_or_else(|_| DEFAULT_TEST_DATABASE_URL.to_string());
+ let pool = PgPool::connect(&url)
+ .await
+ .unwrap_or_else(|e| panic!("this test needs the Django test database at {url}: {e}"));
+ // Random ids, so rows that a failed earlier run left behind cannot collide.
+ let team_id: i32 = rand::thread_rng().gen_range(1_000_000_000..2_000_000_000);
+ let teams = [team_id, team_id + 1, team_id + 2];
+ sqlx::query(
+ "INSERT INTO cohort_realtime_team_access \
+ (team_id, level, purpose, save_triggers_backfill, processing_since, off_since, changed_at) \
+ VALUES ($1, 'processing', 'load_test', false, now(), NULL, now()), \
+ ($2, 'serving', 'pilot', true, now(), NULL, now()), \
+ ($3, 'off', 'load_test', false, now(), now(), now())",
+ )
+ .bind(teams[0])
+ .bind(teams[1])
+ .bind(teams[2])
+ .execute(&pool)
+ .await
+ .expect("insert access rows");
+
+ let (access, mut poller) = TeamAccess::new(TeamAllowlist::Only(HashSet::new()));
+ let polled = poller.poll(&pool).await;
+ // Delete before asserting, so a failed assertion leaves no rows behind.
+ sqlx::query("DELETE FROM cohort_realtime_team_access WHERE team_id = ANY($1)")
+ .bind(&teams[..])
+ .execute(&pool)
+ .await
+ .expect("delete access rows");
+
+ polled.expect("poll the access table");
+ assert!(access.processes(teams[0]));
+ assert!(access.processes(teams[1]));
+ assert!(!access.processes(teams[2]));
+
+ // A blocked read must end on the server's statement timeout (57014, query_canceled), not on the
+ // longer client timeout, which leaves the read running on the server. Keep this check in this
+ // test, because tests run in parallel and the table lock would stall another test's poll.
+ let mut lock = pool.begin().await.expect("begin the lock transaction");
+ sqlx::query("LOCK TABLE cohort_realtime_team_access IN ACCESS EXCLUSIVE MODE")
+ .execute(&mut *lock)
+ .await
+ .expect("lock the access table");
+ let blocked = poller.poll(&pool).await;
+ lock.rollback().await.expect("release the table lock");
+ assert!(
+ matches!(&blocked, Err(PollError::Query(sqlx::Error::Database(e))) if e.code().as_deref() == Some("57014")),
+ "the server must cancel a blocked read: {blocked:?}"
+ );
+}