Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(cohorts): add the realtime cohort team access record (#125592)

Matheus Batista6a6ef9facc83
diff --git a/.github/scripts/check-idor-model-coverage.py b/.github/scripts/check-idor-model-coverage.py
index addd4ffc2665..00ca07c20a34 100644
--- a/.github/scripts/check-idor-model-coverage.py
+++ b/.github/scripts/check-idor-model-coverage.py
@@ -225,6 +225,9 @@ def get_scoped_models() -> tuple[dict[str, set[str]], set[str], set[str], set[st
# events cleanup schedule; no endpoint looks it up by a user-supplied ID.
"TeamEventsRetentionConfig",
"TeamFeatureFlagsConfig",
+ # Staff-only OneToOne extension keyed on the team, written by a management command and read
+ # by the realtime cohort pipeline; no endpoint looks it up by a user-supplied ID.
+ "RealtimeCohortTeamAccess",
# OneToOne extension of Team keyed on team_id, only ever read as get(team=team) via
# get_or_create_team_extension; no endpoint looks it up by a user-supplied ID.
"TeamFeatureFlagPolicyConfig",
diff --git a/.semgrep/rules/security/idor-team-scoped-models.yaml b/.semgrep/rules/security/idor-team-scoped-models.yaml
index e9ccce433d24..7efcccb62f9f 100644
--- a/.semgrep/rules/security/idor-team-scoped-models.yaml
+++ b/.semgrep/rules/security/idor-team-scoped-models.yaml
@@ -1222,6 +1222,7 @@ rules:
|QueueSchedulerState
|QuickFilter
|QuickFilterContext
+ |RealtimeCohortTeamAccess
|RemoteConfig
|ReplayObservation
|ReplayObservationLabel
diff --git a/docs/internal/realtime-cohorts/backfill-coordination.md b/docs/internal/realtime-cohorts/backfill-coordination.md
index f35ad3466635..57a3ed82fc85 100644
--- a/docs/internal/realtime-cohorts/backfill-coordination.md
+++ b/docs/internal/realtime-cohorts/backfill-coordination.md
@@ -152,13 +152,14 @@ The membership consumer does count those markers toward its sweep, see [membersh
Several independent gates decide whether anything happens.
Missing one of them fails quietly, so they are worth knowing.
-| Gate | Effect |
-| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `REALTIME_COHORT_TEAM_ALLOWLIST` | Teams whose cohorts get hash maintenance, stamp invalidation, supersession and any run at all. Unset means no teams in Django, but the shuffler, processor and seeder default to team 2, so set it explicitly on every service. Set but empty means **all** teams. Must match the value the Rust services use |
-| `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST` | Teams whose saves create runs automatically. Unset or empty means **no** teams |
-| Operator attestations | Settings an operator sets to declare that prerequisites hold. They are declarations, not live checks. Behavioral runs need `BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED` and `BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED`. Person runs also need the person TTL and sizing attestations and a positive seed-bytes budget |
-| Person sizing budget | A cohort-scoped person run is refused when the team's active person-run estimates plus its own would exceed the budget. The check reads the active estimates before the run exists and reserves nothing, so two cohorts sized at the same time can together exceed the budget. A team run checks only its own estimate. Hitting the size estimate's scan cap, or the cap on pinned conditions, also refuses |
-| Behavioral scan limit | `create_cohort_backfill_run` refuses a behavioral team run when its pinned event names had more events than `BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY` on the busiest of the last 7 complete UTC days. `--max-scan-events-per-day` overrides it for one invocation, and 0 skips the count and the check. The save path does not check it |
+| Gate | Effect |
+| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `REALTIME_COHORT_TEAM_ALLOWLIST` | Teams whose cohorts get hash maintenance, stamp invalidation, supersession and any run at all. Unset means no teams in Django, but the shuffler, processor and seeder default to team 2, so set it explicitly on every service. Set but empty means **all** teams. Django puts a listed team at the `serving` level, and adds the teams that have a team access row. The Rust services' lists must name both, but never list a team granted `processing` here, because that makes it `serving` |
+| Team access rows | `cohort_realtime_team_access`, written only by `realtime_cohort_team_access grant\|revoke\|list`. Django reads it as a union with `REALTIME_COHORT_TEAM_ALLOWLIST`; the shuffler, processor and seeder read only their env lists, so a row changes Django alone. Until they read it, keep their env lists in step with the rows. A team granted only here gets runs nobody claims, each holding its cohort's run slot. A team revoked only here loses Django's edit-time supersede while Rust still processes it, so its readiness stamps outlive later edits. A row at `processing` gets the Django behavior above, but the team's users see no realtime state: no readiness, no realtime batch audience, and no backfilled cohort in flags or workflows, even one stamped before the grant, so a workflow that already uses one fails its next save. A row at `serving` also shows them. `revoke` sets `off`, which hides the stamps the same way, and a revoked team cannot be granted again. `grant` and `revoke` refuse a team on the env list |
+| `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST` | Teams whose saves create runs automatically. Unset or empty means **no** teams. A granted row with `save_triggers_backfill` adds its team, so `resave_cohorts --no-backfill-triggers`, not this setting, is what stops a resave from queueing runs |
+| Operator attestations | Settings an operator sets to declare that prerequisites hold. They are declarations, not live checks. Behavioral runs need `BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED` and `BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED`. Person runs also need the person TTL and sizing attestations and a positive seed-bytes budget |
+| Person sizing budget | A cohort-scoped person run is refused when the team's active person-run estimates plus its own would exceed the budget. The check reads the active estimates before the run exists and reserves nothing, so two cohorts sized at the same time can together exceed the budget. A team run checks only its own estimate. Hitting the size estimate's scan cap, or the cap on pinned conditions, also refuses |
+| Behavioral scan limit | `create_cohort_backfill_run` refuses a behavioral team run when its pinned event names had more events than `BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY` on the busiest of the last 7 complete UTC days. `--max-scan-events-per-day` overrides it for one invocation, and 0 skips the count and the check. The save path does not check it |
The Rust services have their own switches, and Django cannot see them.
A team must reach the processor's copy of `REALTIME_COHORT_TEAM_ALLOWLIST` before the seeder's.
diff --git a/docs/internal/realtime-cohorts/completion-and-readiness.md b/docs/internal/realtime-cohorts/completion-and-readiness.md
index fa7b578234f9..2f3161835bca 100644
--- a/docs/internal/realtime-cohorts/completion-and-readiness.md
+++ b/docs/internal/realtime-cohorts/completion-and-readiness.md
@@ -218,11 +218,12 @@ Person readiness may be enabled only once every region's flags service runs that
A `person_metadata` leaf makes a cohort non-realtime in the first place, so such a cohort is never compatible.
A cohort made only of references needs no stamp, so it is never compatible either.
+On a team whose access row is at `processing` or `off` and that is not on `REALTIME_COHORT_TEAM_ALLOWLIST`, no stamp counts, even one from before the grant, so its cohorts are never compatible.
The flag API accepts a cohort with event-based criteria, or one that depends on such a cohort, only when the realtime targeting rollout flag is on for the user and every such cohort is flag-compatible.
The same rollout flag lets the flag editor's cohort picker list these cohorts.
-Behind the same rollout flag, for allowlisted teams, the cohort API's `realtime` field reports one state per cohort that has event or person criteria: `static`, `person_properties`, `daily`, `building`, `rebuilding`, `ready`, or `needs_attention`.
+Behind the same rollout flag, for teams at the `serving` level, the cohort API's `realtime` field reports one state per cohort that has event or person criteria: `static`, `person_properties`, `daily`, `building`, `rebuilding`, `ready`, or `needs_attention`.
### In the feature flags service
diff --git a/docs/internal/realtime-cohorts/definitions-and-eligibility.md b/docs/internal/realtime-cohorts/definitions-and-eligibility.md
index ff6690f08f4c..b66dacf2d83d 100644
--- a/docs/internal/realtime-cohorts/definitions-and-eligibility.md
+++ b/docs/internal/realtime-cohorts/definitions-and-eligibility.md
@@ -110,7 +110,7 @@ The realtime membership serves two readers: feature flags on teams enabled in th
### Shape hashes
-For realtime, non-static, non-deleted cohorts on teams in `REALTIME_COHORT_TEAM_ALLOWLIST`, every save that includes `filters` also maintains three hash columns on the cohort row.
+For realtime, non-static, non-deleted cohorts on teams the pipeline processes ([gates](backfill-coordination.md#gates)), every save that includes `filters` also maintains three hash columns on the cohort row.
| Column | What it fingerprints | Read by |
| ------------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
diff --git a/posthog/api/cohort.py b/posthog/api/cohort.py
index 3f71c844ec32..65164754b06e 100644
--- a/posthog/api/cohort.py
+++ b/posthog/api/cohort.py
@@ -125,7 +125,7 @@ from products.cohorts.backend.realtime_state import (
CohortHistoryBuildPhase,
CohortRealtimeReadiness,
CohortRealtimeState,
- has_realtime_state,
+ cohorts_with_realtime_state,
resolve_realtime_readiness,
)
from products.feature_flags.backend.facade.config import ConfigFormatError, UnsupportedConfig, decode_config
@@ -1001,7 +1001,7 @@ def _resolve_realtime_readiness_if_in_rollout(
Eligibility is checked before the rollout flag, and not after, so a project the pipeline does
not cover never pays for a flag evaluation that cannot change its answer.
"""
- relevant = [cohort for cohort in cohorts if has_realtime_state(cohort)]
+ relevant = cohorts_with_realtime_state(cohorts)
if not relevant or not _realtime_targeting_enabled(context):
return {}
return resolve_realtime_readiness(relevant)
diff --git a/posthog/api/test/test_cohort.py b/posthog/api/test/test_cohort.py
index 6d4b925660a4..8987f83e4169 100644
--- a/posthog/api/test/test_cohort.py
+++ b/posthog/api/test/test_cohort.py
@@ -59,6 +59,11 @@ from products.cohorts.backend.models.backfill import CohortBackfillKind, CohortB
from products.cohorts.backend.models.calculation_history import CohortCalculationHistory
from products.cohorts.backend.models.cohort import DEFAULT_COHORT_INSERT_BATCH_SIZE, Cohort, CohortType
from products.cohorts.backend.models.dependencies import cohort_backfill_pending_key, find_behavioral_cohorts
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
from products.cohorts.backend.models.util import CohortErrorCode, count_cohort_members, list_cohort_member_ids
from products.exports.backend.api.test.test_exports import TestExportMixin
from products.feature_flags.backend.facade.enums import FlagEvaluationsMode
@@ -226,6 +231,7 @@ class TestCohort(TestExportMixin, ClickhouseTestMixin, APIBaseTest, QueryMatchin
"cohort_type": None,
"condition_type": None,
"realtime_enabled": False,
+ "realtime_level": "off",
},
team=ANY,
request=ANY,
@@ -273,18 +279,37 @@ class TestCohort(TestExportMixin, ClickhouseTestMixin, APIBaseTest, QueryMatchin
"cohort_type": None,
"condition_type": None,
"realtime_enabled": False,
+ "realtime_level": "off",
"updated_by_creator": True,
},
team=ANY,
request=ANY,
)
+ @parameterized.expand(
+ [
+ ("team_off_the_pipeline", None, False, "off"),
+ # A load-test team runs the pipeline, so it is enabled, but adoption counts only serving teams.
+ ("team_granted_processing", RealtimeCohortLevel.PROCESSING, True, "processing"),
+ ]
+ )
@patch("django.db.transaction.on_commit", side_effect=lambda func: func())
@patch("posthog.api.cohort.report_user_action")
@patch("posthog.tasks.calculate_cohort.calculate_cohort_ch.delay")
def test_cohort_created_reports_realtime_classification(
- self, patch_calculate_cohort, patch_capture, patch_on_commit
+ self,
+ _name: str,
+ access_level: RealtimeCohortLevel | None,
+ expected_enabled: bool,
+ expected_level: str,
+ patch_calculate_cohort,
+ patch_capture,
+ patch_on_commit,
):
+ if access_level is not None:
+ RealtimeCohortTeamAccess.objects.create(
+ team=self.team, level=access_level, purpose=RealtimeCohortAccessPurpose.LOAD_TEST
+ )
response = self.client.post(
f"/api/projects/{self.team.id}/cohorts",
data={
@@ -322,8 +347,7 @@ class TestCohort(TestExportMixin, ClickhouseTestMixin, APIBaseTest, QueryMatchin
reported["condition_type"],
{"person_properties": False, "behavioral": True, "lifecycle": False, "cohorts": False},
)
- # Realtime-eligible filters on a team the realtime pipeline does not cover.
- self.assertFalse(reported["realtime_enabled"])
+ self.assertEqual((reported["realtime_enabled"], reported["realtime_level"]), (expected_enabled, expected_level))
@patch("django.db.transaction.on_commit", side_effect=lambda func: func())
@patch("posthog.api.cohort.report_user_action")
@@ -428,7 +452,7 @@ class TestCohort(TestExportMixin, ClickhouseTestMixin, APIBaseTest, QueryMatchin
)
self.assertEqual(response.status_code, 201, response.content)
- with self.assertNumQueries(11):
+ with self.assertNumQueries(12):
response = self.client.get(f"/api/projects/{self.team.id}/cohorts")
assert len(response.json()["results"]) == 1
@@ -443,7 +467,7 @@ class TestCohort(TestExportMixin, ClickhouseTestMixin, APIBaseTest, QueryMatchin
)
self.assertEqual(response.status_code, 201, response.content)
- with self.assertNumQueries(11):
+ with self.assertNumQueries(12):
response = self.client.get(f"/api/projects/{self.team.id}/cohorts")
assert len(response.json()["results"]) == 3
diff --git a/posthog/management/commands/resave_cohorts.py b/posthog/management/commands/resave_cohorts.py
index 2a56f2dba814..88e229e1f4ed 100644
--- a/posthog/management/commands/resave_cohorts.py
+++ b/posthog/management/commands/resave_cohorts.py
@@ -17,7 +17,11 @@ from posthog.models.team.team import Team
from products.cohorts.backend.models.cohort import Cohort, CohortOrEmpty, CohortType
from products.cohorts.backend.models.util import get_all_cohort_dependencies, sort_cohorts_topologically
-from products.cohorts.backend.realtime_teams import is_realtime_cohort_team, realtime_allowlist_matches_every_team
+from products.cohorts.backend.realtime_teams import (
+ backfill_triggers_suppressed,
+ is_processing_team,
+ realtime_allowlist_matches_every_team,
+)
from products.feature_flags.backend.flags_cache import coalesced_cohort_flags_cache_rebuilds
logger = structlog.get_logger(__name__)
@@ -147,7 +151,12 @@ class Command(BaseCommand):
parser.add_argument(
"--realtime-allowlist",
action="store_true",
- help="Process the teams matched by REALTIME_COHORT_TEAM_ALLOWLIST.",
+ help="Process the teams the realtime pipeline processes: REALTIME_COHORT_TEAM_ALLOWLIST plus granted teams.",
+ )
+ parser.add_argument(
+ "--no-backfill-triggers",
+ action="store_true",
+ help="Enqueue no backfill run for a resaved cohort, whatever the team's trigger settings say.",
)
parser.add_argument(
"--batch-size",
@@ -166,6 +175,9 @@ class Command(BaseCommand):
selection = self._resolve_teams(options)
dry_run: bool = bool(options.get("dry_run"))
batch_size: int = int(options.get("batch_size") or 500)
+ backfill_triggers: AbstractContextManager[None] = (
+ backfill_triggers_suppressed() if options.get("no_backfill_triggers") else nullcontext()
+ )
# Log start of operation
logger.info(
@@ -194,7 +206,7 @@ class Command(BaseCommand):
total_teams = teams_qs.count()
# Process each team separately
- with termination_unwinds() as coalesce_rebuilds:
+ with termination_unwinds() as coalesce_rebuilds, backfill_triggers:
for team in teams_qs:
teams_processed += 1
logger.info(
@@ -354,15 +366,15 @@ class Command(BaseCommand):
allowed = tuple(
candidate
for candidate in Team.objects.order_by("id").values_list("id", flat=True)
- if is_realtime_cohort_team(candidate)
+ if is_processing_team(candidate)
)
# An allowlist that matches nothing would resave nothing and still report success, which
# during a rollout reads the same as "every cohort is classified".
if not allowed:
- raise CommandError("REALTIME_COHORT_TEAM_ALLOWLIST matches no teams")
+ raise CommandError("--realtime-allowlist matches no teams")
return TeamSelection(
team_ids=allowed,
- label=f"{len(allowed)} realtime-allowlisted teams",
+ label=f"{len(allowed)} realtime-processed teams",
explicit=True,
)
diff --git a/posthog/management/commands/test/test_resave_cohorts.py b/posthog/management/commands/test/test_resave_cohorts.py
index 97dcc8a99b86..4003189374ce 100644
--- a/posthog/management/commands/test/test_resave_cohorts.py
+++ b/posthog/management/commands/test/test_resave_cohorts.py
@@ -20,6 +20,11 @@ from posthog.models.team.team import Team
from posthog.test.db_context_capturing import capture_db_queries
from products.cohorts.backend.models.cohort import Cohort
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
from common.hogvm.python.operation import HOGQL_BYTECODE_VERSION
@@ -714,13 +719,18 @@ class TestResaveCohortsCommandTeamSelection(BaseTest):
assert cohort_b.condition_type is not None
assert cohort_c.condition_type is None
- def test_realtime_allowlist_processes_only_allowlisted_teams(self):
+ @parameterized.expand([("env_listed", True), ("granted", False)])
+ def test_realtime_allowlist_processes_only_realtime_teams(self, _name: str, env_listed: bool):
team_b: Team = Team.objects.create(organization=self.organization)
cohort_a = Cohort.objects.create(team=self.team, name="rt_a", filters=_make_realtime_filters())
cohort_b = Cohort.objects.create(team=team_b, name="rt_b", filters=_make_realtime_filters())
Cohort.objects.filter(id__in=[cohort_a.id, cohort_b.id]).update(condition_type=None)
+ if not env_listed:
+ RealtimeCohortTeamAccess.objects.create(
+ team=team_b, level=RealtimeCohortLevel.PROCESSING, purpose=RealtimeCohortAccessPurpose.LOAD_TEST
+ )
- with override_settings(REALTIME_COHORT_TEAM_ALLOWLIST=str(team_b.id)):
+ with override_settings(REALTIME_COHORT_TEAM_ALLOWLIST=str(team_b.id) if env_listed else "none"):
call_command("resave_cohorts", realtime_allowlist=True)
cohort_a.refresh_from_db()
@@ -728,6 +738,34 @@ class TestResaveCohortsCommandTeamSelection(BaseTest):
assert cohort_b.condition_type is not None
assert cohort_a.condition_type is None
+ @parameterized.expand([("triggers_on", False, 1), ("triggers_suppressed", True, 0)])
+ @override_settings(REALTIME_COHORT_TEAM_ALLOWLIST="none", COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST="none")
+ def test_no_backfill_triggers_overrides_a_granted_team_trigger(
+ self, _name: str, no_backfill_triggers: bool, expected_enqueues: int
+ ):
+ access = RealtimeCohortTeamAccess.objects.create(
+ team=self.team, level=RealtimeCohortLevel.PROCESSING, purpose=RealtimeCohortAccessPurpose.LOAD_TEST
+ )
+ person_filters = {
+ "properties": {
+ "type": "AND",
+ "values": [{"type": "person", "key": "email", "operator": "exact", "value": "[email protected]"}],
+ }
+ }
+ cohort = Cohort.objects.create(team=self.team, name="rt", filters=person_filters)
+ # A stale person hash makes the resave an edit of the person shape, which enqueues a run.
+ Cohort.objects.filter(id=cohort.id).update(condition_type=None, person_filters_shape_hash="stale")
+ access.save_triggers_backfill = True
+ access.save()
+
+ with (
+ patch("django.db.transaction.on_commit", side_effect=lambda func: func()),
+ patch("posthog.tasks.calculate_cohort.trigger_cohort_backfill_run_task.apply_async") as enqueue,
+ ):
+ call_command("resave_cohorts", team_id=[self.team.id], no_backfill_triggers=no_backfill_triggers)
+
+ assert enqueue.call_count == expected_enqueues
+
@override_settings(REALTIME_COHORT_TEAM_ALLOWLIST="all")
def test_realtime_allowlist_matching_every_team_processes_every_team(self):
team_b: Team = Team.objects.create(organization=self.organization)
diff --git a/posthog/test/test_cohort_model.py b/posthog/test/test_cohort_model.py
index 9119ce8abcb7..06d3ad29d1ee 100644
--- a/posthog/test/test_cohort_model.py
+++ b/posthog/test/test_cohort_model.py
@@ -19,6 +19,11 @@ from posthog.test.persons import add_cohort_members, create_person
from products.cohorts.backend.models.cohort import Cohort, CohortConditionFlags, CohortType, ImportResolution
from products.cohorts.backend.models.sql import GET_COHORTPEOPLE_BY_COHORT_ID
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
from products.cohorts.backend.models.util import count_cohort_members, list_cohort_member_ids
from products.event_definitions.backend.models.property_definition import PropertyDefinition, PropertyType
@@ -947,6 +952,31 @@ class TestCohortIsFlagCompatible(BaseTest):
)
self.assertEqual(cohort.is_flag_compatible, expected)
+ @parameterized.expand(
+ [
+ # A load-test team's runs stamp its cohorts too, but its flags never read membership.
+ ("processing", RealtimeCohortLevel.PROCESSING, False),
+ # After a revoke no edit clears the stamps, so they must not open flags and workflows.
+ ("revoked", RealtimeCohortLevel.OFF, False),
+ ("serving", RealtimeCohortLevel.SERVING, True),
+ # The legacy realtime workflow stamped cohorts on teams that have no access at all.
+ ("no_access", None, True),
+ ]
+ )
+ @override_settings(REALTIME_COHORT_TEAM_ALLOWLIST="none")
+ def test_stamps_do_not_count_on_a_row_below_serving(self, _label, level, expected):
+ if level is not None:
+ RealtimeCohortTeamAccess.objects.create(
+ team=self.team, level=level, purpose=RealtimeCohortAccessPurpose.LOAD_TEST
+ )
+ cohort = Cohort.objects.create(
+ team=self.team,
+ filters=_BEHAVIORAL_FILTERS,
+ cohort_type=CohortType.REALTIME,
+ last_backfill_events_at=_FIXED_TS,
+ )
+ self.assertEqual(cohort.is_flag_compatible, expected)
+
class TestCohortComputeConditionType(SimpleTestCase):
@parameterized.expand(
diff --git a/products/cohorts/backend/backfill/runs.py b/products/cohorts/backend/backfill/runs.py
index 37800ee04a72..9d8482b09502 100644
--- a/products/cohorts/backend/backfill/runs.py
+++ b/products/cohorts/backend/backfill/runs.py
@@ -39,7 +39,7 @@ from products.cohorts.backend.models.backfill import (
from products.cohorts.backend.models.cohort import Cohort, CohortType
from products.cohorts.backend.models.leaf_shape import walk_filter_leaves
from products.cohorts.backend.parity.eligibility import structural_exclusion
-from products.cohorts.backend.realtime_teams import is_realtime_cohort_team
+from products.cohorts.backend.realtime_teams import is_processing_team
logger = structlog.get_logger(__name__)
@@ -292,7 +292,7 @@ def attempt_backfill_run_for_cohort(team_id: int, cohort_id: int, trigger_kind:
static in the meantime refuses instead of creating a second run for the same slot. The reason
comes back on the attempt so the signal path can label its metric with it.
"""
- if not is_realtime_cohort_team(team_id):
+ if not is_processing_team(team_id, fresh=True):
return BackfillRunAttempt.refused(BackfillRefusalReason.TEAM_NOT_REALTIME)
try:
@@ -377,8 +377,8 @@ def create_team_backfill_run(
boundary_at: datetime | None = None,
scan_estimate: BehavioralScanEstimate | None = None,
) -> CohortBackfillRun:
- if not is_realtime_cohort_team(team_id):
- raise ValueError(f"Team {team_id} is not in the realtime cohort allowlist")
+ if not is_processing_team(team_id, fresh=True):
+ raise ValueError(f"Team {team_id} is not processed by the realtime cohort pipeline")
boundary_at = _validate_boundary_at(trigger_kind, boundary_at)
@@ -480,7 +480,7 @@ def attempt_person_backfill_run_for_cohort(
The reason for a refusal comes back on the attempt, so the signal path can label its metric
with it.
"""
- if not is_realtime_cohort_team(team_id):
+ if not is_processing_team(team_id, fresh=True):
return BackfillRunAttempt.refused(BackfillRefusalReason.TEAM_NOT_REALTIME)
horizon_days = (
@@ -677,8 +677,8 @@ def create_person_team_backfill_run(
created_by_id: int | None = None,
boundary_at: datetime | None = None,
) -> CohortBackfillRun:
- if not is_realtime_cohort_team(team_id):
- raise ValueError(f"Team {team_id} is not in the realtime cohort allowlist")
+ if not is_processing_team(team_id, fresh=True):
+ raise ValueError(f"Team {team_id} is not processed by the realtime cohort pipeline")
if person_horizon_days < 1:
raise ValueError("person_horizon_days must be at least 1")
diff --git a/products/cohorts/backend/backfill/test/test_management_command.py b/products/cohorts/backend/backfill/test/test_management_command.py
index 74dbb42cc9ab..b68230a677d8 100644
--- a/products/cohorts/backend/backfill/test/test_management_command.py
+++ b/products/cohorts/backend/backfill/test/test_management_command.py
@@ -364,7 +364,7 @@ class TestCreateCohortBackfillRunCommand(BaseTest):
@override_settings(REALTIME_COHORT_TEAM_ALLOWLIST="none")
def test_non_allowlisted_team_errors(self) -> None:
- with self.assertRaisesMessage(CommandError, "realtime cohort allowlist"):
+ with self.assertRaisesMessage(CommandError, "not processed by the realtime cohort pipeline"):
call_command("create_cohort_backfill_run", team_id=self.team.id, trigger="team_enablement")
@mock.patch("products.cohorts.backend.backfill.runs.estimate_person_seed_topic_bytes")
diff --git a/products/cohorts/backend/management/commands/create_cohort_backfill_run.py b/products/cohorts/backend/management/commands/create_cohort_backfill_run.py
index 1edd958577ee..7967018d6138 100644
--- a/products/cohorts/backend/management/commands/create_cohort_backfill_run.py
+++ b/products/cohorts/backend/management/commands/create_cohort_backfill_run.py
@@ -41,7 +41,7 @@ from products.cohorts.backend.backfill.sizing import (
from products.cohorts.backend.models.backfill import CohortBackfillKind, CohortBackfillRunCohort, CohortBackfillTrigger
from products.cohorts.backend.models.cohort import Cohort
from products.cohorts.backend.models.leaf_shape import walk_filter_leaves
-from products.cohorts.backend.realtime_teams import is_realtime_cohort_team
+from products.cohorts.backend.realtime_teams import is_processing_team
def _parse_boundary_at(value: str | None) -> datetime | None:
@@ -107,8 +107,8 @@ class Command(BaseCommand):
if max_scan_events_per_day < 0:
raise CommandError("--max-scan-events-per-day must be 0 or more")
- if not is_realtime_cohort_team(team_id):
- raise CommandError(f"Team {team_id} is not in the realtime cohort allowlist")
+ if not is_processing_team(team_id):
+ raise CommandError(f"Team {team_id} is not processed by the realtime cohort pipeline")
if kind == CohortBackfillKind.PERSON_PROPERTY:
assert person_horizon_days is not None
diff --git a/products/cohorts/backend/management/commands/realtime_cohort_team_access.py b/products/cohorts/backend/management/commands/realtime_cohort_team_access.py
new file mode 100644
index 000000000000..50652787e974
--- /dev/null
+++ b/products/cohorts/backend/management/commands/realtime_cohort_team_access.py
@@ -0,0 +1,112 @@
+"""Grant, revoke and list team access to the realtime cohort pipeline, meant for a toolbox pod.
+
+Only Django reads these rows, as a union with its `REALTIME_COHORT_TEAM_ALLOWLIST`. The shuffler,
+processor and seeder read their own env lists, so every grant or revoke needs the same change there.
+
+A grant at `processing` gives the team Django's side of the pipeline while its users see none of it,
+which is what a load test needs. `serving` also shows them realtime cohort state. `revoke --purpose
+load_test` sets every load-test row to `off` at once, and ends no run.
+
+A team on the env list is `serving` whatever its row says, so `grant` and `revoke` refuse it.
+"""
+
+import argparse
+from datetime import datetime
+from typing import Any
+
+from django.core.management.base import BaseCommand, CommandError, CommandParser
+
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
+from products.cohorts.backend.realtime_teams import (
+ GrantLevel,
+ grant_realtime_cohort_access,
+ revoke_realtime_cohort_access,
+)
+
+GRANT_LEVELS: dict[str, GrantLevel] = {
+ RealtimeCohortLevel.PROCESSING.value: RealtimeCohortLevel.PROCESSING,
+ RealtimeCohortLevel.SERVING.value: RealtimeCohortLevel.SERVING,
+}
+LIST_COLUMNS = ("team_id", "level", "purpose", "save_triggers_backfill", "processing_since", "off_since", "changed_at")
+
+
+class Command(BaseCommand):
+ help = "Grant, revoke and list team access to the realtime cohort pipeline."
+
+ def add_arguments(self, parser: CommandParser) -> None:
+ subparsers = parser.add_subparsers(dest="action", required=True)
+
+ grant = subparsers.add_parser("grant", help="Write the teams' access rows, all or none.")
+ grant.add_argument("--teams", type=int, nargs="+", required=True)
+ grant.add_argument("--purpose", choices=RealtimeCohortAccessPurpose.values, required=True)
+ # Required, so that re-running a grant cannot quietly move a serving team down to processing
+ # or turn its triggers off.
+ grant.add_argument(
+ "--level",
+ choices=list(GRANT_LEVELS),
+ required=True,
+ help="processing runs the pipeline unseen by the team's users; serving also shows them its state.",
+ )
+ grant.add_argument(
+ "--save-triggers-backfill",
+ action=argparse.BooleanOptionalAction,
+ required=True,
+ help="Whether a cohort save in these teams enqueues its backfill run.",
+ )
+
+ revoke = subparsers.add_parser("revoke", help="Set the selected rows to off, all or none.")
+ selector = revoke.add_mutually_exclusive_group(required=True)
+ selector.add_argument("--teams", type=int, nargs="+")
+ selector.add_argument("--purpose", choices=RealtimeCohortAccessPurpose.values)
+
+ subparsers.add_parser("list", help="Print every access row.")
+
+ def handle(self, *args: Any, **options: Any) -> None:
+ try:
+ match options["action"]:
+ case "grant":
+ self._grant(options)
+ case "revoke":
+ self._revoke(options)
+ case "list":
+ self._list()
+ except ValueError as error:
+ raise CommandError(str(error)) from error
+
+ def _grant(self, options: dict[str, Any]) -> None:
+ granted = grant_realtime_cohort_access(
+ options["teams"],
+ level=GRANT_LEVELS[options["level"]],
+ purpose=RealtimeCohortAccessPurpose(options["purpose"]),
+ save_triggers_backfill=options["save_triggers_backfill"],
+ )
+ self.stdout.write(f"Granted {options['level']} to {_team_list(granted)}")
+
+ def _revoke(self, options: dict[str, Any]) -> None:
+ if options["teams"] is not None:
+ rows = RealtimeCohortTeamAccess.objects.filter(team_id__in=options["teams"])
+ else:
+ rows = RealtimeCohortTeamAccess.objects.filter(purpose=options["purpose"])
+ revoked = revoke_realtime_cohort_access(rows)
+ self.stdout.write(f"Revoked {_team_list(revoked)}" if revoked else "No granted team matched")
+ if options["teams"] is not None and (not_granted := sorted(set(options["teams"]) - set(revoked))):
+ self.stdout.write(f"Not granted, left as they were: {_team_list(not_granted)}")
+
+ def _list(self) -> None:
+ self.stdout.write("\t".join(LIST_COLUMNS))
+ for row in RealtimeCohortTeamAccess.objects.order_by("team_id"):
+ values = [str(row.team_id), row.level, row.purpose, str(row.save_triggers_backfill)]
+ values += [_timestamp(row.processing_since), _timestamp(row.off_since), _timestamp(row.changed_at)]
+ self.stdout.write("\t".join(values))
+
+
+def _team_list(team_ids: list[int]) -> str:
+ return f"{len(team_ids)} teams: {', '.join(str(team_id) for team_id in team_ids)}"
+
+
+def _timestamp(value: datetime | None) -> str:
+ return value.isoformat(timespec="seconds") if value is not None else "-"
diff --git a/products/cohorts/backend/migrations/0017_realtime_cohort_team_access.py b/products/cohorts/backend/migrations/0017_realtime_cohort_team_access.py
new file mode 100644
index 000000000000..9d694d12015d
--- /dev/null
+++ b/products/cohorts/backend/migrations/0017_realtime_cohort_team_access.py
@@ -0,0 +1,55 @@
+# Generated by Django 5.2.18 on 2026-10-28 21:02
+
+import django.db.models.deletion
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+ dependencies = [
+ ("cohorts", "0016_cohort_static_calculating_idx"),
+ ("posthog", "1415_organizationaiterms"),
+ ]
+
+ operations = [
+ migrations.CreateModel(
+ name="RealtimeCohortTeamAccess",
+ fields=[
+ (
+ "team",
+ models.OneToOneField(
+ db_constraint=False,
+ on_delete=django.db.models.deletion.DO_NOTHING,
+ primary_key=True,
+ related_name="+",
+ serialize=False,
+ to="posthog.team",
+ ),
+ ),
+ (
+ "level",
+ models.CharField(
+ choices=[
+ ("off", "Off"),
+ ("processing", "Processing"),
+ ("serving", "Serving"),
+ ],
+ max_length=16,
+ ),
+ ),
+ (
+ "purpose",
+ models.CharField(
+ choices=[("pilot", "Pilot"), ("load_test", "Load test")],
+ max_length=16,
+ ),
+ ),
+ ("save_triggers_backfill", models.BooleanField(default=False)),
+ ("processing_since", models.DateTimeField(null=True)),
+ ("off_since", models.DateTimeField(null=True)),
+ ("changed_at", models.DateTimeField(auto_now=True)),
+ ],
+ options={
+ "db_table": "cohort_realtime_team_access",
+ },
+ ),
+ ]
diff --git a/products/cohorts/backend/migrations/max_migration.txt b/products/cohorts/backend/migrations/max_migration.txt
index 10167e233c78..0e7c8aee6d1f 100644
--- a/products/cohorts/backend/migrations/max_migration.txt
+++ b/products/cohorts/backend/migrations/max_migration.txt
@@ -1 +1 @@
-0016_cohort_static_calculating_idx
+0017_realtime_cohort_team_access
diff --git a/products/cohorts/backend/models/__init__.py b/products/cohorts/backend/models/__init__.py
index 129cf99b223c..c15dbc9317ee 100644
--- a/products/cohorts/backend/models/__init__.py
+++ b/products/cohorts/backend/models/__init__.py
@@ -20,6 +20,7 @@ from .cohort import (
CohortType,
get_or_create_internal_test_users_cohort,
)
+from .team_access import RealtimeCohortAccessPurpose, RealtimeCohortLevel, RealtimeCohortTeamAccess
__all__ = [
"DEFAULT_COHORT_INSERT_BATCH_SIZE",
@@ -38,5 +39,8 @@ __all__ = [
"CohortOrEmpty",
"CohortPeople",
"CohortType",
+ "RealtimeCohortAccessPurpose",
+ "RealtimeCohortLevel",
+ "RealtimeCohortTeamAccess",
"get_or_create_internal_test_users_cohort",
]
diff --git a/products/cohorts/backend/models/cohort.py b/products/cohorts/backend/models/cohort.py
index 8f7ffaa7a5d3..aa56665db864 100644
--- a/products/cohorts/backend/models/cohort.py
+++ b/products/cohorts/backend/models/cohort.py
@@ -43,7 +43,8 @@ from products.cohorts.backend.models.leaf_shape import (
extract_person_composition_hash,
extract_person_leaf_shape_hash,
)
-from products.cohorts.backend.realtime_teams import is_realtime_cohort_team
+from products.cohorts.backend.models.team_access import RealtimeCohortLevel
+from products.cohorts.backend.realtime_teams import is_processing_team, realtime_level, team_honors_readiness_stamps
if TYPE_CHECKING:
from posthog.models.team import Team
@@ -370,12 +371,14 @@ class Cohort(FileSystemSyncMixin, RootTeamMixin, models.Model):
self._person_shape_changed = False
maintained_update_fields = set(update_fields) if update_fields is not None else None
try:
- if not self.team_id or not is_realtime_cohort_team(self.team_id):
- return maintained_update_fields
if self.cohort_type != CohortType.REALTIME or self.is_static or self.deleted:
return maintained_update_fields
if maintained_update_fields is not None and "filters" not in maintained_update_fields:
return maintained_update_fields
+ # Read the row, not the cached snapshot: right after a grant, a stale snapshot would skip
+ # the supersede for an edit while the uncached run creators already accept the team.
+ if not self.team_id or not is_processing_team(self.team_id, fresh=True):
+ return maintained_update_fields
new_shape_hash = extract_leaf_shape_hash(self.filters)
new_behavioral_shape_hash = extract_behavioral_leaf_shape_hash(self.filters)
@@ -608,12 +611,18 @@ class Cohort(FileSystemSyncMixin, RootTeamMixin, models.Model):
Every backfill its current filters need has to have landed: person property and
person_metadata filters need `last_backfill_person_properties_at`, behavioral event filters
need `last_backfill_events_at`, and a cohort with both needs both.
+
+ Stamps do not count on a team whose access row is below `serving`
+ (`team_honors_readiness_stamps`). The pipeline stamps a `processing` team's cohorts, but the
+ flags service reads no membership for it, and its users must not see realtime cohorts.
"""
if self.cohort_type != CohortType.REALTIME:
return False
stamps = self._required_backfill_stamps()
- return bool(stamps) and all(stamp is not None for stamp in stamps.values())
+ if not stamps or any(stamp is None for stamp in stamps.values()):
+ return False
+ return team_honors_readiness_stamps(self.team_id)
@property
def realtime_ready_at(self) -> Optional[datetime]:
@@ -694,6 +703,7 @@ class Cohort(FileSystemSyncMixin, RootTeamMixin, models.Model):
return PropertyGroup(PropertyOperatorType.AND, cast(list[Property], []))
def get_analytics_metadata(self):
+ level = realtime_level(self.team_id) if self.team_id else RealtimeCohortLevel.OFF
return {
"filters": self.properties.to_dict(),
"name_length": len(self.name) if self.name else 0,
@@ -706,7 +716,10 @@ class Cohort(FileSystemSyncMixin, RootTeamMixin, models.Model):
# `cohort_type` reads `realtime` whenever the filters are realtime-eligible, on every
# team. This reports whether the team runs the realtime pipeline at all, so a consumer
# can tell an eligible cohort from one that realtime membership is maintained for.
- "realtime_enabled": bool(self.team_id) and is_realtime_cohort_team(self.team_id),
+ "realtime_enabled": level != RealtimeCohortLevel.OFF,
+ # `processing` teams run the pipeline while their users see none of it, so adoption
+ # counts only `serving`.
+ "realtime_level": level.value,
}
def _safe_reset_calculating_state(self, completed_version: int) -> None:
diff --git a/products/cohorts/backend/models/dependencies.py b/products/cohorts/backend/models/dependencies.py
index 7332b9771a78..8b4117ef6f0b 100644
--- a/products/cohorts/backend/models/dependencies.py
+++ b/products/cohorts/backend/models/dependencies.py
@@ -19,7 +19,11 @@ from posthog.redis import get_client as get_redis_client
from products.cohorts.backend.models.backfill import CohortBackfillKind, CohortBackfillTrigger
from products.cohorts.backend.models.cohort import Cohort, CohortType, is_cohort_recalculation_only_save
-from products.cohorts.backend.realtime_teams import is_cohort_backfill_trigger_team, is_realtime_cohort_team
+from products.cohorts.backend.realtime_teams import (
+ is_cohort_backfill_trigger_team,
+ is_processing_team,
+ team_honors_readiness_stamps,
+)
logger = get_logger(__name__)
DEPENDENCY_CACHE_TIMEOUT = 7 * 24 * 60 * 60 # 1 week
@@ -76,8 +80,8 @@ COHORT_REALTIME_STATE_ORPHANED_COUNTER = Counter(
labelnames=["reason"],
)
-# `not_realtime_team` is the one outcome with no log line: it fires on every save of a
-# trigger-allowlisted team the realtime allowlist misses, and logging that per save would be noise
+# `not_realtime_team` is the one outcome with no log line: it fires on every triggering save of a
+# trigger-allowlisted team the pipeline does not process, and logging that per save would be noise
# while the misconfiguration itself deserves a signal an operator can alert on.
COHORT_BACKFILL_TRIGGER_COUNTER = Counter(
"posthog_cohort_backfill_trigger_total",
@@ -203,6 +207,7 @@ def _compute_flag_excluded_behavioral_cohort_ids(team_id: int, *, allow_realtime
.filter(Q(_filters_text__icontains="behavioral") | Q(_filters_text__icontains="cohort"))
.only(
"id",
+ "team_id",
"is_static",
"filters",
"cohort_type",
@@ -218,7 +223,11 @@ def get_flag_excluded_behavioral_cohort_ids(team_id: int, *, allow_realtime_back
"""Behavioral (flag-incompatible) cohort ids for a team, cached across requests."""
# feature_enabled can return None when the flag can't be evaluated; normalize so the
# cache key is stable and the compute path sees a real bool.
- allow_realtime_backfilled = bool(allow_realtime_backfilled)
+ #
+ # A team that does not honor readiness stamps has no flag-compatible cohort, so it uses the cache
+ # entry for allow_realtime_backfilled=False. A grant or a revoke then moves the team to the other
+ # entry instead of reading one computed before the change.
+ allow_realtime_backfilled = bool(allow_realtime_backfilled) and team_honors_readiness_stamps(team_id)
cache_key = _behavioral_cohort_ids_key(team_id, allow_realtime_backfilled)
cached = cache.get(cache_key)
if cached is not None: # empty list is a valid cached result, not a miss
@@ -626,11 +635,11 @@ def cohort_behavioral_shape_changed_supersede(sender, instance, **kwargs):
Part of the Django-to-Rust contract, not a leftover of the removed Python backfill trigger: a run
row pins the cohort's filters and shape hashes (see backfill/runs.py), and rust/cohort-seeder
claims those rows to replay history. An edit mid-run leaves the seeder working from a definition
- that no longer exists, so the run has to be marked superseded. `is_realtime_cohort_team` explains
- why the allowlist bounds this: Django's edit-time invalidation must cover exactly the teams Rust
- maintains realtime membership for.
+ that no longer exists, so the run has to be marked superseded. `is_processing_team` bounds this,
+ because Django's edit-time invalidation must cover exactly the teams Rust maintains realtime
+ membership for.
- `_leaf_shape_changed` is only set for allowlisted realtime, non-static, non-deleted cohorts on a
+ `_leaf_shape_changed` is only set for processed realtime, non-static, non-deleted cohorts on a
real `filters` change, so it is the whole guard.
"""
if not instance._leaf_shape_changed:
@@ -650,7 +659,7 @@ def cohort_person_shape_changed_supersede(sender, instance, **kwargs):
readiness over a backfill whose seeded state went stale during the B window; only the
`superseded_at` set here refuses that stamp.
- `_person_shape_changed` is only set for allowlisted realtime, non-static, non-deleted cohorts on
+ `_person_shape_changed` is only set for processed realtime, non-static, non-deleted cohorts on
a real `filters` change, so it is the whole guard.
"""
if not instance._person_shape_changed:
@@ -675,16 +684,17 @@ def cohort_behavioral_shape_changed_backfill(sender, instance, **kwargs):
task on every create that the creators are guaranteed to refuse, with nothing naming the cause.
"""
try:
+ # Decided in memory first, so a save that triggers nothing never reads the team's access.
+ trigger_kind = _backfill_trigger_kind(instance, kwargs, shape_changed=instance._leaf_shape_changed)
+ if trigger_kind is None:
+ return
if not is_cohort_backfill_trigger_team(instance.team_id):
return
- if not is_realtime_cohort_team(instance.team_id):
+ if not is_processing_team(instance.team_id, fresh=True):
COHORT_BACKFILL_TRIGGER_COUNTER.labels(
backfill_kind=CohortBackfillKind.BEHAVIORAL, outcome="not_realtime_team"
).inc()
return
- trigger_kind = _backfill_trigger_kind(instance, kwargs, shape_changed=instance._leaf_shape_changed)
- if trigger_kind is None:
- return
ineligibility = _backfill_ineligibility_reason(instance, CohortBackfillKind.BEHAVIORAL)
if ineligibility is not None:
_report_backfill_ineligible(instance, CohortBackfillKind.BEHAVIORAL, ineligibility)
@@ -709,16 +719,17 @@ def cohort_person_shape_changed_backfill(sender, instance, **kwargs):
different stores and stamp different readiness columns.
"""
try:
+ # Decided in memory first, so a save that triggers nothing never reads the team's access.
+ trigger_kind = _backfill_trigger_kind(instance, kwargs, shape_changed=instance._person_shape_changed)
+ if trigger_kind is None:
+ return
if not is_cohort_backfill_trigger_team(instance.team_id):
return
- if not is_realtime_cohort_team(instance.team_id):
+ if not is_processing_team(instance.team_id, fresh=True):
COHORT_BACKFILL_TRIGGER_COUNTER.labels(
backfill_kind=CohortBackfillKind.PERSON_PROPERTY, outcome="not_realtime_team"
).inc()
return
- trigger_kind = _backfill_trigger_kind(instance, kwargs, shape_changed=instance._person_shape_changed)
- if trigger_kind is None:
- return
ineligibility = _backfill_ineligibility_reason(instance, CohortBackfillKind.PERSON_PROPERTY)
if ineligibility is not None:
_report_backfill_ineligible(instance, CohortBackfillKind.PERSON_PROPERTY, ineligibility)
diff --git a/products/cohorts/backend/models/team_access.py b/products/cohorts/backend/models/team_access.py
new file mode 100644
index 000000000000..c49bc5f25b8f
--- /dev/null
+++ b/products/cohorts/backend/models/team_access.py
@@ -0,0 +1,45 @@
+from django.db import models
+
+
+class RealtimeCohortLevel(models.TextChoices):
+ OFF = "off", "Off"
+ # The pipeline maintains membership for the team, but nothing the team's users see reads it.
+ PROCESSING = "processing", "Processing"
+ # Processing, plus the realtime surfaces the team's users see.
+ SERVING = "serving", "Serving"
+
+
+class RealtimeCohortAccessPurpose(models.TextChoices):
+ PILOT = "pilot", "Pilot"
+ LOAD_TEST = "load_test", "Load test"
+
+
+class RealtimeCohortTeamAccess(models.Model):
+ """A team's access to the realtime cohort pipeline, granted on top of `REALTIME_COHORT_TEAM_ALLOWLIST`.
+
+ Staff write it only through the `realtime_cohort_team_access` command. Only Django reads it, so the
+ Rust services' env lists must change with it. A missing row means `off`. Never read it through
+ `get_or_create_team_extension`, because that creates the row.
+
+ Rows are never deleted: a revoke sets `off`, and a team deletion leaves the row (`DO_NOTHING`).
+ So when a team's row appears on one read and is missing on a later one, the later read failed.
+ A reader treats that as an error, not as a revoke.
+ """
+
+ # db_constraint=False: a real FK constraint would take a SHARE ROW EXCLUSIVE lock on
+ # posthog_team (a hot table) while migrating. The column stays int4, which the Rust services decode.
+ team = models.OneToOneField(
+ "posthog.Team", on_delete=models.DO_NOTHING, primary_key=True, db_constraint=False, related_name="+"
+ )
+ level = models.CharField(max_length=16, choices=RealtimeCohortLevel.choices)
+ purpose = models.CharField(max_length=16, choices=RealtimeCohortAccessPurpose.choices)
+ # The row's half of `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST`: whether a cohort save enqueues a backfill.
+ save_triggers_backfill = models.BooleanField(default=False)
+ # Set only on a move from off to on, and kept through later level changes.
+ processing_since = models.DateTimeField(null=True)
+ # Set only on a move to off.
+ off_since = models.DateTimeField(null=True)
+ changed_at = models.DateTimeField(auto_now=True)
+
+ class Meta:
+ db_table = "cohort_realtime_team_access"
diff --git a/products/cohorts/backend/models/test/test_behavioral_backfill_dependencies.py b/products/cohorts/backend/models/test/test_behavioral_backfill_dependencies.py
index 300a17712f65..64a39d700a51 100644
--- a/products/cohorts/backend/models/test/test_behavioral_backfill_dependencies.py
+++ b/products/cohorts/backend/models/test/test_behavioral_backfill_dependencies.py
@@ -12,6 +12,12 @@ from products.cohorts.backend.models.dependencies import (
COHORT_BACKFILL_TRIGGER_COUNTER,
COHORT_REALTIME_STATE_ORPHANED_COUNTER,
)
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
+from products.cohorts.backend.realtime_teams import _access_rows, is_processing_team
# The catalog drops a leaf with no bytecode or a `conditionHash` that is not 16 characters, and
# `_calculate_realtime_support` grants `cohort_type=REALTIME` only when every leaf compiled to
@@ -260,6 +266,26 @@ class TestBehavioralBackfillDependencies(BaseTest):
self.assertEqual(cohort.last_realtime_cohort_calculation_at, ready_at)
self.assertEqual(self._orphan_count(), before)
+ @override_settings(REALTIME_COHORT_TEAM_ALLOWLIST="none")
+ def test_edit_right_after_a_grant_maintains_shape_hashes(self) -> None:
+ cohort = self._cohort(7)
+ _access_rows.clear_cache()
+ self.addCleanup(_access_rows.clear_cache)
+
+ with mock.patch("posthog.cache_utils.TEST", False):
+ assert not is_processing_team(self.team.pk)
+ RealtimeCohortTeamAccess.objects.create(
+ team=self.team, level=RealtimeCohortLevel.PROCESSING, purpose=RealtimeCohortAccessPurpose.LOAD_TEST
+ )
+ # The snapshot still predates the grant, so only a fresh read sees the team.
+ assert not is_processing_team(self.team.pk)
+
+ cohort.filters = self._filters(30)
+ cohort.save()
+
+ cohort.refresh_from_db()
+ self.assertIsNotNone(cohort.behavioral_filters_shape_hash)
+
def _redis(self, *set_results: bool) -> mock.Mock:
redis = mock.Mock()
if set_results:
diff --git a/products/cohorts/backend/models/test/test_dependencies.py b/products/cohorts/backend/models/test/test_dependencies.py
index 114c1f25c573..9c9c206e9fdb 100644
--- a/products/cohorts/backend/models/test/test_dependencies.py
+++ b/products/cohorts/backend/models/test/test_dependencies.py
@@ -24,6 +24,11 @@ from products.cohorts.backend.models.dependencies import (
get_flag_excluded_behavioral_cohort_ids,
warm_team_cohort_dependency_cache,
)
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
class TestCohortDependencies(BaseTest):
@@ -721,6 +726,15 @@ class TestFlagExcludedBehavioralCohortIds(BaseTest):
self.assertIn(realtime_behavioral.id, excluded_without)
self.assertNotIn(realtime_behavioral.id, excluded_with)
+ # A grant writes no cohort, so nothing invalidates the cached answer from before it.
+ RealtimeCohortTeamAccess.objects.create(
+ team=self.team, level=RealtimeCohortLevel.PROCESSING, purpose=RealtimeCohortAccessPurpose.LOAD_TEST
+ )
+ self.assertIn(
+ realtime_behavioral.id,
+ get_flag_excluded_behavioral_cohort_ids(self.team.id, allow_realtime_backfilled=True),
+ )
+
def test_detects_behavioral_nested_in_group_and_two_hop_referrers(self) -> None:
# Behavioral node nested inside an inner AND group exercises the check_property_values recursion.
behavioral = Cohort.objects.create(
diff --git a/products/cohorts/backend/realtime_state.py b/products/cohorts/backend/realtime_state.py
index 8f22be691178..61e5693e8bd3 100644
--- a/products/cohorts/backend/realtime_state.py
+++ b/products/cohorts/backend/realtime_state.py
@@ -42,7 +42,7 @@ from products.cohorts.backend.models.backfill import (
)
from products.cohorts.backend.models.cohort import Cohort, CohortType
from products.cohorts.backend.models.dependencies import cohort_backfill_pending_key
-from products.cohorts.backend.realtime_teams import is_realtime_cohort_team
+from products.cohorts.backend.realtime_teams import is_serving_team
logger = structlog.get_logger(__name__)
@@ -401,25 +401,33 @@ def _resolve_for_team(team_id: int, cohorts: Sequence[Cohort]) -> dict[int, Coho
def has_realtime_state(cohort: Cohort) -> bool:
"""Whether this cohort can have a flag targeting state at all.
- False on a team that does not run the pipeline, because its flags can never read realtime
- membership, and for a cohort whose criteria are neither event-based nor person properties: a
- cohort reference alone leaves the realtime evaluator no leaf to key membership on, and nothing
- in the flag API turns on it either. Neither has anything to say, so neither pays for the run
- lookups, or the rollout flag evaluation, that saying it costs.
+ False on a team below the `serving` level, because its flags never read realtime membership
+ and its users must not see the pipeline. False for a cohort whose criteria are neither
+ event-based nor person properties: a cohort reference alone leaves the realtime evaluator no
+ leaf to key membership on, and nothing in the flag API turns on it either. Neither has anything
+ to say, so neither pays for the run lookups, or the rollout flag evaluation, that saying it costs.
- In-memory only: the allowlist is a parsed setting and the criteria come off the cohort row.
+ The criteria come off the cohort row, and the team's level from a per-process snapshot.
"""
- if cohort.pk is None or not is_realtime_cohort_team(cohort.team_id):
- return False
- return _has_event_criteria(cohort) or _has_person_criteria(cohort)
+ return bool(cohorts_with_realtime_state([cohort]))
+
+
+def cohorts_with_realtime_state(cohorts: Sequence[Cohort]) -> list[Cohort]:
+ """The cohorts ``has_realtime_state`` admits, reading each team's level once per call."""
+ saved = [cohort for cohort in cohorts if cohort.pk is not None]
+ serving_teams = {team_id for team_id in {cohort.team_id for cohort in saved} if is_serving_team(team_id)}
+ return [
+ cohort
+ for cohort in saved
+ if cohort.team_id in serving_teams and (_has_event_criteria(cohort) or _has_person_criteria(cohort))
+ ]
def resolve_realtime_readiness(cohorts: Sequence[Cohort]) -> dict[int, CohortRealtimeReadiness]:
"""One readiness per cohort id, for the cohorts ``has_realtime_state`` admits."""
by_team: dict[int, list[Cohort]] = defaultdict(list)
- for cohort in cohorts:
- if has_realtime_state(cohort):
- by_team[cohort.team_id].append(cohort)
+ for cohort in cohorts_with_realtime_state(cohorts):
+ by_team[cohort.team_id].append(cohort)
readiness: dict[int, CohortRealtimeReadiness] = {}
for team_id, team_cohorts in by_team.items():
diff --git a/products/cohorts/backend/realtime_teams.py b/products/cohorts/backend/realtime_teams.py
index 5adbc615e02f..7e4585535a9c 100644
--- a/products/cohorts/backend/realtime_teams.py
+++ b/products/cohorts/backend/realtime_teams.py
@@ -1,8 +1,30 @@
+from collections.abc import Iterable, Iterator
+from contextlib import contextmanager
+from contextvars import ContextVar
+from datetime import timedelta
+from typing import Literal
+
from django.conf import settings
+from django.db import transaction
+from django.db.models import QuerySet
+from django.utils import timezone
+
+from posthog.cache_utils import cache_for
+from posthog.models.team.team import Team
+
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
# Matches the Rust range-span cap in rust/common/types/src/cohort.rs.
_MAX_RANGE_SPAN = 100_000
+GrantLevel = Literal[RealtimeCohortLevel.PROCESSING, RealtimeCohortLevel.SERVING]
+
+_backfill_triggers_suppressed: ContextVar[bool] = ContextVar("cohort_backfill_triggers_suppressed", default=False)
+
def _matches_every_team(raw_allowlist: str) -> bool:
raw_allowlist = raw_allowlist.strip()
@@ -50,15 +72,173 @@ def realtime_allowlist_matches_every_team() -> bool:
return _matches_every_team(settings.REALTIME_COHORT_TEAM_ALLOWLIST)
-def is_realtime_cohort_team(team_id: int) -> bool:
- """Whether the realtime-cohort pipeline is scoped to ``team_id``."""
+def realtime_allowlist_matches(team_id: int) -> bool:
+ """Whether ``REALTIME_COHORT_TEAM_ALLOWLIST`` names ``team_id``, whatever its access row says."""
return _team_in_allowlist(settings.REALTIME_COHORT_TEAM_ALLOWLIST, team_id)
+# Cohort saves and every row of the cohort list read this, so it is cached per process rather than
+# queried per call. `cache_for` is off under tests, so a test reads the rows it wrote. It keeps `off`
+# rows, because `team_honors_readiness_stamps` must tell a revoked team from one that never had a row.
+@cache_for(timedelta(seconds=30))
+def _access_rows() -> dict[int, RealtimeCohortTeamAccess]:
+ return {row.team_id: row for row in RealtimeCohortTeamAccess.objects.all()}
+
+
+def _access_row(team_id: int, *, fresh: bool) -> RealtimeCohortTeamAccess | None:
+ if fresh:
+ return RealtimeCohortTeamAccess.objects.filter(team_id=team_id).first()
+ return _access_rows().get(team_id)
+
+
+def realtime_level(team_id: int, *, fresh: bool = False) -> RealtimeCohortLevel:
+ """The team's effective level: the union of the env allowlist and the team's access row.
+
+ The env allowlist grants `serving`, because before the levels existed it gated the screens users
+ see as well as the pipeline. The row comes from a per-process snapshot up to 30 seconds old,
+ unless ``fresh`` reads it as it is now.
+ """
+ if realtime_allowlist_matches(team_id):
+ return RealtimeCohortLevel.SERVING
+ row = _access_row(team_id, fresh=fresh)
+ return RealtimeCohortLevel(row.level) if row is not None else RealtimeCohortLevel.OFF
+
+
+def is_processing_team(team_id: int, *, fresh: bool = False) -> bool:
+ """Whether the realtime-cohort pipeline maintains membership for ``team_id``.
+
+ The run creators and the cohort save path pass ``fresh``. A creator reading the snapshot could
+ create a run after a revoke, and an edit reading it right after a grant could skip the supersede
+ of the team's runs, or the trigger of a new one.
+ """
+ return realtime_level(team_id, fresh=fresh) != RealtimeCohortLevel.OFF
+
+
+def is_serving_team(team_id: int) -> bool:
+ """Whether ``team_id``'s users see realtime cohort state: readiness, and audiences that wait on it."""
+ return realtime_level(team_id) == RealtimeCohortLevel.SERVING
+
+
+def team_honors_readiness_stamps(team_id: int) -> bool:
+ """Whether ``team_id``'s backfill stamps can make its cohorts usable in flags and workflows.
+
+ Not while the team's row is below `serving`. At `processing` its users must not see realtime
+ cohorts, and after a revoke no edit clears the stamps. A team with no row keeps the stamps it had
+ before the levels existed.
+ """
+ if realtime_allowlist_matches(team_id):
+ return True
+ row = _access_rows().get(team_id)
+ return row is None or row.level == RealtimeCohortLevel.SERVING
+
+
def is_cohort_backfill_trigger_team(team_id: int) -> bool:
- """Whether saving a cohort in ``team_id`` should enqueue backfill runs for it."""
- if not settings.COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST.strip():
- # No Rust service parses this setting, so it can fail closed on empty where the realtime
- # allowlist cannot: a set-but-empty value means no teams, not every team.
+ """Whether saving a cohort in ``team_id`` should enqueue backfill runs for it.
+
+ Reads the row as it is now, like the edit-time supersede. Right after a grant, an edit supersedes
+ the cohort's runs and clears its stamps, and a stale snapshot would then enqueue no new run. The
+ edit has already stored the new shape hashes, so no later save enqueues one either.
+ """
+ if _backfill_triggers_suppressed.get():
return False
- return _team_in_allowlist(settings.COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST, team_id)
+ # No Rust service parses this setting, so it can fail closed on empty where the realtime
+ # allowlist cannot: a set-but-empty value means no teams, not every team.
+ trigger_allowlist = settings.COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST
+ if trigger_allowlist.strip() and _team_in_allowlist(trigger_allowlist, team_id):
+ return True
+ return (
+ RealtimeCohortTeamAccess.objects.filter(team_id=team_id, save_triggers_backfill=True)
+ .exclude(level=RealtimeCohortLevel.OFF)
+ .exists()
+ )
+
+
+@contextmanager
+def backfill_triggers_suppressed() -> Iterator[None]:
+ """Cohort saves inside enqueue no backfill run on any team.
+
+ Setting `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST=none` does not do this, because a granted team's
+ `save_triggers_backfill` still enqueues.
+ """
+ token = _backfill_triggers_suppressed.set(True)
+ try:
+ yield
+ finally:
+ _backfill_triggers_suppressed.reset(token)
+
+
+def grant_realtime_cohort_access(
+ team_ids: Iterable[int],
+ *,
+ level: GrantLevel,
+ purpose: RealtimeCohortAccessPurpose,
+ save_triggers_backfill: bool,
+) -> list[int]:
+ """Write a row at ``level`` for every team, or for none of them. Returns the team ids.
+
+ A revoked team is refused, because nothing purges the pipeline state a revoke leaves behind,
+ and a re-grant would build on that stale state.
+ """
+ requested = sorted(set(team_ids))
+ with transaction.atomic():
+ parent_by_team = dict(Team.objects.filter(id__in=requested).values_list("id", "parent_team_id"))
+ _refuse("No team with id", [team_id for team_id in requested if team_id not in parent_by_team])
+ # Cohorts live on the root team (`RootTeamMixin`), so a row on a child environment covers nothing.
+ _refuse("Not a root team", [team_id for team_id, parent_id in parent_by_team.items() if parent_id is not None])
+ # The env list makes a team `serving` whatever its row says, and `revoke` refuses a listed
+ # team. So a row here would change nothing, and would block `revoke --purpose` for every team.
+ _refuse(
+ "On REALTIME_COHORT_TEAM_ALLOWLIST",
+ [team_id for team_id in requested if realtime_allowlist_matches(team_id)],
+ )
+ rows = {
+ row.team_id: row
+ for row in RealtimeCohortTeamAccess.objects.select_for_update().filter(team_id__in=requested)
+ }
+ _refuse(
+ "Revoked, and its pipeline state was never purged",
+ [team_id for team_id, row in rows.items() if row.level == RealtimeCohortLevel.OFF],
+ )
+ # `revoke --purpose` selects by it, so a re-grant that relabels a pilot as a load test would
+ # put the pilot under the load test's stop command.
+ _refuse(
+ f"Granted for another purpose than {purpose.value}",
+ [team_id for team_id, row in rows.items() if row.purpose != purpose],
+ )
+
+ now = timezone.now()
+ for team_id in requested:
+ row = rows.get(team_id) or RealtimeCohortTeamAccess(team_id=team_id, processing_since=now)
+ row.level = level
+ row.purpose = purpose
+ row.save_triggers_backfill = save_triggers_backfill
+ # `select_for_update` cannot lock a row that does not exist yet. A plain insert makes a
+ # concurrent first grant of the team fail on the primary key, instead of overwriting the
+ # purpose the other grant wrote.
+ row.save(force_insert=team_id not in rows)
+ return requested
+
+
+def revoke_realtime_cohort_access(rows: QuerySet[RealtimeCohortTeamAccess]) -> list[int]:
+ """Set every granted row in ``rows`` to `off`, or none of them. Returns the revoked team ids.
+
+ A team on `REALTIME_COHORT_TEAM_ALLOWLIST` is refused. The env list keeps it processing whatever
+ the row says, so an `off` row would tell every reader of the table the opposite of the truth.
+ """
+ with transaction.atomic():
+ revoked = sorted(
+ rows.exclude(level=RealtimeCohortLevel.OFF).select_for_update().values_list("team_id", flat=True)
+ )
+ _refuse(
+ "On REALTIME_COHORT_TEAM_ALLOWLIST", [team_id for team_id in revoked if realtime_allowlist_matches(team_id)]
+ )
+ now = timezone.now()
+ RealtimeCohortTeamAccess.objects.filter(team_id__in=revoked).update(
+ level=RealtimeCohortLevel.OFF, off_since=now, changed_at=now
+ )
+ return revoked
+
+
+def _refuse(reason: str, team_ids: list[int]) -> None:
+ if team_ids:
+ raise ValueError(f"{reason}: {', '.join(str(team_id) for team_id in sorted(team_ids))}")
diff --git a/products/cohorts/backend/test/test_realtime_cohort_team_access_command.py b/products/cohorts/backend/test/test_realtime_cohort_team_access_command.py
new file mode 100644
index 000000000000..9d90ca89021e
--- /dev/null
+++ b/products/cohorts/backend/test/test_realtime_cohort_team_access_command.py
@@ -0,0 +1,141 @@
+from datetime import UTC, datetime
+from io import StringIO
+
+import time_machine
+from posthog.test.base import BaseTest
+
+from django.core.management import call_command
+from django.core.management.base import CommandError
+from django.test import override_settings
+
+from parameterized import parameterized
+
+from posthog.models.team.team import Team
+
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
+
+GRANTED_AT = datetime(2026, 10, 28, 12, 0, tzinfo=UTC)
+PROMOTED_AT = datetime(2026, 10, 28, 13, 0, tzinfo=UTC)
+REVOKED_AT = datetime(2026, 10, 28, 14, 0, tzinfo=UTC)
+
+
+@override_settings(REALTIME_COHORT_TEAM_ALLOWLIST="none")
+class TestRealtimeCohortTeamAccessCommand(BaseTest):
+ def _access(self, team_id: int) -> RealtimeCohortTeamAccess:
+ return RealtimeCohortTeamAccess.objects.get(team_id=team_id)
+
+ def _grant(
+ self, *team_ids: int, purpose: str = "load_test", level: str = "processing", triggers: bool = False
+ ) -> None:
+ teams = [str(team_id) for team_id in team_ids]
+ call_command(
+ "realtime_cohort_team_access",
+ "grant",
+ "--teams",
+ *teams,
+ "--purpose",
+ purpose,
+ "--level",
+ level,
+ "--save-triggers-backfill" if triggers else "--no-save-triggers-backfill",
+ )
+
+ def test_grant_promote_revoke_then_refuse_a_regrant(self) -> None:
+ with time_machine.travel(GRANTED_AT, tick=False):
+ self._grant(self.team.pk)
+ with time_machine.travel(PROMOTED_AT, tick=False):
+ self._grant(self.team.pk, level="serving", triggers=True)
+
+ promoted = self._access(self.team.pk)
+ assert (promoted.level, promoted.save_triggers_backfill, promoted.processing_since, promoted.off_since) == (
+ RealtimeCohortLevel.SERVING,
+ True,
+ GRANTED_AT,
+ None,
+ )
+
+ with time_machine.travel(REVOKED_AT, tick=False):
+ call_command("realtime_cohort_team_access", "revoke", "--teams", str(self.team.pk))
+
+ revoked = self._access(self.team.pk)
+ assert (revoked.level, revoked.processing_since, revoked.off_since) == (
+ RealtimeCohortLevel.OFF,
+ GRANTED_AT,
+ REVOKED_AT,
+ )
+
+ with self.assertRaisesMessage(
+ CommandError, f"Revoked, and its pipeline state was never purged: {self.team.pk}"
+ ):
+ self._grant(self.team.pk)
+ assert self._access(self.team.pk).level == RealtimeCohortLevel.OFF
+
+ out = StringIO()
+ call_command("realtime_cohort_team_access", "list", stdout=out)
+ assert f"{self.team.pk}\toff\tload_test\tTrue" in out.getvalue()
+
+ @parameterized.expand(
+ [
+ ("unknown_team", "No team with id"),
+ ("child_environment", "Not a root team"),
+ # The load test's stop command revokes by purpose, so it must never reach a pilot.
+ ("granted_pilot", "Granted for another purpose than load_test"),
+ # `revoke` refuses a listed team, so its row would block the stop command for every team.
+ ("env_listed", "On REALTIME_COHORT_TEAM_ALLOWLIST"),
+ ]
+ )
+ def test_grant_refuses_every_team_when_one_is_refused(self, kind: str, message: str) -> None:
+ allowlist = "none"
+ if kind == "unknown_team":
+ refused_id = 999_999_999
+ elif kind == "child_environment":
+ refused_id = Team.objects.create(organization=self.organization, parent_team=self.team).pk
+ elif kind == "env_listed":
+ refused_id = Team.objects.create(organization=self.organization).pk
+ allowlist = str(refused_id)
+ else:
+ refused_id = Team.objects.create(organization=self.organization).pk
+ RealtimeCohortTeamAccess.objects.create(
+ team_id=refused_id, level=RealtimeCohortLevel.SERVING, purpose=RealtimeCohortAccessPurpose.PILOT
+ )
+
+ with (
+ override_settings(REALTIME_COHORT_TEAM_ALLOWLIST=allowlist),
+ self.assertRaisesMessage(CommandError, f"{message}: {refused_id}"),
+ ):
+ self._grant(self.team.pk, refused_id)
+
+ assert not RealtimeCohortTeamAccess.objects.filter(team=self.team).exists()
+
+ def test_revoke_refuses_a_team_the_env_list_keeps_on(self) -> None:
+ RealtimeCohortTeamAccess.objects.create(
+ team=self.team, level=RealtimeCohortLevel.SERVING, purpose=RealtimeCohortAccessPurpose.PILOT
+ )
+
+ with (
+ override_settings(REALTIME_COHORT_TEAM_ALLOWLIST=str(self.team.pk)),
+ self.assertRaisesMessage(CommandError, f"On REALTIME_COHORT_TEAM_ALLOWLIST: {self.team.pk}"),
+ ):
+ call_command("realtime_cohort_team_access", "revoke", "--purpose", "pilot")
+
+ assert self._access(self.team.pk).level == RealtimeCohortLevel.SERVING
+
+ def test_revoke_by_purpose_leaves_other_purposes_on(self) -> None:
+ pilot = Team.objects.create(organization=self.organization)
+ RealtimeCohortTeamAccess.objects.create(
+ team=self.team, level=RealtimeCohortLevel.PROCESSING, purpose=RealtimeCohortAccessPurpose.LOAD_TEST
+ )
+ RealtimeCohortTeamAccess.objects.create(
+ team=pilot, level=RealtimeCohortLevel.SERVING, purpose=RealtimeCohortAccessPurpose.PILOT
+ )
+
+ call_command("realtime_cohort_team_access", "revoke", "--purpose", "load_test")
+
+ assert (self._access(self.team.pk).level, self._access(pilot.pk).level) == (
+ RealtimeCohortLevel.OFF,
+ RealtimeCohortLevel.SERVING,
+ )
diff --git a/products/cohorts/backend/test/test_realtime_state.py b/products/cohorts/backend/test/test_realtime_state.py
index 617487add72d..19ef1380e4a3 100644
--- a/products/cohorts/backend/test/test_realtime_state.py
+++ b/products/cohorts/backend/test/test_realtime_state.py
@@ -22,6 +22,11 @@ from products.cohorts.backend.models.backfill import (
)
from products.cohorts.backend.models.cohort import Cohort, CohortType
from products.cohorts.backend.models.dependencies import cohort_backfill_pending_key
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
from products.cohorts.backend.realtime_state import (
CohortHistoryBuildPhase,
CohortRealtimeState,
@@ -402,12 +407,24 @@ class TestRealtimeReadiness(BaseTest):
assert self._state(cohort) == CohortRealtimeState.DAILY
+ @parameterized.expand(
+ [
+ ("off_the_pipeline", None),
+ # A load-test team runs the pipeline, but the cohort list must not show its users a state.
+ ("processing_only", RealtimeCohortLevel.PROCESSING),
+ ]
+ )
@override_settings(REALTIME_COHORT_TEAM_ALLOWLIST="none")
- def test_team_without_the_pipeline_has_no_realtime_state(self) -> None:
+ def test_team_not_serving_has_no_realtime_state(self, _name: str, access_level: RealtimeCohortLevel | None) -> None:
# Cohort type is computed for every team, so without this gate every cohort in the fleet
# would claim a realtime state the team's flags can never read.
+ if access_level is not None:
+ RealtimeCohortTeamAccess.objects.create(
+ team=self.team, level=access_level, purpose=RealtimeCohortAccessPurpose.LOAD_TEST
+ )
cohort = self._cohort()
- with self.assertNumQueries(0):
+ # The one query reads the team's access row, which production serves from a per-process cache.
+ with self.assertNumQueries(1):
assert resolve_realtime_readiness([cohort]) == {}
def test_resolving_a_page_costs_the_same_queries_as_one_cohort(self) -> None:
diff --git a/products/cohorts/backend/test/test_realtime_teams.py b/products/cohorts/backend/test/test_realtime_teams.py
index 05a6601593c3..e8c6e9ae86cb 100644
--- a/products/cohorts/backend/test/test_realtime_teams.py
+++ b/products/cohorts/backend/test/test_realtime_teams.py
@@ -1,6 +1,7 @@
import os
import importlib
+from posthog.test.base import BaseTest
from unittest import mock
from django.test import SimpleTestCase, override_settings
@@ -9,10 +10,21 @@ from parameterized import parameterized
import posthog.settings.cohorts as cohorts_settings
-from products.cohorts.backend.realtime_teams import is_cohort_backfill_trigger_team, is_realtime_cohort_team
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
+from products.cohorts.backend.realtime_teams import (
+ backfill_triggers_suppressed,
+ is_cohort_backfill_trigger_team,
+ is_processing_team,
+ is_serving_team,
+ realtime_allowlist_matches,
+)
-class TestIsRealtimeCohortTeam(SimpleTestCase):
+class TestRealtimeAllowlistMatches(SimpleTestCase):
@parameterized.expand(
[
("all_keyword", "all", 999, True),
@@ -41,10 +53,43 @@ class TestIsRealtimeCohortTeam(SimpleTestCase):
)
def test_membership_matches_rust_grammar(self, _name: str, allowlist: str, team_id: int, expected: bool) -> None:
with override_settings(REALTIME_COHORT_TEAM_ALLOWLIST=allowlist):
- self.assertEqual(is_realtime_cohort_team(team_id), expected)
+ self.assertEqual(realtime_allowlist_matches(team_id), expected)
-class TestIsCohortBackfillTriggerTeam(SimpleTestCase):
+class TestRealtimeLevel(BaseTest):
+ @parameterized.expand(
+ [
+ ("no_row", "none", None, (False, False)),
+ ("revoked_row", "none", RealtimeCohortLevel.OFF, (False, False)),
+ # A load-test team runs the pipeline while its users see none of it.
+ ("processing_row", "none", RealtimeCohortLevel.PROCESSING, (True, False)),
+ ("serving_row", "none", RealtimeCohortLevel.SERVING, (True, True)),
+ # The env list gated the screens users see before the levels existed, so it keeps doing so.
+ ("env_listed_without_row", "team", None, (True, True)),
+ ("env_listed_over_processing_row", "team", RealtimeCohortLevel.PROCESSING, (True, True)),
+ ]
+ )
+ def test_level_is_the_union_of_env_list_and_row(
+ self, _name: str, allowlist: str, level: RealtimeCohortLevel | None, expected: tuple[bool, bool]
+ ) -> None:
+ if level is not None:
+ RealtimeCohortTeamAccess.objects.create(
+ team=self.team, level=level, purpose=RealtimeCohortAccessPurpose.PILOT
+ )
+
+ with override_settings(REALTIME_COHORT_TEAM_ALLOWLIST=str(self.team.pk) if allowlist == "team" else allowlist):
+ processing, serving = expected
+ self.assertEqual(
+ (
+ is_processing_team(self.team.pk),
+ is_processing_team(self.team.pk, fresh=True),
+ is_serving_team(self.team.pk),
+ ),
+ (processing, processing, serving),
+ )
+
+
+class TestIsCohortBackfillTriggerTeam(BaseTest):
@parameterized.expand(
[
# Only Python parses this setting, so unlike the realtime allowlist it fails closed on a
@@ -60,6 +105,39 @@ class TestIsCohortBackfillTriggerTeam(SimpleTestCase):
with override_settings(COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST=allowlist):
self.assertEqual(is_cohort_backfill_trigger_team(team_id), expected)
+ @parameterized.expand(
+ [
+ ("granted_with_triggers", RealtimeCohortLevel.PROCESSING, True, True),
+ ("granted_without_triggers", RealtimeCohortLevel.PROCESSING, False, False),
+ ("revoked_with_triggers", RealtimeCohortLevel.OFF, True, False),
+ ]
+ )
+ @override_settings(COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST="none")
+ def test_row_triggers_only_while_granted(
+ self, _name: str, level: RealtimeCohortLevel, save_triggers_backfill: bool, expected: bool
+ ) -> None:
+ RealtimeCohortTeamAccess.objects.create(
+ team=self.team,
+ level=level,
+ purpose=RealtimeCohortAccessPurpose.LOAD_TEST,
+ save_triggers_backfill=save_triggers_backfill,
+ )
+
+ self.assertEqual(is_cohort_backfill_trigger_team(self.team.pk), expected)
+
+ @override_settings(COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST="all")
+ def test_suppression_overrides_every_trigger_source(self) -> None:
+ RealtimeCohortTeamAccess.objects.create(
+ team=self.team,
+ level=RealtimeCohortLevel.PROCESSING,
+ purpose=RealtimeCohortAccessPurpose.LOAD_TEST,
+ save_triggers_backfill=True,
+ )
+
+ with backfill_triggers_suppressed():
+ self.assertFalse(is_cohort_backfill_trigger_team(self.team.pk))
+ self.assertTrue(is_cohort_backfill_trigger_team(self.team.pk))
+
class TestRealtimeCohortAllowlistSetting(SimpleTestCase):
def _reload_allowlist(self) -> str:
diff --git a/products/feature_flags/backend/api/test/__snapshots__/test_feature_flag.ambr b/products/feature_flags/backend/api/test/__snapshots__/test_feature_flag.ambr
index 5a07ab73fad4..5b61429afc84 100644
--- a/products/feature_flags/backend/api/test/__snapshots__/test_feature_flag.ambr
+++ b/products/feature_flags/backend/api/test/__snapshots__/test_feature_flag.ambr
@@ -1495,6 +1495,18 @@
'''
# ---
# name: TestFeatureFlag.test_creating_static_cohort.26
+ '''
+ SELECT "cohort_realtime_team_access"."team_id",
+ "cohort_realtime_team_access"."level",
+ "cohort_realtime_team_access"."purpose",
+ "cohort_realtime_team_access"."save_triggers_backfill",
+ "cohort_realtime_team_access"."processing_since",
+ "cohort_realtime_team_access"."off_since",
+ "cohort_realtime_team_access"."changed_at"
+ FROM "cohort_realtime_team_access"
+ '''
+# ---
+# name: TestFeatureFlag.test_creating_static_cohort.27
'''
SELECT "posthog_team"."id",
"posthog_team"."uuid",
@@ -1592,7 +1604,7 @@
LIMIT 21
'''
# ---
-# name: TestFeatureFlag.test_creating_static_cohort.27
+# name: TestFeatureFlag.test_creating_static_cohort.28
'''
SELECT "posthog_user"."id",
"posthog_user"."password",
@@ -1639,7 +1651,7 @@
LIMIT 21
'''
# ---
-# name: TestFeatureFlag.test_creating_static_cohort.28
+# name: TestFeatureFlag.test_creating_static_cohort.29
'''
SELECT "posthog_cohortcalculationhistory"."id",
"posthog_cohortcalculationhistory"."team_id",
@@ -1660,7 +1672,35 @@
LIMIT 1
'''
# ---
-# name: TestFeatureFlag.test_creating_static_cohort.29
+# name: TestFeatureFlag.test_creating_static_cohort.3
+ '''
+ SELECT "posthog_project"."updated_at",
+ "posthog_project"."id",
+ "posthog_project"."organization_id",
+ "posthog_project"."name",
+ "posthog_project"."logo_media_id",
+ "posthog_project"."created_at",
+ "posthog_project"."product_description",
+ "posthog_project"."is_pending_deletion",
+ "posthog_project"."deletion_scheduled_at"
+ FROM "posthog_project"
+ WHERE "posthog_project"."id" = 99999
+ LIMIT 21
+ '''
+# ---
+# name: TestFeatureFlag.test_creating_static_cohort.30
+ '''
+ SELECT "cohort_realtime_team_access"."team_id",
+ "cohort_realtime_team_access"."level",
+ "cohort_realtime_team_access"."purpose",
+ "cohort_realtime_team_access"."save_triggers_backfill",
+ "cohort_realtime_team_access"."processing_since",
+ "cohort_realtime_team_access"."off_since",
+ "cohort_realtime_team_access"."changed_at"
+ FROM "cohort_realtime_team_access"
+ '''
+# ---
+# name: TestFeatureFlag.test_creating_static_cohort.31
'''
SELECT "posthog_experiment"."id",
"posthog_experiment"."name",
@@ -1704,23 +1744,7 @@
WHERE "posthog_experiment"."exposure_cohort_id" = 99999
'''
# ---
-# name: TestFeatureFlag.test_creating_static_cohort.3
- '''
- SELECT "posthog_project"."updated_at",
- "posthog_project"."id",
- "posthog_project"."organization_id",
- "posthog_project"."name",
- "posthog_project"."logo_media_id",
- "posthog_project"."created_at",
- "posthog_project"."product_description",
- "posthog_project"."is_pending_deletion",
- "posthog_project"."deletion_scheduled_at"
- FROM "posthog_project"
- WHERE "posthog_project"."id" = 99999
- LIMIT 21
- '''
-# ---
-# name: TestFeatureFlag.test_creating_static_cohort.30
+# name: TestFeatureFlag.test_creating_static_cohort.32
'''
/* user_id:0 celery:posthog.tasks.calculate_cohort.insert_cohort_from_feature_flag */
SELECT person_id
@@ -1731,7 +1755,7 @@
GROUP BY person_id
'''
# ---
-# name: TestFeatureFlag.test_creating_static_cohort.31
+# name: TestFeatureFlag.test_creating_static_cohort.33
'''
/* user_id:0 request:_snapshot_ */
SELECT persons.id AS id
diff --git a/products/workflows/backend/services/batch_audience_cohorts.py b/products/workflows/backend/services/batch_audience_cohorts.py
index 969461560555..a26cf9b38b53 100644
--- a/products/workflows/backend/services/batch_audience_cohorts.py
+++ b/products/workflows/backend/services/batch_audience_cohorts.py
@@ -1,7 +1,7 @@
from products.cohorts.backend.models.cohort import Cohort, CohortType
from products.cohorts.backend.models.util import get_all_cohort_dependencies
from products.cohorts.backend.realtime_state import CohortRealtimeState, resolve_realtime_readiness
-from products.cohorts.backend.realtime_teams import is_realtime_cohort_team
+from products.cohorts.backend.realtime_teams import is_serving_team
from products.workflows.backend.facade.contracts import RealtimeCohortStatus
@@ -86,15 +86,18 @@ def realtime_audience_cohort(project_id: int, filters: object) -> Cohort | None:
def realtime_cohort_can_finish(cohort: Cohort) -> bool:
- """Whether the realtime pipeline runs for this cohort's team, so the cohort can become ready at all."""
- return is_realtime_cohort_team(cohort.team_id)
+ """Whether this cohort's team is at the `serving` level, so the cohort can become ready at all.
+
+ A `processing` team runs the pipeline, but its users must not see a realtime audience come alive.
+ """
+ return is_serving_team(cohort.team_id)
def realtime_cohort_is_ready(cohort: Cohort) -> bool:
"""Whether the realtime pipeline holds this cohort's full membership and keeps it current.
- The backfill stamps stay set when a team leaves the realtime allowlist, but the pipeline then
- stops updating the membership, so the team has to be on the allowlist too.
+ The backfill stamps stay set when a team stops serving, but the pipeline may then stop updating
+ the membership, so the team has to be serving too.
"""
return cohort.is_flag_compatible and realtime_cohort_can_finish(cohort)
@@ -108,8 +111,8 @@ def realtime_cohort_status(cohort: Cohort) -> RealtimeCohortStatus:
readiness = resolve_realtime_readiness([cohort]).get(cohort.pk)
if realtime_cohort_is_ready(cohort):
state: str = CohortRealtimeState.READY
- elif not is_realtime_cohort_team(cohort.team_id):
- # The pipeline doesn't run for this team, so the cohort never finishes calculating.
+ elif not is_serving_team(cohort.team_id):
+ # The pipeline doesn't serve this team, so the cohort never finishes calculating.
state = CohortRealtimeState.NEEDS_ATTENTION
elif readiness is not None and readiness.state in _REPORTED_PENDING_STATES:
state = readiness.state
diff --git a/products/workflows/backend/test/test_batch_audience.py b/products/workflows/backend/test/test_batch_audience.py
index 6f9fba4036b3..b7a9066c415b 100644
--- a/products/workflows/backend/test/test_batch_audience.py
+++ b/products/workflows/backend/test/test_batch_audience.py
@@ -9,6 +9,11 @@ from posthog.redis import get_client as get_redis_client
from products.cohorts.backend.models.backfill import CohortBackfillKind, CohortBackfillTrigger
from products.cohorts.backend.models.cohort import Cohort, CohortType
from products.cohorts.backend.models.dependencies import cohort_backfill_pending_key
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
from products.feature_flags.backend.user_blast_radius import PERSON_BATCH_SIZE, get_user_blast_radius_persons
from products.workflows.backend.services.batch_audience import audience_page_size, get_batch_audience_person_ids
from products.workflows.backend.services.blast_radius import get_audience_size
@@ -92,12 +97,21 @@ class TestBatchAudience(ClickhouseTestMixin, BaseTest):
("nothing_building", "behavioral", "all", False, "needs_attention"),
# The pipeline doesn't run for the team, so the cohort never finishes calculating.
("team_off_realtime_allowlist", "behavioral", "none", True, "needs_attention"),
+ # A load-test team runs the pipeline, but its users must not see the audience start building.
+ (
+ "team_granted_processing_only",
+ "behavioral",
+ "none",
+ True,
+ "needs_attention",
+ RealtimeCohortLevel.PROCESSING,
+ ),
# The cohorts product reports this one as person_properties, which the response doesn't declare.
("person_property_cohort_on_a_behavioral_cohort", "person_on_behavioral", "all", False, "building"),
]
)
def test_realtime_cohort_audience_size_estimates_from_the_cohort_count(
- self, _name, shape, allowlist, build_queued, expected_state
+ self, _name, shape, allowlist, build_queued, expected_state, access_level=None
):
# The send reads realtime membership from a Postgres database Django can't reach, so the
# preview estimates from the cohort's periodic count and reports no email dedupe.
@@ -129,6 +143,10 @@ class TestBatchAudience(ClickhouseTestMixin, BaseTest):
count=2,
filters={"properties": {"type": "AND", "values": leaves}},
)
+ if access_level is not None:
+ RealtimeCohortTeamAccess.objects.create(
+ team=self.team, level=access_level, purpose=RealtimeCohortAccessPurpose.LOAD_TEST
+ )
if build_queued:
get_redis_client().set(
cohort_backfill_pending_key(cohort.pk, CohortBackfillKind.BEHAVIORAL),
diff --git a/products/workflows/backend/tests/api/test_hog_flow.py b/products/workflows/backend/tests/api/test_hog_flow.py
index be670dfc71b8..ac4a0f19adc1 100644
--- a/products/workflows/backend/tests/api/test_hog_flow.py
+++ b/products/workflows/backend/tests/api/test_hog_flow.py
@@ -50,6 +50,11 @@ from products.cohorts.backend.models.backfill import (
CohortBackfillTrigger,
)
from products.cohorts.backend.models.cohort import Cohort, CohortType
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
from products.feature_flags.backend.user_blast_radius import BlastRadiusResult
from products.messaging.backend.facade.categories import update_category
from products.messaging.backend.facade.testing import create_message_category_for_test
@@ -4548,6 +4553,8 @@ class TestHogFlowAPI(APIBaseTest):
("realtime_reference_only", "batch", "reference_only", None, True, 400, "can't evaluate"),
# The realtime pipeline doesn't run for the team, so the cohort never becomes ready.
("team_off_realtime_pipeline", "batch", "off_pipeline", None, True, 400, "would never start"),
+ # A load-test team runs the pipeline, but its users must not see a realtime audience start.
+ ("team_granted_processing_only", "batch", "processing_only", None, True, 400, "would never start"),
]
)
@override_settings(REALTIME_COHORT_TEAM_ALLOWLIST="all")
@@ -4594,7 +4601,12 @@ class TestHogFlowAPI(APIBaseTest):
],
}
- with override_settings(REALTIME_COHORT_TEAM_ALLOWLIST="none" if cohort_kind == "off_pipeline" else "all"):
+ if cohort_kind == "processing_only":
+ RealtimeCohortTeamAccess.objects.create(
+ team=self.team, level=RealtimeCohortLevel.PROCESSING, purpose=RealtimeCohortAccessPurpose.LOAD_TEST
+ )
+ off_env_list = cohort_kind in ("off_pipeline", "processing_only")
+ with override_settings(REALTIME_COHORT_TEAM_ALLOWLIST="none" if off_env_list else "all"):
response = self.client.post(f"/api/projects/{self.team.id}/hog_flows", hog_flow)
assert response.status_code == expected_status, response.json()