Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(cohorts): enable a team's realtime cohorts in one command (#129804)

Matheus Batista110de6352051
diff --git a/docs/internal/realtime-cohorts/backfill-coordination.md b/docs/internal/realtime-cohorts/backfill-coordination.md
index 3467bdf0a259..42ca1ab40385 100644
--- a/docs/internal/realtime-cohorts/backfill-coordination.md
+++ b/docs/internal/realtime-cohorts/backfill-coordination.md
@@ -165,14 +165,14 @@ The membership consumer does count those markers toward its sweep, see [membersh
Several independent gates decide whether anything happens.
Missing one of them fails quietly, so they are worth knowing.
-| Gate | Effect |
-| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `REALTIME_COHORT_TEAM_ALLOWLIST` | Teams whose cohorts get hash maintenance, stamp invalidation, supersession and any run at all. Unset means no teams in Django, but the shuffler, processor and seeder default to team 2, so set it explicitly on every service. Set but empty means **all** teams. Django puts a listed team at the `serving` level, and adds the teams that have a team access row. The shufflers, the processor and the seeder add those teams too, the seeder 120 seconds after the grant. Never list a team granted `processing` here, because that makes it `serving` |
-| Team access rows | `cohort_realtime_team_access`, written only by `realtime_cohort_team_access grant\|revoke\|list`. Django, the shufflers, the processor and the seeder read it as a union with their `REALTIME_COHORT_TEAM_ALLOWLIST`. Node ingestion does not read the table, and emits person merges only for the teams in its `PERSON_MERGE_EVENTS_TEAM_ALLOWLIST`. Set that list to `*`, because the processor drops the merges of teams it does not process. The seeder works on a granted team's runs only once `processing_since` is 120 seconds old. After a revoke the seeder stops working on the team's runs, unless its own env list names the team, and each run keeps its cohort's run slot. The processor drops a revoked team from its catalog, and stage 2 GC then deletes the team's state without emitting `left`, so the team's `cohort_membership` rows stay `in_cohort=true`. A shuffler whose env list does not name the team stops forwarding its events. A row at `processing` gets the Django behavior above, but the team's users see no realtime state: no readiness, no realtime batch audience, and no backfilled cohort in flags or workflows, even one stamped before the grant, so a workflow that already uses one fails its next save. A row at `serving` also shows them, and the flags service routes the team's cohorts to the membership table once its team metadata cache carries the level. `revoke` sets `off`, which hides the stamps the same way, and a revoked team cannot be granted again. `grant` and `revoke` refuse a team on the env list |
-| `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST` | Teams whose saves create runs automatically. Unset or empty means **no** teams. A granted row with `save_triggers_backfill` adds its team, so `resave_cohorts --no-backfill-triggers`, not this setting, is what stops a resave from queueing runs |
-| Operator attestations | Settings an operator sets to declare that prerequisites hold. They are declarations, not live checks. Behavioral runs need `BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED` and `BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED`. Person runs also need the person TTL and sizing attestations and a positive seed-bytes budget |
-| Person sizing budget | A cohort-scoped person run is refused when the team's active person-run estimates plus its own would exceed the budget. The check reads the active estimates before the run exists and reserves nothing, so two cohorts sized at the same time can together exceed the budget. A team run checks only its own estimate. Hitting the size estimate's scan cap, or the cap on pinned conditions, also refuses |
-| Behavioral scan limit | `create_cohort_backfill_run` refuses a behavioral team run when its pinned event names had more events than `BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY` on the busiest of the last 7 complete UTC days. `--max-scan-events-per-day` overrides it for one invocation, and 0 skips the count and the check. The save path does not check it |
+| Gate | Effect |
+| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `REALTIME_COHORT_TEAM_ALLOWLIST` | Teams whose cohorts get hash maintenance, stamp invalidation, supersession and any run at all. Unset means no teams in Django, but the shuffler, processor and seeder default to team 2, so set it explicitly on every service. Set but empty means **all** teams. Django puts a listed team at the `serving` level, and adds the teams that have a team access row. The shufflers, the processor and the seeder add those teams too, the seeder 120 seconds after the grant. Never list a team granted `processing` here, because that makes it `serving` |
+| Team access rows | `cohort_realtime_team_access`, written only by `realtime_cohort_team_access enable\|grant\|revoke`. Django, the shufflers, the processor and the seeder read it as a union with their `REALTIME_COHORT_TEAM_ALLOWLIST`. Node ingestion does not read the table, and emits person merges only for the teams in its `PERSON_MERGE_EVENTS_TEAM_ALLOWLIST`. Set that list to `*`, because the processor drops the merges of teams it does not process. The seeder works on a granted team's runs only once `processing_since` is 120 seconds old. After a revoke the seeder stops working on the team's runs, unless its own env list names the team, and each run keeps its cohort's run slot. The processor drops a revoked team from its catalog, and stage 2 GC then deletes the team's state without emitting `left`, so the team's `cohort_membership` rows stay `in_cohort=true`. A shuffler whose env list does not name the team stops forwarding its events. A row at `processing` gets the Django behavior above, but the team's users see no realtime state: no readiness, no realtime batch audience, and no backfilled cohort in flags or workflows, even one stamped before the grant, so a workflow that already uses one fails its next save. A row at `serving` also shows them, and the flags service routes the team's cohorts to the membership table once its team metadata cache carries the level. `revoke` sets `off`, which hides the stamps the same way, and a revoked team cannot be granted again. `grant` and `revoke` refuse a team on the env list, and their `--purpose` does not take `beta`. `enable` is how a beta team gets realtime cohorts, and the only way to a beta row. Before it writes anything, it leaves a team already at `serving` as it is. It refuses what `grant` refuses, every team while an operator attestation is missing, and a team with no row whose cohorts carry readiness stamps, because those stamps come from a time the env list named the team, and the pipeline no longer maintains that membership. Then it resaves the team's cohorts, fills their shape hashes, grants `processing` with `save_triggers_backfill`, and creates one team run per kind. It grants `serving` only when neither run was refused, so a refused team stays at `processing`. Running `enable` again after resolving the refusal keeps each team run created since the grant that is waiting to seed, seeding, or has stamped readiness, and creates only the missing one |
+| `COHORT_BACKFILL_TRIGGER_TEAM_ALLOWLIST` | Teams whose saves create runs automatically. Unset or empty means **no** teams. A granted row with `save_triggers_backfill` adds its team, so `resave_cohorts --no-backfill-triggers`, not this setting, is what stops a resave from queueing runs |
+| Operator attestations | Settings an operator sets to declare that prerequisites hold. They are declarations, not live checks. Behavioral runs need `BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED` and `BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED`. Person runs also need the person TTL and sizing attestations and a positive seed-bytes budget |
+| Person sizing budget | A cohort-scoped person run is refused when the team's active person-run estimates plus its own would exceed the budget. The check reads the active estimates before the run exists and reserves nothing, so two cohorts sized at the same time can together exceed the budget. A team run checks only its own estimate. Hitting the size estimate's scan cap, or the cap on pinned conditions, also refuses |
+| Behavioral scan limit | `create_cohort_backfill_run` and `realtime_cohort_team_access enable` refuse a behavioral team run when its pinned event names had more events than `BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY` on the busiest of the last 7 complete UTC days. `--max-scan-events-per-day` overrides it for one `create_cohort_backfill_run` invocation, and 0 skips the count and the check. `create_team_backfill_run` itself and the save path do not check it |
The Rust services have their own switches, and Django cannot see them.
The processor must hold a team before the seeder works on the team's runs.
diff --git a/docs/internal/realtime-cohorts/membership-output-and-readers.md b/docs/internal/realtime-cohorts/membership-output-and-readers.md
index 5b949d364b0e..932ecd41e694 100644
--- a/docs/internal/realtime-cohorts/membership-output-and-readers.md
+++ b/docs/internal/realtime-cohorts/membership-output-and-readers.md
@@ -213,12 +213,13 @@ A cache entry without the level reads as `off`.
`condition_type` is derived by Django on save.
A cohort whose `condition_type` is empty is never routed, and resaving it fills the column.
+A behavioral run refuses such a cohort, so it never reads as ready.
Every other non-static cohort is evaluated dynamically from the person's properties, where a behavioral condition can never match.
The team metadata cache that the flags service reads carries the team's access row level as `realtime_cohort_level`, or `off` when the team has no row.
It carries the row's level, not Django's effective level, because `REALTIME_COHORT_TEAM_ALLOWLIST` can name every team.
-`realtime_cohort_team_access grant` and `revoke` queue a refresh of the team's entry after they commit.
+`realtime_cohort_team_access enable`, `grant` and `revoke` queue a refresh of the team's entry after they commit.
The hourly team metadata verification repairs an entry whose level still differs from the row.
### How it reads
diff --git a/posthog/cohort_resave.py b/posthog/cohort_resave.py
new file mode 100644
index 000000000000..f95dd6ca18f6
--- /dev/null
+++ b/posthog/cohort_resave.py
@@ -0,0 +1,348 @@
+"""The per-team cohort resave, shared by `resave_cohorts` and realtime cohort enablement.
+
+It lives in core because it coalesces flags cache rebuilds, and the cohorts product cannot depend on
+`products.feature_flags`.
+"""
+
+from __future__ import annotations
+
+import signal
+import threading
+from collections.abc import Iterator, Sequence
+from contextlib import AbstractContextManager, contextmanager, nullcontext
+from types import FrameType
+from typing import Any
+
+import structlog
+
+from posthog.api.cohort import validate_filters_and_compute_realtime_support
+from posthog.dataclasses import frozen
+from posthog.models.team.team import Team
+
+from products.cohorts.backend.models.cohort import Cohort, CohortOrEmpty, CohortType
+from products.cohorts.backend.models.util import (
+ get_all_cohort_dependencies,
+ legacy_groups_to_filters,
+ sort_cohorts_topologically,
+)
+from products.feature_flags.backend.flags_cache import coalesced_cohort_flags_cache_rebuilds
+
+logger = structlog.get_logger(__name__)
+
+# The cohort types the realtime membership gate routes, so a null condition_type changes how they
+# evaluate. See `uses_realtime_membership` in rust/feature-flags/src/cohorts/cohort_models.rs.
+REALTIME_GATED_COHORT_TYPES = (CohortType.REALTIME, CohortType.BEHAVIORAL)
+
+# The catchable signals that end a run without unwinding it. SIGTERM reaches the command when it
+# is the container's main process or is killed by hand. A run in a toolbox `kubectl exec` shell
+# never gets SIGTERM, because a pod stop signals only the main process, but a hangup of that
+# shell's terminal ends it with SIGHUP.
+UNWIND_SIGNALS = (signal.SIGTERM, signal.SIGHUP)
+
+
+@contextmanager
+def termination_unwinds() -> Iterator[bool]:
+ """Raise on SIGTERM and SIGHUP instead of dying in place, when they are this process's to take.
+
+ Either signal otherwise kills the process between a team's cohort saves committing and its
+ coalesced cache rebuild being enqueued, and nothing sweeps that back: the service-cache
+ verifier does not compare cohorts, and a rerun finds nothing left to change. Unwinding lets
+ the coalescing block dispatch the teams it has already recorded. `KeyboardInterrupt` because
+ the per-cohort `except Exception` must not swallow it. SIGKILL cannot be caught, so an
+ expired toolbox claim or an OOM kill still loses the rebuild of the team in progress.
+
+ Yields whether the unwind is in place. The admin view runs the command inside a web worker,
+ which owns its own shutdown, so the caller must not coalesce rebuilds it cannot flush there.
+ """
+ if threading.current_thread() is not threading.main_thread():
+ # Only the main thread may install a handler, and a server that hands sync views to a
+ # threadpool runs the admin view off it.
+ yield False
+ return
+
+ dispositions = {signum: signal.getsignal(signum) for signum in UNWIND_SIGNALS}
+ if any(disposition not in (signal.SIG_DFL, signal.SIG_IGN) for disposition in dispositions.values()):
+ # Something else owns one of them, a web worker serving the admin view for one. Taking it
+ # over would replace that shutdown path, and a handler installed outside Python reads
+ # back as `None`, which cannot be restored afterwards.
+ yield False
+ return
+
+ def raise_interrupt(signum: int, frame: FrameType | None) -> None:
+ raise KeyboardInterrupt(f"received {signal.Signals(signum).name}")
+
+ # An ignored signal cannot end the run, so it stays ignored. `nohup` ignores SIGHUP, and a
+ # run started under it must survive the hangup it asked to survive.
+ taken = [signum for signum, disposition in dispositions.items() if disposition is signal.SIG_DFL]
+ for signum in taken:
+ signal.signal(signum, raise_interrupt)
+ try:
+ yield True
+ finally:
+ for signum in taken:
+ signal.signal(signum, signal.SIG_DFL)
+
+
+@frozen
+class CohortResaveStats:
+ total: int
+ changed: int
+ errors: int
+ validation_errors: int
+ prospective_realtime: int
+ # The team's cohorts were resaved but its cache rebuild never reached the broker, so the
+ # team keeps a stale flags cache.
+ rebuild_dispatch_failed: bool
+
+
+@frozen
+class CohortResaveOutcome:
+ changed: bool
+ validation_failed: bool
+ prospective_realtime: bool
+
+
+UNTOUCHED_COHORT = CohortResaveOutcome(changed=False, validation_failed=False, prospective_realtime=False)
+
+
+def unclassified_cohort_ids(team_ids: Sequence[int] | None) -> list[int]:
+ """Cohorts the realtime gate routes that still have a null `condition_type`, read back from
+ the database.
+
+ Reading persisted state covers every way a cohort ends up unclassified: a save that raised, a
+ filter set the validator rejected, and a well-formed filter group with no leaf conditions,
+ which the resave classifies as realtime and leaves with a null `condition_type`. It only
+ means anything after a run that persisted, so the caller skips it on a dry run.
+ """
+ qs = Cohort.objects.filter(cohort_type__in=REALTIME_GATED_COHORT_TYPES, condition_type__isnull=True)
+ if team_ids is not None:
+ qs = qs.filter(team_id__in=team_ids)
+ return list(qs.order_by("id").values_list("id", flat=True))
+
+
+def resave_team_cohorts(team: Team, *, batch_size: int, dry_run: bool, coalesce_rebuilds: bool) -> CohortResaveStats:
+ """Process all cohorts for a single team."""
+ all_cohorts = _load_team_cohorts(team, batch_size)
+ # Convert groups-only cohorts up front so the dependency map, the sort, and the recompute
+ # all read their filters. Static cohorts never evaluate filters, so they stay as they are.
+ # A cohort whose groups can't be converted stays unfixed, so it counts as an error.
+ unconvertible = 0
+ for loaded in all_cohorts:
+ if not loaded.filters and loaded.groups and not loaded.is_static:
+ loaded.filters = legacy_groups_to_filters(loaded.groups)
+ if not loaded.filters:
+ unconvertible += 1
+ logger.error("cohort_resave_groups_unconvertible", cohort_id=loaded.id, team_id=team.id)
+ seen_cohorts_cache: dict[int, CohortOrEmpty] = {c.id: c for c in all_cohorts}
+ cohort_dependencies, unreadable_ids = _build_dependency_map(team, all_cohorts, seen_cohorts_cache)
+
+ # Sort cohorts topologically - dependencies first, then dependents. The sort parses the
+ # filters of every cohort it reaches, so the unreadable ones stay out of it.
+ sorted_cohort_ids = sort_cohorts_topologically(
+ {c.id for c in all_cohorts if c.id not in unreadable_ids}, seen_cohorts_cache
+ )
+
+ total = len(unreadable_ids)
+ changed = 0
+ errors = len(unreadable_ids) + unconvertible
+ validation_errors = 0
+ prospective_realtime = 0
+
+ # Each save fires two whole-team flags-cache rebuilds, so the block coalesces them into one
+ # dispatch per team. Scoping it to one team means that team reads a stale cache only while
+ # its own cohorts are being resaved, rather than until the whole run finishes. Without the
+ # signal unwind nothing flushes the block on shutdown, so an unprotected run keeps the
+ # per-save enqueues instead.
+ rebuilds: AbstractContextManager[set[int]] = (
+ coalesced_cohort_flags_cache_rebuilds() if coalesce_rebuilds else nullcontext(set())
+ )
+ with rebuilds as stale_cache_teams:
+ for cohort_id in sorted_cohort_ids:
+ cohort = seen_cohorts_cache.get(cohort_id)
+ if not cohort:
+ continue
+
+ total += 1
+ try:
+ outcome = _recompute_cohort(cohort, team, seen_cohorts_cache, cohort_dependencies)
+ if outcome.validation_failed:
+ validation_errors += 1
+ # Counted before the save, so a realtime cohort whose save raises still reports as one.
+ if outcome.prospective_realtime:
+ prospective_realtime += 1
+ if outcome.changed:
+ if not dry_run:
+ cohort.save(update_fields=["filters", "cohort_type", "condition_type"])
+ changed += 1
+ except Exception as err:
+ errors += 1
+ logger.error(
+ "cohort_resave_error",
+ cohort_id=cohort.id,
+ team_id=team.id,
+ error=str(err),
+ exc_info=True,
+ )
+
+ return CohortResaveStats(
+ total=total,
+ changed=changed,
+ errors=errors,
+ validation_errors=validation_errors,
+ prospective_realtime=prospective_realtime,
+ rebuild_dispatch_failed=bool(stale_cache_teams),
+ )
+
+
+def _load_team_cohorts(team: Team, batch_size: int) -> list[Cohort]:
+ """Read every cohort on the team, one page at a time."""
+ # No `select_related("team")`: the caller already holds the team object and passes it into
+ # `_recompute_cohort`, so joining it in here would only hydrate the deprecated taxonomy
+ # columns (which TOAST out to megabytes) onto every retained cohort row. The forward-FK
+ # defer that `TeamManager` applies does not reach a `select_related` JOIN.
+ base_qs = Cohort.objects.filter(team=team).order_by("id")
+ all_cohorts: list[Cohort] = []
+ last_id = 0
+
+ while True:
+ batch = list(base_qs.filter(id__gt=last_id)[:batch_size])
+ if not batch:
+ return all_cohorts
+ all_cohorts.extend(batch)
+ last_id = batch[-1].id
+
+
+def _build_dependency_map(
+ team: Team, all_cohorts: list[Cohort], seen_cohorts_cache: dict[int, CohortOrEmpty]
+) -> tuple[dict[int, set[int]], set[int]]:
+ """Map each cohort id to every cohort id it depends on, recursively: A->B->C makes A depend on
+ both B and C.
+
+ Also return the ids of the cohorts whose dependencies cannot be read. Malformed filters make
+ the lookup raise for that cohort and for every cohort that references it. The run skips
+ these cohorts and counts them as errors, so one bad cohort does not stop the run.
+ """
+ cohort_dependencies: dict[int, set[int]] = {}
+ unreadable_ids: set[int] = set()
+ for cohort in all_cohorts:
+ if not cohort.filters:
+ continue
+ try:
+ dependencies = get_all_cohort_dependencies(cohort, seen_cohorts_cache=seen_cohorts_cache)
+ except Exception as err:
+ unreadable_ids.add(cohort.id)
+ logger.error(
+ "cohort_resave_error",
+ cohort_id=cohort.id,
+ team_id=team.id,
+ error=str(err),
+ exc_info=True,
+ )
+ continue
+ cohort_dependencies[cohort.id] = {dep.id for dep in dependencies}
+ return cohort_dependencies, unreadable_ids
+
+
+def _recompute_cohort(
+ cohort: Cohort,
+ team: Team,
+ seen_cohorts_cache: dict[int, CohortOrEmpty],
+ cohort_dependencies: dict[int, set[int]],
+) -> CohortResaveOutcome:
+ """Recompute one cohort's filters, cohort_type, and condition_type in memory.
+
+ This only mutates the cohort in memory and returns the outcome; the caller persists the
+ change and counts it.
+ """
+ filters = cohort.filters
+ # Nothing to recompute without filters.
+ if not filters:
+ return UNTOUCHED_COHORT
+
+ # Compute the new filters with inline bytecode and cohort_type
+ # Use defensive validation with detailed error reporting
+ clean_filters, computed_type, validation_error_list = validate_filters_and_compute_realtime_support(
+ filters, team, current_cohort_type=cohort.cohort_type, cohort_count=cohort.count
+ )
+
+ # If validation failed but we got the original filters back, log the issue and skip
+ if validation_error_list:
+ logger.warning(
+ "cohort_validation_skipped",
+ cohort_id=cohort.id,
+ team_id=cohort.team_id,
+ reason="Invalid filter structure - keeping original filters",
+ )
+ return CohortResaveOutcome(changed=False, validation_failed=True, prospective_realtime=False)
+
+ if computed_type == "realtime" and _references_block_realtime(filters, seen_cohorts_cache, cohort_dependencies):
+ computed_type = None
+
+ computed_condition_type = Cohort.compute_condition_type(clean_filters)
+
+ # Decide if there is any change worth persisting/reporting
+ will_change = (
+ clean_filters != filters
+ or computed_type != cohort.cohort_type
+ or computed_condition_type != cohort.condition_type
+ )
+
+ # ALWAYS update in-memory for dependency checking
+ cohort.filters = clean_filters
+ cohort.cohort_type = computed_type
+ cohort.condition_type = computed_condition_type
+
+ return CohortResaveOutcome(
+ changed=will_change,
+ validation_failed=False,
+ prospective_realtime=computed_type == "realtime",
+ )
+
+
+def _references_block_realtime(
+ filters: dict[str, Any],
+ seen_cohorts_cache: dict[int, CohortOrEmpty],
+ cohort_dependencies: dict[int, set[int]],
+) -> bool:
+ """Whether a directly referenced cohort stops this one from evaluating in realtime."""
+ for ref_id in _get_direct_cohort_references(filters):
+ ref_cohort = seen_cohorts_cache.get(ref_id)
+ if not ref_cohort:
+ continue
+ # Static cohorts cannot be realtime, so any cohort referencing them can't be realtime
+ if ref_cohort.is_static:
+ return True
+ # Cohorts without filters (empty cohorts) can be considered realtime-compatible
+ # since they match no one (always false)
+ if not ref_cohort.filters:
+ continue
+ # If any directly referenced cohort has dependencies, this cannot be realtime
+ if cohort_dependencies.get(ref_id):
+ return True
+ # Also check if the referenced cohort is not realtime
+ if ref_cohort.cohort_type != "realtime":
+ return True
+ return False
+
+
+def _get_direct_cohort_references(filters: dict[str, Any]) -> set[int]:
+ """Get only the direct cohort references from filters (not transitive)."""
+ referenced_ids = set()
+
+ if isinstance(filters, dict):
+ properties = filters.get("properties", {})
+ if isinstance(properties, dict):
+ values = properties.get("values", [])
+ if isinstance(values, list):
+ for value in values:
+ if isinstance(value, dict):
+ if value.get("type") == "cohort" and value.get("value"):
+ try:
+ referenced_ids.add(int(value["value"]))
+ except (ValueError, TypeError):
+ pass
+ # Recursively check nested groups
+ if "values" in value:
+ referenced_ids.update(_get_direct_cohort_references({"properties": value}))
+
+ return referenced_ids
diff --git a/posthog/management/commands/resave_cohorts.py b/posthog/management/commands/resave_cohorts.py
index cb6af363ff6a..100beb518010 100644
--- a/posthog/management/commands/resave_cohorts.py
+++ b/posthog/management/commands/resave_cohorts.py
@@ -1,91 +1,27 @@
from __future__ import annotations
-import signal
-import threading
-from collections.abc import Iterator
-from contextlib import AbstractContextManager, contextmanager, nullcontext
-from types import FrameType
+from contextlib import AbstractContextManager, nullcontext
from typing import Any
from django.core.management.base import BaseCommand, CommandError, CommandParser
import structlog
-from posthog.api.cohort import validate_filters_and_compute_realtime_support
+from posthog.cohort_resave import resave_team_cohorts, termination_unwinds, unclassified_cohort_ids
from posthog.dataclasses import frozen
from posthog.models.team.team import Team
-from products.cohorts.backend.models.cohort import Cohort, CohortOrEmpty, CohortType
-from products.cohorts.backend.models.util import (
- get_all_cohort_dependencies,
- legacy_groups_to_filters,
- sort_cohorts_topologically,
-)
from products.cohorts.backend.realtime_teams import (
backfill_triggers_suppressed,
is_processing_team,
realtime_allowlist_matches_every_team,
)
-from products.feature_flags.backend.flags_cache import coalesced_cohort_flags_cache_rebuilds
logger = structlog.get_logger(__name__)
# How many ids to name in a failure message before summarizing the rest.
ID_REPORT_LIMIT = 20
-# The cohort types the realtime membership gate routes, so a null condition_type changes how they
-# evaluate. See `uses_realtime_membership` in rust/feature-flags/src/cohorts/cohort_models.rs.
-REALTIME_GATED_COHORT_TYPES = (CohortType.REALTIME, CohortType.BEHAVIORAL)
-
-# The catchable signals that end a run without unwinding it. SIGTERM reaches the command when it
-# is the container's main process or is killed by hand. A run in a toolbox `kubectl exec` shell
-# never gets SIGTERM, because a pod stop signals only the main process, but a hangup of that
-# shell's terminal ends it with SIGHUP.
-UNWIND_SIGNALS = (signal.SIGTERM, signal.SIGHUP)
-
-
-@contextmanager
-def termination_unwinds() -> Iterator[bool]:
- """Raise on SIGTERM and SIGHUP instead of dying in place, when they are this command's to take.
-
- Either signal otherwise kills the process between a team's cohort saves committing and its
- coalesced cache rebuild being enqueued, and nothing sweeps that back: the service-cache
- verifier does not compare cohorts, and a rerun finds nothing left to change. Unwinding lets
- the coalescing block dispatch the teams it has already recorded. `KeyboardInterrupt` because
- the per-cohort `except Exception` must not swallow it. SIGKILL cannot be caught, so an
- expired toolbox claim or an OOM kill still loses the rebuild of the team in progress.
-
- Yields whether the unwind is in place. The admin view runs the command inside a web worker,
- which owns its own shutdown, so the caller must not coalesce rebuilds it cannot flush there.
- """
- if threading.current_thread() is not threading.main_thread():
- # Only the main thread may install a handler, and a server that hands sync views to a
- # threadpool runs the admin view off it.
- yield False
- return
-
- dispositions = {signum: signal.getsignal(signum) for signum in UNWIND_SIGNALS}
- if any(disposition not in (signal.SIG_DFL, signal.SIG_IGN) for disposition in dispositions.values()):
- # Something else owns one of them, a web worker serving the admin view for one. Taking it
- # over would replace that shutdown path, and a handler installed outside Python reads
- # back as `None`, which cannot be restored afterwards.
- yield False
- return
-
- def raise_interrupt(signum: int, frame: FrameType | None) -> None:
- raise KeyboardInterrupt(f"received {signal.Signals(signum).name}")
-
- # An ignored signal cannot end the run, so it stays ignored. `nohup` ignores SIGHUP, and a
- # run started under it must survive the hangup it asked to survive.
- taken = [signum for signum, disposition in dispositions.items() if disposition is signal.SIG_DFL]
- for signum in taken:
- signal.signal(signum, raise_interrupt)
- try:
- yield True
- finally:
- for signum in taken:
- signal.signal(signum, signal.SIG_DFL)
-
def _summarize_ids(ids: list[int]) -> str:
shown = ", ".join(str(value) for value in ids[:ID_REPORT_LIMIT])
@@ -111,28 +47,6 @@ class StaleFlagsCacheError(IncompleteResaveError):
"""The run left teams whose flags cache rebuild never reached the broker."""
-@frozen
-class CohortResaveStats:
- total: int
- changed: int
- errors: int
- validation_errors: int
- prospective_realtime: int
- # The team's cohorts were resaved but its cache rebuild never reached the broker, so the
- # team keeps a stale flags cache.
- rebuild_dispatch_failed: bool
-
-
-@frozen
-class CohortResaveOutcome:
- changed: bool
- validation_failed: bool
- prospective_realtime: bool
-
-
-UNTOUCHED_COHORT = CohortResaveOutcome(changed=False, validation_failed=False, prospective_realtime=False)
-
-
@frozen
class TeamSelection:
team_ids: tuple[int, ...] | None
@@ -220,7 +134,9 @@ class Command(BaseCommand):
)
self.stdout.write(f"Processing team {team.id} ({teams_processed}/{total_teams})")
- stats = self._process_team_cohorts(team, batch_size, dry_run, coalesce_rebuilds)
+ stats = resave_team_cohorts(
+ team, batch_size=batch_size, dry_run=dry_run, coalesce_rebuilds=coalesce_rebuilds
+ )
# Accumulate stats
global_total += stats.total
@@ -256,7 +172,7 @@ class Command(BaseCommand):
# A dry run persists nothing, so there is no post-run state to verify. `None` keeps that
# apart from an empty list, which would claim the check ran and found nothing.
- unclassified = None if dry_run else self._unclassified_cohort_ids(selection)
+ unclassified = None if dry_run else unclassified_cohort_ids(selection.team_ids)
# Log final summary
change_pct = round((global_changed / global_total * 100), 2) if global_total > 0 else 0
@@ -297,20 +213,6 @@ class Command(BaseCommand):
if stale_cache_team_ids:
self._fail_on_stale_cache(stale_cache_team_ids)
- def _unclassified_cohort_ids(self, selection: TeamSelection) -> list[int]:
- """Cohorts the realtime gate routes that still have a null `condition_type`, read back from
- the database.
-
- Reading persisted state covers every way a cohort ends up unclassified: a save that raised, a
- filter set the validator rejected, and a well-formed filter group with no leaf conditions,
- which the resave classifies as realtime and leaves with a null `condition_type`. It only
- means anything after a run that persisted, so the caller skips it on a dry run.
- """
- qs = Cohort.objects.filter(cohort_type__in=REALTIME_GATED_COHORT_TYPES, condition_type__isnull=True)
- if selection.team_ids is not None:
- qs = qs.filter(team_id__in=selection.team_ids)
- return list(qs.order_by("id").values_list("id", flat=True))
-
def _fail_on_unclassified(self, cohort_ids: list[int]) -> None:
"""Refuse to report success while a gated cohort still has a null `condition_type`.
@@ -383,231 +285,3 @@ class Command(BaseCommand):
)
return TeamSelection(team_ids=None, label="all teams", explicit=False)
-
- def _process_team_cohorts(
- self, team: Team, batch_size: int, dry_run: bool, coalesce_rebuilds: bool
- ) -> CohortResaveStats:
- """Process all cohorts for a single team."""
- all_cohorts = self._load_team_cohorts(team, batch_size)
- # Convert groups-only cohorts up front so the dependency map, the sort, and the recompute
- # all read their filters. Static cohorts never evaluate filters, so they stay as they are.
- # A cohort whose groups can't be converted stays unfixed, so it counts as an error.
- unconvertible = 0
- for loaded in all_cohorts:
- if not loaded.filters and loaded.groups and not loaded.is_static:
- loaded.filters = legacy_groups_to_filters(loaded.groups)
- if not loaded.filters:
- unconvertible += 1
- logger.error("cohort_resave_groups_unconvertible", cohort_id=loaded.id, team_id=team.id)
- seen_cohorts_cache: dict[int, CohortOrEmpty] = {c.id: c for c in all_cohorts}
- cohort_dependencies, unreadable_ids = self._build_dependency_map(team, all_cohorts, seen_cohorts_cache)
-
- # Sort cohorts topologically - dependencies first, then dependents. The sort parses the
- # filters of every cohort it reaches, so the unreadable ones stay out of it.
- sorted_cohort_ids = sort_cohorts_topologically(
- {c.id for c in all_cohorts if c.id not in unreadable_ids}, seen_cohorts_cache
- )
-
- total = len(unreadable_ids)
- changed = 0
- errors = len(unreadable_ids) + unconvertible
- validation_errors = 0
- prospective_realtime = 0
-
- # Each save fires two whole-team flags-cache rebuilds, so the block coalesces them into one
- # dispatch per team. Scoping it to one team means that team reads a stale cache only while
- # its own cohorts are being resaved, rather than until the whole run finishes. Without the
- # signal unwind nothing flushes the block on shutdown, so an unprotected run keeps the
- # per-save enqueues instead.
- rebuilds: AbstractContextManager[set[int]] = (
- coalesced_cohort_flags_cache_rebuilds() if coalesce_rebuilds else nullcontext(set())
- )
- with rebuilds as stale_cache_teams:
- for cohort_id in sorted_cohort_ids:
- cohort = seen_cohorts_cache.get(cohort_id)
- if not cohort:
- continue
-
- total += 1
- try:
- outcome = self._recompute_cohort(cohort, team, seen_cohorts_cache, cohort_dependencies)
- if outcome.validation_failed:
- validation_errors += 1
- # Counted before the save, so a realtime cohort whose save raises still reports as one.
- if outcome.prospective_realtime:
- prospective_realtime += 1
- if outcome.changed:
- if not dry_run:
- cohort.save(update_fields=["filters", "cohort_type", "condition_type"])
- changed += 1
- except Exception as err:
- errors += 1
- logger.error(
- "cohort_resave_error",
- cohort_id=cohort.id,
- team_id=team.id,
- error=str(err),
- exc_info=True,
- )
-
- return CohortResaveStats(
- total=total,
- changed=changed,
- errors=errors,
- validation_errors=validation_errors,
- prospective_realtime=prospective_realtime,
- rebuild_dispatch_failed=bool(stale_cache_teams),
- )
-
- def _load_team_cohorts(self, team: Team, batch_size: int) -> list[Cohort]:
- """Read every cohort on the team, one page at a time."""
- # No `select_related("team")`: the caller already holds the team object and passes it into
- # `_recompute_cohort`, so joining it in here would only hydrate the deprecated taxonomy
- # columns (which TOAST out to megabytes) onto every retained cohort row. The forward-FK
- # defer that `TeamManager` applies does not reach a `select_related` JOIN.
- base_qs = Cohort.objects.filter(team=team).order_by("id")
- all_cohorts: list[Cohort] = []
- last_id = 0
-
- while True:
- batch = list(base_qs.filter(id__gt=last_id)[:batch_size])
- if not batch:
- return all_cohorts
- all_cohorts.extend(batch)
- last_id = batch[-1].id
-
- def _build_dependency_map(
- self, team: Team, all_cohorts: list[Cohort], seen_cohorts_cache: dict[int, CohortOrEmpty]
- ) -> tuple[dict[int, set[int]], set[int]]:
- """Map each cohort id to every cohort id it depends on, recursively: A->B->C makes A depend on
- both B and C.
-
- Also return the ids of the cohorts whose dependencies cannot be read. Malformed filters make
- the lookup raise for that cohort and for every cohort that references it. The run skips
- these cohorts and counts them as errors, so one bad cohort does not stop the run.
- """
- cohort_dependencies: dict[int, set[int]] = {}
- unreadable_ids: set[int] = set()
- for cohort in all_cohorts:
- if not cohort.filters:
- continue
- try:
- dependencies = get_all_cohort_dependencies(cohort, seen_cohorts_cache=seen_cohorts_cache)
- except Exception as err:
- unreadable_ids.add(cohort.id)
- logger.error(
- "cohort_resave_error",
- cohort_id=cohort.id,
- team_id=team.id,
- error=str(err),
- exc_info=True,
- )
- continue
- cohort_dependencies[cohort.id] = {dep.id for dep in dependencies}
- return cohort_dependencies, unreadable_ids
-
- def _recompute_cohort(
- self,
- cohort: Cohort,
- team: Team,
- seen_cohorts_cache: dict[int, CohortOrEmpty],
- cohort_dependencies: dict[int, set[int]],
- ) -> CohortResaveOutcome:
- """Recompute one cohort's filters, cohort_type, and condition_type in memory.
-
- This only mutates the cohort in memory and returns the outcome; the caller persists the
- change and counts it.
- """
- filters = cohort.filters
- # Nothing to recompute without filters.
- if not filters:
- return UNTOUCHED_COHORT
-
- # Compute the new filters with inline bytecode and cohort_type
- # Use defensive validation with detailed error reporting
- clean_filters, computed_type, validation_error_list = validate_filters_and_compute_realtime_support(
- filters, team, current_cohort_type=cohort.cohort_type, cohort_count=cohort.count
- )
-
- # If validation failed but we got the original filters back, log the issue and skip
- if validation_error_list:
- logger.warning(
- "cohort_validation_skipped",
- cohort_id=cohort.id,
- team_id=cohort.team_id,
- reason="Invalid filter structure - keeping original filters",
- )
- return CohortResaveOutcome(changed=False, validation_failed=True, prospective_realtime=False)
-
- if computed_type == "realtime" and self._references_block_realtime(
- filters, seen_cohorts_cache, cohort_dependencies
- ):
- computed_type = None
-
- computed_condition_type = Cohort.compute_condition_type(clean_filters)
-
- # Decide if there is any change worth persisting/reporting
- will_change = (
- clean_filters != filters
- or computed_type != cohort.cohort_type
- or computed_condition_type != cohort.condition_type
- )
-
- # ALWAYS update in-memory for dependency checking
- cohort.filters = clean_filters
- cohort.cohort_type = computed_type
- cohort.condition_type = computed_condition_type
-
- return CohortResaveOutcome(
- changed=will_change,
- validation_failed=False,
- prospective_realtime=computed_type == "realtime",
- )
-
- def _references_block_realtime(
- self,
- filters: dict[str, Any],
- seen_cohorts_cache: dict[int, CohortOrEmpty],
- cohort_dependencies: dict[int, set[int]],
- ) -> bool:
- """Whether a directly referenced cohort stops this one from evaluating in realtime."""
- for ref_id in self._get_direct_cohort_references(filters):
- ref_cohort = seen_cohorts_cache.get(ref_id)
- if not ref_cohort:
- continue
- # Static cohorts cannot be realtime, so any cohort referencing them can't be realtime
- if ref_cohort.is_static:
- return True
- # Cohorts without filters (empty cohorts) can be considered realtime-compatible
- # since they match no one (always false)
- if not ref_cohort.filters:
- continue
- # If any directly referenced cohort has dependencies, this cannot be realtime
- if cohort_dependencies.get(ref_id):
- return True
- # Also check if the referenced cohort is not realtime
- if ref_cohort.cohort_type != "realtime":
- return True
- return False
-
- def _get_direct_cohort_references(self, filters: dict[str, Any]) -> set[int]:
- """Get only the direct cohort references from filters (not transitive)."""
- referenced_ids = set()
-
- if isinstance(filters, dict):
- properties = filters.get("properties", {})
- if isinstance(properties, dict):
- values = properties.get("values", [])
- if isinstance(values, list):
- for value in values:
- if isinstance(value, dict):
- if value.get("type") == "cohort" and value.get("value"):
- try:
- referenced_ids.add(int(value["value"]))
- except (ValueError, TypeError):
- pass
- # Recursively check nested groups
- if "values" in value:
- referenced_ids.update(self._get_direct_cohort_references({"properties": value}))
-
- return referenced_ids
diff --git a/posthog/management/commands/test/test_resave_cohorts.py b/posthog/management/commands/test/test_resave_cohorts.py
index 04dff712ef0c..b10328195514 100644
--- a/posthog/management/commands/test/test_resave_cohorts.py
+++ b/posthog/management/commands/test/test_resave_cohorts.py
@@ -15,7 +15,8 @@ from django.test import SimpleTestCase, override_settings
from parameterized import parameterized
-from posthog.management.commands.resave_cohorts import UNWIND_SIGNALS, StaleFlagsCacheError, termination_unwinds
+from posthog.cohort_resave import UNWIND_SIGNALS, termination_unwinds
+from posthog.management.commands.resave_cohorts import StaleFlagsCacheError
from posthog.models.team.team import Team
from posthog.test.db_context_capturing import capture_db_queries
diff --git a/products/cohorts/backend/backfill/runs.py b/products/cohorts/backend/backfill/runs.py
index 5d8d4dffda9a..ef1d05603d29 100644
--- a/products/cohorts/backend/backfill/runs.py
+++ b/products/cohorts/backend/backfill/runs.py
@@ -159,6 +159,10 @@ def behavioral_backfill_ineligibility_reason(cohort: Cohort) -> str | None:
properties = (cohort.filters or {}).get("properties")
if not any(leaf.get("type") == "behavioral" for leaf in walk_filter_leaves(properties)):
return NO_BEHAVIORAL_FILTER
+ # The flags service reads membership only for a cohort whose `condition_type` flags a behavioral
+ # leaf. A stamp on an unclassified cohort makes it read as ready while flags match nobody.
+ if cohort.condition_type is None:
+ return "has a null condition_type"
return _catalog_refusal_reason(cohort)
diff --git a/products/cohorts/backend/backfill/test/test_runs.py b/products/cohorts/backend/backfill/test/test_runs.py
index 9319237554f3..63dfe5753b78 100644
--- a/products/cohorts/backend/backfill/test/test_runs.py
+++ b/products/cohorts/backend/backfill/test/test_runs.py
@@ -290,9 +290,12 @@ class TestBackfillRuns(BaseTest):
def test_team_run_excludes_an_unseedable_cohort_and_refuses_it_by_id(self) -> None:
seedable = self._cohort()
unseedable = self._unseedable_cohort(self._WINDOWLESS_LEAF)
+ unclassified = self._cohort("$autocapture")
+ Cohort.objects.filter(id=unclassified.id).update(condition_type=None)
- with self.assertRaisesMessage(ValueError, "not eligible realtime behavioral cohorts"):
- create_team_backfill_run(self.team.id, "team_enablement", [unseedable.id])
+ for refused in (unseedable, unclassified):
+ with self.assertRaisesMessage(ValueError, "not eligible realtime behavioral cohorts"):
+ create_team_backfill_run(self.team.id, "team_enablement", [refused.id])
run = create_team_backfill_run(self.team.id, "team_enablement")
diff --git a/products/cohorts/backend/management/commands/realtime_cohort_team_access.py b/products/cohorts/backend/management/commands/realtime_cohort_team_access.py
index cd4fd4b9922a..511f0771c02b 100644
--- a/products/cohorts/backend/management/commands/realtime_cohort_team_access.py
+++ b/products/cohorts/backend/management/commands/realtime_cohort_team_access.py
@@ -1,13 +1,17 @@
-"""Grant, revoke and list team access to the realtime cohort pipeline, meant for a toolbox pod.
+"""Enable, grant, revoke and list team access to the realtime cohort pipeline, meant for a toolbox pod.
-Django and the shufflers read these rows, each as a union with its own `REALTIME_COHORT_TEAM_ALLOWLIST`.
-The processor and seeder read only their env lists, so every grant or revoke needs the same change there.
+Django, the shufflers, the processor and the seeder read these rows, each as a union with its own
+`REALTIME_COHORT_TEAM_ALLOWLIST`.
-A grant at `processing` gives the team Django's side of the pipeline while its users see none of it,
-which is what a load test needs. `serving` also shows them realtime cohort state. `revoke --purpose
-load_test` sets every load-test row to `off` at once, and ends no run.
+`enable` is how a beta team gets realtime cohorts: it prepares the team's cohorts, grants `processing`,
+creates the team backfill runs, and moves the team to `serving` once neither run was refused. Run it again
+after resolving a refusal.
-A team on the env list is `serving` whatever its row says, so `grant` and `revoke` refuse it.
+A grant at `processing` gives the team the pipeline while its users see none of it, which is what a load
+test needs. `serving` also shows them realtime cohort state. `revoke --purpose load_test` sets every
+load-test row to `off` at once, and ends no run.
+
+A team on the env list is `serving` whatever its row says, so `enable`, `grant` and `revoke` refuse it.
"""
import argparse
@@ -21,6 +25,16 @@ from products.cohorts.backend.models.team_access import (
RealtimeCohortLevel,
RealtimeCohortTeamAccess,
)
+from products.cohorts.backend.realtime_enablement import (
+ AlreadyServing,
+ NothingToBackfill,
+ RunAlreadyCreated,
+ RunCreated,
+ RunOutcome,
+ RunRefused,
+ TeamEnablement,
+ enable_realtime_cohorts,
+)
from products.cohorts.backend.realtime_teams import (
GrantLevel,
grant_realtime_cohort_access,
@@ -32,17 +46,28 @@ GRANT_LEVELS: dict[str, GrantLevel] = {
RealtimeCohortLevel.SERVING.value: RealtimeCohortLevel.SERVING,
}
LIST_COLUMNS = ("team_id", "level", "purpose", "save_triggers_backfill", "processing_since", "off_since", "changed_at")
+# `enable` alone writes beta rows, so a grant cannot skip the preparation it does. A revoked team cannot be
+# granted again, so beta customer teams are revoked one id at a time.
+NON_BETA_PURPOSES = [
+ purpose for purpose in RealtimeCohortAccessPurpose.values if purpose != RealtimeCohortAccessPurpose.BETA
+]
class Command(BaseCommand):
- help = "Grant, revoke and list team access to the realtime cohort pipeline."
+ help = "Enable, grant, revoke and list team access to the realtime cohort pipeline."
def add_arguments(self, parser: CommandParser) -> None:
subparsers = parser.add_subparsers(dest="action", required=True)
+ enable = subparsers.add_parser(
+ "enable",
+ help="Prepare each beta team's cohorts and backfills, and move it to serving once nothing was refused.",
+ )
+ enable.add_argument("--teams", type=int, nargs="+", required=True)
+
grant = subparsers.add_parser("grant", help="Write the teams' access rows, all or none.")
grant.add_argument("--teams", type=int, nargs="+", required=True)
- grant.add_argument("--purpose", choices=RealtimeCohortAccessPurpose.values, required=True)
+ grant.add_argument("--purpose", choices=NON_BETA_PURPOSES, required=True)
# Required, so that re-running a grant cannot quietly move a serving team down to processing
# or turn its triggers off.
grant.add_argument(
@@ -61,13 +86,15 @@ class Command(BaseCommand):
revoke = subparsers.add_parser("revoke", help="Set the selected rows to off, all or none.")
selector = revoke.add_mutually_exclusive_group(required=True)
selector.add_argument("--teams", type=int, nargs="+")
- selector.add_argument("--purpose", choices=RealtimeCohortAccessPurpose.values)
+ selector.add_argument("--purpose", choices=NON_BETA_PURPOSES)
subparsers.add_parser("list", help="Print every access row.")
def handle(self, *args: Any, **options: Any) -> None:
try:
match options["action"]:
+ case "enable":
+ self._enable(options["teams"])
case "grant":
self._grant(options)
case "revoke":
@@ -77,6 +104,53 @@ class Command(BaseCommand):
except ValueError as error:
raise CommandError(str(error)) from error
+ def _enable(self, team_ids: list[int]) -> None:
+ not_serving: list[int] = []
+ stale_flags_cache: list[int] = []
+ for team_id in sorted(set(team_ids)):
+ try:
+ result = enable_realtime_cohorts(team_id, purpose=RealtimeCohortAccessPurpose.BETA)
+ except ValueError as error:
+ self.stdout.write(f"Team {team_id}: refused, {error}")
+ not_serving.append(team_id)
+ continue
+ match result:
+ case AlreadyServing():
+ self.stdout.write(f"Team {team_id}: already serving")
+ case TeamEnablement() as enablement:
+ self._write_enablement(team_id, enablement)
+ if not enablement.promoted:
+ not_serving.append(team_id)
+ if enablement.resave.rebuild_dispatch_failed:
+ stale_flags_cache.append(team_id)
+ failures = []
+ if not_serving:
+ failures.append(f"Not enabled: {', '.join(str(team_id) for team_id in not_serving)}.")
+ # A rerun is a no-op once the team serves, so it cannot repair this.
+ if stale_flags_cache:
+ failures.append(
+ f"Flags cache rebuild not enqueued for {', '.join(str(team_id) for team_id in stale_flags_cache)}, "
+ "so enqueue update_team_flags_cache and update_team_service_flags_cache for them."
+ )
+ if failures:
+ raise CommandError(" ".join(failures))
+
+ def _write_enablement(self, team_id: int, enablement: TeamEnablement) -> None:
+ level = RealtimeCohortLevel.SERVING if enablement.promoted else RealtimeCohortLevel.PROCESSING
+ self.stdout.write(f"Team {team_id}: {level}")
+ resave = enablement.resave
+ resave_line = (
+ f" Resave: {resave.total} cohorts, {resave.changed} changed, {resave.errors} errors, "
+ f"{resave.validation_errors} validation errors"
+ )
+ if resave.rebuild_dispatch_failed:
+ resave_line += ", flags cache rebuild not enqueued"
+ self.stdout.write(resave_line)
+ for refusal in enablement.refused_cohorts:
+ self.stdout.write(f" Not backfilled: cohort {refusal.cohort_id}, {refusal.kind}, {refusal.reason}")
+ self.stdout.write(f" Behavioral run: {_run_outcome(enablement.behavioral)}")
+ self.stdout.write(f" Person run: {_run_outcome(enablement.person)}")
+
def _grant(self, options: dict[str, Any]) -> None:
granted = grant_realtime_cohort_access(
options["teams"],
@@ -104,6 +178,18 @@ class Command(BaseCommand):
self.stdout.write("\t".join(values))
+def _run_outcome(outcome: RunOutcome) -> str:
+ match outcome:
+ case RunCreated(run_id=run_id):
+ return f"created {run_id}"
+ case RunAlreadyCreated(run_id=run_id):
+ return f"already created {run_id}"
+ case NothingToBackfill():
+ return "nothing to backfill"
+ case RunRefused(reason=reason):
+ return f"refused, {reason}"
+
+
def _team_list(team_ids: list[int]) -> str:
return f"{len(team_ids)} teams: {', '.join(str(team_id) for team_id in team_ids)}"
diff --git a/products/cohorts/backend/migrations/0018_realtime_cohort_team_access_beta_purpose.py b/products/cohorts/backend/migrations/0018_realtime_cohort_team_access_beta_purpose.py
new file mode 100644
index 000000000000..8b9eff97cf5e
--- /dev/null
+++ b/products/cohorts/backend/migrations/0018_realtime_cohort_team_access_beta_purpose.py
@@ -0,0 +1,24 @@
+# Generated by Django 6.1.3 on 2026-11-04 23:59
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+ dependencies = [
+ ("cohorts", "0017_realtime_cohort_team_access"),
+ ]
+
+ operations = [
+ migrations.AlterField(
+ model_name="realtimecohortteamaccess",
+ name="purpose",
+ field=models.CharField(
+ choices=[
+ ("pilot", "Pilot"),
+ ("load_test", "Load test"),
+ ("beta", "Beta"),
+ ],
+ max_length=16,
+ ),
+ ),
+ ]
diff --git a/products/cohorts/backend/migrations/max_migration.txt b/products/cohorts/backend/migrations/max_migration.txt
index 0e7c8aee6d1f..e6851137529a 100644
--- a/products/cohorts/backend/migrations/max_migration.txt
+++ b/products/cohorts/backend/migrations/max_migration.txt
@@ -1 +1 @@
-0017_realtime_cohort_team_access
+0018_realtime_cohort_team_access_beta_purpose
diff --git a/products/cohorts/backend/models/team_access.py b/products/cohorts/backend/models/team_access.py
index bd8c754f134f..d20b29b8e112 100644
--- a/products/cohorts/backend/models/team_access.py
+++ b/products/cohorts/backend/models/team_access.py
@@ -14,14 +14,15 @@ class RealtimeCohortLevel(models.TextChoices):
class RealtimeCohortAccessPurpose(models.TextChoices):
PILOT = "pilot", "Pilot"
LOAD_TEST = "load_test", "Load test"
+ BETA = "beta", "Beta"
class RealtimeCohortTeamAccess(models.Model):
"""A team's access to the realtime cohort pipeline, granted on top of `REALTIME_COHORT_TEAM_ALLOWLIST`.
- Staff write it only through the `realtime_cohort_team_access` command. Django and the shufflers read
- it. The processor and the seeder do not, so their env lists must change with it. A missing row means
- `off`. Never read it through `get_or_create_team_extension`, because that creates the row.
+ Staff write it only through the `realtime_cohort_team_access` command. Django, the shufflers, the
+ processor and the seeder read it. A missing row means `off`. Never read it through
+ `get_or_create_team_extension`, because that creates the row.
Rows are never deleted: a revoke sets `off`, and a team deletion leaves the row (`DO_NOTHING`).
So when a team's row appears on one read and is missing on a later one, the later read failed.
diff --git a/products/cohorts/backend/realtime_enablement.py b/products/cohorts/backend/realtime_enablement.py
new file mode 100644
index 000000000000..5f082dbf8bd5
--- /dev/null
+++ b/products/cohorts/backend/realtime_enablement.py
@@ -0,0 +1,265 @@
+from collections.abc import Callable
+from datetime import datetime
+from uuid import UUID
+
+from django.conf import settings
+from django.db.models import Q
+
+import structlog
+
+from posthog.cohort_resave import CohortResaveStats, resave_team_cohorts, termination_unwinds
+from posthog.dataclasses import frozen
+from posthog.models.team.team import Team
+
+from products.cohorts.backend.backfill.pinning import pin_behavioral_run
+from products.cohorts.backend.backfill.readiness import ensure_filters_shape_hash
+from products.cohorts.backend.backfill.runs import (
+ INAPPLICABLE_BACKFILL_REASONS,
+ behavioral_backfill_ineligibility_reason,
+ check_person_run_preconditions,
+ create_person_team_backfill_run,
+ create_team_backfill_run,
+ person_backfill_ineligibility_reason,
+)
+from products.cohorts.backend.backfill.sizing import estimate_behavioral_scan_events
+from products.cohorts.backend.models.backfill import (
+ CohortBackfillKind,
+ CohortBackfillRun,
+ CohortBackfillRunStatus,
+ CohortBackfillScope,
+ CohortBackfillTrigger,
+)
+from products.cohorts.backend.models.cohort import Cohort, CohortType
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
+from products.cohorts.backend.realtime_teams import grant_realtime_cohort_access, grantable_access_rows
+
+logger = structlog.get_logger(__name__)
+
+_RESAVE_BATCH_SIZE = 500
+
+# A team run that will seed, or has stamped readiness. A rerun after a refusal keeps it, so it does not
+# seed that kind again or check it again against the scan limit. Not `blocked`, which the seeder never claims.
+_KEPT_RUN_STATUSES = (
+ CohortBackfillRunStatus.AWAITING_BOUNDARY,
+ CohortBackfillRunStatus.SEEDING,
+ CohortBackfillRunStatus.RECONCILING,
+ CohortBackfillRunStatus.TRAILING,
+ CohortBackfillRunStatus.COMPLETED,
+)
+
+
+@frozen
+class RunCreated:
+ run_id: UUID
+
+
+@frozen
+class RunAlreadyCreated:
+ run_id: UUID
+
+
+@frozen
+class NothingToBackfill:
+ pass
+
+
+@frozen
+class RunRefused:
+ reason: str
+
+
+type RunOutcome = RunCreated | RunAlreadyCreated | NothingToBackfill | RunRefused
+
+
+@frozen
+class CohortRefusal:
+ cohort_id: int
+ kind: CohortBackfillKind
+ reason: str
+
+
+@frozen
+class TeamEnablement:
+ resave: CohortResaveStats
+ refused_cohorts: tuple[CohortRefusal, ...]
+ behavioral: RunOutcome
+ person: RunOutcome
+
+ @property
+ def promoted(self) -> bool:
+ return not isinstance(self.behavioral, RunRefused) and not isinstance(self.person, RunRefused)
+
+
+@frozen
+class AlreadyServing:
+ pass
+
+
+def enable_realtime_cohorts(team_id: int, *, purpose: RealtimeCohortAccessPurpose) -> TeamEnablement | AlreadyServing:
+ """Put the team at `processing` with its cohorts ready for the pipeline, create its two team
+ backfill runs, and promote it to `serving` only when neither run was refused.
+
+ Raises ``ValueError`` before it writes anything for a team the grant refuses, when an operator
+ attestation is missing, or when a first call finds readiness stamps. A call after a refusal keeps
+ the runs earlier calls created.
+ """
+ row = grantable_access_rows([team_id], purpose, RealtimeCohortTeamAccess.objects.all()).get(team_id)
+ # The steps below go through `processing`, which would hide a serving team's realtime state.
+ if row is not None and row.level == RealtimeCohortLevel.SERVING:
+ return AlreadyServing()
+ # Without them neither team run can be created, so the team would only sit at `processing`.
+ _, missing = check_person_run_preconditions()
+ if missing:
+ raise ValueError(f"Missing operator attestations: {', '.join(missing)}")
+ if row is None:
+ # A team with no row was stamped by a run from a time the env list named it, or by the legacy
+ # calculation workflow, whose stamp the flags service also accepts. Nothing maintains that
+ # membership, and `serving` would route flags to it before the new runs seed.
+ stamped = list(
+ Cohort.objects.filter(
+ Q(last_backfill_events_at__isnull=False)
+ | Q(last_backfill_person_properties_at__isnull=False)
+ | Q(last_realtime_cohort_calculation_at__isnull=False),
+ team_id=team_id,
+ )
+ .order_by("id")
+ .values_list("id", flat=True)
+ )
+ if stamped:
+ raise ValueError(
+ "Cohorts stamped ready before the grant, by a pipeline that stopped maintaining them: "
+ f"{', '.join(str(cohort_id) for cohort_id in stamped)}"
+ )
+
+ team = Team.objects.get(id=team_id)
+ with termination_unwinds() as coalesce_rebuilds:
+ resave = resave_team_cohorts(
+ team, batch_size=_RESAVE_BATCH_SIZE, dry_run=False, coalesce_rebuilds=coalesce_rebuilds
+ )
+ for cohort in _realtime_cohorts(team_id):
+ ensure_filters_shape_hash(cohort)
+ # After the resave and the hashes, because the processor loads the team's catalog within seconds of
+ # the grant. A cohort that is stale or hash-less in that first load waits for the next refresh.
+ grant_realtime_cohort_access(
+ [team_id], level=RealtimeCohortLevel.PROCESSING, purpose=purpose, save_triggers_backfill=True
+ )
+ granted_at = RealtimeCohortTeamAccess.objects.get(team_id=team_id).processing_since
+ assert granted_at is not None, "grant sets processing_since on the row it inserts"
+
+ cohorts = _realtime_cohorts(team_id)
+ behavioral_cohorts, behavioral_refusals = _judge(
+ cohorts, CohortBackfillKind.BEHAVIORAL, behavioral_backfill_ineligibility_reason
+ )
+ person_cohorts, person_refusals = _judge(
+ cohorts, CohortBackfillKind.PERSON_PROPERTY, person_backfill_ineligibility_reason
+ )
+ enablement = TeamEnablement(
+ resave=resave,
+ refused_cohorts=(*behavioral_refusals, *person_refusals),
+ behavioral=_team_run(
+ team_id,
+ CohortBackfillKind.BEHAVIORAL,
+ behavioral_cohorts,
+ granted_at,
+ lambda: _create_behavioral_team_run(team, behavioral_cohorts),
+ ),
+ person=_team_run(
+ team_id,
+ CohortBackfillKind.PERSON_PROPERTY,
+ person_cohorts,
+ granted_at,
+ lambda: create_person_team_backfill_run(
+ team_id,
+ CohortBackfillTrigger.TEAM_ENABLEMENT,
+ settings.BEHAVIORAL_BACKFILL_PERSON_DEFAULT_HORIZON_DAYS,
+ ),
+ ),
+ )
+ if enablement.promoted:
+ grant_realtime_cohort_access(
+ [team_id], level=RealtimeCohortLevel.SERVING, purpose=purpose, save_triggers_backfill=True
+ )
+ return enablement
+
+
+def _realtime_cohorts(team_id: int) -> list[Cohort]:
+ return list(
+ Cohort.objects.filter(
+ team_id=team_id, cohort_type=CohortType.REALTIME, is_static=False, deleted=False
+ ).order_by("id")
+ )
+
+
+def _judge(
+ cohorts: list[Cohort], kind: CohortBackfillKind, ineligibility_reason: Callable[[Cohort], str | None]
+) -> tuple[list[Cohort], list[CohortRefusal]]:
+ eligible: list[Cohort] = []
+ refused: list[CohortRefusal] = []
+ for cohort in cohorts:
+ reason = ineligibility_reason(cohort)
+ if reason is None:
+ eligible.append(cohort)
+ elif reason not in INAPPLICABLE_BACKFILL_REASONS:
+ refused.append(CohortRefusal(cohort_id=cohort.id, kind=kind, reason=reason))
+ return eligible, refused
+
+
+def _team_run(
+ team_id: int,
+ kind: CohortBackfillKind,
+ eligible: list[Cohort],
+ granted_at: datetime,
+ create: Callable[[], CohortBackfillRun],
+) -> RunOutcome:
+ if not eligible:
+ return NothingToBackfill()
+ # Runs from before the grant, such as from a time the env list named the team, seeded state the
+ # pipeline has since dropped.
+ kept = (
+ CohortBackfillRun.objects.for_team(team_id)
+ .filter(
+ scope=CohortBackfillScope.TEAM,
+ backfill_kind=kind,
+ status__in=_KEPT_RUN_STATUSES,
+ created_at__gte=granted_at,
+ )
+ .order_by("-created_at")
+ .first()
+ )
+ if kept is not None:
+ return RunAlreadyCreated(run_id=kept.id)
+ # A creator writes its run in one transaction, so a failure of any kind leaves no run behind. The
+ # ClickHouse sizing scans fail with errors no narrower type covers.
+ try:
+ return RunCreated(run_id=create().id)
+ except ValueError as error:
+ return RunRefused(reason=str(error))
+ except Exception as error:
+ logger.exception("realtime_enablement_run_failed", team_id=team_id, backfill_kind=kind)
+ return RunRefused(reason=str(error))
+
+
+def _create_behavioral_team_run(team: Team, eligible: list[Cohort]) -> CohortBackfillRun:
+ """`create_team_backfill_run` behind the scan limit of `create_cohort_backfill_run`, refusing with
+ ``ValueError`` like the person creator."""
+ max_events_per_day = settings.BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY
+ if not max_events_per_day:
+ return create_team_backfill_run(team.id, CohortBackfillTrigger.TEAM_ENABLEMENT)
+ pinned = pin_behavioral_run(eligible, team)
+ estimate = estimate_behavioral_scan_events(
+ team.id,
+ pinned["event_names"],
+ flag_calls_table=pinned["flag_calls_table"],
+ max_events_per_day=max_events_per_day,
+ )
+ if estimate.over_limit:
+ raise ValueError(
+ f"The run would scan {estimate.peak_day_events} events on its busiest day ({estimate.peak_day}), "
+ f"above the limit of {estimate.max_events_per_day}. Accept the volume with create_cohort_backfill_run "
+ "--max-scan-events-per-day, then run enable again."
+ )
+ return create_team_backfill_run(team.id, CohortBackfillTrigger.TEAM_ENABLEMENT, scan_estimate=estimate)
diff --git a/products/cohorts/backend/realtime_teams.py b/products/cohorts/backend/realtime_teams.py
index a1df4f94c298..2c9520d5c66d 100644
--- a/products/cohorts/backend/realtime_teams.py
+++ b/products/cohorts/backend/realtime_teams.py
@@ -167,6 +167,37 @@ def backfill_triggers_suppressed() -> Iterator[None]:
_backfill_triggers_suppressed.reset(token)
+def grantable_access_rows(
+ team_ids: list[int], purpose: RealtimeCohortAccessPurpose, access: QuerySet[RealtimeCohortTeamAccess]
+) -> dict[int, RealtimeCohortTeamAccess]:
+ """The teams' existing rows read from ``access``. Raises ``ValueError`` if a grant for ``purpose``
+ must not write a row for one of the teams."""
+ parent_by_team = dict(Team.objects.filter(id__in=team_ids).values_list("id", "parent_team_id"))
+ _refuse("No team with id", [team_id for team_id in team_ids if team_id not in parent_by_team])
+ # Cohorts live on the root team (`RootTeamMixin`), so a row on a child environment covers nothing.
+ _refuse("Not a root team", [team_id for team_id, parent_id in parent_by_team.items() if parent_id is not None])
+ # The env list makes a team `serving` whatever its row says, and `revoke` refuses a listed
+ # team. So a row here would change nothing, and would block `revoke --purpose` for every team.
+ _refuse(
+ "On REALTIME_COHORT_TEAM_ALLOWLIST",
+ [team_id for team_id in team_ids if realtime_allowlist_matches(team_id)],
+ )
+ rows = {row.team_id: row for row in access.filter(team_id__in=team_ids)}
+ # The Rust readers in rust/common/cohort-team-access treat `off` as final, so allowing this
+ # needs a change there first.
+ _refuse(
+ "Revoked, and its pipeline state was never purged",
+ [team_id for team_id, row in rows.items() if row.level == RealtimeCohortLevel.OFF],
+ )
+ # `revoke --purpose` selects by it, so a re-grant that relabels a pilot as a load test would
+ # put the pilot under the load test's stop command.
+ _refuse(
+ f"Granted for another purpose than {purpose.value}",
+ [team_id for team_id, row in rows.items() if row.purpose != purpose],
+ )
+ return rows
+
+
def grant_realtime_cohort_access(
team_ids: Iterable[int],
*,
@@ -181,33 +212,7 @@ def grant_realtime_cohort_access(
"""
requested = sorted(set(team_ids))
with transaction.atomic():
- parent_by_team = dict(Team.objects.filter(id__in=requested).values_list("id", "parent_team_id"))
- _refuse("No team with id", [team_id for team_id in requested if team_id not in parent_by_team])
- # Cohorts live on the root team (`RootTeamMixin`), so a row on a child environment covers nothing.
- _refuse("Not a root team", [team_id for team_id, parent_id in parent_by_team.items() if parent_id is not None])
- # The env list makes a team `serving` whatever its row says, and `revoke` refuses a listed
- # team. So a row here would change nothing, and would block `revoke --purpose` for every team.
- _refuse(
- "On REALTIME_COHORT_TEAM_ALLOWLIST",
- [team_id for team_id in requested if realtime_allowlist_matches(team_id)],
- )
- rows = {
- row.team_id: row
- for row in RealtimeCohortTeamAccess.objects.select_for_update().filter(team_id__in=requested)
- }
- # The Rust readers in rust/common/cohort-team-access treat `off` as final, so allowing this
- # needs a change there first.
- _refuse(
- "Revoked, and its pipeline state was never purged",
- [team_id for team_id, row in rows.items() if row.level == RealtimeCohortLevel.OFF],
- )
- # `revoke --purpose` selects by it, so a re-grant that relabels a pilot as a load test would
- # put the pilot under the load test's stop command.
- _refuse(
- f"Granted for another purpose than {purpose.value}",
- [team_id for team_id, row in rows.items() if row.purpose != purpose],
- )
-
+ rows = grantable_access_rows(requested, purpose, RealtimeCohortTeamAccess.objects.select_for_update())
now = timezone.now()
for team_id in requested:
row = rows.get(team_id) or RealtimeCohortTeamAccess(team_id=team_id, processing_since=now)
diff --git a/products/cohorts/backend/test/test_realtime_cohort_team_access_command.py b/products/cohorts/backend/test/test_realtime_cohort_team_access_command.py
index 064856052c16..88983f945230 100644
--- a/products/cohorts/backend/test/test_realtime_cohort_team_access_command.py
+++ b/products/cohorts/backend/test/test_realtime_cohort_team_access_command.py
@@ -1,5 +1,6 @@
from datetime import UTC, datetime
from io import StringIO
+from typing import Any
import time_machine
from posthog.test.base import BaseTest
@@ -13,6 +14,7 @@ from parameterized import parameterized
from posthog.models.team.team import Team
+from products.cohorts.backend.models.cohort import Cohort
from products.cohorts.backend.models.team_access import (
RealtimeCohortAccessPurpose,
RealtimeCohortLevel,
@@ -140,6 +142,25 @@ class TestRealtimeCohortTeamAccessCommand(BaseTest):
assert self._access(self.team.pk).level == RealtimeCohortLevel.SERVING
+ @parameterized.expand(
+ [
+ (
+ "grant",
+ ["grant", "--teams", "{team}", "--purpose", "beta", "--level", "serving", "--save-triggers-backfill"],
+ ),
+ ("revoke", ["revoke", "--purpose", "beta"]),
+ ]
+ )
+ def test_only_enable_writes_beta_rows(self, _name: str, args: list[str]) -> None:
+ RealtimeCohortTeamAccess.objects.create(
+ team=self.team, level=RealtimeCohortLevel.PROCESSING, purpose=RealtimeCohortAccessPurpose.BETA
+ )
+
+ with self.assertRaisesMessage(CommandError, "invalid choice: 'beta'"):
+ call_command("realtime_cohort_team_access", *(arg.format(team=self.team.pk) for arg in args))
+
+ assert self._access(self.team.pk).level == RealtimeCohortLevel.PROCESSING
+
def test_revoke_by_purpose_leaves_other_purposes_on(self) -> None:
pilot = Team.objects.create(organization=self.organization)
RealtimeCohortTeamAccess.objects.create(
@@ -155,3 +176,66 @@ class TestRealtimeCohortTeamAccessCommand(BaseTest):
RealtimeCohortLevel.OFF,
RealtimeCohortLevel.SERVING,
)
+
+ @override_settings(
+ BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED=True,
+ BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED=True,
+ BEHAVIORAL_BACKFILL_PERSON_SIZING_ATTESTED=True,
+ BEHAVIORAL_BACKFILL_PERSON_TTL_ATTESTED=True,
+ BEHAVIORAL_BACKFILL_PERSON_TOPIC_BYTES_BUDGET=1_000_000,
+ )
+ @patch(
+ "products.cohorts.backend.realtime_enablement.estimate_behavioral_scan_events",
+ side_effect=RuntimeError("ClickHouse timeout"),
+ )
+ def test_enable_handles_each_team_and_fails_when_one_is_not_enabled(self, _scan_estimate: Any) -> None:
+ held, serving, enabled = (Team.objects.create(organization=self.organization) for _ in range(3))
+ env_listed_cohort = Cohort.objects.create(
+ team=self.team,
+ filters={
+ "properties": {
+ "type": "AND",
+ "values": [{"type": "person", "key": "email", "operator": "exact", "value": "[email protected]"}],
+ }
+ },
+ )
+ Cohort.objects.filter(id=env_listed_cohort.id).update(condition_type=None)
+ Cohort.objects.create(
+ team=held,
+ filters={
+ "properties": {
+ "type": "AND",
+ "values": [
+ {
+ "type": "behavioral",
+ "key": "$pageview",
+ "value": "performed_event",
+ "event_type": "events",
+ "time_value": 7,
+ "time_interval": "day",
+ }
+ ],
+ }
+ },
+ )
+ RealtimeCohortTeamAccess.objects.create(
+ team=serving, level=RealtimeCohortLevel.SERVING, purpose=RealtimeCohortAccessPurpose.BETA
+ )
+
+ out = StringIO()
+ with (
+ override_settings(REALTIME_COHORT_TEAM_ALLOWLIST=str(self.team.pk)),
+ self.assertRaisesMessage(CommandError, f"Not enabled: {self.team.pk}, {held.pk}."),
+ ):
+ teams = [str(team.pk) for team in (enabled, serving, held, self.team)]
+ call_command("realtime_cohort_team_access", "enable", "--teams", *teams, stdout=out)
+
+ output = out.getvalue()
+ assert f"Team {self.team.pk}: refused, On REALTIME_COHORT_TEAM_ALLOWLIST: {self.team.pk}" in output
+ assert f"Team {serving.pk}: already serving" in output
+ assert Cohort.objects.get(id=env_listed_cohort.id).condition_type is None
+ assert [self._access(team.pk).level for team in (held, serving, enabled)] == [
+ RealtimeCohortLevel.PROCESSING,
+ RealtimeCohortLevel.SERVING,
+ RealtimeCohortLevel.SERVING,
+ ]
diff --git a/products/cohorts/backend/test/test_realtime_enablement.py b/products/cohorts/backend/test/test_realtime_enablement.py
new file mode 100644
index 000000000000..657ed52acbb8
--- /dev/null
+++ b/products/cohorts/backend/test/test_realtime_enablement.py
@@ -0,0 +1,222 @@
+from datetime import UTC, date, datetime
+from typing import Any
+
+from posthog.test.base import BaseTest
+from unittest import mock
+
+from django.test import override_settings
+
+from parameterized import parameterized
+
+from products.cohorts.backend.backfill.flag_calls_table import FlagCallsTable
+from products.cohorts.backend.backfill.runs import create_team_backfill_run
+from products.cohorts.backend.backfill.sizing import BehavioralScanEstimate, PersonSeedEstimate
+from products.cohorts.backend.models.backfill import (
+ CohortBackfillKind,
+ CohortBackfillRun,
+ CohortBackfillRunCohort,
+ CohortBackfillRunStatus,
+ CohortBackfillScope,
+ CohortBackfillTrigger,
+)
+from products.cohorts.backend.models.cohort import Cohort
+from products.cohorts.backend.models.team_access import (
+ RealtimeCohortAccessPurpose,
+ RealtimeCohortLevel,
+ RealtimeCohortTeamAccess,
+)
+from products.cohorts.backend.realtime_enablement import (
+ CohortRefusal,
+ RunAlreadyCreated,
+ RunCreated,
+ RunRefused,
+ TeamEnablement,
+ enable_realtime_cohorts,
+)
+
+PERSON_LEAF = {"type": "person", "key": "email", "operator": "exact", "value": "[email protected]"}
+PAGEVIEW_LEAF = {
+ "type": "behavioral",
+ "key": "$pageview",
+ "value": "performed_event",
+ "event_type": "events",
+ "time_value": 7,
+ "time_interval": "day",
+}
+
+
+@override_settings(
+ REALTIME_COHORT_TEAM_ALLOWLIST="none",
+ BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED=True,
+ BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED=True,
+ BEHAVIORAL_BACKFILL_PERSON_SIZING_ATTESTED=True,
+ BEHAVIORAL_BACKFILL_PERSON_TTL_ATTESTED=True,
+ BEHAVIORAL_BACKFILL_PERSON_TOPIC_BYTES_BUDGET=1_000_000,
+ BEHAVIORAL_BACKFILL_MAX_SCAN_EVENTS_PER_DAY=1_000,
+)
+class TestEnableRealtimeCohorts(BaseTest):
+ def setUp(self) -> None:
+ super().setUp()
+ self.peak_day_events = 500
+ scan = mock.patch(
+ "products.cohorts.backend.realtime_enablement.estimate_behavioral_scan_events",
+ side_effect=self._scan_estimate,
+ )
+ sizing = mock.patch(
+ "products.cohorts.backend.backfill.runs.estimate_person_seed_topic_bytes",
+ return_value=PersonSeedEstimate(
+ estimated_persons=10,
+ pinned_condition_count=1,
+ bytes_per_seed=294,
+ estimated_topic_bytes=2_940,
+ budget_bytes=1_000_000,
+ ),
+ )
+ for patcher in (scan, sizing):
+ patcher.start()
+ self.addCleanup(patcher.stop)
+
+ def _scan_estimate(
+ self, team_id: int, event_names: list[str], *, flag_calls_table: FlagCallsTable, max_events_per_day: int
+ ) -> BehavioralScanEstimate:
+ return BehavioralScanEstimate(
+ days_sampled=7,
+ peak_day=date(2026, 11, 1),
+ peak_day_events=self.peak_day_events,
+ peak_day_events_by_name={"$pageview": self.peak_day_events},
+ max_events_per_day=max_events_per_day,
+ )
+
+ def _cohort(self, *leaves: dict[str, Any]) -> Cohort:
+ cohort = Cohort.objects.create(team=self.team, filters={"properties": {"type": "AND", "values": list(leaves)}})
+ # Cohorts saved before the column existed have no `condition_type`.
+ Cohort.objects.filter(id=cohort.id).update(condition_type=None)
+ return cohort
+
+ def _level(self) -> str:
+ return RealtimeCohortTeamAccess.objects.get(team=self.team).level
+
+ def _team_runs(self, kind: CohortBackfillKind) -> list[CohortBackfillRun]:
+ return list(CohortBackfillRun.objects.for_team(self.team.id).filter(backfill_kind=kind))
+
+ def _enable(self) -> TeamEnablement:
+ enablement = enable_realtime_cohorts(self.team.id, purpose=RealtimeCohortAccessPurpose.BETA)
+ assert isinstance(enablement, TeamEnablement)
+ return enablement
+
+ def test_enables_a_team_in_one_call(self) -> None:
+ mixed = self._cohort(PERSON_LEAF, PAGEVIEW_LEAF)
+ person_only = self._cohort(PERSON_LEAF)
+ # The API accepts a behavioral leaf with no time window, and the catalog drops it.
+ windowless = self._cohort({key: value for key, value in PAGEVIEW_LEAF.items() if not key.startswith("time_")})
+ CohortBackfillRun.objects.for_team(self.team.id).create(
+ team_id=self.team.id,
+ backfill_kind=CohortBackfillKind.BEHAVIORAL,
+ trigger_kind=CohortBackfillTrigger.TEAM_ENABLEMENT,
+ scope=CohortBackfillScope.TEAM,
+ status=CohortBackfillRunStatus.COMPLETED,
+ timezone="UTC",
+ created_at=datetime(2026, 1, 1, tzinfo=UTC),
+ )
+
+ enablement = self._enable()
+
+ access = RealtimeCohortTeamAccess.objects.get(team=self.team)
+ assert (access.level, access.purpose, access.save_triggers_backfill) == (
+ RealtimeCohortLevel.SERVING,
+ RealtimeCohortAccessPurpose.BETA,
+ True,
+ )
+ cohorts = list(Cohort.objects.filter(team=self.team).order_by("id"))
+ assert [cohort.condition_type and cohort.condition_type["behavioral"] for cohort in cohorts] == [
+ True,
+ False,
+ True,
+ ]
+ assert all(
+ None
+ not in (cohort.filters_shape_hash, cohort.behavioral_filters_shape_hash, cohort.person_filters_shape_hash)
+ for cohort in cohorts
+ )
+
+ assert isinstance(enablement.behavioral, RunCreated)
+ assert isinstance(enablement.person, RunCreated)
+ runs = CohortBackfillRun.objects.for_team(self.team.id)
+ behavioral_run = runs.get(id=enablement.behavioral.run_id)
+ person_run = runs.get(id=enablement.person.run_id)
+ assert {(run.scope, run.status, run.trigger_kind) for run in (behavioral_run, person_run)} == {
+ (CohortBackfillScope.TEAM, CohortBackfillRunStatus.AWAITING_BOUNDARY, CohortBackfillTrigger.TEAM_ENABLEMENT)
+ }
+ participations = CohortBackfillRunCohort.objects.for_team(self.team.id)
+ assert sorted(participations.filter(run=behavioral_run).values_list("cohort_id", flat=True)) == [mixed.id]
+ assert sorted(participations.filter(run=person_run).values_list("cohort_id", flat=True)) == [
+ mixed.id,
+ person_only.id,
+ ]
+ assert enablement.refused_cohorts == (
+ CohortRefusal(
+ cohort_id=windowless.id,
+ kind=CohortBackfillKind.BEHAVIORAL,
+ reason="has a filter the realtime catalog drops (unsupported_state_variant)",
+ ),
+ )
+
+ def test_a_refused_run_keeps_the_team_processing_until_a_rerun_has_both_runs(self) -> None:
+ self._cohort(PERSON_LEAF, PAGEVIEW_LEAF)
+ self.peak_day_events = 2_000
+
+ first = self._enable()
+
+ assert isinstance(first.behavioral, RunRefused)
+ assert "2000 events on its busiest day (2026-11-01), above the limit of 1000" in first.behavioral.reason
+ [person_run] = self._team_runs(CohortBackfillKind.PERSON_PROPERTY)
+ assert first.person == RunCreated(run_id=person_run.id)
+ assert self._team_runs(CohortBackfillKind.BEHAVIORAL) == []
+ assert self._level() == RealtimeCohortLevel.PROCESSING
+
+ # The operator accepts the scan volume through `create_cohort_backfill_run
+ # --max-scan-events-per-day 0`. That run finishes, and the person run fails.
+ behavioral_run = create_team_backfill_run(self.team.id, CohortBackfillTrigger.TEAM_ENABLEMENT)
+ runs = CohortBackfillRun.objects.for_team(self.team.id)
+ runs.filter(id=behavioral_run.id).update(status=CohortBackfillRunStatus.COMPLETED)
+ runs.filter(id=person_run.id).update(status=CohortBackfillRunStatus.FAILED)
+ second = self._enable()
+
+ assert second.behavioral == RunAlreadyCreated(run_id=behavioral_run.id)
+ assert isinstance(second.person, RunCreated)
+ assert self._level() == RealtimeCohortLevel.SERVING
+
+ @parameterized.expand(
+ [
+ (
+ "missing_attestation",
+ {"BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED": False},
+ {},
+ "Missing operator attestations: processor durability",
+ ),
+ (
+ "stamped_before_the_grant",
+ {},
+ {"last_backfill_events_at": datetime(2026, 1, 1, tzinfo=UTC)},
+ "Cohorts stamped ready before the grant",
+ ),
+ (
+ "calculated_before_the_grant",
+ {},
+ {"last_realtime_cohort_calculation_at": datetime(2026, 1, 1, tzinfo=UTC)},
+ "Cohorts stamped ready before the grant",
+ ),
+ ]
+ )
+ def test_refuses_the_team_before_any_write(
+ self, _name: str, overrides: dict[str, Any], stamps: dict[str, datetime], message: str
+ ) -> None:
+ cohort = self._cohort(PERSON_LEAF, PAGEVIEW_LEAF)
+ Cohort.objects.filter(id=cohort.id).update(**stamps)
+
+ with override_settings(**overrides), self.assertRaisesMessage(ValueError, message):
+ enable_realtime_cohorts(self.team.id, purpose=RealtimeCohortAccessPurpose.BETA)
+
+ assert not RealtimeCohortTeamAccess.objects.filter(team=self.team).exists()
+ assert not CohortBackfillRun.objects.for_team(self.team.id).exists()
+ assert Cohort.objects.get(id=cohort.id).condition_type is None