:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(cohorts): pin the flag-calls table on behavioral backfill runs (#125201)
Phil Haacke27be6db9fb3
diff --git a/docs/internal/realtime-cohorts/backfill-coordination.md b/docs/internal/realtime-cohorts/backfill-coordination.mdindex 8e7db16bef51..124827c89e94 100644--- a/docs/internal/realtime-cohorts/backfill-coordination.md+++ b/docs/internal/realtime-cohorts/backfill-coordination.md@@ -132,12 +132,25 @@ The processor walks the cohort with its own current catalog and checks only theThe creator pins:- on each participation: the cohort's full `filters` JSON, its definition hash, and the shape hash of the run's kind,-- on a behavioral run: every behavioral condition to replay, with its condition hash, window fields, operator, threshold, window in days and event name, plus the union of event names to scan,+- on a behavioral run: every behavioral condition to replay, with its condition hash, window fields, operator, threshold, window in days and event name, plus the union of event names to scan and `flag_calls_table`, the table the seeder reads `$feature_flag_called` from,- on a person run: every person condition hash to evaluate, and a **horizon**: only persons whose record changed within the last N days are scanned.The seeder builds its catalog from these pinned filters with the same `cohort-core` code the processor uses.The finalizer later compares the pinned hash, and the pinned filters, with the cohort's current ones.+`flag_calls_table` is `events` or `flag_evaluations`, and `schema_version` stays 1.+The seeder reads a missing key as `events` and fails the run on any other value, so both behavioral creators write the key on every run.+The creator pins `flag_evaluations` only when every cohort in the run that has a `$feature_flag_called` condition meets both rules:++- The batch calculation reads every flag call of that cohort from `flag_evaluations`. The creator compiles the cohort's batch query to check this, so the seeder and the batch calculation read the same table.+- Every `$feature_flag_called` window in that cohort is relative and has no end date, so it moves with the run's boundary. `flag_evaluations` drops rows after 90 days. When the seeder scans a day that a relative window still covers, the table still holds that day. The first day of a fixed date range can leave the table before the seeder scans it, while `events` still holds it. This rule does not apply to an organization whose flag calls are only in `flag_evaluations`, because `events` holds none of its new flag calls.++A cohort without a `$feature_flag_called` condition does not affect the choice.+When a cohort's batch query does not compile, the creator refuses the run.+The save path refuses with reason `flag_calls_table_unresolved`, and the team creator raises an error that names the cohort.+In an organization whose flag calls are only in `flag_evaluations`, a run that pinned `events` would give the cohort an empty flag-call history that still counts as backfilled.+The pin does not change after the creator writes it, so every chunk and every retry reads the same table even if the organization's flag evaluations mode changes during the run.+An edit that invalidates the run's kind supersedes it.A cohort-scoped run then stops at the seeder's next lease renewal.In a team-scoped run, the seeder drops the edited cohort's conditions and carries on for the rest.@@ -196,6 +209,7 @@ Two management commands cover operator work.It can narrow the run to chosen cohorts, which is how an operator re-drives one cohort whose trigger was lost.It refuses when an attestation is missing or when a target cohort already has an open participation, and it supports a dry run.For a behavioral run it first counts the team's events on the pinned event names over the last 7 complete UTC days, through the `events` primary key on the offline cluster.+ When the run will pin `flag_evaluations`, it counts `$feature_flag_called` from `flag_evaluations` instead.It prints the busiest day and its largest events, refuses above the behavioral scan limit, and records the count in the run's `preconditions`.The count is a recent sample, not a bound: an older day, or a day in the run's timezone, can hold more events.The seeder reads fewer rows when every condition on an event name filters on event property values.diff --git a/posthog/api/cohort.py b/posthog/api/cohort.pyindex 2e7181ab8c7b..7ef6628023d5 100644--- a/posthog/api/cohort.py+++ b/posthog/api/cohort.py@@ -38,7 +38,7 @@ from posthog.schema import ActorsQuery, HogQLQuery, ProductKeyfrom posthog.hogql.compiler.bytecode import create_bytecodefrom posthog.hogql.constants import CSV_EXPORT_LIMIT, LimitContextfrom posthog.hogql.context import HogQLContext-from posthog.hogql.errors import BaseHogQLError, TableAccessDeniedError+from posthog.hogql.errors import TableAccessDeniedErrorfrom posthog.hogql.printer import prepare_ast_for_printingfrom posthog.hogql.property import PERSON_METADATA_FIELDS, property_to_exprfrom posthog.hogql.transforms.flag_calls_from_flag_evaluations import FLAG_EVALUATIONS_COMPLETE_DAYS@@ -66,6 +66,7 @@ from posthog.helpers.trigram_search import ()from posthog.hogql_queries.actors_query_runner import ActorsQueryRunnerfrom posthog.hogql_queries.hogql_cohort_query import (+ FLAG_CALL_READS_COMPILE_ERRORS,FlagCallReads,HogQLCohortQuery,batch_flag_call_reads,@@ -1102,7 +1103,7 @@ def _flag_call_reads_or_none(criteria: Sequence[Property], team: Team) -> FlagCavalues = [{**prop.to_dict(), "negation": False} for prop in criteria]try:return batch_flag_call_reads(Cohort(team=team, filters={"properties": {"type": "OR", "values": values}}), team)- except (BaseHogQLError, ValidationError, ValueError):+ except FLAG_CALL_READS_COMPILE_ERRORS:return Noneexcept Exception as error:# A malformed filter value can fail an assert in the HogQL printer. The guard refuses the save, so the APIdiff --git a/posthog/hogql_queries/hogql_cohort_query.py b/posthog/hogql_queries/hogql_cohort_query.pyindex a89341252bcd..fca73c037840 100644--- a/posthog/hogql_queries/hogql_cohort_query.py+++ b/posthog/hogql_queries/hogql_cohort_query.py@@ -36,6 +36,7 @@ from posthog.hogql import astfrom posthog.hogql.ast import SelectQuery, SelectSetNode, SelectSetQueryfrom posthog.hogql.constants import HogQLGlobalSettings, LimitContextfrom posthog.hogql.context import HogQLContext+from posthog.hogql.errors import BaseHogQLErrorfrom posthog.hogql.flag_called_warnings import count_flag_calls_read_from_eventsfrom posthog.hogql.parser import parse_selectfrom posthog.hogql.printer import prepare_and_print_ast@@ -53,6 +54,8 @@ from posthog.ph_client import feature_enabled_or_falsefrom posthog.types import AnyPropertyFilterfrom products.cohorts.backend.models.cohort import Cohort+from products.feature_flags.backend.facade.enums import FlagEvaluationsMode+from products.feature_flags.backend.facade.flags import get_flag_evaluations_read_modeINTERVAL_TO_SECONDS = {"minute": 60,@@ -797,20 +800,33 @@ class HogQLCohortQuery:return condition.query+# What batch_flag_call_reads raises when the cohort's batch query does not compile.+FLAG_CALL_READS_COMPILE_ERRORS = (BaseHogQLError, ValidationError, ValueError)++@frozenclass FlagCallReads:+ only_in_flag_evaluations: boolreads_from_events: int+def batch_reads_flag_calls_from_events_only(team: Team) -> bool:+ """Whether the batch calculation reads every flag call from events for the team's organization, whatever the+ cohort. It reads the same mode as the catalog build, so it answers without compiling a cohort."""+ return get_flag_evaluations_read_mode(team.organization_id) == FlagEvaluationsMode.EVENTS++def batch_flag_call_reads(cohort: Cohort, team: Team) -> FlagCallReads:- """How many flag-call reads the batch query for this cohort leaves on events.+ """Whether the organization's flag calls are only in flag_evaluations, and how many flag-call reads the batch query+ for this cohort leaves on events.It compiles the cohort the way the batch calculation does, so it follows the rewrite's rules and its fallback toevents when a field doesn't resolve on flag_evaluations. It raises whatever the compile raises."""executor = HogQLCohortQuery(cohort=cohort, team=team).get_query_executor(bypass_warehouse_access_control=True)- executor.generate_clickhouse_sql()+ _, context = executor.generate_clickhouse_sql()prepared = executor.clickhouse_prepared_astreturn FlagCallReads(+ only_in_flag_evaluations=context.database is not None and context.database.flag_calls_only_in_flag_evaluations,reads_from_events=count_flag_calls_read_from_events(prepared) if prepared is not None else 0,)diff --git a/posthog/tasks/calculate_cohort.py b/posthog/tasks/calculate_cohort.pyindex 25d7b2365c54..514b00363686 100644--- a/posthog/tasks/calculate_cohort.py+++ b/posthog/tasks/calculate_cohort.py@@ -1336,6 +1336,8 @@ COHORT_BACKFILL_REFUSAL_OUTCOMES: dict[BackfillRefusalReason, str] = {# its own.BackfillRefusalReason.SIZING_SCAN_CAP_EXCEEDED: "refused_ineligible",BackfillRefusalReason.DEFINITION_CHANGED: "refused_transient",+ # The cohort's batch query fails to compile on every trigger until the cohort or the compiler changes.+ BackfillRefusalReason.FLAG_CALLS_TABLE_UNRESOLVED: "refused_ineligible",}diff --git a/products/cohorts/backend/backfill/flag_calls_table.py b/products/cohorts/backend/backfill/flag_calls_table.pynew file mode 100644index 000000000000..3cc2fca2fb62--- /dev/null+++ b/products/cohorts/backend/backfill/flag_calls_table.py@@ -0,0 +1,126 @@+import math+from collections.abc import Iterable+from datetime import UTC, datetime+from enum import StrEnum+from typing import Any++from posthog.hogql.transforms.flag_calls_from_flag_evaluations import FLAG_EVALUATIONS_COMPLETE_DAYS++from posthog.exceptions_capture import capture_exception+from posthog.models.flag_evaluations.sql import FLAG_EVALUATIONS_SOURCE_EVENT, FLAG_EVALUATIONS_TABLE+from posthog.models.team.team import Team++from products.cohorts.backend.models.cohort import Cohort+from products.cohorts.backend.parity.eligibility import resolve_behavioral_window+from products.cohorts.backend.parity.tzdates import day_of_instant++from common.hogvm.python.utils import HogVMException++# The seeder scans team-local days and includes the oldest day of the window. A team east of UTC starts its day on+# the UTC date before. A scan that starts at most this many days back reads only days that flag_evaluations holds,+# in every timezone. The API's save guard for flag-call windows uses the same limit.+_MAX_FLAG_EVALUATIONS_DAYS_BACK = FLAG_EVALUATIONS_COMPLETE_DAYS - 1+++class FlagCallsTable(StrEnum):+ EVENTS = "events"+ FLAG_EVALUATIONS = FLAG_EVALUATIONS_TABLE+++class FlagCallsTableUnresolved(ValueError):+ pass+++def _scan_window(condition: dict[str, Any], team: Team) -> tuple[bool, float]:+ """Whether the condition's window slides with the run's boundary, and how many team-local days before today the+ seeder's scan for it starts."""+ window = resolve_behavioral_window(condition)+ if window is None:+ return False, math.inf+ if window.kind != "explicit":+ return True, window.days+ start = condition.get("explicit_datetime")+ if not isinstance(start, str):+ return False, math.inf+ parsed = datetime.fromisoformat(start)+ start_day = day_of_instant(parsed, team.timezone_info) if parsed.tzinfo else parsed.date()+ return False, (day_of_instant(datetime.now(UTC), team.timezone_info) - start_day).days+++def _can_seed_from_flag_evaluations(cohort: Cohort, conditions: list[dict[str, Any]], team: Team) -> bool:+ # hogql_cohort_query imports posthog.tasks.calculate_cohort, which imports this module through runs.+ from posthog.hogql_queries.hogql_cohort_query import ( # noqa: PLC0415+ FLAG_CALL_READS_COMPILE_ERRORS,+ batch_flag_call_reads,+ batch_reads_flag_calls_from_events_only,+ )++ if batch_reads_flag_calls_from_events_only(team):+ return False++ # A malformed filter value can also fail an assert in the HogQL printer or fail in the HogQL VM. These errors+ # repeat on every try, so the run is refused. Any other error, such as a database timeout, reaches the trigger+ # task, which retries it.+ repeating_errors = (*FLAG_CALL_READS_COMPILE_ERRORS, AssertionError, HogVMException)+ try:+ reads = batch_flag_call_reads(cohort, team)+ except repeating_errors as error:+ if not isinstance(error, FLAG_CALL_READS_COMPILE_ERRORS):+ capture_exception(error, additional_properties={"team_id": team.id, "cohort_id": cohort.id})+ raise FlagCallsTableUnresolved(+ f"The batch query of cohort {cohort.id} fails to build, so the run can't pick the table to read "+ f"its flag calls from: {error}"+ ) from error+ scans = [_scan_window(condition, team) for condition in conditions]+ within_ttl = all(days_back <= _MAX_FLAG_EVALUATIONS_DAYS_BACK for _, days_back in scans)+ if reads.only_in_flag_evaluations:+ if reads.reads_from_events != 0:+ raise FlagCallsTableUnresolved(+ f"The batch query of cohort {cohort.id} reads flag calls from events, which holds no new flag calls"+ )+ if not within_ttl:+ raise FlagCallsTableUnresolved(+ f"A flag-call window of cohort {cohort.id} starts before the oldest day flag_evaluations keeps, and "+ "events holds no new flag calls"+ )+ return True+ return reads.reads_from_events == 0 and within_ttl and all(slides for slides, _ in scans)+++def pinned_flag_calls_table(+ cohorts: Iterable[Cohort], conditions: Iterable[dict[str, Any]], team: Team+) -> FlagCallsTable:+ """The table cohort-seeder reads `$feature_flag_called` from in a run over these cohorts and their pinned+ conditions.++ One table covers the whole run. It is flag_evaluations only when every cohort with a flag-call condition meets+ all three rules. A cohort without a flag-call condition does not count.++ - The batch calculation reads every flag call of the cohort from flag_evaluations. The seeder then reads the+ same rows as the batch calculation.+ - Every flag-call window starts at most _MAX_FLAG_EVALUATIONS_DAYS_BACK days before today. The seeder fails a+ run that scans a day older than the flag_evaluations TTL.+ - Every flag-call window slides with the run's boundary, unless the organization's flag calls are only in+ flag_evaluations. The first day of a fixed range can leave flag_evaluations before the seeder scans it, while+ events still holds it. When events holds no new flag calls, flag_evaluations is the only table that has any+ of them.++ A run for an organization that reads flag calls from events pins events without compiling any cohort.++ Raises FlagCallsTableUnresolved when a cohort's batch query fails to build with an error that repeats on every+ try, or when the organization's flag calls are only in flag_evaluations and the batch query still reads flag+ calls from events or a flag-call window starts too far back. Pinning events there would give the cohort an empty+ flag-call history that still counts as backfilled. Any other error from building the batch query, such as a+ database timeout, reaches the caller.+ """+ flag_call_conditions: dict[int, list[dict[str, Any]]] = {}+ for condition in conditions:+ if condition["event_name"] == FLAG_EVALUATIONS_SOURCE_EVENT:+ flag_call_conditions.setdefault(condition["cohort_id"], []).append(condition)+ # A list, not a generator, so all() can't stop at a False verdict before a later cohort raises.+ verdicts = [+ _can_seed_from_flag_evaluations(cohort, flag_call_conditions[cohort.id], team)+ for cohort in cohorts+ if cohort.id in flag_call_conditions+ ]+ return FlagCallsTable.FLAG_EVALUATIONS if verdicts and all(verdicts) else FlagCallsTable.EVENTSdiff --git a/products/cohorts/backend/backfill/pinning.py b/products/cohorts/backend/backfill/pinning.pyindex 310daaee23e3..1642a4cf67ac 100644--- a/products/cohorts/backend/backfill/pinning.py+++ b/products/cohorts/backend/backfill/pinning.py@@ -3,6 +3,9 @@ from typing import Anyimport structlog+from posthog.models.team.team import Team++from products.cohorts.backend.backfill.flag_calls_table import pinned_flag_calls_tablefrom products.cohorts.backend.models.cohort import Cohortfrom products.cohorts.backend.models.leaf_shape import walk_filter_leavesfrom products.cohorts.backend.parity.eligibility import is_action_key, leaf_drop_reason@@ -77,6 +80,19 @@ def pin_conditions_for_cohorts(cohorts: Iterable[Cohort]) -> tuple[dict[str, Anyreturn {"schema_version": 1, "conditions": conditions}, sorted(event_names)+def pin_behavioral_run(cohorts: list[Cohort], team: Team) -> dict[str, Any]:+ """The `pinned` payload of a behavioral run over these cohorts.++ Raises FlagCallsTableUnresolved when a flag-call cohort's batch query does not compile.+ """+ pinned, event_names = pin_conditions_for_cohorts(cohorts)+ return {+ **pinned,+ "event_names": event_names,+ "flag_calls_table": pinned_flag_calls_table(cohorts, pinned["conditions"], team),+ }++def pin_person_conditions_for_cohorts(cohorts: Iterable[Cohort], *, max_conditions: int) -> dict[str, Any]:conditions: list[dict[str, Any]] = []diff --git a/products/cohorts/backend/backfill/runs.py b/products/cohorts/backend/backfill/runs.pyindex 9d8482b09502..5d8d4dffda9a 100644--- a/products/cohorts/backend/backfill/runs.py+++ b/products/cohorts/backend/backfill/runs.py@@ -14,10 +14,11 @@ import structlogfrom posthog.dataclasses import frozenfrom posthog.models.team.team import Team+from products.cohorts.backend.backfill.flag_calls_table import FlagCallsTableUnresolvedfrom products.cohorts.backend.backfill.pinning import (PersonPinningCapExceeded,leaf_unpinnable_reason,- pin_conditions_for_cohorts,+ pin_behavioral_run,pin_person_conditions_for_cohorts,)from products.cohorts.backend.backfill.readiness import ensure_filters_shape_hash@@ -185,11 +186,6 @@ def _run_status(preconditions_missing: list[str]) -> tuple[str, str]:return CohortBackfillRunStatus.AWAITING_BOUNDARY, ""-def _pinned_payload(cohorts: Iterable[Cohort]) -> dict[str, Any]:- pinned, event_names = pin_conditions_for_cohorts(cohorts)- return {**pinned, "event_names": event_names}--def _active_participation_cohort_ids(team_id: int, cohort_ids: Iterable[int], *, kind: CohortBackfillKind) -> set[int]:return set(CohortBackfillRunCohort.objects.for_team(team_id)@@ -262,6 +258,7 @@ class BackfillRefusalReason(StrEnum):OVER_BUDGET = "over_budget"SIZING_SCAN_CAP_EXCEEDED = "sizing_scan_cap_exceeded"DEFINITION_CHANGED = "definition_changed"+ FLAG_CALLS_TABLE_UNRESOLVED = "flag_calls_table_unresolved"@frozen@@ -319,6 +316,17 @@ def attempt_backfill_run_for_cohort(team_id: int, cohort_id: int, trigger_kind:if _has_active_cohort_run(team_id, cohort_id, kind=CohortBackfillKind.BEHAVIORAL):return BackfillRunAttempt.refused(BackfillRefusalReason.RUN_SLOT_OCCUPIED)+ try:+ pinned = pin_behavioral_run([cohort], cohort.team)+ except FlagCallsTableUnresolved as error:+ logger.warning(+ "cohort_behavioral_backfill_flag_calls_table_unresolved",+ team_id=team_id,+ cohort_id=cohort_id,+ error=str(error),+ )+ return BackfillRunAttempt.refused(BackfillRefusalReason.FLAG_CALLS_TABLE_UNRESOLVED)+filters_shape_hash = ensure_filters_shape_hash(cohort)behavioral_filters_shape_hash = cohort.behavioral_filters_shape_hash or ""preconditions, missing = check_run_preconditions()@@ -331,7 +339,7 @@ def attempt_backfill_run_for_cohort(team_id: int, cohort_id: int, trigger_kind:scope=CohortBackfillScope.COHORT,status=status,timezone=cohort.team.timezone,- pinned=_pinned_payload([cohort]),+ pinned=pinned,preconditions=preconditions,blocked_reason=blocked_reason,)@@ -385,11 +393,15 @@ def create_team_backfill_run(requested_ids = set(cohort_ids) if cohort_ids is not None else Nonewith transaction.atomic():team = Team.objects.get(id=team_id)- queryset = Cohort.objects.select_for_update(of=("self",)).filter(- team_id=team_id,- cohort_type=CohortType.REALTIME,- is_static=False,- deleted=False,+ queryset = (+ Cohort.objects.select_for_update(of=("self",))+ .select_related("team")+ .filter(+ team_id=team_id,+ cohort_type=CohortType.REALTIME,+ is_static=False,+ deleted=False,+ ))if requested_ids is not None:queryset = queryset.filter(id__in=requested_ids)@@ -409,6 +421,7 @@ def create_team_backfill_run(if _has_active_team_run(team_id, kind=CohortBackfillKind.BEHAVIORAL):raise ValueError(f"Team {team_id} already has an active team backfill run (behavioral)")+ pinned = pin_behavioral_run(cohorts, team)hashes: dict[int, str] = {}behavioral_hashes: dict[int, str] = {}for cohort in cohorts:@@ -427,7 +440,7 @@ def create_team_backfill_run(status=status,boundary_at=boundary_at,timezone=team.timezone,- pinned=_pinned_payload(cohorts),+ pinned=pinned,preconditions=preconditions,blocked_reason=blocked_reason,)diff --git a/products/cohorts/backend/backfill/sizing.py b/products/cohorts/backend/backfill/sizing.pyindex 2bd8c8115591..cc2a8cdcf618 100644--- a/products/cohorts/backend/backfill/sizing.py+++ b/products/cohorts/backend/backfill/sizing.py@@ -11,6 +11,9 @@ from posthog.clickhouse.workload import Workloadfrom posthog.dataclasses import frozenfrom posthog.errors import InternalCHQueryErrorfrom posthog.exceptions import ClickHouseQueryTimeOut+from posthog.models.flag_evaluations.sql import FLAG_EVALUATIONS_SOURCE_EVENT++from products.cohorts.backend.backfill.flag_calls_table import FlagCallsTable# Calibrated against a serialized `PersonSeed` (rust/cohort-core/src/seed/person.rs): the fixed# envelope (schema_version, kind, team_id, person_id, scanned_at_ms, run_id, claim_epoch) plus one@@ -155,28 +158,55 @@ def estimate_behavioral_scan_events(team_id: int,event_names: Sequence[str],*,+ flag_calls_table: FlagCallsTable,max_events_per_day: int,) -> BehavioralScanEstimate:- """Reads `events` because that is the table the seeder scans."""+ """Counts each event name in the table that cohort-seeder scans it from."""+ until = django_timezone.now().astimezone(UTC).replace(hour=0, minute=0, second=0, microsecond=0)+ window = {"team_id": team_id, "since": until - timedelta(days=BEHAVIORAL_SCAN_ESTIMATE_DAYS), "until": until}+ queries: list[tuple[str, dict[str, Any]]] = []+ counts_flag_calls_on_flag_evaluations = (+ flag_calls_table == FlagCallsTable.FLAG_EVALUATIONS and FLAG_EVALUATIONS_SOURCE_EVENT in event_names+ )+ names_on_events = [+ name+ for name in event_names+ if not (counts_flag_calls_on_flag_evaluations and name == FLAG_EVALUATIONS_SOURCE_EVENT)+ ]+ if counts_flag_calls_on_flag_evaluations:+ queries.append(+ (+ """+ SELECT toDate(timestamp) AS day, %(flag_called)s AS event, count() AS events+ FROM flag_evaluations+ WHERE team_id = %(team_id)s+ AND timestamp >= %(since)s+ AND timestamp < %(until)s+ GROUP BY day+ """,+ {**window, "flag_called": FLAG_EVALUATIONS_SOURCE_EVENT},+ )+ )+ if names_on_events:+ queries.append(+ (+ """+ SELECT toDate(timestamp) AS day, event, count() AS events+ FROM events+ WHERE team_id = %(team_id)s+ AND timestamp >= %(since)s+ AND timestamp < %(until)s+ AND event IN %(event_names)s+ GROUP BY day, event+ """,+ {**window, "event_names": names_on_events},+ )+ )by_day: dict[date, dict[str, int]] = {}- if event_names:- until = django_timezone.now().astimezone(UTC).replace(hour=0, minute=0, second=0, microsecond=0)+ for query, params in queries:rows = sync_execute(- """- SELECT toDate(timestamp) AS day, event, count() AS events- FROM events- WHERE team_id = %(team_id)s- AND timestamp >= %(since)s- AND timestamp < %(until)s- AND event IN %(event_names)s- GROUP BY day, event- """,- {- "team_id": team_id,- "since": until - timedelta(days=BEHAVIORAL_SCAN_ESTIMATE_DAYS),- "until": until,- "event_names": list(event_names),- },+ query,+ params,settings={"max_execution_time": _BEHAVIORAL_SCAN_ESTIMATE_MAX_SECONDS},workload=Workload.OFFLINE,team_id=team_id,diff --git a/products/cohorts/backend/backfill/test/test_management_command.py b/products/cohorts/backend/backfill/test/test_management_command.pyindex b68230a677d8..e37bde4633de 100644--- a/products/cohorts/backend/backfill/test/test_management_command.py+++ b/products/cohorts/backend/backfill/test/test_management_command.py@@ -18,6 +18,7 @@ from posthog.exceptions import (ClickHouseQueryTimeOut,)+from products.cohorts.backend.backfill.flag_calls_table import FlagCallsTablefrom products.cohorts.backend.backfill.sizing import (BehavioralScanEstimate,PersonSeedEstimate,@@ -30,6 +31,8 @@ from products.cohorts.backend.models.backfill import (CohortBackfillRunStatus,)from products.cohorts.backend.models.cohort import Cohort, CohortType+from products.feature_flags.backend.facade.enums import FlagEvaluationsMode+from products.feature_flags.backend.models.organization_feature_flags_config import OrganizationFeatureFlagsConfig# The catalog drops a leaf with no bytecode or a `conditionHash` that is not 16 characters, and# `_calculate_realtime_support` grants `cohort_type=REALTIME` only when every leaf compiled to@@ -56,7 +59,7 @@ class TestCreateCohortBackfillRunCommand(BaseTest):self.addCleanup(patcher.stop)def _scan_estimate(- self, team_id: int, event_names: list[str], *, max_events_per_day: int+ self, team_id: int, event_names: list[str], *, flag_calls_table: FlagCallsTable, max_events_per_day: int) -> BehavioralScanEstimate:return BehavioralScanEstimate(days_sampled=7,@@ -179,6 +182,45 @@ class TestCreateCohortBackfillRunCommand(BaseTest):[selected.id],)+ # The batch calculation swaps flag calls to flag_evaluations in READ_FLAG_EVALUATIONS mode only on the legacy+ # events table.+ @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)+ def test_the_scan_estimate_counts_flag_calls_from_the_table_the_run_pins(self) -> None:+ OrganizationFeatureFlagsConfig.objects.update_or_create(+ organization=self.organization,+ defaults={"flag_evaluations_mode": FlagEvaluationsMode.READ_FLAG_EVALUATIONS},+ )+ Cohort.objects.create(+ team=self.team,+ cohort_type=CohortType.REALTIME,+ filters={+ "properties": {+ "type": "AND",+ "values": [+ {+ "type": "behavioral",+ "key": "$feature_flag_called",+ "event_type": "events",+ "value": "performed_event",+ "conditionHash": "flagcall00000001",+ "time_value": 7,+ "time_interval": "day",+ "event_filters": [+ {"key": "$feature_flag", "type": "event", "value": "beta-feature", "operator": "exact"}+ ],+ "bytecode": _BYTECODE,+ }+ ],+ }+ },+ )++ call_command("create_cohort_backfill_run", team_id=self.team.id, trigger="team_enablement")++ run = CohortBackfillRun.objects.for_team(self.team.id).get()+ self.assertEqual(run.pinned["flag_calls_table"], FlagCallsTable.FLAG_EVALUATIONS)+ self.assertEqual(self.scan_estimate.call_args.kwargs["flag_calls_table"], FlagCallsTable.FLAG_EVALUATIONS)+def test_disaster_recovery_boundary_is_persisted_without_promoting_the_run(self) -> None:self._cohort("$pageview")diff --git a/products/cohorts/backend/backfill/test/test_runs.py b/products/cohorts/backend/backfill/test/test_runs.pyindex 9eb23f8ef318..9319237554f3 100644--- a/products/cohorts/backend/backfill/test/test_runs.py+++ b/products/cohorts/backend/backfill/test/test_runs.py@@ -4,11 +4,13 @@ from datetime import UTC, datetime, timedeltafrom posthog.test.base import BaseTestfrom unittest import mock-from django.db import IntegrityError, transaction+from django.db import IntegrityError, OperationalError, transactionfrom django.test import override_settingsfrom parameterized import parameterized+from posthog.hogql_queries.hogql_cohort_query import FlagCallReads+from products.cohorts.backend.backfill.pinning import PersonPinningCapExceededfrom products.cohorts.backend.backfill.runs import (BackfillRefusalReason,@@ -32,6 +34,8 @@ from products.cohorts.backend.models.backfill import (CohortBackfillScope,)from products.cohorts.backend.models.cohort import Cohort, CohortType+from products.feature_flags.backend.facade.enums import FlagEvaluationsMode+from products.feature_flags.backend.models.organization_feature_flags_config import OrganizationFeatureFlagsConfig# The catalog drops a leaf with no bytecode or a `conditionHash` that is not 16 characters, and# `_calculate_realtime_support` grants `cohort_type=REALTIME` only when every leaf compiled to@@ -317,6 +321,219 @@ class TestBackfillRuns(BaseTest):self.assertIsNotNone(create_backfill_run_for_cohort(self.team.id, cohort.id, "cohort_edited"))+_FLAG_KEY_FILTER = {"key": "$feature_flag", "type": "event", "value": "beta-feature", "operator": "exact"}+_ANY_FLAG_CALL_LEAF = {+ "type": "behavioral",+ "key": "$feature_flag_called",+ "event_type": "events",+ "value": "performed_event",+ "conditionHash": "flagcall00000001",+ "time_value": 30,+ "time_interval": "day",+ "bytecode": _BYTECODE,+}+_FLAG_CALL_LEAF = {**_ANY_FLAG_CALL_LEAF, "event_filters": [_FLAG_KEY_FILTER]}+_PAGEVIEW_LEAF = {+ "type": "behavioral",+ "key": "$pageview",+ "event_type": "events",+ "value": "performed_event",+ "conditionHash": "pageview00000001",+ "time_value": 7,+ "time_interval": "day",+ "bytecode": _BYTECODE,+}+_UNPARSEABLE_FILTER = {"key": "properties.x = (", "type": "hogql"}+_TEN_DAYS_AGO = (datetime.now(UTC) - timedelta(days=10)).strftime("%Y-%m-%d")+_120_DAYS_AGO = (datetime.now(UTC) - timedelta(days=120)).strftime("%Y-%m-%d")+_FLAG_CALL_TABLE_CASES = [+ ("swapped", (_FLAG_CALL_LEAF,), FlagEvaluationsMode.READ_FLAG_EVALUATIONS, "flag_evaluations"),+ ("events_mode", (_FLAG_CALL_LEAF,), FlagEvaluationsMode.EVENTS, "events"),+ # The run can only read events here, so a batch query that fails to compile does not refuse it.+ (+ "events_mode_batch_query_does_not_compile",+ ({**_FLAG_CALL_LEAF, "event_filters": [_FLAG_KEY_FILTER, _UNPARSEABLE_FILTER]},),+ FlagEvaluationsMode.EVENTS,+ "events",+ ),+ ("no_flag_key", (_ANY_FLAG_CALL_LEAF,), FlagEvaluationsMode.READ_FLAG_EVALUATIONS, "events"),+ (+ "window_past_coverage",+ ({**_FLAG_CALL_LEAF, "time_value": 120},),+ FlagEvaluationsMode.READ_FLAG_EVALUATIONS,+ "events",+ ),+ # The batch calculation swaps this range today. The range does not move with the run's boundary. Its first day+ # can leave flag_evaluations before the seeder scans it.+ (+ "fixed_range",+ ({**_FLAG_CALL_LEAF, "explicit_datetime": _TEN_DAYS_AGO},),+ FlagEvaluationsMode.READ_FLAG_EVALUATIONS,+ "events",+ ),+ (+ "only_in_flag_evaluations",+ (_ANY_FLAG_CALL_LEAF,),+ FlagEvaluationsMode.FLAG_EVALUATIONS_ONLY,+ "flag_evaluations",+ ),+ # events holds no new flag calls in this mode. flag_evaluations is the only table that has any of them.+ (+ "fixed_range_only_in_flag_evaluations",+ ({**_FLAG_CALL_LEAF, "explicit_datetime": _TEN_DAYS_AGO},),+ FlagEvaluationsMode.FLAG_EVALUATIONS_ONLY,+ "flag_evaluations",+ ),+ ("no_flag_calls", (_PAGEVIEW_LEAF,), FlagEvaluationsMode.READ_FLAG_EVALUATIONS, "events"),+ (+ "sliding_and_fixed_range_in_one_cohort",+ (_FLAG_CALL_LEAF, {**_FLAG_CALL_LEAF, "conditionHash": "flagcall00000002", "explicit_datetime": _TEN_DAYS_AGO}),+ FlagEvaluationsMode.READ_FLAG_EVALUATIONS,+ "events",+ ),+]+++# The batch calculation swaps flag calls to flag_evaluations in READ_FLAG_EVALUATIONS mode only on the legacy events+# table. These cases pin that table, so they do not depend on which events schema CI runs.+@override_settings(+ REALTIME_COHORT_TEAM_ALLOWLIST="all",+ BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED=True,+ BEHAVIORAL_BACKFILL_DURABILITY_ATTESTED=True,+ CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False,+)+class TestBackfillRunFlagCallsTable(BaseTest):+ def _cohort(self, *leaves: dict) -> Cohort:+ return Cohort.objects.create(+ team=self.team,+ name="flag callers",+ cohort_type=CohortType.REALTIME,+ filters={"properties": {"type": "AND", "values": list(leaves)}},+ )++ def _set_flag_evaluations_mode(self, mode: FlagEvaluationsMode) -> None:+ OrganizationFeatureFlagsConfig.objects.update_or_create(+ organization=self.organization, defaults={"flag_evaluations_mode": mode}+ )++ @parameterized.expand(+ [+ (f"{scope}_{name}", scope, leaves, mode, expected)+ for scope in ("cohort", "team")+ for name, leaves, mode, expected in _FLAG_CALL_TABLE_CASES+ ]+ )+ def test_run_pins_the_table_the_batch_calculation_reads_flag_calls_from(+ self,+ _name: str,+ scope: str,+ leaves: tuple[dict, ...],+ mode: FlagEvaluationsMode,+ expected: str,+ ) -> None:+ self._set_flag_evaluations_mode(mode)+ cohort = self._cohort(*leaves)++ if scope == "team":+ run: CohortBackfillRun | None = create_team_backfill_run(self.team.id, "team_enablement")+ else:+ run = create_backfill_run_for_cohort(self.team.id, cohort.id, "cohort_created")++ assert run is not None+ # cohort-seeder reads a missing key as events. Every run must state the table.+ self.assertEqual(run.pinned["flag_calls_table"], expected)+ self.assertEqual(run.pinned["schema_version"], 1)++ @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=True)+ def test_run_pins_events_when_the_batch_calculation_reads_the_native_json_events_table(self) -> None:+ # The batch calculation keeps reading events_json in this mode, so the seeder must read events too.+ self._set_flag_evaluations_mode(FlagEvaluationsMode.READ_FLAG_EVALUATIONS)+ cohort = self._cohort(_FLAG_CALL_LEAF)++ run = create_backfill_run_for_cohort(self.team.id, cohort.id, "cohort_created")++ assert run is not None+ self.assertEqual(run.pinned["flag_calls_table"], "events")++ @parameterized.expand(+ [+ ("with_a_cohort_without_flag_calls", _PAGEVIEW_LEAF, "flag_evaluations"),+ ("with_a_flag_call_cohort_that_reads_events", _ANY_FLAG_CALL_LEAF, "events"),+ ]+ )+ def test_team_run_pins_flag_evaluations_only_when_every_flag_call_cohort_reads_it(+ self, _name: str, other_leaf: dict, expected: str+ ) -> None:+ self._set_flag_evaluations_mode(FlagEvaluationsMode.READ_FLAG_EVALUATIONS)+ self._cohort(_FLAG_CALL_LEAF)+ self._cohort({**other_leaf, "conditionHash": "othercohort00001"})++ run = create_team_backfill_run(self.team.id, "team_enablement")++ self.assertEqual(run.pinned["flag_calls_table"], expected)++ @parameterized.expand(+ [+ (+ "batch_query_does_not_compile",+ {+ **_FLAG_CALL_LEAF,+ "event_filters": [_FLAG_KEY_FILTER, {"key": "goe_0.properties.plan = 'pro'", "type": "hogql"}],+ },+ ),+ # A placeholder fails in the HogQL VM, outside the compile errors the guard names.+ (+ "batch_query_raises_outside_the_compile_errors",+ {**_FLAG_CALL_LEAF, "event_filters": [_FLAG_KEY_FILTER, {"key": "{x}", "type": "hogql"}]},+ ),+ # The seeder fails a run that scans a day older than the flag_evaluations TTL.+ ("sliding_window_past_the_ttl", {**_FLAG_CALL_LEAF, "time_value": 120}),+ ("fixed_range_past_the_ttl", {**_FLAG_CALL_LEAF, "explicit_datetime": _120_DAYS_AGO}),+ ]+ )+ def test_a_flag_call_cohort_that_no_table_can_backfill_gets_no_run(self, _name: str, leaf: dict) -> None:+ # Pinning events instead would give the cohort an empty flag-call history that still counts as backfilled.+ self._set_flag_evaluations_mode(FlagEvaluationsMode.FLAG_EVALUATIONS_ONLY)+ cohort = self._cohort(leaf)++ attempt = attempt_backfill_run_for_cohort(self.team.id, cohort.id, "cohort_created")+ with self.assertRaisesMessage(ValueError, f"cohort {cohort.id}"):+ create_team_backfill_run(self.team.id, "team_enablement")++ self.assertEqual(attempt.reason, BackfillRefusalReason.FLAG_CALLS_TABLE_UNRESOLVED)+ self.assertEqual(CohortBackfillRun.objects.for_team(self.team.id).count(), 0)++ def test_a_flag_call_cohort_that_still_reads_events_when_events_has_no_new_flag_calls_gets_no_run(self) -> None:+ # A scan the rewrite cannot swap reads a table that holds no new flag calls, so pinning events there would+ # give the cohort an empty flag-call history that still counts as backfilled.+ self._set_flag_evaluations_mode(FlagEvaluationsMode.FLAG_EVALUATIONS_ONLY)+ cohort = self._cohort(_FLAG_CALL_LEAF)++ with mock.patch(+ "posthog.hogql_queries.hogql_cohort_query.batch_flag_call_reads",+ return_value=FlagCallReads(only_in_flag_evaluations=True, reads_from_events=1),+ ):+ attempt = attempt_backfill_run_for_cohort(self.team.id, cohort.id, "cohort_created")++ self.assertEqual(attempt.reason, BackfillRefusalReason.FLAG_CALLS_TABLE_UNRESOLVED)++ @parameterized.expand(+ [+ ("database_timeout", OperationalError("canceling statement due to statement timeout")),+ ("connection_failure", ConnectionError("connection reset by peer")),+ ]+ )+ def test_an_error_that_may_pass_on_a_retry_reaches_the_caller(self, _name: str, error: Exception) -> None:+ self._set_flag_evaluations_mode(FlagEvaluationsMode.FLAG_EVALUATIONS_ONLY)+ cohort = self._cohort(_FLAG_CALL_LEAF)++ with mock.patch("posthog.hogql_queries.hogql_cohort_query.batch_flag_call_reads", side_effect=error):+ with self.assertRaises(type(error)):+ attempt_backfill_run_for_cohort(self.team.id, cohort.id, "cohort_created")++ self.assertEqual(CohortBackfillRun.objects.for_team(self.team.id).count(), 0)++@override_settings(REALTIME_COHORT_TEAM_ALLOWLIST="all",BEHAVIORAL_BACKFILL_MERGE_GATE_ATTESTED=True,diff --git a/products/cohorts/backend/backfill/test/test_sizing.py b/products/cohorts/backend/backfill/test/test_sizing.pyindex 17a6010c404c..039547c809e0 100644--- a/products/cohorts/backend/backfill/test/test_sizing.py+++ b/products/cohorts/backend/backfill/test/test_sizing.py@@ -11,6 +11,7 @@ from posthog.clickhouse.workload import Workloadfrom posthog.errors import InternalCHQueryErrorfrom posthog.exceptions import ClickHouseQueryTimeOut+from products.cohorts.backend.backfill.flag_calls_table import FlagCallsTablefrom products.cohorts.backend.backfill.sizing import (PersonSeedEstimateScanCapExceeded,estimate_behavioral_scan_events,@@ -105,7 +106,9 @@ class TestBehavioralScanEstimate(SimpleTestCase):(date(2026, 9, 28), "signup", 30),]- estimate = estimate_behavioral_scan_events(7, ["$pageview", "signup"], max_events_per_day=50)+ estimate = estimate_behavioral_scan_events(+ 7, ["$pageview", "signup"], flag_calls_table=FlagCallsTable.EVENTS, max_events_per_day=50+ )self.assertEqual(estimate.peak_day, date(2026, 9, 28))self.assertEqual(estimate.peak_day_events, 60)@@ -125,8 +128,39 @@ class TestBehavioralScanEstimate(SimpleTestCase):@mock.patch("products.cohorts.backend.backfill.sizing.sync_execute")def test_no_event_names_skip_the_query(self, sync_execute: mock.Mock) -> None:- estimate = estimate_behavioral_scan_events(7, [], max_events_per_day=50)+ estimate = estimate_behavioral_scan_events(7, [], flag_calls_table=FlagCallsTable.EVENTS, max_events_per_day=50)sync_execute.assert_not_called()self.assertIsNone(estimate.peak_day)self.assertFalse(estimate.over_limit)++ @parameterized.expand(+ [+ ("events", FlagCallsTable.EVENTS, {"events": ["$feature_flag_called", "$pageview"]}),+ (+ "flag_evaluations",+ FlagCallsTable.FLAG_EVALUATIONS,+ {"events": ["$pageview"], "flag_evaluations": ["$feature_flag_called"]},+ ),+ ]+ )+ @time_machine.travel(datetime(2026, 9, 29, 15, 30, tzinfo=UTC), tick=False)+ def test_flag_calls_are_counted_from_the_table_the_run_scans(+ self, _name: str, flag_calls_table: FlagCallsTable, expected_reads: dict[str, list[str]]+ ) -> None:+ reads: dict[str, list[str]] = {}++ def execute(query: str, params: dict, **kwargs: object) -> list[tuple]:+ table = "flag_evaluations" if "FROM flag_evaluations" in query else "events"+ reads[table] = sorted(params.get("event_names", ["$feature_flag_called"]))+ if table == "flag_evaluations" or "$feature_flag_called" in params["event_names"]:+ return [(date(2026, 9, 28), "$feature_flag_called", 70)]+ return [(date(2026, 9, 28), "$pageview", 30)]++ with mock.patch("products.cohorts.backend.backfill.sizing.sync_execute", side_effect=execute):+ estimate = estimate_behavioral_scan_events(+ 7, ["$pageview", "$feature_flag_called"], flag_calls_table=flag_calls_table, max_events_per_day=50+ )++ self.assertEqual(reads, expected_reads)+ self.assertEqual(estimate.peak_day_events_by_name["$feature_flag_called"], 70)diff --git a/products/cohorts/backend/management/commands/create_cohort_backfill_run.py b/products/cohorts/backend/management/commands/create_cohort_backfill_run.pyindex 7967018d6138..3681f3f67233 100644--- a/products/cohorts/backend/management/commands/create_cohort_backfill_run.py+++ b/products/cohorts/backend/management/commands/create_cohort_backfill_run.py@@ -17,9 +17,10 @@ from posthog.exceptions import ()from posthog.models.team.team import Team+from products.cohorts.backend.backfill.flag_calls_table import FlagCallsTableUnresolvedfrom products.cohorts.backend.backfill.pinning import (PersonPinningCapExceeded,- pin_conditions_for_cohorts,+ pin_behavioral_run,pin_person_conditions_for_cohorts,)from products.cohorts.backend.backfill.runs import (@@ -129,14 +130,14 @@ class Command(BaseCommand):cohort_ids = options.get("cohort_ids")if options["dry_run"]:cohorts = self._dry_run_cohorts(team_id, cohort_ids, behavioral_backfill_ineligibility_reason, "behavioral")- pinned, event_names = pin_conditions_for_cohorts(cohorts)+ pinned = self._pin(team_id, cohorts)self.stdout.write(f"Dry run: {len(cohorts)} cohorts, {len(pinned['conditions'])} conditions, "- f"{len(event_names)} event names"+ f"{len(pinned['event_names'])} event names, flag calls from {pinned['flag_calls_table']}")if max_scan_events_per_day:self._write_scan_estimate(- self._scan_estimate(team_id, event_names, max_scan_events_per_day), len(event_names)+ self._scan_estimate(team_id, pinned, max_scan_events_per_day), len(pinned["event_names"]))else:self.stdout.write("Scan estimate: skipped, because the limit is 0")@@ -151,9 +152,9 @@ class Command(BaseCommand):for cohort, reason in judge_team_cohorts(team_id, cohort_ids, behavioral_backfill_ineligibility_reason)if reason is None]- _, event_names = pin_conditions_for_cohorts(eligible)- scan_estimate = self._scan_estimate(team_id, event_names, max_scan_events_per_day)- self._write_scan_estimate(scan_estimate, len(event_names))+ pinned = self._pin(team_id, eligible)+ scan_estimate = self._scan_estimate(team_id, pinned, max_scan_events_per_day)+ self._write_scan_estimate(scan_estimate, len(pinned["event_names"]))if scan_estimate.over_limit:raise CommandError(f"The run would scan {scan_estimate.peak_day_events} events on its busiest day "@@ -271,9 +272,20 @@ class Command(BaseCommand):f"would refuse: {verdict}")- def _scan_estimate(self, team_id: int, event_names: list[str], max_events_per_day: int) -> BehavioralScanEstimate:+ def _pin(self, team_id: int, cohorts: list[Cohort]) -> dict[str, Any]:try:- return estimate_behavioral_scan_events(team_id, event_names, max_events_per_day=max_events_per_day)+ return pin_behavioral_run(cohorts, Team.objects.get(id=team_id))+ except (Team.DoesNotExist, FlagCallsTableUnresolved) as error:+ raise CommandError(str(error)) from error++ def _scan_estimate(self, team_id: int, pinned: dict[str, Any], max_events_per_day: int) -> BehavioralScanEstimate:+ try:+ return estimate_behavioral_scan_events(+ team_id,+ pinned["event_names"],+ flag_calls_table=pinned["flag_calls_table"],+ max_events_per_day=max_events_per_day,+ )except (ClickHouseQueryTimeOut,ClickHouseEstimatedQueryExecutionTimeTooLong,