:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(flags): match flag_evaluations rows with no events copy by predicate (#117395)
Phil Haack7a4fd5ac5a1f
diff --git a/docs/internal/clickhouse-deletion-coverage.md b/docs/internal/clickhouse-deletion-coverage.mdindex ab2443618487..58f40fe5abf7 100644--- a/docs/internal/clickhouse-deletion-coverage.md+++ b/docs/internal/clickhouse-deletion-coverage.md@@ -95,7 +95,7 @@ Leaving one out stays possible, and `PERSON_ID_REWRITE_EXEMPT` is where that dec- `sharded_events` — all sweeps.- `sharded_events_json` — all sweeps, on the events cluster, through patch parts instead of mutations (`uses_patch_parts`). Optional: only present after the native-JSON migration.-- `sharded_flag_evaluations` — person, team, queued-uuid, immediate and deferred event removal, and property removal of event and person properties (below). Immediate event removal skips it when the request carries a HogQL predicate. Deferred event removal with a predicate reaches only the rows that have an events copy (below). Property removal with a predicate does not sweep it. Optional.+- `sharded_flag_evaluations` — person, team, queued-uuid, immediate and deferred event removal, and property removal of event and person properties (below). Immediate event removal with a HogQL predicate does not sweep it, and refuses while a matching row has no events copy. Deferred event removal with a predicate reaches the rows that have an events copy through their uuids, and the rest through the predicate compiled for this table (below). Property removal with a predicate does not sweep it. Optional.- `sharded_posthog_document_embeddings_<model>` — event and team deletion, through `delete_event_documents`. An embedded document is keyed by the id of the thing it describes (`document_id`), and an Event deletion's key is that same id, so the pending dictionary is joined on `(team_id, Event, document_id)`. Every per-model table listed by the error tracking facade's `document_embedding_tables` is swept and counted.Native property-removal requests fail when the selected rows retain a requested permanent or temporary property, or a matching person `$set`/`$set_once` instruction.@@ -192,7 +192,7 @@ Each expired row that drops took its property with it.#### HogQL predicates-`compile_hogql_predicate` compiles against the events schema only (see the HogQL section below), so the job cannot apply a request's predicate to this table.+Property removal compiles a request's predicate against the events schema only. Event removal compiles one for this table's rows with no events copy (see the HogQL section below), but property removal does not use it, so the job cannot apply a request's predicate to this table.A property removal with a predicate leaves `flag_evaluations` out of the sweep, and `get_property_removal_shards` and `verify_property_removal` refuse the request while the table holds rows that match its other criteria.Sweeping the table without the predicate would erase rows the predicate excludes.@@ -213,49 +213,79 @@ The copy carries every other column through unchanged.`flag_evaluations` stays absent from `MATERIALIZATION_VALID_TABLES`.The job finds typed columns through their comments in `system.columns`, not through that set, and adding the table there enrolls it in `materialize()` and the HogQL property planner, which is a separate change.-### Immediate event removal with a HogQL predicate skips `flag_evaluations`+### Immediate event removal with a HogQL predicate does not sweep `flag_evaluations`-`get_event_removal_shards` leaves `flag_evaluations` out of an immediate request that carries a HogQL predicate, so the request neither sweeps the table nor checks it for matching rows.-The rows age out with the table's TTL. The table partitions by month with `ttl_only_drop_parts = 1`, so a part drops only once its newest row expires: the real wait is up to about 120 days, not the 90-day TTL.-The skip exists because of the HogQL gap below: without it, the gate refuses every immediate request with a predicate whose team has matching `$feature_flag_called` rows.+`get_event_removal_shards` leaves `flag_evaluations` out of the sweep for an immediate request that carries a HogQL predicate.+A row with an events copy ages out with the table's TTL. The table partitions by month with `ttl_only_drop_parts = 1`, so a part drops only once its newest row expires: the real wait is up to about 120 days, not the 90-day TTL.The events copies that a deferred request deletes through do not help here, because the immediate path deletes the events rows without queueing their uuids.+A row with no events copy is checked instead.+The request refuses while such a row matches the predicate compiled for this table (below), or matches the request's other criteria when the predicate cannot run on this table.+That turns a request from an org on `flag_evaluations_mode` 2, whose flag calls have no events copy, into a refusal rather than a completed request that leaves every flag call behind.+Re-filing the request as deferred deletes those rows.+A fresh run of a request whose earlier run already deleted events also refuses, because those events were the copies of matching rows.+The admin Retry button starts such a fresh run.+"Re-execute from failure" on the earlier Dagster run finishes the request, because it reuses the output of `get_event_removal_shards` and does not run the check again.An immediate request without a predicate sweeps the table and verifies it like the events tables, because its delete predicate names only `team_id`, `timestamp` and `event`.-### Event removal with a HogQL predicate reaches `flag_evaluations` only through events copies+### Event removal with a HogQL predicate on `flag_evaluations`-`compile_hogql_predicate` resolves every predicate against the events HogQL table and emits events-specific physical columns.-Its only axis of variation is legacy vs native-JSON events, so while the dag does compile one fragment per target, no target selects a different table root.-Whether a given fragment would run against `flag_evaluations` depends on the predicate and the team's modifiers: one naming only `event` or `distinct_id` would, one reaching a `mat_*` column or a property-group map would not, and nothing validates which.-The dag does not guess.-It never runs a fragment against the table.-An immediate request with a predicate skips the table (above).-A HogQL table definition for `flag_evaluations` does not change that, because nothing routes compilation to a table.--A deferred request with a predicate reaches the table through `events` instead.A flag-evaluation row is a fork of its `$feature_flag_called` event, with the same uuid, and both tables shard on `sipHash64(distinct_id)`, so the two copies share a shard.`DeletionTarget.copied_from` records that relationship.-The queue fill reads uuids from `sharded_events` with the full predicate, and the drain deletes those uuids from every target, so it deletes the flag row of every event the predicate names.+A deferred request with a predicate reaches a row that has an events copy through `events`.+The queue fill reads uuids from `sharded_events` with the events-compiled predicate, and the drain deletes those uuids from every target, so it deletes the flag row of every event the predicate names.A flag row whose event the predicate does not name keeps its events copy and stays.-That only works for a row that has an events copy.+A row with no events copy needs the predicate itself.An org on `flag_evaluations_mode` 2 stores its flag calls only in `flag_evaluations`, and a deletion that skipped the table can leave a flag row whose event is gone.-The gate therefore counts the matching rows that have no copy in `sharded_events`, shard-locally, and refuses the request when it finds any.+`DeletionTarget.uncopied_rows_hogql_table` names the HogQL table that compiles the predicate for those rows, `posthog.flag_evaluations` here.+The compile adds that table even when the org flag hides it from queries.+The queue fill reads the uuids of the rows with no copy in `sharded_events` that the compiled predicate matches, shard-locally and in 7-day windows, and queues them with the rest.+A row read this way already existed when its uuid was queued, so the drain's `inserted_at` bound cannot skip it.+The check for an events copy reads `sharded_events` only.+In production, `sharded_events_json` holds no `$feature_flag_called` rows: the deployed ingestion view and the backfill both leave the event out, so a flag row has no copy there.+The view this repo defines (`EVENTS_JSON_TABLE_MV_SQL`) still writes the event, so local, CI and self-hosted deployments do keep a copy.+There, a flag row can keep its `sharded_events_json` copy after its `sharded_events` copy is gone, when an earlier immediate request matched the two events tables differently.+Queueing that row would also delete its `sharded_events_json` copy, even when the native-JSON compile of the predicate excludes it, because the drain deletes a queued uuid from every target.+The check cannot look for that copy row by row, because `sharded_events_json` can sit on another cluster, where a shard-local subquery cannot reach it.+So before a deferred request queues rows with no `sharded_events` copy, the gate counts the flag calls in the request's window on `sharded_events_json`.+It counts through the `events_json` proxy, and also on the storage table through its own handle when that table sits on another cluster, because the proxy can read an empty table there.+When it finds any, the request refuses while a row with no `sharded_events` copy matches the compiled predicate.+Production stores no flag calls in `sharded_events_json`, so the gate finds none there.++The compiled predicate never decides for a row that has an events copy.+The drain deletes a queued uuid from every target, so a flag row that matched the flag-compiled predicate would take its events copy with it, even where the events-compiled predicate does not name that event.+The two compiles can disagree.+On cloud, `propertyGroupsMode` is `optimized`, and events reads a custom property, an `$ai_` property or a `$feature/` property from a property-group map such as `properties_group_custom`.+The map is built with `JSONExtractKeysAndValues(properties, 'String')`, which decodes JSON escapes, so a stored `"a\"b"` reads as `a"b`.+For `=`, `!=` and `IN`, the `optimized` rewrite also compares a typed property as a string.+`flag_evaluations` has no property-group map, so it reads the raw JSON with the surrounding quotes stripped, which reads `a\"b`, and it casts a typed property to its defined type.+On cloud, events reads some `$` properties from `mat_` or `dmat_string_` columns instead, and those differ on edge values: the column reads an empty string and the string `"null"` as NULL, and a nullable column decodes escapes.+So with `$browser` materialized, `properties.$browser = ''` excludes a row that stores `""` on events and matches it on `flag_evaluations`.+The compile therefore refuses a predicate whenever the events compile of it reads a property-group map or a materialized column.+Every property the compile accepts reads raw JSON on both tables.+It also refuses one that needs a join, such as `person_id`, because the fragment is spliced into a bare WHERE that has no join.+`person.properties` and `elements_chain` do not resolve on the table, so those predicates fail to compile.++When the predicate cannot run on the table, the gate counts the rows with no copy in `sharded_events` that match the request's other criteria, shard-locally, and refuses the request when it finds any.Requests without a predicate, immediate or deferred, are swept normally.A deferred one reads the table's own uuids into the queue.-Verification of a deferred request with a predicate, when the table holds rows in the request's window, makes two counts: the request's queue rows that the drain has not marked deleted, and the table's rows with no copy in `sharded_events`.+Verification of a deferred request with a predicate, when the table holds rows in the request's window, makes two counts.+The first is the request's queue rows that the drain has not marked deleted.+The second is the table's rows with no copy in `sharded_events` that the compiled predicate matches, or that match the request's other criteria when the predicate cannot run on the table.The queue rows hold the request until `deletes_job` drains them, including a queued row whose event a property removal changed so that the predicate no longer matches it.The rows with no copy hold the request after the drain.-A flag row stamped after its uuid was queued falls outside the drain's `inserted_at` bound, so it survives the drain.+A flag row stamped after its events copy's uuid was queued falls outside the drain's `inserted_at` bound, so it survives the drain.The drain still deletes its events copy.The rows with no copy therefore include it and keep the request in QUEUED.A re-filed request does not reach that row.-The gate refuses one with the predicate, because the row has no events copy.-A deferred one without the predicate queues the uuid, but the drain skips it while the first drain's `is_deleted` marker lasts, which is 3 months.+A deferred one queues the uuid, but the drain skips it while the first drain's `is_deleted` marker lasts, which is 3 months.Only an immediate request without the predicate deletes the row, and it also deletes every other row in its window that its event filter names.The verify sweep therefore keeps checking a deferred request with a predicate for `lookback_days` after its window falls behind the table's TTL horizon, so the request moves to COMPLETED once the TTL drops the row.The rows with no copy also keep a request the gate refused, which queued nothing, from passing verification as a FAILED request.Counting every row that matches the request's other criteria would hold the request in QUEUED until the TTL, whenever the team has flag calls the predicate does not name.+The second count uses the same compiled predicate as the queue fill, so it cannot see the two compiles disagreeing.+The compile's refusals are the guard against that.## Producer prerequisite: person_id paritydiff --git a/posthog/dags/data_deletion_requests.py b/posthog/dags/data_deletion_requests.pyindex 886e628dcfc2..83ae1f462609 100644--- a/posthog/dags/data_deletion_requests.py+++ b/posthog/dags/data_deletion_requests.py@@ -1,7 +1,7 @@import jsonimport timeimport logging-from collections.abc import Callable, Collection, Iterator+from collections.abc import Callable, Iteratorfrom dataclasses import dataclass, fieldfrom datetime import UTC, datetime, timedeltafrom functools import partial@@ -32,6 +32,7 @@ from posthog.clickhouse.client.connection import ClickHouseUser, NodeRolefrom posthog.clickhouse.cluster import (ClickhouseCluster,LightweightDeleteMutationRunner,+ Query,wait_for_patch_part_replication,)from posthog.clickhouse.events_json import UNPARSEABLE_PROPERTIES_KEY@@ -47,6 +48,7 @@ from posthog.models.data_deletion_request import (ExecutionMode,RequestStatus,RequestType,+ UncopiedRowsPredicate,auto_approve_pending_requests,compile_hogql_predicate,event_match_sql_fragment,@@ -55,12 +57,11 @@ from posthog.models.data_deletion_request import (portable_event_removal_where,queued_requests_to_verify,reaches_target_through_event_uuids,- rows_without_events_copy_criteria,+ uncopied_rows_predicate,verify_queued_request,)from posthog.models.deletion_targets import (COVERAGE_DOC,- FLAG_EVALUATIONS,PERSONAL_DATA_TARGETS,DeletionTarget,NotCounted,@@ -529,21 +530,43 @@ def execute_hogql_event_deletion(_HOGQL_UNSWEEPABLE_REASON = (- "the request carries a HogQL predicate, which only compiles against the events schema "- "(compile_hogql_predicate resolves every predicate against the events HogQL table, varying only "- "legacy vs native-JSON, and nothing checks the result against this table's columns). "+ "the request carries a HogQL predicate, and this job compiles it against the events schema only, "+ "so it cannot apply the predicate to this table. ""To proceed, re-file the request without the predicate, or narrow its events to ones this "f"table never stores. See {COVERAGE_DOC}.")-_HOGQL_NO_EVENTS_COPY_REASON = (- "the request carries a HogQL predicate, which only compiles against the events schema, and these "- "rows have no copy in the events table. The deferred queue deletes rows here through the uuids of "- "their events copies, so it cannot decide whether the predicate names these rows. Such rows belong to an "- "org whose flag calls are stored only in this table, or to events an earlier deletion removed without "- f"these rows. To proceed, re-file the request without the predicate. See {COVERAGE_DOC}."+_IMMEDIATE_HOGQL_NO_EVENTS_COPY_REASON = (+ "the request carries a HogQL predicate and runs immediately, and these rows have no copy in the events "+ "table. An immediate request with a predicate does not delete rows here. To proceed, re-file the request "+ f"as deferred, which queues these rows. See {COVERAGE_DOC}.")+_IMMEDIATE_HOGQL_RETRY_HINT = (+ " If an earlier run of this request already deleted events, finish it with Re-execute from failure on "+ "that Dagster run, which does not run this check again."+)+++def _hogql_unchecked_events_copy_reason(target: DeletionTarget, tables: list[str]) -> str:+ return (+ f"the request carries a HogQL predicate, and these rows have no copy in {target.copied_from}, but "+ f"{', '.join(tables)} holds flag calls in the request's window. The deferred queue deletes a queued uuid "+ "from every table, so queueing these rows could also delete a copy there that the predicate excludes. "+ "This job cannot check those copies row by row. To proceed, re-file the request without the predicate. "+ f"See {COVERAGE_DOC}."+ )+++def _hogql_no_events_copy_reason(error: str) -> str:+ return (+ f"the request carries a HogQL predicate that cannot run on this table ({error}), and these rows have "+ "no copy in the events table, so the predicate cannot reach them through their events copies either. "+ "Such rows belong to an org whose flag calls are stored only in this table, or to events an earlier "+ "deletion removed without these rows. To proceed, re-file the request without the predicate, or as a "+ f"deferred request with a predicate this table can run. See {COVERAGE_DOC}."+ )+def _refuse_unsweepable(cluster: ClickhouseCluster,@@ -636,52 +659,103 @@ def _verify_swept(raise dagster.Failure(description=f"Deletion request {deletion_request.request_id}: {exc}") from exc-def _event_removal_placements(+@frozen+class EventRemovalPlacements:+ """Where an event-removal request reads or deletes rows, settled before it deletes or queues any."""++ # Targets the request's own criteria run on.+ swept: list[TargetPlacement]+ # Targets whose rows with no events copy a deferred request queues, each with its predicate for those rows.+ uncopied: list[tuple[TargetPlacement, UncopiedRowsPredicate]]+++def _holds_rows(cluster: ClickhouseCluster, placement: TargetPlacement, predicate: str, params: dict) -> bool:+ """Whether ``placement``'s target holds rows that match ``predicate``.++ Reads the Distributed proxy, and also the storage table through its own handle when that table sits+ on another cluster. The proxy reads only the cluster its engine names, so on a deployment whose+ storage moved it can read an empty table, as ``_count_unswept_rows`` in posthog/dags/deletes.py notes.+ """+ if count_surviving_rows(cluster, placement.target, predicate, params):+ return True+ if placement.cluster is cluster:+ return False+ query = Query(+ surviving_rows_sql(placement.target.data_table, predicate), params, settings={"max_execution_time": "1800"}+ )+ return any(rows and rows[0][0] for rows in placement.cluster.map_one_host_per_shard(query).result().values())+++def _events_tables_outside_copy_check(cluster: ClickhouseCluster,- deletion_request: DeletionRequestContext,- *,- skip_targets: Collection[DeletionTarget] = (),-) -> list[TargetPlacement]:- """Targets this event-removal request can sweep, each with the handle that reaches it."""+ placements: list[TargetPlacement],+ target: DeletionTarget,+ criteria: tuple[str, dict],+) -> list[str]:+ """The events tables other than ``target.copied_from`` that hold events ``target`` copies, within ``criteria``.++ The copy check reads only ``target.copied_from``, so a row it finds with no copy can still have one+ in another events table. That table can sit on another cluster, where a shard-local subquery cannot+ reach it.+ """+ predicate, params = criteria+ if target.stored_events is not None:+ predicate += " AND event IN %(copied_events)s"+ params = {**params, "copied_events": sorted(target.stored_events)}+ return [+ p.target.data_table+ for p in placements+ if p.target.hogql_schema is not None+ and p.target.data_table != target.copied_from+ and _holds_rows(cluster, p, predicate, params)+ ]+++def _event_removal_placements(+ cluster: ClickhouseCluster, deletion_request: DeletionRequestContext+) -> EventRemovalPlacements:+ """Where this event-removal request reads or deletes rows, each target with the handle that reaches it."""events = [] if deletion_request.delete_all_events else deletion_request.events# A target that can't hold any of the named events has nothing to sweep, and mutations serialize# per table, so enqueueing a no-op one would queue in front of real work.- # Skipped targets are dropped before resolve_placements, which raises for an unreachable target- # that still holds rows.- targets = [t for t in PERSONAL_DATA_TARGETS if t not in skip_targets]- placements = [p for p in resolve_placements(cluster, targets) if p.target.may_hold_any_of(events)]+ placements = [p for p in resolve_placements(cluster) if p.target.may_hold_any_of(events)]if not deletion_request.hogql_predicate:- return placements+ return EventRemovalPlacements(swept=placements, uncopied=[])- unsweepable = [p.target for p in placements if not p.target.accepts_hogql_predicate]+ unsweepable = [p for p in placements if not p.target.accepts_hogql_predicate]criteria = portable_event_removal_where(deletion_request)- reachable = [t for t in unsweepable if reaches_target_through_event_uuids(deletion_request, t)]-_refuse_unsweepable(cluster,- [t for t in unsweepable if t not in reachable],+ [p.target for p in unsweepable if p.target.copied_from is None],deletion_request,_same_criteria(criteria),reason=_HOGQL_UNSWEEPABLE_REASON,)- # The copy check reads the uuid of every matching event, so it runs only on tables that hold rows- # in the request's window.- for target in [t for t in reachable if count_surviving_rows(cluster, t, *criteria)]:- for window_criteria in rows_without_events_copy_criteria(deletion_request, target):- _refuse_unsweepable(- cluster,- [target],- deletion_request,- _same_criteria(window_criteria),- reason=_HOGQL_NO_EVENTS_COPY_REASON,- )- return [p for p in placements if p.target.accepts_hogql_predicate]--# An immediate request with a HogQL predicate leaves flag_evaluations rows to the table's TTL. The-# predicate does not compile against that table. Without the skip, the unsweepable-rows gate would-# refuse every such request whose team has matching $feature_flag_called rows.-_IMMEDIATE_HOGQL_SKIP_TARGETS = (FLAG_EVALUATIONS,)+ uncopied: list[tuple[TargetPlacement, UncopiedRowsPredicate]] = []+ for placement in [p for p in unsweepable if p.target.copied_from is not None]:+ target = placement.target+ uncopied_predicate = uncopied_rows_predicate(deletion_request, target)+ # The copy check reads the uuid of every matching event, so it runs only when the target holds+ # matching rows in the request's window. This count also runs the compiled fragment before+ # anything is queued, so a fragment that ClickHouse rejects fails the request with an empty queue.+ if not count_surviving_rows(cluster, target, *uncopied_predicate.narrow(criteria)):+ continue+ deferred = reaches_target_through_event_uuids(deletion_request, target)+ if uncopied_predicate.compiled is None:+ reason = _hogql_no_events_copy_reason(uncopied_predicate.error)+ elif not deferred:+ reason = _IMMEDIATE_HOGQL_NO_EVENTS_COPY_REASON+ elif tables := _events_tables_outside_copy_check(cluster, placements, target, criteria):+ reason = _hogql_unchecked_events_copy_reason(target, tables)+ else:+ uncopied.append((placement, uncopied_predicate))+ continue+ if not deferred:+ reason += _IMMEDIATE_HOGQL_RETRY_HINT+ for window in uncopied_predicate.windows(deletion_request, target):+ _refuse_unsweepable(cluster, [target], deletion_request, _same_criteria(window), reason=reason)+ return EventRemovalPlacements(swept=[p for p in placements if p.target.accepts_hogql_predicate], uncopied=uncopied)@frozen@@ -739,11 +813,12 @@ def _queue_events_for_deferred_deletion(# personal-data table. Flag-evaluation rows still have to be read on their own: they mirror a# subset of events, so a uuid there may not be in sharded_events once routing moves those# events off it.- placements = [p for p in _event_removal_placements(cluster, deletion_request) if p.target.queue_uuid_candidates]+ placements = _event_removal_placements(cluster, deletion_request)+ swept = [p for p in placements.swept if p.target.queue_uuid_candidates]# Both halves of the INSERT are host-local: the source table and the queue it feeds. A source# on another cluster has no host that holds both, and reading it through its Distributed proxy# instead would pull every matching uuid across the wire into one shard's queue.- stranded = [p.target for p in placements if p.cluster is not cluster]+ stranded = [p.target for p in [*swept, *(p for p, _ in placements.uncopied)] if p.cluster is not cluster]if stranded:raise dagster.Failure(description=(@@ -752,25 +827,29 @@ def _queue_events_for_deferred_deletion(f"{cluster.data_cluster_name!r} and those tables are not. See {COVERAGE_DOC}."))- sources = [p.target for p in placements]+ events_where = event_removal_where(deletion_request)+ reads = [(p.target.data_table, *events_where) for p in swept]+ for placement, uncopied_predicate in placements.uncopied:+ reads += [+ (placement.target.data_table, *window)+ for window in uncopied_predicate.windows(deletion_request, placement.target)+ ]db = django_settings.CLICKHOUSE_DATABASEshards = sorted(cluster.shards)- predicate, params = event_removal_where(deletion_request)- params["data_deletion_request_id"] = deletion_request.request_iddef run_on_shard(client: Client) -> int:- for source in sources:+ for data_table, predicate, params in reads:# nosemgrep: clickhouse-fstring-param-audit (all interpolated values are internal constants/settings)client.execute(f"INSERT INTO {db}.{ADHOC_EVENTS_DELETION_TABLE} (team_id, uuid, data_deletion_request_id) "f"SELECT team_id, uuid, toUUID(%(data_deletion_request_id)s) "- f"FROM {db}.{source.data_table} WHERE {predicate}",- params,+ f"FROM {db}.{data_table} WHERE {predicate}",+ {**params, "data_deletion_request_id": deletion_request.request_id},settings={"max_execution_time": 1800},)row = client.execute(f"SELECT count() FROM {db}.{ADHOC_EVENTS_DELETION_TABLE} WHERE team_id = %(team_id)s AND is_deleted = 0",- {"team_id": params["team_id"]},+ {"team_id": deletion_request.team_id},)return row[0][0] if row else 0@@ -790,7 +869,7 @@ def _queue_events_for_deferred_deletion({"mode": dagster.MetadataValue.text("deferred"),"queued_rows": dagster.MetadataValue.int(total_queued),- "queued_from": dagster.MetadataValue.text(", ".join(s.data_table for s in sources)),+ "queued_from": dagster.MetadataValue.text(", ".join(dict.fromkeys(table for table, *_ in reads))),})@@ -808,8 +887,7 @@ def get_event_removal_shards(if deletion_request.execution_mode == ExecutionMode.DEFERRED.value:return- skip_targets = _IMMEDIATE_HOGQL_SKIP_TARGETS if deletion_request.hogql_predicate else ()- placements = _event_removal_placements(cluster, deletion_request, skip_targets=skip_targets)+ placements = _event_removal_placements(cluster, deletion_request).swept# placement.cluster, not the job's handle: shard numbers are per cluster.shards = [EventRemovalShard(data_table=placement.target.data_table, shard_num=shard_num)diff --git a/posthog/dags/tests/test_data_deletion_requests.py b/posthog/dags/tests/test_data_deletion_requests.pyindex a9297b6876ab..c4999f50338a 100644--- a/posthog/dags/tests/test_data_deletion_requests.py+++ b/posthog/dags/tests/test_data_deletion_requests.py@@ -32,6 +32,7 @@ from posthog.dags.data_deletion_requests import (PersonRemovalContext,PropertyRemovalTarget,_cleaned_select_list,+ _holds_rows,_presence_params,_property_removal_where,_refuse_property_removal_unsweepable,@@ -172,6 +173,19 @@ def _truncate_adhoc_events_deletion(client: Client) -> None:client.execute(f"TRUNCATE TABLE IF EXISTS {ADHOC_EVENTS_DELETION_TABLE}")+def _insert_events_json_rows(rows: list[tuple], client: Client) -> None:+ for team_id, event, uuid, timestamp, properties in rows:+ client.execute(+ "INSERT INTO sharded_events_json (team_id, event, uuid, timestamp, properties) "+ "SELECT %(team_id)s, %(event)s, %(uuid)s, %(timestamp)s, %(properties)s",+ {"team_id": team_id, "event": event, "uuid": uuid, "timestamp": timestamp, "properties": properties},+ )+++def _truncate_events_json(client: Client) -> None:+ client.execute("TRUNCATE TABLE IF EXISTS sharded_events_json")++def _adhoc_pending_uuids(team_id: int, client: Client) -> set:result = client.execute(f"SELECT uuid FROM {ADHOC_EVENTS_DELETION_TABLE} FINAL WHERE team_id = %(team_id)s AND is_deleted = 0",@@ -2575,11 +2589,16 @@ def _delete_uuid(table: str, uuid: UUID, client: Client) -> None:)-def _deferred_hogql_flag_request(name: str) -> tuple[DataDeletionRequest, DeletionRequestContext]:+def _deferred_hogql_flag_request(+ name: str, hogql_predicate: str = "properties.$browser = 'Chrome'"+) -> tuple[DataDeletionRequest, DeletionRequestContext]:from posthog.models.organization import Organizationfrom posthog.models.team import Team- team = Team.objects.create(organization=Organization.objects.create(name=name), name=name)+ # OPTIMIZED is the cloud setting. It makes events read a non-$ key from a property-group map.+ team = Team.objects.create(+ organization=Organization.objects.create(name=name), name=name, modifiers={"propertyGroupsMode": "optimized"}+ )now = datetime.now(UTC)request = DataDeletionRequest.objects.create(team_id=team.id,@@ -2587,7 +2606,7 @@ def _deferred_hogql_flag_request(name: str) -> tuple[DataDeletionRequest, Deletidelete_all_events=True,start_time=now - timedelta(days=14),end_time=now + timedelta(minutes=1),- hogql_predicate="properties.$browser = 'Chrome'",+ hogql_predicate=hogql_predicate,status=RequestStatus.QUEUED,execution_mode=ExecutionMode.DEFERRED.value,)@@ -2595,24 +2614,37 @@ def _deferred_hogql_flag_request(name: str) -> tuple[DataDeletionRequest, Deleti@pytest.mark.django_db+ "hogql_predicate",+ [+ pytest.param("properties.$browser = 'Chrome'", id="predicate_runs_on_flag_evaluations"),+ pytest.param(+ "properties.$browser = 'Chrome' AND properties.plan = 'pro'",+ id="predicate_cannot_run_on_flag_evaluations",+ ),+ ],+)def test_deferred_event_removal_with_a_hogql_predicate_reaches_flag_evaluations_through_event_uuids(- cluster: ClickhouseCluster,+ cluster: ClickhouseCluster, hogql_predicate: str) -> None:from posthog.models.data_deletion_request import verify_queued_request- request, ctx = _deferred_hogql_flag_request("test-deferred-hogql-flag-copies")+ request, ctx = _deferred_hogql_flag_request("test-deferred-hogql-flag-copies", hogql_predicate)now = datetime.now(UTC)earlier = now - timedelta(days=10)- chrome, firefox, earlier_chrome = uuid4(), uuid4(), uuid4()+ chrome, firefox, earlier_chrome, drifted = uuid4(), uuid4(), uuid4(), uuid4()+ # Both predicates match these properties, so both cases queue the same uuids.+ chrome_pro = '{"$browser": "Chrome", "plan": "pro"}'cluster.any_host(_truncate_adhoc_events_deletion).result()cluster.any_host(_truncate_flag_evaluations).result()cluster.any_host(partial(_insert_events_with_properties,[- (ctx.team_id, FLAG_EVALUATIONS_SOURCE_EVENT, chrome, now, '{"$browser": "Chrome"}'),+ (ctx.team_id, FLAG_EVALUATIONS_SOURCE_EVENT, chrome, now, chrome_pro),(ctx.team_id, FLAG_EVALUATIONS_SOURCE_EVENT, firefox, now, '{"$browser": "Firefox"}'),- (ctx.team_id, FLAG_EVALUATIONS_SOURCE_EVENT, earlier_chrome, earlier, '{"$browser": "Chrome"}'),+ (ctx.team_id, FLAG_EVALUATIONS_SOURCE_EVENT, earlier_chrome, earlier, chrome_pro),+ (ctx.team_id, FLAG_EVALUATIONS_SOURCE_EVENT, drifted, now, '{"$browser": "Firefox"}'),],)).result()@@ -2620,9 +2652,12 @@ def test_deferred_event_removal_with_a_hogql_predicate_reaches_flag_evaluations_partial(_insert_flag_evaluations_with_properties,[- (ctx.team_id, "", '{"$browser": "Chrome"}', str(chrome), now, now),+ (ctx.team_id, "", chrome_pro, str(chrome), now, now),(ctx.team_id, "", '{"$browser": "Firefox"}', str(firefox), now, now),- (ctx.team_id, "", '{"$browser": "Chrome"}', str(earlier_chrome), earlier, earlier),+ (ctx.team_id, "", chrome_pro, str(earlier_chrome), earlier, earlier),+ # The predicate matches this flag row and not its events copy. The drain deletes a queued+ # uuid from every table, so queueing it would also delete an events row the predicate excludes.+ (ctx.team_id, "", chrome_pro, str(drifted), now, now),],)).result()@@ -2665,30 +2700,116 @@ def test_deferred_event_removal_with_a_hogql_predicate_reaches_flag_evaluations_@pytest.mark.django_db-def test_deferred_event_removal_with_a_hogql_predicate_refuses_flag_rows_without_an_event(- cluster: ClickhouseCluster,+ "hogql_predicate, refusal",+ [+ pytest.param("properties.$browser = 'Chrome'", None, id="predicate_runs_on_flag_evaluations"),+ pytest.param(+ "properties.$browser = 'Chrome' AND properties.plan = 'pro'",+ "property-group map",+ id="predicate_cannot_run_on_flag_evaluations",+ ),+ ],+)+def test_deferred_event_removal_with_a_hogql_predicate_handles_flag_rows_without_an_event(+ cluster: ClickhouseCluster, hogql_predicate: str, refusal: str | None) -> None:from posthog.models.data_deletion_request import verify_queued_request- request, ctx = _deferred_hogql_flag_request("test-deferred-hogql-flag-no-copy")+ request, ctx = _deferred_hogql_flag_request("test-deferred-hogql-flag-no-copy", hogql_predicate)earlier = datetime.now(UTC) - timedelta(days=10)+ chrome, firefox = uuid4(), uuid4()+ cluster.any_host(_truncate_adhoc_events_deletion).result()cluster.any_host(_truncate_flag_evaluations).result()cluster.any_host(partial(_insert_flag_evaluations_with_properties,- [(ctx.team_id, "", '{"$browser": "Chrome"}', str(uuid4()), earlier, earlier)],+ [+ (ctx.team_id, "", '{"$browser": "Chrome"}', str(chrome), earlier, earlier),+ (ctx.team_id, "", '{"$browser": "Firefox"}', str(firefox), earlier, earlier),+ ],)).result()try:- with pytest.raises(dagster.Failure, match="have no copy in the events table"):+ with pytest.raises(dagster.Failure, match=refusal) if refusal else nullcontext():_run_event_deletion(cluster, ctx)-- request.status = RequestStatus.FAILED- request.save(update_fields=["status"])+ assert cluster.any_host(partial(_adhoc_pending_uuids, ctx.team_id)).result() == (set() if refusal else {chrome})assert verify_queued_request(request).promoted is False++ # Emptying the queue stands in for the drain marking the queued uuids deleted.+ cluster.any_host(_truncate_adhoc_events_deletion).result()+ cluster.any_host(partial(_delete_uuid, FLAG_EVALUATIONS_DATA_TABLE, chrome)).result()+ # The Firefox row holds the request only when verification cannot narrow by the predicate.+ assert verify_queued_request(request).promoted is (refusal is None)+ finally:+ cluster.any_host(_truncate_adhoc_events_deletion).result()+ cluster.any_host(_truncate_flag_evaluations).result()+++ "has_events_copy, native_properties, refusal",+ [+ # The predicate excludes the native copy. The drain deletes a queued uuid from every table, so+ # queueing the flag row would delete that copy too.+ pytest.param(False, '{"$browser": "Firefox"}', "sharded_events_json holds flag calls", id="only_a_native_copy"),+ pytest.param(True, '{"$browser": "Chrome"}', None, id="events_and_native_copies"),+ ],+)+def test_deferred_event_removal_with_a_hogql_predicate_and_a_native_events_copy(+ cluster: ClickhouseCluster, has_events_copy: bool, native_properties: str, refusal: str | None+) -> None:+ _, ctx = _deferred_hogql_flag_request("test-deferred-hogql-flag-native-copy")+ earlier = datetime.now(UTC) - timedelta(days=10)+ flag_uuid = uuid4()+ cluster.any_host(_truncate_adhoc_events_deletion).result()+ cluster.any_host(_truncate_flag_evaluations).result()+ cluster.any_host(_truncate_events_json).result()+ cluster.any_host(+ partial(+ _insert_flag_evaluations_with_properties,+ [(ctx.team_id, "", '{"$browser": "Chrome"}', str(flag_uuid), earlier, earlier)],+ )+ ).result()+ if has_events_copy:+ cluster.any_host(+ partial(+ _insert_events_with_properties,+ [(ctx.team_id, FLAG_EVALUATIONS_SOURCE_EVENT, flag_uuid, earlier, '{"$browser": "Chrome"}')],+ )+ ).result()+ cluster.any_host(+ partial(+ _insert_events_json_rows,+ [(ctx.team_id, FLAG_EVALUATIONS_SOURCE_EVENT, flag_uuid, earlier, native_properties)],+ )+ ).result()++ try:+ with pytest.raises(dagster.Failure, match=refusal) if refusal else nullcontext():+ _run_event_deletion(cluster, ctx)+ assert cluster.any_host(partial(_adhoc_pending_uuids, ctx.team_id)).result() == (+ set() if refusal else {flag_uuid}+ )finally:+ cluster.any_host(_truncate_adhoc_events_deletion).result()cluster.any_host(_truncate_flag_evaluations).result()+ cluster.any_host(_truncate_events_json).result()+++def test_holds_rows_reads_storage_on_another_cluster_when_the_proxy_reads_empty() -> None:+ def resolved(value: object) -> Future:+ future: Future = Future()+ future.set_result(value)+ return future++ cluster = Mock(spec=ClickhouseCluster)+ cluster.any_host_by_role.return_value = resolved([(0,)])+ sibling = Mock(spec=ClickhouseCluster)+ sibling.map_one_host_per_shard.return_value = resolved({"events-host": [(3,)]})++ assert _holds_rows(cluster, TargetPlacement(target=EVENTS_JSON, cluster=sibling), "team_id = 1", {})@pytest.mark.django_db@@ -2995,11 +3116,34 @@ def test_full_job_property_removal_survives_the_ttl_dropping_reingested_flag_eva@pytest.mark.django_db@pytest.mark.parametrize(- "events, delete_all_events, hogql_predicate, surviving_flag_evaluations",+ "events, delete_all_events, hogql_predicate, flag_row_has_events_copy, refusal",[- pytest.param([FLAG_EVALUATIONS_SOURCE_EVENT], False, "", 0, id="named_event"),- pytest.param([], True, "", 0, id="all_events"),- pytest.param([FLAG_EVALUATIONS_SOURCE_EVENT], False, "properties.$browser = 'Chrome'", 1, id="hogql_predicate"),+ pytest.param([FLAG_EVALUATIONS_SOURCE_EVENT], False, "", False, None, id="named_event"),+ pytest.param([], True, "", False, None, id="all_events"),+ pytest.param(+ [FLAG_EVALUATIONS_SOURCE_EVENT],+ False,+ "properties.$browser = 'Chrome'",+ True,+ None,+ id="hogql_predicate_leaves_rows_with_an_events_copy_to_the_ttl",+ ),+ pytest.param(+ [FLAG_EVALUATIONS_SOURCE_EVENT],+ False,+ "properties.$browser = 'Chrome'",+ False,+ "as deferred",+ id="hogql_predicate_refuses_matching_rows_without_an_events_copy",+ ),+ pytest.param(+ [FLAG_EVALUATIONS_SOURCE_EVENT],+ False,+ "properties.$browser = 'Chrome' AND NOT elements_chain ILIKE '%checkout%'",+ False,+ "elements_chain",+ id="hogql_predicate_that_cannot_run_on_flag_evaluations_refuses_rows_without_an_events_copy",+ ),],)def test_immediate_event_deletion_of_flag_evaluations(@@ -3007,26 +3151,29 @@ def test_immediate_event_deletion_of_flag_evaluations(events: list[str],delete_all_events: bool,hogql_predicate: str,- surviving_flag_evaluations: int,+ flag_row_has_events_copy: bool,+ refusal: str | None,) -> None:from posthog.models.organization import Organizationfrom posthog.models.team import Teamorg = Organization.objects.create(name="test-org-immediate-flag-evaluations")team = Team.objects.create(organization=org, name="test-team-immediate-flag-evaluations")- now = datetime.now()+ now = datetime.now(UTC)+ event_uuid = uuid4()cluster.any_host(_truncate_flag_evaluations).result()cluster.any_host(partial(_insert_events_with_properties,- [(team.id, FLAG_EVALUATIONS_SOURCE_EVENT, uuid4(), now, '{"$browser": "Chrome"}')],+ [(team.id, FLAG_EVALUATIONS_SOURCE_EVENT, event_uuid, now, '{"$browser": "Chrome"}')],)).result()+ flag_uuid = event_uuid if flag_row_has_events_copy else uuid4()cluster.any_host(partial(_insert_flag_evaluations_with_properties,- [(team.id, "someone", '{"$browser": "Chrome"}', str(uuid4()), now, now)],+ [(team.id, "someone", '{"$browser": "Chrome"}', str(flag_uuid), now, now)],)).result()@@ -3039,10 +3186,17 @@ def test_immediate_event_deletion_of_flag_evaluations(delete_all_events=delete_all_events,hogql_predicate=hogql_predicate,)- _run_event_deletion(cluster, deletion_ctx)+ with pytest.raises(dagster.Failure, match=refusal) if refusal else nullcontext():+ _run_event_deletion(cluster, deletion_ctx)- assert cluster.any_host(partial(_count_events_by_name, team.id, FLAG_EVALUATIONS_SOURCE_EVENT)).result() == 0- assert len(cluster.any_host(partial(_flag_evaluation_person_ids, team.id)).result()) == surviving_flag_evaluations+ # A refusal comes before any delete, so the events row survives it too.+ assert cluster.any_host(partial(_count_events_by_name, team.id, FLAG_EVALUATIONS_SOURCE_EVENT)).result() == (+ 1 if refusal else 0+ )+ # An immediate request with a predicate never deletes flag rows.+ assert len(cluster.any_host(partial(_flag_evaluation_person_ids, team.id)).result()) == (+ 1 if hogql_predicate else 0+ )cluster.any_host(_truncate_flag_evaluations).result()diff --git a/posthog/hogql/database/database.py b/posthog/hogql/database/database.pyindex e9755a367334..62d640831a3e 100644--- a/posthog/hogql/database/database.py+++ b/posthog/hogql/database/database.py@@ -1791,10 +1791,13 @@ class Database(BaseModel):trigger: str = "direct",allowed_system_tables: Collection[str] | None = None,schema_table_names: set[str] | None = None,+ include_flag_evaluations_table: bool | None = None,) -> Database:"""Build a query catalog, or a reduced catalog for schema serialization only.Query execution must omit schema_table_names because it can prune unrelated warehouse tables.+ ``include_flag_evaluations_table`` overrides the organization flag that decides whether+ ``posthog.flag_evaluations`` is in the catalog. None follows the flag."""build_started = time.perf_counter()@@ -1855,6 +1858,11 @@ class Database(BaseModel):sources,is_hogql_warehouse_access_control_enabled=_evaluate_warehouse_access_control_flag(cast("Team", team)),)+ if include_flag_evaluations_table is not None:+ # A direct-connection catalog has no "posthog" node to hold the table.+ sources = dataclasses.replace(+ sources, include_flag_evaluations_table=include_flag_evaluations_table and connection_id is None+ )with _OTEL_DATABASE.timed_histogram_twin(HOGQL_DATABASE_BUILD_DURATION_SECONDS, {"phase": "build_from_sources"}diff --git a/posthog/hogql/hogql.py b/posthog/hogql/hogql.pyindex 0f752e94251f..d4e2ddafa208 100644--- a/posthog/hogql/hogql.py+++ b/posthog/hogql/hogql.py@@ -19,8 +19,16 @@ def translate_hogql(events_table_alias: Optional[str] = None,events_table_use_new_schema: Optional[bool] = None,placeholders: Optional[dict[str, ast.Expr]] = None,+ table_name: str = "events",+ forbid_joins: bool = False,) -> str:- """Translate a HogQL expression into a ClickHouse expression."""+ """Translate a HogQL expression into a ClickHouse expression.++ ``table_name`` is the HogQL table the expression's fields resolve against, such as+ ``posthog.flag_evaluations``. The result holds only the expression, so a join it needs is lost.+ With ``forbid_joins`` such an expression raises QueryError instead of printing a reference to a+ table alias that does not exist.+ """if query == "":raise QueryError("Empty query")@@ -32,7 +40,7 @@ def translate_hogql(context.modifiers.personsOnEventsMode = alias_poe_mode_for_legacy(actual_poe_mode)if events_table_use_new_schema is not None:context.use_new_events_schema = events_table_use_new_schema- # Create a fake query that selects from "events" to have fields to select from.+ # Create a fake query that selects from the table to have fields to select from.if context.database is None:if context.team_id is None:raise ValueError("Cannot translate HogQL for a filter with no team specified")@@ -43,7 +51,8 @@ def translate_hogql()node = parse_expr(query, placeholders=placeholders)- select_query = ast.SelectQuery(select=[node], select_from=ast.JoinExpr(table=ast.Field(chain=["events"])))+ chain: list[str | int] = [*table_name.split(".")]+ select_query = ast.SelectQuery(select=[node], select_from=ast.JoinExpr(table=ast.Field(chain=chain)))if events_table_alias is not None and isinstance(select_query.select_from, ast.JoinExpr):select_query.select_from.alias = events_table_alias@@ -52,6 +61,12 @@ def translate_hogql(ast.SelectQuery,prepare_ast_for_printing(select_query, context=context, dialect=dialect, stack=[select_query]),)+ if (+ forbid_joins+ and prepared_select_query.select_from is not None+ and prepared_select_query.select_from.next_join is not None+ ):+ raise QueryError("The expression needs a join to another table, which is not supported here")return print_prepared_ast(prepared_select_query.select[0],context=context,diff --git a/posthog/models/data_deletion_request.py b/posthog/models/data_deletion_request.pyindex 779ec0dce89e..ebf3415eb46a 100644--- a/posthog/models/data_deletion_request.py+++ b/posthog/models/data_deletion_request.py@@ -1,3 +1,4 @@+import refrom collections.abc import Callablefrom dataclasses import dataclassfrom datetime import datetime, timedelta@@ -11,6 +12,7 @@ from django.db import modelsfrom django.db.models import Ffrom django.utils import timezone+from posthog.dataclasses import frozenfrom posthog.models.utils import UUIDModelif TYPE_CHECKING:@@ -33,7 +35,7 @@ def _hogql_predicate_text(obj) -> str:return (getattr(obj, "hogql_predicate", "") or "").strip()-def compile_hogql_predicate(obj, use_new_events_schema: bool = False) -> tuple[str, dict]:+def compile_hogql_predicate(obj, use_new_events_schema: bool = False, hogql_table: str = "events") -> tuple[str, dict]:"""Parse and compile ``obj.hogql_predicate`` into a ClickHouse SQL fragment.Returns ``(sql_fragment, extra_params)``. Both are empty when the predicate@@ -53,6 +55,10 @@ def compile_hogql_predicate(obj, use_new_events_schema: bool = False) -> tuple[s(``events_json`` / ``sharded_events_json``) — JSON subcolumn reads instead ofJSONExtract/materialized-column reads. Deletions target both physical events tables, socallers compile one fragment per table.++ ``hogql_table`` resolves the predicate against another HogQL table, such as+ ``posthog.flag_evaluations``. There the compile also raises ValidationError for a predicate that+ needs a join, such as one on ``person_id``, because the fragment is spliced into a bare WHERE."""predicate = _hogql_predicate_text(obj)if not predicate:@@ -63,12 +69,15 @@ def compile_hogql_predicate(obj, use_new_events_schema: bool = False) -> tuple[sfrom posthog.schema import PersonsOnEventsModefrom posthog.hogql.context import HogQLContext+ from posthog.hogql.database.database import Databasefrom posthog.hogql.hogql import translate_hogqlfrom posthog.hogql.modifiers import create_default_modifiers_for_teamfrom posthog.hogql.parser import parse_exprfrom posthog.models.team import Team+ on_events = hogql_table == "events"+try:parse_expr(predicate)except Exception as exc:@@ -103,9 +112,15 @@ def compile_hogql_predicate(obj, use_new_events_schema: bool = False) -> tuple[sraise ValidationError({"hogql_predicate": "team no longer exists; cannot validate the predicate."}) from excmodifiers = create_default_modifiers_for_team(team)modifiers.personsOnEventsMode = PersonsOnEventsMode.PERSON_ID_OVERRIDE_PROPERTIES_ON_EVENTS+ # The organization flag that hides posthog.flag_evaluations from queries does not apply here.+ # The rows that table holds still have to be deletable.+ database = (+ None if on_events else Database.create_for(team=team, modifiers=modifiers, include_flag_evaluations_table=True)+ )context = HogQLContext(team_id=obj.team_id,team=team,+ database=database,modifiers=modifiers,within_non_hogql_query=True,enable_select_queries=True,@@ -119,6 +134,8 @@ def compile_hogql_predicate(obj, use_new_events_schema: bool = False) -> tuple[scontext,dialect="clickhouse",events_table_use_new_schema=use_new_events_schema,+ table_name=hogql_table,+ forbid_joins=not on_events,)except ImportError:# A failed import means the runtime environment is broken (e.g. a Dagster worker that@@ -131,6 +148,45 @@ def compile_hogql_predicate(obj, use_new_events_schema: bool = False) -> tuple[sreturn sql, dict(context.values)+def _property_group_columns_read(events_sql: str) -> list[str]:+ """The property-group map columns that an events-compiled fragment reads."""+ from posthog.clickhouse.property_groups import event_property_group_definitions++ columns = [+ definition.get_column_name(source_column, group_name)+ for source_column, groups in event_property_group_definitions.items()+ for group_name, definition in groups.items()+ ]+ return sorted(column for column in columns if re.search(rf"\b{re.escape(column)}\b", events_sql))+++# A team's READY dynamic slot makes events read a property from dmat_string_<n>. The slot columns carry no+# column_materializer comment, so the registry lookup below does not list them.+_DYNAMIC_MATERIALIZED_COLUMN = re.compile(r"(?<![\w$])dmat_string_\d+(?![\w$])")+++def _materialized_columns_read(events_sql: str) -> list[str]:+ """The events materialized columns that an events-compiled fragment reads.++ The registry holds the ``mat_`` columns and the built-in ``$session_id``, ``$window_id`` and+ ``$group_0`` to ``$group_4`` columns. The dynamic ``dmat_string_`` slots are matched by name. A persons+ subquery reads ``pmat_`` columns the same way on both tables, so those are left out.+ """+ from posthog.clickhouse.materialized_columns import get_enabled_materialized_columns_by_table++ names = {column.name for column in get_enabled_materialized_columns_by_table().get("events", {}).values()}+ registered = {name for name in names if re.search(rf"(?<![\w$]){re.escape(name)}(?![\w$])", events_sql)}+ return sorted(registered | set(_DYNAMIC_MATERIALIZED_COLUMN.findall(events_sql)))+++_ELEMENTS_COLUMN = re.compile(r"(?<![\w$])elements_chain\w*")+++def _elements_columns_read(events_sql: str) -> list[str]:+ """The element columns that an events-compiled fragment reads."""+ return sorted(set(_ELEMENTS_COLUMN.findall(events_sql)))++# Compiling a predicate builds the full HogQL schema graph, which is slow. The Django admin# recompiles it on every stats/preview render, so cache the result in the shared Django cache# (Redis). Keyed by request id and guarded by the predicate text it was compiled from, so a@@ -216,8 +272,8 @@ def reaches_target_through_event_uuids(obj, target: "DeletionTarget") -> bool:"""Whether ``obj`` deletes ``target``'s rows through the uuids of their events copies.A deferred request queues the uuids of the events its HogQL predicate matches, and the drain- deletes those uuids from every target. That is how the predicate, which only compiles against- events, reaches a target that holds copies of events.+ deletes those uuids from every target. That is how the predicate reaches the rows of a target+ that have an events copy."""return (target.copied_from is not None@@ -284,6 +340,79 @@ def rows_without_events_copy_criteria(obj, target: "DeletionTarget") -> list[tupreturn [(predicate, {**params, "start_time": s, "end_time": e}) for s, e in windows]+def _and_fragment(criteria: tuple[str, dict], fragment: tuple[str, dict]) -> tuple[str, dict]:+ """``criteria`` with a compiled HogQL fragment ANDed on, or unchanged when the fragment is empty."""+ predicate, params = criteria+ sql, values = fragment+ if not sql:+ return criteria+ return f"{predicate} AND ({sql})", {**params, **values}+++@frozen+class UncopiedRowsPredicate:+ """A request's HogQL predicate compiled for the rows of a target that have no events copy."""++ # The compiled fragment and its params. None when the predicate cannot run on those rows.+ compiled: tuple[str, dict] | None+ # Why the predicate cannot run there. Empty when it can.+ error: str = ""++ def narrow(self, criteria: tuple[str, dict]) -> tuple[str, dict]:+ """``criteria`` with the compiled fragment ANDed on, or unchanged when there is none."""+ return criteria if self.compiled is None else _and_fragment(criteria, self.compiled)++ def windows(self, obj, target: "DeletionTarget") -> list[tuple[str, dict]]:+ """``rows_without_events_copy_criteria`` narrowed by the compiled fragment.++ The gate, the queue fill and verification all read these criteria, so verification counts+ the rows that the queue fill read.+ """+ return [self.narrow(window) for window in rows_without_events_copy_criteria(obj, target)]+++def uncopied_rows_predicate(obj, target: "DeletionTarget") -> UncopiedRowsPredicate:+ """``obj``'s HogQL predicate compiled for the rows of ``target`` that have no copy in ``target.copied_from``.++ The predicate cannot run there when events reads any part of it from a property-group map or a+ materialized column, which only the events tables have. The map decodes JSON escapes, so a stored+ ``"a\\"b"`` reads as ``a"b`` from the map and as ``a\\"b`` from raw JSON. With+ ``propertyGroupsMode`` set to ``optimized``, events also compares a map value as a string for+ ``=``, ``!=`` and ``IN``, where a raw JSON read casts a typed property to its defined type. A+ materialized column reads an empty string and the string ``"null"`` as NULL, and a nullable one+ decodes JSON escapes. So the two tables could match different rows.+ """+ if target.uncopied_rows_hogql_table is None:+ return UncopiedRowsPredicate(compiled=None, error="This table has no HogQL table to compile it against.")+ try:+ events_sql, _ = compile_hogql_predicate(obj)+ if element_columns := _elements_columns_read(events_sql):+ return UncopiedRowsPredicate(+ compiled=None,+ error=f"Events reads it from {', '.join(element_columns)}. {target.uncopied_rows_hogql_table} stores "+ "no elements and reads them as empty, so the two can match different rows.",+ )+ if map_columns := _property_group_columns_read(events_sql):+ return UncopiedRowsPredicate(+ compiled=None,+ error=f"Events reads it from the property-group map {', '.join(map_columns)}, which decodes JSON "+ f"escapes and compares typed values as strings. {target.uncopied_rows_hogql_table} reads the raw "+ "JSON, so the two can match different rows.",+ )+ if materialized_columns := _materialized_columns_read(events_sql):+ return UncopiedRowsPredicate(+ compiled=None,+ error=f"Events reads it from the materialized column {', '.join(materialized_columns)}, which "+ f"reads an empty string and the string 'null' as NULL. {target.uncopied_rows_hogql_table} reads "+ "the raw JSON, so the two can match different rows.",+ )+ return UncopiedRowsPredicate(+ compiled=compile_hogql_predicate(obj, hogql_table=target.uncopied_rows_hogql_table)+ )+ except ValidationError as exc:+ return UncopiedRowsPredicate(compiled=None, error=" ".join(exc.messages))++def event_removal_where(obj, use_new_events_schema: bool = False) -> tuple[str, dict]:"""Full WHERE predicate + params for event-removal queries.@@ -294,13 +423,9 @@ def event_removal_where(obj, use_new_events_schema: bool = False) -> tuple[str,proxy or the local ``sharded_events`` MergeTree. Pass ``use_new_events_schema``when the query targets the native-JSON events tables."""- predicate, params = portable_event_removal_where(obj)- parts = [predicate]- hogql_sql, hogql_values = compile_hogql_predicate(obj, use_new_events_schema=use_new_events_schema)- if hogql_sql:- parts.append(f"AND ({hogql_sql})")- params.update(hogql_values)- return " ".join(p for p in parts if p), params+ return _and_fragment(+ portable_event_removal_where(obj), compile_hogql_predicate(obj, use_new_events_schema=use_new_events_schema)+ )class RequestType(models.TextChoices):@@ -931,7 +1056,10 @@ def count_remaining_matching_events(request: "DataDeletionRequest") -> int:On a deferred request with a HogQL predicate, a target that cannot take the compiled fragment butholds copies of events gets two counts. The first is the request's queue rows that the drain hasnot marked deleted, because the drain deletes the target's rows through those uuids. The second- is the target's rows with no events copy, which the drain cannot reach. Any other target that+ is the target's rows with no events copy that the predicate compiled for the target matches. When+ the predicate cannot run on the target, it is every such row, which the drain cannot reach. That+ count uses the fragment the queue fill read with, so it cannot see the two tables compiling the+ predicate differently. Any other target thatcannot take the fragment is counted with the portable predicate instead, which matches asuperset. That can only hold a request in QUEUED, never promote one early. It also means anon-zero count is not proof that rows were missed: for a HogQL request the superset can match@@ -970,7 +1098,13 @@ def count_remaining_matching_events(request: "DataDeletionRequest") -> int:continue# docs/internal/clickhouse-deletion-coverage.md says which count holds the request at each stage.total += count_pending_hogql_event_removals(request)- criteria = rows_without_events_copy_criteria(request, target)+ uncopied = uncopied_rows_predicate(request, target)+ # Each window's copy check reads the uuid of every events copy in it, so it runs only when a+ # row in the range can match the compiled fragment.+ matchable = uncopied.compiled is None or count_rows(+ target.read_table, *uncopied.narrow((predicate, params))+ )+ criteria = uncopied.windows(request, target) if matchable else []else:criteria = [portable_event_removal_where(request)]total += sum(count_rows(target.read_table, predicate, params) for predicate, params in criteria)diff --git a/posthog/models/deletion_targets.py b/posthog/models/deletion_targets.pyindex 01cef5e16ff3..108045945793 100644--- a/posthog/models/deletion_targets.py+++ b/posthog/models/deletion_targets.py@@ -40,6 +40,8 @@ from posthog.models.flag_evaluations.sql import (COVERAGE_DOC = "docs/internal/clickhouse-deletion-coverage.md"+FLAG_EVALUATIONS_HOGQL_TABLE = "posthog.flag_evaluations"+# ClickHouse's CLUSTER_DOESNT_EXIST, which clickhouse_driver does not name._CLUSTER_DOESNT_EXIST = 701@@ -96,10 +98,10 @@ class DeletionTarget:# Guarded on system.tables: the table sits behind a migration that may not have run everywhere.optional: bool = False# Which events-schema variant compile_hogql_predicate emits for this table, None where no- # variant is known to run against its physical columns. Every predicate resolves against the- # events HogQL table, and the only choice is legacy versus native-JSON. A fragment names- # physical columns (mat_*, the property-group maps, or JSON subcolumns), so it only runs- # against the schema it was compiled for.+ # variant is known to run against its physical columns. A predicate that applies to every row+ # here resolves against the events HogQL table, and the only choice is legacy versus native-JSON.+ # A fragment names physical columns (mat_*, the property-group maps, or JSON subcolumns), so it+ # only runs against the schema it was compiled for.hogql_schema: HogQLSchema | None = None# True where the property-removal job sweeps the table. Its verification fails while any copy of# a named property that the table keeps survives the rewrite. A property removal refuses while a@@ -120,6 +122,11 @@ class DeletionTarget:# shard as the row, because the check for rows without a copy reads both tables shard-locally.# A deferred event removal with a HogQL predicate deletes rows here through the uuids of their copies.copied_from: str | None = None+ # The HogQL table that a request's predicate compiles against for the rows here that have no copy+ # in copied_from. None where those rows cannot take a predicate. A row with a copy never takes it.+ # The drain deletes a queued uuid from every target, so a predicate that matched a row here and+ # not its copy would delete the copy too.+ uncopied_rows_hogql_table: str | None = None# The event names this table can hold, None meaning unconstrained. Lets a request naming other# events skip this table without querying it.stored_events: frozenset[str] | None = None@@ -214,11 +221,12 @@ EVENTS_JSON = DeletionTarget(# Flag-evaluation telemetry carries the same person_id and group payload as events, so team and# queued-uuid sweeps must reach it, and a person sweep matches on person_id like every other# events-shaped table. Property removal rewrites its event properties and its person properties. A-# HogQL predicate does not compile against the table, so a request that carries one never runs it-# there. Immediate event removal with a predicate leaves the matching rows to the TTL. Deferred event-# removal with a predicate deletes the rows through the uuids of their events copies, and refuses-# while the table holds a matching row with no copy. Property removal with a predicate refuses while-# the table holds matching rows. docs/internal/clickhouse-deletion-coverage.md explains these limits.+# HogQL predicate runs here only on rows with no events copy, compiled against the table's own HogQL+# table. Deferred event removal with a predicate deletes the rows with a copy through the uuids of+# their events copies, and queues the rows without one that the compiled predicate matches.+# Immediate event removal with a predicate leaves the rows with a copy to the TTL, and refuses while+# a row without one matches. Property removal with a predicate refuses while the table holds+# matching rows. docs/internal/clickhouse-deletion-coverage.md explains these limits.FLAG_EVALUATIONS = DeletionTarget(data_table=FLAG_EVALUATIONS_DATA_TABLE,read_table=FLAG_EVALUATIONS_TABLE,@@ -227,6 +235,7 @@ FLAG_EVALUATIONS = DeletionTarget(accepts_person_id_rewrite=True,# Both tables shard on sipHash64(distinct_id), so a flag call and its event share a shard.copied_from=EVENTS_DATA_TABLE(),+ uncopied_rows_hogql_table=FLAG_EVALUATIONS_HOGQL_TABLE,stored_events=frozenset({FLAG_EVALUATIONS_SOURCE_EVENT}),ttl_days=FLAG_EVALUATIONS_TTL_DAYS,)diff --git a/posthog/models/test/test_data_deletion_request.py b/posthog/models/test/test_data_deletion_request.pyindex 5d2d3db4aed5..4636badb69f0 100644--- a/posthog/models/test/test_data_deletion_request.py+++ b/posthog/models/test/test_data_deletion_request.py@@ -5,12 +5,14 @@ from itertools import pairwiseimport pytestimport time_machine+from posthog.test.base import materializedfrom unittest.mock import patchfrom django.core.exceptions import ValidationErrorfrom django.test import override_settingsfrom django.utils import timezone+from posthog.clickhouse.client import sync_executefrom posthog.models import data_deletion_request as ddrfrom posthog.models.data_deletion_request import (DataDeletionRequest,@@ -18,12 +20,17 @@ from posthog.models.data_deletion_request import (RequestType,auto_approve_blocker,cached_compile_hogql_predicate,+ compile_hogql_predicate,invalidate_compiled_predicate_cache,+ uncopied_rows_predicate,)-from posthog.models.deletion_targets import FLAG_EVALUATIONS+from posthog.models.deletion_targets import FLAG_EVALUATIONS, FLAG_EVALUATIONS_HOGQL_TABLE+from posthog.models.flag_evaluations.sql import FLAG_EVALUATIONS_SOURCE_EVENTfrom posthog.models.organization import Organizationfrom posthog.models.team import Team+from products.event_definitions.backend.models import PropertyDefinition+LOCMEM_CACHE = {"default": {"BACKEND": "django.core.cache.backends.locmem.LocMemCache", "LOCATION": "ddr-test"}}TEAM_ID = 99999@@ -320,6 +327,161 @@ def test_compile_hogql_predicate_boolean_person_property_not_coerced(team):assert "transform(" not in sql+def materialized_browser():+ with materialized("events", "$browser"):+ yield+++def dmat_slot_for_plan(team):+ from posthog.models import MaterializedColumnSlot, MaterializedColumnSlotState, PropertyDefinition++ from products.event_definitions.backend.models.property_definition import PropertyType++ prop_def = PropertyDefinition.objects.create(+ team=team,+ project_id=team.project_id,+ name="plan",+ property_type=PropertyType.String,+ type=PropertyDefinition.Type.EVENT,+ )+ MaterializedColumnSlot.objects.create(+ team=team, property_definition=prop_def, slot_index=0, state=MaterializedColumnSlotState.READY+ )+++def _flag_call_request(team: Team, hogql_predicate: str) -> DataDeletionRequest:+ # OPTIMIZED is the cloud setting. It makes events read non-$ keys and $feature/ keys from+ # property-group maps, which flag_evaluations does not have.+ team.modifiers = {"propertyGroupsMode": "optimized"}+ team.save()+ return DataDeletionRequest(+ **_base_kwargs(team_id=team.id, events=[FLAG_EVALUATIONS_SOURCE_EVENT], hogql_predicate=hogql_predicate)+ )++[email protected]("table_visible", [True, False], ids=["org_sees_the_table", "org_flag_hides_the_table"])+def test_compile_hogql_predicate_against_flag_evaluations_reads_its_own_columns(+ team, materialized_browser, table_visible+):+ request = _flag_call_request(team, "properties.$browser = 'Chrome'")+ events_sql, _ = compile_hogql_predicate(request)++ with patch(+ "products.feature_flags.backend.facade.flags.is_flag_evaluations_table_enabled", return_value=table_visible+ ):+ flag_sql, values = compile_hogql_predicate(request, hogql_table=FLAG_EVALUATIONS_HOGQL_TABLE)++ assert "mat_$browser" in events_sql+ assert "mat_$browser" not in flag_sql+ assert "Chrome" in values.values()+++ "predicate, error_match",+ [+ ("person_id = '5f1c9e2a-0b7d-4e63-9a48-2c3d1e7f6b90'", "join"),+ ("person.id = '5f1c9e2a-0b7d-4e63-9a48-2c3d1e7f6b90'", "join"),+ # Exposing elements_chain as '' on flag_evaluations would make this match every row.+ ("NOT elements_chain ILIKE '%checkout%'", "elements_chain"),+ ("properties.plan = 'pro'", "property-group map"),+ ("properties.`$feature/beta` = 'true'", "property-group map"),+ # Events reads a non-$ person property from person_properties_map_custom.+ ("person.properties.email = '[email protected]'", "person_properties_map_custom"),+ # mat_$browser reads a stored "" as NULL, and the raw JSON read on flag_evaluations reads it as "".+ ("properties.$browser = ''", "materialized column mat_$browser"),+ ("properties.$browser IS NULL", "materialized column mat_$browser"),+ # The built-in $group_0 column scrubs the same way, with no mat_ prefix.+ ("properties.$group_0 IS NULL", "materialized column $group_0"),+ # A team's READY dynamic slot makes events read dmat_string_0, which the column registry does not list.+ ("properties.plan IS NULL", "materialized column dmat_string_0"),+ ],+ ids=[+ "person_id",+ "person_id_through_person",+ "elements_chain",+ "custom_property",+ "feature_flag_property",+ "custom_person_property",+ "materialized_property_equals_empty_string",+ "materialized_property_is_null",+ "builtin_group_column_is_null",+ "dynamic_slot_is_null",+ ],+)+def test_uncopied_rows_predicate_refuses_what_flag_evaluations_rows_cannot_answer(+ team, request, predicate, error_match+):+ if "$browser" in predicate:+ request.getfixturevalue("materialized_browser")+ if predicate == "properties.plan IS NULL":+ request.getfixturevalue("dmat_slot_for_plan")+ uncopied = uncopied_rows_predicate(_flag_call_request(team, predicate), FLAG_EVALUATIONS)++ assert uncopied.compiled is None+ assert error_match in uncopied.error+++# A missing key, a JSON null and an escaped quote are the rows where two ways of reading a property can+# disagree.+_PARITY_ROW_PROPERTIES = [+ '{"$plan": "pro"}',+ '{"$plan": "free"}',+ '{"$plan": ""}',+ '{"$plan": null}',+ "{}",+ '{"$plan": "a\\"b"}',+ '{"$plan": 5}',+ '{"$plan": true}',+]+++def _matches_by_row(read_table: str, team_id: int, fragment: str, values: dict) -> dict[str, bool]:+ rows = sync_execute(+ f"SELECT properties, ({fragment}) FROM {read_table} WHERE team_id = %(team_id)s",+ {**values, "team_id": team_id},+ )+ return {properties: bool(matched) for properties, matched in rows}+++ "property_type, predicate",+ [+ pytest.param(None, "properties.$plan = 'pro'", id="equals"),+ pytest.param(None, "properties.$plan = ''", id="equals_empty_string"),+ pytest.param(None, "properties.$plan = 'a\"b'", id="equals_escaped_quote"),+ pytest.param(None, "properties.$plan != 'pro'", id="not_equals"),+ pytest.param(None, "properties.$plan IN ('pro', 'team')", id="in"),+ pytest.param(None, "properties.$plan > 'free'", id="greater"),+ pytest.param(None, "properties.$plan IS NULL", id="is_null"),+ pytest.param(None, "properties.$plan LIKE 'pr%'", id="like"),+ pytest.param("Numeric", "properties.$plan > 1", id="numeric_greater"),+ pytest.param("Boolean", "properties.$plan = true", id="boolean_equals"),+ pytest.param(None, "person.properties.$plan = 'pro'", id="person_property_equals"),+ ],+)+def test_flag_evaluations_predicate_matches_the_rows_the_events_predicate_matches(team, property_type, predicate):+ if property_type is not None:+ PropertyDefinition.objects.create(+ team=team, name="$plan", type=PropertyDefinition.Type.EVENT, property_type=property_type+ )+ request = _flag_call_request(team, predicate)+ events_sql, events_values = compile_hogql_predicate(request)+ assert "properties_group_" not in events_sql+ flag_sql, flag_values = compile_hogql_predicate(request, hogql_table=FLAG_EVALUATIONS_HOGQL_TABLE)++ now = timezone.now()+ rows = [(team.id, FLAG_EVALUATIONS_SOURCE_EVENT, uuid_lib.uuid4(), now, p, p) for p in _PARITY_ROW_PROPERTIES]+ columns = "(team_id, event, uuid, timestamp, properties, person_properties)"+ sync_execute(f"INSERT INTO writable_events {columns} VALUES", rows)+ sync_execute(f"INSERT INTO writable_flag_evaluations {columns} VALUES", rows)++ events_matches = _matches_by_row("events", team.id, events_sql, events_values)+ assert events_matches.keys() == set(_PARITY_ROW_PROPERTIES)+ assert _matches_by_row("flag_evaluations", team.id, flag_sql, flag_values) == events_matches++def test_rendered_count_query_substitutes_parameters():from posthog.admin.admins.data_deletion_request_admin import build_deletion_count_queryfrom posthog.clickhouse.client.escape import substitute_params_for_display