:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(dags): delete sessions past events retention in the monthly cleanup (#124029)
Paweł Szczurfdadabe25ec0
diff --git a/.github/workflows/ci-dagster.yml b/.github/workflows/ci-dagster.ymlindex 287ad3e84780..e7040bc8ff48 100644--- a/.github/workflows/ci-dagster.yml+++ b/.github/workflows/ci-dagster.yml@@ -195,6 +195,7 @@ jobs:- 'posthog/event_usage.py'- 'posthog/products.py'- 'posthog/cloud_utils.py'+ - 'posthog/run_mode.py'- 'posthog/ph_client.py'- 'products/managed_warehouse/backend/**'# Backward compatibility for unrebased PRs; remove after August 14, 2026.diff --git a/docs/internal/clickhouse-deletion-coverage.md b/docs/internal/clickhouse-deletion-coverage.mdindex 3dd5932f74e9..d97875cc3190 100644--- a/docs/internal/clickhouse-deletion-coverage.md+++ b/docs/internal/clickhouse-deletion-coverage.md@@ -316,7 +316,12 @@ When one of those runs starts during a copy, the shard stops, and its error name`_fetch_stats` counts only the events tables. It feeds `AUTO_APPROVE_MAX_EVENTS`, a cost heuristic rather than a completeness claim, so a request auto-approved as small may move somewhat more rows than measured.-`cleanup_old_events_by_partition` stays events-only. It enforces the events retention that staff set for each team, and every other personal-data table already expires sooner under its own TTL.+`cleanup_old_events_by_partition` stays events-only. It enforces the events retention that staff set for each team.+The sessions tables are built from events and have no TTL, so `cleanup_old_sessions_by_partition` deletes them with the same retention after the events are gone.+In deployed cloud they are `sessions` (v1), `raw_sessions` (v2) and `raw_sessions_v3` on the sessions satellite cluster, named by `CLICKHOUSE_SESSIONS_CLUSTER`.+Elsewhere they are `sharded_sessions`, `sharded_raw_sessions` and `sharded_raw_sessions_v3` on the main cluster.+The job skips any of these tables that does not exist.+Other personal-data tables expire sooner under their own TTL.It deletes a month only when every event in it is older than the retention. A month that is only partly past the retention waits for a later run, for manual runs too.Staff set the retention in Django admin, on the team and organization pages:@@ -328,12 +333,14 @@ Staff set the retention in Django admin, on the team and organization pages:- The Events retention page under Custom tools (`/admin/events-retention/`) lists every team that has a value of its own or belongs to an organization with a config.It shows each team's billing months (`Team.event_retention_months`), its effective retention, where that comes from, and the first month the cleanup keeps, 100 teams per page.-`monthly_retention_events_cleanup_job` has two steps:+`monthly_retention_events_cleanup_job` has four steps, in this order:- `plan_old_events_cleanup` reads every team's effective retention when the run starts and logs each one as organization, team and months. It then finds the months that hold events for each team and keeps the ones entirely past that team's retention. It logs the teams due in each month.- `cleanup_old_events_by_partition` deletes each planned month for the teams due in that month only. A team with 13 months and a team with 18 months share the older months, and the 13-month team also has months of its own.+- `plan_old_sessions_cleanup` starts after the events are deleted. It finds the sessions partitions entirely past each team's retention, so a month whose events an earlier run deleted still gets its sessions deleted.+- `cleanup_old_sessions_by_partition` deletes each planned sessions month for the teams due in it. Sessions partition by the month a session started, so a session that continues into a kept month is deleted with the due month.-A manual run can narrow the plan with `team_ids` and `partitions`. A listed team with no retention is ignored.+A manual run can narrow each plan with `team_ids` and `partitions`, set separately on the events and the sessions plan steps. A listed team with no retention is ignored.`monthly_retention_events_cleanup_schedule` starts one run on the 3rd of each month at 21:00 local time (Europe/Paris in EU, America/Los_Angeles elsewhere), and skips when no retention is set. It is stopped by default.The job runs `ensure_no_concurrent_deletes_run` before it plans, and that guard also runs in `deletes_job`. Each job fails at the start while a `deletes_job`, cleanup or squash run is executing, so the two never mutate the events tables at the same time.Both jobs and `squash_person_overrides` carry the `deletes_job_concurrency` tag. The Dagster deployment settings put a run-queue limit of 1 on that tag, so a second run of any of the three queues instead of starting.diff --git a/posthog/dags/deletes.py b/posthog/dags/deletes.pyindex 7d49223c18ca..4460569ab810 100644--- a/posthog/dags/deletes.py+++ b/posthog/dags/deletes.py@@ -63,6 +63,16 @@ from posthog.models.person.sql import (PERSON_STATIC_COHORT_TABLE,PERSONS_TABLE,)+from posthog.models.raw_sessions.sessions_v2 import (+ SHARDED_RAW_SESSIONS_DATA_TABLE,+ TABLE_BASE_NAME as RAW_SESSIONS_TABLE_BASE_NAME,+)+from posthog.models.raw_sessions.sessions_v3 import SHARDED_RAW_SESSIONS_TABLE_V3, TABLE_BASE_NAME_V3+from posthog.models.sessions.sql import (+ SESSIONS_DATA_TABLE,+ TABLE_BASE_NAME as SESSIONS_TABLE_BASE_NAME,+)+from posthog.run_mode import run_modefrom products.error_tracking.backend.facade.api import (DocumentEmbeddingTable,@@ -168,6 +178,58 @@ class OldEventsCleanupPlan:partitions: list[PartitionCleanup]+@frozen+class OldSessionsCleanupPlan:+ partitions: list[PartitionCleanup]+++@frozen+class SessionsPlacement:+ table: str+ cluster: ClickhouseCluster+++# Every sessions table partitions by toYYYYMM of a session timestamp: the first event for v1, the+# session start from the uuidv7 session id for v2 and v3. A due events month therefore names the+# same partition here. A v2 or v3 session that starts in a due month keeps no row, even when it+# continues into the next month.+SESSIONS_RETENTION_TABLES: tuple[str, ...] = (+ SESSIONS_DATA_TABLE(),+ SHARDED_RAW_SESSIONS_DATA_TABLE(),+ SHARDED_RAW_SESSIONS_TABLE_V3(),+)+SATELLITE_SESSIONS_RETENTION_TABLES: tuple[str, ...] = (+ SESSIONS_TABLE_BASE_NAME,+ RAW_SESSIONS_TABLE_BASE_NAME,+ TABLE_BASE_NAME_V3,+)+++def _sessions_placements(cluster: ClickhouseCluster) -> list[SessionsPlacement]:+ placements = [+ SessionsPlacement(table=table, cluster=cluster)+ for table in SESSIONS_RETENTION_TABLES+ if _any_node_has(cluster, table)+ ]+ # Deployed cloud keeps every sessions table on the sessions satellite, under its base name.+ # Other deployments have no such cluster, which is the gate the sessions v3 migrations use.+ if run_mode().is_deployed_cloud:+ satellite = cluster.sibling(settings.CLICKHOUSE_SESSIONS_CLUSTER, NodeRole.SESSIONS)+ placements.extend(+ SessionsPlacement(table=table, cluster=satellite)+ for table in SATELLITE_SESSIONS_RETENTION_TABLES+ if _any_node_has(satellite, table)+ )+ return placements+++def _run_on_every_shard(cluster: ClickhouseCluster, runner: LightweightDeleteMutationRunner) -> None:+ shard_mutations = cluster.map_one_host_per_shard(runner).result()+ cluster.map_all_hosts_in_shards(+ {host.shard_num: mutation.wait for host, mutation in shard_mutations.items() if host.shard_num is not None}+ ).result()++def _yyyymm(moment: datetime) -> int:return moment.year * 100 + moment.month@@ -1639,6 +1701,107 @@ def cleanup_old_events_by_partition()[email protected](ins={"start_after": dagster.In(dagster.Nothing)})+def plan_old_sessions_cleanup(+ context: dagster.OpExecutionContext,+ config: MonthlyCleanupConfig,+ cluster: dagster.ResourceParam[ClickhouseCluster],+) -> OldSessionsCleanupPlan:+ """Find every team with an events retention, and for each month the teams whose sessions in it are all past it."""+ retention = effective_events_retention()+ if config.team_ids:+ if ignored := sorted(set(config.team_ids) - {item.team_id for item in retention}):+ context.log.info(f"Ignoring teams with no events retention: {ignored}")+ retention = [item for item in retention if item.team_id in config.team_ids]+ if not retention:+ context.log.info("No team has an events retention to enforce")+ return OldSessionsCleanupPlan(partitions=[])++ now = datetime.now(UTC).replace(tzinfo=None)+ team_cutoffs = {item.team_id: _yyyymm(whole_month_cutoff(now, item.months)) for item in retention}++ # Each sessions version partitions on a different expression of the session start, so the+ # month comes from the partition id.+ partition_filter = "AND toUInt32(_partition_id) IN %(partitions)s" if config.partitions else ""+ parameters = {+ "team_ids": sorted(team_cutoffs),+ "partitions": config.partitions,+ "cutoff_partition": max(team_cutoffs.values()),+ }+ team_months: set[tuple[int, int]] = set()+ for sessions in _sessions_placements(cluster):+ query = f"""+ SELECT DISTINCT team_id, toUInt32(_partition_id) AS partition+ FROM {sessions.table}+ WHERE team_id IN %(team_ids)s+ AND toUInt32(_partition_id) < %(cutoff_partition)s+ {partition_filter}+ """+ results = sessions.cluster.map_one_host_per_shard(Query(query, parameters=parameters)).result()+ team_months.update((team_id, partition) for rows in results.values() for team_id, partition in rows)++ teams_by_partition: dict[int, list[int]] = defaultdict(list)+ for team_id, partition in sorted(team_months):+ if partition < team_cutoffs[team_id]:+ teams_by_partition[partition].append(team_id)++ plan = OldSessionsCleanupPlan(+ partitions=[+ PartitionCleanup(partition=partition, team_ids=team_ids)+ for partition, team_ids in sorted(teams_by_partition.items(), reverse=True)+ ]+ )+ for partition_cleanup in plan.partitions:+ context.log.info(f"{partition_cleanup.partition}: teams {partition_cleanup.team_ids}")++ context.add_output_metadata(+ {+ "teams": dagster.MetadataValue.int(len(team_cutoffs)),+ "partitions": dagster.MetadataValue.int(len(plan.partitions)),+ }+ )+ return plan+++def cleanup_old_sessions_by_partition(+ context: dagster.OpExecutionContext,+ cluster: dagster.ResourceParam[ClickhouseCluster],+ plan: OldSessionsCleanupPlan,+) -> None:+ """Delete each planned month's sessions for the teams the plan lists for that month."""+ if not plan.partitions:+ context.log.info("No sessions partitions found to clean up")+ return++ total_partitions = len(plan.partitions)+ placements = _sessions_placements(cluster)+ reuse_floor = _mutation_reuse_floor(cluster)++ for idx, item in enumerate(plan.partitions, 1):+ context.log.info(f"Deleting sessions {item.partition} for teams {item.team_ids} ({idx}/{total_partitions})")+ for sessions in placements:+ delete_mutation_runner = LightweightDeleteMutationRunner(+ table=sessions.table,+ predicate="team_id IN %(team_ids)s",+ parameters={"team_ids": item.team_ids},+ partition=str(item.partition),+ settings={"lightweight_deletes_sync": 0},+ reuse_since=reuse_floor,+ )+ _run_on_every_shard(sessions.cluster, delete_mutation_runner)+ context.log.info(f"Completed sessions deletion for partition {item.partition}")++ context.add_output_metadata(+ {+ "partitions_processed": dagster.MetadataValue.int(total_partitions),+ "team_ids": dagster.MetadataValue.text(+ ", ".join(str(tid) for tid in sorted({tid for item in plan.partitions for tid in item.team_ids}))+ ),+ }+ )++@dagster.job(tags={"owner": JobOwners.TEAM_CLICKHOUSE.value,@@ -1646,8 +1809,11 @@ def cleanup_old_events_by_partition(})def monthly_retention_events_cleanup_job():- """Delete events past each team's events retention set in Django admin, a whole month at a time."""- cleanup_old_events_by_partition(plan_old_events_cleanup(start_after=ensure_no_concurrent_deletes_run()))+ """Delete events and sessions past each team's events retention set in Django admin, a whole month at a time."""+ events_deleted = cleanup_old_events_by_partition(+ plan_old_events_cleanup(start_after=ensure_no_concurrent_deletes_run())+ )+ cleanup_old_sessions_by_partition(plan_old_sessions_cleanup(start_after=events_deleted))@dagster.schedule(diff --git a/posthog/dags/tests/test_deletes.py b/posthog/dags/tests/test_deletes.pyindex a77b084320d7..6a8caa1f6aee 100644--- a/posthog/dags/tests/test_deletes.py+++ b/posthog/dags/tests/test_deletes.py@@ -20,6 +20,7 @@ from posthog.dags.deletes import (_DELETE_PREDICATE,_SURVIVOR_COUNT_ATTEMPTS,DELETES_RUN_CONFIG,+ SESSIONS_RETENTION_TABLES,AdhocEventDeletesDictionary,AdhocEventDeletesTable,DeleteConfig,@@ -698,6 +699,41 @@ def _count_by_team_and_partition(cluster: ClickhouseCluster, team_ids: list[int]return cluster.any_host(count).result()+def _insert_sessions(cluster: ClickhouseCluster, sessions: list[tuple[int, datetime]]) -> None:+ def insert(client: Client) -> None:+ rows = [+ (team_id, started_at, int(started_at.replace(tzinfo=UTC).timestamp() * 1000) << 80)+ for team_id, started_at in sessions+ ]+ v1, v2, v3 = SESSIONS_RETENTION_TABLES+ client.execute(+ f"INSERT INTO {v1} (team_id, session_id, min_timestamp) VALUES",+ [(team_id, str(UUID(int=session_id_v7)), started_at) for team_id, started_at, session_id_v7 in rows],+ )+ for table in (v2, v3):+ client.execute(+ f"INSERT INTO {table} (team_id, session_id_v7) VALUES",+ [(team_id, session_id_v7) for team_id, _, session_id_v7 in rows],+ )++ cluster.any_host(insert).result()+++def _sessions_by_team_and_partition(cluster: ClickhouseCluster, team_ids: list[int]) -> dict[str, set[tuple[int, int]]]:+ def read(client: Client) -> dict[str, set[tuple[int, int]]]:+ return {+ table: set(+ client.execute(+ f"SELECT DISTINCT team_id, toUInt32(_partition_id) FROM {table} WHERE team_id IN %(team_ids)s",+ {"team_ids": team_ids},+ )+ )+ for table in SESSIONS_RETENTION_TABLES+ }++ return cluster.any_host(read).result()++def _teams_with_13_and_18_months() -> tuple[Team, Team, Team]:organization = Organization.objects.create(name="Retention")thirteen = Team.objects.create(organization=organization, name="13 months")@@ -784,6 +820,17 @@ def test_monthly_retention_events_cleanup_job(cluster: ClickhouseCluster):+ _events(eighteen.id, _first_hour_of_month(18), 200, 8)+ _events(no_retention.id, _mid_month(20), 0, 9),)+ _insert_sessions(+ cluster,+ [+ (thirteen.id, _mid_month(15)),+ (thirteen.id, _first_hour_of_month(13)),+ (thirteen.id, _mid_month(24)),+ (eighteen.id, _mid_month(20)),+ (eighteen.id, _mid_month(15)),+ (no_retention.id, _mid_month(20)),+ ],+ )monthly_retention_events_cleanup_job.execute_in_process(resources={"cluster": cluster})@@ -793,6 +840,14 @@ def test_monthly_retention_events_cleanup_job(cluster: ClickhouseCluster):(eighteen.id, _partition(_first_hour_of_month(18))): 8,(no_retention.id, _partition(_mid_month(20))): 9,}+ surviving_sessions = {+ (thirteen.id, _partition(_first_hour_of_month(13))),+ (eighteen.id, _partition(_mid_month(15))),+ (no_retention.id, _partition(_mid_month(20))),+ }+ assert _sessions_by_team_and_partition(cluster, [thirteen.id, eighteen.id, no_retention.id]) == dict.fromkeys(+ SESSIONS_RETENTION_TABLES, surviving_sessions+ )@pytest.mark.django_dbdiff --git a/posthog/models/raw_sessions/sessions_v3.py b/posthog/models/raw_sessions/sessions_v3.pyindex 265348911e71..7752a6d1c460 100644--- a/posthog/models/raw_sessions/sessions_v3.py+++ b/posthog/models/raw_sessions/sessions_v3.py@@ -492,13 +492,13 @@ def KAFKA_RAW_SESSIONS_V3_TABLE_SQL():def WRITABLE_RAW_SESSIONS_INGESTION_TABLE_SQL_V3():# On the ingestion layer the writable points at the sessions satellite cluster, where the- # live raw_sessions_v3 MergeTree lives. Cloud-only, so the cluster name has no settings entry.+ # live raw_sessions_v3 MergeTree lives.return RAW_SESSIONS_TABLE_BASE_SQL_V3.format(table_name=WRITABLE_RAW_SESSIONS_TABLE_V3(),engine=Distributed(data_table=TABLE_BASE_NAME_V3,sharding_key="cityHash64(session_id_v7)",- cluster="sessions",+ cluster=settings.CLICKHOUSE_SESSIONS_CLUSTER,),session_timestamp_modifier="DEFAULT",max_hosts=SESSION_V3_MAX_HOSTS_PER_SESSION,diff --git a/posthog/settings/data_stores.py b/posthog/settings/data_stores.pyindex cf5e288bacea..a68c5925c82a 100644--- a/posthog/settings/data_stores.py+++ b/posthog/settings/data_stores.py@@ -343,6 +343,8 @@ CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA: bool = TEST and get_from_env(# query_log_archive's single data table lives on the OPS cluster; every cluster's# Distributed read/write tables route to it via this cluster name.CLICKHOUSE_OPS_CLUSTER: str = os.getenv("CLICKHOUSE_OPS_CLUSTER", "ops")+# In deployed cloud, the live raw_sessions_v3 MergeTree is on this satellite cluster.+CLICKHOUSE_SESSIONS_CLUSTER: str = os.getenv("CLICKHOUSE_SESSIONS_CLUSTER", "sessions")# Opt-in flag for the multinode ClickHouse smoke-test stack. When true, migrations# respect their declared NodeRole(s) instead of being collapsed to NodeRole.ALL,# so a per-cluster topology can actually exercise routing.