:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(dags): run monthly retention cleanup on the 3rd at 9pm local (#123288)
Paweł Szczur78478d8f6356
diff --git a/docs/internal/clickhouse-deletion-coverage.md b/docs/internal/clickhouse-deletion-coverage.mdindex 1e5066aca05d..48182ab66ecd 100644--- a/docs/internal/clickhouse-deletion-coverage.md+++ b/docs/internal/clickhouse-deletion-coverage.md@@ -326,13 +326,13 @@ Staff set the retention in Django admin, on the team and organization pages:- The Events retention page under Custom tools (`/admin/events-retention/`) lists every team that has a value of its own or belongs to an organization with a config.It shows each team's billing months (`Team.event_retention_months`), its effective retention, where that comes from, and the first month the cleanup keeps, 100 teams per page.-`monthly_old_events_cleanup_job` has two steps:+`monthly_retention_events_cleanup_job` has two steps:- `plan_old_events_cleanup` reads every team's effective retention when the run starts and logs each one as organization, team and months. It then finds the months that hold events for each team and keeps the ones entirely past that team's retention. It logs the teams due in each month.- `cleanup_old_events_by_partition` deletes each planned month for the teams due in that month only. A team with 13 months and a team with 18 months share the older months, and the 13-month team also has months of its own.A manual run can narrow the plan with `team_ids` and `partitions`. A listed team with no retention is ignored.-`monthly_old_events_cleanup_schedule` starts one run on the 1st of each month, in every region, and skips when no retention is set. It is stopped by default.+`monthly_retention_events_cleanup_schedule` starts one run on the 3rd of each month at 21:00 local time (Europe/Paris in EU, America/Los_Angeles elsewhere), and skips when no retention is set. It is stopped by default.The job runs `ensure_no_concurrent_deletes_run` before it plans, and that guard also runs in `deletes_job`. Each job fails at the start while a `deletes_job`, cleanup or squash run is executing, so the two never mutate the events tables at the same time.Both jobs and `squash_person_overrides` carry the `deletes_job_concurrency` tag. The Dagster deployment settings put a run-queue limit of 1 on that tag, so a second run of any of the three queues instead of starting.`sharded_events_json` keeps events dated before 2020, because it stores them in its 202001 partition and the delete addresses the event's own month.diff --git a/posthog/dags/deletes.py b/posthog/dags/deletes.pyindex bc2656061828..616b3a132e6b 100644--- a/posthog/dags/deletes.py+++ b/posthog/dags/deletes.py@@ -415,7 +415,7 @@ def ensure_no_concurrent_deletes_run(context: dagster.OpExecutionContext) -> Non# A queued run has done nothing yet. Its own guard sees this run once it starts.blockers = describe_runs(context.instance,- (deletes_job.name, monthly_old_events_cleanup_job.name, squash_person_overrides.name),+ (deletes_job.name, monthly_retention_events_cleanup_job.name, squash_person_overrides.name),statuses=EXECUTING_RUN_STATUSES,exclude_run_id=context.run_id,)@@ -1323,7 +1323,7 @@ def ensure_deletes_job_can_start(context: dagster.OpExecutionContext) -> None:# launching another run would collide, not which of two started runs came first.blockers = describe_runs(context.instance,- (deletes_job.name, monthly_old_events_cleanup_job.name, squash_person_overrides.name, context.job_name),+ (deletes_job.name, monthly_retention_events_cleanup_job.name, squash_person_overrides.name, context.job_name),exclude_run_id=context.run_id,)if blockers:@@ -1500,18 +1500,18 @@ def cleanup_old_events_by_partition("deletes_job_concurrency": "v1",})-def monthly_old_events_cleanup_job():+def monthly_retention_events_cleanup_job():"""Delete events past each team's events retention set in Django admin, a whole month at a time."""cleanup_old_events_by_partition(plan_old_events_cleanup(start_after=ensure_no_concurrent_deletes_run()))@dagster.schedule(- job=monthly_old_events_cleanup_job,- cron_schedule="0 0 1 * *",- execution_timezone="UTC",+ job=monthly_retention_events_cleanup_job,+ cron_schedule="0 21 3 * *",+ execution_timezone="Europe/Paris" if settings.CLOUD_DEPLOYMENT == "EU" else "America/Los_Angeles",default_status=dagster.DefaultScheduleStatus.STOPPED,)-def monthly_old_events_cleanup_schedule(+def monthly_retention_events_cleanup_schedule(context: dagster.ScheduleEvaluationContext,) -> dagster.RunRequest | dagster.SkipReason:if not effective_events_retention():diff --git a/posthog/dags/locations/clickhouse.py b/posthog/dags/locations/clickhouse.pyindex 0f1c9859a5c0..6b353f9364ab 100644--- a/posthog/dags/locations/clickhouse.py+++ b/posthog/dags/locations/clickhouse.py@@ -39,7 +39,7 @@ defs = dagster.Definitions(drop_materialized_column.drop_materialized_column,deletes.deletes_job,deletes.manual_deletes_job,- deletes.monthly_old_events_cleanup_job,+ deletes.monthly_retention_events_cleanup_job,export_query_log_archive_to_s3.export_query_log_archive_to_s3,backfill_materialized_column.backfill_materialized_column,fix_person_id_overrides.fix_person_id_overrides_job,@@ -67,7 +67,7 @@ defs = dagster.Definitions(part_breaker.break_oversized_parts_schedule,data_deletion_requests.auto_approve_deletion_requests_schedule,warehouse_object_reads_daily.warehouse_object_reads_daily_schedule,- deletes.monthly_old_events_cleanup_schedule,+ deletes.monthly_retention_events_cleanup_schedule,],sensors=[clickhouse_cleanup.run_cleanup_sweep_after_deletes,diff --git a/posthog/dags/person_overrides.py b/posthog/dags/person_overrides.pyindex 52d51b74762f..8c11f16a00a1 100644--- a/posthog/dags/person_overrides.py+++ b/posthog/dags/person_overrides.py@@ -415,7 +415,7 @@ def record_squash_run_failure(context: dagster.HookContext) -> None:@dagster.job(tags={"owner": JobOwners.TEAM_CLICKHOUSE.value,- # Shares the run-queue limit with deletes_job and monthly_old_events_cleanup_job, because+ # Shares the run-queue limit with deletes_job and monthly_retention_events_cleanup_job, because# all three mutate the events tables. A squash run then queues while either of them runs."deletes_job_concurrency": "v1",},diff --git a/posthog/dags/tests/test_deletes.py b/posthog/dags/tests/test_deletes.pyindex 9ef3fd6ffc2c..8ccc2b097515 100644--- a/posthog/dags/tests/test_deletes.py+++ b/posthog/dags/tests/test_deletes.py@@ -39,8 +39,8 @@ from posthog.dags.deletes import (ensure_no_concurrent_deletes_run,manual_deletes_job,mark_deletions_verified,- monthly_old_events_cleanup_job,- monthly_old_events_cleanup_schedule,+ monthly_retention_events_cleanup_job,+ monthly_retention_events_cleanup_schedule,plan_old_events_cleanup,resolve_sweep_targets,run_deletes_after_manual_trigger,@@ -712,7 +712,7 @@ def test_cleanup_old_events_delete_query_format(cluster: ClickhouseCluster, snap@pytest.mark.django_db-def test_monthly_old_events_cleanup_job(cluster: ClickhouseCluster):+def test_monthly_retention_events_cleanup_job(cluster: ClickhouseCluster):thirteen, eighteen, no_retention = _teams_with_13_and_18_months()_insert_events(cluster,@@ -725,7 +725,7 @@ def test_monthly_old_events_cleanup_job(cluster: ClickhouseCluster):+ _events(no_retention.id, _mid_month(20), 0, 9),)- monthly_old_events_cleanup_job.execute_in_process(resources={"cluster": cluster})+ monthly_retention_events_cleanup_job.execute_in_process(resources={"cluster": cluster})assert _count_by_team_and_partition(cluster, [thirteen.id, eighteen.id, no_retention.id]) == {(thirteen.id, _partition(_first_hour_of_month(13))): 5,@@ -737,14 +737,14 @@ def test_monthly_old_events_cleanup_job(cluster: ClickhouseCluster):@pytest.mark.django_db@time_machine.travel("2027-01-01", tick=False)-def test_monthly_old_events_cleanup_schedule_starts_one_run_when_a_retention_is_set():+def test_monthly_retention_events_cleanup_schedule_starts_one_run_when_a_retention_is_set():with dagster.build_schedule_context(scheduled_execution_time=datetime(2027, 1, 1, tzinfo=UTC)) as context:- assert isinstance(monthly_old_events_cleanup_schedule(context), dagster.SkipReason)+ assert isinstance(monthly_retention_events_cleanup_schedule(context), dagster.SkipReason)organization = Organization.objects.create(name="With default")OrganizationEventsRetentionConfig.objects.create(organization=organization, default_events_retention_months=13)Team.objects.create(organization=organization, name="Inherits default")- run_request = monthly_old_events_cleanup_schedule(context)+ run_request = monthly_retention_events_cleanup_schedule(context)assert isinstance(run_request, dagster.RunRequest)assert (run_request.run_key, run_request.run_config) == ("202701", {})@@ -1327,9 +1327,9 @@ def test_the_survivor_count_retries_before_reporting_unknown(failing_attempts: i# A squash run blocks whatever its age: manual_deletes_job's preflight can go stale# before the sensor launches the run, so the guard checks squash again at run start.(deletes_job, squash_person_overrides),- (deletes_job, monthly_old_events_cleanup_job),- (monthly_old_events_cleanup_job, deletes_job),- (monthly_old_events_cleanup_job, squash_person_overrides),+ (deletes_job, monthly_retention_events_cleanup_job),+ (monthly_retention_events_cleanup_job, deletes_job),+ (monthly_retention_events_cleanup_job, squash_person_overrides),],)def test_the_deletes_guard_yields_to_an_executing_run_that_mutates_events(@@ -1356,7 +1356,7 @@ def test_the_deletes_guard_yields_to_an_executing_run_that_mutates_events(# A run that has not started yet blocks here even though the in-job guard ignores it: it# is a collision about to happen, and the manual trigger has launched nothing yet.(squash_person_overrides, dagster.DagsterRunStatus.NOT_STARTED),- (monthly_old_events_cleanup_job, dagster.DagsterRunStatus.STARTED),+ (monthly_retention_events_cleanup_job, dagster.DagsterRunStatus.STARTED),],)def test_the_manual_trigger_refuses_while_the_weekly_chain_is_active(