Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse): import daily groups by team and date (#118274)

Eric Duonged3800e91cbc
diff --git a/docs/internal/hogql-trino-compiler.md b/docs/internal/hogql-trino-compiler.md
index 1d2604e80ea2..e8e1f5483b35 100644
--- a/docs/internal/hogql-trino-compiler.md
+++ b/docs/internal/hogql-trino-compiler.md
@@ -196,19 +196,24 @@ Source metadata describes what HogQL means. Target mappings describe where the c
## Managed warehouse groups
-`duckling_groups_backfill_job` exports a full ClickHouse groups snapshot for each enabled team to Parquet and registers the files in DuckLake.
+`duckling_groups_backfill_job` exports ClickHouse groups to Parquet and registers the files in DuckLake, using full or daily mode like persons.
The full-backfill sensor rotates through at most 25 teams per tick, schedules at most three runs, and retries failures with exponential backoff from 10 to 320 minutes.
-The partition key is the team ID; rerun that partition in Dagster to refresh the snapshot.
-There is no daily groups top-up sensor.
-The sensor defaults to stopped for controlled rollout. Validate a manual team snapshot before starting automatic discovery; see the [groups rollout steps](../../posthog/dags/README_DUCKLINGS.md#groups-backfill).
-Starting the sensor discovers all eligible teams, including existing memberships. A stored running sensor state must be stopped explicitly in Dagster.
+Full exports use a numeric team partition. Daily exports use `team_id_YYYY-MM-DD` and filter `_timestamp` to that UTC day.
+`duckling_groups_daily_backfill_sensor` schedules yesterday's team/date partition for teams with a successful full export.
+The date also appears in the run key and `groups_backfill/refresh_date` tag. Each daily partition tracks its own success and retries; previews do not count.
+The daily sensor checks every 10 minutes with the same scan limits, request cap, active-run checks, and failure backoff as discovery.
+Both sensors default to stopped for controlled rollout. Validate a manual team snapshot before starting automatic discovery; see the [groups rollout steps](../../posthog/dags/README_DUCKLINGS.md#groups-backfill).
+Starting the full-backfill sensor discovers all eligible teams, including existing memberships; the daily sensor only refreshes teams with a successful snapshot. Stored running sensor states must be stopped explicitly in Dagster.
The table name is `posthog.groups_<schema_name>`, derived from the control-plane team membership, including teams with grandfathered events or persons table names.
Legacy teams without a membership use `posthog.groups` for manual runs.
The export reads `groups FINAL` without an explicit sort; deduplication follows the ClickHouse table engine.
-Like the full persons job, it deletes the team's existing rows before exporting and repeats the delete immediately before registering the run's files.
+Like persons, full runs delete the team's existing rows; daily runs delete only that team's rows within the selected UTC `_timestamp` day.
+Both modes delete before exporting and repeat that scoped delete immediately before registering the run's files.
Connection retries replay that delete-and-register unit to avoid duplicate rows when a registration commits without an acknowledgement.
-An empty export clears the team's rows. A failed export or registration can leave an empty or partial team until a retry succeeds.
+An empty export clears the affected partition. A failed export or registration can leave an empty or partial partition until a retry succeeds.
+Daily exports preserve other days' rows and can retain older group versions; HogQL groups queries select the latest version.
+A daily import does not remove old versions when a group disappears from the source. Rerun the full team partition to replace all state.
Registration uses autocommit and does not use generation markers or table comments.
Avoid overlapping manual runs for the same team because an older export can replace newer data.
Setting `cleanup_existing_partition_data: false` appends files instead and disables replay deduplication, matching persons.
diff --git a/posthog/dags/README_DUCKLINGS.md b/posthog/dags/README_DUCKLINGS.md
index edd41b21c508..b4a130c9c768 100644
--- a/posthog/dags/README_DUCKLINGS.md
+++ b/posthog/dags/README_DUCKLINGS.md
@@ -44,18 +44,26 @@ Duckling RDS Catalog (PostgreSQL)
### Groups backfill
-`duckling_groups_backfill_job` runs a full snapshot per team using the team ID as its partition key.
-`duckling_groups_full_backfill_sensor` discovers teams with backfills enabled in a ready warehouse and retries failed runs.
-Rerun a team partition manually to refresh its groups; there is no daily groups top-up sensor.
-Each sensor tick checks at most 25 teams, schedules at most three runs, and advances a cursor so repeated failures cannot block discovery.
+`duckling_groups_backfill_job` supports full-team and daily exports, like persons.
+A numeric team partition (for example, `12345`) exports the full current state.
+A team/date partition (for example, `12345_2026-01-01`) exports groups whose `_timestamp` falls within that UTC day.
+`duckling_groups_full_backfill_sensor` discovers teams with backfills enabled in a ready warehouse and retries failed full runs.
+`duckling_groups_daily_backfill_sensor` creates yesterday's team/date partitions for teams with a successful full backfill.
+The daily date appears in the partition key, run key, and `groups_backfill/refresh_date` tag.
+A successful full run does not replace the daily partition's own success record; previews do not count as completed imports.
+Failed daily runs retry with a new run key after the backoff window. An active full run or an active run for that day blocks another daily request.
+Each sensor tick checks at most 25 teams, performs at most four bounded run-storage reads per team, schedules at most three runs, and advances a cursor so repeated failures cannot block discovery.
+Both sensors check every 10 minutes. The daily sensor can submit at most 18 refreshes per hour; failures and available run capacity can reduce throughput.
Failed snapshots retry with exponential backoff from 10 minutes to a maximum of 320 minutes.
The sensor checks active runs separately from real snapshot history; previews do not hide a successful or active snapshot.
-The export reads ClickHouse groups with `FINAL` and follows the full persons delete-and-register flow.
+The export reads ClickHouse groups with `FINAL` and follows the persons delete-and-register flow.
Deduplication follows the table engine: the highest `_timestamp` wins, with equal timestamps resolved by insertion order.
The export does not apply an explicit sort.
-With the default `cleanup_existing_partition_data: true`, the job deletes the team's rows before export.
-Immediately before registering files, it deletes the team's rows again; reconnect retries replay that delete-and-register unit to avoid duplicate rows.
-Registration uses autocommit, matching persons; failures can leave an empty or partially registered team until a retry succeeds.
+With the default `cleanup_existing_partition_data: true`, a full run deletes the team's rows before export; a daily run deletes only that team's rows in the UTC `_timestamp` day.
+Immediately before registering files, the job repeats the same scoped delete; reconnect retries replay that delete-and-register unit to avoid duplicate rows.
+Daily runs preserve other days' rows and can retain older versions of a group, like persons; HogQL groups queries select the latest version.
+Daily imports do not remove older versions when a source group disappears. Rerun the full team partition to replace the complete state.
+Registration uses autocommit, matching persons; failures can leave an empty or partially registered partition until a retry succeeds.
There is no generation marker or table-comment dependency. Avoid overlapping manual runs for the same team because an older export can replace newer data.
`delete_tables` is refused for the legacy shared `groups` table.
The job supports `dry_run` and `skip_ducklake_registration`.
@@ -64,17 +72,20 @@ As with persons, setting `cleanup_existing_partition_data: false` appends files
Groups exports default to a 100 GiB query memory limit, matching full persons exports.
Override it with `clickhouse_settings.max_memory_usage` when launching a run.
For a memory failure, lower `max_s3_file_fanout` and rerun the same team partition with `delete_tables: false`.
-A failed export leaves the team empty when cleanup is enabled; only a successful export proceeds to registration.
+A failed export leaves the affected partition empty when cleanup is enabled; only a successful export proceeds to registration.
-The groups sensor defaults to stopped. Use a manual snapshot to validate rollout before enabling automatic discovery:
+Both groups sensors default to stopped. Use a manual snapshot to validate rollout before enabling automatic discovery:
-1. Deploy the job and confirm `duckling_groups_full_backfill_sensor` is stopped in Dagster. The default does not override a previously stored running state.
-2. Add the selected team's ID to the `duckling_groups_backfill` dynamic partition set.
+1. Deploy the job and confirm both `duckling_groups_full_backfill_sensor` and `duckling_groups_daily_backfill_sensor` are stopped in Dagster. The default does not override a previously stored running state.
+2. Add the selected team's ID to the `duckling_groups_backfill` dynamic partition set through **Materialize… → Add a partition**.
3. Launch `duckling_groups_backfill_job` for that partition with `dry_run: false`, `skip_ducklake_registration: false`, and `delete_tables: false`.
4. Verify the latest groups per type/key against ClickHouse and run a managed warehouse HogQL groups query. Use Dagster run status and output metadata to inspect registration.
5. Start `duckling_groups_full_backfill_sensor` to discover all teams with `backfill_enabled: true` in ready warehouses. The successful manual snapshot is skipped.
+6. Start `duckling_groups_daily_backfill_sensor` after the initial snapshot succeeds. It imports yesterday's UTC `_timestamp` day under a separate team/date partition.
-Stop the sensor to pause new discovery. Stop queued or active runs separately if they must not finish.
+To retry an older day manually, add its `team_id_YYYY-MM-DD` partition and materialize it with `delete_tables: false`. Like the persons sensor, automatic daily discovery targets yesterday.
+
+Stop both sensors to pause new discovery and daily refreshes. Stop queued or active runs separately if they must not finish.
Groups query mapping is available before the first snapshot completes; queries can fail before table creation or read an empty table during the initial export.
## Partition Strategy
diff --git a/posthog/dags/locations/data_stack.py b/posthog/dags/locations/data_stack.py
index f035cbb8c195..6e845c3eefc5 100644
--- a/posthog/dags/locations/data_stack.py
+++ b/posthog/dags/locations/data_stack.py
@@ -21,6 +21,7 @@ defs = dagster.Definitions(
],
sensors=[
groups_backfill.duckling_groups_full_backfill_sensor,
+ groups_backfill.duckling_groups_daily_backfill_sensor,
events_backfill_to_duckling.duckling_events_daily_backfill_sensor,
events_backfill_to_duckling.duckling_events_full_backfill_sensor,
events_backfill_to_duckling.duckling_persons_daily_backfill_sensor,
diff --git a/products/managed_warehouse/backend/tests/test_groups_backfill.py b/products/managed_warehouse/backend/tests/test_groups_backfill.py
index 916bcd0b6e37..d8480d874a25 100644
--- a/products/managed_warehouse/backend/tests/test_groups_backfill.py
+++ b/products/managed_warehouse/backend/tests/test_groups_backfill.py
@@ -3,7 +3,7 @@ from __future__ import annotations
from collections.abc import Iterator
from datetime import UTC, datetime, timedelta
from pathlib import Path
-from typing import cast
+from typing import TYPE_CHECKING, cast
from uuid import uuid4
import pytest
@@ -26,10 +26,12 @@ from products.managed_warehouse.backend.facade.contracts import (
)
from products.managed_warehouse.dags.groups_backfill import (
GROUPS_JOB_NAME,
+ GROUPS_REFRESH_DATE_TAG,
GROUPS_SNAPSHOT_TAG,
MAX_TEAMS_PER_TICK,
RETRY_BASE_SECONDS,
duckling_groups_backfill,
+ duckling_groups_daily_backfill_sensor,
duckling_groups_full_backfill_sensor,
ensure_groups_table,
export_groups_to_duckling_s3,
@@ -37,8 +39,16 @@ from products.managed_warehouse.dags.groups_backfill import (
run_groups_backfill,
)
+if TYPE_CHECKING:
+ from collections.abc import Sequence
-def test_groups_snapshot_deduplicates_keys_within_each_team_and_group_type(monkeypatch: pytest.MonkeyPatch) -> None:
+ from dagster import RunRequest
+
+
[email protected]("partition_date", [None, datetime(2025, 1, 1), datetime(2025, 1, 2), datetime(2025, 1, 3)])
+def test_groups_snapshot_deduplicates_keys_within_each_team_and_group_type(
+ monkeypatch: pytest.MonkeyPatch, partition_date: datetime | None
+) -> None:
monkeypatch.setattr(settings, "CLICKHOUSE_USE_HTTP", False)
table_name = f"test_groups_snapshot_{uuid4().hex}"
with get_client_from_pool() as client:
@@ -69,19 +79,27 @@ def test_groups_snapshot_deduplicates_keys_within_each_team_and_group_type(monke
(7, 0, "tied", '{"name":"last insert"}', created, latest, 1),
],
)
- query = groups_snapshot_select_sql(7).replace("FROM groups FINAL", f"FROM {table_name} FINAL")
+ query = groups_snapshot_select_sql(7, partition_date).replace(
+ "FROM groups FINAL", f"FROM {table_name} FINAL"
+ )
rows = client.execute(query)
finally:
client.execute(f"DROP TABLE {table_name}")
- assert sorted(row[:6] for row in rows) == [
+ expected = [
(7, 0, "company", '{"name":"latest"}', created, latest),
(7, 0, "tied", '{"name":"last insert"}', created, latest),
(7, 1, "company", '{"name":"another type"}', created, latest),
]
+ assert sorted(row[:6] for row in rows) == [
+ row for row in expected if partition_date is None or row[5] == partition_date
+ ]
@pytest.mark.parametrize("memory_override", [None, 20 * 1024**3])
-def test_groups_export_memory_default_and_operator_override(memory_override: int | None) -> None:
[email protected]("partition_date", [None, datetime(2025, 1, 2)])
+def test_groups_export_memory_default_and_operator_override(
+ memory_override: int | None, partition_date: datetime | None
+) -> None:
client = MagicMock()
client.execute.return_value = [(100,)]
context = MagicMock()
@@ -94,11 +112,12 @@ def test_groups_export_memory_default_and_operator_override(memory_override: int
with patch(
"products.managed_warehouse.dags.groups_backfill.get_s3_url_for_clickhouse", return_value="https://example.com"
):
- result = export_groups_to_duckling_s3(context, client, config, target)
+ result = export_groups_to_duckling_s3(context, client, config, target, partition_date)
export_query = client.execute.call_args.args[0]
export_settings = client.execute.call_args.kwargs["settings"]
- assert result == "s3://unused/backfill/groups/7/synthetic-run/*.parquet"
+ period = "/2025-01-02" if partition_date is not None else ""
+ assert result == f"s3://unused/backfill/groups/7{period}/synthetic-run/*.parquet"
assert "FROM groups FINAL" in export_query
assert "ORDER BY" not in export_query
assert export_settings["max_memory_usage"] == (memory_override if memory_override is not None else 100 * 1024**3)
@@ -157,15 +176,31 @@ def lake(tmp_path: Path) -> Iterator[psycopg.Connection[tuple[object, ...]]]:
"skip_registration",
],
)
-def test_groups_full_backfill_cleans_team_and_replays_registration(
- lake: psycopg.Connection[tuple[object, ...]], tmp_path: Path, scenario: str
[email protected]("daily", [False, True])
+@time_machine.travel(datetime(2026, 1, 1, tzinfo=UTC), tick=False)
+def test_groups_backfill_cleans_only_its_partition_and_replays_registration(
+ lake: psycopg.Connection[tuple[object, ...]], tmp_path: Path, scenario: str, daily: bool
) -> None:
table = f"{DUCKLAKE_ALIAS}.posthog.groups_production"
target = DucklingTarget(team_id=7, organization_id="org", bucket="unused", bucket_region="us-east-1")
- lake.execute(f"INSERT INTO {table} (team_id, group_type_index, group_key) VALUES (7, 0, 'old'), (8, 0, 'other')")
+ date = datetime(2025, 12, 31, tzinfo=UTC)
+ lake.execute(
+ f"INSERT INTO {table} (team_id, group_type_index, group_key, _timestamp) VALUES (%s, %s, %s, %s)",
+ (7, 0, "old", date),
+ )
+ for team, key, timestamp in [
+ (7, "before", date - timedelta(microseconds=1)),
+ (7, "after", date + timedelta(days=1)),
+ (8, "other", date),
+ ]:
+ lake.execute(
+ f"INSERT INTO {table} (team_id, group_type_index, group_key, _timestamp) VALUES (%s, %s, %s, %s)",
+ (team, 0, key, timestamp),
+ )
+ untouched = [(7, "after"), (7, "before"), (8, "other")] if daily else [(8, "other")]
for name in ("a", "b"):
lake.execute(
- f"COPY (SELECT * REPLACE ('{name}' AS group_key) FROM {table} WHERE team_id = 7) TO %s (FORMAT PARQUET)",
+ f"COPY (SELECT * REPLACE ('{name}' AS group_key) FROM {table} WHERE team_id = 7 AND group_key = 'old') TO %s (FORMAT PARQUET)",
(str(tmp_path / f"{name}.parquet"),),
)
valid_second_file = (tmp_path / "b.parquet").read_bytes()
@@ -177,7 +212,7 @@ def test_groups_full_backfill_cleans_team_and_replays_registration(
skip_ducklake_registration=scenario == "skip_registration",
)
context = MagicMock()
- context.partition_key = "7"
+ context.partition_key = "7_2025-12-31" if daily else "7"
context.run.run_id = "synthetic-run"
cluster = MagicMock()
preview = config.dry_run or config.skip_ducklake_registration
@@ -188,7 +223,10 @@ def test_groups_full_backfill_cleans_team_and_replays_registration(
def export_result(**kwargs: object) -> MagicMock:
if not preview and config.cleanup_existing_partition_data:
- assert lake.execute(f"SELECT group_key FROM {table} WHERE team_id = 7").fetchall() == []
+ assert (
+ lake.execute(f"SELECT team_id, group_key FROM {table} ORDER BY team_id, group_key").fetchall()
+ == untouched
+ )
if failing_export:
raise RuntimeError("export failed")
result = MagicMock()
@@ -224,7 +262,7 @@ def test_groups_full_backfill_cleans_team_and_replays_registration(
if scenario in ("invalid_file", "export_failure"):
with pytest.raises(duckdb.Error if scenario == "invalid_file" else RuntimeError):
run_groups_backfill(context, config)
- expected = [(7, "a"), (8, "other")] if scenario == "invalid_file" else [(8, "other")]
+ expected = sorted(untouched + ([(7, "a")] if scenario == "invalid_file" else []))
assert (
lake.execute(f"SELECT team_id, group_key FROM {table} ORDER BY team_id, group_key").fetchall()
== expected
@@ -234,16 +272,20 @@ def test_groups_full_backfill_cleans_team_and_replays_registration(
run_groups_backfill(context, config)
assert bool(connect.call_count) is not preview
- expected_rows = [(8, "other")]
+ expected_rows = list(untouched)
if scenario not in ("empty", "dry_run", "skip_registration"):
expected_rows.extend([(7, "a"), (7, "b")])
if scenario in ("append", "dry_run", "skip_registration"):
expected_rows.append((7, "old"))
+ if not daily:
+ expected_rows.extend([(7, "before"), (7, "after")])
assert lake.execute(f"SELECT team_id, group_key FROM {table} ORDER BY team_id, group_key").fetchall() == sorted(
expected_rows
)
expected_files = 0 if scenario in ("empty", "dry_run", "skip_registration") else 2
- assert context.add_output_metadata.call_args.args[0]["files_registered"] == expected_files
+ metadata = context.add_output_metadata.call_args.args[0]
+ assert metadata["files_registered"] == expected_files
+ assert metadata["export_mode"] == ("daily" if daily else "full")
_NOW = datetime(2026, 1, 1, tzinfo=UTC)
@@ -274,19 +316,23 @@ def _add_run(
*,
snapshot: bool = True,
age_seconds: int = RETRY_BASE_SECONDS + 1,
+ partition_key: str | None = None,
) -> None:
- instance.add_run(
- DagsterRun(
- job_name=GROUPS_JOB_NAME,
- run_id=run_id,
- status=status,
- tags={"dagster/partition": str(team_id), GROUPS_SNAPSHOT_TAG: "true" if snapshot else "false"},
+ with time_machine.travel(datetime.now(UTC) - timedelta(seconds=age_seconds), tick=False):
+ instance.add_run(
+ DagsterRun(
+ job_name=GROUPS_JOB_NAME,
+ run_id=run_id,
+ status=status,
+ tags={
+ "dagster/partition": partition_key or str(team_id),
+ GROUPS_SNAPSHOT_TAG: "true" if snapshot else "false",
+ },
+ )
)
- )
- if status in (DagsterRunStatus.FAILURE, DagsterRunStatus.CANCELED):
- run = instance.get_run_by_id(run_id)
- assert run is not None
- with time_machine.travel(_NOW - timedelta(seconds=age_seconds), tick=False):
+ if status in (DagsterRunStatus.FAILURE, DagsterRunStatus.CANCELED):
+ run = instance.get_run_by_id(run_id)
+ assert run is not None
instance.report_run_failed(run)
@@ -341,12 +387,23 @@ def test_preview_after_snapshot_or_during_active_snapshot_does_not_schedule_anot
assert not result.run_requests
[email protected]("daily", [False, True])
@time_machine.travel(_NOW, tick=False)
-def test_failed_teams_do_not_starve_discovery_and_storage_reads_are_bounded() -> None:
+def test_failed_teams_do_not_starve_discovery_and_storage_reads_are_bounded(daily: bool) -> None:
memberships = [_membership(i) for i in range(1, MAX_TEAMS_PER_TICK + 10)]
+ evaluate = duckling_groups_daily_backfill_sensor if daily else duckling_groups_full_backfill_sensor
with DagsterInstance.ephemeral() as instance:
+ if daily:
+ for row in memberships:
+ _add_run(instance, row.team_id, f"initial-{row.team_id}", DagsterRunStatus.SUCCESS, age_seconds=86400)
for team_id in range(1, 4):
- _add_run(instance, team_id, f"failure-{team_id}", DagsterRunStatus.FAILURE)
+ _add_run(
+ instance,
+ team_id,
+ f"failure-{team_id}",
+ DagsterRunStatus.FAILURE,
+ partition_key=f"{team_id}_2025-12-31" if daily else None,
+ )
with (
patch(
"products.managed_warehouse.dags.groups_backfill.list_enabled_backfill_team_memberships",
@@ -354,30 +411,57 @@ def test_failed_teams_do_not_starve_discovery_and_storage_reads_are_bounded() ->
),
patch("products.managed_warehouse.dags.groups_backfill._repair_stale_running_statuses"),
):
- first = duckling_groups_full_backfill_sensor(build_sensor_context(instance=instance))
+ first = evaluate(build_sensor_context(instance=instance))
assert isinstance(first, SensorResult)
- second = duckling_groups_full_backfill_sensor(build_sensor_context(instance=instance, cursor=first.cursor))
+ second = evaluate(build_sensor_context(instance=instance, cursor=first.cursor))
assert isinstance(second, SensorResult)
- assert [r.partition_key for r in second.run_requests or []] == ["4", "5", "6"]
+ assert [r.partition_key for r in second.run_requests or []] == [
+ f"{team_id}_2025-12-31" if daily else str(team_id) for team_id in (4, 5, 6)
+ ]
for row in memberships:
- _add_run(instance, row.team_id, f"success-{row.team_id}", DagsterRunStatus.SUCCESS)
+ _add_run(
+ instance,
+ row.team_id,
+ f"success-{row.team_id}",
+ DagsterRunStatus.SUCCESS,
+ age_seconds=0,
+ partition_key=f"{row.team_id}_2025-12-31" if daily else None,
+ )
with patch.object(instance, "get_run_records", wraps=instance.get_run_records) as reads:
- scanned = duckling_groups_full_backfill_sensor(build_sensor_context(instance=instance))
+ scanned = evaluate(build_sensor_context(instance=instance))
assert isinstance(scanned, SensorResult)
assert scanned.cursor == str(MAX_TEAMS_PER_TICK)
- assert reads.call_count <= MAX_TEAMS_PER_TICK
- wrapped = duckling_groups_full_backfill_sensor(
- build_sensor_context(instance=instance, cursor=str(memberships[-1].team_id))
- )
+ assert reads.call_count <= MAX_TEAMS_PER_TICK * (2 if daily else 1)
+ wrapped = evaluate(build_sensor_context(instance=instance, cursor=str(memberships[-1].team_id)))
assert isinstance(wrapped, SensorResult)
assert wrapped.cursor == str(MAX_TEAMS_PER_TICK)
[email protected]("age_seconds,expected", [(0, False), (RETRY_BASE_SECONDS + 1, True)])
[email protected]("daily", [False, True])
+ "failures,age_seconds,expected",
+ [
+ (1, 0, False),
+ (1, RETRY_BASE_SECONDS + 1, True),
+ (6, RETRY_BASE_SECONDS + 1, False),
+ (6, RETRY_BASE_SECONDS * 32 + 1, True),
+ ],
+)
@time_machine.travel(_NOW, tick=False)
-def test_failed_snapshot_observes_retry_backoff(age_seconds: int, expected: bool) -> None:
+def test_failed_snapshot_observes_retry_backoff(daily: bool, failures: int, age_seconds: int, expected: bool) -> None:
+ evaluate = duckling_groups_daily_backfill_sensor if daily else duckling_groups_full_backfill_sensor
with DagsterInstance.ephemeral() as instance:
- _add_run(instance, 7, "failed", DagsterRunStatus.FAILURE, age_seconds=age_seconds)
+ if daily:
+ _add_run(instance, 7, "initial", DagsterRunStatus.SUCCESS, age_seconds=86400)
+ for i in range(failures):
+ _add_run(
+ instance,
+ 7,
+ f"failed-{i}",
+ DagsterRunStatus.FAILURE,
+ age_seconds=age_seconds,
+ partition_key="7_2025-12-31" if daily else None,
+ )
with (
patch(
"products.managed_warehouse.dags.groups_backfill.list_enabled_backfill_team_memberships",
@@ -385,20 +469,110 @@ def test_failed_snapshot_observes_retry_backoff(age_seconds: int, expected: bool
),
patch("products.managed_warehouse.dags.groups_backfill._repair_stale_running_statuses"),
):
- result = duckling_groups_full_backfill_sensor(build_sensor_context(instance=instance))
+ result = evaluate(build_sensor_context(instance=instance))
assert isinstance(result, SensorResult)
assert bool(result.run_requests) == expected
+ "initial_age,status,age_seconds,run_kind,expected",
+ [
+ (None, None, 0, "daily", False),
+ (None, DagsterRunStatus.SUCCESS, 0, "preview", False),
+ (None, DagsterRunStatus.FAILURE, RETRY_BASE_SECONDS + 1, "daily", False),
+ (86400, None, 0, "daily", True),
+ (0, None, 0, "daily", True),
+ (86400, DagsterRunStatus.SUCCESS, 0, "preview", True),
+ (86400, DagsterRunStatus.STARTED, 0, "daily", False),
+ (86400, DagsterRunStatus.STARTED, 0, "full", False),
+ (86400, DagsterRunStatus.SUCCESS, 0, "daily", False),
+ (86400, DagsterRunStatus.FAILURE, 0, "daily", False),
+ (86400, DagsterRunStatus.FAILURE, RETRY_BASE_SECONDS + 1, "daily", True),
+ (86400, DagsterRunStatus.CANCELED, RETRY_BASE_SECONDS + 1, "daily", True),
+ ],
+)
+@time_machine.travel(_NOW, tick=False)
+def test_daily_groups_topup_requires_initial_full_success_and_observes_its_own_runs(
+ initial_age: int | None, status: DagsterRunStatus | None, age_seconds: int, run_kind: str, expected: bool
+) -> None:
+ with DagsterInstance.ephemeral() as instance:
+ if initial_age is not None:
+ _add_run(instance, 7, "initial", DagsterRunStatus.SUCCESS, age_seconds=initial_age)
+ if status is not None:
+ _add_run(
+ instance,
+ 7,
+ "latest",
+ status,
+ age_seconds=age_seconds,
+ snapshot=run_kind != "preview",
+ partition_key="7" if run_kind == "full" else "7_2025-12-31",
+ )
+ with (
+ patch(
+ "products.managed_warehouse.dags.groups_backfill.list_enabled_backfill_team_memberships",
+ return_value=[_membership(7)],
+ ),
+ patch("products.managed_warehouse.dags.groups_backfill._repair_stale_running_statuses"),
+ ):
+ result = duckling_groups_daily_backfill_sensor(build_sensor_context(instance=instance))
+ assert isinstance(result, SensorResult)
+ assert bool(result.run_requests) == expected
+ if initial_age is not None:
+ initial = duckling_groups_full_backfill_sensor(build_sensor_context(instance=instance))
+ assert isinstance(initial, SensorResult)
+ assert not initial.run_requests
+
+
+@time_machine.travel(_NOW, tick=False)
+def test_daily_groups_topup_is_partitioned_by_yesterday_and_retries_with_a_new_run_key() -> None:
+ with (
+ DagsterInstance.ephemeral() as instance,
+ patch(
+ "products.managed_warehouse.dags.groups_backfill.list_enabled_backfill_team_memberships",
+ return_value=[_membership(7)],
+ ),
+ patch("products.managed_warehouse.dags.groups_backfill._repair_stale_running_statuses"),
+ ):
+ _add_run(instance, 7, "initial", DagsterRunStatus.SUCCESS, age_seconds=86400)
+
+ def requests() -> Sequence[RunRequest]:
+ result = duckling_groups_daily_backfill_sensor(build_sensor_context(instance=instance))
+ assert isinstance(result, SensorResult)
+ return result.run_requests or []
+
+ first = requests()[0]
+ assert first.partition_key == "7_2025-12-31"
+ result = duckling_groups_daily_backfill_sensor(build_sensor_context(instance=instance))
+ assert isinstance(result, SensorResult)
+ assert result.dynamic_partitions_requests is not None
+ assert result.dynamic_partitions_requests[0].partition_keys == ["7_2025-12-31"]
+ assert first.tags[GROUPS_REFRESH_DATE_TAG] == "2025-12-31"
+ assert requests()[0].run_key == first.run_key
+ _add_run(instance, 7, "failed", DagsterRunStatus.FAILURE, partition_key="7_2025-12-31")
+ assert requests()[0].run_key != first.run_key
+ _add_run(instance, 7, "refreshed", DagsterRunStatus.SUCCESS, age_seconds=0, partition_key="7_2025-12-31")
+ assert not requests()
+ with time_machine.travel(_NOW + timedelta(hours=1), tick=False):
+ _add_run(instance, 7, "failed-rerun", DagsterRunStatus.FAILURE, partition_key="7_2025-12-31")
+ assert requests()[0].run_key not in (first.run_key, "7_2025-12-31_groups_failed")
+ with time_machine.travel(_NOW + timedelta(days=1), tick=False):
+ tomorrow = requests()[0]
+ assert tomorrow.run_key != first.run_key
+ assert tomorrow.partition_key == "7_2026-01-01"
+ assert tomorrow.tags[GROUPS_REFRESH_DATE_TAG] == "2026-01-01"
+
+
@pytest.mark.parametrize("config", [{"dry_run": True}, {"skip_ducklake_registration": True}])
-def test_preview_clears_inherited_snapshot_tag(config: dict[str, bool]) -> None:
[email protected]("partition_key", ["7", "7_2025-12-31"])
+def test_preview_clears_inherited_snapshot_tag(config: dict[str, bool], partition_key: str) -> None:
with DagsterInstance.ephemeral() as instance:
- instance.add_dynamic_partitions("duckling_groups_backfill", ["7"])
+ instance.add_dynamic_partitions("duckling_groups_backfill", [partition_key])
with patch("products.managed_warehouse.dags.groups_backfill.run_groups_backfill"):
result = materialize(
[duckling_groups_backfill],
instance=instance,
- partition_key="7",
+ partition_key=partition_key,
tags={GROUPS_SNAPSHOT_TAG: "true"},
run_config={"ops": {"duckling_groups_backfill": {"config": config}}},
)
diff --git a/products/managed_warehouse/dags/groups_backfill.py b/products/managed_warehouse/dags/groups_backfill.py
index 67798e0df79a..fc4e6a798846 100644
--- a/products/managed_warehouse/dags/groups_backfill.py
+++ b/products/managed_warehouse/dags/groups_backfill.py
@@ -1,3 +1,4 @@
+from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING
from django.utils import timezone
@@ -40,6 +41,8 @@ from posthog.dags.events_backfill_to_duckling import (
_repair_stale_running_statuses,
_resolve_duckling_target,
get_s3_url_for_clickhouse,
+ is_full_export_partition,
+ parse_partition_key_dates,
)
from products.data_warehouse.backend.facade.backfill_status import record_backfill_finished, record_backfill_started
@@ -56,6 +59,7 @@ if TYPE_CHECKING:
GROUPS_DATASET = ManagedWarehouseBackfillPartition.Dataset.GROUPS
GROUPS_JOB_NAME = "duckling_groups_backfill_job"
GROUPS_SNAPSHOT_TAG = "groups_backfill/snapshot"
+GROUPS_REFRESH_DATE_TAG = "groups_backfill/refresh_date"
MAX_TEAMS_PER_TICK = 25
MAX_RUN_REQUESTS_PER_TICK = 3
RETRY_HISTORY_LIMIT = 6
@@ -73,16 +77,21 @@ GROUPS_COLUMN_TYPES = {
}
-def groups_snapshot_select_sql(team_id: int) -> str:
+def groups_snapshot_select_sql(team_id: int, date: datetime | None = None) -> str:
if team_id <= 0:
raise ValueError("Groups backfill requires a positive team ID")
+ date_filter = ""
+ if date is not None:
+ start = escape_clickhouse_string(date.strftime("%Y-%m-%d"))
+ end = escape_clickhouse_string((date + timedelta(days=1)).strftime("%Y-%m-%d"))
+ date_filter = f"AND _timestamp >= toDateTime({start}, 'UTC') AND _timestamp < toDateTime({end}, 'UTC')"
return f"""
SELECT team_id, group_type_index, group_key,
group_properties, created_at,
toDateTime64(_timestamp, 6) AS _timestamp,
now64(6) AS _inserted_at
FROM groups FINAL
- WHERE team_id = {int(team_id)}
+ WHERE team_id = {int(team_id)} {date_filter}
"""
@@ -91,8 +100,10 @@ def export_groups_to_duckling_s3(
client: "Client",
config: DucklingBackfillConfig,
target: DucklingTarget,
+ date: datetime | None = None,
) -> str | None:
- path = f"backfill/groups/{target.team_id}/{context.run.run_id}/{{_partition_id}}.parquet"
+ period = f"/{date.strftime('%Y-%m-%d')}" if date is not None else ""
+ path = f"backfill/groups/{target.team_id}{period}/{context.run.run_id}/{{_partition_id}}.parquet"
s3_url = get_s3_url_for_clickhouse(target.bucket, target.bucket_region, path)
s3_glob = f"s3://{target.bucket}/{path.replace('{_partition_id}', '*')}"
if config.dry_run:
@@ -106,18 +117,22 @@ def export_groups_to_duckling_s3(
**(config.clickhouse_settings or {}),
}
settings.update(PARQUET_WRITER_SETTINGS)
- row_count = client.execute(
- "SELECT count() FROM groups WHERE team_id = %(team_id)s", {"team_id": target.team_id}, settings=settings
- )[0][0]
+ count_sql = "SELECT count() FROM groups WHERE team_id = %(team_id)s"
+ count_params: dict[str, int | str] = {"team_id": target.team_id}
+ if date is not None:
+ count_sql += " AND _timestamp >= toDateTime(%(date)s, 'UTC') AND _timestamp < toDateTime(%(next_date)s, 'UTC')"
+ count_params.update(date=date.strftime("%Y-%m-%d"), next_date=(date + timedelta(days=1)).strftime("%Y-%m-%d"))
+ row_count = client.execute(count_sql, count_params, settings=settings)[0][0]
fanout = _compute_fanout(int(row_count), config.target_rows_per_file, config.max_s3_file_fanout)
export_sql = f"""
INSERT INTO FUNCTION s3({escape_clickhouse_string(s3_url)}, 'Parquet')
PARTITION BY toString(cityHash64(group_type_index, group_key) % {fanout})
- {groups_snapshot_select_sql(target.team_id)}
+ {groups_snapshot_select_sql(target.team_id, date)}
SETTINGS s3_truncate_on_insert=1, use_hive_partitioning=0
"""
# A failed export can leave partial S3 objects; only register files after the whole query succeeds.
- _execute_export_with_retry(client, export_sql, settings, f"team_id={target.team_id}, full groups export")
+ info = f"team_id={target.team_id}, date={date.strftime('%Y-%m-%d') if date is not None else 'full'}"
+ _execute_export_with_retry(client, export_sql, settings, info)
return s3_glob
@@ -144,12 +159,17 @@ def ensure_groups_table(
def delete_groups_partition_data(
- conn: psycopg.Connection[tuple[object, ...]], target: DucklingTarget, table_name: str
+ conn: psycopg.Connection[tuple[object, ...]],
+ target: DucklingTarget,
+ table_name: str,
+ date: datetime | None = None,
) -> None:
- conn.execute(
- sql.SQL("DELETE FROM {} WHERE team_id = %s").format(sql.Identifier(DUCKLAKE_ALIAS, "posthog", table_name)),
- (target.team_id,),
- )
+ query = sql.SQL("DELETE FROM {} WHERE team_id = %s").format(sql.Identifier(DUCKLAKE_ALIAS, "posthog", table_name))
+ params: tuple[object, ...] = (target.team_id,)
+ if date is not None:
+ query += sql.SQL(" AND _timestamp >= %s AND _timestamp < %s")
+ params += (date.strftime("%Y-%m-%d"), (date + timedelta(days=1)).strftime("%Y-%m-%d"))
+ conn.execute(query, params)
def register_groups_files_with_duckling(
@@ -172,57 +192,84 @@ def register_groups_files_with_duckling(
def run_groups_backfill(context: AssetExecutionContext, config: DucklingBackfillConfig) -> None:
- team_id = int(context.partition_key)
+ partition_key = context.partition_key
+ is_full = is_full_export_partition(partition_key)
+ dates: list[datetime | None]
+ if is_full:
+ team_id = int(partition_key)
+ dates = [None]
+ else:
+ team_id, partition_dates = parse_partition_key_dates(partition_key)
+ dates = list(partition_dates)
groups_snapshot_select_sql(team_id)
+ mode = "full" if is_full else "daily"
target = _resolve_duckling_target(team_id)
table_name = resolve_groups_table(team_id)
tags = dagster_tags(context)
should_use_duckgres = not (config.dry_run or config.skip_ducklake_registration)
session = _DuckgresSession(context, target) if should_use_duckgres else None
files_registered = 0
+ partitions_exported = 0
try:
if session is not None:
session.run("ensure groups table", lambda conn: ensure_groups_table(conn, table_name, config))
- if config.cleanup_existing_partition_data:
- session.run("delete all groups", lambda conn: delete_groups_partition_data(conn, target, table_name))
+ for partition_date in dates:
+ date_label = partition_date.strftime("%Y-%m-%d") if partition_date is not None else "full"
+ if session is not None and config.cleanup_existing_partition_data:
- def export(client: "Client") -> str | None:
- with tags_context(kind="dagster", dagster=tags):
- return export_groups_to_duckling_s3(context, client, config, target)
+ def delete_groups_partition(
+ conn: psycopg.Connection[tuple[object, ...]], date: datetime | None = partition_date
+ ) -> None:
+ delete_groups_partition_data(conn, target, table_name, date)
- s3_glob = (
- _get_cluster()
- .any_host_by_role(
- fn=export, workload=Workload.OFFLINE if is_cloud() else Workload.DEFAULT, node_role=NodeRole.DATA
+ session.run(f"delete groups partition {date_label}", delete_groups_partition)
+
+ def export(client: "Client", date: datetime | None = partition_date) -> str | None:
+ with tags_context(kind="dagster", dagster=tags):
+ return export_groups_to_duckling_s3(context, client, config, target, date)
+
+ s3_glob = (
+ _get_cluster()
+ .any_host_by_role(
+ fn=export, workload=Workload.OFFLINE if is_cloud() else Workload.DEFAULT, node_role=NodeRole.DATA
+ )
+ .result()
)
- .result()
- )
- if s3_glob is not None and session is not None:
+ if s3_glob is not None:
+ partitions_exported += 1
+ if session is not None:
- def register_full_groups_files(conn: psycopg.Connection[tuple[object, ...]], glob: str = s3_glob) -> int:
- # Re-clear the team on replay because an unacknowledged registration may have committed.
- if config.cleanup_existing_partition_data:
- delete_groups_partition_data(conn, target, table_name)
- return register_groups_files_with_duckling(context, conn, table_name, glob)
+ def register_groups_files(
+ conn: psycopg.Connection[tuple[object, ...]],
+ glob: str = s3_glob,
+ date: datetime | None = partition_date,
+ ) -> int:
+ # Re-clear the partition on replay because an unacknowledged registration may have committed.
+ if config.cleanup_existing_partition_data:
+ delete_groups_partition_data(conn, target, table_name, date)
+ return register_groups_files_with_duckling(context, conn, table_name, glob)
- files_registered = session.run("register groups files (full)", register_full_groups_files)
+ files_registered += session.run(f"register groups files {date_label}", register_groups_files)
finally:
if session is not None:
session.close()
context.add_output_metadata(
{
"team_id": team_id,
+ "partition_key": partition_key,
+ "export_mode": mode,
"groups_table": table_name,
"files_registered": files_registered,
+ "partitions_exported": partitions_exported,
}
)
if not config.dry_run and not config.skip_ducklake_registration:
record_duckling_backfill_workload(
team_id=team_id,
dataset=GROUPS_DATASET,
- mode="full",
+ mode=mode,
files_registered=files_registered,
- partitions_exported=1,
+ partitions_exported=partitions_exported,
)
@@ -232,11 +279,13 @@ def duckling_groups_backfill(context: AssetExecutionContext, config: DucklingBac
context.instance.add_run_tags(context.run.run_id, {GROUPS_SNAPSHOT_TAG: "false"})
run_groups_backfill(context, config)
return
- team_id = int(context.partition_key)
+ is_full = is_full_export_partition(context.partition_key)
+ team_id = int(context.partition_key) if is_full else parse_partition_key_dates(context.partition_key)[0]
+ mode = "full" if is_full else "daily"
run_id = context.run.run_id
context.instance.add_run_tags(run_id, {GROUPS_SNAPSHOT_TAG: "true"})
record_backfill_started(team_id=team_id, dataset=GROUPS_DATASET, partition_key=context.partition_key, run_id=run_id)
- with track_duckling_backfill(team_id=team_id, dataset=GROUPS_DATASET, mode="full"):
+ with track_duckling_backfill(team_id=team_id, dataset=GROUPS_DATASET, mode=mode):
try:
run_groups_backfill(context, config)
except Exception as error:
@@ -254,11 +303,64 @@ duckling_groups_backfill_job = define_asset_job(
)
-@sensor(job=duckling_groups_backfill_job, minimum_interval_seconds=600, default_status=DefaultSensorStatus.STOPPED)
-def duckling_groups_full_backfill_sensor(context: SensorEvaluationContext) -> SensorResult:
- backfills = sorted(
- list_enabled_backfill_team_memberships("groups_full_backfill_sensor"), key=lambda row: row.team_id
+def _groups_snapshot_run_request(
+ context: SensorEvaluationContext, key: str, *, refresh_daily: bool, now: datetime
+) -> RunRequest | None:
+ full_partition_tags = {"dagster/partition": key}
+ successes = context.instance.get_run_records(
+ filters=RunsFilter(
+ job_name=GROUPS_JOB_NAME,
+ tags={**full_partition_tags, GROUPS_SNAPSHOT_TAG: "true"},
+ statuses=[DagsterRunStatus.SUCCESS],
+ ),
+ limit=1,
+ )
+ if refresh_daily:
+ if not successes:
+ return None
+ if context.instance.get_runs(
+ filters=RunsFilter(job_name=GROUPS_JOB_NAME, tags=full_partition_tags, statuses=ACTIVE_RUN_STATUSES),
+ limit=1,
+ ):
+ return None
+ key = f"{key}_{(now - timedelta(days=1)).date().isoformat()}"
+ elif successes:
+ return None
+ partition_tags = {"dagster/partition": key}
+ if context.instance.get_runs(
+ filters=RunsFilter(job_name=GROUPS_JOB_NAME, tags=partition_tags, statuses=ACTIVE_RUN_STATUSES), limit=1
+ ):
+ return None
+ history = context.instance.get_run_records(
+ filters=RunsFilter(job_name=GROUPS_JOB_NAME, tags={**partition_tags, GROUPS_SNAPSHOT_TAG: "true"}),
+ limit=RETRY_HISTORY_LIMIT,
+ )
+ latest = history[0] if history else None
+ if latest is not None and latest.dagster_run.is_success:
+ return None
+ if latest is not None and latest.dagster_run.status in (DagsterRunStatus.FAILURE, DagsterRunStatus.CANCELED):
+ failures = 0
+ for record in history:
+ if record.dagster_run.status not in (DagsterRunStatus.FAILURE, DagsterRunStatus.CANCELED):
+ break
+ failures += 1
+ retry_after = RETRY_BASE_SECONDS * 2 ** (failures - 1)
+ ended_at = latest.end_time or latest.update_timestamp.timestamp()
+ if now.timestamp() < ended_at + retry_after:
+ return None
+ return RunRequest(
+ partition_key=key,
+ run_key=f"{key}_groups_{latest.dagster_run.run_id if latest else 'new'}",
+ tags={
+ GROUPS_SNAPSHOT_TAG: "true",
+ **({GROUPS_REFRESH_DATE_TAG: key.split("_", 1)[1]} if refresh_daily else {}),
+ },
)
+
+
+def _groups_snapshot_sensor_result(context: SensorEvaluationContext, *, refresh_daily: bool) -> SensorResult:
+ call_site = "groups_daily_backfill_sensor" if refresh_daily else "groups_full_backfill_sensor"
+ backfills = sorted(list_enabled_backfill_team_memberships(call_site), key=lambda row: row.team_id)
if not backfills:
return SensorResult(run_requests=[])
_repair_stale_running_statuses(context, GROUPS_DATASET)
@@ -267,41 +369,16 @@ def duckling_groups_full_backfill_sensor(context: SensorEvaluationContext) -> Se
existing = set(context.instance.get_dynamic_partitions("duckling_groups_backfill"))
new_partitions: list[str] = []
requests: list[RunRequest] = []
+ now = timezone.now().astimezone(UTC)
for membership in candidates[:MAX_TEAMS_PER_TICK]:
after_team = membership.team_id
key = str(membership.team_id)
- active = context.instance.get_runs(
- filters=RunsFilter(job_name=GROUPS_JOB_NAME, tags={"dagster/partition": key}, statuses=ACTIVE_RUN_STATUSES),
- limit=1,
- )
- if active:
+ request = _groups_snapshot_run_request(context, key, refresh_daily=refresh_daily, now=now)
+ if request is None:
continue
- history = context.instance.get_run_records(
- filters=RunsFilter(job_name=GROUPS_JOB_NAME, tags={"dagster/partition": key, GROUPS_SNAPSHOT_TAG: "true"}),
- limit=RETRY_HISTORY_LIMIT,
- )
- latest = history[0] if history else None
- if latest is not None and latest.dagster_run.is_success:
- continue
- if latest is not None:
- failures = 0
- for record in history:
- if record.dagster_run.status not in (DagsterRunStatus.FAILURE, DagsterRunStatus.CANCELED):
- break
- failures += 1
- retry_after = RETRY_BASE_SECONDS * 2 ** max(failures - 1, 0)
- ended_at = latest.end_time or latest.update_timestamp.timestamp()
- if timezone.now().timestamp() < ended_at + retry_after:
- continue
- if key not in existing:
- new_partitions.append(key)
- requests.append(
- RunRequest(
- partition_key=key,
- run_key=f"{key}_groups_{latest.dagster_run.run_id if latest else 'new'}",
- tags={GROUPS_SNAPSHOT_TAG: "true"},
- )
- )
+ if request.partition_key is not None and request.partition_key not in existing:
+ new_partitions.append(request.partition_key)
+ requests.append(request)
if len(requests) >= MAX_RUN_REQUESTS_PER_TICK:
break
return SensorResult(
@@ -311,3 +388,13 @@ def duckling_groups_full_backfill_sensor(context: SensorEvaluationContext) -> Se
if new_partitions
else [],
)
+
+
+@sensor(job=duckling_groups_backfill_job, minimum_interval_seconds=600, default_status=DefaultSensorStatus.STOPPED)
+def duckling_groups_full_backfill_sensor(context: SensorEvaluationContext) -> SensorResult:
+ return _groups_snapshot_sensor_result(context, refresh_daily=False)
+
+
+@sensor(job=duckling_groups_backfill_job, minimum_interval_seconds=600, default_status=DefaultSensorStatus.STOPPED)
+def duckling_groups_daily_backfill_sensor(context: SensorEvaluationContext) -> SensorResult:
+ return _groups_snapshot_sensor_result(context, refresh_daily=True)