:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(managed-warehouse): convert duckgres tables for the trino path (#128556)
Eric Duong8f0475a556cf
diff --git a/posthog/dags/README_DUCKLINGS.md b/posthog/dags/README_DUCKLINGS.mdindex 4c1ca200de59..416ea5d6cdc7 100644--- a/posthog/dags/README_DUCKLINGS.md+++ b/posthog/dags/README_DUCKLINGS.md@@ -22,9 +22,9 @@ Duckling RDS Catalog (PostgreSQL)- **ClickHouse EC2 role**: Allowed in duckling bucket policy for direct S3 writes- **Dagster IRSA role**: Can assume duckling cross-account roles for DuckDB registration-### Warehouses without duckgres+### Registration through Trino-Organizations with the `managed-warehouse-trino-import-registration` flag have no duckgres, so the events, persons, and groups backfills register through the organization's Trino catalog instead.+Organizations with the `managed-warehouse-trino-import-registration` flag register the events, persons, and groups backfills through the organization's Trino catalog instead of duckgres.The ClickHouse export is the same, except that it casts each column to the Parquet type that DuckLake writes for its Trino column, because `add_files` refuses any type conversion.An export-only run (`skip_ducklake_registration: true`) writes the same types, so `add_files` can register its files later.ClickHouse lists the files it exported, because nothing else on the path can list the duckling bucket.@@ -40,11 +40,26 @@ One `add_files` call registers the files of an export and removes the rows they- An export with no files deletes the rows it would replace.- With `cleanup_existing_partition_data: false`, the files are added and no rows are removed.+Tables that duckgres created can hold unsigned columns, which Trino shows as other types, so `add_files` refuses the export's files.+Daily and other Trino runs refuse such a table and fail with a message.+To change an organization over:++1. Run `duckling_trino_conversion_job` for the team, with `dry_run: true` first.+ It widens the columns that DuckLake can widen in place, such as groups `group_type_index`, and reports the tables that still need a rebuild.+ It fails when the events table needs a rebuild, because no snapshot run can rebuild events.+2. A column that DuckLake cannot widen, such as persons `person_version`, needs a full backfill with `rebuild_mismatched_table: true`.+ Run it with the organization on the Trino flag, and without `skip_schema_validation`, because the validation finds the mismatch.+ The run replaces the table after the export and keeps the columns that the export does not write.+ Readers see an empty table until the run registers the files, and a run that fails after the rebuild leaves the table empty until the next full backfill.++The duckgres path accepts the converted layout, so turning the flag off still works.+The call has an import ID, so a retry after a lost response adds nothing.A statement that fails for a transient reason, such as a DuckLake commit conflict, a lost coordinator, or a cluster fault, runs again up to four times.Readers never see a day without rows or with two copies of it, so this path needs no delete-then-register replay.-Day and table replacements remove rows without a team filter, so the Trino path refuses to replace rows in the shared `events`, `persons`, and `groups` tables, or to drop them with `delete_tables: true`.-Trino-created tables have no year/month/day partitioning, because Trino cannot partition a timestamp with time zone by those transforms. Each file holds one day, so file statistics still prune reads.+Day and table replacements remove rows without a team filter, so the Trino path refuses to replace rows in the shared `events`, `persons`, and `groups` tables, or to drop them with `delete_tables: true` or rebuild them.+Trino-created tables have no year/month/day partitioning.+Each file holds one day, so file statistics still prune reads.## Jobs and Sensorsdiff --git a/posthog/dags/events_backfill_to_duckling.py b/posthog/dags/events_backfill_to_duckling.pyindex e5a091cf4ac6..65b5caf57c51 100644--- a/posthog/dags/events_backfill_to_duckling.py+++ b/posthog/dags/events_backfill_to_duckling.py@@ -60,6 +60,7 @@ from dagster import (DagsterRunStatus,DefaultSensorStatus,DynamicPartitionsDefinition,+ OpExecutionContext,RunRequest,RunsFilter,SensorEvaluationContext,@@ -106,6 +107,7 @@ from products.managed_warehouse.backend.facade.client import ()from products.managed_warehouse.backend.facade.contracts import (DuckLakeDayReplacement,+ DuckLakeTableLayoutError,DuckLakeTableReplacement,ManagedWarehouseTeamMembership,)@@ -559,7 +561,7 @@ class _DuckgresSession:MAX_ATTEMPTS = 4- def __init__(self, context: AssetExecutionContext, target: DucklingTarget) -> None:+ def __init__(self, context: AssetExecutionContext | OpExecutionContext, target: DucklingTarget) -> None:self._context = contextself._target = target# Keep one org-scoped credential ID for the session so reconnects@@ -817,6 +819,11 @@ PERSONS_COLUMNS = """now64(6) AS _inserted_at"""+# A BIGINT person_version column refuses UBIGINT files, so a table converted for Trino needs a signed export.+PERSONS_COLUMNS_SIGNED_VERSION = PERSONS_COLUMNS.replace(+ "p.version AS person_version", "toInt64(p.version) AS person_version"+)+# Expected columns in the duckling's persons table for schema validationEXPECTED_DUCKLAKE_PERSONS_COLUMNS = {"team_id",@@ -1035,6 +1042,7 @@ class DucklingBackfillConfig(Config):cleanup_existing_partition_data: bool = True # Delete existing DuckLake data for partition before registeringcreate_tables_if_missing: bool = Truedelete_tables: bool = False # Danger: drops and recreates tables, losing all data+ rebuild_mismatched_table: bool = False # A full Trino persons or groups run replaces a table with other typesdry_run: bool = False# Opt-in per run. A backfill never evaluates the canary feature flag.use_trino_canary: bool = False@@ -1189,6 +1197,14 @@ def table_exists(return False+def _persons_export_columns(conn: psycopg.Connection[Any], target: DucklingTarget) -> str:+ if not table_exists(conn, DUCKLAKE_ALIAS, "posthog", target.persons_table):+ return PERSONS_COLUMNS+ rows = conn.execute(f"DESCRIBE {DUCKLAKE_ALIAS}.posthog.{target.persons_table}").fetchall()+ version_types = {str(row[1]) for row in rows if row[0] == "person_version"}+ return PERSONS_COLUMNS_SIGNED_VERSION if version_types == {"BIGINT"} else PERSONS_COLUMNS++def _set_table_partitioning(conn: psycopg.Connection[Any],alias: str,@@ -2549,7 +2565,7 @@ def _merged_clickhouse_settings(context: AssetExecutionContext, config: Ducklingreraise=True,)def list_exported_files(client: Client, target: DucklingTarget, s3_glob: str, settings: dict[str, Any]) -> list[str]:- """List the files an export wrote, for a warehouse without duckgres to glob them.+ """List the files an export wrote, for the Trino path to register them.ClickHouse wrote the files, and the duckling bucket policy lets it list them. The export globs have theirwildcard in the file name only, so every listed file name joins the directory of the glob.@@ -2600,7 +2616,7 @@ def backfill_through_trino(exports: Sequence[TrinoExport],settings: dict[str, Any],) -> TrinoBackfillResult:- """Export each item from ClickHouse and register its files through Trino, for a warehouse without duckgres.+ """Export each item from ClickHouse and register its files through Trino instead of duckgres.One add_files call adds the files of an export and removes the rows they replace in the same snapshot. A retriedcall with the same import id is a no-op, so this path needs no delete-then-register replay. A run that skips@@ -2613,17 +2629,39 @@ def backfill_through_trino(partitions_exported = 0files_registered = 0register = not (config.dry_run or config.skip_ducklake_registration)+ if config.rebuild_mismatched_table and config.skip_schema_validation:+ raise ValueError(+ "rebuild_mismatched_table needs the schema validation, because the validation finds the mismatch"+ )+ # Only a run that replaces the whole table loses nothing when it rebuilds the table, so only such a run may+ # replace a table whose types differ from the export, and only when the caller asked for it.+ replaces_table = (+ config.rebuild_mismatched_table+ and config.cleanup_existing_partition_data+ and bool(exports)+ and all(isinstance(item.replacement, DuckLakeTableReplacement) for item in exports)+ )+ rebuild = Falsewith (connect_backfill_catalog(target.organization_id, canary=config.use_trino_canary) if register else nullcontext()) as catalog:if catalog is not None:- catalog.ensure_table(- table,- column_types,- drop_first=config.delete_tables,- create_if_missing=config.create_tables_if_missing,- validate=not config.skip_schema_validation,- )+ try:+ catalog.ensure_table(+ table,+ column_types,+ drop_first=config.delete_tables,+ create_if_missing=config.create_tables_if_missing,+ validate=not config.skip_schema_validation,+ )+ except DuckLakeTableLayoutError:+ if not replaces_table:+ raise+ rebuild = True+ context.log.warning(+ f"{table} does not have the column types of the export, so this run rebuilds it before it "+ "registers files"+ )for item in exports:# A new prefix for every export, because an export under an earlier prefix would overwrite files that# DuckLake already registered with their row counts and statistics.@@ -2642,6 +2680,12 @@ def backfill_through_trino(context.log.info(f"Exported {len(files)} file(s) for {item.label} without registering them")continuereplacement = item.replacement if config.cleanup_existing_partition_data else None+ if rebuild:+ # The rebuild waits until the export finished, to keep the window with an empty table short. A file+ # count that add_files refuses fails here, before the rebuild empties the table.+ catalog.validate_file_count(files)+ catalog.replace_table(table, column_types)+ rebuild = Falseif files:catalog.register_files(table, files, import_id=prefix, replacement=replacement)files_registered += len(files)@@ -3180,6 +3224,12 @@ def _run_duckling_persons_backfill(context: AssetExecutionContext, config: Ducklcontext.log.info("Validating duckling persons schema compatibility...")session.run("validate persons schema", lambda c: validate_duckling_persons_schema(context, target, c))+ persons_columns = (+ session.run("read persons column types", lambda c: _persons_export_columns(c, target))+ if session is not None+ else PERSONS_COLUMNS+ )+merged_settings = _merged_clickhouse_settings(context, config)cluster = _get_cluster()@@ -3207,6 +3257,7 @@ def _run_duckling_persons_backfill(context: AssetExecutionContext, config: Ducklteam_id=team_id,run_id=run_id,settings=merged_settings,+ columns=persons_columns,)s3_glob = cluster.any_host_by_role(@@ -3261,6 +3312,7 @@ def _run_duckling_persons_backfill(context: AssetExecutionContext, config: Duckldate=date,run_id=run_id,settings=merged_settings,+ columns=persons_columns,)s3_glob = cluster.any_host_by_role(diff --git a/posthog/dags/locations/data_stack.py b/posthog/dags/locations/data_stack.pyindex 184c03183384..460e43b71210 100644--- a/posthog/dags/locations/data_stack.py+++ b/posthog/dags/locations/data_stack.py@@ -4,7 +4,7 @@ from posthog.dags import events_backfill_to_ducklingfrom products.data_modeling.backend.facade.dags import job_usagefrom products.data_warehouse.backend.facade.dags import managed_viewset_sync-from products.managed_warehouse.backend.facade.dags import groups_backfill+from products.managed_warehouse.backend.facade.dags import groups_backfill, trino_conversionfrom . import loggers, resources@@ -17,6 +17,7 @@ defs = dagster.Definitions(jobs=[job_usage.model_job_usage_job,groups_backfill.duckling_groups_backfill_job,+ trino_conversion.duckling_trino_conversion_job,managed_viewset_sync.sync_managed_viewsets_job,events_backfill_to_duckling.duckling_events_backfill_job,events_backfill_to_duckling.duckling_persons_backfill_job,diff --git a/posthog/dags/test_events_backfill_to_duckling.py b/posthog/dags/test_events_backfill_to_duckling.pyindex 12fb9111b86d..eddedff8fc3a 100644--- a/posthog/dags/test_events_backfill_to_duckling.py+++ b/posthog/dags/test_events_backfill_to_duckling.py@@ -76,6 +76,7 @@ from products.managed_warehouse.backend.facade.api import EARLIEST_BACKFILL_DATEfrom products.managed_warehouse.backend.facade.client import ServiceCredential, ServiceCredentialUnavailablefrom products.managed_warehouse.backend.facade.contracts import (DuckLakeDayReplacement,+ DuckLakeTableLayoutError,DuckLakeTableReplacement,DucklingTables,ManagedWarehouseTableNames,@@ -2626,7 +2627,11 @@ class TestFannedOutLayoutRoundTrip:class _FakeTrinoBackfillCatalog:- def __init__(self) -> None:+ def __init__(self, layout_mismatch: bool = False, max_files: int = 2_000) -> None:+ self.layout_mismatch = layout_mismatch+ self.max_files = max_files+ self.rebuilt = False+ self.calls: list[tuple[Any, ...]] = []self.tables: list[tuple[str, dict[str, str]]] = []self.registered: list[tuple[str, list[str], str, DuckLakeDayReplacement | DuckLakeTableReplacement | None]] = []self.deleted: list[tuple[str, DuckLakeDayReplacement | DuckLakeTableReplacement]] = []@@ -2634,8 +2639,18 @@ class _FakeTrinoBackfillCatalog:def ensure_table(self, table: str, columns: dict[str, str], *, drop_first: bool, create_if_missing: bool, validate: bool) -> None:+ if self.layout_mismatch and validate and not self.rebuilt:+ raise DuckLakeTableLayoutError("the table has unsigned columns")self.tables.append((table, dict(columns)))+ def replace_table(self, table: str, columns: dict[str, str]) -> None:+ self.calls.append(("replace", table))+ self.rebuilt = True++ def validate_file_count(self, files: list[str]) -> None:+ if len(files) > self.max_files:+ raise ValueError(f"The export wrote {len(files)} files, more than {self.max_files}")+def register_files(self,table: str,@@ -2644,6 +2659,7 @@ class _FakeTrinoBackfillCatalog:import_id: str,replacement: DuckLakeDayReplacement | DuckLakeTableReplacement | None,) -> int:+ self.calls.append(("register", table, replacement))self.registered.append((table, list(files), import_id, replacement))return len(files)@@ -2676,6 +2692,49 @@ class _FakeExportClickHouse:return [(10,)]+class TestDuckgresPersonsExportColumns:+ @parameterized.expand(+ [+ ("bigint_column_gets_a_signed_version", "BIGINT", "2", "toInt64(p.version) AS person_version"),+ ("ubigint_column_keeps_the_unsigned_version", "UBIGINT", "2", "p.version AS person_version"),+ ("daily_run_follows_the_table_too", "BIGINT", "2_2026-06-17", "toInt64(p.version) AS person_version"),+ ]+ )+ @time_machine.travel(datetime(2026, 7, 1, tzinfo=UTC), tick=False)+ def test_exports_the_version_type_that_the_persons_table_accepts(+ self, _name, version_type, partition_key, expected_column+ ):+ clickhouse = _FakeExportClickHouse(files_per_export=1)+ cluster = MagicMock()+ cluster.any_host_by_role.side_effect = lambda fn, **_: MagicMock(result=MagicMock(return_value=fn(clickhouse)))+ conn = MagicMock()+ conn.execute.return_value.fetchall.return_value = [("person_version", version_type)]++ def run_op(what: str, op: Any) -> Any:+ return op(conn) if what == "read persons column types" else 0++ context = MagicMock(partition_key=partition_key)+ context.run.run_id = "0123456789abcdef-run"+ target = TestBackfillThroughTrino.TARGET+ with (+ patch("posthog.dags.events_backfill_to_duckling._resolve_duckling_target", return_value=target),+ patch("posthog.dags.events_backfill_to_duckling.backfills_through_trino", return_value=False),+ patch("posthog.dags.events_backfill_to_duckling._get_cluster", return_value=cluster),+ patch("posthog.dags.events_backfill_to_duckling._DuckgresSession") as duckgres,+ patch("posthog.dags.events_backfill_to_duckling.record_duckling_backfill_workload"),+ ):+ duckgres.return_value.run.side_effect = run_op+ _run_duckling_persons_backfill(+ context,+ DucklingBackfillConfig(+ create_tables_if_missing=False, skip_schema_validation=True, cleanup_existing_partition_data=False+ ),+ )++ (insert, _settings) = clickhouse.inserts[0]+ assert expected_column in insert++class TestBackfillThroughTrino:TARGET = DucklingTarget(team_id=2,@@ -2686,8 +2745,15 @@ class TestBackfillThroughTrino:persons_table="persons_team_2",)- def _run(self, run, partition_key: str, clickhouse: _FakeExportClickHouse, config: DucklingBackfillConfig):- catalog = _FakeTrinoBackfillCatalog()+ def _run(+ self,+ run,+ partition_key: str,+ clickhouse: _FakeExportClickHouse,+ config: DucklingBackfillConfig,+ catalog: _FakeTrinoBackfillCatalog | None = None,+ ):+ catalog = catalog or _FakeTrinoBackfillCatalog()cluster = MagicMock()cluster.any_host_by_role.side_effect = lambda fn, **_: MagicMock(result=MagicMock(return_value=fn(clickhouse)))context = MagicMock(partition_key=partition_key)@@ -2757,6 +2823,56 @@ class TestBackfillThroughTrino:assert cast_column in insertassert settings["output_format_parquet_string_as_string"] == 1+ @parameterized.expand(+ [+ ("full_rebuilds_once_then_registers", "2", True, True, ["replace", "register", "register"]),+ ("full_without_rebuild_refuses", "2", False, True, None),+ ("full_without_cleanup_refuses", "2", True, False, None),+ ("day_refuses", "2_2026-06-17", True, True, None),+ ]+ )+ @time_machine.travel(datetime(2026, 7, 1, tzinfo=UTC), tick=False)+ def test_rebuilds_a_table_with_other_column_types_only_when_the_run_replaces_the_table(+ self, _name, partition_key, rebuild, cleanup, expected_calls+ ):+ clickhouse = _FakeExportClickHouse(files_per_export=2)+ catalog = _FakeTrinoBackfillCatalog(layout_mismatch=True)+ config = DucklingBackfillConfig(rebuild_mismatched_table=rebuild, cleanup_existing_partition_data=cleanup)++ def run() -> None:+ self._run(_run_duckling_persons_backfill, partition_key, clickhouse, config, catalog)++ if expected_calls is None:+ with pytest.raises(DuckLakeTableLayoutError):+ run()+ assert "replace" not in [call[0] for call in catalog.calls]+ assert catalog.registered == []+ else:+ run()+ assert [call[0] for call in catalog.calls] == expected_calls+ assert {call[2] for call in catalog.calls if call[0] == "register"} == {DuckLakeTableReplacement()}+ assert len(catalog.registered) == 2++ @parameterized.expand(+ [+ ("too_many_files", 1, DucklingBackfillConfig(rebuild_mismatched_table=True), "more than 1"),+ (+ "no_schema_validation",+ 2_000,+ DucklingBackfillConfig(rebuild_mismatched_table=True, skip_schema_validation=True),+ "needs the schema validation",+ ),+ ]+ )+ @time_machine.travel(datetime(2026, 7, 1, tzinfo=UTC), tick=False)+ def test_refuses_before_it_rebuilds_the_table(self, _name, max_files, config, message):+ catalog = _FakeTrinoBackfillCatalog(layout_mismatch=True, max_files=max_files)++ with pytest.raises(ValueError, match=message):+ self._run(_run_duckling_persons_backfill, "2", _FakeExportClickHouse(files_per_export=2), config, catalog)++ assert catalog.calls == []+@parameterized.expand([("empty_day_deletes_its_rows", 0, True),diff --git a/products/managed_warehouse/backend/backfill_trino.py b/products/managed_warehouse/backend/backfill_trino.pyindex d461aba71661..924b1743cf75 100644--- a/products/managed_warehouse/backend/backfill_trino.py+++ b/products/managed_warehouse/backend/backfill_trino.py@@ -1,8 +1,8 @@"""Register duckling backfill exports in DuckLake through the organization's Trino catalog.-The Dagster backfills use this for warehouses that have no duckgres. One `add_files` call registers the files of an-export and removes the rows they replace in a single DuckLake snapshot, so readers never see a day without rows or-with two copies of it. The call refuses any type conversion, so the export must write the exact Parquet types that+The Dagster backfills use this for organizations with the Trino registration flag. One `add_files` call registers the+files of an export and removes the rows they replace in a single DuckLake snapshot, so readers never see a day without+rows or with two copies of it. The call refuses any type conversion, so the export must write the exact Parquet types thatDuckLake writes for each Trino column."""@@ -22,7 +22,11 @@ from posthog.hogql.escape_sql import escape_trino_identifier, escape_trino_strinfrom posthog.exceptions_capture import capture_exceptionfrom posthog.ph_client import feature_enabled_or_false-from products.managed_warehouse.backend.facade.contracts import DuckLakeDayReplacement, DuckLakeTableReplacement+from products.managed_warehouse.backend.facade.contracts import (+ DuckLakeDayReplacement,+ DuckLakeTableLayoutError,+ DuckLakeTableReplacement,+)from products.managed_warehouse.backend.facade.feature_flags import DUCKLAKE_TRINO_REGISTRATION_FLAGfrom products.managed_warehouse.backend.trino_connection import connect_managed_warehouse_trinofrom products.managed_warehouse.backend.trino_execution import is_transient_trino_error@@ -46,6 +50,7 @@ TRANSIENT_DUCKLAKE_ERROR_NAMES = frozenset({"DUCKLAKE_COMMIT_CONFLICT", "DUCKLAKE_CONCURRENT_MODIFICATION", "DUCKLAKE_COMMIT_FAILED"})STATEMENT_ATTEMPTS = 4+REBUILD_HINT = "Run duckling_trino_conversion_job for the team, or a full backfill with rebuild_mismatched_table: true."Replacement = DuckLakeDayReplacement | DuckLakeTableReplacement@@ -81,6 +86,10 @@ def _refuse_shared_table(table: str, change: str) -> None:)+def _column_definitions(columns: Mapping[str, str]) -> str:+ return ", ".join(f"{escape_trino_identifier(name)} {data_type}" for name, data_type in columns.items())++def _is_transient(error: BaseException) -> bool:# The client already retried each HTTP request, so these mean that the coordinator or the gateway went away.if isinstance(error, TrinoConnectionError | Http502Error | Http503Error | Http504Error):@@ -126,6 +135,7 @@ class TrinoBackfillCatalog:def _run(self, statement: str) -> list[list[Any]]:# Every statement here can run again. DDL has IF [NOT] EXISTS, deleting a range twice deletes nothing more,# and add_files records its import_id, so a repeat of an applied import adds nothing.+ # CREATE OR REPLACE runs before the add_files that fills the table, so a repeat recreates an empty table.cursor = self._connection.cursor()try:cursor.execute(statement)@@ -141,27 +151,44 @@ class TrinoBackfillCatalog:self._run(f"DROP TABLE IF EXISTS {self._table(table)}")if create_if_missing:self._run(f"CREATE SCHEMA IF NOT EXISTS {self._catalog}.{escape_trino_identifier(BACKFILL_SCHEMA)}")- definitions = ", ".join(- f"{escape_trino_identifier(name)} {data_type}" for name, data_type in columns.items()- )- self._run(f"CREATE TABLE IF NOT EXISTS {self._table(table)} ({definitions})")+ self._run(f"CREATE TABLE IF NOT EXISTS {self._table(table)} ({_column_definitions(columns)})")if validate:self._validate(table, columns)def _validate(self, table: str, columns: Mapping[str, str]) -> None:# A table can have more columns than the export, because add_files reads a missing column as NULL.- rows = self._run(- f"SELECT column_name, data_type FROM {self._catalog}.information_schema.columns "- f"WHERE table_schema = {escape_trino_string(BACKFILL_SCHEMA)} AND table_name = {escape_trino_string(table)}"- )- actual = {str(name): str(data_type) for name, data_type in rows}+ actual = self._columns(table)+ if not actual:+ raise ValueError(+ f"Table {BACKFILL_SCHEMA}.{table} does not exist. Run with create_tables_if_missing: true."+ )mismatched = {name: actual.get(name, "missing") for name, data_type in columns.items() if actual.get(name) != data_type}if mismatched:- raise ValueError(+ raise DuckLakeTableLayoutError(f"Table {BACKFILL_SCHEMA}.{table} does not match the backfill export, so add_files would refuse its "- f"files. Expected {dict(columns)}, but these columns differ: {mismatched}"+ f"files. Expected {dict(columns)}, but these columns differ: {mismatched}. {REBUILD_HINT}"+ )++ def _columns(self, table: str) -> dict[str, str]:+ rows = self._run(+ f"SELECT column_name, data_type FROM {self._catalog}.information_schema.columns "+ f"WHERE table_schema = {escape_trino_string(BACKFILL_SCHEMA)} AND table_name = {escape_trino_string(table)}"+ )+ return {str(name): str(data_type) for name, data_type in rows}++ def replace_table(self, table: str, columns: Mapping[str, str]) -> None:+ _refuse_shared_table(table, "rebuild")+ # Views can select a column that the export does not write, so the rebuilt table keeps it.+ kept = {name: data_type for name, data_type in self._columns(table).items() if name not in columns}+ self._run(f"CREATE OR REPLACE TABLE {self._table(table)} ({_column_definitions({**columns, **kept})})")++ def validate_file_count(self, files: Sequence[str]) -> None:+ if len(files) > ADD_FILES_MAX_FILES:+ raise ValueError(+ f"The export wrote {len(files)} files, but one add_files call registers at most "+ f"{ADD_FILES_MAX_FILES}. Lower max_s3_file_fanout.")def register_files(@@ -174,11 +201,7 @@ class TrinoBackfillCatalog:"""if replacement is not None:_refuse_shared_table(table, "replace rows in")- if len(files) > ADD_FILES_MAX_FILES:- raise ValueError(- f"The export wrote {len(files)} files, but one add_files call registers at most "- f"{ADD_FILES_MAX_FILES}. Lower max_s3_file_fanout."- )+ self.validate_file_count(files)options = [f"files => ARRAY[{', '.join(escape_trino_string(path) for path in files)}]",f"import_id => {escape_trino_string(import_id)}",diff --git a/products/managed_warehouse/backend/facade/backfill_trino.py b/products/managed_warehouse/backend/facade/backfill_trino.pyindex 183580cf73b0..491b5ebfe948 100644--- a/products/managed_warehouse/backend/facade/backfill_trino.py+++ b/products/managed_warehouse/backend/facade/backfill_trino.py@@ -1,4 +1,4 @@-"""Register duckling backfill exports in DuckLake through Trino, for warehouses without duckgres."""+"""Register duckling backfill exports in DuckLake through Trino instead of duckgres."""from __future__ import annotationsdiff --git a/products/managed_warehouse/backend/facade/contracts.py b/products/managed_warehouse/backend/facade/contracts.pyindex 6db19bd36917..245e2cc8ce14 100644--- a/products/managed_warehouse/backend/facade/contracts.py+++ b/products/managed_warehouse/backend/facade/contracts.py@@ -25,6 +25,7 @@ __all__ = ["CPUnavailableError","DuckLakeBackfillCatalog","DuckLakeDayReplacement",+ "DuckLakeTableLayoutError","DuckLakeTableReplacement","DuckgresQueryServerConfig","DuckgresStoredBucketConfig",@@ -447,6 +448,10 @@ class DuckLakeTableReplacement:"""All rows of the table, which a full backfill export replaces."""+class DuckLakeTableLayoutError(ValueError):+ """The table's column types differ from the backfill export, so add_files refuses the export's files."""++class DuckLakeBackfillCatalog(Protocol):"""The DuckLake statements a backfill runs in the organization's Trino catalog."""@@ -454,6 +459,10 @@ class DuckLakeBackfillCatalog(Protocol):self, table: str, columns: Mapping[str, str], *, drop_first: bool, create_if_missing: bool, validate: bool) -> None: ...+ def replace_table(self, table: str, columns: Mapping[str, str]) -> None: ...++ def validate_file_count(self, files: Sequence[str]) -> None: ...+def register_files(self,table: str,diff --git a/products/managed_warehouse/backend/facade/dags.py b/products/managed_warehouse/backend/facade/dags.pyindex b1d762b4a15a..977f9db16072 100644--- a/products/managed_warehouse/backend/facade/dags.py+++ b/products/managed_warehouse/backend/facade/dags.py@@ -1,3 +1,3 @@-from products.managed_warehouse.dags import groups_backfill+from products.managed_warehouse.dags import groups_backfill, trino_conversion-__all__ = ["groups_backfill"]+__all__ = ["groups_backfill", "trino_conversion"]diff --git a/products/managed_warehouse/backend/tests/test_backfill_trino.py b/products/managed_warehouse/backend/tests/test_backfill_trino.pyindex cd44643c90c1..612cb2d9fad9 100644--- a/products/managed_warehouse/backend/tests/test_backfill_trino.py+++ b/products/managed_warehouse/backend/tests/test_backfill_trino.py@@ -16,7 +16,11 @@ from products.managed_warehouse.backend.backfill_trino import (TrinoBackfillCatalog,)from products.managed_warehouse.backend.facade.backfill_trino import connect_backfill_catalog-from products.managed_warehouse.backend.facade.contracts import DuckLakeDayReplacement, DuckLakeTableReplacement+from products.managed_warehouse.backend.facade.contracts import (+ DuckLakeDayReplacement,+ DuckLakeTableLayoutError,+ DuckLakeTableReplacement,+)if TYPE_CHECKING:from trino.dbapi import Connection@@ -192,35 +196,73 @@ class TestEnsureTable:'("team_id" bigint, "timestamp" timestamp(6) with time zone, "person_version" bigint)',]- @parameterized.expand([("events",), ("persons",), ("groups",)])- def test_refuses_to_drop_a_table_other_teams_share(self, table: str) -> None:+ def test_rebuilds_the_table_with_the_export_types_and_keeps_other_columns(self) -> None:+ catalog, connection = _catalog({**COLUMNS, "person_version": "decimal(20,0)", "added_later": "varchar"})++ catalog.replace_table("events_team_7", COLUMNS)++ assert connection.statements[-1] == (+ 'CREATE OR REPLACE TABLE "warehouse"."posthog"."events_team_7" '+ '("team_id" bigint, "timestamp" timestamp(6) with time zone, "person_version" bigint, '+ '"added_later" varchar)'+ )++ @parameterized.expand(+ [+ (f"{name}_{table}", table, action)+ for table in ("events", "persons", "groups")+ for name, action in (+ (+ "drop",+ lambda catalog, table: catalog.ensure_table(+ table, COLUMNS, drop_first=True, create_if_missing=True, validate=True+ ),+ ),+ ("rebuild", lambda catalog, table: catalog.replace_table(table, COLUMNS)),+ )+ ]+ )+ def test_refuses_to_drop_or_rebuild_a_table_other_teams_share(self, _name: str, table: str, action) -> None:catalog, connection = _catalog(COLUMNS)with pytest.raises(ValueError, match="shared"):- catalog.ensure_table(table, COLUMNS, drop_first=True, create_if_missing=True, validate=True)+ action(catalog, table)assert connection.statements == []@parameterized.expand([- ("extra_column", {**COLUMNS, "added_later": "varchar"}, None),- ("missing_column", {"team_id": "bigint", "timestamp": "timestamp(6) with time zone"}, "person_version"),- ("unsigned_column", {**COLUMNS, "person_version": "decimal(20,0)"}, "person_version"),+ ("extra_column", {**COLUMNS, "added_later": "varchar"}, None, None),+ (+ "missing_column",+ {"team_id": "bigint", "timestamp": "timestamp(6) with time zone"},+ DuckLakeTableLayoutError,+ "person_version",+ ),+ (+ "unsigned_column",+ {**COLUMNS, "person_version": "decimal(20,0)"},+ DuckLakeTableLayoutError,+ "person_version",+ ),+ ("missing_table", {}, ValueError, "does not exist"),])def test_validates_that_exported_files_match_the_table(- self, _name: str, actual: dict[str, str], mismatch: str | None+ self, _name: str, actual: dict[str, str], error: type[Exception] | None, message: str | None) -> None:catalog, _ = _catalog(actual)def ensure() -> None:catalog.ensure_table("events_team_7", COLUMNS, drop_first=False, create_if_missing=False, validate=True)- if mismatch is None:+ if error is None:+ ensure()+ return+ with pytest.raises(error, match=message) as raised:ensure()- else:- with pytest.raises(ValueError, match=mismatch):- ensure()+ # A rebuild must not create a table that does not exist, so a missing table is not a layout mismatch.+ assert (raised.type is DuckLakeTableLayoutError) == (error is DuckLakeTableLayoutError)def _query_error(error_class: type[TrinoQueryError], name: str, error_type: str) -> TrinoQueryError:diff --git a/products/managed_warehouse/backend/tests/test_groups_backfill.py b/products/managed_warehouse/backend/tests/test_groups_backfill.pyindex bff56ac21826..a6c47240aac4 100644--- a/products/managed_warehouse/backend/tests/test_groups_backfill.py+++ b/products/managed_warehouse/backend/tests/test_groups_backfill.py@@ -704,3 +704,23 @@ def test_preview_clears_inherited_snapshot_tag(config: dict[str, bool], partitiodef test_delete_tables_refuses_legacy_shared_table(lake: psycopg.Connection[tuple[object, ...]]) -> None:with pytest.raises(ValueError, match="shared groups table"):ensure_groups_table(lake, "groups", DucklingBackfillConfig(delete_tables=True))+++ "alter,accepted",+ [+ ("ALTER COLUMN group_type_index SET DATA TYPE SMALLINT", True),+ ("ALTER COLUMN group_type_index SET DATA TYPE INTEGER", False),+ ("ADD COLUMN extra INTEGER", False),+ ],+)+def test_groups_table_check_accepts_only_the_converted_group_type_index(+ lake: psycopg.Connection[tuple[object, ...]], alter: str, accepted: bool+) -> None:+ lake.execute(f"ALTER TABLE {DUCKLAKE_ALIAS}.posthog.groups_production {alter}")++ if accepted:+ ensure_groups_table(lake, "groups_production", DucklingBackfillConfig())+ else:+ with pytest.raises(ValueError, match="schema mismatch"):+ ensure_groups_table(lake, "groups_production", DucklingBackfillConfig())diff --git a/products/managed_warehouse/backend/tests/test_trino_conversion.py b/products/managed_warehouse/backend/tests/test_trino_conversion.pynew file mode 100644index 000000000000..6a7ea1fb07eb--- /dev/null+++ b/products/managed_warehouse/backend/tests/test_trino_conversion.py@@ -0,0 +1,154 @@+import re+from typing import Any++import pytest+from unittest.mock import MagicMock, patch++import psycopg+from dagster import build_op_context+from parameterized import parameterized+from psycopg import sql++from posthog.dags.events_backfill_to_duckling import (+ EVENTS_TABLE_DDL,+ EVENTS_TRINO_COLUMN_TYPES,+ PERSONS_TABLE_DDL,+ PERSONS_TRINO_COLUMN_TYPES,+ DucklingTarget,+)++from products.managed_warehouse.dags.groups_backfill import GROUPS_COLUMN_TYPES, GROUPS_TRINO_COLUMN_TYPES+from products.managed_warehouse.dags.trino_conversion import (+ DucklingTrinoConversionConfig,+ duckling_trino_conversion,+ plan_table_conversion,+)++TARGET = DucklingTarget(+ team_id=2,+ organization_id="org-1",+ bucket="bkt",+ bucket_region="us-east-1",+ events_table="events_team_2",+ persons_table="persons_team_2",+)+++def _ddl_types(ddl: str) -> dict[str, str]:+ columns = re.findall(r"^\s+(\w+) (\w+),?$", ddl, re.MULTILINE)+ return {name: "TIMESTAMP WITH TIME ZONE" if kind == "TIMESTAMPTZ" else kind for name, kind in columns}+++EVENTS_TYPES = _ddl_types(EVENTS_TABLE_DDL)+PERSONS_TYPES = _ddl_types(PERSONS_TABLE_DDL)+CONVERTED_GROUPS_TYPES = {**GROUPS_COLUMN_TYPES, "group_type_index": "SMALLINT"}+++class _FakeConnection:+ def __init__(self, tables: dict[str, dict[str, str]]) -> None:+ self.tables = tables+ self.alters: list[tuple[str, str, str]] = []++ def execute(self, query: str | sql.Composable) -> Any:+ statement = query.as_string() if isinstance(query, sql.Composable) else query+ if statement.startswith("DESCRIBE"):+ table = statement.split(".")[-1].strip('"')+ if table not in self.tables:+ raise psycopg.errors.UndefinedTable(table)+ return MagicMock(fetchall=lambda: list(self.tables[table].items()))+ match = re.fullmatch(+ r'ALTER TABLE "\w+"\."posthog"\."(\w+)" ALTER COLUMN "(\w+)" SET DATA TYPE (.+)', statement+ )+ assert match is not None, statement+ table, column, column_type = match.groups()+ self.alters.append((table, column, column_type))+ self.tables[table][column] = column_type+ return MagicMock()+++class _FakeSession:+ def __init__(self, connection: _FakeConnection) -> None:+ self.connection = connection++ def run(self, what: str, op: Any) -> Any:+ return op(self.connection)++ def close(self) -> None:+ pass+++ [+ (+ "duckgres_groups_widen_the_index",+ "groups",+ GROUPS_COLUMN_TYPES,+ GROUPS_TRINO_COLUMN_TYPES,+ False,+ {"group_type_index": "SMALLINT"},+ {},+ ),+ (+ "duckgres_persons_need_a_rebuild",+ "persons",+ PERSONS_TYPES,+ PERSONS_TRINO_COLUMN_TYPES,+ False,+ {},+ {"person_version": "UBIGINT"},+ ),+ ("duckgres_events_match", "events", EVENTS_TYPES, EVENTS_TRINO_COLUMN_TYPES, False, {}, {}),+ ("converted_groups_need_nothing", "groups", CONVERTED_GROUPS_TYPES, GROUPS_TRINO_COLUMN_TYPES, False, {}, {}),+ (+ "missing_column_needs_a_rebuild",+ "groups",+ {"team_id": "BIGINT"},+ GROUPS_TRINO_COLUMN_TYPES,+ False,+ {},+ {+ "group_type_index": "missing",+ "group_key": "missing",+ "group_properties": "missing",+ "created_at": "missing",+ "_timestamp": "missing",+ "_inserted_at": "missing",+ },+ ),+ ("missing_table", "groups", None, GROUPS_TRINO_COLUMN_TYPES, True, {}, {}),+ ]+)+def test_plan_splits_columns_into_widen_and_rebuild(_name, table, actual, expected, missing, widen, rebuild) -> None:+ plan = plan_table_conversion(table, actual, expected)++ assert (plan.missing, dict(plan.widen), dict(plan.rebuild)) == (missing, widen, rebuild)+++def test_plan_refuses_a_trino_type_without_a_duckdb_mapping() -> None:+ with pytest.raises(ValueError, match="no DuckDB type"):+ plan_table_conversion("t", {"c": "BIGINT"}, {"c": "decimal(20,0)"})++[email protected]([("dry_run", True, 0), ("real_run", False, 1)])+def test_conversion_widens_only_what_the_plan_names_and_only_once(_name, dry_run, expected_alters) -> None:+ connection = _FakeConnection(+ {+ "events_team_2": dict(EVENTS_TYPES),+ "persons_team_2": dict(PERSONS_TYPES),+ "groups_team_2": dict(GROUPS_COLUMN_TYPES),+ }+ )+ config = DucklingTrinoConversionConfig(team_id=2, dry_run=dry_run)++ with (+ patch("products.managed_warehouse.dags.trino_conversion._resolve_duckling_target", return_value=TARGET),+ patch("products.managed_warehouse.dags.trino_conversion.resolve_groups_table", return_value="groups_team_2"),+ patch(+ "products.managed_warehouse.dags.trino_conversion._DuckgresSession", return_value=_FakeSession(connection)+ ),+ ):+ duckling_trino_conversion(build_op_context(), config)+ duckling_trino_conversion(build_op_context(), config)++ assert connection.alters == [("groups_team_2", "group_type_index", "SMALLINT")] * expected_alters+ assert connection.tables["persons_team_2"] == PERSONS_TYPESdiff --git a/products/managed_warehouse/dags/groups_backfill.py b/products/managed_warehouse/dags/groups_backfill.pyindex 1b2ddefa994d..cf3b52f7f377 100644--- a/products/managed_warehouse/dags/groups_backfill.py+++ b/products/managed_warehouse/dags/groups_backfill.py@@ -179,6 +179,9 @@ def ensure_groups_table(if not config.skip_schema_validation:rows = conn.execute(sql.SQL("DESCRIBE {}").format(table)).fetchall()actual = {str(row[0]): str(row[1]) for row in rows}+ # A table converted for Trino holds SMALLINT, and DuckLake accepts this export's UTINYINT files into it.+ if actual.get("group_type_index") == "SMALLINT":+ actual["group_type_index"] = GROUPS_COLUMN_TYPES["group_type_index"]if actual != GROUPS_COLUMN_TYPES:raise ValueError(f"Groups table schema mismatch: expected {GROUPS_COLUMN_TYPES}, got {actual}")diff --git a/products/managed_warehouse/dags/trino_conversion.py b/products/managed_warehouse/dags/trino_conversion.pynew file mode 100644index 000000000000..bae3a713ce8f--- /dev/null+++ b/products/managed_warehouse/dags/trino_conversion.py@@ -0,0 +1,177 @@+from collections.abc import Mapping++import dagster+import psycopg+from psycopg import sql++from posthog.dags.common import JobOwners+from posthog.dags.events_backfill_to_duckling import (+ DUCKLAKE_ALIAS,+ EVENTS_TRINO_COLUMN_TYPES,+ PERSONS_TRINO_COLUMN_TYPES,+ _DuckgresSession,+ _resolve_duckling_target,+ table_exists,+)+from posthog.dataclasses import frozen++from products.managed_warehouse.backend.facade.team_state import resolve_groups_table+from products.managed_warehouse.dags.groups_backfill import GROUPS_TRINO_COLUMN_TYPES++# The names that DuckDB DESCRIBE reports for the Trino types of the backfill exports.+TRINO_TO_DUCKDB_TYPES = {+ "bigint": "BIGINT",+ "integer": "INTEGER",+ "smallint": "SMALLINT",+ "boolean": "BOOLEAN",+ "varchar": "VARCHAR",+ "timestamp(6) with time zone": "TIMESTAMP WITH TIME ZONE",+}++# The in-place type changes of DuckLake that end in a signed type. DuckLake changes UINTEGER only to UBIGINT and+# UBIGINT to no type, so a column of either type needs a rebuild.+PROMOTIONS = frozenset(+ {+ ("TINYINT", "SMALLINT"),+ ("TINYINT", "INTEGER"),+ ("TINYINT", "BIGINT"),+ ("SMALLINT", "INTEGER"),+ ("SMALLINT", "BIGINT"),+ ("INTEGER", "BIGINT"),+ ("UTINYINT", "SMALLINT"),+ ("UTINYINT", "INTEGER"),+ ("UTINYINT", "BIGINT"),+ ("USMALLINT", "INTEGER"),+ ("USMALLINT", "BIGINT"),+ }+)+++class DucklingTrinoConversionConfig(dagster.Config):+ team_id: int+ dry_run: bool = True+++@frozen+class TableConversionPlan:+ table: str+ missing: bool+ widen: Mapping[str, str]+ rebuild: Mapping[str, str]+++def plan_table_conversion(+ table: str, actual: Mapping[str, str] | None, expected_trino: Mapping[str, str]+) -> TableConversionPlan:+ if actual is None:+ return TableConversionPlan(table=table, missing=True, widen={}, rebuild={})+ widen: dict[str, str] = {}+ rebuild: dict[str, str] = {}+ for column, trino_type in expected_trino.items():+ if trino_type not in TRINO_TO_DUCKDB_TYPES:+ raise ValueError(f"The Trino type {trino_type} of column {column} has no DuckDB type mapped")+ target = TRINO_TO_DUCKDB_TYPES[trino_type]+ current = actual.get(column, "missing")+ if current == target:+ continue+ if (current, target) in PROMOTIONS:+ widen[column] = target+ else:+ rebuild[column] = current+ return TableConversionPlan(table=table, missing=False, widen=widen, rebuild=rebuild)+++def _describe(conn: psycopg.Connection[tuple[object, ...]], table: str) -> dict[str, str] | None:+ if not table_exists(conn, DUCKLAKE_ALIAS, "posthog", table):+ return None+ rows = conn.execute(sql.SQL("DESCRIBE {}").format(sql.Identifier(DUCKLAKE_ALIAS, "posthog", table))).fetchall()+ return {str(row[0]): str(row[1]) for row in rows}+++def _widen_column(conn: psycopg.Connection[tuple[object, ...]], table: str, column: str, target: str) -> None:+ # The session can replay this operation after a lost connection, and a repeated ALTER writes a new column version.+ if (_describe(conn, table) or {}).get(column) == target:+ return+ conn.execute(+ sql.SQL("ALTER TABLE {} ALTER COLUMN {} SET DATA TYPE {}").format(+ sql.Identifier(DUCKLAKE_ALIAS, "posthog", table), sql.Identifier(column), sql.SQL(target)+ )+ )+++def _summarize(+ context: dagster.OpExecutionContext, team_id: int, dry_run: bool, plans: Mapping[str, TableConversionPlan]+) -> str:+ verb = "would widen" if dry_run else "widened"+ lines = []+ next_steps = []+ for kind, plan in plans.items():+ if plan.missing:+ lines.append(f"{kind} ({plan.table}): table does not exist, so Trino creates it with the right types")+ continue+ lines.append(+ f"{kind} ({plan.table}): {verb} {dict(plan.widen) or 'nothing'}; needs a rebuild: {dict(plan.rebuild) or 'no'}"+ )+ if plan.rebuild and kind != "events":+ job = f"duckling_{kind}_backfill_job"+ next_steps.append(+ f"Rebuild {kind}: run {job} for partition {team_id} with rebuild_mismatched_table: true, with the org on the Trino flag"+ )+ summary = "\n".join([*lines, *next_steps])+ context.log.info(f"Trino conversion summary for team {team_id}:\n{summary}")+ return summary++[email protected](tags={"owner": JobOwners.TEAM_MANAGED_WAREHOUSE.value})+def duckling_trino_conversion(context: dagster.OpExecutionContext, config: DucklingTrinoConversionConfig) -> None:+ target = _resolve_duckling_target(config.team_id)+ tables = {+ "events": (target.events_table, EVENTS_TRINO_COLUMN_TYPES),+ "persons": (target.persons_table, PERSONS_TRINO_COLUMN_TYPES),+ "groups": (resolve_groups_table(config.team_id), GROUPS_TRINO_COLUMN_TYPES),+ }+ plans: dict[str, TableConversionPlan] = {}+ session = _DuckgresSession(context, target)+ try:+ for kind, (table, expected) in tables.items():++ def describe(conn: psycopg.Connection[tuple[object, ...]], table: str = table) -> dict[str, str] | None:+ return _describe(conn, table)++ plan = plan_table_conversion(table, session.run(f"describe {kind}", describe), expected)+ plans[kind] = plan+ context.log.info(f"Conversion plan for {kind}: {plan}")+ if config.dry_run or not plan.widen:+ continue+ for column, column_type in plan.widen.items():++ def widen(+ conn: psycopg.Connection[tuple[object, ...]],+ table: str = table,+ column: str = column,+ column_type: str = column_type,+ ) -> None:+ _widen_column(conn, table, column, column_type)++ session.run(f"widen {table}.{column}", widen)+ after = session.run(f"describe {kind}", describe) or {}+ wrong = {col: after.get(col) for col, ty in plan.widen.items() if after.get(col) != ty}+ if wrong:+ raise ValueError(f"{table} still has other types than the plan: {wrong}")+ finally:+ session.close()+ summary = _summarize(context, config.team_id, config.dry_run, plans)+ if plans["events"].rebuild:+ raise ValueError(+ f"The events table {plans['events'].table} needs a rebuild that a snapshot run cannot do: "+ f"{dict(plans['events'].rebuild)}"+ )+ context.add_output_metadata({"summary": summary})+++ description="Widens the unsigned columns of a team's duckling tables in place, so Trino can register files",+ tags={"owner": JobOwners.TEAM_MANAGED_WAREHOUSE.value},+)+def duckling_trino_conversion_job() -> None:+ duckling_trino_conversion()