:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): replace synced rows by source file in the loader (#121281)
Tom Owers7d339933c821
diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/README.md b/products/warehouse_sources/backend/temporal/data_imports/pipelines/README.mdindex 5548bceab5af..03d327ca0970 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/README.md+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/README.md@@ -35,6 +35,20 @@ A later attempt of the same workflow run that also reads from the start loads thAn attempt that continues from a cursor merges, because it can read a row again that an earlier attempt already queued.The append keeps one row for each key in a batch, not in the run: a key that the source holds in two batches stays twice until a later run merges that key.+## Incremental runs that replace rows by file++A source that has no primary key can name the origin of each row instead, for example the file it came from.+It sets `SourceResponse.replace_scope` (`ReplaceScope` in `sources/common/typings.py`): a column, the values this run reads again (`replaced`), and the values that are live now (`retained`).+A source sets a scope only while the `warehouse-replace-rows-by-file` flag is on for the schema, because an older loader ignores the scope and fails the keyless batch.++- The pipeline writes the two value sets to `replace_scope.parquet` in the run's folder (`pipeline_v3/replace_scope.py`). Each queue row carries only the column and the manifest path, under `replace_scope` in its metadata.+- Before batch 0 of a fresh attempt writes, the loader deletes each row whose value is in `replaced` or not in `retained` (`pipeline_v3/load/replace_rows.py`). It reads the values in the table from one column, and deletes in commits of up to 1,000 values. Then every batch of the run appends.+- The delete commits do not carry the batch tag, so a batch that failed after its deletes still writes on redelivery. A second delete of the same values finds nothing more to delete.+- The attempt records itself in `sync_type_config["replace_scope_run_uuid"]`, and the loader drops each later batch of an earlier attempt of the same workflow run. The producer also supersedes the queue rows of earlier attempts. The new attempt deletes the rows of every file that an earlier attempt read again.+- A run that reads no rows queues one empty batch, so a scope that only removes rows still reaches the loader.+- A first sync and a reset load into an empty table, so they do not send a scope. A row with no value in the column stays.+- The watermark does not change: it moves only when the run completes, so a failed run reads and deletes the same files again.+## Incremental runs that fail part wayThe final batch of a run promotes the run's staged watermark, so a run that fails after many loaded batches would make the next run read all of them again.diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/extract.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/extract.pyindex 9a01bfc42a5e..f41ed30c474e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/extract.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/extract.py@@ -486,8 +486,9 @@ def validate_incremental_sync(# The Delta merge needs a key to match rows on, so a keyless incremental table fails once a# table already exists. Raise before extraction rather than letting the writer hit it mid-load:# on pipeline v3 the write happens in the load consumer, which can only fail the job, so the- # schema is never paused and the same doomed run repeats on every schedule.- if is_incremental and not is_first_sync and not resource.primary_keys:+ # schema is never paused and the same doomed run repeats on every schedule. A run with a replace+ # scope appends instead of merging, so it needs no key.+ if is_incremental and not is_first_sync and not resource.primary_keys and resource.replace_scope is None:raise MissingPrimaryKeysException()diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/test/test_extract.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/test/test_extract.pyindex 6a88341d4e08..bb5455a5e0c7 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/test/test_extract.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/test/test_extract.py@@ -53,7 +53,11 @@ from products.warehouse_sources.backend.temporal.data_imports.row_tracking imporget_rows,increment_rows,)-from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SortMode, SourceResponse+from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import (+ ReplaceScope,+ SortMode,+ SourceResponse,+)from products.warehouse_sources.backend.temporal.data_imports.util import NonRetryableExceptionfrom products.warehouse_sources.backend.types import IncrementalFieldType@@ -957,13 +961,15 @@ class TestValidateIncrementalSync:[# The failure this guard exists for: a keyless incremental table can never merge into# the Delta table that an earlier run already wrote.- ("keyless_incremental_after_first_sync_raises", True, False, None, True),+ ("keyless_incremental_after_first_sync_raises", True, False, None, False, True),# The first run writes the whole table, so it doesn't need a merge key. Raising here# would break every initial sync of a keyless table.- ("keyless_incremental_first_sync_allowed", True, True, None, False),+ ("keyless_incremental_first_sync_allowed", True, True, None, False, False),# Full refresh overwrites, so it never merges on a key.- ("keyless_full_refresh_allowed", False, False, None, False),- ("incremental_with_key_allowed", True, False, ["id"], False),+ ("keyless_full_refresh_allowed", False, False, None, False, False),+ ("incremental_with_key_allowed", True, False, ["id"], False, False),+ # A replace scope loads as appends, so it never merges on a key.+ ("keyless_incremental_with_a_replace_scope_allowed", True, False, None, True, False),])def test_missing_primary_keys(@@ -972,9 +978,16 @@ class TestValidateIncrementalSync:is_incremental: bool,is_first_sync: bool,primary_keys: list[str] | None,+ replaces_rows: bool,expect_raise: bool,):- resource = MagicMock(primary_keys=primary_keys, has_duplicate_primary_keys=False)+ resource = MagicMock(+ primary_keys=primary_keys,+ has_duplicate_primary_keys=False,+ replace_scope=ReplaceScope(column="_file_name", replaced=frozenset(), retained=frozenset())+ if replaces_rows+ else None,+ )if not expect_raise:validate_incremental_sync(is_incremental, resource, is_first_sync=is_first_sync)diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/delta/test/test_writer.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/delta/test/test_writer.pyindex 7bbe24d1278a..1e8e52715108 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/delta/test/test_writer.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/delta/test/test_writer.py@@ -1,6 +1,7 @@import osimport jsonimport random+import tempfilefrom decimal import Decimalfrom pathlib import Pathfrom types import SimpleNamespace@@ -1924,3 +1925,68 @@ class TestCommitCoversBatch:assert await DeltaWriter(helper).has_batch_been_committed("r", 5) is Trueassert await DeltaWriter(helper).has_batch_been_committed("r", 6) is False+++class TestDeleteWhere:+ _SEED = pa.table(+ {+ "_file_name": ["a.csv", "a.csv", "b.csv", "it's.csv", "c.csv", None],+ "id": [1, 2, 3, 4, 5, 6],+ }+ )++ @parameterized.expand(+ [+ ("in_one_commit", 1000, 1),+ ("in_a_commit_per_two_values", 2, 2),+ ]+ )+ @pytest.mark.asyncio+ async def test_deletes_the_rows_of_the_named_values_only(+ self, _name: str, chunk_size: int, expected_commits: int+ ) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ deltalake.write_deltalake(tmp, self._SEED)+ writer = DeltaWriter(make_local_table_ref(tmp))+ before_commit = MagicMock()++ deleted = await writer.delete_where(+ "_file_name",+ ["b.csv", "it's.csv", "a.csv", "not_in_the_table.csv"],+ chunk_size=chunk_size,+ commit_metadata={"replace_scope_run": "run-1"},+ before_commit=before_commit,+ )++ rows = deltalake.DeltaTable(tmp).to_pyarrow_table().sort_by("id").to_pydict()+ assert deleted == 4+ assert rows == {"_file_name": ["c.csv", None], "id": [5, 6]}+ assert await writer.distinct_values("_file_name") == {"c.csv"}+ deletes = [c for c in deltalake.DeltaTable(tmp).history() if c["operation"] == "DELETE"]+ assert len(deletes) == expected_commits == before_commit.call_count+ # A delete that carried a batch tag would make a redelivered batch skip its own write.+ assert not any(commit_covers_batch(commit, "run-1", 0) for commit in deletes)++ @pytest.mark.asyncio+ async def test_a_delete_retried_after_a_conflict_checks_ownership_again(self) -> None:+ async def retry_once(table: Any, operation_fn: Any, *args: Any, **kwargs: Any) -> Any:+ operation_fn()+ return operation_fn()++ with tempfile.TemporaryDirectory() as tmp:+ deltalake.write_deltalake(tmp, self._SEED)+ writer = DeltaWriter(make_local_table_ref(tmp))+ before_commit = MagicMock()++ with patch(f"{_WRITER_MODULE}.execute_with_conflict_retry", side_effect=retry_once):+ await writer.delete_where("_file_name", ["a.csv"], before_commit=before_commit)++ assert before_commit.call_count == 2++ @parameterized.expand([("no_table", False), ("no_such_column", True)])+ @pytest.mark.asyncio+ async def test_distinct_values_are_unknown_without_the_column(self, _name: str, table_exists: bool) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ if table_exists:+ deltalake.write_deltalake(tmp, pa.table({"id": [1]}))+ assert await DeltaWriter(make_local_table_ref(tmp)).distinct_values("_file_name") is Nonediff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/delta/writer.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/delta/writer.pyindex 845a87a325ad..a6faa1e45750 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/delta/writer.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/delta/writer.py@@ -3,7 +3,7 @@ import timeimport asyncioimport functoolsimport contextlib-from collections.abc import Callable, Sequence+from collections.abc import Callable, Iterable, Sequencefrom typing import TYPE_CHECKING, Any, Literalfrom django.conf import settings@@ -39,6 +39,18 @@ if TYPE_CHECKING:from products.warehouse_sources.backend.temporal.data_imports.pipelines.core.delta.table import DeltaTableRef+# About the size at which a delta-rs `IN (...)` predicate still plans quickly.+DELETE_VALUES_PER_COMMIT = 1000+++def _sql_identifier(name: str) -> str:+ return '"' + name.replace('"', '""') + '"'+++def _sql_string_literal(value: str) -> str:+ return "'" + value.replace("'", "''") + "'"++class DeltaliteFallbackRefusedError(Exception):"""Deltalite failed on a partition too large for the delta-rs MERGE fallback.@@ -815,6 +827,69 @@ class DeltaWriter:self._table.invalidate_cached_table()return restore_metrics+ async def distinct_values(self, column: str) -> set[str] | None:+ """The distinct non-null values of a text `column`, or None when the table or the column is missing.++ The scan reads only that column, one file at a time, so its memory is the set it returns.+ """+ delta_table = await self._table.get_delta_table()+ if delta_table is None or column not in {field.name for field in delta_table.schema().fields}:+ return None+ dataset = delta_table.to_pyarrow_dataset()++ def _scan() -> set[str]:+ values: set[str] = set()+ scanner = dataset.scanner(columns=[column], batch_readahead=1, fragment_readahead=1, use_threads=False)+ for batch in scanner.to_batches():+ values.update(str(value) for value in pc.unique(batch.column(0)).drop_null().to_pylist())+ return values++ return await asyncio.to_thread(_scan)++ async def delete_where(+ self,+ column: str,+ values: Iterable[str],+ *,+ chunk_size: int = DELETE_VALUES_PER_COMMIT,+ commit_metadata: dict[str, str] | None = None,+ before_commit: Callable[[], None] | None = None,+ ) -> int:+ """Delete the rows whose text `column` holds one of `values`, and return how many went.++ Each commit names at most `chunk_size` values, which keeps the predicate small enough to+ plan. A crash between commits leaves some values deleted, so a caller must be able to run+ the same delete again. `before_commit` runs before each commit.++ Never pass a batch tag in `commit_metadata`: `has_batch_been_committed` would then skip the+ write of a batch whose delete committed and whose write did not.+ """+ delta_table = await self._table.get_delta_table()+ ordered = sorted(set(values))+ if delta_table is None or not ordered:+ return 0+ commit_properties = deltalake.CommitProperties(custom_metadata=commit_metadata) if commit_metadata else None+ target = delta_table+ deleted = 0+ for start in range(0, len(ordered), chunk_size):+ literals = ", ".join(_sql_string_literal(value) for value in ordered[start : start + chunk_size])+ predicate = f"{_sql_identifier(column)} IN ({literals})"++ # Inside the operation, so a retry after a commit conflict checks ownership again.+ def _delete(predicate: str = predicate) -> dict[str, Any]:+ if before_commit is not None:+ before_commit()+ return target.delete(predicate, commit_properties=commit_properties)++ metrics = await execute_with_conflict_retry(+ target,+ _delete,+ "delete_where",+ self._logger,+ )+ deleted += int(metrics.get("num_deleted_rows") or 0)+ return deleted+async def has_commit_with_metadata(self, match: dict[str, str], *, scan_limit: int = 50) -> bool:"""Check whether any recent delta commit has custom metadata matching all entries in `match`.diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/coalescing.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/coalescing.pyindex 2e825678cdef..cec6757dae40 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/coalescing.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/coalescing.py@@ -52,6 +52,9 @@ class LoadShape:# batches bound for different destinations are written under different configurations and# must never share a set, the same as two batches with different primary keys.external_destination_ids: tuple[str, ...]+ # A run's replace scope is applied by its first batch, so batches with different scopes, which+ # are batches of different runs, never share a write.+ replace_scope_manifest: str | None = None@frozen@@ -95,6 +98,7 @@ class CoalesceMember:is_first_ever_sync=batch.is_first_ever_sync,cdc_write_mode=metadata.get("cdc_write_mode"),external_destination_ids=_external_ids(metadata.get("external_destination_ids"), batch.destination_ids),+ replace_scope_manifest=_manifest_path(metadata.get("replace_scope")),),)@@ -123,6 +127,7 @@ class CoalesceMember:is_first_ever_sync=signal.is_first_ever_sync,cdc_write_mode=signal.cdc_write_mode,external_destination_ids=_external_ids(signal.external_destination_ids, signal.destination_ids),+ replace_scope_manifest=signal.replace_scope.manifest_path if signal.replace_scope else None,),)@@ -166,6 +171,11 @@ def _arrow_bytes(value: object) -> int | None:return value+def _manifest_path(raw: object) -> str | None:+ path = raw.get("manifest_path") if isinstance(raw, dict) else None+ return path if isinstance(path, str) else None++def _key_tuple(keys: Sequence[str] | None) -> tuple[str, ...] | None:return None if keys is None else tuple(keys)diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/batch_steps.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/batch_steps.pyindex 4bee31597887..019f1d0f3f6b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/batch_steps.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/batch_steps.py@@ -22,6 +22,7 @@ BatchStep = Literal["mark_processed","post_write","compact_during_rebuild",+ "replace_rows",]diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/processor.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/processor.pyindex b94d0ff6e6a2..6f56339a63cf 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/processor.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/processor.py@@ -100,6 +100,10 @@ from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipelineLOADED_PREFIX_PROMOTION_TOTAL,PARQUET_READ_DURATION_SECONDS,)+from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.load.replace_rows import (+ apply_replace_scope,+ replace_scope_applies,+)from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.load.table_handles import (GROUP_TABLE_HANDLES,)@@ -107,6 +111,7 @@ from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipelineExportSignalMessage,SyncTypeLiteral,)+from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.replace_scope import ReplaceScopeRunfrom products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.s3 import (read_parquet,read_parquet_first_values,@@ -1319,7 +1324,8 @@ def _process_message_reported(if export_signal.sync_type == "incremental" and export_signal.cdc_write_mode is None:rebuild = TableRebuildRun(schema.sync_type_config)- if any(rebuild.replaces(run_uuid) for run_uuid, _ in members):+ replace_run = ReplaceScopeRun(schema.sync_type_config)+ if any(rebuild.replaces(run_uuid) or replace_run.replaces(run_uuid) for run_uuid, _ in members):if constituents is not None:raise CoalescingDeclined("a batch belongs to an attempt that a later attempt replaced")logger.info(@@ -1401,6 +1407,14 @@ def _process_message_reported(verify_ownership()append_runs.start_run(export_signal)+ if replace_scope_applies(export_signal):+ # Before the table is read, so the batch is fitted to the table that the deletes leave.+ if verify_ownership is not None:+ verify_ownership()+ report_phase("replace_rows")+ with timer.step("replace_rows"):+ apply_replace_scope(export_signal, schema, delta_table_ref, verify_ownership)+logger.debug("message_received",team_id=export_signal.team_id,@@ -1443,8 +1457,19 @@ def _process_message_reported(with timer.step("table_open"):existing_delta_table = async_to_sync(delta_table_ref.get_delta_table)()- with timer.step("partition"):- pa_table = _apply_partitioning(export_signal, pa_table, existing_delta_table, schema)+ # A run whose source read no rows sends one empty batch only to carry its replace scope.+ is_scope_carrier = export_signal.replace_scope is not None and pa_table.num_rows == 0+ if is_scope_carrier and existing_delta_table is None:+ # Writing the carrier would create a table that holds only the scope column.+ raise ValueError(+ f"The Delta table of schema {export_signal.schema_id} is missing, so the empty batch of "+ f"run {export_signal.run_uuid} cannot apply its replace scope"+ )+ skips_write = is_scope_carrier++ if not skips_write:+ with timer.step("partition"):+ pa_table = _apply_partitioning(export_signal, pa_table, existing_delta_table, schema)# Capture file URIs before write for partial data loading. The listing is O(files in# table), so skip it entirely when no consumer wants it — during a long first sync the@@ -1535,8 +1560,16 @@ def _process_message_reported(primary_keys=primary_keys or [],commit_metadata=commit_metadata,)+ elif skips_write and existing_delta_table is not None:+ delta_table = existing_delta_tableelse:- write_type = _get_write_type(export_signal.sync_type)+ # The scope already removed every row this run brings again, so its batches only add rows.+ # A first sync or a reset keeps the incremental load, whose batch 0 overwrites the table.+ write_type: Literal["incremental", "full_refresh", "append"] = (+ "append"+ if export_signal.replace_scope is not None and not export_signal.is_first_ever_sync+ else _get_write_type(export_signal.sync_type)+ )# First batch should overwrite the table, but only if not resumingshould_overwrite_table = export_signal.batch_index == 0 and not export_signal.is_resumediff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/replace_rows.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/replace_rows.pynew file mode 100644index 000000000000..9dfd45b57d1a--- /dev/null+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/replace_rows.py@@ -0,0 +1,82 @@+"""Applies a run's replace scope before the run's first batch loads (see `ReplaceScope`)."""++from collections.abc import Callable++import structlog+from asgiref.sync import async_to_sync++from products.warehouse_sources.backend.models.external_data_schema import ExternalDataSchema+from products.warehouse_sources.backend.temporal.data_imports.pipelines.core.delta.table import DeltaTableRef+from products.warehouse_sources.backend.temporal.data_imports.pipelines.core.delta.writer import DeltaWriter+from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.messages import ExportSignalMessage+from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.replace_scope import (+ ReplaceScopeRun,+ scope_from_manifest,+)+from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.s3 import read_parquet++logger = structlog.get_logger(__name__)+++def replace_scope_applies(export_signal: ExportSignalMessage) -> bool:+ """Whether this batch deletes rows before it writes.++ Only the first batch of a fresh attempt does. A first sync or a reset writes into an empty+ table, so it has nothing to delete.+ """+ return (+ export_signal.replace_scope is not None+ and export_signal.batch_index == 0+ and not export_signal.is_resume+ and not export_signal.is_first_ever_sync+ )+++def apply_replace_scope(+ export_signal: ExportSignalMessage,+ schema: ExternalDataSchema,+ delta_table_ref: DeltaTableRef,+ verify_ownership: Callable[[], None] | None,+) -> None:+ """Delete the rows of every value the scope replaces and of every value it no longer retains.++ The attempt is recorded first, so a late batch of an earlier attempt is dropped from here on.+ The deletes commit without the batch tag, so a crash before the batch's own write leaves the+ batch to load on redelivery, which deletes the same values again and finds nothing more to+ delete. The values include every file this attempt writes, so a new attempt also removes what+ a failed attempt of the same run appended.+ """+ ref = export_signal.replace_scope+ if ref is None:+ return+ ReplaceScopeRun.record(schema, export_signal.run_uuid)++ writer = DeltaWriter(delta_table_ref)+ present = async_to_sync(writer.distinct_values)(ref.column)+ log = logger.bind(+ team_id=export_signal.team_id,+ external_data_schema_id=export_signal.schema_id,+ run_uuid=export_signal.run_uuid,+ column=ref.column,+ )+ if present is None:+ # Nothing in the table names a value to compare with: the table is new, or an earlier mode+ # wrote it without the column. The source must reset such a table, so this run only appends.+ log.warning("replace_scope_skipped_column_missing")+ return++ scope = scope_from_manifest(ref.column, read_parquet(ref.manifest_path))+ to_delete = (present - scope.retained) | (present & scope.replaced)+ rows_deleted = async_to_sync(writer.delete_where)(+ ref.column,+ to_delete,+ commit_metadata={"replace_scope_run": export_signal.run_uuid},+ before_commit=verify_ownership,+ )+ log.info(+ "replace_scope_applied",+ values_in_table=len(present),+ values_replaced=len(present & scope.replaced),+ values_removed=len(present - scope.retained),+ rows_deleted=rows_deleted,+ )diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/test_processor.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/test_processor.pyindex 7d5ef79f7edd..3793c93c237d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/test_processor.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/load/test_processor.py@@ -58,7 +58,9 @@ from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipelineGroupTableHandles,)from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.messages import ExportSignalMessage+from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.replace_scope import manifest_tablefrom products.warehouse_sources.backend.temporal.data_imports.pipelines.test_mocks import mock_delta_table+from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import ReplaceScopefrom products.warehouse_sources.backend.types import ExternalDataSourceTypefrom products.warehouse_sources_queue.backend.core.batch_consumer import CoalescingDeclinedfrom products.warehouse_sources_queue.backend.core.jobs_db import LoadedPrefix, UnfinishedEarlierBatches@@ -305,6 +307,9 @@ def _message(**overrides: Any) -> dict[str, Any]:_PROCESSOR = "products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.load.processor"+_REPLACE_ROWS = "products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.load.replace_rows"+_REPLACE_SCOPE = "products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.replace_scope"+_IDEMPOTENCY = "products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.load.idempotency"_POST_LOAD_RESULT = PostLoadResult(queryable_folder="folder", table_size_written=True)@@ -1969,6 +1974,203 @@ class TestTableRebuildLoad:assert [name for row_id, name in self._table_rows(tmp) if row_id == 4] == ["later_4"]+FileRows = dict[str, list[int]]+++class TestReplaceScopeLoad:+ # The table before the run: the rows of three files of the bucket.+ _SEED: FileRows = {"a.csv": [1, 2], "b.csv": [3, 4], "c.csv": [5]}++ @staticmethod+ def _table(file_rows: FileRows) -> pa.Table:+ if not file_rows:+ # What the pipeline queues for a run that read no rows.+ return pa.table({"_file_name": pa.array([], pa.string())})+ names = [name for name, ids in file_rows.items() for _ in ids]+ ids = [row_id for row_ids in file_rows.values() for row_id in row_ids]+ return pa.table({"_file_name": pa.array(names, pa.string()), "id": pa.array(ids, pa.int64())})++ @contextmanager+ def _loader(+ self, table_path: str, files: dict[str, pa.Table], config: dict[str, Any], *, redis: bool = False+ ) -> Iterator[None]:+ def local_ref(**kwargs: Any) -> DeltaTableRef:+ ref = make_local_table_ref(table_path)+ ref._is_first_sync = kwargs["is_first_sync"]+ return ref++ def update_config(*args: Any, updates: dict[str, Any], **kwargs: Any) -> dict[str, Any]:+ config.update(updates)+ return config++ @contextmanager+ def no_redis() -> Iterator[None]:+ yield None++ with ExitStack() as stack:+ stack.enter_context(patch(f"{_PROCESSOR}.posthoganalytics"))+ stack.enter_context(patch(f"{_PROCESSOR}.mark_batch_as_processed"))+ if not redis:+ # The redelivery check then reads the table history, which is what finds a landed write.+ stack.enter_context(patch(f"{_PROCESSOR}.is_batch_already_processed", return_value=False))+ stack.enter_context(patch(f"{_IDEMPOTENCY}.get_redis_client", no_redis))+ for module in (_PROCESSOR, _REPLACE_ROWS):+ stack.enter_context(patch(f"{module}.read_parquet", side_effect=lambda path: files[path]))+ stack.enter_context(patch(f"{_REPLACE_SCOPE}.update_sync_type_config_keys", side_effect=update_config))+ stack.enter_context(patch(f"{_PROCESSOR}.DeltaTableRef", side_effect=local_ref))+ job_model = stack.enter_context(patch(f"{_PROCESSOR}.ExternalDataJob"))+ job = MagicMock()+ job.schema.sync_type_config = config+ job_model.objects.prefetch_related.return_value.get.return_value = job+ yield++ def _load(+ self,+ table_path: str,+ attempt: int,+ batches: list[FileRows],+ scope: ReplaceScope,+ *,+ indexes: list[int] | None = None,+ config: dict[str, Any] | None = None,+ is_first_ever_sync: bool = False,+ delivery: int = 1,+ redis: bool = False,+ ) -> None:+ run_uuid = f"{_WORKFLOW_RUN}-a{attempt}"+ manifest_path = f"s3://bucket/{run_uuid}/replace_scope.parquet"+ files = {f"s3://bucket/{run_uuid}/{index}": self._table(rows) for index, rows in enumerate(batches)}+ files[manifest_path] = manifest_table(scope)+ with self._loader(table_path, files, config if config is not None else {}, redis=redis):+ for index in indexes if indexes is not None else range(len(batches)):+ process_message(+ _message(+ run_uuid=run_uuid,+ batch_index=index,+ s3_path=f"s3://bucket/{run_uuid}/{index}",+ primary_keys=None,+ is_first_ever_sync=is_first_ever_sync,+ replace_scope={"column": "_file_name", "manifest_path": manifest_path},+ ),+ attempt=delivery,+ )++ @staticmethod+ def _scope(replaced: set[str], retained: set[str]) -> ReplaceScope:+ return ReplaceScope(column="_file_name", replaced=frozenset(replaced), retained=frozenset(retained))++ @staticmethod+ def _rows_by_file(table_path: str) -> FileRows:+ rows: FileRows = {}+ for name, row_id in sorted(+ zip(*DeltaTable(table_path).to_pyarrow_table().select(["_file_name", "id"]).to_pydict().values())+ ):+ rows.setdefault(name, []).append(row_id)+ return rows++ @staticmethod+ def _operations(table_path: str) -> list[str]:+ return [commit["operation"] for commit in DeltaTable(table_path).history()]++ @parameterized.expand(+ [+ ("a_deleted_file_loses_its_rows", [{}], set(), {"a.csv", "c.csv"}, {"a.csv": [1, 2], "c.csv": [5]}),+ (+ "a_shrunk_file_keeps_only_its_new_rows_and_a_new_file_adds_its_rows",+ [{"b.csv": [3]}, {"d.csv": [7, 8]}],+ {"b.csv", "d.csv"},+ {"a.csv", "b.csv", "c.csv", "d.csv"},+ {"a.csv": [1, 2], "b.csv": [3], "c.csv": [5], "d.csv": [7, 8]},+ ),+ ]+ )+ def test_the_table_mirrors_the_files_of_the_scope(+ self, _name: str, batches: list[FileRows], replaced: set[str], retained: set[str], expected: FileRows+ ) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ write_deltalake(tmp, self._table(self._SEED))++ self._load(tmp, 1, batches, self._scope(replaced, retained))++ assert self._rows_by_file(tmp) == expected+ assert "MERGE" not in self._operations(tmp)++ @parameterized.expand(+ [+ ("an_attempt_replaced_after_two_batches", [0, 1]),+ ("an_attempt_replaced_before_its_first_batch_loads", []),+ ]+ )+ def test_a_retried_run_gives_the_table_of_one_attempt(self, _name: str, first_attempt_batches: list[int]) -> None:+ batches: list[FileRows] = [{"b.csv": [3]}, {"d.csv": [7]}, {"e.csv": [9]}]+ scope = self._scope({"b.csv", "d.csv", "e.csv"}, {"a.csv", "b.csv", "c.csv", "d.csv", "e.csv"})+ with tempfile.TemporaryDirectory() as tmp:+ retried, once = f"{tmp}/retried", f"{tmp}/once"+ write_deltalake(retried, self._table(self._SEED))+ write_deltalake(once, self._table(self._SEED))+ config: dict[str, Any] = {}++ self._load(retried, 1, batches, scope, indexes=first_attempt_batches, config=config)+ self._load(retried, 2, batches, scope, indexes=[0], config=config)+ # A batch of the first attempt that the loader takes after the second attempt started.+ self._load(retried, 1, batches, scope, indexes=[2], config=config)+ self._load(retried, 2, batches, scope, indexes=[1, 2], config=config)+ self._load(once, 1, batches, scope)++ assert self._rows_by_file(retried) == self._rows_by_file(once)+ assert self._rows_by_file(once) == {"a.csv": [1, 2], "b.csv": [3], "c.csv": [5], "d.csv": [7], "e.csv": [9]}++ def test_a_first_batch_redelivered_after_its_deletes_still_writes(self) -> None:+ scope = self._scope({"b.csv"}, {"a.csv", "b.csv", "c.csv"})+ original_write = DeltaWriter.write++ async def worker_dies_once(writer: DeltaWriter, **kwargs: Any) -> DeltaTable:+ if not crashed:+ crashed.append(True)+ raise RuntimeError("the worker died before the write")+ return await original_write(writer, **kwargs)++ crashed: list[bool] = []+ with tempfile.TemporaryDirectory() as tmp:+ write_deltalake(tmp, self._table(self._SEED))+ with patch.object(DeltaWriter, "write", worker_dies_once), pytest.raises(RuntimeError):+ self._load(tmp, 1, [{"b.csv": [3]}], scope, redis=True)+ assert "DELETE" in self._operations(tmp)++ with patch.object(DeltaWriter, "write", worker_dies_once):+ self._load(tmp, 1, [{"b.csv": [3]}], scope, delivery=2, redis=True)++ assert self._rows_by_file(tmp) == {"a.csv": [1, 2], "b.csv": [3], "c.csv": [5]}++ @parameterized.expand([("a_first_sync", True, True), ("no_table", False, False)])+ def test_nothing_is_deleted_without_rows_to_replace(+ self, _name: str, is_first_ever_sync: bool, seeded: bool+ ) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ if seeded:+ write_deltalake(tmp, self._table(self._SEED))++ self._load(+ tmp,+ 1,+ [{"b.csv": [3]}],+ self._scope({"b.csv"}, {"b.csv"}),+ is_first_ever_sync=is_first_ever_sync,+ )++ assert self._rows_by_file(tmp) == {"b.csv": [3]}+ assert "DELETE" not in self._operations(tmp)++ def test_an_empty_batch_never_creates_a_table(self) -> None:+ with tempfile.TemporaryDirectory() as tmp:+ table_path = f"{tmp}/table"++ with pytest.raises(ValueError, match="is missing"):+ self._load(table_path, 1, [{}], self._scope(set(), {"a.csv"}))++ assert not DeltaTable.is_deltatable(table_path)++def _batch_rows(index: int) -> pa.Table:# Each batch updates one row of the batch before it and adds two rows.ids = [index * 2, index * 2 + 1, index * 2 + 2]diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/messages.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/messages.pyindex 38cdd059ddd7..b55e873f5c7d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/messages.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/messages.py@@ -2,6 +2,8 @@ import timefrom dataclasses import asdict, dataclass, fieldfrom typing import Any, Literal, Optional+from posthog.dataclasses import frozen+from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import (PartitionFormat,PartitionMode,@@ -10,6 +12,30 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.typSyncTypeLiteral = Literal["full_refresh", "incremental", "append", "cdc"]+@frozen+class ReplaceScopeRef:+ """Where the loader finds a run's replace scope (see `ReplaceScope`).++ The value sets can hold every file of a bucket, so they travel in a manifest file next to the+ run's batches. Only the manifest's path and the column name go into each queue row.+ """++ column: str+ manifest_path: str++ def to_dict(self) -> dict[str, str]:+ return {"column": self.column, "manifest_path": self.manifest_path}++ @classmethod+ def from_dict(cls, raw: object) -> "ReplaceScopeRef | None":+ if not isinstance(raw, dict):+ return None+ column, manifest_path = raw.get("column"), raw.get("manifest_path")+ if not isinstance(column, str) or not isinstance(manifest_path, str):+ return None+ return cls(column=column, manifest_path=manifest_path)++@dataclass(frozen=True)class ExportSignalMessage:"""One extracted batch's hand-off from the extraction side to the load consumer.@@ -55,6 +81,7 @@ class ExportSignalMessage:# None on a message queued before the producer recorded it, which readers treat as unknown# rather than as none. Resolved once per run, because the consumer reads it per batch.external_destination_ids: Optional[list[str]] = None+ replace_scope: Optional[ReplaceScopeRef] = Nonedef to_dict(self) -> dict[str, Any]:return asdict(self)@@ -92,4 +119,5 @@ class ExportSignalMessage:cdc_table_mode=data.get("cdc_table_mode"),destination_ids=data.get("destination_ids") or [],external_destination_ids=data.get("external_destination_ids"),+ replace_scope=ReplaceScopeRef.from_dict(data.get("replace_scope")),)diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/pipeline.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/pipeline.pyindex 5f2d977dfe7a..c6d8db6c601c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/pipeline.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/pipeline.py@@ -92,6 +92,7 @@ from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipelineIncrementalBatchRangeReader,IncrementalHandoffCheckpoint,)+from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.messages import ReplaceScopeReffrom products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.metrics import (get_batches_produced_metric,get_pipeline_run_duration_metric,@@ -193,6 +194,8 @@ class PipelineV3(Generic[ResumableData]):_prefix_checkpoint: IncrementalHandoffCheckpoint | None = None_job_rows_synced_through_batch: int = 0_kept_rows_synced: int | None = None+ # Set once the run handed its replace scope to the loader.+ _replace_scope_ref: ReplaceScopeRef | None = Nonedef __init__(self,@@ -219,7 +222,10 @@ class PipelineV3(Generic[ResumableData]):# Persisted PK (user override or earlier detection) > live-detected > `id` fallback. Keeps# the merge key stable across runs when live detection (e.g. Snowflake SHOW PRIMARY KEYS)# intermittently returns nothing.- self._resource.primary_keys = resolve_primary_keys(models.schema, self._resource)+ # A replace scope loads as appends, so a key persisted by an earlier mode must not come back.+ self._resource.primary_keys = (+ None if source_response.replace_scope is not None else resolve_primary_keys(models.schema, self._resource)+ )self._job = models.jobself._reset_pipeline = reset_pipeline@@ -274,6 +280,10 @@ class PipelineV3(Generic[ResumableData]):self._source_resume_manager = resumable_source_managerself._resumable_source_manager = resolve_resume_manager(resumable_source_manager, self._resource)+ if self._resumable_source_manager is not None and source_response.replace_scope is not None:+ # A resumed attempt skips the rows before its cursor, and stages no scope, so its keyless+ # batches would merge. Each attempt of a scoped run must read from the start.+ raise ValueError("A source that resumes from its own cursor cannot send a replace scope")self._preemption = preemptionself._resumed_incremental_run_uuid = resumed_incremental_run_uuidself._sent_resumed_run_finalization = False@@ -414,6 +424,7 @@ class PipelineV3(Generic[ResumableData]):The source must promise ascending order, and nothing else may own the restart point: asource that resumes from its own cursor reads that cursor, and a reset reloads the table.A run that writes several tables keeps one queue per table, which one value cannot cover.+ A replace scope makes each attempt delete every file it reads again, so it must read from the start."""return (self._schema.is_incremental@@ -423,6 +434,7 @@ class PipelineV3(Generic[ResumableData]):and self._resumable_source_manager is Noneand not source_response.lanesand source_response.cdc_write_mode is None+ and source_response.replace_scope is Noneand self._run_uuid is not None)@@ -551,6 +563,25 @@ class PipelineV3(Generic[ResumableData]):self._pg_producer.is_first_ever_sync = Trueself._prefix_checkpoint = None+ async def _stage_replace_scope(self, sync_type: SyncTypeLiteral) -> None:+ """Hand the source's replace scope to the loader, before the first batch is queued.++ Not called for a first sync or a reset: the table starts empty, so no row needs to go.+ """+ scope = self._resource.replace_scope+ if scope is None:+ return+ if sync_type != "incremental" or self._resource.lanes:+ await self._logger.awarning("V3 Pipeline: Ignoring a replace scope on a run that does not merge one table")+ return+ manifest_path = await asyncio.to_thread(self._s3_batch_writer.write_replace_scope, scope)+ self._replace_scope_ref = ReplaceScopeRef(column=scope.column, manifest_path=manifest_path)+ self._pg_producer.replace_scope = self._replace_scope_ref++ async def _stage_empty_replace_scope_batch(self, scope: ReplaceScopeRef) -> None:+ """Queue one batch with no rows, so a scope that only removes rows still reaches the loader."""+ await self._stage_batch(pa.table({scope.column: pa.array([], pa.string())}), batch_index=0, row_count=0)+async def _record_table_rebuild(self, run_uuid: str) -> None:"""Record that this attempt loads the table from empty (see `TableRebuildRun`).@@ -825,6 +856,9 @@ class PipelineV3(Generic[ResumableData]):is_cdc_companion=self._maintains_companion_table(),)+ if not is_fresh_sync and not should_resume:+ await self._stage_replace_scope(sync_type)+async def stage_remaining_rows() -> None:nonlocal chunk_index, row_countwhile self._batcher.should_yield(include_incomplete_chunk=True):@@ -948,6 +982,9 @@ class PipelineV3(Generic[ResumableData]):# The source ended, so it holds no rows and its last cursor is safe.self._confirm_resume_state()await stage_remaining_rows()+ if chunk_index == 0 and self._replace_scope_ref is not None:+ await self._stage_empty_replace_scope_batch(self._replace_scope_ref)+ chunk_index += 1await self._finalize(row_count=row_count)# With zero batches, `_finalize` sent no final-batch notification, so the load@@ -1025,6 +1062,8 @@ class PipelineV3(Generic[ResumableData]):*(self._schema.partitioning_keys_override or []),*(self._schema.partitioning_keys or []),*(self._resource.partition_keys or []),+ # The loader deletes by this column, so a row without it could never be replaced.+ *([self._resource.replace_scope.column] if self._resource.replace_scope else []),],self._observed_columns,self._logger,diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/postgres_queue/producer.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/postgres_queue/producer.pyindex 3c0cdee59ff3..a87ed7836e5b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/postgres_queue/producer.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/postgres_queue/producer.py@@ -20,7 +20,10 @@ import psycopgimport structlogfrom structlog.types import FilteringBoundLogger-from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.messages import SyncTypeLiteral+from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.messages import (+ ReplaceScopeRef,+ SyncTypeLiteral,+)from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.s3 import BatchWriteResultfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import (PartitionFormat,@@ -121,6 +124,8 @@ class PostgresProducer:None if external_destination_ids is None else list(external_destination_ids))+ self._replace_scope: ReplaceScopeRef | None = None+with _queue_db_errors():self._conn = _connect_with_retry(database_url)self._batches_sent = 0@@ -141,6 +146,15 @@ class PostgresProducer:def is_first_ever_sync(self, value: bool) -> None:self._is_first_ever_sync = value+ @property+ def replace_scope(self) -> ReplaceScopeRef | None:+ """Set before the run's first batch, so every queue row of the run carries it."""+ return self._replace_scope++ @replace_scope.setter+ def replace_scope(self, value: ReplaceScopeRef | None) -> None:+ self._replace_scope = value+@propertydef has_held_batch(self) -> bool:return self._held is not None@@ -276,6 +290,9 @@ class PostgresProducer:## An append is the same: the loader removes an older attempt's rows when this# run's batch 0 arrives (see `load/append_rollback.py`).+ #+ # A run with a replace scope is the same: its batch 0 deletes the rows of every+ # file it reads again, which covers every file an older attempt of the run read.with _queue_db_errors():superseded = BatchQueue.supersede_other_runs(self._conn,@@ -284,6 +301,7 @@ class PostgresProducer:spare_runs_with_progress=(self._sync_type != "full_refresh"and not (self._sync_type == "append" and settings.DATA_WAREHOUSE_APPEND_ROLLBACK_ENABLED)+ and self._replace_scope is None),)if superseded > 0:@@ -335,6 +353,8 @@ class PostgresProducer:metadata["external_destination_ids"] = self._external_destination_idsif loaded_prefix_cursor is not None:metadata[LOADED_PREFIX_CURSOR_KEY] = loaded_prefix_cursor+ if self._replace_scope is not None:+ metadata["replace_scope"] = self._replace_scope.to_dict()with _queue_db_errors():self._conn.execute(diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/postgres_queue/test_jobs_db.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/postgres_queue/test_jobs_db.pyindex 0b8aa85dd1ea..5c8d1ba70641 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/postgres_queue/test_jobs_db.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/postgres_queue/test_jobs_db.py@@ -13,6 +13,10 @@ from unittest.mock import AsyncMock, patchimport psycopgfrom products.warehouse_sources.backend.temporal.data_imports.metrics import LOCK_TAKEOVER_LATEST_ERROR+from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.messages import (+ ExportSignalMessage,+ ReplaceScopeRef,+)from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.postgres_queue.consumer import (BatchConsumer,ConsumerConfig,@@ -1073,6 +1077,7 @@ class TestPendingBatchToExportSignal:"data_folder": "/tmp/data","primary_keys": ["id"],"cdc_write_mode": "upsert",+ "replace_scope": {"column": "_file_name", "manifest_path": "s3://bucket/replace_scope.parquet"},},latest_attempt=2,)@@ -1092,6 +1097,10 @@ class TestPendingBatchToExportSignal:assert signal["data_folder"] == "/tmp/data"assert signal["primary_keys"] == ["id"]assert signal["cdc_write_mode"] == "upsert"+ # Without the scope the loader would append the run's rows next to the rows they replace.+ assert ExportSignalMessage.from_dict(signal).replace_scope == ReplaceScopeRef(+ column="_file_name", manifest_path="s3://bucket/replace_scope.parquet"+ )@pytest.mark.django_db(transaction=True)diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/postgres_queue/test_producer.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/postgres_queue/test_producer.pyindex 343ab6cf73eb..2aabe2f457f0 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/postgres_queue/test_producer.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/postgres_queue/test_producer.py@@ -7,6 +7,7 @@ from unittest.mock import MagicMock, PropertyMock, patchimport psycopg+from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.messages import ReplaceScopeReffrom products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.postgres_queue.producer import (PostgresProducer,)@@ -110,6 +111,7 @@ class TestPostgresProducerSendBatch:cdc_write_mode="upsert",cdc_table_mode="merge",)+ producer.replace_scope = ReplaceScopeRef(column="_file_name", manifest_path="s3://bucket/manifest.parquet")batch_result = dataclasses.replace(_make_batch_result(batch_index=1), arrow_bytes=4096)producer.send_batch_notification(batch_result)@@ -122,6 +124,7 @@ class TestPostgresProducerSendBatch:assert metadata["partition_count"] == 4assert metadata["cdc_write_mode"] == "upsert"assert metadata["cdc_table_mode"] == "merge"+ assert metadata["replace_scope"] == {"column": "_file_name", "manifest_path": "s3://bucket/manifest.parquet"}assert metadata["timestamp_ns"] == 123456789@@ -184,18 +187,27 @@ class TestPostgresProducerSupersede:)@pytest.mark.parametrize(- "sync_type,spare",- [("full_refresh", False), ("incremental", True), ("append", False), ("cdc", True)],+ "sync_type,replaces_rows,spare",+ [+ ("full_refresh", False, False),+ ("incremental", False, True),+ ("incremental", True, False),+ ("append", False, False),+ ("cdc", False, True),+ ],)- def test_only_full_refresh_and_append_supersede_a_run_that_is_still_loading(- self, settings: Any, sync_type: str, spare: bool+ def test_only_full_refresh_append_and_replace_scope_supersede_a_run_that_is_still_loading(+ self, settings: Any, sync_type: str, replaces_rows: bool, spare: bool) -> None:"""A fresh full_refresh overwrites the table on batch 0, and the loader removes an older- append attempt's rows on batch 0, so the loaded rows of an older attempt are discarded+ append attempt's rows on batch 0, and a run with a replace scope deletes on batch 0 the rows of+ every file an older attempt read, so the loaded rows of an older attempt are discardedeither way and its queued batches are dead weight on the serial per-schema gate. The othersync types keep the sparing rule, because partially merged work survives."""settings.DATA_WAREHOUSE_APPEND_ROLLBACK_ENABLED = Trueproducer = _make_producer(is_resume=False, sync_type=sync_type)+ if replaces_rows:+ producer.replace_scope = ReplaceScopeRef(column="_file_name", manifest_path="s3://bucket/manifest.parquet")batch_result = _make_batch_result(batch_index=0)with patch(diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/replace_scope.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/replace_scope.pynew file mode 100644index 000000000000..df84bb61eef5--- /dev/null+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/replace_scope.py@@ -0,0 +1,66 @@+"""Both halves of a run's replace scope: the manifest the extract side writes and the loader reads,+and the record that lets the loader drop the batches of an attempt that a later attempt replaced.++See `ReplaceScope` for what the loader does with a scope.+"""++from typing import Any++import pyarrow as pa++from products.warehouse_sources.backend.models.external_data_schema import (+ ExternalDataSchema,+ attempt_of,+ update_sync_type_config_keys,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import ReplaceScope++REPLACE_SCOPE_RUN_KEY = "replace_scope_run_uuid"+MANIFEST_FILE_NAME = "replace_scope.parquet"+_REPLACED_COLUMN = "replaced"+_RETAINED_COLUMN = "retained"+++def manifest_table(scope: ReplaceScope) -> pa.Table:+ """The two value sets as two columns. The shorter column is padded with nulls."""+ replaced, retained = sorted(scope.replaced), sorted(scope.retained)+ length = max(len(replaced), len(retained))+ return pa.table(+ {+ _REPLACED_COLUMN: pa.array(replaced + [None] * (length - len(replaced)), pa.string()),+ _RETAINED_COLUMN: pa.array(retained + [None] * (length - len(retained)), pa.string()),+ }+ )+++def scope_from_manifest(column: str, table: pa.Table) -> ReplaceScope:+ def values(name: str) -> frozenset[str]:+ return frozenset(value for value in table.column(name).to_pylist() if value is not None)++ return ReplaceScope(column=column, replaced=values(_REPLACED_COLUMN), retained=values(_RETAINED_COLUMN))+++class ReplaceScopeRun:+ """The newest attempt of a workflow run whose first batch applied a replace scope.++ That batch deleted the rows of every file the attempt reads, which includes every file an+ earlier attempt of the same run read. A batch of an earlier attempt that loads afterwards would+ put rows back that the newer attempt also appends, so the loader drops it.+ """++ def __init__(self, config: dict[str, Any] | None) -> None:+ raw = (config or {}).get(REPLACE_SCOPE_RUN_KEY)+ self._attempt = attempt_of(raw) if isinstance(raw, str) else None++ def replaces(self, run_uuid: str) -> bool:+ """Whether `run_uuid` is an earlier attempt of the workflow run that applied the scope."""+ candidate = attempt_of(run_uuid)+ if self._attempt is None or candidate is None:+ return False+ return self._attempt[0] == candidate[0] and candidate[1] < self._attempt[1]++ @staticmethod+ def record(schema: ExternalDataSchema, run_uuid: str) -> None:+ schema.sync_type_config = update_sync_type_config_keys(+ schema.id, schema.team_id, updates={REPLACE_SCOPE_RUN_KEY: run_uuid}+ )diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/s3/writer.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/s3/writer.pyindex 57baec568703..00c01dc65e64 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/s3/writer.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/s3/writer.py@@ -25,6 +25,10 @@ from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipelineget_s3_write_duration_metric,get_s3_write_errors_metric,)+from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.replace_scope import (+ MANIFEST_FILE_NAME,+ manifest_table,+)from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.s3.common import (BatchWriteResult,cleanup_folder,@@ -34,6 +38,7 @@ from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipelineget_date_partition,strip_s3_protocol,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import ReplaceScopeParquetCompression = Literal["gzip", "bz2", "brotli", "lz4", "zstd", "snappy", "none"]@@ -220,6 +225,17 @@ class S3BatchWriter:return schema_path+ def write_replace_scope(self, scope: ReplaceScope) -> str:+ """Write the scope's value sets next to the run's batches and return the manifest's path."""+ manifest_path = f"{self._base_folder}/{MANIFEST_FILE_NAME}"+ try:+ _write_parquet_to_s3(self._s3, strip_s3_protocol(manifest_path), manifest_table(scope), self._compression)+ except Exception as e:+ if is_object_store_permission_denied(e):+ raise ObjectStorePermissionDeniedError(OBJECT_STORE_PERMISSION_DENIED_MESSAGE) from e+ raise+ return manifest_path+def get_base_folder(self) -> str:return self._base_folderdiff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/test_pipeline.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/test_pipeline.pyindex f70c854f1c90..4fbee007e583 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/test_pipeline.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/test_pipeline.py@@ -35,6 +35,9 @@ from products.warehouse_sources.backend.temporal.data_imports.pipelines.common.pfrom products.warehouse_sources.backend.temporal.data_imports.pipelines.core.abandonable_iterate import (SourceAbandonedError,)+from products.warehouse_sources.backend.temporal.data_imports.pipelines.core.arrow_utils import (+ observe_and_project_table as _real_observe_and_project_table,+)from products.warehouse_sources.backend.temporal.data_imports.pipelines.core.batcher import Batcherfrom products.warehouse_sources.backend.temporal.data_imports.pipelines.core.consts import PARTITION_KEYfrom products.warehouse_sources.backend.temporal.data_imports.pipelines.core.partitioning import (@@ -48,6 +51,7 @@ from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipelineLanedPipelineV3,_LaneWriter,)+from products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.messages import ReplaceScopeReffrom products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.pipeline import (PipelineV3,StagedSchemeWithoutResetError,@@ -64,6 +68,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import (OutputLane,+ ReplaceScope,SortMode,SourceInputs,SourceResponse,@@ -97,7 +102,7 @@ def _make_pipeline() -> PipelineV3:with patch.object(PipelineV3, "__init__", return_value=None):pipeline = PipelineV3.__new__(PipelineV3)- pipeline._resource = MagicMock(name="test_table", primary_keys=["id"], lanes=None)+ pipeline._resource = MagicMock(name="test_table", primary_keys=["id"], lanes=None, replace_scope=None)pipeline._resource_name = "test_table"pipeline._job = MagicMock(team_id=1, workflow_run_id="run-abc", billable=False)pipeline._source = MagicMock(source_type="Postgres")@@ -175,6 +180,7 @@ class TestAttemptScopedRunUuid:partition_mode=None,cdc_write_mode=None,lanes=None,+ replace_scope=None,)with (@@ -317,6 +323,146 @@ class TestTableRebuildAcrossAttempts:assert schema.sync_type_config.get("table_rebuild_run_uuid") == expected_record+class TestReplaceScopeHandOff:+ @pytest.mark.parametrize(+ "table_is_new, expect_scope",+ [+ pytest.param(False, True, id="a_run_on_an_existing_table_hands_the_scope_on"),+ pytest.param(True, False, id="a_first_sync_or_a_reset_has_no_rows_to_replace"),+ ],+ )+ @pytest.mark.asyncio+ async def test_a_run_that_reads_no_rows_still_reaches_the_loader_with_its_scope(+ self, table_is_new: bool, expect_scope: bool+ ) -> None:+ pipeline = _make_pipeline()+ pipeline._run_uuid = "run-abc-a1"+ pipeline._is_incremental = True+ pipeline._schema.table = None if table_is_new else MagicMock()+ pipeline._table = pipeline._schema.table+ pipeline._schema.sync_type_config = {}+ pipeline._delta_table_ref = MagicMock(is_first_sync=table_is_new)+ pipeline._resource.primary_keys = None+ pipeline._resource.has_duplicate_primary_keys = False+ pipeline._last_incremental_field_value = None+ scope = ReplaceScope(column="_file_name", replaced=frozenset(), retained=frozenset({"kept.csv"}))+ pipeline._resource.replace_scope = scope+ s3_writer = cast(MagicMock, pipeline._s3_batch_writer)+ s3_writer.write_replace_scope.return_value = "s3://bucket/run-abc-a1/replace_scope.parquet"+ producer = MagicMock(sync_type="incremental", is_first_ever_sync=False, has_held_batch=False)+ pipeline._pg_producer = producer++ with ExitStack() as stack:+ for name in (+ "reset_rows_synced_if_needed",+ "setup_row_tracking_with_billing_check",+ "persist_primary_keys",+ "handle_reset_or_full_refresh",+ "handle_corrupted_delta_log",+ "finalize_desc_sort_incremental_value",+ "commit_source_cursor",+ ):+ stack.enter_context(patch(f"{_PIPELINE}.{name}", new_callable=AsyncMock))+ stack.enter_context(patch(f"{_PIPELINE}.DeltaMaintenance")).return_value.run_scheduled = AsyncMock()+ stack.enter_context(patch(f"{_PIPELINE}.activity")).in_activity.return_value = False+ stack.enter_context(patch(f"{_TABLE_REBUILD}.update_sync_type_config_keys"))+ stack.enter_context(patch(f"{_PIPELINE}.update_sync_type_config_keys"))+ pipeline._resource.items = MagicMock(return_value=iter([]))+ pipeline._batcher.should_yield.return_value = False # type: ignore[attr-defined]++ await pipeline.run()++ if not expect_scope:+ s3_writer.write_replace_scope.assert_not_called()+ s3_writer.write_batch.assert_not_called()+ return+ s3_writer.write_replace_scope.assert_called_once_with(scope)+ assert producer.replace_scope == ReplaceScopeRef(+ column="_file_name", manifest_path="s3://bucket/run-abc-a1/replace_scope.parquet"+ )+ [(empty_batch, batch_index), _] = s3_writer.write_batch.call_args+ assert (empty_batch.num_rows, empty_batch.column_names, batch_index) == (0, ["_file_name"], 0)+ producer.send_final_batch.assert_called_once()++ def test_a_resumable_source_cannot_send_a_scope(self) -> None:+ resource = MagicMock(+ name="users",+ primary_keys=None,+ lanes=None,+ cdc_write_mode=None,+ replace_scope=ReplaceScope(column="_file_name", replaced=frozenset(), retained=frozenset()),+ )+ manager = MagicMock(can_resume=MagicMock(return_value=False))+ with (+ patch(f"{_PIPELINE}.current_import_attempt", return_value=1),+ patch(f"{_PIPELINE}.current_workflow_id", return_value="wf-1"),+ patch(f"{_PIPELINE}.current_workflow_run_id", return_value="wfrun-abc"),+ patch(f"{_PIPELINE}.S3BatchWriter"),+ patch(f"{_PIPELINE}.PostgresProducer"),+ patch(f"{_PIPELINE}.DeltaTableRef"),+ patch(f"{_PIPELINE}.resolve_resume_manager", return_value=manager),+ pytest.raises(ValueError, match="cannot send a replace scope"),+ ):+ PipelineV3(+ source_response=resource,+ logger=_make_logger(),+ job_id="job-1",+ reset_pipeline=False,+ shutdown_monitor=MagicMock(),+ resumable_source_manager=manager,+ models=ImportJobModels(+ job=MagicMock(team_id=1, workflow_run_id="wfrun-abc", billable=False, id="job-1"),+ schema=ExternalDataSchema(name="users", team_id=1, sync_type="incremental"),+ source=MagicMock(source_type="Postgres"),+ table=None,+ ),+ )++ @pytest.mark.parametrize("has_scope", [False, True])+ def test_a_scoped_run_never_continues_from_a_handoff_checkpoint(self, has_scope: bool) -> None:+ pipeline = _make_pipeline()+ pipeline._run_uuid = "run-abc-a1"+ schema = cast(Any, pipeline._schema)+ schema.is_incremental = True+ schema.incremental_field = "_modified_at"+ scope = ReplaceScope(column="_file_name", replaced=frozenset(), retained=frozenset())+ source_response = MagicMock(+ sort_mode="asc", lanes=None, cdc_write_mode=None, replace_scope=scope if has_scope else None+ )++ assert pipeline._tracks_handoff_checkpoint(source_response, reset_pipeline=False) is not has_scope++ @pytest.mark.asyncio+ async def test_column_selection_keeps_the_replace_scope_column(self) -> None:+ pipeline = _make_pipeline()+ pipeline._uses_delta_write_column_selection = True+ schema = cast(Any, pipeline._schema)+ schema.enabled_columns = ["value"]+ schema.incremental_field = None+ schema.partitioning_keys_override = None+ schema.partitioning_keys = None+ pipeline._resource.primary_keys = None+ pipeline._resource.partition_keys = None+ pipeline._resource.replace_scope = ReplaceScope(column="_file_name", replaced=frozenset(), retained=frozenset())+ table = pa.table({"value": [1], "other": [2], "_file_name": ["a.csv"]})++ class _Stop(Exception):+ pass++ projected: list[pa.Table] = []++ async def project(*args: Any) -> pa.Table:+ projected.append(await _real_observe_and_project_table(*args))+ raise _Stop++ with patch(f"{_PIPELINE}.observe_and_project_table", side_effect=project), pytest.raises(_Stop):+ await pipeline._process_batch(pa_table=table, batch_index=0, row_count=1)++ [result] = projected+ assert "_file_name" in result.column_names+ assert "other" not in result.column_names++class TestExtractionFailureDoesNotCleanupS3:@pytest.mark.asyncioasync def test_s3_files_preserved_when_extraction_fails(self) -> None:@@ -372,6 +518,7 @@ class TestCDCSourceWiring:partition_mode=None,cdc_write_mode=cdc_write_mode,lanes=None,+ replace_scope=None,)base = "products.warehouse_sources.backend.temporal.data_imports.pipelines.pipeline_v3.pipeline"@@ -473,6 +620,7 @@ class TestResetLoadWritesTheQueuedScheme:partition_mode=source_mode,cdc_write_mode=None,lanes=None,+ replace_scope=None,webhook_only=webhook_only,)manager = MagicMock(can_resume=MagicMock(return_value=True)) if can_resume else None@@ -1238,6 +1386,7 @@ class TestDescWatermarkAcrossResumedAttempts:partition_mode="md5",cdc_write_mode=None,lanes=None,+ replace_scope=None,webhook_only=False,sort_mode=sort_mode,)@@ -2109,6 +2258,7 @@ class TestLoadedPrefixCursor:partition_mode=None,cdc_write_mode=cdc_write_mode,lanes=lanes,+ replace_scope=None,sort_mode=sort_mode,webhook_only=False,chunk_size=None,diff --git a/products/warehouse_sources/backend/temporal/data_imports/schema_flags.py b/products/warehouse_sources/backend/temporal/data_imports/schema_flags.pyindex 92441c8bafdf..f9e5726ca407 100644--- a/products/warehouse_sources/backend/temporal/data_imports/schema_flags.py+++ b/products/warehouse_sources/backend/temporal/data_imports/schema_flags.py@@ -19,6 +19,12 @@ if TYPE_CHECKING:from products.warehouse_sources.backend.models.external_data_schema import ExternalDataSchema+# A source sends a replace scope (see `ReplaceScope`) only while this flag is on for the schema. The+# flag stays off until the loaders that apply a scope are live, because an older loader ignores the+# scope and fails the keyless batch.+REPLACE_ROWS_BY_FILE_FLAG = "warehouse-replace-rows-by-file"++def is_schema_flag_enabled(schema: ExternalDataSchema, flag: str) -> bool:"""Evaluate a rollout flag for this schema.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/typings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/typings.pyindex 2bc717ee9a34..9dbcfeb38ccd 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/typings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/typings.py@@ -9,6 +9,8 @@ from typing import TYPE_CHECKING, Any, ClassVar, Literal, Optional, Protocol, Tyfrom structlog.types import FilteringBoundLogger+from posthog.dataclasses import frozen+from products.warehouse_sources.backend.types import ExternalDataSchemaSyncType, IncrementalFieldTypeif TYPE_CHECKING:@@ -72,6 +74,24 @@ class OutputLane:own writer tracks the schema it ends up with."""+@frozen+class ReplaceScope:+ """The destination rows that one incremental run replaces, named by the value of one column.++ The source names each row's origin (for example the file it came from) in `column`. Before the+ run's first batch loads, the loader deletes every row whose value is in `replaced` or is not in+ `retained`. The run's batches then load as appends. So a run replaces the rows of each origin it+ reads again, and removes the rows of each origin that is gone. A row with no value in `column`+ is outside the scope and stays.+ """++ column: str+ replaced: frozenset[str]+ """The origins this run reads again: changed or new. Their old rows go before the new rows load."""+ retained: frozenset[str]+ """Every origin that is live now. The rows of any other origin go."""++@dataclasses.dataclass(frozen=False) # callers mutate `primary_keys` after constructionclass SourceResponse:name: str@@ -134,6 +154,10 @@ class SourceResponse:lanes: Optional[list[OutputLane]] = None"""Tables this response's items feed, when it feeds more than the one `name` alone describes.None means the single lane built from `name` and `cdc_write_mode`."""+ replace_scope: Optional[ReplaceScope] = None+ """Set by an incremental source that replaces rows by origin instead of merging on a primary key+ (see `ReplaceScope`). The run loads as appends and needs no primary key. A first sync or a reset+ ignores it, because the table starts empty."""# Not frozen: nothing mutates it in place today, so freezing it is plausible, but every sourcediff --git a/products/warehouse_sources_queue/backend/core/jobs_db.py b/products/warehouse_sources_queue/backend/core/jobs_db.pyindex bb2faf6982d7..952116c4f620 100644--- a/products/warehouse_sources_queue/backend/core/jobs_db.py+++ b/products/warehouse_sources_queue/backend/core/jobs_db.py@@ -1217,6 +1217,7 @@ class PendingBatch:"cdc_table_mode": self.metadata.get("cdc_table_mode"),"destination_ids": self.destination_ids or [],"external_destination_ids": self.metadata.get("external_destination_ids"),+ "replace_scope": self.metadata.get("replace_scope"),}