:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): coarsen keyed md5 tables for merge parallelism (#121046)
Tom Owers134cdccdb862
diff --git a/posthog/settings/data_warehouse.py b/posthog/settings/data_warehouse.pyindex 17b2e499ebea..967f1b05b025 100644--- a/posthog/settings/data_warehouse.py+++ b/posthog/settings/data_warehouse.py@@ -103,6 +103,18 @@ DATA_WAREHOUSE_MERGE_TARGET_PARTITION_BYTES = get_from_env("DATA_WAREHOUSE_MERGE_TARGET_PARTITION_BYTES", 2_000_000_000, type_cast=int)+# Coarsening of md5 tables that upsert on a primary key. Such an upsert rewrites at least one file in+# every md5 partition, so the partition count, not the partition size, sets the merge cost. For these+# tables the coarsening path uses the merge budget above, and picks the divisor of the current count+# nearest to TARGET, never below MIN. MIN is the memory governor's cap on the partitions that one upsert+# merges in parallel, so a coarser table keeps all of its merge parallelism. TARGET is twice that cap,+# so the parallel partition tasks still share the work when one partition is slow.+# The automatic path uses this policy only when the `data-warehouse-merge-auto-coarsen` feature flag is on+# for the schema. Otherwise it coarsens these tables on the base budget as for any other table.+# A `stage_warehouse_coarsening` nomination always uses this policy.+DATA_WAREHOUSE_MERGE_MD5_PARTITION_TARGET = get_from_env("DATA_WAREHOUSE_MERGE_MD5_PARTITION_TARGET", 8, type_cast=int)+DATA_WAREHOUSE_MERGE_MD5_PARTITION_MIN = get_from_env("DATA_WAREHOUSE_MERGE_MD5_PARTITION_MIN", 4, type_cast=int)+# How often each sync activity self-reports its workload (phase, buffer bytes, RSS) to the warehouse# Redis, for post-mortem enrichment of silent worker deaths. Zero or negative disables reporting# entirely (the fleet kill switch — hooks become no-ops and no thread starts).diff --git a/products/warehouse_sources/backend/temporal/data_imports/README.md b/products/warehouse_sources/backend/temporal/data_imports/README.mdindex 1813c46c2307..564dd77382c5 100644--- a/products/warehouse_sources/backend/temporal/data_imports/README.md+++ b/products/warehouse_sources/backend/temporal/data_imports/README.md@@ -61,7 +61,12 @@ Tuning and gating:- **Repairing the existing backlog.** The automatic coarsening path will not reach a table that keeps recording OOM occurrences, and the tables most in need of coarsening are exactly the ones the unreliable OOM signal keeps firing on. `./manage.py stage_warehouse_coarsening` is the way in: it nominates tables by setting a `coarsen_requested` marker, and the next sync evaluates them. A nomination skips the policy gates (OOM history, layout age, minimum partition count) because an operator has made that call, but never the safety check: the controller still measures the live layout and refuses any target that would not fit the budget, so a nomination can only ever be a no-op. Nominated rewrites report `trigger_reason=coarsening_requested`. Dry run by default, `--limit` defaults low, and it is worth keeping low: a rewrite blocks that table's sync for as long as it runs, which is seconds for small tables and hours for the largest.- **Putting back a table that was split too fine.** A table split from just over the budget cannot pass the coarsening safety check, because its coarser partitions are larger than half the budget. `./manage.py stage_warehouse_repartition --schema-id <id> --format <tier>` covers that case for datetime tables. The operator names the coarser tier, and the command measures the live table and refuses a target whose largest partition would be over the full budget. It then stages the same `repartition_pending` target as the admin page (`trigger_reason=admin`), and the next scheduled sync runs the rewrite. A scheduled full refresh skips the repartition step, so the target then waits for the sync after it. `--schema-id` repeats, and it is a dry run by default.- The budget is tunable via the `DATA_WAREHOUSE_TARGET_PARTITION_BYTES` setting (default ~0.5 GB at-rest → ~10 GB worst-case merge). Worker pods are multi-tenant, so the budget leaves headroom for concurrent merges under the 29 GB pod limit rather than sizing to a single merge.-- Tables that upsert on a primary key (incremental, webhook and xmin sync types) use a larger size budget for the `proactive_threshold` trigger: `DATA_WAREHOUSE_MERGE_TARGET_PARTITION_BYTES` (default ~2 GB at-rest, never below the base budget). These upserts go through deltalite, whose peak memory does not grow with partition size. On an md5 table each upsert touches every partition and rewrites at least one file in each, so a split there only makes every later merge slower. The delta-rs MERGE runs on such a table only for a batch that deltalite fails, and the engine is chosen per batch, so the controller cannot know it in advance. When deltalite fails on a partition above the base budget, the writer refuses the delta-rs MERGE and raises a retryable error, so the larger cap cannot lead to an oversized MERGE (metric outcome `fallback_refused`). The OOM trigger, its floor and coarsening keep the base budget. Full-refresh tables keep the base budget too.+- Tables that upsert on a primary key (incremental, webhook and xmin sync types) use a larger size budget for the `proactive_threshold` trigger: `DATA_WAREHOUSE_MERGE_TARGET_PARTITION_BYTES` (default ~2 GB at-rest, never below the base budget). These upserts go through deltalite, whose peak memory does not grow with partition size. On an md5 table each upsert touches every partition and rewrites at least one file in each, so a split there only makes every later merge slower. The delta-rs MERGE runs on such a table only for a batch that deltalite fails, and the engine is chosen per batch, so the controller cannot know it in advance. When deltalite fails on a partition above the base budget, the writer refuses the delta-rs MERGE and raises a retryable error, so the larger cap cannot lead to an oversized MERGE (metric outcome `fallback_refused`). The OOM trigger and its floor keep the base budget. Full-refresh tables keep the base budget too.+- Coarsening of a keyed md5 table can use the merge budget instead. For these tables the partition count, not the partition size, sets the merge cost, because each upsert rewrites at least one file in every partition.+ - The coarse layout must keep its largest partition at or under 0.75 of the merge budget (`MERGE_COARSEN_TARGET_FRACTION`). The size split then leaves it alone until it grows by a third.+ - Of the divisors that fit, the one nearest `DATA_WAREHOUSE_MERGE_MD5_PARTITION_TARGET` (default 8) wins, never one below `DATA_WAREHOUSE_MERGE_MD5_PARTITION_MIN` (default 4). The floor is the memory governor's cap on partitions merged in parallel.+ - A `stage_warehouse_coarsening` nomination always uses this policy. The automatic path uses it only when the `data-warehouse-merge-auto-coarsen` feature flag is on for the schema (default off), and its trigger is then an eighth of the merge budget. When the flag is off, the automatic path coarsens these tables on the base budget.+ - Datetime and numerical keyed tables keep the base budget, because a batch usually falls in a few of their partitions.- The OOM-history override is tunable via `DATA_WAREHOUSE_REPARTITION_OOM_THRESHOLD` (default 3) and `DATA_WAREHOUSE_REPARTITION_OOM_WINDOW_DAYS` (default 7). An OOM-triggered rewrite of an under-budget table steps one tier finer per cooldown cycle, converging as the (still-recorded) OOMs continue.- The split floor and the coarsening trigger are both derived from the budget rather than tuned separately (`COARSEN_TRIGGER_DIVISOR`), so the two directions stay consistent by construction.- The coarsening gate's own tunables are `COARSEN_OOM_FREE_DAYS` (default 14, deliberately longer than the split window) and `DATA_WAREHOUSE_COARSEN_BLOCK_MERGE_PEAK_BYTES` (default 1 MB), the peak above which a merge-phase death blocks regardless of how the classification rules explained it. `DELTA_COARSEN_DECLINE_TOTAL` records which gate declined, labelled by reason.diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/repartition.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/repartition.pyindex e6fa217ebfff..53ed45004e4e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/repartition.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/repartition.py@@ -851,9 +851,17 @@ def select_coarsen_target(schema: ExternalDataSchema,partition_bytes: dict[str | None, int],target_partition_bytes: int,+ *,+ min_md5_count: int = 1,+ preferred_md5_count: int | None = None,) -> tuple[RepartitionTarget | None, str]:"""Pick the coarsest partition scheme whose largest partition still fits `target_partition_bytes`.+ For md5, `min_md5_count` and `preferred_md5_count` change "coarsest" to "the fitting divisor nearest+ to `preferred_md5_count`, never below `min_md5_count`". A keyed upsert rewrites at least one file in+ each md5 partition, but it also merges up to a fixed number of partitions in parallel, so the fewest+ partitions is not the fastest layout for that table.+The mirror of `select_repartition_target`, for tables left over-fragmented, most of them by thefiner path itself reacting to failures that were never about partition size. Fragmentation is notfree: every partition is its own merge commit, so a table split into thousands of tiny pieces syncs@@ -914,15 +922,25 @@ def select_coarsen_target(divisors.add(low)divisors.add(count // low)divisors.discard(count)- for new_count in sorted(divisors): # ascending, so the coarsest layout that fits wins- if acceptable(_simulate_modulo_coarsening(partition_bytes, new_count)):- return RepartitionTarget(- partition_keys=keys,- trigger_reason="",- partition_mode="md5",- partition_count=new_count,- ), "selected"- return None, "no_coarser_layout_fits"+ allowed = sorted(divisor for divisor in divisors if divisor >= min_md5_count)+ if not allowed:+ return None, "md5_at_partition_floor"+ fitting = [+ new_count for new_count in allowed if acceptable(_simulate_modulo_coarsening(partition_bytes, new_count))+ ]+ if not fitting:+ return None, "no_coarser_layout_fits"+ if preferred_md5_count is None:+ new_count = fitting[0] # ascending, so the coarsest layout that fits wins+ else:+ # On a tie, the larger count wins, because it keeps the partitions smaller.+ new_count = min(fitting, key=lambda divisor: (abs(divisor - preferred_md5_count), -divisor))+ return RepartitionTarget(+ partition_keys=keys,+ trigger_reason="",+ partition_mode="md5",+ partition_count=new_count,+ ), "selected"if mode == "numerical":current_size = schema.partition_sizediff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/repartition_controller.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/repartition_controller.pyindex 3376caba263c..904b54681379 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/repartition_controller.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/repartition_controller.py@@ -48,6 +48,10 @@ if TYPE_CHECKING:# the second is not a decision the first should make.WAREHOUSE_REPARTITION_HOLD_FLAG = "data-warehouse-repartition-hold"+# Lets the automatic coarsening path size keyed md5 tables by merge parallelism. Release it to one+# table with a `schema_id` condition before rolling out wider.+WAREHOUSE_MERGE_AUTO_COARSEN_FLAG = "data-warehouse-merge-auto-coarsen"+# Coarsening gates. The two directions deliberately don't meet: a table is split finer above the budget# and merged coarser only below an eighth of it, and a coarsen aims at half the budget. So a freshly# coarsened table has to double before the finer path can claim it, and a freshly split one has to@@ -118,6 +122,59 @@ def split_budget_bytes(schema: ExternalDataSchema) -> int:return target_partition_bytes()+# A coarsened keyed md5 table aims at this share of the merge budget, not at half of it as other+# tables do. Its partition size costs no merge memory, so the margin only has to keep normal growth from+# sending the table straight back to the size split. A table must grow by a third before that split+# claims it, and the split then waits out the cooldown that the coarsening rewrite starts.+MERGE_COARSEN_TARGET_FRACTION = 0.75+++def merge_auto_coarsen_enabled(schema: ExternalDataSchema) -> bool:+ return is_schema_flag_enabled(schema, WAREHOUSE_MERGE_AUTO_COARSEN_FLAG)+++def merge_md5_partition_min() -> int:+ return max(1, int(getattr(settings, "DATA_WAREHOUSE_MERGE_MD5_PARTITION_MIN", 4)))+++def merge_md5_partition_target() -> int:+ return max(merge_md5_partition_min(), int(getattr(settings, "DATA_WAREHOUSE_MERGE_MD5_PARTITION_TARGET", 8)))+++def coarsens_for_merge_parallelism(+ schema: ExternalDataSchema, *, operator_requested: bool, auto_coarsen: bool | None = None+) -> bool:+ """Whether coarsening sizes this table by the merge budget and the md5 partition target.++ Only a keyed md5 table qualifies. Its upserts hash every batch over all partitions, so each+ partition adds at least one rewritten file to every merge, and its partition size costs no merge+ memory (see `split_budget_bytes`). A batch of a datetime or numerical table usually falls in a few+ partitions, so the partition count does not set its merge cost, and the base budget stays.++ The automatic path uses this policy only when the auto coarsen flag is on for the schema. An+ operator nomination always uses it. A caller that already has the flag result passes it as+ `auto_coarsen`, so the flag is evaluated at most once per run. The evaluation is synchronous+ and reads the database, so async callers run this in a thread.+ """+ if schema.sync_type not in KEYED_MERGE_SYNC_TYPES or schema.partition_mode != "md5":+ return False+ if operator_requested:+ return True+ # The automatic path never coarsens a table this fine, so the flag result could not change anything.+ if (schema.partition_count or 0) < COARSEN_MIN_PARTITIONS:+ return False+ return merge_auto_coarsen_enabled(schema) if auto_coarsen is None else auto_coarsen+++def coarsen_trigger_budget(+ schema: ExternalDataSchema, *, operator_requested: bool, auto_coarsen: bool | None = None+) -> int:+ """The budget that the coarsening trigger compares the largest partition with."""+ if coarsens_for_merge_parallelism(schema, operator_requested=operator_requested, auto_coarsen=auto_coarsen):+ return merge_target_partition_bytes()+ return target_partition_bytes()++def min_splittable_partition_bytes() -> int:"""The smallest partition an OOM-triggered split is allowed to produce.@@ -209,6 +266,11 @@ def pre_extraction_measurement_is_redundant(schema: ExternalDataSchema, job: Extreturn Falseif measurement.get("split_budget") != split_budget_bytes(schema):return False+ # A measurement without this key is older than the key, and was taken against the base budget.+ if measurement.get("coarsen_budget", measurement.get("budget")) != coarsen_trigger_budget(+ schema, operator_requested=False+ ):+ return Falseif (not schema.is_appendand ExternalDataSchemaOOMEvent.recent_count(schema, days=repartition_oom_window_days())@@ -343,8 +405,8 @@ async def maybe_flag_for_coarsening(recent_oom_count: int,logger: FilteringBoundLogger,*,- budget: int,max_bytes: int,+ auto_coarsen: bool | None = None,) -> None:"""Flag an over-fragmented table to be rebuilt into fewer, larger partitions.@@ -359,6 +421,9 @@ async def maybe_flag_for_coarsening(check is `select_coarsen_target`, which measures the live layout and refuses any target that wouldnot fit the budget. Nothing overrides that, so a nomination can only ever be a no-op.+ A keyed md5 table can use the merge budget and the md5 partition target instead of the base+ budget (see `coarsens_for_merge_parallelism`). The safety check then uses that budget too.+Ordered cheapest first — in-memory shape gates, then the in-memory selector, then the database.Post-load detection runs thisfor every within-budget table on every sync, so anything before the selector is fleet-wide cost,@@ -381,6 +446,10 @@ async def maybe_flag_for_coarsening(returnrequested = schema.coarsen_requested+ merge_policy = await asyncio.to_thread(+ coarsens_for_merge_parallelism, schema, operator_requested=requested is not None, auto_coarsen=auto_coarsen+ )+ budget = coarsen_trigger_budget(schema, operator_requested=requested is not None, auto_coarsen=merge_policy)if requested is None:# Below these two the table is not over-fragmented at all, so returning before `_decline`# keeps the metric scoped to the population the rollout is about.@@ -397,9 +466,19 @@ async def maybe_flag_for_coarsening(if layout_age is not None and layout_age < COARSEN_MIN_LAYOUT_AGE_SECONDS:return _decline("layout_too_young")- target, reason = await asyncio.to_thread(- select_coarsen_target, schema, partition_bytes, budget // COARSEN_TARGET_DIVISOR- )+ if merge_policy:+ target, reason = await asyncio.to_thread(+ select_coarsen_target,+ schema,+ partition_bytes,+ int(budget * MERGE_COARSEN_TARGET_FRACTION),+ min_md5_count=merge_md5_partition_min(),+ preferred_md5_count=merge_md5_partition_target(),+ )+ else:+ target, reason = await asyncio.to_thread(+ select_coarsen_target, schema, partition_bytes, budget // COARSEN_TARGET_DIVISOR+ )if requested is not None and target is None:# Consume a refused nomination now: leaving it set would re-evaluate the same table every sync,# and a table the selector refuses today will refuse again until its data changes. A *selected*@@ -533,8 +612,11 @@ async def maybe_flag_for_repartition(# A table with this shape is one that the coarsening path evaluates on each run, and its# gates open with time (layout age, days without an OOM). Such a table is never healthy, so# the next run measures it again.+ nominated = schema.coarsen_requested is not None+ auto_coarsen = await asyncio.to_thread(coarsens_for_merge_parallelism, schema, operator_requested=nominated)+ coarsen_budget = coarsen_trigger_budget(schema, operator_requested=nominated, auto_coarsen=auto_coarsen)coarsen_candidate = (- len(partition_bytes) >= COARSEN_MIN_PARTITIONS and max_bytes * COARSEN_TRIGGER_DIVISOR <= budget+ len(partition_bytes) >= COARSEN_MIN_PARTITIONS and max_bytes * COARSEN_TRIGGER_DIVISOR <= coarsen_budget)await asyncio.to_thread(schema.record_partition_measurement,@@ -544,6 +626,7 @@ async def maybe_flag_for_repartition("phase": phase,"budget": budget,"split_budget": size_split_budget,+ "coarsen_budget": coarsen_budget,"healthy": not over_budget and not oom_triggered and not coarsen_candidate,},)@@ -561,7 +644,7 @@ async def maybe_flag_for_repartition(partition_count=len(partition_bytes),)await maybe_flag_for_coarsening(- schema, source, job, partition_bytes, oom_count, logger, budget=budget, max_bytes=max_bytes+ schema, source, job, partition_bytes, oom_count, logger, max_bytes=max_bytes, auto_coarsen=auto_coarsen)return@@ -591,7 +674,7 @@ async def maybe_flag_for_repartition(# its chance, and this is the only route by which a nominated table reaches coarsening at# all once its OOM count crosses the split threshold.await maybe_flag_for_coarsening(- schema, source, job, partition_bytes, oom_count, logger, budget=budget, max_bytes=max_bytes+ schema, source, job, partition_bytes, oom_count, logger, max_bytes=max_bytes)returndiff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/test_repartition.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/test_repartition.pyindex 9e86d63b971d..c8704b76e4d9 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/test_repartition.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/test_repartition.py@@ -516,6 +516,29 @@ class TestSelectCoarsenTarget:for key, value in expect.items():assert getattr(target, key) == value+ @parameterized.expand(+ [+ # (name, count, target_bytes, min_count, preferred_count, expected_count)+ ("nearest_divisor_below_the_target", 85, 10_000, 4, 8, 5),+ ("nearest_divisor_above_the_target", 100, 10_000, 4, 8, 10),+ ("a_tie_keeps_the_finer_layout", 60, 10_000, 4, 8, 10),+ ("a_divisor_under_the_floor_never_wins", 96, 10_000, 4, 1, 4),+ ("the_size_cap_overrides_the_preference", 64, 400, 4, 4, 8),+ ("no_preference_keeps_the_coarsest_fit", 64, 10_000, 1, None, 1),+ ]+ )+ def test_md5_count_bounds(self, _name, count, target_bytes, min_count, preferred_count, expected_count):+ target, reason = select_coarsen_target(+ _schema(partition_mode="md5", partition_count=count, partitioning_keys=["id"]),+ {str(bucket): 50 for bucket in range(count)},+ target_bytes,+ min_md5_count=min_count,+ preferred_md5_count=preferred_count,+ )+ assert reason == "selected"+ assert target is not None+ assert target.partition_count == expected_count+@parameterized.expand([("hour", "day"),diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/test_repartition_controller.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/test_repartition_controller.pyindex d7f989fcd0f3..ad1f40a1e425 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/test_repartition_controller.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/core/test_repartition_controller.py@@ -4,6 +4,7 @@ import asyncioimport datetimeimport tempfileimport contextlib+from typing import Anyimport pytestfrom unittest.mock import AsyncMock, MagicMock, patch@@ -412,6 +413,7 @@ class TestPartitionMeasurementRecord:"phase": phase,"budget": budget,"split_budget": budget,+ "coarsen_budget": budget,"healthy": healthy,}@@ -450,16 +452,59 @@ class TestPartitionMeasurementRecord:assert measurement is not None and measurement["healthy"] is False@pytest.mark.parametrize(- "previous_status, nominated, sync_type, measured_budget, measured_split_budget, recent_ooms, redundant",+ "previous_status, nominated, sync_type, measured_budget, measured_split_budget, recent_ooms, auto_coarsen, "+ "measured_coarsen_budget, redundant",[- (ExternalDataJob.Status.COMPLETED, False, None, 500, 500, 0, True),- (ExternalDataJob.Status.FAILED, False, None, 500, 500, 0, False),- (ExternalDataJob.Status.COMPLETED, True, None, 500, 500, 0, False),- (ExternalDataJob.Status.COMPLETED, False, None, 400, 500, 0, False),- (ExternalDataJob.Status.COMPLETED, False, None, 500, 500, 3, False),- (ExternalDataJob.Status.COMPLETED, False, ExternalDataSchema.SyncType.INCREMENTAL, 500, 2000, 0, True),- (ExternalDataJob.Status.COMPLETED, False, ExternalDataSchema.SyncType.INCREMENTAL, 500, 3000, 0, False),- (ExternalDataJob.Status.COMPLETED, False, None, 500, None, 0, False),+ (ExternalDataJob.Status.COMPLETED, False, None, 500, 500, 0, False, None, True),+ (ExternalDataJob.Status.FAILED, False, None, 500, 500, 0, False, None, False),+ (ExternalDataJob.Status.COMPLETED, True, None, 500, 500, 0, False, None, False),+ (ExternalDataJob.Status.COMPLETED, False, None, 400, 500, 0, False, None, False),+ (ExternalDataJob.Status.COMPLETED, False, None, 500, 500, 3, False, None, False),+ (+ ExternalDataJob.Status.COMPLETED,+ False,+ ExternalDataSchema.SyncType.INCREMENTAL,+ 500,+ 2000,+ 0,+ False,+ None,+ True,+ ),+ (+ ExternalDataJob.Status.COMPLETED,+ False,+ ExternalDataSchema.SyncType.INCREMENTAL,+ 500,+ 3000,+ 0,+ False,+ None,+ False,+ ),+ (ExternalDataJob.Status.COMPLETED, False, None, 500, None, 0, False, None, False),+ (+ ExternalDataJob.Status.COMPLETED,+ False,+ ExternalDataSchema.SyncType.INCREMENTAL,+ 500,+ 2000,+ 0,+ True,+ None,+ False,+ ),+ (+ ExternalDataJob.Status.COMPLETED,+ False,+ ExternalDataSchema.SyncType.INCREMENTAL,+ 500,+ 2000,+ 0,+ True,+ 2000,+ True,+ ),],ids=["previous_run_completed",@@ -470,6 +515,8 @@ class TestPartitionMeasurementRecord:"merge_table_measured_against_its_split_budget","split_budget_changed_since_the_measurement","measurement_without_a_split_budget",+ "auto_coarsen_flag_turned_on_since_the_measurement",+ "merge_table_measured_against_its_coarsen_budget",],)def test_gate_reads_what_can_change_without_a_write(@@ -481,6 +528,8 @@ class TestPartitionMeasurementRecord:measured_budget: int,measured_split_budget: int | None,recent_ooms: int,+ auto_coarsen: bool,+ measured_coarsen_budget: int | None,redundant: bool,):schema = _make_schema(team, {})@@ -506,15 +555,20 @@ class TestPartitionMeasurementRecord:"phase": "post_load","budget": measured_budget,"split_budget": measured_split_budget,+ **({"coarsen_budget": measured_coarsen_budget} if measured_coarsen_budget else {}),"healthy": True,},+ "partition_mode": "md5",+ "partition_count": ctrl.COARSEN_MIN_PARTITIONS,**({"coarsen_requested": _NOMINATION} if nominated else {}),}with (override_settings(- DATA_WAREHOUSE_TARGET_PARTITION_BYTES=500, DATA_WAREHOUSE_MERGE_TARGET_PARTITION_BYTES=2000+ DATA_WAREHOUSE_TARGET_PARTITION_BYTES=500,+ DATA_WAREHOUSE_MERGE_TARGET_PARTITION_BYTES=2000,),+ patch.object(ctrl, "is_schema_flag_enabled", return_value=auto_coarsen),patch.object(ctrl.ExternalDataSchemaOOMEvent, "recent_count", return_value=recent_ooms),):assert ctrl.pre_extraction_measurement_is_redundant(schema, current) is redundant@@ -852,6 +906,223 @@ class TestCoarsenTrigger:assert schema.coarsen_requested is None+_KEYED = ExternalDataSchema.SyncType.INCREMENTAL+_FULL_REFRESH = ExternalDataSchema.SyncType.FULL_REFRESH+++class TestMergeCoarsenPolicy:+ # Budgets in bytes, small enough to state partition sizes directly: the merge policy caps a coarse+ # partition at 1500 (0.75 of the merge budget), and its automatic trigger fires at 250 or less.+ _BUDGETS = {"DATA_WAREHOUSE_TARGET_PARTITION_BYTES": 500, "DATA_WAREHOUSE_MERGE_TARGET_PARTITION_BYTES": 2000}++ def _schema(+ self, team, sync_type: str, *, partition_count: int, nominated: bool, **overrides+ ) -> ExternalDataSchema:+ schema = _make_schema(+ team,+ {+ "partitioning_enabled": True,+ "partition_mode": "md5",+ "partition_count": partition_count,+ "partitioning_keys": ["id"],+ "last_repartition_at": _days_ago_iso(30),+ **({"coarsen_requested": _NOMINATION} if nominated else {}),+ **overrides,+ },+ )+ schema.sync_type = sync_type+ schema.save()+ return schema++ def _detect(+ self,+ team,+ schema: ExternalDataSchema,+ partition_bytes: dict[str | None, int],+ *,+ flag_enabled: bool = False,+ **settings,+ ) -> MagicMock:+ with (+ override_settings(**{**self._BUDGETS, **settings}),+ patch.object(ctrl, "is_schema_flag_enabled", return_value=flag_enabled) as flag_check,+ patch.object(ctrl, "measure_partition_bytes", return_value=partition_bytes),+ patch.object(ctrl, "capture_repartition_event"),+ ):+ async_to_sync(ctrl.maybe_flag_for_repartition)(+ schema, schema.source, _make_job(team, schema), MagicMock(), logger+ )+ schema.refresh_from_db()+ return flag_check++ @staticmethod+ def _md5_layout(count: int, each: int) -> dict[str | None, int]:+ return {str(bucket): each for bucket in range(count)}++ @pytest.mark.parametrize(+ "sync_type, count, each, settings, expected_count, expected_reason",+ [+ (_KEYED, 85, 60, {}, 5, None),+ (ExternalDataSchema.SyncType.WEBHOOK, 85, 60, {}, 5, None),+ (ExternalDataSchema.SyncType.XMIN, 85, 60, {}, 5, None),+ (_KEYED, 64, 20, {}, 8, None),+ (_KEYED, 85, 100, {}, 17, None),+ (_KEYED, 85, 480, {}, None, "no_coarser_layout_fits"),+ (_KEYED, 16, 10, {}, 4, None),+ (_KEYED, 6, 10, {}, None, "md5_at_partition_floor"),+ (_KEYED, 64, 20, {"DATA_WAREHOUSE_MERGE_MD5_PARTITION_MIN": 16}, 16, None),+ (_FULL_REFRESH, 85, 60, {}, None, "no_coarser_layout_fits"),+ (_FULL_REFRESH, 16, 10, {}, 1, None),+ ],+ ids=[+ "incremental_picks_the_divisor_nearest_the_target",+ "webhook_uses_the_merge_policy",+ "xmin_uses_the_merge_policy",+ "an_exact_divisor_hits_the_target",+ "a_divisor_over_the_headroom_is_skipped_for_a_finer_one",+ "refuses_when_no_divisor_fits_under_the_merge_budget",+ "stops_at_the_floor_where_the_base_policy_goes_to_one",+ "a_table_at_the_floor_has_no_coarser_layout",+ "the_floor_setting_wins_over_the_target",+ "full_refresh_keeps_the_base_budget",+ "full_refresh_still_goes_to_the_coarsest_fit",+ ],+ )+ def test_a_nomination_sizes_keyed_md5_tables_by_merge_parallelism(+ self,+ team,+ sync_type: str,+ count: int,+ each: int,+ settings: dict[str, Any],+ expected_count: int | None,+ expected_reason: str | None,+ ):+ schema = self._schema(+ team, sync_type, partition_count=count, nominated=True, last_repartition_at=_days_ago_iso(0)+ )++ with patch.object(ctrl, "DELTA_COARSEN_DECLINE_TOTAL") as decline_metric:+ self._detect(team, schema, self._md5_layout(count, each), **settings)++ assert schema.coarsen_requested is None+ pending = schema.repartition_pending+ if expected_count is None:+ assert pending is None+ assert decline_metric.labels.call_args.kwargs == {"reason": expected_reason}+ else:+ assert pending is not None+ assert pending["trigger_reason"] == "coarsening_requested"+ assert pending["partition_mode"] == "md5"+ assert pending["partition_count"] == expected_count++ @pytest.mark.parametrize(+ "sync_type, flag_state, each, expected_count",+ [+ (_KEYED, "off", 100, None),+ (_KEYED, "on", 100, 17),+ (_KEYED, "errors", 100, None),+ (_KEYED, "off", 20, 17),+ (_KEYED, "errors", 20, 17),+ (_KEYED, "on", 20, 5),+ (_FULL_REFRESH, "on", 100, None),+ ],+ ids=[+ "off_leaves_a_table_only_the_merge_budget_can_coarsen",+ "on_coarsens_against_the_merge_budget",+ "an_erroring_flag_counts_as_off",+ "off_keeps_the_base_policy",+ "an_erroring_flag_keeps_the_base_policy",+ "on_stops_near_the_target",+ "on_does_not_reach_full_refresh_tables",+ ],+ )+ def test_automatic_merge_coarsening_is_behind_a_flag(+ self, team, sync_type: str, flag_state: str, each: int, expected_count: int | None+ ):+ schema = self._schema(team, sync_type, partition_count=85, nominated=False)+ layout = self._md5_layout(85, each)++ if flag_state == "errors":+ with (+ patch.object(schema_flags.posthoganalytics, "feature_enabled", side_effect=RuntimeError("flags down")),+ patch.object(schema_flags, "capture_exception"),+ override_settings(**self._BUDGETS),+ patch.object(ctrl, "measure_partition_bytes", return_value=layout),+ patch.object(ctrl, "capture_repartition_event"),+ ):+ async_to_sync(ctrl.maybe_flag_for_repartition)(+ schema, schema.source, _make_job(team, schema), MagicMock(), logger+ )+ schema.refresh_from_db()+ else:+ self._detect(team, schema, layout, flag_enabled=flag_state == "on")++ pending = schema.repartition_pending+ if expected_count is None:+ assert pending is None+ else:+ assert pending is not None+ assert pending["trigger_reason"] == "coarsening"+ assert pending["partition_count"] == expected_count++ @pytest.mark.parametrize(+ "sync_type, nominated, count, expected_calls",+ [+ (_KEYED, False, 85, 1),+ (_KEYED, True, 85, 0),+ (_FULL_REFRESH, False, 85, 0),+ (_KEYED, False, 8, 0),+ ],+ ids=[+ "a_keyed_md5_table_evaluates_the_flag_once",+ "a_nomination_skips_the_flag",+ "a_table_the_policy_cannot_reach_skips_the_flag",+ "a_table_too_fine_to_coarsen_skips_the_flag",+ ],+ )+ def test_the_flag_is_evaluated_only_when_it_can_change_the_policy(+ self, team, sync_type: str, nominated: bool, count: int, expected_calls: int+ ):+ schema = self._schema(team, sync_type, partition_count=count, nominated=nominated)++ flag_check = self._detect(team, schema, self._md5_layout(count, 20))++ assert flag_check.call_count == expected_calls+ if expected_calls:+ assert flag_check.call_args.args == (schema, ctrl.WAREHOUSE_MERGE_AUTO_COARSEN_FLAG)++ @pytest.mark.parametrize("flag_enabled", [False, True], ids=["flag_off", "flag_on"])+ @pytest.mark.parametrize("layout_age_days", [0, 2], ids=["just_rewritten", "past_the_cooldown"])+ def test_a_coarsened_table_is_not_split_or_coarsened_again(self, team, flag_enabled: bool, layout_age_days: int):+ schema = self._schema(team, _KEYED, partition_count=64, nominated=True)+ self._detect(team, schema, self._md5_layout(64, 187))+ pending = schema.repartition_pending+ assert pending is not None+ coarse_count = pending["partition_count"]+ assert coarse_count == 8++ # The state a finished swap leaves behind.+ schema.sync_type_config = {+ **schema.sync_type_config,+ "partition_count": coarse_count,+ "last_repartition_at": _days_ago_iso(layout_age_days),+ }+ schema.sync_type_config.pop("repartition_pending")+ schema.save()++ self._detect(+ team,+ schema,+ self._md5_layout(coarse_count, 64 * 187 // coarse_count),+ flag_enabled=flag_enabled,+ )++ assert schema.repartition_pending is None+ measurement = schema.partition_measurement+ assert measurement is not None and measurement["healthy"] is True++# An Exception-derived cancellation, named exactly `CancelledError`: models how `async_to_sync` can# surface a worker-shutdown cancel so it slips past a plain BaseException catch. `_is_cancellation`# keys on the type name, so this must be named `CancelledError`.