:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(signals): refuse the inbox ranking engagement export when the local share collapses (#123455)
posthog[bot]523db1bab2e7
diff --git a/products/signals/dags/inbox_ranking/AGENTS.md b/products/signals/dags/inbox_ranking/AGENTS.mdindex e4d99ee978ba..e88cedefc602 100644--- a/products/signals/dags/inbox_ranking/AGENTS.md+++ b/products/signals/dags/inbox_ranking/AGENTS.md@@ -8,7 +8,7 @@ Read `README.md` first for what the dataset is and how partitions behave. This f- `inbox_report_title_embeddings` is a **leaf**: never add it to `inbox_report_model_data`'s `deps` or give that join a title column, or a failure in the title snapshot fails the training table. Its own `deps=[MODEL_DATA_TABLE]` exists for memory, not data — two full sets of 1536-float vectors must not be held at once in the run pod. Raising the pod memory limit is not the first resort.- `inbox_user_report_interactions` is a **leaf** with its own `USER_INTERACTIONS_SCHEMA_VERSION`: keep it out of `inbox_report_model_data`'s `deps`, and never stamp it with `FEATURE_SCHEMA_VERSION`, or a user-grain change sets off the labels refresh sensor. It writes a row only for a report in this region's Postgres whose team is in `training_consent_team_ids()`.- `inbox_project_vectors` is a **leaf** with its own `PROJECT_VECTORS_SCHEMA_VERSION`: keep it out of `inbox_report_model_data`'s `deps`. Its `deps=[EMBEDDINGS_TABLE, LABELS_TABLE]` keep it after the embeddings snapshot, so the two vector-heavy steps do not share the run pod's memory. `project_vectors` takes a report into a vector only when its vector landed and its UI engagement happened strictly before `as_of`; never widen that to the latest snapshot value. A report born on day D reads the dt=D−1 vectors (see the README).-- `inbox_engagement_export` is a **leaf** with its own `ENGAGEMENT_EXPORT_SCHEMA_VERSION`, read by another region. It writes to the mirror store only under `serving_engagement_key`, because the mirror's write grant covers only `<prefix>/serving/`. Keep its columns exactly `LABEL_COLUMNS` and its rows limited to reports with no `SignalReport` in this region: no team ids, counts, text or vectors leave the region. A mirror failure logs and must not fail the asset.+- `inbox_engagement_export` is a **leaf** with its own `ENGAGEMENT_EXPORT_SCHEMA_VERSION`, read by another region. It writes to the mirror store only under `serving_engagement_key`, because the mirror's write grant covers only `<prefix>/serving/`. Keep its columns exactly `LABEL_COLUMNS` and its rows limited to reports with no `SignalReport` in this region: no team ids, counts, text or vectors leave the region. A mirror failure logs and must not fail the asset. Keep the local-share guard before the write: when the local report lookup finds no local report (`no_local_reports`) or a share below `ENGAGEMENT_EXPORT_MIN_LOCAL_SHARE` (`local_share_collapsed`), the asset writes nothing and still succeeds. Its `inbox_ranking_engagement_export_published` event carries counts only, never a report id or a team id.- `inbox_mirror_project_vectors` (`mirror/dag.py`) is the only inbox-ranking asset that registers on EU (`is_inbox_ranking_mirror_registered()`). It reads this region's report vectors through `report_embeddings_table` and the engagement export through `serving_engagement_key`, and writes only to this region's object store at `serving_project_vectors_key`. Keep it on the same `project_vectors()` and the same consent rule as `inbox_project_vectors`, so the two regions build their vectors the same way. A missing, too old or unknown-schema export fails the asset rather than writing vectors without engagement.- Register new jobs/schedules in `posthog/dags/locations/signals.py` (or `signals_ml.py` for a torch-dependent dag), keep the EU gate (`is_inbox_ranking_registered()`), and remember the ci-dagster paths filter must cover every module a dag file imports (`uv run .github/scripts/check-dagster-paths.py`).diff --git a/products/signals/dags/inbox_ranking/README.md b/products/signals/dags/inbox_ranking/README.mdindex 2f9a41359ddb..f346cb738489 100644--- a/products/signals/dags/inbox_ranking/README.md+++ b/products/signals/dags/inbox_ranking/README.md@@ -89,7 +89,9 @@ The inbox telemetry of every region lands in the US dogfood project, so the `ope- The columns are exactly `LABEL_COLUMNS` from `dataset/project_vectors.py`: `report_id` and the first opened, first UI dismissed and first UI action timestamps. There are no team ids, no counts, no text and no vectors, so the other region passes the file to `project_vectors()` unchanged.- The labels snapshot is cumulative, so each day is a full snapshot as of the end of dt=D.- The key is `serving_engagement_key(prefix, dt)`: `<prefix>/serving/engagement/<dt>/engagement.parquet`. The mirror's write grant covers only `<prefix>/serving/`. The write sets `ACL: bucket-owner-full-control` and stamps the row count, the snapshot date and `ENGAGEMENT_EXPORT_SCHEMA_VERSION` in the object metadata.-- **It is a leaf.** Nothing joins it. A mirror write failure does not fail the asset: it logs, and the output metadata carries `written=false`. The metadata also carries `rows`, `excluded_rows`, `bytes` and `s3_key`.+- **It is a leaf.** Nothing joins it. A mirror write failure does not fail the asset: it logs, and the output metadata carries `written=false`. The metadata also carries `rows`, `excluded_rows` (the rows whose report is in this region), `local_share` (`excluded_rows` over all labels rows), `bytes` and `s3_key`.+- **A collapsed local share refuses the write.** When the local report lookup returns too few rows, every report it misses looks remote, and this region's report ids would go to the other region. So when the labels table has rows and the lookup finds no local report, the skip reason is `no_local_reports`. When `local_share` is below `ENGAGEMENT_EXPORT_MIN_LOCAL_SHARE`, the skip reason is `local_share_collapsed`. A refusal writes nothing, logs at error level, and sets `written=false` and `skip_reason` in the metadata. The asset does not fail: the mirror region uses an older export for up to `ENGAGEMENT_EXPORT_MAX_AGE_DAYS`, then fails on its side. An empty labels table is not refused.+- Each run with the mirror bucket set captures one `inbox_ranking_engagement_export_published` event in the dogfood project through `training/telemetry.py`, also when the write fails or is refused. It carries `partition`, `written`, `skip_reason`, `rows`, `excluded_rows`, `local_share`, `bytes` and `schema_version`, and no report id or team id.- A report deleted in this region keeps its Postgres row with the `deleted` status, so it is not exported. Only a report whose row is gone, for example after its team was deleted, looks remote. Such a row exposes a report id and timestamps, and it never matches a report vector in the other region.- The grant has no delete, and no lifecycle rule covers the prefix, so the daily objects accumulate. Each one holds a report id and a few timestamps per row.diff --git a/products/signals/dags/inbox_ranking/dataset/dag.py b/products/signals/dags/inbox_ranking/dataset/dag.pyindex 55327457b202..da94186989b0 100644--- a/products/signals/dags/inbox_ranking/dataset/dag.py+++ b/products/signals/dags/inbox_ranking/dataset/dag.py@@ -186,6 +186,11 @@ REVIEWERS_SCHEMA_VERSION = 1# The engagement export is read in another region, so its shape moves apart from every local snapshot.ENGAGEMENT_EXPORT_SCHEMA_VERSION = 1+# Most labels rows belong to this region's reports. A share below this floor means the local report+# lookup lost rows, and the export would send this region's report ids to the other region.+ENGAGEMENT_EXPORT_MIN_LOCAL_SHARE = 0.25+NO_LOCAL_REPORTS = "no_local_reports"+LOCAL_SHARE_COLLAPSED = "local_share_collapsed"COMMON_ASSET_KWARGS: dict[str, Any] = {"group_name": "inbox_ranking",@@ -1488,6 +1493,17 @@ def remote_engagement_rows(labels: pa.Table, local_report_ids: Collection[str])return labels.filter(pa.array(keep, type=pa.bool_()))+def engagement_export_refusal(label_rows: int, local_rows: int) -> str | None:+ """Why the export must not be written, or None. An empty labels table has nothing to leak."""+ if label_rows == 0:+ return None+ if local_rows == 0:+ return NO_LOCAL_REPORTS+ if local_rows / label_rows < ENGAGEMENT_EXPORT_MIN_LOCAL_SHARE:+ return LOCAL_SHARE_COLLAPSED+ return None++@dagster.asset(name=ENGAGEMENT_EXPORT,# Nothing joins this asset, so its failure does not fail the training table.@@ -1515,45 +1531,77 @@ def inbox_engagement_export(context: dagster.AssetExecutionContext) -> None:labels = read_parquet(s3_client(), dataset_bucket(), partition_object_key(prefix, LABELS_TABLE, partition_key), list(LABEL_COLUMNS))- local_report_ids = _report_teams(- sorted(valid_report_uuids(set(labels.column("report_id").to_pylist()))), snapshot_end- ).keys()- table = remote_engagement_rows(labels, local_report_ids)- buffer = io.BytesIO()- pq.write_table(table, buffer, compression="zstd")- content = buffer.getvalue()+ # telemetry imports the training modules, which import this module.+ from products.signals.dags.inbox_ranking.training.telemetry import ( # noqa: PLC0415+ capture_training_events,+ engagement_export_event,+ )+ report_ids = labels.column("report_id").to_pylist()+ local_report_ids = _report_teams(sorted(valid_report_uuids(set(report_ids))), snapshot_end).keys()+ local_rows = sum(report_id in local_report_ids for report_id in report_ids)+ local_share = local_rows / labels.num_rows if labels.num_rows else None+ table = remote_engagement_rows(labels, local_report_ids)key = serving_engagement_key(prefix, partition_key)- written = True- try:- serving_mirror_storage().write(- bucket=mirror_bucket,- key=key,- content=content,- extras={- # The mirror bucket is in another account, and its policy refuses a write without this ACL.- "ACL": "bucket-owner-full-control",- "Metadata": {- ROW_COUNT_METADATA_KEY: str(table.num_rows),- SNAPSHOT_DATE_METADATA_KEY: partition_key,- SCHEMA_VERSION_METADATA_KEY: str(ENGAGEMENT_EXPORT_SCHEMA_VERSION),- },- },+ skip_reason = engagement_export_refusal(labels.num_rows, local_rows)+ content = b""+ written = False+ if skip_reason is not None:+ # Nothing is written, so the other region falls back to an older export, then fails on its side.+ context.log.error(+ f"engagement export for {partition_key} refused ({skip_reason}): {local_rows} of "+ f"{labels.num_rows} labels rows have a report in this region")- except Exception as error:- # The mirror is another deployment's store, so an outage there only logs. The other region- # detects the missing day on its side.- context.log.exception(f"engagement export for {partition_key} not written: {error!r}")- written = False+ else:+ buffer = io.BytesIO()+ pq.write_table(table, buffer, compression="zstd")+ content = buffer.getvalue()+ try:+ serving_mirror_storage().write(+ bucket=mirror_bucket,+ key=key,+ content=content,+ extras={+ # The mirror bucket is in another account, and its policy refuses a write without this ACL.+ "ACL": "bucket-owner-full-control",+ "Metadata": {+ ROW_COUNT_METADATA_KEY: str(table.num_rows),+ SNAPSHOT_DATE_METADATA_KEY: partition_key,+ SCHEMA_VERSION_METADATA_KEY: str(ENGAGEMENT_EXPORT_SCHEMA_VERSION),+ },+ },+ )+ written = True+ except Exception as error:+ # The mirror is another deployment's store, so an outage there only logs. The other region+ # detects the missing day on its side.+ context.log.exception(f"engagement export for {partition_key} not written: {error!r}")context.add_output_metadata({"written": dagster.MetadataValue.bool(written),+ **({"skip_reason": dagster.MetadataValue.text(skip_reason)} if skip_reason else {}),"rows": dagster.MetadataValue.int(table.num_rows),- "excluded_rows": dagster.MetadataValue.int(labels.num_rows - table.num_rows),+ "excluded_rows": dagster.MetadataValue.int(local_rows),+ "local_share": dagster.MetadataValue.float(local_share or 0.0),"bytes": dagster.MetadataValue.int(len(content)),"s3_key": dagster.MetadataValue.text(f"s3://{mirror_bucket}/{key}"),})+ capture_training_events(+ context,+ partition_key,+ [+ engagement_export_event(+ written=written,+ skip_reason=skip_reason,+ rows=table.num_rows,+ excluded_rows=local_rows,+ local_share=local_share,+ bytes_written=len(content),+ schema_version=ENGAGEMENT_EXPORT_SCHEMA_VERSION,+ )+ ],+ )def label_provenance_ok(diff --git a/products/signals/dags/inbox_ranking/tests/test_project_vectors.py b/products/signals/dags/inbox_ranking/tests/test_project_vectors.pyindex 6f776cee9ca1..12d706f75115 100644--- a/products/signals/dags/inbox_ranking/tests/test_project_vectors.py+++ b/products/signals/dags/inbox_ranking/tests/test_project_vectors.py@@ -34,7 +34,10 @@ from products.signals.dags.inbox_ranking.dataset.project_vectors import ()from products.signals.dags.inbox_ranking.mirror import dag as mirror_dagfrom products.signals.dags.inbox_ranking.multihead.prototype import project_context-from products.signals.dags.inbox_ranking.training import dag as training_dag+from products.signals.dags.inbox_ranking.training import (+ dag as training_dag,+ telemetry,+)AS_OF = datetime.datetime(2026, 7, 30, tzinfo=datetime.UTC)BEFORE = AS_OF - datetime.timedelta(days=3)@@ -224,29 +227,62 @@ class _MirrorStorage:self.writes.append({"bucket": bucket, "key": key, "content": content, "extras": extras})+REMOTE_REPORT_IDS = [f"00000000-0000-4000-8000-00000000001{index}" for index in range(4)]+++class _CaptureClient:+ def __init__(self) -> None:+ self.calls: list[dict[str, Any]] = []++ def capture(self, **kwargs: Any) -> None:+ self.calls.append(kwargs)++ def shutdown(self) -> None:+ pass++@pytest.mark.parametrize(- "mirror_bucket,mirror_fails",- [("mirror-bucket", False), ("mirror-bucket", True), ("", False)],- ids=["mirror_set", "mirror_down", "no_mirror"],+ "mirror_bucket,mirror_fails,label_ids,local_ids,skip_reason",+ [+ ("mirror-bucket", False, [LOCAL_REPORT_ID, REMOTE_REPORT_ID, "not-a-uuid"], {LOCAL_REPORT_ID}, None),+ ("mirror-bucket", True, [LOCAL_REPORT_ID, REMOTE_REPORT_ID, "not-a-uuid"], {LOCAL_REPORT_ID}, None),+ ("", False, [LOCAL_REPORT_ID, REMOTE_REPORT_ID, "not-a-uuid"], {LOCAL_REPORT_ID}, None),+ ("mirror-bucket", False, [LOCAL_REPORT_ID, REMOTE_REPORT_ID], set(), dag.NO_LOCAL_REPORTS),+ (+ "mirror-bucket",+ False,+ [LOCAL_REPORT_ID, *REMOTE_REPORT_IDS],+ {LOCAL_REPORT_ID},+ dag.LOCAL_SHARE_COLLAPSED,+ ),+ ("mirror-bucket", False, [], set(), None),+ ],+ ids=["mirror_set", "mirror_down", "no_mirror", "no_local_reports", "local_share_collapsed", "empty_labels"],)def test_the_engagement_export_sends_only_remote_reports_to_the_mirror(- monkeypatch: pytest.MonkeyPatch, mirror_bucket: str, mirror_fails: bool+ monkeypatch: pytest.MonkeyPatch,+ mirror_bucket: str,+ mirror_fails: bool,+ label_ids: list[str],+ local_ids: set[str],+ skip_reason: str | None,) -> None:partition_key = BEFORE.date().isoformat()labels = pa.Table.from_pylist([{"snapshot_date": AS_OF.date(), "report_id": report_id, "first_opened_at": BEFORE, "open_count": 1}- for report_id in (LOCAL_REPORT_ID, REMOTE_REPORT_ID, "not-a-uuid")+ for report_id in label_ids],schema=LABELS_SCHEMA,)labels_key = common.partition_object_key("inbox_ranking", dag.LABELS_TABLE, partition_key)mirror = _MirrorStorage(fail=mirror_fails)looked_up: list[list[str]] = []+ capture = _CaptureClient()def fake_report_teams(report_ids: list[str], snapshot_end: datetime.datetime) -> dict[str, int]:looked_up.append(report_ids)- return {LOCAL_REPORT_ID: 2}+ return dict.fromkeys(local_ids & set(report_ids), 2)def mirror_storage() -> _MirrorStorage:assert mirror_bucket, "no mirror is set, so nothing may reach a second store"@@ -262,13 +298,38 @@ def test_the_engagement_export_sends_only_remote_reports_to_the_mirror(monkeypatch.setattr(dag, "serving_mirror_storage", mirror_storage)monkeypatch.setattr(dag.settings, "INBOX_RANKING_DATASET_S3_PREFIX", "inbox_ranking")monkeypatch.setattr(dag.settings, "INBOX_RANKING_SERVING_MIRROR_BUCKET", mirror_bucket)+ monkeypatch.setattr(telemetry, "is_cloud", lambda: True)+ monkeypatch.setattr(telemetry, "get_client", lambda region, **kwargs: capture)- dag.inbox_engagement_export(dagster.build_asset_context(partition_key=partition_key))+ context = dagster.build_asset_context(partition_key=partition_key)+ dag.inbox_engagement_export(context)- if not mirror_bucket or mirror_fails:+ if not mirror_bucket:+ assert mirror.writes == []+ assert capture.calls == []+ return+ metadata = context.get_output_metadata("result")+ written = not mirror_fails and skip_reason is None+ assert metadata["written"].value is written+ assert ("skip_reason" in metadata) is (skip_reason is not None)+ (event,) = capture.calls+ assert event["event"] == telemetry.ENGAGEMENT_EXPORT_PUBLISHED_EVENT+ local_rows = len(local_ids)+ assert {key: event["properties"][key] for key in ("written", "skip_reason", "excluded_rows", "local_share")} == {+ "written": written,+ "skip_reason": skip_reason,+ "excluded_rows": local_rows,+ "local_share": local_rows / len(label_ids) if label_ids else None,+ }+ assert event["properties"]["partition"] == partition_key+ assert event["properties"]["schema_version"] == dag.ENGAGEMENT_EXPORT_SCHEMA_VERSION+ # No report id and no team id leaves the region through the event.+ assert not {"report_id", "report_ids", "team_id", "team_ids"} & set(event["properties"])+ assert not set(label_ids) & {str(value) for value in event["properties"].values()}++ if not written:assert mirror.writes == []return- assert looked_up == [[LOCAL_REPORT_ID, REMOTE_REPORT_ID]](write,) = mirror.writesassert write["bucket"] == mirror_bucketassert write["key"] == serving_engagement_key("inbox_ranking", partition_key)@@ -277,6 +338,10 @@ def test_the_engagement_export_sends_only_remote_reports_to_the_mirror(assert write["extras"]["Metadata"][common.SCHEMA_VERSION_METADATA_KEY] == str(dag.ENGAGEMENT_EXPORT_SCHEMA_VERSION)exported = pq.read_table(io.BytesIO(write["content"]))assert exported.column_names == list(LABEL_COLUMNS)+ if not label_ids:+ assert exported.num_rows == 0+ return+ assert looked_up == [[LOCAL_REPORT_ID, REMOTE_REPORT_ID]]assert exported.to_pylist() == [dict.fromkeys(LABEL_COLUMNS) | {"report_id": REMOTE_REPORT_ID, "first_opened_at": BEFORE}]diff --git a/products/signals/dags/inbox_ranking/training/telemetry.py b/products/signals/dags/inbox_ranking/training/telemetry.pyindex 688ffc2248d9..c13d2e77aff1 100644--- a/products/signals/dags/inbox_ranking/training/telemetry.py+++ b/products/signals/dags/inbox_ranking/training/telemetry.py@@ -38,6 +38,7 @@ UNSEEN_CALIBRATION_EVENT = "inbox_ranking_unseen_calibration"HOLDOUT_CALIBRATION_EVENT = "inbox_ranking_holdout_calibration"SERVING_MANIFEST_PUBLISHED_EVENT = "inbox_ranking_serving_manifest_published"EXPERIMENT_COMPARED_EVENT = "inbox_ranking_experiment_compared"+ENGAGEMENT_EXPORT_PUBLISHED_EVENT = "inbox_ranking_engagement_export_published"# Candidate metadata copied onto every per-head event so a chart can filter or break down on it._CANDIDATE_CONTEXT_KEYS = (@@ -219,6 +220,32 @@ def serving_manifest_event(return TrainingEvent(event=SERVING_MANIFEST_PUBLISHED_EVENT, properties=properties)+def engagement_export_event(+ *,+ written: bool,+ skip_reason: str | None,+ rows: int,+ excluded_rows: int,+ local_share: float | None,+ bytes_written: int,+ schema_version: int,+) -> TrainingEvent:+ """What the day's engagement export sent to the mirror store. It carries counts only, never a+ report id or a team id."""+ return TrainingEvent(+ event=ENGAGEMENT_EXPORT_PUBLISHED_EVENT,+ properties={+ "written": written,+ "skip_reason": skip_reason,+ "rows": rows,+ "excluded_rows": excluded_rows,+ "local_share": local_share,+ "bytes": bytes_written,+ "schema_version": schema_version,+ },+ )++def unseen_head_graded_events(*, run_id: str, grades: Sequence[HeadGrade]) -> list[TrainingEvent]:"""One event per (model, head): the unseen AUC that sits next to the head's holdout AUC. A headwith rows but a single outcome class still reports, with a null AUC, so the series has no gap."""