:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(signals): export eu inbox engagement moments to the mirror bucket (#121373)
posthog[bot]c5355d6ca055
diff --git a/posthog/dags/locations/signals.py b/posthog/dags/locations/signals.pyindex 639e9f53456f..2df2c300d787 100644--- a/posthog/dags/locations/signals.py+++ b/posthog/dags/locations/signals.py@@ -24,6 +24,7 @@ if is_inbox_ranking_registered():inbox_ranking_dataset.inbox_report_reviewers,inbox_ranking_dataset.inbox_user_report_interactions,inbox_ranking_dataset.inbox_project_vectors,+ inbox_ranking_dataset.inbox_engagement_export,inbox_ranking_training.inbox_ranking_serving_manifest,inbox_ranking_served.inbox_ranking_served_scores,inbox_ranking_training.inbox_ranking_unseen_graded,diff --git a/products/signals/backend/ranking/serving_manifest.py b/products/signals/backend/ranking/serving_manifest.pyindex 778722bf015f..8531bf325ec0 100644--- a/products/signals/backend/ranking/serving_manifest.py+++ b/products/signals/backend/ranking/serving_manifest.py@@ -51,6 +51,7 @@ HEAD_FILE_SUFFIX = ".ubj"MANIFEST_FILE = "manifest.json"METADATA_FILE = "metadata.json"PROJECT_VECTORS_FILE = "project_vectors.parquet"+ENGAGEMENT_FILE = "engagement.parquet"_SERVING_SEGMENT = "serving"@@ -71,6 +72,12 @@ def serving_project_vectors_key(prefix: str, partition_key: str) -> str:return f"{prefix}/{_SERVING_SEGMENT}/project_vectors/{partition_key}/{PROJECT_VECTORS_FILE}"+def serving_engagement_key(prefix: str, partition_key: str) -> str:+ """Where one day of the engagement export for another region's reports sits. The US dataset dag+ writes it to the mirror store, and the mirror region reads it to build its own project vectors."""+ return f"{prefix}/{_SERVING_SEGMENT}/engagement/{partition_key}/{ENGAGEMENT_FILE}"++def model_key(model_name: str, model_version: str) -> str:return f"{model_name}@{model_version}"diff --git a/products/signals/dags/inbox_ranking/AGENTS.md b/products/signals/dags/inbox_ranking/AGENTS.mdindex 2080127f4ad6..9065800335d5 100644--- a/products/signals/dags/inbox_ranking/AGENTS.md+++ b/products/signals/dags/inbox_ranking/AGENTS.md@@ -8,6 +8,7 @@ Read `README.md` first for what the dataset is and how partitions behave. This f- `inbox_report_title_embeddings` is a **leaf**: never add it to `inbox_report_model_data`'s `deps` or give that join a title column, or a failure in the title snapshot fails the training table. Its own `deps=[MODEL_DATA_TABLE]` exists for memory, not data — two full sets of 1536-float vectors must not be held at once in the run pod. Raising the pod memory limit is not the first resort.- `inbox_user_report_interactions` is a **leaf** with its own `USER_INTERACTIONS_SCHEMA_VERSION`: keep it out of `inbox_report_model_data`'s `deps`, and never stamp it with `FEATURE_SCHEMA_VERSION`, or a user-grain change sets off the labels refresh sensor. It writes a row only for a report in this region's Postgres whose team is in `training_consent_team_ids()`.- `inbox_project_vectors` is a **leaf** with its own `PROJECT_VECTORS_SCHEMA_VERSION`: keep it out of `inbox_report_model_data`'s `deps`. Its `deps=[EMBEDDINGS_TABLE, LABELS_TABLE]` keep it after the embeddings snapshot, so the two vector-heavy steps do not share the run pod's memory. `project_vectors` takes a report into a vector only when its vector landed and its UI engagement happened strictly before `as_of`; never widen that to the latest snapshot value. A report born on day D reads the dt=D−1 vectors (see the README).+- `inbox_engagement_export` is a **leaf** with its own `ENGAGEMENT_EXPORT_SCHEMA_VERSION`, read by another region. It writes to the mirror store only under `serving_engagement_key`, because the mirror's write grant covers only `<prefix>/serving/`. Keep its columns exactly `LABEL_COLUMNS` and its rows limited to reports with no `SignalReport` in this region: no team ids, counts, text or vectors leave the region. A mirror failure logs and must not fail the asset.- Register new jobs/schedules in `posthog/dags/locations/signals.py` (or `signals_ml.py` for a torch-dependent dag), keep the EU gate (`is_inbox_ranking_registered()`), and remember the ci-dagster paths filter must cover every module a dag file imports (`uv run .github/scripts/check-dagster-paths.py`).## Training dag specificsdiff --git a/products/signals/dags/inbox_ranking/README.md b/products/signals/dags/inbox_ranking/README.mdindex b9f80d82018c..a9f11829b857 100644--- a/products/signals/dags/inbox_ranking/README.md+++ b/products/signals/dags/inbox_ranking/README.md@@ -79,6 +79,20 @@ Point-in-time contract:- The training side can rebuild any past day by calling the function on newer snapshots with an earlier `as_of`. No backfill is needed while the vectors are inside the embedding TTL.- Serving: when any manifest entry reads project vectors, `inbox_ranking_serving_manifest` copies the dt=D day into the deployment's object store before it writes the manifest. The sweep reads that day once per pass, only for the teams it scores. Its `as_of` is the end of dt=D, so it is never newer than a report scored after the publish. Until the next publish, a report scored on day D+2 still reads dt=D, which is one day older than training reads.+### Engagement export++`inbox_engagement_export` sends the UI engagement moments of the other region's reports to the mirror store, so that region can build its own project vectors from its own report embeddings.+The inbox telemetry of every region lands in the US dogfood project, so the `open`, `action` and `dismiss` moments of an EU report exist only in the US labels snapshot.++- It runs only when `INBOX_RANKING_SERVING_MIRROR_BUCKET` is set. Otherwise it logs and writes nothing.+- The rows are the dt=D labels rows whose `report_id` has no `SignalReport` in this region's Postgres before the cutoff. A row whose id is not a UUID is dropped.+- The columns are exactly `LABEL_COLUMNS` from `dataset/project_vectors.py`: `report_id` and the first opened, first UI dismissed and first UI action timestamps. There are no team ids, no counts, no text and no vectors, so the other region passes the file to `project_vectors()` unchanged.+- The labels snapshot is cumulative, so each day is a full snapshot as of the end of dt=D.+- The key is `serving_engagement_key(prefix, dt)`: `<prefix>/serving/engagement/<dt>/engagement.parquet`. The mirror's write grant covers only `<prefix>/serving/`. The write sets `ACL: bucket-owner-full-control` and stamps the row count, the snapshot date and `ENGAGEMENT_EXPORT_SCHEMA_VERSION` in the object metadata.+- **It is a leaf.** Nothing joins it. A mirror write failure does not fail the asset: it logs, and the output metadata carries `written=false`. The metadata also carries `rows`, `excluded_rows`, `bytes` and `s3_key`.+- A report deleted in this region keeps its Postgres row with the `deleted` status, so it is not exported. Only a report whose row is gone, for example after its team was deleted, looks remote. Such a row exposes a report id and timestamps, and it never matches a report vector in the other region.+- The grant has no delete, and no lifecycle rule covers the prefix, so the daily objects accumulate. Each one holds a report id and a few timestamps per row.+### Partition semantics- Partition `dt=D` snapshots the eligible report inventory (promoted before `D+1 00:00 UTC`, or referenced by any label event before it). Every label aggregate is bounded `event_time < D+1 00:00 UTC`.diff --git a/products/signals/dags/inbox_ranking/dataset/dag.py b/products/signals/dags/inbox_ranking/dataset/dag.pyindex 28037f540fb6..7d8ab538e3e1 100644--- a/products/signals/dags/inbox_ranking/dataset/dag.py+++ b/products/signals/dags/inbox_ranking/dataset/dag.py@@ -12,6 +12,9 @@ Nine assets on one daily partition, each writing Parquet under the configured S3inbox_user_report_interactions/v1/dt=D/ cumulative interactions at (report, person) graininbox_project_vectors/v1/dt=D/ per-team mean report vectors of past engagement, by kind+inbox_engagement_export writes no dataset object. It sends the dt=D engagement moments of the other+region's reports to the serving mirror store, for that region's project vectors.+The first four are report grain and feed one table. inbox_signal_embeddings is signal grain and isread on its own, joined to the others by report_id at training time. inbox_report_title_embeddingsis a report-grain leaf: nothing joins it, and no model family trains on it. inbox_report_reviewers is a@@ -56,6 +59,7 @@ Point-in-time caveats, per source:partition holds today's set, flagged by features_observed_at."""+import ioimport jsonimport datetimefrom collections.abc import Collection, Iterator, Mapping@@ -67,6 +71,7 @@ from django.db.models import Count, Minimport dagsterimport pyarrow as paimport pyarrow.compute as pc+import pyarrow.parquet as pqfrom posthog import settingsfrom posthog.clickhouse.client import sync_execute@@ -83,6 +88,7 @@ from products.signals.backend.models import (SignalReportSuggestedReviewer,)from products.signals.backend.ranking.inventory import spine_report_filter+from products.signals.backend.ranking.serving_manifest import serving_engagement_keyfrom products.signals.backend.report_embeddings import (EMBEDDING_DOCUMENT_TYPE,EMBEDDING_PRODUCT,@@ -99,7 +105,10 @@ from products.signals.dags.inbox_ranking.common import (DATASET_VERSION,HUMAN_ACTOR_KINDS,PARQUET_PART_NAME,+ ROW_COUNT_METADATA_KEY,S3_BUCKET_ENV,+ SCHEMA_VERSION_METADATA_KEY,+ SNAPSHOT_DATE_METADATA_KEY,dataset_bucket,dataset_unconfigured,ensure_utc,@@ -116,6 +125,7 @@ from products.signals.dags.inbox_ranking.common import (read_parquet,read_parquet_if_exists,s3_client,+ serving_mirror_storage,skip_unconfigured,snapshot_bounds,write_parquet,@@ -169,11 +179,15 @@ SIGNAL_EMBEDDINGS_TABLE = "inbox_signal_embeddings"REVIEWERS_TABLE = "inbox_report_reviewers"USER_INTERACTIONS_TABLE = "inbox_user_report_interactions"PROJECT_VECTORS_TABLE = "inbox_project_vectors"+ENGAGEMENT_EXPORT = "inbox_engagement_export"# The reviewers asset is a leaf with its own columns, so its schema moves apart from# FEATURE_SCHEMA_VERSION. The same rule applies: additive nullable columns bump it.REVIEWERS_SCHEMA_VERSION = 1+# The engagement export is read in another region, so its shape moves apart from every local snapshot.+ENGAGEMENT_EXPORT_SCHEMA_VERSION = 1+COMMON_ASSET_KWARGS: dict[str, Any] = {"group_name": "inbox_ranking","partitions_def": partition_def,@@ -1462,6 +1476,86 @@ def inbox_project_vectors(context: dagster.AssetExecutionContext) -> None:)+def remote_engagement_rows(labels: pa.Table, local_report_ids: Collection[str]) -> pa.Table:+ """The label rows of reports that do not exist in this region's Postgres.++ Inbox telemetry of every region lands in the US dogfood project, so the labels snapshot holds+ label-only rows for the other region's reports. A row whose id cannot be a report UUID is+ dropped too: no region can match it to a report vector."""+ report_ids = labels.column("report_id").to_pylist()+ valid_ids = valid_report_uuids(set(report_ids))+ keep = [report_id in valid_ids and report_id not in local_report_ids for report_id in report_ids]+ return labels.filter(pa.array(keep, type=pa.bool_()))+++ name=ENGAGEMENT_EXPORT,+ # Nothing joins this asset, so its failure does not fail the training table.+ deps=[LABELS_TABLE],+ **COMMON_ASSET_KWARGS,+)+def inbox_engagement_export(context: dagster.AssetExecutionContext) -> None:+ """Export the UI engagement moments of the other region's reports to the mirror store.++ The other region builds its project vectors from its own report embeddings, but the open,+ action and dismiss moments exist only in this region's labels snapshot. The export carries+ only `LABEL_COLUMNS`: report ids and timestamps, no team ids, no text and no vectors. The+ labels snapshot is cumulative, so each day is a full snapshot as of the end of dt=D.+ """+ if skip_unconfigured(context):+ return+ mirror_bucket = settings.INBOX_RANKING_SERVING_MIRROR_BUCKET+ if not mirror_bucket:+ context.log.info("INBOX_RANKING_SERVING_MIRROR_BUCKET is not set; skipping the engagement export")+ return+ partition_key = context.partition_key+ _, snapshot_end = snapshot_bounds(partition_key)+ prefix = settings.INBOX_RANKING_DATASET_S3_PREFIX++ labels = read_parquet(+ s3_client(), dataset_bucket(), partition_object_key(prefix, LABELS_TABLE, partition_key), list(LABEL_COLUMNS)+ )+ local_report_ids = _report_teams(+ sorted(valid_report_uuids(set(labels.column("report_id").to_pylist()))), snapshot_end+ ).keys()+ table = remote_engagement_rows(labels, local_report_ids)+ buffer = io.BytesIO()+ pq.write_table(table, buffer, compression="zstd")+ content = buffer.getvalue()++ key = serving_engagement_key(prefix, partition_key)+ written = True+ try:+ serving_mirror_storage().write(+ bucket=mirror_bucket,+ key=key,+ content=content,+ extras={+ # The mirror bucket is in another account, and its policy refuses a write without this ACL.+ "ACL": "bucket-owner-full-control",+ "Metadata": {+ ROW_COUNT_METADATA_KEY: str(table.num_rows),+ SNAPSHOT_DATE_METADATA_KEY: partition_key,+ SCHEMA_VERSION_METADATA_KEY: str(ENGAGEMENT_EXPORT_SCHEMA_VERSION),+ },+ },+ )+ except Exception as error:+ # The mirror is another deployment's store, so an outage there only logs. The other region+ # detects the missing day on its side.+ context.log.exception(f"engagement export for {partition_key} not written: {error!r}")+ written = False+ context.add_output_metadata(+ {+ "written": dagster.MetadataValue.bool(written),+ "rows": dagster.MetadataValue.int(table.num_rows),+ "excluded_rows": dagster.MetadataValue.int(labels.num_rows - table.num_rows),+ "bytes": dagster.MetadataValue.int(len(content)),+ "s3_key": dagster.MetadataValue.text(f"s3://{mirror_bucket}/{key}"),+ }+ )++def label_provenance_ok(pg_status: str | None,pg_updated_at: datetime.datetime | None,@@ -1629,6 +1723,7 @@ inbox_ranking_dataset_job = dagster.define_asset_job(REVIEWERS_TABLE,USER_INTERACTIONS_TABLE,PROJECT_VECTORS_TABLE,+ ENGAGEMENT_EXPORT,],# The seven label streams run sequentially and each may take its full 600s query timeout, so an# hour left a slow-but-valid pass no room for the join, the S3 writes, or an asset retry — anddiff --git a/products/signals/dags/inbox_ranking/tests/test_dataset.py b/products/signals/dags/inbox_ranking/tests/test_dataset.pyindex 02dee62dc33c..6e0b7ca91451 100644--- a/products/signals/dags/inbox_ranking/tests/test_dataset.py+++ b/products/signals/dags/inbox_ranking/tests/test_dataset.py@@ -1274,6 +1274,7 @@ def test_the_leaf_snapshots_run_in_the_job_without_feeding_the_join():dag.inbox_report_reviewers,dag.inbox_user_report_interactions,dag.inbox_project_vectors,+ dag.inbox_engagement_export,])model_data_deps = {key.path[-1] for key in dag.inbox_report_model_data.keys_by_input_name.values()}@@ -1282,6 +1283,7 @@ def test_the_leaf_snapshots_run_in_the_job_without_feeding_the_join():dag.REVIEWERS_TABLE,dag.USER_INTERACTIONS_TABLE,dag.PROJECT_VECTORS_TABLE,+ dag.ENGAGEMENT_EXPORT,):assert dagster.AssetKey(leaf) in selectionassert leaf not in model_data_depsdiff --git a/products/signals/dags/inbox_ranking/tests/test_project_vectors.py b/products/signals/dags/inbox_ranking/tests/test_project_vectors.pyindex cd6291ec1c0f..2c7583f17d93 100644--- a/products/signals/dags/inbox_ranking/tests/test_project_vectors.py+++ b/products/signals/dags/inbox_ranking/tests/test_project_vectors.py@@ -17,6 +17,7 @@ from products.signals.backend.ranking.serving_manifest import (SERVED_ROLE,ServingManifest,ServingManifestEntry,+ serving_engagement_key,serving_project_vectors_key,)from products.signals.dags.inbox_ranking import common@@ -24,6 +25,7 @@ from products.signals.dags.inbox_ranking.dataset import dagfrom products.signals.dags.inbox_ranking.dataset.dag import EMBEDDINGS_SCHEMA, LABELS_SCHEMAfrom products.signals.dags.inbox_ranking.dataset.project_vectors import (HISTORY_LIMIT,+ LABEL_COLUMNS,MIN_HISTORY_REPORTS,PROJECT_VECTORS_SCHEMA,PROJECT_VECTORS_SCHEMA_VERSION,@@ -202,3 +204,76 @@ def test_serving_reads_the_published_day_as_training_builds_it(monkeypatch: pytenp.testing.assert_array_equal(served.has_history[0], training.has_history[pair])assert training.has_history[training.index].tolist() == [[1, 1, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]assert not training.vectors[training.index[2]].any()+++LOCAL_REPORT_ID = "00000000-0000-4000-8000-000000000001"+REMOTE_REPORT_ID = "00000000-0000-4000-8000-000000000002"+++class _MirrorStorage:+ def __init__(self, fail: bool = False) -> None:+ self.fail = fail+ self.writes: list[dict[str, Any]] = []++ def write(self, bucket: str, key: str, content: bytes, extras: dict | None) -> None:+ if self.fail:+ raise RuntimeError("mirror unreachable")+ self.writes.append({"bucket": bucket, "key": key, "content": content, "extras": extras})+++ "mirror_bucket,mirror_fails",+ [("mirror-bucket", False), ("mirror-bucket", True), ("", False)],+ ids=["mirror_set", "mirror_down", "no_mirror"],+)+def test_the_engagement_export_sends_only_remote_reports_to_the_mirror(+ monkeypatch: pytest.MonkeyPatch, mirror_bucket: str, mirror_fails: bool+) -> None:+ partition_key = BEFORE.date().isoformat()+ labels = pa.Table.from_pylist(+ [+ {"snapshot_date": AS_OF.date(), "report_id": report_id, "first_opened_at": BEFORE, "open_count": 1}+ for report_id in (LOCAL_REPORT_ID, REMOTE_REPORT_ID, "not-a-uuid")+ ],+ schema=LABELS_SCHEMA,+ )+ labels_key = common.partition_object_key("inbox_ranking", dag.LABELS_TABLE, partition_key)+ mirror = _MirrorStorage(fail=mirror_fails)+ looked_up: list[list[str]] = []++ def fake_report_teams(report_ids: list[str], snapshot_end: datetime.datetime) -> dict[str, int]:+ looked_up.append(report_ids)+ return {LOCAL_REPORT_ID: 2}++ def mirror_storage() -> _MirrorStorage:+ assert mirror_bucket, "no mirror is set, so nothing may reach a second store"+ return mirror++ monkeypatch.setattr(dag, "skip_unconfigured", lambda context: False)+ monkeypatch.setattr(dag, "dataset_bucket", lambda: "test-bucket")+ monkeypatch.setattr(dag, "s3_client", lambda: None)+ monkeypatch.setattr(+ dag, "read_parquet", lambda client, bucket, key, columns=None: {labels_key: labels}[key].select(columns)+ )+ monkeypatch.setattr(dag, "_report_teams", fake_report_teams)+ monkeypatch.setattr(dag, "serving_mirror_storage", mirror_storage)+ monkeypatch.setattr(dag.settings, "INBOX_RANKING_DATASET_S3_PREFIX", "inbox_ranking")+ monkeypatch.setattr(dag.settings, "INBOX_RANKING_SERVING_MIRROR_BUCKET", mirror_bucket)++ dag.inbox_engagement_export(dagster.build_asset_context(partition_key=partition_key))++ if not mirror_bucket or mirror_fails:+ assert mirror.writes == []+ return+ assert looked_up == [[LOCAL_REPORT_ID, REMOTE_REPORT_ID]]+ (write,) = mirror.writes+ assert write["bucket"] == mirror_bucket+ assert write["key"] == serving_engagement_key("inbox_ranking", partition_key)+ # The mirror bucket is in another account, and its policy refuses a write without this ACL.+ assert write["extras"]["ACL"] == "bucket-owner-full-control"+ assert write["extras"]["Metadata"][common.SCHEMA_VERSION_METADATA_KEY] == str(dag.ENGAGEMENT_EXPORT_SCHEMA_VERSION)+ exported = pq.read_table(io.BytesIO(write["content"]))+ assert exported.column_names == list(LABEL_COLUMNS)+ assert exported.to_pylist() == [+ dict.fromkeys(LABEL_COLUMNS) | {"report_id": REMOTE_REPORT_ID, "first_opened_at": BEFORE}+ ]diff --git a/products/signals/dags/inbox_ranking/tests/test_training.py b/products/signals/dags/inbox_ranking/tests/test_training.pyindex 494cd9257b5b..d000d2026983 100644--- a/products/signals/dags/inbox_ranking/tests/test_training.py+++ b/products/signals/dags/inbox_ranking/tests/test_training.py@@ -50,6 +50,7 @@ from products.signals.backend.ranking.serving_manifest import (ServingManifest,ServingManifestEntry,model_key,+ serving_engagement_key,serving_manifest_key,serving_model_prefix,serving_project_vectors_key,@@ -2503,6 +2504,11 @@ def test_the_serving_prefix_layout_is_stable():serving_model_prefix("inbox_ranking", model_key(EMBEDDINGS_MODEL_NAME, "2026-08-19"))== "inbox_ranking/serving/models/report_embeddings_xgb@2026-08-19")+ # The mirror's write grant covers only the serving prefix.+ assert (+ serving_engagement_key("inbox_ranking", "2026-08-19")+ == "inbox_ranking/serving/engagement/2026-08-19/engagement.parquet"+ )def _served_event(report_id: str, at: str, **properties: Any) -> tuple[pd.Timestamp, dict[str, Any]]: