:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(signals): train inbox ranking networks on a 90-day lookback (#120684)
posthog[bot]25a64ebcd1b4
diff --git a/docs/internal/inbox-ranking-training.md b/docs/internal/inbox-ranking-training.mdindex fb7d63202b9e..ca15f9d4e04b 100644--- a/docs/internal/inbox-ranking-training.md+++ b/docs/internal/inbox-ranking-training.md@@ -43,7 +43,7 @@ Each head in `metadata.json` records `cohort` (`everyone`) and `horizon_days`.## Row budgetA feature set can limit the rows one head keeps (`max_examples_per_head`).-The report-embedding family sets it to 100,000.+The report-embedding sets set it to 250,000.The budget limits history, not the rows inside a day:- The head's examples are grouped by report-creation day, newest first.diff --git a/posthog/settings/object_storage.py b/posthog/settings/object_storage.pyindex 8d94d29023cb..7f1649002def 100644--- a/posthog/settings/object_storage.py+++ b/posthog/settings/object_storage.py@@ -105,7 +105,7 @@ INBOX_RANKING_DATASET_S3_PREFIX = os.getenv("INBOX_RANKING_DATASET_S3_PREFIX", "# Training dag (products/signals/dags/inbox_ranking/training): how many daily snapshots back the# examples reach, and how many trailing days of reports grade a candidate. Promotion is manual: a# person runs `inbox_ranking_promote_job`, which writes the served pointer.-INBOX_RANKING_TRAINING_LOOKBACK_DAYS = get_from_env("INBOX_RANKING_TRAINING_LOOKBACK_DAYS", 60, type_cast=int)+INBOX_RANKING_TRAINING_LOOKBACK_DAYS = get_from_env("INBOX_RANKING_TRAINING_LOOKBACK_DAYS", 90, type_cast=int)INBOX_RANKING_TRAINING_HOLDOUT_DAYS = get_from_env("INBOX_RANKING_TRAINING_HOLDOUT_DAYS", 7, type_cast=int)# Labels refresh sensor (products/signals/dags/inbox_ranking/dataset): how many stale labels# partitions one hourly tick rewrites after a FEATURE_SCHEMA_VERSION bump, newest first.diff --git a/products/signals/backend/ranking/features.py b/products/signals/backend/ranking/features.pyindex cd59d5325292..21ed05b718d7 100644--- a/products/signals/backend/ranking/features.py+++ b/products/signals/backend/ranking/features.py@@ -150,11 +150,11 @@ class ReportEmbeddingsFeatureSet(FeatureSet):schema_version = 1feature_names = EMBEDDING_FEATURE_NAMESstate_columns: tuple[str, ...] = ()- # 1536 float32 columns, so a head's Parquet slice and its training matrix both scale with this.- # Sized so every head of this family fits one partition's examples object and the fits stay- # inside the training job's runtime budget. The budget cuts the oldest report-creation days, so- # it must stay large enough to keep the rare heads' positives.- max_examples_per_head = 100_000+ # 1536 float32 columns, so a head's training matrix scales with this. Sized above the largest+ # head over the 90-day lookback, so the budget is a memory guard and does not trim the window.+ # The budget cuts the oldest report-creation days, so a budget that binds starves the rare+ # heads of the positives the longer lookback adds.+ max_examples_per_head = 250_000def __init__(self, *, name: str, extras_key: str) -> None:self.name = namediff --git a/products/signals/dags/inbox_ranking/AGENTS.md b/products/signals/dags/inbox_ranking/AGENTS.mdindex 0b0eb0af28f7..7884c528ff58 100644--- a/products/signals/dags/inbox_ranking/AGENTS.md+++ b/products/signals/dags/inbox_ranking/AGENTS.md@@ -26,7 +26,7 @@ Read `README.md` first for what the dataset is and how partitions behave. This f- `served_score_rows` keeps the earliest scored event per (report, model key) and takes `model_role` from the event's roles. Grades stay per `(model_name, model_version, model_role)` and are never pooled across versions. A re-run that reads no event is refused when the partition already holds rows, and so is one that reads fewer families than the object holds, because the newborn state snapshot behind those rows ages out and the dt=D+horizon grade reads them.- Every scored row carries `head_readable`, and an unreadable head is still graded: a head that misses `min_holdout_positives` on one day's holdout still gets a number from the pooled online grade. Keep the promotion gate reading `readable` from the candidate metadata: an unreadable head must not decide a promotion.- `model_name` is the model recipe, and every model object, scored row, grade and event carries it. A family is an entry in `MODEL_FAMILIES` (`training/unseen.py`) naming the feature set it is fit on, its `learner` and the `location` that trains it; each location's candidate and champion assets walk `families_trained_in(<location>)`, and the grader and the events need no change to gain a family. A candidate is graded only against the served model of its own `model_name`; the served pointer itself can name any family. Do not fold the family into `model_version`, which the dashboard filters as a date.-- A recipe change (features, heads, horizons, learner or training settings, `DATASET_VERSION`) needs a new `model_name`, such as `nn_v2`, because the promotion gate grades a candidate only against the served model of its own name. Every candidate writes `recipe` and `recipe_hash` (`training/recipe.py`), and the candidate asset fails before it writes when the hash differs from the newest hashed version of the name at or before its partition, a re-run of the same partition included. Keep anything that changes daily (dates, `run_id`, counts, metrics) out of the recipe, or every retrain fails. Do not add an override for the check: the fix is a new name.+- A recipe change (features, heads, horizons, learner or training settings, `DATASET_VERSION`) needs a new `model_name`, such as `nn_v3`, because the promotion gate grades a candidate only against the served model of its own name. Every candidate writes `recipe` and `recipe_hash` (`training/recipe.py`), and the candidate asset fails before it writes when the hash differs from the newest hashed version of the name at or before its partition, a re-run of the same partition included. Keep anything that changes daily (dates, `run_id`, counts, metrics) out of the recipe, or every retrain fails. Do not add an override for the check: the fix is a new name.- A family whose `metadata.json`, examples object or side input is missing for a partition is logged and skipped, so one family does not cost every family's series. The champion asset raises only when no family has a candidate at all.- Skipping is not the same as rebuilding from nothing, and the difference is destructive. An empty examples object produces an empty candidate, and the candidate's prefix cleanup then deletes the boosters that partition already holds, which `served.json` or a pin can be pointing at. So when a side input is unavailable, leave the partition's objects alone rather than writing empty ones over them.- The champion is compared to a candidate on the candidate's holdout, through the champion's `<head>.holdout.ubj` (the train-only fit). Keep writing that file: without it the gate falls back to the champion's stored AUC, which was measured on a different set of reports.@@ -38,7 +38,7 @@ Read `README.md` first for what the dataset is and how partitions behave. This f- Publishing order is load-bearing. Every model copy finishes before `manifest.json` is written, so a failed copy leaves the previous manifest and the previous models serving. A model version is immutable, so a version whose `metadata.json` is already at the target is not copied again. A run with nothing to publish, such as a run without `served.json`, writes nothing rather than an empty manifest. The served model passes the same check as a pin (`_pin_metadata`) before composition.- When a manifest entry has `reads_project_vectors`, the dt=D `inbox_project_vectors` day is copied to `serving/project_vectors/<D>/` before `manifest.json`, and the manifest names it in `project_vectors`. A missing day fails the publish, like a missing model file. Never copy project vectors to the mirror: they are per-team data of this deployment, so the mirror's manifest names none. Training and serving build the `project` and `has_history` inputs through the one `project_inputs` in `features.py`; keep `PROJECT_VECTOR_KINDS` and `MIN_PROJECT_HISTORY_REPORTS` there.- `HEADS` holds only the heads the inbox reads (`HeadRanker` in `backend/ranking/inbox_sort.py`). Add a head there only together with a surface that reads it. The promotion gate skips a champion head that is not in `HEADS`, so a trim does not block promotion.-- `nn_v2` and `nn_with_prio` are trained by the `signals_ml` location (`nn/dag.py`), never by the main one. They share one `inbox_ranking_nn_examples` object, and each has its own candidate and champion asset from `family_assets`. The `nn_v2` assets keep the names `inbox_ranking_nn_model_{candidate,champion}`, so their Dagster history continues. A later family's candidate depends on the previous family's champion, for memory and not for data: two candidates at once can exceed the job's memory limit. A family whose set reads a state column the examples object lacks skips with a warning rather than training on zeros. The main examples asset builds only the sets of `families_trained_in(MAIN_LOCATION)`, so registering a set for `signals_ml` does not add a 1536-column object to the main job. torch is imported only inside the candidate asset's body (through `nn/train.py` and `multihead/network.py`). `signals_ml` must import without torch, and `test_multihead_network.py` pins that. Its decision goes through `decide_family_champion`, the same rule as the boosters, graded through the incumbent's `model.holdout.onnx`. Keep `signals_ml` commented out in `.dagster_home/workspace.yaml`, and keep its schedule (04:30 UTC) running by default on prod US only.+- `nn_v3` and `nn_with_prio_v2` are trained by the `signals_ml` location (`nn/dag.py`), never by the main one. They share one `inbox_ranking_nn_examples` object, and each has its own candidate and champion asset from `family_assets`. The `nn_v3` assets keep the names `inbox_ranking_nn_model_{candidate,champion}`, so their Dagster history continues. A later family's candidate depends on the previous family's champion, for memory and not for data: two candidates at once can exceed the job's memory limit. A family whose set reads a state column the examples object lacks skips with a warning rather than training on zeros. The main examples asset builds only the sets of `families_trained_in(MAIN_LOCATION)`, so registering a set for `signals_ml` does not add a 1536-column object to the main job. torch is imported only inside the candidate asset's body (through `nn/train.py` and `multihead/network.py`). `signals_ml` must import without torch, and `test_multihead_network.py` pins that. Its decision goes through `decide_family_champion`, the same rule as the boosters, graded through the incumbent's `model.holdout.onnx`. Keep `signals_ml` commented out in `.dagster_home/workspace.yaml`, and keep its schedule (04:30 UTC) running by default on prod US only.## Invariants — do not breakdiff --git a/products/signals/dags/inbox_ranking/README.md b/products/signals/dags/inbox_ranking/README.mdindex a41c7b39748d..eb6cd89b8e8d 100644--- a/products/signals/dags/inbox_ranking/README.md+++ b/products/signals/dags/inbox_ranking/README.md@@ -150,8 +150,8 @@ s3://<bucket>/<prefix>/└── inbox_ranking_served_scores/v1/dt=YYYY-MM-DD/ # the scoring sweep's scores of the reports born that day, per model key```-- **A model is a `(model_name, model_version, model_role)`.** `model_name` is the recipe: which features, which learner and which training settings: `report_embeddings_v2`, per-head XGBoost, is the one trained here. `nn_v2`, one ONNX network, and `nn_with_prio`, the same network with the report's priority as an input, are trained by the `signals_ml` location, and each `MODEL_FAMILIES` entry names its `learner` and the `location` that trains it. `model_version` is the partition day it was fit on, and `model_role` is the role it was scored or trained under (`candidate` on the training events; `served`, `recent`, `challenger`, `pinned` or `cross_family` on the online grade, and `daily_candidate` on grades written before the `recent` role). Each family owns a prefix under the models path, so two families trained on the same day cannot collide. A candidate is graded only against the served model of its own name. The served pointer can name any family, so a rollback can cross families.-- **A recipe change gets a new `model_name`.** `report_embeddings` and `nn` became `report_embeddings_v2` and `nn_v2` when `action` started to count a thumbs up. A candidate's `metadata.json` carries `recipe` and `recipe_hash`, a stable hash of the recipe (`training/recipe.py`): the learner, the feature set and its schema version, the project-vector kinds and their history settings, the booster params or network config, each head's cohort, label, horizon, label columns and status-label flag, `lookback_days`, `holdout_days`, the example grain and row budget, and `dataset_version`. Dates, `run_id`, row counts and metrics stay out, so a daily retrain keeps the hash. The candidate asset compares its hash with the newest version of the same name, at or before its partition, that has one, and on a mismatch it fails before it writes the partition and names the changed inputs. A re-run of a partition therefore compares with that partition's own saved recipe. The search covers every dataset version, so a `DATASET_VERSION` bump also needs a new name. A version without a hash is not compared. This includes an env change such as `INBOX_RANKING_TRAINING_LOOKBACK_DAYS`. To start a new recipe, add a `MODEL_FAMILIES` entry with a new name (`nn_v2` after `nn`) and remove the old entry: the old versions stay servable by key, and the promotion gate never compares two recipes. Two recipes can train side by side for a comparison, but each costs a manifest slot, so do not do it by default. `recipe_hash` is on `inbox_ranking_candidate_trained` and `inbox_ranking_promotion_decided` (with `incumbent_recipe_hash`), and `inbox_ranking_serving_manifest_published` carries `model_recipe_hashes` per model key.+- **A model is a `(model_name, model_version, model_role)`.** `model_name` is the recipe: which features, which learner and which training settings: `report_embeddings_v2`, per-head XGBoost, is the one trained here. `nn_v3`, one ONNX network, and `nn_with_prio_v2`, the same network with the report's priority as an input, are trained by the `signals_ml` location, and each `MODEL_FAMILIES` entry names its `learner` and the `location` that trains it. `model_version` is the partition day it was fit on, and `model_role` is the role it was scored or trained under (`candidate` on the training events; `served`, `recent`, `challenger`, `pinned` or `cross_family` on the online grade, and `daily_candidate` on grades written before the `recent` role). Each family owns a prefix under the models path, so two families trained on the same day cannot collide. A candidate is graded only against the served model of its own name. The served pointer can name any family, so a rollback can cross families.+- **A recipe change gets a new `model_name`.** `report_embeddings` and `nn` became `report_embeddings_v2` and `nn_v2` when `action` started to count a thumbs up, and `nn_v2` and `nn_with_prio` became `nn_v3` and `nn_with_prio_v2` when the lookback went from 60 to 90 days and the row budget from 100,000 to 250,000. A candidate's `metadata.json` carries `recipe` and `recipe_hash`, a stable hash of the recipe (`training/recipe.py`): the learner, the feature set and its schema version, the project-vector kinds and their history settings, the booster params or network config, each head's cohort, label, horizon, label columns and status-label flag, `lookback_days`, `holdout_days`, the example grain and row budget, and `dataset_version`. Dates, `run_id`, row counts and metrics stay out, so a daily retrain keeps the hash. The candidate asset compares its hash with the newest version of the same name, at or before its partition, that has one, and on a mismatch it fails before it writes the partition and names the changed inputs. A re-run of a partition therefore compares with that partition's own saved recipe. The search covers every dataset version, so a `DATASET_VERSION` bump also needs a new name. A version without a hash is not compared. This includes an env change such as `INBOX_RANKING_TRAINING_LOOKBACK_DAYS`. To start a new recipe, add a `MODEL_FAMILIES` entry with a new name (`nn_v2` after `nn`) and remove the old entry: the old versions stay servable by key, and the promotion gate never compares two recipes. Two recipes can train side by side for a comparison, but each costs a manifest slot, so do not do it by default. `recipe_hash` is on `inbox_ranking_candidate_trained` and `inbox_ranking_promotion_decided` (with `incumbent_recipe_hash`), and `inbox_ranking_serving_manifest_published` carries `model_recipe_hashes` per model key.- **A feature set is one feature universe.** `products/signals/backend/ranking/features.py` holds a `FeatureSet` per universe: its name, its schema version, its ordered feature names, the report-state columns it reads, `build_matrix`, and how many rows it wants per report and per head. `report_embeddings` is the report vector, and `report_embeddings_priority` is that vector plus one column per priority level (`priority_P0` to `priority_P4`, all zero when the report has no judgment), read from the `priority` state column. A candidate records the set it was fit on in its `metadata.json`, so the grader checks a model against its own set rather than one global contract, the examples asset writes one Parquet per set, and the unseen scorer builds one matrix per set and shares it across the families that read it. Adding a set means an entry in `FEATURE_SETS`; a model naming a set this build cannot produce is logged and left unscored. The main examples asset writes a Parquet only for the sets of the families the main location trains, so a set only `signals_ml` reads costs it nothing.- **Examples are one row per report at its birth, by default.** For partition `dt=D` the examples asset reads the report-state and labels snapshots `dt=D-lookback..D` and emits one row per report, on the snapshot of the day it was created, labeled from the snapshot `horizon_days` later (7 for open, 14 for action, 30 for fixed). Features are built by the feature set from that snapshot's state columns plus `age_hours`, the report's age at the snapshot. Labels are aligned to the state spine, so a report with no label event is a negative (all-zero labels), not absent. Label-only rows (no Postgres state) are skipped, as are state rows read long after their snapshot day (backfills carry current Postgres state; see `features_observed_at`) and, for the heads that read the status stream (`action` and `fixed`), rows whose status telemetry fails the dataset's `label_provenance_ok` check. Birth is the moment serving scores a report, and the label that comes out is a per-report probability, which is what the inbox ordering needs. A report born before the window, or one whose birth-day snapshot is missing from it, is no example; `reports_missing_birth_snapshot` on the asset counts the second case, which is a gap in the partitions.- **An outcome already visible on the birth day is a future positive.** A report born on day D has no scoring moment before D, so an outcome visible at D belongs to the moment being built rather than to an earlier one; the labels of D therefore read as their defaults for a report born on D. Most outcomes land on the birth day, so this is where the positives are: censoring on them costs the `pr_created` head most of its training signal. `<set>_<head>_birth_day_positives` on the examples asset, and `birth_day_positives` on the `inbox_ranking_examples_built` event, say how many positives the rule keeps. These rows carry a hindsight the other rows do not: the state snapshot reads `signal_count`, `total_weight`, `run_count` and the text sizes live from Postgres a few hours after day D ends, so a birth-day outcome happened before its own feature read. Both the holdout AUC and the newborn unseen grade therefore read optimistically on these rows, until the scoring sweep's timestamped score log replaces the daily snapshot as this table's source.@@ -170,7 +170,7 @@ s3://<bucket>/<prefix>/```bashdagster job launch -w .dagster_home/workspace.yaml --location posthog.dags.locations.signals -j inbox_ranking_promote_job \- --config-json '{"ops": {"inbox_ranking_promote": {"config": {"model_key": "nn_v2@2026-10-20", "reason": "seed the pointer"}}}}'+ --config-json '{"ops": {"inbox_ranking_promote": {"config": {"model_key": "nn_v3@2026-10-21", "reason": "seed the pointer"}}}}'```- **Serving manifest**: the dag's decision about what the scoring sweep runs, published where the sweep can read it. The Temporal workers have no credential for the dataset bucket, so `inbox_ranking_serving_manifest` copies the models the sweep needs into the deployment's own object store and writes a manifest naming them:@@ -348,21 +348,21 @@ The network leaves out every report that is in any head's holdout, so the compar`nn` is one multi-head network (`multihead/network.py`) over the report vector and the team's `typical`, `open`, `action` and `dismiss` project vectors, with the heads `open`, `action` and `fixed`. It is a model family like the per-head ones: it has its own prefix, the main job's serving manifest names a served, challenger or cross-family `nn` model as an `onnx` entry, and the online grader grades it. The main location's candidate and champion assets never fit it.-Two `nn` families train every day. `nn_v2` reads the `report_embeddings` set. `nn_with_prio` reads `report_embeddings_priority`: its `report` input carries the 1536 vector components and then the five priority levels, and those five columns skip the shared projection and go straight into the trunk input. Neither is served by default; a person serves one with `inbox_ranking_promote_job` after comparing their online grades.+Two `nn` families train every day. `nn_v3` reads the `report_embeddings` set. `nn_with_prio_v2` reads `report_embeddings_priority`: its `report` input carries the 1536 vector components and then the five priority levels, and those five columns skip the shared projection and go straight into the trunk input. Neither is served by default; a person serves one with `inbox_ranking_promote_job` after comparing their online grades.`inbox_ranking_nn_training_job` lives on its own code location, `posthog/dags/locations/signals_ml.py`, which runs on an image with the `inbox-ranking` group. torch is imported only inside the candidate asset's body, so the location module imports on the main image too, and `test_multihead_network.py` fails if torch reaches it. The job writes:```texts3://<bucket>/<prefix>/├── inbox_ranking_nn_examples/v1/dt=YYYY-MM-DD/ # rows, a label and a holdout flag per head, priority; no vectors-├── inbox_ranking_models/v1/nn_v2/+├── inbox_ranking_models/v1/nn_v3/│ ├── dt=YYYY-MM-DD/model.onnx + model.holdout.onnx + metadata.json│ └── champion.json # retired; nothing writes it-└── inbox_ranking_models/v1/nn_with_prio/dt=YYYY-MM-DD/model.onnx + model.holdout.onnx + metadata.json+└── inbox_ranking_models/v1/nn_with_prio_v2/dt=YYYY-MM-DD/model.onnx + model.holdout.onnx + metadata.json```-- **Examples** are the prototype's: each head's `example_moments` over the `report_embeddings` set, capped by `cap_examples` and split by `holdout_mask`, so every head's rows and holdout are the ones its `report_embeddings` champion is graded on. The object holds no vector, and one object serves both families. Each row carries its `priority` from its own snapshot. The candidate and champion assets rebuild the report vectors and the project vectors from the dt=D embeddings and labels snapshots. An object written before the `priority` column existed makes the `nn_with_prio` candidate skip with a warning; re-run the examples asset for that day.-- **Candidate**: the network trains on the rows in no head's holdout and is graded on each head's own holdout through the same `head_metrics` the boosters use, so `metadata.json` carries the same per-head fields (pooled and within-team AUC, average precision, ECE, readability, row counts, example window). Readability uses the boosters' permutation null: three networks fit on permuted labels. `model.onnx` is a refit on every labeled row, and `model.holdout.onnx` is the train-only fit. `metadata.json` has `model_kind: onnx`, the family's `feature_set`, and `output_heads`, the graph's column order. The `nn_with_prio` candidate runs after the `nn_v2` champion, for memory and not for data: each candidate holds an embeddings snapshot and the project vectors.+- **Examples** are the prototype's: each head's `example_moments` over the `report_embeddings` set, capped by `cap_examples` and split by `holdout_mask`, so every head's rows and holdout are the ones its `report_embeddings` champion is graded on. The object holds no vector, and one object serves both families. Each row carries its `priority` from its own snapshot. The candidate and champion assets rebuild the report vectors and the project vectors from the dt=D embeddings and labels snapshots. An object written before the `priority` column existed makes the `nn_with_prio_v2` candidate skip with a warning; re-run the examples asset for that day.+- **Candidate**: the network trains on the rows in no head's holdout and is graded on each head's own holdout through the same `head_metrics` the boosters use, so `metadata.json` carries the same per-head fields (pooled and within-team AUC, average precision, ECE, readability, row counts, example window). Readability uses the boosters' permutation null: three networks fit on permuted labels. `model.onnx` is a refit on every labeled row, and `model.holdout.onnx` is the train-only fit. `metadata.json` has `model_kind: onnx`, the family's `feature_set`, and `output_heads`, the graph's column order. The `nn_with_prio_v2` candidate runs after the `nn_v3` champion, for memory and not for data: each candidate holds an embeddings snapshot and the project vectors.- **Champion**: the same rule as the other families (`decide_family_champion`), recorded and never written to a pointer. The incumbent is the served model when it has the candidate's `model_name`, graded through its `model.holdout.onnx` on the candidate's holdout, with `onnxruntime`.- **A missing partition is a skip.** If the dt=D state, labels or embeddings snapshot is missing, each asset logs and writes nothing, so the partition keeps the objects it holds.- **Schedule**: 04:30 UTC, after the dataset job's 2h budget (02:30 UTC start) and before the 05:00 UTC training job when the run is short enough. A candidate that lands after that day's manifest is paired by the next day's manifest. The schedule runs by default on prod US and is stopped everywhere else.@@ -421,7 +421,7 @@ Bump `RENDER_VERSION` with any change to a filter, a cap or the layout.| --------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- || `INBOX_RANKING_DATASET_S3_BUCKET` | unset | Destination bucket. Unset on Cloud makes every asset log and skip, so the dag can deploy before the bucket exists. Unset elsewhere falls back to the deployment's object-storage service (SeaweedFS in dev and CI). || `INBOX_RANKING_DATASET_S3_PREFIX` | `inbox_ranking` | Key prefix under the bucket. |-| `INBOX_RANKING_TRAINING_LOOKBACK_DAYS` | `60` | How many daily snapshots back the training examples reach. |+| `INBOX_RANKING_TRAINING_LOOKBACK_DAYS` | `90` | How many daily snapshots back the training examples reach. || `INBOX_RANKING_TRAINING_HOLDOUT_DAYS` | `7` | Trailing days of reports that grade a candidate. || `INBOX_RANKING_SERVING_MIRROR_BUCKET` | unset | A second deployment's object-storage bucket that receives a copy of every serving publish, through ambient AWS config. Unset means no mirror. || `INBOX_RANKING_SERVING_MIRROR_REGION` | unset | AWS region of the mirror bucket. |@@ -437,7 +437,7 @@ All reads route to the offline cluster replicas on Cloud (`etl_workload()`), car## Operating it- Backfill any day range from the Dagster UI; partitions start 2026-04-01 (the label epoch). Every asset sits in the `inbox_ranking_etl` pool so concurrent partitions don't each start their own fleet-wide embeddings scan — the pool's limit is a Dagster deployment setting, provisioned with the bucket.-- A label column change needs a `FEATURE_SCHEMA_VERSION` bump, and no manual backfill. The labels asset stamps the version in each object's `feature-schema-version` metadata. Every hour, `inbox_ranking_labels_refresh_sensor` finds labels partitions in the training lookback whose stamp is missing or older, and runs `inbox_ranking_labels_refresh_job` on at most `INBOX_RANKING_LABELS_REFRESH_MAX_RUNS` (default 6) of them, newest first. A 60-day lookback is current again in about 10 hours. The sensor skips the newest day, which the daily schedule writes, and partitions with no labels object. It requests a partition at most once per schema version, so a failed refresh alerts and needs a person. The refresh covers labels only: report state reads current Postgres and embeddings have a TTL, so a rewrite of those is not point-in-time. `pairs_skipped_missing_label_columns` on `inbox_ranking_examples_built` counts the snapshot pairs each head lost to a missing label column while partitions are stale.+- A label column change needs a `FEATURE_SCHEMA_VERSION` bump, and no manual backfill. The labels asset stamps the version in each object's `feature-schema-version` metadata. Every hour, `inbox_ranking_labels_refresh_sensor` finds labels partitions in the training lookback whose stamp is missing or older, and runs `inbox_ranking_labels_refresh_job` on at most `INBOX_RANKING_LABELS_REFRESH_MAX_RUNS` (default 6) of them, newest first. A 90-day lookback is current again in about 15 hours. The sensor skips the newest day, which the daily schedule writes, and partitions with no labels object. It requests a partition at most once per schema version, so a failed refresh alerts and needs a person. The refresh covers labels only: report state reads current Postgres and embeddings have a TTL, so a rewrite of those is not point-in-time. `pairs_skipped_missing_label_columns` on `inbox_ranking_examples_built` counts the snapshot pairs each head lost to a missing label column while partitions are stale.- Do not materialize these assets in-process in the code-location pod. Its memory limit (2Gi) is too small; launch a run instead.- Failures alert `#alerts-self-driving` (owner `team-self-driving`); assets retry twice with a 60s delay before failing a run. A UI-launched materialization runs under Dagster's implicit `__ASSET_JOB`, which carries no owner tag, so alert routing falls back to matching the `inbox_report_`, `inbox_signal_`, and `inbox_ranking_` asset-name prefixes.- Runtime budgets are per job (`dagster/max_runtime`): 2h for the dataset job, 3h for the training job. The dataset needs the 2h: its seven label streams run sequentially, each allowed up to 600s, and the join and S3 writes come after them. The dataset cap ends before training starts at 05:00 UTC, so a stuck dataset run fails before training reads its snapshots.diff --git a/products/signals/dags/inbox_ranking/nn/dag.py b/products/signals/dags/inbox_ranking/nn/dag.pyindex 5d04fca3a9bd..705e7ada8ae5 100644--- a/products/signals/dags/inbox_ranking/nn/dag.py+++ b/products/signals/dags/inbox_ranking/nn/dag.py@@ -1,5 +1,5 @@"""Daily training for the `nn` model families: one multi-head network over the report vector and the-team's project vectors, exported to ONNX. `nn_v2` reads the vector only, and `nn_with_prio` also+team's project vectors, exported to ONNX. `nn_v3` reads the vector only, and `nn_with_prio_v2` alsoreads the report's priority levels.One examples asset, and a candidate and a champion asset per family, on the dataset dag's daily@@ -91,7 +91,7 @@ from products.signals.dags.inbox_ranking.training.train import HoldoutGradefrom products.signals.dags.inbox_ranking.training.unseen import NN_FAMILY, NN_WITH_PRIO_FAMILY, ModelFamilyNN_EXAMPLES_TABLE = "inbox_ranking_nn_examples"-# The `nn_v2` assets keep the names they had as the only family, so their Dagster history continues.+# The `nn_v3` assets keep the names they had as the only family, so their Dagster history continues.NN_CANDIDATE_ASSET = "inbox_ranking_nn_model_candidate"NN_CHAMPION_ASSET = "inbox_ranking_nn_model_champion"NN_WITH_PRIO_CANDIDATE_ASSET = "inbox_ranking_nn_with_prio_model_candidate"diff --git a/products/signals/dags/inbox_ranking/training/unseen.py b/products/signals/dags/inbox_ranking/training/unseen.pyindex 23e8af4e078d..905f941f7ef4 100644--- a/products/signals/dags/inbox_ranking/training/unseen.py+++ b/products/signals/dags/inbox_ranking/training/unseen.py@@ -62,8 +62,8 @@ CANDIDATE_ROLE = "candidate"# A recipe change takes a new name (`training/recipe.py`). The versions of an earlier name stay# servable by key, and their online grades keep their own series.REPORT_EMBEDDINGS_MODEL_NAME = "report_embeddings_v2"-NN_MODEL_NAME = "nn_v2"-NN_WITH_PRIO_MODEL_NAME = "nn_with_prio"+NN_MODEL_NAME = "nn_v3"+NN_WITH_PRIO_MODEL_NAME = "nn_with_prio_v2"# The Dagster code locations that train a family: `signals` on the main image, and `signals_ml` on# the image that carries torch.MAIN_LOCATION = "signals"@@ -121,7 +121,7 @@ class ModelFamily:# `nn` is one multi-head network over the report vector and the team's project vectors. It needs# torch, so the `signals_ml` location trains it and the main location never fits it. It declares# the report embeddings set, because that set builds its report input in training and at serving.-# `nn_with_prio` is the same network with the report's priority levels as extra report inputs.+# `nn_with_prio_v2` is the same network with the report's priority levels as extra report inputs.MODEL_FAMILIES: tuple[ModelFamily, ...] = (ModelFamily(name=REPORT_EMBEDDINGS_MODEL_NAME, feature_set=REPORT_EMBEDDINGS_FEATURE_SET),ModelFamily(