:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(signals): retire the report_embeddings_v2 inbox ranking family (#120687)
posthog[bot]4f63533d5145
diff --git a/docs/internal/inbox-ranking-training.md b/docs/internal/inbox-ranking-training.mdindex 00fff397c559..73be4d0cb4cb 100644--- a/docs/internal/inbox-ranking-training.md+++ b/docs/internal/inbox-ranking-training.md@@ -6,7 +6,7 @@ The artefact's team/report IDs, creation timestamp, and actor attribution identiSelecting the current priority adds no correction. These records are available for future priority learning; they do not introduce a new ranking label or training head.Research already running when a priority is edited preserves that correction; a later research run can reassess it.-The report-embedding ranking family uses one example per report per head, from the report's birth-day snapshot.+The `nn` ranking families use one example per report per head, from the report's birth-day snapshot.This prevents long-lived reports from receiving more weight in training because they appear in more daily snapshots.Birth days use the UTC interval returned by `snapshot_bounds`, including the start and excluding the end.The label comes from the snapshot `horizon_days` later, including outcomes already present on the birth day.@@ -15,11 +15,10 @@ The families train only the heads the inbox reads: `open`, `action` and `fixed`.The dataset still snapshots the label columns of other outcomes, so a head can come back without a labels backfill.Manual resolutions with reason `no_action_needed` count as reasoned resolutions for the `action` head but do not count toward the `fixed` head. They record that the report was accepted without claiming an implementation or a PR merge.-The examples asset records `reports_missing_birth_snapshot` for observed reports born inside the lookback whose birth-day partition is missing.+`reports_missing_birth_snapshot` counts the observed reports born inside the lookback whose birth-day partition is missing.These reports produce no birth-grain example; reports born before the lookback do not contribute to this count.A missing horizon snapshot also prevents an example because its label is unknown.Point-in-time state, label provenance, and embedding availability checks still apply.-Each embedding family reads its own rendering's snapshot, so a report missing from one snapshot is an example for the other family only.Every training series resets on the first partition after deploy.Before and after holdout numbers are not comparable because the example population changes.@@ -56,7 +55,7 @@ The scores stay calibrated to that population, and the holdout stays a clean timA recency cut drops old positives with old negatives, so the budget must stay large enough to keep the rare heads fed.Each head in `metadata.json` records `example_window_start`, the earliest report-creation day kept, and `example_cap_bound`, whether the budget dropped any day.-`inbox_ranking_examples_built` carries both fields, so a chart shows when the budget starts to bind.+`inbox_ranking_candidate_trained` carries both fields per head, so a chart shows when the budget starts to bind.## Promotion@@ -128,7 +127,7 @@ They answer two questions: of the reports a head flags, how many have the outcomA head predicts positive when `score >= classification_threshold`.A score equal to the threshold is a positive prediction.-The threshold is the positive rate of the rows the booster was fit on.+The threshold is the positive rate of the rows the model was fit on.It means "at least as likely as the average fitting example", not a 50% probability.- **Holdout:** the threshold is the positive rate of the train-only rows. It is fixed before the holdout outcomes are read.diff --git a/posthog/dags/locations/signals.py b/posthog/dags/locations/signals.pyindex 26a106e47fc1..639e9f53456f 100644--- a/posthog/dags/locations/signals.py+++ b/posthog/dags/locations/signals.py@@ -24,9 +24,6 @@ if is_inbox_ranking_registered():inbox_ranking_dataset.inbox_report_reviewers,inbox_ranking_dataset.inbox_user_report_interactions,inbox_ranking_dataset.inbox_project_vectors,- inbox_ranking_training.inbox_ranking_training_examples,- inbox_ranking_training.inbox_ranking_model_candidate,- inbox_ranking_training.inbox_ranking_model_champion,inbox_ranking_training.inbox_ranking_serving_manifest,inbox_ranking_served.inbox_ranking_served_scores,inbox_ranking_training.inbox_ranking_unseen_graded,diff --git a/products/signals/dags/inbox_ranking/AGENTS.md b/products/signals/dags/inbox_ranking/AGENTS.mdindex 7884c528ff58..2080127f4ad6 100644--- a/products/signals/dags/inbox_ranking/AGENTS.md+++ b/products/signals/dags/inbox_ranking/AGENTS.md@@ -13,23 +13,23 @@ Read `README.md` first for what the dataset is and how partitions behave. This f## Training dag specifics- `products/signals/backend/ranking/features.py` holds a `FeatureSet` per feature universe (it lives in the backend so the scoring sweep owns it and the dag imports it). The embedding sets are also the serving contract: the sweep builds each model's matrix through the set's own `build_matrix`, and refuses a booster whose `feature_names` or `feature_schema_version` disagree with its set. A change to a set's features means bumping that set's `schema_version`, and the sweep must be able to compute it at scoring time — nothing impression-derived, nothing that needs the label streams.-- A model declares its feature set in `metadata.json` and is checked against that set, not against one global contract. Adding a set means an entry in `FEATURE_SETS`; the examples asset then writes its Parquet under `inbox_ranking_training_examples/v1/<feature_set>/dt=D/`. Do not reintroduce a module-global feature list into the training path: three sets live side by side. A partition whose examples were written before the per-set layout has nothing under `<feature_set>/`, so re-run the examples asset for that day before re-running the candidate on it.+- A model declares its feature set in `metadata.json` and is checked against that set, not against one global contract. Adding a set means an entry in `FEATURE_SETS`. Do not reintroduce a module-global feature list into the training path: several sets live side by side.- A set that reads a side input declares it in `extras_keys`, and reports per-row availability through `buildable`. The example builder drops the rows `buildable` returns False for.-- An embedding set is a `ReportEmbeddingsFeatureSet` parameterized by the `extras` key it reads, one rendering each. A set reads its own key only: never add a fallback to another rendering's vector. `title_embeddings` is retired (`RETIRED_MODEL_NAMES`), so nothing trains on the `inbox_report_title_embeddings` snapshot. `embeddings_extras` loads one set's inputs at a time, because a snapshot carries a vector per live report; keep it that way, so a further embedding family costs runtime rather than peak memory.+- An embedding set is a `ReportEmbeddingsFeatureSet` parameterized by the `extras` key it reads, one rendering each. A set reads its own key only: never add a fallback to another rendering's vector. `title_embeddings` is retired (`RETIRED_MODEL_NAMES`), so nothing trains on the `inbox_report_title_embeddings` snapshot. A snapshot carries a vector per live report, so load one rendering at a time: a further embedding family must cost runtime rather than peak memory.- A **missing** side-input object leaves its key out, so only the sets reading it are skipped and their partitions stay as they are. An **empty** object keeps its key and takes the ordinary thin-input path. Do not collapse the two: the first must not write over a partition, and the second is a day the family really has nothing to fit.- Both `buildable` and `build_matrix` take the moment being built as `as_of`, and a side input that changes over a report's life must honor it. The report vector does: reports are re-embedded when their text changes, so the snapshot's latest vector often postdates the moment, and taking it would train on text that did not exist yet. Never widen a set's side input to the latest value for convenience.-- `example_grain` and `max_examples_per_head` are per set, because the examples object is. Every set defaults to the birth grain so the families target the same scoring moment; side-input availability and row budgets can still change their training populations. `report_embeddings` is the reason the budget exists (1536 columns per row). A budget limits history and never samples inside a day: `cap_examples` keeps whole report-creation days, newest first, so the kept label rate stays the population rate and the scores stay calibrated. Never reintroduce a positive-keeping sample: it raises the base rate the booster fits, and a holdout cut from the same sample hides it.+- `example_grain` and `max_examples_per_head` are per set, because the examples object is. Every set defaults to the birth grain so the families target the same scoring moment; side-input availability and row budgets can still change their training populations. `report_embeddings` is the reason the budget exists (1536 columns per row). A budget limits history and never samples inside a day: `cap_examples` keeps whole report-creation days, newest first, so the kept label rate stays the population rate and the scores stay calibrated. Never reintroduce a positive-keeping sample: it raises the base rate the model fits, and a holdout cut from the same sample hides it.- `feature_vector` (one row, serving) and `feature_frame` (vectorized, training) must agree row for row; a test pins it. Change both together.- Examples are one row per report, on the snapshot of the day it was created, labeled from the snapshot `horizon_days` later. The scoring-moment and report grains stay selectable for re-scoring families, so keep the holdout cut by report: a row-level split leaks near-duplicate snapshots of the same report the moment a set asks for one of them. The moments are chosen before the features are built, so a set under a row budget never builds columns for rows it then drops; keep that order.- `inbox_ranking_models/v1/<model_name>/dt=D/` is history like the dataset partitions: the only mutation is a re-run of the same partition, which replaces the prefix in full (stale head files are deleted). `inbox_ranking_models/v1/served.json` is the only object written outside a partition, and only `inbox_ranking_promote_job` writes it, when a person runs it. No asset may write `served.json` or a `<model_name>/champion.json`: the champion assets record a decision and nothing else. The old `champion.json` files stay as the cross-family role's input.- The model is read in three layers: **training** (train metrics), **test** (the random-by-report holdout, which drives promotion) and **online** (`inbox_ranking_served_scores`, the scoring sweep's live scores of each day's newborns, graded by `inbox_ranking_unseen_graded` at each head's horizon). Do not add an offline rescoring of the newborn pool beside the online layer: the sweep already scores it with every model key the manifest names.- `served_score_rows` keeps the earliest scored event per (report, model key) and takes `model_role` from the event's roles. Grades stay per `(model_name, model_version, model_role)` and are never pooled across versions. A re-run that reads no event is refused when the partition already holds rows, and so is one that reads fewer families than the object holds, because the newborn state snapshot behind those rows ages out and the dt=D+horizon grade reads them.- Every scored row carries `head_readable`, and an unreadable head is still graded: a head that misses `min_holdout_positives` on one day's holdout still gets a number from the pooled online grade. Keep the promotion gate reading `readable` from the candidate metadata: an unreadable head must not decide a promotion.-- `model_name` is the model recipe, and every model object, scored row, grade and event carries it. A family is an entry in `MODEL_FAMILIES` (`training/unseen.py`) naming the feature set it is fit on, its `learner` and the `location` that trains it; each location's candidate and champion assets walk `families_trained_in(<location>)`, and the grader and the events need no change to gain a family. A candidate is graded only against the served model of its own `model_name`; the served pointer itself can name any family. Do not fold the family into `model_version`, which the dashboard filters as a date.+- `model_name` is the model recipe, and every model object, scored row, grade and event carries it. A family is an entry in `MODEL_FAMILIES` (`training/unseen.py`) naming the feature set it is fit on, its `learner` and the `location` that trains it; the `signals_ml` location trains every family (`nn/dag.py`), and the grader and the events need no change to gain a family. `families_trained_in(MAIN_LOCATION)` is empty: the main location fits no model. The per-head XGBoost families are retired (`RETIRED_MODEL_NAMES`), and the sweep keeps the `xgboost` handler so an old version stays servable by key for a rollback. A candidate is graded only against the served model of its own `model_name`; the served pointer itself can name any family. Do not fold the family into `model_version`, which the dashboard filters as a date.- A recipe change (features, heads, horizons, learner or training settings, `DATASET_VERSION`) needs a new `model_name`, such as `nn_v3`, because the promotion gate grades a candidate only against the served model of its own name. Every candidate writes `recipe` and `recipe_hash` (`training/recipe.py`), and the candidate asset fails before it writes when the hash differs from the newest hashed version of the name at or before its partition, a re-run of the same partition included. Keep anything that changes daily (dates, `run_id`, counts, metrics) out of the recipe, or every retrain fails. Do not add an override for the check: the fix is a new name.- A family whose `metadata.json`, examples object or side input is missing for a partition is logged and skipped, so one family does not cost every family's series. The champion asset raises only when no family has a candidate at all.- Skipping is not the same as rebuilding from nothing, and the difference is destructive. An empty examples object produces an empty candidate, and the candidate's prefix cleanup then deletes the boosters that partition already holds, which `served.json` or a pin can be pointing at. So when a side input is unavailable, leave the partition's objects alone rather than writing empty ones over them.-- The champion is compared to a candidate on the candidate's holdout, through the champion's `<head>.holdout.ubj` (the train-only fit). Keep writing that file: without it the gate falls back to the champion's stored AUC, which was measured on a different set of reports.+- The champion is compared to a candidate on the candidate's holdout, through the champion's train-only fit (`model.holdout.onnx`). Keep writing that file: without it the gate falls back to the champion's stored AUC, which was measured on a different set of reports.- The example builder reads labels aligned to the state spine (`assemble_snapshot`): no label row means all-zero labels, not "absent". It drops rows whose `features_observed_at` is a backfill (`STATE_LAG_LIMIT`) and, for status-derived heads, rows that fail `label_provenance_ok`.- The serving path reads the deployment's own object store, never the dataset bucket: the Temporal workers have no credential for that bucket, and keeping it that way is why the manifest asset copies what the sweep needs. `serving/models/<key>/` holds a version's `metadata.json` and its model files, `serving/manifest.json` names the models one pass scores. The holdout boosters grade a promotion and must stay behind.- An entry's files come from `ServingManifestEntry.files`: one `<head>.ubj` per head for `xgboost`, and the one `model.onnx` that `model_file` names for `onnx`. The copy step and the model store read that property, and the pin check reads the same names from each head's `file` in `metadata.json`, so a new model kind with another file layout keeps the two in step. Keep `onnxruntime` and `xgboost` imports inside the handlers' `load`: a test pins that importing the sweep loads neither.@@ -38,7 +38,7 @@ Read `README.md` first for what the dataset is and how partitions behave. This f- Publishing order is load-bearing. Every model copy finishes before `manifest.json` is written, so a failed copy leaves the previous manifest and the previous models serving. A model version is immutable, so a version whose `metadata.json` is already at the target is not copied again. A run with nothing to publish, such as a run without `served.json`, writes nothing rather than an empty manifest. The served model passes the same check as a pin (`_pin_metadata`) before composition.- When a manifest entry has `reads_project_vectors`, the dt=D `inbox_project_vectors` day is copied to `serving/project_vectors/<D>/` before `manifest.json`, and the manifest names it in `project_vectors`. A missing day fails the publish, like a missing model file. Never copy project vectors to the mirror: they are per-team data of this deployment, so the mirror's manifest names none. Training and serving build the `project` and `has_history` inputs through the one `project_inputs` in `features.py`; keep `PROJECT_VECTOR_KINDS` and `MIN_PROJECT_HISTORY_REPORTS` there.- `HEADS` holds only the heads the inbox reads (`HeadRanker` in `backend/ranking/inbox_sort.py`). Add a head there only together with a surface that reads it. The promotion gate skips a champion head that is not in `HEADS`, so a trim does not block promotion.-- `nn_v3` and `nn_with_prio_v2` are trained by the `signals_ml` location (`nn/dag.py`), never by the main one. They share one `inbox_ranking_nn_examples` object, and each has its own candidate and champion asset from `family_assets`. The `nn_v3` assets keep the names `inbox_ranking_nn_model_{candidate,champion}`, so their Dagster history continues. A later family's candidate depends on the previous family's champion, for memory and not for data: two candidates at once can exceed the job's memory limit. A family whose set reads a state column the examples object lacks skips with a warning rather than training on zeros. The main examples asset builds only the sets of `families_trained_in(MAIN_LOCATION)`, so registering a set for `signals_ml` does not add a 1536-column object to the main job. torch is imported only inside the candidate asset's body (through `nn/train.py` and `multihead/network.py`). `signals_ml` must import without torch, and `test_multihead_network.py` pins that. Its decision goes through `decide_family_champion`, the same rule as the boosters, graded through the incumbent's `model.holdout.onnx`. Keep `signals_ml` commented out in `.dagster_home/workspace.yaml`, and keep its schedule (04:30 UTC) running by default on prod US only.+- `nn_v3` and `nn_with_prio_v2` are trained by the `signals_ml` location (`nn/dag.py`), never by the main one. They share one `inbox_ranking_nn_examples` object, and each has its own candidate and champion asset from `family_assets`. The `nn_v3` assets keep the names `inbox_ranking_nn_model_{candidate,champion}`, so their Dagster history continues. A later family's candidate depends on the previous family's champion, for memory and not for data: two candidates at once can exceed the job's memory limit. A family whose set reads a state column the examples object lacks skips with a warning rather than training on zeros. torch is imported only inside the candidate asset's body (through `nn/train.py` and `multihead/network.py`). `signals_ml` must import without torch, and `test_multihead_network.py` pins that. Its decision goes through `decide_family_champion` in `training/dag.py`, graded through the incumbent's `model.holdout.onnx`. Keep `signals_ml` commented out in `.dagster_home/workspace.yaml`, and keep its schedule (04:30 UTC) running by default on prod US only.## Invariants — do not breakdiff --git a/products/signals/dags/inbox_ranking/README.md b/products/signals/dags/inbox_ranking/README.mdindex 556625803e96..b9f80d82018c 100644--- a/products/signals/dags/inbox_ranking/README.md+++ b/products/signals/dags/inbox_ranking/README.md@@ -1,6 +1,6 @@# Inbox ranking Dagster dags-Dagster jobs for the Self-driving Inbox report-ranking model: the **dataset** dag (daily snapshots), and the **training** dag (daily per-head XGBoost candidates, champion pointer, serving manifest and online grades), sibling subpackages sharing `common.py`.+Dagster jobs for the Self-driving Inbox report-ranking model: the **dataset** dag (daily snapshots), the **training** dag (serving manifest and online grades), and the **nn** dag (the daily candidates), sibling subpackages sharing `common.py`.```textinbox_ranking/@@ -10,11 +10,11 @@ inbox_ranking/│ ├── queries.py # HogQL label SQL, embeddings SQL, stream merging│ └── project_vectors.py # per-team context vectors (pure function)├── training/-│ ├── dag.py # examples → candidate → champion assets + job + schedule+│ ├── dag.py # serving manifest → served scores → unseen grade assets + job + schedule, and the plumbing nn shares│ ├── promote.py # the manual promote job that writes served.json│ ├── examples.py # birth-grain training examples over the snapshots│ ├── heads.py # the v0 outcome heads-│ ├── train.py # per-head XGBoost fit + holdout/null metrics+│ ├── train.py # per-head holdout/null metrics that every learner fills│ └── promotion.py # the candidate grading rule├── multihead/ # the multi-head network, its ONNX export, and the offline prototype├── nn/ # the `nn` family: examples → candidate → champion assets, on the `signals_ml` location@@ -137,36 +137,40 @@ Point-in-time caveats, per source:## The training dag-`inbox_ranking_training_job` runs daily at 05:00 UTC on the same partition definition (gated like the dataset job) and writes:+`inbox_ranking_training_job` runs daily at 05:00 UTC on the same partition definition (gated like the dataset job).+It fits no model: the `nn` families train on the `signals_ml` location (see [The nn family](#the-nn-family)).+The job runs three assets: `inbox_ranking_serving_manifest`, `inbox_ranking_served_scores` and `inbox_ranking_unseen_graded`.+The manifest depends on the dt=D `inbox_project_vectors` partition, which it copies.+The models live under the dataset prefix:```texts3://<bucket>/<prefix>/-├── inbox_ranking_training_examples/v1/<feature_set>/dt=YYYY-MM-DD/ # one row per report at its birth, all heads, one parquet per feature set├── inbox_ranking_models/v1/│ ├── served.json # the model key the scoring sweep serves; only the promote job writes it│ └── <model_name>/-│ ├── dt=YYYY-MM-DD/<head>.ubj + <head>.holdout.ubj # the day's candidate: serving fit + train-only fit-│ │ + metadata.json # (a re-run replaces this prefix in full)+│ ├── dt=YYYY-MM-DD/<model files> + metadata.json # the day's candidate (a re-run replaces this prefix in full)│ └── champion.json # retired automatic pointer; read only for the cross-family role└── inbox_ranking_served_scores/v1/dt=YYYY-MM-DD/ # the scoring sweep's scores of the reports born that day, per model key```-- **A model is a `(model_name, model_version, model_role)`.** `model_name` is the recipe: which features, which learner and which training settings: `report_embeddings_v2`, per-head XGBoost, is the one trained here. `nn_v3`, one ONNX network, and `nn_with_prio_v2`, the same network with the report's priority as an input, are trained by the `signals_ml` location, and each `MODEL_FAMILIES` entry names its `learner` and the `location` that trains it. `model_version` is the partition day it was fit on, and `model_role` is the role it was scored or trained under (`candidate` on the training events; `served`, `recent`, `challenger`, `pinned` or `cross_family` on the online grade, and `daily_candidate` on grades written before the `recent` role). Each family owns a prefix under the models path, so two families trained on the same day cannot collide. A candidate is graded only against the served model of its own name. The served pointer can name any family, so a rollback can cross families.+The per-head XGBoost families are retired (`RETIRED_MODEL_NAMES`): `tabular_xgb`, `title_embeddings` and `report_embeddings_v2`.+Their saved versions (`<head>.ubj` + `<head>.holdout.ubj`) stay in the bucket, and the sweep can still serve one by key, so a rollback to one of them is possible.+The grader drops the online rows of a retired family.++- **A model is a `(model_name, model_version, model_role)`.** `model_name` is the recipe: which features, which learner and which training settings: `nn_v3`, one ONNX network, and `nn_with_prio_v2`, the same network with the report's priority as an input, are trained by the `signals_ml` location, and each `MODEL_FAMILIES` entry names its `learner` and the `location` that trains it. `model_version` is the partition day it was fit on, and `model_role` is the role it was scored or trained under (`candidate` on the training events; `served`, `recent`, `challenger`, `pinned` or `cross_family` on the online grade, and `daily_candidate` on grades written before the `recent` role). Each family owns a prefix under the models path, so two families trained on the same day cannot collide. A candidate is graded only against the served model of its own name. The served pointer can name any family, so a rollback can cross families.- **A recipe change gets a new `model_name`.** `report_embeddings` and `nn` became `report_embeddings_v2` and `nn_v2` when `action` started to count a thumbs up, and `nn_v2` and `nn_with_prio` became `nn_v3` and `nn_with_prio_v2` when the lookback went from 60 to 90 days and the row budget from 100,000 to 250,000. A candidate's `metadata.json` carries `recipe` and `recipe_hash`, a stable hash of the recipe (`training/recipe.py`): the learner, the feature set and its schema version, the project-vector kinds and their history settings, the booster params or network config, each head's cohort, label, horizon, label columns and status-label flag, `lookback_days`, `holdout_days`, the example grain and row budget, and `dataset_version`. Dates, `run_id`, row counts and metrics stay out, so a daily retrain keeps the hash. The candidate asset compares its hash with the newest version of the same name, at or before its partition, that has one, and on a mismatch it fails before it writes the partition and names the changed inputs. A re-run of a partition therefore compares with that partition's own saved recipe. The search covers every dataset version, so a `DATASET_VERSION` bump also needs a new name. A version without a hash is not compared. This includes an env change such as `INBOX_RANKING_TRAINING_LOOKBACK_DAYS`. To start a new recipe, add a `MODEL_FAMILIES` entry with a new name (`nn_v2` after `nn`) and remove the old entry: the old versions stay servable by key, and the promotion gate never compares two recipes. Two recipes can train side by side for a comparison, but each costs a manifest slot, so do not do it by default. `recipe_hash` is on `inbox_ranking_candidate_trained` and `inbox_ranking_promotion_decided` (with `incumbent_recipe_hash`), and `inbox_ranking_serving_manifest_published` carries `model_recipe_hashes` per model key.-- **A feature set is one feature universe.** `products/signals/backend/ranking/features.py` holds a `FeatureSet` per universe: its name, its schema version, its ordered feature names, the report-state columns it reads, `build_matrix`, and how many rows it wants per report and per head. `report_embeddings` is the report vector, and `report_embeddings_priority` is that vector plus one column per priority level (`priority_P0` to `priority_P4`, all zero when the report has no judgment), read from the `priority` state column. A candidate records the set it was fit on in its `metadata.json`, so the grader checks a model against its own set rather than one global contract, the examples asset writes one Parquet per set, and the unseen scorer builds one matrix per set and shares it across the families that read it. Adding a set means an entry in `FEATURE_SETS`; a model naming a set this build cannot produce is logged and left unscored. The main examples asset writes a Parquet only for the sets of the families the main location trains, so a set only `signals_ml` reads costs it nothing.-- **Examples are one row per report at its birth, by default.** For partition `dt=D` the examples asset reads the report-state and labels snapshots `dt=D-lookback..D` and emits one row per report, on the snapshot of the day it was created, labeled from the snapshot `horizon_days` later (7 for open, 14 for action, 30 for fixed). Features are built by the feature set from that snapshot's state columns plus `age_hours`, the report's age at the snapshot. Labels are aligned to the state spine, so a report with no label event is a negative (all-zero labels), not absent. Label-only rows (no Postgres state) are skipped, as are state rows read long after their snapshot day (backfills carry current Postgres state; see `features_observed_at`) and, for the heads that read the status stream (`action` and `fixed`), rows whose status telemetry fails the dataset's `label_provenance_ok` check. Birth is the moment serving scores a report, and the label that comes out is a per-report probability, which is what the inbox ordering needs. A report born before the window, or one whose birth-day snapshot is missing from it, is no example; `reports_missing_birth_snapshot` on the asset counts the second case, which is a gap in the partitions.-- **An outcome already visible on the birth day is a future positive.** A report born on day D has no scoring moment before D, so an outcome visible at D belongs to the moment being built rather than to an earlier one; the labels of D therefore read as their defaults for a report born on D. Most outcomes land on the birth day, so this is where the positives are: censoring on them costs the `pr_created` head most of its training signal. `<set>_<head>_birth_day_positives` on the examples asset, and `birth_day_positives` on the `inbox_ranking_examples_built` event, say how many positives the rule keeps. These rows carry a hindsight the other rows do not: the state snapshot reads `signal_count`, `total_weight`, `run_count` and the text sizes live from Postgres a few hours after day D ends, so a birth-day outcome happened before its own feature read. Both the holdout AUC and the newborn unseen grade therefore read optimistically on these rows, until the scoring sweep's timestamped score log replaces the daily snapshot as this table's source.+- **A feature set is one feature universe.** `products/signals/backend/ranking/features.py` holds a `FeatureSet` per universe: its name, its schema version, its ordered feature names, the report-state columns it reads, `build_matrix`, and how many rows it wants per report and per head. `report_embeddings` is the report vector, and `report_embeddings_priority` is that vector plus one column per priority level (`priority_P0` to `priority_P4`, all zero when the report has no judgment), read from the `priority` state column. A candidate records the set it was fit on in its `metadata.json`, so the grader checks a model against its own set rather than one global contract, and the scoring sweep builds one matrix per set and shares it across the families that read it. Adding a set means an entry in `FEATURE_SETS`; a model naming a set this build cannot produce is logged and left unscored.+- **Examples are one row per report at its birth, by default.** For partition `dt=D` the `nn` examples asset reads the report-state and labels snapshots `dt=D-lookback..D` and emits one row per report, on the snapshot of the day it was created, labeled from the snapshot `horizon_days` later (7 for open, 14 for action, 30 for fixed). Features are built by the feature set from that snapshot's state columns plus `age_hours`, the report's age at the snapshot. Labels are aligned to the state spine, so a report with no label event is a negative (all-zero labels), not absent. Label-only rows (no Postgres state) are skipped, as are state rows read long after their snapshot day (backfills carry current Postgres state; see `features_observed_at`) and, for the heads that read the status stream (`action` and `fixed`), rows whose status telemetry fails the dataset's `label_provenance_ok` check. Birth is the moment serving scores a report, and the label that comes out is a per-report probability, which is what the inbox ordering needs. A report born before the window, or one whose birth-day snapshot is missing from it, is no example; `reports_missing_birth_snapshot` counts the second case, which is a gap in the partitions.+- **An outcome already visible on the birth day is a future positive.** A report born on day D has no scoring moment before D, so an outcome visible at D belongs to the moment being built rather than to an earlier one; the labels of D therefore read as their defaults for a report born on D. Most outcomes land on the birth day, so this is where the positives are: censoring on them costs the `pr_created` head most of its training signal. `birth_day_positives` on the unseen grade events says how many positives landed on the birth day. These rows carry a hindsight the other rows do not: the state snapshot reads `signal_count`, `total_weight`, `run_count` and the text sizes live from Postgres a few hours after day D ends, so a birth-day outcome happened before its own feature read. Both the holdout AUC and the newborn unseen grade therefore read optimistically on these rows, until the scoring sweep's timestamped score log replaces the daily snapshot as this table's source.- **Every head asks whether the outcome happened within N days of the scoring moment.**- **A set may ask for another grain**, and cap the rows one head keeps (`max_examples_per_head`, whole report-creation days, newest first). The scoring-moment grain is one row per (report, snapshot) whose head label is still 0 on that snapshot: the serving situation replayed over history, but its label is a hazard conditional on the report still being live, and one long-lived report contributes dozens of near-duplicate rows. The report grain is the first snapshot of the window where the report is a usable moment, which is later than birth only when a side input landed late. Both stay selectable for re-scoring families; nothing ships on them today. The row budget shortens the history a head is fit on and keeps each kept day whole, so the scores stay calibrated to the population of that window. The lookback is the training job's whatever the grain, so positives accrue over the whole window until the budget binds.- **The embedding sets are the serving contract.** Each reads one rendering's vector and nothing else, so the scoring sweep needs only the report's latest vector for that rendering. The sweep must build features through the same module; the booster's `feature_names` are checked against it at load. A model whose metadata names no set is left unscored.- **`report_embeddings` is the report's own vector and nothing else**, read from the dt=D `inbox_report_embeddings` snapshot as a side input. Age is deliberately left out: it is the whole signal of the `recency_auc` line the unseen read already reports, so a gap to that line is content rather than recency. The scoring sweep does not serve this set: it is an offline candidate, and serving it needs the report vector at scoring time.- **A moment only takes a vector that already existed for it.** The snapshot holds the latest vector per report, and a report is re-embedded whenever its text changes (the summary workflow and every re-research run rewrite it), so the latest vector often postdates an earlier moment. Each row carries `embedding_inserted_at`, and a moment keeps the vector only when it landed at or before that snapshot's end; at the birth grain a report whose birth-day snapshot holds only a later vector is no example, and at the report grain it moves to the first snapshot where the vector was already its own. Without that check the family would train on text that did not exist when the report was supposedly scored. A report the snapshot has no vector for at all is not an example either, because the source table's TTL runs from report creation and a long-lived report loses its vector while still live. The scores asset records each set's coverage of the newborn pool, so a thin side input is visible rather than silent.-- **A set whose side input is unavailable is skipped, not rebuilt from nothing.** The examples asset writes no object for it, the candidate asset leaves that family's partition as it stands, and the unseen scorer drops its models for the day. Rebuilding would write an empty examples object, then an empty candidate, and the candidate's prefix cleanup would delete boosters a champion pointer can name.-- **Missing and empty are different inputs.** The case above is a snapshot object that is absent. A snapshot that exists and carries no usable vector keeps its key, so the family builds no row and the ordinary thin-input path applies.-- **One rendering's vectors are held at a time.** The examples asset loads a set's side inputs, builds that set's object, and releases them before the next set. Each snapshot carries a vector per live report, so a further embedding family costs runtime rather than peak memory. The scores asset narrows each snapshot to the newborn pool it scores for the same reason. Raising the pod memory limit is not the first move here.-- **Candidate**: one per family, per-head XGBoost with fixed params, holdout = the last `holdout_days` of reports (cut by report, never by row), AUC + a label-permutation null. Each family trains on the examples of the feature set its registry entry names, and a family whose examples or metadata are missing that day is logged and skipped rather than failing the asset. A head is _readable_ when it has enough holdout positives and clears its null by 0.05. The shipped booster (`<head>.ubj`) is refit on everything; the train-only fit is kept as `<head>.holdout.ubj` so a later candidate can grade this model on its own holdout.-- **Metrics telemetry**: each asset also captures its metrics as events into the dogfood project (the same project the label events land in), through `training/telemetry.py`: `inbox_ranking_examples_built` once per head and feature set, `inbox_ranking_candidate_trained` once per head (`head`, `model_role` always `candidate`, holdout / train AUC, average precision, logloss, positive rate, mean predicted score, the row-weighted decile calibration error, the permutation-null mean and spread, counts, `readable`), `inbox_ranking_holdout_calibration` once per head and holdout score decile `inbox_ranking_promotion_decided` once per run and `inbox_ranking_serving_manifest_published` once per run, all carrying `model_name` and `model_version` and stamped midday UTC on the partition day so re-runs and backfills chart on the day they describe. Per-head stability is a trends insight with a `head` breakdown; a readability drop is an insight alert. `metadata.json` stays the durable record. Capture is best-effort. Local dev runs (`DEBUG`) emit too, under `distinct_id` `inbox_ranking_training_local` with `environment=local`, so filter or break down on `environment` when reading the prod series; any other non-Cloud deployment emits nothing.-- **Champion**: one decision per family, recorded and never written to a pointer. The incumbent is the model `served.json` names when it has the candidate's `model_name`; otherwise there is no paired comparison, and the reason says so. `promotion.decide_promotion` says the candidate would promote when it has a readable head and is within 0.02 AUC and 0.02 holdout ECE of the incumbent on every head the incumbent could read. `would_promote` and `gates_passed` on the `inbox_ranking_promotion_decided` event and the asset metadata carry the decision, and `promoted` is always false. The incumbent's AUCs and ECEs come from its `<head>.holdout.ubj` scored on the candidate's holdout (`paired_champion_grades`), so both models are compared on one set of reports; an incumbent without that file falls back to its stored AUC and skips the calibration check. A head whose shared holdout has fewer than its `min_holdout_positives` is skipped rather than blocking, and is listed in `skipped_heads`; a head the candidate did not train still blocks.+- **A missing side input is a skip, not a rebuild from nothing.** A rebuild would write an empty candidate, and the candidate's prefix cleanup would delete model files that a pointer can name. See [The nn family](#the-nn-family).+- **Candidate**: one per family, holdout = the last `holdout_days` of reports (cut by report, never by row), AUC + a label-permutation null. A head is _readable_ when it has enough holdout positives and clears its null by 0.05. The shipped model is refit on everything, and the train-only fit is kept so a later candidate can grade this model on its own holdout.+- **Metrics telemetry**: each asset also captures its metrics as events into the dogfood project (the same project the label events land in), through `training/telemetry.py`: `inbox_ranking_candidate_trained` once per head (`head`, `model_role` always `candidate`, holdout / train AUC, average precision, logloss, positive rate, mean predicted score, the row-weighted decile calibration error, the permutation-null mean and spread, counts, `readable`), `inbox_ranking_holdout_calibration` once per head and holdout score decile `inbox_ranking_promotion_decided` once per run and `inbox_ranking_serving_manifest_published` once per run, all carrying `model_name` and `model_version` and stamped midday UTC on the partition day so re-runs and backfills chart on the day they describe. Per-head stability is a trends insight with a `head` breakdown; a readability drop is an insight alert. `metadata.json` stays the durable record. Capture is best-effort. Local dev runs (`DEBUG`) emit too, under `distinct_id` `inbox_ranking_training_local` with `environment=local`, so filter or break down on `environment` when reading the prod series; any other non-Cloud deployment emits nothing.+- **Champion**: one decision per family, recorded and never written to a pointer. The incumbent is the model `served.json` names when it has the candidate's `model_name`; otherwise there is no paired comparison, and the reason says so. `promotion.decide_promotion` says the candidate would promote when it has a readable head and is within 0.02 AUC and 0.02 holdout ECE of the incumbent on every head the incumbent could read. `would_promote` and `gates_passed` on the `inbox_ranking_promotion_decided` event and the asset metadata carry the decision, and `promoted` is always false. The incumbent's AUCs and ECEs come from its train-only fit scored on the candidate's holdout, so both models are compared on one set of reports; an incumbent without that fit falls back to its stored AUC and skips the calibration check. A head whose shared holdout has fewer than its `min_holdout_positives` is skipped rather than blocking, and is listed in `skipped_heads`; a head the candidate did not train still blocks.- **Promotion is manual.** `inbox_ranking_promote_job` (`training/promote.py`, `signals` location) takes a `model_key` and a `reason`. It refuses a malformed key, a model this build cannot serve, a missing model file, and a model without every head in `HEADS`, and then writes nothing. Otherwise it writes `served.json` (`model_key`, `promoted_at`, `reason`, `previous_key`, `run_id`), captures `inbox_ranking_model_promoted`, and publishes the serving manifest for the newest partition with project vectors, so the sweep serves the model without waiting for the 05:00 UTC run. Rollback is the same job with an older key, of any family. Launch it from the Dagster UI, or:```bash@@ -266,7 +270,7 @@ See [training example semantics](../../../../docs/internal/inbox-ranking-trainin### The online readThe model is read in three layers: **training** (train metrics), **test** (the random-by-report holdout, which drives promotion) and **online** (the scoring sweep's live scores, graded at each head's horizon).-The holdout grades the recipe, not the model that ships: the shipped booster is refit on train plus holdout, and every holdout row comes from the same snapshots the trainer saw.+The holdout grades the recipe, not the model that ships: the shipped model is refit on train plus holdout, and every holdout row comes from the same snapshots the trainer saw.Two assets add the missing number, from the scores the sweep actually wrote for reports no model trained on.- **`inbox_ranking_served_scores` (dt=D)** (`training/served.py`) takes the dt=D state rows of the reports **created on D**, the pool the rows tag `pool=newborn`. A newborn has no scoring moment before D, and the dt=D examples stop at D minus the head's horizon (7 days at the shortest), so no example can cover it. It reads this deployment's `inbox_ranking_report_scored` events for that pool inside D and keeps the earliest `scored` event per (report, model key), so every model the manifest named is graded: the served model, the served family's `recent` versions, the `pinned` models, and the other families' `cross_family` champions. `model_role` comes from the event's roles: `served` when they hold it, else the first role, else `candidate`. The sweep scores a candidate only after a manifest names it, so a candidate's online cohort starts the day after it was trained, and it stays a `recent` entry until N newer versions replace it. Every head a model fit is scored, readable or not, and each row carries `head_readable` and the model's own `classification_threshold`. One Parquet row per (report, model key, head) carries the score, `age_hours`, `label_at_scoring` (the head outcome that had already happened when the report was scored, which for a newborn is one that landed on its own birth day, so the grade keeps it; a status-label head's rows are still dropped on it, because no scores column carries the scoring-day `label_provenance_ok` that the example builder checks), `model_name`, `model_role`, and `pool`. `served_pool_coverage` and `pool_coverage_by_model` say how much of the pool each model covered. A re-run that reads no event is refused when the partition already holds rows, and so is one that reads fewer families than the object already holds. Rows of a retired family (`RETIRED_MODEL_NAMES`) are dropped on read, and the rewrite guard does not require them.@@ -289,7 +293,7 @@ Compare model families on the within-team AUC when they differ in team-level inp### Running the training job locally-The training job is S3-only, so it can run on a laptop against copies of the prod snapshots. The dataset job cannot: it needs the dogfood project's ClickHouse and cross-region Postgres.+The `nn` training job is S3-only, so it can run on a laptop against copies of the prod snapshots. The dataset job cannot: it needs the dogfood project's ClickHouse and cross-region Postgres.1. With the dev stack up (`bin/start`; the script waits for object storage to accept requests), sync the two prefixes the job reads into the local object-storage bucket. This needs an SSO session with the `secrets-editor` role on `prod-us-secrets`, and the Secrets Manager id of the dataset reader credential (provisioned with the bucket; ask the owning team):@@ -298,26 +302,21 @@ The training job is S3-only, so it can run on a laptop against copies of the proINBOX_RANKING_READER_SECRET_ID=<secret id> products/signals/dags/inbox_ranking/bin/sync_snapshots_local.sh```- This copies `inbox_report_state/v1/dt=*` and `inbox_report_labels/v1/dt=*` to `~/.cache/posthog/inbox_ranking/` and from there into `s3://posthog/inbox_ranking/` on SeaweedFS (`localhost:19000`). `INBOX_RANKING_SYNC_EMBEDDINGS=1` adds `inbox_report_embeddings`, which the `report_embeddings` family needs; it is off by default because that table carries a vector per live report per partition. Without it the job still runs, and that family builds no examples. `INBOX_RANKING_SYNC_TITLE_EMBEDDINGS=1` adds `inbox_report_title_embeddings` on its own switch and for the same reason. No family trains on it. It prints the partition days present in both tables; pick the **newest** of those as the partition to run. The examples asset reads the `INBOX_RANKING_TRAINING_LOOKBACK_DAYS` snapshots behind the partition it runs for, so an early day has little history behind it and trains on almost nothing. Re-runs only move new days.+ This copies `inbox_report_state/v1/dt=*` and `inbox_report_labels/v1/dt=*` to `~/.cache/posthog/inbox_ranking/` and from there into `s3://posthog/inbox_ranking/` on SeaweedFS (`localhost:19000`). `INBOX_RANKING_SYNC_EMBEDDINGS=1` adds `inbox_report_embeddings`, which the `nn` families need; it is off by default because that table carries a vector per live report per partition. Without it the `nn` assets skip the partition. `INBOX_RANKING_SYNC_TITLE_EMBEDDINGS=1` adds `inbox_report_title_embeddings` on its own switch and for the same reason. No family trains on it. It prints the partition days present in both tables; pick the **newest** of those as the partition to run. The `nn` examples asset reads the `INBOX_RANKING_TRAINING_LOOKBACK_DAYS` snapshots behind the partition it runs for, so an early day has little history behind it and trains on almost nothing. Re-runs only move new days.Note that the dev stack's object storage accepts unsigned requests and publishes port 19000 on every host interface, so anything synced here (and the disk cache) is readable by whoever can reach your machine. The snapshots are internal dogfood telemetry, not customer data, but run the sync on a network you trust.2. Make sure `INBOX_RANKING_DATASET_S3_BUCKET` is unset for the Dagster process: `common.s3_client()` then talks to SeaweedFS and `dataset_bucket()` is `posthog`. Checking your shell (`env | grep INBOX_RANKING`) is not enough, because `bin/start` sources `.env.local` into the processes it starts; check that file too. If the variable reaches Dagster, the dag uses your ambient AWS credentials against that bucket, and the read-only credential the sync used protects nothing.-3. `bin/start` runs `dagster dev` on http://localhost:3030 with the signals location loaded; materialize `inbox_ranking_training_job` for that partition from the UI, or from the CLI:-- ```bash- dagster job launch -w .dagster_home/workspace.yaml --location posthog.dags.locations.signals \- -j inbox_ranking_training_job --tags '{"dagster/partition": "2026-08-25"}'- ```+3. Launch `inbox_ranking_nn_training_job` for that partition as [The nn family](#the-nn-family) describes.The schedule is stopped outside prod US, so nothing runs unasked. If the run sits in `QUEUED`, check the daemon log for `Maximum is 10, won't launch more`: runs from a killed `dagster dev` stay `STARTED` forever and count against the local queue. Terminate them from the Runs page (force termination).-4. Read the result from `s3://posthog/inbox_ranking/inbox_ranking_models/v1/report_embeddings_v2/dt=<day>/metadata.json` (per-head AUCs, readability); the examples are at `s3://posthog/inbox_ranking/inbox_ranking_training_examples/v1/report_embeddings/dt=<day>/part-00000.parquet`:+4. Read the result from `s3://posthog/inbox_ranking/inbox_ranking_models/v1/nn_v3/dt=<day>/metadata.json` (per-head AUCs, readability):```bashAWS_ACCESS_KEY_ID=object_storage_root_user AWS_SECRET_ACCESS_KEY=object_storage_root_password \- aws --endpoint-url http://localhost:19000 s3 cp s3://posthog/inbox_ranking/inbox_ranking_models/v1/report_embeddings_v2/dt=2026-08-25/metadata.json -+ aws --endpoint-url http://localhost:19000 s3 cp s3://posthog/inbox_ranking/inbox_ranking_models/v1/nn_v3/dt=2026-08-25/metadata.json -```Nothing here touches the prod bucket: the reader credential is read-only and the dag writes only to the local bucket.@@ -347,7 +346,7 @@ The network leaves out every report that is in any head's holdout, so the compar### The nn family-`nn` is one multi-head network (`multihead/network.py`) over the report vector and the team's `typical`, `open`, `action` and `dismiss` project vectors, with the heads `open`, `action` and `fixed`. It is a model family like the per-head ones: it has its own prefix, the main job's serving manifest names a served, challenger or cross-family `nn` model as an `onnx` entry, and the online grader grades it. The main location's candidate and champion assets never fit it.+`nn` is one multi-head network (`multihead/network.py`) over the report vector and the team's `typical`, `open`, `action` and `dismiss` project vectors, with the heads `open`, `action` and `fixed`. It is a model family like the per-head ones: it has its own prefix, the main job's serving manifest names a served, challenger or cross-family `nn` model as an `onnx` entry, and the online grader grades it. The main location fits no model.Two `nn` families train every day. `nn_v3` reads the `report_embeddings` set. `nn_with_prio_v2` reads `report_embeddings_priority`: its `report` input carries the 1536 vector components and then the five priority levels, and those five columns skip the shared projection and go straight into the trunk input. Neither is served by default; a person serves one with `inbox_ranking_promote_job` after comparing their online grades.@@ -363,7 +362,7 @@ s3://<bucket>/<prefix>/```- **Examples** are the prototype's: each head's `example_moments` over the `report_embeddings` set, capped by `cap_examples` and split by `holdout_mask`, so every head's rows and holdout are the ones its `report_embeddings` champion is graded on. The object holds no vector, and one object serves both families. Each row carries its `priority` from its own snapshot. The candidate and champion assets rebuild the report vectors and the project vectors from the dt=D embeddings and labels snapshots. An object written before the `priority` column existed makes the `nn_with_prio_v2` candidate skip with a warning; re-run the examples asset for that day.-- **Candidate**: the network trains on the rows in no head's holdout and is graded on each head's own holdout through the same `head_metrics` the boosters use, so `metadata.json` carries the same per-head fields (pooled and within-team AUC, average precision, ECE, readability, row counts, example window). Readability uses the boosters' permutation null: three networks fit on permuted labels. `model.onnx` is a refit on every labeled row, and `model.holdout.onnx` is the train-only fit. `metadata.json` has `model_kind: onnx`, the family's `feature_set`, and `output_heads`, the graph's column order. The `nn_with_prio_v2` candidate runs after the `nn_v3` champion, for memory and not for data: each candidate holds an embeddings snapshot and the project vectors.+- **Candidate**: the network trains on the rows in no head's holdout and is graded on each head's own holdout through the shared `head_metrics`, so `metadata.json` carries the same per-head fields as the retired booster families (pooled and within-team AUC, average precision, ECE, readability, row counts, example window). Readability uses a permutation null: three networks fit on permuted labels. `model.onnx` is a refit on every labeled row, and `model.holdout.onnx` is the train-only fit. `metadata.json` has `model_kind: onnx`, the family's `feature_set`, and `output_heads`, the graph's column order. The `nn_with_prio_v2` candidate runs after the `nn_v3` champion, for memory and not for data: each candidate holds an embeddings snapshot and the project vectors.- **Champion**: the same rule as the other families (`decide_family_champion`), recorded and never written to a pointer. The incumbent is the served model when it has the candidate's `model_name`, graded through its `model.holdout.onnx` on the candidate's holdout, with `onnxruntime`.- **A missing partition is a skip.** If the dt=D state, labels or embeddings snapshot is missing, each asset logs and writes nothing, so the partition keeps the objects it holds.- **Schedule**: 04:30 UTC, after the dataset job's 2h budget (02:30 UTC start) and before the 05:00 UTC training job when the run is short enough. A candidate that lands after that day's manifest is paired by the next day's manifest. The schedule runs by default on prod US and is stopped everywhere else.@@ -397,10 +396,10 @@ All reads route to the offline cluster replicas on Cloud (`etl_workload()`), car## Operating it- Backfill any day range from the Dagster UI; partitions start 2026-04-01 (the label epoch). Every asset sits in the `inbox_ranking_etl` pool so concurrent partitions don't each start their own fleet-wide embeddings scan — the pool's limit is a Dagster deployment setting, provisioned with the bucket.-- A label column change needs a `FEATURE_SCHEMA_VERSION` bump, and no manual backfill. The labels asset stamps the version in each object's `feature-schema-version` metadata. Every hour, `inbox_ranking_labels_refresh_sensor` finds labels partitions in the training lookback whose stamp is missing or older, and runs `inbox_ranking_labels_refresh_job` on at most `INBOX_RANKING_LABELS_REFRESH_MAX_RUNS` (default 6) of them, newest first. A 90-day lookback is current again in about 15 hours. The sensor skips the newest day, which the daily schedule writes, and partitions with no labels object. It requests a partition at most once per schema version, so a failed refresh alerts and needs a person. The refresh covers labels only: report state reads current Postgres and embeddings have a TTL, so a rewrite of those is not point-in-time. `pairs_skipped_missing_label_columns` on `inbox_ranking_examples_built` counts the snapshot pairs each head lost to a missing label column while partitions are stale.+- A label column change needs a `FEATURE_SCHEMA_VERSION` bump, and no manual backfill. The labels asset stamps the version in each object's `feature-schema-version` metadata. Every hour, `inbox_ranking_labels_refresh_sensor` finds labels partitions in the training lookback whose stamp is missing or older, and runs `inbox_ranking_labels_refresh_job` on at most `INBOX_RANKING_LABELS_REFRESH_MAX_RUNS` (default 6) of them, newest first. A 90-day lookback is current again in about 15 hours. The sensor skips the newest day, which the daily schedule writes, and partitions with no labels object. It requests a partition at most once per schema version, so a failed refresh alerts and needs a person. The refresh covers labels only: report state reads current Postgres and embeddings have a TTL, so a rewrite of those is not point-in-time. A head loses the snapshot pairs whose label column is missing while partitions are stale.- Do not materialize these assets in-process in the code-location pod. Its memory limit (2Gi) is too small; launch a run instead.- Failures alert `#alerts-self-driving` (owner `team-self-driving`); assets retry twice with a 60s delay before failing a run. A UI-launched materialization runs under Dagster's implicit `__ASSET_JOB`, which carries no owner tag, so alert routing falls back to matching the `inbox_report_`, `inbox_signal_`, and `inbox_ranking_` asset-name prefixes.-- Runtime budgets are per job (`dagster/max_runtime`): 2h for the dataset job, 3h for the training job. The dataset needs the 2h: its seven label streams run sequentially, each allowed up to 600s, and the join and S3 writes come after them. The dataset cap ends before training starts at 05:00 UTC, so a stuck dataset run fails before training reads its snapshots.+- Runtime budgets are per job (`dagster/max_runtime`): 2h for the dataset job, 1h for the training job, 3h for the `nn` training job. The dataset needs the 2h: its seven label streams run sequentially, each allowed up to 600s, and the join and S3 writes come after them. The dataset cap ends before training starts at 05:00 UTC, so a stuck dataset run fails before training reads its snapshots.## Training consent@@ -410,10 +409,10 @@ The dags train only on reports from organizations whose `Organization.is_ai_trai- `inbox_report_state` keeps only the spine reports of those teams. A label event that names another team's report does not pull it back in. That report's labels still land in `inbox_report_model_data` as a label-only row, which training skips.- `inbox_report_embeddings`, `inbox_report_title_embeddings` and `inbox_signal_embeddings` read only those teams, through `team_id IN (...)` in the ClickHouse query.-- The examples asset drops every state row whose `report_team_id` is not in the current set, before it builds examples. The lookback window reaches partitions written before an organization opted out, so this filter is what makes an opt-out take effect on the next training run.+- The `nn` examples asset drops every state row whose `report_team_id` is not in the current set, before it builds examples. The lookback window reaches partitions written before an organization opted out, so this filter is what makes an opt-out take effect on the next training run.The filter follows the current setting, so an organization that opts back in is trained on again.-The state asset and the examples asset record `excluded_no_training_consent_reports` and `excluded_no_training_consent_teams` as metadata, and `inbox_ranking_examples_built` carries the examples counts. They are counts, never ids.+The state asset and the `nn` examples asset record `excluded_no_training_consent_reports` and `excluded_no_training_consent_teams` as metadata. They are counts, never ids.The online grade reads the dt=D state snapshot for its newborn pool, so that pool shrinks to consenting teams too. Scoring in the sweep does not change.The filter does not delete existing partitions or models. A re-run of `inbox_signal_embeddings` stays additive, so rows that an older run wrote remain in that partition. Nothing trains on signal embeddings today, and a future reader must apply the same filter.diff --git a/products/signals/dags/inbox_ranking/bin/sync_snapshots_local.sh b/products/signals/dags/inbox_ranking/bin/sync_snapshots_local.shindex 75ce00721bcc..97384b17c9c5 100755--- a/products/signals/dags/inbox_ranking/bin/sync_snapshots_local.sh+++ b/products/signals/dags/inbox_ranking/bin/sync_snapshots_local.sh@@ -1,14 +1,13 @@#!/usr/bin/env bash# Copy the prod inbox_report_state and inbox_report_labels partitions into local SeaweedFS so-# inbox_ranking_training_job can run from the local Dagster UI against real snapshots.+# inbox_ranking_nn_training_job can run from the local Dagster UI against real snapshots.## products/signals/dags/inbox_ranking/bin/sync_snapshots_local.sh## INBOX_RANKING_SYNC_EMBEDDINGS=1 also copies inbox_report_embeddings, the report vectors the-# report_embeddings family trains on. It is off by default because that table holds a 1536-float-# vector per live report per partition, which is orders of magnitude more to pull than the other-# two together. Without it the training job still runs: the family finds no vector for any report-# and builds no examples, and every other family is unaffected.+# nn families train on. It is off by default because that table holds a 1536-float vector per+# live report per partition, which is orders of magnitude more to pull than the other two+# together. Without it the nn assets skip the partition.## INBOX_RANKING_SYNC_TITLE_EMBEDDINGS=1 copies inbox_report_title_embeddings, the title-only# vectors of the same reports. It is a separate switch for the same reason: each table is a vectordiff --git a/products/signals/dags/inbox_ranking/multihead/local.py b/products/signals/dags/inbox_ranking/multihead/local.pyindex d08aed58bb58..ec96b4672797 100644--- a/products/signals/dags/inbox_ranking/multihead/local.py+++ b/products/signals/dags/inbox_ranking/multihead/local.py@@ -92,7 +92,7 @@ def consenting_snapshots(def report_vectors(embeddings: pa.Table) -> Extras:- """The `report_embeddings` side input, in the shape the dag's `embeddings_extras` builds."""+ """The `report_embeddings` side input, indexed by report_id."""vectors = (embeddings.select(["report_id", EMBEDDING_COLUMN, EMBEDDING_INSERTED_AT_COLUMN]).to_pandas()diff --git a/products/signals/dags/inbox_ranking/multihead/prototype.py b/products/signals/dags/inbox_ranking/multihead/prototype.pyindex 1577d2ec6d68..09845092ae37 100644--- a/products/signals/dags/inbox_ranking/multihead/prototype.py+++ b/products/signals/dags/inbox_ranking/multihead/prototype.py@@ -46,9 +46,19 @@ from products.signals.dags.inbox_ranking.multihead.inputs import NetworkConfig,from products.signals.dags.inbox_ranking.training.calibration import calibration_buckets, expected_calibration_errorfrom products.signals.dags.inbox_ranking.training.examples import Snapshot, cap_examples, example_moments, holdout_maskfrom products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME, Head-from products.signals.dags.inbox_ranking.training.train import XGB_PARAMSfrom products.signals.dags.inbox_ranking.training.within_team import within_team_auc+# The per-head booster settings the retired `report_embeddings_v2` family trained with.+XGB_PARAMS: dict[str, object] = {+ "n_estimators": 200,+ "max_depth": 4,+ "learning_rate": 0.05,+ "subsample": 0.8,+ "colsample_bytree": 0.8,+ "min_child_weight": 5,+ "eval_metric": "auc",+ "n_jobs": 2,+}PROTOTYPE_HEADS: tuple[Head, ...] = tuple(HEADS_BY_NAME[name] for name in ("open", "action", "fixed"))FEATURE_SET = REPORT_EMBEDDINGS_FEATURE_SETdiff --git a/products/signals/dags/inbox_ranking/nn/model.py b/products/signals/dags/inbox_ranking/nn/model.pyindex 0b1ab31fe931..cd92edfe9aed 100644--- a/products/signals/dags/inbox_ranking/nn/model.py+++ b/products/signals/dags/inbox_ranking/nn/model.py@@ -81,8 +81,8 @@ def candidate_metadata() -> dict[str, Any]:"""The `metadata.json` of a `family` candidate, in the layout of the per-head families.- The heads carry the same metric fields as a booster's (`train_head` and this trainer both fill- `HeadMetrics`), so the promotion rule, the events and the manifest read every family alike.+ The heads carry the metric fields of `HeadMetrics`, the fields the per-head booster families+ wrote, so the promotion rule, the events and the manifest read every family alike."""feature_set = family.feature_setmetadata = {diff --git a/products/signals/dags/inbox_ranking/tests/test_nn.py b/products/signals/dags/inbox_ranking/tests/test_nn.pyindex 4907d1fe0b5d..d8e29f47664f 100644--- a/products/signals/dags/inbox_ranking/tests/test_nn.py+++ b/products/signals/dags/inbox_ranking/tests/test_nn.py@@ -29,7 +29,7 @@ from products.signals.backend.ranking.serving_manifest import ()from products.signals.dags.inbox_ranking.common import DATASET_VERSION, partition_object_keyfrom products.signals.dags.inbox_ranking.dataset import project_vectors-from products.signals.dags.inbox_ranking.dataset.dag import EMBEDDINGS_TABLE, LABELS_TABLE+from products.signals.dags.inbox_ranking.dataset.dag import EMBEDDINGS_TABLE, LABELS_TABLE, STATE_TABLEfrom products.signals.dags.inbox_ranking.multihead.inputs import NetworkConfigfrom products.signals.dags.inbox_ranking.multihead.local import report_vectorsfrom products.signals.dags.inbox_ranking.nn import dag as nn_dag@@ -52,13 +52,8 @@ from products.signals.dags.inbox_ranking.nn.model import (from products.signals.dags.inbox_ranking.tests.test_multihead import D0, DAYS, HOLDOUT_DAYS, _SyntheticInboxfrom products.signals.dags.inbox_ranking.training import recipe as recipe_modulefrom products.signals.dags.inbox_ranking.training.dag import (- CHAMPION_FILE,METADATA_FILE,- SERVED_FILE,- booster_recipe,check_recipe,- inbox_ranking_model_candidate,- inbox_ranking_model_champion,model_object_key,served_object_key,)@@ -71,11 +66,12 @@ from products.signals.dags.inbox_ranking.training.unseen import (NN_MODEL_NAME,NN_WITH_PRIO_FAMILY,NN_WITH_PRIO_MODEL_NAME,- REPORT_EMBEDDINGS_MODEL_NAME,ModelFamily,)PREFIX = settings.INBOX_RANKING_DATASET_S3_PREFIX+# A retired per-head XGBoost family. Its saved versions stay servable by key.+XGBOOST_MODEL_NAME = "report_embeddings_v2"PARTITION = "2026-08-19"NOW = datetime.datetime(2026, 8, 20, 6, tzinfo=datetime.UTC)@@ -176,37 +172,16 @@ def no_side_effects(monkeypatch: pytest.MonkeyPatch) -> None:monkeypatch.setattr(f"{module}.read_ranking_overrides", lambda now: NO_OVERRIDES, raising=False)-def test_the_main_location_never_fits_nn_or_writes_a_pointer(monkeypatch, no_side_effects):- # Every family has a candidate and nothing is served, so each decision would promote. Training- # still writes no pointer: only the promote job chooses what the sweep serves.- client = _S3(- {- model_object_key(PREFIX, name, PARTITION, METADATA_FILE): json.dumps(- _xgboost_metadata(name, PARTITION)- ).encode()- for name in (REPORT_EMBEDDINGS_MODEL_NAME, NN_MODEL_NAME)- }- )- monkeypatch.setattr("products.signals.dags.inbox_ranking.training.dag.s3_client", lambda: client)-- with dagster.build_asset_context(partition_key=PARTITION) as context:- inbox_ranking_model_candidate(context)- inbox_ranking_model_champion(context)-- assert not [key for key in client.objects if key.endswith((CHAMPION_FILE, SERVED_FILE))]- assert not [key for key in client.objects if f"/{NN_MODEL_NAME}/" in key and not key.endswith(METADATA_FILE)]--def test_the_manifest_carries_an_nn_champion_as_an_onnx_entry():decision = compose_manifest([FamilyModels(- name=REPORT_EMBEDDINGS_MODEL_NAME,- champion=_xgboost_metadata(REPORT_EMBEDDINGS_MODEL_NAME, "2026-08-15"),+ name=XGBOOST_MODEL_NAME,+ champion=_xgboost_metadata(XGBOOST_MODEL_NAME, "2026-08-15"),),FamilyModels(name=NN_MODEL_NAME, champion=_nn_metadata("2026-08-16")),],- served=_xgboost_metadata(REPORT_EMBEDDINGS_MODEL_NAME, "2026-08-15"),+ served=_xgboost_metadata(XGBOOST_MODEL_NAME, "2026-08-15"),prefix=PREFIX,now=NOW,)@@ -223,8 +198,8 @@ def test_the_manifest_serves_an_nn_model_with_the_xgboost_champion_as_a_challengdecision = compose_manifest([FamilyModels(- name=REPORT_EMBEDDINGS_MODEL_NAME,- champion=_xgboost_metadata(REPORT_EMBEDDINGS_MODEL_NAME, "2026-08-15"),+ name=XGBOOST_MODEL_NAME,+ champion=_xgboost_metadata(XGBOOST_MODEL_NAME, "2026-08-15"),),FamilyModels(name=NN_MODEL_NAME, champion=_nn_metadata("2026-08-16", heads=NN_HEAD_NAMES)),],@@ -246,7 +221,7 @@ def test_the_manifest_serves_an_nn_model_with_the_xgboost_champion_as_a_challeng(NN_MODEL_NAME, "2026-08-16", [SERVED_ROLE]),(NN_MODEL_NAME, PARTITION, [RECENT_ROLE]),(NN_WITH_PRIO_MODEL_NAME, PARTITION, [CHALLENGER_ROLE]),- (REPORT_EMBEDDINGS_MODEL_NAME, "2026-08-15", [CROSS_FAMILY_ROLE]),+ (XGBOOST_MODEL_NAME, "2026-08-15", [CROSS_FAMILY_ROLE]),]served = decision.manifest.servedassert served.model_kind == ONNX_MODEL_KIND@@ -375,10 +350,10 @@ def _action_recipe(head: Head) -> dict[str, Any]:@pytest.mark.parametrize("stored, recipe, patch, changed",[- ({"2026-08-18": "booster"}, "booster", None, None),+ ({"2026-08-18": "network"}, "network", None, None),(- {"2026-08-18": "booster"},- "booster",+ {"2026-08-18": "network"},+ "network",(settings, "INBOX_RANKING_TRAINING_LOOKBACK_DAYS", 120),"lookback_days",),@@ -420,7 +395,6 @@ def _action_recipe(head: Head) -> dict[str, Any]:)def test_check_recipe_refuses_a_new_recipe_under_an_old_name(monkeypatch, stored, recipe, patch, changed):recipes = {- "booster": lambda: booster_recipe(REPORT_EMBEDDINGS_FEATURE_SET),"network": lambda: network_recipe(NN_FAMILY, NetworkConfig(), ()),"network_more_epochs": lambda: network_recipe(NN_FAMILY, NetworkConfig(epochs=NetworkConfig().epochs + 5), ()),"network_with_vectors": lambda: network_recipe(NN_FAMILY, NetworkConfig(), PROJECT_VECTOR_KINDS),@@ -446,9 +420,20 @@ def test_check_recipe_refuses_a_new_recipe_under_an_old_name(monkeypatch, storeddef test_check_recipe_finds_the_recipe_under_an_earlier_dataset_version():# A DATASET_VERSION bump moves the models to a new folder, so searching only the current one# would let the old name train on the new dataset with no recipe to compare against.- earlier = {**booster_recipe(REPORT_EMBEDDINGS_FEATURE_SET), "dataset_version": "v0"}+ earlier = {**network_recipe(NN_FAMILY, NetworkConfig(), ()), "dataset_version": "v0"}objects = dict([_stored_version("family", "2026-08-18", earlier, dataset_version="v0")])with pytest.raises(dagster.Failure) as failure:- check_recipe(_S3(objects), "bucket", PREFIX, "family", PARTITION, booster_recipe(REPORT_EMBEDDINGS_FEATURE_SET))+ check_recipe(+ _S3(objects), "bucket", PREFIX, "family", PARTITION, network_recipe(NN_FAMILY, NetworkConfig(), ())+ )assert "dataset_version" in (failure.value.description or "")+++def test_examples_depend_on_the_whole_lookback_window():+ # An asset backfill must not run examples(D) before the older state and labels days exist.+ for upstream in (STATE_TABLE, LABELS_TABLE):+ mapping = nn_dag.inbox_ranking_nn_examples.get_partition_mapping(dagster.AssetKey(upstream))+ assert isinstance(mapping, dagster.TimeWindowPartitionMapping)+ assert mapping.start_offset == -settings.INBOX_RANKING_TRAINING_LOOKBACK_DAYS+ assert mapping.end_offset == 0diff --git a/products/signals/dags/inbox_ranking/tests/test_promote.py b/products/signals/dags/inbox_ranking/tests/test_promote.pyindex d1c9997abae5..f67ad646a858 100644--- a/products/signals/dags/inbox_ranking/tests/test_promote.py+++ b/products/signals/dags/inbox_ranking/tests/test_promote.py@@ -17,11 +17,13 @@ from products.signals.dags.inbox_ranking.training.dag import METADATA_FILE, modefrom products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAMEfrom products.signals.dags.inbox_ranking.training.promote import PromoteConfig, inbox_ranking_promotefrom products.signals.dags.inbox_ranking.training.telemetry import MODEL_PROMOTED_EVENT-from products.signals.dags.inbox_ranking.training.unseen import REPORT_EMBEDDINGS_MODEL_NAME, ModelFamily+from products.signals.dags.inbox_ranking.training.unseen import ModelFamilyPREFIX = settings.INBOX_RANKING_DATASET_S3_PREFIXSERVED_VERSION = "2026-10-16"OLDER_VERSION = "2026-10-01"+# A retired XGBoost family. A rollback can still serve one of its saved versions.+XGBOOST_MODEL_NAME = "report_embeddings_v2"class _WritableS3(_ModelStoreS3):@@ -31,7 +33,7 @@ class _WritableS3(_ModelStoreS3):def _metadata(version: str, heads: tuple[str, ...]) -> dict[str, Any]:return {- "model_name": REPORT_EMBEDDINGS_MODEL_NAME,+ "model_name": XGBOOST_MODEL_NAME,"model_version": version,"feature_set": REPORT_EMBEDDINGS_FEATURE_SET.name,"feature_schema_version": REPORT_EMBEDDINGS_FEATURE_SET.schema_version,@@ -41,7 +43,7 @@ def _metadata(version: str, heads: tuple[str, ...]) -> dict[str, Any]:def _add_model(objects: dict[str, bytes], version: str, heads: tuple[str, ...], *, files: bool = True) -> None:- folder = model_object_key(PREFIX, REPORT_EMBEDDINGS_MODEL_NAME, version, "")+ folder = model_object_key(PREFIX, XGBOOST_MODEL_NAME, version, "")objects[folder + METADATA_FILE] = json.dumps(_metadata(version, heads)).encode()if files:for head in heads:@@ -53,8 +55,8 @@ def _dataset() -> _WritableS3:objects: dict[str, bytes] = {served_object_key(PREFIX): json.dumps({- "model_key": f"{REPORT_EMBEDDINGS_MODEL_NAME}@{SERVED_VERSION}",- "model_name": REPORT_EMBEDDINGS_MODEL_NAME,+ "model_key": f"{XGBOOST_MODEL_NAME}@{SERVED_VERSION}",+ "model_name": XGBOOST_MODEL_NAME,"model_version": SERVED_VERSION,}).encode(),@@ -74,7 +76,7 @@ def _promote(monkeypatch: pytest.MonkeyPatch, client: _WritableS3, store: _AppObmonkeypatch.setattr(settings, "INBOX_RANKING_SERVING_MIRROR_BUCKET", "")monkeypatch.setattr("products.signals.dags.inbox_ranking.training.dag.MODEL_FAMILIES",- (ModelFamily(name=REPORT_EMBEDDINGS_MODEL_NAME, feature_set=REPORT_EMBEDDINGS_FEATURE_SET),),+ (ModelFamily(name=XGBOOST_MODEL_NAME, feature_set=REPORT_EMBEDDINGS_FEATURE_SET),),)for module in ("dag", "promote"):monkeypatch.setattr(f"products.signals.dags.inbox_ranking.training.{module}.s3_client", lambda: client)@@ -87,12 +89,12 @@ def _promote(monkeypatch: pytest.MonkeyPatch, client: _WritableS3, store: _AppObdef test_a_rollback_to_an_older_version_rewrites_the_pointer_and_publishes_it(monkeypatch):client, store = _dataset(), _AppObjectStore()- capture = _promote(monkeypatch, client, store, f"{REPORT_EMBEDDINGS_MODEL_NAME}@{OLDER_VERSION}")+ capture = _promote(monkeypatch, client, store, f"{XGBOOST_MODEL_NAME}@{OLDER_VERSION}")pointer = json.loads(client.objects[served_object_key(PREFIX)])assert (pointer["model_key"], pointer["previous_key"], pointer["reason"]) == (- f"{REPORT_EMBEDDINGS_MODEL_NAME}@{OLDER_VERSION}",- f"{REPORT_EMBEDDINGS_MODEL_NAME}@{SERVED_VERSION}",+ f"{XGBOOST_MODEL_NAME}@{OLDER_VERSION}",+ f"{XGBOOST_MODEL_NAME}@{SERVED_VERSION}","roll back",)manifest = ServingManifest.model_validate_json(store.objects[serving_manifest_key(PREFIX)])@@ -106,8 +108,8 @@ def test_a_rollback_to_an_older_version_rewrites_the_pointer_and_publishes_it(mo"model_key,message",[("report_embeddings@latest", "not a `<model_name>@<YYYY-MM-DD>` model key"),- (f"{REPORT_EMBEDDINGS_MODEL_NAME}@2026-09-01", "missing model files"),- (f"{REPORT_EMBEDDINGS_MODEL_NAME}@2026-09-02", "has no model for the heads"),+ (f"{XGBOOST_MODEL_NAME}@2026-09-01", "missing model files"),+ (f"{XGBOOST_MODEL_NAME}@2026-09-02", "has no model for the heads"),],ids=["malformed_key", "missing_file", "missing_head"],)diff --git a/products/signals/dags/inbox_ranking/tests/test_training.py b/products/signals/dags/inbox_ranking/tests/test_training.pyindex 6d351c93e1a5..494cd9257b5b 100644--- a/products/signals/dags/inbox_ranking/tests/test_training.py+++ b/products/signals/dags/inbox_ranking/tests/test_training.py@@ -36,7 +36,7 @@ from products.signals.backend.ranking.features import (FeatureSet,feature_set_by_name,)-from products.signals.backend.ranking.model_contract import classification_thresholds, model_mismatch+from products.signals.backend.ranking.model_contract import model_mismatchfrom products.signals.backend.ranking.overrides import RankingOverridesfrom products.signals.backend.ranking.serving_manifest import (CHALLENGER_ROLE,@@ -56,12 +56,7 @@ from products.signals.backend.ranking.serving_manifest import ()from products.signals.backend.ranking.sinks import REPORT_SCORED_EVENTfrom products.signals.dags.inbox_ranking.common import partition_object_key-from products.signals.dags.inbox_ranking.dataset.dag import (- EMBEDDINGS_TABLE,- LABELS_TABLE,- PROJECT_VECTORS_TABLE,- STATE_TABLE,-)+from products.signals.dags.inbox_ranking.dataset.dag import LABELS_TABLE, PROJECT_VECTORS_TABLE, STATE_TABLEfrom products.signals.dags.inbox_ranking.training.calibration import (BUCKETS,calibration_buckets,@@ -69,22 +64,13 @@ from products.signals.dags.inbox_ranking.training.calibration import ()from products.signals.dags.inbox_ranking.training.classification import classification_metricsfrom products.signals.dags.inbox_ranking.training.dag import (- _EXTRA_SNAPSHOT_TABLES,_STATE_READ_COLUMNS,METADATA_FILE,_delete_other_objects,_pinned_models,_publish_manifest,- _train_candidate,- booster_recipe,- candidate_metadata,champion_object_key,- embeddings_extras,- example_windows,- examples_object_key,- examples_table,grade_metadata,- inbox_ranking_training_examples,inbox_ranking_unseen_graded,load_snapshots,model_object_key,@@ -93,7 +79,6 @@ from products.signals.dags.inbox_ranking.training.dag import ()from products.signals.dags.inbox_ranking.training.examples import (STATE_LAG_LIMIT,- ConsentExclusion,Snapshot,assemble_snapshot,birth_day_positives,@@ -111,7 +96,6 @@ from products.signals.dags.inbox_ranking.training.promotion import (PromotionDecision,decide_promotion,)-from products.signals.dags.inbox_ranking.training.recipe import RECIPE_HASH_KEY, recipe_hashfrom products.signals.dags.inbox_ranking.training.served import (event_model_role,served_events,@@ -123,23 +107,16 @@ from products.signals.dags.inbox_ranking.training.telemetry import (DISTINCT_ID,LOCAL_DISTINCT_ID,SERVING_MANIFEST_PUBLISHED_EVENT,- HeadExampleCounts,TrainingEvent,candidate_events,capture_training_events,- examples_events,holdout_calibration_events,promotion_event,unseen_calibration_events,unseen_head_graded_events,unseen_report_graded_events,)-from products.signals.dags.inbox_ranking.training.train import (- HoldoutGrade,- _head_readable,- booster_holdout_grade,- train_head,-)+from products.signals.dags.inbox_ranking.training.train import HoldoutGrade, _head_readablefrom products.signals.dags.inbox_ranking.training.unseen import (CANDIDATE_ROLE,LEGACY_POOL_NAME,@@ -147,7 +124,6 @@ from products.signals.dags.inbox_ranking.training.unseen import (NN_MODEL_NAME,NN_WITH_PRIO_MODEL_NAME,POOL_NAME,- REPORT_EMBEDDINGS_MODEL_NAME,SERVED_SCORES_TABLE,ModelFamily,calibration_rows,@@ -164,7 +140,8 @@ from products.signals.dags.inbox_ranking.training.unseen import ()from products.signals.dags.inbox_ranking.training.within_team import NO_TEAMS, WithinTeamAuc, within_team_auc-EMBEDDINGS_MODEL_NAME = REPORT_EMBEDDINGS_MODEL_NAME+# A per-head XGBoost family on the report vector. The sweep still serves this learner.+EMBEDDINGS_MODEL_NAME = "report_embeddings_xgb"EMBEDDING_FEATURE_SETS = (REPORT_EMBEDDINGS_FEATURE_SET,)@@ -479,165 +456,6 @@ def test_holdout_mask_cuts_by_report_not_by_row():assert holdout_mask(examples, holdout_days=7).tolist() == [False, False, True, True]-def test_train_head_learns_a_separable_signal_and_names_its_features():- head = HEADS_BY_NAME["open"]- rng = np.random.default_rng(0)- n = 2000- created = pd.to_datetime("2026-07-01", utc=True) + pd.to_timedelta(rng.integers(0, 40, n), unit="D")- signal_count = rng.integers(1, 50, n)- rows = pd.DataFrame(- {- "signal_count": signal_count,- "total_weight": rng.random(n),- "run_count": 1,- "title_chars": 40,- "summary_chars": 400,- "priority": "P2",- "actionability": None,- "age_hours": 12.0,- }- )- examples = STATE_FEATURE_SET.build_matrix(rows)- examples.insert(0, "head", head.name)- examples.insert(1, "report_id", [f"r{i}" for i in range(n)])- examples.insert(2, "snapshot_date", D0)- examples.insert(3, "report_created_at", created)- examples.insert(4, "team_id", rng.integers(1, 4, n))- examples["label"] = (signal_count > 25).astype(int)-- trained = train_head(examples, head, feature_names=STATE_FEATURE_SET.feature_names, holdout_days=7)- assert trained is not None- assert trained.metrics.readable- assert trained.metrics.holdout_auc is not None and trained.metrics.holdout_auc > 0.9- assert trained.metrics.null_auc is not None and abs(trained.metrics.null_auc - 0.5) < 0.15- assert trained.metrics.null_auc_std is not None and trained.metrics.null_auc_std < 0.15- assert trained.metrics.train_auc is not None and trained.metrics.train_auc > 0.9- assert trained.metrics.holdout_average_precision is not None and trained.metrics.holdout_average_precision > 0.9- assert trained.metrics.holdout_logloss is not None and trained.metrics.holdout_logloss < 0.5- assert trained.metrics.holdout_positive_rate == pytest.approx(- trained.metrics.holdout_positives / trained.metrics.holdout_rows- )- # The holdout cut is the train-only rate, and the unseen cut is the rate of every refit row.- holdout = trained.metrics.holdout_classification- assert holdout.threshold == pytest.approx(trained.metrics.train_positives / trained.metrics.train_rows)- assert trained.metrics.refit_classification_threshold == pytest.approx(examples["label"].mean())- assert holdout.threshold != trained.metrics.refit_classification_threshold- counts = (holdout.true_positives, holdout.false_positives, holdout.true_negatives, holdout.false_negatives)- assert sum(count or 0 for count in counts) == trained.metrics.holdout_rows- assert holdout.precision is not None and holdout.precision > 0.8- assert holdout.recall is not None and holdout.recall > 0.8- within_team = trained.metrics.holdout_within_team- assert within_team.teams == 3 and within_team.auc is not None and within_team.auc > 0.9- metadata_head = trained.metrics.as_dict()- assert metadata_head["holdout_classification_threshold"] == holdout.threshold- assert metadata_head.items() >= within_team.as_dict(prefix="holdout_").items()- assert classification_thresholds({"heads": [metadata_head]}) == {- head.name: trained.metrics.refit_classification_threshold- }- booster = xgb.Booster()- booster.load_model(bytearray(trained.booster_ubj))- assert booster.feature_names == list(STATE_FEATURE_SET.feature_names)- # The saved holdout fit graded on the same rows must reproduce the stored metric: this is the- # path the champion gate uses to compare two models on one holdout.- assert trained.holdout_booster_ubj is not None- paired = booster_holdout_grade(- trained.holdout_booster_ubj, examples, head, feature_names=STATE_FEATURE_SET.feature_names, holdout_days=7- )- assert paired is not None- assert paired.auc == pytest.approx(trained.metrics.holdout_auc, abs=1e-6)- assert paired.expected_calibration_error == pytest.approx(- trained.metrics.holdout_expected_calibration_error, abs=1e-6- )- # The metadata that ships says what the probability is conditioned on and how much history it saw.- window = {"example_window_start": "2026-08-01", "example_cap_bound": False}- table = examples_table(examples, STATE_FEATURE_SET, {head.name: window})- buffer = io.BytesIO()- pq.write_table(table, buffer)- windows = example_windows(pq.read_table(pa.BufferReader(buffer.getvalue())))- candidate_head = candidate_metadata(- "2026-08-19",- [trained],- model_name=STATE_MODEL_NAME,- feature_set=STATE_FEATURE_SET,- skipped=[],- trained_at=NOW,- run_id="run-1",- windows=windows,- )["heads"][0]- assert {"cohort": head.cohort.__name__, "horizon_days": head.horizon_days, **window}.items() <= (- candidate_head.items()- )- # A booster from another feature schema is not scorable on these examples: the gate must fall- # back to the stored AUC instead of failing the champion asset every day.- other_schema = xgb.XGBClassifier(n_estimators=2).fit(pd.DataFrame({"not_a_feature": [0, 1, 0, 1]}), [0, 1, 0, 1])- other_ubj = bytes(other_schema.get_booster().save_raw("ubj"))- assert (- booster_holdout_grade(other_ubj, examples, head, feature_names=STATE_FEATURE_SET.feature_names, holdout_days=7)- is None- )---def test_train_head_keeps_logloss_on_a_single_class_holdout():- # Only the newest reports fall in the holdout, and they are all negatives: AUC is undefined- # there but logloss is not, and the head must still train and ship.- head = HEADS_BY_NAME["open"]- rng = np.random.default_rng(1)- n = 400- created = pd.to_datetime("2026-07-01", utc=True) + pd.to_timedelta(rng.integers(0, 40, n), unit="D")- signal_count = rng.integers(1, 50, n)- rows = pd.DataFrame(- {- "signal_count": signal_count,- "total_weight": rng.random(n),- "run_count": 1,- "title_chars": 40,- "summary_chars": 400,- "priority": "P2",- "actionability": None,- "age_hours": 12.0,- }- )- examples = STATE_FEATURE_SET.build_matrix(rows)- examples.insert(0, "head", head.name)- examples.insert(1, "report_id", [f"r{i}" for i in range(n)])- examples.insert(2, "snapshot_date", D0)- examples.insert(3, "report_created_at", created)- in_holdout = holdout_mask(examples, 7).to_numpy()- examples["label"] = ((signal_count > 25) & ~in_holdout).astype(int)-- trained = train_head(examples, head, feature_names=STATE_FEATURE_SET.feature_names, holdout_days=7)- assert trained is not None- assert trained.metrics.holdout_positives == 0- assert trained.metrics.holdout_auc is None- assert trained.metrics.holdout_average_precision is None- assert trained.metrics.holdout_logloss is not None and trained.metrics.holdout_logloss > 0- assert not trained.metrics.readable- # With no positive in the holdout, the decile error is the whole mean score.- assert len(trained.calibration) == BUCKETS- assert trained.metrics.holdout_mean_score is not None- assert trained.metrics.holdout_expected_calibration_error == pytest.approx(trained.metrics.holdout_mean_score)- # No holdout positive: recall has no denominator, and the negatives still give specificity.- assert trained.metrics.holdout_classification.recall is None- assert trained.metrics.holdout_classification.false_negatives == 0- assert trained.metrics.holdout_classification.specificity is not None---def test_train_head_returns_none_without_both_classes():- head = HEADS_BY_NAME["open"]- examples = pd.DataFrame(- {- "head": [head.name] * 3,- "report_id": ["a", "b", "c"],- "snapshot_date": D0,- "report_created_at": pd.to_datetime(["2026-08-01"] * 3, utc=True),- "label": [0, 0, 0],- }- )- for name in STATE_FEATURE_SET.feature_names:- examples[name] = 1.0- assert train_head(examples, head, feature_names=STATE_FEATURE_SET.feature_names, holdout_days=7) is None--class _FakeClient:def __init__(self) -> None:self.calls: list[dict[str, Any]] = []@@ -754,7 +572,7 @@ def test_birth_grain_keeps_one_row_per_report_on_the_day_it_was_created(feature_assert moments[moments["report_id"] == "newborn"]["label"].tolist() == [0, 1, 1]-def test_reports_missing_birth_snapshot_counts_only_the_ones_a_partition_gap_costs(monkeypatch):+def test_reports_missing_birth_snapshot_counts_only_the_ones_a_partition_gap_costs():gap = D0 + datetime.timedelta(days=1)dates = [D0 + datetime.timedelta(days=offset) for offset in range(3 + HEADS_BY_NAME["open"].horizon_days)]ids = ["born_in_the_gap", "older", "utc_start", "utc_end", "label_only"]@@ -778,18 +596,6 @@ def test_reports_missing_birth_snapshot_counts_only_the_ones_a_partition_gap_cosexamples = build_examples(snapshots, HEADS_BY_NAME["open"], STATE_FEATURE_SET)assert examples["report_id"].tolist() == ["utc_end"]- module = "products.signals.dags.inbox_ranking.training.dag"- monkeypatch.setattr(f"{module}.skip_unconfigured", lambda context: False)- monkeypatch.setattr(f"{module}.s3_client", lambda: None)- monkeypatch.setattr(f"{module}.snapshot_dates", lambda *args: dates)- monkeypatch.setattr(f"{module}.load_snapshots", lambda *args, **kwargs: snapshots)- monkeypatch.setattr(f"{module}.training_consent_team_ids", lambda: frozenset({1}))- monkeypatch.setattr(f"{module}.embeddings_extras", lambda *args: NO_EXTRAS)- monkeypatch.setattr(f"{module}._write_examples", lambda *args: {})- with dagster.build_asset_context(partition_key=dates[-1].isoformat()) as context:- inbox_ranking_training_examples(context)- assert context.get_output_metadata("result")["reports_missing_birth_snapshot"].value == 2-def test_build_examples_keeps_an_outcome_that_landed_on_the_reports_birth_day():# Many outcomes land the day the report is born. Censoring on the labels of the birth day would@@ -1176,6 +982,7 @@ def test_scored_pool_names_the_definition_a_scores_object_was_written_under(scor(_scores(["a", "b"], model_name=[None, EMBEDDINGS_MODEL_NAME]), [EMBEDDINGS_MODEL_NAME]),# Named rows of a retired family are dropped too, or its grades outlive the family.(_scores(["a", "b"], model_name=["tabular_xgb", EMBEDDINGS_MODEL_NAME]), [EMBEDDINGS_MODEL_NAME]),+ (_scores(["a", "b"], model_name=["report_embeddings_v2", EMBEDDINGS_MODEL_NAME]), [EMBEDDINGS_MODEL_NAME]),],)def test_with_model_names_drops_rows_that_name_no_family(scores, expected):@@ -1279,7 +1086,7 @@ def test_an_empty_scores_write_is_refused_over_a_partition_that_holds_rows(exist([STATE_MODEL_NAME], [], [STATE_MODEL_NAME]),([], [STATE_MODEL_NAME], []),(- ["tabular_xgb", "title_embeddings", EMBEDDINGS_MODEL_NAME],+ ["tabular_xgb", "title_embeddings", "report_embeddings_v2", EMBEDDINGS_MODEL_NAME],[EMBEDDINGS_MODEL_NAME],[],),@@ -1348,7 +1155,7 @@ def test_training_events_capture_gate_and_local_marking(monkeypatch, cloud, debug, expected_distinct_id, expected_environment):client = _patch_capture(monkeypatch, cloud=cloud, debug=debug)- events = [TrainingEvent(event="inbox_ranking_examples_built", properties={"head": "open"})]+ events = [TrainingEvent(event="inbox_ranking_candidate_trained", properties={"head": "open"})]capture_training_events(dagster.build_asset_context(), "2026-08-25", events)if expected_distinct_id is None:assert client.calls == []@@ -1403,24 +1210,6 @@ def test_training_events_carry_the_dashboard_contract(monkeypatch):grades = head_grades(graded, HEADS_BY_NAME["open"], pool=POOL_NAME, scoring_partition="2026-08-18")events = [*candidate_events(metadata),- *examples_events(- partition_key="2026-08-25",- run_id="run-1",- feature_set=STATE_FEATURE_SET.name,- snapshots=20,- backfilled_rows=0,- excluded=ConsentExclusion(reports=4, teams=2),- per_head={- "open": HeadExampleCounts(- rows=10,- positives=2,- birth_day_positives=1,- example_window_start=datetime.date(2026, 7, 1),- example_cap_bound=True,- pairs_skipped_missing_label_columns=0,- )- },- ),promotion_event(partition_key="2026-08-25",run_id="run-1",@@ -1465,7 +1254,7 @@ def test_training_events_carry_the_dashboard_contract(monkeypatch):assert call["timestamp"] == datetime.datetime(2026, 8, 25, 12, tzinfo=datetime.UTC)assert call["properties"]["$process_person_profile"] is Falseassert call["properties"]["model_version"] == "2026-08-25"- # Every model event breaks down on the family; the examples event is per feature set instead.+ # Every model event breaks down on the family.for event_name in ("inbox_ranking_candidate_trained","inbox_ranking_promotion_decided",@@ -1486,19 +1275,6 @@ def test_training_events_carry_the_dashboard_contract(monkeypatch):assert "file" not in candidates[0]["properties"]# A head with nothing to fit still reports, so the readability alert sees a bad day, not a gap.assert {"trained": False, "readable": False}.items() <= candidates[2]["properties"].items()- examples_props = by_event["inbox_ranking_examples_built"][0]["properties"]- # Examples are per feature set, not per family: this is the dimension the counts break down on.- assert {- "head": "open",- "rows": 10,- "positives": 2,- "birth_day_positives": 1,- "example_window_start": "2026-07-01",- "example_cap_bound": True,- "feature_set": STATE_FEATURE_SET.name,- "excluded_no_training_consent_reports": 4,- "excluded_no_training_consent_teams": 2,- }.items() <= examples_props.items()promotion_props = by_event["inbox_ranking_promotion_decided"][0]["properties"]assert {"would_promote": True,@@ -1776,15 +1552,6 @@ def test_load_snapshots_fails_without_the_requested_partition():load_snapshots(_EmptyS3(), "bucket", "inbox_ranking", dates, required=dates[-1])-def test_examples_depend_on_the_whole_lookback_window():- # An asset backfill must not run examples(D) before the older state and labels days exist.- for upstream in (STATE_TABLE, LABELS_TABLE):- mapping = inbox_ranking_training_examples.get_partition_mapping(dagster.AssetKey(upstream))- assert isinstance(mapping, dagster.TimeWindowPartitionMapping)- assert mapping.start_offset == -settings.INBOX_RANKING_TRAINING_LOOKBACK_DAYS- assert mapping.end_offset == 0--def test_model_key_layout_is_stable():# The scoring sweep resolves these keys, and a prefix per family stops two families colliding.assert (@@ -1797,7 +1564,7 @@ def test_model_key_layout_is_stable():)assert (model_object_key("inbox_ranking", EMBEDDINGS_MODEL_NAME, "2026-08-19", "open.ubj")- == "inbox_ranking/inbox_ranking_models/v1/report_embeddings_v2/dt=2026-08-19/open.ubj"+ == "inbox_ranking/inbox_ranking_models/v1/report_embeddings_xgb/dt=2026-08-19/open.ubj")assert snapshot_dates("2026-08-19", 2) == [datetime.date(2026, 8, 17),@@ -1910,27 +1677,6 @@ def test_model_mismatch_checks_a_model_against_its_own_feature_set(metadata, expassert expected is None and mismatch is None or (mismatch is not None and expected in mismatch)[email protected]("family", MODEL_FAMILIES, ids=[family.name for family in MODEL_FAMILIES])-def test_candidate_metadata_declares_the_set_it_was_fit_on(family):- # The trainer's own record must pass the grader's check, or the day's candidate goes unscored.- # Every registered family, because the trainer fits each on the set its registry entry names.- metadata = candidate_metadata(- "2026-08-19",- [],- model_name=family.name,- feature_set=family.feature_set,- skipped=[],- trained_at=NOW,- run_id="run-1",- )- assert metadata["model_name"] == family.name- assert metadata["feature_set"] == family.feature_set.name- assert metadata["feature_schema_version"] == family.feature_set.schema_version- assert model_mismatch(metadata) is None- # The next day's recipe check compares against this hash, so it must be the one the check computes.- assert metadata[RECIPE_HASH_KEY] == recipe_hash(booster_recipe(family.feature_set))--def test_build_examples_carries_the_columns_of_the_set_it_is_given():# The examples Parquet is per feature set, so a second set's object holds its own features.ids = ["a"]@@ -1952,23 +1698,6 @@ def test_build_examples_carries_the_columns_of_the_set_it_is_given():assert examples["age_hours"].tolist() == [12.0]-def test_examples_key_layout_is_per_feature_set():- # Two sets carry different feature columns, so they cannot share a partition's object.- assert (- examples_object_key("inbox_ranking", STATE_FEATURE_SET.name, "2026-08-19")- == "inbox_ranking/inbox_ranking_training_examples/v1/state/dt=2026-08-19/part-00000.parquet"- )- assert (- examples_object_key("inbox_ranking", REPORT_EMBEDDINGS_FEATURE_SET.name, "2026-08-19")- == "inbox_ranking/inbox_ranking_training_examples/v1/report_embeddings/dt=2026-08-19/part-00000.parquet"- )- # And one models prefix per family, so one family's booster never lands on another's.- assert (- model_object_key("inbox_ranking", EMBEDDINGS_MODEL_NAME, "2026-08-19", "open.ubj")- == "inbox_ranking/inbox_ranking_models/v1/report_embeddings_v2/dt=2026-08-19/open.ubj"- )--@pytest.mark.parametrize("feature_set", EMBEDDING_FEATURE_SETS)def test_every_registered_set_is_reachable_by_the_name_a_model_records(feature_set):# A model names its set in `metadata.json`, and a name this build cannot resolve is left@@ -2056,81 +1785,6 @@ def test_a_report_without_this_models_vector_is_not_an_embeddings_example(featurassert feature_set.build_matrix(rows, extras).isna().to_numpy().all()-def _snapshot_objects(**frames: pd.DataFrame | None) -> dict[str, bytes]:- """The dt=D embeddings partitions to serve, by extras key. A key mapped to None has no object."""- return {- partition_object_key("inbox_ranking", _EXTRA_SNAPSHOT_TABLES[key], "2026-08-10"): _parquet(frame)- for key, frame in frames.items()- if frame is not None- }---def _extras_for(objects: dict[str, bytes], keys=_EMBEDDING_EXTRA_KEYS, **kwargs) -> Extras:- return embeddings_extras(- dagster.build_asset_context(), _ParquetS3(objects), "bucket", "inbox_ranking", "2026-08-10", keys, **kwargs- )---def test_a_missing_rendering_is_left_out_while_an_empty_one_keeps_its_key():- # The two cases must not be folded together. A missing snapshot has to leave the family's- # partition as it stands, because rebuilding from nothing writes an empty candidate over- # boosters a champion pointer can name. A snapshot that exists and holds no usable vector is a- # day the family genuinely has nothing to fit, and takes the ordinary thin-input path.- empty = _vector_frame({}).astype({EMBEDDING_COLUMN: object})-- missing = _extras_for(_snapshot_objects(report_embeddings=None))- assert missing == {}- assert REPORT_EMBEDDINGS_FEATURE_SET.missing_extras(missing) == (REPORT_EMBEDDINGS_EXTRA,)-- empty_snapshot = _extras_for(_snapshot_objects(report_embeddings=empty))- assert REPORT_EMBEDDINGS_FEATURE_SET.missing_extras(empty_snapshot) == ()- assert REPORT_EMBEDDINGS_FEATURE_SET.buildable(_state(["a"]), empty_snapshot).tolist() == [False]---def test_a_rendering_is_read_only_when_a_set_asks_for_it():- # The examples asset reads one set's inputs at a time, so a family it is not building must not- # pull a fleet-wide vector table across the network.- objects = _snapshot_objects(report_embeddings=_vector_frame({"a": _embedding()}))- assert set(_extras_for(objects, REPORT_EMBEDDINGS_FEATURE_SET.extras_keys)) == {REPORT_EMBEDDINGS_EXTRA}- assert _extras_for(objects, ()) == {}---def test_a_set_whose_snapshot_is_missing_is_skipped(monkeypatch):- # A failed snapshot must leave the family's examples as they stand, not rebuild them from nothing.- module = "products.signals.dags.inbox_ranking.training.dag"- written: list[str] = []- monkeypatch.setattr(f"{module}.skip_unconfigured", lambda context: False)- monkeypatch.setattr(f"{module}.s3_client", lambda: None)- monkeypatch.setattr(f"{module}.load_snapshots", lambda *args, **kwargs: {})- monkeypatch.setattr(f"{module}.training_consent_team_ids", lambda: frozenset())- monkeypatch.setattr(- f"{module}.embeddings_extras",- lambda context, client, bucket, prefix, partition_key, keys: {- key: _vector_frame({}) for key in keys if key != REPORT_EMBEDDINGS_EXTRA- },- )-- def _record(context, client, bucket, prefix, partition_key, feature_set, *args) -> dict[str, Any]:- written.append(feature_set.name)- return {}-- monkeypatch.setattr(f"{module}._write_examples", _record)-- with dagster.build_asset_context(partition_key="2026-08-10") as context:- inbox_ranking_training_examples(context)- metadata = context.get_output_metadata("result")-- assert written == []- assert metadata[f"{REPORT_EMBEDDINGS_FEATURE_SET.name}_skipped"].value is True---def test_the_embeddings_snapshot_is_upstream_of_the_examples_that_read_it():- # An asset that reads a snapshot without declaring it can run before the day's snapshot lands- # and record the family as having no input.- deps = {key.path[-1] for key in inbox_ranking_training_examples.keys_by_input_name.values()}- assert EMBEDDINGS_TABLE in deps--@pytest.mark.parametrize("feature_set", EMBEDDING_FEATURE_SETS, ids=EMBEDDING_SET_IDS)def test_report_grain_keeps_one_example_per_report_and_needs_a_vector(feature_set):# The scoring-moment grain emits a near-duplicate row per snapshot, which is what 1536 columns@@ -2244,19 +1898,6 @@ def test_a_set_declares_the_side_inputs_it_needs(feature_set, extras, expected):assert feature_set.missing_extras(extras) == expected-def test_a_family_without_examples_keeps_the_partition_it_already_has():- # Rebuilding from a missing examples object would write an empty candidate and delete this- # partition's boosters, which a champion pointer can name. _EmptyS3 has no put_object or- # delete_objects, so any write here raises instead of passing silently.- family = ModelFamily(name=EMBEDDINGS_MODEL_NAME, feature_set=REPORT_EMBEDDINGS_FEATURE_SET)-- metadata = _train_candidate(- dagster.build_asset_context(), _EmptyS3(), "bucket", "inbox_ranking", "2026-08-19", family- )-- assert metadata[f"{EMBEDDINGS_MODEL_NAME}_skipped"].value is True--def _serving_metadata(model_name: str, version: str, *, heads=("open", "action"), readable=True, model_file=None, **overrides):@@ -2860,7 +2501,7 @@ def test_the_serving_prefix_layout_is_stable():assert serving_manifest_key("inbox_ranking") == "inbox_ranking/serving/manifest.json"assert (serving_model_prefix("inbox_ranking", model_key(EMBEDDINGS_MODEL_NAME, "2026-08-19"))- == "inbox_ranking/serving/models/report_embeddings_v2@2026-08-19"+ == "inbox_ranking/serving/models/report_embeddings_xgb@2026-08-19")@@ -2878,12 +2519,12 @@ def _served_events(*events: tuple[pd.Timestamp, dict[str, Any]]) -> pd.DataFrameV1, V2 = "2026-08-08", "2026-08-09"-_V1_EVENT = {"model_name": REPORT_EMBEDDINGS_MODEL_NAME, "model_version": V1, "readable_heads": ["open"]}-_V2_EVENT = {"model_name": REPORT_EMBEDDINGS_MODEL_NAME, "model_version": V2}+_V1_EVENT = {"model_name": EMBEDDINGS_MODEL_NAME, "model_version": V1, "readable_heads": ["open"]}+_V2_EVENT = {"model_name": EMBEDDINGS_MODEL_NAME, "model_version": V2}CANDIDATE_VERSION = "2026-08-10"_CANDIDATE_EVENT = {- "model_key": model_key(REPORT_EMBEDDINGS_MODEL_NAME, CANDIDATE_VERSION),- "model_name": REPORT_EMBEDDINGS_MODEL_NAME,+ "model_key": model_key(EMBEDDINGS_MODEL_NAME, CANDIDATE_VERSION),+ "model_name": EMBEDDINGS_MODEL_NAME,"model_version": CANDIDATE_VERSION,"roles": [RECENT_ROLE],}@@ -2952,14 +2593,14 @@ def test_served_score_rows_keep_the_earliest_score_of_each_newborn_per_model_keyzip(others["report_id"], others["model_name"], others["model_version"], others["model_role"], others["score"])) == [("challenger_only", NN_MODEL_NAME, V1, CROSS_FAMILY_ROLE, 0.5),- ("tie", REPORT_EMBEDDINGS_MODEL_NAME, CANDIDATE_VERSION, RECENT_ROLE, 0.7),+ ("tie", EMBEDDINGS_MODEL_NAME, CANDIDATE_VERSION, RECENT_ROLE, 0.7),]metadata = served_metadata(_state(_SERVED_POOL), rows)assert metadata["served_pool_coverage"] == dagster.MetadataValue.float(4 / 6)assert metadata["pool_coverage_by_model"].value == {- f"{REPORT_EMBEDDINGS_MODEL_NAME}@{V1}/{SERVED_ROLE}": 2 / 6,- f"{REPORT_EMBEDDINGS_MODEL_NAME}@{V2}/{SERVED_ROLE}": 3 / 6,- f"{REPORT_EMBEDDINGS_MODEL_NAME}@{CANDIDATE_VERSION}/{RECENT_ROLE}": 1 / 6,+ f"{EMBEDDINGS_MODEL_NAME}@{V1}/{SERVED_ROLE}": 2 / 6,+ f"{EMBEDDINGS_MODEL_NAME}@{V2}/{SERVED_ROLE}": 3 / 6,+ f"{EMBEDDINGS_MODEL_NAME}@{CANDIDATE_VERSION}/{RECENT_ROLE}": 1 / 6,f"{NN_MODEL_NAME}@{V1}/{CROSS_FAMILY_ROLE}": 1 / 6,}@@ -3009,9 +2650,9 @@ def test_served_scores_grade_per_version_at_the_threshold_that_was_served(monkey}# Every model key the sweep scored is graded on the same scoring day, under the role it scored with.assert set(graded_by_key) == {- (REPORT_EMBEDDINGS_MODEL_NAME, V1, SERVED_ROLE),- (REPORT_EMBEDDINGS_MODEL_NAME, V2, SERVED_ROLE),- (REPORT_EMBEDDINGS_MODEL_NAME, CANDIDATE_VERSION, RECENT_ROLE),+ (EMBEDDINGS_MODEL_NAME, V1, SERVED_ROLE),+ (EMBEDDINGS_MODEL_NAME, V2, SERVED_ROLE),+ (EMBEDDINGS_MODEL_NAME, CANDIDATE_VERSION, RECENT_ROLE),(NN_MODEL_NAME, V1, CROSS_FAMILY_ROLE),}graded = {version: row for (_, version, role), row in graded_by_key.items() if role == SERVED_ROLE}diff --git a/products/signals/dags/inbox_ranking/training/dag.py b/products/signals/dags/inbox_ranking/training/dag.pyindex ae4196ea54db..bd3599d90fd4 100644--- a/products/signals/dags/inbox_ranking/training/dag.py+++ b/products/signals/dags/inbox_ranking/training/dag.py@@ -1,42 +1,30 @@-"""Daily training for the Self-driving Inbox report-ranking model (v0: per-head XGBoost).+"""Daily serving publish and online grading for the Self-driving Inbox report-ranking models.-Three assets on the same daily partition as the dataset dag, each writing under the dataset prefix:+The models are trained by the `signals_ml` location (`nn/dag.py`). This dag fits nothing. Its+assets run on the same daily partition as the dataset dag and write under the dataset prefix:- inbox_ranking_training_examples/v1/<set>/dt=D/ examples over the trailing snapshots, per feature set- inbox_ranking_models/v1/<name>/dt=D/<head>.ubj one booster per head + metadata.json (the candidate)inbox_ranking_models/v1/served.json pointer to the model key the scoring sweep servesinbox_ranking_served_scores/v1/dt=D/ the scoring sweep's scores of the reports born that day-A fifth asset publishes what the scoring sweep serves. The dataset bucket holds the training-history and the Temporal workers cannot reach it, so `inbox_ranking_serving_manifest` copies the-models the sweep needs into the deployment's own object store and writes a manifest naming them+`inbox_ranking_serving_manifest` publishes what the scoring sweep serves. The dataset bucket holds+the training history and the Temporal workers cannot reach it, so the asset copies the models the+sweep needs into the deployment's own object store and writes a manifest naming them(`serving/manifest.json`, `serving/models/<key>/`). The manifest is the dag's serving decision:-which models a pass scores, and which of them the inbox would order on.--Partition dt=D trains on the report-state/labels snapshots dt=D-lookback..D (issue 13's-scoring-moment join, `training/examples.py`), grades each head on the last `holdout_days` of-reports, and refits on everything. The champion asset applies `promotion.decide_promotion` and-records the decision, so the daily candidate series doubles as monitoring. It never moves a-pointer: only `inbox_ranking_promote_job` (`training/promote.py`) writes `served.json`, and a-person runs it. Every candidate is kept under `models/v1/<model_name>/dt=D/`; a re-run of a-partition replaces that prefix in full (stale head files are removed). The incumbent is the served-model when it has the candidate's `model_name`, and it is graded on the candidate's holdout through-its `<head>.holdout.ubj` (the train-only fit), so the rule compares both models on one set of-reports. The `<name>/champion.json` objects that the retired automatic promotion wrote stay in-place, and the manifest still reads them for the cross-family role.--Both the candidate and the champion asset walk `families_trained_in(MAIN_LOCATION)`, so each family trains on the-examples of the feature set it declares and is graded against the served model of its own name. A family whose-examples or metadata are missing that day is logged and skipped: its own series has a gap, and-every other family still trains and gets graded.-The `nn` family is trained by the `signals_ml` location (`nn/dag.py`) and never here.+which models a pass scores, and which of them the inbox would order on. Only+`inbox_ranking_promote_job` (`training/promote.py`) writes `served.json`, and a person runs it.+The `<name>/champion.json` objects that the retired automatic promotion wrote stay in place, and+the manifest still reads them for the cross-family role.++This module also holds the plumbing the `nn` trainer shares: snapshot loading, the recipe check+and `decide_family_champion`, which records each family's promotion decision against the served+model of its own name, graded on the candidate's holdout through the incumbent's train-only fit.Two further assets grade the served models on data no example covers. `inbox_ranking_served_scores`(`training/served.py`) reads the scoring sweep's live scores of every report born on D, one row setper model key the manifest named (`unseen_pool` explains why no example can cover one);`inbox_ranking_unseen_graded` evaluates saved scores daily against the dt=D labels until eachhead's horizon, keeping provisional evaluations separate from mature grades. The holdout AUC-grades the recipe, because the shipped booster is refit on train plus holdout; the baked online+grades the recipe, because the shipped model is refit on train plus holdout; the baked onlineAUC grades each model on reports it never saw. The two are comparable because both apply thesame `Head` cohort, label and horizon."""@@ -44,30 +32,20 @@ same `Head` cohort, label and horizon.import jsonimport datetimefrom collections.abc import Callable, Mapping, Sequence-from types import MappingProxyTypefrom typing import TYPE_CHECKING, Anyimport pandas as pdimport dagster-import pyarrow as pafrom botocore.exceptions import ClientErrorfrom posthog import settingsfrom posthog.dataclasses import frozenfrom posthog.storage import object_storage-from products.signals.backend.ranking.features import (- EMBEDDING_COLUMN,- EMBEDDING_INSERTED_AT_COLUMN,- FEATURE_SETS,- REPORT_EMBEDDINGS_EXTRA,- Extras,- FeatureSet,-)-from products.signals.backend.ranking.model_contract import model_feature_set, model_mismatch, trained_head_files+from products.signals.backend.ranking.features import FEATURE_SETS+from products.signals.backend.ranking.model_contract import model_mismatch, trained_head_filesfrom products.signals.backend.ranking.overrides import read_ranking_overridesfrom products.signals.backend.ranking.serving_manifest import (- DEFAULT_MODEL_KIND,ServingManifest,model_key,serving_manifest_key,@@ -75,7 +53,6 @@ from products.signals.backend.ranking.serving_manifest import ()from products.signals.dags.inbox_ranking.common import (DATASET_VERSION,- PARQUET_PART_NAME,S3_BUCKET_ENV,RunContext,dataset_bucket,@@ -87,29 +64,14 @@ from products.signals.dags.inbox_ranking.common import (s3_client,serving_mirror_storage,skip_unconfigured,- write_parquet,-)-from products.signals.dags.inbox_ranking.consent import training_consent_team_ids-from products.signals.dags.inbox_ranking.dataset.dag import (- EMBEDDINGS_TABLE,- LABELS_TABLE,- PROJECT_VECTORS_TABLE,- STATE_TABLE,)+from products.signals.dags.inbox_ranking.dataset.dag import LABELS_TABLE, PROJECT_VECTORS_TABLE, STATE_TABLEfrom products.signals.dags.inbox_ranking.training.examples import (BASE_STATE_COLUMNS,PROVENANCE_LABEL_COLUMNS,PROVENANCE_STATE_COLUMNS,- ConsentExclusion,- HeadExamples,Snapshot,assemble_snapshot,- birth_day_positives,- build_head_examples,- drop_without_training_consent,- example_columns,- point_in_time_mask,- reports_missing_birth_snapshot,)from products.signals.dags.inbox_ranking.training.heads import (ACTION_LABEL_COLUMNS,@@ -118,19 +80,10 @@ from products.signals.dags.inbox_ranking.training.heads import (HEADS_BY_NAME,)from products.signals.dags.inbox_ranking.training.promotion import decide_promotion-from products.signals.dags.inbox_ranking.training.recipe import (- RECIPE_HASH_KEY,- model_recipe,- recipe_mismatch,- with_recipe,-)+from products.signals.dags.inbox_ranking.training.recipe import RECIPE_HASH_KEY, recipe_mismatchfrom products.signals.dags.inbox_ranking.training.serving import FamilyModels, ManifestDecision, compose_manifestfrom products.signals.dags.inbox_ranking.training.telemetry import (- HeadExampleCounts,- candidate_events,capture_training_events,- examples_events,- holdout_calibration_events,promotion_event,serving_manifest_event,unseen_calibration_events,@@ -138,22 +91,13 @@ from products.signals.dags.inbox_ranking.training.telemetry import (unseen_head_graded_events,unseen_report_graded_events,)-from products.signals.dags.inbox_ranking.training.train import (- XGB_PARAMS,- HoldoutGrade,- TrainedHead,- booster_holdout_grade,- holdout_calibration_rows,- train_head,-)+from products.signals.dags.inbox_ranking.training.train import HoldoutGradefrom products.signals.dags.inbox_ranking.training.unseen import (- MAIN_LOCATION,MODEL_FAMILIES,SERVED_SCORES_TABLE,HeadGrade,ModelFamily,calibration_rows,- families_trained_in,graded_rows,head_grades,missing_label_columns,@@ -165,7 +109,6 @@ from products.signals.dags.inbox_ranking.training.unseen import (if TYPE_CHECKING:from posthog.storage.object_storage import ObjectStorageClient-EXAMPLES_TABLE = "inbox_ranking_training_examples"MODELS_TABLE = "inbox_ranking_models"CHAMPION_FILE = "champion.json"SERVED_FILE = "served.json"@@ -207,12 +150,6 @@ COMMON_ASSET_KWARGS: dict[str, Any] = {}-def examples_object_key(prefix: str, feature_set_name: str, partition_key: str) -> str:- """One examples object per feature set: two sets carry different feature columns, so they- cannot share a Parquet, and a family trains on the object of the set it declares."""- return f"{prefix}/{EXAMPLES_TABLE}/{DATASET_VERSION}/{feature_set_name}/dt={partition_key}/{PARQUET_PART_NAME}"--def model_object_key(prefix: str, model_name: str, partition_key: str, filename: str) -> str:return f"{prefix}/{MODELS_TABLE}/{DATASET_VERSION}/{model_name}/dt={partition_key}/{filename}"@@ -309,11 +246,6 @@ def check_recipe(raise dagster.Failure(description=mismatch, allow_retries=False)-def booster_recipe(feature_set: FeatureSet) -> dict[str, Any]:- """The recipe of a per-head XGBoost family fit on `feature_set`."""- return model_recipe(model_kind=DEFAULT_MODEL_KIND, feature_set=feature_set, heads=HEADS, learner_params=XGB_PARAMS)--def snapshot_dates(partition_key: str, lookback_days: int) -> list[datetime.date]:end = datetime.date.fromisoformat(partition_key)return [end - datetime.timedelta(days=offset) for offset in range(lookback_days, -1, -1)]@@ -344,82 +276,7 @@ def load_snapshots(return snapshots-# The dt=D snapshot table behind each embedding side input, one table per rendering.-_EXTRA_SNAPSHOT_TABLES: Mapping[str, str] = MappingProxyType(- {- REPORT_EMBEDDINGS_EXTRA: EMBEDDINGS_TABLE,- }-)---def embeddings_extras(- context: dagster.AssetExecutionContext,- client,- bucket: str,- prefix: str,- partition_key: str,- extras_keys: Sequence[str],-) -> Extras:- """The dt=D vectors for `extras_keys`, each indexed by report_id and read from its own snapshot.-- One snapshot serves every moment of the run, and each vector carries the moment it landed, so a- moment can only take a vector that already existed for it. A report is re-embedded whenever its- text changes, so the latest vector is often newer than an earlier moment; the set drops those- rather than dressing later text as earlier state. Reading one snapshot per day of the lookback- would recover the superseded vectors, at the cost of pulling a fleet-wide vector table across- the network once per day of the window.-- A **missing** snapshot is not a failure, and not an empty side input either: the key is left- out, so the caller skips only the sets that read it. Rebuilding one of those from nothing would- strip the family's partition. A snapshot that **exists** and carries no usable vector is the- other case, and keeps its key: no row is then buildable, the family's heads read as unfit, and- the ordinary thin-input path applies. The two cannot be folded together, because the first must- leave a partition alone and the second is a day the family genuinely has nothing to fit.- """- extras: dict[str, pd.DataFrame] = {}- for extras_key in extras_keys:- table_name = _EXTRA_SNAPSHOT_TABLES[extras_key]- table = read_parquet_if_exists(- client,- bucket,- partition_object_key(prefix, table_name, partition_key),- columns=["report_id", EMBEDDING_COLUMN, EMBEDDING_INSERTED_AT_COLUMN],- )- if table is None:- context.log.warning(- f"no {table_name} snapshot for dt={partition_key}; the {extras_key} side input is unavailable"- )- continue- vectors = table.to_pandas().set_index("report_id")- # `reindex` refuses a duplicated index, and one report is one document per rendering.- vectors = vectors[~vectors.index.duplicated()]- extras[extras_key] = vectors- return extras---# Parquet schema metadata key of the examples object that carries each head's example window.-EXAMPLE_WINDOWS_METADATA_KEY = b"inbox_ranking_example_windows"---def examples_table(- examples: pd.DataFrame, feature_set: FeatureSet, windows: Mapping[str, Mapping[str, object]] | None = None-) -> pa.Table:- table = pa.Table.from_pandas(examples[list(example_columns(feature_set))], preserve_index=False)- if windows is None:- return table- return table.replace_schema_metadata(- {**(table.schema.metadata or {}), EXAMPLE_WINDOWS_METADATA_KEY: json.dumps(windows).encode()}- )---def example_windows(table: pa.Table) -> dict[str, dict[str, object]]:- """Each head's example window as the examples asset wrote it. An object written before the- window was recorded carries none, so its heads record no window."""- raw = (table.schema.metadata or {}).get(EXAMPLE_WINDOWS_METADATA_KEY)- return json.loads(raw) if raw else {}---# The examples for dt=D read every snapshot back to D-lookback. The default same-partition mapping+# The `nn` examples for dt=D read every snapshot back to D-lookback. The default same-partition mapping# would let an asset backfill run examples(D) as soon as state(D) and labels(D) exist, while the# older days it needs are still pending. Partitions before the label epoch have no upstream to map to._LOOKBACK_MAPPING = dagster.TimeWindowPartitionMapping(@@ -429,431 +286,6 @@ _LOOKBACK_MAPPING = dagster.TimeWindowPartitionMapping()- name=EXAMPLES_TABLE,- deps=[- dagster.AssetDep(STATE_TABLE, partition_mapping=_LOOKBACK_MAPPING),- dagster.AssetDep(LABELS_TABLE, partition_mapping=_LOOKBACK_MAPPING),- # Only dt=D: `embeddings_extras` explains why one snapshot serves the whole window.- EMBEDDINGS_TABLE,- ],- **COMMON_ASSET_KWARGS,-)-def inbox_ranking_training_examples(context: dagster.AssetExecutionContext) -> None:- if skip_unconfigured(context):- return- partition_key = context.partition_key- bucket, prefix, client = dataset_bucket(), settings.INBOX_RANKING_DATASET_S3_PREFIX, s3_client()-- dates = snapshot_dates(partition_key, settings.INBOX_RANKING_TRAINING_LOOKBACK_DAYS)- snapshots = load_snapshots(client, bucket, prefix, dates, required=dates[-1])- context.log.info(f"{len(snapshots)} of {len(dates)} snapshots present")- snapshots, excluded = drop_without_training_consent(snapshots, training_consent_team_ids())- if excluded.reports:- context.log.info(f"{excluded.reports} reports from {excluded.teams} teams without AI training consent excluded")- backfilled_rows = sum(int((~point_in_time_mask(snap.state, snap.date)).sum()) for snap in snapshots.values())- if backfilled_rows:- context.log.warning(f"{backfilled_rows} state rows read after the snapshot window are excluded (backfill)")-- # A gap in the partitions is silent at the birth grain: it removes every report born that day- # from every head, rather than thinning the rows of a report that survives.- unreachable_reports = reports_missing_birth_snapshot(snapshots, dates)- if unreachable_reports:- context.log.warning(f"{unreachable_reports} reports born inside the window have no birth-day snapshot")-- metadata: dict[str, dagster.MetadataValue] = {- "snapshots": dagster.MetadataValue.int(len(snapshots)),- "backfilled_state_rows_excluded": dagster.MetadataValue.int(backfilled_rows),- "reports_missing_birth_snapshot": dagster.MetadataValue.int(unreachable_reports),- "excluded_no_training_consent_reports": dagster.MetadataValue.int(excluded.reports),- "excluded_no_training_consent_teams": dagster.MetadataValue.int(excluded.teams),- }- # Only the sets this location trains on. A set only `signals_ml` reads would cost this job a- # second 1536-column examples object that nothing reads.- main_sets = {family.feature_set.name: family.feature_set for family in families_trained_in(MAIN_LOCATION)}- for feature_set in main_sets.values():- metadata |= _examples_for_set(- context, client, bucket, prefix, partition_key, feature_set, snapshots, backfilled_rows, excluded- )- context.add_output_metadata(metadata)---def _examples_for_set(- context: dagster.AssetExecutionContext,- client,- bucket: str,- prefix: str,- partition_key: str,- feature_set: FeatureSet,- snapshots: Mapping[datetime.date, Snapshot],- backfilled_rows: int,- excluded: ConsentExclusion,-) -> dict[str, dagster.MetadataValue]:- """One feature set's examples for the partition, side inputs included, and its asset metadata.-- The side inputs are read and released inside this call, so the next set's snapshot is read- after this set's is gone. Each embeddings snapshot carries a vector per live report, and a- name still bound in the caller's loop would hold the previous rendering through the next read,- which is what makes the peak scale with the number of families.- """- extras = embeddings_extras(context, client, bucket, prefix, partition_key, feature_set.extras_keys)- missing = feature_set.missing_extras(extras)- # Rebuilding a set from a missing side input would write an empty examples object, which makes- # the next candidate run train nothing, write metadata with no heads, and delete the boosters- # this partition already holds. A champion pointer can name that version, so the partition- # keeps what it has instead.- if missing:- context.log.warning(f"{feature_set.name} examples not rebuilt for dt={partition_key}: no {', '.join(missing)}")- return {f"{feature_set.name}_skipped": dagster.MetadataValue.bool(True)}- return _write_examples(- context, client, bucket, prefix, partition_key, feature_set, snapshots, backfilled_rows, excluded, extras- )---def _write_examples(- context: dagster.AssetExecutionContext,- client,- bucket: str,- prefix: str,- partition_key: str,- feature_set: FeatureSet,- snapshots: Mapping[datetime.date, Snapshot],- backfilled_rows: int,- excluded: ConsentExclusion,- extras: Extras,-) -> dict[str, dagster.MetadataValue]:- """One feature set's examples for the partition, as its own object and its own events, with- the asset metadata to record for it."""- built: dict[str, HeadExamples] = {- head.name: build_head_examples(snapshots, head, feature_set, extras) for head in HEADS- }- per_head = {name: head_examples.examples for name, head_examples in built.items()}- examples = (- pd.concat(per_head.values(), ignore_index=True)- if per_head- else pd.DataFrame(columns=list(example_columns(feature_set)))- )-- key = examples_object_key(prefix, feature_set.name, partition_key)- windows = {name: head_examples.window() for name, head_examples in built.items()}- write_parquet(client, bucket, key, examples_table(examples, feature_set, windows), snapshot_date=partition_key)- counts = {- name: HeadExampleCounts(- rows=len(head_examples.examples),- positives=int(head_examples.examples["label"].sum()),- birth_day_positives=birth_day_positives(head_examples.examples),- example_window_start=head_examples.window_start,- example_cap_bound=head_examples.cap_bound,- pairs_skipped_missing_label_columns=head_examples.pairs_skipped_missing_label_columns,- )- for name, head_examples in built.items()- }- metadata: dict[str, dagster.MetadataValue] = {- f"{feature_set.name}_rows": dagster.MetadataValue.int(len(examples)),- **{- f"{feature_set.name}_{name}_rows": dagster.MetadataValue.int(head_counts.rows)- for name, head_counts in counts.items()- },- **{- f"{feature_set.name}_{name}_positives": dagster.MetadataValue.int(head_counts.positives)- for name, head_counts in counts.items()- },- **{- f"{feature_set.name}_{name}_birth_day_positives": dagster.MetadataValue.int(head_counts.birth_day_positives)- for name, head_counts in counts.items()- },- **{- f"{feature_set.name}_{name}_pairs_skipped_missing_label_columns": dagster.MetadataValue.int(- head_counts.pairs_skipped_missing_label_columns- )- for name, head_counts in counts.items()- },- f"{feature_set.name}_s3_key": dagster.MetadataValue.text(f"s3://{bucket}/{key}"),- }- capture_training_events(- context,- partition_key,- examples_events(- partition_key=partition_key,- run_id=context.run.run_id,- feature_set=feature_set.name,- snapshots=len(snapshots),- backfilled_rows=backfilled_rows,- excluded=excluded,- per_head=counts,- ),- )- return metadata---def candidate_metadata(- partition_key: str,- trained: list[TrainedHead],- *,- model_name: str,- feature_set: FeatureSet,- skipped: list[str],- trained_at: datetime.datetime,- run_id: str,- windows: Mapping[str, Mapping[str, object]] | None = None,-) -> dict[str, Any]:- metadata = {- "model_name": model_name,- "model_version": partition_key,- "dataset_version": DATASET_VERSION,- # The learner a model store loads the booster with. Every family is per-head XGBoost today;- # the field exists so the serving manifest reads it from the source rather than assuming it.- "model_kind": DEFAULT_MODEL_KIND,- # The feature universe this model was fit on. The grader checks the model against this set- # rather than against one global contract, so a second family is not rejected for reading- # different features.- "feature_set": feature_set.name,- "feature_schema_version": feature_set.schema_version,- "feature_names": list(feature_set.feature_names),- "trained_at": trained_at.isoformat(),- "run_id": run_id,- "lookback_days": settings.INBOX_RANKING_TRAINING_LOOKBACK_DAYS,- "holdout_days": settings.INBOX_RANKING_TRAINING_HOLDOUT_DAYS,- "xgb_params": XGB_PARAMS,- # Per head: the cohort a probability is conditioned on, its horizon, and the report-creation- # window its examples cover, so a stored score says what it means without the code.- "heads": [- head.metrics.as_dict()- | {- "cohort": HEADS_BY_NAME[head.head].cohort.__name__,- "horizon_days": HEADS_BY_NAME[head.head].horizon_days,- "example_window_start": (windows or {}).get(head.head, {}).get("example_window_start"),- "example_cap_bound": (windows or {}).get(head.head, {}).get("example_cap_bound"),- "file": f"{head.head}.ubj",- "holdout_file": f"{head.head}.holdout.ubj" if head.holdout_booster_ubj is not None else None,- }- for head in trained- ],- # Heads with nothing to fit on this partition; recorded so the per-head series has no gap.- "skipped_heads": skipped,- }- return with_recipe(metadata, booster_recipe(feature_set))---def paired_champion_grades(- client,- bucket: str,- prefix: str,- champion: dict[str, Any],- examples: pd.DataFrame,- *,- feature_set: FeatureSet,- holdout_days: int,-) -> dict[str, HoldoutGrade]:- """The champion's readable heads graded on the candidate's holdout, through the champion's saved- holdout boosters. Heads without a saved holdout booster are left out and fall back to the- champion's stored AUC in `decide_promotion`."""- grades: dict[str, HoldoutGrade] = {}- for entry in champion.get("heads", []):- head = HEADS_BY_NAME.get(entry.get("head"))- if head is None or not entry.get("readable") or not entry.get("holdout_file"):- continue- body = _read_bytes_if_exists(- client,- bucket,- model_object_key(prefix, champion["model_name"], champion["model_version"], entry["holdout_file"]),- )- if body is None:- continue- grade = booster_holdout_grade(- body, examples, head, feature_names=feature_set.feature_names, holdout_days=holdout_days- )- if grade is not None:- grades[head.name] = grade- return grades--[email protected](name="inbox_ranking_model_candidate", deps=[EXAMPLES_TABLE], **COMMON_ASSET_KWARGS)-def inbox_ranking_model_candidate(context: dagster.AssetExecutionContext) -> None:- if skip_unconfigured(context):- return- partition_key = context.partition_key- bucket, prefix, client = dataset_bucket(), settings.INBOX_RANKING_DATASET_S3_PREFIX, s3_client()-- # Every family's recipe is checked before any family trains, so a refused run writes nothing.- for family in families_trained_in(MAIN_LOCATION):- check_recipe(client, bucket, prefix, family.name, partition_key, booster_recipe(family.feature_set))- metadata: dict[str, dagster.MetadataValue] = {}- for family in families_trained_in(MAIN_LOCATION):- metadata |= _train_candidate(context, client, bucket, prefix, partition_key, family)- context.add_output_metadata(metadata)---def _train_candidate(- context: dagster.AssetExecutionContext,- client,- bucket: str,- prefix: str,- partition_key: str,- family: ModelFamily,-) -> dict[str, dagster.MetadataValue]:- """One family's candidate: its heads fit on the examples of the feature set it declares, its- boosters and `metadata.json` under its own prefix, and the asset metadata to record for it.-- Each family's examples object is read and released in turn rather than all of them up front:- one family's object carries a column per embedding dimension, so holding two at once doubles- the widest thing this asset touches.- """- feature_set = family.feature_set- table = read_parquet_if_exists(client, bucket, examples_object_key(prefix, feature_set.name, partition_key))- # The examples asset skips a set whose side input was unavailable, so its object can be absent.- # Training on nothing would replace this partition's boosters with an empty candidate and leave- # a champion pointer naming deleted files, so the family keeps this partition as it stands.- if table is None:- context.log.warning(- f"no {feature_set.name} examples for dt={partition_key}; {family.name} keeps this partition as it stands"- )- return {f"{family.name}_skipped": dagster.MetadataValue.bool(True)}- windows = example_windows(table)- examples = table.to_pandas()- trained: list[TrainedHead] = []- skipped: list[str] = []- for head in HEADS:- result = train_head(- examples,- head,- feature_names=feature_set.feature_names,- holdout_days=settings.INBOX_RANKING_TRAINING_HOLDOUT_DAYS,- )- if result is None:- context.log.warning(f"{family.name} {head.name}: nothing to fit, skipped")- skipped.append(head.name)- continue- context.log.info(f"{family.name} {head.name}: {result.metrics.as_dict()}")- trained.append(result)-- written: set[str] = set()- for model in trained:- files = {f"{model.head}.ubj": model.booster_ubj}- if model.holdout_booster_ubj is not None:- files[f"{model.head}.holdout.ubj"] = model.holdout_booster_ubj- for filename, body in files.items():- key = model_object_key(prefix, family.name, partition_key, filename)- client.put_object(Bucket=bucket, Key=key, Body=body, ContentType="application/octet-stream")- written.add(key)- metadata = candidate_metadata(- partition_key,- trained,- model_name=family.name,- feature_set=feature_set,- skipped=skipped,- trained_at=datetime.datetime.now(datetime.UTC),- run_id=context.run.run_id,- windows=windows,- )- metadata_key = model_object_key(prefix, family.name, partition_key, METADATA_FILE)- _put_json(client, bucket, metadata_key, metadata)- written.add(metadata_key)- # A re-run that trains fewer heads must not leave the previous run's files behind.- stale = _delete_other_objects(client, bucket, model_object_key(prefix, family.name, partition_key, ""), written)- if stale:- context.log.warning(- f"removed {len(stale)} stale {family.name} objects from a previous run of dt={partition_key}"- )- capture_training_events(- context,- partition_key,- [- *candidate_events(metadata),- *holdout_calibration_events(- partition_key=partition_key,- run_id=context.run.run_id,- model_name=family.name,- rows=holdout_calibration_rows(trained),- ),- ],- )- return {- f"{family.name}_stale_objects_removed": dagster.MetadataValue.int(len(stale)),- f"{family.name}_heads_trained": dagster.MetadataValue.int(len(trained)),- f"{family.name}_heads_readable": dagster.MetadataValue.int(sum(1 for head in trained if head.metrics.readable)),- **{- f"{family.name}_{head.head}_holdout_auc": dagster.MetadataValue.float(head.metrics.holdout_auc)- for head in trained- if head.metrics.holdout_auc is not None- },- **{- f"{family.name}_{head.head}_{name}": (- dagster.MetadataValue.int(value) if isinstance(value, int) else dagster.MetadataValue.float(value)- )- for head in trained- for name, value in head.metrics.holdout_within_team.as_dict(prefix="holdout_").items()- if value is not None- },- f"{family.name}_s3_key": dagster.MetadataValue.text(f"s3://{bucket}/{metadata_key}"),- }--[email protected](name="inbox_ranking_model_champion", deps=["inbox_ranking_model_candidate"], **COMMON_ASSET_KWARGS)-def inbox_ranking_model_champion(context: dagster.AssetExecutionContext) -> None:- if skip_unconfigured(context):- return- partition_key = context.partition_key- bucket, prefix, client = dataset_bucket(), settings.INBOX_RANKING_DATASET_S3_PREFIX, s3_client()-- metadata: dict[str, dagster.MetadataValue] = {}- decided = 0- for family in families_trained_in(MAIN_LOCATION):- family_metadata = _decide_champion(context, client, bucket, prefix, partition_key, family)- if family_metadata is None:- continue- decided += 1- metadata |= family_metadata- # Every family missing its candidate means the candidate asset wrote nothing at all, which is a- # broken partition rather than one family's gap.- if not decided:- raise dagster.Failure(f"no candidate metadata for dt={partition_key}")- context.add_output_metadata(metadata)---def _decide_champion(- context: dagster.AssetExecutionContext,- client,- bucket: str,- prefix: str,- partition_key: str,- family: ModelFamily,-) -> dict[str, dagster.MetadataValue] | None:- """One per-head XGBoost family's promotion decision against the served model of its name, or- None when the family has no candidate for the partition."""-- def grade_champion(champion: Mapping[str, Any]) -> dict[str, HoldoutGrade]:- champion_feature_set = model_feature_set(champion)- if champion_feature_set is None:- context.log.warning(- f"{family.name} champion {champion['model_version']} reads a feature set this build cannot produce"- )- return {}- # The champion is graded on the examples of its own feature set, the set the candidate- # shares: promotion stays inside a family. Without that object there is no shared- # holdout, so the rule falls back to the champion's stored AUC.- examples = read_parquet_if_exists(- client, bucket, examples_object_key(prefix, champion_feature_set.name, partition_key)- )- if examples is None:- context.log.warning(- f"no {champion_feature_set.name} examples for dt={partition_key}; "- f"the {family.name} champion is compared on its stored AUC"- )- return {}- return paired_champion_grades(- client,- bucket,- prefix,- dict(champion),- examples.to_pandas(),- feature_set=champion_feature_set,- holdout_days=settings.INBOX_RANKING_TRAINING_HOLDOUT_DAYS,- )-- return decide_family_champion(context, client, bucket, prefix, partition_key, family, grade_champion=grade_champion)--def served_incumbent(client, bucket: str, prefix: str, model_name: str) -> dict[str, Any] | None:"""The metadata of the served model when it has `model_name`, else None. A candidate is gradedonly against a model of its own name: two names read different features, so a paired grade@@ -971,7 +403,7 @@ class _ServingStore:self.storage.write(bucket=self.bucket, key=key, content=content, extras=extras or None)[email protected](name=SERVING_MANIFEST_ASSET, deps=["inbox_ranking_model_champion"], **COMMON_ASSET_KWARGS)[email protected](name=SERVING_MANIFEST_ASSET, deps=[PROJECT_VECTORS_TABLE], **COMMON_ASSET_KWARGS)def inbox_ranking_serving_manifest(context: dagster.AssetExecutionContext) -> None:"""Publish the models the scoring sweep serves, and the manifest naming them.@@ -1459,30 +891,20 @@ def inbox_ranking_unseen_graded(context: dagster.AssetExecutionContext) -> None:inbox_ranking_training_job = dagster.define_asset_job(name="inbox_ranking_training_job",selection=[- EXAMPLES_TABLE,- "inbox_ranking_model_candidate",- "inbox_ranking_model_champion",SERVING_MANIFEST_ASSET,SERVED_SCORES_TABLE,"inbox_ranking_unseen_graded",],tags={**owner_tags,- # The report-embeddings family fits 1536-column heads, and each head costs a fit per- # permutation draw on top of the two it ships, so the wall clock is now the trainer's- # rather than the ETL's.- "dagster/max_runtime": str(3 * 60 * 60),- # The examples asset holds every snapshot of the lookback window in pandas at once (state- # plus labels per day) before the per-head builders run, plus one rendering's vectors as- # the side input the embedding set being built reads, so the peak grows with the lookback- # and the inventory. It is one rendering at a time rather than one per family, so a further- # embedding family costs runtime and not peak. Growth should surface as a slow run, not an- # OOMKilled pod.+ # The job fits nothing: it copies models, reads one day of scoring events and grades the+ # scores objects back to the longest head horizon against one labels snapshot.+ "dagster/max_runtime": str(60 * 60),"dagster-k8s/config": {"container_config": {"resources": {- "requests": {"memory": "8Gi"},- "limits": {"memory": "24Gi"},+ "requests": {"memory": "2Gi"},+ "limits": {"memory": "8Gi"},}}},diff --git a/products/signals/dags/inbox_ranking/training/telemetry.py b/products/signals/dags/inbox_ranking/training/telemetry.pyindex 18c0219eee2d..93d943b111d1 100644--- a/products/signals/dags/inbox_ranking/training/telemetry.py+++ b/products/signals/dags/inbox_ranking/training/telemetry.py@@ -19,7 +19,6 @@ from posthog.ph_client import get_clientfrom products.signals.backend.ranking.serving_manifest import ServingManifestfrom products.signals.dags.inbox_ranking.common import RunContext, snapshot_bounds-from products.signals.dags.inbox_ranking.training.examples import ConsentExclusionfrom products.signals.dags.inbox_ranking.training.promotion import PromotionDecisionfrom products.signals.dags.inbox_ranking.training.unseen import CANDIDATE_ROLE, HeadGrade@@ -30,7 +29,6 @@ LOCAL_SUFFIX = "_local"LOCAL_DISTINCT_ID = f"{DISTINCT_ID}{LOCAL_SUFFIX}"LOCAL_ENVIRONMENT = "local"CANDIDATE_TRAINED_EVENT = "inbox_ranking_candidate_trained"-EXAMPLES_BUILT_EVENT = "inbox_ranking_examples_built"PROMOTION_DECIDED_EVENT = "inbox_ranking_promotion_decided"MODEL_PROMOTED_EVENT = "inbox_ranking_model_promoted"UNSEEN_HEAD_GRADED_EVENT = "inbox_ranking_unseen_head_graded"@@ -61,20 +59,6 @@ class TrainingEvent:properties: dict[str, object]-@frozen-class HeadExampleCounts:- rows: int- positives: int- # Of the positives, how many sit on their report's birth day: most outcomes land there, so a- # drop in this share is the first sign the birth-day rule stopped keeping them.- birth_day_positives: int- # The earliest report-creation day the head kept, and whether the row budget dropped older days,- # so a chart shows when the budget starts to cut history.- example_window_start: datetime.date | None- example_cap_bound: bool- pairs_skipped_missing_label_columns: int--def candidate_events(metadata: Mapping[str, Any]) -> list[TrainingEvent]:"""One event per head: the head's metrics plus the candidate context. A head the candidatecould not fit still gets an event (`trained` false, `readable` false), so a per-head alert sees@@ -103,47 +87,6 @@ def candidate_events(metadata: Mapping[str, Any]) -> list[TrainingEvent]:return [*trained, *skipped]-def examples_events(- *,- partition_key: str,- run_id: str,- feature_set: str,- snapshots: int,- backfilled_rows: int,- excluded: ConsentExclusion,- per_head: Mapping[str, HeadExampleCounts],-) -> list[TrainingEvent]:- """One event per head with its example and positive counts; the run-level counts repeat on each.-- Examples are per feature set, not per model family: every family on a set trains on one- Parquet, so `feature_set` is the dimension that separates two of these series.- """- return [- TrainingEvent(- event=EXAMPLES_BUILT_EVENT,- properties={- "model_version": partition_key,- "feature_set": feature_set,- "run_id": run_id,- "snapshots": snapshots,- "backfilled_state_rows_excluded": backfilled_rows,- "excluded_no_training_consent_reports": excluded.reports,- "excluded_no_training_consent_teams": excluded.teams,- "head": head,- "rows": counts.rows,- "positives": counts.positives,- "birth_day_positives": counts.birth_day_positives,- "example_window_start": counts.example_window_start.isoformat()- if counts.example_window_start- else None,- "example_cap_bound": counts.example_cap_bound,- "pairs_skipped_missing_label_columns": counts.pairs_skipped_missing_label_columns,- },- )- for head, counts in per_head.items()- ]--def promotion_event(*,partition_key: str,diff --git a/products/signals/dags/inbox_ranking/training/train.py b/products/signals/dags/inbox_ranking/training/train.pyindex 5e9ae18059c1..57307208e173 100644--- a/products/signals/dags/inbox_ranking/training/train.py+++ b/products/signals/dags/inbox_ranking/training/train.py@@ -1,23 +1,16 @@-"""Per-head XGBoost training over the scoring-moment examples.--Deliberately plain: fixed hyperparameters, a time-based holdout by report, AUC plus a-label-permutation null per head. The feature names come from the caller's feature set, so one-learner serves every family; the booster is saved as UBJSON bytes with those names, so the serving-side can assert them against the set the model declares.+"""Per-head grading shared by every learner: holdout metrics, the label-permutation null, and the+readable rule the promotion gate reads."""from collections.abc import Sequenceimport numpy as np-import pandas as pd-import xgboost as xgbfrom sklearn.metrics import average_precision_score, log_loss, roc_auc_scorefrom posthog.dataclasses import frozenfrom products.signals.dags.inbox_ranking.training.calibration import (CalibrationBucket,- bucket_rows,calibration_buckets,expected_calibration_error,)@@ -26,27 +19,16 @@ from products.signals.dags.inbox_ranking.training.classification import (ClassificationMetrics,classification_metrics,)-from products.signals.dags.inbox_ranking.training.examples import holdout_maskfrom products.signals.dags.inbox_ranking.training.heads import Headfrom products.signals.dags.inbox_ranking.training.within_team import NO_TEAMS, WithinTeamAuc, within_team_auc-XGB_PARAMS: dict[str, object] = {- "n_estimators": 200,- "max_depth": 4,- "learning_rate": 0.05,- "subsample": 0.8,- "colsample_bytree": 0.8,- "min_child_weight": 5,- "eval_metric": "auc",- "n_jobs": 2,-}# A readable head must also clear its own permutation null by this much; below it the AUC is# within the noise of this many positives.NULL_MARGIN = 0.05# Each null is a full fit on permuted labels, so this multiplies the training cost per head. One-# draw swings with whichever feature the permuted fit happened to lean on (age_hours dominates),-# which put single-draw nulls at 0.38 on a 35k-row holdout; the mean of a few draws is the number-# the margin is compared against, and the spread is reported so a wide band is visible.+# draw swings with whichever feature the permuted fit happened to lean on, so the mean of a few+# draws is the number the margin is compared against, and the spread is reported so a wide band is+# visible.NULL_PERMUTATIONS = 3@@ -110,43 +92,12 @@ class HeadMetrics:}-@frozen-class TrainedHead:- head: str- booster_ubj: bytes- # The train-only fit that produced the holdout metrics, kept so a later candidate can grade this- # model on its own holdout; None when there were no holdout rows.- holdout_booster_ubj: bytes | None- metrics: HeadMetrics- # Beside the metrics rather than inside them: those go on one event per head, a table needs one- # event per bucket.- calibration: tuple[CalibrationBucket, ...]---def _ubj(model: xgb.XGBClassifier) -> bytes:- return bytes(model.get_booster().save_raw("ubj"))---def _fit(x: pd.DataFrame, y: np.ndarray, seed: int) -> xgb.XGBClassifier:- model = xgb.XGBClassifier(**XGB_PARAMS, random_state=seed)- model.fit(x, y, verbose=False)- return model--def _auc(y: np.ndarray, scores: np.ndarray) -> float | None:if y.sum() == 0 or y.sum() == len(y):return Nonereturn float(roc_auc_score(y, scores))-def _team_ids(rows: pd.DataFrame) -> np.ndarray:- """Each row's team, None where unknown. An examples object written before the column existed- has no team, so its within-team fields stay empty."""- if "team_id" not in rows:- return np.full(len(rows), None, dtype=object)- return rows["team_id"].to_numpy(dtype=object)--def _head_readable(holdout_auc: float | None, null_auc: float | None, holdout_positives: int, *, min_positives: int) -> bool:@@ -207,62 +158,6 @@ def head_metrics(return metrics, calibration-def train_head(- examples: pd.DataFrame, head: Head, *, feature_names: Sequence[str], holdout_days: int, seed: int = 0-) -> TrainedHead | None:- """Fit one head; None when there is nothing to fit (no positive or no negative in train)."""- rows = examples[examples["head"] == head.name]- if rows.empty:- return None- test = holdout_mask(rows, holdout_days).to_numpy()- # float32 is what XGBoost bins internally, and it halves the largest allocation of a wide set.- x = rows[list(feature_names)].astype(np.float32)- y = rows["label"].to_numpy(dtype=int)- x_train, y_train, x_test, y_test = x[~test], y[~test], x[test], y[test]- if len(y_train) == 0 or y_train.sum() == 0 or y_train.sum() == len(y_train):- return None-- model = _fit(x_train, y_train, seed)- holdout_scores = model.predict_proba(x_test)[:, 1] if len(y_test) else np.array([])- holdout_auc = _auc(y_test, holdout_scores) if len(y_test) else None-- null_aucs: list[float] = []- if holdout_auc is not None:- rng = np.random.default_rng(seed)- null_aucs = [- auc- for draw in range(NULL_PERMUTATIONS)- if (auc := _auc(y_test, _fit(x_train, rng.permutation(y_train), seed + draw).predict_proba(x_test)[:, 1]))- is not None- ]-- metrics, calibration = head_metrics(- head,- y_train=y_train,- train_scores=model.predict_proba(x_train)[:, 1],- y_test=y_test,- holdout_scores=holdout_scores,- null_aucs=null_aucs,- refit_positive_rate=float(y.mean()),- holdout_team_ids=_team_ids(rows)[test],- )- # Refit on everything before shipping: the holdout only exists to grade the recipe.- final = _fit(x, y, seed)- return TrainedHead(- head=head.name,- booster_ubj=_ubj(final),- holdout_booster_ubj=_ubj(model) if len(y_test) else None,- metrics=metrics,- calibration=calibration,- )---def holdout_calibration_rows(trained: Sequence[TrainedHead]) -> list[dict[str, object]]:- """One dict per (head, decile) of the candidate's holdout, in the shape the unseen calibration- rows use, so one insight can read the holdout line and the unseen line side by side."""- return [row for model in trained for row in bucket_rows(model.calibration, {"head": model.head})]--@frozenclass HoldoutGrade:auc: float | None@@ -277,27 +172,3 @@ def holdout_grade(y: np.ndarray, scores: np.ndarray) -> HoldoutGrade:expected_calibration_error=expected_calibration_error(calibration_buckets(y.astype(bool), scores)),positives=int(y.sum()),)---def booster_holdout_grade(- booster_ubj: bytes, examples: pd.DataFrame, head: Head, *, feature_names: Sequence[str], holdout_days: int-) -> HoldoutGrade | None:- """AUC and calibration error of a saved booster on `head`'s holdout rows of `examples`: the same- rows `train_head` grades a candidate on, so a champion and a candidate can be compared on one set."""- rows = examples[examples["head"] == head.name]- if rows.empty:- return None- test = holdout_mask(rows, holdout_days).to_numpy()- if not test.any():- return None- booster = xgb.Booster()- booster.load_model(bytearray(booster_ubj))- # The booster's own names, not the set's: a champion trained before an additive schema bump is- # still scorable on its subset. A name the examples lack means the schemas are incompatible,- # and the caller falls back to the stored AUC.- names = list(booster.feature_names or feature_names)- if any(name not in rows for name in names):- return None- x = rows.loc[test, names].astype(np.float32)- y = rows.loc[test, "label"].to_numpy(dtype=int)- return holdout_grade(y, booster.predict(xgb.DMatrix(x, feature_names=names)))diff --git a/products/signals/dags/inbox_ranking/training/unseen.py b/products/signals/dags/inbox_ranking/training/unseen.pyindex 905f941f7ef4..65783ca4129c 100644--- a/products/signals/dags/inbox_ranking/training/unseen.py+++ b/products/signals/dags/inbox_ranking/training/unseen.py@@ -61,7 +61,6 @@ CANDIDATE_ROLE = "candidate"# it was fit on. Both are in the identity, so two families trained on one day stay apart.# A recipe change takes a new name (`training/recipe.py`). The versions of an earlier name stay# servable by key, and their online grades keep their own series.-REPORT_EMBEDDINGS_MODEL_NAME = "report_embeddings_v2"NN_MODEL_NAME = "nn_v3"NN_WITH_PRIO_MODEL_NAME = "nn_with_prio_v2"# The Dagster code locations that train a family: `signals` on the main image, and `signals_ml` on@@ -71,7 +70,7 @@ ML_LOCATION = "signals_ml"# Families this build no longer trains. Saved scores objects still hold their rows, so every reader# drops them through `with_model_names`: the grades stop with the family, and the rewrite guard does# not require rows that no re-run can produce.-RETIRED_MODEL_NAMES = frozenset({"tabular_xgb", "title_embeddings"})+RETIRED_MODEL_NAMES = frozenset({"tabular_xgb", "title_embeddings", "report_embeddings_v2"})# A shuffle plus one AUC rather than a refit, so this sits far above the trainer's NULL_PERMUTATIONS.NULL_PERMUTATIONS = 25@@ -119,11 +118,10 @@ class ModelFamily:# family with no metadata for the day is skipped, so an entry can be added here before its trainer# writes its first candidate, and a family that fails costs its own series rather than every one.# `nn` is one multi-head network over the report vector and the team's project vectors. It needs-# torch, so the `signals_ml` location trains it and the main location never fits it. It declares+# torch, so the `signals_ml` location trains it. The main location fits no family. `nn` declares# the report embeddings set, because that set builds its report input in training and at serving.# `nn_with_prio_v2` is the same network with the report's priority levels as extra report inputs.MODEL_FAMILIES: tuple[ModelFamily, ...] = (- ModelFamily(name=REPORT_EMBEDDINGS_MODEL_NAME, feature_set=REPORT_EMBEDDINGS_FEATURE_SET),ModelFamily(name=NN_MODEL_NAME, feature_set=REPORT_EMBEDDINGS_FEATURE_SET, learner=ONNX_MODEL_KIND, location=ML_LOCATION),