Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(signals): add an isolated experiment lane for inbox ranking nn models (#121614)

posthog[bot]307f476e68a8
diff --git a/posthog/dags/locations/signals_ml.py b/posthog/dags/locations/signals_ml.py
index 72d4f616e649..2fa3d58d9f68 100644
--- a/posthog/dags/locations/signals_ml.py
+++ b/posthog/dags/locations/signals_ml.py
@@ -1,7 +1,10 @@
import dagster
from products.signals.dags.inbox_ranking.common import is_inbox_ranking_registered
-from products.signals.dags.inbox_ranking.nn import dag as inbox_ranking_nn
+from products.signals.dags.inbox_ranking.nn import (
+ dag as inbox_ranking_nn,
+ experiments as inbox_ranking_nn_experiments,
+)
from . import loggers, resources
@@ -16,9 +19,16 @@ if is_inbox_ranking_registered():
inbox_ranking_nn.inbox_ranking_nn_model_champion,
inbox_ranking_nn.inbox_ranking_nn_with_prio_model_candidate,
inbox_ranking_nn.inbox_ranking_nn_with_prio_model_champion,
+ inbox_ranking_nn_experiments.inbox_ranking_nn_experiment_candidates,
+ ],
+ jobs=[
+ inbox_ranking_nn.inbox_ranking_nn_training_job,
+ inbox_ranking_nn_experiments.inbox_ranking_nn_experiments_job,
+ ],
+ schedules=[
+ inbox_ranking_nn.inbox_ranking_nn_training_schedule,
+ inbox_ranking_nn_experiments.inbox_ranking_nn_experiments_schedule,
],
- jobs=[inbox_ranking_nn.inbox_ranking_nn_training_job],
- schedules=[inbox_ranking_nn.inbox_ranking_nn_training_schedule],
loggers=loggers,
resources=resources,
)
diff --git a/products/signals/dags/inbox_ranking/AGENTS.md b/products/signals/dags/inbox_ranking/AGENTS.md
index 9065800335d5..edefdcf77ce3 100644
--- a/products/signals/dags/inbox_ranking/AGENTS.md
+++ b/products/signals/dags/inbox_ranking/AGENTS.md
@@ -26,7 +26,7 @@ Read `README.md` first for what the dataset is and how partitions behave. This f
- The model is read in three layers: **training** (train metrics), **test** (the random-by-report holdout, which drives promotion) and **online** (`inbox_ranking_served_scores`, the scoring sweep's live scores of each day's newborns, graded by `inbox_ranking_unseen_graded` at each head's horizon). Do not add an offline rescoring of the newborn pool beside the online layer: the sweep already scores it with every model key the manifest names.
- `served_score_rows` keeps the earliest scored event per (report, model key) and takes `model_role` from the event's roles. Grades stay per `(model_name, model_version, model_role)` and are never pooled across versions. A re-run that reads no event is refused when the partition already holds rows, and so is one that reads fewer families than the object holds, because the newborn state snapshot behind those rows ages out and the dt=D+horizon grade reads them.
- Every scored row carries `head_readable`, and an unreadable head is still graded: a head that misses `min_holdout_positives` on one day's holdout still gets a number from the pooled online grade. Keep the promotion gate reading `readable` from the candidate metadata: an unreadable head must not decide a promotion.
-- `model_name` is the model recipe, and every model object, scored row, grade and event carries it. A family is an entry in `MODEL_FAMILIES` (`training/unseen.py`) naming the feature set it is fit on, its `learner` and the `location` that trains it; the `signals_ml` location trains every family (`nn/dag.py`), and the grader and the events need no change to gain a family. `families_trained_in(MAIN_LOCATION)` is empty: the main location fits no model. The per-head XGBoost families are retired (`RETIRED_MODEL_NAMES`), and the sweep keeps the `xgboost` handler so an old version stays servable by key for a rollback. A candidate is graded only against the served model of its own `model_name`; the served pointer itself can name any family. Do not fold the family into `model_version`, which the dashboard filters as a date.
+- `model_name` is the model recipe, and every model object, scored row, grade and event carries it. A family is an entry in `MODEL_FAMILIES` (`training/families.py`) naming the feature set it is fit on, its `learner` and the `location` that trains it; the `signals_ml` location trains every family (`nn/dag.py`), and the grader and the events need no change to gain a family. `families_trained_in(MAIN_LOCATION)` is empty: the main location fits no model. The per-head XGBoost families are retired (`RETIRED_MODEL_NAMES`), and the sweep keeps the `xgboost` handler so an old version stays servable by key for a rollback. A candidate is graded only against the served model of its own `model_name`; the served pointer itself can name any family. Do not fold the family into `model_version`, which the dashboard filters as a date.
- A recipe change (features, heads, horizons, learner or training settings, `DATASET_VERSION`) needs a new `model_name`, such as `nn_v3`, because the promotion gate grades a candidate only against the served model of its own name. Every candidate writes `recipe` and `recipe_hash` (`training/recipe.py`), and the candidate asset fails before it writes when the hash differs from the newest hashed version of the name at or before its partition, a re-run of the same partition included. Keep anything that changes daily (dates, `run_id`, counts, metrics) out of the recipe, or every retrain fails. Do not add an override for the check: the fix is a new name.
- A family whose `metadata.json`, examples object or side input is missing for a partition is logged and skipped, so one family does not cost every family's series. The champion asset raises only when no family has a candidate at all.
- Skipping is not the same as rebuilding from nothing, and the difference is destructive. An empty examples object produces an empty candidate, and the candidate's prefix cleanup then deletes the boosters that partition already holds, which `served.json` or a pin can be pointing at. So when a side input is unavailable, leave the partition's objects alone rather than writing empty ones over them.
@@ -35,12 +35,14 @@ Read `README.md` first for what the dataset is and how partitions behave. This f
- The serving path reads the deployment's own object store, never the dataset bucket: the Temporal workers have no credential for that bucket, and keeping it that way is why the manifest asset copies what the sweep needs. `serving/models/<key>/` holds a version's `metadata.json` and its model files, `serving/manifest.json` names the models one pass scores. The holdout boosters grade a promotion and must stay behind.
- An entry's files come from `ServingManifestEntry.files`: one `<head>.ubj` per head for `xgboost`, and the one `model.onnx` that `model_file` names for `onnx`. The copy step and the model store read that property, and the pin check reads the same names from each head's `file` in `metadata.json`, so a new model kind with another file layout keeps the two in step. Keep `onnxruntime` and `xgboost` imports inside the handlers' `load`: a test pins that importing the sweep loads neither.
- The manifest schema and the object keys live in `products/signals/backend/ranking/serving_manifest.py`, next to the feature contract, because the dag writes them and the sweep reads them. Its validators run at write time, so the dag refuses a manifest the sweep could not act on; the entry cap is the `ranking_score` artefact's, so a manifest can never produce a row the artefact refuses.
-- The `recent` role keeps the newest `INBOX_RANKING_SERVING_RECENT_VERSIONS` versions of the served `model_name`, read from the last 2N partitions, so each version gets several days of online scores. The `challenger` role keeps the newest readable version of each other `MODEL_FAMILIES` entry, read from its last 2 partitions. Each one passes the pin check before composition, because one missing file fails the publish. Keep the order under the cap as served, recent, challenger, pinned, cross-family. Each recent entry is one more model per sweep pass, so measure the sweep's duration, peak RSS and scored and deferred counts on the `inbox_ranking_sweep_finished` log before and after a change to N.
+- The `recent` role keeps the newest `INBOX_RANKING_SERVING_RECENT_VERSIONS` versions of the served `model_name`, read from the last 2N partitions, so each version gets several days of online scores. The `challenger` role keeps the newest readable version of each other serving family (`serving_families()`), read from its last 2 partitions. Each one passes the pin check before composition, because one missing file fails the publish. Keep the order under the cap as served, recent, challenger, pinned, cross-family. Each recent entry is one more model per sweep pass, so measure the sweep's duration, peak RSS and scored and deferred counts on the `inbox_ranking_sweep_finished` log before and after a change to N.
- Publishing order is load-bearing. Every model copy finishes before `manifest.json` is written, so a failed copy leaves the previous manifest and the previous models serving. A model version is immutable, so a version whose `metadata.json` is already at the target is not copied again. A run with nothing to publish, such as a run without `served.json`, writes nothing rather than an empty manifest. The served model passes the same check as a pin (`_pin_metadata`) before composition.
- When a manifest entry has `reads_project_vectors`, the dt=D `inbox_project_vectors` day is copied to `serving/project_vectors/<D>/` before `manifest.json`, and the manifest names it in `project_vectors`. A missing day fails the publish, like a missing model file. Never copy project vectors to the mirror: they are per-team data of this deployment, so the mirror's manifest names none. Training and serving build the `project` and `has_history` inputs through the one `project_inputs` in `features.py`; keep `PROJECT_VECTOR_KINDS` and `MIN_PROJECT_HISTORY_REPORTS` there.
- `HEADS` holds only the heads the inbox reads (`HeadRanker` in `backend/ranking/inbox_sort.py`). Add a head there only together with a surface that reads it. The promotion gate skips a champion head that is not in `HEADS`, so a trim does not block promotion.
- `nn_v3` and `nn_with_prio_v2` are trained by the `signals_ml` location (`nn/dag.py`), never by the main one. They share one `inbox_ranking_nn_examples` object, and each has its own candidate and champion asset from `family_assets`. The `nn_v3` assets keep the names `inbox_ranking_nn_model_{candidate,champion}`, so their Dagster history continues. A later family's candidate depends on the previous family's champion, for memory and not for data: two candidates at once can exceed the job's memory limit. A family whose set reads a state column the examples object lacks skips with a warning rather than training on zeros. torch is imported only inside the candidate asset's body (through `nn/train.py` and `multihead/network.py`). `signals_ml` must import without torch, and `test_multihead_network.py` pins that. Its decision goes through `decide_family_champion` in `training/dag.py`, graded through the incumbent's `model.holdout.onnx`. Keep `signals_ml` commented out in `.dagster_home/workspace.yaml`, and keep its schedule (04:30 UTC) running by default on prod US only.
+- An experiment is a `MODEL_FAMILIES` entry with `experiment=True`, a `baseline` serving family and an `expires` day. Only `inbox_ranking_nn_experiments_job` (`nn/experiments.py`) trains it, with its own schedule and resource tags, so a modeling idea never adds to the `nn` job's runtime or memory budget. It reads the day's `inbox_ranking_nn_examples` object and never rebuilds it. An experiment never enters the serving manifest, and `_pin_metadata` refuses its key, so the promote job, the served pointer and the pins cannot name it. To serve its recipe, register it as a serving family under a new `model_name`. Its `metadata.json` carries a `vs_baseline` block: the experiment and the baseline fit with each of `EXPERIMENT_SEEDS` on the same train rows and graded on the same holdout rows. An extra side input for an experiment goes through the registry in `training/side_inputs.py`, which refuses a frame without a timezone-aware `as_of_column` and drops rows known after the partition end. A feature set reads it through `rows_as_of`.
+
## Invariants — do not break
- `dt=` partitions are **immutable snapshots** with deterministic object keys; the only mutation ever applied is an idempotent re-run of the same partition. The exception is `inbox_signal_embeddings`, an emission log whose partition holds only that day's inserts — see the README's signal-grain section before touching it. Its re-run must stay **additive** (union with the existing object): the source drops rows it already archived, so a plain overwrite destroys history that exists nowhere else.
diff --git a/products/signals/dags/inbox_ranking/README.md b/products/signals/dags/inbox_ranking/README.md
index a9f11829b857..3495dc77ad8a 100644
--- a/products/signals/dags/inbox_ranking/README.md
+++ b/products/signals/dags/inbox_ranking/README.md
@@ -206,7 +206,7 @@ Each entry names its `model_kind`, and the sweep loads it through that kind's ha
- `xgboost`, the default for a model without the field: one booster per head, `<head>.ubj`. The entry's `heads` list names the files.
- `onnx`: one `model.onnx` for every head. The entry names it in `model_file`, and `heads` still lists the heads it scores. The graph takes one float32 input, `report`, of shape (batch, feature-set width), and gives one output, `probabilities`, of shape (batch, heads). The column order of that output is `output_heads` in the model's `metadata.json`, and each head entry in `heads` there names `model.onnx` as its `file`. At load, the handler checks the input names and widths against what the model's feature set builds, and a mismatch skips the model like an unloadable booster. The session runs on CPU with a bounded thread count, and `onnxruntime` is imported only when a model loads. The project-vector inputs are not served yet, so the sweep skips an `nn` model, whose graph also takes `project` and `has_history`, until it feeds them.
-The manifest holds one entry per model with its `roles`: `served` for the model `served.json` names, `recent` for each of the newest `INBOX_RANKING_SERVING_RECENT_VERSIONS` (default 2) versions of the served `model_name`, `challenger` for the newest readable version of each other `MODEL_FAMILIES` entry, read from its last 2 days because an `nn` candidate can land after the manifest runs, `pinned` for each model the `pin` override keeps (see [Overrides](#overrides)), and `cross_family` for every other family's `champion.json` that has a readable head. Nothing writes those files anymore, so they name the last version the retired automatic promotion chose. The dag reads the `recent` versions newest first from `<model_name>/dt=D/metadata.json` in the 2N days up to the partition, where N is the setting. It skips a version with no readable head, the served version, and a version that fails the pin check, and it removes duplicate keys. So each version gets online scores for about N days, not for one. A pass scores all of them, so an online paired read and a later interleaving need no rescoring; the entry count is capped at the `ranking_score` artefact's own limit (5). The order is `served`, `recent`, `challenger`, `pinned`, `cross_family`, so the cap drops `cross_family` entries first and `pinned` entries next. Challengers come in family registration order, so when there are more challengers than free entries, the last registered families lose their place. A challenger is how a new recipe gets online scores on the same reports as the served model before anyone promotes it. Each `recent` version adds one model to every sweep pass, so a higher N costs sweep time and pod memory. The schema, the role names and the object keys live in `products/signals/backend/ranking/serving_manifest.py`, and its validators run at write time, so the dag refuses a manifest the sweep could not act on. A model version is immutable, so a version already in the store is not copied again, and `manifest.json` is written last: a failed copy leaves the previous manifest and the previous models serving. When `served.json` is missing, or names a model that fails the pin check, the asset writes nothing and says why, and the previous manifest keeps serving.
+The manifest holds one entry per model with its `roles`: `served` for the model `served.json` names, `recent` for each of the newest `INBOX_RANKING_SERVING_RECENT_VERSIONS` (default 2) versions of the served `model_name`, `challenger` for the newest readable version of each other serving family, read from its last 2 days because an `nn` candidate can land after the manifest runs, `pinned` for each model the `pin` override keeps (see [Overrides](#overrides)), and `cross_family` for every other family's `champion.json` that has a readable head. Nothing writes those files anymore, so they name the last version the retired automatic promotion chose. The dag reads the `recent` versions newest first from `<model_name>/dt=D/metadata.json` in the 2N days up to the partition, where N is the setting. It skips a version with no readable head, the served version, and a version that fails the pin check, and it removes duplicate keys. So each version gets online scores for about N days, not for one. A pass scores all of them, so an online paired read and a later interleaving need no rescoring; the entry count is capped at the `ranking_score` artefact's own limit (5). The order is `served`, `recent`, `challenger`, `pinned`, `cross_family`, so the cap drops `cross_family` entries first and `pinned` entries next. Challengers come in family registration order, so when there are more challengers than free entries, the last registered families lose their place. A challenger is how a new recipe gets online scores on the same reports as the served model before anyone promotes it. Each `recent` version adds one model to every sweep pass, so a higher N costs sweep time and pod memory. The schema, the role names and the object keys live in `products/signals/backend/ranking/serving_manifest.py`, and its validators run at write time, so the dag refuses a manifest the sweep could not act on. A model version is immutable, so a version already in the store is not copied again, and `manifest.json` is written last: a failed copy leaves the previous manifest and the previous models serving. When `served.json` is missing, or names a model that fails the pin check, the asset writes nothing and says why, and the previous manifest keeps serving.
A region that does not train serves the same models through a mirror. When `INBOX_RANKING_SERVING_MIRROR_BUCKET` is set, the asset repeats the publish into that bucket after the primary publish succeeds, in the same order and with the same skip of versions already there. A mirror failure does not fail the asset: it logs, and the `inbox_ranking_serving_manifest_published` event carries `mirror_published=false`. The property is absent when no mirror is set.
@@ -388,6 +388,25 @@ dagster job launch -w .dagster_home/workspace.yaml --location posthog.dags.locat
-j inbox_ranking_nn_training_job --tags '{"dagster/partition": "2026-08-25"}'
```
+### Experiment families
+
+An experiment is a `MODEL_FAMILIES` entry with `experiment=True`, a `baseline` serving family and an `expires` day (`training/families.py`). It tests a modeling idea without a serving family's costs:
+
+- **Its own job.** `inbox_ranking_nn_experiments_job` (`nn/experiments.py`, on `signals_ml`) trains every live experiment, one after the other. It has its own runtime and memory tags, so it never uses the `nn` job's budget. Its schedule (08:00 UTC, after the `nn` job's 3h budget) skips when no experiment is live.
+- **No new examples.** It reads the day's `inbox_ranking_nn_examples` object. A day without that object is skipped, and so is a family after its `expires` day.
+- **Never served.** The serving manifest reads only `serving_families()`. The pin check refuses an experiment key, so the promote job, the served pointer and the pins cannot name one. To serve the recipe, register it as a serving family under a new `model_name`.
+- **Paired read.** The experiment and its baseline each fit once per seed in `EXPERIMENT_SEEDS`, on the same train rows. `metadata.json` holds a `vs_baseline` block per head: each seed's within-team and pooled holdout AUC for both arms, and the mean, min and max of the gaps. One `inbox_ranking_experiment_compared` event per head carries the gaps.
+- **Side inputs.** A side input beyond the report vectors needs a loader in `training/side_inputs.py`. The registry refuses a frame without a timezone-aware `as_of_column` and drops the rows known after the partition end. A feature set reads the frame through `rows_as_of`.
+
+The schedule is a fallback. To read an experiment on a chosen day, launch the job for that partition from the Dagster UI or the CLI:
+
+```bash
+dagster job launch -w .dagster_home/workspace.yaml --location posthog.dags.locations.signals_ml \
+ -j inbox_ranking_nn_experiments_job --tags '{"dagster/partition": "2026-10-20"}'
+```
+
+The newest day you can launch is the newest day the `nn` job finished. Adjacent days share most of their train and holdout rows, so their reads are not independent. Judge a `vs_baseline` gap on spaced days, for example every 7th day.
+
## Configuration
| Setting | Default | Meaning |
diff --git a/products/signals/dags/inbox_ranking/nn/dag.py b/products/signals/dags/inbox_ranking/nn/dag.py
index 705e7ada8ae5..1a0a69750aa6 100644
--- a/products/signals/dags/inbox_ranking/nn/dag.py
+++ b/products/signals/dags/inbox_ranking/nn/dag.py
@@ -49,9 +49,9 @@ from products.signals.dags.inbox_ranking.common import (
from products.signals.dags.inbox_ranking.consent import training_consent_team_ids
from products.signals.dags.inbox_ranking.dataset.dag import EMBEDDINGS_TABLE, LABELS_TABLE, STATE_TABLE
from products.signals.dags.inbox_ranking.dataset.project_vectors import EMBEDDING_COLUMNS, LABEL_COLUMNS
-from products.signals.dags.inbox_ranking.multihead.inputs import NetworkConfig
+from products.signals.dags.inbox_ranking.multihead.inputs import NetworkConfig, NetworkInputs
from products.signals.dags.inbox_ranking.multihead.local import report_vectors
-from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable
+from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable, report_matrix
from products.signals.dags.inbox_ranking.nn.examples import (
NN_HEADS,
build_examples,
@@ -66,6 +66,7 @@ from products.signals.dags.inbox_ranking.nn.model import (
champion_grades,
network_recipe,
network_vector_kinds,
+ vs_baseline_block,
)
from products.signals.dags.inbox_ranking.training.calibration import bucket_rows
from products.signals.dags.inbox_ranking.training.dag import (
@@ -82,13 +83,20 @@ from products.signals.dags.inbox_ranking.training.dag import (
snapshot_dates,
)
from products.signals.dags.inbox_ranking.training.examples import drop_without_training_consent
+from products.signals.dags.inbox_ranking.training.families import (
+ MODEL_FAMILIES,
+ NN_FAMILY,
+ NN_WITH_PRIO_FAMILY,
+ ModelFamily,
+)
+from products.signals.dags.inbox_ranking.training.side_inputs import load_side_inputs
from products.signals.dags.inbox_ranking.training.telemetry import (
candidate_events,
capture_training_events,
+ experiment_comparison_events,
holdout_calibration_events,
)
from products.signals.dags.inbox_ranking.training.train import HoldoutGrade
-from products.signals.dags.inbox_ranking.training.unseen import NN_FAMILY, NN_WITH_PRIO_FAMILY, ModelFamily
NN_EXAMPLES_TABLE = "inbox_ranking_nn_examples"
# The `nn_v3` assets keep the names they had as the only family, so their Dagster history continues.
@@ -99,6 +107,8 @@ NN_WITH_PRIO_CHAMPION_ASSET = "inbox_ranking_nn_with_prio_model_champion"
# The settings the prototype compared against the champion recipe with.
NETWORK_CONFIG = NetworkConfig()
+# The seeds an experiment and its baseline are both fit with for the paired comparison.
+EXPERIMENT_SEEDS = (0, 1, 2)
NN_ASSET_KWARGS: dict[str, Any] = {**COMMON_ASSET_KWARGS, "group_name": "inbox_ranking_nn"}
@@ -189,6 +199,49 @@ def _missing_state_columns(family: ModelFamily, table: ExampleTable) -> list[str
return [column for column in family.feature_set.state_columns if column not in table.rows]
+@frozen
+class _ExperimentInputs:
+ extras: Extras
+ # The baseline family's report input for every examples row, graded against the experiment's.
+ baseline_reports: np.ndarray
+
+
+def _experiment_inputs(
+ context: dagster.AssetExecutionContext,
+ client,
+ bucket: str,
+ prefix: str,
+ partition_key: str,
+ family: ModelFamily,
+ table: ExampleTable,
+ day: DayInputs,
+) -> _ExperimentInputs | None:
+ """The side inputs an experiment reads beside the report vectors, and its baseline's report
+ input, or None when either cannot be built and the experiment keeps the partition as it stands."""
+ baseline = next(candidate for candidate in MODEL_FAMILIES if candidate.name == family.baseline)
+ missing_columns = _missing_state_columns(baseline, table)
+ if missing_columns:
+ context.log.warning(
+ f"the nn examples for dt={partition_key} have no {', '.join(missing_columns)} column for baseline "
+ f"{baseline.name}; {family.name} keeps this partition as it stands"
+ )
+ return None
+ extras = {
+ **day.extras,
+ **load_side_inputs(client, bucket, prefix, partition_key, family.feature_set.missing_extras(day.extras)),
+ }
+ missing_extras = family.feature_set.missing_extras(extras)
+ if missing_extras:
+ context.log.warning(
+ f"no {', '.join(missing_extras)} side input for dt={partition_key}; {family.name} keeps this partition "
+ "as it stands"
+ )
+ return None
+ return _ExperimentInputs(
+ extras=extras, baseline_reports=report_matrix(table.rows, day.extras, baseline.feature_set)
+ )
+
+
def train_candidate(
context: dagster.AssetExecutionContext, client, bucket: str, partition_key: str, family: ModelFamily
) -> dict[str, dagster.MetadataValue]:
@@ -212,7 +265,14 @@ def train_candidate(
day = _day_inputs(context, client, bucket, prefix, partition_key)
if day is None:
return {"skipped": dagster.MetadataValue.bool(True)}
- inputs = network_inputs(examples.table, day.extras, day.labels, day.embeddings, family.feature_set)
+ extras = day.extras
+ baseline_reports = None
+ if family.experiment:
+ experiment = _experiment_inputs(context, client, bucket, prefix, partition_key, family, examples.table, day)
+ if experiment is None:
+ return {"skipped": dagster.MetadataValue.bool(True)}
+ extras, baseline_reports = experiment.extras, experiment.baseline_reports
+ inputs = network_inputs(examples.table, extras, day.labels, day.embeddings, family.feature_set)
del day
check_recipe(
client,
@@ -224,10 +284,21 @@ def train_candidate(
)
from products.signals.dags.inbox_ranking.nn.train import ( # noqa: PLC0415 — keeps torch off the location's import path
+ seed_scores,
train_network,
)
trained = train_network(examples.table, inputs, NN_HEADS, NETWORK_CONFIG)
+ vs_baseline = None
+ if baseline_reports is not None and family.baseline is not None:
+ baseline_inputs = NetworkInputs(reports=baseline_reports, context=inputs.context)
+ vs_baseline = vs_baseline_block(
+ examples.table,
+ baseline=family.baseline,
+ seeds=EXPERIMENT_SEEDS,
+ experiment_scores=seed_scores(examples.table, inputs, NN_HEADS, NETWORK_CONFIG, EXPERIMENT_SEEDS),
+ baseline_scores=seed_scores(examples.table, baseline_inputs, NN_HEADS, NETWORK_CONFIG, EXPERIMENT_SEEDS),
+ )
for head in trained.heads:
context.log.info(f"{family.name} {head.metrics.head}: {head.metrics.as_dict()}")
for name in trained.skipped:
@@ -250,6 +321,8 @@ def train_candidate(
trained_at=datetime.datetime.now(datetime.UTC),
run_id=context.run_id,
)
+ if vs_baseline is not None:
+ metadata = {**metadata, "vs_baseline": vs_baseline}
metadata_key = model_object_key(prefix, family.name, partition_key, METADATA_FILE)
_put_json(client, bucket, metadata_key, metadata)
written.add(metadata_key)
@@ -263,6 +336,7 @@ def train_candidate(
partition_key,
[
*candidate_events(metadata),
+ *experiment_comparison_events(metadata),
*holdout_calibration_events(
partition_key=partition_key,
run_id=context.run_id,
diff --git a/products/signals/dags/inbox_ranking/nn/experiments.py b/products/signals/dags/inbox_ranking/nn/experiments.py
new file mode 100644
index 000000000000..6e48b6c24edb
--- /dev/null
+++ b/products/signals/dags/inbox_ranking/nn/experiments.py
@@ -0,0 +1,115 @@
+"""Daily training for the experiment families (`experiment_families()` in `training/families.py`).
+
+An experiment is a modeling idea under test. It trains in this job, apart from the `nn` training
+job, so it never costs the serving families' runtime or memory budget. It reads the day's
+`inbox_ranking_nn_examples` object that the `nn` job wrote and never rebuilds it:
+
+ inbox_ranking_models/v1/<model_name>/dt=D/ the experiment's graphs and metadata.json, with a vs_baseline block
+
+The `vs_baseline` block fits the experiment and its baseline with each of `EXPERIMENT_SEEDS` on the
+same train rows, and grades both on the same holdout rows, so each seed gives a paired AUC gap.
+An experiment never enters the serving manifest and cannot be promoted or pinned. It stops training
+after its `expires` day, and a partition without examples is skipped and keeps what it holds.
+"""
+
+import datetime
+
+import dagster
+
+from posthog import settings
+
+from products.signals.dags.inbox_ranking.common import (
+ S3_BUCKET_ENV,
+ dataset_bucket,
+ dataset_unconfigured,
+ owner_tags,
+ partition_object_key,
+ s3_client,
+ skip_unconfigured,
+)
+from products.signals.dags.inbox_ranking.nn.dag import NN_ASSET_KWARGS, NN_EXAMPLES_TABLE, train_candidate
+from products.signals.dags.inbox_ranking.training.dag import _object_exists
+from products.signals.dags.inbox_ranking.training.families import ModelFamily, experiment_families
+
+NN_EXPERIMENTS_ASSET = "inbox_ranking_nn_experiment_candidates"
+
+
+def live_experiments(families: tuple[ModelFamily, ...], day: datetime.date) -> tuple[ModelFamily, ...]:
+ return tuple(family for family in families if not family.expired(day))
+
+
+def run_experiments(
+ context: dagster.AssetExecutionContext, client, bucket: str, partition_key: str, families: tuple[ModelFamily, ...]
+) -> dict[str, dagster.MetadataValue]:
+ """Train each live experiment in `families` for dt=`partition_key`, one after the other, and
+ return the asset metadata. An expired experiment and a partition without examples write nothing."""
+ day = datetime.date.fromisoformat(partition_key)
+ metadata: dict[str, dagster.MetadataValue] = {}
+ for family in families:
+ if family.expired(day):
+ context.log.info(f"{family.name} expired on {family.expires}; skipped for dt={partition_key}")
+ metadata[f"{family.name}_skipped"] = dagster.MetadataValue.text(f"expired on {family.expires}")
+ live = live_experiments(families, day)
+ if not live:
+ return {**metadata, "skipped": dagster.MetadataValue.bool(True)}
+ examples_key = partition_object_key(settings.INBOX_RANKING_DATASET_S3_PREFIX, NN_EXAMPLES_TABLE, partition_key)
+ if not _object_exists(client, bucket, examples_key):
+ context.log.warning(f"no nn examples for dt={partition_key}; every experiment skips this partition")
+ return {**metadata, "skipped": dagster.MetadataValue.bool(True)}
+ for family in live:
+ for name, value in train_candidate(context, client, bucket, partition_key, family).items():
+ metadata[f"{family.name}_{name}"] = value
+ return metadata
+
+
[email protected](name=NN_EXPERIMENTS_ASSET, deps=[NN_EXAMPLES_TABLE], **NN_ASSET_KWARGS)
+def inbox_ranking_nn_experiment_candidates(context: dagster.AssetExecutionContext) -> None:
+ if skip_unconfigured(context):
+ return
+ context.add_output_metadata(
+ run_experiments(context, s3_client(), dataset_bucket(), context.partition_key, experiment_families())
+ )
+
+
+inbox_ranking_nn_experiments_job = dagster.define_asset_job(
+ name="inbox_ranking_nn_experiments_job",
+ selection=[NN_EXPERIMENTS_ASSET],
+ tags={
+ **owner_tags,
+ # Per experiment: the five network fits of a candidate, then one fit per seed for each of
+ # the experiment and its baseline. The experiments run one after the other.
+ "dagster/max_runtime": str(4 * 60 * 60),
+ # As the nn job's candidate, plus the baseline's report matrix, held for the paired fits.
+ "dagster-k8s/config": {
+ "container_config": {
+ "resources": {
+ "requests": {"memory": "8Gi"},
+ "limits": {"memory": "32Gi"},
+ }
+ }
+ },
+ },
+)
+
+
+# After the nn job's 3h budget (04:30 UTC start), so the dt=D-1 examples exist.
+ cron_schedule="0 8 * * *",
+ job=inbox_ranking_nn_experiments_job,
+ execution_timezone="UTC",
+ default_status=dagster.DefaultScheduleStatus.RUNNING
+ if settings.CLOUD_DEPLOYMENT == "US"
+ else dagster.DefaultScheduleStatus.STOPPED,
+ tags=owner_tags,
+)
+def inbox_ranking_nn_experiments_schedule(
+ context: dagster.ScheduleEvaluationContext,
+) -> dagster.RunRequest | dagster.SkipReason:
+ if dataset_unconfigured():
+ return dagster.SkipReason(f"{S3_BUCKET_ENV} is not set; skipping until the dedicated bucket is provisioned")
+ previous_day = context.scheduled_execution_time.date() - datetime.timedelta(days=1)
+ if not live_experiments(experiment_families(), previous_day):
+ return dagster.SkipReason(f"no experiment family is live on {previous_day.isoformat()}")
+ return dagster.RunRequest(
+ partition_key=previous_day.isoformat(), run_key=f"nn-experiments-{previous_day.isoformat()}"
+ )
diff --git a/products/signals/dags/inbox_ranking/nn/model.py b/products/signals/dags/inbox_ranking/nn/model.py
index cd92edfe9aed..b2a3a19eab0d 100644
--- a/products/signals/dags/inbox_ranking/nn/model.py
+++ b/products/signals/dags/inbox_ranking/nn/model.py
@@ -24,10 +24,11 @@ from products.signals.dags.inbox_ranking.multihead.inputs import ONNX_INPUTS, ON
from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable
from products.signals.dags.inbox_ranking.nn.examples import NN_HEAD_NAMES, NN_HEADS
from products.signals.dags.inbox_ranking.training.calibration import CalibrationBucket
+from products.signals.dags.inbox_ranking.training.families import ModelFamily
from products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME
from products.signals.dags.inbox_ranking.training.recipe import model_recipe, with_recipe
-from products.signals.dags.inbox_ranking.training.train import HeadMetrics, HoldoutGrade, holdout_grade
-from products.signals.dags.inbox_ranking.training.unseen import ModelFamily
+from products.signals.dags.inbox_ranking.training.train import HeadMetrics, HoldoutGrade, _auc, holdout_grade
+from products.signals.dags.inbox_ranking.training.within_team import within_team_auc
HOLDOUT_MODEL_FILE = "model.holdout.onnx"
@@ -160,3 +161,57 @@ def champion_grades(
continue
grades[name] = holdout_grade(table.labels[rows, index].astype(int), probabilities[name][rows])
return grades
+
+
+def _paired(experiment_aucs: Sequence[float | None], baseline_aucs: Sequence[float | None]) -> dict[str, Any]:
+ deltas = [
+ experiment - base
+ for experiment, base in zip(experiment_aucs, baseline_aucs)
+ if experiment is not None and base is not None
+ ]
+ return {
+ "experiment_aucs": list(experiment_aucs),
+ "baseline_aucs": list(baseline_aucs),
+ "paired_seeds": len(deltas),
+ "mean_delta_auc": float(np.mean(deltas)) if deltas else None,
+ "min_delta_auc": min(deltas) if deltas else None,
+ "max_delta_auc": max(deltas) if deltas else None,
+ "wins": sum(1 for delta in deltas if delta > 0),
+ }
+
+
+def vs_baseline_block(
+ table: ExampleTable,
+ *,
+ baseline: str,
+ seeds: Sequence[int],
+ experiment_scores: Sequence[np.ndarray],
+ baseline_scores: Sequence[np.ndarray],
+) -> dict[str, Any]:
+ """The experiment against its baseline, per head, on the same holdout rows.
+
+ Each seed fits both arms on the same train rows, so the two AUCs of one seed form a pair. One
+ seed's gap can be noise from the network's initialization alone, so the block keeps the range
+ of the gaps across seeds. `within_team` is the read the inbox order depends on, and `pooled` is
+ there for reference. A seed where either arm has no AUC drops out of the pairs.
+ """
+ train_rows = table.network_train_rows()
+ heads: dict[str, dict[str, Any]] = {}
+ for index, name in enumerate(NN_HEAD_NAMES):
+ rows = table.head_rows(index, holdout=True)
+ y = table.labels[rows, index].astype(int)
+ team_ids = table.team_ids(rows)
+ heads[name] = {
+ "train_rows": int((~np.isnan(table.labels[train_rows, index])).sum()),
+ "holdout_rows": len(rows),
+ "holdout_positives": int(y.sum()),
+ "within_team": _paired(
+ [within_team_auc(y, scores[rows, index], team_ids).auc for scores in experiment_scores],
+ [within_team_auc(y, scores[rows, index], team_ids).auc for scores in baseline_scores],
+ ),
+ "pooled": _paired(
+ [_auc(y, scores[rows, index]) for scores in experiment_scores],
+ [_auc(y, scores[rows, index]) for scores in baseline_scores],
+ ),
+ }
+ return {"baseline": baseline, "seeds": list(seeds), "heads": heads}
diff --git a/products/signals/dags/inbox_ranking/nn/train.py b/products/signals/dags/inbox_ranking/nn/train.py
index a353361f55d8..6ed66beefb3f 100644
--- a/products/signals/dags/inbox_ranking/nn/train.py
+++ b/products/signals/dags/inbox_ranking/nn/train.py
@@ -135,3 +135,16 @@ def train_network(
refit_rows=len(refit_rows),
vector_kinds=network_vector_kinds(inputs),
)
+
+
+def seed_scores(
+ table: ExampleTable, inputs: NetworkInputs, heads: Sequence[Head], config: NetworkConfig, seeds: Sequence[int]
+) -> list[np.ndarray]:
+ """One train-only fit per seed, each scoring every row. Two arms fit with the same seeds on the
+ same rows give the paired scores `vs_baseline_block` grades."""
+ train_rows = table.network_train_rows()
+ all_rows = np.arange(len(inputs))
+ return [
+ _fit(inputs, heads, dataclasses.replace(config, seed=seed), table.labels, train_rows).predict(inputs, all_rows)
+ for seed in seeds
+ ]
diff --git a/products/signals/dags/inbox_ranking/tests/test_experiments.py b/products/signals/dags/inbox_ranking/tests/test_experiments.py
new file mode 100644
index 000000000000..5a530ed9895e
--- /dev/null
+++ b/products/signals/dags/inbox_ranking/tests/test_experiments.py
@@ -0,0 +1,182 @@
+import datetime
+
+import pytest
+
+import numpy as np
+import pandas as pd
+import dagster
+
+from posthog import settings
+
+from products.signals.backend.ranking.features import REPORT_EMBEDDINGS_FEATURE_SET
+from products.signals.dags.inbox_ranking.common import partition_object_key
+from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable
+from products.signals.dags.inbox_ranking.nn import experiments
+from products.signals.dags.inbox_ranking.nn.dag import NN_EXAMPLES_TABLE
+from products.signals.dags.inbox_ranking.nn.examples import NN_HEAD_NAMES
+from products.signals.dags.inbox_ranking.nn.model import vs_baseline_block
+from products.signals.dags.inbox_ranking.tests.test_training import _ModelStoreS3
+from products.signals.dags.inbox_ranking.training import side_inputs
+from products.signals.dags.inbox_ranking.training.families import NN_MODEL_NAME, ModelFamily, family_problems
+from products.signals.dags.inbox_ranking.training.side_inputs import (
+ SideInputError,
+ SideInputLoader,
+ load_side_inputs,
+ rows_as_of,
+)
+
+PARTITION = "2026-10-20"
+PREFIX = settings.INBOX_RANKING_DATASET_S3_PREFIX
+SIDE_INPUT = "report_sources"
+
+
+def _experiment(name: str, expires: datetime.date) -> ModelFamily:
+ return ModelFamily(
+ name=name,
+ feature_set=REPORT_EMBEDDINGS_FEATURE_SET,
+ experiment=True,
+ baseline=NN_MODEL_NAME,
+ expires=expires,
+ )
+
+
[email protected]("examples_present", [True, False])
+def test_an_expired_experiment_and_a_day_without_examples_train_nothing(monkeypatch, examples_present):
+ trained: list[str] = []
+
+ def train_candidate(context, client, bucket, partition_key, family):
+ trained.append(family.name)
+ return {"heads_trained": dagster.MetadataValue.int(3)}
+
+ monkeypatch.setattr(experiments, "train_candidate", train_candidate)
+ objects = {partition_object_key(PREFIX, NN_EXAMPLES_TABLE, PARTITION): b"examples"} if examples_present else {}
+ # Expiry is inclusive: the expires day still trains, the day after does not.
+ families = (
+ _experiment("expired", datetime.date(2026, 10, 19)),
+ _experiment("last_day", datetime.date(2026, 10, 20)),
+ )
+
+ with dagster.build_asset_context(partition_key=PARTITION) as context:
+ metadata = experiments.run_experiments(context, _ModelStoreS3(objects), "bucket", PARTITION, families)
+
+ assert trained == (["last_day"] if examples_present else [])
+ assert metadata["expired_skipped"].value == "expired on 2026-10-19"
+ assert ("skipped" in metadata) is not examples_present
+
+
+def test_an_experiment_must_name_a_registered_serving_baseline_and_an_expiry():
+ with pytest.raises(ValueError, match="needs a baseline and an expires day"):
+ ModelFamily(name="open_ended", feature_set=REPORT_EMBEDDINGS_FEATURE_SET, experiment=True)
+ serving = ModelFamily(name=NN_MODEL_NAME, feature_set=REPORT_EMBEDDINGS_FEATURE_SET)
+ chained = ModelFamily(
+ name="chained",
+ feature_set=REPORT_EMBEDDINGS_FEATURE_SET,
+ experiment=True,
+ baseline="first",
+ expires=datetime.date(2026, 12, 31),
+ )
+ assert family_problems((serving, _experiment("first", datetime.date(2026, 12, 31)), chained)) == [
+ "experiment chained names baseline first, which is not a serving family"
+ ]
+
+
+def _sources(known_at: list[str]) -> pd.DataFrame:
+ return pd.DataFrame(
+ {
+ "report_id": [f"r{index}" for index in range(len(known_at))],
+ "known_at": pd.to_datetime(known_at, utc=True),
+ }
+ )
+
+
+def registered(monkeypatch):
+ frames: dict[str, pd.DataFrame | None] = {}
+ monkeypatch.setattr(side_inputs, "SIDE_INPUT_LOADERS", {})
+ side_inputs.register_side_input(
+ SideInputLoader(
+ key=SIDE_INPUT, as_of_column="known_at", load=lambda client, bucket, prefix, day: frames.get(day)
+ )
+ )
+ return frames
+
+
+def test_a_side_input_is_cut_at_the_partition_end_and_read_only_as_of_a_moment(registered):
+ registered[PARTITION] = _sources(["2026-10-19T12:00:00", "2026-10-21T00:00:00", "2026-10-21T00:00:01"])
+
+ extras = load_side_inputs(None, "bucket", PREFIX, PARTITION, [SIDE_INPUT])
+
+ # A row known after the partition end did not exist on dt=D.
+ assert list(extras[SIDE_INPUT]["report_id"]) == ["r0", "r1"]
+ # A moment on dt=D-1 sees only what was known by then, as the report vector rule does.
+ moment = datetime.datetime(2026, 10, 20, tzinfo=datetime.UTC)
+ assert list(rows_as_of(extras, SIDE_INPUT, moment)["report_id"]) == ["r0"]
+ with pytest.raises(SideInputError, match="only as of a moment"):
+ rows_as_of(extras, SIDE_INPUT, None)
+
+
+def test_a_missing_side_input_object_leaves_its_key_out(registered):
+ assert load_side_inputs(None, "bucket", PREFIX, PARTITION, [SIDE_INPUT]) == {}
+
+
+ "frame,message",
+ [
+ (pd.DataFrame({"report_id": ["r0"]}), "has no known_at column"),
+ (
+ pd.DataFrame({"report_id": ["r0"], "known_at": pd.to_datetime([None], utc=True)}),
+ "rows with no known_at",
+ ),
+ (
+ pd.DataFrame({"report_id": ["r0"], "known_at": pd.to_datetime(["2026-10-19"])}),
+ "not a timezone-aware timestamp",
+ ),
+ ],
+ ids=["no_column", "null_moment", "naive_moment"],
+)
+def test_a_side_input_without_a_usable_as_of_column_is_refused(registered, frame, message):
+ registered[PARTITION] = frame
+
+ with pytest.raises(SideInputError, match=message):
+ load_side_inputs(None, "bucket", PREFIX, PARTITION, [SIDE_INPUT])
+
+
+def test_a_side_input_without_a_registered_loader_is_refused(registered):
+ with pytest.raises(SideInputError, match="no registered loader"):
+ load_side_inputs(None, "bucket", PREFIX, PARTITION, ["unregistered"])
+
+
+def test_vs_baseline_pairs_each_seed_inside_each_team_on_the_same_holdout_rows():
+ # Team 1 ranks its own reports perfectly under the experiment's first seed, while the pooled
+ # order mixes the teams. The third head has only negatives on the holdout, so nothing pairs.
+ labels = np.array([[1, 1, 0], [0, 0, 0], [1, 0, 0], [0, 1, 0]], dtype=np.float32)
+ table = ExampleTable(
+ rows=pd.DataFrame({"report_id": ["a", "b", "c", "d"], "report_team_id": [1, 1, 2, 2]}),
+ labels=labels,
+ holdout=np.ones_like(labels, dtype=bool),
+ )
+ perfect = labels.copy()
+ inverted = 1 - labels
+
+ block = vs_baseline_block(
+ table,
+ baseline=NN_MODEL_NAME,
+ seeds=(0, 1),
+ experiment_scores=[perfect, inverted],
+ baseline_scores=[inverted, inverted],
+ )
+
+ first, _, third = NN_HEAD_NAMES
+ paired = {
+ "experiment_aucs": [1.0, 0.0],
+ "baseline_aucs": [0.0, 0.0],
+ "paired_seeds": 2,
+ "mean_delta_auc": 0.5,
+ "min_delta_auc": 0.0,
+ "max_delta_auc": 1.0,
+ "wins": 1,
+ }
+ assert block["heads"][first]["within_team"] == paired
+ assert block["heads"][first]["pooled"] == paired
+ assert block["heads"][third]["within_team"]["paired_seeds"] == 0
+ assert block["heads"][third]["within_team"]["mean_delta_auc"] is None
diff --git a/products/signals/dags/inbox_ranking/tests/test_nn.py b/products/signals/dags/inbox_ranking/tests/test_nn.py
index d8e29f47664f..ec3a8a0c6e9c 100644
--- a/products/signals/dags/inbox_ranking/tests/test_nn.py
+++ b/products/signals/dags/inbox_ranking/tests/test_nn.py
@@ -57,17 +57,17 @@ from products.signals.dags.inbox_ranking.training.dag import (
model_object_key,
served_object_key,
)
-from products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME, Head
-from products.signals.dags.inbox_ranking.training.recipe import RECIPE_HASH_KEY, RECIPE_KEY, model_recipe, recipe_hash
-from products.signals.dags.inbox_ranking.training.serving import FamilyModels, compose_manifest
-from products.signals.dags.inbox_ranking.training.train import HeadMetrics
-from products.signals.dags.inbox_ranking.training.unseen import (
+from products.signals.dags.inbox_ranking.training.families import (
NN_FAMILY,
NN_MODEL_NAME,
NN_WITH_PRIO_FAMILY,
NN_WITH_PRIO_MODEL_NAME,
ModelFamily,
)
+from products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME, Head
+from products.signals.dags.inbox_ranking.training.recipe import RECIPE_HASH_KEY, RECIPE_KEY, model_recipe, recipe_hash
+from products.signals.dags.inbox_ranking.training.serving import FamilyModels, compose_manifest
+from products.signals.dags.inbox_ranking.training.train import HeadMetrics
PREFIX = settings.INBOX_RANKING_DATASET_S3_PREFIX
# A retired per-head XGBoost family. Its saved versions stay servable by key.
@@ -252,7 +252,18 @@ def test_the_nn_decision_is_graded_against_the_served_nn_model_and_writes_nothin
assert client.objects == before
[email protected]("family", [NN_FAMILY, NN_WITH_PRIO_FAMILY], ids=lambda family: family.name)
+_PRIO_EXPERIMENT = ModelFamily(
+ name="nn_with_prio_experiment",
+ feature_set=NN_WITH_PRIO_FAMILY.feature_set,
+ learner=ONNX_MODEL_KIND,
+ location=NN_FAMILY.location,
+ experiment=True,
+ baseline=NN_MODEL_NAME,
+ expires=datetime.date(2026, 12, 31),
+)
+
+
[email protected]("family", [NN_FAMILY, NN_WITH_PRIO_FAMILY, _PRIO_EXPERIMENT], ids=lambda family: family.name)
def test_the_nn_candidate_writes_a_graph_whose_columns_follow_its_recorded_head_order(
monkeypatch, no_side_effects, family
):
@@ -282,6 +293,17 @@ def test_the_nn_candidate_writes_a_graph_whose_columns_follow_its_recorded_head_
assert metadata["feature_set"] == family.feature_set.name
assert metadata[ONNX_OUTPUT_HEADS_KEY] == list(NN_HEAD_NAMES)
assert model_mismatch(metadata) is None
+ # Only an experiment is graded against a baseline, with one paired AUC per seed and head.
+ if family.experiment:
+ assert metadata["vs_baseline"]["baseline"] == NN_MODEL_NAME
+ assert {
+ name: len(grade["within_team"]["experiment_aucs"])
+ == len(grade["within_team"]["baseline_aucs"])
+ == len(nn_dag.EXPERIMENT_SEEDS)
+ for name, grade in metadata["vs_baseline"]["heads"].items()
+ } == dict.fromkeys(NN_HEAD_NAMES, True)
+ else:
+ assert "vs_baseline" not in metadata
assert {head["head"] for head in metadata["heads"]} == set(NN_HEAD_NAMES)
assert {(head["file"], head["holdout_file"]) for head in metadata["heads"]} == {
(ONNX_MODEL_FILE, HOLDOUT_MODEL_FILE)
diff --git a/products/signals/dags/inbox_ranking/tests/test_promote.py b/products/signals/dags/inbox_ranking/tests/test_promote.py
index f67ad646a858..061bef9c4d67 100644
--- a/products/signals/dags/inbox_ranking/tests/test_promote.py
+++ b/products/signals/dags/inbox_ranking/tests/test_promote.py
@@ -14,16 +14,17 @@ from products.signals.dags.inbox_ranking.common import partition_object_key
from products.signals.dags.inbox_ranking.dataset.dag import PROJECT_VECTORS_TABLE
from products.signals.dags.inbox_ranking.tests.test_training import _AppObjectStore, _ModelStoreS3, _patch_capture
from products.signals.dags.inbox_ranking.training.dag import METADATA_FILE, model_object_key, served_object_key
+from products.signals.dags.inbox_ranking.training.families import ModelFamily
from products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME
from products.signals.dags.inbox_ranking.training.promote import PromoteConfig, inbox_ranking_promote
from products.signals.dags.inbox_ranking.training.telemetry import MODEL_PROMOTED_EVENT
-from products.signals.dags.inbox_ranking.training.unseen import ModelFamily
PREFIX = settings.INBOX_RANKING_DATASET_S3_PREFIX
SERVED_VERSION = "2026-10-16"
OLDER_VERSION = "2026-10-01"
# A retired XGBoost family. A rollback can still serve one of its saved versions.
XGBOOST_MODEL_NAME = "report_embeddings_v2"
+EXPERIMENT_MODEL_NAME = "report_embeddings_experiment"
class _WritableS3(_ModelStoreS3):
@@ -31,9 +32,9 @@ class _WritableS3(_ModelStoreS3):
self.objects[Key] = Body
-def _metadata(version: str, heads: tuple[str, ...]) -> dict[str, Any]:
+def _metadata(version: str, heads: tuple[str, ...], model_name: str = XGBOOST_MODEL_NAME) -> dict[str, Any]:
return {
- "model_name": XGBOOST_MODEL_NAME,
+ "model_name": model_name,
"model_version": version,
"feature_set": REPORT_EMBEDDINGS_FEATURE_SET.name,
"feature_schema_version": REPORT_EMBEDDINGS_FEATURE_SET.schema_version,
@@ -42,9 +43,16 @@ def _metadata(version: str, heads: tuple[str, ...]) -> dict[str, Any]:
}
-def _add_model(objects: dict[str, bytes], version: str, heads: tuple[str, ...], *, files: bool = True) -> None:
- folder = model_object_key(PREFIX, XGBOOST_MODEL_NAME, version, "")
- objects[folder + METADATA_FILE] = json.dumps(_metadata(version, heads)).encode()
+def _add_model(
+ objects: dict[str, bytes],
+ version: str,
+ heads: tuple[str, ...],
+ *,
+ files: bool = True,
+ model_name: str = XGBOOST_MODEL_NAME,
+) -> None:
+ folder = model_object_key(PREFIX, model_name, version, "")
+ objects[folder + METADATA_FILE] = json.dumps(_metadata(version, heads, model_name)).encode()
if files:
for head in heads:
objects[folder + f"{head}.ubj"] = b"booster"
@@ -67,6 +75,8 @@ def _dataset() -> _WritableS3:
_add_model(objects, OLDER_VERSION, all_heads)
_add_model(objects, "2026-09-01", all_heads, files=False)
_add_model(objects, "2026-09-02", all_heads[:1])
+ # Every head and every file: only being an experiment keeps it from serving.
+ _add_model(objects, SERVED_VERSION, all_heads, model_name=EXPERIMENT_MODEL_NAME)
return _WritableS3(objects)
@@ -76,7 +86,16 @@ def _promote(monkeypatch: pytest.MonkeyPatch, client: _WritableS3, store: _AppOb
monkeypatch.setattr(settings, "INBOX_RANKING_SERVING_MIRROR_BUCKET", "")
monkeypatch.setattr(
"products.signals.dags.inbox_ranking.training.dag.MODEL_FAMILIES",
- (ModelFamily(name=XGBOOST_MODEL_NAME, feature_set=REPORT_EMBEDDINGS_FEATURE_SET),),
+ (
+ ModelFamily(name=XGBOOST_MODEL_NAME, feature_set=REPORT_EMBEDDINGS_FEATURE_SET),
+ ModelFamily(
+ name=EXPERIMENT_MODEL_NAME,
+ feature_set=REPORT_EMBEDDINGS_FEATURE_SET,
+ experiment=True,
+ baseline=XGBOOST_MODEL_NAME,
+ expires=datetime.date(2026, 12, 31),
+ ),
+ ),
)
for module in ("dag", "promote"):
monkeypatch.setattr(f"products.signals.dags.inbox_ranking.training.{module}.s3_client", lambda: client)
@@ -110,8 +129,9 @@ def test_a_rollback_to_an_older_version_rewrites_the_pointer_and_publishes_it(mo
("report_embeddings@latest", "not a `<model_name>@<YYYY-MM-DD>` model key"),
(f"{XGBOOST_MODEL_NAME}@2026-09-01", "missing model files"),
(f"{XGBOOST_MODEL_NAME}@2026-09-02", "has no model for the heads"),
+ (f"{EXPERIMENT_MODEL_NAME}@{SERVED_VERSION}", "is an experiment family, and an experiment never serves"),
],
- ids=["malformed_key", "missing_file", "missing_head"],
+ ids=["malformed_key", "missing_file", "missing_head", "experiment"],
)
def test_a_model_the_sweep_could_not_serve_is_refused_and_nothing_is_written(monkeypatch, model_key, message):
client, store = _dataset(), _AppObjectStore()
diff --git a/products/signals/dags/inbox_ranking/tests/test_training.py b/products/signals/dags/inbox_ranking/tests/test_training.py
index d000d2026983..ea0f3cf30917 100644
--- a/products/signals/dags/inbox_ranking/tests/test_training.py
+++ b/products/signals/dags/inbox_ranking/tests/test_training.py
@@ -90,6 +90,12 @@ from products.signals.dags.inbox_ranking.training.examples import (
holdout_mask,
reports_missing_birth_snapshot,
)
+from products.signals.dags.inbox_ranking.training.families import (
+ MODEL_FAMILIES,
+ NN_MODEL_NAME,
+ NN_WITH_PRIO_MODEL_NAME,
+ ModelFamily,
+)
from products.signals.dags.inbox_ranking.training.heads import ACTION_LABEL_COLUMNS, HEADS_BY_NAME, Head
from products.signals.dags.inbox_ranking.training.promotion import (
AUC_TOLERANCE,
@@ -121,12 +127,8 @@ from products.signals.dags.inbox_ranking.training.train import HoldoutGrade, _he
from products.signals.dags.inbox_ranking.training.unseen import (
CANDIDATE_ROLE,
LEGACY_POOL_NAME,
- MODEL_FAMILIES,
- NN_MODEL_NAME,
- NN_WITH_PRIO_MODEL_NAME,
POOL_NAME,
SERVED_SCORES_TABLE,
- ModelFamily,
calibration_rows,
chance_band,
empty_scores_write_allowed,
@@ -143,6 +145,15 @@ from products.signals.dags.inbox_ranking.training.within_team import NO_TEAMS, W
# A per-head XGBoost family on the report vector. The sweep still serves this learner.
EMBEDDINGS_MODEL_NAME = "report_embeddings_xgb"
+EXPERIMENT_MODEL_NAME = "report_embeddings_experiment"
+_EMBEDDINGS_FAMILY = ModelFamily(name=EMBEDDINGS_MODEL_NAME, feature_set=REPORT_EMBEDDINGS_FEATURE_SET)
+_EXPERIMENT_FAMILY = ModelFamily(
+ name=EXPERIMENT_MODEL_NAME,
+ feature_set=REPORT_EMBEDDINGS_FEATURE_SET,
+ experiment=True,
+ baseline=EMBEDDINGS_MODEL_NAME,
+ expires=datetime.date(2026, 9, 30),
+)
EMBEDDING_FEATURE_SETS = (REPORT_EMBEDDINGS_FEATURE_SET,)
@@ -2338,6 +2349,9 @@ def test_recent_versions_and_challengers_come_from_the_window_and_skip_a_version
(NN_WITH_PRIO_MODEL_NAME, "2026-08-19", False),
(NN_WITH_PRIO_MODEL_NAME, "2026-08-18", True),
(NN_MODEL_NAME, "2026-08-17", True),
+ # Readable, with every file, and newest in the window: an experiment never serves, so it is
+ # no challenger either.
+ (EXPERIMENT_MODEL_NAME, "2026-08-19", True),
):
folder = model_object_key(prefix, name, version, "")
client.objects[folder + METADATA_FILE] = json.dumps(
@@ -2348,7 +2362,7 @@ def test_recent_versions_and_challengers_come_from_the_window_and_skip_a_version
monkeypatch.setattr(settings, "INBOX_RANKING_SERVING_RECENT_VERSIONS", 3)
store = _AppObjectStore()
- _run_serving_manifest(monkeypatch, store, dataset_client=client, families=MODEL_FAMILIES)
+ _run_serving_manifest(monkeypatch, store, dataset_client=client, families=(*MODEL_FAMILIES, _EXPERIMENT_FAMILY))
manifest = ServingManifest.model_validate_json(store.objects[serving_manifest_key(prefix)])
assert [(entry.model_name, entry.model_version, entry.roles) for entry in manifest.models] == [
@@ -2474,7 +2488,12 @@ def test_a_missing_pin_is_dropped_and_the_manifest_still_publishes(monkeypatch):
missing_key = model_key(EMBEDDINGS_MODEL_NAME, pin_version)
absent_key = model_key(EMBEDDINGS_MODEL_NAME, "2026-07-01")
denied_key = model_key(EMBEDDINGS_MODEL_NAME, "2026-06-01")
- pins = (missing_key, absent_key, denied_key, retired_key)
+ experiment_key = model_key(EXPERIMENT_MODEL_NAME, pin_version)
+ client.objects[model_object_key(prefix, EXPERIMENT_MODEL_NAME, pin_version, METADATA_FILE)] = json.dumps(
+ _serving_metadata(EXPERIMENT_MODEL_NAME, pin_version, heads=("open",), **_FEATURE_FIELDS)
+ ).encode()
+ client.objects[model_object_key(prefix, EXPERIMENT_MODEL_NAME, pin_version, "open.ubj")] = b"booster"
+ pins = (missing_key, absent_key, denied_key, retired_key, experiment_key)
monkeypatch.setattr(
"products.signals.dags.inbox_ranking.training.dag.read_ranking_overrides",
lambda now: RankingOverrides(pin=pins),
@@ -2484,7 +2503,9 @@ def test_a_missing_pin_is_dropped_and_the_manifest_still_publishes(monkeypatch):
)
store = _AppObjectStore()
- _run_serving_manifest(monkeypatch, store, dataset_client=denied_client)
+ _run_serving_manifest(
+ monkeypatch, store, dataset_client=denied_client, families=(_EMBEDDINGS_FAMILY, _EXPERIMENT_FAMILY)
+ )
manifest = ServingManifest.model_validate_json(store.objects[serving_manifest_key(prefix)])
assert not set(pins) & {entry.key for entry in manifest.models}
@@ -2495,6 +2516,7 @@ def test_a_missing_pin_is_dropped_and_the_manifest_still_publishes(monkeypatch):
assert METADATA_FILE in dropped[absent_key]
assert "AccessDenied" in dropped[denied_key]
assert "this build can produce" in dropped[retired_key]
+ assert "is an experiment family" in dropped[experiment_key]
def test_the_serving_prefix_layout_is_stable():
diff --git a/products/signals/dags/inbox_ranking/training/dag.py b/products/signals/dags/inbox_ranking/training/dag.py
index bd3599d90fd4..e94b8ec795c5 100644
--- a/products/signals/dags/inbox_ranking/training/dag.py
+++ b/products/signals/dags/inbox_ranking/training/dag.py
@@ -73,6 +73,12 @@ from products.signals.dags.inbox_ranking.training.examples import (
Snapshot,
assemble_snapshot,
)
+from products.signals.dags.inbox_ranking.training.families import (
+ MODEL_FAMILIES,
+ ModelFamily,
+ experiment_names,
+ serving_families,
+)
from products.signals.dags.inbox_ranking.training.heads import (
ACTION_LABEL_COLUMNS,
HEADS,
@@ -93,10 +99,8 @@ from products.signals.dags.inbox_ranking.training.telemetry import (
)
from products.signals.dags.inbox_ranking.training.train import HoldoutGrade
from products.signals.dags.inbox_ranking.training.unseen import (
- MODEL_FAMILIES,
SERVED_SCORES_TABLE,
HeadGrade,
- ModelFamily,
calibration_rows,
graded_rows,
head_grades,
@@ -436,7 +440,7 @@ def _publish_manifest(context: RunContext, partition_key: str, run_id: str) -> d
name=family.name,
champion=_read_json_if_exists(client, bucket, champion_object_key(prefix, family.name)),
)
- for family in MODEL_FAMILIES
+ for family in serving_families(MODEL_FAMILIES)
]
recent_versions = settings.INBOX_RANKING_SERVING_RECENT_VERSIONS
recent = (
@@ -449,7 +453,7 @@ def _publish_manifest(context: RunContext, partition_key: str, run_id: str) -> d
challengers = (
[
metadata
- for family in MODEL_FAMILIES
+ for family in serving_families(MODEL_FAMILIES)
if family.name != served["model_name"]
for metadata in _recent_models(context, client, bucket, prefix, family.name, partition_key, 1)
]
@@ -642,8 +646,16 @@ def _recent_models(
def _pin_metadata(client, bucket: str, prefix: str, key: str) -> tuple[dict[str, Any] | None, str | None]:
- """The metadata of one pinned model, or None and the reason it cannot be served."""
+ """The metadata of one pinned model, or None and the reason it cannot be served.
+
+ The promote job, the served pointer, the pins and the recent and challenger reads all pass this
+ check, so an experiment family is refused here once for every path."""
model_name, model_version = key.split("@", 1)
+ if model_name in experiment_names(MODEL_FAMILIES):
+ return None, (
+ f"{model_name} is an experiment family, and an experiment never serves. To serve this "
+ "recipe, register it as a serving family under a new model_name"
+ )
metadata = _read_json_if_exists(client, bucket, model_object_key(prefix, model_name, model_version, METADATA_FILE))
if metadata is None:
return None, f"no {METADATA_FILE} in the dataset bucket"
diff --git a/products/signals/dags/inbox_ranking/training/families.py b/products/signals/dags/inbox_ranking/training/families.py
new file mode 100644
index 000000000000..528a8d49cf64
--- /dev/null
+++ b/products/signals/dags/inbox_ranking/training/families.py
@@ -0,0 +1,120 @@
+"""The model families: which features and which learner each one fits, and where.
+
+A family is either a serving family or an experiment. A serving family can be served, enters the
+serving manifest as a challenger, and gets online grades. An experiment trains in its own job
+(`nn/experiments.py`), is graded only against its `baseline` on the same holdout, never enters the
+manifest, and can never be promoted or pinned. An experiment stops training after its `expires`
+day, so a modeling idea cannot grow into a permanent cost by accident. To serve an experiment's
+recipe, register it as a serving family under a new `model_name`.
+"""
+
+import datetime
+from collections.abc import Sequence
+
+from posthog.dataclasses import frozen
+
+from products.signals.backend.ranking.features import (
+ REPORT_EMBEDDINGS_FEATURE_SET,
+ REPORT_EMBEDDINGS_PRIORITY_FEATURE_SET,
+ FeatureSet,
+)
+from products.signals.backend.ranking.serving_manifest import DEFAULT_MODEL_KIND, ONNX_MODEL_KIND
+
+# The model family: which features and which learner, as against `model_version`, the partition day
+# it was fit on. Both are in the identity, so two families trained on one day stay apart.
+# A recipe change takes a new name (`training/recipe.py`). The versions of an earlier name stay
+# servable by key, and their online grades keep their own series.
+NN_MODEL_NAME = "nn_v3"
+NN_WITH_PRIO_MODEL_NAME = "nn_with_prio_v2"
+# The Dagster code locations that train a family: `signals` on the main image, and `signals_ml` on
+# the image that carries torch.
+MAIN_LOCATION = "signals"
+ML_LOCATION = "signals_ml"
+
+
+@frozen
+class ModelFamily:
+ """One family the training jobs fit: its name, the feature set it is fit on, the learner a
+ model store loads it with, and the Dagster code location whose assets train it.
+
+ An experiment names the serving family it is graded against in `baseline`, and the last
+ partition day it trains in `expires`. A serving family has neither."""
+
+ name: str
+ feature_set: FeatureSet
+ learner: str = DEFAULT_MODEL_KIND
+ location: str = MAIN_LOCATION
+ experiment: bool = False
+ baseline: str | None = None
+ expires: datetime.date | None = None
+
+ def __post_init__(self) -> None:
+ if self.experiment and (self.baseline is None or self.expires is None):
+ raise ValueError(f"experiment family {self.name} needs a baseline and an expires day")
+ if not self.experiment and (self.baseline is not None or self.expires is not None):
+ raise ValueError(f"serving family {self.name} cannot have a baseline or an expires day")
+
+ def expired(self, day: datetime.date) -> bool:
+ return self.expires is not None and day > self.expires
+
+
+# The families the training jobs train, in the order they are trained. A
+# family with no metadata for the day is skipped, so an entry can be added here before its trainer
+# writes its first candidate, and a family that fails costs its own series rather than every one.
+# `nn` is one multi-head network over the report vector and the team's project vectors. It needs
+# torch, so the `signals_ml` location trains it. The main location fits no family. `nn` declares
+# the report embeddings set, because that set builds its report input in training and at serving.
+# `nn_with_prio_v2` is the same network with the report's priority levels as extra report inputs.
+MODEL_FAMILIES: tuple[ModelFamily, ...] = (
+ ModelFamily(
+ name=NN_MODEL_NAME, feature_set=REPORT_EMBEDDINGS_FEATURE_SET, learner=ONNX_MODEL_KIND, location=ML_LOCATION
+ ),
+ ModelFamily(
+ name=NN_WITH_PRIO_MODEL_NAME,
+ feature_set=REPORT_EMBEDDINGS_PRIORITY_FEATURE_SET,
+ learner=ONNX_MODEL_KIND,
+ location=ML_LOCATION,
+ ),
+)
+NN_FAMILY = next(family for family in MODEL_FAMILIES if family.name == NN_MODEL_NAME)
+NN_WITH_PRIO_FAMILY = next(family for family in MODEL_FAMILIES if family.name == NN_WITH_PRIO_MODEL_NAME)
+
+
+def serving_families(families: Sequence[ModelFamily] = MODEL_FAMILIES) -> tuple[ModelFamily, ...]:
+ """The families the serving manifest, the promote job and the online grader can read."""
+ return tuple(family for family in families if not family.experiment)
+
+
+def experiment_families(families: Sequence[ModelFamily] = MODEL_FAMILIES) -> tuple[ModelFamily, ...]:
+ """The families the experiments job trains, expired ones included. The job skips an expired one."""
+ return tuple(family for family in families if family.experiment)
+
+
+def experiment_names(families: Sequence[ModelFamily] = MODEL_FAMILIES) -> frozenset[str]:
+ return frozenset(family.name for family in experiment_families(families))
+
+
+def families_trained_in(location: str) -> tuple[ModelFamily, ...]:
+ """The serving families whose candidate and champion assets live in `location`. The serving
+ manifest and the online grader walk every serving family, so a family gains both wherever it
+ trains."""
+ return tuple(family for family in serving_families() if family.location == location)
+
+
+def family_problems(families: Sequence[ModelFamily]) -> list[str]:
+ """Why `families` is not a valid registry: a duplicate name, or an experiment whose baseline is
+ not a serving family in the registry."""
+ names = [family.name for family in families]
+ problems = [f"family {name} is registered twice" for name in sorted({n for n in names if names.count(n) > 1})]
+ serving = {family.name for family in serving_families(families)}
+ problems += [
+ f"experiment {family.name} names baseline {family.baseline}, which is not a serving family"
+ for family in experiment_families(families)
+ if family.baseline not in serving
+ ]
+ return problems
+
+
+_PROBLEMS = family_problems(MODEL_FAMILIES)
+if _PROBLEMS:
+ raise ValueError("; ".join(_PROBLEMS))
diff --git a/products/signals/dags/inbox_ranking/training/side_inputs.py b/products/signals/dags/inbox_ranking/training/side_inputs.py
new file mode 100644
index 000000000000..1e97240ce662
--- /dev/null
+++ b/products/signals/dags/inbox_ranking/training/side_inputs.py
@@ -0,0 +1,84 @@
+"""The registry of side-input loaders an experiment family can read beside the report vectors.
+
+A side input is a per-report frame that a feature set reads through its `extras_keys`. Each row
+must carry the moment it became known, in the column the loader declares as `as_of_column`. The
+registry refuses a frame without that column, with a null in it, or with a naive timestamp, and it
+drops the rows that became known after the partition ends. A feature set then reads the frame
+through `rows_as_of`, which keeps only the rows known at the moment being built. A side input that
+changes over a report's life therefore cannot leak a later value into an earlier example.
+
+A missing object leaves its key out of the result, so only the families that read it skip. An empty
+object keeps its key.
+"""
+
+import datetime
+from collections.abc import Callable, Iterable
+
+import pandas as pd
+
+from posthog.dataclasses import frozen
+
+from products.signals.backend.ranking.features import Extras
+from products.signals.dags.inbox_ranking.common import snapshot_bounds
+
+# Reads the dt=D object of one side input. Returns None when the object was never written.
+LoadFrame = Callable[[object, str, str, str], pd.DataFrame | None]
+
+
+class SideInputError(ValueError):
+ pass
+
+
+@frozen
+class SideInputLoader:
+ key: str
+ as_of_column: str
+ load: LoadFrame
+
+
+SIDE_INPUT_LOADERS: dict[str, SideInputLoader] = {}
+
+
+def register_side_input(loader: SideInputLoader) -> SideInputLoader:
+ if loader.key in SIDE_INPUT_LOADERS:
+ raise SideInputError(f"side input {loader.key} is registered twice")
+ if not loader.as_of_column:
+ raise SideInputError(f"side input {loader.key} declares no as_of_column")
+ SIDE_INPUT_LOADERS[loader.key] = loader
+ return loader
+
+
+def _known_by(frame: pd.DataFrame, loader: SideInputLoader, as_of: datetime.datetime) -> pd.DataFrame:
+ if as_of.tzinfo is None:
+ raise SideInputError(f"side input {loader.key} needs a timezone-aware as_of")
+ if loader.as_of_column not in frame:
+ raise SideInputError(f"side input {loader.key} has no {loader.as_of_column} column")
+ known = frame[loader.as_of_column]
+ if known.isna().any():
+ raise SideInputError(f"side input {loader.key} has rows with no {loader.as_of_column}")
+ if not isinstance(known.dtype, pd.DatetimeTZDtype):
+ raise SideInputError(f"side input {loader.key} column {loader.as_of_column} is not a timezone-aware timestamp")
+ return frame.loc[known <= as_of]
+
+
+def rows_as_of(extras: Extras, key: str, as_of: datetime.datetime | None) -> pd.DataFrame:
+ """The rows of side input `key` known at `as_of`. A feature set reads a registered side input
+ through this, so a moment never sees a value that landed after it."""
+ if as_of is None:
+ raise SideInputError(f"side input {key} is read only as of a moment")
+ return _known_by(extras[key], SIDE_INPUT_LOADERS[key], as_of)
+
+
+def load_side_inputs(client, bucket: str, prefix: str, partition_key: str, keys: Iterable[str]) -> Extras:
+ """The dt=D frame of each side input in `keys`, cut at the end of dt=D. A key without a
+ registered loader raises, and a key whose object is missing is left out."""
+ _, snapshot_end = snapshot_bounds(partition_key)
+ loaded: dict[str, pd.DataFrame] = {}
+ for key in keys:
+ loader = SIDE_INPUT_LOADERS.get(key)
+ if loader is None:
+ raise SideInputError(f"side input {key} has no registered loader")
+ frame = loader.load(client, bucket, prefix, partition_key)
+ if frame is not None:
+ loaded[key] = _known_by(frame, loader, snapshot_end)
+ return loaded
diff --git a/products/signals/dags/inbox_ranking/training/telemetry.py b/products/signals/dags/inbox_ranking/training/telemetry.py
index 93d943b111d1..576e5072c48b 100644
--- a/products/signals/dags/inbox_ranking/training/telemetry.py
+++ b/products/signals/dags/inbox_ranking/training/telemetry.py
@@ -37,6 +37,7 @@ UNSEEN_REPORT_GRADED_EVENT = "inbox_ranking_unseen_report_graded"
UNSEEN_CALIBRATION_EVENT = "inbox_ranking_unseen_calibration"
HOLDOUT_CALIBRATION_EVENT = "inbox_ranking_holdout_calibration"
SERVING_MANIFEST_PUBLISHED_EVENT = "inbox_ranking_serving_manifest_published"
+EXPERIMENT_COMPARED_EVENT = "inbox_ranking_experiment_compared"
# Candidate metadata copied onto every per-head event so a chart can filter or break down on it.
_CANDIDATE_CONTEXT_KEYS = (
@@ -126,6 +127,31 @@ def promotion_event(
)
+def experiment_comparison_events(metadata: Mapping[str, Any]) -> list[TrainingEvent]:
+ """One event per head of an experiment candidate's `vs_baseline` block. The per-seed AUC lists
+ stay in `metadata.json`: an insight cannot chart a JSON array, so the event carries the gaps."""
+ block = metadata.get("vs_baseline") or {}
+ context = {key: metadata.get(key) for key in ("model_name", "model_version", "run_id", "recipe_hash")}
+ return [
+ TrainingEvent(
+ event=EXPERIMENT_COMPARED_EVENT,
+ properties={
+ **context,
+ "baseline_model_name": block.get("baseline"),
+ "seeds": len(block.get("seeds", [])),
+ "head": head,
+ **{key: grade.get(key) for key in ("train_rows", "holdout_rows", "holdout_positives")},
+ **{
+ f"{read}_{key}": (grade.get(read) or {}).get(key)
+ for read in ("within_team", "pooled")
+ for key in ("paired_seeds", "mean_delta_auc", "min_delta_auc", "max_delta_auc", "wins")
+ },
+ },
+ )
+ for head, grade in (block.get("heads") or {}).items()
+ ]
+
+
def model_promoted_event(*, model_key: str, previous_key: str | None, reason: str, run_id: str) -> TrainingEvent:
return TrainingEvent(
event=MODEL_PROMOTED_EVENT,
diff --git a/products/signals/dags/inbox_ranking/training/unseen.py b/products/signals/dags/inbox_ranking/training/unseen.py
index 65783ca4129c..aee6b2569a7d 100644
--- a/products/signals/dags/inbox_ranking/training/unseen.py
+++ b/products/signals/dags/inbox_ranking/training/unseen.py
@@ -22,12 +22,6 @@ from sklearn.metrics import roc_auc_score
from posthog.dataclasses import frozen
-from products.signals.backend.ranking.features import (
- REPORT_EMBEDDINGS_FEATURE_SET,
- REPORT_EMBEDDINGS_PRIORITY_FEATURE_SET,
- FeatureSet,
-)
-from products.signals.backend.ranking.serving_manifest import DEFAULT_MODEL_KIND, ONNX_MODEL_KIND
from products.signals.dags.inbox_ranking.training.calibration import (
CalibrationBucket,
bucket_rows,
@@ -57,16 +51,6 @@ SERVED_SCORES_TABLE = "inbox_ranking_served_scores"
# The role of a trained candidate on the training events, and of a scored row whose event names no role.
CANDIDATE_ROLE = "candidate"
-# The model family: which features and which learner, as against `model_version`, the partition day
-# it was fit on. Both are in the identity, so two families trained on one day stay apart.
-# A recipe change takes a new name (`training/recipe.py`). The versions of an earlier name stay
-# servable by key, and their online grades keep their own series.
-NN_MODEL_NAME = "nn_v3"
-NN_WITH_PRIO_MODEL_NAME = "nn_with_prio_v2"
-# The Dagster code locations that train a family: `signals` on the main image, and `signals_ml` on
-# the image that carries torch.
-MAIN_LOCATION = "signals"
-ML_LOCATION = "signals_ml"
# Families this build no longer trains. Saved scores objects still hold their rows, so every reader
# drops them through `with_model_names`: the grades stop with the family, and the rewrite guard does
# not require rows that no re-run can produce.
@@ -103,45 +87,6 @@ SCORES_SCHEMA = pa.schema(_SCORE_TYPES)
SCORE_COLUMNS = tuple(_SCORE_TYPES)
-@frozen
-class ModelFamily:
- """One family the training jobs fit: its name, the feature set it is fit on, the learner a
- model store loads it with, and the Dagster code location whose assets train it."""
-
- name: str
- feature_set: FeatureSet
- learner: str = DEFAULT_MODEL_KIND
- location: str = MAIN_LOCATION
-
-
-# The families the training jobs train, in the order they are trained. A
-# family with no metadata for the day is skipped, so an entry can be added here before its trainer
-# writes its first candidate, and a family that fails costs its own series rather than every one.
-# `nn` is one multi-head network over the report vector and the team's project vectors. It needs
-# torch, so the `signals_ml` location trains it. The main location fits no family. `nn` declares
-# the report embeddings set, because that set builds its report input in training and at serving.
-# `nn_with_prio_v2` is the same network with the report's priority levels as extra report inputs.
-MODEL_FAMILIES: tuple[ModelFamily, ...] = (
- ModelFamily(
- name=NN_MODEL_NAME, feature_set=REPORT_EMBEDDINGS_FEATURE_SET, learner=ONNX_MODEL_KIND, location=ML_LOCATION
- ),
- ModelFamily(
- name=NN_WITH_PRIO_MODEL_NAME,
- feature_set=REPORT_EMBEDDINGS_PRIORITY_FEATURE_SET,
- learner=ONNX_MODEL_KIND,
- location=ML_LOCATION,
- ),
-)
-NN_FAMILY = next(family for family in MODEL_FAMILIES if family.name == NN_MODEL_NAME)
-NN_WITH_PRIO_FAMILY = next(family for family in MODEL_FAMILIES if family.name == NN_WITH_PRIO_MODEL_NAME)
-
-
-def families_trained_in(location: str) -> tuple[ModelFamily, ...]:
- """The families whose candidate and champion assets live in `location`. The serving manifest
- and the online grader walk every family, so a family gains both wherever it trains."""
- return tuple(family for family in MODEL_FAMILIES if family.location == location)
-
-
@frozen
class HeadGrade:
head: str