Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(signals): name inbox ranking recipes with a recipe hash (#118964)

posthog[bot]003831fa0690
diff --git a/products/signals/dags/inbox_ranking/AGENTS.md b/products/signals/dags/inbox_ranking/AGENTS.md
index 0711c7068364..8bfd9dad7ab3 100644
--- a/products/signals/dags/inbox_ranking/AGENTS.md
+++ b/products/signals/dags/inbox_ranking/AGENTS.md
@@ -25,7 +25,8 @@ Read `README.md` first for what the dataset is and how partitions behave. This f
- The model is read in three layers: **training** (train metrics), **test** (the random-by-report holdout, which drives promotion) and **online** (`inbox_ranking_served_scores`, the scoring sweep's live scores of each day's newborns, graded by `inbox_ranking_unseen_graded` at each head's horizon). Do not add an offline rescoring of the newborn pool beside the online layer: the sweep already scores it with every model key the manifest names.
- `served_score_rows` keeps the earliest scored event per (report, model key) and takes `model_role` from the event's roles. Grades stay per `(model_name, model_version, model_role)` and are never pooled across versions. A re-run that reads no event is refused when the partition already holds rows, and so is one that reads fewer families than the object holds, because the newborn state snapshot behind those rows ages out and the dt=D+horizon grade reads them.
- Every scored row carries `head_readable`, and an unreadable head is still graded: a head that misses `min_holdout_positives` on one day's holdout still gets a number from the pooled online grade. Keep the promotion gate reading `readable` from the candidate metadata: an unreadable head must not decide a promotion.
-- `model_name` is the model family, and every model object, scored row, grade and event carries it. A family is an entry in `MODEL_FAMILIES` (`training/unseen.py`) naming the feature set it is fit on, its `learner` and the `location` that trains it; each location's candidate and champion assets walk `families_trained_in(<location>)`, and the grader and the events need no change to gain a family. A candidate is graded only against the served model of its own `model_name`; the served pointer itself can name any family. Do not fold the family into `model_version`, which the dashboard filters as a date.
+- `model_name` is the model recipe, and every model object, scored row, grade and event carries it. A family is an entry in `MODEL_FAMILIES` (`training/unseen.py`) naming the feature set it is fit on, its `learner` and the `location` that trains it; each location's candidate and champion assets walk `families_trained_in(<location>)`, and the grader and the events need no change to gain a family. A candidate is graded only against the served model of its own `model_name`; the served pointer itself can name any family. Do not fold the family into `model_version`, which the dashboard filters as a date.
+- A recipe change (features, heads, horizons, learner or training settings, `DATASET_VERSION`) needs a new `model_name`, such as `nn_v2`, because the promotion gate grades a candidate only against the served model of its own name. Every candidate writes `recipe` and `recipe_hash` (`training/recipe.py`), and the candidate asset fails before it writes when the hash differs from the newest hashed version of the name at or before its partition, a re-run of the same partition included. Keep anything that changes daily (dates, `run_id`, counts, metrics) out of the recipe, or every retrain fails. Do not add an override for the check: the fix is a new name.
- A family whose `metadata.json`, examples object or side input is missing for a partition is logged and skipped, so one family does not cost every family's series. The champion asset raises only when no family has a candidate at all.
- Skipping is not the same as rebuilding from nothing, and the difference is destructive. An empty examples object produces an empty candidate, and the candidate's prefix cleanup then deletes the boosters that partition already holds, which `served.json` or a pin can be pointing at. So when a side input is unavailable, leave the partition's objects alone rather than writing empty ones over them.
- The champion is compared to a candidate on the candidate's holdout, through the champion's `<head>.holdout.ubj` (the train-only fit). Keep writing that file: without it the gate falls back to the champion's stored AUC, which was measured on a different set of reports.
diff --git a/products/signals/dags/inbox_ranking/README.md b/products/signals/dags/inbox_ranking/README.md
index bf9012ca8054..a1c0109293be 100644
--- a/products/signals/dags/inbox_ranking/README.md
+++ b/products/signals/dags/inbox_ranking/README.md
@@ -149,7 +149,8 @@ s3://<bucket>/<prefix>/
└── inbox_ranking_served_scores/v1/dt=YYYY-MM-DD/ # the scoring sweep's scores of the reports born that day, per model key
```
-- **A model is a `(model_name, model_version, model_role)`.** `model_name` is the family: which features and which learner: `report_embeddings`, per-head XGBoost, is the one trained here. `nn`, one ONNX network, is trained by the `signals_ml` location, and each `MODEL_FAMILIES` entry names its `learner` and the `location` that trains it. `model_version` is the partition day it was fit on, and `model_role` is the role it was scored or trained under (`candidate` on the training events; `served`, `daily_candidate` or `cross_family` on the online grade). Each family owns a prefix under the models path, so two families trained on the same day cannot collide. A candidate is graded only against the served model of its own name. The served pointer can name any family, so a rollback can cross families.
+- **A model is a `(model_name, model_version, model_role)`.** `model_name` is the recipe: which features, which learner and which training settings: `report_embeddings`, per-head XGBoost, is the one trained here. `nn`, one ONNX network, is trained by the `signals_ml` location, and each `MODEL_FAMILIES` entry names its `learner` and the `location` that trains it. `model_version` is the partition day it was fit on, and `model_role` is the role it was scored or trained under (`candidate` on the training events; `served`, `daily_candidate` or `cross_family` on the online grade). Each family owns a prefix under the models path, so two families trained on the same day cannot collide. A candidate is graded only against the served model of its own name. The served pointer can name any family, so a rollback can cross families.
+- **A recipe change gets a new `model_name`.** A candidate's `metadata.json` carries `recipe` and `recipe_hash`, a stable hash of the recipe (`training/recipe.py`): the learner, the feature set and its schema version, the project-vector kinds and their history settings, the booster params or network config, each head's cohort, label, horizon, label columns and status-label flag, `lookback_days`, `holdout_days`, the example grain and row budget, and `dataset_version`. Dates, `run_id`, row counts and metrics stay out, so a daily retrain keeps the hash. The candidate asset compares its hash with the newest version of the same name, at or before its partition, that has one, and on a mismatch it fails before it writes the partition and names the changed inputs. A re-run of a partition therefore compares with that partition's own saved recipe. The search covers every dataset version, so a `DATASET_VERSION` bump also needs a new name. A version without a hash is not compared. This includes an env change such as `INBOX_RANKING_TRAINING_LOOKBACK_DAYS`. To start a new recipe, add a `MODEL_FAMILIES` entry with a new name (`nn_v2` after `nn`) and remove the old entry: the old versions stay servable by key, and the promotion gate never compares two recipes. Two recipes can train side by side for a comparison, but each costs a manifest slot, so do not do it by default. `recipe_hash` is on `inbox_ranking_candidate_trained` and `inbox_ranking_promotion_decided` (with `incumbent_recipe_hash`), and `inbox_ranking_serving_manifest_published` carries `model_recipe_hashes` per model key.
- **A feature set is one feature universe.** `products/signals/backend/ranking/features.py` holds a `FeatureSet` per universe: its name, its schema version, its ordered feature names, the report-state columns it reads, `build_matrix`, and how many rows it wants per report and per head. `report_embeddings` is the one today. A candidate records the set it was fit on in its `metadata.json`, so the grader checks a model against its own set rather than one global contract, the examples asset writes one Parquet per set, and the unseen scorer builds one matrix per set and shares it across the families that read it. Adding a set means an entry in `FEATURE_SETS`; a model naming a set this build cannot produce is logged and left unscored.
- **Examples are one row per report at its birth, by default.** For partition `dt=D` the examples asset reads the report-state and labels snapshots `dt=D-lookback..D` and emits one row per report, on the snapshot of the day it was created, labeled from the snapshot `horizon_days` later (7 for open, 14 for action, 30 for fixed). Features are built by the feature set from that snapshot's state columns plus `age_hours`, the report's age at the snapshot. Labels are aligned to the state spine, so a report with no label event is a negative (all-zero labels), not absent. Label-only rows (no Postgres state) are skipped, as are state rows read long after their snapshot day (backfills carry current Postgres state; see `features_observed_at`) and, for the heads that read the status stream (`action` and `fixed`), rows whose status telemetry fails the dataset's `label_provenance_ok` check. Birth is the moment serving scores a report, and the label that comes out is a per-report probability, which is what the inbox ordering needs. A report born before the window, or one whose birth-day snapshot is missing from it, is no example; `reports_missing_birth_snapshot` on the asset counts the second case, which is a gap in the partitions.
- **An outcome already visible on the birth day is a future positive.** A report born on day D has no scoring moment before D, so an outcome visible at D belongs to the moment being built rather than to an earlier one; the labels of D therefore read as their defaults for a report born on D. Most outcomes land on the birth day, so this is where the positives are: censoring on them costs the `pr_created` head most of its training signal. `<set>_<head>_birth_day_positives` on the examples asset, and `birth_day_positives` on the `inbox_ranking_examples_built` event, say how many positives the rule keeps. These rows carry a hindsight the other rows do not: the state snapshot reads `signal_count`, `total_weight`, `run_count` and the text sizes live from Postgres a few hours after day D ends, so a birth-day outcome happened before its own feature read. Both the holdout AUC and the newborn unseen grade therefore read optimistically on these rows, until the scoring sweep's timestamped score log replaces the daily snapshot as this table's source.
diff --git a/products/signals/dags/inbox_ranking/nn/dag.py b/products/signals/dags/inbox_ranking/nn/dag.py
index c043db695ba0..fed303d1c773 100644
--- a/products/signals/dags/inbox_ranking/nn/dag.py
+++ b/products/signals/dags/inbox_ranking/nn/dag.py
@@ -55,7 +55,13 @@ from products.signals.dags.inbox_ranking.nn.examples import (
network_inputs,
subset,
)
-from products.signals.dags.inbox_ranking.nn.model import HOLDOUT_MODEL_FILE, candidate_metadata, champion_grades
+from products.signals.dags.inbox_ranking.nn.model import (
+ HOLDOUT_MODEL_FILE,
+ candidate_metadata,
+ champion_grades,
+ network_recipe,
+ network_vector_kinds,
+)
from products.signals.dags.inbox_ranking.training.calibration import bucket_rows
from products.signals.dags.inbox_ranking.training.dag import (
_LOOKBACK_MAPPING,
@@ -64,6 +70,7 @@ from products.signals.dags.inbox_ranking.training.dag import (
_delete_other_objects,
_put_json,
_read_bytes_if_exists,
+ check_recipe,
decide_family_champion,
load_snapshots,
model_object_key,
@@ -192,6 +199,14 @@ def train_candidate(
examples = examples_from_arrow(arrow)
inputs = network_inputs(examples.table, day.extras, day.labels, day.embeddings)
del day
+ check_recipe(
+ client,
+ bucket,
+ prefix,
+ NN_FAMILY.name,
+ partition_key,
+ network_recipe(NETWORK_CONFIG, network_vector_kinds(inputs)),
+ )
from products.signals.dags.inbox_ranking.nn.train import ( # noqa: PLC0415 — keeps torch off the location's import path
train_network,
diff --git a/products/signals/dags/inbox_ranking/nn/model.py b/products/signals/dags/inbox_ranking/nn/model.py
index b7e1d2729d15..4f731a8897be 100644
--- a/products/signals/dags/inbox_ranking/nn/model.py
+++ b/products/signals/dags/inbox_ranking/nn/model.py
@@ -19,11 +19,13 @@ from posthog.dataclasses import frozen
from products.signals.backend.ranking.model_store import ONNX_OUTPUT_HEADS_KEY
from products.signals.backend.ranking.serving_manifest import ONNX_MODEL_FILE, ONNX_MODEL_KIND
from products.signals.dags.inbox_ranking.common import DATASET_VERSION
+from products.signals.dags.inbox_ranking.dataset.project_vectors import VECTOR_KINDS
from products.signals.dags.inbox_ranking.multihead.inputs import ONNX_INPUTS, ONNX_OUTPUT, NetworkConfig, NetworkInputs
from products.signals.dags.inbox_ranking.multihead.prototype import FEATURE_SET, ExampleTable
-from products.signals.dags.inbox_ranking.nn.examples import NN_HEAD_NAMES
+from products.signals.dags.inbox_ranking.nn.examples import NN_HEAD_NAMES, NN_HEADS
from products.signals.dags.inbox_ranking.training.calibration import CalibrationBucket
from products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME
+from products.signals.dags.inbox_ranking.training.recipe import model_recipe, with_recipe
from products.signals.dags.inbox_ranking.training.train import HeadMetrics, HoldoutGrade, holdout_grade
from products.signals.dags.inbox_ranking.training.unseen import NN_MODEL_NAME
@@ -52,6 +54,21 @@ class TrainedNetwork:
vector_kinds: tuple[str, ...]
+def network_vector_kinds(inputs: NetworkInputs) -> tuple[str, ...]:
+ """The project-vector kinds a network fit on `inputs` takes, empty for a report-only network."""
+ return VECTOR_KINDS if inputs.context is not None else ()
+
+
+def network_recipe(config: NetworkConfig, vector_kinds: Sequence[str]) -> dict[str, Any]:
+ return model_recipe(
+ model_kind=ONNX_MODEL_KIND,
+ feature_set=FEATURE_SET,
+ heads=NN_HEADS,
+ learner_params=dataclasses.asdict(config),
+ project_vector_kinds=vector_kinds,
+ )
+
+
def candidate_metadata(
partition_key: str,
trained: TrainedNetwork,
@@ -66,7 +83,7 @@ def candidate_metadata(
The heads carry the same metric fields as a booster's (`train_head` and this trainer both fill
`HeadMetrics`), so the promotion rule, the events and the manifest read every family alike.
"""
- return {
+ metadata = {
"model_name": NN_MODEL_NAME,
"model_version": partition_key,
"dataset_version": DATASET_VERSION,
@@ -99,6 +116,7 @@ def candidate_metadata(
],
"skipped_heads": list(trained.skipped),
}
+ return with_recipe(metadata, network_recipe(config, trained.vector_kinds))
def onnx_probabilities(model: bytes, inputs: NetworkInputs, output_heads: Sequence[str]) -> dict[str, np.ndarray]:
diff --git a/products/signals/dags/inbox_ranking/nn/train.py b/products/signals/dags/inbox_ranking/nn/train.py
index 71fb38fbaa0d..6e6f46d58ccc 100644
--- a/products/signals/dags/inbox_ranking/nn/train.py
+++ b/products/signals/dags/inbox_ranking/nn/train.py
@@ -23,7 +23,7 @@ from products.signals.dags.inbox_ranking.dataset.project_vectors import VECTOR_K
from products.signals.dags.inbox_ranking.multihead.inputs import NetworkConfig, NetworkInputs
from products.signals.dags.inbox_ranking.multihead.network import MultiHeadNetwork, export_onnx
from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable
-from products.signals.dags.inbox_ranking.nn.model import NetworkHead, TrainedNetwork
+from products.signals.dags.inbox_ranking.nn.model import NetworkHead, TrainedNetwork, network_vector_kinds
from products.signals.dags.inbox_ranking.training.heads import Head
from products.signals.dags.inbox_ranking.training.train import NULL_PERMUTATIONS, head_metrics
@@ -131,5 +131,5 @@ def train_network(
skipped=tuple(skipped),
network_train_rows=len(train_rows),
refit_rows=len(refit_rows),
- vector_kinds=VECTOR_KINDS if inputs.context is not None else (),
+ vector_kinds=network_vector_kinds(inputs),
)
diff --git a/products/signals/dags/inbox_ranking/tests/test_nn.py b/products/signals/dags/inbox_ranking/tests/test_nn.py
index 9e5a7ca6f9de..c7ec1b5bed2e 100644
--- a/products/signals/dags/inbox_ranking/tests/test_nn.py
+++ b/products/signals/dags/inbox_ranking/tests/test_nn.py
@@ -1,6 +1,7 @@
import io
import json
import datetime
+import dataclasses
from typing import Any
import pytest
@@ -14,6 +15,7 @@ from sklearn.metrics import roc_auc_score
from posthog import settings
+from products.signals.backend.ranking.features import PROJECT_VECTOR_KINDS, REPORT_EMBEDDINGS_FEATURE_SET
from products.signals.backend.ranking.model_contract import model_mismatch
from products.signals.backend.ranking.model_store import ONNX_OUTPUT_HEADS_KEY
from products.signals.backend.ranking.overrides import NO_OVERRIDES
@@ -24,7 +26,8 @@ from products.signals.backend.ranking.serving_manifest import (
ONNX_MODEL_KIND,
SERVED_ROLE,
)
-from products.signals.dags.inbox_ranking.common import partition_object_key
+from products.signals.dags.inbox_ranking.common import DATASET_VERSION, partition_object_key
+from products.signals.dags.inbox_ranking.dataset import project_vectors
from products.signals.dags.inbox_ranking.dataset.dag import EMBEDDINGS_TABLE, LABELS_TABLE
from products.signals.dags.inbox_ranking.multihead.inputs import NetworkConfig
from products.signals.dags.inbox_ranking.multihead.local import report_vectors
@@ -41,18 +44,24 @@ from products.signals.dags.inbox_ranking.nn.model import (
NetworkHead,
TrainedNetwork,
candidate_metadata,
+ network_recipe,
onnx_probabilities,
)
from products.signals.dags.inbox_ranking.tests.test_multihead import D0, DAYS, HOLDOUT_DAYS, _SyntheticInbox
+from products.signals.dags.inbox_ranking.training import recipe as recipe_module
from products.signals.dags.inbox_ranking.training.dag import (
CHAMPION_FILE,
METADATA_FILE,
SERVED_FILE,
+ booster_recipe,
+ check_recipe,
inbox_ranking_model_candidate,
inbox_ranking_model_champion,
model_object_key,
served_object_key,
)
+from products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME, Head
+from products.signals.dags.inbox_ranking.training.recipe import RECIPE_HASH_KEY, RECIPE_KEY, model_recipe, recipe_hash
from products.signals.dags.inbox_ranking.training.serving import FamilyModels, compose_manifest
from products.signals.dags.inbox_ranking.training.train import HeadMetrics
from products.signals.dags.inbox_ranking.training.unseen import NN_MODEL_NAME, REPORT_EMBEDDINGS_MODEL_NAME
@@ -80,8 +89,17 @@ class _S3:
def get_paginator(self, _name):
return self
- def paginate(self, *, Bucket, Prefix):
- yield {"Contents": [{"Key": key} for key in sorted(self.objects) if key.startswith(Prefix)]}
+ def paginate(self, *, Bucket, Prefix, Delimiter=None):
+ keys = [key for key in sorted(self.objects) if key.startswith(Prefix)]
+ if Delimiter is None:
+ yield {"Contents": [{"Key": key} for key in keys]}
+ return
+ folders = {
+ Prefix + key[len(Prefix) :].split(Delimiter)[0] + Delimiter
+ for key in keys
+ if Delimiter in key[len(Prefix) :]
+ }
+ yield {"CommonPrefixes": [{"Prefix": folder} for folder in sorted(folders)]}
def delete_objects(self, *, Bucket, Delete):
for obj in Delete["Objects"]:
@@ -292,3 +310,105 @@ def test_the_nn_candidate_writes_a_graph_whose_columns_follow_its_recorded_head_
auc = roc_auc_score(holdout.labels[rows, index], probabilities[head["head"]][rows])
assert auc == pytest.approx(head["holdout_auc"], abs=1e-4)
assert auc > 0.75
+
+
+def _stored_version(
+ model_name: str, version: str, recipe: dict[str, Any] | None, dataset_version: str = DATASET_VERSION
+) -> tuple[str, bytes]:
+ metadata: dict[str, Any] = {"model_name": model_name, "model_version": version}
+ if recipe is not None:
+ metadata |= {RECIPE_KEY: recipe, RECIPE_HASH_KEY: recipe_hash(recipe)}
+ key = model_object_key(PREFIX, model_name, version, METADATA_FILE)
+ key = key.replace(f"/{DATASET_VERSION}/", f"/{dataset_version}/", 1)
+ return key, json.dumps(metadata).encode()
+
+
+ACTION = HEADS_BY_NAME["action"]
+
+
+def _action_recipe(head: Head) -> dict[str, Any]:
+ return model_recipe(
+ model_kind=ONNX_MODEL_KIND, feature_set=REPORT_EMBEDDINGS_FEATURE_SET, heads=[head], learner_params={}
+ )
+
+
+ "stored, recipe, patch, changed",
+ [
+ ({"2026-08-18": "booster"}, "booster", None, None),
+ (
+ {"2026-08-18": "booster"},
+ "booster",
+ (settings, "INBOX_RANKING_TRAINING_LOOKBACK_DAYS", 120),
+ "lookback_days",
+ ),
+ ({"2026-08-18": "network"}, "network_more_epochs", None, "learner_params.epochs"),
+ ({"2026-08-17": "network", "2026-08-18": None}, "network_more_epochs", None, "learner_params.epochs"),
+ ({"2026-08-18": None}, "network_more_epochs", None, None),
+ ({"2026-08-20": "network"}, "network_more_epochs", None, None),
+ ({PARTITION: "network"}, "network", None, None),
+ ({PARTITION: "network"}, "network_more_epochs", None, "learner_params.epochs"),
+ ({"2026-08-18": "action"}, "action_without_status_labels", None, "heads.action.status_labels"),
+ ({"2026-08-18": "action"}, "action_without_label_columns", None, "heads.action.label_columns"),
+ (
+ {"2026-08-18": "network_with_vectors"},
+ "network_with_vectors",
+ (project_vectors, "HISTORY_LIMIT", 10),
+ "project_vector_settings.history_limit",
+ ),
+ (
+ {"2026-08-18": "network_with_vectors"},
+ "network_with_vectors",
+ (recipe_module, "MIN_PROJECT_HISTORY_REPORTS", 1),
+ "project_vector_settings.min_history_reports",
+ ),
+ ],
+ ids=[
+ "daily_retrain",
+ "env_lookback_change",
+ "network_config_change",
+ "skips_a_version_without_a_hash",
+ "nothing_hashed_to_compare",
+ "ignores_later_versions",
+ "rerun_with_the_same_recipe",
+ "rerun_with_another_recipe",
+ "head_status_labels_change",
+ "head_label_columns_change",
+ "project_history_limit_change",
+ "project_min_history_change",
+ ],
+)
+def test_check_recipe_refuses_a_new_recipe_under_an_old_name(monkeypatch, stored, recipe, patch, changed):
+ recipes = {
+ "booster": lambda: booster_recipe(REPORT_EMBEDDINGS_FEATURE_SET),
+ "network": lambda: network_recipe(NetworkConfig(), ()),
+ "network_more_epochs": lambda: network_recipe(NetworkConfig(epochs=NetworkConfig().epochs + 5), ()),
+ "network_with_vectors": lambda: network_recipe(NetworkConfig(), PROJECT_VECTOR_KINDS),
+ "action": lambda: _action_recipe(ACTION),
+ "action_without_status_labels": lambda: _action_recipe(dataclasses.replace(ACTION, status_labels=False)),
+ "action_without_label_columns": lambda: _action_recipe(dataclasses.replace(ACTION, label_columns=())),
+ }
+ objects = dict(
+ _stored_version("family", version, None if name is None else recipes[name]())
+ for version, name in stored.items()
+ )
+ if patch is not None:
+ monkeypatch.setattr(*patch)
+ check = lambda: check_recipe(_S3(objects), "bucket", PREFIX, "family", PARTITION, recipes[recipe]())
+ if changed is None:
+ check()
+ return
+ with pytest.raises(dagster.Failure) as failure:
+ check()
+ assert changed in (failure.value.description or "")
+
+
+def test_check_recipe_finds_the_recipe_under_an_earlier_dataset_version():
+ # A DATASET_VERSION bump moves the models to a new folder, so searching only the current one
+ # would let the old name train on the new dataset with no recipe to compare against.
+ earlier = {**booster_recipe(REPORT_EMBEDDINGS_FEATURE_SET), "dataset_version": "v0"}
+ objects = dict([_stored_version("family", "2026-08-18", earlier, dataset_version="v0")])
+
+ with pytest.raises(dagster.Failure) as failure:
+ check_recipe(_S3(objects), "bucket", PREFIX, "family", PARTITION, booster_recipe(REPORT_EMBEDDINGS_FEATURE_SET))
+ assert "dataset_version" in (failure.value.description or "")
diff --git a/products/signals/dags/inbox_ranking/tests/test_training.py b/products/signals/dags/inbox_ranking/tests/test_training.py
index 55b3f3feb20c..435b7c6eb045 100644
--- a/products/signals/dags/inbox_ranking/tests/test_training.py
+++ b/products/signals/dags/inbox_ranking/tests/test_training.py
@@ -73,6 +73,7 @@ from products.signals.dags.inbox_ranking.training.dag import (
_pinned_models,
_publish_manifest,
_train_candidate,
+ booster_recipe,
candidate_metadata,
champion_object_key,
embeddings_extras,
@@ -107,6 +108,7 @@ from products.signals.dags.inbox_ranking.training.promotion import (
PromotionDecision,
decide_promotion,
)
+from products.signals.dags.inbox_ranking.training.recipe import RECIPE_HASH_KEY, recipe_hash
from products.signals.dags.inbox_ranking.training.served import (
event_model_role,
served_events,
@@ -1919,6 +1921,8 @@ def test_candidate_metadata_declares_the_set_it_was_fit_on(family):
assert metadata["feature_set"] == family.feature_set.name
assert metadata["feature_schema_version"] == family.feature_set.schema_version
assert model_mismatch(metadata) is None
+ # The next day's recipe check compares against this hash, so it must be the one the check computes.
+ assert metadata[RECIPE_HASH_KEY] == recipe_hash(booster_recipe(family.feature_set))
def test_build_examples_carries_the_columns_of_the_set_it_is_given():
diff --git a/products/signals/dags/inbox_ranking/training/dag.py b/products/signals/dags/inbox_ranking/training/dag.py
index fbf0262ec79b..7b6b9818af4e 100644
--- a/products/signals/dags/inbox_ranking/training/dag.py
+++ b/products/signals/dags/inbox_ranking/training/dag.py
@@ -69,6 +69,7 @@ from products.signals.backend.ranking.overrides import read_ranking_overrides
from products.signals.backend.ranking.serving_manifest import (
DEFAULT_MODEL_KIND,
ServingManifest,
+ model_key,
serving_manifest_key,
serving_project_vectors_key,
)
@@ -117,6 +118,12 @@ from products.signals.dags.inbox_ranking.training.heads import (
HEADS_BY_NAME,
)
from products.signals.dags.inbox_ranking.training.promotion import decide_promotion
+from products.signals.dags.inbox_ranking.training.recipe import (
+ RECIPE_HASH_KEY,
+ model_recipe,
+ recipe_mismatch,
+ with_recipe,
+)
from products.signals.dags.inbox_ranking.training.serving import FamilyModels, ManifestDecision, compose_manifest
from products.signals.dags.inbox_ranking.training.telemetry import (
HeadExampleCounts,
@@ -258,6 +265,56 @@ def read_served_pointer(client, bucket: str, prefix: str) -> dict[str, Any] | No
return _read_json_if_exists(client, bucket, served_object_key(prefix))
+def previous_recipe_metadata(
+ client, bucket: str, prefix: str, model_name: str, partition_key: str
+) -> dict[str, Any] | None:
+ """The `metadata.json` of the newest version of `model_name` at or before `partition_key` that
+ carries a recipe hash, or None. A re-run of a partition compares with that partition's own saved
+ recipe, so a re-run cannot replace a version with another recipe. A version written before the
+ hash existed is passed over.
+
+ Every dataset version is searched, because a `DATASET_VERSION` bump moves the models to a new
+ folder and would otherwise hide the recipe the name was trained with."""
+ paginator = client.get_paginator("list_objects_v2")
+
+ def folders(parent: str) -> list[str]:
+ return [
+ common["Prefix"]
+ for page in paginator.paginate(Bucket=bucket, Prefix=parent, Delimiter="/")
+ for common in page.get("CommonPrefixes", [])
+ ]
+
+ candidates: list[tuple[str, bool, str]] = []
+ for dataset_folder in folders(f"{prefix}/{MODELS_TABLE}/"):
+ version_folder = f"{dataset_folder}{model_name}/dt="
+ is_current = dataset_folder == f"{prefix}/{MODELS_TABLE}/{DATASET_VERSION}/"
+ for folder in folders(version_folder):
+ version = folder[len(version_folder) :].rstrip("/")
+ if version <= partition_key:
+ candidates.append((version, is_current, f"{folder}{METADATA_FILE}"))
+ for _, _, key in sorted(candidates, reverse=True):
+ metadata = _read_json_if_exists(client, bucket, key)
+ if metadata is not None and metadata.get(RECIPE_HASH_KEY):
+ return metadata
+ return None
+
+
+def check_recipe(
+ client, bucket: str, prefix: str, model_name: str, partition_key: str, recipe: Mapping[str, Any]
+) -> None:
+ """Fail the run, without a retry, when `recipe` is not the recipe `model_name` was trained with."""
+ mismatch = recipe_mismatch(
+ model_name, recipe, previous_recipe_metadata(client, bucket, prefix, model_name, partition_key)
+ )
+ if mismatch is not None:
+ raise dagster.Failure(description=mismatch, allow_retries=False)
+
+
+def booster_recipe(feature_set: FeatureSet) -> dict[str, Any]:
+ """The recipe of a per-head XGBoost family fit on `feature_set`."""
+ return model_recipe(model_kind=DEFAULT_MODEL_KIND, feature_set=feature_set, heads=HEADS, learner_params=XGB_PARAMS)
+
+
def snapshot_dates(partition_key: str, lookback_days: int) -> list[datetime.date]:
end = datetime.date.fromisoformat(partition_key)
return [end - datetime.timedelta(days=offset) for offset in range(lookback_days, -1, -1)]
@@ -538,7 +595,7 @@ def candidate_metadata(
run_id: str,
windows: Mapping[str, Mapping[str, object]] | None = None,
) -> dict[str, Any]:
- return {
+ metadata = {
"model_name": model_name,
"model_version": partition_key,
"dataset_version": DATASET_VERSION,
@@ -573,6 +630,7 @@ def candidate_metadata(
# Heads with nothing to fit on this partition; recorded so the per-head series has no gap.
"skipped_heads": skipped,
}
+ return with_recipe(metadata, booster_recipe(feature_set))
def paired_champion_grades(
@@ -615,6 +673,9 @@ def inbox_ranking_model_candidate(context: dagster.AssetExecutionContext) -> Non
partition_key = context.partition_key
bucket, prefix, client = dataset_bucket(), settings.INBOX_RANKING_DATASET_S3_PREFIX, s3_client()
+ # Every family's recipe is checked before any family trains, so a refused run writes nothing.
+ for family in families_trained_in(MAIN_LOCATION):
+ check_recipe(client, bucket, prefix, family.name, partition_key, booster_recipe(family.feature_set))
metadata: dict[str, dagster.MetadataValue] = {}
for family in families_trained_in(MAIN_LOCATION):
metadata |= _train_candidate(context, client, bucket, prefix, partition_key, family)
@@ -867,6 +928,8 @@ def decide_family_champion(
incumbent_champion_version=incumbent_champion_version,
champion_aucs=champion_aucs,
champion_eces=champion_eces,
+ recipe_hash=candidate.get(RECIPE_HASH_KEY),
+ incumbent_recipe_hash=(champion or {}).get(RECIPE_HASH_KEY),
)
],
)
@@ -1001,6 +1064,14 @@ def _publish_manifest(context: RunContext, partition_key: str, run_id: str) -> d
present_keys=publication.present,
bytes_copied=publication.bytes_copied,
mirror_published=mirror_published,
+ recipe_hashes={
+ model_key(metadata["model_name"], metadata["model_version"]): metadata.get(RECIPE_HASH_KEY)
+ for metadata in (
+ *([served] if served is not None else []),
+ *pinned,
+ *(record for family in families for record in (family.candidate, family.champion) if record),
+ )
+ },
)
],
)
diff --git a/products/signals/dags/inbox_ranking/training/recipe.py b/products/signals/dags/inbox_ranking/training/recipe.py
new file mode 100644
index 000000000000..1fcda2a59c95
--- /dev/null
+++ b/products/signals/dags/inbox_ranking/training/recipe.py
@@ -0,0 +1,111 @@
+"""The recipe of a model version: everything that makes one day's fit comparable to the next.
+
+`model_name` names a recipe, and `model_version` is the day it was fit on. A candidate's
+`metadata.json` carries `recipe` (the inputs below, as a dict) and `recipe_hash`, a stable hash of
+that dict. The recipe holds what a daily retrain keeps fixed: the learner, the feature set, the
+learner's settings, each head's label definition and horizon, the example window and budget, and
+the dataset version. It never holds dates, the `run_id`, row counts or metrics.
+
+The promotion gate grades a candidate only against the served model of its own `model_name`, so two
+recipes under one name would be compared as if they were one series. The candidate asset refuses
+that: it compares its hash with the newest version of the same name, at or before its partition,
+that has one, and fails before it writes the partition when they differ. The fix is a new `model_name`, not an override.
+"""
+
+import json
+import hashlib
+from collections.abc import Mapping, Sequence
+from typing import Any
+
+from posthog import settings
+
+from products.signals.backend.ranking.features import MIN_PROJECT_HISTORY_REPORTS, FeatureSet
+from products.signals.dags.inbox_ranking.common import DATASET_VERSION
+from products.signals.dags.inbox_ranking.dataset import project_vectors
+from products.signals.dags.inbox_ranking.training.heads import Head
+
+RECIPE_KEY = "recipe"
+RECIPE_HASH_KEY = "recipe_hash"
+
+
+def model_recipe(
+ *,
+ model_kind: str,
+ feature_set: FeatureSet,
+ heads: Sequence[Head],
+ learner_params: Mapping[str, Any],
+ project_vector_kinds: Sequence[str] = (),
+) -> dict[str, Any]:
+ """`learner_params` are the booster params for XGBoost, or the network config for `nn`."""
+ return {
+ "model_kind": model_kind,
+ "feature_set": feature_set.name,
+ "feature_schema_version": feature_set.schema_version,
+ "project_vector_kinds": list(project_vector_kinds),
+ # Both change the project inputs, so they matter only to a model that reads project vectors.
+ "project_vector_settings": (
+ {
+ "history_limit": project_vectors.HISTORY_LIMIT,
+ "min_history_reports": MIN_PROJECT_HISTORY_REPORTS,
+ }
+ if project_vector_kinds
+ else {}
+ ),
+ "learner_params": dict(learner_params),
+ "heads": {
+ head.name: {
+ "cohort": head.cohort.__name__,
+ "label": head.label.__name__,
+ "horizon_days": head.horizon_days,
+ # Both drop example rows, so a change to either changes what the head is fit on.
+ "label_columns": list(head.label_columns),
+ "status_labels": head.status_labels,
+ }
+ for head in heads
+ },
+ "lookback_days": settings.INBOX_RANKING_TRAINING_LOOKBACK_DAYS,
+ "holdout_days": settings.INBOX_RANKING_TRAINING_HOLDOUT_DAYS,
+ "example_grain": feature_set.example_grain,
+ "max_examples_per_head": feature_set.max_examples_per_head,
+ "dataset_version": DATASET_VERSION,
+ }
+
+
+def recipe_hash(recipe: Mapping[str, Any]) -> str:
+ canonical = json.dumps(recipe, sort_keys=True, separators=(",", ":"), default=str)
+ return hashlib.sha256(canonical.encode()).hexdigest()[:16]
+
+
+def with_recipe(metadata: Mapping[str, Any], recipe: Mapping[str, Any]) -> dict[str, Any]:
+ return {**metadata, RECIPE_KEY: dict(recipe), RECIPE_HASH_KEY: recipe_hash(recipe)}
+
+
+def _flatten(value: Any, path: str = "") -> dict[str, Any]:
+ if isinstance(value, Mapping):
+ flat: dict[str, Any] = {}
+ for key, inner in value.items():
+ flat |= _flatten(inner, f"{path}.{key}" if path else str(key))
+ return flat
+ return {path: value}
+
+
+def changed_recipe_keys(previous: Mapping[str, Any], current: Mapping[str, Any]) -> list[str]:
+ """The dotted paths whose values differ, such as `learner_params.epochs`."""
+ before, after = _flatten(previous), _flatten(current)
+ return sorted(key for key in before.keys() | after.keys() if before.get(key) != after.get(key))
+
+
+def recipe_mismatch(model_name: str, recipe: Mapping[str, Any], previous: Mapping[str, Any] | None) -> str | None:
+ """Why a candidate with `recipe` must not be written under `model_name`, or None when it may.
+
+ `previous` is the metadata of the newest version of the name, at or before this partition, that
+ has a hash. None
+ means there is nothing to compare: the name is new, or its versions predate the hash."""
+ if previous is None or previous.get(RECIPE_HASH_KEY) == recipe_hash(recipe):
+ return None
+ changed = changed_recipe_keys(previous.get(RECIPE_KEY) or {}, recipe)
+ return (
+ f"{model_name} recipe changed since {model_name}@{previous.get('model_version')} "
+ f"(changed: {', '.join(changed) or 'unknown'}). A recipe change needs a new model_name in "
+ "MODEL_FAMILIES, so the promotion gate never compares two recipes."
+ )
diff --git a/products/signals/dags/inbox_ranking/training/telemetry.py b/products/signals/dags/inbox_ranking/training/telemetry.py
index e8590b91168f..e045fe87fe7c 100644
--- a/products/signals/dags/inbox_ranking/training/telemetry.py
+++ b/products/signals/dags/inbox_ranking/training/telemetry.py
@@ -48,6 +48,7 @@ _CANDIDATE_CONTEXT_KEYS = (
"feature_schema_version",
"lookback_days",
"holdout_days",
+ "recipe_hash",
)
# Object names inside the models prefix; meaningless on a chart.
_HEAD_FILE_KEYS = ("file", "holdout_file")
@@ -151,12 +152,15 @@ def promotion_event(
incumbent_champion_version: str,
champion_aucs: Mapping[str, float],
champion_eces: Mapping[str, float],
+ recipe_hash: str | None = None,
+ incumbent_recipe_hash: str | None = None,
) -> TrainingEvent:
"""`champion_aucs` and `champion_eces` were scored by the incumbent on this candidate's holdout.
The incumbent is the served model when it has `model_name`, else `none`.
Training never moves the served pointer, so `promoted` is always false. A person promotes
- through `inbox_ranking_promote_job`, which captures `MODEL_PROMOTED_EVENT`."""
+ through `inbox_ranking_promote_job`, which captures `MODEL_PROMOTED_EVENT`.
+ The two recipe hashes are None for a version written before the hash existed."""
return TrainingEvent(
event=PROMOTION_DECIDED_EVENT,
properties={
@@ -170,6 +174,8 @@ def promotion_event(
"skipped_heads": list(decision.skipped_heads),
"champion_version": incumbent_champion_version,
"incumbent_champion_version": incumbent_champion_version,
+ "recipe_hash": recipe_hash,
+ "incumbent_recipe_hash": incumbent_recipe_hash,
**{f"champion_{head}_auc_on_this_holdout": auc for head, auc in champion_aucs.items()},
**{f"champion_{head}_ece_on_this_holdout": ece for head, ece in champion_eces.items()},
},
@@ -194,10 +200,12 @@ def serving_manifest_event(
present_keys: Sequence[str] = (),
bytes_copied: int = 0,
mirror_published: bool | None = None,
+ recipe_hashes: Mapping[str, str | None] | None = None,
) -> TrainingEvent:
"""What the day's manifest published. A run that wrote nothing still reports, with `reason`:
the manifest is what makes the sweep serve anything, so a silent gap in this series would read
- the same as a day of ordinary publishing."""
+ the same as a day of ordinary publishing. `recipe_hashes` maps a model key to the `recipe_hash`
+ of its `metadata.json`."""
properties: dict[str, object] = {
"model_version": partition_key,
"run_id": run_id,
@@ -214,6 +222,7 @@ def serving_manifest_event(
"served_key": manifest.served.key,
"model_keys": [entry.key for entry in manifest.models],
"model_roles": {entry.key: entry.roles for entry in manifest.models},
+ "model_recipe_hashes": {entry.key: (recipe_hashes or {}).get(entry.key) for entry in manifest.models},
}
if mirror_published is not None:
properties["mirror_published"] = mirror_published