:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(signals): judge inbox ranking experiments per team with an A/A null (#129309)
posthog[bot]c389e828abfb
diff --git a/products/signals/dags/inbox_ranking/AGENTS.md b/products/signals/dags/inbox_ranking/AGENTS.mdindex d02ef0a23bcf..2d9429af3800 100644--- a/products/signals/dags/inbox_ranking/AGENTS.md+++ b/products/signals/dags/inbox_ranking/AGENTS.md@@ -45,7 +45,7 @@ Read `README.md` first for what the dataset is and how partitions behave. This f- `HEADS` holds only the heads the inbox reads (`HeadRanker` in `backend/ranking/inbox_sort.py`). Add a head there only together with a surface that reads it. The promotion gate skips a champion head that is not in `HEADS`, so a trim does not block promotion.- `nn_v3` and `nn_with_prio_v2` are trained by the `signals_ml` location (`nn/dag.py`), never by the main one. They share one `inbox_ranking_nn_examples` object, and each has its own candidate and champion asset from `family_assets`. The `nn_v3` assets keep the names `inbox_ranking_nn_model_{candidate,champion}`, so their Dagster history continues. A later family's candidate depends on the previous family's champion, for memory and not for data: two candidates at once can exceed the job's memory limit. A family whose set reads a state column the examples object lacks skips with a warning rather than training on zeros. torch is imported only inside the candidate asset's body (through `nn/train.py` and `multihead/network.py`). `signals_ml` must import without torch, and `test_multihead_network.py` pins that. Its decision goes through `decide_family_champion` in `training/dag.py`, graded through the incumbent's `model.holdout.onnx`. Keep `signals_ml` commented out in `.dagster_home/workspace.yaml`, and keep its schedule (04:30 UTC) running by default on prod US only.-- An experiment is a `MODEL_FAMILIES` entry with `experiment=True`, a `baseline` serving family and an `expires` day. Only `inbox_ranking_nn_experiments_job` (`nn/experiments.py`) trains it, with its own schedule and resource tags, so a modeling idea never adds to the `nn` job's runtime or memory budget. It reads the day's `inbox_ranking_nn_examples` object and never rebuilds it. An experiment never enters the serving manifest, and `_pin_metadata` refuses its key, so the promote job, the served pointer and the pins cannot name it. To serve its recipe, register it as a serving family under a new `model_name`. Its `metadata.json` carries a `vs_baseline` block: the experiment and the baseline fit with each of `EXPERIMENT_SEEDS` on the same train rows and graded on the same holdout rows. An extra side input for an experiment goes through the registry in `training/side_inputs.py`, which refuses a frame without a timezone-aware `as_of_column` and drops rows known after the partition end. A feature set reads it through `rows_as_of`.+- An experiment is a `MODEL_FAMILIES` entry with `experiment=True`, a `baseline` serving family and an `expires` day. Only `inbox_ranking_nn_experiments_job` (`nn/experiments.py`) trains it, with its own schedule and resource tags, so a modeling idea never adds to the `nn` job's runtime or memory budget. It reads the day's `inbox_ranking_nn_examples` object and never rebuilds it. An experiment never enters the serving manifest, and `_pin_metadata` refuses its key, so the promote job, the served pointer and the pins cannot name it. To serve its recipe, register it as a serving family under a new `model_name`. Its `metadata.json` carries a `vs_baseline` block: the experiment and the baseline fit with each of `EXPERIMENT_SEEDS` on the same train rows and graded on the same holdout rows. An A/A experiment (`nn_with_prio_aa_v1`) copies its baseline's recipe and sets `seed_offset`, so its arm fits on other seeds: on the baseline's own seeds the two arms are one fit and every delta is zero. An extra side input for an experiment goes through the registry in `training/side_inputs.py`, which refuses a frame without a timezone-aware `as_of_column` and drops rows known after the partition end. A feature set reads it through `rows_as_of`.- `inbox_ranking_nn_pair_examples` (`nn/pairs.py`) has one row per (report example, impressed user), with its own `PAIR_EXAMPLES_SCHEMA_VERSION`. Its own job has no schedule, and the experiments job builds it for the same partition before the pair experiments, only on a day with a live pair experiment. Keep the report experiments asset free of a dependency on it, so a failed pair step skips the pair experiments only. It takes its report rows and holdout from the `open` head's rows of the day's `inbox_ranking_nn_examples` object and never rebuilds them. Its impressions and labels come from the dt=B+`PAIR_HORIZON_DAYS` interactions partition, and its user inputs are as of the start of day B, so the target report is never in the user's history. The object holds no vector: a reader rebuilds the user vectors with `pair_user_vectors`. `user_vectors()` lives next to `project_vectors()` and shares `known_before` and `_usable_embeddings` with it; never give it its own as-of rule or embedding gate.- A pair-grain family (`grain="pair"`) is always an experiment, trained by `nn/pair_lane.py` in the experiments job. Its baseline is another pair experiment and its `reference` a report-grain serving family. `family_problems` checks both at import. Keep the report-grain `vs_baseline` block, its events and the report families' recipe free of pair fields: an added recipe key changes every serving family's hash, and `check_recipe` then fails each retrain. Fit the reference on report train rows only, so no holdout pair's report trains it.- A pair set (`PairUserHistoryFeatureSet`) is the user-history contract of a pair family. Its set with `user_history` off is the pair baseline: the same width, the same reviewer flag, and zero in every user column and user offset whatever the pair rows hold. Build each arm's inputs from the pair rows for that arm's set, never from another arm's inputs. The report part may be built once per report, but the user columns and offsets are per pair, so one user's history never reaches another user's pair of the same report. A pair set stays out of `FEATURE_SETS`, because the state loader reads every registered set's columns and the state snapshot has no pair columns.diff --git a/products/signals/dags/inbox_ranking/README.md b/products/signals/dags/inbox_ranking/README.mdindex a658b5d17369..205de476ff57 100644--- a/products/signals/dags/inbox_ranking/README.md+++ b/products/signals/dags/inbox_ranking/README.md@@ -470,7 +470,8 @@ An experiment is a `MODEL_FAMILIES` entry with `experiment=True`, a `baseline` s- **Its own job.** `inbox_ranking_nn_experiments_job` (`nn/experiments.py`, on `signals_ml`) trains every live experiment, one after the other. It has its own runtime and memory tags, so it never uses the `nn` job's budget. It holds three assets, run one step at a time in one process: `inbox_ranking_nn_experiment_candidates` (the report-grain experiments), `inbox_ranking_nn_pair_examples` and `inbox_ranking_nn_pair_experiment_candidates` (the pair-grain experiments, after the pair examples). The report experiments do not depend on the pair examples, so a failed pair step skips only the pair experiments. Its schedule (08:00 UTC, after the `nn` job's 3h budget) selects the report asset when a report experiment is live and the two pair assets when a pair experiment is live, and skips when nothing is live.- **No new examples.** It reads the day's `inbox_ranking_nn_examples` object. A day without that object is skipped, and so is a family after its `expires` day.- **Never served.** The serving manifest reads only `serving_families()`. The pin check refuses an experiment key, so the promote job, the served pointer and the pins cannot name one. To serve the recipe, register it as a serving family under a new `model_name`.-- **Paired read.** The experiment and its baseline each fit once per seed in `EXPERIMENT_SEEDS`, on the same train rows. `metadata.json` holds a `vs_baseline` block per head: each seed's within-team and pooled holdout AUC for both arms, and the mean, min and max of the gaps. One `inbox_ranking_experiment_compared` event per head carries the gaps.+- **Paired read.** The experiment and its baseline each fit once per seed in `EXPERIMENT_SEEDS`, on the same train rows. `metadata.json` holds a `vs_baseline` block per head: each seed's holdout AUC for both arms, and the mean, min and max of the gaps. The reads are `within_team` (pairs pooled over teams), `within_team_macro` (each team weighs the same), the same two without the dogfood project (`within_team_non_dogfood`, `within_team_non_dogfood_macro`), and `pooled` for reference. The non-dogfood macro delta also gets a team bootstrap interval, the method the pair lane uses. One `inbox_ranking_experiment_compared` event per head carries the gaps of every read and the interval's `low`, `high` and `teams`.+- **A/A control.** `nn_with_prio_aa_v1` is `nn_with_prio_v2` against itself. Its `seed_offset` fits its arm on seeds other than the baseline's, so its deltas show how large a delta from noise alone is. Read an experiment's delta against the A/A deltas of the same days before calling it a win.- **Side inputs.** A side input beyond the report vectors needs a loader in `training/side_inputs.py`. The registry refuses a frame without a timezone-aware `as_of_column` and drops the rows known after the partition end. A feature set in the dags reads the frame through `rows_as_of`. A set in `features.py` cannot import the dags, so it applies the same `<= as_of` cut on the loader's column itself.The live report-grain experiment is `nn_with_prio_sources_v1`, graded against `nn_with_prio_v2` until 2026-11-11. The pair-grain experiments are under [User-history families](#user-history-families). Its set, `report_embeddings_priority_sources`, adds the source products of the report's signals: one column per product in a fixed vocabulary, `source_other`, `source_count` and `has_sources`. The `report_sources` loader reads only the metadata columns of `inbox_signal_embeddings`, from 30 days before the training lookback to dt=D. A moment takes the latest emission per `(team_id, signal_id)` inserted at or before it and drops a retracted signal. It never reads the impression labels' `source_products` column, which holds the latest impression's value. The scoring sweep cannot build this set, so the recipe needs serving work before it can become a serving family.diff --git a/products/signals/dags/inbox_ranking/nn/dag.py b/products/signals/dags/inbox_ranking/nn/dag.pyindex 50bc7a05dce0..625c9ded9222 100644--- a/products/signals/dags/inbox_ranking/nn/dag.py+++ b/products/signals/dags/inbox_ranking/nn/dag.py@@ -306,11 +306,13 @@ def train_candidate(vs_baseline = Noneif baseline_reports is not None and family.baseline is not None:baseline_inputs = NetworkInputs(reports=baseline_reports, context=baseline_context or inputs.context)+ experiment_seeds = family.experiment_seeds(EXPERIMENT_SEEDS)vs_baseline = vs_baseline_block(examples.table,baseline=family.baseline,seeds=EXPERIMENT_SEEDS,- experiment_scores=seed_scores(examples.table, inputs, NN_HEADS, NETWORK_CONFIG, EXPERIMENT_SEEDS),+ experiment_seeds=experiment_seeds,+ experiment_scores=seed_scores(examples.table, inputs, NN_HEADS, NETWORK_CONFIG, experiment_seeds),baseline_scores=seed_scores(examples.table, baseline_inputs, NN_HEADS, NETWORK_CONFIG, EXPERIMENT_SEEDS),)metadata = write_candidate(diff --git a/products/signals/dags/inbox_ranking/nn/model.py b/products/signals/dags/inbox_ranking/nn/model.pyindex c11de3f643e7..b0b4e23c5ae7 100644--- a/products/signals/dags/inbox_ranking/nn/model.py+++ b/products/signals/dags/inbox_ranking/nn/model.py@@ -12,6 +12,7 @@ from collections.abc import Mapping, Sequencefrom typing import Anyimport numpy as np+import pandas as pdfrom posthog import settingsfrom posthog.dataclasses import frozen@@ -34,7 +35,7 @@ from products.signals.dags.inbox_ranking.training.families import PAIR_GRAIN, Mofrom products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME, Headfrom products.signals.dags.inbox_ranking.training.recipe import model_recipe, with_recipefrom products.signals.dags.inbox_ranking.training.train import HeadMetrics, HoldoutGrade, _auc, holdout_grade-from products.signals.dags.inbox_ranking.training.within_team import within_team_auc+from products.signals.dags.inbox_ranking.training.within_team import group_aucs, summarize_groups, team_intervalHOLDOUT_MODEL_FILE = "model.holdout.onnx"@@ -253,14 +254,20 @@ def vs_baseline_block(seeds: Sequence[int],experiment_scores: Sequence[np.ndarray],baseline_scores: Sequence[np.ndarray],+ experiment_seeds: Sequence[int] | None = None,) -> dict[str, Any]:"""The experiment against its baseline, per head, on the same holdout rows.Each seed fits both arms on the same train rows, so the two AUCs of one seed form a pair. Oneseed's gap can be noise from the network's initialization alone, so the block keeps the range- of the gaps across seeds. `within_team` is the read the inbox order depends on, and `pooled` is- there for reference. `within_team_non_dogfood` leaves out PostHog's own project, whose inbox use- is not a customer's. A seed where either arm has no AUC drops out of the pairs.+ of the gaps across seeds. `experiment_seeds` are the experiment arm's seeds when they differ from+ the baseline's `seeds`, as in an A/A experiment.++ `within_team` is the read the inbox order depends on, and `pooled` is there for reference.+ `within_team_non_dogfood` leaves out PostHog's own project, whose inbox use is not a customer's.+ Each `_macro` read weighs every team the same, so a few large inboxes cannot carry it. The+ non-dogfood macro delta also gets a team bootstrap interval. A seed where either arm has no AUC+ drops out of the pairs."""train_rows = table.network_train_rows()heads: dict[str, dict[str, Any]] = {}@@ -268,28 +275,35 @@ def vs_baseline_block(rows = table.head_rows(index, holdout=True)y = table.labels[rows, index].astype(int)team_ids = table.team_ids(rows)- customer = team_ids != LABELS_TEAM_ID- heads[name] = {+ grade: dict[str, Any] = {"train_rows": int((~np.isnan(table.labels[train_rows, index])).sum()),"holdout_rows": len(rows),"holdout_positives": int(y.sum()),- "within_team": _paired(- [within_team_auc(y, scores[rows, index], team_ids).auc for scores in experiment_scores],- [within_team_auc(y, scores[rows, index], team_ids).auc for scores in baseline_scores],- ),- "within_team_non_dogfood": _paired(- [- within_team_auc(y[customer], scores[rows, index][customer], team_ids[customer]).auc- for scores in experiment_scores- ],- [- within_team_auc(y[customer], scores[rows, index][customer], team_ids[customer]).auc- for scores in baseline_scores- ],- ),- "pooled": _paired(- [_auc(y, scores[rows, index]) for scores in experiment_scores],- [_auc(y, scores[rows, index]) for scores in baseline_scores],- ),}- return {"baseline": baseline, "seeds": list(seeds), "heads": heads}+ scopes = {"within_team": np.ones(len(rows), dtype=bool), "within_team_non_dogfood": team_ids != LABELS_TEAM_ID}+ for scope, mask in scopes.items():+ scoped_y, scoped_rows, scoped_teams = y[mask], rows[mask], team_ids[mask]+ experiment = [+ group_aucs(scoped_y, scores[scoped_rows, index], scoped_teams) for scores in experiment_scores+ ]+ base = [group_aucs(scoped_y, scores[scoped_rows, index], scoped_teams) for scores in baseline_scores]+ experiment_reads = [summarize_groups(per_team) for per_team in experiment]+ base_reads = [summarize_groups(per_team) for per_team in base]+ grade[scope] = _paired([read.auc for read in experiment_reads], [read.auc for read in base_reads])+ grade[f"{scope}_macro"] = _paired(+ [read.macro for read in experiment_reads], [read.macro for read in base_reads]+ )+ if scope == "within_team_non_dogfood":+ teams = pd.Series(scoped_teams, index=scoped_teams).groupby(level=0).first()+ grade[f"{scope}_macro_team_interval"] = team_interval(experiment, base, teams)+ grade["pooled"] = _paired(+ [_auc(y, scores[rows, index]) for scores in experiment_scores],+ [_auc(y, scores[rows, index]) for scores in baseline_scores],+ )+ heads[name] = grade+ return {+ "baseline": baseline,+ "seeds": list(seeds),+ "experiment_seeds": list(experiment_seeds if experiment_seeds is not None else seeds),+ "heads": heads,+ }diff --git a/products/signals/dags/inbox_ranking/nn/pair_grade.py b/products/signals/dags/inbox_ranking/nn/pair_grade.pyindex c17e7cf9c7fc..4ab9bf90bd1c 100644--- a/products/signals/dags/inbox_ranking/nn/pair_grade.py+++ b/products/signals/dags/inbox_ranking/nn/pair_grade.py@@ -24,15 +24,17 @@ from products.signals.dags.inbox_ranking.dataset.queries import LABELS_TEAM_IDfrom products.signals.dags.inbox_ranking.multihead.prototype import ExampleTablefrom products.signals.dags.inbox_ranking.nn.pairs import HISTORY_BUCKETSfrom products.signals.dags.inbox_ranking.training.families import PAIR_GRAIN-from products.signals.dags.inbox_ranking.training.within_team import group_aucs, summarize_groups+from products.signals.dags.inbox_ranking.training.within_team import (+ BOOTSTRAP_DRAWS,+ group_aucs,+ summarize_groups,+ team_interval,+)# PostHog's own project, where the inbox's builders are its users.DOGFOOD_TEAM_ID = LABELS_TEAM_ID# Prior views from which a user counts as having history, read as one slice as well as the buckets.WITH_HISTORY_MIN_VIEWS = 10-BOOTSTRAP_DRAWS = 1000-BOOTSTRAP_SEED = 0-INTERVAL_PERCENTILES = (2.5, 97.5)ALL = "all"NON_DOGFOOD = "non_dogfood"@@ -78,42 +80,6 @@ def _delta_summary(experiment: Sequence[float | None], other: Sequence[float | N}-def team_interval(- experiment: Sequence[pd.DataFrame | None],- other: Sequence[pd.DataFrame | None],- group_teams: pd.Series,- *,- draws: int = BOOTSTRAP_DRAWS,- seed: int = BOOTSTRAP_SEED,-) -> dict[str, Any]:- """A percentile interval for the macro within-user delta, resampling teams with replacement.-- Each user's delta is its AUC gap averaged over the paired seeds. A resample's macro delta is the- mean over every user of the drawn teams, a team drawn twice counting twice, so it is the same- statistic as the macro delta itself. Fewer than two teams give no interval.- """- per_seed = [- e["u"] / e["pairs"] - o["u"] / o["pairs"]- for e, o in zip(experiment, other)- if e is not None and o is not None and not e.empty- ]- if not per_seed:- return {"low": None, "high": None, "teams": 0, "draws": 0}- per_user = pd.concat(per_seed, axis=1).mean(axis=1)- per_team = (- pd.DataFrame({"delta": per_user.to_numpy(), "team": group_teams.loc[per_user.index].to_numpy()})- .groupby("team")["delta"]- .agg(["sum", "size"])- )- teams = len(per_team)- if teams < 2:- return {"low": None, "high": None, "teams": teams, "draws": 0}- weights = np.random.default_rng(seed).multinomial(teams, np.full(teams, 1 / teams), size=draws)- macro = (weights @ per_team["sum"].to_numpy()) / (weights @ per_team["size"].to_numpy())- low, high = np.percentile(macro, INTERVAL_PERCENTILES)- return {"low": float(low), "high": float(high), "teams": teams, "draws": draws}--def _slice_grade(outcomes: np.ndarray,groups: np.ndarray,diff --git a/products/signals/dags/inbox_ranking/tests/test_experiments.py b/products/signals/dags/inbox_ranking/tests/test_experiments.pyindex 6ca33d11a4d5..a18abd9fe36b 100644--- a/products/signals/dags/inbox_ranking/tests/test_experiments.py+++ b/products/signals/dags/inbox_ranking/tests/test_experiments.py@@ -35,7 +35,7 @@ from products.signals.dags.inbox_ranking.multihead.inputs import NetworkConfigfrom products.signals.dags.inbox_ranking.multihead.local import report_vectorsfrom products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable, project_context, report_matrixfrom products.signals.dags.inbox_ranking.nn import experiments, pair_lane-from products.signals.dags.inbox_ranking.nn.dag import NN_EXAMPLES_TABLE+from products.signals.dags.inbox_ranking.nn.dag import EXPERIMENT_SEEDS, NN_EXAMPLES_TABLEfrom products.signals.dags.inbox_ranking.nn.examples import NN_HEAD_NAMES, NN_HEADS, build_examples, examples_arrowfrom products.signals.dags.inbox_ranking.nn.model import network_recipe, onnx_probabilities, vs_baseline_blockfrom products.signals.dags.inbox_ranking.nn.pair_grade import DOGFOOD_TEAM_ID, pair_vs_baseline_block@@ -63,6 +63,7 @@ from products.signals.dags.inbox_ranking.training.families import (NN_MODEL_NAME,NN_PAIR_BASE_MODEL_NAME,NN_PAIR_USER_MODEL_NAME,+ NN_WITH_PRIO_AA_MODEL_NAME,NN_WITH_PRIO_FAMILY,NN_WITH_PRIO_MODEL_NAME,PAIR_GRAIN,@@ -360,9 +361,16 @@ def test_vs_baseline_pairs_each_seed_inside_each_team_on_the_same_holdout_rows()"holdout_positives",*(f"{read}_{key}"- for read in ("within_team", "pooled")+ for read in (+ "within_team",+ "within_team_non_dogfood",+ "within_team_macro",+ "within_team_non_dogfood_macro",+ "pooled",+ )for key in ("paired_seeds", "mean_delta_auc", "min_delta_auc", "max_delta_auc", "wins")),+ *(f"within_team_non_dogfood_macro_interval_{key}" for key in ("low", "high", "teams")),}paired = {"experiment_aucs": [1.0, 0.0],@@ -389,6 +397,48 @@ def test_vs_baseline_pairs_each_seed_inside_each_team_on_the_same_holdout_rows())assert mixed["heads"][first]["within_team"]["experiment_aucs"] == [0.5]assert mixed["heads"][first]["within_team_non_dogfood"]["experiment_aucs"] == [1.0]+ assert mixed["heads"][first]["within_team_non_dogfood_macro"]["experiment_aucs"] == [1.0]+ [mixed_event, *_] = experiment_comparison_events({"model_name": "experiment", "vs_baseline": mixed})+ assert [+ mixed_event.properties[f"{read}_mean_delta_auc"]+ for read in ("within_team", "within_team_non_dogfood", "within_team_non_dogfood_macro")+ ] == [0.5, 1.0, 1.0]++ # Customer team 1 ranks its four pairs right and customer team 3 ranks its one pair wrong, so the+ # pooled read follows team 1 while the macro read weighs both teams the same.+ uneven_labels = np.zeros((8, len(NN_HEAD_NAMES)), dtype=np.float32)+ uneven_labels[:, 0] = [1, 1, 0, 0, 1, 0, 1, 0]+ uneven = ExampleTable(+ rows=pd.DataFrame(+ {"report_id": list("abcdefgh"), "report_team_id": [1, 1, 1, 1, 3, 3, DOGFOOD_TEAM_ID, DOGFOOD_TEAM_ID]}+ ),+ labels=uneven_labels,+ holdout=np.ones_like(uneven_labels, dtype=bool),+ )+ experiment = uneven_labels.copy()+ experiment[4:6, 0] = 1 - experiment[4:6, 0]+ grade = vs_baseline_block(+ uneven,+ baseline=NN_MODEL_NAME,+ seeds=(0,),+ experiment_scores=[experiment],+ baseline_scores=[np.full_like(uneven_labels, 0.5)],+ )["heads"][first]+ assert grade["within_team_non_dogfood"]["mean_delta_auc"] == pytest.approx(0.3)+ assert grade["within_team_non_dogfood_macro"]["mean_delta_auc"] == pytest.approx(0.0)+ assert grade["within_team_macro"]["mean_delta_auc"] == pytest.approx(1 / 6)+ assert grade["within_team_non_dogfood_macro_team_interval"] == {"low": -0.5, "high": 0.5, "teams": 2, "draws": 1000}+++def test_the_aa_control_is_its_baseline_recipe_fit_on_other_seeds():+ aa = next(family for family in MODEL_FAMILIES if family.name == NN_WITH_PRIO_AA_MODEL_NAME)+ config = NetworkConfig()+ assert aa.baseline == NN_WITH_PRIO_MODEL_NAME+ assert network_recipe(aa, config, aa.project_vector_kinds) == network_recipe(+ NN_WITH_PRIO_FAMILY, config, NN_WITH_PRIO_FAMILY.project_vector_kinds+ )+ # On the baseline's own seeds the two arms are the same fit, and every delta is zero.+ assert set(aa.experiment_seeds(EXPERIMENT_SEEDS)).isdisjoint(EXPERIMENT_SEEDS)def _signal_log(signals: list[tuple[str, str, str, str]]) -> bytes:diff --git a/products/signals/dags/inbox_ranking/tests/test_nn.py b/products/signals/dags/inbox_ranking/tests/test_nn.pyindex 56bc72f84a4b..fbeb59c86aea 100644--- a/products/signals/dags/inbox_ranking/tests/test_nn.py+++ b/products/signals/dags/inbox_ranking/tests/test_nn.py@@ -58,6 +58,7 @@ from products.signals.dags.inbox_ranking.training.dag import (served_object_key,)from products.signals.dags.inbox_ranking.training.families import (+ AA_SEED_OFFSET,MODEL_FAMILIES,NN_FAMILY,NN_MODEL_NAME,@@ -262,6 +263,7 @@ _PRIO_EXPERIMENT = ModelFamily(experiment=True,baseline=NN_MODEL_NAME,expires=datetime.date(2026, 12, 31),+ seed_offset=AA_SEED_OFFSET,)NN_WITH_PRIO_FIXED_FAMILY = next(family for family in MODEL_FAMILIES if family.name == NN_WITH_PRIO_FIXED_MODEL_NAME)@@ -309,6 +311,8 @@ def test_the_nn_candidate_writes_a_graph_whose_columns_follow_its_recorded_head_# Only an experiment is graded against a baseline, with one paired AUC per seed and head.if family.experiment:assert metadata["vs_baseline"]["baseline"] == family.baseline+ assert metadata["vs_baseline"]["seeds"] == list(nn_dag.EXPERIMENT_SEEDS)+ assert metadata["vs_baseline"]["experiment_seeds"] == list(family.experiment_seeds(nn_dag.EXPERIMENT_SEEDS))assert {name: len(grade["within_team"]["experiment_aucs"])== len(grade["within_team"]["baseline_aucs"])diff --git a/products/signals/dags/inbox_ranking/training/families.py b/products/signals/dags/inbox_ranking/training/families.pyindex fe0285a1e9ed..4bafec300cd6 100644--- a/products/signals/dags/inbox_ranking/training/families.py+++ b/products/signals/dags/inbox_ranking/training/families.py@@ -42,6 +42,9 @@ NN_WITH_PRIO_SOURCES_MODEL_NAME = "nn_with_prio_sources_v1"NN_WITH_PRIO_FIXED_MODEL_NAME = "nn_with_prio_fixed_v1"NN_WITH_PRIO_FIXED_NODISMISS_MODEL_NAME = "nn_with_prio_fixed_nodismiss_v1"FIXED_EXPERIMENT_EXPIRES = datetime.date(2026, 11, 27)+NN_WITH_PRIO_AA_MODEL_NAME = "nn_with_prio_aa_v1"+# Far from any baseline seed, so the A/A arm never shares an initialization with its baseline.+AA_SEED_OFFSET = 100NN_PAIR_BASE_MODEL_NAME = "nn_pair_base_v1"NN_PAIR_USER_MODEL_NAME = "nn_pair_user_v1"PAIR_EXPERIMENT_EXPIRES = datetime.date(2026, 11, 25)@@ -67,6 +70,10 @@ class ModelFamily:`project_vector_kinds` are the team context kinds the network reads, in its input order. Only areport-grain experiment can change them. The pair lane reads `PROJECT_VECTOR_KINDS` only.+ `seed_offset` moves the experiment arm's comparison seeds away from the baseline's. Only a+ report-grain experiment can set it. An A/A experiment, with its baseline's exact recipe, sets it+ so that its deltas measure the noise of the comparison alone.+A pair-grain family is always an experiment and declares its `heads` from `PAIR_HEAD_NAMES`. Its`baseline` is another pair experiment, and it names a `baseline`, a `reference`, or both: thefirst pair family of a comparison has no pair baseline and is graded against the reference only."""@@ -83,9 +90,12 @@ class ModelFamily:heads: tuple[str, ...] = ()reference: str | None = Noneproject_vector_kinds: tuple[str, ...] = PROJECT_VECTOR_KINDS+ seed_offset: int = 0def __post_init__(self) -> None:self._check_project_vector_kinds()+ if self.seed_offset and (not self.experiment or self.grain == PAIR_GRAIN):+ raise ValueError(f"family {self.name} sets a seed offset; only a report-grain experiment can")if self.grain == PAIR_GRAIN:self._check_pair()return@@ -127,6 +137,10 @@ class ModelFamily:def expired(self, day: datetime.date) -> bool:return self.expires is not None and day > self.expires+ def experiment_seeds(self, seeds: Sequence[int]) -> tuple[int, ...]:+ """The seeds the experiment arm fits with, paired in order with the baseline's `seeds`."""+ return tuple(seed + self.seed_offset for seed in seeds)+# The families the training jobs train, in the order they are trained. A# family with no metadata for the day is skipped, so an entry can be added here before its trainer@@ -139,6 +153,8 @@ class ModelFamily:# report's signals as of the moment, graded against `nn_with_prio_v2`.# `nn_with_prio_fixed_v1` adds a `fixed` project vector to `nn_with_prio_v2`, because fixes reach# teams whose UI kinds are thin. `nn_with_prio_fixed_nodismiss_v1` swaps `dismiss` for `fixed`.+# `nn_with_prio_aa_v1` is an A/A control: `nn_with_prio_v2` against itself on other seeds, so its+# deltas show how large a null delta is.# `nn_pair_base_v1` and `nn_pair_user_v1` are the user-history comparison at pair grain: one network,# one input shape, the same pairs and seeds, and the same reviewer flag. The base holds every# user-history input at zero, so `nn_pair_user_v1` against it is the uplift of the user's history@@ -182,6 +198,16 @@ MODEL_FAMILIES: tuple[ModelFamily, ...] = (expires=FIXED_EXPERIMENT_EXPIRES,project_vector_kinds=("typical", "open", "action", "fixed"),),+ ModelFamily(+ name=NN_WITH_PRIO_AA_MODEL_NAME,+ feature_set=REPORT_EMBEDDINGS_PRIORITY_FEATURE_SET,+ learner=ONNX_MODEL_KIND,+ location=ML_LOCATION,+ experiment=True,+ baseline=NN_WITH_PRIO_MODEL_NAME,+ expires=datetime.date(2026, 12, 2),+ seed_offset=AA_SEED_OFFSET,+ ),ModelFamily(name=NN_PAIR_BASE_MODEL_NAME,feature_set=PAIR_BASE_FEATURE_SET,diff --git a/products/signals/dags/inbox_ranking/training/telemetry.py b/products/signals/dags/inbox_ranking/training/telemetry.pyindex 203b8d8d8a14..6ce5796eec63 100644--- a/products/signals/dags/inbox_ranking/training/telemetry.py+++ b/products/signals/dags/inbox_ranking/training/telemetry.py@@ -143,6 +143,15 @@ def promotion_event(_DELTA_KEYS = ("paired_seeds", "mean_delta_auc", "min_delta_auc", "max_delta_auc", "wins")+_INTERVAL_KEYS = ("low", "high", "teams")+# Saved insights read `within_team` and `pooled`, so the keys of those two reads never change.+_REPORT_READS = (+ "within_team",+ "within_team_non_dogfood",+ "within_team_macro",+ "within_team_non_dogfood_macro",+ "pooled",+)def _pair_comparison_events(block: Mapping[str, Any], context: Mapping[str, Any]) -> list[TrainingEvent]:@@ -174,8 +183,7 @@ def _pair_comparison_events(block: Mapping[str, Any], context: Mapping[str, Any]}interval = deltas.get("macro_team_interval") or {}properties |= {- f"{comparison}_within_user_macro_interval_{key}": interval.get(key)- for key in ("low", "high", "teams")+ f"{comparison}_within_user_macro_interval_{key}": interval.get(key) for key in _INTERVAL_KEYS}events.append(TrainingEvent(event=EXPERIMENT_COMPARED_EVENT, properties=properties))return events@@ -184,7 +192,8 @@ def _pair_comparison_events(block: Mapping[str, Any], context: Mapping[str, Any]def experiment_comparison_events(metadata: Mapping[str, Any]) -> list[TrainingEvent]:"""One event per head of an experiment candidate's `vs_baseline` block. The per-seed AUC listsstay in `metadata.json`: an insight cannot chart a JSON array, so the event carries the gaps.- A pair experiment's block gives one event per head, scope and slice instead."""+ The non-dogfood macro delta also carries its team interval. A pair experiment's block gives one+ event per head, scope and slice instead."""block = metadata.get("vs_baseline") or {}context = {key: metadata.get(key) for key in ("model_name", "model_version", "run_id", "recipe_hash")}if block.get("grain") == PAIR_GRAIN:@@ -198,10 +207,12 @@ def experiment_comparison_events(metadata: Mapping[str, Any]) -> list[TrainingEv"seeds": len(block.get("seeds", [])),"head": head,**{key: grade.get(key) for key in ("train_rows", "holdout_rows", "holdout_positives")},+ **{f"{read}_{key}": (grade.get(read) or {}).get(key) for read in _REPORT_READS for key in _DELTA_KEYS},**{- f"{read}_{key}": (grade.get(read) or {}).get(key)- for read in ("within_team", "pooled")- for key in ("paired_seeds", "mean_delta_auc", "min_delta_auc", "max_delta_auc", "wins")+ f"within_team_non_dogfood_macro_interval_{key}": (+ grade.get("within_team_non_dogfood_macro_team_interval") or {}+ ).get(key)+ for key in _INTERVAL_KEYS},},)diff --git a/products/signals/dags/inbox_ranking/training/within_team.py b/products/signals/dags/inbox_ranking/training/within_team.pyindex dfea533b13cd..088f40a77ed7 100644--- a/products/signals/dags/inbox_ranking/training/within_team.py+++ b/products/signals/dags/inbox_ranking/training/within_team.py@@ -11,13 +11,23 @@ weighs each team the same, so a few large teams cannot carry it.`within_group_auc` is the same read over any grouping. The pair lane (`nn/pair_grade.py`) groups byuser, because a personal score asks whether it ranks one person's reports better.++`team_interval` is a bootstrap interval for a paired macro delta. It resamples teams, because the+groups of one team share an inbox and are not independent reads."""+from collections.abc import Sequence+from typing import Any+import numpy as npimport pandas as pdfrom posthog.dataclasses import frozen+BOOTSTRAP_DRAWS = 1000+BOOTSTRAP_SEED = 0+INTERVAL_PERCENTILES = (2.5, 97.5)+@frozenclass WithinTeamAuc:@@ -112,3 +122,41 @@ def within_team_auc(outcomes: np.ndarray, scores: np.ndarray, team_ids: np.ndarrif grade.auc is None:return NO_TEAMSreturn WithinTeamAuc(auc=grade.auc, macro=grade.macro, teams=grade.groups, pairs=grade.pairs)+++def team_interval(+ experiment: Sequence[pd.DataFrame | None],+ other: Sequence[pd.DataFrame | None],+ group_teams: pd.Series,+ *,+ draws: int = BOOTSTRAP_DRAWS,+ seed: int = BOOTSTRAP_SEED,+) -> dict[str, Any]:+ """A percentile interval for the macro within-group delta, resampling teams with replacement.++ `experiment` and `other` hold the `group_aucs` of each paired seed, and `group_teams` maps a group+ id to its team. A group that is a team maps to itself. Each group's delta is its AUC gap averaged+ over the paired seeds. A resample's macro delta is the mean over every group of the drawn teams, a+ team drawn twice counting twice, so it is the same statistic as the macro delta itself. Fewer than+ two teams give no interval.+ """+ per_seed = [+ e["u"] / e["pairs"] - o["u"] / o["pairs"]+ for e, o in zip(experiment, other)+ if e is not None and o is not None and not e.empty+ ]+ if not per_seed:+ return {"low": None, "high": None, "teams": 0, "draws": 0}+ per_group = pd.concat(per_seed, axis=1).mean(axis=1)+ per_team = (+ pd.DataFrame({"delta": per_group.to_numpy(), "team": group_teams.loc[per_group.index].to_numpy()})+ .groupby("team")["delta"]+ .agg(["sum", "size"])+ )+ teams = len(per_team)+ if teams < 2:+ return {"low": None, "high": None, "teams": teams, "draws": 0}+ weights = np.random.default_rng(seed).multinomial(teams, np.full(teams, 1 / teams), size=draws)+ macro = (weights @ per_team["sum"].to_numpy()) / (weights @ per_team["size"].to_numpy())+ low, high = np.percentile(macro, INTERVAL_PERCENTILES)+ return {"low": float(low), "high": float(high), "teams": teams, "draws": draws}