Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(signals): pair-grain families and within-user grading in the experiment lane (#125004)

posthog[bot]13792c791745
diff --git a/products/signals/dags/inbox_ranking/AGENTS.md b/products/signals/dags/inbox_ranking/AGENTS.md
index bcb0fab88d70..37636ff02bd8 100644
--- a/products/signals/dags/inbox_ranking/AGENTS.md
+++ b/products/signals/dags/inbox_ranking/AGENTS.md
@@ -45,6 +45,7 @@ Read `README.md` first for what the dataset is and how partitions behave. This f
- An experiment is a `MODEL_FAMILIES` entry with `experiment=True`, a `baseline` serving family and an `expires` day. Only `inbox_ranking_nn_experiments_job` (`nn/experiments.py`) trains it, with its own schedule and resource tags, so a modeling idea never adds to the `nn` job's runtime or memory budget. It reads the day's `inbox_ranking_nn_examples` object and never rebuilds it. An experiment never enters the serving manifest, and `_pin_metadata` refuses its key, so the promote job, the served pointer and the pins cannot name it. To serve its recipe, register it as a serving family under a new `model_name`. Its `metadata.json` carries a `vs_baseline` block: the experiment and the baseline fit with each of `EXPERIMENT_SEEDS` on the same train rows and graded on the same holdout rows. An extra side input for an experiment goes through the registry in `training/side_inputs.py`, which refuses a frame without a timezone-aware `as_of_column` and drops rows known after the partition end. A feature set reads it through `rows_as_of`.
- `inbox_ranking_nn_pair_examples` (`nn/pairs.py`) has one row per (report example, impressed user), with its own `PAIR_EXAMPLES_SCHEMA_VERSION` and its own job without a schedule. It takes its report rows and holdout from the `open` head's rows of the day's `inbox_ranking_nn_examples` object and never rebuilds them. Its impressions and labels come from the dt=B+`PAIR_HORIZON_DAYS` interactions partition, and its user inputs are as of the start of day B, so the target report is never in the user's history. The object holds no vector: a reader rebuilds the user vectors with `pair_user_vectors`. `user_vectors()` lives next to `project_vectors()` and shares `known_before` and `_usable_embeddings` with it; never give it its own as-of rule or embedding gate.
+- A pair-grain family (`grain="pair"`) is always an experiment, trained by `nn/pair_lane.py` in the experiments job. Its baseline is another pair experiment and its `reference` a report-grain serving family. `family_problems` checks both at import. Keep the report-grain `vs_baseline` block, its events and the report families' recipe free of pair fields: an added recipe key changes every serving family's hash, and `check_recipe` then fails each retrain. Fit the reference on report train rows only, so no holdout pair's report trains it.
## Invariants — do not break
diff --git a/products/signals/dags/inbox_ranking/README.md b/products/signals/dags/inbox_ranking/README.md
index 5e6aeb70bcfe..9d8eac9977b1 100644
--- a/products/signals/dags/inbox_ranking/README.md
+++ b/products/signals/dags/inbox_ranking/README.md
@@ -474,6 +474,17 @@ The `open` kind reads `first_viewed_at`, the durable VIEW row, and the `dismiss`
The vector is the plain mean of the user's last `USER_HISTORY_LIMIT` (20) reports of the kind, through the as-of rule and the embedding gate of `project_vectors()`.
A report counts only in the organization of its own team, and a user without usable history gets no row.
+### Pair-grain experiments
+
+A family with `grain="pair"` scores one row per (report, impressed user) from `inbox_ranking_nn_pair_examples`. Pair grain is experiment-only: `ModelFamily` refuses a pair-grain serving family, so the manifest, the promote job and the pins never see one.
+
+- **Registration.** A pair family declares its `heads` from `PAIR_HEAD_NAMES` (`open`, `action`; there is no pair `fixed` label). Its `baseline` is another pair experiment in the registry, and its optional `reference` is a report-grain serving family. It names a baseline, a reference, or both. `family_problems` refuses a baseline of the other grain, an unregistered pair baseline, a baseline cycle and a reference that is not a report-grain serving family.
+- **Lane.** The experiments job routes a pair family to `nn/pair_lane.py`. A day without the pair examples object skips the pair families only, with a log line. A pair takes its report's input and state columns from the report's row of the day's `inbox_ranking_nn_examples` object, and its team's project vectors as of its birth day. The experiment and its baseline fit with each of `EXPERIMENT_SEEDS` on the same train pairs and are graded on the same holdout pairs.
+- **Reference.** For each seed, the reference fits at report grain on the report train rows born on a pair train day, and each holdout pair takes its report's score. The reference fits before the pair matrices are built, so its report matrix and the pair matrices are never in memory together.
+- **Grade.** `vs_baseline` (`nn/pair_grade.py`) has `grain: "pair"` and, per head, within-user AUCs for each arm and seed. A user is one person in one team, and a user with one outcome class is left out. `macro` weighs each user the same, and `pooled` weighs each user by its pairs. The scopes are `all` and `non_dogfood` (without the dogfood project's users). The slices are `all`, the prior-view buckets, `history_10_plus`, `suggested_reviewer` and `not_suggested_reviewer`. `vs_baseline` and `vs_reference` give the mean, min and max delta over the paired seeds, and a team bootstrap interval for the macro delta. One `inbox_ranking_experiment_compared` event per head, scope and slice carries `grain`, `scope`, `slice` and the deltas against both arms. Report-grain experiments keep their block and their events as they were.
+
+No pair family is registered yet. The user-input feature set and the two pair families that read it come in a follow-up.
+
## Configuration
| Setting | Default | Meaning |
diff --git a/products/signals/dags/inbox_ranking/nn/dag.py b/products/signals/dags/inbox_ranking/nn/dag.py
index 1a0a69750aa6..50ce7a70dec2 100644
--- a/products/signals/dags/inbox_ranking/nn/dag.py
+++ b/products/signals/dags/inbox_ranking/nn/dag.py
@@ -62,6 +62,7 @@ from products.signals.dags.inbox_ranking.nn.examples import (
)
from products.signals.dags.inbox_ranking.nn.model import (
HOLDOUT_MODEL_FILE,
+ TrainedNetwork,
candidate_metadata,
champion_grades,
network_recipe,
@@ -299,6 +300,27 @@ def train_candidate(
experiment_scores=seed_scores(examples.table, inputs, NN_HEADS, NETWORK_CONFIG, EXPERIMENT_SEEDS),
baseline_scores=seed_scores(examples.table, baseline_inputs, NN_HEADS, NETWORK_CONFIG, EXPERIMENT_SEEDS),
)
+ return write_candidate(
+ context, client, bucket, partition_key, family, trained, windows=examples.windows, vs_baseline=vs_baseline
+ )
+
+
+def write_candidate(
+ context: dagster.AssetExecutionContext,
+ client,
+ bucket: str,
+ partition_key: str,
+ family: ModelFamily,
+ trained: TrainedNetwork,
+ *,
+ windows: Mapping[str, Mapping[str, object]],
+ vs_baseline: Mapping[str, Any] | None,
+ pair_settings: Mapping[str, Any] | None = None,
+) -> dict[str, dagster.MetadataValue]:
+ """Write a fitted `family` network's graphs and `metadata.json` under `<model_name>/dt=D/`,
+ replacing what a previous run of the partition left, capture its events, and return the asset
+ metadata."""
+ prefix = settings.INBOX_RANKING_DATASET_S3_PREFIX
for head in trained.heads:
context.log.info(f"{family.name} {head.metrics.head}: {head.metrics.as_dict()}")
for name in trained.skipped:
@@ -317,12 +339,13 @@ def train_candidate(
trained,
family=family,
config=NETWORK_CONFIG,
- windows=examples.windows,
+ windows=windows,
trained_at=datetime.datetime.now(datetime.UTC),
run_id=context.run_id,
+ pair_settings=pair_settings,
)
if vs_baseline is not None:
- metadata = {**metadata, "vs_baseline": vs_baseline}
+ metadata = {**metadata, "vs_baseline": dict(vs_baseline)}
metadata_key = model_object_key(prefix, family.name, partition_key, METADATA_FILE)
_put_json(client, bucket, metadata_key, metadata)
written.add(metadata_key)
diff --git a/products/signals/dags/inbox_ranking/nn/experiments.py b/products/signals/dags/inbox_ranking/nn/experiments.py
index 6e48b6c24edb..fcb0e6285ecb 100644
--- a/products/signals/dags/inbox_ranking/nn/experiments.py
+++ b/products/signals/dags/inbox_ranking/nn/experiments.py
@@ -10,6 +10,11 @@ The `vs_baseline` block fits the experiment and its baseline with each of `EXPER
same train rows, and grades both on the same holdout rows, so each seed gives a paired AUC gap.
An experiment never enters the serving manifest and cannot be promoted or pinned. It stops training
after its `expires` day, and a partition without examples is skipped and keeps what it holds.
+
+A pair-grain experiment trains in the pair lane (`nn/pair_lane.py`) on the day's
+`inbox_ranking_nn_pair_examples` object as well, and is graded within user against its pair baseline
+and its report-level reference (`nn/pair_grade.py`). A partition without pair examples skips the pair
+families only.
"""
import datetime
@@ -28,8 +33,9 @@ from products.signals.dags.inbox_ranking.common import (
skip_unconfigured,
)
from products.signals.dags.inbox_ranking.nn.dag import NN_ASSET_KWARGS, NN_EXAMPLES_TABLE, train_candidate
+from products.signals.dags.inbox_ranking.nn.pair_lane import train_pair_candidate
from products.signals.dags.inbox_ranking.training.dag import _object_exists
-from products.signals.dags.inbox_ranking.training.families import ModelFamily, experiment_families
+from products.signals.dags.inbox_ranking.training.families import PAIR_GRAIN, ModelFamily, experiment_families
NN_EXPERIMENTS_ASSET = "inbox_ranking_nn_experiment_candidates"
@@ -57,7 +63,8 @@ def run_experiments(
context.log.warning(f"no nn examples for dt={partition_key}; every experiment skips this partition")
return {**metadata, "skipped": dagster.MetadataValue.bool(True)}
for family in live:
- for name, value in train_candidate(context, client, bucket, partition_key, family).items():
+ train = train_pair_candidate if family.grain == PAIR_GRAIN else train_candidate
+ for name, value in train(context, client, bucket, partition_key, family).items():
metadata[f"{family.name}_{name}"] = value
return metadata
@@ -77,9 +84,14 @@ inbox_ranking_nn_experiments_job = dagster.define_asset_job(
tags={
**owner_tags,
# Per experiment: the five network fits of a candidate, then one fit per seed for each of
- # the experiment and its baseline. The experiments run one after the other.
+ # the experiment and its baseline. The experiments run one after the other. A pair
+ # experiment adds one report-grain fit per seed for its reference, and its fits run over
+ # the pair rows, a few times the report rows.
"dagster/max_runtime": str(4 * 60 * 60),
# As the nn job's candidate, plus the baseline's report matrix, held for the paired fits.
+ # A pair experiment holds its own and its baseline's pair report matrices (pair rows x
+ # 1536 floats each) and fits the reference before building them, so the reference's report
+ # matrix is gone by then. Pair rows a few times the report rows stay well inside the limit.
"dagster-k8s/config": {
"container_config": {
"resources": {
diff --git a/products/signals/dags/inbox_ranking/nn/model.py b/products/signals/dags/inbox_ranking/nn/model.py
index b2a3a19eab0d..973544bab85c 100644
--- a/products/signals/dags/inbox_ranking/nn/model.py
+++ b/products/signals/dags/inbox_ranking/nn/model.py
@@ -24,8 +24,8 @@ from products.signals.dags.inbox_ranking.multihead.inputs import ONNX_INPUTS, ON
from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable
from products.signals.dags.inbox_ranking.nn.examples import NN_HEAD_NAMES, NN_HEADS
from products.signals.dags.inbox_ranking.training.calibration import CalibrationBucket
-from products.signals.dags.inbox_ranking.training.families import ModelFamily
-from products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME
+from products.signals.dags.inbox_ranking.training.families import PAIR_GRAIN, ModelFamily
+from products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME, Head
from products.signals.dags.inbox_ranking.training.recipe import model_recipe, with_recipe
from products.signals.dags.inbox_ranking.training.train import HeadMetrics, HoldoutGrade, _auc, holdout_grade
from products.signals.dags.inbox_ranking.training.within_team import within_team_auc
@@ -60,14 +60,31 @@ def network_vector_kinds(inputs: NetworkInputs) -> tuple[str, ...]:
return VECTOR_KINDS if inputs.context is not None else ()
-def network_recipe(family: ModelFamily, config: NetworkConfig, vector_kinds: Sequence[str]) -> dict[str, Any]:
- return model_recipe(
+def family_heads(family: ModelFamily) -> tuple[Head, ...]:
+ """The heads a `family` network is fit on, in its label and output column order."""
+ if family.grain == PAIR_GRAIN:
+ return tuple(HEADS_BY_NAME[name] for name in family.heads)
+ return NN_HEADS
+
+
+def network_recipe(
+ family: ModelFamily,
+ config: NetworkConfig,
+ vector_kinds: Sequence[str],
+ pair_settings: Mapping[str, Any] | None = None,
+) -> dict[str, Any]:
+ """`pair_settings` is the pair examples' label horizon and schema, for a pair-grain family only,
+ so a report-grain family's recipe and hash stay as they were."""
+ recipe = model_recipe(
model_kind=ONNX_MODEL_KIND,
feature_set=family.feature_set,
- heads=NN_HEADS,
+ heads=family_heads(family),
learner_params=dataclasses.asdict(config),
project_vector_kinds=vector_kinds,
)
+ if family.grain == PAIR_GRAIN:
+ recipe = {**recipe, "grain": PAIR_GRAIN, "pair": dict(pair_settings or {})}
+ return recipe
def candidate_metadata(
@@ -79,11 +96,13 @@ def candidate_metadata(
windows: Mapping[str, Mapping[str, object]],
trained_at: datetime.datetime,
run_id: str,
+ pair_settings: Mapping[str, Any] | None = None,
) -> dict[str, Any]:
"""The `metadata.json` of a `family` candidate, in the layout of the per-head families.
The heads carry the metric fields of `HeadMetrics`, the fields the per-head booster families
- wrote, so the promotion rule, the events and the manifest read every family alike.
+ wrote, so the promotion rule, the events and the manifest read every family alike. A pair-grain
+ candidate also records its grain, and its heads carry the pair labels' horizon.
"""
feature_set = family.feature_set
metadata = {
@@ -109,7 +128,9 @@ def candidate_metadata(
head.metrics.as_dict()
| {
"cohort": HEADS_BY_NAME[head.metrics.head].cohort.__name__,
- "horizon_days": HEADS_BY_NAME[head.metrics.head].horizon_days,
+ "horizon_days": (pair_settings or {}).get(
+ "label_horizon_days", HEADS_BY_NAME[head.metrics.head].horizon_days
+ ),
"example_window_start": windows.get(head.metrics.head, {}).get("example_window_start"),
"example_cap_bound": windows.get(head.metrics.head, {}).get("example_cap_bound"),
"file": ONNX_MODEL_FILE,
@@ -119,7 +140,9 @@ def candidate_metadata(
],
"skipped_heads": list(trained.skipped),
}
- return with_recipe(metadata, network_recipe(family, config, trained.vector_kinds))
+ if family.grain == PAIR_GRAIN:
+ metadata["grain"] = PAIR_GRAIN
+ return with_recipe(metadata, network_recipe(family, config, trained.vector_kinds, pair_settings))
def onnx_probabilities(model: bytes, inputs: NetworkInputs, output_heads: Sequence[str]) -> dict[str, np.ndarray]:
diff --git a/products/signals/dags/inbox_ranking/nn/pair_grade.py b/products/signals/dags/inbox_ranking/nn/pair_grade.py
new file mode 100644
index 000000000000..c17e7cf9c7fc
--- /dev/null
+++ b/products/signals/dags/inbox_ranking/nn/pair_grade.py
@@ -0,0 +1,215 @@
+"""The pair lane's grade: a pair experiment against its pair baseline and its report-level
+reference, within user, on the same holdout pairs.
+
+A personal score is read inside one person's inbox, so a group here is one user in one team: the
+reports that person sees in that project. The AUC is `within_group_auc` over those groups, as a
+macro over users (each user with both outcome classes weighs the same, and a user with one class is
+left out) and pooled over their pairs (each user weighs by its positive x negative pairs).
+
+Every read is per head and per seed, for each arm, in two scopes: every user, and the users outside
+the dogfood project, whose own inbox use is not a customer's. Inside each scope the holdout pairs
+are also sliced by the user's prior views (`HISTORY_BUCKETS`) and by whether the user was a
+suggested reviewer of the report. The deltas are the experiment minus the baseline and the
+experiment minus the reference, over the paired seeds. The macro delta also gets a team bootstrap
+interval: the users of one team share an inbox, so they are not independent reads.
+"""
+
+from collections.abc import Mapping, Sequence
+from typing import Any
+
+import numpy as np
+import pandas as pd
+
+from products.signals.dags.inbox_ranking.dataset.queries import LABELS_TEAM_ID
+from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable
+from products.signals.dags.inbox_ranking.nn.pairs import HISTORY_BUCKETS
+from products.signals.dags.inbox_ranking.training.families import PAIR_GRAIN
+from products.signals.dags.inbox_ranking.training.within_team import group_aucs, summarize_groups
+
+# PostHog's own project, where the inbox's builders are its users.
+DOGFOOD_TEAM_ID = LABELS_TEAM_ID
+# Prior views from which a user counts as having history, read as one slice as well as the buckets.
+WITH_HISTORY_MIN_VIEWS = 10
+BOOTSTRAP_DRAWS = 1000
+BOOTSTRAP_SEED = 0
+INTERVAL_PERCENTILES = (2.5, 97.5)
+
+ALL = "all"
+NON_DOGFOOD = "non_dogfood"
+EXPERIMENT_ARM = "experiment"
+BASELINE_ARM = "baseline"
+REFERENCE_ARM = "reference"
+# The delta block name and the arm the experiment is compared with.
+COMPARISONS = (("vs_baseline", BASELINE_ARM), ("vs_reference", REFERENCE_ARM))
+READS = ("macro", "pooled")
+_GROUP_COLUMNS = ["report_team_id", "user_distinct_id"]
+
+
+def pair_scopes(rows: pd.DataFrame) -> dict[str, np.ndarray]:
+ return {
+ ALL: np.ones(len(rows), dtype=bool),
+ NON_DOGFOOD: rows["report_team_id"].to_numpy() != DOGFOOD_TEAM_ID,
+ }
+
+
+def pair_slices(rows: pd.DataFrame) -> dict[str, np.ndarray]:
+ prior_views = rows["prior_views"].to_numpy()
+ suggested = rows["is_suggested_reviewer"].to_numpy(dtype=bool)
+ slices = {ALL: np.ones(len(rows), dtype=bool)}
+ for name, lowest, highest in HISTORY_BUCKETS:
+ in_bucket = prior_views >= lowest
+ if highest is not None:
+ in_bucket &= prior_views <= highest
+ slices[f"history_{name}"] = in_bucket
+ slices[f"history_{WITH_HISTORY_MIN_VIEWS}_plus"] = prior_views >= WITH_HISTORY_MIN_VIEWS
+ slices["suggested_reviewer"] = suggested
+ slices["not_suggested_reviewer"] = ~suggested
+ return slices
+
+
+def _delta_summary(experiment: Sequence[float | None], other: Sequence[float | None]) -> dict[str, Any]:
+ deltas = [e - o for e, o in zip(experiment, other) if e is not None and o is not None]
+ return {
+ "paired_seeds": len(deltas),
+ "mean_delta_auc": float(np.mean(deltas)) if deltas else None,
+ "min_delta_auc": min(deltas) if deltas else None,
+ "max_delta_auc": max(deltas) if deltas else None,
+ "wins": sum(1 for delta in deltas if delta > 0),
+ }
+
+
+def team_interval(
+ experiment: Sequence[pd.DataFrame | None],
+ other: Sequence[pd.DataFrame | None],
+ group_teams: pd.Series,
+ *,
+ draws: int = BOOTSTRAP_DRAWS,
+ seed: int = BOOTSTRAP_SEED,
+) -> dict[str, Any]:
+ """A percentile interval for the macro within-user delta, resampling teams with replacement.
+
+ Each user's delta is its AUC gap averaged over the paired seeds. A resample's macro delta is the
+ mean over every user of the drawn teams, a team drawn twice counting twice, so it is the same
+ statistic as the macro delta itself. Fewer than two teams give no interval.
+ """
+ per_seed = [
+ e["u"] / e["pairs"] - o["u"] / o["pairs"]
+ for e, o in zip(experiment, other)
+ if e is not None and o is not None and not e.empty
+ ]
+ if not per_seed:
+ return {"low": None, "high": None, "teams": 0, "draws": 0}
+ per_user = pd.concat(per_seed, axis=1).mean(axis=1)
+ per_team = (
+ pd.DataFrame({"delta": per_user.to_numpy(), "team": group_teams.loc[per_user.index].to_numpy()})
+ .groupby("team")["delta"]
+ .agg(["sum", "size"])
+ )
+ teams = len(per_team)
+ if teams < 2:
+ return {"low": None, "high": None, "teams": teams, "draws": 0}
+ weights = np.random.default_rng(seed).multinomial(teams, np.full(teams, 1 / teams), size=draws)
+ macro = (weights @ per_team["sum"].to_numpy()) / (weights @ per_team["size"].to_numpy())
+ low, high = np.percentile(macro, INTERVAL_PERCENTILES)
+ return {"low": float(low), "high": float(high), "teams": teams, "draws": draws}
+
+
+def _slice_grade(
+ outcomes: np.ndarray,
+ groups: np.ndarray,
+ group_teams: pd.Series,
+ arms: Mapping[str, Sequence[np.ndarray] | None],
+ draws: int,
+) -> dict[str, Any]:
+ per_arm = {
+ arm: [group_aucs(outcomes, scores, groups) for scores in arm_scores]
+ for arm, arm_scores in arms.items()
+ if arm_scores is not None
+ }
+ classes = pd.DataFrame({"group": groups, "outcome": outcomes}).groupby("group")["outcome"].agg(["min", "max"])
+ graded_users = classes.index[(classes["min"] == 0) & (classes["max"] == 1)]
+ grade: dict[str, Any] = {
+ "rows": len(outcomes),
+ "positives": int(outcomes.sum()),
+ "users": len(graded_users),
+ "teams": int(group_teams.loc[graded_users].nunique()),
+ }
+ reads: dict[str, dict[str, list[float | None]]] = {}
+ for arm in (EXPERIMENT_ARM, BASELINE_ARM, REFERENCE_ARM):
+ if arm not in per_arm:
+ grade[arm] = None
+ continue
+ summaries = [summarize_groups(per_group) for per_group in per_arm[arm]]
+ reads[arm] = {"macro": [s.macro for s in summaries], "pooled": [s.auc for s in summaries]}
+ grade[arm] = reads[arm]
+ for name, other in COMPARISONS:
+ if other not in per_arm:
+ grade[name] = None
+ continue
+ grade[name] = {
+ **{read: _delta_summary(reads[EXPERIMENT_ARM][read], reads[other][read]) for read in READS},
+ "macro_team_interval": team_interval(per_arm[EXPERIMENT_ARM], per_arm[other], group_teams, draws=draws),
+ }
+ return grade
+
+
+def pair_vs_baseline_block(
+ table: ExampleTable,
+ heads: Sequence[str],
+ *,
+ baseline: str | None,
+ reference: str | None,
+ seeds: Sequence[int],
+ experiment_scores: Sequence[np.ndarray],
+ baseline_scores: Sequence[np.ndarray] | None = None,
+ reference_scores: Sequence[np.ndarray] | None = None,
+ draws: int = BOOTSTRAP_DRAWS,
+) -> dict[str, Any]:
+ """The `vs_baseline` block of a pair experiment, per head, on the same holdout pairs.
+
+ `table` holds one row per pair with labels in `heads` order. Every score array holds a row per
+ pair and a column per head, one array per seed. The reference scores are its report scores
+ broadcast to every pair of the report. An arm that is None (no pair baseline, or no reference)
+ leaves its reads and its deltas None.
+ """
+ train_rows = table.network_train_rows()
+ graded: dict[str, dict[str, Any]] = {}
+ for index, name in enumerate(heads):
+ rows = table.head_rows(index, holdout=True)
+ frame = table.rows.iloc[rows].reset_index(drop=True)
+ outcomes = table.labels[rows, index].astype(int)
+ groups = frame.groupby(_GROUP_COLUMNS, sort=False).ngroup().to_numpy()
+ group_teams = pd.Series(frame["report_team_id"].to_numpy(), index=groups).groupby(level=0).first()
+ arm_scores = {
+ EXPERIMENT_ARM: [scores[rows, index] for scores in experiment_scores],
+ BASELINE_ARM: [scores[rows, index] for scores in baseline_scores] if baseline_scores is not None else None,
+ REFERENCE_ARM: (
+ [scores[rows, index] for scores in reference_scores] if reference_scores is not None else None
+ ),
+ }
+ slices = pair_slices(frame)
+ graded[name] = {
+ "train_rows": int((~np.isnan(table.labels[train_rows, index])).sum()),
+ "holdout_rows": len(rows),
+ "holdout_positives": int(outcomes.sum()),
+ "holdout_users": len(np.unique(groups)),
+ "scopes": {
+ scope: {
+ slice_name: _slice_grade(
+ outcomes[scope_mask & slice_mask],
+ groups[scope_mask & slice_mask],
+ group_teams,
+ {
+ arm: [scores[scope_mask & slice_mask] for scores in per_seed]
+ if per_seed is not None
+ else None
+ for arm, per_seed in arm_scores.items()
+ },
+ draws,
+ )
+ for slice_name, slice_mask in slices.items()
+ }
+ for scope, scope_mask in pair_scopes(frame).items()
+ },
+ }
+ return {"grain": PAIR_GRAIN, "baseline": baseline, "reference": reference, "seeds": list(seeds), "heads": graded}
diff --git a/products/signals/dags/inbox_ranking/nn/pair_lane.py b/products/signals/dags/inbox_ranking/nn/pair_lane.py
new file mode 100644
index 000000000000..e6b304c4c4bf
--- /dev/null
+++ b/products/signals/dags/inbox_ranking/nn/pair_lane.py
@@ -0,0 +1,279 @@
+"""The experiments job's pair lane: fit and grade a pair-grain experiment, without torch at import.
+
+A pair-grain family (`grain="pair"` in `training/families.py`) trains on the day's
+`inbox_ranking_nn_pair_examples` object, one row per (report example, impressed user), and never
+rebuilds it. A pair reads its report's input as the report lane builds it, from the report's row of
+the day's `inbox_ranking_nn_examples` object, and its team's project vectors as of the start of its
+birth day. The experiment and its pair baseline fit with each of `EXPERIMENT_SEEDS` on the same train
+pairs and are graded on the same holdout pairs.
+
+The reference is a report-grain serving family. For each seed it is fit on the report rows of the
+pair train days, at report grain with every `nn` head, and each holdout pair takes its report's
+score. That answers "better than a report-level score" on the same days without loading the served
+model, and the read stays paired on seeds. The reference is fit first and its report inputs are
+dropped before the pair inputs are built, so the two matrices are never held at once.
+
+A partition without pair examples, report examples or the dt=D snapshots keeps what it holds.
+"""
+
+from collections.abc import Sequence
+from typing import Any
+
+import numpy as np
+import pandas as pd
+import dagster
+
+from posthog import settings
+
+from products.signals.backend.ranking.features import Extras, FeatureSet
+from products.signals.dags.inbox_ranking.common import partition_object_key, read_parquet_if_exists
+from products.signals.dags.inbox_ranking.dataset.project_vectors import VECTOR_KINDS
+from products.signals.dags.inbox_ranking.multihead.inputs import NetworkInputs
+from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable, project_context, report_matrix
+from products.signals.dags.inbox_ranking.nn.dag import (
+ EXPERIMENT_SEEDS,
+ NETWORK_CONFIG,
+ NN_EXAMPLES_TABLE,
+ DayInputs,
+ _day_inputs,
+ write_candidate,
+)
+from products.signals.dags.inbox_ranking.nn.examples import NN_HEAD_NAMES, NN_HEADS, examples_from_arrow, network_inputs
+from products.signals.dags.inbox_ranking.nn.model import family_heads, network_recipe
+from products.signals.dags.inbox_ranking.nn.pair_grade import pair_vs_baseline_block
+from products.signals.dags.inbox_ranking.nn.pairs import (
+ NN_PAIR_EXAMPLES_TABLE,
+ PAIR_EXAMPLES_SCHEMA_VERSION,
+ PAIR_HORIZON_DAYS,
+)
+from products.signals.dags.inbox_ranking.training.dag import check_recipe
+from products.signals.dags.inbox_ranking.training.families import MODEL_FAMILIES, ModelFamily
+from products.signals.dags.inbox_ranking.training.side_inputs import load_side_inputs
+
+# The pair labels' horizon and the pair object's schema, in a pair family's recipe.
+PAIR_SETTINGS: dict[str, Any] = {
+ "label_horizon_days": PAIR_HORIZON_DAYS,
+ "examples_schema_version": PAIR_EXAMPLES_SCHEMA_VERSION,
+}
+_REPORT_KEY = ["report_id", "snapshot_date"]
+_PAIR_COLUMNS = [
+ "report_id",
+ "snapshot_date",
+ "report_team_id",
+ "user_distinct_id",
+ "prior_views",
+ "is_suggested_reviewer",
+]
+
+
+def pair_example_table(pairs: pd.DataFrame, report_rows: pd.DataFrame, heads: Sequence[str]) -> ExampleTable:
+ """One row per pair, with labels and holdout flags in `heads` order.
+
+ A pair takes the state columns of its report's examples row (such as `priority`), so a feature
+ set builds its report input as the report lane does. A pair whose report has no examples row is
+ dropped. Every head of a pair shares the report's holdout flag.
+ """
+ state = report_rows.drop(columns=["report_team_id"])
+ merged = pairs.merge(state, on=_REPORT_KEY, how="inner", validate="many_to_one")
+ rows = merged[[*_PAIR_COLUMNS, *(c for c in state.columns if c not in _REPORT_KEY)]].reset_index(drop=True)
+ labels = np.column_stack([merged[f"{head}_label"].to_numpy(dtype=np.float32) for head in heads])
+ holdout = np.repeat(merged["holdout"].to_numpy(dtype=bool)[:, None], len(heads), axis=1)
+ return ExampleTable(
+ rows=rows, labels=labels.reshape(len(rows), len(heads)), holdout=holdout.reshape(len(rows), len(heads))
+ )
+
+
+def report_positions(report_rows: pd.DataFrame, pair_rows: pd.DataFrame) -> np.ndarray:
+ """The position of each pair's report row in `report_rows`, -1 where it has none."""
+ reports = pd.MultiIndex.from_frame(report_rows[_REPORT_KEY])
+ return reports.get_indexer(pd.MultiIndex.from_frame(pair_rows[_REPORT_KEY]))
+
+
+def broadcast_report_scores(
+ report_scores: np.ndarray, positions: np.ndarray, report_heads: Sequence[str], pair_heads: Sequence[str]
+) -> np.ndarray:
+ """(pairs, pair heads) scores: every pair takes its report's score of the same head, NaN where
+ the pair's report has no row."""
+ columns = [list(report_heads).index(head) for head in pair_heads]
+ scores = np.full((len(positions), len(pair_heads)), np.nan, dtype=np.float32)
+ found = positions >= 0
+ scores[found] = report_scores[positions[found]][:, columns]
+ return scores
+
+
+def reference_train_rows(report_table: ExampleTable, pair_table: ExampleTable) -> np.ndarray:
+ """The report rows the reference fits on: the report lane's train rows born on a pair train day.
+ The report train rows are in no head's holdout, so no holdout pair's report is among them."""
+ days = set(pair_table.rows["snapshot_date"].iloc[pair_table.network_train_rows()])
+ train = report_table.network_train_rows()
+ return train[report_table.rows["snapshot_date"].iloc[train].isin(days).to_numpy()]
+
+
+def pair_report_matrix(rows: pd.DataFrame, extras: Extras, feature_set: FeatureSet) -> np.ndarray:
+ """The report input of every pair. Built once per report moment, then repeated for its pairs."""
+ reports = rows.drop_duplicates(_REPORT_KEY).reset_index(drop=True)
+ return report_matrix(reports, extras, feature_set)[report_positions(reports, rows)]
+
+
+def pair_network_inputs(table: ExampleTable, extras: Extras, day: DayInputs, feature_set: FeatureSet) -> NetworkInputs:
+ return NetworkInputs(
+ reports=pair_report_matrix(table.rows, extras, feature_set),
+ context=project_context(table.rows, day.labels, day.embeddings),
+ )
+
+
+def _family(name: str | None) -> ModelFamily | None:
+ return next((family for family in MODEL_FAMILIES if family.name == name), None) if name is not None else None
+
+
+def _skipped(context: dagster.AssetExecutionContext, message: str) -> dict[str, dagster.MetadataValue]:
+ context.log.warning(message)
+ return {"skipped": dagster.MetadataValue.bool(True)}
+
+
+def reference_scores(
+ report_table: ExampleTable,
+ pair_table: ExampleTable,
+ reference: ModelFamily,
+ extras: Extras,
+ day: DayInputs,
+ heads: Sequence[str],
+) -> list[np.ndarray]:
+ """Per seed, the reference fit on the report rows of the pair train days, its report scores
+ broadcast to every pair."""
+ from products.signals.dags.inbox_ranking.nn.train import ( # noqa: PLC0415 — keeps torch off the location's import path
+ seed_scores,
+ )
+
+ inputs = network_inputs(report_table, extras, day.labels, day.embeddings, reference.feature_set)
+ positions = report_positions(report_table.rows, pair_table.rows)
+ scores = seed_scores(
+ report_table,
+ inputs,
+ NN_HEADS,
+ NETWORK_CONFIG,
+ EXPERIMENT_SEEDS,
+ train_rows=reference_train_rows(report_table, pair_table),
+ )
+ return [broadcast_report_scores(seed, positions, NN_HEAD_NAMES, heads) for seed in scores]
+
+
+def train_pair_candidate(
+ context: dagster.AssetExecutionContext, client, bucket: str, partition_key: str, family: ModelFamily
+) -> dict[str, dagster.MetadataValue]:
+ """Fit the day's network for the pair-grain `family`, grade it against its pair baseline and its
+ reference, write its graphs and `metadata.json` under `<model_name>/dt=D/`, and return the asset
+ metadata."""
+ prefix = settings.INBOX_RANKING_DATASET_S3_PREFIX
+ pairs = read_parquet_if_exists(client, bucket, partition_object_key(prefix, NN_PAIR_EXAMPLES_TABLE, partition_key))
+ if pairs is None:
+ return _skipped(
+ context,
+ f"no {NN_PAIR_EXAMPLES_TABLE} for dt={partition_key}; {family.name} keeps this partition as it stands. "
+ f"Run inbox_ranking_nn_pair_examples_job for dt={partition_key}",
+ )
+ report_arrow = read_parquet_if_exists(
+ client, bucket, partition_object_key(prefix, NN_EXAMPLES_TABLE, partition_key)
+ )
+ if report_arrow is None:
+ return _skipped(
+ context, f"no nn examples for dt={partition_key}; {family.name} keeps this partition as it stands"
+ )
+ examples = examples_from_arrow(report_arrow)
+ del report_arrow
+ pair_frame = pairs.to_pandas()
+ del pairs
+ table = pair_example_table(pair_frame, examples.table.rows, family.heads)
+ without_report = len(pair_frame) - len(table.rows)
+ del pair_frame
+ if without_report:
+ context.log.warning(f"{without_report} pairs of dt={partition_key} have no report examples row; dropped")
+
+ baseline, reference = _family(family.baseline), _family(family.reference)
+ for arm in (family, baseline, reference):
+ if arm is None:
+ continue
+ rows = examples.table.rows if arm is reference else table.rows
+ missing = [column for column in arm.feature_set.state_columns if column not in rows]
+ if missing:
+ return _skipped(
+ context,
+ f"the nn examples for dt={partition_key} have no {', '.join(missing)} column for {arm.name}; "
+ f"{family.name} keeps this partition as it stands",
+ )
+ day = _day_inputs(context, client, bucket, prefix, partition_key)
+ if day is None:
+ return {"skipped": dagster.MetadataValue.bool(True)}
+ arms = [arm for arm in (family, baseline, reference) if arm is not None]
+ wanted = sorted({key for arm in arms for key in arm.feature_set.missing_extras(day.extras)})
+ extras = {**day.extras, **load_side_inputs(client, bucket, prefix, partition_key, wanted)}
+ missing_extras = sorted({key for arm in arms for key in arm.feature_set.missing_extras(extras)})
+ if missing_extras:
+ return _skipped(
+ context,
+ f"no {', '.join(missing_extras)} side input for dt={partition_key}; {family.name} keeps this partition "
+ "as it stands",
+ )
+ # Pair inputs always carry the project context, so the graphs take every project-vector kind.
+ check_recipe(
+ client,
+ bucket,
+ prefix,
+ family.name,
+ partition_key,
+ network_recipe(family, NETWORK_CONFIG, VECTOR_KINDS, PAIR_SETTINGS),
+ )
+
+ from products.signals.dags.inbox_ranking.nn.train import ( # noqa: PLC0415 — keeps torch off the location's import path
+ seed_scores,
+ train_network,
+ )
+
+ heads = family_heads(family)
+ has_train_pairs = len(table.network_train_rows()) > 0
+ reference_seed_scores = None
+ if reference is not None and has_train_pairs:
+ reference_seed_scores = reference_scores(examples.table, table, reference, extras, day, family.heads)
+ inputs = pair_network_inputs(table, extras, day, family.feature_set)
+ baseline_reports = pair_report_matrix(table.rows, extras, baseline.feature_set) if baseline is not None else None
+ del day
+
+ trained = train_network(table, inputs, heads, NETWORK_CONFIG)
+ vs_baseline = None
+ if has_train_pairs:
+ vs_baseline = pair_vs_baseline_block(
+ table,
+ family.heads,
+ baseline=family.baseline,
+ reference=family.reference,
+ seeds=EXPERIMENT_SEEDS,
+ experiment_scores=seed_scores(table, inputs, heads, NETWORK_CONFIG, EXPERIMENT_SEEDS),
+ baseline_scores=(
+ seed_scores(
+ table,
+ NetworkInputs(reports=baseline_reports, context=inputs.context),
+ heads,
+ NETWORK_CONFIG,
+ EXPERIMENT_SEEDS,
+ )
+ if baseline_reports is not None
+ else None
+ ),
+ reference_scores=reference_seed_scores,
+ )
+ metadata = write_candidate(
+ context,
+ client,
+ bucket,
+ partition_key,
+ family,
+ trained,
+ windows={name: window for name, window in examples.windows.items() if name in family.heads},
+ vs_baseline=vs_baseline,
+ pair_settings=PAIR_SETTINGS,
+ )
+ return {
+ **metadata,
+ "pair_rows": dagster.MetadataValue.int(len(table.rows)),
+ "pairs_without_report_row": dagster.MetadataValue.int(without_report),
+ }
diff --git a/products/signals/dags/inbox_ranking/nn/pairs.py b/products/signals/dags/inbox_ranking/nn/pairs.py
index b0b903161ca7..f7fe3356a232 100644
--- a/products/signals/dags/inbox_ranking/nn/pairs.py
+++ b/products/signals/dags/inbox_ranking/nn/pairs.py
@@ -54,6 +54,7 @@ from products.signals.dags.inbox_ranking.dataset.project_vectors import (
)
from products.signals.dags.inbox_ranking.nn.dag import NN_ASSET_KWARGS, NN_EXAMPLES_TABLE
from products.signals.dags.inbox_ranking.training.dag import _LOOKBACK_MAPPING
+from products.signals.dags.inbox_ranking.training.families import PAIR_HEAD_NAMES
from products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME
NN_PAIR_EXAMPLES_TABLE = "inbox_ranking_nn_pair_examples"
@@ -62,7 +63,7 @@ PAIR_EXAMPLES_SCHEMA_VERSION = 1
# The report head whose rows, holdout and later label partition the pairs share.
_REPORT_HEAD = HEADS_BY_NAME["open"]
PAIR_HORIZON_DAYS = _REPORT_HEAD.horizon_days
-PAIR_HEADS = ("open", "action")
+PAIR_HEADS = PAIR_HEAD_NAMES
DAYS_SINCE_VIEW_CAP = 90.0
# (name, lowest, highest) prior views per bucket. The within-user grade needs enough rows in the
# buckets with real history to be readable.
diff --git a/products/signals/dags/inbox_ranking/nn/train.py b/products/signals/dags/inbox_ranking/nn/train.py
index 6ed66beefb3f..37d674e1cbd6 100644
--- a/products/signals/dags/inbox_ranking/nn/train.py
+++ b/products/signals/dags/inbox_ranking/nn/train.py
@@ -138,11 +138,19 @@ def train_network(
def seed_scores(
- table: ExampleTable, inputs: NetworkInputs, heads: Sequence[Head], config: NetworkConfig, seeds: Sequence[int]
+ table: ExampleTable,
+ inputs: NetworkInputs,
+ heads: Sequence[Head],
+ config: NetworkConfig,
+ seeds: Sequence[int],
+ train_rows: np.ndarray | None = None,
) -> list[np.ndarray]:
"""One train-only fit per seed, each scoring every row. Two arms fit with the same seeds on the
- same rows give the paired scores `vs_baseline_block` grades."""
- train_rows = table.network_train_rows()
+ same rows give the paired scores `vs_baseline_block` grades. `train_rows` defaults to the
+ table's network train rows; the pair lane's report-level reference narrows them to the pair
+ train days."""
+ if train_rows is None:
+ train_rows = table.network_train_rows()
all_rows = np.arange(len(inputs))
return [
_fit(inputs, heads, dataclasses.replace(config, seed=seed), table.labels, train_rows).predict(inputs, all_rows)
diff --git a/products/signals/dags/inbox_ranking/tests/test_experiments.py b/products/signals/dags/inbox_ranking/tests/test_experiments.py
index e1b7669874ae..e4a75bbfc8f2 100644
--- a/products/signals/dags/inbox_ranking/tests/test_experiments.py
+++ b/products/signals/dags/inbox_ranking/tests/test_experiments.py
@@ -1,4 +1,6 @@
+import json
import datetime
+import dataclasses
import pytest
@@ -17,22 +19,51 @@ from products.signals.backend.ranking.features import (
REPORT_EMBEDDINGS_PRIORITY_SOURCES_FEATURE_SET,
REPORT_SOURCES_EXTRA,
)
+from products.signals.backend.ranking.serving_manifest import ONNX_MODEL_KIND
from products.signals.dags.inbox_ranking.common import partition_object_key
-from products.signals.dags.inbox_ranking.dataset.dag import SIGNAL_EMBEDDINGS_SCHEMA, SIGNAL_EMBEDDINGS_TABLE
+from products.signals.dags.inbox_ranking.dataset.dag import (
+ EMBEDDINGS_TABLE,
+ LABELS_TABLE,
+ SIGNAL_EMBEDDINGS_SCHEMA,
+ SIGNAL_EMBEDDINGS_TABLE,
+)
+from products.signals.dags.inbox_ranking.multihead.inputs import NetworkConfig
+from products.signals.dags.inbox_ranking.multihead.local import report_vectors
from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable, report_matrix
-from products.signals.dags.inbox_ranking.nn import experiments
+from products.signals.dags.inbox_ranking.nn import experiments, pair_lane
from products.signals.dags.inbox_ranking.nn.dag import NN_EXAMPLES_TABLE
-from products.signals.dags.inbox_ranking.nn.examples import NN_HEAD_NAMES
-from products.signals.dags.inbox_ranking.nn.model import vs_baseline_block
+from products.signals.dags.inbox_ranking.nn.examples import NN_HEAD_NAMES, NN_HEADS, build_examples, examples_arrow
+from products.signals.dags.inbox_ranking.nn.model import network_recipe, vs_baseline_block
+from products.signals.dags.inbox_ranking.nn.pair_grade import DOGFOOD_TEAM_ID, pair_vs_baseline_block
+from products.signals.dags.inbox_ranking.nn.pair_lane import (
+ PAIR_SETTINGS,
+ broadcast_report_scores,
+ report_positions,
+ train_pair_candidate,
+)
+from products.signals.dags.inbox_ranking.nn.pairs import NN_PAIR_EXAMPLES_TABLE, PAIR_EXAMPLES_SCHEMA
+from products.signals.dags.inbox_ranking.tests.test_multihead import D0, DAYS, HOLDOUT_DAYS, _SyntheticInbox
+from products.signals.dags.inbox_ranking.tests.test_nn import _S3, _parquet
from products.signals.dags.inbox_ranking.tests.test_training import _ModelStoreS3
from products.signals.dags.inbox_ranking.training import side_inputs
-from products.signals.dags.inbox_ranking.training.families import NN_MODEL_NAME, ModelFamily, family_problems
+from products.signals.dags.inbox_ranking.training.dag import METADATA_FILE, model_object_key
+from products.signals.dags.inbox_ranking.training.families import (
+ NN_MODEL_NAME,
+ NN_WITH_PRIO_FAMILY,
+ NN_WITH_PRIO_MODEL_NAME,
+ PAIR_GRAIN,
+ ModelFamily,
+ family_problems,
+ serving_families,
+)
+from products.signals.dags.inbox_ranking.training.recipe import model_recipe
from products.signals.dags.inbox_ranking.training.side_inputs import (
SideInputError,
SideInputLoader,
load_side_inputs,
rows_as_of,
)
+from products.signals.dags.inbox_ranking.training.telemetry import experiment_comparison_events
PARTITION = "2026-10-20"
PREFIX = settings.INBOX_RANKING_DATASET_S3_PREFIX
@@ -49,6 +80,29 @@ def _experiment(name: str, expires: datetime.date) -> ModelFamily:
)
+def _pair_experiment(
+ name: str,
+ *,
+ baseline: str | None = None,
+ reference: str | None = NN_WITH_PRIO_MODEL_NAME,
+ expires: datetime.date = datetime.date(2026, 12, 31),
+ heads: tuple[str, ...] = ("open", "action"),
+ experiment: bool = True,
+) -> ModelFamily:
+ return ModelFamily(
+ name=name,
+ feature_set=NN_WITH_PRIO_FAMILY.feature_set,
+ learner=ONNX_MODEL_KIND,
+ location=NN_WITH_PRIO_FAMILY.location,
+ experiment=experiment,
+ baseline=baseline,
+ expires=expires,
+ grain=PAIR_GRAIN,
+ heads=heads,
+ reference=reference,
+ )
+
+
@pytest.mark.parametrize("examples_present", [True, False])
def test_an_expired_experiment_and_a_day_without_examples_train_nothing(monkeypatch, examples_present):
trained: list[str] = []
@@ -57,18 +111,24 @@ def test_an_expired_experiment_and_a_day_without_examples_train_nothing(monkeypa
trained.append(family.name)
return {"heads_trained": dagster.MetadataValue.int(3)}
+ def train_pair_candidate(context, client, bucket, partition_key, family):
+ trained.append(f"{family.name} in the pair lane")
+ return {"heads_trained": dagster.MetadataValue.int(2)}
+
monkeypatch.setattr(experiments, "train_candidate", train_candidate)
+ monkeypatch.setattr(experiments, "train_pair_candidate", train_pair_candidate)
objects = {partition_object_key(PREFIX, NN_EXAMPLES_TABLE, PARTITION): b"examples"} if examples_present else {}
# Expiry is inclusive: the expires day still trains, the day after does not.
families = (
_experiment("expired", datetime.date(2026, 10, 19)),
_experiment("last_day", datetime.date(2026, 10, 20)),
+ _pair_experiment("pair", expires=datetime.date(2026, 10, 20)),
)
with dagster.build_asset_context(partition_key=PARTITION) as context:
metadata = experiments.run_experiments(context, _ModelStoreS3(objects), "bucket", PARTITION, families)
- assert trained == (["last_day"] if examples_present else [])
+ assert trained == (["last_day", "pair in the pair lane"] if examples_present else [])
assert metadata["expired_skipped"].value == "expired on 2026-10-19"
assert ("skipped" in metadata) is not examples_present
@@ -89,6 +149,58 @@ def test_an_experiment_must_name_a_registered_serving_baseline_and_an_expiry():
]
+ "build,message",
+ [
+ (lambda: _pair_experiment("served", experiment=False), "must be an experiment"),
+ (lambda: _pair_experiment("unanchored", reference=None), "needs a baseline, a reference, or both"),
+ (lambda: _pair_experiment("fixed_head", heads=("open", "fixed")), "declares heads"),
+ ],
+ ids=["pair_serving_family", "no_baseline_or_reference", "fixed_is_not_a_pair_head"],
+)
+def test_a_pair_family_is_an_experiment_with_pair_heads_and_something_to_grade_against(build, message):
+ with pytest.raises(ValueError, match=message):
+ build()
+
+
+_PRIO_SERVING = ModelFamily(name=NN_WITH_PRIO_MODEL_NAME, feature_set=NN_WITH_PRIO_FAMILY.feature_set)
+
+
+ "families,problems",
+ [
+ ((_PRIO_SERVING, _pair_experiment("pair_base"), _pair_experiment("pair_user", baseline="pair_base")), []),
+ (
+ (_PRIO_SERVING, _pair_experiment("pair_user", baseline=NN_WITH_PRIO_MODEL_NAME)),
+ ["experiment pair_user is pair grain and its baseline nn_with_prio_v2 is report grain"],
+ ),
+ (
+ (_PRIO_SERVING, _pair_experiment("pair_user", baseline="pair_base")),
+ ["experiment pair_user names baseline pair_base, which is not a registered pair experiment"],
+ ),
+ (
+ (
+ _PRIO_SERVING,
+ _pair_experiment("pair_a", baseline="pair_b"),
+ _pair_experiment("pair_b", baseline="pair_a"),
+ ),
+ [
+ "experiment pair_a has a baseline cycle: pair_a -> pair_b -> pair_a",
+ "experiment pair_b has a baseline cycle: pair_b -> pair_a -> pair_b",
+ ],
+ ),
+ (
+ (_PRIO_SERVING, _pair_experiment("pair_base", reference="pair_base")),
+ ["experiment pair_base names reference pair_base, which is not a report-grain serving family"],
+ ),
+ ],
+ ids=["valid_pair_chain", "mixed_grain_baseline", "unregistered_pair_baseline", "baseline_cycle", "pair_reference"],
+)
+def test_a_pair_experiment_names_a_registered_pair_baseline_and_a_report_serving_reference(families, problems):
+ assert family_problems(families) == problems
+ assert all(family.name == NN_WITH_PRIO_MODEL_NAME for family in serving_families(families))
+
+
def _sources(known_at: list[str]) -> pd.DataFrame:
return pd.DataFrame(
{
@@ -176,6 +288,25 @@ def test_vs_baseline_pairs_each_seed_inside_each_team_on_the_same_holdout_rows()
)
first, _, third = NN_HEAD_NAMES
+ # A report-grain experiment's events keep the fields they had before pair grain existed.
+ [event, *_] = experiment_comparison_events({"model_name": "experiment", "vs_baseline": block})
+ assert set(event.properties) == {
+ "model_name",
+ "model_version",
+ "run_id",
+ "recipe_hash",
+ "baseline_model_name",
+ "seeds",
+ "head",
+ "train_rows",
+ "holdout_rows",
+ "holdout_positives",
+ *(
+ f"{read}_{key}"
+ for read in ("within_team", "pooled")
+ for key in ("paired_seeds", "mean_delta_auc", "min_delta_auc", "max_delta_auc", "wins")
+ ),
+ }
paired = {
"experiment_aucs": [1.0, 0.0],
"baseline_aucs": [0.0, 0.0],
@@ -245,3 +376,194 @@ def test_a_late_signal_does_not_leak_into_an_earlier_row():
# The github signal landed on dt=D, after the dt=D-1 row's moment, so only the dt=D row sees it.
# A report with no signals has zero in every source column.
assert matrix[columns].values.tolist() == [[1, 0, 0, 1, 1], [1, 1, 0, 2, 1], [0, 0, 0, 0, 0]]
+
+
+def test_a_report_family_recipe_has_no_pair_keys_so_its_hash_stays_as_it_was():
+ config = NetworkConfig()
+ assert network_recipe(NN_WITH_PRIO_FAMILY, config, ("open",)) == model_recipe(
+ model_kind=ONNX_MODEL_KIND,
+ feature_set=NN_WITH_PRIO_FAMILY.feature_set,
+ heads=NN_HEADS,
+ learner_params=dataclasses.asdict(config),
+ project_vector_kinds=("open",),
+ )
+ pair_recipe = network_recipe(_pair_experiment("pair"), config, ("open",), PAIR_SETTINGS)
+ assert (pair_recipe["grain"], pair_recipe["pair"], list(pair_recipe["heads"])) == (
+ PAIR_GRAIN,
+ PAIR_SETTINGS,
+ ["open", "action"],
+ )
+
+
+def test_a_reference_score_is_broadcast_to_every_pair_of_its_report():
+ day = datetime.date(2026, 10, 1)
+ reports = pd.DataFrame({"report_id": ["a", "b", "c"], "snapshot_date": [day] * 3, "report_team_id": [3, 3, 4]})
+ pairs = pd.DataFrame({"report_id": ["b", "a", "b", "z"], "snapshot_date": [day] * 4})
+ # One column per report-grain head, in NN_HEAD_NAMES order: open, action, fixed.
+ report_scores = np.array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]], dtype=np.float32)
+
+ scores = broadcast_report_scores(report_scores, report_positions(reports, pairs), NN_HEAD_NAMES, ("action", "open"))
+
+ np.testing.assert_allclose(scores[:3], [[0.5, 0.4], [0.2, 0.1], [0.5, 0.4]])
+ # A pair whose report has no row gets no score rather than another report's.
+ assert np.isnan(scores[3]).all()
+
+
+def _pair_frame(users: list[tuple[int, str, int, bool, list[int]]]) -> pd.DataFrame:
+ """One pair per (user, label), each user's reports invented in order."""
+ rows = []
+ for team_id, user, prior_views, suggested, labels in users:
+ for position, label in enumerate(labels):
+ rows.append(
+ {
+ "report_id": f"{user}-{position}",
+ "report_team_id": team_id,
+ "user_distinct_id": user,
+ "prior_views": prior_views,
+ "is_suggested_reviewer": suggested,
+ "label": label,
+ }
+ )
+ return pd.DataFrame(rows)
+
+
+def test_the_pair_grade_is_within_user_with_dogfood_history_and_reviewer_slices_and_a_team_interval():
+ frame = _pair_frame(
+ [
+ (3, "u1", 12, True, [1, 0]),
+ (4, "u3", 20, False, [1, 0, 0]),
+ (DOGFOOD_TEAM_ID, "u4", 15, False, [1, 0]),
+ # One outcome class only: the user has no within-user pair and is left out.
+ (3, "u2", 0, False, [0, 0]),
+ ]
+ )
+ labels = frame[["label"]].to_numpy(dtype=np.float32)
+ table = ExampleTable(rows=frame, labels=labels, holdout=np.ones_like(labels, dtype=bool))
+ perfect = labels.copy()
+ # Right inside u3 (two pairs), wrong inside u1 and u4 (one pair each): macro 1/3, pooled 2/4.
+ baseline = np.array([[0], [1], [1], [0], [0], [0], [1], [0], [0]], dtype=np.float32)
+ # A report-level score that ties inside every user.
+ reference = np.full_like(labels, 0.5)
+
+ block = pair_vs_baseline_block(
+ table,
+ ("open",),
+ baseline="pair_base",
+ reference=NN_WITH_PRIO_MODEL_NAME,
+ seeds=(0, 1),
+ experiment_scores=[perfect, perfect],
+ baseline_scores=[baseline, baseline],
+ reference_scores=[reference, reference],
+ )
+
+ scopes = block["heads"]["open"]["scopes"]
+ every = scopes["all"]["all"]
+ assert (every["users"], every["teams"]) == (3, 3)
+ assert every["baseline"]["macro"] == pytest.approx([1 / 3, 1 / 3])
+ assert every["baseline"]["pooled"] == pytest.approx([0.5, 0.5])
+ assert every["vs_baseline"]["macro"]["mean_delta_auc"] == pytest.approx(2 / 3)
+ assert every["vs_baseline"]["pooled"]["mean_delta_auc"] == pytest.approx(0.5)
+ assert every["vs_reference"]["macro"]["mean_delta_auc"] == pytest.approx(0.5)
+ interval = every["vs_baseline"]["macro_team_interval"]
+ assert interval["teams"] == 3
+ assert 0 <= interval["low"] <= 2 / 3 <= interval["high"] <= 1
+ # Leaving the dogfood project's user out leaves u1 (wrong) and u3 (right) in the 10+ bucket.
+ with_history = scopes["non_dogfood"]["history_10_plus"]
+ assert (with_history["users"], with_history["baseline"]["macro"]) == (2, pytest.approx([0.5, 0.5]))
+ assert scopes["all"]["history_0"]["users"] == 0
+ assert scopes["all"]["history_0"]["vs_baseline"]["macro"]["paired_seeds"] == 0
+ # One team cannot be resampled into an interval.
+ assert scopes["all"]["suggested_reviewer"]["vs_baseline"]["macro_team_interval"]["low"] is None
+
+ events = experiment_comparison_events({"model_name": "pair_user", "vs_baseline": block})
+ assert len(events) == len(scopes) * len(scopes["all"])
+ [event] = [
+ e for e in events if (e.properties["scope"], e.properties["slice"]) == ("non_dogfood", "history_10_plus")
+ ]
+ assert event.properties["grain"] == PAIR_GRAIN
+ assert event.properties["reference_model_name"] == NN_WITH_PRIO_MODEL_NAME
+ assert event.properties["vs_baseline_within_user_macro_mean_delta_auc"] == pytest.approx(0.5)
+ assert event.properties["vs_reference_within_user_macro_mean_delta_auc"] == pytest.approx(0.5)
+
+
+def _pair_examples(report_rows: pd.DataFrame, report_labels: np.ndarray, report_holdout: np.ndarray) -> pa.Table:
+ """Two users per report with an `open` example: the first opens what the report's `open` label
+ says and has history, the second never opens and has none."""
+ open_index, action_index = NN_HEAD_NAMES.index("open"), NN_HEAD_NAMES.index("action")
+ rows = []
+ for position, report in report_rows.iterrows():
+ if np.isnan(report_labels[position, open_index]):
+ continue
+ for user, prior_views in (("reader", 12), ("skimmer", 0)):
+ opened = user == "reader" and bool(report_labels[position, open_index])
+ rows.append(
+ {
+ "snapshot_date": report["snapshot_date"],
+ "report_id": report["report_id"],
+ "report_team_id": int(report["report_team_id"]),
+ "user_distinct_id": f"{user}-{report['report_team_id']}",
+ "holdout": bool(report_holdout[position, open_index]),
+ "open_label": float(opened),
+ "action_label": float(opened and report_labels[position, action_index] == 1),
+ "user_open_reports": prior_views,
+ "user_dismiss_reports": 0,
+ "prior_views": prior_views,
+ "days_since_last_view": 1.0 if prior_views else 90.0,
+ "has_history": bool(prior_views),
+ "is_suggested_reviewer": user == "reader",
+ "suggested_reviewer_source": "reviewers",
+ }
+ )
+ return pa.Table.from_pylist(rows, schema=PAIR_EXAMPLES_SCHEMA)
+
+
[email protected]("pairs_present", [True, False])
+def test_a_pair_experiment_trains_on_pair_rows_and_is_graded_within_user(monkeypatch, pairs_present):
+ pytest.importorskip("torch")
+ monkeypatch.setattr("products.signals.dags.inbox_ranking.nn.dag.capture_training_events", lambda *a, **k: None)
+ monkeypatch.setattr(
+ pair_lane, "NETWORK_CONFIG", NetworkConfig(projection_dim=8, trunk_dim=8, dropout=0.1, epochs=40, batch_size=32)
+ )
+ monkeypatch.setattr(pair_lane, "EXPERIMENT_SEEDS", (0, 1))
+ base = _pair_experiment("nn_pair_stub_base")
+ family = _pair_experiment("nn_pair_stub", baseline=base.name)
+ registry = (NN_WITH_PRIO_FAMILY, base, family)
+ assert family_problems(registry) == []
+ monkeypatch.setattr(pair_lane, "MODEL_FAMILIES", registry)
+ inbox = _SyntheticInbox()
+ embeddings, labels = inbox.embeddings(), inbox.labels()
+ examples = build_examples(inbox.snapshots(), report_vectors(embeddings), holdout_days=HOLDOUT_DAYS)
+ table = examples.table
+ day = (D0 + datetime.timedelta(days=DAYS - 1)).isoformat()
+ objects = {
+ partition_object_key(PREFIX, NN_EXAMPLES_TABLE, day): _parquet(examples_arrow(examples)),
+ partition_object_key(PREFIX, EMBEDDINGS_TABLE, day): _parquet(embeddings),
+ partition_object_key(PREFIX, LABELS_TABLE, day): _parquet(labels),
+ }
+ if pairs_present:
+ objects[partition_object_key(PREFIX, NN_PAIR_EXAMPLES_TABLE, day)] = _parquet(
+ _pair_examples(table.rows, table.labels, table.holdout)
+ )
+ client = _S3(objects)
+
+ with dagster.build_asset_context(partition_key=day) as context:
+ result = train_pair_candidate(context, client, "bucket", day, family)
+
+ metadata_key = model_object_key(PREFIX, family.name, day, METADATA_FILE)
+ if not pairs_present:
+ assert result["skipped"].value is True
+ assert metadata_key not in client.objects
+ return
+ metadata = json.loads(client.objects[metadata_key])
+ assert (metadata["grain"], metadata["recipe"]["grain"]) == (PAIR_GRAIN, PAIR_GRAIN)
+ assert [head["head"] for head in metadata["heads"]] == ["open", "action"]
+ block = metadata["vs_baseline"]
+ assert (block["grain"], block["baseline"], block["reference"]) == (PAIR_GRAIN, base.name, NN_WITH_PRIO_MODEL_NAME)
+ for grade in block["heads"].values():
+ every = grade["scopes"]["all"]["all"]
+ assert every["vs_baseline"]["macro"]["paired_seeds"] == every["vs_reference"]["macro"]["paired_seeds"] == 2
+ # The skimmers never open, so only the readers have both classes; the dogfood readers drop out.
+ assert grade["scopes"]["non_dogfood"]["all"]["users"] < every["users"]
+ opened = block["heads"]["open"]["scopes"]["all"]["all"]
+ assert min(opened["experiment"]["macro"]) > 0.75
+ assert min(opened["reference"]["macro"]) > 0.75
diff --git a/products/signals/dags/inbox_ranking/tests/test_promote.py b/products/signals/dags/inbox_ranking/tests/test_promote.py
index 061bef9c4d67..55ce6a435049 100644
--- a/products/signals/dags/inbox_ranking/tests/test_promote.py
+++ b/products/signals/dags/inbox_ranking/tests/test_promote.py
@@ -14,7 +14,7 @@ from products.signals.dags.inbox_ranking.common import partition_object_key
from products.signals.dags.inbox_ranking.dataset.dag import PROJECT_VECTORS_TABLE
from products.signals.dags.inbox_ranking.tests.test_training import _AppObjectStore, _ModelStoreS3, _patch_capture
from products.signals.dags.inbox_ranking.training.dag import METADATA_FILE, model_object_key, served_object_key
-from products.signals.dags.inbox_ranking.training.families import ModelFamily
+from products.signals.dags.inbox_ranking.training.families import PAIR_GRAIN, ModelFamily
from products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME
from products.signals.dags.inbox_ranking.training.promote import PromoteConfig, inbox_ranking_promote
from products.signals.dags.inbox_ranking.training.telemetry import MODEL_PROMOTED_EVENT
@@ -25,6 +25,7 @@ OLDER_VERSION = "2026-10-01"
# A retired XGBoost family. A rollback can still serve one of its saved versions.
XGBOOST_MODEL_NAME = "report_embeddings_v2"
EXPERIMENT_MODEL_NAME = "report_embeddings_experiment"
+PAIR_EXPERIMENT_MODEL_NAME = "report_embeddings_pair_experiment"
class _WritableS3(_ModelStoreS3):
@@ -77,6 +78,7 @@ def _dataset() -> _WritableS3:
_add_model(objects, "2026-09-02", all_heads[:1])
# Every head and every file: only being an experiment keeps it from serving.
_add_model(objects, SERVED_VERSION, all_heads, model_name=EXPERIMENT_MODEL_NAME)
+ _add_model(objects, SERVED_VERSION, all_heads, model_name=PAIR_EXPERIMENT_MODEL_NAME)
return _WritableS3(objects)
@@ -95,6 +97,15 @@ def _promote(monkeypatch: pytest.MonkeyPatch, client: _WritableS3, store: _AppOb
baseline=XGBOOST_MODEL_NAME,
expires=datetime.date(2026, 12, 31),
),
+ ModelFamily(
+ name=PAIR_EXPERIMENT_MODEL_NAME,
+ feature_set=REPORT_EMBEDDINGS_FEATURE_SET,
+ experiment=True,
+ expires=datetime.date(2026, 12, 31),
+ grain=PAIR_GRAIN,
+ heads=("open", "action"),
+ reference=XGBOOST_MODEL_NAME,
+ ),
),
)
for module in ("dag", "promote"):
@@ -130,8 +141,9 @@ def test_a_rollback_to_an_older_version_rewrites_the_pointer_and_publishes_it(mo
(f"{XGBOOST_MODEL_NAME}@2026-09-01", "missing model files"),
(f"{XGBOOST_MODEL_NAME}@2026-09-02", "has no model for the heads"),
(f"{EXPERIMENT_MODEL_NAME}@{SERVED_VERSION}", "is an experiment family, and an experiment never serves"),
+ (f"{PAIR_EXPERIMENT_MODEL_NAME}@{SERVED_VERSION}", "is an experiment family, and an experiment never serves"),
],
- ids=["malformed_key", "missing_file", "missing_head", "experiment"],
+ ids=["malformed_key", "missing_file", "missing_head", "experiment", "pair_experiment"],
)
def test_a_model_the_sweep_could_not_serve_is_refused_and_nothing_is_written(monkeypatch, model_key, message):
client, store = _dataset(), _AppObjectStore()
diff --git a/products/signals/dags/inbox_ranking/training/families.py b/products/signals/dags/inbox_ranking/training/families.py
index 7df45b995129..c2db886a916d 100644
--- a/products/signals/dags/inbox_ranking/training/families.py
+++ b/products/signals/dags/inbox_ranking/training/families.py
@@ -6,10 +6,17 @@ serving manifest as a challenger, and gets online grades. An experiment trains i
manifest, and can never be promoted or pinned. An experiment stops training after its `expires`
day, so a modeling idea cannot grow into a permanent cost by accident. To serve an experiment's
recipe, register it as a serving family under a new `model_name`.
+
+A family has a grain. A report-grain family scores one row per report, as the inbox reads it. A
+pair-grain family scores one row per (report, impressed user) from `inbox_ranking_nn_pair_examples`,
+and is experiment-only until a serving design reads a personal score. Its baseline is another pair
+experiment, and its optional `reference` is a report-grain serving family fit on the same days and
+graded on the same holdout pairs, so it answers "better than the report-level score".
"""
import datetime
from collections.abc import Sequence
+from typing import Final, Literal
from posthog.dataclasses import frozen
@@ -32,6 +39,11 @@ NN_WITH_PRIO_SOURCES_MODEL_NAME = "nn_with_prio_sources_v1"
# the image that carries torch.
MAIN_LOCATION = "signals"
ML_LOCATION = "signals_ml"
+REPORT_GRAIN: Final = "report"
+PAIR_GRAIN: Final = "pair"
+Grain = Literal["report", "pair"]
+# The heads a pair-grain family can declare: the pair examples carry these labels only.
+PAIR_HEAD_NAMES = ("open", "action")
@frozen
@@ -40,7 +52,11 @@ class ModelFamily:
model store loads it with, and the Dagster code location whose assets train it.
An experiment names the serving family it is graded against in `baseline`, and the last
- partition day it trains in `expires`. A serving family has neither."""
+ partition day it trains in `expires`. A serving family has neither.
+
+ A pair-grain family is always an experiment and declares its `heads` from `PAIR_HEAD_NAMES`. Its
+ `baseline` is another pair experiment, and it names a `baseline`, a `reference`, or both: the
+ first pair family of a comparison has no pair baseline and is graded against the reference only."""
name: str
feature_set: FeatureSet
@@ -49,13 +65,37 @@ class ModelFamily:
experiment: bool = False
baseline: str | None = None
expires: datetime.date | None = None
+ grain: Grain = REPORT_GRAIN
+ # A pair-grain family's heads. A report-grain family trains every `nn` head and declares none.
+ heads: tuple[str, ...] = ()
+ reference: str | None = None
def __post_init__(self) -> None:
+ if self.grain == PAIR_GRAIN:
+ self._check_pair()
+ return
+ if self.heads or self.reference is not None:
+ raise ValueError(f"report-grain family {self.name} cannot declare heads or a reference")
if self.experiment and (self.baseline is None or self.expires is None):
raise ValueError(f"experiment family {self.name} needs a baseline and an expires day")
if not self.experiment and (self.baseline is not None or self.expires is not None):
raise ValueError(f"serving family {self.name} cannot have a baseline or an expires day")
+ def _check_pair(self) -> None:
+ if not self.experiment:
+ raise ValueError(
+ f"pair-grain family {self.name} must be an experiment: no serving design reads a pair score yet"
+ )
+ if self.expires is None:
+ raise ValueError(f"experiment family {self.name} needs an expires day")
+ if self.baseline is None and self.reference is None:
+ raise ValueError(f"pair experiment {self.name} needs a baseline, a reference, or both")
+ if not self.heads or len(set(self.heads)) != len(self.heads) or not set(self.heads) <= set(PAIR_HEAD_NAMES):
+ raise ValueError(
+ f"pair experiment {self.name} declares heads {list(self.heads)}; pick distinct heads from "
+ f"{list(PAIR_HEAD_NAMES)}"
+ )
+
def expired(self, day: datetime.date) -> bool:
return self.expires is not None and day > self.expires
@@ -114,17 +154,58 @@ def families_trained_in(location: str) -> tuple[ModelFamily, ...]:
return tuple(family for family in serving_families() if family.location == location)
+def _baseline_problem(family: ModelFamily, by_name: dict[str, ModelFamily], serving: set[str]) -> str | None:
+ if family.baseline is None:
+ return None
+ baseline = by_name.get(family.baseline)
+ if baseline is not None and baseline.grain != family.grain:
+ return (
+ f"experiment {family.name} is {family.grain} grain and its baseline {baseline.name} is "
+ f"{baseline.grain} grain"
+ )
+ if family.grain == REPORT_GRAIN and family.baseline not in serving:
+ return f"experiment {family.name} names baseline {family.baseline}, which is not a serving family"
+ if family.grain == PAIR_GRAIN and (baseline is None or not baseline.experiment):
+ return f"experiment {family.name} names baseline {family.baseline}, which is not a registered pair experiment"
+ return None
+
+
+def _baseline_cycle(family: ModelFamily, by_name: dict[str, ModelFamily]) -> list[str] | None:
+ """The baseline chain from `family` back to itself, or None when the chain ends."""
+ chain = [family.name]
+ current = family
+ while current.baseline is not None and current.baseline in by_name:
+ if current.baseline in chain:
+ return [*chain, current.baseline] if current.baseline == family.name else None
+ chain.append(current.baseline)
+ current = by_name[current.baseline]
+ return None
+
+
def family_problems(families: Sequence[ModelFamily]) -> list[str]:
- """Why `families` is not a valid registry: a duplicate name, or an experiment whose baseline is
- not a serving family in the registry."""
+ """Why `families` is not a valid registry: a duplicate name, an experiment whose baseline is not
+ a serving family of its grain (a pair experiment's is another registered pair experiment), a
+ baseline of the other grain, a baseline cycle, or a reference that is not a report-grain serving
+ family."""
names = [family.name for family in families]
problems = [f"family {name} is registered twice" for name in sorted({n for n in names if names.count(n) > 1})]
+ by_name = {family.name: family for family in families}
serving = {family.name for family in serving_families(families)}
- problems += [
- f"experiment {family.name} names baseline {family.baseline}, which is not a serving family"
- for family in experiment_families(families)
- if family.baseline not in serving
- ]
+ for family in experiment_families(families):
+ problem = _baseline_problem(family, by_name, serving)
+ if problem is not None:
+ problems.append(problem)
+ cycle = _baseline_cycle(family, by_name)
+ if cycle is not None:
+ problems.append(f"experiment {family.name} has a baseline cycle: {' -> '.join(cycle)}")
+ reference = by_name.get(family.reference) if family.reference is not None else None
+ if family.reference is not None and (
+ reference is None or reference.experiment or reference.grain != REPORT_GRAIN
+ ):
+ problems.append(
+ f"experiment {family.name} names reference {family.reference}, which is not a report-grain "
+ "serving family"
+ )
return problems
diff --git a/products/signals/dags/inbox_ranking/training/telemetry.py b/products/signals/dags/inbox_ranking/training/telemetry.py
index c13d2e77aff1..419fbdf0a5f4 100644
--- a/products/signals/dags/inbox_ranking/training/telemetry.py
+++ b/products/signals/dags/inbox_ranking/training/telemetry.py
@@ -19,6 +19,7 @@ from posthog.ph_client import get_client
from products.signals.backend.ranking.serving_manifest import ServingManifest
from products.signals.dags.inbox_ranking.common import RunContext, snapshot_bounds
+from products.signals.dags.inbox_ranking.training.families import PAIR_GRAIN
from products.signals.dags.inbox_ranking.training.promotion import PromotionDecision
from products.signals.dags.inbox_ranking.training.unseen import CANDIDATE_ROLE, HeadGrade
@@ -141,11 +142,53 @@ def promotion_event(
)
+_DELTA_KEYS = ("paired_seeds", "mean_delta_auc", "min_delta_auc", "max_delta_auc", "wins")
+
+
+def _pair_comparison_events(block: Mapping[str, Any], context: Mapping[str, Any]) -> list[TrainingEvent]:
+ """One event per head, scope and slice of a pair experiment's `vs_baseline` block, so a chart
+ filters to one slice (such as `non_dogfood` users with 10 or more prior views) and reads its
+ within-user deltas against the baseline and the reference."""
+ events: list[TrainingEvent] = []
+ for head, grade in (block.get("heads") or {}).items():
+ for scope, slices in (grade.get("scopes") or {}).items():
+ for slice_name, cell in slices.items():
+ properties: dict[str, Any] = {
+ **context,
+ "grain": block.get("grain"),
+ "baseline_model_name": block.get("baseline"),
+ "reference_model_name": block.get("reference"),
+ "seeds": len(block.get("seeds", [])),
+ "head": head,
+ **{key: grade.get(key) for key in ("train_rows", "holdout_rows", "holdout_positives")},
+ "scope": scope,
+ "slice": slice_name,
+ **{f"slice_{key}": cell.get(key) for key in ("rows", "positives", "users", "teams")},
+ }
+ for comparison in ("vs_baseline", "vs_reference"):
+ deltas = cell.get(comparison) or {}
+ for read in ("macro", "pooled"):
+ properties |= {
+ f"{comparison}_within_user_{read}_{key}": (deltas.get(read) or {}).get(key)
+ for key in _DELTA_KEYS
+ }
+ interval = deltas.get("macro_team_interval") or {}
+ properties |= {
+ f"{comparison}_within_user_macro_interval_{key}": interval.get(key)
+ for key in ("low", "high", "teams")
+ }
+ events.append(TrainingEvent(event=EXPERIMENT_COMPARED_EVENT, properties=properties))
+ return events
+
+
def experiment_comparison_events(metadata: Mapping[str, Any]) -> list[TrainingEvent]:
"""One event per head of an experiment candidate's `vs_baseline` block. The per-seed AUC lists
- stay in `metadata.json`: an insight cannot chart a JSON array, so the event carries the gaps."""
+ stay in `metadata.json`: an insight cannot chart a JSON array, so the event carries the gaps.
+ A pair experiment's block gives one event per head, scope and slice instead."""
block = metadata.get("vs_baseline") or {}
context = {key: metadata.get(key) for key in ("model_name", "model_version", "run_id", "recipe_hash")}
+ if block.get("grain") == PAIR_GRAIN:
+ return _pair_comparison_events(block, context)
return [
TrainingEvent(
event=EXPERIMENT_COMPARED_EVENT,
diff --git a/products/signals/dags/inbox_ranking/training/within_team.py b/products/signals/dags/inbox_ranking/training/within_team.py
index 40667e886617..dfea533b13cd 100644
--- a/products/signals/dags/inbox_ranking/training/within_team.py
+++ b/products/signals/dags/inbox_ranking/training/within_team.py
@@ -8,6 +8,9 @@ AUC well above 0.5 and a within-team AUC of exactly 0.5.
Only the pairs inside one team count here. `auc` weighs each team by its positive x negative pairs,
so it answers "how often does a positive report outrank a negative one in its own inbox". `macro`
weighs each team the same, so a few large teams cannot carry it.
+
+`within_group_auc` is the same read over any grouping. The pair lane (`nn/pair_grade.py`) groups by
+user, because a personal score asks whether it ranks one person's reports better.
"""
import numpy as np
@@ -37,39 +40,75 @@ class WithinTeamAuc:
NO_TEAMS = WithinTeamAuc()
-def within_team_auc(outcomes: np.ndarray, scores: np.ndarray, team_ids: np.ndarray) -> WithinTeamAuc:
- """The Mann-Whitney U of each team, with ties paid 0.5, summed over the teams.
+@frozen
+class WithinGroupAuc:
+ # None when no group has both outcome classes.
+ auc: float | None = None
+ macro: float | None = None
+ # The groups with both outcome classes, and their summed positive x negative pairs.
+ groups: int = 0
+ pairs: int = 0
- A row without a team id is left out. A team with one outcome class has no pair and contributes
- nothing. A non-finite score makes the order undefined, so the result is NO_TEAMS, which is what
- the pooled AUC reports for the same rows.
+
+NO_GROUPS = WithinGroupAuc()
+
+
+def group_aucs(outcomes: np.ndarray, scores: np.ndarray, group_ids: np.ndarray) -> pd.DataFrame | None:
+ """The Mann-Whitney U and the positive x negative pairs of each group with both outcome classes,
+ indexed by group id, with ties paid 0.5.
+
+ A row without a group id is left out. A non-finite score makes the order undefined, so the
+ result is None, which is what the pooled AUC reports for the same rows.
"""
rows = pd.DataFrame(
{
- "team": pd.Series(team_ids, dtype=object),
+ "group": pd.Series(group_ids, dtype=object),
"outcome": np.asarray(outcomes, dtype=bool),
"score": np.asarray(scores, dtype=float),
}
)
- rows = rows.loc[rows["team"].notna()].copy()
- if rows.empty or not np.isfinite(rows["score"].to_numpy()).all():
- return NO_TEAMS
- rank = rows.groupby("team")["score"].rank(method="average")
+ rows = rows.loc[rows["group"].notna()].copy()
+ if not np.isfinite(rows["score"].to_numpy()).all():
+ return None
+ rank = rows.groupby("group")["score"].rank(method="average")
rows["positive_rank"] = rank.where(rows["outcome"], 0.0)
- per_team = rows.groupby("team").agg(
+ per_group = rows.groupby("group").agg(
rows=("outcome", "size"),
positives=("outcome", "sum"),
positive_rank_sum=("positive_rank", "sum"),
)
- per_team["negatives"] = per_team["rows"] - per_team["positives"]
- per_team = per_team[(per_team["positives"] > 0) & (per_team["negatives"] > 0)]
- if per_team.empty:
- return NO_TEAMS
- pairs = per_team["positives"] * per_team["negatives"]
- u = per_team["positive_rank_sum"] - per_team["positives"] * (per_team["positives"] + 1) / 2
- return WithinTeamAuc(
- auc=float(u.sum() / pairs.sum()),
- macro=float((u / pairs).mean()),
- teams=len(per_team),
- pairs=int(pairs.sum()),
+ per_group["negatives"] = per_group["rows"] - per_group["positives"]
+ per_group = per_group[(per_group["positives"] > 0) & (per_group["negatives"] > 0)]
+ return pd.DataFrame(
+ {
+ "u": per_group["positive_rank_sum"] - per_group["positives"] * (per_group["positives"] + 1) / 2,
+ "pairs": per_group["positives"] * per_group["negatives"],
+ }
+ )
+
+
+def summarize_groups(per_group: pd.DataFrame | None) -> WithinGroupAuc:
+ """`auc` weighs each group by its pairs, and `macro` weighs each group the same."""
+ if per_group is None or per_group.empty:
+ return NO_GROUPS
+ return WithinGroupAuc(
+ auc=float(per_group["u"].sum() / per_group["pairs"].sum()),
+ macro=float((per_group["u"] / per_group["pairs"]).mean()),
+ groups=len(per_group),
+ pairs=int(per_group["pairs"].sum()),
)
+
+
+def within_group_auc(outcomes: np.ndarray, scores: np.ndarray, group_ids: np.ndarray) -> WithinGroupAuc:
+ """The Mann-Whitney U of each group, summed over the groups. A group with one outcome class has
+ no pair and contributes nothing."""
+ return summarize_groups(group_aucs(outcomes, scores, group_ids))
+
+
+def within_team_auc(outcomes: np.ndarray, scores: np.ndarray, team_ids: np.ndarray) -> WithinTeamAuc:
+ """`within_group_auc` over teams. A row without a team id is left out, and a non-finite score
+ gives NO_TEAMS."""
+ grade = within_group_auc(outcomes, scores, team_ids)
+ if grade.auc is None:
+ return NO_TEAMS
+ return WithinTeamAuc(auc=grade.auc, macro=grade.macro, teams=grade.groups, pairs=grade.pairs)