Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(signals): add a fixed head and trained cold path to pair ranking (#126172)

posthog[bot]f2621db7ad2d
diff --git a/products/signals/dags/inbox_ranking/README.md b/products/signals/dags/inbox_ranking/README.md
index 205de476ff57..fc6c9a1baa77 100644
--- a/products/signals/dags/inbox_ranking/README.md
+++ b/products/signals/dags/inbox_ranking/README.md
@@ -508,10 +508,13 @@ A report counts only in the organization of its own team, and a user without usa
A family with `grain="pair"` scores one row per (report, impressed user) from `inbox_ranking_nn_pair_examples`. Pair grain is experiment-only: `ModelFamily` refuses a pair-grain serving family, so the manifest, the promote job and the pins never see one.
-- **Registration.** A pair family declares its `heads` from `PAIR_HEAD_NAMES` (`open`, `action`; there is no pair `fixed` label). Its `baseline` is another pair experiment in the registry, and its optional `reference` is a report-grain serving family. It names a baseline, a reference, or both. `family_problems` refuses a baseline of the other grain, an unregistered pair baseline, a baseline cycle and a reference that is not a report-grain serving family.
+- **Registration.** A pair family declares its `heads` from `PAIR_HEAD_NAMES`: the per-viewer `open` and `action`, and the broadcast `fixed`. Its `baseline` is another pair experiment in the registry, and its optional `reference` is a report-grain serving family. It names a baseline, a reference, or both. `family_problems` refuses a baseline of the other grain, an unregistered pair baseline, a baseline cycle, a baseline with other heads or other `pair_training`, and a reference that is not a report-grain serving family.
+- **Broadcast `fixed`.** No person is credited with a fix, so a pair's `fixed` label is its report's own `fixed` label, with the report's own `fixed` holdout flag, from the day's `inbox_ranking_nn_examples` object. Each of the report's pairs weighs 1/pairs in the loss, so a report shown to many people counts once. The pair examples object is unchanged.
- **Lane.** The experiments job routes a pair family to `nn/pair_lane.py`. A day without the pair examples object skips the pair families only, with a log line. A pair takes its report's input and state columns from the report's row of the day's `inbox_ranking_nn_examples` object, and its team's project vectors as of its birth day. The experiment and its baseline fit with each of `EXPERIMENT_SEEDS` on the same train pairs and are graded on the same holdout pairs.
- **Reference.** For each seed, the reference fits at report grain on the report train rows born on a pair train day, and each holdout pair takes its report's score. The reference fits before the pair matrices are built, so its report matrix and the pair matrices are never in memory together.
-- **Grade.** `vs_baseline` (`nn/pair_grade.py`) has `grain: "pair"` and, per head, within-user AUCs for each arm and seed. A user is one person in one team, and a user with one outcome class is left out. `macro` weighs each user the same, and `pooled` weighs each user by its pairs. The scopes are `all` and `non_dogfood` (without the dogfood project's users). The slices are `all`, the prior-view buckets, `history_10_plus`, `suggested_reviewer` and `not_suggested_reviewer`. `vs_baseline` and `vs_reference` give the mean, min and max delta over the paired seeds, and a team bootstrap interval for the macro delta. One `inbox_ranking_experiment_compared` event per head, scope and slice carries `grain`, `scope`, `slice` and the deltas against both arms. Report-grain experiments keep their block and their events as they were.
+- **Grade.** `vs_baseline` (`nn/pair_grade.py`) has `grain: "pair"` and, per head, within-user AUCs for each arm and seed. A user is one person in one team, and a user with one outcome class is left out. `macro` weighs each user the same, and `pooled` weighs each user by its pairs. The scopes are `all` and `non_dogfood` (without the dogfood project's users). The slices are `all`, the prior-view buckets, `history_10_plus`, `suggested_reviewer` and `not_suggested_reviewer`. `vs_baseline`, `vs_reference` and `vs_cold` give the mean, min and max delta over the paired seeds, and a team bootstrap interval for the macro delta. A broadcast head has `group: "team"` and is graded per report, never within user: one row per holdout report, each arm's score the mean over the report's pairs, slice `all` only. One `inbox_ranking_experiment_compared` event per head, scope and slice carries `grain`, `grade`, `group`, `scope`, `slice` and the deltas against each arm. Report-grain experiments keep their block and their events as they were.
+- **Cold path.** A family with `pair_training` (`PairTrainingConfig` in `multihead/inputs.py`) trains its cold score: the same network with every user input at zero. During training, user-input dropout zeroes the user offsets, the `log1p` counts, the days since the last view and `has_user_history` of a share of each batch's rows, and keeps `is_suggested_reviewer` and the report inputs. The cold score also zeroes the reviewer flag, as a caller without a user has none, so every pair of a report gets one cold score. The `cold` arm (`vs_cold`, personal minus cold, per history bucket) reads it within user. The `cold_path` block grades it at report grain against the reference on the report-level labels of the holdout reports with pairs, grouped by team, with events of `grade: "cold_path"`.
+- **Overfitting.** A family with `pair_training` stops early on the newest train days (`validation_share`), keeps the weights of its best validation epoch, and refits on every labeled row for that many epochs. It adds dropout on the projected user offsets and their products with the report. Its candidate records `pair_fit`: the kept epochs and the train, validation and holdout AUC per head, each row weighted by its loss weight. Its recipe records `pair_training`. A family without it trains and hashes as before.
#### User-history families
@@ -523,6 +526,8 @@ A family with `grain="pair"` scores one row per (report, impressed user) from `i
- **Per pair.** The lane builds each arm's inputs from the pair rows. The report part is built once per report, and the user columns and offsets come from each pair's own user, so two users of one report never share them. The user vectors come from `pair_user_vectors` on the dt=D interactions snapshot of teams in `training_consent_team_ids()`, inside each team's organization, as of the start of the birth day. A family that reads user history skips a partition without that snapshot.
- **Metadata.** A pair candidate records `user_vector_kinds`, and its recipe records them with `user_vector_settings`. The asset metadata adds `pair_rows`, `pair_users` and `pair_rows_with_prior_views` per family.
+`nn_pair_user_v2` and `nn_pair_base_v2` are the same comparison with the heads `open`, `action` and `fixed`, a trained cold path and early stopping (`PAIR_TRAINING`). Both train until 2026-12-09. The two arms differ only in user history. Read the cold path first: in `grade=cold_path`, `scope=non_dogfood`, the macro delta against the reference for each head, whose team bootstrap interval should include or sit above zero. Then read `vs_cold` per history bucket, which should not fall below zero beyond seed noise, and the `history_10_plus` go bar above for `open` and `action`. `pair_fit` shows the train to validation to holdout gap.
+
To run both families for one partition, launch the experiments job on the `posthog.dags.locations.signals_ml` location with the tag `dagster/partition` set to the day (`YYYY-MM-DD`), as in the command above. The job needs no run config. A launch without an asset selection runs all three assets, so it also trains the report-grain experiments. To train only the pair families, select `inbox_ranking_nn_pair_examples` and `inbox_ranking_nn_pair_experiment_candidates`. Launch a day the `nn` job finished, and whose dt=D interactions and embeddings snapshots and dt=B+7 interactions partitions exist.
Before the first read, launch `inbox_ranking_nn_pair_examples_job` alone on a recent day and check its metadata: `rows`, `users`, positives per label, rows and users per prior-view bucket, and `suggested_reviewer_share`. The `history_10_plus` holdout needs users with both classes in enough teams for the within-user grade and its team bootstrap to read.
diff --git a/products/signals/dags/inbox_ranking/multihead/inputs.py b/products/signals/dags/inbox_ranking/multihead/inputs.py
index 85c81feb4c7e..9eed90de77a6 100644
--- a/products/signals/dags/inbox_ranking/multihead/inputs.py
+++ b/products/signals/dags/inbox_ranking/multihead/inputs.py
@@ -25,6 +25,39 @@ class NetworkConfig:
seed: int = 0
+@frozen
+class PairTrainingConfig:
+ """How a pair network trains its cold path and holds back its user branch, beside its
+ `NetworkConfig`. A pair family without one trains as the report lane does.
+
+ The cold path is the same network with every user input at zero. User-input dropout zeroes the
+ user offsets and the user-history columns of a share of each batch's rows, so the network learns
+ to score a row without them. It keeps `is_suggested_reviewer` and the report inputs.
+
+ Early stopping holds out the newest train days as a validation split, keeps the weights of the
+ epoch with the lowest validation loss, and refits on every labeled row for that many epochs.
+ """
+
+ user_input_dropout: float = 0.2
+ # Dropout on the projected user offsets and their products with the report, apart from the
+ # trunk's own dropout, because the user branch overfits first.
+ user_dropout: float = 0.5
+ validation_share: float = 0.15
+ patience: int = 3
+ max_epochs: int = 30
+
+ def __post_init__(self) -> None:
+ for name, share in (
+ ("user_input_dropout", self.user_input_dropout),
+ ("user_dropout", self.user_dropout),
+ ("validation_share", self.validation_share),
+ ):
+ if not 0.0 <= share < 1.0:
+ raise ValueError(f"{name} is {share}; pick a share in [0, 1)")
+ if self.patience < 1 or self.max_epochs < 1:
+ raise ValueError("patience and max_epochs must be at least 1")
+
+
@frozen
class ProjectContext:
"""The team's project vectors per (team, moment), shared by every report of that pair."""
diff --git a/products/signals/dags/inbox_ranking/multihead/network.py b/products/signals/dags/inbox_ranking/multihead/network.py
index 079c956c7394..f665229d8d35 100644
--- a/products/signals/dags/inbox_ranking/multihead/network.py
+++ b/products/signals/dags/inbox_ranking/multihead/network.py
@@ -22,10 +22,16 @@ default of zero, the graph is the vector-only one.
A label is NaN where the head has no example for the row, and a NaN label adds no gradient, so one
network trains every head on rows that only some heads cover.
+A pair network can train with a `PairTrainingConfig`: per-row label weights (a broadcast head's
+1/pairs), user-input dropout so the zeroed-user cold path is trained, its own dropout on the user
+parts of the trunk input, and early stopping on validation rows. Every one of them acts in training
+only, so the exported graph has the same inputs and outputs.
+
torch is not a main dependency: it comes from the `inbox-ranking` dependency group. Import this
module only from dev code, never from a module on the Django, Temporal or Dagster import path.
"""
+import copy
import math
from collections.abc import Iterator, Sequence
from contextlib import contextmanager
@@ -37,7 +43,13 @@ from torch import nn
from torch.nn import functional as F
from products.signals.backend.ranking.features import EMBEDDING_DIMENSIONS
-from products.signals.dags.inbox_ranking.multihead.inputs import ONNX_INPUTS, ONNX_OUTPUT, NetworkConfig, NetworkInputs
+from products.signals.dags.inbox_ranking.multihead.inputs import (
+ ONNX_INPUTS,
+ ONNX_OUTPUT,
+ NetworkConfig,
+ NetworkInputs,
+ PairTrainingConfig,
+)
# text-embedding-3 vectors are unit length, so each component is about 1/sqrt(D). Scaling by
# sqrt(D) gives the projection unit-variance inputs, which is what its initialization assumes.
@@ -48,7 +60,14 @@ ONNX_OPSET = 17
class _Model(nn.Module):
def __init__(
- self, *, heads: int, kinds: int, config: NetworkConfig, report_features: int = 0, user_kinds: int = 0
+ self,
+ *,
+ heads: int,
+ kinds: int,
+ config: NetworkConfig,
+ report_features: int = 0,
+ user_kinds: int = 0,
+ user_dropout: float | None = None,
) -> None:
super().__init__()
self.kinds = kinds
@@ -57,6 +76,9 @@ class _Model(nn.Module):
k, h = config.projection_dim, config.trunk_dim
self.projection = nn.Linear(EMBEDDING_DIMENSIONS, k)
self.input_dropout = nn.Dropout(config.dropout)
+ # A dropout module holds no weights and draws nothing at construction, so a network without
+ # one initializes and exports as before.
+ self.user_dropout = nn.Dropout(user_dropout) if user_dropout is not None and user_kinds else None
self.trunk = nn.Linear(k + report_features + kinds * (2 * k + 1) + user_kinds * 2 * k, h)
self.output_dropout = nn.Dropout(config.dropout)
self.heads = nn.Linear(h, heads)
@@ -92,7 +114,10 @@ class _Model(nn.Module):
if user is None:
raise ValueError("a network built with user history needs user offsets")
projected_user = self.projection(user * INPUT_SCALE)
- parts += [projected_user.flatten(1), (projected.unsqueeze(1) * projected_user).flatten(1)]
+ user_parts = torch.cat(
+ [projected_user.flatten(1), (projected.unsqueeze(1) * projected_user).flatten(1)], dim=1
+ )
+ parts.append(self.user_dropout(user_parts) if self.user_dropout is not None else user_parts)
trunk_input = torch.cat(parts, dim=1) if len(parts) > 1 else projected
trunk_out = F.relu(self.trunk(self.input_dropout(trunk_input)))
return self.heads(self.output_dropout(trunk_out))
@@ -109,12 +134,31 @@ class _Probabilities(nn.Module):
return torch.sigmoid(self.model(*inputs))
-def masked_binary_cross_entropy(logits: torch.Tensor, labels: torch.Tensor) -> torch.Tensor:
- """The mean binary cross-entropy over the labels that are not NaN."""
+def masked_binary_cross_entropy(
+ logits: torch.Tensor, labels: torch.Tensor, weights: torch.Tensor | None = None
+) -> torch.Tensor:
+ """The mean binary cross-entropy over the labels that are not NaN, weighted by `weights`
+ (rows, heads) when given."""
observed = ~torch.isnan(labels)
# The NaN labels are replaced before the loss, not masked after it: 0 * NaN is NaN, also in the gradient.
losses = F.binary_cross_entropy_with_logits(logits, torch.where(observed, labels, 0.0), reduction="none")
- return (losses * observed).sum() / observed.sum().clamp(min=1)
+ if weights is None:
+ return (losses * observed).sum() / observed.sum().clamp(min=1)
+ weighted = observed * weights
+ # A batch can hold less than one unit of weight, such as one report's broadcast rows alone.
+ return (losses * weighted).sum() / weighted.sum().clamp(min=1e-8)
+
+
+def drop_user_inputs(
+ tensors: tuple[torch.Tensor, ...], dropped: torch.Tensor, user_columns: Sequence[int]
+) -> tuple[torch.Tensor, ...]:
+ """`tensors` with every user input of the `dropped` rows at zero: the `user_columns` of the
+ report input and the user offsets. The project inputs and every other report column stay."""
+ report, project, has_history, user = tensors
+ columns = torch.zeros(report.shape[1], dtype=torch.bool)
+ columns[list(user_columns)] = True
+ report = report.masked_fill(dropped[:, None] & columns[None, :], 0.0)
+ return report, project, has_history, user.masked_fill(dropped[:, None, None], 0.0)
class MultiHeadNetwork:
@@ -127,7 +171,11 @@ class MultiHeadNetwork:
config: NetworkConfig,
report_features: int = 0,
user_kinds: int = 0,
+ pair: PairTrainingConfig | None = None,
+ user_columns: Sequence[int] = (),
) -> None:
+ """`pair` and `user_columns` are for a pair network: `user_columns` are the report-input
+ columns that user-input dropout zeroes beside the user offsets."""
if user_kinds and not with_context:
raise ValueError("a network that reads user history needs project vectors")
self.head_names = tuple(heads)
@@ -136,6 +184,9 @@ class MultiHeadNetwork:
self.report_features = report_features
self.user_kinds = user_kinds
self.config = config
+ self.pair = pair
+ self.user_columns = tuple(user_columns)
+ self.epochs_kept = 0
# Every random draw (initialization, shuffle, dropout) comes from this seeded state, and the
# global torch state stays as it was.
self._rng_state = torch.Generator().manual_seed(config.seed).get_state()
@@ -146,6 +197,7 @@ class MultiHeadNetwork:
config=config,
report_features=report_features,
user_kinds=user_kinds,
+ user_dropout=pair.user_dropout if pair is not None else None,
)
weights = [parameter for name, parameter in self.model.named_parameters() if name.endswith("weight")]
biases = [parameter for name, parameter in self.model.named_parameters() if name.endswith("bias")]
@@ -158,24 +210,73 @@ class MultiHeadNetwork:
lr=config.learning_rate,
)
- def fit(self, inputs: NetworkInputs, labels: np.ndarray, rows: np.ndarray) -> list[float]:
- """Train on `rows` for the configured epochs; returns the mean loss per epoch."""
+ def fit(
+ self,
+ inputs: NetworkInputs,
+ labels: np.ndarray,
+ rows: np.ndarray,
+ *,
+ weights: np.ndarray | None = None,
+ validation_rows: np.ndarray | None = None,
+ epochs: int | None = None,
+ ) -> list[float]:
+ """Train on `rows`; returns the mean loss per epoch.
+
+ Without `validation_rows` the fit runs `epochs`, by default the configured ones. A pair
+ network with validation rows runs up to its `max_epochs`, stops after `patience` epochs
+ without a lower validation loss, and keeps the weights of its best epoch."""
+ stopping = self.pair is not None and validation_rows is not None and len(validation_rows) > 0
+ if epochs is None:
+ epochs = self.pair.max_epochs if stopping and self.pair is not None else self.config.epochs
history: list[float] = []
- self.model.train()
+ best_loss, best_state, best_epoch = math.inf, None, 0
with self._seeded():
- for _ in range(self.config.epochs):
+ for epoch in range(1, epochs + 1):
+ self.model.train()
losses = []
for batch in self._batches(rows, shuffle=True):
loss = masked_binary_cross_entropy(
- self.model(*self._tensors(inputs, batch)), torch.from_numpy(labels[batch]).float()
+ self.model(*self._train_tensors(inputs, batch)),
+ torch.from_numpy(labels[batch]).float(),
+ torch.from_numpy(weights[batch]).float() if weights is not None else None,
)
self._optimizer.zero_grad()
loss.backward()
self._optimizer.step()
losses.append(loss.item())
history.append(float(np.mean(losses)) if losses else float("nan"))
+ if not stopping or validation_rows is None or self.pair is None:
+ continue
+ validation_loss = self.loss(inputs, labels, validation_rows, weights=weights)
+ if validation_loss < best_loss:
+ best_loss, best_epoch = validation_loss, epoch
+ best_state = copy.deepcopy(self.model.state_dict())
+ elif epoch - best_epoch >= self.pair.patience:
+ break
+ if best_state is not None:
+ self.model.load_state_dict(best_state)
+ self.epochs_kept = best_epoch if best_state is not None else len(history)
return history
+ def loss(
+ self, inputs: NetworkInputs, labels: np.ndarray, rows: np.ndarray, *, weights: np.ndarray | None = None
+ ) -> float:
+ """The masked loss of `rows`, without dropout, over the whole of `rows` at once."""
+ if len(rows) == 0:
+ return math.nan
+ self.model.eval()
+ with torch.no_grad():
+ logits = torch.cat(
+ [self.model(*self._tensors(inputs, batch)) for batch in self._batches(rows, shuffle=False)]
+ )
+ return float(
+ masked_binary_cross_entropy(
+ logits,
+ torch.from_numpy(labels[rows]).float(),
+ torch.from_numpy(weights[rows]).float() if weights is not None else None,
+ )
+ )
+
def predict(self, inputs: NetworkInputs, rows: np.ndarray) -> np.ndarray:
"""(len(rows), heads) probabilities, without dropout."""
if len(rows) == 0:
@@ -189,6 +290,13 @@ class MultiHeadNetwork:
]
)
+ def _train_tensors(self, inputs: NetworkInputs, rows: np.ndarray) -> tuple[torch.Tensor, ...]:
+ tensors = self._tensors(inputs, rows)
+ if self.pair is None or not self.pair.user_input_dropout or not self.user_kinds:
+ return tensors
+ dropped = torch.rand(len(rows)) < self.pair.user_input_dropout
+ return drop_user_inputs(tensors, dropped, self.user_columns)
+
def _tensors(self, inputs: NetworkInputs, rows: np.ndarray) -> tuple[torch.Tensor, ...]:
report = torch.from_numpy(inputs.reports[rows]).float()
if not self.with_context:
diff --git a/products/signals/dags/inbox_ranking/multihead/prototype.py b/products/signals/dags/inbox_ranking/multihead/prototype.py
index 44cf2908fc15..278e1b13c04a 100644
--- a/products/signals/dags/inbox_ranking/multihead/prototype.py
+++ b/products/signals/dags/inbox_ranking/multihead/prototype.py
@@ -78,6 +78,9 @@ class ExampleTable:
labels: np.ndarray
# (rows, heads) bool, the head's own `holdout_mask`.
holdout: np.ndarray
+ # (rows, heads) float32 loss weights, or None for a weight of one everywhere. A pair table sets
+ # a broadcast head's weight to 1/pairs of the report, so each report weighs one in that head.
+ weights: np.ndarray | None = None
def network_train_rows(self) -> np.ndarray:
"""Rows with a label for some head and in no head's holdout."""
diff --git a/products/signals/dags/inbox_ranking/nn/model.py b/products/signals/dags/inbox_ranking/nn/model.py
index b0b4e23c5ae7..9bc576074c34 100644
--- a/products/signals/dags/inbox_ranking/nn/model.py
+++ b/products/signals/dags/inbox_ranking/nn/model.py
@@ -31,7 +31,7 @@ from products.signals.dags.inbox_ranking.multihead.inputs import ONNX_INPUTS, ON
from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable
from products.signals.dags.inbox_ranking.nn.examples import NN_HEAD_NAMES, NN_HEADS
from products.signals.dags.inbox_ranking.training.calibration import CalibrationBucket
-from products.signals.dags.inbox_ranking.training.families import PAIR_GRAIN, ModelFamily
+from products.signals.dags.inbox_ranking.training.families import PAIR_BROADCAST_HEAD_NAMES, PAIR_GRAIN, ModelFamily
from products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME, Head
from products.signals.dags.inbox_ranking.training.recipe import model_recipe, with_recipe
from products.signals.dags.inbox_ranking.training.train import HeadMetrics, HoldoutGrade, _auc, holdout_grade
@@ -46,6 +46,43 @@ class NetworkHead:
calibration: tuple[CalibrationBucket, ...]
+@frozen
+class HeadFitAucs:
+ """One head's AUC on the train-only fit's own fit rows, its validation rows and its holdout,
+ each row weighted by its loss weight, so a broadcast head reads one unit per report."""
+
+ head: str
+ train_auc: float | None
+ validation_auc: float | None
+ holdout_auc: float | None
+
+
+@frozen
+class PairFitRecord:
+ """How a pair network with `PairTrainingConfig` fit: the epochs its early stopping kept, which
+ the refit on every row runs for, and the gap from train to validation to holdout per head."""
+
+ epochs_kept: int
+ fit_rows: int
+ validation_rows: int
+ heads: tuple[HeadFitAucs, ...]
+
+ def as_dict(self) -> dict[str, Any]:
+ return {
+ "epochs_kept": self.epochs_kept,
+ "fit_rows": self.fit_rows,
+ "validation_rows": self.validation_rows,
+ "heads": {
+ head.head: {
+ "train_auc": head.train_auc,
+ "validation_auc": head.validation_auc,
+ "holdout_auc": head.holdout_auc,
+ }
+ for head in self.heads
+ },
+ }
+
+
@frozen
class TrainedNetwork:
# The graphs' output columns, in order. Every head the network was fit on has a column, a
@@ -62,6 +99,7 @@ class TrainedNetwork:
vector_kinds: tuple[str, ...]
# The user-history kinds the graphs take, empty for a network without a `user` input.
user_vector_kinds: tuple[str, ...] = ()
+ pair_fit: PairFitRecord | None = None
def network_vector_kinds(inputs: NetworkInputs) -> tuple[str, ...]:
@@ -105,6 +143,9 @@ def network_recipe(
learner_params=dataclasses.asdict(config),
project_vector_kinds=vector_kinds,
)
+ if family.pair_training is not None:
+ # Only a family that sets it carries the key, so the first pair families keep their hashes.
+ recipe = {**recipe, "pair_training": dataclasses.asdict(family.pair_training)}
if family.grain == PAIR_GRAIN:
recipe = {
**recipe,
@@ -162,8 +203,11 @@ def candidate_metadata(
head.metrics.as_dict()
| {
"cohort": HEADS_BY_NAME[head.metrics.head].cohort.__name__,
- "horizon_days": (pair_settings or {}).get(
- "label_horizon_days", HEADS_BY_NAME[head.metrics.head].horizon_days
+ # A broadcast head keeps its report label, and so the report head's horizon.
+ "horizon_days": (
+ HEADS_BY_NAME[head.metrics.head].horizon_days
+ if head.metrics.head in PAIR_BROADCAST_HEAD_NAMES
+ else (pair_settings or {}).get("label_horizon_days", HEADS_BY_NAME[head.metrics.head].horizon_days)
),
"example_window_start": windows.get(head.metrics.head, {}).get("example_window_start"),
"example_cap_bound": windows.get(head.metrics.head, {}).get("example_cap_bound"),
@@ -177,6 +221,8 @@ def candidate_metadata(
if family.grain == PAIR_GRAIN:
metadata["grain"] = PAIR_GRAIN
metadata["user_vector_kinds"] = list(trained.user_vector_kinds)
+ if trained.pair_fit is not None:
+ metadata["pair_fit"] = trained.pair_fit.as_dict()
return with_recipe(
metadata,
network_recipe(
diff --git a/products/signals/dags/inbox_ranking/nn/pair_grade.py b/products/signals/dags/inbox_ranking/nn/pair_grade.py
index 4ab9bf90bd1c..36788f2af7a0 100644
--- a/products/signals/dags/inbox_ranking/nn/pair_grade.py
+++ b/products/signals/dags/inbox_ranking/nn/pair_grade.py
@@ -12,6 +12,12 @@ are also sliced by the user's prior views (`HISTORY_BUCKETS`) and by whether the
suggested reviewer of the report. The deltas are the experiment minus the baseline and the
experiment minus the reference, over the paired seeds. The macro delta also gets a team bootstrap
interval: the users of one team share an inbox, so they are not independent reads.
+
+A family trained with a cold path also has a `cold` arm: the same networks with every user input at
+zero (`vs_cold` is personal minus cold, per history bucket). A broadcast head such as `fixed` is a
+report outcome, so it is graded per report and never within user: one row per holdout report, each
+arm's score the mean over the report's pairs, grouped by team. The `cold_path` block grades the cold
+score at report grain against the reference, on the report-level labels of the reports with pairs.
"""
from collections.abc import Mapping, Sequence
@@ -20,10 +26,12 @@ from typing import Any
import numpy as np
import pandas as pd
+from posthog.dataclasses import frozen
+
from products.signals.dags.inbox_ranking.dataset.queries import LABELS_TEAM_ID
from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable
from products.signals.dags.inbox_ranking.nn.pairs import HISTORY_BUCKETS
-from products.signals.dags.inbox_ranking.training.families import PAIR_GRAIN
+from products.signals.dags.inbox_ranking.training.families import PAIR_BROADCAST_HEAD_NAMES, PAIR_GRAIN
from products.signals.dags.inbox_ranking.training.within_team import (
BOOTSTRAP_DRAWS,
group_aucs,
@@ -41,10 +49,16 @@ NON_DOGFOOD = "non_dogfood"
EXPERIMENT_ARM = "experiment"
BASELINE_ARM = "baseline"
REFERENCE_ARM = "reference"
+COLD_ARM = "cold"
+ARMS = (EXPERIMENT_ARM, BASELINE_ARM, REFERENCE_ARM, COLD_ARM)
# The delta block name and the arm the experiment is compared with.
-COMPARISONS = (("vs_baseline", BASELINE_ARM), ("vs_reference", REFERENCE_ARM))
+COMPARISONS = (("vs_baseline", BASELINE_ARM), ("vs_reference", REFERENCE_ARM), ("vs_cold", COLD_ARM))
READS = ("macro", "pooled")
+# What one group of a grade is: one user in one team, or one team for a per-report grade.
+USER_GROUP = "user"
+TEAM_GROUP = "team"
_GROUP_COLUMNS = ["report_team_id", "user_distinct_id"]
+_REPORT_KEY = ["report_id", "snapshot_date"]
def pair_scopes(rows: pd.DataFrame) -> dict[str, np.ndarray]:
@@ -101,7 +115,7 @@ def _slice_grade(
"teams": int(group_teams.loc[graded_users].nunique()),
}
reads: dict[str, dict[str, list[float | None]]] = {}
- for arm in (EXPERIMENT_ARM, BASELINE_ARM, REFERENCE_ARM):
+ for arm in ARMS:
if arm not in per_arm:
grade[arm] = None
continue
@@ -119,6 +133,67 @@ def _slice_grade(
return grade
+def _scoped_grades(
+ frame: pd.DataFrame,
+ outcomes: np.ndarray,
+ groups: np.ndarray,
+ arm_scores: Mapping[str, Sequence[np.ndarray] | None],
+ slices: Mapping[str, np.ndarray],
+ draws: int,
+) -> dict[str, dict[str, Any]]:
+ """Every scope and slice of one head's rows. `frame` holds `report_team_id` per row."""
+ group_teams = pd.Series(frame["report_team_id"].to_numpy(), index=groups).groupby(level=0).first()
+ return {
+ scope: {
+ slice_name: _slice_grade(
+ outcomes[scope_mask & slice_mask],
+ groups[scope_mask & slice_mask],
+ group_teams,
+ {
+ arm: [scores[scope_mask & slice_mask] for scores in per_seed] if per_seed is not None else None
+ for arm, per_seed in arm_scores.items()
+ },
+ draws,
+ )
+ for slice_name, slice_mask in slices.items()
+ }
+ for scope, scope_mask in pair_scopes(frame).items()
+ }
+
+
+def _team_groups(frame: pd.DataFrame) -> np.ndarray:
+ return frame.groupby("report_team_id", sort=False).ngroup().to_numpy()
+
+
+@frozen
+class _ReportRows:
+ """One row per report of a broadcast head's holdout pairs."""
+
+ # The report's first pair row, for its `report_team_id`.
+ frame: pd.DataFrame
+ outcomes: np.ndarray
+ # Per arm and seed, the mean score over the report's pairs; None for an arm without scores.
+ scores: dict[str, list[np.ndarray] | None]
+
+
+def _per_report(
+ frame: pd.DataFrame, outcomes: np.ndarray, arm_scores: Mapping[str, Sequence[np.ndarray] | None]
+) -> _ReportRows:
+ report = frame.groupby(_REPORT_KEY, sort=False).ngroup().to_numpy()
+ first = pd.Series(np.arange(len(report))).groupby(report).first().to_numpy()
+
+ def mean(scores: np.ndarray) -> np.ndarray:
+ return pd.Series(scores).groupby(report).mean().to_numpy()
+
+ return _ReportRows(
+ frame=frame.iloc[first].reset_index(drop=True),
+ outcomes=outcomes[first],
+ scores={
+ arm: [mean(s) for s in per_seed] if per_seed is not None else None for arm, per_seed in arm_scores.items()
+ },
+ )
+
+
def pair_vs_baseline_block(
table: ExampleTable,
heads: Sequence[str],
@@ -129,13 +204,15 @@ def pair_vs_baseline_block(
experiment_scores: Sequence[np.ndarray],
baseline_scores: Sequence[np.ndarray] | None = None,
reference_scores: Sequence[np.ndarray] | None = None,
+ cold_scores: Sequence[np.ndarray] | None = None,
draws: int = BOOTSTRAP_DRAWS,
) -> dict[str, Any]:
"""The `vs_baseline` block of a pair experiment, per head, on the same holdout pairs.
`table` holds one row per pair with labels in `heads` order. Every score array holds a row per
pair and a column per head, one array per seed. The reference scores are its report scores
- broadcast to every pair of the report. An arm that is None (no pair baseline, or no reference)
+ broadcast to every pair of the report, and the cold scores the experiment's own networks with
+ every user input at zero. An arm that is None (no pair baseline, no reference, or no cold path)
leaves its reads and its deltas None.
"""
train_rows = table.network_train_rows()
@@ -144,38 +221,86 @@ def pair_vs_baseline_block(
rows = table.head_rows(index, holdout=True)
frame = table.rows.iloc[rows].reset_index(drop=True)
outcomes = table.labels[rows, index].astype(int)
- groups = frame.groupby(_GROUP_COLUMNS, sort=False).ngroup().to_numpy()
- group_teams = pd.Series(frame["report_team_id"].to_numpy(), index=groups).groupby(level=0).first()
arm_scores = {
- EXPERIMENT_ARM: [scores[rows, index] for scores in experiment_scores],
- BASELINE_ARM: [scores[rows, index] for scores in baseline_scores] if baseline_scores is not None else None,
+ arm: [s[rows, index] for s in per_seed] if per_seed is not None else None
+ for arm, per_seed in (
+ (EXPERIMENT_ARM, experiment_scores),
+ (BASELINE_ARM, baseline_scores),
+ (REFERENCE_ARM, reference_scores),
+ (COLD_ARM, cold_scores),
+ )
+ }
+ head: dict[str, Any] = {
+ "train_rows": int((~np.isnan(table.labels[train_rows, index])).sum()),
+ "holdout_rows": len(rows),
+ "holdout_positives": int(outcomes.sum()),
+ }
+ if name in PAIR_BROADCAST_HEAD_NAMES:
+ reports = _per_report(frame, outcomes, arm_scores)
+ head |= {
+ "group": TEAM_GROUP,
+ "holdout_reports": len(reports.frame),
+ "scopes": _scoped_grades(
+ reports.frame,
+ reports.outcomes,
+ _team_groups(reports.frame),
+ reports.scores,
+ {ALL: np.ones(len(reports.frame), dtype=bool)},
+ draws,
+ ),
+ }
+ else:
+ groups = frame.groupby(_GROUP_COLUMNS, sort=False).ngroup().to_numpy()
+ head |= {
+ "group": USER_GROUP,
+ "holdout_users": len(np.unique(groups)),
+ "scopes": _scoped_grades(frame, outcomes, groups, arm_scores, pair_slices(frame), draws),
+ }
+ graded[name] = head
+ return {"grain": PAIR_GRAIN, "baseline": baseline, "reference": reference, "seeds": list(seeds), "heads": graded}
+
+
+def cold_path_block(
+ report_table: ExampleTable,
+ report_heads: Sequence[str],
+ pair_report_positions: np.ndarray,
+ heads: Sequence[str],
+ *,
+ reference: str | None,
+ cold_scores: Sequence[np.ndarray],
+ reference_scores: Sequence[np.ndarray] | None,
+ draws: int = BOOTSTRAP_DRAWS,
+) -> dict[str, Any]:
+ """The cold score against the report-level reference at report grain, per head, within team.
+
+ `report_table` holds the report rows with labels in `report_heads` order, and
+ `pair_report_positions` each pair's row in it (-1 for none). `cold_scores` are per seed, a row per
+ pair and a column per head of `heads`; every pair of a report has the same cold score. The
+ `reference_scores` are per seed, a row per report and a column per report head. A head reads the
+ reports in its own report-level holdout that have a pair.
+ """
+ first_pair = pd.Series(np.arange(len(pair_report_positions))).groupby(pair_report_positions).first()
+ first_pair = first_pair[first_pair.index >= 0]
+ graded: dict[str, dict[str, Any]] = {}
+ for index, name in enumerate(heads):
+ report_index = list(report_heads).index(name)
+ holdout = report_table.head_rows(report_index, holdout=True)
+ rows = holdout[np.isin(holdout, first_pair.index.to_numpy())]
+ pairs = first_pair.loc[rows].to_numpy()
+ frame = report_table.rows.iloc[rows].reset_index(drop=True)
+ outcomes = report_table.labels[rows, report_index].astype(int)
+ arm_scores = {
+ EXPERIMENT_ARM: [scores[pairs, index] for scores in cold_scores],
REFERENCE_ARM: (
- [scores[rows, index] for scores in reference_scores] if reference_scores is not None else None
+ [scores[rows, report_index] for scores in reference_scores] if reference_scores is not None else None
),
}
- slices = pair_slices(frame)
graded[name] = {
- "train_rows": int((~np.isnan(table.labels[train_rows, index])).sum()),
+ "group": TEAM_GROUP,
"holdout_rows": len(rows),
"holdout_positives": int(outcomes.sum()),
- "holdout_users": len(np.unique(groups)),
- "scopes": {
- scope: {
- slice_name: _slice_grade(
- outcomes[scope_mask & slice_mask],
- groups[scope_mask & slice_mask],
- group_teams,
- {
- arm: [scores[scope_mask & slice_mask] for scores in per_seed]
- if per_seed is not None
- else None
- for arm, per_seed in arm_scores.items()
- },
- draws,
- )
- for slice_name, slice_mask in slices.items()
- }
- for scope, scope_mask in pair_scopes(frame).items()
- },
+ "scopes": _scoped_grades(
+ frame, outcomes, _team_groups(frame), arm_scores, {ALL: np.ones(len(rows), dtype=bool)}, draws
+ ),
}
- return {"grain": PAIR_GRAIN, "baseline": baseline, "reference": reference, "seeds": list(seeds), "heads": graded}
+ return {"reference": reference, "heads": graded}
diff --git a/products/signals/dags/inbox_ranking/nn/pair_lane.py b/products/signals/dags/inbox_ranking/nn/pair_lane.py
index 9125fa61c53d..705c89d7092c 100644
--- a/products/signals/dags/inbox_ranking/nn/pair_lane.py
+++ b/products/signals/dags/inbox_ranking/nn/pair_lane.py
@@ -15,6 +15,13 @@ user columns and zero offsets, whatever the pair rows hold. The user vectors are
dt=D interactions and embeddings snapshots through `pair_user_vectors`, on the history of teams with
training consent only, as the pair examples asset builds them.
+A broadcast head (`fixed`) takes the report's own label and holdout flag from the report's examples
+row, on every pair of the report, with a loss weight of 1/pairs so each report weighs one. A family
+with `pair_training` fits through a `PairFit`, and its seed networks also score every pair with each
+user input at zero, including `is_suggested_reviewer`, as a caller without a user would. That cold
+score is one per report, and is graded within user against the personal score and at report grain
+against the reference.
+
The reference is a report-grain serving family. For each seed it is fit on the report rows of the
pair train days, at report grain with every `nn` head, and each holdout pair takes its report's
score. That answers "better than a report-level score" on the same days without loading the served
@@ -40,6 +47,8 @@ from posthog.dataclasses import frozen
from products.signals.backend.ranking.features import (
EMBEDDING_DIMENSIONS,
PAIR_STATE_COLUMNS,
+ SUGGESTED_REVIEWER_COLUMN,
+ USER_HISTORY_FEATURE_NAMES,
USER_HISTORY_KINDS,
USER_REPORTS_COLUMNS,
Extras,
@@ -63,7 +72,7 @@ from products.signals.dags.inbox_ranking.nn.dag import (
)
from products.signals.dags.inbox_ranking.nn.examples import NN_HEAD_NAMES, NN_HEADS, examples_from_arrow, network_inputs
from products.signals.dags.inbox_ranking.nn.model import family_heads, family_user_kinds, network_recipe
-from products.signals.dags.inbox_ranking.nn.pair_grade import pair_vs_baseline_block
+from products.signals.dags.inbox_ranking.nn.pair_grade import cold_path_block, pair_vs_baseline_block
from products.signals.dags.inbox_ranking.nn.pairs import (
NN_PAIR_EXAMPLES_TABLE,
PAIR_EXAMPLES_SCHEMA_VERSION,
@@ -72,7 +81,7 @@ from products.signals.dags.inbox_ranking.nn.pairs import (
team_organizations,
)
from products.signals.dags.inbox_ranking.training.dag import check_recipe
-from products.signals.dags.inbox_ranking.training.families import MODEL_FAMILIES, ModelFamily
+from products.signals.dags.inbox_ranking.training.families import MODEL_FAMILIES, PAIR_BROADCAST_HEAD_NAMES, ModelFamily
from products.signals.dags.inbox_ranking.training.side_inputs import load_side_inputs
# The pair labels' horizon and the pair object's schema, in a pair family's recipe.
@@ -103,20 +112,61 @@ class UserHistory:
organizations: Mapping[int, str]
-def pair_example_table(pairs: pd.DataFrame, report_rows: pd.DataFrame, heads: Sequence[str]) -> ExampleTable:
+def report_head_frame(report_table: ExampleTable, heads: Sequence[str]) -> pd.DataFrame:
+ """The report key, then each of `heads`' report-level label and holdout flag, one row per report
+ row of `report_table` (labels in `NN_HEAD_NAMES` order)."""
+ frame = report_table.rows[_REPORT_KEY].copy()
+ for head in heads:
+ index = NN_HEAD_NAMES.index(head)
+ frame[f"{head}_label"] = report_table.labels[:, index]
+ frame[f"{head}_holdout"] = report_table.holdout[:, index]
+ return frame
+
+
+def pair_example_table(
+ pairs: pd.DataFrame,
+ report_rows: pd.DataFrame,
+ heads: Sequence[str],
+ report_heads: pd.DataFrame | None = None,
+) -> ExampleTable:
"""One row per pair, with labels and holdout flags in `heads` order.
A pair takes the state columns of its report's examples row (such as `priority`), so a feature
set builds its report input as the report lane does. A pair whose report has no examples row is
- dropped. Every head of a pair shares the report's holdout flag.
+ dropped. A per-viewer head reads the pair's own label and the report's `open` holdout flag.
+
+ A broadcast head reads its report's own label and holdout flag from `report_heads`
+ (`report_head_frame`), and weighs 1/pairs of the report in the loss, so the report weighs one.
+ The table carries weights only when it has a broadcast head.
"""
+ broadcast = [head for head in heads if head in PAIR_BROADCAST_HEAD_NAMES]
+ if broadcast and report_heads is None:
+ raise ValueError(f"the broadcast heads {broadcast} need the report heads frame")
state = report_rows.drop(columns=["report_team_id"])
merged = pairs.merge(state, on=_REPORT_KEY, how="inner", validate="many_to_one")
+ if broadcast and report_heads is not None:
+ columns = [*_REPORT_KEY, *(f"{head}_{part}" for head in broadcast for part in ("label", "holdout"))]
+ merged = merged.merge(report_heads[columns], on=_REPORT_KEY, how="left", validate="many_to_one")
rows = merged[[*_PAIR_COLUMNS, *(c for c in state.columns if c not in _REPORT_KEY)]].reset_index(drop=True)
labels = np.column_stack([merged[f"{head}_label"].to_numpy(dtype=np.float32) for head in heads])
- holdout = np.repeat(merged["holdout"].to_numpy(dtype=bool)[:, None], len(heads), axis=1)
+ holdout = np.column_stack(
+ [
+ merged[f"{head}_holdout" if head in broadcast else "holdout"].fillna(False).to_numpy(dtype=bool)
+ for head in heads
+ ]
+ )
+ weights = None
+ if broadcast:
+ pairs_of_report = merged.groupby(_REPORT_KEY)["report_id"].transform("size").to_numpy(dtype=np.float32)
+ weights = np.column_stack(
+ [1.0 / pairs_of_report if head in broadcast else np.ones(len(merged), dtype=np.float32) for head in heads]
+ ).astype(np.float32)
+ weights = weights.reshape(len(rows), len(heads))
return ExampleTable(
- rows=rows, labels=labels.reshape(len(rows), len(heads)), holdout=holdout.reshape(len(rows), len(heads))
+ rows=rows,
+ labels=labels.reshape(len(rows), len(heads)),
+ holdout=holdout.reshape(len(rows), len(heads)),
+ weights=weights,
)
@@ -216,6 +266,25 @@ def pair_network_inputs(
return NetworkInputs(reports=pair_report_matrix(table.rows, extras, feature_set), context=project, users=users)
+def user_input_columns(feature_set: FeatureSet) -> tuple[int, ...]:
+ """The report-input columns of a pair set that hold user history, `is_suggested_reviewer` not
+ among them; empty for any other set."""
+ if not isinstance(feature_set, PairUserHistoryFeatureSet):
+ return ()
+ return tuple(feature_set.feature_names.index(name) for name in USER_HISTORY_FEATURE_NAMES)
+
+
+def cold_network_inputs(inputs: NetworkInputs, feature_set: FeatureSet) -> NetworkInputs:
+ """`inputs` as a caller without a user gives them: every user-history column, the reviewer flag
+ and the user offsets at zero. The report and project inputs stay, so every pair of a report gets
+ one cold score."""
+ if not isinstance(feature_set, PairUserHistoryFeatureSet):
+ raise ValueError(f"the {feature_set.name} feature set has no user inputs to zero")
+ reports = inputs.reports.copy()
+ reports[:, [*user_input_columns(feature_set), feature_set.feature_names.index(SUGGESTED_REVIEWER_COLUMN)]] = 0.0
+ return NetworkInputs(reports=reports, context=inputs.context, users=zero_user_context(len(reports)))
+
+
def load_user_history(client, bucket: str, prefix: str, partition_key: str) -> UserHistory | None:
"""The dt=D interactions snapshot of the teams with training consent, and their organizations;
None when the snapshot is missing."""
@@ -258,17 +327,15 @@ def reference_scores(
reference: ModelFamily,
extras: Extras,
day: DayInputs,
- heads: Sequence[str],
) -> list[np.ndarray]:
- """Per seed, the reference fit on the report rows of the pair train days, its report scores
- broadcast to every pair."""
+ """Per seed, the reference fit on the report rows of the pair train days, scoring every report
+ row with a column per `nn` head."""
from products.signals.dags.inbox_ranking.nn.train import ( # noqa: PLC0415 — keeps torch off the location's import path
seed_scores,
)
inputs = network_inputs(report_table, extras, day.labels, day.embeddings, reference.feature_set)
- positions = report_positions(report_table.rows, pair_table.rows)
- scores = seed_scores(
+ return seed_scores(
report_table,
inputs,
NN_HEADS,
@@ -276,7 +343,6 @@ def reference_scores(
EXPERIMENT_SEEDS,
train_rows=reference_train_rows(report_table, pair_table),
)
- return [broadcast_report_scores(seed, positions, NN_HEAD_NAMES, heads) for seed in scores]
def train_pair_candidate(
@@ -304,7 +370,9 @@ def train_pair_candidate(
del report_arrow
pair_frame = pairs.to_pandas()
del pairs
- table = pair_example_table(pair_frame, examples.table.rows, family.heads)
+ table = pair_example_table(
+ pair_frame, examples.table.rows, family.heads, report_head_frame(examples.table, family.heads)
+ )
without_report = len(pair_frame) - len(table.rows)
del pair_frame
if without_report:
@@ -357,15 +425,23 @@ def train_pair_candidate(
)
from products.signals.dags.inbox_ranking.nn.train import ( # noqa: PLC0415 — keeps torch off the location's import path
+ PairFit,
+ seed_networks,
seed_scores,
train_network,
)
heads = family_heads(family)
+ pair = (
+ PairFit(config=family.pair_training, user_columns=user_input_columns(family.feature_set))
+ if family.pair_training is not None
+ else None
+ )
has_train_pairs = len(table.network_train_rows()) > 0
- reference_seed_scores = None
+ positions = report_positions(examples.table.rows, table.rows)
+ reference_report_scores = None
if reference is not None and has_train_pairs:
- reference_seed_scores = reference_scores(examples.table, table, reference, extras, day, family.heads)
+ reference_report_scores = reference_scores(examples.table, table, reference, extras, day)
project = project_context(table.rows, day.labels, day.embeddings)
inputs = pair_network_inputs(table, extras, family.feature_set, project, history, day.embeddings)
baseline_inputs = (
@@ -375,23 +451,50 @@ def train_pair_candidate(
)
del day, history
- trained = train_network(table, inputs, heads, NETWORK_CONFIG)
+ trained = train_network(table, inputs, heads, NETWORK_CONFIG, pair)
vs_baseline = None
if has_train_pairs:
+ baseline_scores = (
+ seed_scores(table, baseline_inputs, heads, NETWORK_CONFIG, EXPERIMENT_SEEDS, pair=pair)
+ if baseline_inputs is not None
+ else None
+ )
+ # The cold inputs copy the pair report matrix, so the baseline's is gone before they exist.
+ del baseline_inputs
+ all_rows = np.arange(len(table.rows))
+ networks = seed_networks(table, inputs, heads, NETWORK_CONFIG, EXPERIMENT_SEEDS, pair=pair)
+ experiment_scores = [network.predict(inputs, all_rows) for network in networks]
+ cold_scores = None
+ if pair is not None:
+ cold_inputs = cold_network_inputs(inputs, family.feature_set)
+ cold_scores = [network.predict(cold_inputs, all_rows) for network in networks]
+ del cold_inputs
vs_baseline = pair_vs_baseline_block(
table,
family.heads,
baseline=family.baseline,
reference=family.reference,
seeds=EXPERIMENT_SEEDS,
- experiment_scores=seed_scores(table, inputs, heads, NETWORK_CONFIG, EXPERIMENT_SEEDS),
- baseline_scores=(
- seed_scores(table, baseline_inputs, heads, NETWORK_CONFIG, EXPERIMENT_SEEDS)
- if baseline_inputs is not None
+ experiment_scores=experiment_scores,
+ baseline_scores=baseline_scores,
+ reference_scores=(
+ [broadcast_report_scores(s, positions, NN_HEAD_NAMES, family.heads) for s in reference_report_scores]
+ if reference_report_scores is not None
else None
),
- reference_scores=reference_seed_scores,
+ cold_scores=cold_scores,
)
+ if cold_scores is not None:
+ vs_baseline["cold_path"] = cold_path_block(
+ examples.table,
+ NN_HEAD_NAMES,
+ positions,
+ family.heads,
+ reference=family.reference,
+ cold_scores=cold_scores,
+ reference_scores=reference_report_scores,
+ )
+ del networks
metadata = write_candidate(
context,
client,
diff --git a/products/signals/dags/inbox_ranking/nn/pairs.py b/products/signals/dags/inbox_ranking/nn/pairs.py
index 54e4a5f81ad4..0c7f3804a229 100644
--- a/products/signals/dags/inbox_ranking/nn/pairs.py
+++ b/products/signals/dags/inbox_ranking/nn/pairs.py
@@ -60,7 +60,7 @@ from products.signals.dags.inbox_ranking.dataset.project_vectors import (
)
from products.signals.dags.inbox_ranking.nn.dag import NN_ASSET_KWARGS, NN_EXAMPLES_TABLE
from products.signals.dags.inbox_ranking.training.dag import _LOOKBACK_MAPPING
-from products.signals.dags.inbox_ranking.training.families import PAIR_HEAD_NAMES
+from products.signals.dags.inbox_ranking.training.families import PAIR_VIEWER_HEAD_NAMES
from products.signals.dags.inbox_ranking.training.heads import HEADS_BY_NAME
NN_PAIR_EXAMPLES_TABLE = "inbox_ranking_nn_pair_examples"
@@ -69,7 +69,7 @@ PAIR_EXAMPLES_SCHEMA_VERSION = 1
# The report head whose rows, holdout and later label partition the pairs share.
_REPORT_HEAD = HEADS_BY_NAME["open"]
PAIR_HORIZON_DAYS = _REPORT_HEAD.horizon_days
-PAIR_HEADS = PAIR_HEAD_NAMES
+PAIR_HEADS = PAIR_VIEWER_HEAD_NAMES
DAYS_SINCE_VIEW_CAP = DAYS_SINCE_LAST_VIEW_CAP
REVIEWERS_SOURCE = "reviewers"
IMPRESSIONS_SOURCE = "impressions"
diff --git a/products/signals/dags/inbox_ranking/nn/train.py b/products/signals/dags/inbox_ranking/nn/train.py
index ed5344a55a77..838f0f93d4ed 100644
--- a/products/signals/dags/inbox_ranking/nn/train.py
+++ b/products/signals/dags/inbox_ranking/nn/train.py
@@ -6,6 +6,10 @@ holdout AUC of `NULL_PERMUTATIONS` networks fit on permuted labels by `NULL_MARG
ships is a refit on every labeled row, as a booster is, and the train-only graph is kept to grade
this model as a later champion.
+A pair network fit with a `PairFit` holds the newest train days out as validation rows, stops early
+on them, and its refit runs for the epochs the train-only fit kept. It reads the table's loss
+weights, and records each head's train, validation and holdout AUC.
+
This module imports torch through `multihead/network.py`. Import it only inside an asset body.
"""
@@ -20,11 +24,13 @@ from sklearn.metrics import roc_auc_score
from posthog.dataclasses import frozen
from products.signals.backend.ranking.features import EMBEDDING_DIMENSIONS
-from products.signals.dags.inbox_ranking.multihead.inputs import NetworkConfig, NetworkInputs
+from products.signals.dags.inbox_ranking.multihead.inputs import NetworkConfig, NetworkInputs, PairTrainingConfig
from products.signals.dags.inbox_ranking.multihead.network import MultiHeadNetwork, export_onnx
from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable
from products.signals.dags.inbox_ranking.nn.model import (
+ HeadFitAucs,
NetworkHead,
+ PairFitRecord,
TrainedNetwork,
network_user_kinds,
network_vector_kinds,
@@ -33,8 +39,44 @@ from products.signals.dags.inbox_ranking.training.heads import Head
from products.signals.dags.inbox_ranking.training.train import NULL_PERMUTATIONS, head_metrics
+@frozen
+class PairFit:
+ """A pair network's training settings and the report-input columns its user-input dropout zeroes."""
+
+ config: PairTrainingConfig
+ user_columns: tuple[int, ...]
+
+
+@frozen
+class _FitRows:
+ fit: np.ndarray
+ # The rows early stopping reads, empty without a `PairFit`.
+ validation: np.ndarray
+
+
+def validation_split(table: ExampleTable, train_rows: np.ndarray, share: float) -> _FitRows:
+ """The newest `share` of the train rows' report days as validation rows, at least one day, and
+ the rest as fit rows. Cut by day like the holdout, so a report never straddles the two and the
+ validation rows are the nearest in time to the holdout. A single train day gives no validation."""
+ days = table.rows["snapshot_date"].to_numpy()[train_rows]
+ unique = np.unique(days)
+ if share <= 0 or len(unique) < 2:
+ return _FitRows(fit=train_rows, validation=train_rows[:0])
+ cut = unique[-max(1, round(share * len(unique)))]
+ newest = days >= cut
+ return _FitRows(fit=train_rows[~newest], validation=train_rows[newest])
+
+
def _fit(
- inputs: NetworkInputs, heads: Sequence[Head], config: NetworkConfig, labels: np.ndarray, rows: np.ndarray
+ inputs: NetworkInputs,
+ heads: Sequence[Head],
+ config: NetworkConfig,
+ labels: np.ndarray,
+ rows: _FitRows,
+ *,
+ pair: PairFit | None = None,
+ weights: np.ndarray | None = None,
+ epochs: int | None = None,
) -> MultiHeadNetwork:
network = MultiHeadNetwork(
heads=[head.name for head in heads],
@@ -43,11 +85,19 @@ def _fit(
config=config,
report_features=inputs.reports.shape[1] - EMBEDDING_DIMENSIONS,
user_kinds=inputs.users.vectors.shape[1] if inputs.users is not None else 0,
+ pair=pair.config if pair is not None else None,
+ user_columns=pair.user_columns if pair is not None else (),
)
- network.fit(inputs, labels, rows)
+ network.fit(inputs, labels, rows.fit, weights=weights, validation_rows=rows.validation, epochs=epochs)
return network
+def _fit_rows(table: ExampleTable, train_rows: np.ndarray, pair: PairFit | None) -> _FitRows:
+ if pair is None:
+ return _FitRows(fit=train_rows, validation=train_rows[:0])
+ return validation_split(table, train_rows, pair.config.validation_share)
+
+
def _onnx_bytes(network: MultiHeadNetwork) -> bytes:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "model.onnx"
@@ -55,10 +105,10 @@ def _onnx_bytes(network: MultiHeadNetwork) -> bytes:
return path.read_bytes()
-def _auc(y: np.ndarray, scores: np.ndarray) -> float | None:
+def _auc(y: np.ndarray, scores: np.ndarray, weights: np.ndarray | None = None) -> float | None:
if len(y) == 0 or y.sum() in (0, len(y)):
return None
- return float(roc_auc_score(y, scores))
+ return float(roc_auc_score(y, scores, sample_weight=weights))
@frozen
@@ -66,6 +116,31 @@ class GradedHeads:
holdout_network: MultiHeadNetwork
trained: list[NetworkHead]
skipped: list[str]
+ pair_fit: PairFitRecord | None = None
+
+
+def _pair_fit_record(
+ table: ExampleTable, heads: Sequence[Head], scores: np.ndarray, rows: _FitRows, epochs_kept: int
+) -> PairFitRecord:
+ def head_auc(index: int, head_rows: np.ndarray) -> float | None:
+ head_rows = head_rows[~np.isnan(table.labels[head_rows, index])]
+ weights = table.weights[head_rows, index] if table.weights is not None else None
+ return _auc(table.labels[head_rows, index].astype(int), scores[head_rows, index], weights)
+
+ return PairFitRecord(
+ epochs_kept=epochs_kept,
+ fit_rows=len(rows.fit),
+ validation_rows=len(rows.validation),
+ heads=tuple(
+ HeadFitAucs(
+ head=head.name,
+ train_auc=head_auc(index, rows.fit),
+ validation_auc=head_auc(index, rows.validation),
+ holdout_auc=head_auc(index, table.head_rows(index, holdout=True)),
+ )
+ for index, head in enumerate(heads)
+ ),
+ )
def _graded_heads(
@@ -74,18 +149,29 @@ def _graded_heads(
heads: Sequence[Head],
config: NetworkConfig,
train_rows: np.ndarray,
+ pair: PairFit | None = None,
) -> GradedHeads:
"""The train-only network, each head it can grade, and each head with nothing to fit."""
all_rows = np.arange(len(inputs))
- holdout_network = _fit(inputs, heads, config, table.labels, train_rows)
+ rows = _fit_rows(table, train_rows, pair)
+ holdout_network = _fit(inputs, heads, config, table.labels, rows, pair=pair, weights=table.weights)
scores = holdout_network.predict(inputs, all_rows)
rng = np.random.default_rng(config.seed)
null_scores = []
for draw in range(NULL_PERMUTATIONS):
+ order = rng.permutation(train_rows)
permuted = table.labels.copy()
- permuted[train_rows] = table.labels[rng.permutation(train_rows)]
+ permuted[train_rows] = table.labels[order]
+ permuted_weights = None
+ if table.weights is not None:
+ permuted_weights = table.weights.copy()
+ permuted_weights[train_rows] = table.weights[order]
null_config = dataclasses.replace(config, seed=config.seed + draw)
- null_scores.append(_fit(inputs, heads, null_config, permuted, train_rows).predict(inputs, all_rows))
+ null_scores.append(
+ _fit(inputs, heads, null_config, permuted, rows, pair=pair, weights=permuted_weights).predict(
+ inputs, all_rows
+ )
+ )
trained: list[NetworkHead] = []
skipped: list[str] = []
@@ -112,23 +198,45 @@ def _graded_heads(
holdout_team_ids=table.team_ids(test),
)
trained.append(NetworkHead(metrics=metrics, calibration=calibration))
- return GradedHeads(holdout_network=holdout_network, trained=trained, skipped=skipped)
+ return GradedHeads(
+ holdout_network=holdout_network,
+ trained=trained,
+ skipped=skipped,
+ pair_fit=(
+ _pair_fit_record(table, heads, scores, rows, holdout_network.epochs_kept) if pair is not None else None
+ ),
+ )
def train_network(
- table: ExampleTable, inputs: NetworkInputs, heads: Sequence[Head], config: NetworkConfig
+ table: ExampleTable,
+ inputs: NetworkInputs,
+ heads: Sequence[Head],
+ config: NetworkConfig,
+ pair: PairFit | None = None,
) -> TrainedNetwork:
"""`heads` is the column order of `table`'s labels and of the graphs' output."""
output_heads = tuple(head.name for head in heads)
train_rows = table.network_train_rows()
refit_rows = np.flatnonzero((~np.isnan(table.labels)).any(axis=1))
- graded = _graded_heads(table, inputs, heads, config, train_rows) if len(train_rows) else None
+ graded = _graded_heads(table, inputs, heads, config, train_rows, pair) if len(train_rows) else None
trained = graded.trained if graded is not None else []
skipped = graded.skipped if graded is not None else list(output_heads)
model_onnx = holdout_model_onnx = None
if graded is not None and trained:
- # Refit on everything before shipping: the holdout only exists to grade the recipe.
- model_onnx = _onnx_bytes(_fit(inputs, heads, config, table.labels, refit_rows))
+ # Refit on everything before shipping: the holdout only exists to grade the recipe. With
+ # nothing left to validate on, the refit runs the epochs the train-only fit kept.
+ refit = _fit(
+ inputs,
+ heads,
+ config,
+ table.labels,
+ _FitRows(fit=refit_rows, validation=refit_rows[:0]),
+ pair=pair,
+ weights=table.weights,
+ epochs=graded.holdout_network.epochs_kept if pair is not None else None,
+ )
+ model_onnx = _onnx_bytes(refit)
holdout_model_onnx = _onnx_bytes(graded.holdout_network)
return TrainedNetwork(
output_heads=output_heads,
@@ -140,9 +248,32 @@ def train_network(
refit_rows=len(refit_rows),
vector_kinds=network_vector_kinds(inputs),
user_vector_kinds=network_user_kinds(inputs),
+ pair_fit=graded.pair_fit if graded is not None else None,
)
+def seed_networks(
+ table: ExampleTable,
+ inputs: NetworkInputs,
+ heads: Sequence[Head],
+ config: NetworkConfig,
+ seeds: Sequence[int],
+ train_rows: np.ndarray | None = None,
+ pair: PairFit | None = None,
+) -> list[MultiHeadNetwork]:
+ """One train-only fit per seed. `train_rows` defaults to the table's network train rows; the
+ pair lane's report-level reference narrows them to the pair train days."""
+ if train_rows is None:
+ train_rows = table.network_train_rows()
+ rows = _fit_rows(table, train_rows, pair)
+ return [
+ _fit(
+ inputs, heads, dataclasses.replace(config, seed=seed), table.labels, rows, pair=pair, weights=table.weights
+ )
+ for seed in seeds
+ ]
+
+
def seed_scores(
table: ExampleTable,
inputs: NetworkInputs,
@@ -150,15 +281,12 @@ def seed_scores(
config: NetworkConfig,
seeds: Sequence[int],
train_rows: np.ndarray | None = None,
+ pair: PairFit | None = None,
) -> list[np.ndarray]:
"""One train-only fit per seed, each scoring every row. Two arms fit with the same seeds on the
- same rows give the paired scores `vs_baseline_block` grades. `train_rows` defaults to the
- table's network train rows; the pair lane's report-level reference narrows them to the pair
- train days."""
- if train_rows is None:
- train_rows = table.network_train_rows()
+ same rows give the paired scores `vs_baseline_block` grades."""
all_rows = np.arange(len(inputs))
return [
- _fit(inputs, heads, dataclasses.replace(config, seed=seed), table.labels, train_rows).predict(inputs, all_rows)
- for seed in seeds
+ network.predict(inputs, all_rows)
+ for network in seed_networks(table, inputs, heads, config, seeds, train_rows, pair)
]
diff --git a/products/signals/dags/inbox_ranking/tests/test_experiments.py b/products/signals/dags/inbox_ranking/tests/test_experiments.py
index a18abd9fe36b..c609f1d1f56c 100644
--- a/products/signals/dags/inbox_ranking/tests/test_experiments.py
+++ b/products/signals/dags/inbox_ranking/tests/test_experiments.py
@@ -31,7 +31,7 @@ from products.signals.dags.inbox_ranking.dataset.dag import (
USER_INTERACTIONS_SCHEMA,
USER_INTERACTIONS_TABLE,
)
-from products.signals.dags.inbox_ranking.multihead.inputs import NetworkConfig
+from products.signals.dags.inbox_ranking.multihead.inputs import NetworkConfig, PairTrainingConfig
from products.signals.dags.inbox_ranking.multihead.local import report_vectors
from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable, project_context, report_matrix
from products.signals.dags.inbox_ranking.nn import experiments, pair_lane
@@ -63,10 +63,13 @@ from products.signals.dags.inbox_ranking.training.families import (
NN_MODEL_NAME,
NN_PAIR_BASE_MODEL_NAME,
NN_PAIR_USER_MODEL_NAME,
+ NN_PAIR_USER_V2_MODEL_NAME,
NN_WITH_PRIO_AA_MODEL_NAME,
NN_WITH_PRIO_FAMILY,
NN_WITH_PRIO_MODEL_NAME,
PAIR_GRAIN,
+ PAIR_HEAD_NAMES,
+ PAIR_TRAINING,
ModelFamily,
family_problems,
serving_families,
@@ -212,9 +215,9 @@ def test_a_run_builds_pair_examples_only_for_a_live_pair_experiment_and_trains_i
[
(lambda: _pair_experiment("served", experiment=False), "must be an experiment"),
(lambda: _pair_experiment("unanchored", reference=None), "needs a baseline, a reference, or both"),
- (lambda: _pair_experiment("fixed_head", heads=("open", "fixed")), "declares heads"),
+ (lambda: _pair_experiment("dismiss_head", heads=("open", "dismiss")), "declares heads"),
],
- ids=["pair_serving_family", "no_baseline_or_reference", "fixed_is_not_a_pair_head"],
+ ids=["pair_serving_family", "no_baseline_or_reference", "dismiss_is_not_a_pair_head"],
)
def test_a_pair_family_is_an_experiment_with_pair_heads_and_something_to_grade_against(build, message):
with pytest.raises(ValueError, match=message):
@@ -251,8 +254,30 @@ _PRIO_SERVING = ModelFamily(name=NN_WITH_PRIO_MODEL_NAME, feature_set=NN_WITH_PR
(_PRIO_SERVING, _pair_experiment("pair_base", reference="pair_base")),
["experiment pair_base names reference pair_base, which is not a report-grain serving family"],
),
+ (
+ (
+ _PRIO_SERVING,
+ _pair_experiment("pair_base"),
+ dataclasses.replace(
+ _pair_experiment("pair_user", baseline="pair_base"),
+ heads=PAIR_HEAD_NAMES,
+ pair_training=PAIR_TRAINING,
+ ),
+ ),
+ [
+ "pair experiment pair_user and its baseline pair_base differ in heads or pair training; "
+ "the two arms must differ only in user history"
+ ],
+ ),
+ ],
+ ids=[
+ "valid_pair_chain",
+ "mixed_grain_baseline",
+ "unregistered_pair_baseline",
+ "baseline_cycle",
+ "pair_reference",
+ "arms_differ_beyond_user_history",
],
- ids=["valid_pair_chain", "mixed_grain_baseline", "unregistered_pair_baseline", "baseline_cycle", "pair_reference"],
)
def test_a_pair_experiment_names_a_registered_pair_baseline_and_a_report_serving_reference(families, problems):
assert family_problems(families) == problems
@@ -512,6 +537,12 @@ def test_a_report_family_recipe_has_no_pair_keys_so_its_hash_stays_as_it_was():
PAIR_SETTINGS,
["open", "action"],
)
+ # The first pair families keep their hashes; a family with a cold path names its settings.
+ assert "pair_training" not in pair_recipe
+ cold_path_recipe = network_recipe(
+ dataclasses.replace(_pair_experiment("pair"), pair_training=PAIR_TRAINING), config, ("open",), PAIR_SETTINGS
+ )
+ assert cold_path_recipe["pair_training"] == dataclasses.asdict(PAIR_TRAINING)
def test_a_reference_score_is_broadcast_to_every_pair_of_its_report():
@@ -605,6 +636,46 @@ def test_the_pair_grade_is_within_user_with_dogfood_history_and_reviewer_slices_
assert event.properties["vs_reference_within_user_macro_mean_delta_auc"] == pytest.approx(0.5)
+def test_a_broadcast_head_is_graded_once_per_report_inside_each_team():
+ day = datetime.date(2026, 10, 1)
+ # (team, report, fixed label, the experiment's score of each of its pairs)
+ reports = [(3, "r1", 1, [0.9, 0.1, 0.8]), (3, "r2", 0, [0.5, 0.5]), (4, "r3", 1, [0.2]), (4, "r4", 0, [0.4, 0.1])]
+ rows, labels, scores = [], [], []
+ for team_id, report_id, label, pair_scores in reports:
+ for position, score in enumerate(pair_scores):
+ rows.append(
+ {
+ "report_id": report_id,
+ "snapshot_date": day,
+ "report_team_id": team_id,
+ "user_distinct_id": f"{report_id}-u{position}",
+ "prior_views": 0,
+ "is_suggested_reviewer": False,
+ }
+ )
+ labels.append([label])
+ scores.append([score])
+ label_array = np.array(labels, dtype=np.float32)
+ table = ExampleTable(rows=pd.DataFrame(rows), labels=label_array, holdout=np.ones_like(label_array, dtype=bool))
+
+ block = pair_vs_baseline_block(
+ table,
+ ("fixed",),
+ baseline=None,
+ reference=None,
+ seeds=(0,),
+ experiment_scores=[np.array(scores, dtype=np.float32)],
+ )
+
+ fixed = block["heads"]["fixed"]
+ assert (fixed["group"], fixed["holdout_reports"], fixed["holdout_rows"]) == ("team", 4, 8)
+ every = fixed["scopes"]["all"]["all"]
+ # Mean pair scores rank r1 over r2 in team 3, and r4 over r3 in team 4. Graded within user, no
+ # user would have both classes.
+ assert (every["rows"], every["users"]) == (4, 2)
+ assert every["experiment"]["macro"] == pytest.approx([0.5])
+
+
def _pair_examples(report_rows: pd.DataFrame, report_labels: np.ndarray, report_holdout: np.ndarray) -> pa.Table:
"""Two users per report with an `open` example: the first opens what the report's `open` label
says and has history, the second never opens and has none."""
@@ -654,8 +725,13 @@ def _reader_history(inbox: _SyntheticInbox, day: str) -> pa.Table:
@pytest.mark.parametrize(
"model_name,pairs_present",
- [(NN_PAIR_USER_MODEL_NAME, True), (NN_PAIR_BASE_MODEL_NAME, True), (NN_PAIR_USER_MODEL_NAME, False)],
- ids=["user_family", "base_family", "user_family_without_pairs"],
+ [
+ (NN_PAIR_USER_MODEL_NAME, True),
+ (NN_PAIR_BASE_MODEL_NAME, True),
+ (NN_PAIR_USER_MODEL_NAME, False),
+ (NN_PAIR_USER_V2_MODEL_NAME, True),
+ ],
+ ids=["user_family", "base_family", "user_family_without_pairs", "cold_path_family"],
)
def test_a_registered_pair_family_trains_scores_and_is_graded_within_user(monkeypatch, model_name, pairs_present):
pytest.importorskip("torch")
@@ -667,6 +743,16 @@ def test_a_registered_pair_family_trains_scores_and_is_graded_within_user(monkey
monkeypatch.setattr(pair_lane, "training_consent_team_ids", lambda: frozenset(TEAMS))
monkeypatch.setattr(pair_lane, "team_organizations", lambda team_ids: {t: f"org-{t}" for t in team_ids})
family = next(family for family in MODEL_FAMILIES if family.name == model_name)
+ if family.pair_training is not None:
+ family = dataclasses.replace(family, pair_training=PairTrainingConfig(max_epochs=12, patience=2))
+ monkeypatch.setattr(
+ pair_lane,
+ "MODEL_FAMILIES",
+ tuple(
+ dataclasses.replace(f, pair_training=family.pair_training) if f.pair_training is not None else f
+ for f in MODEL_FAMILIES
+ ),
+ )
inbox = _SyntheticInbox()
embeddings, labels = inbox.embeddings(), inbox.labels()
examples = build_examples(inbox.snapshots(), report_vectors(embeddings), holdout_days=HOLDOUT_DAYS)
@@ -694,14 +780,15 @@ def test_a_registered_pair_family_trains_scores_and_is_graded_within_user(monkey
metadata = json.loads(client.objects[metadata_key])
assert (metadata["grain"], metadata["recipe"]["grain"]) == (PAIR_GRAIN, PAIR_GRAIN)
assert (metadata["feature_set"], metadata["user_vector_kinds"]) == (family.feature_set.name, ["open", "dismiss"])
- assert [head["head"] for head in metadata["heads"]] == ["open", "action"]
+ assert [head["head"] for head in metadata["heads"]] == list(family.heads)
block = metadata["vs_baseline"]
assert (block["grain"], block["baseline"], block["reference"]) == (
PAIR_GRAIN,
family.baseline,
NN_WITH_PRIO_MODEL_NAME,
)
- for grade in block["heads"].values():
+ for name in ("open", "action"):
+ grade = block["heads"][name]
every = grade["scopes"]["all"]["all"]
assert every["vs_reference"]["macro"]["paired_seeds"] == 2
if family.baseline is None:
@@ -714,7 +801,9 @@ def test_a_registered_pair_family_trains_scores_and_is_graded_within_user(monkey
assert min(opened["experiment"]["macro"]) > 0.65
assert min(opened["reference"]["macro"]) > 0.75
- pair_table = pair_lane.pair_example_table(pairs.to_pandas(), table.rows, family.heads)
+ pair_table = pair_lane.pair_example_table(
+ pairs.to_pandas(), table.rows, family.heads, pair_lane.report_head_frame(table, family.heads)
+ )
history = pair_lane.load_user_history(client, "bucket", PREFIX, day)
inputs = pair_lane.pair_network_inputs(
pair_table,
@@ -732,3 +821,30 @@ def test_a_registered_pair_family_trains_scores_and_is_graded_within_user(monkey
cold_start = (pair_table.rows["prior_views"] == 0).to_numpy()
assert cold_start.any()
assert all(np.isfinite(head_scores).all() for head_scores in scores.values())
+
+ if family.pair_training is None:
+ assert "pair_fit" not in metadata and "cold_path" not in block and "pair_training" not in metadata["recipe"]
+ assert opened["vs_cold"] is None
+ return
+ fit = metadata["pair_fit"]
+ assert 1 <= fit["epochs_kept"] <= family.pair_training.max_epochs and fit["validation_rows"] > 0
+ assert set(fit["heads"]) == set(family.heads)
+ assert all(fit["heads"][head]["train_auc"] is not None for head in family.heads)
+ assert opened["vs_cold"]["macro"]["paired_seeds"] == 2
+ assert block["heads"]["fixed"]["group"] == "team"
+ fixed_horizon = next(head for head in metadata["heads"] if head["head"] == "fixed")["horizon_days"]
+ assert fixed_horizon == next(head for head in NN_HEADS if head.name == "fixed").horizon_days
+ cold_path = block["cold_path"]
+ assert (cold_path["reference"], set(cold_path["heads"])) == (NN_WITH_PRIO_MODEL_NAME, set(family.heads))
+ assert cold_path["heads"]["open"]["scopes"]["all"]["all"]["vs_reference"]["macro"]["paired_seeds"] == 2
+ events = experiment_comparison_events(metadata)
+ assert {event.properties["grade"] for event in events} == {"personal", "cold_path"}
+
+ cold = onnx_probabilities(
+ client.objects[model_object_key(PREFIX, family.name, day, ONNX_MODEL_FILE)],
+ pair_lane.cold_network_inputs(inputs, family.feature_set),
+ metadata[ONNX_OUTPUT_HEADS_KEY],
+ )
+ for head_scores in cold.values():
+ per_report = pd.Series(head_scores).groupby(pair_table.rows["report_id"].to_numpy())
+ np.testing.assert_allclose(per_report.max() - per_report.min(), 0.0, atol=1e-6)
diff --git a/products/signals/dags/inbox_ranking/tests/test_multihead_network.py b/products/signals/dags/inbox_ranking/tests/test_multihead_network.py
index 88143ca5ad9c..ed36d77cff52 100644
--- a/products/signals/dags/inbox_ranking/tests/test_multihead_network.py
+++ b/products/signals/dags/inbox_ranking/tests/test_multihead_network.py
@@ -8,20 +8,29 @@ import pytest
import numpy as np
from parameterized import parameterized
-from products.signals.backend.ranking.features import EMBEDDING_DIMENSIONS, USER_HISTORY_KINDS
+from products.signals.backend.ranking.features import (
+ EMBEDDING_DIMENSIONS,
+ PAIR_FEATURE_NAMES,
+ PAIR_USER_HISTORY_FEATURE_SET,
+ SUGGESTED_REVIEWER_COLUMN,
+ USER_HISTORY_KINDS,
+)
from products.signals.dags.inbox_ranking.dataset.project_vectors import VECTOR_KINDS
from products.signals.dags.inbox_ranking.multihead.inputs import (
NetworkConfig,
NetworkInputs,
+ PairTrainingConfig,
ProjectContext,
UserContext,
)
+from products.signals.dags.inbox_ranking.nn.pair_lane import user_input_columns
torch = pytest.importorskip("torch")
onnxruntime = pytest.importorskip("onnxruntime")
from products.signals.dags.inbox_ranking.multihead.network import ( # noqa: E402
MultiHeadNetwork,
+ drop_user_inputs,
export_onnx,
masked_binary_cross_entropy,
)
@@ -87,6 +96,61 @@ def test_a_nan_label_adds_no_gradient_and_no_loss() -> None:
assert logits.grad[1, 2].item() == 0.0
+def test_a_report_weighs_one_in_the_loss_whatever_its_number_of_pairs() -> None:
+ logits = torch.tensor([[2.0], [2.0], [2.0], [-1.0]])
+ labels = torch.tensor([[1.0], [1.0], [1.0], [0.0]])
+ weights = torch.tensor([[1 / 3], [1 / 3], [1 / 3], [1.0]])
+
+ per_report = masked_binary_cross_entropy(torch.tensor([[2.0], [-1.0]]), torch.tensor([[1.0], [0.0]]))
+
+ assert masked_binary_cross_entropy(logits, labels, weights).item() == pytest.approx(per_report.item())
+
+
+def test_user_input_dropout_zeroes_every_user_input_and_nothing_else() -> None:
+ names = PAIR_USER_HISTORY_FEATURE_SET.feature_names
+ columns = user_input_columns(PAIR_USER_HISTORY_FEATURE_SET)
+ reviewer = names.index(SUGGESTED_REVIEWER_COLUMN)
+ assert reviewer not in columns
+ assert {*columns, reviewer} == {names.index(name) for name in PAIR_FEATURE_NAMES}
+ generator = torch.Generator().manual_seed(0)
+ tensors = (
+ torch.rand(4, len(names), generator=generator) + 0.1,
+ torch.rand(4, KINDS, EMBEDDING_DIMENSIONS, generator=generator),
+ torch.ones(4, KINDS),
+ torch.rand(4, len(USER_HISTORY_KINDS), EMBEDDING_DIMENSIONS, generator=generator) + 0.1,
+ )
+ dropped = torch.tensor([True, False, True, False])
+
+ report, project, has_history, user = drop_user_inputs(tensors, dropped, columns)
+
+ kept = [column for column in range(len(names)) if column not in columns]
+ assert not report[dropped][:, list(columns)].any()
+ assert not user[dropped].any()
+ assert torch.equal(report[:, kept], tensors[0][:, kept])
+ assert torch.equal(report[~dropped], tensors[0][~dropped])
+ assert torch.equal(user[~dropped], tensors[3][~dropped])
+ assert torch.equal(project, tensors[1]) and torch.equal(has_history, tensors[2])
+
+
+def test_early_stopping_keeps_the_weights_of_the_best_validation_epoch() -> None:
+ # The validation rows repeat the fit rows with flipped labels, so every epoch that fits the
+ # fit rows better raises the validation loss after the first.
+ fit_inputs = _inputs(32, with_context=False, seed=4)
+ inputs = NetworkInputs(reports=np.concatenate([fit_inputs.reports, fit_inputs.reports]))
+ fit_labels = np.random.default_rng(5).integers(0, 2, size=(32, len(HEADS))).astype(np.float32)
+ labels = np.concatenate([fit_labels, 1.0 - fit_labels])
+ pair = PairTrainingConfig(patience=2, max_epochs=20)
+ config = NetworkConfig(projection_dim=8, trunk_dim=8, dropout=0.0, learning_rate=1e-2, batch_size=16)
+
+ stopped = MultiHeadNetwork(heads=HEADS, with_context=False, kinds=KINDS, config=config, pair=pair)
+ history = stopped.fit(inputs, labels, np.arange(32), validation_rows=np.arange(32, 64))
+ one_epoch = MultiHeadNetwork(heads=HEADS, with_context=False, kinds=KINDS, config=config, pair=pair)
+ one_epoch.fit(inputs, labels, np.arange(32), epochs=1)
+
+ assert (stopped.epochs_kept, len(history)) == (1, 1 + pair.patience)
+ np.testing.assert_allclose(stopped.predict(inputs, np.arange(32)), one_epoch.predict(inputs, np.arange(32)))
+
+
@parameterized.expand([("report_only", False), ("with_project_vectors", True)])
def test_a_seeded_fit_is_repeatable(_name: str, with_context: bool) -> None:
inputs = _inputs(64, with_context=with_context)
diff --git a/products/signals/dags/inbox_ranking/tests/test_pairs.py b/products/signals/dags/inbox_ranking/tests/test_pairs.py
index 2365858b50cf..63a0b3f1348c 100644
--- a/products/signals/dags/inbox_ranking/tests/test_pairs.py
+++ b/products/signals/dags/inbox_ranking/tests/test_pairs.py
@@ -25,9 +25,16 @@ from products.signals.dags.inbox_ranking.dataset.dag import (
from products.signals.dags.inbox_ranking.dataset.project_vectors import USER_VECTOR_KINDS, VECTOR_KINDS
from products.signals.dags.inbox_ranking.multihead.inputs import ProjectContext
from products.signals.dags.inbox_ranking.multihead.local import report_vectors
+from products.signals.dags.inbox_ranking.multihead.prototype import ExampleTable
from products.signals.dags.inbox_ranking.nn import pair_lane
from products.signals.dags.inbox_ranking.nn.dag import NN_EXAMPLES_TABLE
-from products.signals.dags.inbox_ranking.nn.pair_lane import load_user_history, pair_example_table, pair_network_inputs
+from products.signals.dags.inbox_ranking.nn.examples import NN_HEAD_NAMES
+from products.signals.dags.inbox_ranking.nn.pair_lane import (
+ load_user_history,
+ pair_example_table,
+ pair_network_inputs,
+ report_head_frame,
+)
from products.signals.dags.inbox_ranking.nn.pairs import (
IMPRESSIONS_SOURCE,
NN_PAIR_EXAMPLES_TABLE,
@@ -312,3 +319,49 @@ def test_each_pair_arm_builds_its_own_inputs_and_only_the_user_arm_reads_history
assert not offsets[row["u5"]].any()
# Without history the user reads as the longest time since a view, and nothing else.
np.testing.assert_allclose(user.reports[row["u5"], width:reviewer], [0.0, 0.0, 1.0, 0.0])
+
+
+def test_a_broadcast_head_takes_its_reports_own_label_and_weighs_one_per_report() -> None:
+ reports = pd.DataFrame(
+ {"report_id": ["a", "b"], "snapshot_date": [BIRTH] * 2, "report_team_id": [2, 2], "priority": ["P1", "P2"]}
+ )
+ # Report a's `fixed` label is 1 and in its own holdout; report b's has not matured.
+ report_labels = np.full((2, len(NN_HEAD_NAMES)), np.nan, dtype=np.float32)
+ report_holdout = np.zeros((2, len(NN_HEAD_NAMES)), dtype=bool)
+ fixed = NN_HEAD_NAMES.index("fixed")
+ report_labels[0, fixed], report_holdout[0, fixed] = 1.0, True
+ report_table = ExampleTable(rows=reports, labels=report_labels, holdout=report_holdout)
+ pairs = pd.DataFrame(
+ [
+ {
+ "report_id": report_id,
+ "snapshot_date": BIRTH,
+ "report_team_id": 2,
+ "user_distinct_id": user,
+ "holdout": False,
+ "open_label": float(user == "u1"),
+ "action_label": 0.0,
+ "user_open_reports": 0,
+ "user_dismiss_reports": 0,
+ "prior_views": 0,
+ "days_since_last_view": 90.0,
+ "has_history": False,
+ "is_suggested_reviewer": False,
+ }
+ for report_id, user in (("a", "u1"), ("a", "u2"), ("a", "u3"), ("b", "u1"))
+ ]
+ )
+ heads = ("open", "action", "fixed")
+
+ table = pair_example_table(pairs, reports, heads, report_head_frame(report_table, heads))
+
+ assert table.weights is not None
+ by_report = table.rows["report_id"].to_numpy()
+ np.testing.assert_allclose(table.labels[by_report == "a", 2], [1.0, 1.0, 1.0])
+ assert np.isnan(table.labels[by_report == "b", 2]).all()
+ np.testing.assert_allclose(table.labels[:, 0], [1.0, 0.0, 0.0, 1.0])
+ assert table.weights[by_report == "a", 2].sum() == pytest.approx(1.0)
+ np.testing.assert_array_equal(table.weights[:, :2], np.ones((4, 2)))
+ np.testing.assert_array_equal(table.holdout[:, 2], [True, True, True, False])
+ assert not table.holdout[:, :2].any()
+ assert pair_example_table(pairs, reports, ("open", "action")).weights is None
diff --git a/products/signals/dags/inbox_ranking/training/families.py b/products/signals/dags/inbox_ranking/training/families.py
index 4bafec300cd6..878bf1218d75 100644
--- a/products/signals/dags/inbox_ranking/training/families.py
+++ b/products/signals/dags/inbox_ranking/training/families.py
@@ -12,6 +12,11 @@ pair-grain family scores one row per (report, impressed user) from `inbox_rankin
and is experiment-only until a serving design reads a personal score. Its baseline is another pair
experiment, and its optional `reference` is a report-grain serving family fit on the same days and
graded on the same holdout pairs, so it answers "better than the report-level score".
+
+A pair family's heads are per viewer (`open`, `action`, from the pair examples) or broadcast
+(`fixed`): the report's own label on every pair of the report, weighted 1/pairs so a report shown to
+many people counts once. A pair family with `pair_training` also trains its cold path (the same
+network with every user input at zero) and stops early on a validation split of its train days.
"""
import datetime
@@ -31,6 +36,7 @@ from products.signals.backend.ranking.features import (
FeatureSet,
)
from products.signals.backend.ranking.serving_manifest import DEFAULT_MODEL_KIND, ONNX_MODEL_KIND
+from products.signals.dags.inbox_ranking.multihead.inputs import PairTrainingConfig
# The model family: which features and which learner, as against `model_version`, the partition day
# it was fit on. Both are in the identity, so two families trained on one day stay apart.
@@ -48,6 +54,10 @@ AA_SEED_OFFSET = 100
NN_PAIR_BASE_MODEL_NAME = "nn_pair_base_v1"
NN_PAIR_USER_MODEL_NAME = "nn_pair_user_v1"
PAIR_EXPERIMENT_EXPIRES = datetime.date(2026, 11, 25)
+NN_PAIR_BASE_V2_MODEL_NAME = "nn_pair_base_v2"
+NN_PAIR_USER_V2_MODEL_NAME = "nn_pair_user_v2"
+# Room for three reads at least seven days apart.
+PAIR_V2_EXPERIMENT_EXPIRES = datetime.date(2026, 12, 9)
# The Dagster code locations that train a family: `signals` on the main image, and `signals_ml` on
# the image that carries torch.
MAIN_LOCATION = "signals"
@@ -55,8 +65,12 @@ ML_LOCATION = "signals_ml"
REPORT_GRAIN: Final = "report"
PAIR_GRAIN: Final = "pair"
Grain = Literal["report", "pair"]
-# The heads a pair-grain family can declare: the pair examples carry these labels only.
-PAIR_HEAD_NAMES = ("open", "action")
+# The per-viewer heads, whose labels the pair examples carry.
+PAIR_VIEWER_HEAD_NAMES = ("open", "action")
+# The report heads whose own label goes to every pair of the report. No person is credited with it.
+PAIR_BROADCAST_HEAD_NAMES = ("fixed",)
+PAIR_HEAD_NAMES = (*PAIR_VIEWER_HEAD_NAMES, *PAIR_BROADCAST_HEAD_NAMES)
+PAIR_TRAINING = PairTrainingConfig()
@frozen
@@ -76,7 +90,9 @@ class ModelFamily:
A pair-grain family is always an experiment and declares its `heads` from `PAIR_HEAD_NAMES`. Its
`baseline` is another pair experiment, and it names a `baseline`, a `reference`, or both: the
- first pair family of a comparison has no pair baseline and is graded against the reference only."""
+ first pair family of a comparison has no pair baseline and is graded against the reference only.
+ Only a pair-grain family can set `pair_training`. A family without it trains as the first pair
+ families did, so their recipes stay as they were."""
name: str
feature_set: FeatureSet
@@ -91,6 +107,7 @@ class ModelFamily:
reference: str | None = None
project_vector_kinds: tuple[str, ...] = PROJECT_VECTOR_KINDS
seed_offset: int = 0
+ pair_training: PairTrainingConfig | None = None
def __post_init__(self) -> None:
self._check_project_vector_kinds()
@@ -99,8 +116,8 @@ class ModelFamily:
if self.grain == PAIR_GRAIN:
self._check_pair()
return
- if self.heads or self.reference is not None:
- raise ValueError(f"report-grain family {self.name} cannot declare heads or a reference")
+ if self.heads or self.reference is not None or self.pair_training is not None:
+ raise ValueError(f"report-grain family {self.name} cannot declare heads, a reference or pair training")
if self.experiment and (self.baseline is None or self.expires is None):
raise ValueError(f"experiment family {self.name} needs a baseline and an expires day")
if not self.experiment and (self.baseline is not None or self.expires is not None):
@@ -159,6 +176,9 @@ class ModelFamily:
# one input shape, the same pairs and seeds, and the same reviewer flag. The base holds every
# user-history input at zero, so `nn_pair_user_v1` against it is the uplift of the user's history
# alone. Both are also read against the report-level `nn_with_prio_v2`.
+# `nn_pair_base_v2` and `nn_pair_user_v2` are the same comparison with a broadcast `fixed` head,
+# user-input dropout for the cold path, and early stopping with stronger dropout on the user branch.
+# The two arms again differ only in user history.
MODEL_FAMILIES: tuple[ModelFamily, ...] = (
ModelFamily(
name=NN_MODEL_NAME, feature_set=REPORT_EMBEDDINGS_FEATURE_SET, learner=ONNX_MODEL_KIND, location=ML_LOCATION
@@ -216,7 +236,7 @@ MODEL_FAMILIES: tuple[ModelFamily, ...] = (
experiment=True,
expires=PAIR_EXPERIMENT_EXPIRES,
grain=PAIR_GRAIN,
- heads=PAIR_HEAD_NAMES,
+ heads=PAIR_VIEWER_HEAD_NAMES,
reference=NN_WITH_PRIO_MODEL_NAME,
),
ModelFamily(
@@ -228,8 +248,33 @@ MODEL_FAMILIES: tuple[ModelFamily, ...] = (
baseline=NN_PAIR_BASE_MODEL_NAME,
expires=PAIR_EXPERIMENT_EXPIRES,
grain=PAIR_GRAIN,
+ heads=PAIR_VIEWER_HEAD_NAMES,
+ reference=NN_WITH_PRIO_MODEL_NAME,
+ ),
+ ModelFamily(
+ name=NN_PAIR_BASE_V2_MODEL_NAME,
+ feature_set=PAIR_BASE_FEATURE_SET,
+ learner=ONNX_MODEL_KIND,
+ location=ML_LOCATION,
+ experiment=True,
+ expires=PAIR_V2_EXPERIMENT_EXPIRES,
+ grain=PAIR_GRAIN,
heads=PAIR_HEAD_NAMES,
reference=NN_WITH_PRIO_MODEL_NAME,
+ pair_training=PAIR_TRAINING,
+ ),
+ ModelFamily(
+ name=NN_PAIR_USER_V2_MODEL_NAME,
+ feature_set=PAIR_USER_HISTORY_FEATURE_SET,
+ learner=ONNX_MODEL_KIND,
+ location=ML_LOCATION,
+ experiment=True,
+ baseline=NN_PAIR_BASE_V2_MODEL_NAME,
+ expires=PAIR_V2_EXPERIMENT_EXPIRES,
+ grain=PAIR_GRAIN,
+ heads=PAIR_HEAD_NAMES,
+ reference=NN_WITH_PRIO_MODEL_NAME,
+ pair_training=PAIR_TRAINING,
),
)
NN_FAMILY = next(family for family in MODEL_FAMILIES if family.name == NN_MODEL_NAME)
@@ -270,6 +315,15 @@ def _baseline_problem(family: ModelFamily, by_name: dict[str, ModelFamily], serv
return f"experiment {family.name} names baseline {family.baseline}, which is not a serving family"
if family.grain == PAIR_GRAIN and (baseline is None or not baseline.experiment):
return f"experiment {family.name} names baseline {family.baseline}, which is not a registered pair experiment"
+ if (
+ family.grain == PAIR_GRAIN
+ and baseline is not None
+ and (baseline.heads, baseline.pair_training) != (family.heads, family.pair_training)
+ ):
+ return (
+ f"pair experiment {family.name} and its baseline {baseline.name} differ in heads or pair training; "
+ "the two arms must differ only in user history"
+ )
return None
diff --git a/products/signals/dags/inbox_ranking/training/telemetry.py b/products/signals/dags/inbox_ranking/training/telemetry.py
index 6ce5796eec63..cdcf30f52552 100644
--- a/products/signals/dags/inbox_ranking/training/telemetry.py
+++ b/products/signals/dags/inbox_ranking/training/telemetry.py
@@ -154,41 +154,66 @@ _REPORT_READS = (
)
+PERSONAL_GRADE = "personal"
+COLD_PATH_GRADE = "cold_path"
+
+
def _pair_comparison_events(block: Mapping[str, Any], context: Mapping[str, Any]) -> list[TrainingEvent]:
"""One event per head, scope and slice of a pair experiment's `vs_baseline` block, so a chart
filters to one slice (such as `non_dogfood` users with 10 or more prior views) and reads its
- within-user deltas against the baseline and the reference."""
+ within-user deltas against the baseline, the reference and the cold score. A `cold_path` block
+ adds one event per head and scope with `grade` set to `cold_path`. `group` says what the
+ `within_user` reads group by: a user, or a team for a per-report grade."""
events: list[TrainingEvent] = []
- for head, grade in (block.get("heads") or {}).items():
- for scope, slices in (grade.get("scopes") or {}).items():
- for slice_name, cell in slices.items():
- properties: dict[str, Any] = {
- **context,
- "grain": block.get("grain"),
- "baseline_model_name": block.get("baseline"),
- "reference_model_name": block.get("reference"),
- "seeds": len(block.get("seeds", [])),
- "head": head,
- **{key: grade.get(key) for key in ("train_rows", "holdout_rows", "holdout_positives")},
- "scope": scope,
- "slice": slice_name,
- **{f"slice_{key}": cell.get(key) for key in ("rows", "positives", "users", "teams")},
- }
- for comparison in ("vs_baseline", "vs_reference"):
- deltas = cell.get(comparison) or {}
- for read in ("macro", "pooled"):
- properties |= {
- f"{comparison}_within_user_{read}_{key}": (deltas.get(read) or {}).get(key)
- for key in _DELTA_KEYS
- }
- interval = deltas.get("macro_team_interval") or {}
- properties |= {
- f"{comparison}_within_user_macro_interval_{key}": interval.get(key) for key in _INTERVAL_KEYS
- }
- events.append(TrainingEvent(event=EXPERIMENT_COMPARED_EVENT, properties=properties))
+ grades = (
+ (PERSONAL_GRADE, block.get("heads") or {}),
+ (COLD_PATH_GRADE, (block.get("cold_path") or {}).get("heads") or {}),
+ )
+ for grade_name, heads in grades:
+ for head, grade in heads.items():
+ for scope, slices in (grade.get("scopes") or {}).items():
+ for slice_name, cell in slices.items():
+ events.append(
+ _pair_comparison_event(block, context, grade_name, head, grade, scope, slice_name, cell)
+ )
return events
+def _pair_comparison_event(
+ block: Mapping[str, Any],
+ context: Mapping[str, Any],
+ grade_name: str,
+ head: str,
+ grade: Mapping[str, Any],
+ scope: str,
+ slice_name: str,
+ cell: Mapping[str, Any],
+) -> TrainingEvent:
+ properties: dict[str, Any] = {
+ **context,
+ "grain": block.get("grain"),
+ "baseline_model_name": block.get("baseline"),
+ "reference_model_name": block.get("reference"),
+ "seeds": len(block.get("seeds", [])),
+ "grade": grade_name,
+ "group": grade.get("group", "user"),
+ "head": head,
+ **{key: grade.get(key) for key in ("train_rows", "holdout_rows", "holdout_positives")},
+ "scope": scope,
+ "slice": slice_name,
+ **{f"slice_{key}": cell.get(key) for key in ("rows", "positives", "users", "teams")},
+ }
+ for comparison in ("vs_baseline", "vs_reference", "vs_cold"):
+ deltas = cell.get(comparison) or {}
+ for read in ("macro", "pooled"):
+ properties |= {
+ f"{comparison}_within_user_{read}_{key}": (deltas.get(read) or {}).get(key) for key in _DELTA_KEYS
+ }
+ interval = deltas.get("macro_team_interval") or {}
+ properties |= {f"{comparison}_within_user_macro_interval_{key}": interval.get(key) for key in _INTERVAL_KEYS}
+ return TrainingEvent(event=EXPERIMENT_COMPARED_EVENT, properties=properties)
+
+
def experiment_comparison_events(metadata: Mapping[str, Any]) -> list[TrainingEvent]:
"""One event per head of an experiment candidate's `vs_baseline` block. The per-seed AUC lists
stay in `metadata.json`: an insight cannot chart a JSON array, so the event carries the gaps.