Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

15.7 KB
Raw Blame

Inbox ranking training examples

Priority edits in the web inbox append a priority_judgment artefact instead of changing the original assessment. The correction's adjustment records previous_priority, previous_judgment_id, and source="inbox_sidebar". The artefact's team/report IDs, creation timestamp, and actor attribution identify the correction and its author; the linked judgment retains the original explanation and dollar estimate. Selecting the current priority adds no correction. These records are available for future priority learning; they do not introduce a new ranking label or training head. Research already running when a priority is edited preserves that correction; a later research run can reassess it.

The nn ranking families use one example per report per head, from the report's birth-day snapshot. This prevents long-lived reports from receiving more weight in training because they appear in more daily snapshots. Birth days use the UTC interval returned by snapshot_bounds, including the start and excluding the end. The label comes from the snapshot horizon_days later, including outcomes already present on the birth day.

The families train only the heads the inbox reads: open, action and fixed. The dataset still snapshots the label columns of other outcomes, so a head can come back without a labels backfill. Manual resolutions with reason no_action_needed count as reasoned resolutions for the action head but do not count toward the fixed head. They record that the report was accepted without claiming an implementation or a PR merge.

reports_missing_birth_snapshot counts the observed reports born inside the lookback whose birth-day partition is missing. These reports produce no birth-grain example; reports born before the lookback do not contribute to this count. A missing horizon snapshot also prevents an example because its label is unknown. Point-in-time state, label provenance, and embedding availability checks still apply.

Every training series resets on the first partition after deploy. Before and after holdout numbers are not comparable because the example population changes. The daily snapshots and selectable scoring-moment and report grains remain available for future models.

See the ranking DAG README for the feature-set contract and operating instructions.

Cached Inbox ordering

The inbox-sort-cache flag controls reads from the cached priority and ranking columns. The flag stays off during deployment. API processes and workers update the cache even when the flag is off.

After every API process and worker uses the new cache writers, run python manage.py backfill_inbox_sort_cache --team-id <team-id> for each team in the rollout. The command uses locked batches and can run again after an interruption. Complete the backfill for every team in an organization before you enable the flag for that organization. Keep the flag off if any old writer remains. Old writers do not update the cache. Disable the flag to return to artefact queries if ordering differs during rollout.

What each probability means

Each head is conditioned on its cohort (training/heads.py). Every head uses every report as its cohort, so the probabilities are over the same reports and can be compared. Each head is graded only on its cohort.

Head Probability Horizon
open P(opened) over every report 7d
action P(acted on) over every report 14d
fixed P(fixed) over every report 30d

Each head in metadata.json records cohort (everyone) and horizon_days.

Row budget

A feature set can limit the rows one head keeps (max_examples_per_head). The report-embedding sets set it to 250,000. The budget limits history, not the rows inside a day:

  • The head's examples are grouped by report-creation day, newest first.
  • Whole days are kept while the running total stays within the budget, and every older day is dropped.
  • The newest day is always kept, even when it alone exceeds the budget.

Every kept day keeps all of its positives and negatives, so the kept label rate is the population rate of those days. The scores stay calibrated to that population, and the holdout stays a clean time split. A recency cut drops old positives with old negatives, so the budget must stay large enough to keep the rare heads fed.

Each head in metadata.json records example_window_start, the earliest report-creation day kept, and example_cap_bound, whether the budget dropped any day. inbox_ranking_candidate_trained carries both fields per head, so a chart shows when the budget starts to bind.

Promotion

Promotion is manual. A person runs inbox_ranking_promote_job with a model_key and a reason, and the job writes inbox_ranking_models/v1/served.json and publishes the serving manifest. Training never moves that pointer.

Each run still grades every candidate against the served model of its own model_name. The decision says the candidate would promote only when, on every head the served model could read:

  • its holdout AUC is at most AUC_TOLERANCE below the served model's, and
  • its holdout calibration error (ECE) is at most ECE_TOLERANCE above the served model's.

Both incumbent numbers come from the served model's <head>.holdout.ubj scored on the candidate's holdout. A head without a paired incumbent ECE skips the calibration check. inbox_ranking_promotion_decided carries the decision as would_promote and gates_passed, with promoted always false, and the paired values as champion_<head>_auc_on_this_holdout and champion_<head>_ece_on_this_holdout.

Baked and unbaked unseen metrics

inbox_ranking_unseen_graded evaluates the saved predictions for each scoring cohort every day, from its birth-day snapshot through each head's horizon. It reads each scores partition once per run and never re-scores an older cohort with a newer model. The first read has observed_days=0; the unit is the difference between UTC snapshot dates, not a full day of observation for every report.

  • Baked: inbox_ranking_unseen_head_graded, inbox_ranking_unseen_calibration, and inbox_ranking_unseen_report_graded retain their full-horizon semantics.
  • Daily evaluations: inbox_ranking_unseen_head_evaluated carries the head metrics at each observation age, including the final baked evaluation. Filter is_mature=false for unbaked results and is_mature=true for baked results.

The daily event includes scoring_partition, evaluation_partition, observed_days, horizon_days, is_mature, and evaluated_at, alongside the model family, version, role and pool. rows counts eligible reports, scored_rows counts the original scored reports, and cohort_coverage is their ratio. Eligibility can grow as reports receive impressions or opens; label availability and provenance rules still apply. Missing scores or required label columns produce an explicit skip in the asset metadata, not an all-negative grade. An empty eligible cohort emits zero rows and null metrics; a cohort with only one outcome class emits counts and a null AUC. readable continues to describe the model's holdout readability, independently of maturity or the evaluation's sample size.

Reading the two versions

For a cohort-date time series, group by (scoring_partition, pool, head, model_name, model_version, model_role). Select the complete event with the greatest (evaluation_partition, evaluated_at) in each group, rather than averaging or summing daily revisions. Use run_id as a tie-breaker for an identical evaluation timestamp. Select the complete event even when its AUC is null, so a previously defined AUC does not hide an empty or single-class revision. Plot scoring_partition on the x-axis and distinguish the unbaked tail with is_mature. Keep cohort age, horizon, rows and positives visible in the tooltip. The event timestamp remains the evaluation partition's midday UTC, so default event-time trends describe the evaluation day rather than the scoring cohort.

For a history at a fixed age, filter observed_days first and retain the newest evaluated_at for each group. For example, compare day-one cohorts with other day-one cohorts, while keeping the baked series alongside them. Model comparisons need the same reports, observation age and pool. The daily stream contains aggregate metrics only; per-report and per-decile events remain baked-only to bound telemetry volume.

An unbaked negative means the outcome has not happened in the available snapshot. Later outcomes can move AUC in either direction, and early evaluations can favor reports whose outcomes arrive quickly. Early calibration error compares a full-horizon prediction with incomplete outcomes; use baked results for calibration decisions and definitive regression judgments. Daily evaluations do not change training labels, champion promotion or serving. They stop at the head's horizon; a retry evaluates the named snapshot rather than extending that horizon to the retry date.

Verification after deployment

Check that the new event has an age-zero evaluation for each scored head and subsequent evaluations through its horizon. For the final evaluation, compare counts and metrics with the matching baked event. Check missing-partition metadata and grading runtime; the grader reads at most max(horizon_days) + 1 scores objects per run. Existing baked-only consumers need no filter changes. New charts must opt into the daily event and select one revision per cohort as described above.

Classification metrics

Every fitted head reports classification metrics next to AUC and calibration. They answer two questions: of the reports a head flags, how many have the outcome (precision), and of the outcomes, how many the head flags (recall).

Threshold

A head predicts positive when score >= classification_threshold. A score equal to the threshold is a positive prediction. The threshold is the positive rate of the rows the model was fit on. It means "at least as likely as the average fitting example", not a 50% probability.

  • Holdout: the threshold is the positive rate of the train-only rows. It is fixed before the holdout outcomes are read.
  • Unseen: the threshold is the positive rate of every row the refit was fit on. The candidate saves it per head as refit_classification_threshold in metadata.json. The scorer copies it onto each saved score as classification_threshold, and every daily and mature grade of those scores reads that value.
  • A champion keeps its own threshold. A candidate and a champion graded on the same reports use different cuts.

The threshold is specific to the head, family, version and fitting population. A row budget keeps only the newest report-creation days, so the rate is that window's rate, not the rate of the full lookback. Do not read the threshold as population prevalence, and do not compute it again from evaluation labels.

Metrics

The metrics use the same eligible rows as the other metrics of the grade.

  • Counts: true_positives, false_positives, true_negatives, false_negatives.
  • Ratios: precision, recall, f1, specificity, accuracy, balanced_accuracy (the mean of recall and specificity), predicted_positive_rate.
  • inbox_ranking_candidate_trained carries them with a holdout_ prefix, plus refit_classification_threshold.
  • inbox_ranking_unseen_head_graded and inbox_ranking_unseen_head_evaluated carry them without a prefix.

Null values have three causes:

  • A ratio with a zero denominator is null. For example, a cohort without positives has a null recall.
  • An empty cohort with a known threshold has zero counts and null ratios.
  • A model or scores object saved before thresholds existed has null classification fields. Its other metrics stay.

Readability and maturity rules do not change. An unbaked negative can still become a positive, so unbaked precision and recall are provisional. These metrics are evaluation-only: they do not change inbox order or champion promotion.

Pooling

Read families and maturity levels apart. To pool cohorts or days, select one revision per cohort as described in Reading the two versions, sum the confusion counts, and then compute the ratios from the sums. Never average daily precision, recall or F1. A pooled line over several versions uses the frozen threshold of each version, so label it that way.

After deployment and a new training and scoring run, check that the candidate events carry numeric holdout_ fields and refit_classification_threshold, and that the unseen events carry classification_threshold. Check the mature grades as each head's horizon becomes available.

Online classification metrics

The online grade reads the scores the scoring sweep wrote, for every model key the serving manifest names. No asset rescores the newborn pool offline.

Score events

Each inbox_ranking_report_scored event carries the served threshold of its model:

  • threshold_<head>: the head's refit_classification_threshold from the serving copy's metadata. A serving copy is immutable per model key, so this is the threshold that scored the report.
  • predicted_<head>: p_<head> >= threshold_<head>. A score equal to the threshold is a positive prediction.
  • readable_heads: the heads whose holdout was readable.

A head without a saved threshold has no threshold_ or predicted_ property. Models trained before thresholds existed have none, and no other value stands in for one.

The online grade

inbox_ranking_served_scores writes inbox_ranking_served_scores/v1/dt=D/ in the scores schema.

  • Population: D's newborn pool.
  • Score: per report and model key, the earliest scored event in D with this deployment's environment.
  • model_role: served when the event's roles hold it, else the first role (recent, cross_family, pinned), else candidate. Grades written before the recent role carry daily_candidate for the same read.
  • classification_threshold comes from threshold_<head>, and is null when the event has none.
  • Asset metadata: served_pool_coverage, pool_coverage_by_model, rows per model version, and the heads without a threshold.

inbox_ranking_unseen_graded reads this object for each scoring partition. Its events carry the row's model_role, with no new event type. A missing object is a skip in the asset metadata, not a failure.

Caveats:

  • A manual promotion can change the served model part of the way through D, so one day's cohort can split across two versions, and one report can hold a row for each. Grades stay per (model_name, model_version, model_role). Never pool them across versions.
  • The sweep scores a candidate only after a manifest names it, so a candidate's online cohort starts the day after it was trained. A new manifest does not rescore reports that already have a score. Only a change of the served key does that. So a candidate scores only the reports that are new or have a new vector after its manifest is published. pool_coverage_by_model shows how much of the pool each key covered.
  • A report first scored after D ends, for example when its vector arrived late, is not in D's rows. served_pool_coverage shows this for the served model. A later score never fills it in.
  • The sweep scores with the vector current at scoring time, so two keys that scored one report at different times did not always read the same vector.
  • A deployment where the sweep is off writes an empty object with coverage 0 and grades nothing.

After deployment, check that inbox_ranking_unseen_head_evaluated holds more than one model key for one scoring day, each with its model_role. Until a model trained with thresholds is served, the events have no threshold properties and the grades have null classification fields.