Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(research): add a position-prior regression check for score questions (#131) (#259)

GeneLab42dbf0f3851a
diff --git a/research/eval/README.md b/research/eval/README.md
index 6470bbda..adf2e7f6 100644
--- a/research/eval/README.md
+++ b/research/eval/README.md
@@ -214,3 +214,107 @@ one 2.14.0. The clamped run differs from the committed file on `ece` and `mean_c
`accuracy`, `macro_f1` and `n` are identical in all three columns by construction: scaling
logits by any positive temperature does not change the argmax. That is why a re-run can settle
the calibration question without reopening the accuracy numbers.
+
+---
+
+## Presentation checks (`presentation_checks.py`)
+
+A label-free regression check for the `score` position prior in #131.
+`laya-multilingual` rarely picks the first-listed `score` level, and the fix is a
+position-balanced retrain. This script says whether a retrained checkpoint removed
+the prior. Every input is fixed in the file (10 short English states written for it),
+so it needs no dataset and no labels.
+
+```bash
+python research/eval/presentation_checks.py --model convaiinnovations/laya --subfolder multilingual
+python research/eval/presentation_checks.py --model ./retrained-checkpoint --out report.json
+```
+
+Exit status: `0` every check passed, `1` a check failed, `2` the harness disagrees with
+`Agent.system_one` by more than `1e-3` (nothing else is trusted then). CPU is the
+default device: fp32 and deterministic, which is what the thresholds were set on.
+
+### The two checks
+
+| check | input | metric | gate |
+|---|---|---|---|
+| `score_slot0_identical` | one `score` question whose K levels all carry the same text; texts `moderate` and `a request`, K = 3, 4, 5 | raw slot-0 marker logit minus the mean over the K slots, averaged over 10 states × 6 configurations | `>= -0.20` |
+| `score_first_slot_permuted` | `Not urgent` / `Soon` / `Work is blocked` in all 6 orders, per state | share of the 60 decisions whose argmax is the first slot | `>= 0.15` |
+
+`score_slot0_identical` is the identical-option control from @AlKor13 in #131. With
+identical texts the rendered options differ only by position and by the `level N:`
+prefix that `render_options` always emits, so a checkpoint without a slot prior has
+no reason to prefer or avoid any slot.
+
+`score_first_slot_permuted` presents every order of the three levels, so each level
+sits in each slot exactly twice per state. A checkpoint whose answer does not depend
+on the order picks the first slot in exactly 1/3 of the decisions, whatever the states
+say; the rate moves only through order dependence.
+
+Both read raw marker logits (before temperature) through `laya_eval.score_cases`,
+and the script first compares that path with `Agent.system_one` on every state.
+
+### Measured on the shipped checkpoints
+
+CPU, fp32, `convaiinnovations/laya@1c5edc1`, laya 0.3.7. Full output, per state and
+per configuration: `research/results/presentation_checks_shipped.json`.
+
+| checkpoint | `score_slot0_identical` (leave-one-out) | `score_first_slot_permuted` (leave-one-out) | verdict |
+|---|---|---|---|
+| `laya` (english) | **+0.664** (+0.520 .. +0.741) | **0.217** (0.204 .. 0.241) | PASS |
+| `laya-multilingual` | **−0.492** (−0.563 .. −0.425) | **0.017** (0.000 .. 0.019) | FAIL |
+
+Parity with `Agent.system_one`: max |Δp| 4.98e-5 (multilingual) and 4.92e-5 (english),
+which is the 4-decimal rounding of `system_one`'s probabilities.
+
+### Thresholds
+
+The gates were set from the leave-one-out ranges above, not tuned to them. Two
+conditions were fixed before the 10-state run:
+
+1. the current multilingual checkpoint fails and the english checkpoint passes in
+ **every** leave-one-out subset, and
+2. the worst leave-one-out value of each checkpoint clears the threshold by at least
+ 0.10 logit (slot 0) and 0.05 (first-slot rate, 3 of 60 decisions).
+
+| check | threshold | multilingual worst → margin | english worst → margin |
+|---|---|---|---|
+| `score_slot0_identical` | −0.20 | −0.425 → 0.225 | +0.520 → 0.720 |
+| `score_first_slot_permuted` | 0.15 | 0.019 → 0.131 | 0.204 → 0.054 |
+
+The tightest margin is the english first-slot rate, at 0.054 against the 0.05 rule.
+An order-invariant checkpoint sits at exactly 0.333 on that check.
+
+### Tests
+
+`research/eval/test_presentation_checks.py` runs offline, with scripted logits in
+place of a checkpoint:
+
+```bash
+python research/eval/test_presentation_checks.py # 69 passed, 0 failed
+```
+
+It pins the fixed inputs and both gates. It checks that the identical-option
+questions render as `level i: <same text>`, and that every level sits in every slot
+exactly twice. It also checks the metric arithmetic by hand, the leave-one-out
+bounds, the one-sided gates, and the exit codes. A scripted slot-0 hole fails both
+checks, and an order-invariant model scores exactly 1/3.
+
+### Limits
+
+* **The gate is one-sided because the english checkpoint is not flat either.** With
+ identical options it prefers the early slots, more strongly as K grows: slot 0 sits
+ +0.10 / +0.41 / +0.85 above the mean at K = 3 / 4 / 5 with `moderate`, and
+ +0.21 / +0.74 / +1.68 with `a request`. At K = 3 with `moderate` it is close to flat,
+ which matches the #131 control. A two-sided "no position effect" gate would fail
+ the english checkpoint, so the check asks the narrower question #131 is about:
+ whether slot 0 is suppressed.
+ (multilingual: −0.75 / −0.52 / −0.25 and −0.58 / −0.47 / −0.37.)
+* Passing is not accuracy. A checkpoint can clear both gates and still rank urgency
+ badly; this checks one known failure, not `score` quality.
+* English only, `score` only, 10 states. The states are short support messages, so a
+ checkpoint's behaviour on long inputs or other languages is not covered here.
+* Thresholds were set on CPU fp32. On CUDA, `Agent` runs the forward pass under
+ reduced-precision autocast and `score_cases` does not. The parity check reports that
+ difference instead of hiding it.
+* New checks are one function each, registered in `CHECKS`.
diff --git a/research/eval/presentation_checks.py b/research/eval/presentation_checks.py
new file mode 100644
index 00000000..5afabca4
--- /dev/null
+++ b/research/eval/presentation_checks.py
@@ -0,0 +1,289 @@
+"""Presentation-sensitivity regression checks for a Laya checkpoint.
+
+Why this exists
+---------------
+``laya-multilingual``'s ``score`` head has a learned prior against the first-listed
+level (#131). The fix is a position-balanced retrain, and this is the check that says
+whether a retrained checkpoint removed the prior. It needs no labelled data: every
+input is fixed in this file, and both checks compare the checkpoint against itself
+under different presentations of the same question.
+
+Checks
+------
+``score_slot0_identical``
+ The identical-option control from @AlKor13 in #131. A ``score`` question whose K
+ levels all carry the same text, so the rendered options differ only by position
+ and the ``level N:`` prefix ``render_options`` always emits. The metric is the raw
+ marker logit of slot 0 minus the mean over the K slots, averaged over every state
+ and every (text, K) configuration. A checkpoint with no slot-0 deficit sits at or
+ above 0; the gate is one-sided (see "Limits" in README.md).
+
+``score_first_slot_permuted``
+ Three real levels presented in all 3! = 6 orders for each state. Every level sits
+ in every slot exactly twice per state, so a checkpoint whose answer does not
+ depend on the order picks the first slot in exactly 1/3 of the decisions. The
+ metric is the observed first-slot rate.
+
+Both metrics read raw marker logits (before temperature) through
+``laya_eval.score_cases``, the same forward pass ``research/scripts/bench_local.py``
+uses. ``parity`` checks that path against ``Agent.system_one`` before anything is
+reported.
+
+Usage
+-----
+ python research/eval/presentation_checks.py --model convaiinnovations/laya
+ python research/eval/presentation_checks.py --model convaiinnovations/laya \\
+ --subfolder multilingual --out multilingual.json
+
+Exit status: 0 every check passed, 1 a check failed, 2 the harness could not be
+trusted (parity with ``Agent.system_one`` above ``PARITY_TOL``).
+"""
+from __future__ import annotations
+
+import argparse
+import itertools
+import json
+import os
+import sys
+import time
+from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple
+
+# Same path handling as laya_eval.py: allow running this file directly.
+_REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+if _REPO_ROOT not in sys.path:
+ sys.path.insert(0, _REPO_ROOT)
+
+from research.eval.laya_eval import score_cases, softmax_t, temperature_for # noqa: E402
+
+# Fixed English states written for this check. Urgency varies on purpose: a slot
+# prior has to show through states whose honest answer is low, middle and high.
+STATES = (
+ "Hi, whenever you get a chance, could you update the billing address on our account "
+ "to 12 Harbour Street? No rush at all.",
+ "Our invoice for March shows two charges for the same seat. Could you look into it "
+ "this week?",
+ "The CSV export has been failing since this morning and our finance team cannot close "
+ "the month until it works.",
+ "Just wanted to say thanks for the onboarding call yesterday. Nothing needed from "
+ "your side.",
+ "We are thinking about adding ten more licences next quarter. Can someone send pricing "
+ "when convenient?",
+ "Login is down for every user in our company. Nobody can reach the dashboard and our "
+ "customers are waiting.",
+ "The mobile app sometimes shows the wrong time zone on reports. It is annoying but we "
+ "can work around it.",
+ "Our API key was revoked by mistake and every production request is returning 401 "
+ "errors right now.",
+ "Payroll runs tomorrow morning and the integration still rejects our employee file "
+ "with a schema error.",
+ "Could you add a dark mode to the admin screens at some point? Several of us would "
+ "like it.",
+)
+INSTRUCTIONS = "How urgent is this request?"
+IDENTICAL_TEXTS = ("moderate", "a request")
+IDENTICAL_KS = (3, 4, 5)
+LEVELS = ("Not urgent", "Soon", "Work is blocked")
+
+# Gates. Set from the shipped checkpoints' leave-one-out ranges (README.md, "Thresholds").
+SLOT0_MIN = -0.20
+FIRST_SLOT_MIN = 0.15
+PARITY_TOL = 1e-3
+
+ScoreFn = Callable[[str, Sequence[Dict[str, Any]]], List[Any]]
+
+
+def identical_question(text: str, k: int) -> Dict[str, Any]:
+ return {"type": "score", "instructions": INSTRUCTIONS, "criteria": [text] * k}
+
+
+def permuted_questions() -> List[Tuple[Tuple[int, ...], Dict[str, Any]]]:
+ """Every order of LEVELS, as (order, question); order[slot] indexes LEVELS."""
+ return [(order, {"type": "score", "instructions": INSTRUCTIONS,
+ "criteria": [LEVELS[i] for i in order]})
+ for order in itertools.permutations(range(len(LEVELS)))]
+
+
+def mean(xs: Sequence[float]) -> float:
+ return sum(xs) / len(xs)
+
+
+def leave_one_out(per_state: Sequence[float]) -> Tuple[float, float]:
+ """(min, max) of the metric recomputed with each state left out in turn."""
+ n = len(per_state)
+ total = sum(per_state)
+ loo = [(total - x) / (n - 1) for x in per_state]
+ return min(loo), max(loo)
+
+
+def passes(value: float, threshold: float) -> bool:
+ return value >= threshold
+
+
+def check_score_slot0_identical(score_fn: ScoreFn, states: Sequence[str] = STATES) -> Dict[str, Any]:
+ configs = [(text, k) for text in IDENTICAL_TEXTS for k in IDENTICAL_KS]
+ questions = [identical_question(text, k) for text, k in configs]
+ per_state: List[float] = []
+ per_config: Dict[str, List[float]] = {"%s/K=%d" % c: [] for c in configs}
+ per_slot: Dict[str, List[float]] = {"%s/K=%d" % (t, k): [0.0] * k for t, k in configs}
+ for state in states:
+ slot0 = []
+ for (text, k), z in zip(configs, score_fn(state, questions)):
+ name = "%s/K=%d" % (text, k)
+ z = [float(v) for v in z] # numpy float32 would survive into the JSON report
+ centred = [v - mean(z) for v in z]
+ slot0.append(centred[0])
+ per_config[name].append(centred[0])
+ per_slot[name] = [a + b / len(states) for a, b in zip(per_slot[name], centred)]
+ per_state.append(mean(slot0))
+ metric = mean(per_state)
+ lo, hi = leave_one_out(per_state)
+ return {
+ "metric": round(metric, 4),
+ "threshold": SLOT0_MIN,
+ "passed": passes(metric, SLOT0_MIN),
+ "leave_one_out": [round(lo, 4), round(hi, 4)],
+ "per_state": [round(x, 4) for x in per_state],
+ "per_config": {name: round(mean(v), 4) for name, v in per_config.items()},
+ "per_slot_centred": {name: [round(x, 4) for x in v] for name, v in per_slot.items()},
+ }
+
+
+def check_score_first_slot_permuted(score_fn: ScoreFn, states: Sequence[str] = STATES) -> Dict[str, Any]:
+ design = permuted_questions()
+ questions = [q for _order, q in design]
+ by_slot = [0] * len(LEVELS)
+ by_label = {label: 0 for label in LEVELS}
+ per_state: List[float] = []
+ order_invariant = 0
+ for state in states:
+ picked = []
+ for (order, _q), z in zip(design, score_fn(state, questions)):
+ slot = max(range(len(z)), key=lambda i: float(z[i]))
+ by_slot[slot] += 1
+ by_label[LEVELS[order[slot]]] += 1
+ picked.append((slot, LEVELS[order[slot]]))
+ per_state.append(sum(1 for slot, _ in picked if slot == 0) / len(design))
+ order_invariant += len({label for _, label in picked}) == 1
+ metric = mean(per_state)
+ lo, hi = leave_one_out(per_state)
+ return {
+ "metric": round(metric, 4),
+ "threshold": FIRST_SLOT_MIN,
+ "passed": passes(metric, FIRST_SLOT_MIN),
+ "leave_one_out": [round(lo, 4), round(hi, 4)],
+ "decisions": len(states) * len(design),
+ "per_state": [round(x, 4) for x in per_state],
+ "argmax_by_slot": by_slot,
+ "picks_by_label": by_label,
+ "order_invariant_states": order_invariant,
+ }
+
+
+CHECKS: Dict[str, Callable[..., Dict[str, Any]]] = {
+ "score_slot0_identical": check_score_slot0_identical,
+ "score_first_slot_permuted": check_score_first_slot_permuted,
+}
+
+
+def run_checks(score_fn: ScoreFn, names: Optional[Sequence[str]] = None,
+ states: Sequence[str] = STATES) -> Dict[str, Any]:
+ names = list(names or CHECKS)
+ results = {name: CHECKS[name](score_fn, states) for name in names}
+ return {"checks": results, "passed": all(r["passed"] for r in results.values())}
+
+
+def exit_code(report: Dict[str, Any], parity: float) -> int:
+ if not parity <= PARITY_TOL:
+ return 2
+ return 0 if report["passed"] else 1
+
+
+def agent_score_fn(agent) -> ScoreFn:
+ """Raw marker logits for several questions on one state, one forward pass."""
+ def score(state, questions):
+ return score_cases(agent, [(state, {str(i): q for i, q in enumerate(questions)})])
+ return score
+
+
+def parity(agent, states: Sequence[str] = STATES) -> float:
+ """Max |p| difference between this harness and Agent.system_one.
+
+ system_one rounds probabilities to 4 decimals, so agreement shows up as <= 5e-5.
+ """
+ from laya.common import QTYPES
+
+ questions = {"identical": identical_question(IDENTICAL_TEXTS[0], 3),
+ "levels": {"type": "score", "instructions": INSTRUCTIONS, "criteria": list(LEVELS)}}
+ score = agent_score_fn(agent)
+ worst = 0.0
+ for state in states:
+ public = agent.system_one(state, questions)["answers"]
+ for (qid, _q), z in zip(questions.items(), score(state, list(questions.values()))):
+ p = softmax_t(z, temperature_for(agent, QTYPES["score"], len(z)))
+ got = [public[qid]["probabilities"][str(i)] for i in range(len(z))]
+ worst = max(worst, max(abs(a - float(b)) for a, b in zip(got, p)))
+ return worst
+
+
+def main(argv: Optional[Sequence[str]] = None) -> int:
+ parser = argparse.ArgumentParser(
+ prog="presentation-checks",
+ description="Label-free presentation-sensitivity checks for a Laya checkpoint.")
+ parser.add_argument("--model", default="convaiinnovations/laya",
+ help="checkpoint repo id or local path")
+ parser.add_argument("--subfolder", default=None,
+ help="checkpoint subfolder, e.g. multilingual")
+ parser.add_argument("--device", default="cpu",
+ help="default cpu: fp32 and deterministic, which is what the thresholds were set on")
+ parser.add_argument("--checks", default=",".join(CHECKS),
+ help="comma-separated subset of: %s" % ", ".join(CHECKS))
+ parser.add_argument("--out", default=None, help="write the JSON report here")
+ args = parser.parse_args(argv)
+
+ names = [x.strip() for x in args.checks.split(",") if x.strip()]
+ unknown = [x for x in names if x not in CHECKS]
+ if unknown:
+ print("unknown checks: %s" % ", ".join(unknown), file=sys.stderr)
+ return 2
+
+ import laya
+
+ started = time.time()
+ agent = laya.load(args.model, device=args.device, subfolder=args.subfolder)
+ agent.model.eval()
+ worst = parity(agent)
+ report = run_checks(agent_score_fn(agent), names)
+ payload = {
+ "config": {
+ "model": args.model,
+ "subfolder": args.subfolder,
+ "device": str(agent.device),
+ "states": len(STATES),
+ "instructions": INSTRUCTIONS,
+ "identical_texts": list(IDENTICAL_TEXTS),
+ "identical_ks": list(IDENTICAL_KS),
+ "levels": list(LEVELS),
+ "laya_version": getattr(laya, "__version__", "unknown"),
+ },
+ "parity_max_abs_diff": worst,
+ **report,
+ "seconds": round(time.time() - started, 1),
+ }
+
+ for name, r in report["checks"].items():
+ print(" %-26s %8.4f >= %5.2f %s (leave-one-out %.4f .. %.4f)"
+ % (name, r["metric"], r["threshold"], "PASS" if r["passed"] else "FAIL",
+ r["leave_one_out"][0], r["leave_one_out"][1]))
+ print(" parity vs Agent.system_one: max |dp| %.2e (tolerance %.0e)" % (worst, PARITY_TOL))
+ code = exit_code(report, worst)
+ print(" verdict: %s" % {0: "PASS", 1: "FAIL", 2: "HARNESS MISMATCH"}[code])
+
+ if args.out:
+ with open(args.out, "w", encoding="utf-8") as fh:
+ json.dump(payload, fh, indent=1, ensure_ascii=False)
+ fh.write("\n")
+ return code
+
+
+if __name__ == "__main__": # pragma: no cover
+ sys.exit(main())
diff --git a/research/eval/test_presentation_checks.py b/research/eval/test_presentation_checks.py
new file mode 100644
index 00000000..d376da55
--- /dev/null
+++ b/research/eval/test_presentation_checks.py
@@ -0,0 +1,168 @@
+"""Offline tests for the presentation checks.
+
+No checkpoint and no network: the checks take a scoring function, and every test
+here passes a scripted one. The model-dependent paths (`agent_score_fn`, `parity`)
+run only from the CLI against a real checkpoint.
+
+Run: python research/eval/test_presentation_checks.py
+"""
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
+
+from laya.common import render_options # noqa: E402
+from research.eval.presentation_checks import ( # noqa: E402
+ CHECKS, FIRST_SLOT_MIN, IDENTICAL_KS, IDENTICAL_TEXTS, INSTRUCTIONS, LEVELS, PARITY_TOL,
+ SLOT0_MIN, STATES, check_score_first_slot_permuted, check_score_slot0_identical, exit_code,
+ identical_question, leave_one_out, passes, permuted_questions, run_checks,
+)
+
+PASS, FAIL = [], []
+
+
+def check(name, got, want):
+ if got == want:
+ PASS.append(name)
+ else:
+ FAIL.append("%s:\n got %r\n want %r" % (name, got, want))
+
+
+def check_true(name, cond, detail=""):
+ if cond:
+ PASS.append(name)
+ else:
+ FAIL.append("%s %s" % (name, detail))
+
+
+def scripted(position_bias, label_score=None):
+ """A scoring function: logit = position_bias[slot] + label_score[option text]."""
+ label_score = label_score or {}
+
+ def score(_state, questions):
+ return [[position_bias[i] + label_score.get(text, 0.0) for i, text in enumerate(q["criteria"])]
+ for q in questions]
+ return score
+
+
+FLAT = scripted([0.0] * 5)
+NO_SLOT0 = scripted([-2.0, 0.5, 0.5, 0.5, 0.5], {"Not urgent": 0.0, "Soon": 0.3, "Work is blocked": 0.6})
+EARLY = scripted([1.5, 0.8, 0.0, -0.8, -1.5]) # english-like: prefers early slots
+BY_LABEL = scripted([0.0] * 5, {"Not urgent": 0.0, "Soon": 1.0, "Work is blocked": 2.0})
+
+
+# ------------------------------------------------------------------- the inputs
+check("states/ten fixed states", len(STATES), 10)
+check("states/no duplicates", len(set(STATES)), len(STATES))
+check_true("states/plain ASCII", all(s.isascii() for s in STATES))
+check("const/instructions", INSTRUCTIONS, "How urgent is this request?")
+check("const/identical texts", IDENTICAL_TEXTS, ("moderate", "a request"))
+check("const/identical K", IDENTICAL_KS, (3, 4, 5))
+check("const/levels", LEVELS, ("Not urgent", "Soon", "Work is blocked"))
+check("const/slot-0 gate", SLOT0_MIN, -0.20)
+check("const/first-slot gate", FIRST_SLOT_MIN, 0.15)
+check("const/parity tolerance", PARITY_TOL, 1e-3)
+check("const/registered checks", sorted(CHECKS), ["score_first_slot_permuted", "score_slot0_identical"])
+
+
+# ------------------------------------------- identical options differ by position only
+for text in IDENTICAL_TEXTS:
+ for k in IDENTICAL_KS:
+ q = identical_question(text, k)
+ rendered = render_options({"t": q["type"], "ins": q["instructions"], "crit": q["criteria"]})
+ check("identical/%s K=%d renders level-indexed" % (text, k),
+ rendered, ["level %d: %s" % (i, text) for i in range(k)])
+ check("identical/%s K=%d one text after the prefix" % (text, k),
+ {r.split(": ", 1)[1] for r in rendered}, {text})
+
+
+# ------------------------------------------------------- the all-permutation design
+design = permuted_questions()
+check("perm/3! orders", len(design), 6)
+check("perm/orders are distinct", len({order for order, _ in design}), 6)
+for slot in range(len(LEVELS)):
+ for li, label in enumerate(LEVELS):
+ check("perm/%s in slot %d twice" % (label, slot),
+ sum(1 for order, _ in design if order[slot] == li), 2)
+check_true("perm/criteria follow the order",
+ all(q["criteria"] == [LEVELS[i] for i in order] for order, q in design))
+
+
+# ------------------------------------------------------------ slot-0, identical options
+r = check_score_slot0_identical(FLAT)
+check("slot0/flat logits sit at zero", r["metric"], 0.0)
+check("slot0/flat passes", r["passed"], True)
+
+r = check_score_slot0_identical(NO_SLOT0)
+check_true("slot0/suppressed slot 0 is negative", r["metric"] < SLOT0_MIN, str(r["metric"]))
+check("slot0/suppressed slot 0 fails", r["passed"], False)
+# centred slot 0 = -2.0 - mean(bias[:k]); K=3: -5/3, K=4: -1.875, K=5: -2.0, same for both texts
+check("slot0/suppressed metric by hand", r["metric"], round((-5 / 3 - 1.875 - 2.0) / 3, 4))
+check("slot0/per-slot means are centred",
+ all(abs(sum(v)) < 1e-3 for v in r["per_slot_centred"].values()), True)
+
+r = check_score_slot0_identical(EARLY)
+check_true("slot0/early-slot preference is positive", r["metric"] > 0, str(r["metric"]))
+check("slot0/one-sided: early-slot preference passes", r["passed"], True)
+
+check("slot0/one row per state", len(r["per_state"]), len(STATES))
+check("slot0/one entry per (text, K)", len(r["per_config"]), len(IDENTICAL_TEXTS) * len(IDENTICAL_KS))
+check("slot0/metric is the mean of per_state",
+ round(sum(r["per_state"]) / len(r["per_state"]), 4), r["metric"])
+
+
+# ------------------------------------------------------------ first slot, permuted
+r = check_score_first_slot_permuted(BY_LABEL)
+check("first/order-invariant model picks slot 0 in exactly 1/3", r["metric"], round(1 / 3, 4))
+check("first/order-invariant passes", r["passed"], True)
+check("first/every state order-invariant", r["order_invariant_states"], len(STATES))
+check("first/picks follow the label", r["picks_by_label"],
+ {"Not urgent": 0, "Soon": 0, "Work is blocked": 6 * len(STATES)})
+check("first/decisions", r["decisions"], 6 * len(STATES))
+
+r = check_score_first_slot_permuted(NO_SLOT0)
+check("first/slot-0 hole gives zero", r["metric"], 0.0)
+check("first/slot-0 hole fails", r["passed"], False)
+check("first/argmax never lands in slot 0", r["argmax_by_slot"][0], 0)
+check_true("first/order dependence is visible", r["order_invariant_states"] == 0,
+ str(r["order_invariant_states"]))
+
+r = check_score_first_slot_permuted(EARLY)
+check("first/one-sided: always slot 0 passes", (r["metric"], r["passed"]), (1.0, True))
+
+
+# ------------------------------------------------------------ leave-one-out and gates
+check("loo/by hand", leave_one_out([1.0, 2.0, 3.0, 4.0]), (2.0, 3.0))
+check("loo/constant", leave_one_out([0.5, 0.5, 0.5]), (0.5, 0.5))
+r = check_score_slot0_identical(NO_SLOT0)
+check_true("loo/brackets the metric",
+ r["leave_one_out"][0] <= r["metric"] <= r["leave_one_out"][1], str(r))
+check("gate/at the threshold passes", passes(SLOT0_MIN, SLOT0_MIN), True)
+check("gate/just below fails", passes(SLOT0_MIN - 1e-6, SLOT0_MIN), False)
+
+
+# ------------------------------------------------------------------- run and exit
+calls = []
+
+
+def counting(state, questions):
+ calls.append(len(questions))
+ return FLAT(state, questions)
+
+
+run_checks(counting)
+check("run/one forward per state per check", len(calls), 2 * len(STATES))
+check("run/passes only when every check does", run_checks(BY_LABEL)["passed"], True)
+check("run/one failing check fails the run", run_checks(NO_SLOT0)["passed"], False)
+check("run/subset by name", sorted(run_checks(FLAT, ["score_slot0_identical"])["checks"]),
+ ["score_slot0_identical"])
+check("exit/pass", exit_code({"passed": True}, 4.9e-5), 0)
+check("exit/fail", exit_code({"passed": False}, 4.9e-5), 1)
+check("exit/parity outranks the verdict", exit_code({"passed": True}, 2e-3), 2)
+check("exit/nan parity is a mismatch", exit_code({"passed": True}, float("nan")), 2)
+
+
+print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
+for f in FAIL:
+ print(" FAIL " + f)
+sys.exit(1 if FAIL else 0)
diff --git a/research/results/presentation_checks_shipped.json b/research/results/presentation_checks_shipped.json
new file mode 100644
index 00000000..da98ff83
--- /dev/null
+++ b/research/results/presentation_checks_shipped.json
@@ -0,0 +1,121 @@
+{
+ "about": "research/eval/presentation_checks.py on the shipped checkpoints; the thresholds in that file were set from the leave_one_out ranges below",
+ "revision": "convaiinnovations/laya@1c5edc17a7acd8701df6fc341c0d179f1c62c982",
+ "environment": {
+ "device": "cpu",
+ "dtype": "float32",
+ "torch": "2.11.0",
+ "transformers": "5.17.0",
+ "python": "3.11.9"
+ },
+ "english": {
+ "config": {
+ "model": "convaiinnovations/laya",
+ "subfolder": null,
+ "device": "cpu",
+ "states": 10,
+ "instructions": "How urgent is this request?",
+ "identical_texts": ["moderate", "a request"],
+ "identical_ks": [3, 4, 5],
+ "levels": ["Not urgent", "Soon", "Work is blocked"],
+ "laya_version": "0.3.7"
+ },
+ "parity_max_abs_diff": 4.9224502247733476e-05,
+ "checks": {
+ "score_slot0_identical": {
+ "metric": 0.6638,
+ "threshold": -0.2,
+ "passed": true,
+ "leave_one_out": [0.5203, 0.7405],
+ "per_state": [0.5923, 0.656, 0.222, 1.9554, -0.0263, 0.4189, 1.0751, 0.7183, 0.828, 0.1986],
+ "per_config": {
+ "moderate/K=3": 0.0981,
+ "moderate/K=4": 0.4054,
+ "moderate/K=5": 0.8496,
+ "a request/K=3": 0.2079,
+ "a request/K=4": 0.7411,
+ "a request/K=5": 1.6808
+ },
+ "per_slot_centred": {
+ "moderate/K=3": [0.0981, 0.0789, -0.1769],
+ "moderate/K=4": [0.4054, 0.284, -0.2765, -0.4129],
+ "moderate/K=5": [0.8496, 0.6907, 0.0293, -0.5735, -0.9963],
+ "a request/K=3": [0.2079, -0.2148, 0.0069],
+ "a request/K=4": [0.7411, 0.6621, -0.73, -0.6732],
+ "a request/K=5": [1.6808, 1.4661, 0.1138, -1.3483, -1.9125]
+ }
+ },
+ "score_first_slot_permuted": {
+ "metric": 0.2167,
+ "threshold": 0.15,
+ "passed": true,
+ "leave_one_out": [0.2037, 0.2407],
+ "decisions": 60,
+ "per_state": [0.3333, 0.0, 0.3333, 0.3333, 0.3333, 0.3333, 0.1667, 0.3333, 0.0, 0.0],
+ "argmax_by_slot": [13, 18, 29],
+ "picks_by_label": {
+ "Not urgent": 34,
+ "Soon": 6,
+ "Work is blocked": 20
+ },
+ "order_invariant_states": 5
+ }
+ },
+ "passed": true
+ },
+ "multilingual": {
+ "config": {
+ "model": "convaiinnovations/laya",
+ "subfolder": "multilingual",
+ "device": "cpu",
+ "states": 10,
+ "instructions": "How urgent is this request?",
+ "identical_texts": ["moderate", "a request"],
+ "identical_ks": [3, 4, 5],
+ "levels": ["Not urgent", "Soon", "Work is blocked"],
+ "laya_version": "0.3.7"
+ },
+ "parity_max_abs_diff": 4.97927037618473e-05,
+ "checks": {
+ "score_slot0_identical": {
+ "metric": -0.4917,
+ "threshold": -0.2,
+ "passed": false,
+ "leave_one_out": [-0.563, -0.4253],
+ "per_state": [-0.2235, -0.434, -0.6344, -0.2102, -1.0894, -0.9533, 0.1498, -0.2656, -0.4718, -0.7846],
+ "per_config": {
+ "moderate/K=3": -0.7506,
+ "moderate/K=4": -0.5224,
+ "moderate/K=5": -0.2548,
+ "a request/K=3": -0.5806,
+ "a request/K=4": -0.4701,
+ "a request/K=5": -0.3717
+ },
+ "per_slot_centred": {
+ "moderate/K=3": [-0.7506, 0.7242, 0.0263],
+ "moderate/K=4": [-0.5224, 1.0311, 0.151, -0.6598],
+ "moderate/K=5": [-0.2548, 1.0529, 0.3262, -0.4503, -0.6739],
+ "a request/K=3": [-0.5806, 0.5528, 0.0278],
+ "a request/K=4": [-0.4701, 0.5605, 0.1682, -0.2586],
+ "a request/K=5": [-0.3717, 0.4884, 0.2081, -0.0723, -0.2525]
+ }
+ },
+ "score_first_slot_permuted": {
+ "metric": 0.0167,
+ "threshold": 0.15,
+ "passed": false,
+ "leave_one_out": [0.0, 0.0185],
+ "decisions": 60,
+ "per_state": [0.0, 0.0, 0.0, 0.1667, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
+ "argmax_by_slot": [1, 28, 31],
+ "picks_by_label": {
+ "Not urgent": 19,
+ "Soon": 11,
+ "Work is blocked": 30
+ },
+ "order_invariant_states": 0
+ }
+ },
+ "passed": false
+ }
+}