Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(research): add a position-prior regression check for score questions (#131) (#259)
GeneLab42dbf0f3851a
diff --git a/research/eval/README.md b/research/eval/README.mdindex 6470bbda..adf2e7f6 100644--- a/research/eval/README.md+++ b/research/eval/README.md@@ -214,3 +214,107 @@ one 2.14.0. The clamped run differs from the committed file on `ece` and `mean_c`accuracy`, `macro_f1` and `n` are identical in all three columns by construction: scalinglogits by any positive temperature does not change the argmax. That is why a re-run can settlethe calibration question without reopening the accuracy numbers.++---++## Presentation checks (`presentation_checks.py`)++A label-free regression check for the `score` position prior in #131.+`laya-multilingual` rarely picks the first-listed `score` level, and the fix is a+position-balanced retrain. This script says whether a retrained checkpoint removed+the prior. Every input is fixed in the file (10 short English states written for it),+so it needs no dataset and no labels.++```bash+python research/eval/presentation_checks.py --model convaiinnovations/laya --subfolder multilingual+python research/eval/presentation_checks.py --model ./retrained-checkpoint --out report.json+```++Exit status: `0` every check passed, `1` a check failed, `2` the harness disagrees with+`Agent.system_one` by more than `1e-3` (nothing else is trusted then). CPU is the+default device: fp32 and deterministic, which is what the thresholds were set on.++### The two checks++| check | input | metric | gate |+|---|---|---|---|+| `score_slot0_identical` | one `score` question whose K levels all carry the same text; texts `moderate` and `a request`, K = 3, 4, 5 | raw slot-0 marker logit minus the mean over the K slots, averaged over 10 states × 6 configurations | `>= -0.20` |+| `score_first_slot_permuted` | `Not urgent` / `Soon` / `Work is blocked` in all 6 orders, per state | share of the 60 decisions whose argmax is the first slot | `>= 0.15` |++`score_slot0_identical` is the identical-option control from @AlKor13 in #131. With+identical texts the rendered options differ only by position and by the `level N:`+prefix that `render_options` always emits, so a checkpoint without a slot prior has+no reason to prefer or avoid any slot.++`score_first_slot_permuted` presents every order of the three levels, so each level+sits in each slot exactly twice per state. A checkpoint whose answer does not depend+on the order picks the first slot in exactly 1/3 of the decisions, whatever the states+say; the rate moves only through order dependence.++Both read raw marker logits (before temperature) through `laya_eval.score_cases`,+and the script first compares that path with `Agent.system_one` on every state.++### Measured on the shipped checkpoints++CPU, fp32, `convaiinnovations/laya@1c5edc1`, laya 0.3.7. Full output, per state and+per configuration: `research/results/presentation_checks_shipped.json`.++| checkpoint | `score_slot0_identical` (leave-one-out) | `score_first_slot_permuted` (leave-one-out) | verdict |+|---|---|---|---|+| `laya` (english) | **+0.664** (+0.520 .. +0.741) | **0.217** (0.204 .. 0.241) | PASS |+| `laya-multilingual` | **−0.492** (−0.563 .. −0.425) | **0.017** (0.000 .. 0.019) | FAIL |++Parity with `Agent.system_one`: max |Δp| 4.98e-5 (multilingual) and 4.92e-5 (english),+which is the 4-decimal rounding of `system_one`'s probabilities.++### Thresholds++The gates were set from the leave-one-out ranges above, not tuned to them. Two+conditions were fixed before the 10-state run:++1. the current multilingual checkpoint fails and the english checkpoint passes in+ **every** leave-one-out subset, and+2. the worst leave-one-out value of each checkpoint clears the threshold by at least+ 0.10 logit (slot 0) and 0.05 (first-slot rate, 3 of 60 decisions).++| check | threshold | multilingual worst → margin | english worst → margin |+|---|---|---|---|+| `score_slot0_identical` | −0.20 | −0.425 → 0.225 | +0.520 → 0.720 |+| `score_first_slot_permuted` | 0.15 | 0.019 → 0.131 | 0.204 → 0.054 |++The tightest margin is the english first-slot rate, at 0.054 against the 0.05 rule.+An order-invariant checkpoint sits at exactly 0.333 on that check.++### Tests++`research/eval/test_presentation_checks.py` runs offline, with scripted logits in+place of a checkpoint:++```bash+python research/eval/test_presentation_checks.py # 69 passed, 0 failed+```++It pins the fixed inputs and both gates. It checks that the identical-option+questions render as `level i: <same text>`, and that every level sits in every slot+exactly twice. It also checks the metric arithmetic by hand, the leave-one-out+bounds, the one-sided gates, and the exit codes. A scripted slot-0 hole fails both+checks, and an order-invariant model scores exactly 1/3.++### Limits++* **The gate is one-sided because the english checkpoint is not flat either.** With+ identical options it prefers the early slots, more strongly as K grows: slot 0 sits+ +0.10 / +0.41 / +0.85 above the mean at K = 3 / 4 / 5 with `moderate`, and+ +0.21 / +0.74 / +1.68 with `a request`. At K = 3 with `moderate` it is close to flat,+ which matches the #131 control. A two-sided "no position effect" gate would fail+ the english checkpoint, so the check asks the narrower question #131 is about:+ whether slot 0 is suppressed.+ (multilingual: −0.75 / −0.52 / −0.25 and −0.58 / −0.47 / −0.37.)+* Passing is not accuracy. A checkpoint can clear both gates and still rank urgency+ badly; this checks one known failure, not `score` quality.+* English only, `score` only, 10 states. The states are short support messages, so a+ checkpoint's behaviour on long inputs or other languages is not covered here.+* Thresholds were set on CPU fp32. On CUDA, `Agent` runs the forward pass under+ reduced-precision autocast and `score_cases` does not. The parity check reports that+ difference instead of hiding it.+* New checks are one function each, registered in `CHECKS`.diff --git a/research/eval/presentation_checks.py b/research/eval/presentation_checks.pynew file mode 100644index 00000000..5afabca4--- /dev/null+++ b/research/eval/presentation_checks.py@@ -0,0 +1,289 @@+"""Presentation-sensitivity regression checks for a Laya checkpoint.++Why this exists+---------------+``laya-multilingual``'s ``score`` head has a learned prior against the first-listed+level (#131). The fix is a position-balanced retrain, and this is the check that says+whether a retrained checkpoint removed the prior. It needs no labelled data: every+input is fixed in this file, and both checks compare the checkpoint against itself+under different presentations of the same question.++Checks+------+``score_slot0_identical``+ The identical-option control from @AlKor13 in #131. A ``score`` question whose K+ levels all carry the same text, so the rendered options differ only by position+ and the ``level N:`` prefix ``render_options`` always emits. The metric is the raw+ marker logit of slot 0 minus the mean over the K slots, averaged over every state+ and every (text, K) configuration. A checkpoint with no slot-0 deficit sits at or+ above 0; the gate is one-sided (see "Limits" in README.md).++``score_first_slot_permuted``+ Three real levels presented in all 3! = 6 orders for each state. Every level sits+ in every slot exactly twice per state, so a checkpoint whose answer does not+ depend on the order picks the first slot in exactly 1/3 of the decisions. The+ metric is the observed first-slot rate.++Both metrics read raw marker logits (before temperature) through+``laya_eval.score_cases``, the same forward pass ``research/scripts/bench_local.py``+uses. ``parity`` checks that path against ``Agent.system_one`` before anything is+reported.++Usage+-----+ python research/eval/presentation_checks.py --model convaiinnovations/laya+ python research/eval/presentation_checks.py --model convaiinnovations/laya \\+ --subfolder multilingual --out multilingual.json++Exit status: 0 every check passed, 1 a check failed, 2 the harness could not be+trusted (parity with ``Agent.system_one`` above ``PARITY_TOL``).+"""+from __future__ import annotations++import argparse+import itertools+import json+import os+import sys+import time+from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple++# Same path handling as laya_eval.py: allow running this file directly.+_REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+if _REPO_ROOT not in sys.path:+ sys.path.insert(0, _REPO_ROOT)++from research.eval.laya_eval import score_cases, softmax_t, temperature_for # noqa: E402++# Fixed English states written for this check. Urgency varies on purpose: a slot+# prior has to show through states whose honest answer is low, middle and high.+STATES = (+ "Hi, whenever you get a chance, could you update the billing address on our account "+ "to 12 Harbour Street? No rush at all.",+ "Our invoice for March shows two charges for the same seat. Could you look into it "+ "this week?",+ "The CSV export has been failing since this morning and our finance team cannot close "+ "the month until it works.",+ "Just wanted to say thanks for the onboarding call yesterday. Nothing needed from "+ "your side.",+ "We are thinking about adding ten more licences next quarter. Can someone send pricing "+ "when convenient?",+ "Login is down for every user in our company. Nobody can reach the dashboard and our "+ "customers are waiting.",+ "The mobile app sometimes shows the wrong time zone on reports. It is annoying but we "+ "can work around it.",+ "Our API key was revoked by mistake and every production request is returning 401 "+ "errors right now.",+ "Payroll runs tomorrow morning and the integration still rejects our employee file "+ "with a schema error.",+ "Could you add a dark mode to the admin screens at some point? Several of us would "+ "like it.",+)+INSTRUCTIONS = "How urgent is this request?"+IDENTICAL_TEXTS = ("moderate", "a request")+IDENTICAL_KS = (3, 4, 5)+LEVELS = ("Not urgent", "Soon", "Work is blocked")++# Gates. Set from the shipped checkpoints' leave-one-out ranges (README.md, "Thresholds").+SLOT0_MIN = -0.20+FIRST_SLOT_MIN = 0.15+PARITY_TOL = 1e-3++ScoreFn = Callable[[str, Sequence[Dict[str, Any]]], List[Any]]+++def identical_question(text: str, k: int) -> Dict[str, Any]:+ return {"type": "score", "instructions": INSTRUCTIONS, "criteria": [text] * k}+++def permuted_questions() -> List[Tuple[Tuple[int, ...], Dict[str, Any]]]:+ """Every order of LEVELS, as (order, question); order[slot] indexes LEVELS."""+ return [(order, {"type": "score", "instructions": INSTRUCTIONS,+ "criteria": [LEVELS[i] for i in order]})+ for order in itertools.permutations(range(len(LEVELS)))]+++def mean(xs: Sequence[float]) -> float:+ return sum(xs) / len(xs)+++def leave_one_out(per_state: Sequence[float]) -> Tuple[float, float]:+ """(min, max) of the metric recomputed with each state left out in turn."""+ n = len(per_state)+ total = sum(per_state)+ loo = [(total - x) / (n - 1) for x in per_state]+ return min(loo), max(loo)+++def passes(value: float, threshold: float) -> bool:+ return value >= threshold+++def check_score_slot0_identical(score_fn: ScoreFn, states: Sequence[str] = STATES) -> Dict[str, Any]:+ configs = [(text, k) for text in IDENTICAL_TEXTS for k in IDENTICAL_KS]+ questions = [identical_question(text, k) for text, k in configs]+ per_state: List[float] = []+ per_config: Dict[str, List[float]] = {"%s/K=%d" % c: [] for c in configs}+ per_slot: Dict[str, List[float]] = {"%s/K=%d" % (t, k): [0.0] * k for t, k in configs}+ for state in states:+ slot0 = []+ for (text, k), z in zip(configs, score_fn(state, questions)):+ name = "%s/K=%d" % (text, k)+ z = [float(v) for v in z] # numpy float32 would survive into the JSON report+ centred = [v - mean(z) for v in z]+ slot0.append(centred[0])+ per_config[name].append(centred[0])+ per_slot[name] = [a + b / len(states) for a, b in zip(per_slot[name], centred)]+ per_state.append(mean(slot0))+ metric = mean(per_state)+ lo, hi = leave_one_out(per_state)+ return {+ "metric": round(metric, 4),+ "threshold": SLOT0_MIN,+ "passed": passes(metric, SLOT0_MIN),+ "leave_one_out": [round(lo, 4), round(hi, 4)],+ "per_state": [round(x, 4) for x in per_state],+ "per_config": {name: round(mean(v), 4) for name, v in per_config.items()},+ "per_slot_centred": {name: [round(x, 4) for x in v] for name, v in per_slot.items()},+ }+++def check_score_first_slot_permuted(score_fn: ScoreFn, states: Sequence[str] = STATES) -> Dict[str, Any]:+ design = permuted_questions()+ questions = [q for _order, q in design]+ by_slot = [0] * len(LEVELS)+ by_label = {label: 0 for label in LEVELS}+ per_state: List[float] = []+ order_invariant = 0+ for state in states:+ picked = []+ for (order, _q), z in zip(design, score_fn(state, questions)):+ slot = max(range(len(z)), key=lambda i: float(z[i]))+ by_slot[slot] += 1+ by_label[LEVELS[order[slot]]] += 1+ picked.append((slot, LEVELS[order[slot]]))+ per_state.append(sum(1 for slot, _ in picked if slot == 0) / len(design))+ order_invariant += len({label for _, label in picked}) == 1+ metric = mean(per_state)+ lo, hi = leave_one_out(per_state)+ return {+ "metric": round(metric, 4),+ "threshold": FIRST_SLOT_MIN,+ "passed": passes(metric, FIRST_SLOT_MIN),+ "leave_one_out": [round(lo, 4), round(hi, 4)],+ "decisions": len(states) * len(design),+ "per_state": [round(x, 4) for x in per_state],+ "argmax_by_slot": by_slot,+ "picks_by_label": by_label,+ "order_invariant_states": order_invariant,+ }+++CHECKS: Dict[str, Callable[..., Dict[str, Any]]] = {+ "score_slot0_identical": check_score_slot0_identical,+ "score_first_slot_permuted": check_score_first_slot_permuted,+}+++def run_checks(score_fn: ScoreFn, names: Optional[Sequence[str]] = None,+ states: Sequence[str] = STATES) -> Dict[str, Any]:+ names = list(names or CHECKS)+ results = {name: CHECKS[name](score_fn, states) for name in names}+ return {"checks": results, "passed": all(r["passed"] for r in results.values())}+++def exit_code(report: Dict[str, Any], parity: float) -> int:+ if not parity <= PARITY_TOL:+ return 2+ return 0 if report["passed"] else 1+++def agent_score_fn(agent) -> ScoreFn:+ """Raw marker logits for several questions on one state, one forward pass."""+ def score(state, questions):+ return score_cases(agent, [(state, {str(i): q for i, q in enumerate(questions)})])+ return score+++def parity(agent, states: Sequence[str] = STATES) -> float:+ """Max |p| difference between this harness and Agent.system_one.++ system_one rounds probabilities to 4 decimals, so agreement shows up as <= 5e-5.+ """+ from laya.common import QTYPES++ questions = {"identical": identical_question(IDENTICAL_TEXTS[0], 3),+ "levels": {"type": "score", "instructions": INSTRUCTIONS, "criteria": list(LEVELS)}}+ score = agent_score_fn(agent)+ worst = 0.0+ for state in states:+ public = agent.system_one(state, questions)["answers"]+ for (qid, _q), z in zip(questions.items(), score(state, list(questions.values()))):+ p = softmax_t(z, temperature_for(agent, QTYPES["score"], len(z)))+ got = [public[qid]["probabilities"][str(i)] for i in range(len(z))]+ worst = max(worst, max(abs(a - float(b)) for a, b in zip(got, p)))+ return worst+++def main(argv: Optional[Sequence[str]] = None) -> int:+ parser = argparse.ArgumentParser(+ prog="presentation-checks",+ description="Label-free presentation-sensitivity checks for a Laya checkpoint.")+ parser.add_argument("--model", default="convaiinnovations/laya",+ help="checkpoint repo id or local path")+ parser.add_argument("--subfolder", default=None,+ help="checkpoint subfolder, e.g. multilingual")+ parser.add_argument("--device", default="cpu",+ help="default cpu: fp32 and deterministic, which is what the thresholds were set on")+ parser.add_argument("--checks", default=",".join(CHECKS),+ help="comma-separated subset of: %s" % ", ".join(CHECKS))+ parser.add_argument("--out", default=None, help="write the JSON report here")+ args = parser.parse_args(argv)++ names = [x.strip() for x in args.checks.split(",") if x.strip()]+ unknown = [x for x in names if x not in CHECKS]+ if unknown:+ print("unknown checks: %s" % ", ".join(unknown), file=sys.stderr)+ return 2++ import laya++ started = time.time()+ agent = laya.load(args.model, device=args.device, subfolder=args.subfolder)+ agent.model.eval()+ worst = parity(agent)+ report = run_checks(agent_score_fn(agent), names)+ payload = {+ "config": {+ "model": args.model,+ "subfolder": args.subfolder,+ "device": str(agent.device),+ "states": len(STATES),+ "instructions": INSTRUCTIONS,+ "identical_texts": list(IDENTICAL_TEXTS),+ "identical_ks": list(IDENTICAL_KS),+ "levels": list(LEVELS),+ "laya_version": getattr(laya, "__version__", "unknown"),+ },+ "parity_max_abs_diff": worst,+ **report,+ "seconds": round(time.time() - started, 1),+ }++ for name, r in report["checks"].items():+ print(" %-26s %8.4f >= %5.2f %s (leave-one-out %.4f .. %.4f)"+ % (name, r["metric"], r["threshold"], "PASS" if r["passed"] else "FAIL",+ r["leave_one_out"][0], r["leave_one_out"][1]))+ print(" parity vs Agent.system_one: max |dp| %.2e (tolerance %.0e)" % (worst, PARITY_TOL))+ code = exit_code(report, worst)+ print(" verdict: %s" % {0: "PASS", 1: "FAIL", 2: "HARNESS MISMATCH"}[code])++ if args.out:+ with open(args.out, "w", encoding="utf-8") as fh:+ json.dump(payload, fh, indent=1, ensure_ascii=False)+ fh.write("\n")+ return code+++if __name__ == "__main__": # pragma: no cover+ sys.exit(main())diff --git a/research/eval/test_presentation_checks.py b/research/eval/test_presentation_checks.pynew file mode 100644index 00000000..d376da55--- /dev/null+++ b/research/eval/test_presentation_checks.py@@ -0,0 +1,168 @@+"""Offline tests for the presentation checks.++No checkpoint and no network: the checks take a scoring function, and every test+here passes a scripted one. The model-dependent paths (`agent_score_fn`, `parity`)+run only from the CLI against a real checkpoint.++Run: python research/eval/test_presentation_checks.py+"""+import os+import sys++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))++from laya.common import render_options # noqa: E402+from research.eval.presentation_checks import ( # noqa: E402+ CHECKS, FIRST_SLOT_MIN, IDENTICAL_KS, IDENTICAL_TEXTS, INSTRUCTIONS, LEVELS, PARITY_TOL,+ SLOT0_MIN, STATES, check_score_first_slot_permuted, check_score_slot0_identical, exit_code,+ identical_question, leave_one_out, passes, permuted_questions, run_checks,+)++PASS, FAIL = [], []+++def check(name, got, want):+ if got == want:+ PASS.append(name)+ else:+ FAIL.append("%s:\n got %r\n want %r" % (name, got, want))+++def check_true(name, cond, detail=""):+ if cond:+ PASS.append(name)+ else:+ FAIL.append("%s %s" % (name, detail))+++def scripted(position_bias, label_score=None):+ """A scoring function: logit = position_bias[slot] + label_score[option text]."""+ label_score = label_score or {}++ def score(_state, questions):+ return [[position_bias[i] + label_score.get(text, 0.0) for i, text in enumerate(q["criteria"])]+ for q in questions]+ return score+++FLAT = scripted([0.0] * 5)+NO_SLOT0 = scripted([-2.0, 0.5, 0.5, 0.5, 0.5], {"Not urgent": 0.0, "Soon": 0.3, "Work is blocked": 0.6})+EARLY = scripted([1.5, 0.8, 0.0, -0.8, -1.5]) # english-like: prefers early slots+BY_LABEL = scripted([0.0] * 5, {"Not urgent": 0.0, "Soon": 1.0, "Work is blocked": 2.0})+++# ------------------------------------------------------------------- the inputs+check("states/ten fixed states", len(STATES), 10)+check("states/no duplicates", len(set(STATES)), len(STATES))+check_true("states/plain ASCII", all(s.isascii() for s in STATES))+check("const/instructions", INSTRUCTIONS, "How urgent is this request?")+check("const/identical texts", IDENTICAL_TEXTS, ("moderate", "a request"))+check("const/identical K", IDENTICAL_KS, (3, 4, 5))+check("const/levels", LEVELS, ("Not urgent", "Soon", "Work is blocked"))+check("const/slot-0 gate", SLOT0_MIN, -0.20)+check("const/first-slot gate", FIRST_SLOT_MIN, 0.15)+check("const/parity tolerance", PARITY_TOL, 1e-3)+check("const/registered checks", sorted(CHECKS), ["score_first_slot_permuted", "score_slot0_identical"])+++# ------------------------------------------- identical options differ by position only+for text in IDENTICAL_TEXTS:+ for k in IDENTICAL_KS:+ q = identical_question(text, k)+ rendered = render_options({"t": q["type"], "ins": q["instructions"], "crit": q["criteria"]})+ check("identical/%s K=%d renders level-indexed" % (text, k),+ rendered, ["level %d: %s" % (i, text) for i in range(k)])+ check("identical/%s K=%d one text after the prefix" % (text, k),+ {r.split(": ", 1)[1] for r in rendered}, {text})+++# ------------------------------------------------------- the all-permutation design+design = permuted_questions()+check("perm/3! orders", len(design), 6)+check("perm/orders are distinct", len({order for order, _ in design}), 6)+for slot in range(len(LEVELS)):+ for li, label in enumerate(LEVELS):+ check("perm/%s in slot %d twice" % (label, slot),+ sum(1 for order, _ in design if order[slot] == li), 2)+check_true("perm/criteria follow the order",+ all(q["criteria"] == [LEVELS[i] for i in order] for order, q in design))+++# ------------------------------------------------------------ slot-0, identical options+r = check_score_slot0_identical(FLAT)+check("slot0/flat logits sit at zero", r["metric"], 0.0)+check("slot0/flat passes", r["passed"], True)++r = check_score_slot0_identical(NO_SLOT0)+check_true("slot0/suppressed slot 0 is negative", r["metric"] < SLOT0_MIN, str(r["metric"]))+check("slot0/suppressed slot 0 fails", r["passed"], False)+# centred slot 0 = -2.0 - mean(bias[:k]); K=3: -5/3, K=4: -1.875, K=5: -2.0, same for both texts+check("slot0/suppressed metric by hand", r["metric"], round((-5 / 3 - 1.875 - 2.0) / 3, 4))+check("slot0/per-slot means are centred",+ all(abs(sum(v)) < 1e-3 for v in r["per_slot_centred"].values()), True)++r = check_score_slot0_identical(EARLY)+check_true("slot0/early-slot preference is positive", r["metric"] > 0, str(r["metric"]))+check("slot0/one-sided: early-slot preference passes", r["passed"], True)++check("slot0/one row per state", len(r["per_state"]), len(STATES))+check("slot0/one entry per (text, K)", len(r["per_config"]), len(IDENTICAL_TEXTS) * len(IDENTICAL_KS))+check("slot0/metric is the mean of per_state",+ round(sum(r["per_state"]) / len(r["per_state"]), 4), r["metric"])+++# ------------------------------------------------------------ first slot, permuted+r = check_score_first_slot_permuted(BY_LABEL)+check("first/order-invariant model picks slot 0 in exactly 1/3", r["metric"], round(1 / 3, 4))+check("first/order-invariant passes", r["passed"], True)+check("first/every state order-invariant", r["order_invariant_states"], len(STATES))+check("first/picks follow the label", r["picks_by_label"],+ {"Not urgent": 0, "Soon": 0, "Work is blocked": 6 * len(STATES)})+check("first/decisions", r["decisions"], 6 * len(STATES))++r = check_score_first_slot_permuted(NO_SLOT0)+check("first/slot-0 hole gives zero", r["metric"], 0.0)+check("first/slot-0 hole fails", r["passed"], False)+check("first/argmax never lands in slot 0", r["argmax_by_slot"][0], 0)+check_true("first/order dependence is visible", r["order_invariant_states"] == 0,+ str(r["order_invariant_states"]))++r = check_score_first_slot_permuted(EARLY)+check("first/one-sided: always slot 0 passes", (r["metric"], r["passed"]), (1.0, True))+++# ------------------------------------------------------------ leave-one-out and gates+check("loo/by hand", leave_one_out([1.0, 2.0, 3.0, 4.0]), (2.0, 3.0))+check("loo/constant", leave_one_out([0.5, 0.5, 0.5]), (0.5, 0.5))+r = check_score_slot0_identical(NO_SLOT0)+check_true("loo/brackets the metric",+ r["leave_one_out"][0] <= r["metric"] <= r["leave_one_out"][1], str(r))+check("gate/at the threshold passes", passes(SLOT0_MIN, SLOT0_MIN), True)+check("gate/just below fails", passes(SLOT0_MIN - 1e-6, SLOT0_MIN), False)+++# ------------------------------------------------------------------- run and exit+calls = []+++def counting(state, questions):+ calls.append(len(questions))+ return FLAT(state, questions)+++run_checks(counting)+check("run/one forward per state per check", len(calls), 2 * len(STATES))+check("run/passes only when every check does", run_checks(BY_LABEL)["passed"], True)+check("run/one failing check fails the run", run_checks(NO_SLOT0)["passed"], False)+check("run/subset by name", sorted(run_checks(FLAT, ["score_slot0_identical"])["checks"]),+ ["score_slot0_identical"])+check("exit/pass", exit_code({"passed": True}, 4.9e-5), 0)+check("exit/fail", exit_code({"passed": False}, 4.9e-5), 1)+check("exit/parity outranks the verdict", exit_code({"passed": True}, 2e-3), 2)+check("exit/nan parity is a mismatch", exit_code({"passed": True}, float("nan")), 2)+++print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))+for f in FAIL:+ print(" FAIL " + f)+sys.exit(1 if FAIL else 0)diff --git a/research/results/presentation_checks_shipped.json b/research/results/presentation_checks_shipped.jsonnew file mode 100644index 00000000..da98ff83--- /dev/null+++ b/research/results/presentation_checks_shipped.json@@ -0,0 +1,121 @@+{+ "about": "research/eval/presentation_checks.py on the shipped checkpoints; the thresholds in that file were set from the leave_one_out ranges below",+ "revision": "convaiinnovations/laya@1c5edc17a7acd8701df6fc341c0d179f1c62c982",+ "environment": {+ "device": "cpu",+ "dtype": "float32",+ "torch": "2.11.0",+ "transformers": "5.17.0",+ "python": "3.11.9"+ },+ "english": {+ "config": {+ "model": "convaiinnovations/laya",+ "subfolder": null,+ "device": "cpu",+ "states": 10,+ "instructions": "How urgent is this request?",+ "identical_texts": ["moderate", "a request"],+ "identical_ks": [3, 4, 5],+ "levels": ["Not urgent", "Soon", "Work is blocked"],+ "laya_version": "0.3.7"+ },+ "parity_max_abs_diff": 4.9224502247733476e-05,+ "checks": {+ "score_slot0_identical": {+ "metric": 0.6638,+ "threshold": -0.2,+ "passed": true,+ "leave_one_out": [0.5203, 0.7405],+ "per_state": [0.5923, 0.656, 0.222, 1.9554, -0.0263, 0.4189, 1.0751, 0.7183, 0.828, 0.1986],+ "per_config": {+ "moderate/K=3": 0.0981,+ "moderate/K=4": 0.4054,+ "moderate/K=5": 0.8496,+ "a request/K=3": 0.2079,+ "a request/K=4": 0.7411,+ "a request/K=5": 1.6808+ },+ "per_slot_centred": {+ "moderate/K=3": [0.0981, 0.0789, -0.1769],+ "moderate/K=4": [0.4054, 0.284, -0.2765, -0.4129],+ "moderate/K=5": [0.8496, 0.6907, 0.0293, -0.5735, -0.9963],+ "a request/K=3": [0.2079, -0.2148, 0.0069],+ "a request/K=4": [0.7411, 0.6621, -0.73, -0.6732],+ "a request/K=5": [1.6808, 1.4661, 0.1138, -1.3483, -1.9125]+ }+ },+ "score_first_slot_permuted": {+ "metric": 0.2167,+ "threshold": 0.15,+ "passed": true,+ "leave_one_out": [0.2037, 0.2407],+ "decisions": 60,+ "per_state": [0.3333, 0.0, 0.3333, 0.3333, 0.3333, 0.3333, 0.1667, 0.3333, 0.0, 0.0],+ "argmax_by_slot": [13, 18, 29],+ "picks_by_label": {+ "Not urgent": 34,+ "Soon": 6,+ "Work is blocked": 20+ },+ "order_invariant_states": 5+ }+ },+ "passed": true+ },+ "multilingual": {+ "config": {+ "model": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "device": "cpu",+ "states": 10,+ "instructions": "How urgent is this request?",+ "identical_texts": ["moderate", "a request"],+ "identical_ks": [3, 4, 5],+ "levels": ["Not urgent", "Soon", "Work is blocked"],+ "laya_version": "0.3.7"+ },+ "parity_max_abs_diff": 4.97927037618473e-05,+ "checks": {+ "score_slot0_identical": {+ "metric": -0.4917,+ "threshold": -0.2,+ "passed": false,+ "leave_one_out": [-0.563, -0.4253],+ "per_state": [-0.2235, -0.434, -0.6344, -0.2102, -1.0894, -0.9533, 0.1498, -0.2656, -0.4718, -0.7846],+ "per_config": {+ "moderate/K=3": -0.7506,+ "moderate/K=4": -0.5224,+ "moderate/K=5": -0.2548,+ "a request/K=3": -0.5806,+ "a request/K=4": -0.4701,+ "a request/K=5": -0.3717+ },+ "per_slot_centred": {+ "moderate/K=3": [-0.7506, 0.7242, 0.0263],+ "moderate/K=4": [-0.5224, 1.0311, 0.151, -0.6598],+ "moderate/K=5": [-0.2548, 1.0529, 0.3262, -0.4503, -0.6739],+ "a request/K=3": [-0.5806, 0.5528, 0.0278],+ "a request/K=4": [-0.4701, 0.5605, 0.1682, -0.2586],+ "a request/K=5": [-0.3717, 0.4884, 0.2081, -0.0723, -0.2525]+ }+ },+ "score_first_slot_permuted": {+ "metric": 0.0167,+ "threshold": 0.15,+ "passed": false,+ "leave_one_out": [0.0, 0.0185],+ "decisions": 60,+ "per_state": [0.0, 0.0, 0.0, 0.1667, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],+ "argmax_by_slot": [1, 28, 31],+ "picks_by_label": {+ "Not urgent": 19,+ "Soon": 11,+ "Work is blocked": 30+ },+ "order_invariant_states": 0+ }+ },+ "passed": false+ }+}