Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(evals): add an act-head diagnostic over labelled reports
Bruce-Yii5be358d49b7d
diff --git a/research/evals/README.md b/research/evals/README.mdindex 62790bc3..b1be119e 100644--- a/research/evals/README.md+++ b/research/evals/README.md@@ -10,6 +10,14 @@ Datasets, baselines and thresholds for the `laya-evals` harness and its CI gate.baselines in `research/results/`.- `check_regression.py`: adapts a `research/eval/laya_eval.py` report into`laya.evals.EvalReport` and compares it to a committed baseline.+- `act_head_eval.py`: scores `answer.action.act_probability` against labelled+ decisions in a report, as the bar a future act-head retraining has to beat.+ Pure arithmetic over a report `laya-evals run --json` already wrote: no second+ model load, and no change to model, routing or `laya.evals` code. The current+ checkpoints never learned a useful act head, so this measures, it does not+ fix, and a number here is not a claim the signal is informative today.+- `test_act_head_eval.py`: weight-free tests for the above. Like the+ `research/eval/test_*.py` suites, run it directly; CI does not collect it.## Format@@ -39,6 +47,29 @@ Adding a dataset: point `--baseline` at a report you have reviewed, keep the tol`thresholds.json`, and commit both beside the dataset, so a quality change is a reviewablediff.+## Act-head diagnostic++```bash+laya-evals run data.jsonl --model english --json report.json+python research/evals/act_head_eval.py report.json --markdown act.md+```++Answers one question: does `act_probability` rank correct decisions above incorrect ones?+It reports the scorable count, the correct/incorrect split, the tie-correct ROC-AUC of+`act_probability`, the same AUC for the calibrated `confidence` on **exactly** the same+decisions as a control, and the observed min/max/unique of `act_probability` so a signal+pinned at one value is visible rather than averaged away.++Cases are bucketed explicitly and nothing is folded into a denominator: a `score` answer has+no binary `correct` and is skipped with a count, a missing or non-numeric+`act_probability` is reported as missing rather than read as `0.0`, and an AUC is `null`+with a reason when either class is absent, since ROC-AUC is undefined there.++The AUC is the Mann-Whitney U form with mid-ranks for ties, which agrees with+`sklearn.metrics.roc_auc_score` and needs only numpy. Ties are not a rounding detail here:+the current signal is constant, so a tie-correct implementation is the difference between+reporting a failure and reporting 0.5 as if it were a measurement.+## The real labelled setThe maintainer's 396-decision English set and the 51-language sweep are the authoritativediff --git a/research/evals/act_head_eval.py b/research/evals/act_head_eval.pynew file mode 100644index 00000000..a216ab96--- /dev/null+++ b/research/evals/act_head_eval.py@@ -0,0 +1,290 @@+"""Score `act_probability` against labelled decisions, as a bar for a future act head.++`answer.action.act_probability` is what the released checkpoints emit from an act head they+were never trained to use, and #185 records that it sits at ~1.0 on almost every input and+that the raw signal runs *against* correctness. This harness does not try to fix that, and+it does not claim the signal is useful today. Its only job is to make the question+measurable, so that a retrained head has a number to beat:++ **Does `act_probability` rank correct decisions above incorrect ones on a labelled+ evaluation set?**++## Why this reads a report instead of running a model++`laya-evals run --json REPORT` already writes, per case, the full `answer` (which carries+`answer.action.act_probability`), the calibrated `confidence`, and the binary `correct`.+So the diagnostic is pure arithmetic over a report that already exists:++* no checkpoint is loaded and no forward pass is run a second time;+* model, routing and eval code are untouched, and `laya.evals` gains no public API;+* any future checkpoint or retraining run produces an apples-to-apples number, because the+ input format is the one the harness already commits.++## Case selection++Every input case lands in exactly one bucket, and nothing is silently folded into a+denominator:++| bucket | meaning |+|---|---|+| `paired` | usable for both AUCs |+| `no_binary_correctness` | `correct` is not a bool -- a `score` answer, where correctness is not unambiguous |+| `missing_act_probability` | `action.act_probability` absent or not a real number |+| `missing_confidence` | `correct` and `act_probability` fine, but no numeric `confidence` |+| `malformed_case` | the entry is not a case object at all |++The `paired` set is used for **both** the `act_probability` AUC and the `confidence`+control, so the comparator covers exactly the same decisions by construction rather than by+convention. A `score` answer is skipped because `laya.evals` deliberately leaves `correct`+null for it; there is no canonical binary definition for a score, and inventing one here+would be a different experiment.++## Metric++`roc_auc` is the tie-correct ROC-AUC, the Mann-Whitney U form with mid-ranks for tied+scores. It agrees with `sklearn.metrics.roc_auc_score` but needs only numpy, which the+metric math in `laya.evals` already depends on. Concretely:++ AUC = (sum of mid-ranks of the correct cases - n_pos * (n_pos + 1) / 2) / (n_pos * n_neg)++Ties get the average of the ranks they span, so a tied correct case is not credited with+beating a tied incorrect one; with every score tied and both classes present the result is+exactly 0.5.++ROC-AUC is undefined when one class is absent, and this harness reports `null` with a+`auc_null_reason` instead of inventing a number. A constant or near-constant+`act_probability` cannot rank anything, so the summary also reports the observed minimum,+maximum and unique-value count plus a `saturated` flag: a signal pinned at one value is a+failed diagnostic, not a 0.5 result to be averaged away.++ python research/evals/act_head_eval.py report.json [--json out.json] [--markdown out.md]+"""+from __future__ import annotations++import argparse+import json+import sys+from typing import Any, Dict, List, Optional, Sequence++import numpy as np++SCHEMA = "laya-evals-report/act-head-eval/1"++# Below this spread every act_probability is the same number to the precision the report+# stores (4 decimals), so the signal cannot separate anything.+_SATURATION_EPS = 1e-9++_UNSCORABLE = "one class is absent: ROC-AUC is undefined"+_NO_CASES = "no scorable cases"+++def _is_number(value: Any) -> bool:+ """True for a real number. `bool` is an `int` subclass, and is not a measurement."""+ return isinstance(value, (int, float)) and not isinstance(value, bool)+++def roc_auc(scores: Sequence[float], labels: Sequence[bool]) -> Optional[float]:+ """Tie-correct ROC-AUC of ``scores`` against binary ``labels``.++ None when either class is absent, or when there is nothing to score: ROC-AUC needs+ both a positive and a negative case to be defined at all.+ """+ y = np.asarray(labels, dtype=bool)+ s = np.asarray(scores, dtype=float)+ if y.shape != s.shape:+ raise ValueError("scores and labels must be the same length")+ n_pos = int(y.sum())+ n_neg = int(y.shape[0] - n_pos)+ if n_pos == 0 or n_neg == 0:+ return None++ # Mid-ranks over the sorted scores, so a group of tied scores shares the average of+ # the 1-based ranks it spans.+ order = np.argsort(s, kind="mergesort")+ ordered = s[order]+ ranks = np.empty(s.shape[0], dtype=float)+ start = 0+ while start < ordered.shape[0]:+ end = start+ while end + 1 < ordered.shape[0] and ordered[end + 1] == ordered[start]:+ end += 1+ ranks[order[start:end + 1]] = (start + end) / 2.0 + 1.0+ start = end + 1++ rank_sum = float(ranks[y].sum())+ return (rank_sum - n_pos * (n_pos + 1) / 2.0) / (n_pos * n_neg)+++def _case_act_probability(case: Dict[str, Any]) -> Any:+ answer = case.get("answer")+ if not isinstance(answer, dict):+ return None+ action = answer.get("action")+ if not isinstance(action, dict):+ return None+ return action.get("act_probability")+++def analyse_cases(cases: Sequence[Any]) -> Dict[str, Any]:+ """Score one report's cases, bucketing every case exactly once.++ Returns a dict that is JSON-serialisable as-is: counts, both AUCs (or ``None`` with a+ reason), and the spread of ``act_probability`` needed to spot a saturated signal.+ """+ paired: List[Dict[str, Any]] = []+ skipped = {+ "no_binary_correctness": 0,+ "missing_act_probability": 0,+ "missing_confidence": 0,+ "malformed_case": 0,+ }++ for case in cases:+ if not isinstance(case, dict):+ skipped["malformed_case"] += 1+ continue+ correct = case.get("correct")+ if not isinstance(correct, bool):+ # A score answer, or a report without the field: correctness is not binary here.+ skipped["no_binary_correctness"] += 1+ continue+ act = _case_act_probability(case)+ if not _is_number(act):+ # Never coerced to 0.0: a missing act_probability is missing evidence, and+ # reading it as "not acting" would quietly invent a decision.+ skipped["missing_act_probability"] += 1+ continue+ confidence = case.get("confidence")+ if not _is_number(confidence):+ # Dropped from the paired set rather than back-filled, so the act AUC and the+ # confidence control stay on exactly the same decisions.+ skipped["missing_confidence"] += 1+ continue+ paired.append({"act": float(act), "confidence": float(confidence), "correct": correct})++ labels = [p["correct"] for p in paired]+ acts = [p["act"] for p in paired]+ confs = [p["confidence"] for p in paired]++ act_auc = roc_auc(acts, labels) if paired else None+ confidence_auc = roc_auc(confs, labels) if paired else None+ if not paired:+ reason = _NO_CASES+ elif act_auc is None:+ reason = _UNSCORABLE+ else:+ reason = None++ summary: Dict[str, Any] = {+ "schema": SCHEMA,+ "total_cases": len(cases),+ "paired": len(paired),+ "paired_cases": len(paired),+ "correct": sum(1 for p in paired if p["correct"]),+ "incorrect": sum(1 for p in paired if not p["correct"]),+ "act_auc": act_auc,+ "confidence_auc": confidence_auc,+ "auc_null_reason": reason,+ "act_min": min(acts) if acts else None,+ "act_max": max(acts) if acts else None,+ "act_unique": len(set(acts)) if acts else 0,+ }+ summary["skipped"] = skipped+ summary["saturated"] = bool(+ acts and (len(set(acts)) <= 1 or (max(acts) - min(acts)) < _SATURATION_EPS)+ )+ return summary+++def report_from_document(document: Any) -> Dict[str, Any]:+ """Analyse an `EvalReport.to_json()` document.++ Raises ValueError when the document is not a report, so a wrong file is a clear error+ rather than a set of zeros that reads like a result.+ """+ if not isinstance(document, dict):+ raise ValueError("report must be a JSON object")+ cases = document.get("cases")+ if not isinstance(cases, list):+ raise ValueError("report has no 'cases' list: is this a laya-evals --json report?")+ analysis = analyse_cases(cases)+ return {+ "schema": SCHEMA,+ "config": document.get("config") or {},+ "source_overall": document.get("overall") or {},+ "analysis": analysis,+ }+++def format_markdown(result: Dict[str, Any]) -> str:+ """A short human summary of an analysed report."""+ a = result["analysis"]+ lines = [+ "# act-head diagnostic",+ "",+ "Does `act_probability` rank correct decisions above incorrect ones?",+ "",+ "| measure | value |",+ "|---|---|",+ "| cases in report | %d |" % a["total_cases"],+ "| scorable (paired) decisions | %d |" % a["paired"],+ "| correct / incorrect | %d / %d |" % (a["correct"], a["incorrect"]),+ "| ROC-AUC of `act_probability` | %s |" % _fmt(a["act_auc"]),+ "| ROC-AUC of calibrated `confidence` (same cases) | %s |" % _fmt(a["confidence_auc"]),+ "| `act_probability` min / max / unique | %s / %s / %d |"+ % (_fmt(a["act_min"]), _fmt(a["act_max"]), a["act_unique"]),+ "| saturated signal | %s |" % ("yes" if a["saturated"] else "no"),+ ]+ if a["auc_null_reason"]:+ lines.append("")+ lines.append("No AUC reported: %s." % a["auc_null_reason"])+ skipped = a["skipped"]+ if any(skipped.values()):+ lines.append("")+ lines.append("Skipped cases, excluded from every count above:")+ for key in ("no_binary_correctness", "missing_act_probability",+ "missing_confidence", "malformed_case"):+ if skipped[key]:+ lines.append("* `%s`: %d" % (key, skipped[key]))+ lines.extend([+ "",+ "This is a diagnostic only. It does not change `act_probability`, and a score",+ "here is not a claim that the current act head is informative.",+ "",+ ])+ return "\n".join(lines)+++def _fmt(value: Optional[float]) -> str:+ return "n/a" if value is None else "%.4f" % value+++def main(argv: Optional[List[str]] = None) -> int:+ parser = argparse.ArgumentParser(+ description="Score act_probability against labelled decisions in a laya-evals report."+ )+ parser.add_argument("report", help="the JSON written by `laya-evals run --json`")+ parser.add_argument("--json", dest="json_out", help="write the full result here")+ parser.add_argument("--markdown", dest="md_out", help="write a human summary here")+ args = parser.parse_args(argv)++ with open(args.report, "r", encoding="utf-8") as handle:+ document = json.load(handle)+ result = report_from_document(document)++ if args.json_out:+ with open(args.json_out, "w", encoding="utf-8") as handle:+ json.dump(result, handle, indent=2, sort_keys=True)+ handle.write("\n")+ if args.md_out:+ with open(args.md_out, "w", encoding="utf-8") as handle:+ handle.write(format_markdown(result))+ if not args.json_out and not args.md_out:+ print(json.dumps(result, indent=2, sort_keys=True))+ else:+ print(format_markdown(result))+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/research/evals/test_act_head_eval.py b/research/evals/test_act_head_eval.pynew file mode 100644index 00000000..8d982dc7--- /dev/null+++ b/research/evals/test_act_head_eval.py@@ -0,0 +1,228 @@+"""Offline tests for the act-head diagnostic in `research/evals/act_head_eval.py`.++No checkpoint and no network: every function under test is pure arithmetic over a+report that `laya-evals --json` already wrote, so this runs offline.++The fixtures here are tiny and synthetic. They exist to pin the metric mechanics+only -- they are **not** a measurement of the released checkpoints and say nothing+about act-head quality. The real labelled sets are not committed to this repository.++Run: python research/evals/test_act_head_eval.py+"""+import json+import os+import sys++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))++from research.evals.act_head_eval import ( # noqa: E402+ analyse_cases,+ format_markdown,+ roc_auc,+)++PASS, FAIL = [], []+++def check(name, got, want):+ if got == want:+ PASS.append(name)+ else:+ FAIL.append("%s:\n got %r\n want %r" % (name, got, want))+++def check_true(name, cond, detail=""):+ if cond:+ PASS.append(name)+ else:+ FAIL.append("%s%s" % (name, (" -- " + detail) if detail else ""))+++def check_close(name, got, want, tol=1e-9):+ if got is None:+ FAIL.append("%s:\n got None\n want ~%r" % (name, want))+ elif abs(float(got) - want) > tol:+ FAIL.append("%s:\n got %r\n want ~%r" % (name, got, want))+ else:+ PASS.append(name)+++def case(act, correct, confidence=None, qid="q", answer_type="choice", drop_action=False):+ """One `laya-evals` report case, shaped like `EvalReport.to_json()["cases"][i]`."""+ answer = {"type": answer_type}+ if correct is not None and answer_type == "choice":+ answer["choice"] = "a"+ if not drop_action:+ answer["action"] = {} if act is None else {"act_probability": act}+ else:+ answer["action"] = {}+ case_obj = {"qid": qid, "answer": answer, "correct": correct}+ if confidence is not None:+ case_obj["confidence"] = confidence+ return case_obj+++# ------------------------------------------------------------------ roc_auc math+# Perfect separation: every correct case outranks every incorrect one.+check("auc/perfect", roc_auc([0.9, 0.8, 0.2, 0.1], [True, True, False, False]), 1.0)+# Exactly reversed: the score runs against correctness, which is the failure #185 reports.+check("auc/reversed", roc_auc([0.1, 0.2, 0.8, 0.9], [True, True, False, False]), 0.0)+# All scores tied, both classes present: every pairing is a tie, so exactly 0.5.+check("auc/all_tied", roc_auc([1.0, 1.0, 1.0, 1.0], [True, True, False, False]), 0.5)+# Partial ties get mid-ranks, so a tied positive does not beat a tied negative outright.+check_close("auc/partial_tie", roc_auc([0.5, 0.5, 0.9], [True, False, True]), 0.75)+check_close("auc/single_positive", roc_auc([0.3, 0.7], [False, True]), 1.0)+# One class absent: ROC-AUC is undefined, and must not be faked as 0.5 or 0.0.+check("auc/all_positive", roc_auc([0.1, 0.9], [True, True]), None)+check("auc/all_negative", roc_auc([0.1, 0.9], [False, False]), None)+check("auc/empty", roc_auc([], []), None)+# A single tied pair is the smallest well-defined case.+check("auc/one_each_tied", roc_auc([0.7, 0.7], [True, False]), 0.5)++# ------------------------------------------------------------------ case bucketing+# A perfect ranking through the full report path, with the confidence control.+perfect = analyse_cases([+ case(0.99, True, 0.9), case(0.80, True, 0.8),+ case(0.20, False, 0.3), case(0.01, False, 0.1),+])+check("analyse/paired_all", perfect["paired"], 4)+check("analyse/correct", perfect["correct"], 2)+check("analyse/incorrect", perfect["incorrect"], 2)+check("analyse/act_auc", perfect["act_auc"], 1.0)+check("analyse/confidence_auc", perfect["confidence_auc"], 1.0)+check_true("analyse/not_saturated", perfect["saturated"] is False, repr(perfect))+check("analyse/unique", perfect["act_unique"], 4)++# Reversed act_probability, but a confidence that ranks correctly: this is the shape+# of the measurement in #185, where the raw signal runs against correctness.+reversed_act = analyse_cases([+ case(0.10, True, 0.9), case(0.20, True, 0.7),+ case(0.80, False, 0.3), case(0.90, False, 0.1),+])+check("analyse/reversed_act_auc", reversed_act["act_auc"], 0.0)+check("analyse/reversed_confidence_auc", reversed_act["confidence_auc"], 1.0)+check("analyse/reversed_same_case_set", reversed_act["paired"], 4)++# The documented failure mode: act_probability saturated at 1.0 on every input.+saturated = analyse_cases([+ case(1.0, True, 0.9), case(1.0, False, 0.4), case(1.0, True, 0.8), case(1.0, False, 0.2),+])+check("analyse/saturated_detected", saturated["saturated"], True)+check("analyse/saturated_unique", saturated["act_unique"], 1)+check("analyse/saturated_auc", saturated["act_auc"], 0.5)+check("analyse/saturated_min_max", (saturated["act_min"], saturated["act_max"]), (1.0, 1.0))+check_true("analyse/saturated_min_max_json_safe",+ json.dumps(saturated, sort_keys=True) is not None)++# `correct=null` (a score answer) is counted and skipped, never folded into the denominator.+with_nulls = analyse_cases([+ case(0.9, True, 0.8),+ case(0.1, False, 0.2),+ case(0.5, None, 0.6, answer_type="score"),+ case(0.4, None, 0.3, answer_type="score"),+])+check("analyse/null_correctness_skipped", with_nulls["paired"], 2)+check("analyse/null_correctness_counted", with_nulls["skipped"]["no_binary_correctness"], 2)+check("analyse/null_correctness_auc", with_nulls["act_auc"], 1.0)++# Missing act_probability is explicit, never silently read as 0.0.+missing = analyse_cases([+ case(0.9, True, 0.8),+ case(None, False, 0.3, drop_action=True),+ case(0.1, False, 0.2),+])+check("analyse/missing_act_counted", missing["skipped"]["missing_act_probability"], 1)+check("analyse/missing_act_paired", missing["paired"], 2)+check("analyse/missing_act_auc", missing["act_auc"], 1.0)+check_true("analyse/missing_act_min_ignores_missing",+ missing["act_min"] == 0.1 and missing["act_max"] == 0.9, repr(missing))++# A non-numeric act_probability is malformed, not zero.+malformed = analyse_cases([+ case("not-a-number", True, 0.8),+ case(0.2, False, 0.3),+])+check("analyse/malformed_act_counted", malformed["skipped"]["missing_act_probability"], 1)+check("analyse/malformed_act_paired", malformed["paired"], 1)++# A bool in the act_probability slot is malformed, not the integer 1.+boolish = analyse_cases([case(True, True, 0.8), case(0.2, False, 0.3)])+check("analyse/bool_act_counted", boolish["skipped"]["missing_act_probability"], 1)++# Missing confidence drops the case from BOTH AUCs, so the control stays on one set.+no_conf = analyse_cases([+ case(0.9, True, 0.8),+ case(0.1, False),+ case(0.2, True, 0.1),+])+check("analyse/missing_confidence_counted", no_conf["skipped"]["missing_confidence"], 1)+check("analyse/missing_confidence_paired", no_conf["paired"], 2)+check("analyse/missing_confidence_same_set", no_conf["paired_cases"], 2)++# All-correct or all-incorrect cannot produce a fake AUC.+all_correct = analyse_cases([case(0.9, True, 0.8), case(0.4, True, 0.6)])+check("analyse/all_correct_auc", all_correct["act_auc"], None)+check("analyse/all_correct_confidence_auc", all_correct["confidence_auc"], None)+check("analyse/all_correct_null_reason", all_correct["auc_null_reason"],+ "one class is absent: ROC-AUC is undefined")+all_wrong = analyse_cases([case(0.9, False, 0.8), case(0.4, False, 0.6)])+check("analyse/all_wrong_auc", all_wrong["act_auc"], None)++# An empty report is a clean null state, not a crash.+empty = analyse_cases([])+check("analyse/empty_paired", empty["paired"], 0)+check("analyse/empty_auc", empty["act_auc"], None)+check("analyse/empty_reason", empty["auc_null_reason"], "no scorable cases")++# Non-dict / malformed case entries are counted, not fatal.+junk = analyse_cases([None, "nope", 5, case(0.9, True, 0.8)])+check("analyse/junk_counted", junk["skipped"]["malformed_case"], 3)+check("analyse/junk_paired", junk["paired"], 1)++# ------------------------------------------------------------------ report plumbing+REPORT = {+ "config": {"model": "english"},+ "overall": {"accuracy": 0.5},+ "cases": [+ case(0.9, True, 0.8, qid="a"),+ case(0.1, False, 0.2, qid="b"),+ case(0.5, None, 0.6, qid="c", answer_type="score"),+ ],+}+++def _load(payload):+ return json.loads(json.dumps(payload))+++from research.evals.act_head_eval import report_from_document # noqa: E402++loaded = report_from_document(_load(REPORT))+check("report/cases_read", loaded["analysis"]["total_cases"], 3)+check("report/config_kept", loaded["config"], {"model": "english"})+check("report/analyse_paired", loaded["analysis"]["paired"], 2)+check("report/schema", loaded["schema"], "laya-evals-report/act-head-eval/1")++# The same document twice gives byte-identical JSON, so a committed result is a real diff.+first = json.dumps(report_from_document(_load(REPORT)), sort_keys=True)+second = json.dumps(report_from_document(_load(REPORT)), sort_keys=True)+check("report/deterministic", first, second)++# A document without `cases` is rejected clearly instead of silently reporting zeros.+try:+ report_from_document({"overall": {}})+ FAIL.append("report/missing_cases should raise")+except ValueError as exc:+ check_true("report/missing_cases_raises", "cases" in str(exc), str(exc))++markdown = format_markdown(loaded)+check_true("report/markdown_has_question", "act_probability" in markdown, markdown[:200])+check_true("report/markdown_flags_saturation", "saturat" in markdown.lower(), markdown[:400])++print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))+for f in FAIL:+ print(" FAIL " + f)+if not FAIL:+ print("all act-head eval tests passed")+sys.exit(1 if FAIL else 0)