Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(evals): add an act-head diagnostic over labelled reports

Bruce-Yii5be358d49b7d
diff --git a/research/evals/README.md b/research/evals/README.md
index 62790bc3..b1be119e 100644
--- a/research/evals/README.md
+++ b/research/evals/README.md
@@ -10,6 +10,14 @@ Datasets, baselines and thresholds for the `laya-evals` harness and its CI gate.
baselines in `research/results/`.
- `check_regression.py`: adapts a `research/eval/laya_eval.py` report into
`laya.evals.EvalReport` and compares it to a committed baseline.
+- `act_head_eval.py`: scores `answer.action.act_probability` against labelled
+ decisions in a report, as the bar a future act-head retraining has to beat.
+ Pure arithmetic over a report `laya-evals run --json` already wrote: no second
+ model load, and no change to model, routing or `laya.evals` code. The current
+ checkpoints never learned a useful act head, so this measures, it does not
+ fix, and a number here is not a claim the signal is informative today.
+- `test_act_head_eval.py`: weight-free tests for the above. Like the
+ `research/eval/test_*.py` suites, run it directly; CI does not collect it.
## Format
@@ -39,6 +47,29 @@ Adding a dataset: point `--baseline` at a report you have reviewed, keep the tol
`thresholds.json`, and commit both beside the dataset, so a quality change is a reviewable
diff.
+## Act-head diagnostic
+
+```bash
+laya-evals run data.jsonl --model english --json report.json
+python research/evals/act_head_eval.py report.json --markdown act.md
+```
+
+Answers one question: does `act_probability` rank correct decisions above incorrect ones?
+It reports the scorable count, the correct/incorrect split, the tie-correct ROC-AUC of
+`act_probability`, the same AUC for the calibrated `confidence` on **exactly** the same
+decisions as a control, and the observed min/max/unique of `act_probability` so a signal
+pinned at one value is visible rather than averaged away.
+
+Cases are bucketed explicitly and nothing is folded into a denominator: a `score` answer has
+no binary `correct` and is skipped with a count, a missing or non-numeric
+`act_probability` is reported as missing rather than read as `0.0`, and an AUC is `null`
+with a reason when either class is absent, since ROC-AUC is undefined there.
+
+The AUC is the Mann-Whitney U form with mid-ranks for ties, which agrees with
+`sklearn.metrics.roc_auc_score` and needs only numpy. Ties are not a rounding detail here:
+the current signal is constant, so a tie-correct implementation is the difference between
+reporting a failure and reporting 0.5 as if it were a measurement.
+
## The real labelled set
The maintainer's 396-decision English set and the 51-language sweep are the authoritative
diff --git a/research/evals/act_head_eval.py b/research/evals/act_head_eval.py
new file mode 100644
index 00000000..a216ab96
--- /dev/null
+++ b/research/evals/act_head_eval.py
@@ -0,0 +1,290 @@
+"""Score `act_probability` against labelled decisions, as a bar for a future act head.
+
+`answer.action.act_probability` is what the released checkpoints emit from an act head they
+were never trained to use, and #185 records that it sits at ~1.0 on almost every input and
+that the raw signal runs *against* correctness. This harness does not try to fix that, and
+it does not claim the signal is useful today. Its only job is to make the question
+measurable, so that a retrained head has a number to beat:
+
+ **Does `act_probability` rank correct decisions above incorrect ones on a labelled
+ evaluation set?**
+
+## Why this reads a report instead of running a model
+
+`laya-evals run --json REPORT` already writes, per case, the full `answer` (which carries
+`answer.action.act_probability`), the calibrated `confidence`, and the binary `correct`.
+So the diagnostic is pure arithmetic over a report that already exists:
+
+* no checkpoint is loaded and no forward pass is run a second time;
+* model, routing and eval code are untouched, and `laya.evals` gains no public API;
+* any future checkpoint or retraining run produces an apples-to-apples number, because the
+ input format is the one the harness already commits.
+
+## Case selection
+
+Every input case lands in exactly one bucket, and nothing is silently folded into a
+denominator:
+
+| bucket | meaning |
+|---|---|
+| `paired` | usable for both AUCs |
+| `no_binary_correctness` | `correct` is not a bool -- a `score` answer, where correctness is not unambiguous |
+| `missing_act_probability` | `action.act_probability` absent or not a real number |
+| `missing_confidence` | `correct` and `act_probability` fine, but no numeric `confidence` |
+| `malformed_case` | the entry is not a case object at all |
+
+The `paired` set is used for **both** the `act_probability` AUC and the `confidence`
+control, so the comparator covers exactly the same decisions by construction rather than by
+convention. A `score` answer is skipped because `laya.evals` deliberately leaves `correct`
+null for it; there is no canonical binary definition for a score, and inventing one here
+would be a different experiment.
+
+## Metric
+
+`roc_auc` is the tie-correct ROC-AUC, the Mann-Whitney U form with mid-ranks for tied
+scores. It agrees with `sklearn.metrics.roc_auc_score` but needs only numpy, which the
+metric math in `laya.evals` already depends on. Concretely:
+
+ AUC = (sum of mid-ranks of the correct cases - n_pos * (n_pos + 1) / 2) / (n_pos * n_neg)
+
+Ties get the average of the ranks they span, so a tied correct case is not credited with
+beating a tied incorrect one; with every score tied and both classes present the result is
+exactly 0.5.
+
+ROC-AUC is undefined when one class is absent, and this harness reports `null` with a
+`auc_null_reason` instead of inventing a number. A constant or near-constant
+`act_probability` cannot rank anything, so the summary also reports the observed minimum,
+maximum and unique-value count plus a `saturated` flag: a signal pinned at one value is a
+failed diagnostic, not a 0.5 result to be averaged away.
+
+ python research/evals/act_head_eval.py report.json [--json out.json] [--markdown out.md]
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from typing import Any, Dict, List, Optional, Sequence
+
+import numpy as np
+
+SCHEMA = "laya-evals-report/act-head-eval/1"
+
+# Below this spread every act_probability is the same number to the precision the report
+# stores (4 decimals), so the signal cannot separate anything.
+_SATURATION_EPS = 1e-9
+
+_UNSCORABLE = "one class is absent: ROC-AUC is undefined"
+_NO_CASES = "no scorable cases"
+
+
+def _is_number(value: Any) -> bool:
+ """True for a real number. `bool` is an `int` subclass, and is not a measurement."""
+ return isinstance(value, (int, float)) and not isinstance(value, bool)
+
+
+def roc_auc(scores: Sequence[float], labels: Sequence[bool]) -> Optional[float]:
+ """Tie-correct ROC-AUC of ``scores`` against binary ``labels``.
+
+ None when either class is absent, or when there is nothing to score: ROC-AUC needs
+ both a positive and a negative case to be defined at all.
+ """
+ y = np.asarray(labels, dtype=bool)
+ s = np.asarray(scores, dtype=float)
+ if y.shape != s.shape:
+ raise ValueError("scores and labels must be the same length")
+ n_pos = int(y.sum())
+ n_neg = int(y.shape[0] - n_pos)
+ if n_pos == 0 or n_neg == 0:
+ return None
+
+ # Mid-ranks over the sorted scores, so a group of tied scores shares the average of
+ # the 1-based ranks it spans.
+ order = np.argsort(s, kind="mergesort")
+ ordered = s[order]
+ ranks = np.empty(s.shape[0], dtype=float)
+ start = 0
+ while start < ordered.shape[0]:
+ end = start
+ while end + 1 < ordered.shape[0] and ordered[end + 1] == ordered[start]:
+ end += 1
+ ranks[order[start:end + 1]] = (start + end) / 2.0 + 1.0
+ start = end + 1
+
+ rank_sum = float(ranks[y].sum())
+ return (rank_sum - n_pos * (n_pos + 1) / 2.0) / (n_pos * n_neg)
+
+
+def _case_act_probability(case: Dict[str, Any]) -> Any:
+ answer = case.get("answer")
+ if not isinstance(answer, dict):
+ return None
+ action = answer.get("action")
+ if not isinstance(action, dict):
+ return None
+ return action.get("act_probability")
+
+
+def analyse_cases(cases: Sequence[Any]) -> Dict[str, Any]:
+ """Score one report's cases, bucketing every case exactly once.
+
+ Returns a dict that is JSON-serialisable as-is: counts, both AUCs (or ``None`` with a
+ reason), and the spread of ``act_probability`` needed to spot a saturated signal.
+ """
+ paired: List[Dict[str, Any]] = []
+ skipped = {
+ "no_binary_correctness": 0,
+ "missing_act_probability": 0,
+ "missing_confidence": 0,
+ "malformed_case": 0,
+ }
+
+ for case in cases:
+ if not isinstance(case, dict):
+ skipped["malformed_case"] += 1
+ continue
+ correct = case.get("correct")
+ if not isinstance(correct, bool):
+ # A score answer, or a report without the field: correctness is not binary here.
+ skipped["no_binary_correctness"] += 1
+ continue
+ act = _case_act_probability(case)
+ if not _is_number(act):
+ # Never coerced to 0.0: a missing act_probability is missing evidence, and
+ # reading it as "not acting" would quietly invent a decision.
+ skipped["missing_act_probability"] += 1
+ continue
+ confidence = case.get("confidence")
+ if not _is_number(confidence):
+ # Dropped from the paired set rather than back-filled, so the act AUC and the
+ # confidence control stay on exactly the same decisions.
+ skipped["missing_confidence"] += 1
+ continue
+ paired.append({"act": float(act), "confidence": float(confidence), "correct": correct})
+
+ labels = [p["correct"] for p in paired]
+ acts = [p["act"] for p in paired]
+ confs = [p["confidence"] for p in paired]
+
+ act_auc = roc_auc(acts, labels) if paired else None
+ confidence_auc = roc_auc(confs, labels) if paired else None
+ if not paired:
+ reason = _NO_CASES
+ elif act_auc is None:
+ reason = _UNSCORABLE
+ else:
+ reason = None
+
+ summary: Dict[str, Any] = {
+ "schema": SCHEMA,
+ "total_cases": len(cases),
+ "paired": len(paired),
+ "paired_cases": len(paired),
+ "correct": sum(1 for p in paired if p["correct"]),
+ "incorrect": sum(1 for p in paired if not p["correct"]),
+ "act_auc": act_auc,
+ "confidence_auc": confidence_auc,
+ "auc_null_reason": reason,
+ "act_min": min(acts) if acts else None,
+ "act_max": max(acts) if acts else None,
+ "act_unique": len(set(acts)) if acts else 0,
+ }
+ summary["skipped"] = skipped
+ summary["saturated"] = bool(
+ acts and (len(set(acts)) <= 1 or (max(acts) - min(acts)) < _SATURATION_EPS)
+ )
+ return summary
+
+
+def report_from_document(document: Any) -> Dict[str, Any]:
+ """Analyse an `EvalReport.to_json()` document.
+
+ Raises ValueError when the document is not a report, so a wrong file is a clear error
+ rather than a set of zeros that reads like a result.
+ """
+ if not isinstance(document, dict):
+ raise ValueError("report must be a JSON object")
+ cases = document.get("cases")
+ if not isinstance(cases, list):
+ raise ValueError("report has no 'cases' list: is this a laya-evals --json report?")
+ analysis = analyse_cases(cases)
+ return {
+ "schema": SCHEMA,
+ "config": document.get("config") or {},
+ "source_overall": document.get("overall") or {},
+ "analysis": analysis,
+ }
+
+
+def format_markdown(result: Dict[str, Any]) -> str:
+ """A short human summary of an analysed report."""
+ a = result["analysis"]
+ lines = [
+ "# act-head diagnostic",
+ "",
+ "Does `act_probability` rank correct decisions above incorrect ones?",
+ "",
+ "| measure | value |",
+ "|---|---|",
+ "| cases in report | %d |" % a["total_cases"],
+ "| scorable (paired) decisions | %d |" % a["paired"],
+ "| correct / incorrect | %d / %d |" % (a["correct"], a["incorrect"]),
+ "| ROC-AUC of `act_probability` | %s |" % _fmt(a["act_auc"]),
+ "| ROC-AUC of calibrated `confidence` (same cases) | %s |" % _fmt(a["confidence_auc"]),
+ "| `act_probability` min / max / unique | %s / %s / %d |"
+ % (_fmt(a["act_min"]), _fmt(a["act_max"]), a["act_unique"]),
+ "| saturated signal | %s |" % ("yes" if a["saturated"] else "no"),
+ ]
+ if a["auc_null_reason"]:
+ lines.append("")
+ lines.append("No AUC reported: %s." % a["auc_null_reason"])
+ skipped = a["skipped"]
+ if any(skipped.values()):
+ lines.append("")
+ lines.append("Skipped cases, excluded from every count above:")
+ for key in ("no_binary_correctness", "missing_act_probability",
+ "missing_confidence", "malformed_case"):
+ if skipped[key]:
+ lines.append("* `%s`: %d" % (key, skipped[key]))
+ lines.extend([
+ "",
+ "This is a diagnostic only. It does not change `act_probability`, and a score",
+ "here is not a claim that the current act head is informative.",
+ "",
+ ])
+ return "\n".join(lines)
+
+
+def _fmt(value: Optional[float]) -> str:
+ return "n/a" if value is None else "%.4f" % value
+
+
+def main(argv: Optional[List[str]] = None) -> int:
+ parser = argparse.ArgumentParser(
+ description="Score act_probability against labelled decisions in a laya-evals report."
+ )
+ parser.add_argument("report", help="the JSON written by `laya-evals run --json`")
+ parser.add_argument("--json", dest="json_out", help="write the full result here")
+ parser.add_argument("--markdown", dest="md_out", help="write a human summary here")
+ args = parser.parse_args(argv)
+
+ with open(args.report, "r", encoding="utf-8") as handle:
+ document = json.load(handle)
+ result = report_from_document(document)
+
+ if args.json_out:
+ with open(args.json_out, "w", encoding="utf-8") as handle:
+ json.dump(result, handle, indent=2, sort_keys=True)
+ handle.write("\n")
+ if args.md_out:
+ with open(args.md_out, "w", encoding="utf-8") as handle:
+ handle.write(format_markdown(result))
+ if not args.json_out and not args.md_out:
+ print(json.dumps(result, indent=2, sort_keys=True))
+ else:
+ print(format_markdown(result))
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/research/evals/test_act_head_eval.py b/research/evals/test_act_head_eval.py
new file mode 100644
index 00000000..8d982dc7
--- /dev/null
+++ b/research/evals/test_act_head_eval.py
@@ -0,0 +1,228 @@
+"""Offline tests for the act-head diagnostic in `research/evals/act_head_eval.py`.
+
+No checkpoint and no network: every function under test is pure arithmetic over a
+report that `laya-evals --json` already wrote, so this runs offline.
+
+The fixtures here are tiny and synthetic. They exist to pin the metric mechanics
+only -- they are **not** a measurement of the released checkpoints and say nothing
+about act-head quality. The real labelled sets are not committed to this repository.
+
+Run: python research/evals/test_act_head_eval.py
+"""
+import json
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from research.evals.act_head_eval import ( # noqa: E402
+ analyse_cases,
+ format_markdown,
+ roc_auc,
+)
+
+PASS, FAIL = [], []
+
+
+def check(name, got, want):
+ if got == want:
+ PASS.append(name)
+ else:
+ FAIL.append("%s:\n got %r\n want %r" % (name, got, want))
+
+
+def check_true(name, cond, detail=""):
+ if cond:
+ PASS.append(name)
+ else:
+ FAIL.append("%s%s" % (name, (" -- " + detail) if detail else ""))
+
+
+def check_close(name, got, want, tol=1e-9):
+ if got is None:
+ FAIL.append("%s:\n got None\n want ~%r" % (name, want))
+ elif abs(float(got) - want) > tol:
+ FAIL.append("%s:\n got %r\n want ~%r" % (name, got, want))
+ else:
+ PASS.append(name)
+
+
+def case(act, correct, confidence=None, qid="q", answer_type="choice", drop_action=False):
+ """One `laya-evals` report case, shaped like `EvalReport.to_json()["cases"][i]`."""
+ answer = {"type": answer_type}
+ if correct is not None and answer_type == "choice":
+ answer["choice"] = "a"
+ if not drop_action:
+ answer["action"] = {} if act is None else {"act_probability": act}
+ else:
+ answer["action"] = {}
+ case_obj = {"qid": qid, "answer": answer, "correct": correct}
+ if confidence is not None:
+ case_obj["confidence"] = confidence
+ return case_obj
+
+
+# ------------------------------------------------------------------ roc_auc math
+# Perfect separation: every correct case outranks every incorrect one.
+check("auc/perfect", roc_auc([0.9, 0.8, 0.2, 0.1], [True, True, False, False]), 1.0)
+# Exactly reversed: the score runs against correctness, which is the failure #185 reports.
+check("auc/reversed", roc_auc([0.1, 0.2, 0.8, 0.9], [True, True, False, False]), 0.0)
+# All scores tied, both classes present: every pairing is a tie, so exactly 0.5.
+check("auc/all_tied", roc_auc([1.0, 1.0, 1.0, 1.0], [True, True, False, False]), 0.5)
+# Partial ties get mid-ranks, so a tied positive does not beat a tied negative outright.
+check_close("auc/partial_tie", roc_auc([0.5, 0.5, 0.9], [True, False, True]), 0.75)
+check_close("auc/single_positive", roc_auc([0.3, 0.7], [False, True]), 1.0)
+# One class absent: ROC-AUC is undefined, and must not be faked as 0.5 or 0.0.
+check("auc/all_positive", roc_auc([0.1, 0.9], [True, True]), None)
+check("auc/all_negative", roc_auc([0.1, 0.9], [False, False]), None)
+check("auc/empty", roc_auc([], []), None)
+# A single tied pair is the smallest well-defined case.
+check("auc/one_each_tied", roc_auc([0.7, 0.7], [True, False]), 0.5)
+
+# ------------------------------------------------------------------ case bucketing
+# A perfect ranking through the full report path, with the confidence control.
+perfect = analyse_cases([
+ case(0.99, True, 0.9), case(0.80, True, 0.8),
+ case(0.20, False, 0.3), case(0.01, False, 0.1),
+])
+check("analyse/paired_all", perfect["paired"], 4)
+check("analyse/correct", perfect["correct"], 2)
+check("analyse/incorrect", perfect["incorrect"], 2)
+check("analyse/act_auc", perfect["act_auc"], 1.0)
+check("analyse/confidence_auc", perfect["confidence_auc"], 1.0)
+check_true("analyse/not_saturated", perfect["saturated"] is False, repr(perfect))
+check("analyse/unique", perfect["act_unique"], 4)
+
+# Reversed act_probability, but a confidence that ranks correctly: this is the shape
+# of the measurement in #185, where the raw signal runs against correctness.
+reversed_act = analyse_cases([
+ case(0.10, True, 0.9), case(0.20, True, 0.7),
+ case(0.80, False, 0.3), case(0.90, False, 0.1),
+])
+check("analyse/reversed_act_auc", reversed_act["act_auc"], 0.0)
+check("analyse/reversed_confidence_auc", reversed_act["confidence_auc"], 1.0)
+check("analyse/reversed_same_case_set", reversed_act["paired"], 4)
+
+# The documented failure mode: act_probability saturated at 1.0 on every input.
+saturated = analyse_cases([
+ case(1.0, True, 0.9), case(1.0, False, 0.4), case(1.0, True, 0.8), case(1.0, False, 0.2),
+])
+check("analyse/saturated_detected", saturated["saturated"], True)
+check("analyse/saturated_unique", saturated["act_unique"], 1)
+check("analyse/saturated_auc", saturated["act_auc"], 0.5)
+check("analyse/saturated_min_max", (saturated["act_min"], saturated["act_max"]), (1.0, 1.0))
+check_true("analyse/saturated_min_max_json_safe",
+ json.dumps(saturated, sort_keys=True) is not None)
+
+# `correct=null` (a score answer) is counted and skipped, never folded into the denominator.
+with_nulls = analyse_cases([
+ case(0.9, True, 0.8),
+ case(0.1, False, 0.2),
+ case(0.5, None, 0.6, answer_type="score"),
+ case(0.4, None, 0.3, answer_type="score"),
+])
+check("analyse/null_correctness_skipped", with_nulls["paired"], 2)
+check("analyse/null_correctness_counted", with_nulls["skipped"]["no_binary_correctness"], 2)
+check("analyse/null_correctness_auc", with_nulls["act_auc"], 1.0)
+
+# Missing act_probability is explicit, never silently read as 0.0.
+missing = analyse_cases([
+ case(0.9, True, 0.8),
+ case(None, False, 0.3, drop_action=True),
+ case(0.1, False, 0.2),
+])
+check("analyse/missing_act_counted", missing["skipped"]["missing_act_probability"], 1)
+check("analyse/missing_act_paired", missing["paired"], 2)
+check("analyse/missing_act_auc", missing["act_auc"], 1.0)
+check_true("analyse/missing_act_min_ignores_missing",
+ missing["act_min"] == 0.1 and missing["act_max"] == 0.9, repr(missing))
+
+# A non-numeric act_probability is malformed, not zero.
+malformed = analyse_cases([
+ case("not-a-number", True, 0.8),
+ case(0.2, False, 0.3),
+])
+check("analyse/malformed_act_counted", malformed["skipped"]["missing_act_probability"], 1)
+check("analyse/malformed_act_paired", malformed["paired"], 1)
+
+# A bool in the act_probability slot is malformed, not the integer 1.
+boolish = analyse_cases([case(True, True, 0.8), case(0.2, False, 0.3)])
+check("analyse/bool_act_counted", boolish["skipped"]["missing_act_probability"], 1)
+
+# Missing confidence drops the case from BOTH AUCs, so the control stays on one set.
+no_conf = analyse_cases([
+ case(0.9, True, 0.8),
+ case(0.1, False),
+ case(0.2, True, 0.1),
+])
+check("analyse/missing_confidence_counted", no_conf["skipped"]["missing_confidence"], 1)
+check("analyse/missing_confidence_paired", no_conf["paired"], 2)
+check("analyse/missing_confidence_same_set", no_conf["paired_cases"], 2)
+
+# All-correct or all-incorrect cannot produce a fake AUC.
+all_correct = analyse_cases([case(0.9, True, 0.8), case(0.4, True, 0.6)])
+check("analyse/all_correct_auc", all_correct["act_auc"], None)
+check("analyse/all_correct_confidence_auc", all_correct["confidence_auc"], None)
+check("analyse/all_correct_null_reason", all_correct["auc_null_reason"],
+ "one class is absent: ROC-AUC is undefined")
+all_wrong = analyse_cases([case(0.9, False, 0.8), case(0.4, False, 0.6)])
+check("analyse/all_wrong_auc", all_wrong["act_auc"], None)
+
+# An empty report is a clean null state, not a crash.
+empty = analyse_cases([])
+check("analyse/empty_paired", empty["paired"], 0)
+check("analyse/empty_auc", empty["act_auc"], None)
+check("analyse/empty_reason", empty["auc_null_reason"], "no scorable cases")
+
+# Non-dict / malformed case entries are counted, not fatal.
+junk = analyse_cases([None, "nope", 5, case(0.9, True, 0.8)])
+check("analyse/junk_counted", junk["skipped"]["malformed_case"], 3)
+check("analyse/junk_paired", junk["paired"], 1)
+
+# ------------------------------------------------------------------ report plumbing
+REPORT = {
+ "config": {"model": "english"},
+ "overall": {"accuracy": 0.5},
+ "cases": [
+ case(0.9, True, 0.8, qid="a"),
+ case(0.1, False, 0.2, qid="b"),
+ case(0.5, None, 0.6, qid="c", answer_type="score"),
+ ],
+}
+
+
+def _load(payload):
+ return json.loads(json.dumps(payload))
+
+
+from research.evals.act_head_eval import report_from_document # noqa: E402
+
+loaded = report_from_document(_load(REPORT))
+check("report/cases_read", loaded["analysis"]["total_cases"], 3)
+check("report/config_kept", loaded["config"], {"model": "english"})
+check("report/analyse_paired", loaded["analysis"]["paired"], 2)
+check("report/schema", loaded["schema"], "laya-evals-report/act-head-eval/1")
+
+# The same document twice gives byte-identical JSON, so a committed result is a real diff.
+first = json.dumps(report_from_document(_load(REPORT)), sort_keys=True)
+second = json.dumps(report_from_document(_load(REPORT)), sort_keys=True)
+check("report/deterministic", first, second)
+
+# A document without `cases` is rejected clearly instead of silently reporting zeros.
+try:
+ report_from_document({"overall": {}})
+ FAIL.append("report/missing_cases should raise")
+except ValueError as exc:
+ check_true("report/missing_cases_raises", "cases" in str(exc), str(exc))
+
+markdown = format_markdown(loaded)
+check_true("report/markdown_has_question", "act_probability" in markdown, markdown[:200])
+check_true("report/markdown_flags_saturation", "saturat" in markdown.lower(), markdown[:400])
+
+print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
+for f in FAIL:
+ print(" FAIL " + f)
+if not FAIL:
+ print("all act-head eval tests passed")
+sys.exit(1 if FAIL else 0)