Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(evals): a run identity, and a gate that refuses a baseline it cannot match
Bruce-Yiiba9054b23ac8
diff --git a/docs/evals.md b/docs/evals.mdindex 57bf788e..e8430636 100644--- a/docs/evals.md+++ b/docs/evals.md@@ -124,6 +124,34 @@ metrics per slice value, so a regression in one language or one question is visireading the aggregate. The `model` slice holds the checkpoint that answered each row: the`Router`'s own choice per request, or the runner's `model` for a runner that does not route.+## Run identity++`run` records what it measured in the report's `config` block, so the artifact a reviewer reads+is reviewable on its own:++| key | meaning |+|---|---|+| `schema` | the report shape, `laya-evals-report/1`, so a consumer can refuse one it cannot read |+| `dataset` | the path as typed -- a name, not a hash |+| `dataset_sha256` | the sha256 of the dataset bytes that were parsed |+| `questions_sha256` | a fingerprint of the question schema: every question's id, type and `criteria`, over the whole dataset |+| `laya_version` | the `laya` that computed the numbers |+| `thresholds` | the gate this run applied: `min`, `max` and `baseline_tolerance` |+| `revisions` | the commit each checkpoint that answered was loaded from (see [below](#baseline-and-ci-gate)) |++`dataset` is a path, and a path is not an identity: a dataset can be edited in place, moved, or+refetched under the same name, and a CI cache can hand two runs the same filename and different+bytes. `questions_sha256` covers what was *asked* rather than how many rows there were, so scoring+more states on the same questions stays comparable to a baseline, while renaming a `choice` option+label does not -- `criteria` is the decision space, and the metamorphic checks in+`research/eval/metamorphic.py` exist because a label rename flips answers. `instructions` is+excluded on purpose: rewording prose changes the prompt, not the question being asked.++Nothing time-bearing is recorded, so a report is still byte-reproducible for a fixed runner.++`REPORT_SCHEMA`, `questions_fingerprint(dataset)` and `file_fingerprint(path)` are public, so a+caller driving `laya.evals.evaluate` directly gets the same identity a CLI run does.+## Baseline and CI gate- Keep the dataset, a baseline report (`--json` output you have reviewed), and the tolerances@@ -133,6 +161,21 @@ reading the aggregate. The `model` slice holds the checkpoint that answered eachdrops into CI unchanged. `laya.evals.EvalReport.compare` and `assert_regression` expose thesame logic for tests.+The metric gate answers "did the numbers move". It cannot answer "were these the same numbers",+because `compare` reads `overall` and only `overall` -- so a baseline recorded against one dataset+would pass a candidate scored on another, with identical arithmetic. `EvalReport.comparable_to`+closes that: it compares `schema`, `dataset_sha256` and `questions_sha256`, and `run --baseline`+and `compare` still print every delta, then fail with a non-zero exit naming the key and both+values:++```text+FAIL: baseline is not comparable: dataset_sha256 (dataset bytes): baseline is <sha>, this run is <sha>+```++A key missing on either side is *unknown*, not a conflict, so every report written before the+identity existed keeps comparing exactly as it did. That includes the scheduled gate's baseline+below, which comes from `research/eval/` and has no `config.schema` at all.+Two CI surfaces use this:- a weight-free job in `.github/workflows/ci.yml` runs `tests/test_evals.py` anddiff --git a/docs/staged-adoption.md b/docs/staged-adoption.mdindex 9f3bed9a..eebb51c7 100644--- a/docs/staged-adoption.md+++ b/docs/staged-adoption.md@@ -78,6 +78,13 @@ owner with the policy. Re-evaluate it when those inputs change. Confidence ordernot establish that a decision is correct, and high confidence is never execution permission byitself.+Four of those six come from the evaluation harness: a `laya-evals run --json` report records the+checkpoint commit that answered (`config.revisions`), the dataset's bytes and question schema+(`config.dataset_sha256`, `config.questions_sha256`) and the gate it was scored under+(`config.thresholds`). Calibration method and owner are the application's to record. See+[Evaluation harness](evals.md) for the run identity and the comparability check between a baseline+and a candidate.+The README's [Automated Confidence Gating](https://github.com/NandhaKishorM/laya#automated-confidence-gating),[Calibration](https://github.com/NandhaKishorM/laya#calibration), and [Honest limits](https://github.com/NandhaKishorM/laya#honest-limits) sectionsgive the existing calibration and confidence context. Keep irreversible or high-cost actions behind@@ -128,6 +135,8 @@ real request## See also- [Prediction hooks](hooks/index.md) — the extension seam for audit, metrics and gating.+- [Evaluation harness](evals.md) — the run identity a baseline carries and the comparability check+ between a baseline and a candidate.- [Hook API reference](hooks/api.md) — `PredictContext` fields and lifecycle events.- [Tracing](hooks/tracing.md) — `run_id` and span correlation.- [README: Automated Confidence Gating](https://github.com/NandhaKishorM/laya#automated-confidence-gating) — confidence is adiff --git a/laya/evals.py b/laya/evals.pyindex 050d8c8f..a02243c2 100644--- a/laya/evals.py+++ b/laya/evals.py@@ -10,9 +10,18 @@ checkpoint in tests.The report is deterministic for a fixed runner: the same dataset produces the same numbers, and``EvalReport.compare`` turns a baseline into a pass/fail with the per-metric deltas, which is whatthe CI gate consumes.++`compare` reads ``overall`` and nothing else, so a run that says nothing about *what it measured*+makes a gate that can only compare arithmetic. Every report therefore carries a run identity in+``config`` -- ``schema``, ``questions_sha256``, ``laya_version`` and (from the CLI)+``dataset_sha256`` and the ``thresholds`` actually applied -- and+``EvalReport.comparable_to`` refuses a comparison between two runs that are not the same+measurement. The identity is deliberately free of anything time-bearing, so a report stays+byte-reproducible for a fixed runner."""from __future__ import annotations+import hashlibimport inspectimport jsonimport statistics@@ -22,6 +31,34 @@ from typing import Any, Dict, Iterable, List, Optional, Sequence, Tupleimport numpy as np+#: The shape of the report this module writes, so a consumer can refuse one it cannot read.+#: ``research/evals/act_head_eval.py`` publishes a report under its own tag off this prefix.+REPORT_SCHEMA = "laya-evals-report/1"++#: The keys that decide whether two reports are the same measurement. A key missing+#: on either side is unknown rather than a conflict, so a baseline committed before these existed+#: keeps comparing exactly as it did. ``schema`` is also read at the top level, which is where+#: ``research/evals/act_head_eval.py`` puts its own tag.+_IDENTITY_KEYS = ("schema", "dataset_sha256", "questions_sha256")++_IDENTITY_LABELS = {+ "schema": "report schema",+ "dataset_sha256": "dataset bytes",+ "questions_sha256": "question schema",+}+++def _identity_of(document: Any) -> Dict[str, Any]:+ """Pull the identity keys out of a report, from ``config`` or from the top level."""+ if not isinstance(document, dict):+ return {}+ config = document.get("config")+ found = dict(config) if isinstance(config, dict) else {}+ for key in _IDENTITY_KEYS:+ if found.get(key) is None and document.get(key) is not None:+ found[key] = document[key]+ return found+class EvalError(ValueError):"""A dataset or report is malformed; the message names the row or field."""@@ -88,6 +125,80 @@ class Dataset:return cls(examples)+def _canonical(value: Any) -> str:+ return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False,+ default=str)+++def questions_fingerprint(dataset: "Dataset") -> str:+ """A stable hash of *what was asked*, over every question in `dataset`.++ This is the question-schema identity `docs/staged-adoption.md` asks an operator to record+ with the policy, and the part of the run identity a report can compute for itself: the+ dataset file hash is the CLI's, but the questions are parsed here.++ It is a function of the decision space, not of the rows, so scoring more states on the same+ questions stays comparable to a baseline. A `choice` option label lives in `criteria` and so+ counts as a different question -- which is the case that matters, since+ `research/eval/metamorphic.py` exists because renaming a label flips answers. Free-text+ `instructions` are excluded: rewording them changes the prompt, not the decision space, and+ a baseline should survive a copy edit.+ """+ schemas = set()+ for example in dataset.examples:+ for qid, question in example.questions.items():+ body = question if isinstance(question, dict) else {}+ schemas.add(_canonical({"qid": qid, "type": body.get("type"),+ "criteria": body.get("criteria")}))+ digest = hashlib.sha256()+ for schema in sorted(schemas):+ digest.update(schema.encode("utf-8"))+ digest.update(b"\n")+ return digest.hexdigest()+++def file_fingerprint(path: str) -> str:+ """The sha256 of a file's bytes -- the dataset identity, where the path is only a name.++ `config.dataset` is the path as typed, and two datasets share a path across a rebase, a CI+ cache or a colleague's checkout. Raises `EvalError` when the file cannot be read, so a report+ never carries a hash of something other than the bytes that were parsed.+ """+ digest = hashlib.sha256()+ try:+ with open(path, "rb") as handle:+ for block in iter(lambda: handle.read(65536), b""):+ digest.update(block)+ except OSError as exc:+ raise EvalError("%s cannot be read for a fingerprint: %s" % (path, exc)) from exc+ return digest.hexdigest()+++def _library_version() -> Optional[str]:+ """`laya.__version__`, read lazily so this module stays importable on its own.++ `laya/__init__.py` defines the string at import time, so this is a module lookup rather+ than a load of anything torch-backed -- but it is imported inside the call to keep+ `laya.evals` free of an import cycle, and a failure here must not fail a run that has+ already scored every row.+ """+ try:+ import laya+ except Exception: # pragma: no cover - identity only+ return None+ return getattr(laya, "__version__", None)+++def _run_identity(dataset: "Dataset") -> Dict[str, Any]:+ """The `config` keys this module can fill in on its own, with no knowledge of the CLI."""+ identity: Dict[str, Any] = {"schema": REPORT_SCHEMA,+ "questions_sha256": questions_fingerprint(dataset)}+ version = _library_version()+ if version is not None:+ identity["laya_version"] = version+ return identity++# --------------------------------------------------------------------------- evaluatorsclass Evaluator:"""Per-answer metric. `score` returns a number, or None when it does not apply."""@@ -240,6 +351,34 @@ class EvalReport:lines.append("| %s | %s |" % (value, " | ".join(cells)))return "\n".join(lines) + "\n"+ def comparable_to(self, baseline: Dict[str, Any]) -> Tuple[bool, List[str]]:+ """Is `baseline` the same measurement as this report? Returns (ok, reasons).++ `compare` reads `overall` and only `overall`, so two reports of different datasets,+ different question schemas or different report shapes produce identical arithmetic and+ an identical pass. That is the failure `docs/staged-adoption.md` runs at: a gate that+ cannot tell which experiment produced a number cannot support a promotion decision.++ A key absent on either side is *unknown*, not a conflict, so every baseline committed+ before the identity existed -- including the scheduled gate's+ `research/results/eval_english_51_languages.json`, which comes from `research/eval/` and+ has no `config.schema` -- keeps comparing exactly as it did. Only a key present on both+ sides with different values refuses the comparison, and each reason names the key and+ both values so the failure is actionable from the console alone.++ `laya_version` and `thresholds` are recorded in `config` but deliberately not compared+ here: a patch release must not invalidate a committed baseline.+ """+ other = _identity_of(baseline)+ reasons: List[str] = []+ for key in _IDENTITY_KEYS:+ here, there = self.config.get(key), other.get(key)+ if here is None or there is None or here == there:+ continue+ reasons.append("%s (%s): baseline is %s, this run is %s"+ % (key, _IDENTITY_LABELS[key], there, here))+ return (not reasons), reasons+def compare(self, baseline: Dict[str, Any], tolerances: Optional[Dict[str, float]] = None,) -> Tuple[bool, Dict[str, Dict[str, float]]]:"""Compare `overall` to a baseline report's `overall`. Returns (ok, deltas).@@ -461,6 +600,10 @@ def evaluate(runner: Any, dataset: Dataset, evaluators: Optional[Sequence[Evaluareport.overall["latency_p50_ms"], report.overall["latency_p95_ms"] = _percentiles(waits)(report.overall["cost_per_decision_p50_ms"],report.overall["cost_per_decision_p95_ms"]) = _percentiles(shares)+ # The run identity, stamped here rather than left to the caller: everything below is+ # computable from `dataset`, so a programmatic `evaluate` is as identifiable as a CLI run.+ # The CLI adds `dataset_sha256` and `thresholds`, which need the file path and the gate.+ report.config = dict(report.config, **_run_identity(dataset))report.config = dict(report.config, timing={"latency_metric": "per request: the wall time of the call that returned it, unsplit","cost_metric": "per decision: that call divided by its own chunk size",diff --git a/laya/evals_cli.py b/laya/evals_cli.pyindex d2a3a8ff..e2ed02af 100644--- a/laya/evals_cli.py+++ b/laya/evals_cli.py@@ -204,6 +204,21 @@ def _print_deltas(deltas: Dict[str, Dict[str, Any]]) -> None:% (metric, delta["baseline"], delta["value"], delta["diff"], delta["tolerance"]))+def _comparability_failure(report: evals.EvalReport, baseline: Dict[str, Any]) -> Optional[str]:+ """The refusal to compare two runs that are not the same measurement, as one failure line.++ The metric gate answers "did the numbers move"; it cannot answer "were these the same+ numbers", because `EvalReport.compare` reads `overall` and only `overall`. So a baseline+ recorded against one dataset passes a candidate scored on another, with identical+ arithmetic. The gate still prints its deltas -- the reviewer wants to see both facts -- but+ an incomparable pair is not a pass.+ """+ ok, reasons = report.comparable_to(baseline)+ if ok:+ return None+ return "baseline is not comparable: " + "; ".join(reasons)++def _cmd_validate(args) -> int:dataset = evals.Dataset.from_jsonl(args.dataset)questions = sorted({qid for example in dataset.examples for qid in example.questions})@@ -262,7 +277,12 @@ def _cmd_run(args) -> int:# download.raise EvalError(str(exc)) from excrunner = RouterRunner(router)- config = {"dataset": args.dataset, "model": args.model, "device": args.device}+ config = {"dataset": args.dataset, "model": args.model, "device": args.device,+ # The path above is the name, not the data. Hashing the bytes here is what makes+ # a committed baseline reviewable: the dataset can be edited in place, moved or+ # refetched under the same name, and a reviewer comparing two reports needs the+ # report -- not a file mtime -- to say so.+ "dataset_sha256": evals.file_fingerprint(args.dataset)}if args.onnx:config["onnx"] = args.onnxif extra:@@ -286,14 +306,25 @@ def _cmd_run(args) -> int:mins[key] = args.min_accuracyif args.max_ece is not None:maxs["ece"] = args.max_ece+ tolerances = _parse_pairs(args.tolerance)+ # The gate this run actually applied, recorded with the numbers it produced. A baseline whose+ # numbers came from a different gate is a different claim, and `docs/staged-adoption.md`+ # asks for exactly this alongside the checkpoint version and the evaluation set.+ report.config = dict(report.config,+ thresholds={"min": mins, "max": maxs, "baseline_tolerance": tolerances})failures = _check_thresholds(report.overall, mins, maxs)if args.baseline:baseline = _load_report(args.baseline)- ok, deltas = report.compare(baseline, _parse_pairs(args.tolerance))+ ok, deltas = report.compare(baseline, tolerances)_print_deltas(deltas)if not ok:failures.append("baseline comparison failed")+ # Checked after the deltas are printed, so a reviewer sees what moved as well as why the+ # two runs are not the same experiment.+ refusal = _comparability_failure(report, baseline)+ if refusal:+ failures.append(refusal)for name in sorted(report.overall):print("%-18s %.4f" % (name, report.overall[name]))@@ -322,7 +353,13 @@ def _cmd_compare(args) -> int:baseline = _load_report(args.baseline)ok, deltas = report.compare(baseline, _parse_pairs(args.tolerance))_print_deltas(deltas)- return 0 if ok else 1+ # `compare` is the offline re-check of a report a reviewer already read, so the same+ # comparability refusal has to apply here: a saved report that cannot say which experiment+ # it came from must not pass a gate either.+ refusal = _comparability_failure(report, baseline)+ if refusal:+ print("FAIL: " + refusal, file=sys.stderr)+ return 0 if ok and not refusal else 1def main(argv: Optional[Sequence[str]] = None) -> int:diff --git a/tests/test_evals.py b/tests/test_evals.pyindex ce0db32a..2c227aca 100644--- a/tests/test_evals.py+++ b/tests/test_evals.py@@ -7,6 +7,8 @@ import reimport pytest+import laya+from laya import evalsfrom laya.evals import (ChoiceAccuracy,Dataset,@@ -586,6 +588,141 @@ def test_default_evaluators_cover_the_three_types():assert {"choice_accuracy", "noul_accuracy", "score_mae", "mean_confidence"} <= names+# ------------------------------------------------------- run identity / comparability+def _identified(**config):+ return EvalReport(config={"schema": evals.REPORT_SCHEMA, **config},+ overall={"choice_accuracy": 0.8})+++def test_evaluate_records_what_it_measured():+ """`config` is the artifact a reviewer reads, so it has to say which run produced it.++ `config.dataset` is the path as typed. Two datasets share a path across a rebase, a CI+ cache or a colleague's checkout, and `docs/evals.md` tells the reviewer to commit the+ dataset and the baseline together and read the result as a diff -- which is only reviewable+ if the report says which bytes were scored.+ """+ dataset = Dataset([Example("s1", Q, {"intent": "a"}), Example("s2", Q, {"intent": "b"})])+ report = evaluate(StubRunner({"s1": {"intent": choice_answer("a")},+ "s2": {"intent": choice_answer("a")}}), dataset)+ config = report.config+ assert config["schema"] == evals.REPORT_SCHEMA+ assert config["laya_version"] == laya.__version__+ assert len(config["questions_sha256"]) == 64, "the question schema is part of the identity"++ # Determinism: the identity must carry nothing time-bearing, or a re-run of the same dataset+ # stops producing the same report and the artifact a reviewer diffs becomes noise. The+ # `*_ms` metrics are wall clock and always differ; only the config block is asserted here.+ again = evaluate(StubRunner({"s1": {"intent": choice_answer("a")},+ "s2": {"intent": choice_answer("a")}}), dataset)+ assert {k: v for k, v in config.items() if k != "timing"} == \+ {k: v for k, v in again.config.items() if k != "timing"}+++def test_questions_fingerprint_follows_the_question_not_the_row():+ """The fingerprint covers the question schema, so it is a function of what was asked.++ A run that scores a different dataset must not collide with a baseline, and a run that+ scores the same questions on more rows must still be comparable to it. Renaming a `choice`+ label is a different question, not a different row: `research/eval/metamorphic.py` exists+ because option order and label rename flip answers.+ """+ base = Dataset([Example("s1", Q, {"intent": "a"})])+ more_rows = Dataset([Example("s1", Q, {"intent": "a"}), Example("s2", Q, {"intent": "a"})])+ assert evals.questions_fingerprint(base) == evals.questions_fingerprint(more_rows)++ renamed = Dataset([Example("s1", {"intent": {"type": "choice", "instructions": "?",+ "criteria": {"a": "x", "c": "y"}}},+ {"intent": "a"})])+ assert evals.questions_fingerprint(renamed) != evals.questions_fingerprint(base)++ retyped = Dataset([Example("s1", {"flag": {"type": "noul", "instructions": "?"}},+ {"flag": True})])+ assert evals.questions_fingerprint(retyped) != evals.questions_fingerprint(base)+++def test_comparable_to_refuses_two_different_runs():+ """`compare` reads `overall` only, so without this a gate passes two different experiments.++ Both reports score 0.8, so the arithmetic is identical -- but they are not the same+ measurement, and a promotion decision made on that is unfalsifiable. Every field+ `docs/staged-adoption.md` tells the operator to record with the policy has to be able to+ stop the comparison.+ """+ baseline = _identified(dataset_sha256="a" * 64, questions_sha256="q" * 64)+ same = _identified(dataset_sha256="a" * 64, questions_sha256="q" * 64)+ ok, reasons = same.comparable_to(baseline.to_json())+ assert ok and reasons == [], reasons++ for field, other in (("schema", "laya-evals-report/act-head-eval/1"),+ ("dataset_sha256", "b" * 64),+ ("questions_sha256", "z" * 64)):+ candidate = _identified(dataset_sha256="a" * 64, questions_sha256="q" * 64)+ candidate.config[field] = other+ ok, reasons = candidate.comparable_to(baseline.to_json())+ assert not ok, field+ # The key is named, so the failure is greppable against the `config` a reviewer reads.+ assert any(field in reason for reason in reasons), (field, reasons)++ # The reason names both values, or a reviewer still has to open two files to act on it.+ swapped = _identified(dataset_sha256="c" * 64, questions_sha256="q" * 64)+ ok, reasons = swapped.comparable_to(baseline.to_json())+ assert "c" * 64 in " ".join(reasons) and "a" * 64 in " ".join(reasons)+++def test_comparable_to_ignores_what_it_cannot_see():+ """A field missing on one side is unknown, not a conflict.++ Every baseline committed before this existed -- including the scheduled gate's+ `research/results/eval_english_51_languages.json`, which comes from `research/eval/` and+ has no `config.schema` at all -- has to keep comparing exactly as it did.+ """+ legacy = {"overall": {"choice_accuracy": 0.8}, "config": {"dataset": "old.jsonl"}}+ report = _identified(dataset_sha256="a" * 64, questions_sha256="q" * 64)+ ok, reasons = report.comparable_to(legacy)+ assert ok and reasons == [], reasons+ assert report.compare(legacy, {"choice_accuracy": 0.0})[0], "the metric gate still runs"++ # Two reports that both predate the identity fields are as comparable as they ever were.+ ok, reasons = EvalReport(config={"dataset": "a.jsonl"}).comparable_to(legacy)+ assert ok and reasons == [], reasons++ # A bare metric dict is a baseline `compare` already accepts, so it stays acceptable here.+ ok, reasons = report.comparable_to({"choice_accuracy": 0.8})+ assert ok and reasons == [], reasons+++def test_a_comparing_report_does_not_read_a_shape_it_does_not_know():+ # `research/evals/act_head_eval.py` publishes a report under its own schema tag. A consumer+ # that cannot read it must be told, not left to compare a different metric space.+ foreign = {"schema": "laya-evals-report/act-head-eval/1",+ "overall": {"choice_accuracy": 0.8}, "cases": [], "slices": {}}+ ok, reasons = _identified(dataset_sha256="a" * 64).comparable_to(foreign)+ assert not ok and any("act-head-eval" in reason for reason in reasons), reasons+++def test_docs_document_the_run_identity_and_the_refusal():+ """The docs table is derived against the code, not transcribed, so it cannot drift.++ `docs/evals.md` is what an operator reads before deciding what to record with a policy, so+ every key the report actually writes has to appear there.+ """+ from pathlib import Path++ page = (Path(__file__).resolve().parent.parent / "docs" / "evals.md").read_text(encoding="utf-8")+ dataset = Dataset([Example("s1", Q, {"intent": "a"})])+ report = evaluate(StubRunner({"s1": {"intent": choice_answer("a")}}), dataset)+ for key in report.config:+ if key != "timing":+ assert "`%s`" % key in page, key+ # The keys the CLI adds, and the programmatic entry points, are part of the same contract.+ for key in ("dataset_sha256", "thresholds", "revisions"):+ assert "`%s`" % key in page, key+ for name in ("REPORT_SCHEMA", "questions_fingerprint", "file_fingerprint", "comparable_to"):+ assert name in page, name+ assert "not comparable" in page, "the refusal the gate prints is documented"++# --------------------------------------------------------------- CLIdef _write_dataset(tmp_path, rows):path = tmp_path / "dataset.jsonl"@@ -635,6 +772,97 @@ def test_cli_rejects_a_malformed_tolerance():evals_cli._parse_pairs(["choice_accuracy"])+def test_cli_run_records_the_run_identity(monkeypatch, tmp_path):+ """The `--json` artifact is what a reviewer reads, so it has to say which run produced it.++ `docs/evals.md` tells the reviewer to commit the dataset and a reviewed baseline together and+ read the result as a diff. That only works if the report carries the dataset's bytes and the+ question schema, not just the path, and if it records the gate the numbers came out of.+ """+ from laya import evals_cli++ _patch_router(monkeypatch)+ dataset = _write_dataset(tmp_path, RUN_ROWS)+ out = tmp_path / "report.json"+ assert evals_cli.main(["run", dataset, "--min-accuracy", "0.5", "--max-ece", "0.9",+ "--tolerance", "choice_accuracy=0.02", "--json", str(out)]) == 0+ config = json.loads(out.read_text(encoding="utf-8"))["config"]+ assert config["schema"] == evals.REPORT_SCHEMA+ assert config["laya_version"] == laya.__version__+ assert config["dataset"] == dataset, "the path stays a name; the hash carries the bytes"+ assert config["dataset_sha256"] == evals.file_fingerprint(dataset)+ assert config["questions_sha256"] == evals.questions_fingerprint(evals.Dataset.from_jsonl(dataset))+ assert config["thresholds"]["min"] == {"choice_accuracy": 0.5}+ assert config["thresholds"]["max"] == {"ece": 0.9}+ assert config["thresholds"]["baseline_tolerance"] == {"choice_accuracy": 0.02}+++def test_cli_run_refuses_a_baseline_from_a_different_dataset(monkeypatch, tmp_path, capsys):+ """The gate that a promotion decision rests on: two different runs cannot pass as one.++ Both runs score the same rows with the same stub, so every metric matches and the baseline+ comparison passes. But the baseline was recorded against different bytes, so the number being+ promoted was never measured on the data the gate claims to have validated. A pass here is+ arithmetic, not evidence.+ """+ from laya import evals_cli++ _patch_router(monkeypatch)+ original = _write_dataset(tmp_path, RUN_ROWS)+ baseline_out = tmp_path / "baseline.json"+ assert evals_cli.main(["run", original, "--json", str(baseline_out)]) == 0+ capsys.readouterr()++ # The same file name, one extra row: exactly what an edited dataset looks like in a diff.+ edited = tmp_path / "edited.jsonl"+ edited.write_text("\n".join(json.dumps(row) for row in RUN_ROWS)+ + "\n" + json.dumps(RUN_ROWS[0]), encoding="utf-8")+ assert evals_cli.main(["run", str(edited), "--baseline", str(baseline_out),+ "--tolerance", "choice_accuracy=0.05"]) == 1+ err = capsys.readouterr().err+ assert "baseline is not comparable" in err+ assert "dataset_sha256" in err, "the failure names the key a reviewer can check"+++def test_cli_run_still_accepts_a_baseline_with_no_identity(monkeypatch, tmp_path, capsys):+ """Every baseline committed before the identity existed keeps passing exactly as it did.++ The scheduled gate compares against `research/results/eval_english_51_languages.json`, which+ comes from `research/eval/` and has no `config.schema`. A key missing on one side is unknown,+ not a conflict -- otherwise this change would break the repo's own CI on its first run.+ """+ from laya import evals_cli++ _patch_router(monkeypatch)+ dataset = _write_dataset(tmp_path, RUN_ROWS)+ legacy = tmp_path / "legacy.json"+ legacy.write_text(json.dumps({"config": {"dataset": "somewhere/else.jsonl"},+ "overall": {"choice_accuracy": 1.0}}), encoding="utf-8")+ assert evals_cli.main(["run", dataset, "--baseline", str(legacy),+ "--tolerance", "choice_accuracy=0.05"]) == 0+ assert "not comparable" not in capsys.readouterr().err+++def test_cli_compare_refuses_two_different_runs(tmp_path, capsys):+ from laya import evals_cli++ report = {"config": {"schema": evals.REPORT_SCHEMA, "dataset_sha256": "a" * 64,+ "questions_sha256": "q" * 64},+ "overall": {"choice_accuracy": 0.8}, "cases": [], "slices": {}}+ baseline = {"config": {"schema": evals.REPORT_SCHEMA, "dataset_sha256": "b" * 64,+ "questions_sha256": "q" * 64},+ "overall": {"choice_accuracy": 0.8}, "cases": [], "slices": {}}+ (tmp_path / "r.json").write_text(json.dumps(report), encoding="utf-8")+ (tmp_path / "b.json").write_text(json.dumps(baseline), encoding="utf-8")+ # Identical `overall`, and the default tolerance is an exact match, so the metric gate passes.+ assert evals_cli.main(["compare", str(tmp_path / "r.json"),+ "--baseline", str(tmp_path / "b.json")]) == 1+ captured = capsys.readouterr()+ assert "baseline is not comparable" in captured.err and "dataset_sha256" in captured.err+ # The deltas are still printed, so the reviewer sees what moved as well as why it is not a pass.+ assert "diff=" in captured.out and "choice_accuracy" in captured.out++# ------------------------------------------------------------------ CLI run# One score row inside 0.25 of its label, one 0.3 away (inside 0.5 but not 0.25), plus a choice and# an noul row so every default metric is in the report too.diff --git a/tests/test_evals_api.py b/tests/test_evals_api.pyindex 84337f91..6a08a780 100644--- a/tests/test_evals_api.py+++ b/tests/test_evals_api.py@@ -39,9 +39,16 @@ check("default evaluator names",check_true("ScoreWithin names its tolerance", evals.ScoreWithin(0.25).name == "score_within_0.25")# --------------------------------------------------------------- exports / callables-for name in ("Dataset", "Example", "EvalError", "EvalReport", "evaluate", "ece", "assert_regression"):+for name in ("Dataset", "Example", "EvalError", "EvalReport", "evaluate", "ece", "assert_regression",+ "REPORT_SCHEMA", "questions_fingerprint", "file_fingerprint"):check_true("laya.evals.%s exists" % name, hasattr(evals, name))+check("REPORT_SCHEMA", evals.REPORT_SCHEMA, "laya-evals-report/1")+check_true("EvalReport.comparable_to is part of the report contract",+ callable(getattr(evals.EvalReport, "comparable_to", None)))+check("EvalReport fields are unchanged", [f.name for f in dataclasses.fields(evals.EvalReport)],+ ["config", "overall", "slices", "cases"])+# --------------------------------------------------------------- torch stays out of importprobe = subprocess.run([sys.executable, "-c",