Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(evals): a run identity, and a gate that refuses a baseline it cannot match

Bruce-Yiiba9054b23ac8
diff --git a/docs/evals.md b/docs/evals.md
index 57bf788e..e8430636 100644
--- a/docs/evals.md
+++ b/docs/evals.md
@@ -124,6 +124,34 @@ metrics per slice value, so a regression in one language or one question is visi
reading the aggregate. The `model` slice holds the checkpoint that answered each row: the
`Router`'s own choice per request, or the runner's `model` for a runner that does not route.
+## Run identity
+
+`run` records what it measured in the report's `config` block, so the artifact a reviewer reads
+is reviewable on its own:
+
+| key | meaning |
+|---|---|
+| `schema` | the report shape, `laya-evals-report/1`, so a consumer can refuse one it cannot read |
+| `dataset` | the path as typed -- a name, not a hash |
+| `dataset_sha256` | the sha256 of the dataset bytes that were parsed |
+| `questions_sha256` | a fingerprint of the question schema: every question's id, type and `criteria`, over the whole dataset |
+| `laya_version` | the `laya` that computed the numbers |
+| `thresholds` | the gate this run applied: `min`, `max` and `baseline_tolerance` |
+| `revisions` | the commit each checkpoint that answered was loaded from (see [below](#baseline-and-ci-gate)) |
+
+`dataset` is a path, and a path is not an identity: a dataset can be edited in place, moved, or
+refetched under the same name, and a CI cache can hand two runs the same filename and different
+bytes. `questions_sha256` covers what was *asked* rather than how many rows there were, so scoring
+more states on the same questions stays comparable to a baseline, while renaming a `choice` option
+label does not -- `criteria` is the decision space, and the metamorphic checks in
+`research/eval/metamorphic.py` exist because a label rename flips answers. `instructions` is
+excluded on purpose: rewording prose changes the prompt, not the question being asked.
+
+Nothing time-bearing is recorded, so a report is still byte-reproducible for a fixed runner.
+
+`REPORT_SCHEMA`, `questions_fingerprint(dataset)` and `file_fingerprint(path)` are public, so a
+caller driving `laya.evals.evaluate` directly gets the same identity a CLI run does.
+
## Baseline and CI gate
- Keep the dataset, a baseline report (`--json` output you have reviewed), and the tolerances
@@ -133,6 +161,21 @@ reading the aggregate. The `model` slice holds the checkpoint that answered each
drops into CI unchanged. `laya.evals.EvalReport.compare` and `assert_regression` expose the
same logic for tests.
+The metric gate answers "did the numbers move". It cannot answer "were these the same numbers",
+because `compare` reads `overall` and only `overall` -- so a baseline recorded against one dataset
+would pass a candidate scored on another, with identical arithmetic. `EvalReport.comparable_to`
+closes that: it compares `schema`, `dataset_sha256` and `questions_sha256`, and `run --baseline`
+and `compare` still print every delta, then fail with a non-zero exit naming the key and both
+values:
+
+```text
+FAIL: baseline is not comparable: dataset_sha256 (dataset bytes): baseline is <sha>, this run is <sha>
+```
+
+A key missing on either side is *unknown*, not a conflict, so every report written before the
+identity existed keeps comparing exactly as it did. That includes the scheduled gate's baseline
+below, which comes from `research/eval/` and has no `config.schema` at all.
+
Two CI surfaces use this:
- a weight-free job in `.github/workflows/ci.yml` runs `tests/test_evals.py` and
diff --git a/docs/staged-adoption.md b/docs/staged-adoption.md
index 9f3bed9a..eebb51c7 100644
--- a/docs/staged-adoption.md
+++ b/docs/staged-adoption.md
@@ -78,6 +78,13 @@ owner with the policy. Re-evaluate it when those inputs change. Confidence order
not establish that a decision is correct, and high confidence is never execution permission by
itself.
+Four of those six come from the evaluation harness: a `laya-evals run --json` report records the
+checkpoint commit that answered (`config.revisions`), the dataset's bytes and question schema
+(`config.dataset_sha256`, `config.questions_sha256`) and the gate it was scored under
+(`config.thresholds`). Calibration method and owner are the application's to record. See
+[Evaluation harness](evals.md) for the run identity and the comparability check between a baseline
+and a candidate.
+
The README's [Automated Confidence Gating](https://github.com/NandhaKishorM/laya#automated-confidence-gating),
[Calibration](https://github.com/NandhaKishorM/laya#calibration), and [Honest limits](https://github.com/NandhaKishorM/laya#honest-limits) sections
give the existing calibration and confidence context. Keep irreversible or high-cost actions behind
@@ -128,6 +135,8 @@ real request
## See also
- [Prediction hooks](hooks/index.md) — the extension seam for audit, metrics and gating.
+- [Evaluation harness](evals.md) — the run identity a baseline carries and the comparability check
+ between a baseline and a candidate.
- [Hook API reference](hooks/api.md) — `PredictContext` fields and lifecycle events.
- [Tracing](hooks/tracing.md) — `run_id` and span correlation.
- [README: Automated Confidence Gating](https://github.com/NandhaKishorM/laya#automated-confidence-gating) — confidence is a
diff --git a/laya/evals.py b/laya/evals.py
index 050d8c8f..a02243c2 100644
--- a/laya/evals.py
+++ b/laya/evals.py
@@ -10,9 +10,18 @@ checkpoint in tests.
The report is deterministic for a fixed runner: the same dataset produces the same numbers, and
``EvalReport.compare`` turns a baseline into a pass/fail with the per-metric deltas, which is what
the CI gate consumes.
+
+`compare` reads ``overall`` and nothing else, so a run that says nothing about *what it measured*
+makes a gate that can only compare arithmetic. Every report therefore carries a run identity in
+``config`` -- ``schema``, ``questions_sha256``, ``laya_version`` and (from the CLI)
+``dataset_sha256`` and the ``thresholds`` actually applied -- and
+``EvalReport.comparable_to`` refuses a comparison between two runs that are not the same
+measurement. The identity is deliberately free of anything time-bearing, so a report stays
+byte-reproducible for a fixed runner.
"""
from __future__ import annotations
+import hashlib
import inspect
import json
import statistics
@@ -22,6 +31,34 @@ from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple
import numpy as np
+#: The shape of the report this module writes, so a consumer can refuse one it cannot read.
+#: ``research/evals/act_head_eval.py`` publishes a report under its own tag off this prefix.
+REPORT_SCHEMA = "laya-evals-report/1"
+
+#: The keys that decide whether two reports are the same measurement. A key missing
+#: on either side is unknown rather than a conflict, so a baseline committed before these existed
+#: keeps comparing exactly as it did. ``schema`` is also read at the top level, which is where
+#: ``research/evals/act_head_eval.py`` puts its own tag.
+_IDENTITY_KEYS = ("schema", "dataset_sha256", "questions_sha256")
+
+_IDENTITY_LABELS = {
+ "schema": "report schema",
+ "dataset_sha256": "dataset bytes",
+ "questions_sha256": "question schema",
+}
+
+
+def _identity_of(document: Any) -> Dict[str, Any]:
+ """Pull the identity keys out of a report, from ``config`` or from the top level."""
+ if not isinstance(document, dict):
+ return {}
+ config = document.get("config")
+ found = dict(config) if isinstance(config, dict) else {}
+ for key in _IDENTITY_KEYS:
+ if found.get(key) is None and document.get(key) is not None:
+ found[key] = document[key]
+ return found
+
class EvalError(ValueError):
"""A dataset or report is malformed; the message names the row or field."""
@@ -88,6 +125,80 @@ class Dataset:
return cls(examples)
+def _canonical(value: Any) -> str:
+ return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False,
+ default=str)
+
+
+def questions_fingerprint(dataset: "Dataset") -> str:
+ """A stable hash of *what was asked*, over every question in `dataset`.
+
+ This is the question-schema identity `docs/staged-adoption.md` asks an operator to record
+ with the policy, and the part of the run identity a report can compute for itself: the
+ dataset file hash is the CLI's, but the questions are parsed here.
+
+ It is a function of the decision space, not of the rows, so scoring more states on the same
+ questions stays comparable to a baseline. A `choice` option label lives in `criteria` and so
+ counts as a different question -- which is the case that matters, since
+ `research/eval/metamorphic.py` exists because renaming a label flips answers. Free-text
+ `instructions` are excluded: rewording them changes the prompt, not the decision space, and
+ a baseline should survive a copy edit.
+ """
+ schemas = set()
+ for example in dataset.examples:
+ for qid, question in example.questions.items():
+ body = question if isinstance(question, dict) else {}
+ schemas.add(_canonical({"qid": qid, "type": body.get("type"),
+ "criteria": body.get("criteria")}))
+ digest = hashlib.sha256()
+ for schema in sorted(schemas):
+ digest.update(schema.encode("utf-8"))
+ digest.update(b"\n")
+ return digest.hexdigest()
+
+
+def file_fingerprint(path: str) -> str:
+ """The sha256 of a file's bytes -- the dataset identity, where the path is only a name.
+
+ `config.dataset` is the path as typed, and two datasets share a path across a rebase, a CI
+ cache or a colleague's checkout. Raises `EvalError` when the file cannot be read, so a report
+ never carries a hash of something other than the bytes that were parsed.
+ """
+ digest = hashlib.sha256()
+ try:
+ with open(path, "rb") as handle:
+ for block in iter(lambda: handle.read(65536), b""):
+ digest.update(block)
+ except OSError as exc:
+ raise EvalError("%s cannot be read for a fingerprint: %s" % (path, exc)) from exc
+ return digest.hexdigest()
+
+
+def _library_version() -> Optional[str]:
+ """`laya.__version__`, read lazily so this module stays importable on its own.
+
+ `laya/__init__.py` defines the string at import time, so this is a module lookup rather
+ than a load of anything torch-backed -- but it is imported inside the call to keep
+ `laya.evals` free of an import cycle, and a failure here must not fail a run that has
+ already scored every row.
+ """
+ try:
+ import laya
+ except Exception: # pragma: no cover - identity only
+ return None
+ return getattr(laya, "__version__", None)
+
+
+def _run_identity(dataset: "Dataset") -> Dict[str, Any]:
+ """The `config` keys this module can fill in on its own, with no knowledge of the CLI."""
+ identity: Dict[str, Any] = {"schema": REPORT_SCHEMA,
+ "questions_sha256": questions_fingerprint(dataset)}
+ version = _library_version()
+ if version is not None:
+ identity["laya_version"] = version
+ return identity
+
+
# --------------------------------------------------------------------------- evaluators
class Evaluator:
"""Per-answer metric. `score` returns a number, or None when it does not apply."""
@@ -240,6 +351,34 @@ class EvalReport:
lines.append("| %s | %s |" % (value, " | ".join(cells)))
return "\n".join(lines) + "\n"
+ def comparable_to(self, baseline: Dict[str, Any]) -> Tuple[bool, List[str]]:
+ """Is `baseline` the same measurement as this report? Returns (ok, reasons).
+
+ `compare` reads `overall` and only `overall`, so two reports of different datasets,
+ different question schemas or different report shapes produce identical arithmetic and
+ an identical pass. That is the failure `docs/staged-adoption.md` runs at: a gate that
+ cannot tell which experiment produced a number cannot support a promotion decision.
+
+ A key absent on either side is *unknown*, not a conflict, so every baseline committed
+ before the identity existed -- including the scheduled gate's
+ `research/results/eval_english_51_languages.json`, which comes from `research/eval/` and
+ has no `config.schema` -- keeps comparing exactly as it did. Only a key present on both
+ sides with different values refuses the comparison, and each reason names the key and
+ both values so the failure is actionable from the console alone.
+
+ `laya_version` and `thresholds` are recorded in `config` but deliberately not compared
+ here: a patch release must not invalidate a committed baseline.
+ """
+ other = _identity_of(baseline)
+ reasons: List[str] = []
+ for key in _IDENTITY_KEYS:
+ here, there = self.config.get(key), other.get(key)
+ if here is None or there is None or here == there:
+ continue
+ reasons.append("%s (%s): baseline is %s, this run is %s"
+ % (key, _IDENTITY_LABELS[key], there, here))
+ return (not reasons), reasons
+
def compare(self, baseline: Dict[str, Any], tolerances: Optional[Dict[str, float]] = None,
) -> Tuple[bool, Dict[str, Dict[str, float]]]:
"""Compare `overall` to a baseline report's `overall`. Returns (ok, deltas).
@@ -461,6 +600,10 @@ def evaluate(runner: Any, dataset: Dataset, evaluators: Optional[Sequence[Evalua
report.overall["latency_p50_ms"], report.overall["latency_p95_ms"] = _percentiles(waits)
(report.overall["cost_per_decision_p50_ms"],
report.overall["cost_per_decision_p95_ms"]) = _percentiles(shares)
+ # The run identity, stamped here rather than left to the caller: everything below is
+ # computable from `dataset`, so a programmatic `evaluate` is as identifiable as a CLI run.
+ # The CLI adds `dataset_sha256` and `thresholds`, which need the file path and the gate.
+ report.config = dict(report.config, **_run_identity(dataset))
report.config = dict(report.config, timing={
"latency_metric": "per request: the wall time of the call that returned it, unsplit",
"cost_metric": "per decision: that call divided by its own chunk size",
diff --git a/laya/evals_cli.py b/laya/evals_cli.py
index d2a3a8ff..e2ed02af 100644
--- a/laya/evals_cli.py
+++ b/laya/evals_cli.py
@@ -204,6 +204,21 @@ def _print_deltas(deltas: Dict[str, Dict[str, Any]]) -> None:
% (metric, delta["baseline"], delta["value"], delta["diff"], delta["tolerance"]))
+def _comparability_failure(report: evals.EvalReport, baseline: Dict[str, Any]) -> Optional[str]:
+ """The refusal to compare two runs that are not the same measurement, as one failure line.
+
+ The metric gate answers "did the numbers move"; it cannot answer "were these the same
+ numbers", because `EvalReport.compare` reads `overall` and only `overall`. So a baseline
+ recorded against one dataset passes a candidate scored on another, with identical
+ arithmetic. The gate still prints its deltas -- the reviewer wants to see both facts -- but
+ an incomparable pair is not a pass.
+ """
+ ok, reasons = report.comparable_to(baseline)
+ if ok:
+ return None
+ return "baseline is not comparable: " + "; ".join(reasons)
+
+
def _cmd_validate(args) -> int:
dataset = evals.Dataset.from_jsonl(args.dataset)
questions = sorted({qid for example in dataset.examples for qid in example.questions})
@@ -262,7 +277,12 @@ def _cmd_run(args) -> int:
# download.
raise EvalError(str(exc)) from exc
runner = RouterRunner(router)
- config = {"dataset": args.dataset, "model": args.model, "device": args.device}
+ config = {"dataset": args.dataset, "model": args.model, "device": args.device,
+ # The path above is the name, not the data. Hashing the bytes here is what makes
+ # a committed baseline reviewable: the dataset can be edited in place, moved or
+ # refetched under the same name, and a reviewer comparing two reports needs the
+ # report -- not a file mtime -- to say so.
+ "dataset_sha256": evals.file_fingerprint(args.dataset)}
if args.onnx:
config["onnx"] = args.onnx
if extra:
@@ -286,14 +306,25 @@ def _cmd_run(args) -> int:
mins[key] = args.min_accuracy
if args.max_ece is not None:
maxs["ece"] = args.max_ece
+ tolerances = _parse_pairs(args.tolerance)
+ # The gate this run actually applied, recorded with the numbers it produced. A baseline whose
+ # numbers came from a different gate is a different claim, and `docs/staged-adoption.md`
+ # asks for exactly this alongside the checkpoint version and the evaluation set.
+ report.config = dict(report.config,
+ thresholds={"min": mins, "max": maxs, "baseline_tolerance": tolerances})
failures = _check_thresholds(report.overall, mins, maxs)
if args.baseline:
baseline = _load_report(args.baseline)
- ok, deltas = report.compare(baseline, _parse_pairs(args.tolerance))
+ ok, deltas = report.compare(baseline, tolerances)
_print_deltas(deltas)
if not ok:
failures.append("baseline comparison failed")
+ # Checked after the deltas are printed, so a reviewer sees what moved as well as why the
+ # two runs are not the same experiment.
+ refusal = _comparability_failure(report, baseline)
+ if refusal:
+ failures.append(refusal)
for name in sorted(report.overall):
print("%-18s %.4f" % (name, report.overall[name]))
@@ -322,7 +353,13 @@ def _cmd_compare(args) -> int:
baseline = _load_report(args.baseline)
ok, deltas = report.compare(baseline, _parse_pairs(args.tolerance))
_print_deltas(deltas)
- return 0 if ok else 1
+ # `compare` is the offline re-check of a report a reviewer already read, so the same
+ # comparability refusal has to apply here: a saved report that cannot say which experiment
+ # it came from must not pass a gate either.
+ refusal = _comparability_failure(report, baseline)
+ if refusal:
+ print("FAIL: " + refusal, file=sys.stderr)
+ return 0 if ok and not refusal else 1
def main(argv: Optional[Sequence[str]] = None) -> int:
diff --git a/tests/test_evals.py b/tests/test_evals.py
index ce0db32a..2c227aca 100644
--- a/tests/test_evals.py
+++ b/tests/test_evals.py
@@ -7,6 +7,8 @@ import re
import pytest
+import laya
+from laya import evals
from laya.evals import (
ChoiceAccuracy,
Dataset,
@@ -586,6 +588,141 @@ def test_default_evaluators_cover_the_three_types():
assert {"choice_accuracy", "noul_accuracy", "score_mae", "mean_confidence"} <= names
+# ------------------------------------------------------- run identity / comparability
+def _identified(**config):
+ return EvalReport(config={"schema": evals.REPORT_SCHEMA, **config},
+ overall={"choice_accuracy": 0.8})
+
+
+def test_evaluate_records_what_it_measured():
+ """`config` is the artifact a reviewer reads, so it has to say which run produced it.
+
+ `config.dataset` is the path as typed. Two datasets share a path across a rebase, a CI
+ cache or a colleague's checkout, and `docs/evals.md` tells the reviewer to commit the
+ dataset and the baseline together and read the result as a diff -- which is only reviewable
+ if the report says which bytes were scored.
+ """
+ dataset = Dataset([Example("s1", Q, {"intent": "a"}), Example("s2", Q, {"intent": "b"})])
+ report = evaluate(StubRunner({"s1": {"intent": choice_answer("a")},
+ "s2": {"intent": choice_answer("a")}}), dataset)
+ config = report.config
+ assert config["schema"] == evals.REPORT_SCHEMA
+ assert config["laya_version"] == laya.__version__
+ assert len(config["questions_sha256"]) == 64, "the question schema is part of the identity"
+
+ # Determinism: the identity must carry nothing time-bearing, or a re-run of the same dataset
+ # stops producing the same report and the artifact a reviewer diffs becomes noise. The
+ # `*_ms` metrics are wall clock and always differ; only the config block is asserted here.
+ again = evaluate(StubRunner({"s1": {"intent": choice_answer("a")},
+ "s2": {"intent": choice_answer("a")}}), dataset)
+ assert {k: v for k, v in config.items() if k != "timing"} == \
+ {k: v for k, v in again.config.items() if k != "timing"}
+
+
+def test_questions_fingerprint_follows_the_question_not_the_row():
+ """The fingerprint covers the question schema, so it is a function of what was asked.
+
+ A run that scores a different dataset must not collide with a baseline, and a run that
+ scores the same questions on more rows must still be comparable to it. Renaming a `choice`
+ label is a different question, not a different row: `research/eval/metamorphic.py` exists
+ because option order and label rename flip answers.
+ """
+ base = Dataset([Example("s1", Q, {"intent": "a"})])
+ more_rows = Dataset([Example("s1", Q, {"intent": "a"}), Example("s2", Q, {"intent": "a"})])
+ assert evals.questions_fingerprint(base) == evals.questions_fingerprint(more_rows)
+
+ renamed = Dataset([Example("s1", {"intent": {"type": "choice", "instructions": "?",
+ "criteria": {"a": "x", "c": "y"}}},
+ {"intent": "a"})])
+ assert evals.questions_fingerprint(renamed) != evals.questions_fingerprint(base)
+
+ retyped = Dataset([Example("s1", {"flag": {"type": "noul", "instructions": "?"}},
+ {"flag": True})])
+ assert evals.questions_fingerprint(retyped) != evals.questions_fingerprint(base)
+
+
+def test_comparable_to_refuses_two_different_runs():
+ """`compare` reads `overall` only, so without this a gate passes two different experiments.
+
+ Both reports score 0.8, so the arithmetic is identical -- but they are not the same
+ measurement, and a promotion decision made on that is unfalsifiable. Every field
+ `docs/staged-adoption.md` tells the operator to record with the policy has to be able to
+ stop the comparison.
+ """
+ baseline = _identified(dataset_sha256="a" * 64, questions_sha256="q" * 64)
+ same = _identified(dataset_sha256="a" * 64, questions_sha256="q" * 64)
+ ok, reasons = same.comparable_to(baseline.to_json())
+ assert ok and reasons == [], reasons
+
+ for field, other in (("schema", "laya-evals-report/act-head-eval/1"),
+ ("dataset_sha256", "b" * 64),
+ ("questions_sha256", "z" * 64)):
+ candidate = _identified(dataset_sha256="a" * 64, questions_sha256="q" * 64)
+ candidate.config[field] = other
+ ok, reasons = candidate.comparable_to(baseline.to_json())
+ assert not ok, field
+ # The key is named, so the failure is greppable against the `config` a reviewer reads.
+ assert any(field in reason for reason in reasons), (field, reasons)
+
+ # The reason names both values, or a reviewer still has to open two files to act on it.
+ swapped = _identified(dataset_sha256="c" * 64, questions_sha256="q" * 64)
+ ok, reasons = swapped.comparable_to(baseline.to_json())
+ assert "c" * 64 in " ".join(reasons) and "a" * 64 in " ".join(reasons)
+
+
+def test_comparable_to_ignores_what_it_cannot_see():
+ """A field missing on one side is unknown, not a conflict.
+
+ Every baseline committed before this existed -- including the scheduled gate's
+ `research/results/eval_english_51_languages.json`, which comes from `research/eval/` and
+ has no `config.schema` at all -- has to keep comparing exactly as it did.
+ """
+ legacy = {"overall": {"choice_accuracy": 0.8}, "config": {"dataset": "old.jsonl"}}
+ report = _identified(dataset_sha256="a" * 64, questions_sha256="q" * 64)
+ ok, reasons = report.comparable_to(legacy)
+ assert ok and reasons == [], reasons
+ assert report.compare(legacy, {"choice_accuracy": 0.0})[0], "the metric gate still runs"
+
+ # Two reports that both predate the identity fields are as comparable as they ever were.
+ ok, reasons = EvalReport(config={"dataset": "a.jsonl"}).comparable_to(legacy)
+ assert ok and reasons == [], reasons
+
+ # A bare metric dict is a baseline `compare` already accepts, so it stays acceptable here.
+ ok, reasons = report.comparable_to({"choice_accuracy": 0.8})
+ assert ok and reasons == [], reasons
+
+
+def test_a_comparing_report_does_not_read_a_shape_it_does_not_know():
+ # `research/evals/act_head_eval.py` publishes a report under its own schema tag. A consumer
+ # that cannot read it must be told, not left to compare a different metric space.
+ foreign = {"schema": "laya-evals-report/act-head-eval/1",
+ "overall": {"choice_accuracy": 0.8}, "cases": [], "slices": {}}
+ ok, reasons = _identified(dataset_sha256="a" * 64).comparable_to(foreign)
+ assert not ok and any("act-head-eval" in reason for reason in reasons), reasons
+
+
+def test_docs_document_the_run_identity_and_the_refusal():
+ """The docs table is derived against the code, not transcribed, so it cannot drift.
+
+ `docs/evals.md` is what an operator reads before deciding what to record with a policy, so
+ every key the report actually writes has to appear there.
+ """
+ from pathlib import Path
+
+ page = (Path(__file__).resolve().parent.parent / "docs" / "evals.md").read_text(encoding="utf-8")
+ dataset = Dataset([Example("s1", Q, {"intent": "a"})])
+ report = evaluate(StubRunner({"s1": {"intent": choice_answer("a")}}), dataset)
+ for key in report.config:
+ if key != "timing":
+ assert "`%s`" % key in page, key
+ # The keys the CLI adds, and the programmatic entry points, are part of the same contract.
+ for key in ("dataset_sha256", "thresholds", "revisions"):
+ assert "`%s`" % key in page, key
+ for name in ("REPORT_SCHEMA", "questions_fingerprint", "file_fingerprint", "comparable_to"):
+ assert name in page, name
+ assert "not comparable" in page, "the refusal the gate prints is documented"
+
+
# --------------------------------------------------------------- CLI
def _write_dataset(tmp_path, rows):
path = tmp_path / "dataset.jsonl"
@@ -635,6 +772,97 @@ def test_cli_rejects_a_malformed_tolerance():
evals_cli._parse_pairs(["choice_accuracy"])
+def test_cli_run_records_the_run_identity(monkeypatch, tmp_path):
+ """The `--json` artifact is what a reviewer reads, so it has to say which run produced it.
+
+ `docs/evals.md` tells the reviewer to commit the dataset and a reviewed baseline together and
+ read the result as a diff. That only works if the report carries the dataset's bytes and the
+ question schema, not just the path, and if it records the gate the numbers came out of.
+ """
+ from laya import evals_cli
+
+ _patch_router(monkeypatch)
+ dataset = _write_dataset(tmp_path, RUN_ROWS)
+ out = tmp_path / "report.json"
+ assert evals_cli.main(["run", dataset, "--min-accuracy", "0.5", "--max-ece", "0.9",
+ "--tolerance", "choice_accuracy=0.02", "--json", str(out)]) == 0
+ config = json.loads(out.read_text(encoding="utf-8"))["config"]
+ assert config["schema"] == evals.REPORT_SCHEMA
+ assert config["laya_version"] == laya.__version__
+ assert config["dataset"] == dataset, "the path stays a name; the hash carries the bytes"
+ assert config["dataset_sha256"] == evals.file_fingerprint(dataset)
+ assert config["questions_sha256"] == evals.questions_fingerprint(evals.Dataset.from_jsonl(dataset))
+ assert config["thresholds"]["min"] == {"choice_accuracy": 0.5}
+ assert config["thresholds"]["max"] == {"ece": 0.9}
+ assert config["thresholds"]["baseline_tolerance"] == {"choice_accuracy": 0.02}
+
+
+def test_cli_run_refuses_a_baseline_from_a_different_dataset(monkeypatch, tmp_path, capsys):
+ """The gate that a promotion decision rests on: two different runs cannot pass as one.
+
+ Both runs score the same rows with the same stub, so every metric matches and the baseline
+ comparison passes. But the baseline was recorded against different bytes, so the number being
+ promoted was never measured on the data the gate claims to have validated. A pass here is
+ arithmetic, not evidence.
+ """
+ from laya import evals_cli
+
+ _patch_router(monkeypatch)
+ original = _write_dataset(tmp_path, RUN_ROWS)
+ baseline_out = tmp_path / "baseline.json"
+ assert evals_cli.main(["run", original, "--json", str(baseline_out)]) == 0
+ capsys.readouterr()
+
+ # The same file name, one extra row: exactly what an edited dataset looks like in a diff.
+ edited = tmp_path / "edited.jsonl"
+ edited.write_text("\n".join(json.dumps(row) for row in RUN_ROWS)
+ + "\n" + json.dumps(RUN_ROWS[0]), encoding="utf-8")
+ assert evals_cli.main(["run", str(edited), "--baseline", str(baseline_out),
+ "--tolerance", "choice_accuracy=0.05"]) == 1
+ err = capsys.readouterr().err
+ assert "baseline is not comparable" in err
+ assert "dataset_sha256" in err, "the failure names the key a reviewer can check"
+
+
+def test_cli_run_still_accepts_a_baseline_with_no_identity(monkeypatch, tmp_path, capsys):
+ """Every baseline committed before the identity existed keeps passing exactly as it did.
+
+ The scheduled gate compares against `research/results/eval_english_51_languages.json`, which
+ comes from `research/eval/` and has no `config.schema`. A key missing on one side is unknown,
+ not a conflict -- otherwise this change would break the repo's own CI on its first run.
+ """
+ from laya import evals_cli
+
+ _patch_router(monkeypatch)
+ dataset = _write_dataset(tmp_path, RUN_ROWS)
+ legacy = tmp_path / "legacy.json"
+ legacy.write_text(json.dumps({"config": {"dataset": "somewhere/else.jsonl"},
+ "overall": {"choice_accuracy": 1.0}}), encoding="utf-8")
+ assert evals_cli.main(["run", dataset, "--baseline", str(legacy),
+ "--tolerance", "choice_accuracy=0.05"]) == 0
+ assert "not comparable" not in capsys.readouterr().err
+
+
+def test_cli_compare_refuses_two_different_runs(tmp_path, capsys):
+ from laya import evals_cli
+
+ report = {"config": {"schema": evals.REPORT_SCHEMA, "dataset_sha256": "a" * 64,
+ "questions_sha256": "q" * 64},
+ "overall": {"choice_accuracy": 0.8}, "cases": [], "slices": {}}
+ baseline = {"config": {"schema": evals.REPORT_SCHEMA, "dataset_sha256": "b" * 64,
+ "questions_sha256": "q" * 64},
+ "overall": {"choice_accuracy": 0.8}, "cases": [], "slices": {}}
+ (tmp_path / "r.json").write_text(json.dumps(report), encoding="utf-8")
+ (tmp_path / "b.json").write_text(json.dumps(baseline), encoding="utf-8")
+ # Identical `overall`, and the default tolerance is an exact match, so the metric gate passes.
+ assert evals_cli.main(["compare", str(tmp_path / "r.json"),
+ "--baseline", str(tmp_path / "b.json")]) == 1
+ captured = capsys.readouterr()
+ assert "baseline is not comparable" in captured.err and "dataset_sha256" in captured.err
+ # The deltas are still printed, so the reviewer sees what moved as well as why it is not a pass.
+ assert "diff=" in captured.out and "choice_accuracy" in captured.out
+
+
# ------------------------------------------------------------------ CLI run
# One score row inside 0.25 of its label, one 0.3 away (inside 0.5 but not 0.25), plus a choice and
# an noul row so every default metric is in the report too.
diff --git a/tests/test_evals_api.py b/tests/test_evals_api.py
index 84337f91..6a08a780 100644
--- a/tests/test_evals_api.py
+++ b/tests/test_evals_api.py
@@ -39,9 +39,16 @@ check("default evaluator names",
check_true("ScoreWithin names its tolerance", evals.ScoreWithin(0.25).name == "score_within_0.25")
# --------------------------------------------------------------- exports / callables
-for name in ("Dataset", "Example", "EvalError", "EvalReport", "evaluate", "ece", "assert_regression"):
+for name in ("Dataset", "Example", "EvalError", "EvalReport", "evaluate", "ece", "assert_regression",
+ "REPORT_SCHEMA", "questions_fingerprint", "file_fingerprint"):
check_true("laya.evals.%s exists" % name, hasattr(evals, name))
+check("REPORT_SCHEMA", evals.REPORT_SCHEMA, "laya-evals-report/1")
+check_true("EvalReport.comparable_to is part of the report contract",
+ callable(getattr(evals.EvalReport, "comparable_to", None)))
+check("EvalReport fields are unchanged", [f.name for f in dataclasses.fields(evals.EvalReport)],
+ ["config", "overall", "slices", "cases"])
+
# --------------------------------------------------------------- torch stays out of import
probe = subprocess.run(
[sys.executable, "-c",