Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(evals): compare student agreement with repeated teacher decisions

xp5aae58cf00af
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 28d71a39..4a7bfe7e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -164,7 +164,7 @@ jobs:
pip install -e . pytest
- name: Eval harness tests
- run: python -m pytest tests/test_evals.py -q
+ run: python -m pytest tests/test_evals.py tests/test_evals_agreement.py -q
- name: Shortlist attribution tests
run: python -m pytest tests/test_evals_shortlist.py -q
diff --git a/docs/evals.md b/docs/evals.md
index 9fa7351d..dcba75b8 100644
--- a/docs/evals.md
+++ b/docs/evals.md
@@ -29,6 +29,117 @@ Exit codes: `0` on success, `1` when a threshold or a baseline tolerance fails,
usage error. `run` prints the overall metrics and any requested slices to stdout, and writes
the full report and a Markdown summary when `--json` / `--markdown` are given.
+## Comparing a student with repeated teacher decisions
+
+When a teacher gives different answers to the same input, a single saved answer
+does not describe its repeatability. `compare_agreement` compares a student and
+an independent teacher repeat against one fixed reference teacher observation.
+It takes three saved runs and needs no model weights or teacher service:
+
+```bash
+laya-evals agreement reference.json repeat.json student.json \
+ --bootstrap-samples 2000 --seed 0 --json agreement.json
+```
+
+Without `--json`, the command writes the report to stdout. `--seed` controls only
+the bootstrap, not model generation. Success exits `0`; malformed or misaligned
+runs exit `2`. This command does not apply a replacement or quality gate.
+
+Each input is a JSON object with the following shape:
+
+```json
+{
+ "schema": "laya-agreement-run/1",
+ "run_id": "teacher-reference",
+ "provenance": {
+ "model": "teacher-checkpoint",
+ "revision": null,
+ "settings": {"temperature": 0.7, "prompt_template": "Choose one label."},
+ "seed": 42
+ },
+ "cases": [{
+ "id": "ticket-001",
+ "state": "I was charged twice.",
+ "questions": {"route": {
+ "type": "choice",
+ "instructions": "Select the team responsible for this request.",
+ "criteria": ["billing", "support"]
+ }},
+ "answers": {"route": "billing"},
+ "language": "en",
+ "tags": ["duplicate-charge"]
+ }]
+}
+```
+
+Use a distinct `run_id` for each observation run. All three runs must contain
+exactly the same unique case IDs, states, question definitions, language and tags;
+row order may differ. A state must be non-null JSON. Question definitions preserve option order and accept
+non-empty criteria dictionaries or lists of distinct string labels. Answers must
+cover every question. A student answer may be `null` for abstention; teacher
+answers must be labels. Missing cases, unknown labels, non-finite JSON values and
+duplicate JSON object keys in CLI inputs are errors, rather than omitted samples.
+
+The two teacher runs must declare identical model, revision and settings. Put
+sampling parameters and the prompt template in `settings`, and repetition seeds
+outside it. Other metadata, such as the dataset revision and generation script,
+is preserved. Record immutable model revisions when available; explicitly use
+`null` when unknown. The report flags unknown revisions but cannot verify that
+the recorded provenance is accurate or that the teacher calls were independent.
+Input object order is significant, matching Laya's serialization of structured
+states and questions; keep it identical across the runs.
+
+To run a local student after validating the teacher records:
+
+```python
+import json
+import laya
+from laya.evals import evaluate_agreement
+
+with open("reference.json", encoding="utf-8") as handle:
+ reference = json.load(handle)
+with open("repeat.json", encoding="utf-8") as handle:
+ repeat = json.load(handle)
+revision = "YOUR_CHECKPOINT_COMMIT"
+with laya.Router(device="cpu", revision=revision) as runner:
+ report = evaluate_agreement(
+ runner, reference, repeat, model="english",
+ student_provenance={
+ "model": "convaiinnovations/laya", "revision": revision,
+ "settings": {"device": "cpu"},
+ },
+ )
+print(report["overall"])
+```
+
+The runner uses the same `predict(state, questions, model=...)` protocol as the
+labelled harness (for example, `laya.Router`). Inputs are
+copied before inference. Runner failures propagate; invalid or incomplete outputs
+raise `EvalError`. A choice flagged `abstention="abstained"` becomes a null
+observation, while its underlying choice must still be a valid label.
+
+The separate `laya-agreement-report/1` report records provenance, input hashes,
+individual paired decisions and rates sliced by question ID, language and tag.
+For every decision, teacher repeat agreement is `repeat == reference`, student
+agreement is `student == reference`, and the paired delta is their difference.
+All rates use the full decision count: abstentions contribute zero agreement and
+reduce `student_coverage`. The 95% percentile interval resamples whole case IDs,
+keeping their questions together, and recomputes the ratio of decision sums. It
+is deterministic for fixed inputs, seed and bootstrap count (100–100000). With
+fewer than two cases the interval is `null`. Slice rates have no intervals.
+
+Agreement is not accuracy, and teacher repeatability is not a theoretical ceiling.
+Both models can agree on a wrong answer. The interval describes variation across
+the supplied cases with their saved observations; it does not capture additional
+teacher generations. An interval containing zero does not establish equivalence.
+Inspect slices and individual decisions for label skew, and use labelled evals
+for correctness or calibration claims.
+
+[`examples/evals/teacher_agreement.py`](https://github.com/NandhaKishorM/laya/blob/main/examples/evals/teacher_agreement.py)
+replays two published teacher observations from a pinned public dataset against
+a local Laya checkpoint, saving the three runs and their report. Its small fixed
+sample demonstrates the workflow, not deployment readiness.
+
## Attributing shortlist errors
For a labelled high-cardinality choice set, `laya.evals_shortlist.evaluate_shortlist`
diff --git a/examples/evals/teacher_agreement.py b/examples/evals/teacher_agreement.py
new file mode 100644
index 00000000..5dc50d86
--- /dev/null
+++ b/examples/evals/teacher_agreement.py
@@ -0,0 +1,226 @@
+"""Compare a local Laya checkpoint with two published teacher observations.
+
+From a checkout::
+
+ python -m pip install -e . pyarrow
+ python examples/evals/teacher_agreement.py --output-dir /tmp/laya-agreement
+ laya-evals agreement /tmp/laya-agreement/reference.json \
+ /tmp/laya-agreement/repeat.json /tmp/laya-agreement/student.json
+
+The first 32 test rows and teacher repeats 0/1 are fixed before student inference.
+The parquet is downloaded once, checked against its SHA256, and kept in the output
+directory. Pass --parquet to reuse a downloaded copy. Only the example needs pyarrow.
+The pinned English Laya checkpoint runs locally on CPU; no teacher API is called.
+
+Data credit: LocalGate's MIT-licensed localgate/mmlu-pro-closed dataset, derived
+from TIGER-Lab/MMLU-Pro. The publisher's protocol and generation manifest are at:
+https://huggingface.co/datasets/localgate/mmlu-pro-closed/blob/de3c3d7bd8870e6ec48f03284f3e48042bae9ecf/protocol.md
+https://huggingface.co/datasets/localgate/mmlu-pro-closed/blob/de3c3d7bd8870e6ec48f03284f3e48042bae9ecf/provenance/labels_manifest.json
+
+Teacher picks are real saved generations, not reconstructed from aggregate scores.
+The publisher did not record the teacher's immutable model revision; null preserves
+that uncertainty. Laya receives the same question and ordered options as typed
+inputs, not the teacher's chain-of-thought prompt. This small panel demonstrates
+the evaluation pipeline: agreement is neither accuracy nor a performance ceiling.
+"""
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import sys
+import urllib.request
+from copy import deepcopy
+from pathlib import Path
+
+DATASET = "localgate/mmlu-pro-closed"
+DATASET_REVISION = "de3c3d7bd8870e6ec48f03284f3e48042bae9ecf"
+PARQUET_SHA256 = "7b1ee7627f19a6ac322aa677a6483549e5ea6e136b6992dba6da0f8a2227176f"
+PARQUET_URL = (
+ f"https://huggingface.co/datasets/{DATASET}/resolve/{DATASET_REVISION}"
+ "/data/test-00000-of-00001.parquet"
+)
+PROTOCOL_SHA256 = "6f0ac97a38225f523a61803d25d6868e90813ef6873a8170a7e2f2a9e056ef9b"
+MODEL = "convaiinnovations/laya"
+MODEL_REVISION = "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851"
+COUNT = 32
+PROMPT_TEMPLATE = (
+ "<bos><|turn>system\n"
+ "The following are multiple choice questions (with answers) about {category}. "
+ 'Think step by step and then finish your answer with "the answer is (X)" '
+ "where X is the correct letter choice.<turn|>\n"
+ "<|turn>user\n"
+ "Question:\n{question}\nOptions:\n{ordered_options}\n"
+ "Answer: Let's think step by step.<turn|>\n"
+ "<|turn>model\n"
+)
+
+
+def write_json(path: Path, value: object) -> None:
+ with path.open("w", encoding="utf-8", newline="\n") as handle:
+ json.dump(value, handle, ensure_ascii=False, indent=2, allow_nan=False)
+ handle.write("\n")
+
+
+def teacher_runs(parquet: Path) -> tuple[dict, dict]:
+ import pyarrow.parquet as pq
+
+ if parquet.exists():
+ data = parquet.read_bytes()
+ else:
+ request = urllib.request.Request(
+ PARQUET_URL, headers={"User-Agent": "Laya-agreement-example"}
+ )
+ with urllib.request.urlopen(request, timeout=120) as response:
+ data = response.read()
+ actual = hashlib.sha256(data).hexdigest()
+ if actual != PARQUET_SHA256:
+ raise ValueError(f"parquet SHA256 mismatch: {actual}")
+ if not parquet.exists():
+ parquet.parent.mkdir(parents=True, exist_ok=True)
+ parquet.write_bytes(data)
+
+ table = pq.read_table(parquet)
+ if table.num_rows < COUNT:
+ raise ValueError(f"expected at least {COUNT} rows in the pinned parquet")
+ rows = table.slice(0, COUNT).to_pylist()
+ if len({row["question_id"] for row in rows}) != COUNT:
+ raise ValueError("duplicate question_id in the fixed selection")
+ runs = []
+ for repeat, seed in ((0, 42), (1, 43)):
+ cases = []
+ for row in rows:
+ criteria = dict(zip("ABCDEFGHIJ", row["options"]))
+ if row["k"] != 5 or len(row["picks"]) != 5:
+ raise ValueError(f"expected five teacher observations for {row['question_id']}")
+ if row["picks"][repeat] not in criteria:
+ raise ValueError(
+ f"invalid teacher choice for {row['question_id']}, repeat {repeat}; "
+ "the fixed selection is not filtered or backfilled"
+ )
+ cases.append({
+ "id": str(row["question_id"]),
+ "state": row["question"],
+ "questions": {"answer": {
+ "type": "choice",
+ "instructions": "Choose the correct answer to the question.",
+ "criteria": criteria,
+ }},
+ "answers": {"answer": row["picks"][repeat]},
+ "tags": [row["category"]],
+ "language": "en",
+ })
+ runs.append({
+ "schema": "laya-agreement-run/1",
+ "run_id": f"gemma-seed-{seed}",
+ "provenance": {
+ "model": "google/gemma-4-E2B-it",
+ "revision": None,
+ "settings": {
+ "do_sample": True,
+ "temperature": 1.0,
+ "top_p": 0.95,
+ "top_k": 64,
+ "max_gen_toks": 6144,
+ "until": ["Question:"],
+ "thinking": False,
+ "num_fewshot": 0,
+ "prompt_template": PROMPT_TEMPLATE,
+ "option_line_template": "{letter}. {option}",
+ "protocol_sha256": PROTOCOL_SHA256,
+ },
+ "seed": seed,
+ "source": {
+ "dataset": DATASET,
+ "revision": DATASET_REVISION,
+ "split": "test",
+ "selection": "first 32 rows in the pinned parquet",
+ "file_sha256": PARQUET_SHA256,
+ "teacher_revision_note": "not recorded in the publisher's manifest",
+ },
+ },
+ "cases": cases,
+ })
+ return runs[0], runs[1]
+
+
+class LocalRunner:
+ """Adapt one loaded Agent to the evaluation runner's model keyword."""
+
+ def __init__(self, agent):
+ self.agent = agent
+ self.calls = 0
+
+ def predict(self, state, questions, *, model=None):
+ if model is not None:
+ raise ValueError("this example uses a single fixed checkpoint")
+ result = self.agent.predict(state, questions, lang="en")
+ self.calls += 1
+ if self.calls % 8 == 0:
+ print(f"Evaluated {self.calls}/{COUNT} cases", file=sys.stderr, flush=True)
+ return result
+
+
+def student_run(reference: dict, report: dict) -> dict:
+ """Persist actual report observations for offline CLI replay, without inference."""
+ cases = deepcopy(reference["cases"])
+ by_id = {case["id"]: case for case in cases}
+ for case in cases:
+ case["answers"] = {}
+ for record in report["cases"]:
+ by_id[record["case_id"]]["answers"][record["qid"]] = record["student"]
+ return {
+ "schema": "laya-agreement-run/1",
+ "run_id": report["config"]["run_ids"]["student"],
+ "provenance": report["config"]["provenance"]["student"],
+ "cases": cases,
+ }
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--output-dir", type=Path, required=True)
+ parser.add_argument("--parquet", type=Path, help="reuse the pinned test parquet")
+ args = parser.parse_args()
+ args.output_dir.mkdir(parents=True, exist_ok=True)
+ parquet = args.parquet or args.output_dir / "teacher-test.parquet"
+ reference, repeat = teacher_runs(parquet)
+ write_json(args.output_dir / "reference.json", reference)
+ write_json(args.output_dir / "repeat.json", repeat)
+
+ import torch
+ import transformers
+
+ import laya
+ from laya import evals
+
+ torch.set_num_threads(2)
+ torch.manual_seed(0)
+ provenance = {
+ "model": MODEL,
+ "revision": MODEL_REVISION,
+ "settings": {
+ "device": "cpu", "backend": "eager", "lang": "en",
+ "min_confidence": None, "torch_threads": 2,
+ "laya_version": laya.__version__, "torch_version": str(torch.__version__),
+ "transformers_version": transformers.__version__,
+ },
+ "seed": 0,
+ }
+ print(f"Loading {MODEL}@{MODEL_REVISION} on CPU", file=sys.stderr, flush=True)
+ with laya.load(MODEL, device="cpu", revision=MODEL_REVISION, backend="eager") as agent:
+ report = evals.evaluate_agreement(
+ LocalRunner(agent), reference, repeat, student_provenance=provenance,
+ bootstrap_samples=2000, seed=0,
+ )
+ student = student_run(reference, report)
+ # The saved run must reproduce the full report, including input fingerprints.
+ if evals.compare_agreement(reference, repeat, student) != report:
+ raise RuntimeError("saved observations did not reproduce the live evaluation")
+ write_json(args.output_dir / "student.json", student)
+ write_json(args.output_dir / "report.json", report)
+ print(json.dumps(report["overall"], indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/laya/_agreement.py b/laya/_agreement.py
new file mode 100644
index 00000000..07eff427
--- /dev/null
+++ b/laya/_agreement.py
@@ -0,0 +1,316 @@
+"""Offline, paired teacher-repeat and student agreement on choice decisions.
+
+Teacher observations are comparison targets, not ground truth. Nothing here computes accuracy
+or calibration, calls a teacher service, or estimates a ceiling on student performance.
+"""
+from copy import deepcopy
+import hashlib
+import json
+from typing import Any, Dict, List, Optional
+
+import numpy as np
+
+from .evals import EvalError
+
+
+RUN_SCHEMA = "laya-agreement-run/1"
+REPORT_SCHEMA = "laya-agreement-report/1"
+
+
+def _text(value: Any, where: str) -> None:
+ if not isinstance(value, str) or not value.strip():
+ raise EvalError("%s must be a non-empty string" % where)
+
+
+def _json(value: Any, where: str) -> str:
+ """Strict JSON, preserving object order because it can change model inputs."""
+ def check(item):
+ if type(item) is dict:
+ for key, child in item.items():
+ if not isinstance(key, str):
+ raise ValueError("object keys must be strings")
+ check(child)
+ elif type(item) is list:
+ for child in item:
+ check(child)
+ elif type(item) not in (str, int, float, bool, type(None)):
+ raise ValueError("unsupported value of type %s" % type(item).__name__)
+
+ try:
+ check(value)
+ encoded = json.dumps(value, ensure_ascii=False, allow_nan=False, separators=(",", ":"))
+ encoded.encode("utf-8")
+ return encoded
+ except (ValueError, TypeError, RecursionError, UnicodeError) as exc:
+ raise EvalError("%s must be strict JSON: %s" % (where, exc)) from exc
+
+
+def _fingerprint(value: Any) -> str:
+ return hashlib.sha256(_json(value, "fingerprint input").encode("utf-8")).hexdigest()
+
+
+def _controls(bootstrap_samples: int, seed: int) -> None:
+ if type(bootstrap_samples) is not int or not 100 <= bootstrap_samples <= 100000:
+ raise EvalError("bootstrap_samples must be an integer from 100 to 100000")
+ if type(seed) is not int or seed < 0:
+ raise EvalError("seed must be a non-negative integer")
+
+
+def _provenance(value: Any, where: str) -> None:
+ if not isinstance(value, dict):
+ raise EvalError("%s must be an object" % where)
+ _text(value.get("model"), where + ".model")
+ if "revision" not in value:
+ raise EvalError("%s needs revision (use null when unknown)" % where)
+ if value["revision"] is not None:
+ _text(value["revision"], where + ".revision")
+ if not isinstance(value.get("settings"), dict):
+ raise EvalError("%s.settings must be an object" % where)
+ _json(value, where)
+
+
+def _labels(question: Any, where: str) -> List[str]:
+ if not isinstance(question, dict) or question.get("type") != "choice":
+ raise EvalError("%s must be a choice question" % where)
+ instructions = question.get("instructions")
+ if (instructions is None or not isinstance(instructions, (str, dict, list, int, float))
+ or (isinstance(instructions, str) and not instructions.strip())
+ or (isinstance(instructions, (dict, list)) and not instructions)):
+ raise EvalError("%s needs non-empty instructions" % where)
+ criteria = question.get("criteria")
+ if not isinstance(criteria, (dict, list)) or not criteria:
+ raise EvalError("%s.criteria must be a non-empty object or list of string labels" % where)
+ # Agent._to_internal preserves dict keys and expands a list to {label: None}.
+ # This protocol accepts string labels only, so saved JSON and runtime answers agree.
+ labels = list(criteria)
+ if any(not isinstance(label, str) for label in labels):
+ raise EvalError("%s.criteria labels must be strings" % where)
+ if len(set(labels)) != len(labels):
+ raise EvalError("%s.criteria contains duplicate labels" % where)
+ if "labels" in question:
+ raise EvalError("%s: labels is only supported for noul questions" % where)
+ if "option_order" in question:
+ order = question["option_order"]
+ if (not isinstance(order, list) or any(type(i) is not int for i in order)
+ or sorted(order) != list(range(len(labels)))):
+ raise EvalError("%s.option_order must be a permutation of the option indices" % where)
+ return labels
+
+
+def _run(value: Any, role: str, student: bool = False) -> Dict[str, Any]:
+ if not isinstance(value, dict) or value.get("schema") != RUN_SCHEMA:
+ raise EvalError("%s.schema must be %r" % (role, RUN_SCHEMA))
+ _json(value, role)
+ _text(value.get("run_id"), role + ".run_id")
+ _provenance(value.get("provenance"), role + ".provenance")
+ cases = value.get("cases")
+ if not isinstance(cases, list) or not cases:
+ raise EvalError("%s.cases must be a non-empty list" % role)
+ by_id = {}
+ for index, case in enumerate(cases):
+ where = "%s.cases[%d]" % (role, index)
+ if not isinstance(case, dict):
+ raise EvalError("%s must be an object" % where)
+ _text(case.get("id"), where + ".id")
+ case_id = case["id"]
+ if case_id in by_id:
+ raise EvalError("%s: duplicate case id %r" % (role, case_id))
+ if case.get("state") is None:
+ raise EvalError("%s needs a non-null state" % where)
+ questions, answers = case.get("questions"), case.get("answers")
+ if not isinstance(questions, dict) or not questions:
+ raise EvalError("%s.questions must be a non-empty object" % where)
+ if not isinstance(answers, dict) or set(answers) != set(questions):
+ raise EvalError("%s.answers must have exactly the question ids" % where)
+ for qid, question in questions.items():
+ _text(qid, where + ".questions key")
+ labels = _labels(question, "%s.questions[%r]" % (where, qid))
+ answer = answers[qid]
+ if answer is None and student:
+ continue
+ if not isinstance(answer, str) or answer not in labels:
+ raise EvalError("%s.answers[%r] must be a choice label%s"
+ % (where, qid, " or null" if student else ""))
+ if case.get("language") is not None and not isinstance(case["language"], str):
+ raise EvalError("%s.language must be a string or null" % where)
+ tags = case.get("tags", [])
+ if not isinstance(tags, list) or any(not isinstance(tag, str) for tag in tags):
+ raise EvalError("%s.tags must be a list of strings" % where)
+ by_id[case_id] = case
+ # Sorting cases makes the artifact identity independent of JSONL/array row order.
+ normalized = dict(value, cases=[by_id[key] for key in sorted(by_id)])
+ return {"run": value, "by_id": by_id, "fingerprint": _fingerprint(normalized)}
+
+
+def _align(reference: Dict, other: Dict, role: str) -> None:
+ if set(reference["by_id"]) != set(other["by_id"]):
+ raise EvalError("%s case ids must exactly match reference case ids" % role)
+ for case_id, case in reference["by_id"].items():
+ candidate = other["by_id"][case_id]
+ for key in ("state", "questions"):
+ if _json(case[key], key) != _json(candidate[key], key):
+ raise EvalError("%s case %r has different %s (including option order)"
+ % (role, case_id, key))
+ # Slice labels belong to the inputs, not to the model being compared.
+ if (case.get("language") != candidate.get("language")
+ or set(case.get("tags", [])) != set(candidate.get("tags", []))):
+ raise EvalError("%s case %r has different language or tags" % (role, case_id))
+
+
+def _teachers(reference: Any, repeat: Any) -> tuple:
+ ref, rep = _run(reference, "reference"), _run(repeat, "repeat")
+ if reference["run_id"] == repeat["run_id"]:
+ raise EvalError("reference and repeat need distinct run_id values")
+ for key in ("model", "revision", "settings"):
+ if _json(reference["provenance"][key], key) != _json(repeat["provenance"][key], key):
+ raise EvalError("teacher provenance.%s must match between reference and repeat" % key)
+ _align(ref, rep, "repeat")
+ return ref, rep
+
+
+def _aggregate(records: List[Dict]) -> Dict[str, Any]:
+ n = len(records)
+ teacher = sum(r["teacher_repeat_agreement"] for r in records)
+ student = sum(r["student_teacher_agreement"] for r in records)
+ return {
+ "n_cases": len({r["case_id"] for r in records}), "n_decisions": n,
+ "teacher_repeat_agreement": teacher / n,
+ "student_teacher_agreement": student / n,
+ "student_coverage": sum(r["accepted"] for r in records) / n,
+ "paired_agreement_delta": (student - teacher) / n,
+ }
+
+
+def _interval(records: List[Dict], samples: int, seed: int) -> Optional[List[float]]:
+ totals = {}
+ for record in records:
+ cluster = totals.setdefault(record["case_id"], [0, 0])
+ cluster[0] += record["delta"]
+ cluster[1] += 1
+ if len(totals) < 2:
+ return None
+ clusters = np.asarray([totals[key] for key in sorted(totals)], dtype=np.int64)
+ rng = np.random.default_rng(seed)
+ draws = np.empty(samples, dtype=float)
+ # Each replicate resamples whole cases. Ratio of sums keeps the estimand per decision
+ # when cases have different question counts; batching bounds the index matrix memory.
+ batch = max(1, min(samples, 131072 // len(clusters)))
+ for start in range(0, samples, batch):
+ stop = min(samples, start + batch)
+ indices = rng.integers(len(clusters), size=(stop - start, len(clusters)))
+ sums = clusters[indices].sum(axis=1)
+ draws[start:stop] = sums[:, 0] / sums[:, 1]
+ return np.percentile(draws, [2.5, 97.5]).tolist()
+
+
+def _report(ref: Dict, rep: Dict, stu: Dict, bootstrap_samples: int, seed: int) -> Dict[str, Any]:
+ records = []
+ for case_id in sorted(ref["by_id"]):
+ case = ref["by_id"][case_id]
+ input_fingerprint = _fingerprint({"state": case["state"], "questions": case["questions"]})
+ for qid in sorted(case["questions"]):
+ first = case["answers"][qid]
+ second = rep["by_id"][case_id]["answers"][qid]
+ student = stu["by_id"][case_id]["answers"][qid]
+ teacher_agreement, student_agreement = second == first, student == first
+ records.append({
+ "case_id": case_id, "qid": qid, "teacher_reference": first,
+ "teacher_repeat": second, "student": student,
+ "teacher_repeat_agreement": teacher_agreement,
+ "student_teacher_agreement": student_agreement, "accepted": student is not None,
+ "delta": int(student_agreement) - int(teacher_agreement),
+ "input_fingerprint": input_fingerprint, "language": case.get("language"),
+ "tags": sorted(set(case.get("tags", []))),
+ })
+ overall = _aggregate(records)
+ overall["paired_delta_ci95"] = _interval(records, bootstrap_samples, seed)
+ slices = {}
+ for dimension in ("qid", "language", "tag"):
+ groups = {}
+ for record in records:
+ values = record["tags"] if dimension == "tag" else [record[dimension]]
+ for value in values:
+ if value is not None:
+ groups.setdefault(value, []).append(record)
+ slices[dimension] = {key: _aggregate(groups[key]) for key in sorted(groups)}
+ runs = {"reference": ref, "repeat": rep, "student": stu}
+ limitations = [
+ "Agreement is not accuracy, a student performance ceiling, or evidence that a teacher can be replaced.",
+ "Teacher independence and provenance are declared by the caller, not verified by this report.",
+ "The paired interval resamples original cases with their saved observations; it does not resample teacher calls.",
+ ]
+ unknown = [role for role, data in runs.items() if data["run"]["provenance"]["revision"] is None]
+ if unknown:
+ limitations.append("Unknown model revision for %s; the observations cannot be fully regenerated from this provenance."
+ % ", ".join(unknown))
+ if overall["n_cases"] < 2:
+ limitations.append("A paired interval needs at least two original cases; paired_delta_ci95 is null.")
+ return {
+ "schema": REPORT_SCHEMA,
+ "config": {
+ "protocol": "teacher-reference-paired/1", "bootstrap_samples": bootstrap_samples,
+ "seed": seed, "cluster_unit": "case_id", "confidence_level": 0.95,
+ "interval_method": "paired case bootstrap, percentile, ratio of decision sums",
+ "provenance": {role: deepcopy(data["run"]["provenance"]) for role, data in runs.items()},
+ "run_ids": {role: data["run"]["run_id"] for role, data in runs.items()},
+ "input_fingerprints": {role: data["fingerprint"] for role, data in runs.items()},
+ },
+ "overall": overall, "slices": slices, "cases": records, "limitations": limitations,
+ }
+
+
+def compare_agreement(reference: Dict, repeat: Dict, student: Dict, *,
+ bootstrap_samples: int = 2000, seed: int = 0) -> Dict[str, Any]:
+ """Compare three saved choice runs, paired by case id and question id.
+
+ A student null is an abstention: it contributes zero agreement and zero coverage on the
+ same denominator as answered decisions. Missing cases or answers are errors.
+ """
+ _controls(bootstrap_samples, seed)
+ ref, rep = _teachers(reference, repeat)
+ stu = _run(student, "student", student=True)
+ if student["run_id"] in (reference["run_id"], repeat["run_id"]):
+ raise EvalError("student needs a run_id distinct from both teacher runs")
+ _align(ref, stu, "student")
+ return _report(ref, rep, stu, bootstrap_samples, seed)
+
+
+def evaluate_agreement(runner: Any, reference: Dict, repeat: Dict, *,
+ student_provenance: Dict, model: Optional[str] = None,
+ bootstrap_samples: int = 2000, seed: int = 0) -> Dict[str, Any]:
+ """Run a local student once per case after validating both teacher observations.
+
+ Provenance describes the caller's runner; no checkpoint revision is inferred or fabricated.
+ The runner receives copied inputs and its exceptions propagate without dropping cases.
+ """
+ _controls(bootstrap_samples, seed)
+ ref, rep = _teachers(reference, repeat)
+ _provenance(student_provenance, "student_provenance")
+ if model is not None:
+ _text(model, "model")
+ cases = []
+ for case_id in sorted(ref["by_id"]):
+ case = ref["by_id"][case_id]
+ result = runner.predict(deepcopy(case["state"]), deepcopy(case["questions"]), model=model)
+ answers = result.get("answers") if isinstance(result, dict) else None
+ if not isinstance(answers, dict) or set(answers) != set(case["questions"]):
+ raise EvalError("student case %r must return exactly the question ids" % case_id)
+ choices = {}
+ for qid, question in case["questions"].items():
+ answer = answers[qid]
+ where = "student case %r question %r" % (case_id, qid)
+ if not isinstance(answer, dict) or answer.get("type") != "choice":
+ raise EvalError("%s must return a choice answer" % where)
+ choice = answer.get("choice")
+ if not isinstance(choice, str) or choice not in _labels(question, where):
+ raise EvalError("%s returned an invalid choice label" % where)
+ choices[qid] = None if answer.get("abstention") == "abstained" else choice
+ cases.append(dict(deepcopy(case), answers=choices))
+ run_id, suffix = "student", 0
+ while run_id in (reference["run_id"], repeat["run_id"]):
+ suffix += 1
+ run_id = "student-%d" % suffix
+ student = {"schema": RUN_SCHEMA, "run_id": run_id,
+ "provenance": deepcopy(student_provenance), "cases": cases}
+ stu = _run(student, "student", student=True)
+ return _report(ref, rep, stu, bootstrap_samples, seed)
diff --git a/laya/evals.py b/laya/evals.py
index 53035722..b066859c 100644
--- a/laya/evals.py
+++ b/laya/evals.py
@@ -66,6 +66,38 @@ class EvalError(ValueError):
"""A dataset or report is malformed; the message names the row or field."""
+def compare_agreement(reference: Dict, repeat: Dict, student: Dict, *,
+ bootstrap_samples: int = 2000, seed: int = 0) -> Dict[str, Any]:
+ """Compare saved teacher reference, teacher repeat, and student choice observations.
+
+ Inputs use ``laya-agreement-run/1``; see the evaluation guide for the schema.
+ All runs must contain the same cases and questions. Student nulls count as
+ disagreements and reduce coverage. The report includes a paired, case-clustered
+ bootstrap interval (100–100000 replicates), provenance, and individual decisions.
+ Teacher agreement is not ground-truth accuracy or a ceiling on student performance.
+ """
+ from ._agreement import compare_agreement as compare
+
+ return compare(reference, repeat, student, bootstrap_samples=bootstrap_samples, seed=seed)
+
+
+def evaluate_agreement(runner: Any, reference: Dict, repeat: Dict, *,
+ student_provenance: Dict, model: Optional[str] = None,
+ bootstrap_samples: int = 2000, seed: int = 0) -> Dict[str, Any]:
+ """Evaluate a local student's agreement with two saved teacher observations.
+
+ Validate both teacher runs before calling ``runner.predict(state, questions,
+ model=model)`` once per original case. Record the caller's student provenance;
+ use a null revision when it is unknown. Runner errors propagate, and missing or
+ invalid answers raise ``EvalError`` instead of dropping decisions. An answer's
+ ``abstention='abstained'`` becomes a null student observation.
+ """
+ from ._agreement import evaluate_agreement as evaluate_student
+
+ return evaluate_student(runner, reference, repeat, student_provenance=student_provenance,
+ model=model, bootstrap_samples=bootstrap_samples, seed=seed)
+
+
@dataclass
class Example:
"""One labelled decision: a state, its questions, and the expected answer per question id."""
diff --git a/laya/evals_cli.py b/laya/evals_cli.py
index 174eb220..0d8ff5ee 100644
--- a/laya/evals_cli.py
+++ b/laya/evals_cli.py
@@ -225,6 +225,17 @@ def _build_parser() -> argparse.ArgumentParser:
run.add_argument("--json", dest="json_out", help="write the full report JSON here")
run.add_argument("--markdown", dest="markdown_out", help="write a Markdown summary here")
+ agreement = sub.add_parser("agreement", help="compare saved teacher and student choice runs")
+ agreement.add_argument("reference", help="reference teacher run JSON")
+ agreement.add_argument("repeat", help="independent repeat teacher run JSON")
+ agreement.add_argument("student", help="student run JSON on the same cases")
+ agreement.add_argument("--bootstrap-samples", type=int, default=2000,
+ help="case bootstrap replicates, 100–100000 (default '2000')")
+ agreement.add_argument("--seed", type=int, default=0,
+ help="non-negative bootstrap seed (default '0')")
+ agreement.add_argument("--json", dest="json_out", metavar="PATH",
+ help="write the agreement report to PATH instead of stdout")
+
compare = sub.add_parser("compare", help="compare a report JSON against a baseline")
compare.add_argument("report", help="a report JSON written by `laya-evals run --json`")
compare.add_argument("--baseline", required=True,
@@ -488,6 +499,35 @@ def _cmd_compare(args) -> int:
return 0 if ok and not failures else 1
+def _cmd_agreement(args) -> int:
+ def unique_object(pairs):
+ obj = {}
+ for key, value in pairs:
+ if key in obj:
+ raise ValueError("duplicate JSON key %r" % key)
+ obj[key] = value
+ return obj
+
+ runs = []
+ for path in (args.reference, args.repeat, args.student):
+ try:
+ with open(path, encoding="utf-8") as handle:
+ runs.append(json.load(handle, object_pairs_hook=unique_object))
+ except (OSError, ValueError, RecursionError) as exc:
+ raise EvalError("cannot read agreement run %r: %s" % (path, exc)) from exc
+ report = evals.compare_agreement(*runs, bootstrap_samples=args.bootstrap_samples, seed=args.seed)
+ encoded = json.dumps(report, ensure_ascii=False, indent=2, allow_nan=False) + "\n"
+ if args.json_out:
+ try:
+ with open(args.json_out, "w", encoding="utf-8", newline="\n") as handle:
+ handle.write(encoded)
+ except OSError as exc:
+ raise EvalError("cannot write agreement report %r: %s" % (args.json_out, exc)) from exc
+ else:
+ print(encoded, end="")
+ return 0
+
+
def main(argv: Optional[Sequence[str]] = None) -> int:
args = _build_parser().parse_args(argv)
# Same reason as `laya.cli.main`: redirected output is encoded with the locale's codec by
@@ -497,6 +537,8 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8")
try:
+ if args.command == "agreement":
+ return _cmd_agreement(args)
if args.command == "validate":
return _cmd_validate(args)
if args.command == "run":
diff --git a/scripts/test_suites.py b/scripts/test_suites.py
index 089fdf77..c64b38d4 100644
--- a/scripts/test_suites.py
+++ b/scripts/test_suites.py
@@ -104,6 +104,7 @@ SCRIPT_SUITES = [
]
PYTEST_SUITES = [
+ "tests/test_evals_agreement.py",
"tests/test_serve.py",
"tests/test_mcp_remote.py",
"tests/test_router_batch.py",
diff --git a/tests/test_evals_agreement.py b/tests/test_evals_agreement.py
new file mode 100644
index 00000000..394bf0aa
--- /dev/null
+++ b/tests/test_evals_agreement.py
@@ -0,0 +1,357 @@
+"""Model-free checks for paired agreement and case-level uncertainty."""
+from copy import deepcopy
+import json
+from pathlib import Path
+import subprocess
+import sys
+
+import pytest
+
+from laya import evals
+
+
+def run_document(run_id, rows, *, teacher=True):
+ cases = []
+ for index, labels in enumerate(rows):
+ questions = {
+ "q%d" % q: {"type": "choice", "instructions": "Choose a route.",
+ "criteria": {"a": "Accept", "b": "Defer"}}
+ for q in range(len(labels))
+ }
+ cases.append({
+ "id": "case-%d" % index,
+ "state": {"text": "request-%d" % index, "context": ["original"]},
+ "questions": questions,
+ "answers": dict(zip(questions, labels)),
+ "tags": ["routing"], "language": "en",
+ })
+ return {"schema": "laya-agreement-run/1", "run_id": run_id,
+ "provenance": {"model": "teacher" if teacher else "student",
+ "revision": "pinned", "settings": {"temperature": 0.5}},
+ "cases": cases}
+
+
+def documents(reference=(("a",), ("b",)), repeat=None, student=None):
+ return (run_document("reference", reference),
+ run_document("repeat", repeat if repeat is not None else reference),
+ run_document("student", student if student is not None else reference,
+ teacher=False))
+
+
+def compare(runs, **kwargs):
+ return evals.compare_agreement(*runs, **kwargs)
+
+
+def test_paired_rates_use_every_decision_and_preserve_abstention_denominator():
+ runs = documents((("a", "a", "a"), ("b", "a")),
+ (("a", "b", "a"), ("b", "a")),
+ (("a", "b", None), ("b", "b")))
+ report = compare(runs)
+ assert report["schema"] == "laya-agreement-report/1"
+ overall = report["overall"]
+ assert overall["n_cases"] == 2
+ assert overall["n_decisions"] == 5
+ assert overall["teacher_repeat_agreement"] == pytest.approx(4 / 5)
+ assert overall["student_teacher_agreement"] == pytest.approx(2 / 5)
+ assert overall["student_coverage"] == pytest.approx(4 / 5)
+ assert overall["paired_agreement_delta"] == pytest.approx(-2 / 5)
+ assert len(report["cases"]) == 5
+ observed = {(row["case_id"], row["qid"]):
+ (row["teacher_reference"], row["teacher_repeat"], row["student"],
+ row["accepted"], row["delta"]) for row in report["cases"]}
+ assert observed == {
+ ("case-0", "q0"): ("a", "a", "a", True, 0),
+ ("case-0", "q1"): ("a", "b", "b", True, 0),
+ ("case-0", "q2"): ("a", "a", None, False, -1),
+ ("case-1", "q0"): ("b", "b", "b", True, 0),
+ ("case-1", "q1"): ("a", "a", "b", True, -1),
+ }
+ assert report["slices"]["qid"]["q2"]["n_decisions"] == 1
+ assert report["slices"]["qid"]["q2"]["student_coverage"] == 0.0
+ assert not {"accuracy", "choice_accuracy", "ece", "brier", "aurc"} & overall.keys()
+ assert json.loads(json.dumps(report)) == report
+
+
[email protected]("repeat,student,agreement", [("a", "a", 1.0), ("b", "b", 0.0)])
+def test_perfect_and_zero_agreement(repeat, student, agreement):
+ overall = compare(documents((("a",), ("a",)),
+ ((repeat,), (repeat,)),
+ ((student,), (student,))))["overall"]
+ assert overall["teacher_repeat_agreement"] == agreement
+ assert overall["student_teacher_agreement"] == agreement
+ assert overall["paired_agreement_delta"] == 0.0
+ assert tuple(overall["paired_delta_ci95"]) == (0.0, 0.0)
+
+
+def test_student_abstention_is_not_a_missing_case_or_an_agreeing_answer():
+ overall = compare(documents(student=((None,), (None,))))["overall"]
+ assert overall["n_cases"] == overall["n_decisions"] == 2
+ assert overall["student_coverage"] == 0.0
+ assert overall["student_teacher_agreement"] == 0.0
+ assert overall["paired_agreement_delta"] == -1.0
+ assert tuple(overall["paired_delta_ci95"]) == (-1.0, -1.0)
+
+
+def test_alignment_uses_case_ids_and_comparison_does_not_mutate_inputs():
+ runs = documents((("a",), ("b",), ("a",)),
+ (("a",), ("a",), ("b",)),
+ (("b",), ("b",), ("a",)))
+ original = deepcopy(runs)
+ expected = compare(runs, seed=31)["overall"]
+ assert runs == original
+ runs[0]["cases"].reverse()
+ runs[1]["cases"] = runs[1]["cases"][1:] + runs[1]["cases"][:1]
+ assert compare(runs, seed=31)["overall"] == expected
+
+
+def test_one_case_has_no_interval_even_with_many_questions():
+ overall = compare(documents((("a",) * 30,)))["overall"]
+ assert overall["n_cases"] == 1
+ assert overall["n_decisions"] == 30
+ assert overall["paired_delta_ci95"] is None
+
+
+def test_bootstrap_resamples_whole_cases_not_correlated_questions():
+ # There are just two independent cases. Resampling two cases can give only
+ # delta -1, 0 or +1, regardless of how many questions each case contains.
+ runs = documents((("a",) * 25, ("a",) * 25),
+ (("a",) * 25, ("b",) * 25),
+ (("b",) * 25, ("a",) * 25))
+ report = compare(runs, bootstrap_samples=2000, seed=17)
+ overall = report["overall"]
+ assert overall["paired_agreement_delta"] == 0.0
+ assert tuple(overall["paired_delta_ci95"]) == (-1.0, 1.0)
+ assert compare(runs, bootstrap_samples=2000, seed=17) == report
+
+
+def test_bootstrap_preserves_decision_weights_with_unequal_case_sizes():
+ runs = documents((("a",) * 100, ("a",), ("a",), ("a",)),
+ (("b",) * 100, ("a",), ("a",), ("a",)),
+ (("a",) * 100, ("b",), ("b",), ("b",)))
+ overall = compare(runs, bootstrap_samples=5000, seed=23)["overall"]
+ assert overall["paired_agreement_delta"] == pytest.approx(97 / 103)
+ # K large-case draws follows Binomial(4, 1/4). The 2.5th/97.5th percentiles
+ # have K=0/3: ratios -4/4 and (300-1)/(300+1), not case-mean rates -1/0.5.
+ assert overall["paired_delta_ci95"] == pytest.approx([-1.0, 299 / 301])
+
+
[email protected]("side", [0, 1, 2])
[email protected]("change", ["missing", "duplicate", "unknown"])
+def test_case_sets_must_match_exactly(side, change):
+ runs = documents()
+ if change == "missing":
+ runs[side]["cases"].pop()
+ elif change == "duplicate":
+ runs[side]["cases"].append(deepcopy(runs[side]["cases"][0]))
+ else:
+ runs[side]["cases"][0]["id"] = "unrelated"
+ with pytest.raises(ValueError):
+ compare(runs)
+
+
[email protected]("change", ["state", "prompt", "criteria_order", "criteria_text", "qid"])
+def test_input_and_question_drift_are_rejected(change):
+ runs = documents()
+ case = runs[1]["cases"][0]
+ question = case["questions"]["q0"]
+ if change == "state":
+ case["state"]["context"].append("changed")
+ elif change == "prompt":
+ question["instructions"] = "Choose the opposite route."
+ elif change == "criteria_order":
+ question["criteria"] = {"b": "Defer", "a": "Accept"}
+ elif change == "criteria_text":
+ question["criteria"]["a"] = "Reject"
+ else:
+ case["questions"]["other"] = case["questions"].pop("q0")
+ case["answers"]["other"] = case["answers"].pop("q0")
+ with pytest.raises(ValueError):
+ compare(runs)
+
+
[email protected]("key,value", [("model", "other"), ("revision", "other-pin"),
+ ("settings", {"temperature": 0.9})])
+def test_teacher_configuration_must_match(key, value):
+ runs = documents()
+ runs[1]["provenance"][key] = value
+ with pytest.raises(ValueError):
+ compare(runs)
+
+
+def test_teacher_seed_may_differ():
+ runs = documents()
+ runs[0]["provenance"]["seed"] = 5
+ runs[1]["provenance"]["seed"] = 8
+ assert compare(runs)["overall"]["teacher_repeat_agreement"] == 1.0
+
+
+def test_unknown_teacher_revision_remains_unknown_and_disclosed():
+ runs = documents()
+ for run in runs[:2]:
+ run["provenance"]["revision"] = None
+ report = compare(runs)
+ assert report["config"]["provenance"]["reference"]["revision"] is None
+ assert report["config"]["provenance"]["repeat"]["revision"] is None
+ assert any("revision" in note.lower() for note in report["limitations"])
+
+
[email protected]("name,value", [("bootstrap_samples", 99), ("bootstrap_samples", 100001),
+ ("bootstrap_samples", True), ("bootstrap_samples", 100.5), ("bootstrap_samples", float("inf")),
+ ("seed", -1), ("seed", True), ("seed", 1.5), ("seed", float("nan"))])
+def test_bootstrap_options_require_bounded_integer_values(name, value):
+ with pytest.raises(ValueError):
+ compare(documents(), **{name: value})
+
+
[email protected]("change", ["schema", "empty", "score", "teacher_abstention",
+ "unknown_label", "missing_answer", "extra_answer", "nonfinite"])
+def test_malformed_runs_are_rejected(change):
+ runs = documents()
+ case = runs[0]["cases"][0]
+ if change == "schema":
+ runs[0]["schema"] = "laya-agreement-run/999"
+ elif change == "empty":
+ for run in runs:
+ run["cases"] = []
+ elif change == "score":
+ case["questions"]["q0"]["type"] = "score"
+ elif change == "teacher_abstention":
+ case["answers"]["q0"] = None
+ elif change == "unknown_label":
+ case["answers"]["q0"] = "unlisted"
+ elif change == "missing_answer":
+ case["answers"].pop("q0")
+ elif change == "extra_answer":
+ case["answers"]["other"] = "a"
+ else:
+ for run in runs:
+ run["provenance"]["settings"]["temperature"] = float("nan")
+ with pytest.raises(ValueError):
+ compare(runs)
+
+
+_DEFAULT_RESULT = object()
+
+
+class RecordingRunner:
+ def __init__(self, result=_DEFAULT_RESULT, mutate=False):
+ self.calls = []
+ self.result = result
+ self.mutate = mutate
+
+ def predict(self, state, questions, model=None):
+ self.calls.append((deepcopy(state), deepcopy(questions), model))
+ answers = {qid: {"type": "choice", "choice": "a"} for qid in questions}
+ if self.mutate:
+ state["context"].append("runner mutation")
+ questions["q0"]["criteria"]["a"] = "runner mutation"
+ return {"answers": answers} if self.result is _DEFAULT_RESULT else self.result
+
+
+def evaluate(runner, runs, **kwargs):
+ return evals.evaluate_agreement(runner, runs[0], runs[1],
+ student_provenance=runs[2]["provenance"], **kwargs)
+
+
+def test_evaluate_forwards_model_and_isolates_inputs_from_runner_mutation():
+ runs = documents()
+ original = deepcopy(runs)
+ runner = RecordingRunner(mutate=True)
+ report = evaluate(runner, runs, model="local-checkpoint")
+ assert runs == original
+ assert len(runner.calls) == 2
+ assert all(call[2] == "local-checkpoint" for call in runner.calls)
+ assert report["overall"]["student_teacher_agreement"] == 0.5
+
+
+def test_evaluate_treats_abstained_argmax_as_uncovered():
+ runner = RecordingRunner({"answers": {"q0": {"type": "choice", "choice": "a",
+ "abstention": "abstained"}}})
+ overall = evaluate(runner, documents())["overall"]
+ assert overall["student_coverage"] == overall["student_teacher_agreement"] == 0.0
+
+
[email protected]("result", [None, "invalid", {}, {"answers": {}},
+ {"answers": {"q0": {"type": "choice", "choice": "unlisted"}}},
+ {"answers": {"q0": {"type": "score", "score": 0.4}}},
+ {"answers": {"q0": {"type": "choice", "choice": None}}}])
+def test_evaluate_rejects_invalid_runner_results(result):
+ with pytest.raises(ValueError):
+ evaluate(RecordingRunner(result), documents())
+
+
[email protected]("invalid", ["alignment", "bootstrap", "provenance", "null_state"])
+def test_evaluate_validates_every_input_before_first_runner_call(invalid):
+ runs = documents()
+ kwargs = {}
+ if invalid == "alignment":
+ runs[1]["cases"][-1]["state"] = "changed final case"
+ elif invalid == "bootstrap":
+ kwargs["bootstrap_samples"] = True
+ elif invalid == "null_state":
+ for run in runs:
+ run["cases"][-1]["state"] = None
+ else:
+ runs[2]["provenance"]["settings"] = {"temperature": float("inf")}
+ runner = RecordingRunner()
+ with pytest.raises(ValueError):
+ evaluate(runner, runs, **kwargs)
+ assert runner.calls == []
+
+
+def cli(*args):
+ return subprocess.run([sys.executable, "-m", "laya.evals_cli", "agreement", *map(str, args)],
+ cwd=Path(__file__).resolve().parents[1], capture_output=True,
+ text=True, encoding="utf-8", timeout=30)
+
+
+def saved_runs(tmp_path):
+ runs = documents()
+ for run in runs:
+ run["cases"][0]["state"] = "退款申请"
+ paths = [tmp_path / (run["run_id"] + ".json") for run in runs]
+ for path, run in zip(paths, runs):
+ path.write_text(json.dumps(run, ensure_ascii=False), encoding="utf-8")
+ return paths, runs
+
+
+def test_cli_saved_run_replay_matches_public_api(tmp_path):
+ paths, runs = saved_runs(tmp_path)
+ expected = compare(runs, bootstrap_samples=100, seed=12)
+ args = [*paths, "--bootstrap-samples", "100", "--seed", "12"]
+ result = cli(*args)
+ assert result.returncode == 0, result.stderr
+ assert json.loads(result.stdout) == expected
+ output = tmp_path / "结果.json"
+ result = cli(*args, "--json", output)
+ assert result.returncode == 0, result.stderr
+ assert not result.stdout
+ assert json.loads(output.read_text(encoding="utf-8")) == expected
+
+
[email protected]("invalid", ["duplicate_key", "nan", "malformed", "nested", "missing", "misaligned"])
+def test_cli_refuses_bad_inputs_without_a_report(tmp_path, invalid):
+ paths, runs = saved_runs(tmp_path)
+ if invalid == "duplicate_key":
+ # Reject ambiguity even when the duplicate has the same value.
+ raw = paths[0].read_text(encoding="utf-8")
+ paths[0].write_text(raw.replace('"schema":', '"schema":"laya-agreement-run/1","schema":', 1),
+ encoding="utf-8")
+ elif invalid == "nan":
+ runs[0]["provenance"]["settings"]["temperature"] = float("nan")
+ paths[0].write_text(json.dumps(runs[0]), encoding="utf-8")
+ elif invalid == "malformed":
+ paths[0].write_text("{broken", encoding="utf-8")
+ elif invalid == "nested":
+ paths[0].write_text("[" * 5000 + "0" + "]" * 5000, encoding="utf-8")
+ elif invalid == "missing":
+ paths[0].unlink()
+ else:
+ runs[2]["cases"].pop()
+ paths[2].write_text(json.dumps(runs[2]), encoding="utf-8")
+ output = tmp_path / "report.json"
+ result = cli(*paths, "--json", output)
+ assert result.returncode == 2
+ assert "laya-evals:" in result.stderr
+ assert "Traceback" not in result.stderr
+ assert not output.exists()
diff --git a/tests/test_evals_api.py b/tests/test_evals_api.py
index 1861d9a3..c7fee1ff 100644
--- a/tests/test_evals_api.py
+++ b/tests/test_evals_api.py
@@ -44,6 +44,7 @@ check_true("ScoreWithin names its tolerance", evals.ScoreWithin(0.25).name == "s
# --------------------------------------------------------------- exports / callables
for name in ("Dataset", "Example", "EvalError", "EvalReport", "evaluate", "ece", "assert_regression",
"REPORT_SCHEMA", "questions_fingerprint", "file_fingerprint",
+ "compare_agreement", "evaluate_agreement",
"brier", "aurc", "selective_accuracy", "is_confidence_metric",
# `_eval_policy` imports these three, so they are a contract between two modules in
# this package, not internals -- a rename would break the release gate, not just a
@@ -158,7 +159,7 @@ def _subparsers(parser):
_subs = _subparsers(evals_cli._build_parser())
-check("laya-evals subcommands", sorted(_subs), ["compare", "evidence", "run", "validate"])
+check("laya-evals subcommands", sorted(_subs), ["agreement", "compare", "evidence", "run", "validate"])
_unhelped, _unstated, _options = [], [], {}
for _sub, _parser in sorted(_subs.items()):
diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.py
index 41c53485..9b4b0c6d 100644
--- a/tests/test_hooks_api.py
+++ b/tests/test_hooks_api.py
@@ -1512,6 +1512,16 @@ finally:
_hooks_mod.clear_default_hooks()
+from laya.evals import compare_agreement, evaluate_agreement # noqa: E402
+
+check("compare_agreement signature", list(inspect.signature(compare_agreement).parameters),
+ ["reference", "repeat", "student", "bootstrap_samples", "seed"])
+check("evaluate_agreement signature", list(inspect.signature(evaluate_agreement).parameters),
+ ["runner", "reference", "repeat", "student_provenance", "model", "bootstrap_samples", "seed"])
+check("agreement provenance is required",
+ inspect.signature(evaluate_agreement).parameters["student_provenance"].default,
+ inspect.Parameter.empty)
+
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
print(" FAIL", f)