Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(evals): compare student agreement with repeated teacher decisions
xp5aae58cf00af
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 28d71a39..4a7bfe7e 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -164,7 +164,7 @@ jobs:pip install -e . pytest- name: Eval harness tests- run: python -m pytest tests/test_evals.py -q+ run: python -m pytest tests/test_evals.py tests/test_evals_agreement.py -q- name: Shortlist attribution testsrun: python -m pytest tests/test_evals_shortlist.py -qdiff --git a/docs/evals.md b/docs/evals.mdindex 9fa7351d..dcba75b8 100644--- a/docs/evals.md+++ b/docs/evals.md@@ -29,6 +29,117 @@ Exit codes: `0` on success, `1` when a threshold or a baseline tolerance fails,usage error. `run` prints the overall metrics and any requested slices to stdout, and writesthe full report and a Markdown summary when `--json` / `--markdown` are given.+## Comparing a student with repeated teacher decisions++When a teacher gives different answers to the same input, a single saved answer+does not describe its repeatability. `compare_agreement` compares a student and+an independent teacher repeat against one fixed reference teacher observation.+It takes three saved runs and needs no model weights or teacher service:++```bash+laya-evals agreement reference.json repeat.json student.json \+ --bootstrap-samples 2000 --seed 0 --json agreement.json+```++Without `--json`, the command writes the report to stdout. `--seed` controls only+the bootstrap, not model generation. Success exits `0`; malformed or misaligned+runs exit `2`. This command does not apply a replacement or quality gate.++Each input is a JSON object with the following shape:++```json+{+ "schema": "laya-agreement-run/1",+ "run_id": "teacher-reference",+ "provenance": {+ "model": "teacher-checkpoint",+ "revision": null,+ "settings": {"temperature": 0.7, "prompt_template": "Choose one label."},+ "seed": 42+ },+ "cases": [{+ "id": "ticket-001",+ "state": "I was charged twice.",+ "questions": {"route": {+ "type": "choice",+ "instructions": "Select the team responsible for this request.",+ "criteria": ["billing", "support"]+ }},+ "answers": {"route": "billing"},+ "language": "en",+ "tags": ["duplicate-charge"]+ }]+}+```++Use a distinct `run_id` for each observation run. All three runs must contain+exactly the same unique case IDs, states, question definitions, language and tags;+row order may differ. A state must be non-null JSON. Question definitions preserve option order and accept+non-empty criteria dictionaries or lists of distinct string labels. Answers must+cover every question. A student answer may be `null` for abstention; teacher+answers must be labels. Missing cases, unknown labels, non-finite JSON values and+duplicate JSON object keys in CLI inputs are errors, rather than omitted samples.++The two teacher runs must declare identical model, revision and settings. Put+sampling parameters and the prompt template in `settings`, and repetition seeds+outside it. Other metadata, such as the dataset revision and generation script,+is preserved. Record immutable model revisions when available; explicitly use+`null` when unknown. The report flags unknown revisions but cannot verify that+the recorded provenance is accurate or that the teacher calls were independent.+Input object order is significant, matching Laya's serialization of structured+states and questions; keep it identical across the runs.++To run a local student after validating the teacher records:++```python+import json+import laya+from laya.evals import evaluate_agreement++with open("reference.json", encoding="utf-8") as handle:+ reference = json.load(handle)+with open("repeat.json", encoding="utf-8") as handle:+ repeat = json.load(handle)+revision = "YOUR_CHECKPOINT_COMMIT"+with laya.Router(device="cpu", revision=revision) as runner:+ report = evaluate_agreement(+ runner, reference, repeat, model="english",+ student_provenance={+ "model": "convaiinnovations/laya", "revision": revision,+ "settings": {"device": "cpu"},+ },+ )+print(report["overall"])+```++The runner uses the same `predict(state, questions, model=...)` protocol as the+labelled harness (for example, `laya.Router`). Inputs are+copied before inference. Runner failures propagate; invalid or incomplete outputs+raise `EvalError`. A choice flagged `abstention="abstained"` becomes a null+observation, while its underlying choice must still be a valid label.++The separate `laya-agreement-report/1` report records provenance, input hashes,+individual paired decisions and rates sliced by question ID, language and tag.+For every decision, teacher repeat agreement is `repeat == reference`, student+agreement is `student == reference`, and the paired delta is their difference.+All rates use the full decision count: abstentions contribute zero agreement and+reduce `student_coverage`. The 95% percentile interval resamples whole case IDs,+keeping their questions together, and recomputes the ratio of decision sums. It+is deterministic for fixed inputs, seed and bootstrap count (100–100000). With+fewer than two cases the interval is `null`. Slice rates have no intervals.++Agreement is not accuracy, and teacher repeatability is not a theoretical ceiling.+Both models can agree on a wrong answer. The interval describes variation across+the supplied cases with their saved observations; it does not capture additional+teacher generations. An interval containing zero does not establish equivalence.+Inspect slices and individual decisions for label skew, and use labelled evals+for correctness or calibration claims.++[`examples/evals/teacher_agreement.py`](https://github.com/NandhaKishorM/laya/blob/main/examples/evals/teacher_agreement.py)+replays two published teacher observations from a pinned public dataset against+a local Laya checkpoint, saving the three runs and their report. Its small fixed+sample demonstrates the workflow, not deployment readiness.+## Attributing shortlist errorsFor a labelled high-cardinality choice set, `laya.evals_shortlist.evaluate_shortlist`diff --git a/examples/evals/teacher_agreement.py b/examples/evals/teacher_agreement.pynew file mode 100644index 00000000..5dc50d86--- /dev/null+++ b/examples/evals/teacher_agreement.py@@ -0,0 +1,226 @@+"""Compare a local Laya checkpoint with two published teacher observations.++From a checkout::++ python -m pip install -e . pyarrow+ python examples/evals/teacher_agreement.py --output-dir /tmp/laya-agreement+ laya-evals agreement /tmp/laya-agreement/reference.json \+ /tmp/laya-agreement/repeat.json /tmp/laya-agreement/student.json++The first 32 test rows and teacher repeats 0/1 are fixed before student inference.+The parquet is downloaded once, checked against its SHA256, and kept in the output+directory. Pass --parquet to reuse a downloaded copy. Only the example needs pyarrow.+The pinned English Laya checkpoint runs locally on CPU; no teacher API is called.++Data credit: LocalGate's MIT-licensed localgate/mmlu-pro-closed dataset, derived+from TIGER-Lab/MMLU-Pro. The publisher's protocol and generation manifest are at:+https://huggingface.co/datasets/localgate/mmlu-pro-closed/blob/de3c3d7bd8870e6ec48f03284f3e48042bae9ecf/protocol.md+https://huggingface.co/datasets/localgate/mmlu-pro-closed/blob/de3c3d7bd8870e6ec48f03284f3e48042bae9ecf/provenance/labels_manifest.json++Teacher picks are real saved generations, not reconstructed from aggregate scores.+The publisher did not record the teacher's immutable model revision; null preserves+that uncertainty. Laya receives the same question and ordered options as typed+inputs, not the teacher's chain-of-thought prompt. This small panel demonstrates+the evaluation pipeline: agreement is neither accuracy nor a performance ceiling.+"""+from __future__ import annotations++import argparse+import hashlib+import json+import sys+import urllib.request+from copy import deepcopy+from pathlib import Path++DATASET = "localgate/mmlu-pro-closed"+DATASET_REVISION = "de3c3d7bd8870e6ec48f03284f3e48042bae9ecf"+PARQUET_SHA256 = "7b1ee7627f19a6ac322aa677a6483549e5ea6e136b6992dba6da0f8a2227176f"+PARQUET_URL = (+ f"https://huggingface.co/datasets/{DATASET}/resolve/{DATASET_REVISION}"+ "/data/test-00000-of-00001.parquet"+)+PROTOCOL_SHA256 = "6f0ac97a38225f523a61803d25d6868e90813ef6873a8170a7e2f2a9e056ef9b"+MODEL = "convaiinnovations/laya"+MODEL_REVISION = "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851"+COUNT = 32+PROMPT_TEMPLATE = (+ "<bos><|turn>system\n"+ "The following are multiple choice questions (with answers) about {category}. "+ 'Think step by step and then finish your answer with "the answer is (X)" '+ "where X is the correct letter choice.<turn|>\n"+ "<|turn>user\n"+ "Question:\n{question}\nOptions:\n{ordered_options}\n"+ "Answer: Let's think step by step.<turn|>\n"+ "<|turn>model\n"+)+++def write_json(path: Path, value: object) -> None:+ with path.open("w", encoding="utf-8", newline="\n") as handle:+ json.dump(value, handle, ensure_ascii=False, indent=2, allow_nan=False)+ handle.write("\n")+++def teacher_runs(parquet: Path) -> tuple[dict, dict]:+ import pyarrow.parquet as pq++ if parquet.exists():+ data = parquet.read_bytes()+ else:+ request = urllib.request.Request(+ PARQUET_URL, headers={"User-Agent": "Laya-agreement-example"}+ )+ with urllib.request.urlopen(request, timeout=120) as response:+ data = response.read()+ actual = hashlib.sha256(data).hexdigest()+ if actual != PARQUET_SHA256:+ raise ValueError(f"parquet SHA256 mismatch: {actual}")+ if not parquet.exists():+ parquet.parent.mkdir(parents=True, exist_ok=True)+ parquet.write_bytes(data)++ table = pq.read_table(parquet)+ if table.num_rows < COUNT:+ raise ValueError(f"expected at least {COUNT} rows in the pinned parquet")+ rows = table.slice(0, COUNT).to_pylist()+ if len({row["question_id"] for row in rows}) != COUNT:+ raise ValueError("duplicate question_id in the fixed selection")+ runs = []+ for repeat, seed in ((0, 42), (1, 43)):+ cases = []+ for row in rows:+ criteria = dict(zip("ABCDEFGHIJ", row["options"]))+ if row["k"] != 5 or len(row["picks"]) != 5:+ raise ValueError(f"expected five teacher observations for {row['question_id']}")+ if row["picks"][repeat] not in criteria:+ raise ValueError(+ f"invalid teacher choice for {row['question_id']}, repeat {repeat}; "+ "the fixed selection is not filtered or backfilled"+ )+ cases.append({+ "id": str(row["question_id"]),+ "state": row["question"],+ "questions": {"answer": {+ "type": "choice",+ "instructions": "Choose the correct answer to the question.",+ "criteria": criteria,+ }},+ "answers": {"answer": row["picks"][repeat]},+ "tags": [row["category"]],+ "language": "en",+ })+ runs.append({+ "schema": "laya-agreement-run/1",+ "run_id": f"gemma-seed-{seed}",+ "provenance": {+ "model": "google/gemma-4-E2B-it",+ "revision": None,+ "settings": {+ "do_sample": True,+ "temperature": 1.0,+ "top_p": 0.95,+ "top_k": 64,+ "max_gen_toks": 6144,+ "until": ["Question:"],+ "thinking": False,+ "num_fewshot": 0,+ "prompt_template": PROMPT_TEMPLATE,+ "option_line_template": "{letter}. {option}",+ "protocol_sha256": PROTOCOL_SHA256,+ },+ "seed": seed,+ "source": {+ "dataset": DATASET,+ "revision": DATASET_REVISION,+ "split": "test",+ "selection": "first 32 rows in the pinned parquet",+ "file_sha256": PARQUET_SHA256,+ "teacher_revision_note": "not recorded in the publisher's manifest",+ },+ },+ "cases": cases,+ })+ return runs[0], runs[1]+++class LocalRunner:+ """Adapt one loaded Agent to the evaluation runner's model keyword."""++ def __init__(self, agent):+ self.agent = agent+ self.calls = 0++ def predict(self, state, questions, *, model=None):+ if model is not None:+ raise ValueError("this example uses a single fixed checkpoint")+ result = self.agent.predict(state, questions, lang="en")+ self.calls += 1+ if self.calls % 8 == 0:+ print(f"Evaluated {self.calls}/{COUNT} cases", file=sys.stderr, flush=True)+ return result+++def student_run(reference: dict, report: dict) -> dict:+ """Persist actual report observations for offline CLI replay, without inference."""+ cases = deepcopy(reference["cases"])+ by_id = {case["id"]: case for case in cases}+ for case in cases:+ case["answers"] = {}+ for record in report["cases"]:+ by_id[record["case_id"]]["answers"][record["qid"]] = record["student"]+ return {+ "schema": "laya-agreement-run/1",+ "run_id": report["config"]["run_ids"]["student"],+ "provenance": report["config"]["provenance"]["student"],+ "cases": cases,+ }+++def main() -> None:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--output-dir", type=Path, required=True)+ parser.add_argument("--parquet", type=Path, help="reuse the pinned test parquet")+ args = parser.parse_args()+ args.output_dir.mkdir(parents=True, exist_ok=True)+ parquet = args.parquet or args.output_dir / "teacher-test.parquet"+ reference, repeat = teacher_runs(parquet)+ write_json(args.output_dir / "reference.json", reference)+ write_json(args.output_dir / "repeat.json", repeat)++ import torch+ import transformers++ import laya+ from laya import evals++ torch.set_num_threads(2)+ torch.manual_seed(0)+ provenance = {+ "model": MODEL,+ "revision": MODEL_REVISION,+ "settings": {+ "device": "cpu", "backend": "eager", "lang": "en",+ "min_confidence": None, "torch_threads": 2,+ "laya_version": laya.__version__, "torch_version": str(torch.__version__),+ "transformers_version": transformers.__version__,+ },+ "seed": 0,+ }+ print(f"Loading {MODEL}@{MODEL_REVISION} on CPU", file=sys.stderr, flush=True)+ with laya.load(MODEL, device="cpu", revision=MODEL_REVISION, backend="eager") as agent:+ report = evals.evaluate_agreement(+ LocalRunner(agent), reference, repeat, student_provenance=provenance,+ bootstrap_samples=2000, seed=0,+ )+ student = student_run(reference, report)+ # The saved run must reproduce the full report, including input fingerprints.+ if evals.compare_agreement(reference, repeat, student) != report:+ raise RuntimeError("saved observations did not reproduce the live evaluation")+ write_json(args.output_dir / "student.json", student)+ write_json(args.output_dir / "report.json", report)+ print(json.dumps(report["overall"], indent=2))+++if __name__ == "__main__":+ main()diff --git a/laya/_agreement.py b/laya/_agreement.pynew file mode 100644index 00000000..07eff427--- /dev/null+++ b/laya/_agreement.py@@ -0,0 +1,316 @@+"""Offline, paired teacher-repeat and student agreement on choice decisions.++Teacher observations are comparison targets, not ground truth. Nothing here computes accuracy+or calibration, calls a teacher service, or estimates a ceiling on student performance.+"""+from copy import deepcopy+import hashlib+import json+from typing import Any, Dict, List, Optional++import numpy as np++from .evals import EvalError+++RUN_SCHEMA = "laya-agreement-run/1"+REPORT_SCHEMA = "laya-agreement-report/1"+++def _text(value: Any, where: str) -> None:+ if not isinstance(value, str) or not value.strip():+ raise EvalError("%s must be a non-empty string" % where)+++def _json(value: Any, where: str) -> str:+ """Strict JSON, preserving object order because it can change model inputs."""+ def check(item):+ if type(item) is dict:+ for key, child in item.items():+ if not isinstance(key, str):+ raise ValueError("object keys must be strings")+ check(child)+ elif type(item) is list:+ for child in item:+ check(child)+ elif type(item) not in (str, int, float, bool, type(None)):+ raise ValueError("unsupported value of type %s" % type(item).__name__)++ try:+ check(value)+ encoded = json.dumps(value, ensure_ascii=False, allow_nan=False, separators=(",", ":"))+ encoded.encode("utf-8")+ return encoded+ except (ValueError, TypeError, RecursionError, UnicodeError) as exc:+ raise EvalError("%s must be strict JSON: %s" % (where, exc)) from exc+++def _fingerprint(value: Any) -> str:+ return hashlib.sha256(_json(value, "fingerprint input").encode("utf-8")).hexdigest()+++def _controls(bootstrap_samples: int, seed: int) -> None:+ if type(bootstrap_samples) is not int or not 100 <= bootstrap_samples <= 100000:+ raise EvalError("bootstrap_samples must be an integer from 100 to 100000")+ if type(seed) is not int or seed < 0:+ raise EvalError("seed must be a non-negative integer")+++def _provenance(value: Any, where: str) -> None:+ if not isinstance(value, dict):+ raise EvalError("%s must be an object" % where)+ _text(value.get("model"), where + ".model")+ if "revision" not in value:+ raise EvalError("%s needs revision (use null when unknown)" % where)+ if value["revision"] is not None:+ _text(value["revision"], where + ".revision")+ if not isinstance(value.get("settings"), dict):+ raise EvalError("%s.settings must be an object" % where)+ _json(value, where)+++def _labels(question: Any, where: str) -> List[str]:+ if not isinstance(question, dict) or question.get("type") != "choice":+ raise EvalError("%s must be a choice question" % where)+ instructions = question.get("instructions")+ if (instructions is None or not isinstance(instructions, (str, dict, list, int, float))+ or (isinstance(instructions, str) and not instructions.strip())+ or (isinstance(instructions, (dict, list)) and not instructions)):+ raise EvalError("%s needs non-empty instructions" % where)+ criteria = question.get("criteria")+ if not isinstance(criteria, (dict, list)) or not criteria:+ raise EvalError("%s.criteria must be a non-empty object or list of string labels" % where)+ # Agent._to_internal preserves dict keys and expands a list to {label: None}.+ # This protocol accepts string labels only, so saved JSON and runtime answers agree.+ labels = list(criteria)+ if any(not isinstance(label, str) for label in labels):+ raise EvalError("%s.criteria labels must be strings" % where)+ if len(set(labels)) != len(labels):+ raise EvalError("%s.criteria contains duplicate labels" % where)+ if "labels" in question:+ raise EvalError("%s: labels is only supported for noul questions" % where)+ if "option_order" in question:+ order = question["option_order"]+ if (not isinstance(order, list) or any(type(i) is not int for i in order)+ or sorted(order) != list(range(len(labels)))):+ raise EvalError("%s.option_order must be a permutation of the option indices" % where)+ return labels+++def _run(value: Any, role: str, student: bool = False) -> Dict[str, Any]:+ if not isinstance(value, dict) or value.get("schema") != RUN_SCHEMA:+ raise EvalError("%s.schema must be %r" % (role, RUN_SCHEMA))+ _json(value, role)+ _text(value.get("run_id"), role + ".run_id")+ _provenance(value.get("provenance"), role + ".provenance")+ cases = value.get("cases")+ if not isinstance(cases, list) or not cases:+ raise EvalError("%s.cases must be a non-empty list" % role)+ by_id = {}+ for index, case in enumerate(cases):+ where = "%s.cases[%d]" % (role, index)+ if not isinstance(case, dict):+ raise EvalError("%s must be an object" % where)+ _text(case.get("id"), where + ".id")+ case_id = case["id"]+ if case_id in by_id:+ raise EvalError("%s: duplicate case id %r" % (role, case_id))+ if case.get("state") is None:+ raise EvalError("%s needs a non-null state" % where)+ questions, answers = case.get("questions"), case.get("answers")+ if not isinstance(questions, dict) or not questions:+ raise EvalError("%s.questions must be a non-empty object" % where)+ if not isinstance(answers, dict) or set(answers) != set(questions):+ raise EvalError("%s.answers must have exactly the question ids" % where)+ for qid, question in questions.items():+ _text(qid, where + ".questions key")+ labels = _labels(question, "%s.questions[%r]" % (where, qid))+ answer = answers[qid]+ if answer is None and student:+ continue+ if not isinstance(answer, str) or answer not in labels:+ raise EvalError("%s.answers[%r] must be a choice label%s"+ % (where, qid, " or null" if student else ""))+ if case.get("language") is not None and not isinstance(case["language"], str):+ raise EvalError("%s.language must be a string or null" % where)+ tags = case.get("tags", [])+ if not isinstance(tags, list) or any(not isinstance(tag, str) for tag in tags):+ raise EvalError("%s.tags must be a list of strings" % where)+ by_id[case_id] = case+ # Sorting cases makes the artifact identity independent of JSONL/array row order.+ normalized = dict(value, cases=[by_id[key] for key in sorted(by_id)])+ return {"run": value, "by_id": by_id, "fingerprint": _fingerprint(normalized)}+++def _align(reference: Dict, other: Dict, role: str) -> None:+ if set(reference["by_id"]) != set(other["by_id"]):+ raise EvalError("%s case ids must exactly match reference case ids" % role)+ for case_id, case in reference["by_id"].items():+ candidate = other["by_id"][case_id]+ for key in ("state", "questions"):+ if _json(case[key], key) != _json(candidate[key], key):+ raise EvalError("%s case %r has different %s (including option order)"+ % (role, case_id, key))+ # Slice labels belong to the inputs, not to the model being compared.+ if (case.get("language") != candidate.get("language")+ or set(case.get("tags", [])) != set(candidate.get("tags", []))):+ raise EvalError("%s case %r has different language or tags" % (role, case_id))+++def _teachers(reference: Any, repeat: Any) -> tuple:+ ref, rep = _run(reference, "reference"), _run(repeat, "repeat")+ if reference["run_id"] == repeat["run_id"]:+ raise EvalError("reference and repeat need distinct run_id values")+ for key in ("model", "revision", "settings"):+ if _json(reference["provenance"][key], key) != _json(repeat["provenance"][key], key):+ raise EvalError("teacher provenance.%s must match between reference and repeat" % key)+ _align(ref, rep, "repeat")+ return ref, rep+++def _aggregate(records: List[Dict]) -> Dict[str, Any]:+ n = len(records)+ teacher = sum(r["teacher_repeat_agreement"] for r in records)+ student = sum(r["student_teacher_agreement"] for r in records)+ return {+ "n_cases": len({r["case_id"] for r in records}), "n_decisions": n,+ "teacher_repeat_agreement": teacher / n,+ "student_teacher_agreement": student / n,+ "student_coverage": sum(r["accepted"] for r in records) / n,+ "paired_agreement_delta": (student - teacher) / n,+ }+++def _interval(records: List[Dict], samples: int, seed: int) -> Optional[List[float]]:+ totals = {}+ for record in records:+ cluster = totals.setdefault(record["case_id"], [0, 0])+ cluster[0] += record["delta"]+ cluster[1] += 1+ if len(totals) < 2:+ return None+ clusters = np.asarray([totals[key] for key in sorted(totals)], dtype=np.int64)+ rng = np.random.default_rng(seed)+ draws = np.empty(samples, dtype=float)+ # Each replicate resamples whole cases. Ratio of sums keeps the estimand per decision+ # when cases have different question counts; batching bounds the index matrix memory.+ batch = max(1, min(samples, 131072 // len(clusters)))+ for start in range(0, samples, batch):+ stop = min(samples, start + batch)+ indices = rng.integers(len(clusters), size=(stop - start, len(clusters)))+ sums = clusters[indices].sum(axis=1)+ draws[start:stop] = sums[:, 0] / sums[:, 1]+ return np.percentile(draws, [2.5, 97.5]).tolist()+++def _report(ref: Dict, rep: Dict, stu: Dict, bootstrap_samples: int, seed: int) -> Dict[str, Any]:+ records = []+ for case_id in sorted(ref["by_id"]):+ case = ref["by_id"][case_id]+ input_fingerprint = _fingerprint({"state": case["state"], "questions": case["questions"]})+ for qid in sorted(case["questions"]):+ first = case["answers"][qid]+ second = rep["by_id"][case_id]["answers"][qid]+ student = stu["by_id"][case_id]["answers"][qid]+ teacher_agreement, student_agreement = second == first, student == first+ records.append({+ "case_id": case_id, "qid": qid, "teacher_reference": first,+ "teacher_repeat": second, "student": student,+ "teacher_repeat_agreement": teacher_agreement,+ "student_teacher_agreement": student_agreement, "accepted": student is not None,+ "delta": int(student_agreement) - int(teacher_agreement),+ "input_fingerprint": input_fingerprint, "language": case.get("language"),+ "tags": sorted(set(case.get("tags", []))),+ })+ overall = _aggregate(records)+ overall["paired_delta_ci95"] = _interval(records, bootstrap_samples, seed)+ slices = {}+ for dimension in ("qid", "language", "tag"):+ groups = {}+ for record in records:+ values = record["tags"] if dimension == "tag" else [record[dimension]]+ for value in values:+ if value is not None:+ groups.setdefault(value, []).append(record)+ slices[dimension] = {key: _aggregate(groups[key]) for key in sorted(groups)}+ runs = {"reference": ref, "repeat": rep, "student": stu}+ limitations = [+ "Agreement is not accuracy, a student performance ceiling, or evidence that a teacher can be replaced.",+ "Teacher independence and provenance are declared by the caller, not verified by this report.",+ "The paired interval resamples original cases with their saved observations; it does not resample teacher calls.",+ ]+ unknown = [role for role, data in runs.items() if data["run"]["provenance"]["revision"] is None]+ if unknown:+ limitations.append("Unknown model revision for %s; the observations cannot be fully regenerated from this provenance."+ % ", ".join(unknown))+ if overall["n_cases"] < 2:+ limitations.append("A paired interval needs at least two original cases; paired_delta_ci95 is null.")+ return {+ "schema": REPORT_SCHEMA,+ "config": {+ "protocol": "teacher-reference-paired/1", "bootstrap_samples": bootstrap_samples,+ "seed": seed, "cluster_unit": "case_id", "confidence_level": 0.95,+ "interval_method": "paired case bootstrap, percentile, ratio of decision sums",+ "provenance": {role: deepcopy(data["run"]["provenance"]) for role, data in runs.items()},+ "run_ids": {role: data["run"]["run_id"] for role, data in runs.items()},+ "input_fingerprints": {role: data["fingerprint"] for role, data in runs.items()},+ },+ "overall": overall, "slices": slices, "cases": records, "limitations": limitations,+ }+++def compare_agreement(reference: Dict, repeat: Dict, student: Dict, *,+ bootstrap_samples: int = 2000, seed: int = 0) -> Dict[str, Any]:+ """Compare three saved choice runs, paired by case id and question id.++ A student null is an abstention: it contributes zero agreement and zero coverage on the+ same denominator as answered decisions. Missing cases or answers are errors.+ """+ _controls(bootstrap_samples, seed)+ ref, rep = _teachers(reference, repeat)+ stu = _run(student, "student", student=True)+ if student["run_id"] in (reference["run_id"], repeat["run_id"]):+ raise EvalError("student needs a run_id distinct from both teacher runs")+ _align(ref, stu, "student")+ return _report(ref, rep, stu, bootstrap_samples, seed)+++def evaluate_agreement(runner: Any, reference: Dict, repeat: Dict, *,+ student_provenance: Dict, model: Optional[str] = None,+ bootstrap_samples: int = 2000, seed: int = 0) -> Dict[str, Any]:+ """Run a local student once per case after validating both teacher observations.++ Provenance describes the caller's runner; no checkpoint revision is inferred or fabricated.+ The runner receives copied inputs and its exceptions propagate without dropping cases.+ """+ _controls(bootstrap_samples, seed)+ ref, rep = _teachers(reference, repeat)+ _provenance(student_provenance, "student_provenance")+ if model is not None:+ _text(model, "model")+ cases = []+ for case_id in sorted(ref["by_id"]):+ case = ref["by_id"][case_id]+ result = runner.predict(deepcopy(case["state"]), deepcopy(case["questions"]), model=model)+ answers = result.get("answers") if isinstance(result, dict) else None+ if not isinstance(answers, dict) or set(answers) != set(case["questions"]):+ raise EvalError("student case %r must return exactly the question ids" % case_id)+ choices = {}+ for qid, question in case["questions"].items():+ answer = answers[qid]+ where = "student case %r question %r" % (case_id, qid)+ if not isinstance(answer, dict) or answer.get("type") != "choice":+ raise EvalError("%s must return a choice answer" % where)+ choice = answer.get("choice")+ if not isinstance(choice, str) or choice not in _labels(question, where):+ raise EvalError("%s returned an invalid choice label" % where)+ choices[qid] = None if answer.get("abstention") == "abstained" else choice+ cases.append(dict(deepcopy(case), answers=choices))+ run_id, suffix = "student", 0+ while run_id in (reference["run_id"], repeat["run_id"]):+ suffix += 1+ run_id = "student-%d" % suffix+ student = {"schema": RUN_SCHEMA, "run_id": run_id,+ "provenance": deepcopy(student_provenance), "cases": cases}+ stu = _run(student, "student", student=True)+ return _report(ref, rep, stu, bootstrap_samples, seed)diff --git a/laya/evals.py b/laya/evals.pyindex 53035722..b066859c 100644--- a/laya/evals.py+++ b/laya/evals.py@@ -66,6 +66,38 @@ class EvalError(ValueError):"""A dataset or report is malformed; the message names the row or field."""+def compare_agreement(reference: Dict, repeat: Dict, student: Dict, *,+ bootstrap_samples: int = 2000, seed: int = 0) -> Dict[str, Any]:+ """Compare saved teacher reference, teacher repeat, and student choice observations.++ Inputs use ``laya-agreement-run/1``; see the evaluation guide for the schema.+ All runs must contain the same cases and questions. Student nulls count as+ disagreements and reduce coverage. The report includes a paired, case-clustered+ bootstrap interval (100–100000 replicates), provenance, and individual decisions.+ Teacher agreement is not ground-truth accuracy or a ceiling on student performance.+ """+ from ._agreement import compare_agreement as compare++ return compare(reference, repeat, student, bootstrap_samples=bootstrap_samples, seed=seed)+++def evaluate_agreement(runner: Any, reference: Dict, repeat: Dict, *,+ student_provenance: Dict, model: Optional[str] = None,+ bootstrap_samples: int = 2000, seed: int = 0) -> Dict[str, Any]:+ """Evaluate a local student's agreement with two saved teacher observations.++ Validate both teacher runs before calling ``runner.predict(state, questions,+ model=model)`` once per original case. Record the caller's student provenance;+ use a null revision when it is unknown. Runner errors propagate, and missing or+ invalid answers raise ``EvalError`` instead of dropping decisions. An answer's+ ``abstention='abstained'`` becomes a null student observation.+ """+ from ._agreement import evaluate_agreement as evaluate_student++ return evaluate_student(runner, reference, repeat, student_provenance=student_provenance,+ model=model, bootstrap_samples=bootstrap_samples, seed=seed)++@dataclassclass Example:"""One labelled decision: a state, its questions, and the expected answer per question id."""diff --git a/laya/evals_cli.py b/laya/evals_cli.pyindex 174eb220..0d8ff5ee 100644--- a/laya/evals_cli.py+++ b/laya/evals_cli.py@@ -225,6 +225,17 @@ def _build_parser() -> argparse.ArgumentParser:run.add_argument("--json", dest="json_out", help="write the full report JSON here")run.add_argument("--markdown", dest="markdown_out", help="write a Markdown summary here")+ agreement = sub.add_parser("agreement", help="compare saved teacher and student choice runs")+ agreement.add_argument("reference", help="reference teacher run JSON")+ agreement.add_argument("repeat", help="independent repeat teacher run JSON")+ agreement.add_argument("student", help="student run JSON on the same cases")+ agreement.add_argument("--bootstrap-samples", type=int, default=2000,+ help="case bootstrap replicates, 100–100000 (default '2000')")+ agreement.add_argument("--seed", type=int, default=0,+ help="non-negative bootstrap seed (default '0')")+ agreement.add_argument("--json", dest="json_out", metavar="PATH",+ help="write the agreement report to PATH instead of stdout")+compare = sub.add_parser("compare", help="compare a report JSON against a baseline")compare.add_argument("report", help="a report JSON written by `laya-evals run --json`")compare.add_argument("--baseline", required=True,@@ -488,6 +499,35 @@ def _cmd_compare(args) -> int:return 0 if ok and not failures else 1+def _cmd_agreement(args) -> int:+ def unique_object(pairs):+ obj = {}+ for key, value in pairs:+ if key in obj:+ raise ValueError("duplicate JSON key %r" % key)+ obj[key] = value+ return obj++ runs = []+ for path in (args.reference, args.repeat, args.student):+ try:+ with open(path, encoding="utf-8") as handle:+ runs.append(json.load(handle, object_pairs_hook=unique_object))+ except (OSError, ValueError, RecursionError) as exc:+ raise EvalError("cannot read agreement run %r: %s" % (path, exc)) from exc+ report = evals.compare_agreement(*runs, bootstrap_samples=args.bootstrap_samples, seed=args.seed)+ encoded = json.dumps(report, ensure_ascii=False, indent=2, allow_nan=False) + "\n"+ if args.json_out:+ try:+ with open(args.json_out, "w", encoding="utf-8", newline="\n") as handle:+ handle.write(encoded)+ except OSError as exc:+ raise EvalError("cannot write agreement report %r: %s" % (args.json_out, exc)) from exc+ else:+ print(encoded, end="")+ return 0++def main(argv: Optional[Sequence[str]] = None) -> int:args = _build_parser().parse_args(argv)# Same reason as `laya.cli.main`: redirected output is encoded with the locale's codec by@@ -497,6 +537,8 @@ def main(argv: Optional[Sequence[str]] = None) -> int:if hasattr(sys.stdout, "reconfigure"):sys.stdout.reconfigure(encoding="utf-8")try:+ if args.command == "agreement":+ return _cmd_agreement(args)if args.command == "validate":return _cmd_validate(args)if args.command == "run":diff --git a/scripts/test_suites.py b/scripts/test_suites.pyindex 089fdf77..c64b38d4 100644--- a/scripts/test_suites.py+++ b/scripts/test_suites.py@@ -104,6 +104,7 @@ SCRIPT_SUITES = []PYTEST_SUITES = [+ "tests/test_evals_agreement.py","tests/test_serve.py","tests/test_mcp_remote.py","tests/test_router_batch.py",diff --git a/tests/test_evals_agreement.py b/tests/test_evals_agreement.pynew file mode 100644index 00000000..394bf0aa--- /dev/null+++ b/tests/test_evals_agreement.py@@ -0,0 +1,357 @@+"""Model-free checks for paired agreement and case-level uncertainty."""+from copy import deepcopy+import json+from pathlib import Path+import subprocess+import sys++import pytest++from laya import evals+++def run_document(run_id, rows, *, teacher=True):+ cases = []+ for index, labels in enumerate(rows):+ questions = {+ "q%d" % q: {"type": "choice", "instructions": "Choose a route.",+ "criteria": {"a": "Accept", "b": "Defer"}}+ for q in range(len(labels))+ }+ cases.append({+ "id": "case-%d" % index,+ "state": {"text": "request-%d" % index, "context": ["original"]},+ "questions": questions,+ "answers": dict(zip(questions, labels)),+ "tags": ["routing"], "language": "en",+ })+ return {"schema": "laya-agreement-run/1", "run_id": run_id,+ "provenance": {"model": "teacher" if teacher else "student",+ "revision": "pinned", "settings": {"temperature": 0.5}},+ "cases": cases}+++def documents(reference=(("a",), ("b",)), repeat=None, student=None):+ return (run_document("reference", reference),+ run_document("repeat", repeat if repeat is not None else reference),+ run_document("student", student if student is not None else reference,+ teacher=False))+++def compare(runs, **kwargs):+ return evals.compare_agreement(*runs, **kwargs)+++def test_paired_rates_use_every_decision_and_preserve_abstention_denominator():+ runs = documents((("a", "a", "a"), ("b", "a")),+ (("a", "b", "a"), ("b", "a")),+ (("a", "b", None), ("b", "b")))+ report = compare(runs)+ assert report["schema"] == "laya-agreement-report/1"+ overall = report["overall"]+ assert overall["n_cases"] == 2+ assert overall["n_decisions"] == 5+ assert overall["teacher_repeat_agreement"] == pytest.approx(4 / 5)+ assert overall["student_teacher_agreement"] == pytest.approx(2 / 5)+ assert overall["student_coverage"] == pytest.approx(4 / 5)+ assert overall["paired_agreement_delta"] == pytest.approx(-2 / 5)+ assert len(report["cases"]) == 5+ observed = {(row["case_id"], row["qid"]):+ (row["teacher_reference"], row["teacher_repeat"], row["student"],+ row["accepted"], row["delta"]) for row in report["cases"]}+ assert observed == {+ ("case-0", "q0"): ("a", "a", "a", True, 0),+ ("case-0", "q1"): ("a", "b", "b", True, 0),+ ("case-0", "q2"): ("a", "a", None, False, -1),+ ("case-1", "q0"): ("b", "b", "b", True, 0),+ ("case-1", "q1"): ("a", "a", "b", True, -1),+ }+ assert report["slices"]["qid"]["q2"]["n_decisions"] == 1+ assert report["slices"]["qid"]["q2"]["student_coverage"] == 0.0+ assert not {"accuracy", "choice_accuracy", "ece", "brier", "aurc"} & overall.keys()+ assert json.loads(json.dumps(report)) == report++[email protected]("repeat,student,agreement", [("a", "a", 1.0), ("b", "b", 0.0)])+def test_perfect_and_zero_agreement(repeat, student, agreement):+ overall = compare(documents((("a",), ("a",)),+ ((repeat,), (repeat,)),+ ((student,), (student,))))["overall"]+ assert overall["teacher_repeat_agreement"] == agreement+ assert overall["student_teacher_agreement"] == agreement+ assert overall["paired_agreement_delta"] == 0.0+ assert tuple(overall["paired_delta_ci95"]) == (0.0, 0.0)+++def test_student_abstention_is_not_a_missing_case_or_an_agreeing_answer():+ overall = compare(documents(student=((None,), (None,))))["overall"]+ assert overall["n_cases"] == overall["n_decisions"] == 2+ assert overall["student_coverage"] == 0.0+ assert overall["student_teacher_agreement"] == 0.0+ assert overall["paired_agreement_delta"] == -1.0+ assert tuple(overall["paired_delta_ci95"]) == (-1.0, -1.0)+++def test_alignment_uses_case_ids_and_comparison_does_not_mutate_inputs():+ runs = documents((("a",), ("b",), ("a",)),+ (("a",), ("a",), ("b",)),+ (("b",), ("b",), ("a",)))+ original = deepcopy(runs)+ expected = compare(runs, seed=31)["overall"]+ assert runs == original+ runs[0]["cases"].reverse()+ runs[1]["cases"] = runs[1]["cases"][1:] + runs[1]["cases"][:1]+ assert compare(runs, seed=31)["overall"] == expected+++def test_one_case_has_no_interval_even_with_many_questions():+ overall = compare(documents((("a",) * 30,)))["overall"]+ assert overall["n_cases"] == 1+ assert overall["n_decisions"] == 30+ assert overall["paired_delta_ci95"] is None+++def test_bootstrap_resamples_whole_cases_not_correlated_questions():+ # There are just two independent cases. Resampling two cases can give only+ # delta -1, 0 or +1, regardless of how many questions each case contains.+ runs = documents((("a",) * 25, ("a",) * 25),+ (("a",) * 25, ("b",) * 25),+ (("b",) * 25, ("a",) * 25))+ report = compare(runs, bootstrap_samples=2000, seed=17)+ overall = report["overall"]+ assert overall["paired_agreement_delta"] == 0.0+ assert tuple(overall["paired_delta_ci95"]) == (-1.0, 1.0)+ assert compare(runs, bootstrap_samples=2000, seed=17) == report+++def test_bootstrap_preserves_decision_weights_with_unequal_case_sizes():+ runs = documents((("a",) * 100, ("a",), ("a",), ("a",)),+ (("b",) * 100, ("a",), ("a",), ("a",)),+ (("a",) * 100, ("b",), ("b",), ("b",)))+ overall = compare(runs, bootstrap_samples=5000, seed=23)["overall"]+ assert overall["paired_agreement_delta"] == pytest.approx(97 / 103)+ # K large-case draws follows Binomial(4, 1/4). The 2.5th/97.5th percentiles+ # have K=0/3: ratios -4/4 and (300-1)/(300+1), not case-mean rates -1/0.5.+ assert overall["paired_delta_ci95"] == pytest.approx([-1.0, 299 / 301])++[email protected]("side", [0, 1, 2])[email protected]("change", ["missing", "duplicate", "unknown"])+def test_case_sets_must_match_exactly(side, change):+ runs = documents()+ if change == "missing":+ runs[side]["cases"].pop()+ elif change == "duplicate":+ runs[side]["cases"].append(deepcopy(runs[side]["cases"][0]))+ else:+ runs[side]["cases"][0]["id"] = "unrelated"+ with pytest.raises(ValueError):+ compare(runs)++[email protected]("change", ["state", "prompt", "criteria_order", "criteria_text", "qid"])+def test_input_and_question_drift_are_rejected(change):+ runs = documents()+ case = runs[1]["cases"][0]+ question = case["questions"]["q0"]+ if change == "state":+ case["state"]["context"].append("changed")+ elif change == "prompt":+ question["instructions"] = "Choose the opposite route."+ elif change == "criteria_order":+ question["criteria"] = {"b": "Defer", "a": "Accept"}+ elif change == "criteria_text":+ question["criteria"]["a"] = "Reject"+ else:+ case["questions"]["other"] = case["questions"].pop("q0")+ case["answers"]["other"] = case["answers"].pop("q0")+ with pytest.raises(ValueError):+ compare(runs)++[email protected]("key,value", [("model", "other"), ("revision", "other-pin"),+ ("settings", {"temperature": 0.9})])+def test_teacher_configuration_must_match(key, value):+ runs = documents()+ runs[1]["provenance"][key] = value+ with pytest.raises(ValueError):+ compare(runs)+++def test_teacher_seed_may_differ():+ runs = documents()+ runs[0]["provenance"]["seed"] = 5+ runs[1]["provenance"]["seed"] = 8+ assert compare(runs)["overall"]["teacher_repeat_agreement"] == 1.0+++def test_unknown_teacher_revision_remains_unknown_and_disclosed():+ runs = documents()+ for run in runs[:2]:+ run["provenance"]["revision"] = None+ report = compare(runs)+ assert report["config"]["provenance"]["reference"]["revision"] is None+ assert report["config"]["provenance"]["repeat"]["revision"] is None+ assert any("revision" in note.lower() for note in report["limitations"])++[email protected]("name,value", [("bootstrap_samples", 99), ("bootstrap_samples", 100001),+ ("bootstrap_samples", True), ("bootstrap_samples", 100.5), ("bootstrap_samples", float("inf")),+ ("seed", -1), ("seed", True), ("seed", 1.5), ("seed", float("nan"))])+def test_bootstrap_options_require_bounded_integer_values(name, value):+ with pytest.raises(ValueError):+ compare(documents(), **{name: value})++[email protected]("change", ["schema", "empty", "score", "teacher_abstention",+ "unknown_label", "missing_answer", "extra_answer", "nonfinite"])+def test_malformed_runs_are_rejected(change):+ runs = documents()+ case = runs[0]["cases"][0]+ if change == "schema":+ runs[0]["schema"] = "laya-agreement-run/999"+ elif change == "empty":+ for run in runs:+ run["cases"] = []+ elif change == "score":+ case["questions"]["q0"]["type"] = "score"+ elif change == "teacher_abstention":+ case["answers"]["q0"] = None+ elif change == "unknown_label":+ case["answers"]["q0"] = "unlisted"+ elif change == "missing_answer":+ case["answers"].pop("q0")+ elif change == "extra_answer":+ case["answers"]["other"] = "a"+ else:+ for run in runs:+ run["provenance"]["settings"]["temperature"] = float("nan")+ with pytest.raises(ValueError):+ compare(runs)+++_DEFAULT_RESULT = object()+++class RecordingRunner:+ def __init__(self, result=_DEFAULT_RESULT, mutate=False):+ self.calls = []+ self.result = result+ self.mutate = mutate++ def predict(self, state, questions, model=None):+ self.calls.append((deepcopy(state), deepcopy(questions), model))+ answers = {qid: {"type": "choice", "choice": "a"} for qid in questions}+ if self.mutate:+ state["context"].append("runner mutation")+ questions["q0"]["criteria"]["a"] = "runner mutation"+ return {"answers": answers} if self.result is _DEFAULT_RESULT else self.result+++def evaluate(runner, runs, **kwargs):+ return evals.evaluate_agreement(runner, runs[0], runs[1],+ student_provenance=runs[2]["provenance"], **kwargs)+++def test_evaluate_forwards_model_and_isolates_inputs_from_runner_mutation():+ runs = documents()+ original = deepcopy(runs)+ runner = RecordingRunner(mutate=True)+ report = evaluate(runner, runs, model="local-checkpoint")+ assert runs == original+ assert len(runner.calls) == 2+ assert all(call[2] == "local-checkpoint" for call in runner.calls)+ assert report["overall"]["student_teacher_agreement"] == 0.5+++def test_evaluate_treats_abstained_argmax_as_uncovered():+ runner = RecordingRunner({"answers": {"q0": {"type": "choice", "choice": "a",+ "abstention": "abstained"}}})+ overall = evaluate(runner, documents())["overall"]+ assert overall["student_coverage"] == overall["student_teacher_agreement"] == 0.0++[email protected]("result", [None, "invalid", {}, {"answers": {}},+ {"answers": {"q0": {"type": "choice", "choice": "unlisted"}}},+ {"answers": {"q0": {"type": "score", "score": 0.4}}},+ {"answers": {"q0": {"type": "choice", "choice": None}}}])+def test_evaluate_rejects_invalid_runner_results(result):+ with pytest.raises(ValueError):+ evaluate(RecordingRunner(result), documents())++[email protected]("invalid", ["alignment", "bootstrap", "provenance", "null_state"])+def test_evaluate_validates_every_input_before_first_runner_call(invalid):+ runs = documents()+ kwargs = {}+ if invalid == "alignment":+ runs[1]["cases"][-1]["state"] = "changed final case"+ elif invalid == "bootstrap":+ kwargs["bootstrap_samples"] = True+ elif invalid == "null_state":+ for run in runs:+ run["cases"][-1]["state"] = None+ else:+ runs[2]["provenance"]["settings"] = {"temperature": float("inf")}+ runner = RecordingRunner()+ with pytest.raises(ValueError):+ evaluate(runner, runs, **kwargs)+ assert runner.calls == []+++def cli(*args):+ return subprocess.run([sys.executable, "-m", "laya.evals_cli", "agreement", *map(str, args)],+ cwd=Path(__file__).resolve().parents[1], capture_output=True,+ text=True, encoding="utf-8", timeout=30)+++def saved_runs(tmp_path):+ runs = documents()+ for run in runs:+ run["cases"][0]["state"] = "退款申请"+ paths = [tmp_path / (run["run_id"] + ".json") for run in runs]+ for path, run in zip(paths, runs):+ path.write_text(json.dumps(run, ensure_ascii=False), encoding="utf-8")+ return paths, runs+++def test_cli_saved_run_replay_matches_public_api(tmp_path):+ paths, runs = saved_runs(tmp_path)+ expected = compare(runs, bootstrap_samples=100, seed=12)+ args = [*paths, "--bootstrap-samples", "100", "--seed", "12"]+ result = cli(*args)+ assert result.returncode == 0, result.stderr+ assert json.loads(result.stdout) == expected+ output = tmp_path / "结果.json"+ result = cli(*args, "--json", output)+ assert result.returncode == 0, result.stderr+ assert not result.stdout+ assert json.loads(output.read_text(encoding="utf-8")) == expected++[email protected]("invalid", ["duplicate_key", "nan", "malformed", "nested", "missing", "misaligned"])+def test_cli_refuses_bad_inputs_without_a_report(tmp_path, invalid):+ paths, runs = saved_runs(tmp_path)+ if invalid == "duplicate_key":+ # Reject ambiguity even when the duplicate has the same value.+ raw = paths[0].read_text(encoding="utf-8")+ paths[0].write_text(raw.replace('"schema":', '"schema":"laya-agreement-run/1","schema":', 1),+ encoding="utf-8")+ elif invalid == "nan":+ runs[0]["provenance"]["settings"]["temperature"] = float("nan")+ paths[0].write_text(json.dumps(runs[0]), encoding="utf-8")+ elif invalid == "malformed":+ paths[0].write_text("{broken", encoding="utf-8")+ elif invalid == "nested":+ paths[0].write_text("[" * 5000 + "0" + "]" * 5000, encoding="utf-8")+ elif invalid == "missing":+ paths[0].unlink()+ else:+ runs[2]["cases"].pop()+ paths[2].write_text(json.dumps(runs[2]), encoding="utf-8")+ output = tmp_path / "report.json"+ result = cli(*paths, "--json", output)+ assert result.returncode == 2+ assert "laya-evals:" in result.stderr+ assert "Traceback" not in result.stderr+ assert not output.exists()diff --git a/tests/test_evals_api.py b/tests/test_evals_api.pyindex 1861d9a3..c7fee1ff 100644--- a/tests/test_evals_api.py+++ b/tests/test_evals_api.py@@ -44,6 +44,7 @@ check_true("ScoreWithin names its tolerance", evals.ScoreWithin(0.25).name == "s# --------------------------------------------------------------- exports / callablesfor name in ("Dataset", "Example", "EvalError", "EvalReport", "evaluate", "ece", "assert_regression","REPORT_SCHEMA", "questions_fingerprint", "file_fingerprint",+ "compare_agreement", "evaluate_agreement","brier", "aurc", "selective_accuracy", "is_confidence_metric",# `_eval_policy` imports these three, so they are a contract between two modules in# this package, not internals -- a rename would break the release gate, not just a@@ -158,7 +159,7 @@ def _subparsers(parser):_subs = _subparsers(evals_cli._build_parser())-check("laya-evals subcommands", sorted(_subs), ["compare", "evidence", "run", "validate"])+check("laya-evals subcommands", sorted(_subs), ["agreement", "compare", "evidence", "run", "validate"])_unhelped, _unstated, _options = [], [], {}for _sub, _parser in sorted(_subs.items()):diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.pyindex 41c53485..9b4b0c6d 100644--- a/tests/test_hooks_api.py+++ b/tests/test_hooks_api.py@@ -1512,6 +1512,16 @@ finally:_hooks_mod.clear_default_hooks()+from laya.evals import compare_agreement, evaluate_agreement # noqa: E402++check("compare_agreement signature", list(inspect.signature(compare_agreement).parameters),+ ["reference", "repeat", "student", "bootstrap_samples", "seed"])+check("evaluate_agreement signature", list(inspect.signature(evaluate_agreement).parameters),+ ["runner", "reference", "repeat", "student_provenance", "model", "bootstrap_samples", "seed"])+check("agreement provenance is required",+ inspect.signature(evaluate_agreement).parameters["student_provenance"].default,+ inspect.Parameter.empty)+print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))for f in FAIL:print(" FAIL", f)