Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
research(eval): add metamorphic option-permutation robustness checks (#269)
Reskyumeb731f2eb9bbc
diff --git a/research/eval/README.md b/research/eval/README.mdindex adf2e7f6..b5b3e407 100644--- a/research/eval/README.md+++ b/research/eval/README.md@@ -318,3 +318,72 @@ checks, and an order-invariant model scores exactly 1/3.reduced-precision autocast and `score_cases` does not. The parity check reports thatdifference instead of hiding it.* New checks are one function each, registered in `CHECKS`.+++## Metamorphic option-order robustness (experimental)++`metamorphic.py` adds the initial scope of+[#244](https://github.com/NandhaKishorM/laya/issues/244): **choice option order robustness**, without changing model/runtime behavior. Label renaming, neutral labels, paraphrases, structured-state permutations, `score` and `noul` perturbations are intentionally deferred. Run from the repository root after installing Laya and `datasets`:++```bash+python -m research.eval.metamorphic --model convaiinnovations/laya \+ --langs en --per-lang 100 --n-opts 20 --batch-size 16 --out robustness.json+python -m research.eval.metamorphic --model convaiinnovations/laya \+ --subfolder multilingual --langs en --out multilingual-robustness.json+python -m unittest research.eval.test_metamorphic -v+```++Each MASSIVE case uses the existing harness's sampler and produces two inputs:++1. The unchanged baseline.+2. One seeded shuffle of option order; if the shuffle is the identity, a one-slot+ rotation is used. This is a bounded diagnostic, not exhaustive permutation testing+ or a uniform draw over all nonidentity permutations.++Instructions and state are otherwise unchanged. Option key/value pairs are moved together during permutation. Every result is mapped back to the original semantic option order **before** predictions and metrics are computed. Exact ties choose the first canonical option. The RNG starts fresh per language; `--seed` controls both sampling and transformations. `--batch-size` bounds the number of forward-pass inputs and does not alter the generated variants. Model inference may still have small floating-point differences across devices and batch sizes.++The JSON contains `config`, per-language `report`, and full `cases`. Each case saves its original input, canonical keys and optional gold index; each variant saves its presented keys, explicit `canonical_to_transformed` and `transformed_to_canonical` label mappings, slot-to-canonical indices, complete **canonical-order** probability vector, prediction, confidence, correctness (or `null`), and comparison to baseline. Probabilities are not rounded. The config records model/subfolder, temperature mode and values, truncation settings, dataset, seed and batch size. For reproducible checkpoint comparisons, use a pinned local snapshot and retain the environment versions alongside the report. `--unclamped` has the same meaning as in `laya_eval`. If any language fails, its error is saved and the command exits nonzero while retaining successful languages.++Metrics are grouped under `option_order` and `overall`:++| Metric | Definition |+|---|---|+| `semantic_agreement_rate` | Fraction of baseline/variant pairs with the same canonical argmax |+| `mean_probability_drift` | Mean absolute probability change across options, then pairs |+| `max_probability_drift` | Largest absolute change of any option across all pairs |+| `mean_js_divergence` | Mean Jensen-Shannon divergence using natural logs, in `[0, ln(2)]` |+| `mean_confidence_drift` | Mean signed change of maximum probability, variant minus baseline |+| `mean_absolute_confidence_drift` | Mean magnitude of that confidence change |+| `worst_confidence_increase_on_disagreement` | Largest positive confidence change among changed decisions, or zero if none |++`overall` is pair-weighted, not a fraction of cases where *all* variants agree. `quality` separately reports accuracy and the existing harness's 15-bin ECE for baseline and each transformation on labelled cases only. Empty groups contain `n: 0`; unlabelled quality groups contain `n_labelled: 0` without inventing an accuracy or ECE. Robustness agreement is not a correctness measure: consistently wrong predictions can be perfectly invariant.++For another corpus, the Python API accepts `(state, questions)` cases in the same shape as the harness, and a callback returning probability vectors in presented option order:++```python+from research.eval.metamorphic import evaluate, model_scorer+agent.model.eval()+result = evaluate(cases, model_scorer(agent), gold_indices=None, seed=13)+```++The first version intentionally accepts only **one choice question per case**, with at least two options. Label renaming and other metamorphic transforms are intentionally deferred as proposed in the issue.++For an explicit single-case experiment, the same implementation exposes:++```python+from research.eval.metamorphic import (+ MetamorphicCase, permute_options,+ evaluate_variants, compare_predictions,+)++case = MetamorphicCase(state, questions, gold_index=None)+variants = [permute_options(case, seed=42)]+agent.model.eval()+results = evaluate_variants(agent, case, variants)+report = compare_predictions(baseline=results.baseline, variants=results.variants)+```++For offline tests, pass `agent=None, score=fake_scorer` to `evaluate_variants`. The scorer takes a batch of harness `(state, questions)` inputs and returns one probability vector per input. The public transformations return independent copies and explicit mappings in both directions, including identity label mappings for order-only transformations.++**Semantic agreement and distribution stability are different properties.**+A shift from `[0.91, 0.06, 0.03]` to `[0.88, 0.08, 0.04]` preserves the decision while showing nonzero drift. Switching the winner is reported as disagreement, regardless of whether confidence rises or falls. No metric here automatically classifies either observation as a bug; acceptable variation depends on the use case, and the report deliberately defines no universal pass/fail threshold.diff --git a/research/eval/metamorphic.py b/research/eval/metamorphic.pynew file mode 100644index 00000000..097ee1c5--- /dev/null+++ b/research/eval/metamorphic.py@@ -0,0 +1,344 @@+"""Small, reproducible choice-invariance diagnostics (issue #244).++Run from the checkout: python -m research.eval.metamorphic --help.+Pure transformations/metrics can also be used without a model or gold labels.+"""+from __future__ import annotations++import argparse+from copy import deepcopy+from dataclasses import dataclass+import json+import math+import random+from typing import Any, Callable, Sequence++from . import laya_eval as harness+++@dataclass+class MetamorphicCase:+ """One canonical choice decision in the existing harness input format."""++ state: Any+ questions: dict+ gold_index: int | None = None++ def __post_init__(self):+ if len(self.questions) != 1:+ raise ValueError("expected exactly one choice question per case")+ question = next(iter(self.questions.values()))+ criteria = question.get("criteria")+ if question.get("type") != "choice" or not isinstance(criteria, dict):+ raise ValueError("expected choice criteria as a dictionary")+ if len(criteria) < 2:+ raise ValueError("at least two options are required")+ if any(not isinstance(k, str) or not k.strip() for k in criteria):+ raise ValueError("option keys must be nonempty strings")+ if self.gold_index is not None and (+ type(self.gold_index) is not int or not 0 <= self.gold_index < len(criteria)):+ raise ValueError("gold index outside canonical options")++ @property+ def option_keys(self):+ return list(next(iter(self.questions.values()))["criteria"])++ def as_pair(self):+ return self.state, self.questions+++@dataclass+class MetamorphicVariant:+ """Transformed input plus an explicit bidirectional semantic label mapping."""++ kind: str+ case: MetamorphicCase+ canonical_to_transformed: dict[str, str]++ @property+ def transformed_to_canonical(self):+ return {v: k for k, v in self.canonical_to_transformed.items()}++ def as_record(self):+ mapping = self.canonical_to_transformed+ inverse = self.transformed_to_canonical+ keys = list(mapping)+ if len(inverse) != len(mapping) or set(inverse) != set(self.case.option_keys):+ raise ValueError("label mapping must be a bijection over transformed options")+ return {"kind": self.kind, "case": self.case.as_pair(),+ "canonical_to_transformed": dict(mapping),+ "transformed_to_canonical": inverse,+ "canonical_indices": [keys.index(inverse[k]) for k in self.case.option_keys]}+++def _transform(case, kind, order, labels):+ questions = deepcopy(case.questions)+ question = next(iter(questions.values()))+ criteria = question["criteria"]+ keys = case.option_keys+ mapping = dict(zip(keys, labels))+ question["criteria"] = {mapping[keys[i]]: deepcopy(criteria[keys[i]]) for i in order}+ gold = None if case.gold_index is None else order.index(case.gold_index)+ return MetamorphicVariant(kind, MetamorphicCase(deepcopy(case.state), questions, gold), mapping)+++def permute_options(case: MetamorphicCase, seed: int = harness.SEED):+ """One deterministic nonidentity shuffle; identity falls back to rotation."""+ order = list(range(len(case.option_keys)))+ random.Random(seed).shuffle(order)+ if order == list(range(len(order))):+ order = order[1:] + order[:1]+ return _transform(case, "option_order", order, case.option_keys)+++def _baseline(case):+ return _transform(case, "baseline", list(range(len(case.option_keys))), case.option_keys)+++def make_variants(case, rng: random.Random):+ """Adapt the harness tuple format to baseline plus option permutation.++ Label renaming, paraphrases, and other metamorphic transforms are intentionally deferred.+ """+ canonical = MetamorphicCase(*case)+ return [v.as_record() for v in (+ _baseline(canonical), permute_options(canonical, rng.getrandbits(64)))]+++def canonicalize(probabilities, canonical_indices):+ """Validate a probability vector and restore canonical semantic ordering."""+ values = [float(p) for p in probabilities]+ n = len(values)+ if not n or any(type(i) is not int for i in canonical_indices) or sorted(canonical_indices) != list(range(n)):+ raise ValueError("probabilities and canonical mapping must be a bijection")+ if any(not math.isfinite(p) or p < 0 or p > 1 for p in values):+ raise ValueError("probabilities must be finite values in [0, 1]")+ if not math.isclose(sum(values), 1.0, abs_tol=1e-6, rel_tol=0):+ raise ValueError("probabilities must sum to one")+ restored = [0.0] * n+ for p, index in zip(values, canonical_indices):+ restored[index] = p+ return restored+++def distribution_metrics(baseline, variant):+ """Drift in canonical space; JS uses natural logs (range 0..ln(2)).++ Ties choose the first canonical option, so reordering tied slots does not+ itself count as disagreement. Confidence is the maximum probability;+ confidence drift is signed (variant minus baseline).+ """+ p = canonicalize(baseline, range(len(baseline)))+ q = canonicalize(variant, range(len(baseline)))+ middle = [(a + b) / 2 for a, b in zip(p, q)]+ js = sum(0.5 * x * math.log(x / m)+ for distribution in (p, q)+ for x, m in zip(distribution, middle) if x > 0)+ drift = [abs(a - b) for a, b in zip(p, q)]+ return {+ "semantic_agreement": max(range(len(p)), key=p.__getitem__) ==+ max(range(len(q)), key=q.__getitem__),+ "mean_probability_drift": sum(drift) / len(drift),+ "max_probability_drift": max(drift),+ "js_divergence": max(0.0, js),+ "confidence_drift": max(q) - max(p),+ }+++def summarise_pairs(pairs):+ """Pair-weighted summary; absent observations are not perfect agreement."""+ if not pairs:+ return {"n": 0}+ return {+ "n": len(pairs),+ "semantic_agreement_rate": sum(p["semantic_agreement"] for p in pairs) / len(pairs),+ "mean_probability_drift": sum(p["mean_probability_drift"] for p in pairs) / len(pairs),+ "max_probability_drift": max(p["max_probability_drift"] for p in pairs),+ "mean_js_divergence": sum(p["js_divergence"] for p in pairs) / len(pairs),+ "mean_confidence_drift": sum(p["confidence_drift"] for p in pairs) / len(pairs),+ "mean_absolute_confidence_drift": sum(abs(p["confidence_drift"]) for p in pairs) / len(pairs),+ "worst_confidence_increase_on_disagreement": max(+ [0.0] + [p["confidence_drift"] for p in pairs if not p["semantic_agreement"]]),+ }+++def evaluate(cases, score: Callable, gold_indices=None, seed=harness.SEED,+ batch_size=16):+ """Score cases via ``score(batch) -> probability vectors in presented order``.++ Optional gold indices refer to original option order; entries may be None.+ Batches bound inference memory. RNG state and variants do not depend on batch+ size. Returned records retain inputs, mappings and full precision vectors.+ """+ cases = list(cases)+ rng = random.Random(seed)+ generated = [make_variants(case, rng) for case in cases]+ return _evaluate_generated(cases, generated, score, gold_indices, batch_size)+++def _evaluate_generated(cases, generated_by_case, score, gold_indices, batch_size):+ if batch_size < 1:+ raise ValueError("batch_size must be positive")+ cases = list(cases)+ golds = [None] * len(cases) if gold_indices is None else list(gold_indices)+ if len(golds) != len(cases):+ raise ValueError("one gold index is required per case")+ variants, records = [], []+ for index, (case, gold) in enumerate(zip(cases, golds)):+ generated = generated_by_case[index]+ keys = list(next(iter(case[1].values()))["criteria"])+ if gold is not None and (type(gold) is not int or not 0 <= gold < len(keys)):+ raise ValueError("gold index outside canonical options")+ records.append({"index": index, "state": deepcopy(case[0]),+ "questions": deepcopy(case[1]), "options": keys,+ "gold_index": gold, "variants": []})+ variants.extend((index, v) for v in generated)+ for start in range(0, len(variants), batch_size):+ batch = variants[start:start + batch_size]+ vectors = list(score([v["case"] for _, v in batch]))+ if len(vectors) != len(batch):+ raise ValueError("scorer returned the wrong number of probability vectors")+ for (index, v), raw in zip(batch, vectors):+ probs = canonicalize(raw, v["canonical_indices"])+ pred = max(range(len(probs)), key=probs.__getitem__)+ records[index]["variants"].append({+ "kind": v["kind"],+ "presented_options": list(next(iter(v["case"][1].values()))["criteria"]),+ "canonical_indices": v["canonical_indices"],+ "canonical_to_transformed": v["canonical_to_transformed"],+ "transformed_to_canonical": v["transformed_to_canonical"],+ "probabilities": probs, "pred_index": pred,+ "pred_label": records[index]["options"][pred],+ "confidence": max(probs),+ "correct": None if golds[index] is None else pred == golds[index],+ })+ return {"report": _report(records), "cases": records}+++def _report(records):+ groups = {"option_order": []}+ for record in records:+ baseline = record["variants"][0]["probabilities"]+ for variant in record["variants"][1:]:+ variant["comparison"] = distribution_metrics(baseline, variant["probabilities"])+ groups[variant["kind"]].append(variant["comparison"])+ report = {kind: summarise_pairs(pairs) for kind, pairs in groups.items()}+ report["overall"] = summarise_pairs([p for pairs in groups.values() for p in pairs])+ report["quality"] = {}+ for kind in ("baseline", "option_order"):+ labelled = [v for r in records for v in r["variants"]+ if v["kind"] == kind and v["correct"] is not None]+ quality = {"n_labelled": len(labelled)}+ if labelled:+ quality.update(accuracy=sum(v["correct"] for v in labelled) / len(labelled),+ ece=harness.ece([v["confidence"] for v in labelled],+ [v["correct"] for v in labelled]))+ report["quality"][kind] = quality+ return report+++def model_scorer(agent, unclamped=False):+ """Reuse the harness's raw logits and per-option-count temperatures."""+ from laya.common import QTYPES++ def score(cases):+ return [harness.softmax_t(z, harness.temperature_for(+ agent, QTYPES["choice"], len(z), unclamped))+ for z in harness.score_cases(agent, cases)]+ return score+++@dataclass+class MetamorphicResults:+ baseline: dict+ variants: list[dict]+++def evaluate_variants(agent, case: MetamorphicCase, variants, *, score=None,+ unclamped=False, batch_size=16):+ """Evaluate explicit transforms; inject ``score`` for offline stub models.++ With a real agent, call ``agent.model.eval()`` first, as with the harness.+ All output vectors are restored to the original case's canonical ordering.+ """+ generated = [_baseline(case).as_record()]+ for variant in variants:+ if variant.kind not in ("option_order",):+ raise ValueError("unsupported transformation kind")+ if list(variant.canonical_to_transformed) != case.option_keys:+ raise ValueError("variant mapping does not match the canonical case")+ generated.append(variant.as_record())+ result = _evaluate_generated([case.as_pair()], [generated],+ score if score is not None else model_scorer(agent, unclamped),+ [case.gold_index], batch_size)+ predictions = result["cases"][0]["variants"]+ return MetamorphicResults(predictions[0], predictions[1:])+++def compare_predictions(*, baseline, variants):+ """Report semantic agreement separately from continuous stability metrics.++ No drift threshold or automatic bug classification is applied.+ """+ return _report([{"variants": [deepcopy(baseline), *deepcopy(variants)]}])+++def main(argv: Sequence[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--model", default="convaiinnovations/laya")+ parser.add_argument("--subfolder", default=None)+ parser.add_argument("--device", default=None)+ parser.add_argument("--langs", default="en", help="comma-separated MASSIVE configs, or all")+ parser.add_argument("--per-lang", type=int, default=harness.PER_LANG)+ parser.add_argument("--n-opts", type=int, default=harness.N_OPTS)+ parser.add_argument("--seed", type=int, default=harness.SEED)+ parser.add_argument("--batch-size", type=int, default=16)+ parser.add_argument("--unclamped", action="store_true")+ parser.add_argument("--out", required=True, help="JSON report path")+ args = parser.parse_args(argv)+ if args.per_lang < 1 or args.n_opts < 2 or args.batch_size < 1:+ parser.error("per-lang/batch-size must be positive and n-opts must be at least 2")+ langs = (harness.available_languages() if args.langs.strip().lower() == "all"+ else list(dict.fromkeys(x.strip() for x in args.langs.split(",") if x.strip())))+ if not langs:+ parser.error("no languages selected")+ import laya++ agent = laya.load(args.model, device=args.device, subfolder=args.subfolder)+ agent.model.eval()+ payload: dict[str, Any] = {+ "config": {**vars(args), "dataset": harness.DATASET, "split": "test",+ "device": str(agent.device), "laya_version": laya.__version__,+ "max_len": agent.cfg.get("max_len"),+ "head_max_len": agent.cfg.get("head_max_len"),+ "temperature": list(agent.temperature_raw if args.unclamped else agent.temperature),+ "temperature_by_options": dict(agent.temperature_by_options_raw if args.unclamped+ else agent.temperature_by_options),+ "permutations_per_case": 1, "js_log_base": "e"},+ "report": {}, "cases": [],+ }+ failed = False+ for lang in langs:+ try:+ rows = harness.load_language(lang)+ cases, gold, _ = harness.build_suite(+ rows, sorted({r["label_text"] for r in rows}), args.per_lang, args.n_opts, args.seed)+ if not cases:+ raise ValueError("dataset returned no cases")+ result = evaluate(cases, model_scorer(agent, args.unclamped), gold,+ args.seed, args.batch_size)+ payload["report"][lang] = result["report"]+ payload["cases"].extend({"lang": lang, **r} for r in result["cases"])+ print(lang, json.dumps(result["report"]), flush=True)+ except Exception as exc:+ failed = True+ payload["report"][lang] = {"error": str(exc)}+ print("%s FAILED: %s" % (lang, exc), flush=True)+ with open(args.out, "w", encoding="utf-8") as stream:+ json.dump(payload, stream, ensure_ascii=False, indent=2, allow_nan=False)+ stream.write("\n")+ return 1 if failed else 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/research/eval/test_metamorphic.py b/research/eval/test_metamorphic.pynew file mode 100644index 00000000..3ac16679--- /dev/null+++ b/research/eval/test_metamorphic.py@@ -0,0 +1,201 @@+"""Offline tests: python -m unittest research.eval.test_metamorphic -v."""+from contextlib import redirect_stdout+from copy import deepcopy+from io import StringIO+import json+import math+from pathlib import Path+import random+import tempfile+import unittest+from unittest.mock import MagicMock, patch++from research.eval import metamorphic as m++CASE = ({"utterance": "I was charged twice"}, {"intent": {+ "type": "choice", "instructions": "Choose the request category.",+ "criteria": {"billing": "payment issues", "technical": "software help", "sales": "new purchases"},+}})+++def semantic_scorer(cases):+ weights = {"payment issues": 0.7, "software help": 0.2, "new purchases": 0.1}+ return [[weights[v] for v in q["intent"]["criteria"].values()] for _, q in cases]+++class TransformTests(unittest.TestCase):+ def test_mapping_and_unchanged_inputs(self):+ original = deepcopy(CASE)+ baseline, order = m.make_variants(CASE, random.Random(13))+ self.assertEqual(CASE, original)+ self.assertEqual(baseline["case"], CASE)+ self.assertNotEqual(order["canonical_indices"], [0, 1, 2])+ for variant in (baseline, order):+ criteria = variant["case"][1]["intent"]["criteria"]+ descriptions = list(CASE[1]["intent"]["criteria"].values())+ self.assertEqual(list(criteria.values()),+ [descriptions[i] for i in variant["canonical_indices"]])+ order["case"][0]["utterance"] = "changed"+ self.assertEqual(CASE, original)++ def test_identity_shuffle_fallback(self):+ with patch("random.Random.shuffle", return_value=None):+ variant = m.permute_options(m.MetamorphicCase(*CASE), seed=42)+ self.assertEqual(variant.as_record()["canonical_indices"], [1, 2, 0])++ def test_explicit_api_mapping_and_gold(self):+ case = m.MetamorphicCase(*CASE, gold_index=0)+ permuted = m.permute_options(case, seed=42)+ self.assertEqual(permuted, m.permute_options(case, seed=42))+ self.assertEqual(permuted.canonical_to_transformed,+ {"billing": "billing", "technical": "technical", "sales": "sales"})+ self.assertEqual(permuted.transformed_to_canonical["billing"], "billing")+ self.assertEqual(permuted.case.option_keys[permuted.case.gold_index], "billing")+ result = m.evaluate_variants(None, case, [permuted], score=semantic_scorer)+ report = m.compare_predictions(baseline=result.baseline, variants=result.variants)+ self.assertEqual(report["overall"]["semantic_agreement_rate"], 1)+ self.assertEqual(report["overall"]["max_probability_drift"], 0)++ def test_many_options_and_nested_descriptions(self):+ case = deepcopy(CASE)+ case[1]["intent"]["criteria"] = {str(i): {"description": [str(i)]} for i in range(28)}+ order = m.make_variants(case, random.Random(0))[1]+ criteria = order["case"][1]["intent"]["criteria"]+ self.assertEqual(len(criteria), 28)++ def test_none_or_blank_descriptions_accepted(self):+ for criteria in ({"a": None, "b": "two"}, {"a": " ", "b": ""}, {"a": None, "b": None}):+ case = deepcopy(CASE)+ case[1]["intent"]["criteria"] = criteria+ variants = m.make_variants(case, random.Random(0))+ self.assertEqual(len(variants), 2)++ def test_invalid_cases(self):+ for criteria in ({}, {"a": "one"}, {1: "one", "b": "two"}, {" ": "one", "b": "two"}, {"": "one", "b": "two"}):+ case = deepcopy(CASE)+ case[1]["intent"]["criteria"] = criteria+ with self.subTest(criteria=criteria), self.assertRaises(ValueError):+ m.make_variants(case, random.Random(0))+ for questions in ({}, {"a": CASE[1]["intent"], "b": CASE[1]["intent"]},+ {"a": {"type": "noul", "criteria": {"false": "no", "true": "yes"}}}):+ with self.assertRaises(ValueError):+ m.make_variants(({}, questions), random.Random(0))+++class MetricTests(unittest.TestCase):+ def test_inverse_mapping(self):+ self.assertEqual(m.canonicalize([0.1, 0.7, 0.2], [2, 0, 1]), [0.7, 0.2, 0.1])++ def test_invalid_distributions(self):+ for vector, mapping in (([], []), ([0.2, 0.8], [0, 0]),+ ([0.2, 0.8], [0]), ([0.2, 0.8], [0, 2]),+ ([float("nan"), 0.5], [0, 1]),+ ([float("inf"), 0], [0, 1]),+ ([-0.1, 1.1], [0, 1]), ([0.2, 0.2], [0, 1])):+ with self.subTest(vector=vector), self.assertRaises(ValueError):+ m.canonicalize(vector, mapping)++ def test_known_divergence_and_drift(self):+ result = m.distribution_metrics([1, 0], [0, 1])+ self.assertFalse(result["semantic_agreement"])+ self.assertAlmostEqual(result["js_divergence"], math.log(2))+ self.assertEqual(result["mean_probability_drift"], 1)+ self.assertEqual(result["confidence_drift"], 0)+ identical = m.distribution_metrics([0.5, 0.5, 0], [0.5, 0.5, 0])+ self.assertTrue(identical["semantic_agreement"])+ self.assertEqual(identical["js_divergence"], 0)++ def test_agreement_is_distinct_from_stability(self):+ same_winner = m.distribution_metrics([0.91, 0.06, 0.03], [0.88, 0.08, 0.04])+ self.assertTrue(same_winner["semantic_agreement"])+ self.assertGreater(same_winner["mean_probability_drift"], 0)+ self.assertGreater(same_winner["js_divergence"], 0)+ changed_winner = m.distribution_metrics([0.91, 0.06, 0.03], [0.08, 0.86, 0.06])+ self.assertFalse(changed_winner["semantic_agreement"])+ self.assertLess(changed_winner["confidence_drift"], 0)++ def test_summary_and_confidence_increase(self):+ pairs = [m.distribution_metrics([0.6, 0.4], [0.1, 0.9]), m.distribution_metrics([0.6, 0.4], [0.8, 0.2])]+ summary = m.summarise_pairs(pairs)+ self.assertEqual(summary["semantic_agreement_rate"], 0.5)+ self.assertAlmostEqual(summary["mean_probability_drift"], 0.35)+ self.assertAlmostEqual(summary["max_probability_drift"], 0.5)+ self.assertAlmostEqual(summary["mean_confidence_drift"], 0.25)+ self.assertAlmostEqual(summary["worst_confidence_increase_on_disagreement"], 0.3)+ self.assertEqual(m.summarise_pairs([]), {"n": 0})+ decreasing = m.summarise_pairs([m.distribution_metrics([0.9, 0.1], [0.4, 0.6])])+ self.assertEqual(decreasing["worst_confidence_increase_on_disagreement"], 0)+++class EvaluationTests(unittest.TestCase):+ def test_invariance_batching_seed_gold_and_json(self):+ outputs = []+ for batch_size in (1, 2, 16):+ sizes = []+ def score(cases):+ sizes.append(len(cases))+ return semantic_scorer(cases)+ result = m.evaluate([CASE, CASE], score, [0, None], batch_size=batch_size)+ self.assertLessEqual(max(sizes), batch_size)+ self.assertEqual(result["report"]["overall"]["semantic_agreement_rate"], 1)+ self.assertEqual(result["report"]["overall"]["max_probability_drift"], 0)+ self.assertEqual(result["report"]["quality"]["baseline"]["accuracy"], 1)+ self.assertEqual(result["report"]["quality"]["baseline"]["n_labelled"], 1)+ self.assertAlmostEqual(result["report"]["quality"]["baseline"]["ece"], 0.3)+ outputs.append(result)+ self.assertEqual(outputs[0], outputs[1])+ self.assertEqual(outputs[0], outputs[2])+ self.assertEqual(json.loads(json.dumps(outputs[0], allow_nan=False)), outputs[0])++ def test_position_bias_detected(self):+ report = m.evaluate([CASE], lambda cases: [[1, 0, 0] for _ in cases])["report"]+ self.assertEqual(report["option_order"]["semantic_agreement_rate"], 0)+ self.assertEqual(report["quality"]["baseline"], {"n_labelled": 0})++ def test_ties_use_canonical_order(self):+ result = m.evaluate([CASE], lambda cases: [[1/3] * 3 for _ in cases])+ self.assertEqual(result["report"]["overall"]["semantic_agreement_rate"], 1)++ def test_empty_bad_scorers_and_gold(self):+ self.assertEqual(m.evaluate([], lambda _: self.fail("should not score"))["report"]["overall"], {"n": 0})+ for score in (lambda cases: [], lambda cases: [[1, 0] for _ in cases]):+ with self.assertRaises(ValueError):+ m.evaluate([CASE], score)+ for gold in ([], [3], [-1], [True], [0.5]):+ with self.assertRaises(ValueError):+ m.evaluate([CASE], semantic_scorer, gold)+ with self.assertRaises(ValueError):+ m.evaluate([CASE], semantic_scorer, batch_size=0)++ def test_model_adapter_temperature(self):+ class Agent:+ temperature = [1.0] * 3+ temperature_raw = [0.5] * 3+ temperature_by_options = {}+ temperature_by_options_raw = {}+ with patch.object(m.harness, "score_cases", return_value=[[0, math.log(3)]]):+ self.assertAlmostEqual(m.model_scorer(Agent())([CASE])[0][1], 0.75)+ self.assertAlmostEqual(m.model_scorer(Agent(), True)([CASE])[0][1], 0.9)++ def test_cli_partial_failure_writes_report_and_fails(self):+ agent = MagicMock()+ agent.device = "cpu"+ agent.cfg = {"max_len": 512, "head_max_len": 192}+ agent.temperature = [1.0] * 3+ agent.temperature_by_options = {}+ rows = [{"text": "x", "label_text": "a"}, {"text": "y", "label_text": "b"}]+ with tempfile.TemporaryDirectory() as folder:+ out = Path(folder) / "report.json"+ with redirect_stdout(StringIO()), patch("laya.load", return_value=agent), \+ patch.object(m.harness, "load_language", side_effect=[rows, RuntimeError("offline")]), \+ patch.object(m, "model_scorer", return_value=lambda cases: [[0.6, 0.4] for _ in cases]):+ code = m.main(["--langs", "en,de", "--n-opts", "2", "--out", str(out)])+ payload = json.loads(out.read_text(encoding="utf-8"))+ self.assertEqual(code, 1)+ self.assertEqual(payload["report"]["de"], {"error": "offline"})+ self.assertEqual(len(payload["cases"]), 2)+ self.assertEqual(payload["config"]["temperature"], [1.0] * 3)+++if __name__ == "__main__":+ unittest.main()