Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

research(eval): add metamorphic option-permutation robustness checks (#269)

Reskyumeb731f2eb9bbc
diff --git a/research/eval/README.md b/research/eval/README.md
index adf2e7f6..b5b3e407 100644
--- a/research/eval/README.md
+++ b/research/eval/README.md
@@ -318,3 +318,72 @@ checks, and an order-invariant model scores exactly 1/3.
reduced-precision autocast and `score_cases` does not. The parity check reports that
difference instead of hiding it.
* New checks are one function each, registered in `CHECKS`.
+
+
+## Metamorphic option-order robustness (experimental)
+
+`metamorphic.py` adds the initial scope of
+[#244](https://github.com/NandhaKishorM/laya/issues/244): **choice option order robustness**, without changing model/runtime behavior. Label renaming, neutral labels, paraphrases, structured-state permutations, `score` and `noul` perturbations are intentionally deferred. Run from the repository root after installing Laya and `datasets`:
+
+```bash
+python -m research.eval.metamorphic --model convaiinnovations/laya \
+ --langs en --per-lang 100 --n-opts 20 --batch-size 16 --out robustness.json
+python -m research.eval.metamorphic --model convaiinnovations/laya \
+ --subfolder multilingual --langs en --out multilingual-robustness.json
+python -m unittest research.eval.test_metamorphic -v
+```
+
+Each MASSIVE case uses the existing harness's sampler and produces two inputs:
+
+1. The unchanged baseline.
+2. One seeded shuffle of option order; if the shuffle is the identity, a one-slot
+ rotation is used. This is a bounded diagnostic, not exhaustive permutation testing
+ or a uniform draw over all nonidentity permutations.
+
+Instructions and state are otherwise unchanged. Option key/value pairs are moved together during permutation. Every result is mapped back to the original semantic option order **before** predictions and metrics are computed. Exact ties choose the first canonical option. The RNG starts fresh per language; `--seed` controls both sampling and transformations. `--batch-size` bounds the number of forward-pass inputs and does not alter the generated variants. Model inference may still have small floating-point differences across devices and batch sizes.
+
+The JSON contains `config`, per-language `report`, and full `cases`. Each case saves its original input, canonical keys and optional gold index; each variant saves its presented keys, explicit `canonical_to_transformed` and `transformed_to_canonical` label mappings, slot-to-canonical indices, complete **canonical-order** probability vector, prediction, confidence, correctness (or `null`), and comparison to baseline. Probabilities are not rounded. The config records model/subfolder, temperature mode and values, truncation settings, dataset, seed and batch size. For reproducible checkpoint comparisons, use a pinned local snapshot and retain the environment versions alongside the report. `--unclamped` has the same meaning as in `laya_eval`. If any language fails, its error is saved and the command exits nonzero while retaining successful languages.
+
+Metrics are grouped under `option_order` and `overall`:
+
+| Metric | Definition |
+|---|---|
+| `semantic_agreement_rate` | Fraction of baseline/variant pairs with the same canonical argmax |
+| `mean_probability_drift` | Mean absolute probability change across options, then pairs |
+| `max_probability_drift` | Largest absolute change of any option across all pairs |
+| `mean_js_divergence` | Mean Jensen-Shannon divergence using natural logs, in `[0, ln(2)]` |
+| `mean_confidence_drift` | Mean signed change of maximum probability, variant minus baseline |
+| `mean_absolute_confidence_drift` | Mean magnitude of that confidence change |
+| `worst_confidence_increase_on_disagreement` | Largest positive confidence change among changed decisions, or zero if none |
+
+`overall` is pair-weighted, not a fraction of cases where *all* variants agree. `quality` separately reports accuracy and the existing harness's 15-bin ECE for baseline and each transformation on labelled cases only. Empty groups contain `n: 0`; unlabelled quality groups contain `n_labelled: 0` without inventing an accuracy or ECE. Robustness agreement is not a correctness measure: consistently wrong predictions can be perfectly invariant.
+
+For another corpus, the Python API accepts `(state, questions)` cases in the same shape as the harness, and a callback returning probability vectors in presented option order:
+
+```python
+from research.eval.metamorphic import evaluate, model_scorer
+agent.model.eval()
+result = evaluate(cases, model_scorer(agent), gold_indices=None, seed=13)
+```
+
+The first version intentionally accepts only **one choice question per case**, with at least two options. Label renaming and other metamorphic transforms are intentionally deferred as proposed in the issue.
+
+For an explicit single-case experiment, the same implementation exposes:
+
+```python
+from research.eval.metamorphic import (
+ MetamorphicCase, permute_options,
+ evaluate_variants, compare_predictions,
+)
+
+case = MetamorphicCase(state, questions, gold_index=None)
+variants = [permute_options(case, seed=42)]
+agent.model.eval()
+results = evaluate_variants(agent, case, variants)
+report = compare_predictions(baseline=results.baseline, variants=results.variants)
+```
+
+For offline tests, pass `agent=None, score=fake_scorer` to `evaluate_variants`. The scorer takes a batch of harness `(state, questions)` inputs and returns one probability vector per input. The public transformations return independent copies and explicit mappings in both directions, including identity label mappings for order-only transformations.
+
+**Semantic agreement and distribution stability are different properties.**
+A shift from `[0.91, 0.06, 0.03]` to `[0.88, 0.08, 0.04]` preserves the decision while showing nonzero drift. Switching the winner is reported as disagreement, regardless of whether confidence rises or falls. No metric here automatically classifies either observation as a bug; acceptable variation depends on the use case, and the report deliberately defines no universal pass/fail threshold.
diff --git a/research/eval/metamorphic.py b/research/eval/metamorphic.py
new file mode 100644
index 00000000..097ee1c5
--- /dev/null
+++ b/research/eval/metamorphic.py
@@ -0,0 +1,344 @@
+"""Small, reproducible choice-invariance diagnostics (issue #244).
+
+Run from the checkout: python -m research.eval.metamorphic --help.
+Pure transformations/metrics can also be used without a model or gold labels.
+"""
+from __future__ import annotations
+
+import argparse
+from copy import deepcopy
+from dataclasses import dataclass
+import json
+import math
+import random
+from typing import Any, Callable, Sequence
+
+from . import laya_eval as harness
+
+
+@dataclass
+class MetamorphicCase:
+ """One canonical choice decision in the existing harness input format."""
+
+ state: Any
+ questions: dict
+ gold_index: int | None = None
+
+ def __post_init__(self):
+ if len(self.questions) != 1:
+ raise ValueError("expected exactly one choice question per case")
+ question = next(iter(self.questions.values()))
+ criteria = question.get("criteria")
+ if question.get("type") != "choice" or not isinstance(criteria, dict):
+ raise ValueError("expected choice criteria as a dictionary")
+ if len(criteria) < 2:
+ raise ValueError("at least two options are required")
+ if any(not isinstance(k, str) or not k.strip() for k in criteria):
+ raise ValueError("option keys must be nonempty strings")
+ if self.gold_index is not None and (
+ type(self.gold_index) is not int or not 0 <= self.gold_index < len(criteria)):
+ raise ValueError("gold index outside canonical options")
+
+ @property
+ def option_keys(self):
+ return list(next(iter(self.questions.values()))["criteria"])
+
+ def as_pair(self):
+ return self.state, self.questions
+
+
+@dataclass
+class MetamorphicVariant:
+ """Transformed input plus an explicit bidirectional semantic label mapping."""
+
+ kind: str
+ case: MetamorphicCase
+ canonical_to_transformed: dict[str, str]
+
+ @property
+ def transformed_to_canonical(self):
+ return {v: k for k, v in self.canonical_to_transformed.items()}
+
+ def as_record(self):
+ mapping = self.canonical_to_transformed
+ inverse = self.transformed_to_canonical
+ keys = list(mapping)
+ if len(inverse) != len(mapping) or set(inverse) != set(self.case.option_keys):
+ raise ValueError("label mapping must be a bijection over transformed options")
+ return {"kind": self.kind, "case": self.case.as_pair(),
+ "canonical_to_transformed": dict(mapping),
+ "transformed_to_canonical": inverse,
+ "canonical_indices": [keys.index(inverse[k]) for k in self.case.option_keys]}
+
+
+def _transform(case, kind, order, labels):
+ questions = deepcopy(case.questions)
+ question = next(iter(questions.values()))
+ criteria = question["criteria"]
+ keys = case.option_keys
+ mapping = dict(zip(keys, labels))
+ question["criteria"] = {mapping[keys[i]]: deepcopy(criteria[keys[i]]) for i in order}
+ gold = None if case.gold_index is None else order.index(case.gold_index)
+ return MetamorphicVariant(kind, MetamorphicCase(deepcopy(case.state), questions, gold), mapping)
+
+
+def permute_options(case: MetamorphicCase, seed: int = harness.SEED):
+ """One deterministic nonidentity shuffle; identity falls back to rotation."""
+ order = list(range(len(case.option_keys)))
+ random.Random(seed).shuffle(order)
+ if order == list(range(len(order))):
+ order = order[1:] + order[:1]
+ return _transform(case, "option_order", order, case.option_keys)
+
+
+def _baseline(case):
+ return _transform(case, "baseline", list(range(len(case.option_keys))), case.option_keys)
+
+
+def make_variants(case, rng: random.Random):
+ """Adapt the harness tuple format to baseline plus option permutation.
+
+ Label renaming, paraphrases, and other metamorphic transforms are intentionally deferred.
+ """
+ canonical = MetamorphicCase(*case)
+ return [v.as_record() for v in (
+ _baseline(canonical), permute_options(canonical, rng.getrandbits(64)))]
+
+
+def canonicalize(probabilities, canonical_indices):
+ """Validate a probability vector and restore canonical semantic ordering."""
+ values = [float(p) for p in probabilities]
+ n = len(values)
+ if not n or any(type(i) is not int for i in canonical_indices) or sorted(canonical_indices) != list(range(n)):
+ raise ValueError("probabilities and canonical mapping must be a bijection")
+ if any(not math.isfinite(p) or p < 0 or p > 1 for p in values):
+ raise ValueError("probabilities must be finite values in [0, 1]")
+ if not math.isclose(sum(values), 1.0, abs_tol=1e-6, rel_tol=0):
+ raise ValueError("probabilities must sum to one")
+ restored = [0.0] * n
+ for p, index in zip(values, canonical_indices):
+ restored[index] = p
+ return restored
+
+
+def distribution_metrics(baseline, variant):
+ """Drift in canonical space; JS uses natural logs (range 0..ln(2)).
+
+ Ties choose the first canonical option, so reordering tied slots does not
+ itself count as disagreement. Confidence is the maximum probability;
+ confidence drift is signed (variant minus baseline).
+ """
+ p = canonicalize(baseline, range(len(baseline)))
+ q = canonicalize(variant, range(len(baseline)))
+ middle = [(a + b) / 2 for a, b in zip(p, q)]
+ js = sum(0.5 * x * math.log(x / m)
+ for distribution in (p, q)
+ for x, m in zip(distribution, middle) if x > 0)
+ drift = [abs(a - b) for a, b in zip(p, q)]
+ return {
+ "semantic_agreement": max(range(len(p)), key=p.__getitem__) ==
+ max(range(len(q)), key=q.__getitem__),
+ "mean_probability_drift": sum(drift) / len(drift),
+ "max_probability_drift": max(drift),
+ "js_divergence": max(0.0, js),
+ "confidence_drift": max(q) - max(p),
+ }
+
+
+def summarise_pairs(pairs):
+ """Pair-weighted summary; absent observations are not perfect agreement."""
+ if not pairs:
+ return {"n": 0}
+ return {
+ "n": len(pairs),
+ "semantic_agreement_rate": sum(p["semantic_agreement"] for p in pairs) / len(pairs),
+ "mean_probability_drift": sum(p["mean_probability_drift"] for p in pairs) / len(pairs),
+ "max_probability_drift": max(p["max_probability_drift"] for p in pairs),
+ "mean_js_divergence": sum(p["js_divergence"] for p in pairs) / len(pairs),
+ "mean_confidence_drift": sum(p["confidence_drift"] for p in pairs) / len(pairs),
+ "mean_absolute_confidence_drift": sum(abs(p["confidence_drift"]) for p in pairs) / len(pairs),
+ "worst_confidence_increase_on_disagreement": max(
+ [0.0] + [p["confidence_drift"] for p in pairs if not p["semantic_agreement"]]),
+ }
+
+
+def evaluate(cases, score: Callable, gold_indices=None, seed=harness.SEED,
+ batch_size=16):
+ """Score cases via ``score(batch) -> probability vectors in presented order``.
+
+ Optional gold indices refer to original option order; entries may be None.
+ Batches bound inference memory. RNG state and variants do not depend on batch
+ size. Returned records retain inputs, mappings and full precision vectors.
+ """
+ cases = list(cases)
+ rng = random.Random(seed)
+ generated = [make_variants(case, rng) for case in cases]
+ return _evaluate_generated(cases, generated, score, gold_indices, batch_size)
+
+
+def _evaluate_generated(cases, generated_by_case, score, gold_indices, batch_size):
+ if batch_size < 1:
+ raise ValueError("batch_size must be positive")
+ cases = list(cases)
+ golds = [None] * len(cases) if gold_indices is None else list(gold_indices)
+ if len(golds) != len(cases):
+ raise ValueError("one gold index is required per case")
+ variants, records = [], []
+ for index, (case, gold) in enumerate(zip(cases, golds)):
+ generated = generated_by_case[index]
+ keys = list(next(iter(case[1].values()))["criteria"])
+ if gold is not None and (type(gold) is not int or not 0 <= gold < len(keys)):
+ raise ValueError("gold index outside canonical options")
+ records.append({"index": index, "state": deepcopy(case[0]),
+ "questions": deepcopy(case[1]), "options": keys,
+ "gold_index": gold, "variants": []})
+ variants.extend((index, v) for v in generated)
+ for start in range(0, len(variants), batch_size):
+ batch = variants[start:start + batch_size]
+ vectors = list(score([v["case"] for _, v in batch]))
+ if len(vectors) != len(batch):
+ raise ValueError("scorer returned the wrong number of probability vectors")
+ for (index, v), raw in zip(batch, vectors):
+ probs = canonicalize(raw, v["canonical_indices"])
+ pred = max(range(len(probs)), key=probs.__getitem__)
+ records[index]["variants"].append({
+ "kind": v["kind"],
+ "presented_options": list(next(iter(v["case"][1].values()))["criteria"]),
+ "canonical_indices": v["canonical_indices"],
+ "canonical_to_transformed": v["canonical_to_transformed"],
+ "transformed_to_canonical": v["transformed_to_canonical"],
+ "probabilities": probs, "pred_index": pred,
+ "pred_label": records[index]["options"][pred],
+ "confidence": max(probs),
+ "correct": None if golds[index] is None else pred == golds[index],
+ })
+ return {"report": _report(records), "cases": records}
+
+
+def _report(records):
+ groups = {"option_order": []}
+ for record in records:
+ baseline = record["variants"][0]["probabilities"]
+ for variant in record["variants"][1:]:
+ variant["comparison"] = distribution_metrics(baseline, variant["probabilities"])
+ groups[variant["kind"]].append(variant["comparison"])
+ report = {kind: summarise_pairs(pairs) for kind, pairs in groups.items()}
+ report["overall"] = summarise_pairs([p for pairs in groups.values() for p in pairs])
+ report["quality"] = {}
+ for kind in ("baseline", "option_order"):
+ labelled = [v for r in records for v in r["variants"]
+ if v["kind"] == kind and v["correct"] is not None]
+ quality = {"n_labelled": len(labelled)}
+ if labelled:
+ quality.update(accuracy=sum(v["correct"] for v in labelled) / len(labelled),
+ ece=harness.ece([v["confidence"] for v in labelled],
+ [v["correct"] for v in labelled]))
+ report["quality"][kind] = quality
+ return report
+
+
+def model_scorer(agent, unclamped=False):
+ """Reuse the harness's raw logits and per-option-count temperatures."""
+ from laya.common import QTYPES
+
+ def score(cases):
+ return [harness.softmax_t(z, harness.temperature_for(
+ agent, QTYPES["choice"], len(z), unclamped))
+ for z in harness.score_cases(agent, cases)]
+ return score
+
+
+@dataclass
+class MetamorphicResults:
+ baseline: dict
+ variants: list[dict]
+
+
+def evaluate_variants(agent, case: MetamorphicCase, variants, *, score=None,
+ unclamped=False, batch_size=16):
+ """Evaluate explicit transforms; inject ``score`` for offline stub models.
+
+ With a real agent, call ``agent.model.eval()`` first, as with the harness.
+ All output vectors are restored to the original case's canonical ordering.
+ """
+ generated = [_baseline(case).as_record()]
+ for variant in variants:
+ if variant.kind not in ("option_order",):
+ raise ValueError("unsupported transformation kind")
+ if list(variant.canonical_to_transformed) != case.option_keys:
+ raise ValueError("variant mapping does not match the canonical case")
+ generated.append(variant.as_record())
+ result = _evaluate_generated([case.as_pair()], [generated],
+ score if score is not None else model_scorer(agent, unclamped),
+ [case.gold_index], batch_size)
+ predictions = result["cases"][0]["variants"]
+ return MetamorphicResults(predictions[0], predictions[1:])
+
+
+def compare_predictions(*, baseline, variants):
+ """Report semantic agreement separately from continuous stability metrics.
+
+ No drift threshold or automatic bug classification is applied.
+ """
+ return _report([{"variants": [deepcopy(baseline), *deepcopy(variants)]}])
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--model", default="convaiinnovations/laya")
+ parser.add_argument("--subfolder", default=None)
+ parser.add_argument("--device", default=None)
+ parser.add_argument("--langs", default="en", help="comma-separated MASSIVE configs, or all")
+ parser.add_argument("--per-lang", type=int, default=harness.PER_LANG)
+ parser.add_argument("--n-opts", type=int, default=harness.N_OPTS)
+ parser.add_argument("--seed", type=int, default=harness.SEED)
+ parser.add_argument("--batch-size", type=int, default=16)
+ parser.add_argument("--unclamped", action="store_true")
+ parser.add_argument("--out", required=True, help="JSON report path")
+ args = parser.parse_args(argv)
+ if args.per_lang < 1 or args.n_opts < 2 or args.batch_size < 1:
+ parser.error("per-lang/batch-size must be positive and n-opts must be at least 2")
+ langs = (harness.available_languages() if args.langs.strip().lower() == "all"
+ else list(dict.fromkeys(x.strip() for x in args.langs.split(",") if x.strip())))
+ if not langs:
+ parser.error("no languages selected")
+ import laya
+
+ agent = laya.load(args.model, device=args.device, subfolder=args.subfolder)
+ agent.model.eval()
+ payload: dict[str, Any] = {
+ "config": {**vars(args), "dataset": harness.DATASET, "split": "test",
+ "device": str(agent.device), "laya_version": laya.__version__,
+ "max_len": agent.cfg.get("max_len"),
+ "head_max_len": agent.cfg.get("head_max_len"),
+ "temperature": list(agent.temperature_raw if args.unclamped else agent.temperature),
+ "temperature_by_options": dict(agent.temperature_by_options_raw if args.unclamped
+ else agent.temperature_by_options),
+ "permutations_per_case": 1, "js_log_base": "e"},
+ "report": {}, "cases": [],
+ }
+ failed = False
+ for lang in langs:
+ try:
+ rows = harness.load_language(lang)
+ cases, gold, _ = harness.build_suite(
+ rows, sorted({r["label_text"] for r in rows}), args.per_lang, args.n_opts, args.seed)
+ if not cases:
+ raise ValueError("dataset returned no cases")
+ result = evaluate(cases, model_scorer(agent, args.unclamped), gold,
+ args.seed, args.batch_size)
+ payload["report"][lang] = result["report"]
+ payload["cases"].extend({"lang": lang, **r} for r in result["cases"])
+ print(lang, json.dumps(result["report"]), flush=True)
+ except Exception as exc:
+ failed = True
+ payload["report"][lang] = {"error": str(exc)}
+ print("%s FAILED: %s" % (lang, exc), flush=True)
+ with open(args.out, "w", encoding="utf-8") as stream:
+ json.dump(payload, stream, ensure_ascii=False, indent=2, allow_nan=False)
+ stream.write("\n")
+ return 1 if failed else 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/research/eval/test_metamorphic.py b/research/eval/test_metamorphic.py
new file mode 100644
index 00000000..3ac16679
--- /dev/null
+++ b/research/eval/test_metamorphic.py
@@ -0,0 +1,201 @@
+"""Offline tests: python -m unittest research.eval.test_metamorphic -v."""
+from contextlib import redirect_stdout
+from copy import deepcopy
+from io import StringIO
+import json
+import math
+from pathlib import Path
+import random
+import tempfile
+import unittest
+from unittest.mock import MagicMock, patch
+
+from research.eval import metamorphic as m
+
+CASE = ({"utterance": "I was charged twice"}, {"intent": {
+ "type": "choice", "instructions": "Choose the request category.",
+ "criteria": {"billing": "payment issues", "technical": "software help", "sales": "new purchases"},
+}})
+
+
+def semantic_scorer(cases):
+ weights = {"payment issues": 0.7, "software help": 0.2, "new purchases": 0.1}
+ return [[weights[v] for v in q["intent"]["criteria"].values()] for _, q in cases]
+
+
+class TransformTests(unittest.TestCase):
+ def test_mapping_and_unchanged_inputs(self):
+ original = deepcopy(CASE)
+ baseline, order = m.make_variants(CASE, random.Random(13))
+ self.assertEqual(CASE, original)
+ self.assertEqual(baseline["case"], CASE)
+ self.assertNotEqual(order["canonical_indices"], [0, 1, 2])
+ for variant in (baseline, order):
+ criteria = variant["case"][1]["intent"]["criteria"]
+ descriptions = list(CASE[1]["intent"]["criteria"].values())
+ self.assertEqual(list(criteria.values()),
+ [descriptions[i] for i in variant["canonical_indices"]])
+ order["case"][0]["utterance"] = "changed"
+ self.assertEqual(CASE, original)
+
+ def test_identity_shuffle_fallback(self):
+ with patch("random.Random.shuffle", return_value=None):
+ variant = m.permute_options(m.MetamorphicCase(*CASE), seed=42)
+ self.assertEqual(variant.as_record()["canonical_indices"], [1, 2, 0])
+
+ def test_explicit_api_mapping_and_gold(self):
+ case = m.MetamorphicCase(*CASE, gold_index=0)
+ permuted = m.permute_options(case, seed=42)
+ self.assertEqual(permuted, m.permute_options(case, seed=42))
+ self.assertEqual(permuted.canonical_to_transformed,
+ {"billing": "billing", "technical": "technical", "sales": "sales"})
+ self.assertEqual(permuted.transformed_to_canonical["billing"], "billing")
+ self.assertEqual(permuted.case.option_keys[permuted.case.gold_index], "billing")
+ result = m.evaluate_variants(None, case, [permuted], score=semantic_scorer)
+ report = m.compare_predictions(baseline=result.baseline, variants=result.variants)
+ self.assertEqual(report["overall"]["semantic_agreement_rate"], 1)
+ self.assertEqual(report["overall"]["max_probability_drift"], 0)
+
+ def test_many_options_and_nested_descriptions(self):
+ case = deepcopy(CASE)
+ case[1]["intent"]["criteria"] = {str(i): {"description": [str(i)]} for i in range(28)}
+ order = m.make_variants(case, random.Random(0))[1]
+ criteria = order["case"][1]["intent"]["criteria"]
+ self.assertEqual(len(criteria), 28)
+
+ def test_none_or_blank_descriptions_accepted(self):
+ for criteria in ({"a": None, "b": "two"}, {"a": " ", "b": ""}, {"a": None, "b": None}):
+ case = deepcopy(CASE)
+ case[1]["intent"]["criteria"] = criteria
+ variants = m.make_variants(case, random.Random(0))
+ self.assertEqual(len(variants), 2)
+
+ def test_invalid_cases(self):
+ for criteria in ({}, {"a": "one"}, {1: "one", "b": "two"}, {" ": "one", "b": "two"}, {"": "one", "b": "two"}):
+ case = deepcopy(CASE)
+ case[1]["intent"]["criteria"] = criteria
+ with self.subTest(criteria=criteria), self.assertRaises(ValueError):
+ m.make_variants(case, random.Random(0))
+ for questions in ({}, {"a": CASE[1]["intent"], "b": CASE[1]["intent"]},
+ {"a": {"type": "noul", "criteria": {"false": "no", "true": "yes"}}}):
+ with self.assertRaises(ValueError):
+ m.make_variants(({}, questions), random.Random(0))
+
+
+class MetricTests(unittest.TestCase):
+ def test_inverse_mapping(self):
+ self.assertEqual(m.canonicalize([0.1, 0.7, 0.2], [2, 0, 1]), [0.7, 0.2, 0.1])
+
+ def test_invalid_distributions(self):
+ for vector, mapping in (([], []), ([0.2, 0.8], [0, 0]),
+ ([0.2, 0.8], [0]), ([0.2, 0.8], [0, 2]),
+ ([float("nan"), 0.5], [0, 1]),
+ ([float("inf"), 0], [0, 1]),
+ ([-0.1, 1.1], [0, 1]), ([0.2, 0.2], [0, 1])):
+ with self.subTest(vector=vector), self.assertRaises(ValueError):
+ m.canonicalize(vector, mapping)
+
+ def test_known_divergence_and_drift(self):
+ result = m.distribution_metrics([1, 0], [0, 1])
+ self.assertFalse(result["semantic_agreement"])
+ self.assertAlmostEqual(result["js_divergence"], math.log(2))
+ self.assertEqual(result["mean_probability_drift"], 1)
+ self.assertEqual(result["confidence_drift"], 0)
+ identical = m.distribution_metrics([0.5, 0.5, 0], [0.5, 0.5, 0])
+ self.assertTrue(identical["semantic_agreement"])
+ self.assertEqual(identical["js_divergence"], 0)
+
+ def test_agreement_is_distinct_from_stability(self):
+ same_winner = m.distribution_metrics([0.91, 0.06, 0.03], [0.88, 0.08, 0.04])
+ self.assertTrue(same_winner["semantic_agreement"])
+ self.assertGreater(same_winner["mean_probability_drift"], 0)
+ self.assertGreater(same_winner["js_divergence"], 0)
+ changed_winner = m.distribution_metrics([0.91, 0.06, 0.03], [0.08, 0.86, 0.06])
+ self.assertFalse(changed_winner["semantic_agreement"])
+ self.assertLess(changed_winner["confidence_drift"], 0)
+
+ def test_summary_and_confidence_increase(self):
+ pairs = [m.distribution_metrics([0.6, 0.4], [0.1, 0.9]), m.distribution_metrics([0.6, 0.4], [0.8, 0.2])]
+ summary = m.summarise_pairs(pairs)
+ self.assertEqual(summary["semantic_agreement_rate"], 0.5)
+ self.assertAlmostEqual(summary["mean_probability_drift"], 0.35)
+ self.assertAlmostEqual(summary["max_probability_drift"], 0.5)
+ self.assertAlmostEqual(summary["mean_confidence_drift"], 0.25)
+ self.assertAlmostEqual(summary["worst_confidence_increase_on_disagreement"], 0.3)
+ self.assertEqual(m.summarise_pairs([]), {"n": 0})
+ decreasing = m.summarise_pairs([m.distribution_metrics([0.9, 0.1], [0.4, 0.6])])
+ self.assertEqual(decreasing["worst_confidence_increase_on_disagreement"], 0)
+
+
+class EvaluationTests(unittest.TestCase):
+ def test_invariance_batching_seed_gold_and_json(self):
+ outputs = []
+ for batch_size in (1, 2, 16):
+ sizes = []
+ def score(cases):
+ sizes.append(len(cases))
+ return semantic_scorer(cases)
+ result = m.evaluate([CASE, CASE], score, [0, None], batch_size=batch_size)
+ self.assertLessEqual(max(sizes), batch_size)
+ self.assertEqual(result["report"]["overall"]["semantic_agreement_rate"], 1)
+ self.assertEqual(result["report"]["overall"]["max_probability_drift"], 0)
+ self.assertEqual(result["report"]["quality"]["baseline"]["accuracy"], 1)
+ self.assertEqual(result["report"]["quality"]["baseline"]["n_labelled"], 1)
+ self.assertAlmostEqual(result["report"]["quality"]["baseline"]["ece"], 0.3)
+ outputs.append(result)
+ self.assertEqual(outputs[0], outputs[1])
+ self.assertEqual(outputs[0], outputs[2])
+ self.assertEqual(json.loads(json.dumps(outputs[0], allow_nan=False)), outputs[0])
+
+ def test_position_bias_detected(self):
+ report = m.evaluate([CASE], lambda cases: [[1, 0, 0] for _ in cases])["report"]
+ self.assertEqual(report["option_order"]["semantic_agreement_rate"], 0)
+ self.assertEqual(report["quality"]["baseline"], {"n_labelled": 0})
+
+ def test_ties_use_canonical_order(self):
+ result = m.evaluate([CASE], lambda cases: [[1/3] * 3 for _ in cases])
+ self.assertEqual(result["report"]["overall"]["semantic_agreement_rate"], 1)
+
+ def test_empty_bad_scorers_and_gold(self):
+ self.assertEqual(m.evaluate([], lambda _: self.fail("should not score"))["report"]["overall"], {"n": 0})
+ for score in (lambda cases: [], lambda cases: [[1, 0] for _ in cases]):
+ with self.assertRaises(ValueError):
+ m.evaluate([CASE], score)
+ for gold in ([], [3], [-1], [True], [0.5]):
+ with self.assertRaises(ValueError):
+ m.evaluate([CASE], semantic_scorer, gold)
+ with self.assertRaises(ValueError):
+ m.evaluate([CASE], semantic_scorer, batch_size=0)
+
+ def test_model_adapter_temperature(self):
+ class Agent:
+ temperature = [1.0] * 3
+ temperature_raw = [0.5] * 3
+ temperature_by_options = {}
+ temperature_by_options_raw = {}
+ with patch.object(m.harness, "score_cases", return_value=[[0, math.log(3)]]):
+ self.assertAlmostEqual(m.model_scorer(Agent())([CASE])[0][1], 0.75)
+ self.assertAlmostEqual(m.model_scorer(Agent(), True)([CASE])[0][1], 0.9)
+
+ def test_cli_partial_failure_writes_report_and_fails(self):
+ agent = MagicMock()
+ agent.device = "cpu"
+ agent.cfg = {"max_len": 512, "head_max_len": 192}
+ agent.temperature = [1.0] * 3
+ agent.temperature_by_options = {}
+ rows = [{"text": "x", "label_text": "a"}, {"text": "y", "label_text": "b"}]
+ with tempfile.TemporaryDirectory() as folder:
+ out = Path(folder) / "report.json"
+ with redirect_stdout(StringIO()), patch("laya.load", return_value=agent), \
+ patch.object(m.harness, "load_language", side_effect=[rows, RuntimeError("offline")]), \
+ patch.object(m, "model_scorer", return_value=lambda cases: [[0.6, 0.4] for _ in cases]):
+ code = m.main(["--langs", "en,de", "--n-opts", "2", "--out", str(out)])
+ payload = json.loads(out.read_text(encoding="utf-8"))
+ self.assertEqual(code, 1)
+ self.assertEqual(payload["report"]["de"], {"error": "offline"})
+ self.assertEqual(len(payload["cases"]), 2)
+ self.assertEqual(payload["config"]["temperature"], [1.0] * 3)
+
+
+if __name__ == "__main__":
+ unittest.main()