Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(eval): metamorphic label-renaming invariance for choice (#512)
Nikhid39e0e400ae1
diff --git a/research/eval/README.md b/research/eval/README.mdindex b5b3e407..3ba5f01a 100644--- a/research/eval/README.md+++ b/research/eval/README.md@@ -323,7 +323,7 @@ checks, and an order-invariant model scores exactly 1/3.## Metamorphic option-order robustness (experimental)`metamorphic.py` adds the initial scope of-[#244](https://github.com/NandhaKishorM/laya/issues/244): **choice option order robustness**, without changing model/runtime behavior. Label renaming, neutral labels, paraphrases, structured-state permutations, `score` and `noul` perturbations are intentionally deferred. Run from the repository root after installing Laya and `datasets`:+[#244](https://github.com/NandhaKishorM/laya/issues/244): **choice option order robustness**, and the label-renaming transformation from [#512](https://github.com/NandhaKishorM/laya/issues/512), without changing model/runtime behavior. Paraphrases, structured-state permutations, `score` and `noul` perturbations are intentionally deferred. Run from the repository root after installing Laya and `datasets`:```bashpython -m research.eval.metamorphic --model convaiinnovations/laya \@@ -333,18 +333,23 @@ python -m research.eval.metamorphic --model convaiinnovations/laya \python -m unittest research.eval.test_metamorphic -v```-Each MASSIVE case uses the existing harness's sampler and produces two inputs:+Each MASSIVE case uses the existing harness's sampler and produces three inputs:1. The unchanged baseline.2. One seeded shuffle of option order; if the shuffle is the identity, a one-slotrotation is used. This is a bounded diagnostic, not exhaustive permutation testingor a uniform draw over all nonidentity permutations.+3. A deterministic label rename: the option at each position keeps its slot and+ description, and its model-facing key becomes an opaque label (`A`, `B`, `C`, ...+ `Z`, then `key_26`, `key_27`, ...). Order and semantics are unchanged, so any+ drift in the `label_rename` group isolates lexical-label sensitivity (the failure+ mode of #156) from the position sensitivity measured by `option_order`.-Instructions and state are otherwise unchanged. Option key/value pairs are moved together during permutation. Every result is mapped back to the original semantic option order **before** predictions and metrics are computed. Exact ties choose the first canonical option. The RNG starts fresh per language; `--seed` controls both sampling and transformations. `--batch-size` bounds the number of forward-pass inputs and does not alter the generated variants. Model inference may still have small floating-point differences across devices and batch sizes.+Instructions and state are otherwise unchanged. Option key/value pairs are moved together during permutation. Every result is mapped back to the original semantic option order **before** predictions and metrics are computed, and each variant records the explicit bidirectional mapping so the comparison is auditable. Exact ties choose the first canonical option. The RNG starts fresh per language; `--seed` controls both sampling and transformations. `--batch-size` bounds the number of forward-pass inputs and does not alter the generated variants. Model inference may still have small floating-point differences across devices and batch sizes.The JSON contains `config`, per-language `report`, and full `cases`. Each case saves its original input, canonical keys and optional gold index; each variant saves its presented keys, explicit `canonical_to_transformed` and `transformed_to_canonical` label mappings, slot-to-canonical indices, complete **canonical-order** probability vector, prediction, confidence, correctness (or `null`), and comparison to baseline. Probabilities are not rounded. The config records model/subfolder, temperature mode and values, truncation settings, dataset, seed and batch size. For reproducible checkpoint comparisons, use a pinned local snapshot and retain the environment versions alongside the report. `--unclamped` has the same meaning as in `laya_eval`. If any language fails, its error is saved and the command exits nonzero while retaining successful languages.-Metrics are grouped under `option_order` and `overall`:+Metrics are grouped under `option_order`, `label_rename` and `overall`:| Metric | Definition ||---|---|@@ -366,18 +371,18 @@ agent.model.eval()result = evaluate(cases, model_scorer(agent), gold_indices=None, seed=13)```-The first version intentionally accepts only **one choice question per case**, with at least two options. Label renaming and other metamorphic transforms are intentionally deferred as proposed in the issue.+The first version intentionally accepts only **one choice question per case**, with at least two options. Paraphrases and other metamorphic transforms are intentionally deferred as proposed in the issues.For an explicit single-case experiment, the same implementation exposes:```pythonfrom research.eval.metamorphic import (- MetamorphicCase, permute_options,+ MetamorphicCase, permute_options, rename_labels,evaluate_variants, compare_predictions,)case = MetamorphicCase(state, questions, gold_index=None)-variants = [permute_options(case, seed=42)]+variants = [permute_options(case, seed=42), rename_labels(case)]agent.model.eval()results = evaluate_variants(agent, case, variants)report = compare_predictions(baseline=results.baseline, variants=results.variants)diff --git a/research/eval/metamorphic.py b/research/eval/metamorphic.pyindex 097ee1c5..d54befdf 100644--- a/research/eval/metamorphic.py+++ b/research/eval/metamorphic.py@@ -91,18 +91,38 @@ def permute_options(case: MetamorphicCase, seed: int = harness.SEED):return _transform(case, "option_order", order, case.option_keys)+def opaque_labels(count: int, alphabet: str = "ABCDEFGHIJKLMNOPQRSTUVWXYZ") -> list[str]:+ """Deterministic neutral labels: A, B, ... Z, then key_26, key_27, ..."""+ if count < 1:+ raise ValueError("at least one label is required")+ if len(set(alphabet)) < count:+ return [f"key_{i}" for i in range(count)]+ return list(alphabet[:count])+++def rename_labels(case: MetamorphicCase, alphabet: str = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"):+ """Replace model-facing labels with neutral opaque ones; semantics and order unchanged.++ The option at canonical position i is renamed to opaque_labels[i], so the+ presented order is identical and only the label tokens differ (issue #512).+ """+ labels = opaque_labels(len(case.option_keys), alphabet)+ return _transform(case, "label_rename", list(range(len(case.option_keys))), labels)++def _baseline(case):return _transform(case, "baseline", list(range(len(case.option_keys))), case.option_keys)def make_variants(case, rng: random.Random):- """Adapt the harness tuple format to baseline plus option permutation.+ """Adapt the harness tuple format to baseline, option permutation, label rename.- Label renaming, paraphrases, and other metamorphic transforms are intentionally deferred.+ Paraphrases and other metamorphic transforms are intentionally deferred."""canonical = MetamorphicCase(*case)return [v.as_record() for v in (- _baseline(canonical), permute_options(canonical, rng.getrandbits(64)))]+ _baseline(canonical), permute_options(canonical, rng.getrandbits(64)),+ rename_labels(canonical))]def canonicalize(probabilities, canonical_indices):@@ -216,7 +236,7 @@ def _evaluate_generated(cases, generated_by_case, score, gold_indices, batch_sizdef _report(records):- groups = {"option_order": []}+ groups = {"option_order": [], "label_rename": []}for record in records:baseline = record["variants"][0]["probabilities"]for variant in record["variants"][1:]:@@ -225,7 +245,7 @@ def _report(records):report = {kind: summarise_pairs(pairs) for kind, pairs in groups.items()}report["overall"] = summarise_pairs([p for pairs in groups.values() for p in pairs])report["quality"] = {}- for kind in ("baseline", "option_order"):+ for kind in ("baseline", "option_order", "label_rename"):labelled = [v for r in records for v in r["variants"]if v["kind"] == kind and v["correct"] is not None]quality = {"n_labelled": len(labelled)}@@ -263,7 +283,7 @@ def evaluate_variants(agent, case: MetamorphicCase, variants, *, score=None,"""generated = [_baseline(case).as_record()]for variant in variants:- if variant.kind not in ("option_order",):+ if variant.kind not in ("option_order", "label_rename"):raise ValueError("unsupported transformation kind")if list(variant.canonical_to_transformed) != case.option_keys:raise ValueError("variant mapping does not match the canonical case")@@ -314,7 +334,7 @@ def main(argv: Sequence[str] | None = None) -> int:"temperature": list(agent.temperature_raw if args.unclamped else agent.temperature),"temperature_by_options": dict(agent.temperature_by_options_raw if args.unclampedelse agent.temperature_by_options),- "permutations_per_case": 1, "js_log_base": "e"},+ "permutations_per_case": 1, "label_renames_per_case": 1, "js_log_base": "e"},"report": {}, "cases": [],}failed = Falsediff --git a/research/eval/test_metamorphic.py b/research/eval/test_metamorphic.pyindex 3ac16679..f9df757c 100644--- a/research/eval/test_metamorphic.py+++ b/research/eval/test_metamorphic.py@@ -26,11 +26,12 @@ def semantic_scorer(cases):class TransformTests(unittest.TestCase):def test_mapping_and_unchanged_inputs(self):original = deepcopy(CASE)- baseline, order = m.make_variants(CASE, random.Random(13))+ baseline, order, rename = m.make_variants(CASE, random.Random(13))self.assertEqual(CASE, original)self.assertEqual(baseline["case"], CASE)self.assertNotEqual(order["canonical_indices"], [0, 1, 2])- for variant in (baseline, order):+ self.assertEqual(rename["kind"], "label_rename")+ for variant in (baseline, order, rename):criteria = variant["case"][1]["intent"]["criteria"]descriptions = list(CASE[1]["intent"]["criteria"].values())self.assertEqual(list(criteria.values()),@@ -38,6 +39,50 @@ class TransformTests(unittest.TestCase):order["case"][0]["utterance"] = "changed"self.assertEqual(CASE, original)+ def test_rename_labels_preserves_order_and_descriptions(self):+ case = m.MetamorphicCase(*CASE)+ variant = m.rename_labels(case)+ self.assertEqual(variant.kind, "label_rename")+ self.assertEqual(variant.canonical_to_transformed,+ {"billing": "A", "technical": "B", "sales": "C"})+ criteria = variant.case.questions["intent"]["criteria"]+ self.assertEqual(list(criteria), ["A", "B", "C"])+ self.assertEqual(list(criteria.values()),+ list(CASE[1]["intent"]["criteria"].values()))+ self.assertEqual(variant.as_record()["canonical_indices"], [0, 1, 2])+ again = m.rename_labels(case)+ self.assertEqual(variant, again)++ def test_rename_labels_fallback_beyond_alphabet(self):+ case = deepcopy(CASE)+ case[1]["intent"]["criteria"] = {str(i): f"description {i}" for i in range(30)}+ variant = m.rename_labels(m.MetamorphicCase(*case))+ self.assertEqual(variant.canonical_to_transformed["0"], "key_0")+ self.assertEqual(variant.canonical_to_transformed["29"], "key_29")+ self.assertEqual(len(set(variant.canonical_to_transformed.values())), 30)++ def test_rename_labels_exposes_lexical_sensitivity(self):+ # A scorer keyed on the label tokens (not the descriptions) is+ # invariant under permutation but drifts under renaming.+ def label_scorer(cases):+ # Opaque labels pull the probability mass flat AND move the winner:+ # baseline picks billing (0.7), the renamed input picks B (0.4),+ # which canonicalizes back to technical.+ weights = {"billing": 0.7, "technical": 0.2, "sales": 0.1,+ "A": 0.3, "B": 0.4, "C": 0.3}+ return [[weights[v] for v in q["intent"]["criteria"]] for _, q in cases]+ case = m.MetamorphicCase(*CASE, gold_index=0)+ result = m.evaluate_variants(None, case,+ [m.permute_options(case, seed=42), m.rename_labels(case)],+ score=label_scorer)+ self.assertEqual(result.variants[0]["probabilities"], [0.7, 0.2, 0.1])+ self.assertEqual(result.variants[1]["probabilities"], [0.3, 0.4, 0.3])+ report = m.compare_predictions(baseline=result.baseline, variants=result.variants)+ self.assertEqual(report["option_order"]["semantic_agreement_rate"], 1)+ self.assertEqual(report["option_order"]["max_probability_drift"], 0)+ self.assertEqual(report["label_rename"]["semantic_agreement_rate"], 0)+ self.assertGreater(report["label_rename"]["max_probability_drift"], 0.3)+def test_identity_shuffle_fallback(self):with patch("random.Random.shuffle", return_value=None):variant = m.permute_options(m.MetamorphicCase(*CASE), seed=42)@@ -68,7 +113,7 @@ class TransformTests(unittest.TestCase):case = deepcopy(CASE)case[1]["intent"]["criteria"] = criteriavariants = m.make_variants(case, random.Random(0))- self.assertEqual(len(variants), 2)+ self.assertEqual(len(variants), 3)def test_invalid_cases(self):for criteria in ({}, {"a": "one"}, {1: "one", "b": "two"}, {" ": "one", "b": "two"}, {"": "one", "b": "two"}):@@ -147,9 +192,13 @@ class EvaluationTests(unittest.TestCase):self.assertEqual(outputs[0], outputs[2])self.assertEqual(json.loads(json.dumps(outputs[0], allow_nan=False)), outputs[0])- def test_position_bias_detected(self):+ def test_position_and_label_bias_detected(self):report = m.evaluate([CASE], lambda cases: [[1, 0, 0] for _ in cases])["report"]self.assertEqual(report["option_order"]["semantic_agreement_rate"], 0)+ # A pure position scorer is untouched by renaming: order is preserved,+ # so any drift in the label_rename group can only come from labels.+ self.assertEqual(report["label_rename"]["semantic_agreement_rate"], 1)+ self.assertEqual(report["label_rename"]["max_probability_drift"], 0)self.assertEqual(report["quality"]["baseline"], {"n_labelled": 0})def test_ties_use_canonical_order(self):