Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(eval): report budget-confounded metamorphic comparisons
Reskyumec47ece74de7b
diff --git a/research/eval/README.md b/research/eval/README.mdindex 3ba5f01a..6921e781 100644--- a/research/eval/README.md+++ b/research/eval/README.md@@ -363,6 +363,25 @@ Metrics are grouped under `option_order`, `label_rename` and `overall`:`overall` is pair-weighted, not a fraction of cases where *all* variants agree. `quality` separately reports accuracy and the existing harness's 15-bin ECE for baseline and each transformation on labelled cases only. Empty groups contain `n: 0`; unlabelled quality groups contain `n_labelled: 0` without inventing an accuracy or ECE. Robustness agreement is not a correctness measure: consistently wrong predictions can be perfectly invariant.+### Option budget and rendering information++At high option counts Laya's head token budget can cut the options themselves, and two options that share a prefix can come out of the cut as the same token span, which removes the question's ability to tell them apart. This matters for `label_rename`: a shorter label leaves more room for its description, so the baseline and the renamed input can survive the budget with **different content**. For `iot_hue_lightoff: iot hue lightoff` a roomy budget keeps the semantic description on both sides, but a tight budget can keep `iot_` on the baseline and `A: t` on the renamed variant -- no longer information-equivalent.++This section is an evaluation-side follow-up motivated by [#543](https://github.com/NandhaKishorM/laya/issues/543), [#517](https://github.com/NandhaKishorM/laya/issues/517) and [#569](https://github.com/NandhaKishorM/laya/issues/569). #543 raised option collapse at high option counts and was closed as addressed at runtime in v0.3.21 via #569/#542: the model side now reports, for an individual inference, which options lost a token span of their own. That report does not say whether the baseline and transformed variants of a *metamorphic comparison* still carry equivalent option information, which is the question this diagnostic answers -- per comparison, so the resulting drift is not read as pure lexical-label sensitivity.++With a real model, the CLI and `evaluate_variants()` derive a budget probe from the agent (`agent.cfg`) and attach a budget diagnostic to every variant, measured by rendering the options through the same tokenizer/`build_sequence` budget path the model input uses. `evaluate()` has no agent, so it probes only when it is passed an explicit `budget`:++* `budget` per variant: `options`, `distinct_spans` (how many options still have a token span of their own; fewer than `options` means some options can no longer be told apart), `tokens_per_option` (the uniform re-cap applied when the head budget runs out, or `null` when none was), `instruction_tokens`, `span_classes` (which presented options share a span), `retained_text` (the characters the surviving span covers, `null` when unattributable), `retained_descriptions` (the description characters inside that prefix, ignoring the label), `description_present` (whether each option carried description text at all -- `None`, empty and whitespace-only descriptions are absent; numbers, booleans and structured values render as text and count as present), `truncated` and `tail_truncated`.+* `budget_comparison` per pair: both sides' `options`, `distinct_spans`, `tokens_per_option` and `instruction_tokens`, their `retained_descriptions` aligned back to canonical order, both sides' `description_present` on `label_rename` pairs, `budget_confounded` (`true`, `false`, or `null` when the rendering could not be measured or attributed), and `reasons` naming what decided it: `clean`, `option_count`, `distinct_spans`, `instruction_tokens`, `tail_truncation`, `collision_partition` (the two sides group the options into indistinguishable spans differently, even when the number of collapsed options matches), `retained_description` or `unverified_retained_text` (with the affected canonical `slots`), `missing_semantic_description` (a `label_rename` pair whose options never had description text, with the affected canonical `slots`), or `measurement_error`.+* `report["budget"]`: `clean`/`confounded`/`unknown` counts and `confounded_rate` for `option_order`, `label_rename` and `overall`. The rate is over verifiable pairs only and is `null` when there are none, so an all-`unknown` run cannot read as clean.++Two conditions make a comparison's validity unknown and are reported as such, never as clean:++* **Renaming options that have no descriptions.** Choice criteria may legitimately carry `None`, empty or whitespace-only descriptions. Then the label is the only semantic content the option had, and rewriting it removes exactly what the transform promises to preserve, so `budget_confounded` is `null` with `missing_semantic_description` even when both sides report empty `retained_descriptions` and the budget never truncates anything. `option_order` pairs are exempt: a permutation moves every key/value pair together, so the content is preserved whatever the descriptions are, and the diagnostic never invents an `unknown` for a permutation.+* **Unattributable spans.** `retained_text`/`retained_descriptions` are attributed from character prefixes, and the only sound attribution is an exact witness: a prefix whose encoding equals the surviving span token for token. A growing prefix's token count is not guaranteed to be non-decreasing -- a BPE merge can encode a longer prefix to fewer tokens, and lossy normalization can drop characters -- so witnesses need not be adjacent and the search is global, over every character prefix. Exactly one witness is required: none, several, or an option longer than the probe's documented length bound (`metamorphic._ATTRIBUTION_LIMIT` characters) leaves the slot `null` and the pair unknown -- a conservative result instead of a possibly false clean one.++The diagnostic is deterministic and defines no threshold. It never erases or overrides the observed model metrics: a confounded pair still reports its `semantic_agreement` and drift, and `budget_confounded: true` with `semantic_agreement: false` (or with perfect agreement) are both representable. It does **not** prove semantic equivalence -- it reports whether the rendered option content still matches well enough for the drift to be read as lexical-label sensitivity. Options that the budget collapsed on both sides for the same reason stay clean, because option order and label renaming move or rewrite labels by design; only *content* differences count. `tokens_per_option` is reported but never confounds by itself: when the re-cap does not bite, the model sees the same sequence either way. Comparisons the probe cannot measure are reported as unknown rather than assumed clean. Without a model there is no tokenizer to probe, so `evaluate(..., budget=None)` leaves the report byte-identical to earlier versions.+For another corpus, the Python API accepts `(state, questions)` cases in the same shape as the harness, and a callback returning probability vectors in presented option order:```python@@ -388,7 +407,7 @@ results = evaluate_variants(agent, case, variants)report = compare_predictions(baseline=results.baseline, variants=results.variants)```-For offline tests, pass `agent=None, score=fake_scorer` to `evaluate_variants`. The scorer takes a batch of harness `(state, questions)` inputs and returns one probability vector per input. The public transformations return independent copies and explicit mappings in both directions, including identity label mappings for order-only transformations.+For offline tests, pass `agent=None, score=fake_scorer` to `evaluate_variants`. The scorer takes a batch of harness `(state, questions)` inputs and returns one probability vector per input. The public transformations return independent copies and explicit mappings in both directions, including identity label mappings for order-only transformations. Both entry points accept `budget=BudgetProbe(tok, max_len=..., head_max_len=...)` to attach the budget diagnostic to a stub tokenizer. `evaluate_variants()` derives one from `agent.cfg` automatically when a real agent is used, and skips it when a `score` is injected; `evaluate()` has no agent and probes only when a `budget` is passed explicitly.**Semantic agreement and distribution stability are different properties.**-A shift from `[0.91, 0.06, 0.03]` to `[0.88, 0.08, 0.04]` preserves the decision while showing nonzero drift. Switching the winner is reported as disagreement, regardless of whether confidence rises or falls. No metric here automatically classifies either observation as a bug; acceptable variation depends on the use case, and the report deliberately defines no universal pass/fail threshold.+A shift from `[0.91, 0.06, 0.03]` to `[0.88, 0.08, 0.04]` preserves the decision while showing nonzero drift. Switching the winner is reported as disagreement, regardless of whether confidence rises or falls. No metric here automatically classifies either observation as a bug; acceptable variation depends on the use case, and the report deliberately defines no universal pass/fail threshold. **Experimental validity is a third property.** A `budget_confounded` comparison is still recorded with its metrics, but its drift no longer isolates lexical-label sensitivity.diff --git a/research/eval/laya_eval.py b/research/eval/laya_eval.pyindex 0a2fab16..2900c1cd 100644--- a/research/eval/laya_eval.py+++ b/research/eval/laya_eval.py@@ -117,6 +117,15 @@ def available_languages() -> List[str]:# ---------------------------------------------------------------------- scoring+def internal_question(qdef: Dict[str, Any]) -> Dict[str, Any]:+ """The `build_sequence` question dict for one harness question definition.++ Shared with the metamorphic budget probe so a diagnostic measures exactly the+ question that gets scored.+ """+ return {"t": qdef["type"], "ins": qdef["instructions"], "crit": qdef.get("criteria")}++def score_cases(agent, cases) -> List[Any]:"""Raw marker logits per case, from one collated forward pass.@@ -132,9 +141,7 @@ def score_cases(agent, cases) -> List[Any]:items = []for state, questions in cases:for _qid, qdef in questions.items():- q = {"t": qdef["type"],- "ins": qdef["instructions"],- "crit": qdef.get("criteria")}+ q = internal_question(qdef)ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)if len(markers) != len(render_options(q)):raise ValueError("marker/option count mismatch; question exceeds head_max_len")diff --git a/research/eval/metamorphic.py b/research/eval/metamorphic.pyindex d54befdf..65ea5c8f 100644--- a/research/eval/metamorphic.py+++ b/research/eval/metamorphic.py@@ -183,20 +183,23 @@ def summarise_pairs(pairs):def evaluate(cases, score: Callable, gold_indices=None, seed=harness.SEED,- batch_size=16):+ batch_size=16, budget: BudgetProbe | None = None):"""Score cases via ``score(batch) -> probability vectors in presented order``.Optional gold indices refer to original option order; entries may be None.Batches bound inference memory. RNG state and variants do not depend on batchsize. Returned records retain inputs, mappings and full precision vectors.+ An optional ``budget`` probe checks whether both sides of each comparison still+ carry the same rendered option information (#543/#517/#569)."""cases = list(cases)rng = random.Random(seed)generated = [make_variants(case, rng) for case in cases]- return _evaluate_generated(cases, generated, score, gold_indices, batch_size)+ return _evaluate_generated(cases, generated, score, gold_indices, batch_size, budget)-def _evaluate_generated(cases, generated_by_case, score, gold_indices, batch_size):+def _evaluate_generated(cases, generated_by_case, score, gold_indices, batch_size,+ budget=None):if batch_size < 1:raise ValueError("batch_size must be positive")cases = list(cases)@@ -221,7 +224,7 @@ def _evaluate_generated(cases, generated_by_case, score, gold_indices, batch_sizfor (index, v), raw in zip(batch, vectors):probs = canonicalize(raw, v["canonical_indices"])pred = max(range(len(probs)), key=probs.__getitem__)- records[index]["variants"].append({+ record = {"kind": v["kind"],"presented_options": list(next(iter(v["case"][1].values()))["criteria"]),"canonical_indices": v["canonical_indices"],@@ -231,19 +234,34 @@ def _evaluate_generated(cases, generated_by_case, score, gold_indices, batch_siz"pred_label": records[index]["options"][pred],"confidence": max(probs),"correct": None if golds[index] is None else pred == golds[index],- })+ }+ if budget is not None:+ record["budget"] = budget.measure(*v["case"])+ records[index]["variants"].append(record)return {"report": _report(records), "cases": records}def _report(records):groups = {"option_order": [], "label_rename": []}+ budget_groups = {"option_order": [], "label_rename": []}for record in records:- baseline = record["variants"][0]["probabilities"]+ baseline = record["variants"][0]for variant in record["variants"][1:]:- variant["comparison"] = distribution_metrics(baseline, variant["probabilities"])+ variant["comparison"] = distribution_metrics(+ baseline["probabilities"], variant["probabilities"])groups[variant["kind"]].append(variant["comparison"])+ if "budget" in baseline and "budget" in variant:+ variant["budget_comparison"] = compare_budgets(+ baseline["budget"], variant["budget"], variant["canonical_indices"],+ kind=variant["kind"])+ budget_groups[variant["kind"]].append(variant["budget_comparison"])report = {kind: summarise_pairs(pairs) for kind, pairs in groups.items()}report["overall"] = summarise_pairs([p for pairs in groups.values() for p in pairs])+ if any(budget_groups.values()):+ report["budget"] = {kind: summarise_budgets(pairs)+ for kind, pairs in budget_groups.items()}+ report["budget"]["overall"] = summarise_budgets(+ [p for pairs in budget_groups.values() for p in pairs])report["quality"] = {}for kind in ("baseline", "option_order", "label_rename"):labelled = [v for r in records for v in r["variants"]@@ -268,6 +286,255 @@ def model_scorer(agent, unclamped=False):return score+_ATTRIBUTION_LIMIT = 4096+"""Longest option text (characters) whose surviving span is attributed at all."""+++def _retained_prefix(encode, text, kept, full):+ """Character prefix of `text` the surviving token span `kept` stands for, or None.++ The only sound claim is an exact witness: a prefix whose encoding equals `kept`+ token for token. Prefix token counts are not guaranteed to be non-decreasing --+ a BPE merge can encode a longer prefix to fewer tokens, and lossy normalization+ can drop characters -- so no bisection point identifies the witness, witnesses+ need not be adjacent, and a single exact witness is required: with several, which+ characters the model kept is undecidable. Every prefix is therefore checked, and+ anything above `_ATTRIBUTION_LIMIT` characters is left unattributed rather than+ searched.++ None means the span could not be attributed to a character boundary, which the+ caller must not read as "nothing was lost".+ """+ if full[:len(kept)] != kept:+ return None+ if full == kept:+ return text+ if len(text) > _ATTRIBUTION_LIMIT:+ return None+ witnesses = [size for size in range(len(text)) if encode(text[:size]) == kept]+ return text[:witnesses[0]] if len(witnesses) == 1 else None+++def _retained_description(retained, label, rendered):+ """Description characters a truncated option kept, or None when unverifiable."""+ if retained is None or not rendered.startswith(retained):+ return None+ marker = label + ": "+ start = len(marker) if rendered.startswith(marker) else len(rendered)+ return rendered[start:len(retained)]+++def _description_present(value):+ """Whether a criterion value carries description text at all.++ None and blank strings mean "no description" -- whitespace renders as blanks, so+ it is missing too. Anything structured (dict, list, number, boolean) renders as+ JSON text and counts as present; `0` and `False` are legitimate criterion values+ (see `render_options`), not absences.+ """+ if value is None:+ return False+ from laya.common import render_criterion++ return render_criterion(value).strip() != ""+++def _to_canonical(values, canonical_indices):+ """Restore presented-order values to canonical order.++ `canonical_indices[p]` is the canonical slot of the option presented at position+ `p` (the same direction `canonicalize` uses for probabilities), so the inverse+ assignment is a scatter, not a gather: gathering is only correct when the+ permutation is its own inverse.+ """+ restored = [None] * len(values)+ for presented, canonical in enumerate(canonical_indices):+ restored[canonical] = values[presented]+ return restored+++def _canonical_partition(classes, canonical_indices):+ """Span classes relabelled by first canonical occurrence, so permutations compare equal."""+ relabel = {}+ return [relabel.setdefault(class_, len(relabel))+ for class_ in _to_canonical(classes, canonical_indices)]+++class BudgetProbe:+ """Renders options exactly as the model sees them and reports what survived.++ A metamorphic comparison is only information-equivalent if both sides still present+ the same option content after rendering and budgeting; this probe measures that per+ pair. It follows the option-collapse question of #543 (closed; addressed at runtime+ in v0.3.21 via #569/#542) on the evaluation side, where #569's per-inference report+ cannot answer whether baseline and transformed variants agree.++ `build_sequence` caps every option and, when the head budget runs out, re-caps them+ all to the same length, so a shorter label leaves more room for its description. The+ probe reads the option spans back from one `build_sequence` call with an empty state,+ which closes the last option's span -- in a scored sequence it runs on into the+ serialized state and always looks distinguishable (#538) -- and checks them against+ the stats that same call returns.+ """++ def __init__(self, tok, max_len=512, head_max_len=192):+ self.tok = tok+ self.max_len = max_len+ self.head_max_len = head_max_len++ @classmethod+ def from_agent(cls, agent):+ cfg = getattr(agent, "cfg", None) or {}+ return cls(agent.tok, cfg.get("max_len", 512), cfg.get("head_max_len", 192))++ def measure(self, state, questions):+ """One case's option spans, span classes and retained descriptions.++ Returns {"error": ...} instead of raising: a tokenizer the probe cannot drive,+ or a question the budget cuts apart, has to leave the comparison unverifiable+ rather than decide it.+ """+ try:+ return self._measure(state, questions)+ except Exception as exc:+ return {"error": "%s: %s" % (type(exc).__name__, exc)}++ def _measure(self, state, questions):+ from laya.common import build_sequence, encode_text, render_options++ qdef = next(iter(questions.values()))+ q = harness.internal_question(qdef)+ ids, markers, usage = build_sequence(+ self.tok, state, q, self.max_len, self.head_max_len, state_ids=[], return_stats=True)+ rendered = render_options(q)+ if len(markers) != len(rendered):+ # score_cases raises on the same condition: options the head budget accepted+ # were dropped again by max_len, so these are not the spans that get scored.+ raise ValueError("markers %d != options %d" % (len(markers), len(rendered)))+ sep = self.tok.sep_token_id+ # With an empty state the last two ids are the option-block and state separators;+ # anything else at max_len means max_len truncated the option block itself.+ tail_truncated = len(ids) == self.max_len and ids[-2:] != [sep, sep]+ spans = [ids[markers[i] + 1:markers[i + 1]] for i in range(len(markers) - 1)]+ spans.append(ids[markers[-1] + 1:] if tail_truncated else ids[markers[-1] + 1:-2])+ distinct = len({tuple(span) for span in spans})+ if not tail_truncated and distinct != usage["options_distinct"]:+ raise ValueError("recovered spans disagree with build_sequence stats")++ def encode(part):+ return encode_text(self.tok, " " + part, add_special_tokens=False)["input_ids"]++ labels = list(qdef["criteria"])+ description_present = [_description_present(value) for value in qdef["criteria"].values()]+ classes, class_of, retained_descriptions = [], {}, []+ retained_characters = []+ truncated = False+ for position, span in enumerate(spans):+ text = rendered[position].replace(self.tok.mask_token, " ")+ full = encode(text)+ truncated = truncated or len(span) < len(full)+ retained = _retained_prefix(encode, text, list(span), full)+ retained_characters.append(retained)+ retained_descriptions.append(_retained_description(retained, labels[position], text))+ key = tuple(span)+ if key not in class_of:+ class_of[key] = len(class_of)+ classes.append(class_of[key])+ return {+ "options": len(spans),+ "distinct_spans": distinct,+ "tokens_per_option": usage["tokens_per_option"],+ "instruction_tokens": markers[0] - 2,+ "span_classes": classes,+ "retained_text": retained_characters,+ "retained_descriptions": retained_descriptions,+ "description_present": description_present,+ "truncated": truncated,+ "tail_truncated": tail_truncated,+ }+++def compare_budgets(baseline, variant, canonical_indices, *, kind):+ """Whether both sides of a comparison still carry the same option information.++ `baseline` and `variant` are `BudgetProbe.measure` results; the baseline must be in+ canonical order, and `canonical_indices` maps the variant's presented slots back to+ canonical ones (the direction `canonicalize` uses). `kind` is the comparison's+ transformation: a `label_rename` over options whose descriptions are missing removes+ the only semantic content there was, so its verdict is unknown regardless of what the+ rendering did; an `option_order` permutation moves every key/value pair together and+ stays valid. `budget_confounded` is True (the rendering no longer carries the same+ content, so drift cannot be read as lexical sensitivity alone), False, or None when+ the rendering could not be measured or attributed. `tokens_per_option` is reported+ but never confounds by itself: when the re-cap bites it changes the spans reported+ here, and when it does not the model sees the same sequence either way.+ """+ comparison = {key: {"baseline": baseline.get(key), "variant": variant.get(key)}+ for key in ("options", "distinct_spans", "tokens_per_option",+ "instruction_tokens")}+ errors = {side: measurement["error"] for side, measurement in+ (("baseline", baseline), ("variant", variant)) if "error" in measurement}+ if errors:+ return {**comparison, "retained_descriptions": None, "budget_confounded": None,+ "reasons": [{"code": "measurement_error", "errors": errors}]}+ baseline_retained = baseline["retained_descriptions"]+ variant_retained = _to_canonical(variant["retained_descriptions"], canonical_indices)+ comparison["retained_descriptions"] = {"baseline": baseline_retained,+ "variant": variant_retained}+ reasons = []+ if baseline["options"] != variant["options"]:+ reasons.append({"code": "option_count"})+ if baseline["distinct_spans"] != variant["distinct_spans"]:+ reasons.append({"code": "distinct_spans"})+ if baseline["instruction_tokens"] != variant["instruction_tokens"]:+ reasons.append({"code": "instruction_tokens"})+ if baseline["tail_truncated"] or variant["tail_truncated"]:+ reasons.append({"code": "tail_truncation"})+ if baseline["options"] == variant["options"] and _canonical_partition(+ baseline["span_classes"], range(len(baseline["span_classes"]))) != (+ _canonical_partition(variant["span_classes"], canonical_indices)):+ reasons.append({"code": "collision_partition"})+ pairs = list(zip(baseline_retained, variant_retained))+ unverified = [slot for slot, pair in enumerate(pairs) if None in pair]+ differing = [slot for slot, pair in enumerate(pairs)+ if None not in pair and pair[0] != pair[1]]+ if differing:+ reasons.append({"code": "retained_description", "slots": differing})+ if kind == "label_rename":+ # Descriptions are moved to the same canonical slots by the rename, so the two+ # sides' presence vectors are compared as-is once the variant is restored. The+ # model observation is kept; only the experiment's validity is unknown.+ presence = _to_canonical(variant["description_present"], canonical_indices)+ comparison["description_present"] = {"baseline": baseline["description_present"],+ "variant": presence}+ missing = [slot for slot, (base, var) in enumerate(+ zip(baseline["description_present"], presence)) if not base or not var]+ if missing:+ return {**comparison, "budget_confounded": None,+ "reasons": [{"code": "missing_semantic_description", "slots": missing}]}+ if reasons:+ return {**comparison, "budget_confounded": True, "reasons": reasons}+ if unverified:+ return {**comparison, "budget_confounded": None,+ "reasons": [{"code": "unverified_retained_text", "slots": unverified}]}+ return {**comparison, "budget_confounded": False, "reasons": [{"code": "clean"}]}+++def summarise_budgets(pairs):+ """Budget verdict counts per group; `confounded_rate` is over verifiable pairs only."""+ if not pairs:+ return {"n": 0}+ verifiable = [pair for pair in pairs if pair["budget_confounded"] is not None]+ return {+ "n": len(pairs),+ "clean": sum(pair["budget_confounded"] is False for pair in pairs),+ "confounded": sum(pair["budget_confounded"] is True for pair in pairs),+ "unknown": sum(pair["budget_confounded"] is None for pair in pairs),+ "confounded_rate": (sum(pair["budget_confounded"] is True for pair in verifiable)+ / len(verifiable) if verifiable else None),+ }++@dataclassclass MetamorphicResults:baseline: dict@@ -275,12 +542,15 @@ class MetamorphicResults:def evaluate_variants(agent, case: MetamorphicCase, variants, *, score=None,- unclamped=False, batch_size=16):+ unclamped=False, batch_size=16, budget=None):"""Evaluate explicit transforms; inject ``score`` for offline stub models.With a real agent, call ``agent.model.eval()`` first, as with the harness.All output vectors are restored to the original case's canonical ordering.+ A real agent is probed for the option budget by default; a stub ``score``+ is left unprobed unless ``budget`` is passed explicitly."""+ probe = budget if budget is not None else (BudgetProbe.from_agent(agent) if score is None else None)generated = [_baseline(case).as_record()]for variant in variants:if variant.kind not in ("option_order", "label_rename"):@@ -290,7 +560,7 @@ def evaluate_variants(agent, case: MetamorphicCase, variants, *, score=None,generated.append(variant.as_record())result = _evaluate_generated([case.as_pair()], [generated],score if score is not None else model_scorer(agent, unclamped),- [case.gold_index], batch_size)+ [case.gold_index], batch_size, probe)predictions = result["cases"][0]["variants"]return MetamorphicResults(predictions[0], predictions[1:])@@ -326,6 +596,7 @@ def main(argv: Sequence[str] | None = None) -> int:agent = laya.load(args.model, device=args.device, subfolder=args.subfolder)agent.model.eval()+ probe = BudgetProbe.from_agent(agent)payload: dict[str, Any] = {"config": {**vars(args), "dataset": harness.DATASET, "split": "test","device": str(agent.device), "laya_version": laya.__version__,@@ -346,7 +617,7 @@ def main(argv: Sequence[str] | None = None) -> int:if not cases:raise ValueError("dataset returned no cases")result = evaluate(cases, model_scorer(agent, args.unclamped), gold,- args.seed, args.batch_size)+ args.seed, args.batch_size, probe)payload["report"][lang] = result["report"]payload["cases"].extend({"lang": lang, **r} for r in result["cases"])print(lang, json.dumps(result["report"]), flush=True)diff --git a/research/eval/test_metamorphic.py b/research/eval/test_metamorphic.pyindex f9df757c..1ca5cb0f 100644--- a/research/eval/test_metamorphic.py+++ b/research/eval/test_metamorphic.py@@ -246,5 +246,330 @@ class EvaluationTests(unittest.TestCase):self.assertEqual(payload["config"]["temperature"], [1.0] * 3)+class CharTok:+ """One id per character, so option spans are predictable without model weights."""++ cls_token_id = 2+ sep_token_id = 3+ mask_token = "[MASK]"+ mask_token_id = 1++ def __call__(self, text, add_special_tokens=False, truncation=False, max_length=None):+ ids = [10 + (ord(char) % 90) for char in text]+ return {"input_ids": ids[:max_length] if truncation and max_length is not None else ids}+++class DriftingTok(CharTok):+ """A truncating tokenizer whose capped output is not a slice of the uncapped one."""++ def __call__(self, text, add_special_tokens=False, truncation=False, max_length=None):+ result = super().__call__(text, add_special_tokens, truncation, max_length)+ if truncation and max_length is not None and len(text) > max_length:+ result["input_ids"][-1] = 99+ return result+++class DroppingTok(CharTok):+ """One id per character, but the tokenizer discards "~", so some prefixes+ encode to the same span as their shorter neighbours."""++ def __call__(self, text, add_special_tokens=False, truncation=False, max_length=None):+ ids = [ord(char) for char in text if char != "~"]+ return {"input_ids": ids[:max_length] if truncation and max_length is not None else ids}+++def lamp_case(labels, descriptions, instructions="Choose the lamp command."):+ return ({"utterance": "the room is dark"},+ {"intent": {"type": "choice", "instructions": instructions,+ "criteria": dict(zip(labels, descriptions))}})+++def uniform_scorer(cases):+ return [[1.0 / len(q["intent"]["criteria"])] * len(q["intent"]["criteria"]) for _, q in cases]+++def tight_case():+ """Labels that share a four-character prefix, so a tight budget merges their spans."""+ return lamp_case(["iot_hue_light%d" % i for i in range(8)],+ ["turn off lamp %d" % i for i in range(8)])+++class BudgetTests(unittest.TestCase):+ def test_roomy_case_is_not_budget_confounded(self):+ case = m.MetamorphicCase(*lamp_case(+ ["billing", "technical", "sales"],+ ["payment issues", "software help", "new purchases"]))+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=192)+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)+ variants = result["cases"][0]["variants"]+ for variant in variants:+ self.assertEqual(variant["budget"]["distinct_spans"], 3)+ self.assertIsNone(variant["budget"]["tokens_per_option"])+ self.assertEqual(variants[2]["budget"]["retained_descriptions"],+ ["payment issues", "software help", "new purchases"])+ self.assertIs(variants[2]["budget_comparison"]["budget_confounded"], False)+ self.assertEqual(variants[2]["budget_comparison"]["reasons"], [{"code": "clean"}])+ self.assertEqual(result["report"]["budget"]["label_rename"],+ {"n": 1, "clean": 1, "confounded": 0, "unknown": 0, "confounded_rate": 0.0})+ self.assertEqual(result["report"]["budget"]["overall"]["n"], 2)++ def test_tight_budget_exposes_collapsed_spans_and_content(self):+ # At head_max_len=64 the per-option re-cap keeps four characters, so the baseline+ # labels collapse to "iot_" and the renamed option keeps "t" of its description+ # instead (#543/#569).+ case = m.MetamorphicCase(*tight_case())+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=64)+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)+ variants = result["cases"][0]["variants"]+ baseline, rename = variants[0], variants[2]+ self.assertEqual(baseline["budget"]["distinct_spans"], 1)+ self.assertEqual(baseline["budget"]["retained_text"], ["iot_"] * 8)+ self.assertEqual(baseline["budget"]["retained_descriptions"], [""] * 8)+ self.assertEqual(rename["budget"]["distinct_spans"], 8)+ self.assertEqual(rename["budget"]["retained_text"], ["%s: t" % label for label in "ABCDEFGH"])+ self.assertEqual(rename["budget"]["retained_descriptions"], ["t"] * 8)+ comparison = rename["budget_comparison"]+ self.assertIs(comparison["budget_confounded"], True)+ self.assertEqual([reason["code"] for reason in comparison["reasons"]],+ ["distinct_spans", "collision_partition", "retained_description"])+ self.assertEqual(comparison["reasons"][-1]["slots"], list(range(8)))+ # Permuting the same collapsing options still moves every key/value pair together.+ self.assertIs(variants[1]["budget_comparison"]["budget_confounded"], False)+ self.assertEqual(result["report"]["budget"]["label_rename"],+ {"n": 1, "clean": 0, "confounded": 1, "unknown": 0, "confounded_rate": 1.0})+ self.assertEqual(result["report"]["budget"]["overall"]["confounded_rate"], 0.5)++ def test_permutation_alone_is_never_information_loss(self):+ case = m.MetamorphicCase(*lamp_case(["o%03d" % i for i in range(28)],+ ["description %d" % i for i in range(28)]))+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=184)+ permuted = m.permute_options(case, seed=7)+ result = m.evaluate_variants(None, case, [permuted], score=uniform_scorer, budget=probe)+ self.assertEqual(result.baseline["budget"]["distinct_spans"], 28)+ self.assertEqual(result.variants[0]["budget"]["distinct_spans"], 28)+ self.assertEqual(result.variants[0]["budget_comparison"]["reasons"], [{"code": "clean"}])++ def test_both_sides_collapse_differently(self):+ # Equal collision counts, different collided slots: a count-only diagnostic+ # would call this clean, so the partition comparison has to catch it.+ case = m.MetamorphicCase(*lamp_case(+ ["aaaa1", "aaaa2", "cccc", "dddd"], ["one", "two", "three", "four"]))+ labels = ["bbbb", "aaaa3", "aaaa4", "dddd"]+ variant_case = m.MetamorphicCase(*lamp_case(labels, ["one", "two", "three", "four"]))+ mapping = dict(zip(case.option_keys, labels))+ variant = m.MetamorphicVariant("label_rename", variant_case, mapping)+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=38)+ result = m.evaluate_variants(None, case, [variant], score=uniform_scorer, budget=probe)+ baseline, variant_record = result.baseline, result.variants[0]+ self.assertEqual(baseline["budget"]["distinct_spans"], 3)+ self.assertEqual(variant_record["budget"]["distinct_spans"], 3)+ self.assertEqual(baseline["budget"]["span_classes"], [0, 0, 1, 2])+ self.assertEqual(variant_record["budget"]["span_classes"], [0, 1, 1, 2])+ comparison = variant_record["budget_comparison"]+ self.assertIs(comparison["budget_confounded"], True)+ self.assertEqual([reason["code"] for reason in comparison["reasons"]],+ ["collision_partition"])++ def test_only_the_variant_collapses(self):+ # The mirror of the tight case: the reverse direction, where the baseline+ # labels stay distinct and the longer renamed labels push the differing+ # character past the re-cap, has to be reported the same way.+ baseline_case = m.MetamorphicCase(*lamp_case(["a", "b", "c", "d"], ["turn off lamp"] * 4))+ variant_case = m.MetamorphicCase(*lamp_case(+ ["iot_hue_light%d" % i for i in range(4)], ["turn off lamp"] * 4))+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=40)+ baseline = probe.measure(*baseline_case.as_pair())+ variant = probe.measure(*variant_case.as_pair())+ self.assertEqual(baseline["retained_text"], ["%s: t" % label for label in "abcd"])+ self.assertEqual(baseline["distinct_spans"], 4)+ self.assertEqual(variant["retained_text"], ["iot_"] * 4)+ self.assertEqual(variant["distinct_spans"], 1)+ comparison = m.compare_budgets(baseline, variant, [0, 1, 2, 3], kind="label_rename")+ self.assertIs(comparison["budget_confounded"], True)+ self.assertEqual([reason["code"] for reason in comparison["reasons"]],+ ["distinct_spans", "collision_partition", "retained_description"])++ def test_non_involutive_permutation_keeps_descriptions_aligned(self):+ # canonical A B C presented B C A: [1, 2, 0] is a 3-cycle, not its own inverse,+ # so indexing the variant by it scrambles the slots instead of restoring them.+ case = m.MetamorphicCase(*lamp_case(+ ["A", "B", "C"], ["alpha description", "beta description", "gamma description"]))+ variant = m.MetamorphicVariant(+ "option_order",+ m.MetamorphicCase(*lamp_case(["B", "C", "A"],+ ["beta description", "gamma description", "alpha description"])),+ {"A": "A", "B": "B", "C": "C"})+ self.assertEqual(variant.as_record()["canonical_indices"], [1, 2, 0])+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=192)+ result = m.evaluate_variants(None, case, [variant], score=uniform_scorer, budget=probe)+ record = result.variants[0]+ self.assertEqual(record["budget"]["retained_descriptions"],+ ["beta description", "gamma description", "alpha description"])+ self.assertIs(record["budget_comparison"]["budget_confounded"], False)+ self.assertEqual(record["budget_comparison"]["reasons"], [{"code": "clean"}])++ def test_non_involutive_permutation_keeps_collapsed_partitions_aligned(self):+ # The same 3-cycle with labels that collide under the re-cap: the span classes+ # have to be restored to canonical order before the partitions are compared.+ case = m.MetamorphicCase(*lamp_case(+ ["aaaa1", "aaaa2", "cccc"], ["one", "two", "three"]))+ variant = m.MetamorphicVariant(+ "option_order",+ m.MetamorphicCase(*lamp_case(["aaaa2", "cccc", "aaaa1"], ["two", "three", "one"])),+ {"aaaa1": "aaaa1", "aaaa2": "aaaa2", "cccc": "cccc"})+ self.assertEqual(variant.as_record()["canonical_indices"], [1, 2, 0])+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=35)+ result = m.evaluate_variants(None, case, [variant], score=uniform_scorer, budget=probe)+ self.assertEqual(result.baseline["budget"]["span_classes"], [0, 0, 1])+ record = result.variants[0]+ self.assertEqual(record["budget"]["span_classes"], [0, 1, 0])+ self.assertIs(record["budget_comparison"]["budget_confounded"], False)+ self.assertEqual(record["budget_comparison"]["reasons"], [{"code": "clean"}])++ def test_ambiguous_prefix_attribution_is_unknown(self):+ # DroppingTok discards "~", so "aaaa: b" and "aaaa: b~" encode to the same span:+ # which characters the model kept is undecidable, and neither may be asserted.+ case = m.MetamorphicCase(*lamp_case(["aaaa", "cccc"], ["b~zzz", "d~zzz"]))+ probe = m.BudgetProbe(DroppingTok(), max_len=512, head_max_len=34)+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)+ variants = result["cases"][0]["variants"]+ self.assertTrue(variants[0]["budget"]["truncated"])+ self.assertEqual(variants[0]["budget"]["retained_text"], [None, None])+ self.assertEqual(variants[1]["budget"]["retained_text"], [None, None])+ for variant in variants[1:]:+ comparison = variant["budget_comparison"]+ self.assertIsNone(comparison["budget_confounded"])+ self.assertEqual(comparison["reasons"],+ [{"code": "unverified_retained_text", "slots": [0, 1]}])++ def test_attribution_requires_a_globally_unique_witness(self):+ # Two prefixes encode to the same span 15 characters apart, and the length+ # profile dips back to the span's length only at the second one: a search+ # bounded to the neighbourhood of the bisection point sees a single witness+ # and would attribute the span to it.+ text, kept = "x" * 40, [9, 9]+ table = {size: [1] for size in range(5)}+ table.update({size: [0, 0, 0] for size in range(5, 40)})+ table[5], table[20] = list(kept), list(kept)+ table[40] = kept + [0, 0]++ def encode(part):+ return table[len(part)]++ self.assertEqual(encode(text[:5]), kept)+ self.assertEqual(encode(text[:20]), kept)+ self.assertIsNone(m._retained_prefix(encode, text, kept, encode(text)))++ def test_overlong_options_are_not_attributed(self):+ # Above the documented length bound the prefix search is skipped rather than+ # guessed, even though the unique witness here would be trivially findable.+ text = "y" * (m._ATTRIBUTION_LIMIT + 1)++ def encode(part):+ return [7] * len(part)++ self.assertIsNone(m._retained_prefix(encode, text, [7], encode(text)))++ def test_renaming_without_descriptions_is_unknown_not_clean(self):+ # Descriptions may legitimately be absent; then the label is the only semantic+ # content, and renaming it makes the experiment meaningless, not clean.+ case = m.MetamorphicCase(*lamp_case(["billing", "technical", "sales"], [None, None, None]))+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=192)+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)+ variants = result["cases"][0]["variants"]+ baseline, renamed = variants[0], variants[2]+ self.assertEqual(baseline["budget"]["description_present"], [False] * 3)+ self.assertEqual(baseline["budget"]["retained_descriptions"], [""] * 3)+ self.assertEqual(renamed["presented_options"], ["A", "B", "C"])+ comparison = renamed["budget_comparison"]+ self.assertIsNone(comparison["budget_confounded"])+ self.assertEqual(comparison["reasons"],+ [{"code": "missing_semantic_description", "slots": [0, 1, 2]}])+ self.assertEqual(comparison["retained_descriptions"],+ {"baseline": [""] * 3, "variant": [""] * 3})+ # The model observation is untouched: the stub sees the same distribution.+ self.assertTrue(renamed["comparison"]["semantic_agreement"])+ self.assertEqual(renamed["probabilities"], [1 / 3] * 3)+ # Permuting label-only options moves every key/value pair together and stays valid.+ self.assertIs(variants[1]["budget_comparison"]["budget_confounded"], False)+ self.assertEqual(result["report"]["budget"]["label_rename"],+ {"n": 1, "clean": 0, "confounded": 0, "unknown": 1, "confounded_rate": None})++ def test_blank_descriptions_are_missing_too(self):+ for blank in ("", " ", "\t"):+ with self.subTest(description=repr(blank)):+ case = m.MetamorphicCase(*lamp_case(["billing", "technical"], [blank, "software help"]))+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=192)+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)+ comparison = result["cases"][0]["variants"][2]["budget_comparison"]+ self.assertIsNone(comparison["budget_confounded"])+ self.assertEqual(comparison["reasons"],+ [{"code": "missing_semantic_description", "slots": [0]}])++ def test_structured_descriptions_are_not_missing(self):+ case = m.MetamorphicCase(*lamp_case(+ ["billing", "technical", "sales"],+ [{"description": ["payment issues"]}, 0, False]))+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=192)+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)+ renamed = result["cases"][0]["variants"][2]+ self.assertEqual(renamed["budget"]["description_present"], [True, True, True])+ self.assertIs(renamed["budget_comparison"]["budget_confounded"], False)+ self.assertEqual(renamed["budget_comparison"]["reasons"], [{"code": "clean"}])++ def test_description_presence_classification(self):+ for value, expected in ((None, False), ("", False), (" ", False), ("\t", False),+ ("payment issues", True), (" padded ", True),+ (0, True), (False, True), ({}, True), ([], True)):+ with self.subTest(value=value):+ self.assertIs(m._description_present(value), expected)++ def test_unverifiable_rendering_stays_conservative(self):+ # Two long options whose spans the tokenizer cap cuts mid-option: the probe cannot+ # attribute the surviving span to a character boundary, so it reports unknown.+ case = m.MetamorphicCase(*lamp_case(["alpha", "beta"],+ ["description " * 8, "description " * 8]))+ probe = m.BudgetProbe(DriftingTok(), max_len=512, head_max_len=192)+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)+ for variant in result["cases"][0]["variants"]:+ self.assertTrue(variant["budget"]["truncated"])+ self.assertIsNone(variant["budget"]["retained_text"][0])+ for variant in result["cases"][0]["variants"][1:]:+ comparison = variant["budget_comparison"]+ self.assertIsNone(comparison["budget_confounded"])+ self.assertEqual(comparison["reasons"][0]["code"], "unverified_retained_text")+ self.assertEqual(result["report"]["budget"]["overall"],+ {"n": 2, "clean": 0, "confounded": 0, "unknown": 2, "confounded_rate": None})++ def test_broken_tokenizer_is_reported_not_raised(self):+ case = m.MetamorphicCase(*CASE)+ probe = m.BudgetProbe(object(), max_len=512, head_max_len=192)+ result = m.evaluate([case.as_pair()], semantic_scorer, budget=probe)+ comparison = result["cases"][0]["variants"][1]["budget_comparison"]+ self.assertIsNone(comparison["budget_confounded"])+ self.assertEqual(comparison["reasons"][0]["code"], "measurement_error")+ self.assertIsNone(result["report"]["budget"]["overall"]["confounded_rate"])++ def test_without_a_probe_nothing_changes(self):+ case = m.MetamorphicCase(*CASE)+ result = m.evaluate([case.as_pair()], semantic_scorer)+ self.assertNotIn("budget", result["report"])+ for variant in result["cases"][0]["variants"]:+ self.assertNotIn("budget", variant)+ self.assertNotIn("budget_comparison", variant)++ def test_probe_is_deterministic_and_serialisable(self):+ case = m.MetamorphicCase(*tight_case())+ first = m.evaluate([case.as_pair()], uniform_scorer,+ budget=m.BudgetProbe(CharTok(), max_len=512, head_max_len=64))+ second = m.evaluate([case.as_pair()], uniform_scorer,+ budget=m.BudgetProbe(CharTok(), max_len=512, head_max_len=64))+ self.assertEqual(first, second)+ self.assertEqual(json.loads(json.dumps(first, allow_nan=False)), first)+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=64)+ self.assertEqual(probe.measure(*case.as_pair()), probe.measure(*case.as_pair()))+ self.assertEqual(probe.measure(*case.as_pair()), m.BudgetProbe(CharTok(), 512, 64).measure(*case.as_pair()))++if __name__ == "__main__":unittest.main()