Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(eval): report budget-confounded metamorphic comparisons

Reskyumec47ece74de7b
diff --git a/research/eval/README.md b/research/eval/README.md
index 3ba5f01a..6921e781 100644
--- a/research/eval/README.md
+++ b/research/eval/README.md
@@ -363,6 +363,25 @@ Metrics are grouped under `option_order`, `label_rename` and `overall`:
`overall` is pair-weighted, not a fraction of cases where *all* variants agree. `quality` separately reports accuracy and the existing harness's 15-bin ECE for baseline and each transformation on labelled cases only. Empty groups contain `n: 0`; unlabelled quality groups contain `n_labelled: 0` without inventing an accuracy or ECE. Robustness agreement is not a correctness measure: consistently wrong predictions can be perfectly invariant.
+### Option budget and rendering information
+
+At high option counts Laya's head token budget can cut the options themselves, and two options that share a prefix can come out of the cut as the same token span, which removes the question's ability to tell them apart. This matters for `label_rename`: a shorter label leaves more room for its description, so the baseline and the renamed input can survive the budget with **different content**. For `iot_hue_lightoff: iot hue lightoff` a roomy budget keeps the semantic description on both sides, but a tight budget can keep `iot_` on the baseline and `A: t` on the renamed variant -- no longer information-equivalent.
+
+This section is an evaluation-side follow-up motivated by [#543](https://github.com/NandhaKishorM/laya/issues/543), [#517](https://github.com/NandhaKishorM/laya/issues/517) and [#569](https://github.com/NandhaKishorM/laya/issues/569). #543 raised option collapse at high option counts and was closed as addressed at runtime in v0.3.21 via #569/#542: the model side now reports, for an individual inference, which options lost a token span of their own. That report does not say whether the baseline and transformed variants of a *metamorphic comparison* still carry equivalent option information, which is the question this diagnostic answers -- per comparison, so the resulting drift is not read as pure lexical-label sensitivity.
+
+With a real model, the CLI and `evaluate_variants()` derive a budget probe from the agent (`agent.cfg`) and attach a budget diagnostic to every variant, measured by rendering the options through the same tokenizer/`build_sequence` budget path the model input uses. `evaluate()` has no agent, so it probes only when it is passed an explicit `budget`:
+
+* `budget` per variant: `options`, `distinct_spans` (how many options still have a token span of their own; fewer than `options` means some options can no longer be told apart), `tokens_per_option` (the uniform re-cap applied when the head budget runs out, or `null` when none was), `instruction_tokens`, `span_classes` (which presented options share a span), `retained_text` (the characters the surviving span covers, `null` when unattributable), `retained_descriptions` (the description characters inside that prefix, ignoring the label), `description_present` (whether each option carried description text at all -- `None`, empty and whitespace-only descriptions are absent; numbers, booleans and structured values render as text and count as present), `truncated` and `tail_truncated`.
+* `budget_comparison` per pair: both sides' `options`, `distinct_spans`, `tokens_per_option` and `instruction_tokens`, their `retained_descriptions` aligned back to canonical order, both sides' `description_present` on `label_rename` pairs, `budget_confounded` (`true`, `false`, or `null` when the rendering could not be measured or attributed), and `reasons` naming what decided it: `clean`, `option_count`, `distinct_spans`, `instruction_tokens`, `tail_truncation`, `collision_partition` (the two sides group the options into indistinguishable spans differently, even when the number of collapsed options matches), `retained_description` or `unverified_retained_text` (with the affected canonical `slots`), `missing_semantic_description` (a `label_rename` pair whose options never had description text, with the affected canonical `slots`), or `measurement_error`.
+* `report["budget"]`: `clean`/`confounded`/`unknown` counts and `confounded_rate` for `option_order`, `label_rename` and `overall`. The rate is over verifiable pairs only and is `null` when there are none, so an all-`unknown` run cannot read as clean.
+
+Two conditions make a comparison's validity unknown and are reported as such, never as clean:
+
+* **Renaming options that have no descriptions.** Choice criteria may legitimately carry `None`, empty or whitespace-only descriptions. Then the label is the only semantic content the option had, and rewriting it removes exactly what the transform promises to preserve, so `budget_confounded` is `null` with `missing_semantic_description` even when both sides report empty `retained_descriptions` and the budget never truncates anything. `option_order` pairs are exempt: a permutation moves every key/value pair together, so the content is preserved whatever the descriptions are, and the diagnostic never invents an `unknown` for a permutation.
+* **Unattributable spans.** `retained_text`/`retained_descriptions` are attributed from character prefixes, and the only sound attribution is an exact witness: a prefix whose encoding equals the surviving span token for token. A growing prefix's token count is not guaranteed to be non-decreasing -- a BPE merge can encode a longer prefix to fewer tokens, and lossy normalization can drop characters -- so witnesses need not be adjacent and the search is global, over every character prefix. Exactly one witness is required: none, several, or an option longer than the probe's documented length bound (`metamorphic._ATTRIBUTION_LIMIT` characters) leaves the slot `null` and the pair unknown -- a conservative result instead of a possibly false clean one.
+
+The diagnostic is deterministic and defines no threshold. It never erases or overrides the observed model metrics: a confounded pair still reports its `semantic_agreement` and drift, and `budget_confounded: true` with `semantic_agreement: false` (or with perfect agreement) are both representable. It does **not** prove semantic equivalence -- it reports whether the rendered option content still matches well enough for the drift to be read as lexical-label sensitivity. Options that the budget collapsed on both sides for the same reason stay clean, because option order and label renaming move or rewrite labels by design; only *content* differences count. `tokens_per_option` is reported but never confounds by itself: when the re-cap does not bite, the model sees the same sequence either way. Comparisons the probe cannot measure are reported as unknown rather than assumed clean. Without a model there is no tokenizer to probe, so `evaluate(..., budget=None)` leaves the report byte-identical to earlier versions.
+
For another corpus, the Python API accepts `(state, questions)` cases in the same shape as the harness, and a callback returning probability vectors in presented option order:
```python
@@ -388,7 +407,7 @@ results = evaluate_variants(agent, case, variants)
report = compare_predictions(baseline=results.baseline, variants=results.variants)
```
-For offline tests, pass `agent=None, score=fake_scorer` to `evaluate_variants`. The scorer takes a batch of harness `(state, questions)` inputs and returns one probability vector per input. The public transformations return independent copies and explicit mappings in both directions, including identity label mappings for order-only transformations.
+For offline tests, pass `agent=None, score=fake_scorer` to `evaluate_variants`. The scorer takes a batch of harness `(state, questions)` inputs and returns one probability vector per input. The public transformations return independent copies and explicit mappings in both directions, including identity label mappings for order-only transformations. Both entry points accept `budget=BudgetProbe(tok, max_len=..., head_max_len=...)` to attach the budget diagnostic to a stub tokenizer. `evaluate_variants()` derives one from `agent.cfg` automatically when a real agent is used, and skips it when a `score` is injected; `evaluate()` has no agent and probes only when a `budget` is passed explicitly.
**Semantic agreement and distribution stability are different properties.**
-A shift from `[0.91, 0.06, 0.03]` to `[0.88, 0.08, 0.04]` preserves the decision while showing nonzero drift. Switching the winner is reported as disagreement, regardless of whether confidence rises or falls. No metric here automatically classifies either observation as a bug; acceptable variation depends on the use case, and the report deliberately defines no universal pass/fail threshold.
+A shift from `[0.91, 0.06, 0.03]` to `[0.88, 0.08, 0.04]` preserves the decision while showing nonzero drift. Switching the winner is reported as disagreement, regardless of whether confidence rises or falls. No metric here automatically classifies either observation as a bug; acceptable variation depends on the use case, and the report deliberately defines no universal pass/fail threshold. **Experimental validity is a third property.** A `budget_confounded` comparison is still recorded with its metrics, but its drift no longer isolates lexical-label sensitivity.
diff --git a/research/eval/laya_eval.py b/research/eval/laya_eval.py
index 0a2fab16..2900c1cd 100644
--- a/research/eval/laya_eval.py
+++ b/research/eval/laya_eval.py
@@ -117,6 +117,15 @@ def available_languages() -> List[str]:
# ---------------------------------------------------------------------- scoring
+def internal_question(qdef: Dict[str, Any]) -> Dict[str, Any]:
+ """The `build_sequence` question dict for one harness question definition.
+
+ Shared with the metamorphic budget probe so a diagnostic measures exactly the
+ question that gets scored.
+ """
+ return {"t": qdef["type"], "ins": qdef["instructions"], "crit": qdef.get("criteria")}
+
+
def score_cases(agent, cases) -> List[Any]:
"""Raw marker logits per case, from one collated forward pass.
@@ -132,9 +141,7 @@ def score_cases(agent, cases) -> List[Any]:
items = []
for state, questions in cases:
for _qid, qdef in questions.items():
- q = {"t": qdef["type"],
- "ins": qdef["instructions"],
- "crit": qdef.get("criteria")}
+ q = internal_question(qdef)
ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)
if len(markers) != len(render_options(q)):
raise ValueError("marker/option count mismatch; question exceeds head_max_len")
diff --git a/research/eval/metamorphic.py b/research/eval/metamorphic.py
index d54befdf..65ea5c8f 100644
--- a/research/eval/metamorphic.py
+++ b/research/eval/metamorphic.py
@@ -183,20 +183,23 @@ def summarise_pairs(pairs):
def evaluate(cases, score: Callable, gold_indices=None, seed=harness.SEED,
- batch_size=16):
+ batch_size=16, budget: BudgetProbe | None = None):
"""Score cases via ``score(batch) -> probability vectors in presented order``.
Optional gold indices refer to original option order; entries may be None.
Batches bound inference memory. RNG state and variants do not depend on batch
size. Returned records retain inputs, mappings and full precision vectors.
+ An optional ``budget`` probe checks whether both sides of each comparison still
+ carry the same rendered option information (#543/#517/#569).
"""
cases = list(cases)
rng = random.Random(seed)
generated = [make_variants(case, rng) for case in cases]
- return _evaluate_generated(cases, generated, score, gold_indices, batch_size)
+ return _evaluate_generated(cases, generated, score, gold_indices, batch_size, budget)
-def _evaluate_generated(cases, generated_by_case, score, gold_indices, batch_size):
+def _evaluate_generated(cases, generated_by_case, score, gold_indices, batch_size,
+ budget=None):
if batch_size < 1:
raise ValueError("batch_size must be positive")
cases = list(cases)
@@ -221,7 +224,7 @@ def _evaluate_generated(cases, generated_by_case, score, gold_indices, batch_siz
for (index, v), raw in zip(batch, vectors):
probs = canonicalize(raw, v["canonical_indices"])
pred = max(range(len(probs)), key=probs.__getitem__)
- records[index]["variants"].append({
+ record = {
"kind": v["kind"],
"presented_options": list(next(iter(v["case"][1].values()))["criteria"]),
"canonical_indices": v["canonical_indices"],
@@ -231,19 +234,34 @@ def _evaluate_generated(cases, generated_by_case, score, gold_indices, batch_siz
"pred_label": records[index]["options"][pred],
"confidence": max(probs),
"correct": None if golds[index] is None else pred == golds[index],
- })
+ }
+ if budget is not None:
+ record["budget"] = budget.measure(*v["case"])
+ records[index]["variants"].append(record)
return {"report": _report(records), "cases": records}
def _report(records):
groups = {"option_order": [], "label_rename": []}
+ budget_groups = {"option_order": [], "label_rename": []}
for record in records:
- baseline = record["variants"][0]["probabilities"]
+ baseline = record["variants"][0]
for variant in record["variants"][1:]:
- variant["comparison"] = distribution_metrics(baseline, variant["probabilities"])
+ variant["comparison"] = distribution_metrics(
+ baseline["probabilities"], variant["probabilities"])
groups[variant["kind"]].append(variant["comparison"])
+ if "budget" in baseline and "budget" in variant:
+ variant["budget_comparison"] = compare_budgets(
+ baseline["budget"], variant["budget"], variant["canonical_indices"],
+ kind=variant["kind"])
+ budget_groups[variant["kind"]].append(variant["budget_comparison"])
report = {kind: summarise_pairs(pairs) for kind, pairs in groups.items()}
report["overall"] = summarise_pairs([p for pairs in groups.values() for p in pairs])
+ if any(budget_groups.values()):
+ report["budget"] = {kind: summarise_budgets(pairs)
+ for kind, pairs in budget_groups.items()}
+ report["budget"]["overall"] = summarise_budgets(
+ [p for pairs in budget_groups.values() for p in pairs])
report["quality"] = {}
for kind in ("baseline", "option_order", "label_rename"):
labelled = [v for r in records for v in r["variants"]
@@ -268,6 +286,255 @@ def model_scorer(agent, unclamped=False):
return score
+_ATTRIBUTION_LIMIT = 4096
+"""Longest option text (characters) whose surviving span is attributed at all."""
+
+
+def _retained_prefix(encode, text, kept, full):
+ """Character prefix of `text` the surviving token span `kept` stands for, or None.
+
+ The only sound claim is an exact witness: a prefix whose encoding equals `kept`
+ token for token. Prefix token counts are not guaranteed to be non-decreasing --
+ a BPE merge can encode a longer prefix to fewer tokens, and lossy normalization
+ can drop characters -- so no bisection point identifies the witness, witnesses
+ need not be adjacent, and a single exact witness is required: with several, which
+ characters the model kept is undecidable. Every prefix is therefore checked, and
+ anything above `_ATTRIBUTION_LIMIT` characters is left unattributed rather than
+ searched.
+
+ None means the span could not be attributed to a character boundary, which the
+ caller must not read as "nothing was lost".
+ """
+ if full[:len(kept)] != kept:
+ return None
+ if full == kept:
+ return text
+ if len(text) > _ATTRIBUTION_LIMIT:
+ return None
+ witnesses = [size for size in range(len(text)) if encode(text[:size]) == kept]
+ return text[:witnesses[0]] if len(witnesses) == 1 else None
+
+
+def _retained_description(retained, label, rendered):
+ """Description characters a truncated option kept, or None when unverifiable."""
+ if retained is None or not rendered.startswith(retained):
+ return None
+ marker = label + ": "
+ start = len(marker) if rendered.startswith(marker) else len(rendered)
+ return rendered[start:len(retained)]
+
+
+def _description_present(value):
+ """Whether a criterion value carries description text at all.
+
+ None and blank strings mean "no description" -- whitespace renders as blanks, so
+ it is missing too. Anything structured (dict, list, number, boolean) renders as
+ JSON text and counts as present; `0` and `False` are legitimate criterion values
+ (see `render_options`), not absences.
+ """
+ if value is None:
+ return False
+ from laya.common import render_criterion
+
+ return render_criterion(value).strip() != ""
+
+
+def _to_canonical(values, canonical_indices):
+ """Restore presented-order values to canonical order.
+
+ `canonical_indices[p]` is the canonical slot of the option presented at position
+ `p` (the same direction `canonicalize` uses for probabilities), so the inverse
+ assignment is a scatter, not a gather: gathering is only correct when the
+ permutation is its own inverse.
+ """
+ restored = [None] * len(values)
+ for presented, canonical in enumerate(canonical_indices):
+ restored[canonical] = values[presented]
+ return restored
+
+
+def _canonical_partition(classes, canonical_indices):
+ """Span classes relabelled by first canonical occurrence, so permutations compare equal."""
+ relabel = {}
+ return [relabel.setdefault(class_, len(relabel))
+ for class_ in _to_canonical(classes, canonical_indices)]
+
+
+class BudgetProbe:
+ """Renders options exactly as the model sees them and reports what survived.
+
+ A metamorphic comparison is only information-equivalent if both sides still present
+ the same option content after rendering and budgeting; this probe measures that per
+ pair. It follows the option-collapse question of #543 (closed; addressed at runtime
+ in v0.3.21 via #569/#542) on the evaluation side, where #569's per-inference report
+ cannot answer whether baseline and transformed variants agree.
+
+ `build_sequence` caps every option and, when the head budget runs out, re-caps them
+ all to the same length, so a shorter label leaves more room for its description. The
+ probe reads the option spans back from one `build_sequence` call with an empty state,
+ which closes the last option's span -- in a scored sequence it runs on into the
+ serialized state and always looks distinguishable (#538) -- and checks them against
+ the stats that same call returns.
+ """
+
+ def __init__(self, tok, max_len=512, head_max_len=192):
+ self.tok = tok
+ self.max_len = max_len
+ self.head_max_len = head_max_len
+
+ @classmethod
+ def from_agent(cls, agent):
+ cfg = getattr(agent, "cfg", None) or {}
+ return cls(agent.tok, cfg.get("max_len", 512), cfg.get("head_max_len", 192))
+
+ def measure(self, state, questions):
+ """One case's option spans, span classes and retained descriptions.
+
+ Returns {"error": ...} instead of raising: a tokenizer the probe cannot drive,
+ or a question the budget cuts apart, has to leave the comparison unverifiable
+ rather than decide it.
+ """
+ try:
+ return self._measure(state, questions)
+ except Exception as exc:
+ return {"error": "%s: %s" % (type(exc).__name__, exc)}
+
+ def _measure(self, state, questions):
+ from laya.common import build_sequence, encode_text, render_options
+
+ qdef = next(iter(questions.values()))
+ q = harness.internal_question(qdef)
+ ids, markers, usage = build_sequence(
+ self.tok, state, q, self.max_len, self.head_max_len, state_ids=[], return_stats=True)
+ rendered = render_options(q)
+ if len(markers) != len(rendered):
+ # score_cases raises on the same condition: options the head budget accepted
+ # were dropped again by max_len, so these are not the spans that get scored.
+ raise ValueError("markers %d != options %d" % (len(markers), len(rendered)))
+ sep = self.tok.sep_token_id
+ # With an empty state the last two ids are the option-block and state separators;
+ # anything else at max_len means max_len truncated the option block itself.
+ tail_truncated = len(ids) == self.max_len and ids[-2:] != [sep, sep]
+ spans = [ids[markers[i] + 1:markers[i + 1]] for i in range(len(markers) - 1)]
+ spans.append(ids[markers[-1] + 1:] if tail_truncated else ids[markers[-1] + 1:-2])
+ distinct = len({tuple(span) for span in spans})
+ if not tail_truncated and distinct != usage["options_distinct"]:
+ raise ValueError("recovered spans disagree with build_sequence stats")
+
+ def encode(part):
+ return encode_text(self.tok, " " + part, add_special_tokens=False)["input_ids"]
+
+ labels = list(qdef["criteria"])
+ description_present = [_description_present(value) for value in qdef["criteria"].values()]
+ classes, class_of, retained_descriptions = [], {}, []
+ retained_characters = []
+ truncated = False
+ for position, span in enumerate(spans):
+ text = rendered[position].replace(self.tok.mask_token, " ")
+ full = encode(text)
+ truncated = truncated or len(span) < len(full)
+ retained = _retained_prefix(encode, text, list(span), full)
+ retained_characters.append(retained)
+ retained_descriptions.append(_retained_description(retained, labels[position], text))
+ key = tuple(span)
+ if key not in class_of:
+ class_of[key] = len(class_of)
+ classes.append(class_of[key])
+ return {
+ "options": len(spans),
+ "distinct_spans": distinct,
+ "tokens_per_option": usage["tokens_per_option"],
+ "instruction_tokens": markers[0] - 2,
+ "span_classes": classes,
+ "retained_text": retained_characters,
+ "retained_descriptions": retained_descriptions,
+ "description_present": description_present,
+ "truncated": truncated,
+ "tail_truncated": tail_truncated,
+ }
+
+
+def compare_budgets(baseline, variant, canonical_indices, *, kind):
+ """Whether both sides of a comparison still carry the same option information.
+
+ `baseline` and `variant` are `BudgetProbe.measure` results; the baseline must be in
+ canonical order, and `canonical_indices` maps the variant's presented slots back to
+ canonical ones (the direction `canonicalize` uses). `kind` is the comparison's
+ transformation: a `label_rename` over options whose descriptions are missing removes
+ the only semantic content there was, so its verdict is unknown regardless of what the
+ rendering did; an `option_order` permutation moves every key/value pair together and
+ stays valid. `budget_confounded` is True (the rendering no longer carries the same
+ content, so drift cannot be read as lexical sensitivity alone), False, or None when
+ the rendering could not be measured or attributed. `tokens_per_option` is reported
+ but never confounds by itself: when the re-cap bites it changes the spans reported
+ here, and when it does not the model sees the same sequence either way.
+ """
+ comparison = {key: {"baseline": baseline.get(key), "variant": variant.get(key)}
+ for key in ("options", "distinct_spans", "tokens_per_option",
+ "instruction_tokens")}
+ errors = {side: measurement["error"] for side, measurement in
+ (("baseline", baseline), ("variant", variant)) if "error" in measurement}
+ if errors:
+ return {**comparison, "retained_descriptions": None, "budget_confounded": None,
+ "reasons": [{"code": "measurement_error", "errors": errors}]}
+ baseline_retained = baseline["retained_descriptions"]
+ variant_retained = _to_canonical(variant["retained_descriptions"], canonical_indices)
+ comparison["retained_descriptions"] = {"baseline": baseline_retained,
+ "variant": variant_retained}
+ reasons = []
+ if baseline["options"] != variant["options"]:
+ reasons.append({"code": "option_count"})
+ if baseline["distinct_spans"] != variant["distinct_spans"]:
+ reasons.append({"code": "distinct_spans"})
+ if baseline["instruction_tokens"] != variant["instruction_tokens"]:
+ reasons.append({"code": "instruction_tokens"})
+ if baseline["tail_truncated"] or variant["tail_truncated"]:
+ reasons.append({"code": "tail_truncation"})
+ if baseline["options"] == variant["options"] and _canonical_partition(
+ baseline["span_classes"], range(len(baseline["span_classes"]))) != (
+ _canonical_partition(variant["span_classes"], canonical_indices)):
+ reasons.append({"code": "collision_partition"})
+ pairs = list(zip(baseline_retained, variant_retained))
+ unverified = [slot for slot, pair in enumerate(pairs) if None in pair]
+ differing = [slot for slot, pair in enumerate(pairs)
+ if None not in pair and pair[0] != pair[1]]
+ if differing:
+ reasons.append({"code": "retained_description", "slots": differing})
+ if kind == "label_rename":
+ # Descriptions are moved to the same canonical slots by the rename, so the two
+ # sides' presence vectors are compared as-is once the variant is restored. The
+ # model observation is kept; only the experiment's validity is unknown.
+ presence = _to_canonical(variant["description_present"], canonical_indices)
+ comparison["description_present"] = {"baseline": baseline["description_present"],
+ "variant": presence}
+ missing = [slot for slot, (base, var) in enumerate(
+ zip(baseline["description_present"], presence)) if not base or not var]
+ if missing:
+ return {**comparison, "budget_confounded": None,
+ "reasons": [{"code": "missing_semantic_description", "slots": missing}]}
+ if reasons:
+ return {**comparison, "budget_confounded": True, "reasons": reasons}
+ if unverified:
+ return {**comparison, "budget_confounded": None,
+ "reasons": [{"code": "unverified_retained_text", "slots": unverified}]}
+ return {**comparison, "budget_confounded": False, "reasons": [{"code": "clean"}]}
+
+
+def summarise_budgets(pairs):
+ """Budget verdict counts per group; `confounded_rate` is over verifiable pairs only."""
+ if not pairs:
+ return {"n": 0}
+ verifiable = [pair for pair in pairs if pair["budget_confounded"] is not None]
+ return {
+ "n": len(pairs),
+ "clean": sum(pair["budget_confounded"] is False for pair in pairs),
+ "confounded": sum(pair["budget_confounded"] is True for pair in pairs),
+ "unknown": sum(pair["budget_confounded"] is None for pair in pairs),
+ "confounded_rate": (sum(pair["budget_confounded"] is True for pair in verifiable)
+ / len(verifiable) if verifiable else None),
+ }
+
+
@dataclass
class MetamorphicResults:
baseline: dict
@@ -275,12 +542,15 @@ class MetamorphicResults:
def evaluate_variants(agent, case: MetamorphicCase, variants, *, score=None,
- unclamped=False, batch_size=16):
+ unclamped=False, batch_size=16, budget=None):
"""Evaluate explicit transforms; inject ``score`` for offline stub models.
With a real agent, call ``agent.model.eval()`` first, as with the harness.
All output vectors are restored to the original case's canonical ordering.
+ A real agent is probed for the option budget by default; a stub ``score``
+ is left unprobed unless ``budget`` is passed explicitly.
"""
+ probe = budget if budget is not None else (BudgetProbe.from_agent(agent) if score is None else None)
generated = [_baseline(case).as_record()]
for variant in variants:
if variant.kind not in ("option_order", "label_rename"):
@@ -290,7 +560,7 @@ def evaluate_variants(agent, case: MetamorphicCase, variants, *, score=None,
generated.append(variant.as_record())
result = _evaluate_generated([case.as_pair()], [generated],
score if score is not None else model_scorer(agent, unclamped),
- [case.gold_index], batch_size)
+ [case.gold_index], batch_size, probe)
predictions = result["cases"][0]["variants"]
return MetamorphicResults(predictions[0], predictions[1:])
@@ -326,6 +596,7 @@ def main(argv: Sequence[str] | None = None) -> int:
agent = laya.load(args.model, device=args.device, subfolder=args.subfolder)
agent.model.eval()
+ probe = BudgetProbe.from_agent(agent)
payload: dict[str, Any] = {
"config": {**vars(args), "dataset": harness.DATASET, "split": "test",
"device": str(agent.device), "laya_version": laya.__version__,
@@ -346,7 +617,7 @@ def main(argv: Sequence[str] | None = None) -> int:
if not cases:
raise ValueError("dataset returned no cases")
result = evaluate(cases, model_scorer(agent, args.unclamped), gold,
- args.seed, args.batch_size)
+ args.seed, args.batch_size, probe)
payload["report"][lang] = result["report"]
payload["cases"].extend({"lang": lang, **r} for r in result["cases"])
print(lang, json.dumps(result["report"]), flush=True)
diff --git a/research/eval/test_metamorphic.py b/research/eval/test_metamorphic.py
index f9df757c..1ca5cb0f 100644
--- a/research/eval/test_metamorphic.py
+++ b/research/eval/test_metamorphic.py
@@ -246,5 +246,330 @@ class EvaluationTests(unittest.TestCase):
self.assertEqual(payload["config"]["temperature"], [1.0] * 3)
+class CharTok:
+ """One id per character, so option spans are predictable without model weights."""
+
+ cls_token_id = 2
+ sep_token_id = 3
+ mask_token = "[MASK]"
+ mask_token_id = 1
+
+ def __call__(self, text, add_special_tokens=False, truncation=False, max_length=None):
+ ids = [10 + (ord(char) % 90) for char in text]
+ return {"input_ids": ids[:max_length] if truncation and max_length is not None else ids}
+
+
+class DriftingTok(CharTok):
+ """A truncating tokenizer whose capped output is not a slice of the uncapped one."""
+
+ def __call__(self, text, add_special_tokens=False, truncation=False, max_length=None):
+ result = super().__call__(text, add_special_tokens, truncation, max_length)
+ if truncation and max_length is not None and len(text) > max_length:
+ result["input_ids"][-1] = 99
+ return result
+
+
+class DroppingTok(CharTok):
+ """One id per character, but the tokenizer discards "~", so some prefixes
+ encode to the same span as their shorter neighbours."""
+
+ def __call__(self, text, add_special_tokens=False, truncation=False, max_length=None):
+ ids = [ord(char) for char in text if char != "~"]
+ return {"input_ids": ids[:max_length] if truncation and max_length is not None else ids}
+
+
+def lamp_case(labels, descriptions, instructions="Choose the lamp command."):
+ return ({"utterance": "the room is dark"},
+ {"intent": {"type": "choice", "instructions": instructions,
+ "criteria": dict(zip(labels, descriptions))}})
+
+
+def uniform_scorer(cases):
+ return [[1.0 / len(q["intent"]["criteria"])] * len(q["intent"]["criteria"]) for _, q in cases]
+
+
+def tight_case():
+ """Labels that share a four-character prefix, so a tight budget merges their spans."""
+ return lamp_case(["iot_hue_light%d" % i for i in range(8)],
+ ["turn off lamp %d" % i for i in range(8)])
+
+
+class BudgetTests(unittest.TestCase):
+ def test_roomy_case_is_not_budget_confounded(self):
+ case = m.MetamorphicCase(*lamp_case(
+ ["billing", "technical", "sales"],
+ ["payment issues", "software help", "new purchases"]))
+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=192)
+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)
+ variants = result["cases"][0]["variants"]
+ for variant in variants:
+ self.assertEqual(variant["budget"]["distinct_spans"], 3)
+ self.assertIsNone(variant["budget"]["tokens_per_option"])
+ self.assertEqual(variants[2]["budget"]["retained_descriptions"],
+ ["payment issues", "software help", "new purchases"])
+ self.assertIs(variants[2]["budget_comparison"]["budget_confounded"], False)
+ self.assertEqual(variants[2]["budget_comparison"]["reasons"], [{"code": "clean"}])
+ self.assertEqual(result["report"]["budget"]["label_rename"],
+ {"n": 1, "clean": 1, "confounded": 0, "unknown": 0, "confounded_rate": 0.0})
+ self.assertEqual(result["report"]["budget"]["overall"]["n"], 2)
+
+ def test_tight_budget_exposes_collapsed_spans_and_content(self):
+ # At head_max_len=64 the per-option re-cap keeps four characters, so the baseline
+ # labels collapse to "iot_" and the renamed option keeps "t" of its description
+ # instead (#543/#569).
+ case = m.MetamorphicCase(*tight_case())
+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=64)
+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)
+ variants = result["cases"][0]["variants"]
+ baseline, rename = variants[0], variants[2]
+ self.assertEqual(baseline["budget"]["distinct_spans"], 1)
+ self.assertEqual(baseline["budget"]["retained_text"], ["iot_"] * 8)
+ self.assertEqual(baseline["budget"]["retained_descriptions"], [""] * 8)
+ self.assertEqual(rename["budget"]["distinct_spans"], 8)
+ self.assertEqual(rename["budget"]["retained_text"], ["%s: t" % label for label in "ABCDEFGH"])
+ self.assertEqual(rename["budget"]["retained_descriptions"], ["t"] * 8)
+ comparison = rename["budget_comparison"]
+ self.assertIs(comparison["budget_confounded"], True)
+ self.assertEqual([reason["code"] for reason in comparison["reasons"]],
+ ["distinct_spans", "collision_partition", "retained_description"])
+ self.assertEqual(comparison["reasons"][-1]["slots"], list(range(8)))
+ # Permuting the same collapsing options still moves every key/value pair together.
+ self.assertIs(variants[1]["budget_comparison"]["budget_confounded"], False)
+ self.assertEqual(result["report"]["budget"]["label_rename"],
+ {"n": 1, "clean": 0, "confounded": 1, "unknown": 0, "confounded_rate": 1.0})
+ self.assertEqual(result["report"]["budget"]["overall"]["confounded_rate"], 0.5)
+
+ def test_permutation_alone_is_never_information_loss(self):
+ case = m.MetamorphicCase(*lamp_case(["o%03d" % i for i in range(28)],
+ ["description %d" % i for i in range(28)]))
+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=184)
+ permuted = m.permute_options(case, seed=7)
+ result = m.evaluate_variants(None, case, [permuted], score=uniform_scorer, budget=probe)
+ self.assertEqual(result.baseline["budget"]["distinct_spans"], 28)
+ self.assertEqual(result.variants[0]["budget"]["distinct_spans"], 28)
+ self.assertEqual(result.variants[0]["budget_comparison"]["reasons"], [{"code": "clean"}])
+
+ def test_both_sides_collapse_differently(self):
+ # Equal collision counts, different collided slots: a count-only diagnostic
+ # would call this clean, so the partition comparison has to catch it.
+ case = m.MetamorphicCase(*lamp_case(
+ ["aaaa1", "aaaa2", "cccc", "dddd"], ["one", "two", "three", "four"]))
+ labels = ["bbbb", "aaaa3", "aaaa4", "dddd"]
+ variant_case = m.MetamorphicCase(*lamp_case(labels, ["one", "two", "three", "four"]))
+ mapping = dict(zip(case.option_keys, labels))
+ variant = m.MetamorphicVariant("label_rename", variant_case, mapping)
+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=38)
+ result = m.evaluate_variants(None, case, [variant], score=uniform_scorer, budget=probe)
+ baseline, variant_record = result.baseline, result.variants[0]
+ self.assertEqual(baseline["budget"]["distinct_spans"], 3)
+ self.assertEqual(variant_record["budget"]["distinct_spans"], 3)
+ self.assertEqual(baseline["budget"]["span_classes"], [0, 0, 1, 2])
+ self.assertEqual(variant_record["budget"]["span_classes"], [0, 1, 1, 2])
+ comparison = variant_record["budget_comparison"]
+ self.assertIs(comparison["budget_confounded"], True)
+ self.assertEqual([reason["code"] for reason in comparison["reasons"]],
+ ["collision_partition"])
+
+ def test_only_the_variant_collapses(self):
+ # The mirror of the tight case: the reverse direction, where the baseline
+ # labels stay distinct and the longer renamed labels push the differing
+ # character past the re-cap, has to be reported the same way.
+ baseline_case = m.MetamorphicCase(*lamp_case(["a", "b", "c", "d"], ["turn off lamp"] * 4))
+ variant_case = m.MetamorphicCase(*lamp_case(
+ ["iot_hue_light%d" % i for i in range(4)], ["turn off lamp"] * 4))
+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=40)
+ baseline = probe.measure(*baseline_case.as_pair())
+ variant = probe.measure(*variant_case.as_pair())
+ self.assertEqual(baseline["retained_text"], ["%s: t" % label for label in "abcd"])
+ self.assertEqual(baseline["distinct_spans"], 4)
+ self.assertEqual(variant["retained_text"], ["iot_"] * 4)
+ self.assertEqual(variant["distinct_spans"], 1)
+ comparison = m.compare_budgets(baseline, variant, [0, 1, 2, 3], kind="label_rename")
+ self.assertIs(comparison["budget_confounded"], True)
+ self.assertEqual([reason["code"] for reason in comparison["reasons"]],
+ ["distinct_spans", "collision_partition", "retained_description"])
+
+ def test_non_involutive_permutation_keeps_descriptions_aligned(self):
+ # canonical A B C presented B C A: [1, 2, 0] is a 3-cycle, not its own inverse,
+ # so indexing the variant by it scrambles the slots instead of restoring them.
+ case = m.MetamorphicCase(*lamp_case(
+ ["A", "B", "C"], ["alpha description", "beta description", "gamma description"]))
+ variant = m.MetamorphicVariant(
+ "option_order",
+ m.MetamorphicCase(*lamp_case(["B", "C", "A"],
+ ["beta description", "gamma description", "alpha description"])),
+ {"A": "A", "B": "B", "C": "C"})
+ self.assertEqual(variant.as_record()["canonical_indices"], [1, 2, 0])
+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=192)
+ result = m.evaluate_variants(None, case, [variant], score=uniform_scorer, budget=probe)
+ record = result.variants[0]
+ self.assertEqual(record["budget"]["retained_descriptions"],
+ ["beta description", "gamma description", "alpha description"])
+ self.assertIs(record["budget_comparison"]["budget_confounded"], False)
+ self.assertEqual(record["budget_comparison"]["reasons"], [{"code": "clean"}])
+
+ def test_non_involutive_permutation_keeps_collapsed_partitions_aligned(self):
+ # The same 3-cycle with labels that collide under the re-cap: the span classes
+ # have to be restored to canonical order before the partitions are compared.
+ case = m.MetamorphicCase(*lamp_case(
+ ["aaaa1", "aaaa2", "cccc"], ["one", "two", "three"]))
+ variant = m.MetamorphicVariant(
+ "option_order",
+ m.MetamorphicCase(*lamp_case(["aaaa2", "cccc", "aaaa1"], ["two", "three", "one"])),
+ {"aaaa1": "aaaa1", "aaaa2": "aaaa2", "cccc": "cccc"})
+ self.assertEqual(variant.as_record()["canonical_indices"], [1, 2, 0])
+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=35)
+ result = m.evaluate_variants(None, case, [variant], score=uniform_scorer, budget=probe)
+ self.assertEqual(result.baseline["budget"]["span_classes"], [0, 0, 1])
+ record = result.variants[0]
+ self.assertEqual(record["budget"]["span_classes"], [0, 1, 0])
+ self.assertIs(record["budget_comparison"]["budget_confounded"], False)
+ self.assertEqual(record["budget_comparison"]["reasons"], [{"code": "clean"}])
+
+ def test_ambiguous_prefix_attribution_is_unknown(self):
+ # DroppingTok discards "~", so "aaaa: b" and "aaaa: b~" encode to the same span:
+ # which characters the model kept is undecidable, and neither may be asserted.
+ case = m.MetamorphicCase(*lamp_case(["aaaa", "cccc"], ["b~zzz", "d~zzz"]))
+ probe = m.BudgetProbe(DroppingTok(), max_len=512, head_max_len=34)
+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)
+ variants = result["cases"][0]["variants"]
+ self.assertTrue(variants[0]["budget"]["truncated"])
+ self.assertEqual(variants[0]["budget"]["retained_text"], [None, None])
+ self.assertEqual(variants[1]["budget"]["retained_text"], [None, None])
+ for variant in variants[1:]:
+ comparison = variant["budget_comparison"]
+ self.assertIsNone(comparison["budget_confounded"])
+ self.assertEqual(comparison["reasons"],
+ [{"code": "unverified_retained_text", "slots": [0, 1]}])
+
+ def test_attribution_requires_a_globally_unique_witness(self):
+ # Two prefixes encode to the same span 15 characters apart, and the length
+ # profile dips back to the span's length only at the second one: a search
+ # bounded to the neighbourhood of the bisection point sees a single witness
+ # and would attribute the span to it.
+ text, kept = "x" * 40, [9, 9]
+ table = {size: [1] for size in range(5)}
+ table.update({size: [0, 0, 0] for size in range(5, 40)})
+ table[5], table[20] = list(kept), list(kept)
+ table[40] = kept + [0, 0]
+
+ def encode(part):
+ return table[len(part)]
+
+ self.assertEqual(encode(text[:5]), kept)
+ self.assertEqual(encode(text[:20]), kept)
+ self.assertIsNone(m._retained_prefix(encode, text, kept, encode(text)))
+
+ def test_overlong_options_are_not_attributed(self):
+ # Above the documented length bound the prefix search is skipped rather than
+ # guessed, even though the unique witness here would be trivially findable.
+ text = "y" * (m._ATTRIBUTION_LIMIT + 1)
+
+ def encode(part):
+ return [7] * len(part)
+
+ self.assertIsNone(m._retained_prefix(encode, text, [7], encode(text)))
+
+ def test_renaming_without_descriptions_is_unknown_not_clean(self):
+ # Descriptions may legitimately be absent; then the label is the only semantic
+ # content, and renaming it makes the experiment meaningless, not clean.
+ case = m.MetamorphicCase(*lamp_case(["billing", "technical", "sales"], [None, None, None]))
+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=192)
+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)
+ variants = result["cases"][0]["variants"]
+ baseline, renamed = variants[0], variants[2]
+ self.assertEqual(baseline["budget"]["description_present"], [False] * 3)
+ self.assertEqual(baseline["budget"]["retained_descriptions"], [""] * 3)
+ self.assertEqual(renamed["presented_options"], ["A", "B", "C"])
+ comparison = renamed["budget_comparison"]
+ self.assertIsNone(comparison["budget_confounded"])
+ self.assertEqual(comparison["reasons"],
+ [{"code": "missing_semantic_description", "slots": [0, 1, 2]}])
+ self.assertEqual(comparison["retained_descriptions"],
+ {"baseline": [""] * 3, "variant": [""] * 3})
+ # The model observation is untouched: the stub sees the same distribution.
+ self.assertTrue(renamed["comparison"]["semantic_agreement"])
+ self.assertEqual(renamed["probabilities"], [1 / 3] * 3)
+ # Permuting label-only options moves every key/value pair together and stays valid.
+ self.assertIs(variants[1]["budget_comparison"]["budget_confounded"], False)
+ self.assertEqual(result["report"]["budget"]["label_rename"],
+ {"n": 1, "clean": 0, "confounded": 0, "unknown": 1, "confounded_rate": None})
+
+ def test_blank_descriptions_are_missing_too(self):
+ for blank in ("", " ", "\t"):
+ with self.subTest(description=repr(blank)):
+ case = m.MetamorphicCase(*lamp_case(["billing", "technical"], [blank, "software help"]))
+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=192)
+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)
+ comparison = result["cases"][0]["variants"][2]["budget_comparison"]
+ self.assertIsNone(comparison["budget_confounded"])
+ self.assertEqual(comparison["reasons"],
+ [{"code": "missing_semantic_description", "slots": [0]}])
+
+ def test_structured_descriptions_are_not_missing(self):
+ case = m.MetamorphicCase(*lamp_case(
+ ["billing", "technical", "sales"],
+ [{"description": ["payment issues"]}, 0, False]))
+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=192)
+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)
+ renamed = result["cases"][0]["variants"][2]
+ self.assertEqual(renamed["budget"]["description_present"], [True, True, True])
+ self.assertIs(renamed["budget_comparison"]["budget_confounded"], False)
+ self.assertEqual(renamed["budget_comparison"]["reasons"], [{"code": "clean"}])
+
+ def test_description_presence_classification(self):
+ for value, expected in ((None, False), ("", False), (" ", False), ("\t", False),
+ ("payment issues", True), (" padded ", True),
+ (0, True), (False, True), ({}, True), ([], True)):
+ with self.subTest(value=value):
+ self.assertIs(m._description_present(value), expected)
+
+ def test_unverifiable_rendering_stays_conservative(self):
+ # Two long options whose spans the tokenizer cap cuts mid-option: the probe cannot
+ # attribute the surviving span to a character boundary, so it reports unknown.
+ case = m.MetamorphicCase(*lamp_case(["alpha", "beta"],
+ ["description " * 8, "description " * 8]))
+ probe = m.BudgetProbe(DriftingTok(), max_len=512, head_max_len=192)
+ result = m.evaluate([case.as_pair()], uniform_scorer, budget=probe)
+ for variant in result["cases"][0]["variants"]:
+ self.assertTrue(variant["budget"]["truncated"])
+ self.assertIsNone(variant["budget"]["retained_text"][0])
+ for variant in result["cases"][0]["variants"][1:]:
+ comparison = variant["budget_comparison"]
+ self.assertIsNone(comparison["budget_confounded"])
+ self.assertEqual(comparison["reasons"][0]["code"], "unverified_retained_text")
+ self.assertEqual(result["report"]["budget"]["overall"],
+ {"n": 2, "clean": 0, "confounded": 0, "unknown": 2, "confounded_rate": None})
+
+ def test_broken_tokenizer_is_reported_not_raised(self):
+ case = m.MetamorphicCase(*CASE)
+ probe = m.BudgetProbe(object(), max_len=512, head_max_len=192)
+ result = m.evaluate([case.as_pair()], semantic_scorer, budget=probe)
+ comparison = result["cases"][0]["variants"][1]["budget_comparison"]
+ self.assertIsNone(comparison["budget_confounded"])
+ self.assertEqual(comparison["reasons"][0]["code"], "measurement_error")
+ self.assertIsNone(result["report"]["budget"]["overall"]["confounded_rate"])
+
+ def test_without_a_probe_nothing_changes(self):
+ case = m.MetamorphicCase(*CASE)
+ result = m.evaluate([case.as_pair()], semantic_scorer)
+ self.assertNotIn("budget", result["report"])
+ for variant in result["cases"][0]["variants"]:
+ self.assertNotIn("budget", variant)
+ self.assertNotIn("budget_comparison", variant)
+
+ def test_probe_is_deterministic_and_serialisable(self):
+ case = m.MetamorphicCase(*tight_case())
+ first = m.evaluate([case.as_pair()], uniform_scorer,
+ budget=m.BudgetProbe(CharTok(), max_len=512, head_max_len=64))
+ second = m.evaluate([case.as_pair()], uniform_scorer,
+ budget=m.BudgetProbe(CharTok(), max_len=512, head_max_len=64))
+ self.assertEqual(first, second)
+ self.assertEqual(json.loads(json.dumps(first, allow_nan=False)), first)
+ probe = m.BudgetProbe(CharTok(), max_len=512, head_max_len=64)
+ self.assertEqual(probe.measure(*case.as_pair()), probe.measure(*case.as_pair()))
+ self.assertEqual(probe.measure(*case.as_pair()), m.BudgetProbe(CharTok(), 512, 64).measure(*case.as_pair()))
+
+
if __name__ == "__main__":
unittest.main()