Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(research): add a reproducible per-language evaluation harness (#210)

PerryLink2d2c15ce8896
diff --git a/research/__init__.py b/research/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/research/eval/README.md b/research/eval/README.md
new file mode 100644
index 00000000..d54b3200
--- /dev/null
+++ b/research/eval/README.md
@@ -0,0 +1,172 @@
+# laya-eval — a reproducible per-language evaluation harness
+
+An independent harness for measuring a Laya checkpoint: per-language accuracy and
+calibration, with machine-readable per-case output.
+
+It exists because the repository's own benchmark scripts are research code. They
+download every checkpoint, run every part, and print tables. There was no small,
+reproducible harness a third party could point at a checkpoint to answer "how does
+this model do on my language, and can I trust its confidence?" — and no per-case
+record behind the published numbers, so they could not be re-derived without a GPU
+and the original environment.
+
+This addresses the ask in
+[#35](https://github.com/NandhaKishorM/laya/issues/35):
+
+> A fixed prompt format plus a per-language ECE report is exactly what the repo
+> lacks ... Per-case JSON would be very welcome too.
+
+## Install
+
+Nothing beyond a normal Laya install, plus `datasets`:
+
+```bash
+pip install laya datasets
+```
+
+The harness is deliberately not part of the `laya` package: it is evaluation code,
+it pulls a dataset, and `import laya` should stay dependency-light.
+
+## Use
+
+```bash
+# one language
+python research/eval/laya_eval.py --model convaiinnovations/laya --langs en
+
+# several, with a JSON report
+python research/eval/laya_eval.py --model convaiinnovations/laya \
+ --langs en,de,ro --out report.json
+
+# every MASSIVE language
+python research/eval/laya_eval.py --model convaiinnovations/laya --langs all --out all.json
+
+# the multilingual checkpoint
+python research/eval/laya_eval.py --model convaiinnovations/laya \
+ --subfolder multilingual --langs all --out multilingual.json
+
+# a local checkpoint
+python research/eval/laya_eval.py --model ./my-finetune --langs en
+```
+
+Output, per language:
+
+```
+ en n=100 acc=0.8200 macro_f1=0.7876 ece=0.1789 conf=0.9989 (36.7s)
+
+ macro over 51 languages: acc=... ece=... f1=...
+```
+
+and a JSON document with four parts:
+
+| key | contents |
+|---|---|
+| `config` | checkpoint, device, `max_len`, `head_max_len`, dataset, `per_lang`, `n_opts`, seed, the fixed instructions, the temperatures in force, laya version |
+| `report` | per language: `n`, `accuracy`, `macro_f1`, `ece`, `mean_confidence`, `acc_at_50_coverage`, `temperature` |
+| `summary` | macro accuracy / ECE / macro-F1 over the languages that ran |
+| `cases` | every individual decision |
+
+Each case carries `state`, `instructions`, `options`, `gold_index`, `gold_label`,
+`pred_index`, `pred_label`, `probability`, `p_gold`, `confidence`, `correct` and the
+`temperature` used. That is enough to re-derive every number in `report` from the
+file alone, with no model and no network:
+
+```python
+import json
+d = json.load(open("report.json"))
+n = len(d["cases"])
+acc = sum(c["correct"] for c in d["cases"]) / n
+assert abs(acc - d["report"]["en"]["accuracy"]) < 5e-5
+```
+
+## Method
+
+Chosen so results are comparable with the published tables, which is the point of a
+second implementation:
+
+| | |
+|---|---|
+| dataset | `mteb/amazon_massive_intent`, split `test` |
+| sampling | first `--per-lang` rows (default 100); `random.Random(13)` created **fresh per language** |
+| options | `--n-opts` (default 20): the gold label plus `rng.sample` of the others, then shuffled |
+| prompt | `What is the user asking for in \`utterance\`?` |
+| option text | label with `_` → space and `.` → `: ` |
+| metrics | accuracy, macro-F1, ECE over 15 equal-width confidence bins, mean confidence, accuracy at 50% coverage |
+| temperature | the bucket `Agent` would apply, selected by `(question type, option count)` |
+
+`--unclamped` scores with the checkpoint's **raw** bucket temperatures instead of the
+clamped ones `Agent` applies. That is what reproduces the committed sweep, and it is
+also how the two can be compared.
+
+## Verification
+
+Checked against the committed sweep, not only against itself. Both checkpoints over
+**all 51 languages** (`--langs all --per-lang 100 --n-opts 20`), per-language accuracy
+compared against `research/results/cpu_51_language_sweep.json`:
+
+| checkpoint | per-language accuracy identical | `macro_accuracy` committed → mine | `macro_ece` committed → mine |
+|---|---|---|---|
+| **english** | **51 / 51** | 0.2269 → **0.2269** | 0.7331 → 0.5709 |
+| multilingual | 6 / 51 | 0.3661 → 0.4008 | 0.3869 → 0.3911 |
+
+The english checkpoint reproduces every per-language accuracy, not just the macro.
+Those are deterministic outputs on a fixed sample, so they can only agree if the
+sampling, prompt text, option construction and inference path are all identical to
+the committed run.
+
+The `macro_ece` gap on english is the temperature clamp — `choice:11+` is `0.1006`
+raw and `0.5` as served ([#208](https://github.com/NandhaKishorM/laya/issues/208)).
+`--unclamped` exists so both regimes can be produced from one tool. The single-language
+view is the same result in miniature (`--langs en --unclamped`):
+
+| metric | committed | `--unclamped` | default |
+|---|---|---|---|
+| `accuracy` | 0.82 | 0.82 | 0.82 |
+| `macro_f1` | 0.7876 | 0.7876 | 0.7876 |
+| `ece` | 0.1789 | **0.1789** | 0.1382 |
+| `mean_confidence` | 0.9989 | **0.9989** | 0.9582 |
+| `acc_at_50_coverage` | 0.94 | **0.94** | 0.98 |
+
+### The multilingual checkpoint no longer matches its committed row
+
+45 of 51 multilingual accuracies differ, so this is not a plumbing accident here —
+the same code reproduces english 51/51. Most of the movement is upward
+(`bn` 0.29→0.45, `kn` 0.15→0.30, `fa` 0.39→0.51), a few downward (`sv` 0.57→0.49).
+`macro_ece` barely moves (0.3869→0.3911), consistent with the multilingual checkpoint
+having an empty `temperature_by_options`, so the clamp cannot explain it.
+
+Ruled out: the option sets (identical digest to the english run), the weights
+(bundled and standalone multilingual are byte-identical, all 170 tensors
+`torch.equal`), the dataset (revision `940fd47a`, last modified 2026-02-24), and
+`build_sequence` (unchanged since `v0.2.0`). It is in the multilingual inference path
+between `laya 0.2.0` and `0.3.6` and is **not** reconciled. Flagged rather than hidden.
+
+Related: **`head_max_len` is load-bearing for accuracy**, not just for option
+truncation. The english checkpoint at its shipped `head_max_len=192` scores 0.82;
+forcing 256 or 512 drops it to 0.79.
+
+## Tests
+
+`research/eval/test_laya_eval.py` covers the pure functions and runs offline — no
+checkpoint, no network:
+
+```bash
+python research/eval/test_laya_eval.py # 47 passed, 0 failed
+```
+
+It pins the upstream constants (seed 13, 20 options, the exact instruction string),
+the determinism of the sampler, that a fresh RNG per language is used, and the
+metric arithmetic, including the `confidence == 0.0` bin boundary that this harness
+shares with `bench_local.py` and `laya.common.ece_score`.
+
+## Limits
+
+* MASSIVE intent only. The same shape applies to `scenario` and to XNLI, but neither
+ is wired up here.
+* `per_lang=100` is the published setting, not a statistical one. Per-language ECE on
+ 100 cases is noisy; raise `--per-lang` and say so when quoting a number.
+* The English checkpoint collapses on non-Latin scripts (see `BENCHMARKS.md`), so a
+ low score in one language is not by itself evidence of a misroute — check
+ `laya.lang.analyse` for the script before concluding which checkpoint was used.
+* `confidence == 0.0` is not binned, matching `laya.common.ece_score` and
+ `bench_local.py`. If [#39](https://github.com/NandhaKishorM/laya/pull/39) lands
+ with a different boundary, this should follow it.
diff --git a/research/eval/__init__.py b/research/eval/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/research/eval/laya_eval.py b/research/eval/laya_eval.py
new file mode 100644
index 00000000..1b7d5b30
--- /dev/null
+++ b/research/eval/laya_eval.py
@@ -0,0 +1,370 @@
+"""laya-eval: an independent, reproducible evaluation harness for Laya checkpoints.
+
+Why this exists
+---------------
+The repository's own benchmark scripts are research code: they load everything,
+run all parts, and print tables. There is no small, reproducible harness a third
+party can point at a Laya checkpoint to get a per-language accuracy and ECE
+report plus machine-readable per-case output. This is that harness.
+
+Provenance
+----------
+The prompt format, sampling and metrics follow ``research/scripts/bench_local.py``
+so that results are comparable with the published tables:
+
+* dataset ``mteb/amazon_massive_intent``, split ``test``
+* sampling first ``--per-lang`` rows, ``random.Random(SEED)`` fresh per language
+* options ``--n-opts`` labels: the gold one plus ``rng.sample`` of the rest
+* prompt "What is the user asking for in `utterance`?"
+* rendering option keys with ``_`` -> `` `` and ``.`` -> ``: ``
+* metrics accuracy, macro-F1, ECE (15 bins) on the temperature-scaled softmax
+
+Verified against ``research/results/cpu_51_language_sweep.json``: with the raw
+pre-clamp temperatures, ``en`` reproduces n=100 / accuracy 0.82 / macro_f1 0.7876
+/ ece 0.1789 / mean_confidence 0.9989 exactly.
+
+Usage
+-----
+ python -m laya_eval --model convaiinnovations/laya --langs en,de,ro
+ python -m laya_eval --model ./local-checkpoint --langs all --out report.json
+ python -m laya_eval --model convaiinnovations/laya --subfolder multilingual --langs all
+
+Output is a JSON document with a ``config`` block, a per-language ``report`` and
+a ``cases`` list carrying every individual decision, so a number in the report can
+be re-derived without re-running the model.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import random
+import sys
+import time
+from typing import Any, Dict, List, Optional, Sequence, Tuple
+
+# Running this file directly puts research/eval/ on sys.path, not the repo root, so
+# `import laya` would fail. Allow both `python research/eval/laya_eval.py` and
+# `python -m research.eval.laya_eval` to find the package.
+_REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+if _REPO_ROOT not in sys.path:
+ sys.path.insert(0, _REPO_ROOT)
+
+SEED = 13
+N_OPTS = 20
+PER_LANG = 100
+INSTRUCTIONS = "What is the user asking for in `utterance`?"
+DATASET = "mteb/amazon_massive_intent"
+ECE_BINS = 15
+
+
+# --------------------------------------------------------------------- sampling
+def render_label(key: str) -> str:
+ """Option text for one intent label, matching bench_local.py:176."""
+ return key.replace("_", " ").replace(".", ": ")
+
+
+def build_case(text: str, gold: str, pool: Sequence[str], rng: random.Random,
+ n_opts: int) -> Tuple[Dict[str, Any], Dict[str, Any], int]:
+ """One case: a state, a 20-option choice question, and the gold option index."""
+ keys = [gold] + rng.sample(list(pool), min(n_opts - 1, len(pool)))
+ rng.shuffle(keys)
+ state = {"utterance": text}
+ question = {
+ "intent": {
+ "type": "choice",
+ "instructions": INSTRUCTIONS,
+ "criteria": {k: render_label(k) for k in keys},
+ }
+ }
+ return state, question, keys.index(gold)
+
+
+def build_suite(rows: Sequence[Dict[str, Any]], labels: Sequence[str],
+ per_lang: int, n_opts: int, seed: int = SEED):
+ """Deterministic suite for one language. Returns (cases, gold, option_keys)."""
+ labels = sorted(labels)
+ rng = random.Random(seed) # fresh per language, as upstream
+ cases, gold, option_keys = [], [], []
+ for row in list(rows)[:per_lang]:
+ pool = [x for x in labels if x != row["label_text"]]
+ state, question, gold_idx = build_case(row["text"], row["label_text"],
+ pool, rng, n_opts)
+ keys = list(question["intent"]["criteria"])
+ cases.append((state, question))
+ gold.append(gold_idx)
+ option_keys.append(keys)
+ return cases, gold, option_keys
+
+
+def load_language(lang: str, split: str = "test"):
+ """Load one language config. Raises with a readable message if unavailable."""
+ from datasets import load_dataset
+ try:
+ ds = load_dataset(DATASET, lang, split=split)
+ except Exception as exc: # pragma: no cover - network path
+ raise RuntimeError(
+ "could not load %s config %r: %s" % (DATASET, lang, exc)
+ ) from exc
+ return [{"text": r["text"], "label_text": r["label_text"]} for r in ds]
+
+
+def available_languages() -> List[str]:
+ """Language configs the dataset exposes, excluding the aggregate 'default'."""
+ from datasets import get_dataset_config_names
+ names = get_dataset_config_names(DATASET)
+ return sorted(n for n in names if n != "default")
+
+
+# ---------------------------------------------------------------------- scoring
+def score_cases(agent, cases) -> List[Any]:
+ """Raw marker logits per case, from one collated forward pass.
+
+ Mirrors bench_local.py:54-97: build every sequence, collate, run the model
+ once, and slice each row to its own marker count. No temperature is applied
+ here so the caller can score under more than one regime from one pass.
+ """
+ import torch
+ from laya.common import QTYPES, build_sequence, collate_items, render_options
+
+ max_len = agent.cfg.get("max_len", 512)
+ head_max_len = agent.cfg.get("head_max_len", 192)
+ items = []
+ for state, questions in cases:
+ for _qid, qdef in questions.items():
+ q = {"t": qdef["type"],
+ "ins": qdef["instructions"],
+ "crit": qdef.get("criteria")}
+ ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)
+ if len(markers) != len(render_options(q)):
+ raise ValueError("marker/option count mismatch; question exceeds head_max_len")
+ items.append({"ids": ids, "markers": markers, "qtype": QTYPES[q["t"]]})
+ batch = collate_items([items], agent.tok.pad_token_id)
+ with torch.no_grad():
+ logits, _ = agent.model(
+ batch["input_ids"].to(agent.device),
+ batch["attention_mask"].to(agent.device),
+ batch["marker_pos"].to(agent.device),
+ batch["marker_mask"].to(agent.device),
+ batch["qtype"].to(agent.device),
+ )
+ logits = logits.float().cpu().numpy()
+ return [logits[i, :len(it["markers"])] for i, it in enumerate(items)]
+
+
+def softmax_t(z, temperature: float = 1.0):
+ import numpy as np
+ z = np.asarray(z, dtype=float) / max(1e-3, float(temperature))
+ e = np.exp(z - z.max())
+ return e / e.sum()
+
+
+def temperature_for(agent, qtype: int, k: int, unclamped: bool = False) -> float:
+ """The temperature to score this bucket with.
+
+ By default this is what ``Agent`` applies, i.e. the checkpoint's bucket clamped
+ to ``[TEMP_MIN, TEMP_MAX]``. With ``unclamped=True`` it is the checkpoint's raw
+ value, which is what the committed pre-#42 sweep was produced with.
+ """
+ from laya.common import temp_bucket
+ bucket = temp_bucket(qtype, k)
+ if unclamped:
+ return float(agent.temperature_by_options_raw.get(
+ bucket, agent.temperature_raw[qtype]))
+ return float(agent.temperature_by_options.get(bucket, agent.temperature[qtype]))
+
+
+def ece(confidence, correct, bins: int = ECE_BINS) -> float:
+ """Expected calibration error over equal-width confidence bins."""
+ import numpy as np
+ confidence = np.asarray(confidence, dtype=float)
+ correct = np.asarray(correct, dtype=float)
+ if not len(confidence):
+ return float("nan")
+ total, edges = 0.0, np.linspace(0.0, 1.0, bins + 1)
+ for lo, hi in zip(edges[:-1], edges[1:]):
+ sel = (confidence > lo) & (confidence <= hi)
+ if sel.any():
+ total += sel.mean() * abs(confidence[sel].mean() - correct[sel].mean())
+ return float(total)
+
+
+def macro_f1(gold, pred) -> float:
+ import numpy as np
+ gold, pred = np.asarray(gold), np.asarray(pred)
+ scores = []
+ for cls in sorted(set(gold.tolist()) | set(pred.tolist())):
+ tp = int(((pred == cls) & (gold == cls)).sum())
+ fp = int(((pred == cls) & (gold != cls)).sum())
+ fn = int(((pred != cls) & (gold == cls)).sum())
+ scores.append(2 * tp / max(1, 2 * tp + fp + fn))
+ return float(np.mean(scores)) if scores else float("nan")
+
+
+def summarise(confidences, corrects, golds, preds) -> Dict[str, float]:
+ """The metric block, matching bench_local.py:132-144."""
+ import numpy as np
+ confidences = np.asarray(confidences, dtype=float)
+ corrects = np.asarray(corrects, dtype=float)
+ n = len(confidences)
+ if not n:
+ return {"n": 0}
+ half = max(1, n // 2)
+ order = np.argsort(-confidences)[:half]
+ return {
+ "n": n,
+ "accuracy": round(float(corrects.mean()), 4),
+ "macro_f1": round(macro_f1(golds, preds), 4),
+ "ece": round(ece(confidences, corrects), 4),
+ "mean_confidence": round(float(confidences.mean()), 4),
+ "acc_at_50_coverage": round(float(corrects[order].mean()), 4),
+ }
+
+
+# ------------------------------------------------------------------------ runner
+def run_language(agent, lang: str, per_lang: int, n_opts: int, seed: int = SEED,
+ unclamped: bool = False) -> Dict[str, Any]:
+ """Evaluate one language and return its report plus per-case records."""
+ import numpy as np
+ from laya.common import QTYPES
+
+ rows = load_language(lang)
+ cases, gold, option_keys = build_suite(
+ rows, sorted({r["label_text"] for r in rows}), per_lang, n_opts, seed)
+ logits = score_cases(agent, cases)
+
+ confidences, corrects, preds, records = [], [], [], []
+ for i, z in enumerate(logits):
+ k = len(z)
+ temperature = temperature_for(agent, QTYPES["choice"], k, unclamped)
+ probs = softmax_t(z, temperature)
+ pred = int(np.argmax(probs))
+ correct = int(pred == gold[i])
+ confidences.append(float(probs.max()))
+ corrects.append(float(correct))
+ preds.append(pred)
+ records.append({
+ "lang": lang,
+ "index": i,
+ "state": cases[i][0],
+ "instructions": INSTRUCTIONS,
+ "options": option_keys[i],
+ "gold_index": int(gold[i]),
+ "gold_label": option_keys[i][gold[i]],
+ "pred_index": pred,
+ "pred_label": option_keys[i][pred],
+ "probability": round(float(probs[pred]), 6),
+ "p_gold": round(float(probs[gold[i]]), 6),
+ "confidence": round(float(probs.max()), 6),
+ "correct": correct,
+ "temperature": round(float(temperature), 6),
+ })
+
+ report = summarise(confidences, corrects, gold, preds)
+ report["temperature"] = round(
+ float(temperature_for(agent, QTYPES["choice"], n_opts, unclamped)), 6)
+ return {"report": report, "cases": records}
+
+
+def main(argv: Optional[Sequence[str]] = None) -> int:
+ parser = argparse.ArgumentParser(
+ prog="laya-eval",
+ description="Per-language accuracy and calibration report for a Laya checkpoint.")
+ parser.add_argument("--model", default="convaiinnovations/laya",
+ help="checkpoint repo id or local path")
+ parser.add_argument("--subfolder", default=None,
+ help="checkpoint subfolder, e.g. multilingual")
+ parser.add_argument("--device", default=None, help="cpu, cuda, mps (default: auto)")
+ parser.add_argument("--langs", default="en",
+ help="comma-separated configs, or 'all'")
+ parser.add_argument("--per-lang", type=int, default=PER_LANG)
+ parser.add_argument("--n-opts", type=int, default=N_OPTS)
+ parser.add_argument("--seed", type=int, default=SEED)
+ parser.add_argument("--out", default=None, help="write the JSON report here")
+ parser.add_argument("--no-cases", action="store_true",
+ help="omit per-case records (smaller file)")
+ parser.add_argument("--unclamped", action="store_true",
+ help="score with the checkpoint's RAW bucket temperatures instead "
+ "of the clamped ones Agent applies. This is what reproduces "
+ "the committed pre-#42 sweep")
+ args = parser.parse_args(argv)
+
+ if args.langs.strip().lower() == "all":
+ try:
+ langs = available_languages()
+ except Exception as exc:
+ print("could not list dataset configs: %s" % exc, file=sys.stderr)
+ return 2
+ else:
+ langs = [x.strip() for x in args.langs.split(",") if x.strip()]
+ if not langs:
+ print("no languages selected", file=sys.stderr)
+ return 2
+
+ import laya
+
+ started = time.time()
+ agent = laya.load(args.model, device=args.device, subfolder=args.subfolder)
+ agent.model.eval()
+ payload: Dict[str, Any] = {
+ "config": {
+ "model": args.model,
+ "subfolder": args.subfolder,
+ "device": str(agent.device),
+ "max_len": agent.cfg.get("max_len"),
+ "head_max_len": agent.cfg.get("head_max_len"),
+ "dataset": DATASET,
+ "split": "test",
+ "per_lang": args.per_lang,
+ "n_opts": args.n_opts,
+ "seed": args.seed,
+ "instructions": INSTRUCTIONS,
+ "temperatures": dict(agent.temperature_by_options_raw) if args.unclamped
+ else dict(agent.temperature_by_options),
+ "laya_version": getattr(laya, "__version__", "unknown"),
+ },
+ "report": {},
+ "cases": [],
+ }
+
+ for lang in langs:
+ t0 = time.time()
+ try:
+ out = run_language(agent, lang, args.per_lang, args.n_opts,
+ args.seed, args.unclamped)
+ except Exception as exc:
+ print(" %-8s FAILED: %s" % (lang, str(exc)[:110]), file=sys.stderr)
+ payload["report"][lang] = {"error": str(exc)[:200]}
+ continue
+ payload["report"][lang] = out["report"]
+ if not args.no_cases:
+ payload["cases"].extend(out["cases"])
+ r = out["report"]
+ print(" %-8s n=%-4d acc=%.4f macro_f1=%.4f ece=%.4f conf=%.4f (%.1fs)"
+ % (lang, r.get("n", 0), r.get("accuracy", float("nan")),
+ r.get("macro_f1", float("nan")), r.get("ece", float("nan")),
+ r.get("mean_confidence", float("nan")), time.time() - t0),
+ flush=True)
+
+ scored = {k: v for k, v in payload["report"].items() if "error" not in v and v.get("n")}
+ if scored:
+ payload["summary"] = {
+ "languages": len(scored),
+ "macro_accuracy": round(sum(v["accuracy"] for v in scored.values()) / len(scored), 4),
+ "macro_ece": round(sum(v["ece"] for v in scored.values()) / len(scored), 4),
+ "macro_f1": round(sum(v["macro_f1"] for v in scored.values()) / len(scored), 4),
+ "seconds": round(time.time() - started, 1),
+ }
+ print("\n macro over %d languages: acc=%.4f ece=%.4f f1=%.4f"
+ % (len(scored), payload["summary"]["macro_accuracy"],
+ payload["summary"]["macro_ece"], payload["summary"]["macro_f1"]))
+
+ if args.out:
+ with open(args.out, "w", encoding="utf-8") as fh:
+ json.dump(payload, fh, ensure_ascii=False, indent=1)
+ print(" wrote %s" % args.out)
+ return 0
+
+
+if __name__ == "__main__": # pragma: no cover
+ raise SystemExit(main())
diff --git a/research/eval/test_laya_eval.py b/research/eval/test_laya_eval.py
new file mode 100644
index 00000000..d7b74bf6
--- /dev/null
+++ b/research/eval/test_laya_eval.py
@@ -0,0 +1,162 @@
+"""Offline tests for the layered evaluation harness.
+
+No checkpoint and no network: every function under test is pure, so this runs in
+CI next to the other suites. The model-dependent paths (`score_cases`,
+`run_language`) are exercised by `tests/test_local_e2e.py` when a checkpoint is
+present.
+
+Run: python research/eval/test_laya_eval.py
+"""
+import os
+import random
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
+
+from research.eval.laya_eval import ( # noqa: E402
+ ECE_BINS, INSTRUCTIONS, N_OPTS, SEED, build_case, build_suite, ece, macro_f1,
+ render_label, summarise, temperature_for,
+)
+
+PASS, FAIL = [], []
+
+
+def check(name, got, want):
+ if got == want:
+ PASS.append(name)
+ else:
+ FAIL.append("%s:\n got %r\n want %r" % (name, got, want))
+
+
+def check_true(name, cond, detail=""):
+ if cond:
+ PASS.append(name)
+ else:
+ FAIL.append("%s %s" % (name, detail))
+
+
+# --------------------------------------------------------------- label rendering
+check("render/underscores become spaces", render_label("alarm_set"), "alarm set")
+check("render/dots become colon-space", render_label("news.»"), "news: »".replace("»", "»"))
+check("render/query_definition", render_label("qa_definition"), "qa definition")
+check("render/no underscores left", "_" in render_label("a_b_c"), False)
+# must match bench_local.py:176 exactly
+check("render/upstream parity", render_label("datetime_query"), "datetime query")
+
+
+# ------------------------------------------------------------------ one case
+_rng = random.Random(SEED)
+_state, _q, _gold_idx = build_case("wake me up at five am", "alarm_set",
+ ["a", "b", "c"], _rng, 4)
+check("case/state key is utterance", list(_state), ["utterance"])
+check("case/single question id", list(_q), ["intent"])
+check("case/type is choice", _q["intent"]["type"], "choice")
+check("case/instructions fixed", _q["intent"]["instructions"], INSTRUCTIONS)
+check("case/option count", len(_q["intent"]["criteria"]), 4)
+check("case/gold is present", "alarm_set" in _q["intent"]["criteria"], True)
+check("case/gold index points at gold",
+ list(_q["intent"]["criteria"])[_gold_idx], "alarm_set")
+
+
+# ------------------------------------------------- determinism and seed parity
+ROWS = [{"text": "t%d" % i, "label_text": "lab%d" % (i % 7)} for i in range(30)]
+LABELS = sorted({r["label_text"] for r in ROWS})
+
+a, b, c = build_suite(ROWS, LABELS, 10, 5, SEED)
+d, e, f = build_suite(ROWS, LABELS, 10, 5, SEED)
+check("suite/same seed same cases", a == d, True)
+check("suite/same seed same gold", b == e, True)
+check("suite/same seed same options", c == f, True)
+
+g, h, i = build_suite(ROWS, LABELS, 10, 5, SEED + 1)
+check_true("suite/different seed different options", c != i)
+
+# a fresh Random(SEED) per language is what upstream does; two languages with the
+# same row content must therefore produce identical option sets
+ROWS_B = [dict(r) for r in ROWS]
+_b1 = build_suite(ROWS, LABELS, 10, 5, SEED)[2]
+_b2 = build_suite(ROWS_B, LABELS, 10, 5, SEED)[2]
+check("suite/rng reset per language", _b1, _b2)
+
+check("suite/per_lang caps cases", len(a), 10)
+check("suite/gold within options",
+ all(0 <= gi < len(opts) for gi, opts in zip(b, c)), True)
+check("suite/every gold is in its option set",
+ all(opts[gi] == ROWS[i]["label_text"] for i, (gi, opts) in enumerate(zip(b, c))), True)
+check("suite/options are distinct",
+ all(len(set(o)) == len(o) for o in c), True)
+check("suite/n_opts respected", all(len(o) == 5 for o in c), True)
+
+# more options requested than labels available: must not crash or duplicate
+tiny = [{"text": "x", "label_text": "only"}]
+_tc, _tg, _to = build_suite(tiny, ["only"], 1, 20, SEED)
+check("suite/fewer labels than n_opts", len(_to[0]), 1)
+
+
+# --------------------------------------------------------------------- metrics
+import math # noqa: E402
+
+check_true("ece/empty is nan", math.isnan(ece([], [])))
+check("ece/perfect on one bin", round(ece([0.95] * 50, [1.0] * 50), 4), 0.05)
+check("ece/all wrong and confident", round(ece([0.95] * 50, [0.0] * 50), 4), 0.95)
+check("ece/normalised by count", round(ece([0.9] * 100, [1.0] * 100), 4),
+ round(ece([0.9] * 1000, [1.0] * 1000), 4))
+check("ece/one sample", round(ece([1.0], [1.0]), 4), 0.0)
+
+# Known boundary: the bin test is `conf > lo`, so conf == 0.0 falls in no bin and
+# contributes nothing. `laya.common.ece_score` and `research/scripts/bench_local.py`
+# bin identically, so this harness deliberately matches them rather than diverging --
+# comparability with the published tables is the point. PR #39 addresses the same
+# boundary in `laya.common`; if it lands, this harness should follow it.
+check("ece/conf==0.0 is not binned (matches upstream)",
+ round(ece([0.0, 0.0], [1.0, 1.0]), 4), 0.0)
+check_true("ece/conf slightly above 0 IS binned",
+ ece([1e-9, 1e-9], [1.0, 1.0]) > 0.0)
+
+check_true("f1/empty is nan", math.isnan(macro_f1([], [])))
+check("f1/perfect", macro_f1([0, 1, 2], [0, 1, 2]), 1.0)
+check("f1/all wrong", round(macro_f1([0, 1], [1, 0]), 4), 0.0)
+check("f1/disjoint labels", round(macro_f1([0], [1]), 4), 0.0)
+
+_s = summarise([0.9, 0.8, 0.7, 0.6], [1.0, 1.0, 0.0, 0.0], [0, 1, 2, 3], [0, 1, 3, 2])
+check("summary/n", _s["n"], 4)
+check("summary/accuracy", _s["accuracy"], 0.5)
+check("summary/mean_confidence", _s["mean_confidence"], 0.75)
+check("summary/acc_at_50_coverage takes the confident half",
+ _s["acc_at_50_coverage"], 1.0)
+check("summary/empty", summarise([], [], [], []), {"n": 0})
+
+
+# ------------------------------------------------------- temperature selection
+class _FakeAgent:
+ temperature_by_options = {"choice:11+": 0.5}
+ temperature = [1.0, 1.0, 1.0]
+ temperature_by_options_raw = {"choice:11+": 0.10058280825614929}
+ temperature_raw = [1.0, 1.0, 1.0]
+
+
+_fa = _FakeAgent()
+from laya.common import QTYPES # noqa: E402
+
+check("temp/clamped path is used by default",
+ temperature_for(_fa, QTYPES["choice"], 20), 0.5)
+check("temp/raw path under unclamped",
+ round(temperature_for(_fa, QTYPES["choice"], 20, unclamped=True), 6), 0.100583)
+check("temp/falls back to the per-type list",
+ temperature_for(_fa, QTYPES["choice"], 3), 1.0)
+check("temp/raw falls back too",
+ temperature_for(_fa, QTYPES["choice"], 3, unclamped=True), 1.0)
+
+
+# ------------------------------------------------------------------- constants
+check("const/seed matches upstream", SEED, 13)
+check("const/n_opts matches upstream", N_OPTS, 20)
+check("const/bins", ECE_BINS, 15)
+check("const/instructions match bench_local.py",
+ INSTRUCTIONS, "What is the user asking for in `utterance`?")
+
+
+print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
+for f in FAIL:
+ print(" FAIL " + f)
+sys.exit(1 if FAIL else 0)
diff --git a/research/results/eval_english_51_languages.json b/research/results/eval_english_51_languages.json
new file mode 100644
index 00000000..06f9b784
--- /dev/null
+++ b/research/results/eval_english_51_languages.json
@@ -0,0 +1,493 @@
+{
+ "cases_note": "per-case records omitted from this file to keep the diff reviewable; regenerate with research/eval/laya_eval.py --langs all --out <path>",
+ "config": {
+ "dataset": "mteb/amazon_massive_intent",
+ "device": "cpu",
+ "head_max_len": 192,
+ "instructions": "What is the user asking for in `utterance`?",
+ "laya_version": "0.3.6",
+ "max_len": 512,
+ "model": "convaiinnovations/laya",
+ "n_opts": 20,
+ "per_lang": 100,
+ "seed": 13,
+ "split": "test",
+ "subfolder": null,
+ "temperatures": {
+ "choice:11+": 0.5,
+ "choice:2": 1.9063563346862793,
+ "choice:3-5": 1.7601518630981445,
+ "choice:6-10": 1.0000158548355103,
+ "noul:2": 1.983399510383606,
+ "score:3-5": 1.2514300346374512
+ }
+ },
+ "report": {
+ "af": {
+ "acc_at_50_coverage": 0.42,
+ "accuracy": 0.29,
+ "ece": 0.5588,
+ "macro_f1": 0.2874,
+ "mean_confidence": 0.8488,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "am": {
+ "acc_at_50_coverage": 0.12,
+ "accuracy": 0.12,
+ "ece": 0.6804,
+ "macro_f1": 0.1122,
+ "mean_confidence": 0.7873,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "ar": {
+ "acc_at_50_coverage": 0.12,
+ "accuracy": 0.11,
+ "ece": 0.5795,
+ "macro_f1": 0.093,
+ "mean_confidence": 0.6833,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "az": {
+ "acc_at_50_coverage": 0.14,
+ "accuracy": 0.1,
+ "ece": 0.5983,
+ "macro_f1": 0.0629,
+ "mean_confidence": 0.6983,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "bn": {
+ "acc_at_50_coverage": 0.12,
+ "accuracy": 0.08,
+ "ece": 0.6267,
+ "macro_f1": 0.056,
+ "mean_confidence": 0.7067,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "cy": {
+ "acc_at_50_coverage": 0.14,
+ "accuracy": 0.12,
+ "ece": 0.6452,
+ "macro_f1": 0.0906,
+ "mean_confidence": 0.7652,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "da": {
+ "acc_at_50_coverage": 0.52,
+ "accuracy": 0.35,
+ "ece": 0.526,
+ "macro_f1": 0.3082,
+ "mean_confidence": 0.876,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "de": {
+ "acc_at_50_coverage": 0.68,
+ "accuracy": 0.42,
+ "ece": 0.452,
+ "macro_f1": 0.387,
+ "mean_confidence": 0.872,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "el": {
+ "acc_at_50_coverage": 0.2,
+ "accuracy": 0.13,
+ "ece": 0.6789,
+ "macro_f1": 0.1,
+ "mean_confidence": 0.8089,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "en": {
+ "acc_at_50_coverage": 0.98,
+ "accuracy": 0.82,
+ "ece": 0.1382,
+ "macro_f1": 0.7876,
+ "mean_confidence": 0.9582,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "es": {
+ "acc_at_50_coverage": 0.62,
+ "accuracy": 0.51,
+ "ece": 0.4225,
+ "macro_f1": 0.4767,
+ "mean_confidence": 0.9245,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "fa": {
+ "acc_at_50_coverage": 0.16,
+ "accuracy": 0.14,
+ "ece": 0.6331,
+ "macro_f1": 0.1197,
+ "mean_confidence": 0.7731,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "fi": {
+ "acc_at_50_coverage": 0.16,
+ "accuracy": 0.13,
+ "ece": 0.7189,
+ "macro_f1": 0.1213,
+ "mean_confidence": 0.8489,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "fr": {
+ "acc_at_50_coverage": 0.78,
+ "accuracy": 0.59,
+ "ece": 0.2995,
+ "macro_f1": 0.5822,
+ "mean_confidence": 0.8895,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "he": {
+ "acc_at_50_coverage": 0.02,
+ "accuracy": 0.06,
+ "ece": 0.7379,
+ "macro_f1": 0.039,
+ "mean_confidence": 0.7842,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "hi": {
+ "acc_at_50_coverage": 0.08,
+ "accuracy": 0.1,
+ "ece": 0.6416,
+ "macro_f1": 0.0758,
+ "mean_confidence": 0.7416,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "hu": {
+ "acc_at_50_coverage": 0.06,
+ "accuracy": 0.09,
+ "ece": 0.6783,
+ "macro_f1": 0.0813,
+ "mean_confidence": 0.7683,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "hy": {
+ "acc_at_50_coverage": 0.04,
+ "accuracy": 0.05,
+ "ece": 0.5709,
+ "macro_f1": 0.0355,
+ "mean_confidence": 0.6209,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "id": {
+ "acc_at_50_coverage": 0.48,
+ "accuracy": 0.36,
+ "ece": 0.487,
+ "macro_f1": 0.3283,
+ "mean_confidence": 0.8304,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "is": {
+ "acc_at_50_coverage": 0.16,
+ "accuracy": 0.11,
+ "ece": 0.6641,
+ "macro_f1": 0.0893,
+ "mean_confidence": 0.7741,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "it": {
+ "acc_at_50_coverage": 0.52,
+ "accuracy": 0.34,
+ "ece": 0.5156,
+ "macro_f1": 0.3101,
+ "mean_confidence": 0.8534,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "ja": {
+ "acc_at_50_coverage": 0.76,
+ "accuracy": 0.53,
+ "ece": 0.3657,
+ "macro_f1": 0.4942,
+ "mean_confidence": 0.8957,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "jv": {
+ "acc_at_50_coverage": 0.16,
+ "accuracy": 0.16,
+ "ece": 0.67,
+ "macro_f1": 0.1537,
+ "mean_confidence": 0.83,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "ka": {
+ "acc_at_50_coverage": 0.1,
+ "accuracy": 0.09,
+ "ece": 0.6102,
+ "macro_f1": 0.0742,
+ "mean_confidence": 0.7002,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "km": {
+ "acc_at_50_coverage": 0.0,
+ "accuracy": 0.0,
+ "ece": 0.7051,
+ "macro_f1": 0.0,
+ "mean_confidence": 0.7051,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "kn": {
+ "acc_at_50_coverage": 0.08,
+ "accuracy": 0.11,
+ "ece": 0.5903,
+ "macro_f1": 0.0913,
+ "mean_confidence": 0.695,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "ko": {
+ "acc_at_50_coverage": 0.14,
+ "accuracy": 0.11,
+ "ece": 0.6804,
+ "macro_f1": 0.0943,
+ "mean_confidence": 0.7904,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "lv": {
+ "acc_at_50_coverage": 0.1,
+ "accuracy": 0.1,
+ "ece": 0.6659,
+ "macro_f1": 0.0786,
+ "mean_confidence": 0.7659,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "ml": {
+ "acc_at_50_coverage": 0.1,
+ "accuracy": 0.07,
+ "ece": 0.601,
+ "macro_f1": 0.0503,
+ "mean_confidence": 0.671,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "mn": {
+ "acc_at_50_coverage": 0.14,
+ "accuracy": 0.13,
+ "ece": 0.6704,
+ "macro_f1": 0.1213,
+ "mean_confidence": 0.7977,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "ms": {
+ "acc_at_50_coverage": 0.4,
+ "accuracy": 0.27,
+ "ece": 0.5064,
+ "macro_f1": 0.2372,
+ "mean_confidence": 0.7764,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "my": {
+ "acc_at_50_coverage": 0.08,
+ "accuracy": 0.06,
+ "ece": 0.5797,
+ "macro_f1": 0.0455,
+ "mean_confidence": 0.6397,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "nb": {
+ "acc_at_50_coverage": 0.52,
+ "accuracy": 0.33,
+ "ece": 0.5179,
+ "macro_f1": 0.3025,
+ "mean_confidence": 0.8479,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "nl": {
+ "acc_at_50_coverage": 0.56,
+ "accuracy": 0.39,
+ "ece": 0.5025,
+ "macro_f1": 0.3968,
+ "mean_confidence": 0.8831,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "pl": {
+ "acc_at_50_coverage": 0.36,
+ "accuracy": 0.24,
+ "ece": 0.5695,
+ "macro_f1": 0.2033,
+ "mean_confidence": 0.7951,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "pt": {
+ "acc_at_50_coverage": 0.66,
+ "accuracy": 0.47,
+ "ece": 0.4317,
+ "macro_f1": 0.4213,
+ "mean_confidence": 0.8894,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "ro": {
+ "acc_at_50_coverage": 0.48,
+ "accuracy": 0.33,
+ "ece": 0.5499,
+ "macro_f1": 0.2976,
+ "mean_confidence": 0.8668,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "ru": {
+ "acc_at_50_coverage": 0.42,
+ "accuracy": 0.31,
+ "ece": 0.5946,
+ "macro_f1": 0.3084,
+ "mean_confidence": 0.9046,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "sl": {
+ "acc_at_50_coverage": 0.26,
+ "accuracy": 0.2,
+ "ece": 0.5994,
+ "macro_f1": 0.1679,
+ "mean_confidence": 0.7994,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "sq": {
+ "acc_at_50_coverage": 0.28,
+ "accuracy": 0.21,
+ "ece": 0.6031,
+ "macro_f1": 0.1667,
+ "mean_confidence": 0.8131,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "sv": {
+ "acc_at_50_coverage": 0.5,
+ "accuracy": 0.38,
+ "ece": 0.4874,
+ "macro_f1": 0.3809,
+ "mean_confidence": 0.8501,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "sw": {
+ "acc_at_50_coverage": 0.14,
+ "accuracy": 0.13,
+ "ece": 0.6318,
+ "macro_f1": 0.1149,
+ "mean_confidence": 0.7511,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "ta": {
+ "acc_at_50_coverage": 0.12,
+ "accuracy": 0.12,
+ "ece": 0.5541,
+ "macro_f1": 0.0971,
+ "mean_confidence": 0.6741,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "te": {
+ "acc_at_50_coverage": 0.06,
+ "accuracy": 0.09,
+ "ece": 0.6689,
+ "macro_f1": 0.0722,
+ "mean_confidence": 0.7568,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "th": {
+ "acc_at_50_coverage": 0.1,
+ "accuracy": 0.08,
+ "ece": 0.7179,
+ "macro_f1": 0.048,
+ "mean_confidence": 0.7979,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "tl": {
+ "acc_at_50_coverage": 0.46,
+ "accuracy": 0.29,
+ "ece": 0.5121,
+ "macro_f1": 0.249,
+ "mean_confidence": 0.8021,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "tr": {
+ "acc_at_50_coverage": 0.2,
+ "accuracy": 0.14,
+ "ece": 0.5874,
+ "macro_f1": 0.0967,
+ "mean_confidence": 0.7274,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "ur": {
+ "acc_at_50_coverage": 0.08,
+ "accuracy": 0.07,
+ "ece": 0.6428,
+ "macro_f1": 0.0645,
+ "mean_confidence": 0.7128,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "vi": {
+ "acc_at_50_coverage": 0.06,
+ "accuracy": 0.06,
+ "ece": 0.69,
+ "macro_f1": 0.0498,
+ "mean_confidence": 0.75,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "zh-CN": {
+ "acc_at_50_coverage": 0.84,
+ "accuracy": 0.62,
+ "ece": 0.3198,
+ "macro_f1": 0.6175,
+ "mean_confidence": 0.9244,
+ "n": 100,
+ "temperature": 0.5
+ },
+ "zh-TW": {
+ "acc_at_50_coverage": 0.72,
+ "accuracy": 0.46,
+ "ece": 0.4344,
+ "macro_f1": 0.4294,
+ "mean_confidence": 0.8944,
+ "n": 100,
+ "temperature": 0.5
+ }
+ },
+ "summary": {
+ "languages": 51,
+ "macro_accuracy": 0.2269,
+ "macro_ece": 0.5709,
+ "macro_f1": 0.2053,
+ "seconds": 2017.8
+ }
+}
\ No newline at end of file