Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(research): add a reproducible per-language evaluation harness (#210)
PerryLink2d2c15ce8896
diff --git a/research/__init__.py b/research/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/research/eval/README.md b/research/eval/README.mdnew file mode 100644index 00000000..d54b3200--- /dev/null+++ b/research/eval/README.md@@ -0,0 +1,172 @@+# laya-eval — a reproducible per-language evaluation harness++An independent harness for measuring a Laya checkpoint: per-language accuracy and+calibration, with machine-readable per-case output.++It exists because the repository's own benchmark scripts are research code. They+download every checkpoint, run every part, and print tables. There was no small,+reproducible harness a third party could point at a checkpoint to answer "how does+this model do on my language, and can I trust its confidence?" — and no per-case+record behind the published numbers, so they could not be re-derived without a GPU+and the original environment.++This addresses the ask in+[#35](https://github.com/NandhaKishorM/laya/issues/35):++> A fixed prompt format plus a per-language ECE report is exactly what the repo+> lacks ... Per-case JSON would be very welcome too.++## Install++Nothing beyond a normal Laya install, plus `datasets`:++```bash+pip install laya datasets+```++The harness is deliberately not part of the `laya` package: it is evaluation code,+it pulls a dataset, and `import laya` should stay dependency-light.++## Use++```bash+# one language+python research/eval/laya_eval.py --model convaiinnovations/laya --langs en++# several, with a JSON report+python research/eval/laya_eval.py --model convaiinnovations/laya \+ --langs en,de,ro --out report.json++# every MASSIVE language+python research/eval/laya_eval.py --model convaiinnovations/laya --langs all --out all.json++# the multilingual checkpoint+python research/eval/laya_eval.py --model convaiinnovations/laya \+ --subfolder multilingual --langs all --out multilingual.json++# a local checkpoint+python research/eval/laya_eval.py --model ./my-finetune --langs en+```++Output, per language:++```+ en n=100 acc=0.8200 macro_f1=0.7876 ece=0.1789 conf=0.9989 (36.7s)++ macro over 51 languages: acc=... ece=... f1=...+```++and a JSON document with four parts:++| key | contents |+|---|---|+| `config` | checkpoint, device, `max_len`, `head_max_len`, dataset, `per_lang`, `n_opts`, seed, the fixed instructions, the temperatures in force, laya version |+| `report` | per language: `n`, `accuracy`, `macro_f1`, `ece`, `mean_confidence`, `acc_at_50_coverage`, `temperature` |+| `summary` | macro accuracy / ECE / macro-F1 over the languages that ran |+| `cases` | every individual decision |++Each case carries `state`, `instructions`, `options`, `gold_index`, `gold_label`,+`pred_index`, `pred_label`, `probability`, `p_gold`, `confidence`, `correct` and the+`temperature` used. That is enough to re-derive every number in `report` from the+file alone, with no model and no network:++```python+import json+d = json.load(open("report.json"))+n = len(d["cases"])+acc = sum(c["correct"] for c in d["cases"]) / n+assert abs(acc - d["report"]["en"]["accuracy"]) < 5e-5+```++## Method++Chosen so results are comparable with the published tables, which is the point of a+second implementation:++| | |+|---|---|+| dataset | `mteb/amazon_massive_intent`, split `test` |+| sampling | first `--per-lang` rows (default 100); `random.Random(13)` created **fresh per language** |+| options | `--n-opts` (default 20): the gold label plus `rng.sample` of the others, then shuffled |+| prompt | `What is the user asking for in \`utterance\`?` |+| option text | label with `_` → space and `.` → `: ` |+| metrics | accuracy, macro-F1, ECE over 15 equal-width confidence bins, mean confidence, accuracy at 50% coverage |+| temperature | the bucket `Agent` would apply, selected by `(question type, option count)` |++`--unclamped` scores with the checkpoint's **raw** bucket temperatures instead of the+clamped ones `Agent` applies. That is what reproduces the committed sweep, and it is+also how the two can be compared.++## Verification++Checked against the committed sweep, not only against itself. Both checkpoints over+**all 51 languages** (`--langs all --per-lang 100 --n-opts 20`), per-language accuracy+compared against `research/results/cpu_51_language_sweep.json`:++| checkpoint | per-language accuracy identical | `macro_accuracy` committed → mine | `macro_ece` committed → mine |+|---|---|---|---|+| **english** | **51 / 51** | 0.2269 → **0.2269** | 0.7331 → 0.5709 |+| multilingual | 6 / 51 | 0.3661 → 0.4008 | 0.3869 → 0.3911 |++The english checkpoint reproduces every per-language accuracy, not just the macro.+Those are deterministic outputs on a fixed sample, so they can only agree if the+sampling, prompt text, option construction and inference path are all identical to+the committed run.++The `macro_ece` gap on english is the temperature clamp — `choice:11+` is `0.1006`+raw and `0.5` as served ([#208](https://github.com/NandhaKishorM/laya/issues/208)).+`--unclamped` exists so both regimes can be produced from one tool. The single-language+view is the same result in miniature (`--langs en --unclamped`):++| metric | committed | `--unclamped` | default |+|---|---|---|---|+| `accuracy` | 0.82 | 0.82 | 0.82 |+| `macro_f1` | 0.7876 | 0.7876 | 0.7876 |+| `ece` | 0.1789 | **0.1789** | 0.1382 |+| `mean_confidence` | 0.9989 | **0.9989** | 0.9582 |+| `acc_at_50_coverage` | 0.94 | **0.94** | 0.98 |++### The multilingual checkpoint no longer matches its committed row++45 of 51 multilingual accuracies differ, so this is not a plumbing accident here —+the same code reproduces english 51/51. Most of the movement is upward+(`bn` 0.29→0.45, `kn` 0.15→0.30, `fa` 0.39→0.51), a few downward (`sv` 0.57→0.49).+`macro_ece` barely moves (0.3869→0.3911), consistent with the multilingual checkpoint+having an empty `temperature_by_options`, so the clamp cannot explain it.++Ruled out: the option sets (identical digest to the english run), the weights+(bundled and standalone multilingual are byte-identical, all 170 tensors+`torch.equal`), the dataset (revision `940fd47a`, last modified 2026-02-24), and+`build_sequence` (unchanged since `v0.2.0`). It is in the multilingual inference path+between `laya 0.2.0` and `0.3.6` and is **not** reconciled. Flagged rather than hidden.++Related: **`head_max_len` is load-bearing for accuracy**, not just for option+truncation. The english checkpoint at its shipped `head_max_len=192` scores 0.82;+forcing 256 or 512 drops it to 0.79.++## Tests++`research/eval/test_laya_eval.py` covers the pure functions and runs offline — no+checkpoint, no network:++```bash+python research/eval/test_laya_eval.py # 47 passed, 0 failed+```++It pins the upstream constants (seed 13, 20 options, the exact instruction string),+the determinism of the sampler, that a fresh RNG per language is used, and the+metric arithmetic, including the `confidence == 0.0` bin boundary that this harness+shares with `bench_local.py` and `laya.common.ece_score`.++## Limits++* MASSIVE intent only. The same shape applies to `scenario` and to XNLI, but neither+ is wired up here.+* `per_lang=100` is the published setting, not a statistical one. Per-language ECE on+ 100 cases is noisy; raise `--per-lang` and say so when quoting a number.+* The English checkpoint collapses on non-Latin scripts (see `BENCHMARKS.md`), so a+ low score in one language is not by itself evidence of a misroute — check+ `laya.lang.analyse` for the script before concluding which checkpoint was used.+* `confidence == 0.0` is not binned, matching `laya.common.ece_score` and+ `bench_local.py`. If [#39](https://github.com/NandhaKishorM/laya/pull/39) lands+ with a different boundary, this should follow it.diff --git a/research/eval/__init__.py b/research/eval/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/research/eval/laya_eval.py b/research/eval/laya_eval.pynew file mode 100644index 00000000..1b7d5b30--- /dev/null+++ b/research/eval/laya_eval.py@@ -0,0 +1,370 @@+"""laya-eval: an independent, reproducible evaluation harness for Laya checkpoints.++Why this exists+---------------+The repository's own benchmark scripts are research code: they load everything,+run all parts, and print tables. There is no small, reproducible harness a third+party can point at a Laya checkpoint to get a per-language accuracy and ECE+report plus machine-readable per-case output. This is that harness.++Provenance+----------+The prompt format, sampling and metrics follow ``research/scripts/bench_local.py``+so that results are comparable with the published tables:++* dataset ``mteb/amazon_massive_intent``, split ``test``+* sampling first ``--per-lang`` rows, ``random.Random(SEED)`` fresh per language+* options ``--n-opts`` labels: the gold one plus ``rng.sample`` of the rest+* prompt "What is the user asking for in `utterance`?"+* rendering option keys with ``_`` -> `` `` and ``.`` -> ``: ``+* metrics accuracy, macro-F1, ECE (15 bins) on the temperature-scaled softmax++Verified against ``research/results/cpu_51_language_sweep.json``: with the raw+pre-clamp temperatures, ``en`` reproduces n=100 / accuracy 0.82 / macro_f1 0.7876+/ ece 0.1789 / mean_confidence 0.9989 exactly.++Usage+-----+ python -m laya_eval --model convaiinnovations/laya --langs en,de,ro+ python -m laya_eval --model ./local-checkpoint --langs all --out report.json+ python -m laya_eval --model convaiinnovations/laya --subfolder multilingual --langs all++Output is a JSON document with a ``config`` block, a per-language ``report`` and+a ``cases`` list carrying every individual decision, so a number in the report can+be re-derived without re-running the model.+"""+from __future__ import annotations++import argparse+import json+import os+import random+import sys+import time+from typing import Any, Dict, List, Optional, Sequence, Tuple++# Running this file directly puts research/eval/ on sys.path, not the repo root, so+# `import laya` would fail. Allow both `python research/eval/laya_eval.py` and+# `python -m research.eval.laya_eval` to find the package.+_REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+if _REPO_ROOT not in sys.path:+ sys.path.insert(0, _REPO_ROOT)++SEED = 13+N_OPTS = 20+PER_LANG = 100+INSTRUCTIONS = "What is the user asking for in `utterance`?"+DATASET = "mteb/amazon_massive_intent"+ECE_BINS = 15+++# --------------------------------------------------------------------- sampling+def render_label(key: str) -> str:+ """Option text for one intent label, matching bench_local.py:176."""+ return key.replace("_", " ").replace(".", ": ")+++def build_case(text: str, gold: str, pool: Sequence[str], rng: random.Random,+ n_opts: int) -> Tuple[Dict[str, Any], Dict[str, Any], int]:+ """One case: a state, a 20-option choice question, and the gold option index."""+ keys = [gold] + rng.sample(list(pool), min(n_opts - 1, len(pool)))+ rng.shuffle(keys)+ state = {"utterance": text}+ question = {+ "intent": {+ "type": "choice",+ "instructions": INSTRUCTIONS,+ "criteria": {k: render_label(k) for k in keys},+ }+ }+ return state, question, keys.index(gold)+++def build_suite(rows: Sequence[Dict[str, Any]], labels: Sequence[str],+ per_lang: int, n_opts: int, seed: int = SEED):+ """Deterministic suite for one language. Returns (cases, gold, option_keys)."""+ labels = sorted(labels)+ rng = random.Random(seed) # fresh per language, as upstream+ cases, gold, option_keys = [], [], []+ for row in list(rows)[:per_lang]:+ pool = [x for x in labels if x != row["label_text"]]+ state, question, gold_idx = build_case(row["text"], row["label_text"],+ pool, rng, n_opts)+ keys = list(question["intent"]["criteria"])+ cases.append((state, question))+ gold.append(gold_idx)+ option_keys.append(keys)+ return cases, gold, option_keys+++def load_language(lang: str, split: str = "test"):+ """Load one language config. Raises with a readable message if unavailable."""+ from datasets import load_dataset+ try:+ ds = load_dataset(DATASET, lang, split=split)+ except Exception as exc: # pragma: no cover - network path+ raise RuntimeError(+ "could not load %s config %r: %s" % (DATASET, lang, exc)+ ) from exc+ return [{"text": r["text"], "label_text": r["label_text"]} for r in ds]+++def available_languages() -> List[str]:+ """Language configs the dataset exposes, excluding the aggregate 'default'."""+ from datasets import get_dataset_config_names+ names = get_dataset_config_names(DATASET)+ return sorted(n for n in names if n != "default")+++# ---------------------------------------------------------------------- scoring+def score_cases(agent, cases) -> List[Any]:+ """Raw marker logits per case, from one collated forward pass.++ Mirrors bench_local.py:54-97: build every sequence, collate, run the model+ once, and slice each row to its own marker count. No temperature is applied+ here so the caller can score under more than one regime from one pass.+ """+ import torch+ from laya.common import QTYPES, build_sequence, collate_items, render_options++ max_len = agent.cfg.get("max_len", 512)+ head_max_len = agent.cfg.get("head_max_len", 192)+ items = []+ for state, questions in cases:+ for _qid, qdef in questions.items():+ q = {"t": qdef["type"],+ "ins": qdef["instructions"],+ "crit": qdef.get("criteria")}+ ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)+ if len(markers) != len(render_options(q)):+ raise ValueError("marker/option count mismatch; question exceeds head_max_len")+ items.append({"ids": ids, "markers": markers, "qtype": QTYPES[q["t"]]})+ batch = collate_items([items], agent.tok.pad_token_id)+ with torch.no_grad():+ logits, _ = agent.model(+ batch["input_ids"].to(agent.device),+ batch["attention_mask"].to(agent.device),+ batch["marker_pos"].to(agent.device),+ batch["marker_mask"].to(agent.device),+ batch["qtype"].to(agent.device),+ )+ logits = logits.float().cpu().numpy()+ return [logits[i, :len(it["markers"])] for i, it in enumerate(items)]+++def softmax_t(z, temperature: float = 1.0):+ import numpy as np+ z = np.asarray(z, dtype=float) / max(1e-3, float(temperature))+ e = np.exp(z - z.max())+ return e / e.sum()+++def temperature_for(agent, qtype: int, k: int, unclamped: bool = False) -> float:+ """The temperature to score this bucket with.++ By default this is what ``Agent`` applies, i.e. the checkpoint's bucket clamped+ to ``[TEMP_MIN, TEMP_MAX]``. With ``unclamped=True`` it is the checkpoint's raw+ value, which is what the committed pre-#42 sweep was produced with.+ """+ from laya.common import temp_bucket+ bucket = temp_bucket(qtype, k)+ if unclamped:+ return float(agent.temperature_by_options_raw.get(+ bucket, agent.temperature_raw[qtype]))+ return float(agent.temperature_by_options.get(bucket, agent.temperature[qtype]))+++def ece(confidence, correct, bins: int = ECE_BINS) -> float:+ """Expected calibration error over equal-width confidence bins."""+ import numpy as np+ confidence = np.asarray(confidence, dtype=float)+ correct = np.asarray(correct, dtype=float)+ if not len(confidence):+ return float("nan")+ total, edges = 0.0, np.linspace(0.0, 1.0, bins + 1)+ for lo, hi in zip(edges[:-1], edges[1:]):+ sel = (confidence > lo) & (confidence <= hi)+ if sel.any():+ total += sel.mean() * abs(confidence[sel].mean() - correct[sel].mean())+ return float(total)+++def macro_f1(gold, pred) -> float:+ import numpy as np+ gold, pred = np.asarray(gold), np.asarray(pred)+ scores = []+ for cls in sorted(set(gold.tolist()) | set(pred.tolist())):+ tp = int(((pred == cls) & (gold == cls)).sum())+ fp = int(((pred == cls) & (gold != cls)).sum())+ fn = int(((pred != cls) & (gold == cls)).sum())+ scores.append(2 * tp / max(1, 2 * tp + fp + fn))+ return float(np.mean(scores)) if scores else float("nan")+++def summarise(confidences, corrects, golds, preds) -> Dict[str, float]:+ """The metric block, matching bench_local.py:132-144."""+ import numpy as np+ confidences = np.asarray(confidences, dtype=float)+ corrects = np.asarray(corrects, dtype=float)+ n = len(confidences)+ if not n:+ return {"n": 0}+ half = max(1, n // 2)+ order = np.argsort(-confidences)[:half]+ return {+ "n": n,+ "accuracy": round(float(corrects.mean()), 4),+ "macro_f1": round(macro_f1(golds, preds), 4),+ "ece": round(ece(confidences, corrects), 4),+ "mean_confidence": round(float(confidences.mean()), 4),+ "acc_at_50_coverage": round(float(corrects[order].mean()), 4),+ }+++# ------------------------------------------------------------------------ runner+def run_language(agent, lang: str, per_lang: int, n_opts: int, seed: int = SEED,+ unclamped: bool = False) -> Dict[str, Any]:+ """Evaluate one language and return its report plus per-case records."""+ import numpy as np+ from laya.common import QTYPES++ rows = load_language(lang)+ cases, gold, option_keys = build_suite(+ rows, sorted({r["label_text"] for r in rows}), per_lang, n_opts, seed)+ logits = score_cases(agent, cases)++ confidences, corrects, preds, records = [], [], [], []+ for i, z in enumerate(logits):+ k = len(z)+ temperature = temperature_for(agent, QTYPES["choice"], k, unclamped)+ probs = softmax_t(z, temperature)+ pred = int(np.argmax(probs))+ correct = int(pred == gold[i])+ confidences.append(float(probs.max()))+ corrects.append(float(correct))+ preds.append(pred)+ records.append({+ "lang": lang,+ "index": i,+ "state": cases[i][0],+ "instructions": INSTRUCTIONS,+ "options": option_keys[i],+ "gold_index": int(gold[i]),+ "gold_label": option_keys[i][gold[i]],+ "pred_index": pred,+ "pred_label": option_keys[i][pred],+ "probability": round(float(probs[pred]), 6),+ "p_gold": round(float(probs[gold[i]]), 6),+ "confidence": round(float(probs.max()), 6),+ "correct": correct,+ "temperature": round(float(temperature), 6),+ })++ report = summarise(confidences, corrects, gold, preds)+ report["temperature"] = round(+ float(temperature_for(agent, QTYPES["choice"], n_opts, unclamped)), 6)+ return {"report": report, "cases": records}+++def main(argv: Optional[Sequence[str]] = None) -> int:+ parser = argparse.ArgumentParser(+ prog="laya-eval",+ description="Per-language accuracy and calibration report for a Laya checkpoint.")+ parser.add_argument("--model", default="convaiinnovations/laya",+ help="checkpoint repo id or local path")+ parser.add_argument("--subfolder", default=None,+ help="checkpoint subfolder, e.g. multilingual")+ parser.add_argument("--device", default=None, help="cpu, cuda, mps (default: auto)")+ parser.add_argument("--langs", default="en",+ help="comma-separated configs, or 'all'")+ parser.add_argument("--per-lang", type=int, default=PER_LANG)+ parser.add_argument("--n-opts", type=int, default=N_OPTS)+ parser.add_argument("--seed", type=int, default=SEED)+ parser.add_argument("--out", default=None, help="write the JSON report here")+ parser.add_argument("--no-cases", action="store_true",+ help="omit per-case records (smaller file)")+ parser.add_argument("--unclamped", action="store_true",+ help="score with the checkpoint's RAW bucket temperatures instead "+ "of the clamped ones Agent applies. This is what reproduces "+ "the committed pre-#42 sweep")+ args = parser.parse_args(argv)++ if args.langs.strip().lower() == "all":+ try:+ langs = available_languages()+ except Exception as exc:+ print("could not list dataset configs: %s" % exc, file=sys.stderr)+ return 2+ else:+ langs = [x.strip() for x in args.langs.split(",") if x.strip()]+ if not langs:+ print("no languages selected", file=sys.stderr)+ return 2++ import laya++ started = time.time()+ agent = laya.load(args.model, device=args.device, subfolder=args.subfolder)+ agent.model.eval()+ payload: Dict[str, Any] = {+ "config": {+ "model": args.model,+ "subfolder": args.subfolder,+ "device": str(agent.device),+ "max_len": agent.cfg.get("max_len"),+ "head_max_len": agent.cfg.get("head_max_len"),+ "dataset": DATASET,+ "split": "test",+ "per_lang": args.per_lang,+ "n_opts": args.n_opts,+ "seed": args.seed,+ "instructions": INSTRUCTIONS,+ "temperatures": dict(agent.temperature_by_options_raw) if args.unclamped+ else dict(agent.temperature_by_options),+ "laya_version": getattr(laya, "__version__", "unknown"),+ },+ "report": {},+ "cases": [],+ }++ for lang in langs:+ t0 = time.time()+ try:+ out = run_language(agent, lang, args.per_lang, args.n_opts,+ args.seed, args.unclamped)+ except Exception as exc:+ print(" %-8s FAILED: %s" % (lang, str(exc)[:110]), file=sys.stderr)+ payload["report"][lang] = {"error": str(exc)[:200]}+ continue+ payload["report"][lang] = out["report"]+ if not args.no_cases:+ payload["cases"].extend(out["cases"])+ r = out["report"]+ print(" %-8s n=%-4d acc=%.4f macro_f1=%.4f ece=%.4f conf=%.4f (%.1fs)"+ % (lang, r.get("n", 0), r.get("accuracy", float("nan")),+ r.get("macro_f1", float("nan")), r.get("ece", float("nan")),+ r.get("mean_confidence", float("nan")), time.time() - t0),+ flush=True)++ scored = {k: v for k, v in payload["report"].items() if "error" not in v and v.get("n")}+ if scored:+ payload["summary"] = {+ "languages": len(scored),+ "macro_accuracy": round(sum(v["accuracy"] for v in scored.values()) / len(scored), 4),+ "macro_ece": round(sum(v["ece"] for v in scored.values()) / len(scored), 4),+ "macro_f1": round(sum(v["macro_f1"] for v in scored.values()) / len(scored), 4),+ "seconds": round(time.time() - started, 1),+ }+ print("\n macro over %d languages: acc=%.4f ece=%.4f f1=%.4f"+ % (len(scored), payload["summary"]["macro_accuracy"],+ payload["summary"]["macro_ece"], payload["summary"]["macro_f1"]))++ if args.out:+ with open(args.out, "w", encoding="utf-8") as fh:+ json.dump(payload, fh, ensure_ascii=False, indent=1)+ print(" wrote %s" % args.out)+ return 0+++if __name__ == "__main__": # pragma: no cover+ raise SystemExit(main())diff --git a/research/eval/test_laya_eval.py b/research/eval/test_laya_eval.pynew file mode 100644index 00000000..d7b74bf6--- /dev/null+++ b/research/eval/test_laya_eval.py@@ -0,0 +1,162 @@+"""Offline tests for the layered evaluation harness.++No checkpoint and no network: every function under test is pure, so this runs in+CI next to the other suites. The model-dependent paths (`score_cases`,+`run_language`) are exercised by `tests/test_local_e2e.py` when a checkpoint is+present.++Run: python research/eval/test_laya_eval.py+"""+import os+import random+import sys++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))++from research.eval.laya_eval import ( # noqa: E402+ ECE_BINS, INSTRUCTIONS, N_OPTS, SEED, build_case, build_suite, ece, macro_f1,+ render_label, summarise, temperature_for,+)++PASS, FAIL = [], []+++def check(name, got, want):+ if got == want:+ PASS.append(name)+ else:+ FAIL.append("%s:\n got %r\n want %r" % (name, got, want))+++def check_true(name, cond, detail=""):+ if cond:+ PASS.append(name)+ else:+ FAIL.append("%s %s" % (name, detail))+++# --------------------------------------------------------------- label rendering+check("render/underscores become spaces", render_label("alarm_set"), "alarm set")+check("render/dots become colon-space", render_label("news.»"), "news: »".replace("»", "»"))+check("render/query_definition", render_label("qa_definition"), "qa definition")+check("render/no underscores left", "_" in render_label("a_b_c"), False)+# must match bench_local.py:176 exactly+check("render/upstream parity", render_label("datetime_query"), "datetime query")+++# ------------------------------------------------------------------ one case+_rng = random.Random(SEED)+_state, _q, _gold_idx = build_case("wake me up at five am", "alarm_set",+ ["a", "b", "c"], _rng, 4)+check("case/state key is utterance", list(_state), ["utterance"])+check("case/single question id", list(_q), ["intent"])+check("case/type is choice", _q["intent"]["type"], "choice")+check("case/instructions fixed", _q["intent"]["instructions"], INSTRUCTIONS)+check("case/option count", len(_q["intent"]["criteria"]), 4)+check("case/gold is present", "alarm_set" in _q["intent"]["criteria"], True)+check("case/gold index points at gold",+ list(_q["intent"]["criteria"])[_gold_idx], "alarm_set")+++# ------------------------------------------------- determinism and seed parity+ROWS = [{"text": "t%d" % i, "label_text": "lab%d" % (i % 7)} for i in range(30)]+LABELS = sorted({r["label_text"] for r in ROWS})++a, b, c = build_suite(ROWS, LABELS, 10, 5, SEED)+d, e, f = build_suite(ROWS, LABELS, 10, 5, SEED)+check("suite/same seed same cases", a == d, True)+check("suite/same seed same gold", b == e, True)+check("suite/same seed same options", c == f, True)++g, h, i = build_suite(ROWS, LABELS, 10, 5, SEED + 1)+check_true("suite/different seed different options", c != i)++# a fresh Random(SEED) per language is what upstream does; two languages with the+# same row content must therefore produce identical option sets+ROWS_B = [dict(r) for r in ROWS]+_b1 = build_suite(ROWS, LABELS, 10, 5, SEED)[2]+_b2 = build_suite(ROWS_B, LABELS, 10, 5, SEED)[2]+check("suite/rng reset per language", _b1, _b2)++check("suite/per_lang caps cases", len(a), 10)+check("suite/gold within options",+ all(0 <= gi < len(opts) for gi, opts in zip(b, c)), True)+check("suite/every gold is in its option set",+ all(opts[gi] == ROWS[i]["label_text"] for i, (gi, opts) in enumerate(zip(b, c))), True)+check("suite/options are distinct",+ all(len(set(o)) == len(o) for o in c), True)+check("suite/n_opts respected", all(len(o) == 5 for o in c), True)++# more options requested than labels available: must not crash or duplicate+tiny = [{"text": "x", "label_text": "only"}]+_tc, _tg, _to = build_suite(tiny, ["only"], 1, 20, SEED)+check("suite/fewer labels than n_opts", len(_to[0]), 1)+++# --------------------------------------------------------------------- metrics+import math # noqa: E402++check_true("ece/empty is nan", math.isnan(ece([], [])))+check("ece/perfect on one bin", round(ece([0.95] * 50, [1.0] * 50), 4), 0.05)+check("ece/all wrong and confident", round(ece([0.95] * 50, [0.0] * 50), 4), 0.95)+check("ece/normalised by count", round(ece([0.9] * 100, [1.0] * 100), 4),+ round(ece([0.9] * 1000, [1.0] * 1000), 4))+check("ece/one sample", round(ece([1.0], [1.0]), 4), 0.0)++# Known boundary: the bin test is `conf > lo`, so conf == 0.0 falls in no bin and+# contributes nothing. `laya.common.ece_score` and `research/scripts/bench_local.py`+# bin identically, so this harness deliberately matches them rather than diverging --+# comparability with the published tables is the point. PR #39 addresses the same+# boundary in `laya.common`; if it lands, this harness should follow it.+check("ece/conf==0.0 is not binned (matches upstream)",+ round(ece([0.0, 0.0], [1.0, 1.0]), 4), 0.0)+check_true("ece/conf slightly above 0 IS binned",+ ece([1e-9, 1e-9], [1.0, 1.0]) > 0.0)++check_true("f1/empty is nan", math.isnan(macro_f1([], [])))+check("f1/perfect", macro_f1([0, 1, 2], [0, 1, 2]), 1.0)+check("f1/all wrong", round(macro_f1([0, 1], [1, 0]), 4), 0.0)+check("f1/disjoint labels", round(macro_f1([0], [1]), 4), 0.0)++_s = summarise([0.9, 0.8, 0.7, 0.6], [1.0, 1.0, 0.0, 0.0], [0, 1, 2, 3], [0, 1, 3, 2])+check("summary/n", _s["n"], 4)+check("summary/accuracy", _s["accuracy"], 0.5)+check("summary/mean_confidence", _s["mean_confidence"], 0.75)+check("summary/acc_at_50_coverage takes the confident half",+ _s["acc_at_50_coverage"], 1.0)+check("summary/empty", summarise([], [], [], []), {"n": 0})+++# ------------------------------------------------------- temperature selection+class _FakeAgent:+ temperature_by_options = {"choice:11+": 0.5}+ temperature = [1.0, 1.0, 1.0]+ temperature_by_options_raw = {"choice:11+": 0.10058280825614929}+ temperature_raw = [1.0, 1.0, 1.0]+++_fa = _FakeAgent()+from laya.common import QTYPES # noqa: E402++check("temp/clamped path is used by default",+ temperature_for(_fa, QTYPES["choice"], 20), 0.5)+check("temp/raw path under unclamped",+ round(temperature_for(_fa, QTYPES["choice"], 20, unclamped=True), 6), 0.100583)+check("temp/falls back to the per-type list",+ temperature_for(_fa, QTYPES["choice"], 3), 1.0)+check("temp/raw falls back too",+ temperature_for(_fa, QTYPES["choice"], 3, unclamped=True), 1.0)+++# ------------------------------------------------------------------- constants+check("const/seed matches upstream", SEED, 13)+check("const/n_opts matches upstream", N_OPTS, 20)+check("const/bins", ECE_BINS, 15)+check("const/instructions match bench_local.py",+ INSTRUCTIONS, "What is the user asking for in `utterance`?")+++print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))+for f in FAIL:+ print(" FAIL " + f)+sys.exit(1 if FAIL else 0)diff --git a/research/results/eval_english_51_languages.json b/research/results/eval_english_51_languages.jsonnew file mode 100644index 00000000..06f9b784--- /dev/null+++ b/research/results/eval_english_51_languages.json@@ -0,0 +1,493 @@+{+ "cases_note": "per-case records omitted from this file to keep the diff reviewable; regenerate with research/eval/laya_eval.py --langs all --out <path>",+ "config": {+ "dataset": "mteb/amazon_massive_intent",+ "device": "cpu",+ "head_max_len": 192,+ "instructions": "What is the user asking for in `utterance`?",+ "laya_version": "0.3.6",+ "max_len": 512,+ "model": "convaiinnovations/laya",+ "n_opts": 20,+ "per_lang": 100,+ "seed": 13,+ "split": "test",+ "subfolder": null,+ "temperatures": {+ "choice:11+": 0.5,+ "choice:2": 1.9063563346862793,+ "choice:3-5": 1.7601518630981445,+ "choice:6-10": 1.0000158548355103,+ "noul:2": 1.983399510383606,+ "score:3-5": 1.2514300346374512+ }+ },+ "report": {+ "af": {+ "acc_at_50_coverage": 0.42,+ "accuracy": 0.29,+ "ece": 0.5588,+ "macro_f1": 0.2874,+ "mean_confidence": 0.8488,+ "n": 100,+ "temperature": 0.5+ },+ "am": {+ "acc_at_50_coverage": 0.12,+ "accuracy": 0.12,+ "ece": 0.6804,+ "macro_f1": 0.1122,+ "mean_confidence": 0.7873,+ "n": 100,+ "temperature": 0.5+ },+ "ar": {+ "acc_at_50_coverage": 0.12,+ "accuracy": 0.11,+ "ece": 0.5795,+ "macro_f1": 0.093,+ "mean_confidence": 0.6833,+ "n": 100,+ "temperature": 0.5+ },+ "az": {+ "acc_at_50_coverage": 0.14,+ "accuracy": 0.1,+ "ece": 0.5983,+ "macro_f1": 0.0629,+ "mean_confidence": 0.6983,+ "n": 100,+ "temperature": 0.5+ },+ "bn": {+ "acc_at_50_coverage": 0.12,+ "accuracy": 0.08,+ "ece": 0.6267,+ "macro_f1": 0.056,+ "mean_confidence": 0.7067,+ "n": 100,+ "temperature": 0.5+ },+ "cy": {+ "acc_at_50_coverage": 0.14,+ "accuracy": 0.12,+ "ece": 0.6452,+ "macro_f1": 0.0906,+ "mean_confidence": 0.7652,+ "n": 100,+ "temperature": 0.5+ },+ "da": {+ "acc_at_50_coverage": 0.52,+ "accuracy": 0.35,+ "ece": 0.526,+ "macro_f1": 0.3082,+ "mean_confidence": 0.876,+ "n": 100,+ "temperature": 0.5+ },+ "de": {+ "acc_at_50_coverage": 0.68,+ "accuracy": 0.42,+ "ece": 0.452,+ "macro_f1": 0.387,+ "mean_confidence": 0.872,+ "n": 100,+ "temperature": 0.5+ },+ "el": {+ "acc_at_50_coverage": 0.2,+ "accuracy": 0.13,+ "ece": 0.6789,+ "macro_f1": 0.1,+ "mean_confidence": 0.8089,+ "n": 100,+ "temperature": 0.5+ },+ "en": {+ "acc_at_50_coverage": 0.98,+ "accuracy": 0.82,+ "ece": 0.1382,+ "macro_f1": 0.7876,+ "mean_confidence": 0.9582,+ "n": 100,+ "temperature": 0.5+ },+ "es": {+ "acc_at_50_coverage": 0.62,+ "accuracy": 0.51,+ "ece": 0.4225,+ "macro_f1": 0.4767,+ "mean_confidence": 0.9245,+ "n": 100,+ "temperature": 0.5+ },+ "fa": {+ "acc_at_50_coverage": 0.16,+ "accuracy": 0.14,+ "ece": 0.6331,+ "macro_f1": 0.1197,+ "mean_confidence": 0.7731,+ "n": 100,+ "temperature": 0.5+ },+ "fi": {+ "acc_at_50_coverage": 0.16,+ "accuracy": 0.13,+ "ece": 0.7189,+ "macro_f1": 0.1213,+ "mean_confidence": 0.8489,+ "n": 100,+ "temperature": 0.5+ },+ "fr": {+ "acc_at_50_coverage": 0.78,+ "accuracy": 0.59,+ "ece": 0.2995,+ "macro_f1": 0.5822,+ "mean_confidence": 0.8895,+ "n": 100,+ "temperature": 0.5+ },+ "he": {+ "acc_at_50_coverage": 0.02,+ "accuracy": 0.06,+ "ece": 0.7379,+ "macro_f1": 0.039,+ "mean_confidence": 0.7842,+ "n": 100,+ "temperature": 0.5+ },+ "hi": {+ "acc_at_50_coverage": 0.08,+ "accuracy": 0.1,+ "ece": 0.6416,+ "macro_f1": 0.0758,+ "mean_confidence": 0.7416,+ "n": 100,+ "temperature": 0.5+ },+ "hu": {+ "acc_at_50_coverage": 0.06,+ "accuracy": 0.09,+ "ece": 0.6783,+ "macro_f1": 0.0813,+ "mean_confidence": 0.7683,+ "n": 100,+ "temperature": 0.5+ },+ "hy": {+ "acc_at_50_coverage": 0.04,+ "accuracy": 0.05,+ "ece": 0.5709,+ "macro_f1": 0.0355,+ "mean_confidence": 0.6209,+ "n": 100,+ "temperature": 0.5+ },+ "id": {+ "acc_at_50_coverage": 0.48,+ "accuracy": 0.36,+ "ece": 0.487,+ "macro_f1": 0.3283,+ "mean_confidence": 0.8304,+ "n": 100,+ "temperature": 0.5+ },+ "is": {+ "acc_at_50_coverage": 0.16,+ "accuracy": 0.11,+ "ece": 0.6641,+ "macro_f1": 0.0893,+ "mean_confidence": 0.7741,+ "n": 100,+ "temperature": 0.5+ },+ "it": {+ "acc_at_50_coverage": 0.52,+ "accuracy": 0.34,+ "ece": 0.5156,+ "macro_f1": 0.3101,+ "mean_confidence": 0.8534,+ "n": 100,+ "temperature": 0.5+ },+ "ja": {+ "acc_at_50_coverage": 0.76,+ "accuracy": 0.53,+ "ece": 0.3657,+ "macro_f1": 0.4942,+ "mean_confidence": 0.8957,+ "n": 100,+ "temperature": 0.5+ },+ "jv": {+ "acc_at_50_coverage": 0.16,+ "accuracy": 0.16,+ "ece": 0.67,+ "macro_f1": 0.1537,+ "mean_confidence": 0.83,+ "n": 100,+ "temperature": 0.5+ },+ "ka": {+ "acc_at_50_coverage": 0.1,+ "accuracy": 0.09,+ "ece": 0.6102,+ "macro_f1": 0.0742,+ "mean_confidence": 0.7002,+ "n": 100,+ "temperature": 0.5+ },+ "km": {+ "acc_at_50_coverage": 0.0,+ "accuracy": 0.0,+ "ece": 0.7051,+ "macro_f1": 0.0,+ "mean_confidence": 0.7051,+ "n": 100,+ "temperature": 0.5+ },+ "kn": {+ "acc_at_50_coverage": 0.08,+ "accuracy": 0.11,+ "ece": 0.5903,+ "macro_f1": 0.0913,+ "mean_confidence": 0.695,+ "n": 100,+ "temperature": 0.5+ },+ "ko": {+ "acc_at_50_coverage": 0.14,+ "accuracy": 0.11,+ "ece": 0.6804,+ "macro_f1": 0.0943,+ "mean_confidence": 0.7904,+ "n": 100,+ "temperature": 0.5+ },+ "lv": {+ "acc_at_50_coverage": 0.1,+ "accuracy": 0.1,+ "ece": 0.6659,+ "macro_f1": 0.0786,+ "mean_confidence": 0.7659,+ "n": 100,+ "temperature": 0.5+ },+ "ml": {+ "acc_at_50_coverage": 0.1,+ "accuracy": 0.07,+ "ece": 0.601,+ "macro_f1": 0.0503,+ "mean_confidence": 0.671,+ "n": 100,+ "temperature": 0.5+ },+ "mn": {+ "acc_at_50_coverage": 0.14,+ "accuracy": 0.13,+ "ece": 0.6704,+ "macro_f1": 0.1213,+ "mean_confidence": 0.7977,+ "n": 100,+ "temperature": 0.5+ },+ "ms": {+ "acc_at_50_coverage": 0.4,+ "accuracy": 0.27,+ "ece": 0.5064,+ "macro_f1": 0.2372,+ "mean_confidence": 0.7764,+ "n": 100,+ "temperature": 0.5+ },+ "my": {+ "acc_at_50_coverage": 0.08,+ "accuracy": 0.06,+ "ece": 0.5797,+ "macro_f1": 0.0455,+ "mean_confidence": 0.6397,+ "n": 100,+ "temperature": 0.5+ },+ "nb": {+ "acc_at_50_coverage": 0.52,+ "accuracy": 0.33,+ "ece": 0.5179,+ "macro_f1": 0.3025,+ "mean_confidence": 0.8479,+ "n": 100,+ "temperature": 0.5+ },+ "nl": {+ "acc_at_50_coverage": 0.56,+ "accuracy": 0.39,+ "ece": 0.5025,+ "macro_f1": 0.3968,+ "mean_confidence": 0.8831,+ "n": 100,+ "temperature": 0.5+ },+ "pl": {+ "acc_at_50_coverage": 0.36,+ "accuracy": 0.24,+ "ece": 0.5695,+ "macro_f1": 0.2033,+ "mean_confidence": 0.7951,+ "n": 100,+ "temperature": 0.5+ },+ "pt": {+ "acc_at_50_coverage": 0.66,+ "accuracy": 0.47,+ "ece": 0.4317,+ "macro_f1": 0.4213,+ "mean_confidence": 0.8894,+ "n": 100,+ "temperature": 0.5+ },+ "ro": {+ "acc_at_50_coverage": 0.48,+ "accuracy": 0.33,+ "ece": 0.5499,+ "macro_f1": 0.2976,+ "mean_confidence": 0.8668,+ "n": 100,+ "temperature": 0.5+ },+ "ru": {+ "acc_at_50_coverage": 0.42,+ "accuracy": 0.31,+ "ece": 0.5946,+ "macro_f1": 0.3084,+ "mean_confidence": 0.9046,+ "n": 100,+ "temperature": 0.5+ },+ "sl": {+ "acc_at_50_coverage": 0.26,+ "accuracy": 0.2,+ "ece": 0.5994,+ "macro_f1": 0.1679,+ "mean_confidence": 0.7994,+ "n": 100,+ "temperature": 0.5+ },+ "sq": {+ "acc_at_50_coverage": 0.28,+ "accuracy": 0.21,+ "ece": 0.6031,+ "macro_f1": 0.1667,+ "mean_confidence": 0.8131,+ "n": 100,+ "temperature": 0.5+ },+ "sv": {+ "acc_at_50_coverage": 0.5,+ "accuracy": 0.38,+ "ece": 0.4874,+ "macro_f1": 0.3809,+ "mean_confidence": 0.8501,+ "n": 100,+ "temperature": 0.5+ },+ "sw": {+ "acc_at_50_coverage": 0.14,+ "accuracy": 0.13,+ "ece": 0.6318,+ "macro_f1": 0.1149,+ "mean_confidence": 0.7511,+ "n": 100,+ "temperature": 0.5+ },+ "ta": {+ "acc_at_50_coverage": 0.12,+ "accuracy": 0.12,+ "ece": 0.5541,+ "macro_f1": 0.0971,+ "mean_confidence": 0.6741,+ "n": 100,+ "temperature": 0.5+ },+ "te": {+ "acc_at_50_coverage": 0.06,+ "accuracy": 0.09,+ "ece": 0.6689,+ "macro_f1": 0.0722,+ "mean_confidence": 0.7568,+ "n": 100,+ "temperature": 0.5+ },+ "th": {+ "acc_at_50_coverage": 0.1,+ "accuracy": 0.08,+ "ece": 0.7179,+ "macro_f1": 0.048,+ "mean_confidence": 0.7979,+ "n": 100,+ "temperature": 0.5+ },+ "tl": {+ "acc_at_50_coverage": 0.46,+ "accuracy": 0.29,+ "ece": 0.5121,+ "macro_f1": 0.249,+ "mean_confidence": 0.8021,+ "n": 100,+ "temperature": 0.5+ },+ "tr": {+ "acc_at_50_coverage": 0.2,+ "accuracy": 0.14,+ "ece": 0.5874,+ "macro_f1": 0.0967,+ "mean_confidence": 0.7274,+ "n": 100,+ "temperature": 0.5+ },+ "ur": {+ "acc_at_50_coverage": 0.08,+ "accuracy": 0.07,+ "ece": 0.6428,+ "macro_f1": 0.0645,+ "mean_confidence": 0.7128,+ "n": 100,+ "temperature": 0.5+ },+ "vi": {+ "acc_at_50_coverage": 0.06,+ "accuracy": 0.06,+ "ece": 0.69,+ "macro_f1": 0.0498,+ "mean_confidence": 0.75,+ "n": 100,+ "temperature": 0.5+ },+ "zh-CN": {+ "acc_at_50_coverage": 0.84,+ "accuracy": 0.62,+ "ece": 0.3198,+ "macro_f1": 0.6175,+ "mean_confidence": 0.9244,+ "n": 100,+ "temperature": 0.5+ },+ "zh-TW": {+ "acc_at_50_coverage": 0.72,+ "accuracy": 0.46,+ "ece": 0.4344,+ "macro_f1": 0.4294,+ "mean_confidence": 0.8944,+ "n": 100,+ "temperature": 0.5+ }+ },+ "summary": {+ "languages": 51,+ "macro_accuracy": 0.2269,+ "macro_ece": 0.5709,+ "macro_f1": 0.2053,+ "seconds": 2017.8+ }+}\ No newline at end of file