Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

research: benchmark harnesses and raw results (#14)

Nandakishoree760389dc69
diff --git a/research/README.md b/research/README.md
new file mode 100644
index 00000000..7abd50c8
--- /dev/null
+++ b/research/README.md
@@ -0,0 +1,60 @@
+# Research
+
+Benchmark harnesses and raw results for the Laya checkpoints. This branch is the evidence behind
+the numbers quoted in the main README — nothing here is imported by the `laya` package.
+
+## Scripts
+
+| file | what it does |
+|---|---|
+| `scripts/laya_benchmark_colab.ipynb` | the full head-to-head on a Colab T4: typed-decisions, MASSIVE intent + scenario (14 languages), XNLI (15), English suites, latency, option-order robustness, calibration repair. Writes one JSON. |
+| `scripts/build_benchmark_nb.py` | generator for that notebook (edit here, not the `.ipynb`) |
+| `scripts/bench_local.py` | CPU sweep: MASSIVE intent across **all 51 languages**, plus typed-decisions on all three checkpoints |
+| `scripts/bench_apps.py` | the six application workflows (support triage, email + phishing, guardrails, RAG relevance, moderation, model routing) plus the datasets where public Jev numbers exist |
+| `scripts/bench_latency.py` | inference speed including what routing costs: detection overhead, hot path, cold-swap, mixed-language throughput at several `max_loaded` settings |
+| `scripts/make_plots.py` | renders `assets/laya_benchmark.png` from the result JSONs |
+
+Everything runs with `USE_TF=0` — `transformers` probes for TensorFlow at import, and when TF is
+installed its abseil runtime can deadlock model construction on macOS/Python 3.9.
+
+## Results
+
+| file | contents |
+|---|---|
+| `results/t4_colab_benchmark.json` | 17,416 questions on one T4, both checkpoints, identical questions per model |
+| `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options |
+
+## Headline findings
+
+**Routing takes Laya from 23 to 45 of 51 languages.** On MASSIVE intent (20 options, random =
+0.050) the English checkpoint macro-averages 0.227 and clears 3x random on 23 of 51 languages;
+the multilingual checkpoint reaches 0.366 and clears it on 45.
+
+**The English checkpoint's confidence gives no warning when it cannot read the input.** Khmer:
+0.000 accuracy at 0.952 mean confidence. Macro ECE 0.733 across 51 languages, with mean
+confidence never dropping below 0.885 at any accuracy level. This is why routing has to happen
+*before* the forward pass — confidence gating cannot catch it.
+
+**Both checkpoints ship over-confident.** Refitting one temperature per (question type, option
+count) on held-out data moves mean ECE 0.466 -> 0.081 (`laya`) and 0.314 -> 0.106
+(`laya-multilingual`, which ships with no fitted temperatures at all).
+
+**The base checkpoints are near chance on typed-decisions zero-shot** — 0.362 and 0.352 against
+a 0.318 random baseline and a 0.461 majority-class baseline. The published 0.766 belongs to the
+checkpoint fine-tuned on that benchmark's own training split.
+
+**Speed.** 32.8 ms for one question and 7.2 ms/question at batch 10 on a T4; 103–332 questions/s
+batched.
+
+## On comparisons with Jev
+
+There is no TypeSafe API credential in this project, so **Jev was never run here**. Every Jev
+figure quoted is third-party published, with different sample sizes and prompts:
+
+- [AbdelStark/jev-benchmarks](https://github.com/AbdelStark/jev-benchmarks) — AG News 0.910,
+ Banking77 0.870, DAIR Emotion 0.480 (Brier 0.846, NLL 5.588, zero probability on the true label
+ for 16% of examples)
+- [nibzard/decision-model-benchmark](https://github.com/nibzard/decision-model-benchmark) — ECE
+ 0.246 (worst in that study), banking77 0.763, option-order flip rate 13%, latency 264–276 ms p50
+
+Treat those as indicative, not a controlled head-to-head.
diff --git a/research/results/cpu_51_language_sweep.json b/research/results/cpu_51_language_sweep.json
new file mode 100644
index 00000000..1cb7d5ed
--- /dev/null
+++ b/research/results/cpu_51_language_sweep.json
@@ -0,0 +1,1430 @@
+{
+ "meta": {
+ "timestamp": "2026-09-19 11:04:45",
+ "device": "cpu",
+ "torch": "2.8.0",
+ "laya": "0.2.0",
+ "threads": 4
+ },
+ "part_a": {
+ "config": {
+ "languages": [
+ "af",
+ "am",
+ "ar",
+ "az",
+ "bn",
+ "cy",
+ "da",
+ "de",
+ "el",
+ "en",
+ "es",
+ "fa",
+ "fi",
+ "fr",
+ "he",
+ "hi",
+ "hu",
+ "hy",
+ "id",
+ "is",
+ "it",
+ "ja",
+ "jv",
+ "ka",
+ "km",
+ "kn",
+ "ko",
+ "lv",
+ "ml",
+ "mn",
+ "ms",
+ "my",
+ "nb",
+ "nl",
+ "pl",
+ "pt",
+ "ro",
+ "ru",
+ "sl",
+ "sq",
+ "sv",
+ "sw",
+ "ta",
+ "te",
+ "th",
+ "tl",
+ "tr",
+ "ur",
+ "vi",
+ "zh-CN",
+ "zh-TW"
+ ],
+ "per_lang": 100,
+ "n_options": 20,
+ "seed": 13
+ },
+ "by_model": {
+ "english": {
+ "per_language": {
+ "af": {
+ "n": 100,
+ "accuracy": 0.29,
+ "macro_f1": 0.2874,
+ "ece": 0.6873,
+ "brier": 1.3787,
+ "nll": 16.3516,
+ "mean_confidence": 0.9752,
+ "acc_at_50_coverage": 0.42,
+ "seconds": 11.6,
+ "dropped": 0
+ },
+ "am": {
+ "n": 100,
+ "accuracy": 0.12,
+ "macro_f1": 0.1122,
+ "ece": 0.8252,
+ "brier": 1.6644,
+ "nll": 21.6026,
+ "mean_confidence": 0.9452,
+ "acc_at_50_coverage": 0.14,
+ "seconds": 14.6,
+ "dropped": 0
+ },
+ "ar": {
+ "n": 100,
+ "accuracy": 0.11,
+ "macro_f1": 0.093,
+ "ece": 0.8001,
+ "brier": 1.6539,
+ "nll": 20.5087,
+ "mean_confidence": 0.8976,
+ "acc_at_50_coverage": 0.12,
+ "seconds": 11.8,
+ "dropped": 0
+ },
+ "az": {
+ "n": 100,
+ "accuracy": 0.1,
+ "macro_f1": 0.0629,
+ "ece": 0.8247,
+ "brier": 1.6966,
+ "nll": 21.7232,
+ "mean_confidence": 0.9247,
+ "acc_at_50_coverage": 0.14,
+ "seconds": 11.9,
+ "dropped": 0
+ },
+ "bn": {
+ "n": 100,
+ "accuracy": 0.08,
+ "macro_f1": 0.056,
+ "ece": 0.8646,
+ "brier": 1.7373,
+ "nll": 20.8691,
+ "mean_confidence": 0.9446,
+ "acc_at_50_coverage": 0.1,
+ "seconds": 13.9,
+ "dropped": 0
+ },
+ "cy": {
+ "n": 100,
+ "accuracy": 0.12,
+ "macro_f1": 0.0906,
+ "ece": 0.8409,
+ "brier": 1.7029,
+ "nll": 21.2255,
+ "mean_confidence": 0.9609,
+ "acc_at_50_coverage": 0.14,
+ "seconds": 11.8,
+ "dropped": 0
+ },
+ "da": {
+ "n": 100,
+ "accuracy": 0.35,
+ "macro_f1": 0.3082,
+ "ece": 0.6263,
+ "brier": 1.2633,
+ "nll": 16.2409,
+ "mean_confidence": 0.9763,
+ "acc_at_50_coverage": 0.52,
+ "seconds": 12.0,
+ "dropped": 0
+ },
+ "de": {
+ "n": 100,
+ "accuracy": 0.42,
+ "macro_f1": 0.387,
+ "ece": 0.5584,
+ "brier": 1.126,
+ "nll": 13.7538,
+ "mean_confidence": 0.9784,
+ "acc_at_50_coverage": 0.66,
+ "seconds": 14.1,
+ "dropped": 0
+ },
+ "el": {
+ "n": 100,
+ "accuracy": 0.13,
+ "macro_f1": 0.1,
+ "ece": 0.8387,
+ "brier": 1.6854,
+ "nll": 21.1672,
+ "mean_confidence": 0.9687,
+ "acc_at_50_coverage": 0.16,
+ "seconds": 15.1,
+ "dropped": 0
+ },
+ "en": {
+ "n": 100,
+ "accuracy": 0.82,
+ "macro_f1": 0.7876,
+ "ece": 0.1789,
+ "brier": 0.3563,
+ "nll": 3.5193,
+ "mean_confidence": 0.9989,
+ "acc_at_50_coverage": 0.94,
+ "seconds": 16.1,
+ "dropped": 0
+ },
+ "es": {
+ "n": 100,
+ "accuracy": 0.51,
+ "macro_f1": 0.4767,
+ "ece": 0.4796,
+ "brier": 0.9706,
+ "nll": 11.846,
+ "mean_confidence": 0.9896,
+ "acc_at_50_coverage": 0.6,
+ "seconds": 20.3,
+ "dropped": 0
+ },
+ "fa": {
+ "n": 100,
+ "accuracy": 0.14,
+ "macro_f1": 0.1197,
+ "ece": 0.8196,
+ "brier": 1.6664,
+ "nll": 21.65,
+ "mean_confidence": 0.9414,
+ "acc_at_50_coverage": 0.16,
+ "seconds": 24.5,
+ "dropped": 0
+ },
+ "fi": {
+ "n": 100,
+ "accuracy": 0.13,
+ "macro_f1": 0.1213,
+ "ece": 0.8486,
+ "brier": 1.7071,
+ "nll": 21.923,
+ "mean_confidence": 0.9786,
+ "acc_at_50_coverage": 0.16,
+ "seconds": 20.1,
+ "dropped": 0
+ },
+ "fr": {
+ "n": 100,
+ "accuracy": 0.59,
+ "macro_f1": 0.5822,
+ "ece": 0.3882,
+ "brier": 0.7935,
+ "nll": 9.1132,
+ "mean_confidence": 0.9782,
+ "acc_at_50_coverage": 0.78,
+ "seconds": 19.4,
+ "dropped": 0
+ },
+ "he": {
+ "n": 100,
+ "accuracy": 0.06,
+ "macro_f1": 0.039,
+ "ece": 0.9111,
+ "brier": 1.8321,
+ "nll": 24.2157,
+ "mean_confidence": 0.9643,
+ "acc_at_50_coverage": 0.04,
+ "seconds": 20.1,
+ "dropped": 0
+ },
+ "hi": {
+ "n": 100,
+ "accuracy": 0.1,
+ "macro_f1": 0.0758,
+ "ece": 0.8504,
+ "brier": 1.7169,
+ "nll": 21.3541,
+ "mean_confidence": 0.9408,
+ "acc_at_50_coverage": 0.1,
+ "seconds": 21.1,
+ "dropped": 0
+ },
+ "hu": {
+ "n": 100,
+ "accuracy": 0.09,
+ "macro_f1": 0.0813,
+ "ece": 0.8571,
+ "brier": 1.7503,
+ "nll": 23.2612,
+ "mean_confidence": 0.9471,
+ "acc_at_50_coverage": 0.06,
+ "seconds": 18.9,
+ "dropped": 0
+ },
+ "hy": {
+ "n": 100,
+ "accuracy": 0.05,
+ "macro_f1": 0.0355,
+ "ece": 0.8348,
+ "brier": 1.7324,
+ "nll": 20.695,
+ "mean_confidence": 0.8848,
+ "acc_at_50_coverage": 0.08,
+ "seconds": 22.5,
+ "dropped": 0
+ },
+ "id": {
+ "n": 100,
+ "accuracy": 0.36,
+ "macro_f1": 0.3283,
+ "ece": 0.6133,
+ "brier": 1.2484,
+ "nll": 14.6851,
+ "mean_confidence": 0.9619,
+ "acc_at_50_coverage": 0.48,
+ "seconds": 18.7,
+ "dropped": 0
+ },
+ "is": {
+ "n": 100,
+ "accuracy": 0.11,
+ "macro_f1": 0.0893,
+ "ece": 0.8351,
+ "brier": 1.6975,
+ "nll": 21.9127,
+ "mean_confidence": 0.9451,
+ "acc_at_50_coverage": 0.16,
+ "seconds": 19.2,
+ "dropped": 0
+ },
+ "it": {
+ "n": 100,
+ "accuracy": 0.34,
+ "macro_f1": 0.3101,
+ "ece": 0.6474,
+ "brier": 1.2866,
+ "nll": 15.5383,
+ "mean_confidence": 0.9735,
+ "acc_at_50_coverage": 0.52,
+ "seconds": 19.2,
+ "dropped": 0
+ },
+ "ja": {
+ "n": 100,
+ "accuracy": 0.53,
+ "macro_f1": 0.4942,
+ "ece": 0.4599,
+ "brier": 0.9247,
+ "nll": 10.8681,
+ "mean_confidence": 0.9801,
+ "acc_at_50_coverage": 0.76,
+ "seconds": 19.9,
+ "dropped": 0
+ },
+ "jv": {
+ "n": 100,
+ "accuracy": 0.16,
+ "macro_f1": 0.1537,
+ "ece": 0.8035,
+ "brier": 1.6288,
+ "nll": 20.6859,
+ "mean_confidence": 0.9635,
+ "acc_at_50_coverage": 0.14,
+ "seconds": 19.2,
+ "dropped": 0
+ },
+ "ka": {
+ "n": 100,
+ "accuracy": 0.09,
+ "macro_f1": 0.0742,
+ "ece": 0.8448,
+ "brier": 1.72,
+ "nll": 21.2473,
+ "mean_confidence": 0.9348,
+ "acc_at_50_coverage": 0.1,
+ "seconds": 22.0,
+ "dropped": 0
+ },
+ "km": {
+ "n": 100,
+ "accuracy": 0.0,
+ "macro_f1": 0.0,
+ "ece": 0.9515,
+ "brier": 1.9219,
+ "nll": 23.2123,
+ "mean_confidence": 0.9515,
+ "acc_at_50_coverage": 0.0,
+ "seconds": 25.4,
+ "dropped": 0
+ },
+ "kn": {
+ "n": 100,
+ "accuracy": 0.11,
+ "macro_f1": 0.0913,
+ "ece": 0.8416,
+ "brier": 1.7051,
+ "nll": 20.4279,
+ "mean_confidence": 0.9421,
+ "acc_at_50_coverage": 0.06,
+ "seconds": 24.3,
+ "dropped": 0
+ },
+ "ko": {
+ "n": 100,
+ "accuracy": 0.11,
+ "macro_f1": 0.0943,
+ "ece": 0.8498,
+ "brier": 1.7193,
+ "nll": 22.1195,
+ "mean_confidence": 0.9598,
+ "acc_at_50_coverage": 0.14,
+ "seconds": 20.1,
+ "dropped": 0
+ },
+ "lv": {
+ "n": 100,
+ "accuracy": 0.1,
+ "macro_f1": 0.0786,
+ "ece": 0.8466,
+ "brier": 1.7322,
+ "nll": 21.4978,
+ "mean_confidence": 0.9466,
+ "acc_at_50_coverage": 0.1,
+ "seconds": 19.4,
+ "dropped": 0
+ },
+ "ml": {
+ "n": 100,
+ "accuracy": 0.07,
+ "macro_f1": 0.0503,
+ "ece": 0.8572,
+ "brier": 1.7485,
+ "nll": 20.0485,
+ "mean_confidence": 0.9272,
+ "acc_at_50_coverage": 0.1,
+ "seconds": 24.6,
+ "dropped": 0
+ },
+ "mn": {
+ "n": 100,
+ "accuracy": 0.13,
+ "macro_f1": 0.1213,
+ "ece": 0.837,
+ "brier": 1.7109,
+ "nll": 21.5421,
+ "mean_confidence": 0.9591,
+ "acc_at_50_coverage": 0.14,
+ "seconds": 20.6,
+ "dropped": 0
+ },
+ "ms": {
+ "n": 100,
+ "accuracy": 0.27,
+ "macro_f1": 0.2372,
+ "ece": 0.6877,
+ "brier": 1.4027,
+ "nll": 16.277,
+ "mean_confidence": 0.9577,
+ "acc_at_50_coverage": 0.42,
+ "seconds": 19.2,
+ "dropped": 0
+ },
+ "my": {
+ "n": 100,
+ "accuracy": 0.06,
+ "macro_f1": 0.0455,
+ "ece": 0.8605,
+ "brier": 1.7635,
+ "nll": 19.7847,
+ "mean_confidence": 0.9205,
+ "acc_at_50_coverage": 0.08,
+ "seconds": 24.7,
+ "dropped": 0
+ },
+ "nb": {
+ "n": 100,
+ "accuracy": 0.33,
+ "macro_f1": 0.3025,
+ "ece": 0.6484,
+ "brier": 1.302,
+ "nll": 16.4489,
+ "mean_confidence": 0.9692,
+ "acc_at_50_coverage": 0.5,
+ "seconds": 19.2,
+ "dropped": 0
+ },
+ "nl": {
+ "n": 100,
+ "accuracy": 0.39,
+ "macro_f1": 0.3968,
+ "ece": 0.5914,
+ "brier": 1.1943,
+ "nll": 14.4159,
+ "mean_confidence": 0.9745,
+ "acc_at_50_coverage": 0.54,
+ "seconds": 20.7,
+ "dropped": 0
+ },
+ "pl": {
+ "n": 100,
+ "accuracy": 0.24,
+ "macro_f1": 0.2033,
+ "ece": 0.7129,
+ "brier": 1.4653,
+ "nll": 18.4945,
+ "mean_confidence": 0.9529,
+ "acc_at_50_coverage": 0.34,
+ "seconds": 20.8,
+ "dropped": 0
+ },
+ "pt": {
+ "n": 100,
+ "accuracy": 0.47,
+ "macro_f1": 0.4213,
+ "ece": 0.5116,
+ "brier": 1.0273,
+ "nll": 12.7024,
+ "mean_confidence": 0.9718,
+ "acc_at_50_coverage": 0.66,
+ "seconds": 20.8,
+ "dropped": 0
+ },
+ "ro": {
+ "n": 100,
+ "accuracy": 0.33,
+ "macro_f1": 0.2976,
+ "ece": 0.6577,
+ "brier": 1.3076,
+ "nll": 15.9637,
+ "mean_confidence": 0.9824,
+ "acc_at_50_coverage": 0.48,
+ "seconds": 20.0,
+ "dropped": 0
+ },
+ "ru": {
+ "n": 100,
+ "accuracy": 0.31,
+ "macro_f1": 0.3084,
+ "ece": 0.668,
+ "brier": 1.3452,
+ "nll": 17.0527,
+ "mean_confidence": 0.978,
+ "acc_at_50_coverage": 0.42,
+ "seconds": 20.5,
+ "dropped": 0
+ },
+ "sl": {
+ "n": 100,
+ "accuracy": 0.2,
+ "macro_f1": 0.1679,
+ "ece": 0.7561,
+ "brier": 1.5293,
+ "nll": 19.7372,
+ "mean_confidence": 0.9561,
+ "acc_at_50_coverage": 0.26,
+ "seconds": 20.2,
+ "dropped": 0
+ },
+ "sq": {
+ "n": 100,
+ "accuracy": 0.21,
+ "macro_f1": 0.1667,
+ "ece": 0.7549,
+ "brier": 1.5259,
+ "nll": 19.5615,
+ "mean_confidence": 0.9649,
+ "acc_at_50_coverage": 0.28,
+ "seconds": 20.5,
+ "dropped": 0
+ },
+ "sv": {
+ "n": 100,
+ "accuracy": 0.38,
+ "macro_f1": 0.3809,
+ "ece": 0.5956,
+ "brier": 1.2168,
+ "nll": 14.3656,
+ "mean_confidence": 0.9706,
+ "acc_at_50_coverage": 0.5,
+ "seconds": 19.9,
+ "dropped": 0
+ },
+ "sw": {
+ "n": 100,
+ "accuracy": 0.13,
+ "macro_f1": 0.1149,
+ "ece": 0.8282,
+ "brier": 1.6934,
+ "nll": 21.2625,
+ "mean_confidence": 0.9562,
+ "acc_at_50_coverage": 0.14,
+ "seconds": 21.2,
+ "dropped": 0
+ },
+ "ta": {
+ "n": 100,
+ "accuracy": 0.12,
+ "macro_f1": 0.0971,
+ "ece": 0.8224,
+ "brier": 1.6768,
+ "nll": 19.6604,
+ "mean_confidence": 0.9424,
+ "acc_at_50_coverage": 0.12,
+ "seconds": 25.6,
+ "dropped": 0
+ },
+ "te": {
+ "n": 100,
+ "accuracy": 0.09,
+ "macro_f1": 0.0722,
+ "ece": 0.8575,
+ "brier": 1.7384,
+ "nll": 21.777,
+ "mean_confidence": 0.9475,
+ "acc_at_50_coverage": 0.06,
+ "seconds": 25.2,
+ "dropped": 0
+ },
+ "th": {
+ "n": 100,
+ "accuracy": 0.08,
+ "macro_f1": 0.048,
+ "ece": 0.8814,
+ "brier": 1.7728,
+ "nll": 22.4846,
+ "mean_confidence": 0.9614,
+ "acc_at_50_coverage": 0.1,
+ "seconds": 22.9,
+ "dropped": 0
+ },
+ "tl": {
+ "n": 100,
+ "accuracy": 0.29,
+ "macro_f1": 0.249,
+ "ece": 0.6756,
+ "brier": 1.3666,
+ "nll": 16.3214,
+ "mean_confidence": 0.9656,
+ "acc_at_50_coverage": 0.46,
+ "seconds": 20.7,
+ "dropped": 0
+ },
+ "tr": {
+ "n": 100,
+ "accuracy": 0.14,
+ "macro_f1": 0.0967,
+ "ece": 0.788,
+ "brier": 1.6064,
+ "nll": 19.6609,
+ "mean_confidence": 0.928,
+ "acc_at_50_coverage": 0.2,
+ "seconds": 20.7,
+ "dropped": 0
+ },
+ "ur": {
+ "n": 100,
+ "accuracy": 0.07,
+ "macro_f1": 0.0645,
+ "ece": 0.8826,
+ "brier": 1.7822,
+ "nll": 21.4571,
+ "mean_confidence": 0.9526,
+ "acc_at_50_coverage": 0.08,
+ "seconds": 23.2,
+ "dropped": 0
+ },
+ "vi": {
+ "n": 100,
+ "accuracy": 0.06,
+ "macro_f1": 0.0498,
+ "ece": 0.8914,
+ "brier": 1.8055,
+ "nll": 22.6885,
+ "mean_confidence": 0.9514,
+ "acc_at_50_coverage": 0.06,
+ "seconds": 22.9,
+ "dropped": 0
+ },
+ "zh-CN": {
+ "n": 100,
+ "accuracy": 0.62,
+ "macro_f1": 0.6175,
+ "ece": 0.376,
+ "brier": 0.7521,
+ "nll": 8.5938,
+ "mean_confidence": 0.9885,
+ "acc_at_50_coverage": 0.86,
+ "seconds": 22.0,
+ "dropped": 0
+ },
+ "zh-TW": {
+ "n": 100,
+ "accuracy": 0.46,
+ "macro_f1": 0.4294,
+ "ece": 0.5198,
+ "brier": 1.0483,
+ "nll": 12.3023,
+ "mean_confidence": 0.9798,
+ "acc_at_50_coverage": 0.74,
+ "seconds": 21.1,
+ "dropped": 0
+ }
+ },
+ "macro_accuracy": 0.2269,
+ "macro_ece": 0.7331,
+ "n_languages": 51,
+ "languages_above_random": 23
+ },
+ "multilingual": {
+ "per_language": {
+ "af": {
+ "n": 100,
+ "accuracy": 0.35,
+ "macro_f1": 0.3133,
+ "ece": 0.4845,
+ "brier": 1.0466,
+ "nll": 3.5729,
+ "mean_confidence": 0.8028,
+ "acc_at_50_coverage": 0.54,
+ "seconds": 9.0,
+ "dropped": 0
+ },
+ "am": {
+ "n": 100,
+ "accuracy": 0.11,
+ "macro_f1": 0.1046,
+ "ece": 0.4626,
+ "brier": 1.2477,
+ "nll": 4.265,
+ "mean_confidence": 0.5706,
+ "acc_at_50_coverage": 0.12,
+ "seconds": 9.7,
+ "dropped": 0
+ },
+ "ar": {
+ "n": 100,
+ "accuracy": 0.4,
+ "macro_f1": 0.3926,
+ "ece": 0.341,
+ "brier": 0.8619,
+ "nll": 2.697,
+ "mean_confidence": 0.731,
+ "acc_at_50_coverage": 0.6,
+ "seconds": 8.8,
+ "dropped": 0
+ },
+ "az": {
+ "n": 100,
+ "accuracy": 0.3,
+ "macro_f1": 0.2693,
+ "ece": 0.3685,
+ "brier": 0.9758,
+ "nll": 3.1016,
+ "mean_confidence": 0.6685,
+ "acc_at_50_coverage": 0.46,
+ "seconds": 9.7,
+ "dropped": 0
+ },
+ "bn": {
+ "n": 100,
+ "accuracy": 0.29,
+ "macro_f1": 0.2654,
+ "ece": 0.4085,
+ "brier": 1.0428,
+ "nll": 3.2654,
+ "mean_confidence": 0.6944,
+ "acc_at_50_coverage": 0.4,
+ "seconds": 10.3,
+ "dropped": 0
+ },
+ "cy": {
+ "n": 100,
+ "accuracy": 0.16,
+ "macro_f1": 0.1432,
+ "ece": 0.5905,
+ "brier": 1.3315,
+ "nll": 5.0927,
+ "mean_confidence": 0.7149,
+ "acc_at_50_coverage": 0.18,
+ "seconds": 9.4,
+ "dropped": 0
+ },
+ "da": {
+ "n": 100,
+ "accuracy": 0.5,
+ "macro_f1": 0.4682,
+ "ece": 0.2628,
+ "brier": 0.7433,
+ "nll": 2.0483,
+ "mean_confidence": 0.7544,
+ "acc_at_50_coverage": 0.68,
+ "seconds": 9.3,
+ "dropped": 0
+ },
+ "de": {
+ "n": 100,
+ "accuracy": 0.5,
+ "macro_f1": 0.4847,
+ "ece": 0.3014,
+ "brier": 0.752,
+ "nll": 2.4918,
+ "mean_confidence": 0.784,
+ "acc_at_50_coverage": 0.7,
+ "seconds": 8.8,
+ "dropped": 0
+ },
+ "el": {
+ "n": 100,
+ "accuracy": 0.38,
+ "macro_f1": 0.3564,
+ "ece": 0.3832,
+ "brier": 0.9572,
+ "nll": 3.235,
+ "mean_confidence": 0.7507,
+ "acc_at_50_coverage": 0.52,
+ "seconds": 9.3,
+ "dropped": 0
+ },
+ "en": {
+ "n": 100,
+ "accuracy": 0.68,
+ "macro_f1": 0.6282,
+ "ece": 0.209,
+ "brier": 0.5052,
+ "nll": 1.4497,
+ "mean_confidence": 0.8719,
+ "acc_at_50_coverage": 0.96,
+ "seconds": 8.6,
+ "dropped": 0
+ },
+ "es": {
+ "n": 100,
+ "accuracy": 0.53,
+ "macro_f1": 0.4863,
+ "ece": 0.2749,
+ "brier": 0.6896,
+ "nll": 2.0238,
+ "mean_confidence": 0.8049,
+ "acc_at_50_coverage": 0.76,
+ "seconds": 9.0,
+ "dropped": 0
+ },
+ "fa": {
+ "n": 100,
+ "accuracy": 0.39,
+ "macro_f1": 0.3751,
+ "ece": 0.3986,
+ "brier": 0.9086,
+ "nll": 2.7913,
+ "mean_confidence": 0.7886,
+ "acc_at_50_coverage": 0.6,
+ "seconds": 9.0,
+ "dropped": 0
+ },
+ "fi": {
+ "n": 100,
+ "accuracy": 0.29,
+ "macro_f1": 0.2533,
+ "ece": 0.4364,
+ "brier": 1.0851,
+ "nll": 3.7674,
+ "mean_confidence": 0.7197,
+ "acc_at_50_coverage": 0.38,
+ "seconds": 9.2,
+ "dropped": 0
+ },
+ "fr": {
+ "n": 100,
+ "accuracy": 0.54,
+ "macro_f1": 0.5199,
+ "ece": 0.277,
+ "brier": 0.6613,
+ "nll": 1.9862,
+ "mean_confidence": 0.8074,
+ "acc_at_50_coverage": 0.8,
+ "seconds": 9.1,
+ "dropped": 0
+ },
+ "he": {
+ "n": 100,
+ "accuracy": 0.4,
+ "macro_f1": 0.3794,
+ "ece": 0.3503,
+ "brier": 0.9297,
+ "nll": 2.6466,
+ "mean_confidence": 0.7302,
+ "acc_at_50_coverage": 0.5,
+ "seconds": 8.9,
+ "dropped": 0
+ },
+ "hi": {
+ "n": 100,
+ "accuracy": 0.43,
+ "macro_f1": 0.397,
+ "ece": 0.3206,
+ "brier": 0.8304,
+ "nll": 2.7293,
+ "mean_confidence": 0.7506,
+ "acc_at_50_coverage": 0.64,
+ "seconds": 8.9,
+ "dropped": 0
+ },
+ "hu": {
+ "n": 100,
+ "accuracy": 0.29,
+ "macro_f1": 0.2654,
+ "ece": 0.4221,
+ "brier": 1.041,
+ "nll": 3.5242,
+ "mean_confidence": 0.7121,
+ "acc_at_50_coverage": 0.44,
+ "seconds": 8.6,
+ "dropped": 0
+ },
+ "hy": {
+ "n": 100,
+ "accuracy": 0.15,
+ "macro_f1": 0.1298,
+ "ece": 0.5062,
+ "brier": 1.2346,
+ "nll": 4.0964,
+ "mean_confidence": 0.6469,
+ "acc_at_50_coverage": 0.22,
+ "seconds": 9.2,
+ "dropped": 0
+ },
+ "id": {
+ "n": 100,
+ "accuracy": 0.51,
+ "macro_f1": 0.4955,
+ "ece": 0.3048,
+ "brier": 0.7362,
+ "nll": 2.197,
+ "mean_confidence": 0.8028,
+ "acc_at_50_coverage": 0.72,
+ "seconds": 8.8,
+ "dropped": 0
+ },
+ "is": {
+ "n": 100,
+ "accuracy": 0.3,
+ "macro_f1": 0.2831,
+ "ece": 0.4692,
+ "brier": 1.1123,
+ "nll": 4.2693,
+ "mean_confidence": 0.7551,
+ "acc_at_50_coverage": 0.4,
+ "seconds": 8.9,
+ "dropped": 0
+ },
+ "it": {
+ "n": 100,
+ "accuracy": 0.5,
+ "macro_f1": 0.4773,
+ "ece": 0.302,
+ "brier": 0.7537,
+ "nll": 2.5765,
+ "mean_confidence": 0.7825,
+ "acc_at_50_coverage": 0.74,
+ "seconds": 8.3,
+ "dropped": 0
+ },
+ "ja": {
+ "n": 100,
+ "accuracy": 0.64,
+ "macro_f1": 0.5923,
+ "ece": 0.228,
+ "brier": 0.5661,
+ "nll": 1.5915,
+ "mean_confidence": 0.8576,
+ "acc_at_50_coverage": 0.84,
+ "seconds": 8.1,
+ "dropped": 0
+ },
+ "jv": {
+ "n": 100,
+ "accuracy": 0.27,
+ "macro_f1": 0.2348,
+ "ece": 0.5063,
+ "brier": 1.1532,
+ "nll": 3.9457,
+ "mean_confidence": 0.7588,
+ "acc_at_50_coverage": 0.34,
+ "seconds": 8.7,
+ "dropped": 0
+ },
+ "ka": {
+ "n": 100,
+ "accuracy": 0.11,
+ "macro_f1": 0.1059,
+ "ece": 0.5278,
+ "brier": 1.3013,
+ "nll": 4.4255,
+ "mean_confidence": 0.6378,
+ "acc_at_50_coverage": 0.14,
+ "seconds": 9.2,
+ "dropped": 0
+ },
+ "km": {
+ "n": 100,
+ "accuracy": 0.18,
+ "macro_f1": 0.1745,
+ "ece": 0.4123,
+ "brier": 1.1383,
+ "nll": 3.5727,
+ "mean_confidence": 0.5762,
+ "acc_at_50_coverage": 0.22,
+ "seconds": 9.7,
+ "dropped": 0
+ },
+ "kn": {
+ "n": 100,
+ "accuracy": 0.15,
+ "macro_f1": 0.1388,
+ "ece": 0.4371,
+ "brier": 1.1512,
+ "nll": 3.4801,
+ "mean_confidence": 0.5804,
+ "acc_at_50_coverage": 0.2,
+ "seconds": 8.9,
+ "dropped": 0
+ },
+ "ko": {
+ "n": 100,
+ "accuracy": 0.45,
+ "macro_f1": 0.3927,
+ "ece": 0.3293,
+ "brier": 0.8278,
+ "nll": 2.5605,
+ "mean_confidence": 0.7297,
+ "acc_at_50_coverage": 0.6,
+ "seconds": 8.5,
+ "dropped": 0
+ },
+ "lv": {
+ "n": 100,
+ "accuracy": 0.32,
+ "macro_f1": 0.3023,
+ "ece": 0.4804,
+ "brier": 1.1257,
+ "nll": 3.8958,
+ "mean_confidence": 0.7779,
+ "acc_at_50_coverage": 0.4,
+ "seconds": 8.5,
+ "dropped": 0
+ },
+ "ml": {
+ "n": 100,
+ "accuracy": 0.24,
+ "macro_f1": 0.2188,
+ "ece": 0.4142,
+ "brier": 1.1209,
+ "nll": 3.7403,
+ "mean_confidence": 0.6486,
+ "acc_at_50_coverage": 0.3,
+ "seconds": 9.1,
+ "dropped": 0
+ },
+ "mn": {
+ "n": 100,
+ "accuracy": 0.1,
+ "macro_f1": 0.0735,
+ "ece": 0.5581,
+ "brier": 1.3302,
+ "nll": 4.8227,
+ "mean_confidence": 0.6562,
+ "acc_at_50_coverage": 0.1,
+ "seconds": 8.7,
+ "dropped": 0
+ },
+ "ms": {
+ "n": 100,
+ "accuracy": 0.43,
+ "macro_f1": 0.4032,
+ "ece": 0.3924,
+ "brier": 0.916,
+ "nll": 2.9229,
+ "mean_confidence": 0.8224,
+ "acc_at_50_coverage": 0.62,
+ "seconds": 8.4,
+ "dropped": 0
+ },
+ "my": {
+ "n": 100,
+ "accuracy": 0.12,
+ "macro_f1": 0.1185,
+ "ece": 0.455,
+ "brier": 1.2102,
+ "nll": 3.8596,
+ "mean_confidence": 0.575,
+ "acc_at_50_coverage": 0.16,
+ "seconds": 9.3,
+ "dropped": 0
+ },
+ "nb": {
+ "n": 100,
+ "accuracy": 0.53,
+ "macro_f1": 0.4796,
+ "ece": 0.3273,
+ "brier": 0.7368,
+ "nll": 2.7576,
+ "mean_confidence": 0.8573,
+ "acc_at_50_coverage": 0.74,
+ "seconds": 8.1,
+ "dropped": 0
+ },
+ "nl": {
+ "n": 100,
+ "accuracy": 0.45,
+ "macro_f1": 0.3861,
+ "ece": 0.3775,
+ "brier": 0.8735,
+ "nll": 3.2134,
+ "mean_confidence": 0.8137,
+ "acc_at_50_coverage": 0.64,
+ "seconds": 8.4,
+ "dropped": 0
+ },
+ "pl": {
+ "n": 100,
+ "accuracy": 0.51,
+ "macro_f1": 0.4859,
+ "ece": 0.3501,
+ "brier": 0.7774,
+ "nll": 2.3803,
+ "mean_confidence": 0.8356,
+ "acc_at_50_coverage": 0.7,
+ "seconds": 8.2,
+ "dropped": 0
+ },
+ "pt": {
+ "n": 100,
+ "accuracy": 0.45,
+ "macro_f1": 0.4108,
+ "ece": 0.3419,
+ "brier": 0.8515,
+ "nll": 2.5313,
+ "mean_confidence": 0.7863,
+ "acc_at_50_coverage": 0.6,
+ "seconds": 8.0,
+ "dropped": 0
+ },
+ "ro": {
+ "n": 100,
+ "accuracy": 0.35,
+ "macro_f1": 0.3269,
+ "ece": 0.4035,
+ "brier": 0.9823,
+ "nll": 3.2401,
+ "mean_confidence": 0.7501,
+ "acc_at_50_coverage": 0.54,
+ "seconds": 8.0,
+ "dropped": 0
+ },
+ "ru": {
+ "n": 100,
+ "accuracy": 0.54,
+ "macro_f1": 0.4872,
+ "ece": 0.3163,
+ "brier": 0.7624,
+ "nll": 2.5212,
+ "mean_confidence": 0.8405,
+ "acc_at_50_coverage": 0.7,
+ "seconds": 8.0,
+ "dropped": 0
+ },
+ "sl": {
+ "n": 100,
+ "accuracy": 0.33,
+ "macro_f1": 0.2981,
+ "ece": 0.4335,
+ "brier": 1.0212,
+ "nll": 3.1538,
+ "mean_confidence": 0.7635,
+ "acc_at_50_coverage": 0.48,
+ "seconds": 8.1,
+ "dropped": 0
+ },
+ "sq": {
+ "n": 100,
+ "accuracy": 0.26,
+ "macro_f1": 0.2275,
+ "ece": 0.4764,
+ "brier": 1.1099,
+ "nll": 3.4562,
+ "mean_confidence": 0.7348,
+ "acc_at_50_coverage": 0.4,
+ "seconds": 8.2,
+ "dropped": 0
+ },
+ "sv": {
+ "n": 100,
+ "accuracy": 0.57,
+ "macro_f1": 0.524,
+ "ece": 0.2757,
+ "brier": 0.7141,
+ "nll": 2.3393,
+ "mean_confidence": 0.8191,
+ "acc_at_50_coverage": 0.74,
+ "seconds": 7.8,
+ "dropped": 0
+ },
+ "sw": {
+ "n": 100,
+ "accuracy": 0.18,
+ "macro_f1": 0.1741,
+ "ece": 0.5495,
+ "brier": 1.265,
+ "nll": 4.8324,
+ "mean_confidence": 0.7295,
+ "acc_at_50_coverage": 0.28,
+ "seconds": 8.1,
+ "dropped": 0
+ },
+ "ta": {
+ "n": 100,
+ "accuracy": 0.23,
+ "macro_f1": 0.1945,
+ "ece": 0.3968,
+ "brier": 1.0924,
+ "nll": 3.4722,
+ "mean_confidence": 0.6268,
+ "acc_at_50_coverage": 0.28,
+ "seconds": 8.5,
+ "dropped": 0
+ },
+ "te": {
+ "n": 100,
+ "accuracy": 0.22,
+ "macro_f1": 0.1973,
+ "ece": 0.3705,
+ "brier": 1.0643,
+ "nll": 3.348,
+ "mean_confidence": 0.5905,
+ "acc_at_50_coverage": 0.28,
+ "seconds": 8.5,
+ "dropped": 0
+ },
+ "th": {
+ "n": 100,
+ "accuracy": 0.48,
+ "macro_f1": 0.4513,
+ "ece": 0.3357,
+ "brier": 0.8119,
+ "nll": 2.5203,
+ "mean_confidence": 0.7844,
+ "acc_at_50_coverage": 0.62,
+ "seconds": 8.1,
+ "dropped": 0
+ },
+ "tl": {
+ "n": 100,
+ "accuracy": 0.36,
+ "macro_f1": 0.332,
+ "ece": 0.3738,
+ "brier": 0.9662,
+ "nll": 3.2804,
+ "mean_confidence": 0.7338,
+ "acc_at_50_coverage": 0.48,
+ "seconds": 8.1,
+ "dropped": 0
+ },
+ "tr": {
+ "n": 100,
+ "accuracy": 0.37,
+ "macro_f1": 0.3398,
+ "ece": 0.417,
+ "brier": 0.9744,
+ "nll": 3.1177,
+ "mean_confidence": 0.7784,
+ "acc_at_50_coverage": 0.5,
+ "seconds": 7.8,
+ "dropped": 0
+ },
+ "ur": {
+ "n": 100,
+ "accuracy": 0.4,
+ "macro_f1": 0.3774,
+ "ece": 0.3111,
+ "brier": 0.8551,
+ "nll": 2.7536,
+ "mean_confidence": 0.7111,
+ "acc_at_50_coverage": 0.58,
+ "seconds": 8.0,
+ "dropped": 0
+ },
+ "vi": {
+ "n": 100,
+ "accuracy": 0.26,
+ "macro_f1": 0.2238,
+ "ece": 0.5214,
+ "brier": 1.145,
+ "nll": 3.627,
+ "mean_confidence": 0.7722,
+ "acc_at_50_coverage": 0.4,
+ "seconds": 7.9,
+ "dropped": 0
+ },
+ "zh-CN": {
+ "n": 100,
+ "accuracy": 0.63,
+ "macro_f1": 0.5936,
+ "ece": 0.2123,
+ "brier": 0.5221,
+ "nll": 1.714,
+ "mean_confidence": 0.8246,
+ "acc_at_50_coverage": 0.88,
+ "seconds": 7.7,
+ "dropped": 0
+ },
+ "zh-TW": {
+ "n": 100,
+ "accuracy": 0.54,
+ "macro_f1": 0.4956,
+ "ece": 0.327,
+ "brier": 0.7166,
+ "nll": 2.0914,
+ "mean_confidence": 0.83,
+ "acc_at_50_coverage": 0.8,
+ "seconds": 7.7,
+ "dropped": 0
+ }
+ },
+ "macro_accuracy": 0.3661,
+ "macro_ece": 0.3869,
+ "n_languages": 51,
+ "languages_above_random": 45
+ }
+ }
+ },
+ "part_b": {
+ "reference_points": {
+ "jev_1.13.0_published": {
+ "accuracy": 0.727,
+ "soft_accuracy": 0.58,
+ "brier": 0.148,
+ "ece": 0.144,
+ "score_mae": 0.391,
+ "ms_per_case": 710,
+ "source": "figure quoted in the laya repo's own comparison table"
+ },
+ "teacher_self_agreement_ceiling": {
+ "accuracy": 0.735
+ },
+ "modernbert_base_specialist": {
+ "accuracy": 0.646
+ },
+ "random_guess": {
+ "accuracy": 0.3175
+ },
+ "per_question_majority_class": {
+ "accuracy": 0.461
+ },
+ "note": "Jev was NOT run here - no TypeSafe API credential is available. These are published numbers reproduced for context, not a measured head-to-head."
+ },
+ "by_model": {
+ "english": {
+ "n": 2000,
+ "accuracy": 0.3615,
+ "macro_f1": 0.3266,
+ "ece": 0.1747,
+ "brier": 0.7497,
+ "nll": 1.3227,
+ "mean_confidence": 0.5362,
+ "acc_at_50_coverage": 0.428,
+ "soft_accuracy": 0.3315,
+ "brier_vs_soft": 0.3155,
+ "score_mae": 0.6937,
+ "within_1_level": 0.7525,
+ "seconds": 557.0,
+ "dropped": 0,
+ "ms_per_case": 1392.5,
+ "by_workflow": {
+ "agent_trace_observability": {
+ "n": 500,
+ "accuracy": 0.388,
+ "macro_f1": 0.3315,
+ "ece": 0.1999,
+ "brier": 0.7477,
+ "nll": 1.3386,
+ "mean_confidence": 0.576,
+ "acc_at_50_coverage": 0.448
+ },
+ "customer_service": {
+ "n": 500,
+ "accuracy": 0.382,
+ "macro_f1": 0.3446,
+ "ece": 0.1067,
+ "brier": 0.7051,
+ "nll": 1.295,
+ "mean_confidence": 0.4783,
+ "acc_at_50_coverage": 0.52
+ },
+ "invoice_processing": {
+ "n": 500,
+ "accuracy": 0.36,
+ "macro_f1": 0.2176,
+ "ece": 0.2281,
+ "brier": 0.7889,
+ "nll": 1.3281,
+ "mean_confidence": 0.5458,
+ "acc_at_50_coverage": 0.292
+ },
+ "security_incidents": {
+ "n": 500,
+ "accuracy": 0.316,
+ "macro_f1": 0.2115,
+ "ece": 0.2287,
+ "brier": 0.757,
+ "nll": 1.3292,
+ "mean_confidence": 0.5445,
+ "acc_at_50_coverage": 0.428
+ }
+ },
+ "by_question_type": {
+ "choice": {
+ "n": 600,
+ "accuracy": 0.2883,
+ "macro_f1": 0.2958,
+ "ece": 0.1681,
+ "brier": 0.8079,
+ "nll": 1.5603,
+ "mean_confidence": 0.4527,
+ "acc_at_50_coverage": 0.3867
+ },
+ "noul": {
+ "n": 600,
+ "accuracy": 0.4867,
+ "macro_f1": 0.4681,
+ "ece": 0.1747,
+ "brier": 0.6068,
+ "nll": 0.8473,
+ "mean_confidence": 0.6454,
+ "acc_at_50_coverage": 0.43
+ },
+ "score": {
+ "n": 800,
+ "accuracy": 0.3225,
+ "macro_f1": 0.2413,
+ "ece": 0.2065,
+ "brier": 0.8131,
+ "nll": 1.5011,
+ "mean_confidence": 0.5168,
+ "acc_at_50_coverage": 0.335
+ }
+ }
+ }
+ }
+ }
+}
\ No newline at end of file
diff --git a/research/results/t4_colab_benchmark.json b/research/results/t4_colab_benchmark.json
new file mode 100644
index 00000000..ddc400a4
--- /dev/null
+++ b/research/results/t4_colab_benchmark.json
@@ -0,0 +1,4425 @@
+{
+ "meta": {
+ "seed": 13,
+ "n_opts": 20,
+ "per_lang": 300,
+ "timestamp": "2026-09-19 04:45:43",
+ "gpu": "Tesla T4",
+ "torch": "2.11.0+cu128",
+ "transformers": "5.17.0",
+ "models": {
+ "laya": {
+ "repo": "convaiinnovations/laya",
+ "params_total_m": 421.29,
+ "params_encoder_m": 394.78,
+ "params_head_m": 26.51,
+ "hidden_size": 1024,
+ "num_layers": 28,
+ "vocab_size": 50368,
+ "max_len": 512,
+ "head_max_len": 192,
+ "temperature": [
+ 1.6369030475616455,
+ 1.2514300346374512,
+ 1.983399510383606
+ ],
+ "temperature_by_options": {
+ "choice:3-5": 1.7601518630981445,
+ "choice:6-10": 1.0000158548355103,
+ "score:3-5": 1.2514300346374512,
+ "noul:2": 1.983399510383606,
+ "choice:11+": 0.10058280825614929,
+ "choice:2": 1.9063563346862793
+ },
+ "training": {
+ "updates": 7313,
+ "epochs_completed": 1,
+ "hours": 1.96,
+ "world_size": 1,
+ "fine_tuned_from_checkpoint": true
+ },
+ "download_s": 10.7,
+ "tokenizer_config_patched": []
+ },
+ "laya-multilingual": {
+ "repo": "convaiinnovations/laya-multilingual",
+ "params_total_m": 321.91,
+ "params_encoder_m": 306.94,
+ "params_head_m": 14.97,
+ "hidden_size": 768,
+ "num_layers": 22,
+ "vocab_size": 256000,
+ "max_len": 1024,
+ "head_max_len": 256,
+ "temperature": [
+ 1.0,
+ 1.0,
+ 1.0
+ ],
+ "temperature_by_options": {},
+ "training": {
+ "updates": 15987,
+ "epochs_completed": 4,
+ "hours": 4.97,
+ "world_size": 1,
+ "fine_tuned_from_checkpoint": false
+ },
+ "download_s": 8.3,
+ "tokenizer_config_patched": [
+ "extra_special_tokens list->dict"
+ ]
+ }
+ }
+ },
+ "suites": {
+ "typed_decisions": {
+ "laya": {
+ "calibrated": {
+ "n": 2000,
+ "accuracy": 0.362,
+ "macro_f1": 0.3273577076400338,
+ "ece": 0.17410090455022942,
+ "brier": 0.7496143221875959,
+ "nll": 1.3224634387781617,
+ "aurc": 0.5928492917684455,
+ "mean_confidence": 0.5361009045502294,
+ "acc_at_50_coverage": 0.427,
+ "acc_at_80_coverage": 0.386875
+ },
+ "raw": {
+ "n": 2000,
+ "accuracy": 0.362,
+ "macro_f1": 0.3273577076400338,
+ "ece": 0.2648801700275417,
+ "brier": 0.8277820380915699,
+ "nll": 1.53173039009466,
+ "aurc": 0.5998599200683264,
+ "mean_confidence": 0.6268801700275417,
+ "acc_at_50_coverage": 0.425,
+ "acc_at_80_coverage": 0.384375
+ },
+ "seconds": 36.2,
+ "dropped_questions": 0,
+ "questions_per_second": 55.2,
+ "soft_acc": 0.3314854404264519,
+ "brier_soft": 0.31543323975432846,
+ "tv": 0.40975100160837324,
+ "kl": 0.5667172541116379,
+ "score_mae": 0.6936800032000978,
+ "within_1": 0.7525,
+ "by_question_type": {
+ "choice": {
+ "n": 600,
+ "accuracy": 0.29,
+ "macro_f1": 0.2976158635239495,
+ "ece": 0.16639379710456426,
+ "brier": 0.8078685799684945,
+ "nll": 1.5601982379145576,
+ "aurc": 0.6165177886981781,
+ "mean_confidence": 0.45269924967840924,
+ "acc_at_50_coverage": 0.38666666666666666,
+ "acc_at_80_coverage": 0.32708333333333334
+ },
+ "score": {
+ "n": 800,
+ "accuracy": 0.3225,
+ "macro_f1": 0.24129818605373515,
+ "ece": 0.2047018946740388,
+ "brier": 0.8130431610474794,
+ "nll": 1.5009182409941182,
+ "aurc": 0.6560815223692904,
+ "mean_confidence": 0.5166723088589777,
+ "acc_at_50_coverage": 0.335,
+ "acc_at_80_coverage": 0.3109375
+ },
+ "noul": {
+ "n": 600,
+ "accuracy": 0.4866666666666667,
+ "macro_f1": 0.46813409467673606,
+ "ece": 0.17554133251857967,
+ "brier": 0.6067882792601861,
+ "nll": 0.8467889033538241,
+ "aurc": 0.5565114121040284,
+ "mean_confidence": 0.6454073536770518,
+ "acc_at_50_coverage": 0.43,
+ "acc_at_80_coverage": 0.49583333333333335
+ }
+ }
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 2000,
+ "accuracy": 0.3515,
+ "macro_f1": 0.34975321170122525,
+ "ece": 0.31443177617478313,
+ "brier": 0.8904323210391285,
+ "nll": 1.8242712912901595,
+ "aurc": 0.5562863247537705,
+ "mean_confidence": 0.6629025476146312,
+ "acc_at_50_coverage": 0.414,
+ "acc_at_80_coverage": 0.368125
+ },
+ "raw": {
+ "n": 2000,
+ "accuracy": 0.3515,
+ "macro_f1": 0.34975321170122525,
+ "ece": 0.31443177617478313,
+ "brier": 0.8904323210391285,
+ "nll": 1.8242712912901595,
+ "aurc": 0.5562863247537705,
+ "mean_confidence": 0.6629025476146312,
+ "acc_at_50_coverage": 0.414,
+ "acc_at_80_coverage": 0.368125
+ },
+ "seconds": 19.35,
+ "dropped_questions": 0,
+ "questions_per_second": 103.4,
+ "soft_acc": 0.3280210927253606,
+ "brier_soft": 0.462641406109187,
+ "tv": 0.46852647440176376,
+ "kl": 1.0768631881828523,
+ "score_mae": 0.7604419071688604,
+ "within_1": 0.7,
+ "by_question_type": {
+ "choice": {
+ "n": 600,
+ "accuracy": 0.295,
+ "macro_f1": 0.3242803546952691,
+ "ece": 0.36240144724213347,
+ "brier": 0.9885012541895022,
+ "nll": 2.3854267310642197,
+ "aurc": 0.5947836135918948,
+ "mean_confidence": 0.6559877759584947,
+ "acc_at_50_coverage": 0.32,
+ "acc_at_80_coverage": 0.28541666666666665
+ },
+ "score": {
+ "n": 800,
+ "accuracy": 0.285,
+ "macro_f1": 0.2545688462591573,
+ "ece": 0.2324908307616596,
+ "brier": 0.8672059845074727,
+ "nll": 1.7060067133863015,
+ "aurc": 0.7180285302053313,
+ "mean_confidence": 0.5109780128240088,
+ "acc_at_50_coverage": 0.26,
+ "acc_at_80_coverage": 0.2765625
+ },
+ "noul": {
+ "n": 600,
+ "accuracy": 0.49666666666666665,
+ "macro_f1": 0.45384837962962965,
+ "ece": 0.37571669899159743,
+ "brier": 0.8233318365976293,
+ "nll": 1.4208019553879097,
+ "aurc": 0.48235294849361254,
+ "mean_confidence": 0.872383365658264,
+ "acc_at_50_coverage": 0.47,
+ "acc_at_80_coverage": 0.4979166666666667
+ }
+ }
+ }
+ },
+ "massive_intent.en": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.7833333333333333,
+ "macro_f1": 0.7859904974658811,
+ "ece": 0.2142657822979579,
+ "brier": 0.4285638518597769,
+ "nll": 4.783053086201328,
+ "aurc": 0.10066152754017459,
+ "mean_confidence": 0.994229890548954,
+ "acc_at_50_coverage": 0.9066666666666666,
+ "acc_at_80_coverage": 0.8916666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.7833333333333333,
+ "macro_f1": 0.7859904974658811,
+ "ece": 0.12625541044799943,
+ "brier": 0.3323165550840956,
+ "nll": 1.040553004538537,
+ "aurc": 0.0889005283504489,
+ "mean_confidence": 0.9066535593761121,
+ "acc_at_50_coverage": 0.94,
+ "acc_at_80_coverage": 0.8875
+ },
+ "seconds": 3.45,
+ "dropped_questions": 0,
+ "questions_per_second": 87.1
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.6566666666666666,
+ "macro_f1": 0.6613789797689034,
+ "ece": 0.2366905012718295,
+ "brier": 0.5754824338960343,
+ "nll": 2.092763427064851,
+ "aurc": 0.17723907538979836,
+ "mean_confidence": 0.8847088454845109,
+ "acc_at_50_coverage": 0.86,
+ "acc_at_80_coverage": 0.7291666666666666
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.6566666666666666,
+ "macro_f1": 0.6613789797689034,
+ "ece": 0.2366905012718295,
+ "brier": 0.5754824338960343,
+ "nll": 2.092763427064851,
+ "aurc": 0.17723907538979836,
+ "mean_confidence": 0.8847088454845109,
+ "acc_at_50_coverage": 0.86,
+ "acc_at_80_coverage": 0.7291666666666666
+ },
+ "seconds": 1.95,
+ "dropped_questions": 0,
+ "questions_per_second": 153.9
+ }
+ },
+ "massive_intent.de": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.42,
+ "macro_f1": 0.41942609239230483,
+ "ece": 0.5572355019418889,
+ "brier": 1.1276064052483479,
+ "nll": 13.457262315998419,
+ "aurc": 0.40545585846660126,
+ "mean_confidence": 0.9772355019418888,
+ "acc_at_50_coverage": 0.62,
+ "acc_at_80_coverage": 0.475
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.42,
+ "macro_f1": 0.41942609239230483,
+ "ece": 0.32121714127531525,
+ "brier": 0.8544067217522678,
+ "nll": 2.744549528377823,
+ "aurc": 0.35925213513126003,
+ "mean_confidence": 0.7358504676886946,
+ "acc_at_50_coverage": 0.6266666666666667,
+ "acc_at_80_coverage": 0.4791666666666667
+ },
+ "seconds": 3.61,
+ "dropped_questions": 0,
+ "questions_per_second": 83.2
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.4666666666666667,
+ "macro_f1": 0.4664031848395659,
+ "ece": 0.3671645464311559,
+ "brier": 0.849063341128718,
+ "nll": 2.793117284229578,
+ "aurc": 0.3189612559140324,
+ "mean_confidence": 0.8284870971670211,
+ "acc_at_50_coverage": 0.68,
+ "acc_at_80_coverage": 0.5291666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.4666666666666667,
+ "macro_f1": 0.4664031848395659,
+ "ece": 0.3671645464311559,
+ "brier": 0.849063341128718,
+ "nll": 2.793117284229578,
+ "aurc": 0.3189612559140324,
+ "mean_confidence": 0.8284870971670211,
+ "acc_at_50_coverage": 0.68,
+ "acc_at_80_coverage": 0.5291666666666667
+ },
+ "seconds": 2.0,
+ "dropped_questions": 0,
+ "questions_per_second": 150.1
+ }
+ },
+ "massive_intent.fr": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.4866666666666667,
+ "macro_f1": 0.49028575480881303,
+ "ece": 0.4903791740210341,
+ "brier": 0.9864560461649728,
+ "nll": 11.056989529764705,
+ "aurc": 0.3663541633047382,
+ "mean_confidence": 0.9770458406877007,
+ "acc_at_50_coverage": 0.6733333333333333,
+ "acc_at_80_coverage": 0.5583333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.4866666666666667,
+ "macro_f1": 0.49028575480881303,
+ "ece": 0.2933044954853109,
+ "brier": 0.7463862431004725,
+ "nll": 2.0693345449320564,
+ "aurc": 0.2894619278805127,
+ "mean_confidence": 0.7750594706553864,
+ "acc_at_50_coverage": 0.6866666666666666,
+ "acc_at_80_coverage": 0.5625
+ },
+ "seconds": 3.64,
+ "dropped_questions": 0,
+ "questions_per_second": 82.4
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.5533333333333333,
+ "macro_f1": 0.5512106277128809,
+ "ece": 0.2782850354212724,
+ "brier": 0.6978064231902851,
+ "nll": 2.575724948009992,
+ "aurc": 0.29284529217235095,
+ "mean_confidence": 0.8273814788760117,
+ "acc_at_50_coverage": 0.74,
+ "acc_at_80_coverage": 0.6416666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.5533333333333333,
+ "macro_f1": 0.5512106277128809,
+ "ece": 0.2782850354212724,
+ "brier": 0.6978064231902851,
+ "nll": 2.575724948009992,
+ "aurc": 0.29284529217235095,
+ "mean_confidence": 0.8273814788760117,
+ "acc_at_50_coverage": 0.74,
+ "acc_at_80_coverage": 0.6416666666666667
+ },
+ "seconds": 2.03,
+ "dropped_questions": 0,
+ "questions_per_second": 148.0
+ }
+ },
+ "massive_intent.es": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.48,
+ "macro_f1": 0.47803545309485507,
+ "ece": 0.5070839008158101,
+ "brier": 1.0169852227862164,
+ "nll": 12.023140744919116,
+ "aurc": 0.41148437275049987,
+ "mean_confidence": 0.97951395435739,
+ "acc_at_50_coverage": 0.6333333333333333,
+ "acc_at_80_coverage": 0.5291666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.48,
+ "macro_f1": 0.47803545309485507,
+ "ece": 0.3130614775019185,
+ "brier": 0.8095850837679279,
+ "nll": 2.5207261343558645,
+ "aurc": 0.34546548311114267,
+ "mean_confidence": 0.7758544205302613,
+ "acc_at_50_coverage": 0.64,
+ "acc_at_80_coverage": 0.525
+ },
+ "seconds": 3.59,
+ "dropped_questions": 0,
+ "questions_per_second": 83.5
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.5033333333333333,
+ "macro_f1": 0.49673783269846206,
+ "ece": 0.3057029621982169,
+ "brier": 0.7731615819701524,
+ "nll": 2.464950590788325,
+ "aurc": 0.3298720500466257,
+ "mean_confidence": 0.8090362955315501,
+ "acc_at_50_coverage": 0.6933333333333334,
+ "acc_at_80_coverage": 0.5583333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.5033333333333333,
+ "macro_f1": 0.49673783269846206,
+ "ece": 0.3057029621982169,
+ "brier": 0.7731615819701524,
+ "nll": 2.464950590788325,
+ "aurc": 0.3298720500466257,
+ "mean_confidence": 0.8090362955315501,
+ "acc_at_50_coverage": 0.6933333333333334,
+ "acc_at_80_coverage": 0.5583333333333333
+ },
+ "seconds": 2.02,
+ "dropped_questions": 0,
+ "questions_per_second": 148.2
+ }
+ },
+ "massive_intent.pt": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.45666666666666667,
+ "macro_f1": 0.4503225203161018,
+ "ece": 0.5258154421900451,
+ "brier": 1.0615125009722581,
+ "nll": 12.57833633824028,
+ "aurc": 0.38008688374084976,
+ "mean_confidence": 0.9824821088567119,
+ "acc_at_50_coverage": 0.6533333333333333,
+ "acc_at_80_coverage": 0.5291666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.45666666666666667,
+ "macro_f1": 0.4503225203161018,
+ "ece": 0.3446569932959994,
+ "brier": 0.8274315021487898,
+ "nll": 2.762952285380389,
+ "aurc": 0.345993489129298,
+ "mean_confidence": 0.7959313580649547,
+ "acc_at_50_coverage": 0.64,
+ "acc_at_80_coverage": 0.5416666666666666
+ },
+ "seconds": 3.68,
+ "dropped_questions": 0,
+ "questions_per_second": 81.5
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.49333333333333335,
+ "macro_f1": 0.4876915658908163,
+ "ece": 0.34213246762505234,
+ "brier": 0.8232388037791051,
+ "nll": 2.8230107127485624,
+ "aurc": 0.31534173225577855,
+ "mean_confidence": 0.8248538216832458,
+ "acc_at_50_coverage": 0.66,
+ "acc_at_80_coverage": 0.5625
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.49333333333333335,
+ "macro_f1": 0.4876915658908163,
+ "ece": 0.34213246762505234,
+ "brier": 0.8232388037791051,
+ "nll": 2.8230107127485624,
+ "aurc": 0.31534173225577855,
+ "mean_confidence": 0.8248538216832458,
+ "acc_at_50_coverage": 0.66,
+ "acc_at_80_coverage": 0.5625
+ },
+ "seconds": 2.02,
+ "dropped_questions": 0,
+ "questions_per_second": 148.2
+ }
+ },
+ "massive_intent.ru": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.29333333333333333,
+ "macro_f1": 0.29312146555618207,
+ "ece": 0.6849768215676151,
+ "brier": 1.3819963575438161,
+ "nll": 17.166320983364137,
+ "aurc": 0.6050616643967031,
+ "mean_confidence": 0.9783101549009484,
+ "acc_at_50_coverage": 0.41333333333333333,
+ "acc_at_80_coverage": 0.3458333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.29333333333333333,
+ "macro_f1": 0.29312146555618207,
+ "ece": 0.44761075495748537,
+ "brier": 1.0753941894333248,
+ "nll": 3.5739178816102837,
+ "aurc": 0.5521122930922301,
+ "mean_confidence": 0.7409440882908187,
+ "acc_at_50_coverage": 0.41333333333333333,
+ "acc_at_80_coverage": 0.35
+ },
+ "seconds": 3.89,
+ "dropped_questions": 0,
+ "questions_per_second": 77.1
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.5,
+ "macro_f1": 0.5061202589086473,
+ "ece": 0.3233476303888164,
+ "brier": 0.7951925984574719,
+ "nll": 2.6843260934306636,
+ "aurc": 0.3438499231303233,
+ "mean_confidence": 0.8233476303888164,
+ "acc_at_50_coverage": 0.66,
+ "acc_at_80_coverage": 0.5708333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.5,
+ "macro_f1": 0.5061202589086473,
+ "ece": 0.3233476303888164,
+ "brier": 0.7951925984574719,
+ "nll": 2.6843260934306636,
+ "aurc": 0.3438499231303233,
+ "mean_confidence": 0.8233476303888164,
+ "acc_at_50_coverage": 0.66,
+ "acc_at_80_coverage": 0.5708333333333333
+ },
+ "seconds": 2.02,
+ "dropped_questions": 0,
+ "questions_per_second": 148.6
+ }
+ },
+ "massive_intent.tr": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.16,
+ "macro_f1": 0.1486039563594193,
+ "ece": 0.7897886355179764,
+ "brier": 1.603631422859012,
+ "nll": 20.466186122493088,
+ "aurc": 0.746369008854861,
+ "mean_confidence": 0.9497886355179764,
+ "acc_at_50_coverage": 0.24,
+ "acc_at_80_coverage": 0.1875
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.16,
+ "macro_f1": 0.1486039563594193,
+ "ece": 0.4083382750097803,
+ "brier": 1.1535723588235036,
+ "nll": 4.133708635893034,
+ "aurc": 0.6850322055704485,
+ "mean_confidence": 0.5658787393235634,
+ "acc_at_50_coverage": 0.23333333333333334,
+ "acc_at_80_coverage": 0.1875
+ },
+ "seconds": 3.87,
+ "dropped_questions": 0,
+ "questions_per_second": 77.5
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.43666666666666665,
+ "macro_f1": 0.4349967052180575,
+ "ece": 0.35706493438546416,
+ "brier": 0.9040762205611305,
+ "nll": 3.0042063806753494,
+ "aurc": 0.3818364137902104,
+ "mean_confidence": 0.793731601052131,
+ "acc_at_50_coverage": 0.6,
+ "acc_at_80_coverage": 0.49166666666666664
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.43666666666666665,
+ "macro_f1": 0.4349967052180575,
+ "ece": 0.35706493438546416,
+ "brier": 0.9040762205611305,
+ "nll": 3.0042063806753494,
+ "aurc": 0.3818364137902104,
+ "mean_confidence": 0.793731601052131,
+ "acc_at_50_coverage": 0.6,
+ "acc_at_80_coverage": 0.49166666666666664
+ },
+ "seconds": 2.0,
+ "dropped_questions": 0,
+ "questions_per_second": 150.1
+ }
+ },
+ "massive_intent.ar": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.12666666666666668,
+ "macro_f1": 0.1256503611446384,
+ "ece": 0.798641770944787,
+ "brier": 1.6462925546066582,
+ "nll": 20.856901740883874,
+ "aurc": 0.8392220838559886,
+ "mean_confidence": 0.9253084376114538,
+ "acc_at_50_coverage": 0.16666666666666666,
+ "acc_at_80_coverage": 0.12916666666666668
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.12666666666666668,
+ "macro_f1": 0.1256503611446384,
+ "ece": 0.39476726769861925,
+ "brier": 1.177780937942809,
+ "nll": 4.090553130199672,
+ "aurc": 0.8014559695763626,
+ "mean_confidence": 0.5214339343652858,
+ "acc_at_50_coverage": 0.16,
+ "acc_at_80_coverage": 0.1375
+ },
+ "seconds": 3.99,
+ "dropped_questions": 0,
+ "questions_per_second": 75.1
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.38333333333333336,
+ "macro_f1": 0.3773237673233435,
+ "ece": 0.3670310695379155,
+ "brier": 0.9436406092973436,
+ "nll": 3.0610236443302123,
+ "aurc": 0.4345891218357328,
+ "mean_confidence": 0.7489822251667393,
+ "acc_at_50_coverage": 0.5333333333333333,
+ "acc_at_80_coverage": 0.44583333333333336
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.38333333333333336,
+ "macro_f1": 0.3773237673233435,
+ "ece": 0.3670310695379155,
+ "brier": 0.9436406092973436,
+ "nll": 3.0610236443302123,
+ "aurc": 0.4345891218357328,
+ "mean_confidence": 0.7489822251667393,
+ "acc_at_50_coverage": 0.5333333333333333,
+ "acc_at_80_coverage": 0.44583333333333336
+ },
+ "seconds": 1.96,
+ "dropped_questions": 0,
+ "questions_per_second": 153.1
+ }
+ },
+ "massive_intent.hi": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.1,
+ "macro_f1": 0.0953787060033515,
+ "ece": 0.8431830079618857,
+ "brier": 1.724777262836915,
+ "nll": 21.299563769404223,
+ "aurc": 0.904048278658199,
+ "mean_confidence": 0.9431830079618855,
+ "acc_at_50_coverage": 0.10666666666666667,
+ "acc_at_80_coverage": 0.1125
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.1,
+ "macro_f1": 0.0953787060033515,
+ "ece": 0.4348448694638475,
+ "brier": 1.2351111778950616,
+ "nll": 4.272234684416419,
+ "aurc": 0.8813871989083324,
+ "mean_confidence": 0.5315443656017397,
+ "acc_at_50_coverage": 0.1,
+ "acc_at_80_coverage": 0.10833333333333334
+ },
+ "seconds": 4.34,
+ "dropped_questions": 0,
+ "questions_per_second": 69.2
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.38666666666666666,
+ "macro_f1": 0.3799765136142003,
+ "ece": 0.39705202860146693,
+ "brier": 0.966182445618435,
+ "nll": 3.1871501391728065,
+ "aurc": 0.4557389301411769,
+ "mean_confidence": 0.7817389664743709,
+ "acc_at_50_coverage": 0.52,
+ "acc_at_80_coverage": 0.44583333333333336
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.38666666666666666,
+ "macro_f1": 0.3799765136142003,
+ "ece": 0.39705202860146693,
+ "brier": 0.966182445618435,
+ "nll": 3.1871501391728065,
+ "aurc": 0.4557389301411769,
+ "mean_confidence": 0.7817389664743709,
+ "acc_at_50_coverage": 0.52,
+ "acc_at_80_coverage": 0.44583333333333336
+ },
+ "seconds": 1.98,
+ "dropped_questions": 0,
+ "questions_per_second": 151.2
+ }
+ },
+ "massive_intent.ta": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.11333333333333333,
+ "macro_f1": 0.10519330875322393,
+ "ece": 0.8241359277747448,
+ "brier": 1.6826711980977516,
+ "nll": 19.965294983018907,
+ "aurc": 0.8536231181038471,
+ "mean_confidence": 0.935663685936003,
+ "acc_at_50_coverage": 0.12666666666666668,
+ "acc_at_80_coverage": 0.12916666666666668
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.11333333333333333,
+ "macro_f1": 0.10519330875322393,
+ "ece": 0.31129690461029336,
+ "brier": 1.1128552391640942,
+ "nll": 3.7176514878944316,
+ "aurc": 0.8557360412466305,
+ "mean_confidence": 0.4246302379436267,
+ "acc_at_50_coverage": 0.12666666666666668,
+ "acc_at_80_coverage": 0.12916666666666668
+ },
+ "seconds": 4.85,
+ "dropped_questions": 0,
+ "questions_per_second": 61.9
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.25,
+ "macro_f1": 0.2475709870276944,
+ "ece": 0.48455217697434594,
+ "brier": 1.1784996530243754,
+ "nll": 3.996754995032367,
+ "aurc": 0.6777780939794569,
+ "mean_confidence": 0.7345521769743459,
+ "acc_at_50_coverage": 0.32,
+ "acc_at_80_coverage": 0.275
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.25,
+ "macro_f1": 0.2475709870276944,
+ "ece": 0.48455217697434594,
+ "brier": 1.1784996530243754,
+ "nll": 3.996754995032367,
+ "aurc": 0.6777780939794569,
+ "mean_confidence": 0.7345521769743459,
+ "acc_at_50_coverage": 0.32,
+ "acc_at_80_coverage": 0.275
+ },
+ "seconds": 2.01,
+ "dropped_questions": 0,
+ "questions_per_second": 149.3
+ }
+ },
+ "massive_intent.zh-CN": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.5933333333333334,
+ "macro_f1": 0.5964552527791429,
+ "ece": 0.39916859064752297,
+ "brier": 0.80660057431863,
+ "nll": 9.704434459218685,
+ "aurc": 0.24511533386115847,
+ "mean_confidence": 0.9889531214305413,
+ "acc_at_50_coverage": 0.7733333333333333,
+ "acc_at_80_coverage": 0.6666666666666666
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.5933333333333334,
+ "macro_f1": 0.5964552527791429,
+ "ece": 0.24586806288232513,
+ "brier": 0.6523252570127208,
+ "nll": 2.0586656002191677,
+ "aurc": 0.2229554358525133,
+ "mean_confidence": 0.8328217574294239,
+ "acc_at_50_coverage": 0.7666666666666667,
+ "acc_at_80_coverage": 0.6708333333333333
+ },
+ "seconds": 4.01,
+ "dropped_questions": 0,
+ "questions_per_second": 74.7
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.6066666666666667,
+ "macro_f1": 0.6066977290299708,
+ "ece": 0.2624488813566047,
+ "brier": 0.6395126198165556,
+ "nll": 2.057482622773751,
+ "aurc": 0.20966586558217612,
+ "mean_confidence": 0.8679142837146541,
+ "acc_at_50_coverage": 0.84,
+ "acc_at_80_coverage": 0.6958333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.6066666666666667,
+ "macro_f1": 0.6066977290299708,
+ "ece": 0.2624488813566047,
+ "brier": 0.6395126198165556,
+ "nll": 2.057482622773751,
+ "aurc": 0.20966586558217612,
+ "mean_confidence": 0.8679142837146541,
+ "acc_at_50_coverage": 0.84,
+ "acc_at_80_coverage": 0.6958333333333333
+ },
+ "seconds": 1.92,
+ "dropped_questions": 0,
+ "questions_per_second": 155.9
+ }
+ },
+ "massive_intent.ja": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.5466666666666666,
+ "macro_f1": 0.5505137504336817,
+ "ece": 0.43745213971398433,
+ "brier": 0.8910538194571062,
+ "nll": 10.610208949690762,
+ "aurc": 0.27361814455380085,
+ "mean_confidence": 0.9841188063806509,
+ "acc_at_50_coverage": 0.74,
+ "acc_at_80_coverage": 0.6
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.5466666666666666,
+ "macro_f1": 0.5505137504336817,
+ "ece": 0.23469877865675798,
+ "brier": 0.6962990571747073,
+ "nll": 2.1285255990386305,
+ "aurc": 0.26818040251979264,
+ "mean_confidence": 0.7813654453234246,
+ "acc_at_50_coverage": 0.7533333333333333,
+ "acc_at_80_coverage": 0.625
+ },
+ "seconds": 4.08,
+ "dropped_questions": 0,
+ "questions_per_second": 73.5
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.5866666666666667,
+ "macro_f1": 0.5789792898949173,
+ "ece": 0.26880170740133336,
+ "brier": 0.6641315368078033,
+ "nll": 2.1563007642554584,
+ "aurc": 0.2606575513014464,
+ "mean_confidence": 0.8537674763577928,
+ "acc_at_50_coverage": 0.7666666666666667,
+ "acc_at_80_coverage": 0.6541666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.5866666666666667,
+ "macro_f1": 0.5789792898949173,
+ "ece": 0.26880170740133336,
+ "brier": 0.6641315368078033,
+ "nll": 2.1563007642554584,
+ "aurc": 0.2606575513014464,
+ "mean_confidence": 0.8537674763577928,
+ "acc_at_50_coverage": 0.7666666666666667,
+ "acc_at_80_coverage": 0.6541666666666667
+ },
+ "seconds": 1.91,
+ "dropped_questions": 0,
+ "questions_per_second": 157.0
+ }
+ },
+ "massive_intent.ko": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.10333333333333333,
+ "macro_f1": 0.09534820721043308,
+ "ece": 0.8496262680408053,
+ "brier": 1.718330722538468,
+ "nll": 21.741823832218525,
+ "aurc": 0.8696438316894232,
+ "mean_confidence": 0.9529596013741385,
+ "acc_at_50_coverage": 0.13333333333333333,
+ "acc_at_80_coverage": 0.12083333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.10333333333333333,
+ "macro_f1": 0.09534820721043308,
+ "ece": 0.45471327863474253,
+ "brier": 1.236188621010311,
+ "nll": 4.2678925994892944,
+ "aurc": 0.8530241092992829,
+ "mean_confidence": 0.5580466119680759,
+ "acc_at_50_coverage": 0.14,
+ "acc_at_80_coverage": 0.12083333333333333
+ },
+ "seconds": 4.4,
+ "dropped_questions": 0,
+ "questions_per_second": 68.1
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.49,
+ "macro_f1": 0.488708216378764,
+ "ece": 0.3084237491405182,
+ "brier": 0.7840216027433781,
+ "nll": 2.48264831603104,
+ "aurc": 0.3197043609942588,
+ "mean_confidence": 0.7984237491405181,
+ "acc_at_50_coverage": 0.68,
+ "acc_at_80_coverage": 0.5833333333333334
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.49,
+ "macro_f1": 0.488708216378764,
+ "ece": 0.3084237491405182,
+ "brier": 0.7840216027433781,
+ "nll": 2.48264831603104,
+ "aurc": 0.3197043609942588,
+ "mean_confidence": 0.7984237491405181,
+ "acc_at_50_coverage": 0.68,
+ "acc_at_80_coverage": 0.5833333333333334
+ },
+ "seconds": 1.93,
+ "dropped_questions": 0,
+ "questions_per_second": 155.2
+ }
+ },
+ "massive_intent.sw": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.10333333333333333,
+ "macro_f1": 0.09566516861510858,
+ "ece": 0.8512619986530077,
+ "brier": 1.7306965908645546,
+ "nll": 21.490747833569763,
+ "aurc": 0.8578274070062489,
+ "mean_confidence": 0.9539542441153954,
+ "acc_at_50_coverage": 0.12666666666666668,
+ "acc_at_80_coverage": 0.10833333333333334
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.10333333333333333,
+ "macro_f1": 0.09566516861510858,
+ "ece": 0.3882574858223093,
+ "brier": 1.17321258150963,
+ "nll": 4.096486897113687,
+ "aurc": 0.8556569352727932,
+ "mean_confidence": 0.49159081915564257,
+ "acc_at_50_coverage": 0.12,
+ "acc_at_80_coverage": 0.1125
+ },
+ "seconds": 4.2,
+ "dropped_questions": 0,
+ "questions_per_second": 71.5
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.21,
+ "macro_f1": 0.21000860513059322,
+ "ece": 0.4905093053428704,
+ "brier": 1.1966822466692135,
+ "nll": 4.229903642570186,
+ "aurc": 0.6159026094360324,
+ "mean_confidence": 0.7005093053428704,
+ "acc_at_50_coverage": 0.30666666666666664,
+ "acc_at_80_coverage": 0.24166666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.21,
+ "macro_f1": 0.21000860513059322,
+ "ece": 0.4905093053428704,
+ "brier": 1.1966822466692135,
+ "nll": 4.229903642570186,
+ "aurc": 0.6159026094360324,
+ "mean_confidence": 0.7005093053428704,
+ "acc_at_50_coverage": 0.30666666666666664,
+ "acc_at_80_coverage": 0.24166666666666667
+ },
+ "seconds": 1.92,
+ "dropped_questions": 0,
+ "questions_per_second": 155.9
+ }
+ },
+ "massive_scenario.en": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.6033333333333334,
+ "macro_f1": 0.5944362769884591,
+ "ece": 0.3927976732517713,
+ "brier": 0.7882183191086455,
+ "nll": 9.781255719005527,
+ "aurc": 0.31821612105676933,
+ "mean_confidence": 0.9926578571022835,
+ "acc_at_50_coverage": 0.7133333333333334,
+ "acc_at_80_coverage": 0.6375
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.6033333333333334,
+ "macro_f1": 0.5944362769884591,
+ "ece": 0.2951358132155798,
+ "brier": 0.6844123055005451,
+ "nll": 2.0128865847342325,
+ "aurc": 0.19393035010708404,
+ "mean_confidence": 0.8966549050977662,
+ "acc_at_50_coverage": 0.7933333333333333,
+ "acc_at_80_coverage": 0.6541666666666667
+ },
+ "seconds": 2.26,
+ "dropped_questions": 0,
+ "questions_per_second": 132.6
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.56,
+ "macro_f1": 0.5547673096787787,
+ "ece": 0.3073935231585113,
+ "brier": 0.7005044027556065,
+ "nll": 2.445474960645575,
+ "aurc": 0.17889742521042204,
+ "mean_confidence": 0.8673935231585114,
+ "acc_at_50_coverage": 0.8466666666666667,
+ "acc_at_80_coverage": 0.6708333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.56,
+ "macro_f1": 0.5547673096787787,
+ "ece": 0.3073935231585113,
+ "brier": 0.7005044027556065,
+ "nll": 2.445474960645575,
+ "aurc": 0.17889742521042204,
+ "mean_confidence": 0.8673935231585114,
+ "acc_at_50_coverage": 0.8466666666666667,
+ "acc_at_80_coverage": 0.6708333333333333
+ },
+ "seconds": 0.93,
+ "dropped_questions": 0,
+ "questions_per_second": 322.3
+ }
+ },
+ "massive_scenario.de": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.2966666666666667,
+ "macro_f1": 0.2877606145118102,
+ "ece": 0.6683327369921522,
+ "brier": 1.3641775508951617,
+ "nll": 16.096091375368577,
+ "aurc": 0.5605746102500764,
+ "mean_confidence": 0.9649994036588188,
+ "acc_at_50_coverage": 0.41333333333333333,
+ "acc_at_80_coverage": 0.32083333333333336
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.2966666666666667,
+ "macro_f1": 0.2877606145118102,
+ "ece": 0.42285314991027423,
+ "brier": 1.0454422585082612,
+ "nll": 3.1653200488989492,
+ "aurc": 0.5145568812300169,
+ "mean_confidence": 0.7195198165769409,
+ "acc_at_50_coverage": 0.42,
+ "acc_at_80_coverage": 0.32083333333333336
+ },
+ "seconds": 2.43,
+ "dropped_questions": 0,
+ "questions_per_second": 123.3
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.4766666666666667,
+ "macro_f1": 0.4659465881614311,
+ "ece": 0.3621706701744643,
+ "brier": 0.8598210325231435,
+ "nll": 2.7897092657552736,
+ "aurc": 0.33911404853855776,
+ "mean_confidence": 0.8388373368411309,
+ "acc_at_50_coverage": 0.68,
+ "acc_at_80_coverage": 0.5291666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.4766666666666667,
+ "macro_f1": 0.4659465881614311,
+ "ece": 0.3621706701744643,
+ "brier": 0.8598210325231435,
+ "nll": 2.7897092657552736,
+ "aurc": 0.33911404853855776,
+ "mean_confidence": 0.8388373368411309,
+ "acc_at_50_coverage": 0.68,
+ "acc_at_80_coverage": 0.5291666666666667
+ },
+ "seconds": 0.95,
+ "dropped_questions": 0,
+ "questions_per_second": 314.7
+ }
+ },
+ "massive_scenario.fr": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.46,
+ "macro_f1": 0.4519283547157385,
+ "ece": 0.5235926580481265,
+ "brier": 1.0568531962889085,
+ "nll": 11.84809119688847,
+ "aurc": 0.3935424667756691,
+ "mean_confidence": 0.9835926580481263,
+ "acc_at_50_coverage": 0.5733333333333334,
+ "acc_at_80_coverage": 0.5041666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.46,
+ "macro_f1": 0.4519283547157385,
+ "ece": 0.327470752681133,
+ "brier": 0.8270249223142718,
+ "nll": 2.2711318154752536,
+ "aurc": 0.3471555564698994,
+ "mean_confidence": 0.7777987624031797,
+ "acc_at_50_coverage": 0.5866666666666667,
+ "acc_at_80_coverage": 0.4875
+ },
+ "seconds": 2.47,
+ "dropped_questions": 0,
+ "questions_per_second": 121.5
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.4533333333333333,
+ "macro_f1": 0.45184505601719327,
+ "ece": 0.41141516589821114,
+ "brier": 0.9011266954092387,
+ "nll": 2.969407019443928,
+ "aurc": 0.31162388267444885,
+ "mean_confidence": 0.8647484992315443,
+ "acc_at_50_coverage": 0.6733333333333333,
+ "acc_at_80_coverage": 0.5166666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.4533333333333333,
+ "macro_f1": 0.45184505601719327,
+ "ece": 0.41141516589821114,
+ "brier": 0.9011266954092387,
+ "nll": 2.969407019443928,
+ "aurc": 0.31162388267444885,
+ "mean_confidence": 0.8647484992315443,
+ "acc_at_50_coverage": 0.6733333333333333,
+ "acc_at_80_coverage": 0.5166666666666667
+ },
+ "seconds": 0.96,
+ "dropped_questions": 0,
+ "questions_per_second": 311.1
+ }
+ },
+ "massive_scenario.es": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.38333333333333336,
+ "macro_f1": 0.3808258971995832,
+ "ece": 0.5943658153687433,
+ "brier": 1.2053445709534798,
+ "nll": 14.592647728493581,
+ "aurc": 0.5137364183153083,
+ "mean_confidence": 0.9776991487020765,
+ "acc_at_50_coverage": 0.5,
+ "acc_at_80_coverage": 0.42083333333333334
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.38333333333333336,
+ "macro_f1": 0.3808258971995832,
+ "ece": 0.4061125326810553,
+ "brier": 0.974631055323556,
+ "nll": 3.0268021168724615,
+ "aurc": 0.449181477550745,
+ "mean_confidence": 0.7872946293297955,
+ "acc_at_50_coverage": 0.5266666666666666,
+ "acc_at_80_coverage": 0.42916666666666664
+ },
+ "seconds": 2.42,
+ "dropped_questions": 0,
+ "questions_per_second": 123.9
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.44666666666666666,
+ "macro_f1": 0.442167638299833,
+ "ece": 0.4069640053563983,
+ "brier": 0.8821714663948459,
+ "nll": 3.0290174890601005,
+ "aurc": 0.29734844288332196,
+ "mean_confidence": 0.8479267644688439,
+ "acc_at_50_coverage": 0.7133333333333334,
+ "acc_at_80_coverage": 0.5291666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.44666666666666666,
+ "macro_f1": 0.442167638299833,
+ "ece": 0.4069640053563983,
+ "brier": 0.8821714663948459,
+ "nll": 3.0290174890601005,
+ "aurc": 0.29734844288332196,
+ "mean_confidence": 0.8479267644688439,
+ "acc_at_50_coverage": 0.7133333333333334,
+ "acc_at_80_coverage": 0.5291666666666667
+ },
+ "seconds": 0.96,
+ "dropped_questions": 0,
+ "questions_per_second": 313.0
+ }
+ },
+ "massive_scenario.pt": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.42,
+ "macro_f1": 0.41507318506283947,
+ "ece": 0.5676626727748507,
+ "brier": 1.142251765714677,
+ "nll": 14.21340070224241,
+ "aurc": 0.450532843481489,
+ "mean_confidence": 0.9855712309838316,
+ "acc_at_50_coverage": 0.5933333333333334,
+ "acc_at_80_coverage": 0.45
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.42,
+ "macro_f1": 0.41507318506283947,
+ "ece": 0.37499435280399795,
+ "brier": 0.9247850663292572,
+ "nll": 2.9814680709745818,
+ "aurc": 0.40034927925250896,
+ "mean_confidence": 0.7906760763571875,
+ "acc_at_50_coverage": 0.5866666666666667,
+ "acc_at_80_coverage": 0.4791666666666667
+ },
+ "seconds": 2.48,
+ "dropped_questions": 0,
+ "questions_per_second": 120.7
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.4633333333333333,
+ "macro_f1": 0.4583159082261799,
+ "ece": 0.399902067877031,
+ "brier": 0.9173335266917587,
+ "nll": 3.300888761245623,
+ "aurc": 0.3732814777085433,
+ "mean_confidence": 0.861442229013749,
+ "acc_at_50_coverage": 0.6133333333333333,
+ "acc_at_80_coverage": 0.5125
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.4633333333333333,
+ "macro_f1": 0.4583159082261799,
+ "ece": 0.399902067877031,
+ "brier": 0.9173335266917587,
+ "nll": 3.300888761245623,
+ "aurc": 0.3732814777085433,
+ "mean_confidence": 0.861442229013749,
+ "acc_at_50_coverage": 0.6133333333333333,
+ "acc_at_80_coverage": 0.5125
+ },
+ "seconds": 0.95,
+ "dropped_questions": 0,
+ "questions_per_second": 316.5
+ }
+ },
+ "massive_scenario.ru": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.34,
+ "macro_f1": 0.3379920393729521,
+ "ece": 0.6439844360330014,
+ "brier": 1.292337140048209,
+ "nll": 15.821192298551425,
+ "aurc": 0.5736417959888973,
+ "mean_confidence": 0.9775710458281587,
+ "acc_at_50_coverage": 0.3933333333333333,
+ "acc_at_80_coverage": 0.375
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.34,
+ "macro_f1": 0.3379920393729521,
+ "ece": 0.39153995940719355,
+ "brier": 1.0187874517914235,
+ "nll": 3.373532527817828,
+ "aurc": 0.5612235825848404,
+ "mean_confidence": 0.7278895131347629,
+ "acc_at_50_coverage": 0.4266666666666667,
+ "acc_at_80_coverage": 0.37916666666666665
+ },
+ "seconds": 2.62,
+ "dropped_questions": 0,
+ "questions_per_second": 114.6
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.47333333333333333,
+ "macro_f1": 0.46948646596979554,
+ "ece": 0.38306263750978925,
+ "brier": 0.8780662503308873,
+ "nll": 3.1476885318084227,
+ "aurc": 0.3678024075289724,
+ "mean_confidence": 0.8563959708431225,
+ "acc_at_50_coverage": 0.6466666666666666,
+ "acc_at_80_coverage": 0.55
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.47333333333333333,
+ "macro_f1": 0.46948646596979554,
+ "ece": 0.38306263750978925,
+ "brier": 0.8780662503308873,
+ "nll": 3.1476885318084227,
+ "aurc": 0.3678024075289724,
+ "mean_confidence": 0.8563959708431225,
+ "acc_at_50_coverage": 0.6466666666666666,
+ "acc_at_80_coverage": 0.55
+ },
+ "seconds": 0.94,
+ "dropped_questions": 0,
+ "questions_per_second": 319.5
+ }
+ },
+ "massive_scenario.tr": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.22666666666666666,
+ "macro_f1": 0.22930908041261722,
+ "ece": 0.7344336039568069,
+ "brier": 1.4942577998212168,
+ "nll": 18.367193357732436,
+ "aurc": 0.6448873480624936,
+ "mean_confidence": 0.9611002706234736,
+ "acc_at_50_coverage": 0.3333333333333333,
+ "acc_at_80_coverage": 0.25416666666666665
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.22666666666666666,
+ "macro_f1": 0.22930908041261722,
+ "ece": 0.3714210891197136,
+ "brier": 1.0763457483169783,
+ "nll": 3.8042266690014293,
+ "aurc": 0.5847453259269151,
+ "mean_confidence": 0.597705349646592,
+ "acc_at_50_coverage": 0.32666666666666666,
+ "acc_at_80_coverage": 0.24583333333333332
+ },
+ "seconds": 2.55,
+ "dropped_questions": 0,
+ "questions_per_second": 117.5
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.48333333333333334,
+ "macro_f1": 0.48279825510729374,
+ "ece": 0.3510541120298612,
+ "brier": 0.8351881304472127,
+ "nll": 2.5433498577236873,
+ "aurc": 0.3075129394182914,
+ "mean_confidence": 0.8343874453631944,
+ "acc_at_50_coverage": 0.6733333333333333,
+ "acc_at_80_coverage": 0.55
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.48333333333333334,
+ "macro_f1": 0.48279825510729374,
+ "ece": 0.3510541120298612,
+ "brier": 0.8351881304472127,
+ "nll": 2.5433498577236873,
+ "aurc": 0.3075129394182914,
+ "mean_confidence": 0.8343874453631944,
+ "acc_at_50_coverage": 0.6733333333333333,
+ "acc_at_80_coverage": 0.55
+ },
+ "seconds": 0.94,
+ "dropped_questions": 0,
+ "questions_per_second": 320.1
+ }
+ },
+ "massive_scenario.ar": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.14,
+ "macro_f1": 0.1394507004001941,
+ "ece": 0.8210527427159336,
+ "brier": 1.6685223063858712,
+ "nll": 20.83073025736417,
+ "aurc": 0.8878838031394582,
+ "mean_confidence": 0.9610527427159338,
+ "acc_at_50_coverage": 0.10666666666666667,
+ "acc_at_80_coverage": 0.1125
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.14,
+ "macro_f1": 0.1394507004001941,
+ "ece": 0.42293347604019144,
+ "brier": 1.2073659505782506,
+ "nll": 4.070635463836291,
+ "aurc": 0.8709828618752677,
+ "mean_confidence": 0.562436447123142,
+ "acc_at_50_coverage": 0.12666666666666668,
+ "acc_at_80_coverage": 0.11666666666666667
+ },
+ "seconds": 2.58,
+ "dropped_questions": 0,
+ "questions_per_second": 116.1
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.39666666666666667,
+ "macro_f1": 0.3953884343454503,
+ "ece": 0.41013940064528187,
+ "brier": 0.9604693183837674,
+ "nll": 3.244261200284413,
+ "aurc": 0.41538539360629756,
+ "mean_confidence": 0.8001018238229396,
+ "acc_at_50_coverage": 0.56,
+ "acc_at_80_coverage": 0.45416666666666666
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.39666666666666667,
+ "macro_f1": 0.3953884343454503,
+ "ece": 0.41013940064528187,
+ "brier": 0.9604693183837674,
+ "nll": 3.244261200284413,
+ "aurc": 0.41538539360629756,
+ "mean_confidence": 0.8001018238229396,
+ "acc_at_50_coverage": 0.56,
+ "acc_at_80_coverage": 0.45416666666666666
+ },
+ "seconds": 0.95,
+ "dropped_questions": 0,
+ "questions_per_second": 317.2
+ }
+ },
+ "massive_scenario.hi": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.09666666666666666,
+ "macro_f1": 0.09218738365400807,
+ "ece": 0.8656490043549154,
+ "brier": 1.750804430993744,
+ "nll": 22.13358390954595,
+ "aurc": 0.8898096195789185,
+ "mean_confidence": 0.962315671021582,
+ "acc_at_50_coverage": 0.10666666666666667,
+ "acc_at_80_coverage": 0.1
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.09666666666666666,
+ "macro_f1": 0.09218738365400807,
+ "ece": 0.4678827983908522,
+ "brier": 1.2401910045098414,
+ "nll": 4.3450989701505796,
+ "aurc": 0.8733878480349362,
+ "mean_confidence": 0.5645494650575188,
+ "acc_at_50_coverage": 0.10666666666666667,
+ "acc_at_80_coverage": 0.10833333333333334
+ },
+ "seconds": 3.0,
+ "dropped_questions": 0,
+ "questions_per_second": 100.1
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.44666666666666666,
+ "macro_f1": 0.44807505691508126,
+ "ece": 0.3695063961757436,
+ "brier": 0.9068063475197149,
+ "nll": 3.0242455495430627,
+ "aurc": 0.3385449178688652,
+ "mean_confidence": 0.808062239575728,
+ "acc_at_50_coverage": 0.5933333333333334,
+ "acc_at_80_coverage": 0.4875
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.44666666666666666,
+ "macro_f1": 0.44807505691508126,
+ "ece": 0.3695063961757436,
+ "brier": 0.9068063475197149,
+ "nll": 3.0242455495430627,
+ "aurc": 0.3385449178688652,
+ "mean_confidence": 0.808062239575728,
+ "acc_at_50_coverage": 0.5933333333333334,
+ "acc_at_80_coverage": 0.4875
+ },
+ "seconds": 0.99,
+ "dropped_questions": 0,
+ "questions_per_second": 301.8
+ }
+ },
+ "massive_scenario.ta": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.10333333333333333,
+ "macro_f1": 0.10123211331251272,
+ "ece": 0.874681700038002,
+ "brier": 1.7602726601511418,
+ "nll": 22.372137100940222,
+ "aurc": 0.9231935952280065,
+ "mean_confidence": 0.9780150333713353,
+ "acc_at_50_coverage": 0.06666666666666667,
+ "acc_at_80_coverage": 0.09583333333333334
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.10333333333333333,
+ "macro_f1": 0.10123211331251272,
+ "ece": 0.5694229234759011,
+ "brier": 1.3300067700786005,
+ "nll": 4.4484547544084645,
+ "aurc": 0.9140831228744111,
+ "mean_confidence": 0.6727562568092345,
+ "acc_at_50_coverage": 0.06666666666666667,
+ "acc_at_80_coverage": 0.10416666666666667
+ },
+ "seconds": 3.47,
+ "dropped_questions": 0,
+ "questions_per_second": 86.5
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.33666666666666667,
+ "macro_f1": 0.3252572891630063,
+ "ece": 0.42554588538467264,
+ "brier": 1.0566269769548338,
+ "nll": 3.5072212892827976,
+ "aurc": 0.4944285972644023,
+ "mean_confidence": 0.7619340369391754,
+ "acc_at_50_coverage": 0.44666666666666666,
+ "acc_at_80_coverage": 0.37083333333333335
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.33666666666666667,
+ "macro_f1": 0.3252572891630063,
+ "ece": 0.42554588538467264,
+ "brier": 1.0566269769548338,
+ "nll": 3.5072212892827976,
+ "aurc": 0.4944285972644023,
+ "mean_confidence": 0.7619340369391754,
+ "acc_at_50_coverage": 0.44666666666666666,
+ "acc_at_80_coverage": 0.37083333333333335
+ },
+ "seconds": 1.05,
+ "dropped_questions": 0,
+ "questions_per_second": 285.1
+ }
+ },
+ "massive_scenario.zh-CN": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.4866666666666667,
+ "macro_f1": 0.4818270425098754,
+ "ece": 0.49165408019083584,
+ "brier": 1.003183435429874,
+ "nll": 11.86876499252355,
+ "aurc": 0.37479171862970995,
+ "mean_confidence": 0.9783207468575025,
+ "acc_at_50_coverage": 0.62,
+ "acc_at_80_coverage": 0.525
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.4866666666666667,
+ "macro_f1": 0.4818270425098754,
+ "ece": 0.32380128046122,
+ "brier": 0.817820474064045,
+ "nll": 2.3817626237262144,
+ "aurc": 0.31927257748382765,
+ "mean_confidence": 0.8055432886305594,
+ "acc_at_50_coverage": 0.6266666666666667,
+ "acc_at_80_coverage": 0.5375
+ },
+ "seconds": 2.44,
+ "dropped_questions": 0,
+ "questions_per_second": 123.1
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.49666666666666665,
+ "macro_f1": 0.4946656595501705,
+ "ece": 0.37089746453804967,
+ "brier": 0.8238693890006955,
+ "nll": 2.7291576173922643,
+ "aurc": 0.26099321675377096,
+ "mean_confidence": 0.8675641312047164,
+ "acc_at_50_coverage": 0.72,
+ "acc_at_80_coverage": 0.575
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.49666666666666665,
+ "macro_f1": 0.4946656595501705,
+ "ece": 0.37089746453804967,
+ "brier": 0.8238693890006955,
+ "nll": 2.7291576173922643,
+ "aurc": 0.26099321675377096,
+ "mean_confidence": 0.8675641312047164,
+ "acc_at_50_coverage": 0.72,
+ "acc_at_80_coverage": 0.575
+ },
+ "seconds": 0.91,
+ "dropped_questions": 0,
+ "questions_per_second": 329.1
+ }
+ },
+ "massive_scenario.ja": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.4666666666666667,
+ "macro_f1": 0.463871866192089,
+ "ece": 0.5111735469099957,
+ "brier": 1.0434086140512933,
+ "nll": 11.8613739437226,
+ "aurc": 0.36754816891195335,
+ "mean_confidence": 0.9778402135766623,
+ "acc_at_50_coverage": 0.64,
+ "acc_at_80_coverage": 0.5208333333333334
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.4666666666666667,
+ "macro_f1": 0.463871866192089,
+ "ece": 0.31980380234551753,
+ "brier": 0.808630329162578,
+ "nll": 2.293172250253408,
+ "aurc": 0.29887623886089426,
+ "mean_confidence": 0.7864704690121842,
+ "acc_at_50_coverage": 0.6533333333333333,
+ "acc_at_80_coverage": 0.5166666666666667
+ },
+ "seconds": 2.49,
+ "dropped_questions": 0,
+ "questions_per_second": 120.7
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.5066666666666667,
+ "macro_f1": 0.5055826834039463,
+ "ece": 0.3361473942723203,
+ "brier": 0.8127611007453828,
+ "nll": 2.8748737144562644,
+ "aurc": 0.2559812861674103,
+ "mean_confidence": 0.8428140609389871,
+ "acc_at_50_coverage": 0.74,
+ "acc_at_80_coverage": 0.5708333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.5066666666666667,
+ "macro_f1": 0.5055826834039463,
+ "ece": 0.3361473942723203,
+ "brier": 0.8127611007453828,
+ "nll": 2.8748737144562644,
+ "aurc": 0.2559812861674103,
+ "mean_confidence": 0.8428140609389871,
+ "acc_at_50_coverage": 0.74,
+ "acc_at_80_coverage": 0.5708333333333333
+ },
+ "seconds": 0.92,
+ "dropped_questions": 0,
+ "questions_per_second": 324.7
+ }
+ },
+ "massive_scenario.ko": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.12666666666666668,
+ "macro_f1": 0.12095669847529399,
+ "ece": 0.840051417707468,
+ "brier": 1.6997114892267453,
+ "nll": 21.805391671020374,
+ "aurc": 0.8374933680329082,
+ "mean_confidence": 0.9667180843741346,
+ "acc_at_50_coverage": 0.13333333333333333,
+ "acc_at_80_coverage": 0.125
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.12666666666666668,
+ "macro_f1": 0.12095669847529399,
+ "ece": 0.48904771206917413,
+ "brier": 1.2507464283639262,
+ "nll": 4.361151892016419,
+ "aurc": 0.8100520077414101,
+ "mean_confidence": 0.6114450666752915,
+ "acc_at_50_coverage": 0.13333333333333333,
+ "acc_at_80_coverage": 0.12916666666666668
+ },
+ "seconds": 2.62,
+ "dropped_questions": 0,
+ "questions_per_second": 114.5
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.42333333333333334,
+ "macro_f1": 0.4132135713403089,
+ "ece": 0.384623058336728,
+ "brier": 0.9237245564942307,
+ "nll": 3.1511491600254296,
+ "aurc": 0.3293478211036618,
+ "mean_confidence": 0.8079563916700613,
+ "acc_at_50_coverage": 0.6133333333333333,
+ "acc_at_80_coverage": 0.4875
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.42333333333333334,
+ "macro_f1": 0.4132135713403089,
+ "ece": 0.384623058336728,
+ "brier": 0.9237245564942307,
+ "nll": 3.1511491600254296,
+ "aurc": 0.3293478211036618,
+ "mean_confidence": 0.8079563916700613,
+ "acc_at_50_coverage": 0.6133333333333333,
+ "acc_at_80_coverage": 0.4875
+ },
+ "seconds": 0.97,
+ "dropped_questions": 0,
+ "questions_per_second": 309.4
+ }
+ },
+ "massive_scenario.sw": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.1,
+ "macro_f1": 0.0957338933732812,
+ "ece": 0.8603138356126518,
+ "brier": 1.7409367961677737,
+ "nll": 20.656402774670834,
+ "aurc": 0.8851119406852863,
+ "mean_confidence": 0.9603138356126518,
+ "acc_at_50_coverage": 0.10666666666666667,
+ "acc_at_80_coverage": 0.0875
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.1,
+ "macro_f1": 0.0957338933732812,
+ "ece": 0.4739598046143587,
+ "brier": 1.219804489946537,
+ "nll": 4.306388937324703,
+ "aurc": 0.88140076168849,
+ "mean_confidence": 0.5739598046143588,
+ "acc_at_50_coverage": 0.1,
+ "acc_at_80_coverage": 0.09166666666666666
+ },
+ "seconds": 2.37,
+ "dropped_questions": 0,
+ "questions_per_second": 126.8
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.3,
+ "macro_f1": 0.29610211511953605,
+ "ece": 0.48034089961981297,
+ "brier": 1.1406992541134244,
+ "nll": 4.430710868073057,
+ "aurc": 0.6298565574065912,
+ "mean_confidence": 0.771683877944962,
+ "acc_at_50_coverage": 0.38,
+ "acc_at_80_coverage": 0.32916666666666666
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.3,
+ "macro_f1": 0.29610211511953605,
+ "ece": 0.48034089961981297,
+ "brier": 1.1406992541134244,
+ "nll": 4.430710868073057,
+ "aurc": 0.6298565574065912,
+ "mean_confidence": 0.771683877944962,
+ "acc_at_50_coverage": 0.38,
+ "acc_at_80_coverage": 0.32916666666666666
+ },
+ "seconds": 1.15,
+ "dropped_questions": 0,
+ "questions_per_second": 260.4
+ }
+ },
+ "xnli.en": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.86,
+ "macro_f1": 0.8611876188602533,
+ "ece": 0.06851500868396258,
+ "brier": 0.22050539483309456,
+ "nll": 0.39876324404370733,
+ "aurc": 0.03817391536388231,
+ "mean_confidence": 0.920764748584588,
+ "acc_at_50_coverage": 0.98,
+ "acc_at_80_coverage": 0.9208333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.86,
+ "macro_f1": 0.8611876188602533,
+ "ece": 0.11742718550023877,
+ "brier": 0.24798918341393375,
+ "nll": 0.5741518550133503,
+ "aurc": 0.03948252276018887,
+ "mean_confidence": 0.9737337063751195,
+ "acc_at_50_coverage": 0.98,
+ "acc_at_80_coverage": 0.9166666666666666
+ },
+ "seconds": 2.14,
+ "dropped_questions": 0,
+ "questions_per_second": 139.9
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.8433333333333334,
+ "macro_f1": 0.843059132192939,
+ "ece": 0.08620842684402336,
+ "brier": 0.23008455799877037,
+ "nll": 0.4457800463487143,
+ "aurc": 0.03935656889260041,
+ "mean_confidence": 0.9294384876362892,
+ "acc_at_50_coverage": 0.9733333333333334,
+ "acc_at_80_coverage": 0.925
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.8433333333333334,
+ "macro_f1": 0.843059132192939,
+ "ece": 0.08620842684402336,
+ "brier": 0.23008455799877037,
+ "nll": 0.4457800463487143,
+ "aurc": 0.03935656889260041,
+ "mean_confidence": 0.9294384876362892,
+ "acc_at_50_coverage": 0.9733333333333334,
+ "acc_at_80_coverage": 0.925
+ },
+ "seconds": 0.9,
+ "dropped_questions": 0,
+ "questions_per_second": 331.5
+ }
+ },
+ "xnli.de": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.64,
+ "macro_f1": 0.6326035374850881,
+ "ece": 0.21072499301639186,
+ "brier": 0.5388959699858508,
+ "nll": 0.9682222930794052,
+ "aurc": 0.20711301581726796,
+ "mean_confidence": 0.8426243028174698,
+ "acc_at_50_coverage": 0.8066666666666666,
+ "acc_at_80_coverage": 0.7166666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.64,
+ "macro_f1": 0.6326035374850881,
+ "ece": 0.29767984180382207,
+ "brier": 0.6210199479399825,
+ "nll": 1.454373830978189,
+ "aurc": 0.20956478611441695,
+ "mean_confidence": 0.9301385123819258,
+ "acc_at_50_coverage": 0.8,
+ "acc_at_80_coverage": 0.7166666666666667
+ },
+ "seconds": 2.56,
+ "dropped_questions": 0,
+ "questions_per_second": 117.4
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.7933333333333333,
+ "macro_f1": 0.7929169579018676,
+ "ece": 0.13507943353849292,
+ "brier": 0.3391052729394329,
+ "nll": 0.7604748461013707,
+ "aurc": 0.08005420779482897,
+ "mean_confidence": 0.9235221794766247,
+ "acc_at_50_coverage": 0.9133333333333333,
+ "acc_at_80_coverage": 0.8666666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.7933333333333333,
+ "macro_f1": 0.7929169579018676,
+ "ece": 0.13507943353849292,
+ "brier": 0.3391052729394329,
+ "nll": 0.7604748461013707,
+ "aurc": 0.08005420779482897,
+ "mean_confidence": 0.9235221794766247,
+ "acc_at_50_coverage": 0.9133333333333333,
+ "acc_at_80_coverage": 0.8666666666666667
+ },
+ "seconds": 0.93,
+ "dropped_questions": 0,
+ "questions_per_second": 321.9
+ }
+ },
+ "xnli.fr": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.68,
+ "macro_f1": 0.6767434402445008,
+ "ece": 0.18792262418329336,
+ "brier": 0.49136694862107977,
+ "nll": 0.892734589072878,
+ "aurc": 0.16347816642024618,
+ "mean_confidence": 0.8642486447883189,
+ "acc_at_50_coverage": 0.8266666666666667,
+ "acc_at_80_coverage": 0.7375
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.68,
+ "macro_f1": 0.6767434402445008,
+ "ece": 0.258012379095395,
+ "brier": 0.5539348541031027,
+ "nll": 1.358595547094244,
+ "aurc": 0.16696330194238168,
+ "mean_confidence": 0.9380123790953951,
+ "acc_at_50_coverage": 0.82,
+ "acc_at_80_coverage": 0.7375
+ },
+ "seconds": 2.59,
+ "dropped_questions": 0,
+ "questions_per_second": 115.7
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.7666666666666667,
+ "macro_f1": 0.7659139737754787,
+ "ece": 0.16007730354280114,
+ "brier": 0.36889637585192164,
+ "nll": 0.7854969087097513,
+ "aurc": 0.08175402414317233,
+ "mean_confidence": 0.9220045466100965,
+ "acc_at_50_coverage": 0.9333333333333333,
+ "acc_at_80_coverage": 0.8583333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.7666666666666667,
+ "macro_f1": 0.7659139737754787,
+ "ece": 0.16007730354280114,
+ "brier": 0.36889637585192164,
+ "nll": 0.7854969087097513,
+ "aurc": 0.08175402414317233,
+ "mean_confidence": 0.9220045466100965,
+ "acc_at_50_coverage": 0.9333333333333333,
+ "acc_at_80_coverage": 0.8583333333333333
+ },
+ "seconds": 1.01,
+ "dropped_questions": 0,
+ "questions_per_second": 297.7
+ }
+ },
+ "xnli.es": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.6833333333333333,
+ "macro_f1": 0.6818501257866464,
+ "ece": 0.17655950701994094,
+ "brier": 0.47418149963116757,
+ "nll": 0.8472352910712136,
+ "aurc": 0.14227331792807743,
+ "mean_confidence": 0.8598928403532743,
+ "acc_at_50_coverage": 0.8666666666666667,
+ "acc_at_80_coverage": 0.7583333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.6833333333333333,
+ "macro_f1": 0.6818501257866464,
+ "ece": 0.25490116584860123,
+ "brier": 0.544935497456347,
+ "nll": 1.2691842047169257,
+ "aurc": 0.14385415213323569,
+ "mean_confidence": 0.9382344991819344,
+ "acc_at_50_coverage": 0.8666666666666667,
+ "acc_at_80_coverage": 0.75
+ },
+ "seconds": 2.44,
+ "dropped_questions": 0,
+ "questions_per_second": 123.1
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.8,
+ "macro_f1": 0.800149169688347,
+ "ece": 0.13461852130061025,
+ "brier": 0.32386057803816204,
+ "nll": 0.6853079173514695,
+ "aurc": 0.07282326021925968,
+ "mean_confidence": 0.9188244095771106,
+ "acc_at_50_coverage": 0.9533333333333334,
+ "acc_at_80_coverage": 0.8708333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.8,
+ "macro_f1": 0.800149169688347,
+ "ece": 0.13461852130061025,
+ "brier": 0.32386057803816204,
+ "nll": 0.6853079173514695,
+ "aurc": 0.07282326021925968,
+ "mean_confidence": 0.9188244095771106,
+ "acc_at_50_coverage": 0.9533333333333334,
+ "acc_at_80_coverage": 0.8708333333333333
+ },
+ "seconds": 0.95,
+ "dropped_questions": 0,
+ "questions_per_second": 314.9
+ }
+ },
+ "xnli.ru": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.61,
+ "macro_f1": 0.6006412950587708,
+ "ece": 0.2168477327377505,
+ "brier": 0.6069292023023662,
+ "nll": 1.1253612446761034,
+ "aurc": 0.26181387796722144,
+ "mean_confidence": 0.8262401584516464,
+ "acc_at_50_coverage": 0.7066666666666667,
+ "acc_at_80_coverage": 0.65
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.61,
+ "macro_f1": 0.6006412950587708,
+ "ece": 0.30976518905362826,
+ "brier": 0.6837205620638928,
+ "nll": 1.7063311206038916,
+ "aurc": 0.26347910681159836,
+ "mean_confidence": 0.9172463227137924,
+ "acc_at_50_coverage": 0.7066666666666667,
+ "acc_at_80_coverage": 0.65
+ },
+ "seconds": 3.11,
+ "dropped_questions": 0,
+ "questions_per_second": 96.5
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.74,
+ "macro_f1": 0.734120855267125,
+ "ece": 0.17782000732522102,
+ "brier": 0.42816474330946314,
+ "nll": 1.04445604970244,
+ "aurc": 0.13628990016553602,
+ "mean_confidence": 0.9120620401690492,
+ "acc_at_50_coverage": 0.8733333333333333,
+ "acc_at_80_coverage": 0.8083333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.74,
+ "macro_f1": 0.734120855267125,
+ "ece": 0.17782000732522102,
+ "brier": 0.42816474330946314,
+ "nll": 1.04445604970244,
+ "aurc": 0.13628990016553602,
+ "mean_confidence": 0.9120620401690492,
+ "acc_at_50_coverage": 0.8733333333333333,
+ "acc_at_80_coverage": 0.8083333333333333
+ },
+ "seconds": 0.99,
+ "dropped_questions": 0,
+ "questions_per_second": 302.3
+ }
+ },
+ "xnli.tr": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.38333333333333336,
+ "macro_f1": 0.3659172368849788,
+ "ece": 0.3563651548153499,
+ "brier": 0.8459639079784079,
+ "nll": 1.4686604597336481,
+ "aurc": 0.535768027732775,
+ "mean_confidence": 0.7396984881486833,
+ "acc_at_50_coverage": 0.4666666666666667,
+ "acc_at_80_coverage": 0.4125
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.38333333333333336,
+ "macro_f1": 0.3659172368849788,
+ "ece": 0.4763532577379035,
+ "brier": 0.9961971299685647,
+ "nll": 2.178321712537001,
+ "aurc": 0.5370329291137014,
+ "mean_confidence": 0.859686591071237,
+ "acc_at_50_coverage": 0.46,
+ "acc_at_80_coverage": 0.4125
+ },
+ "seconds": 2.78,
+ "dropped_questions": 0,
+ "questions_per_second": 107.8
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.74,
+ "macro_f1": 0.7375732440652106,
+ "ece": 0.19369106633776462,
+ "brier": 0.4476175522553707,
+ "nll": 1.0262754243753656,
+ "aurc": 0.137635591019981,
+ "mean_confidence": 0.9240012000590464,
+ "acc_at_50_coverage": 0.8933333333333333,
+ "acc_at_80_coverage": 0.7916666666666666
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.74,
+ "macro_f1": 0.7375732440652106,
+ "ece": 0.19369106633776462,
+ "brier": 0.4476175522553707,
+ "nll": 1.0262754243753656,
+ "aurc": 0.137635591019981,
+ "mean_confidence": 0.9240012000590464,
+ "acc_at_50_coverage": 0.8933333333333333,
+ "acc_at_80_coverage": 0.7916666666666666
+ },
+ "seconds": 0.93,
+ "dropped_questions": 0,
+ "questions_per_second": 322.2
+ }
+ },
+ "xnli.ar": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.44,
+ "macro_f1": 0.4134423398629324,
+ "ece": 0.28478987850626714,
+ "brier": 0.7813091075197995,
+ "nll": 1.3379267752927917,
+ "aurc": 0.4768315716677202,
+ "mean_confidence": 0.7147199743240977,
+ "acc_at_50_coverage": 0.48,
+ "acc_at_80_coverage": 0.45416666666666666
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.44,
+ "macro_f1": 0.4134423398629324,
+ "ece": 0.4083698884381125,
+ "brier": 0.9167379504417611,
+ "nll": 1.9116379283319231,
+ "aurc": 0.47824095488177326,
+ "mean_confidence": 0.8399337873849,
+ "acc_at_50_coverage": 0.48,
+ "acc_at_80_coverage": 0.45
+ },
+ "seconds": 3.32,
+ "dropped_questions": 0,
+ "questions_per_second": 90.5
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.7266666666666667,
+ "macro_f1": 0.7261659274776471,
+ "ece": 0.19939197696989705,
+ "brier": 0.4625657093860309,
+ "nll": 1.09160837839155,
+ "aurc": 0.12492089138118848,
+ "mean_confidence": 0.916744606923775,
+ "acc_at_50_coverage": 0.8733333333333333,
+ "acc_at_80_coverage": 0.775
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.7266666666666667,
+ "macro_f1": 0.7261659274776471,
+ "ece": 0.19939197696989705,
+ "brier": 0.4625657093860309,
+ "nll": 1.09160837839155,
+ "aurc": 0.12492089138118848,
+ "mean_confidence": 0.916744606923775,
+ "acc_at_50_coverage": 0.8733333333333333,
+ "acc_at_80_coverage": 0.775
+ },
+ "seconds": 0.98,
+ "dropped_questions": 0,
+ "questions_per_second": 307.2
+ }
+ },
+ "xnli.hi": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.4,
+ "macro_f1": 0.39253301861737616,
+ "ece": 0.29034949231012497,
+ "brier": 0.8294244668128908,
+ "nll": 1.3949523678726539,
+ "aurc": 0.5942668839361914,
+ "mean_confidence": 0.6769456621737453,
+ "acc_at_50_coverage": 0.36666666666666664,
+ "acc_at_80_coverage": 0.37916666666666665
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.4,
+ "macro_f1": 0.39253301861737616,
+ "ece": 0.4142800361803396,
+ "brier": 0.9722077032311629,
+ "nll": 1.9630018094084252,
+ "aurc": 0.6002955569729772,
+ "mean_confidence": 0.8005386005987778,
+ "acc_at_50_coverage": 0.36666666666666664,
+ "acc_at_80_coverage": 0.3875
+ },
+ "seconds": 4.37,
+ "dropped_questions": 0,
+ "questions_per_second": 68.6
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.66,
+ "macro_f1": 0.6568312214327381,
+ "ece": 0.24207442048070107,
+ "brier": 0.5353331393161586,
+ "nll": 1.2911493476619622,
+ "aurc": 0.1737586182765312,
+ "mean_confidence": 0.9020744204807011,
+ "acc_at_50_coverage": 0.8,
+ "acc_at_80_coverage": 0.7333333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.66,
+ "macro_f1": 0.6568312214327381,
+ "ece": 0.24207442048070107,
+ "brier": 0.5353331393161586,
+ "nll": 1.2911493476619622,
+ "aurc": 0.1737586182765312,
+ "mean_confidence": 0.9020744204807011,
+ "acc_at_50_coverage": 0.8,
+ "acc_at_80_coverage": 0.7333333333333333
+ },
+ "seconds": 1.19,
+ "dropped_questions": 0,
+ "questions_per_second": 252.8
+ }
+ },
+ "xnli.ur": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.37333333333333335,
+ "macro_f1": 0.3356565779421737,
+ "ece": 0.32197986258438044,
+ "brier": 0.8422166660081521,
+ "nll": 1.4332254995755682,
+ "aurc": 0.549492087108305,
+ "mean_confidence": 0.6953131959177138,
+ "acc_at_50_coverage": 0.4066666666666667,
+ "acc_at_80_coverage": 0.375
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.37333333333333335,
+ "macro_f1": 0.3356565779421737,
+ "ece": 0.4421501948612806,
+ "brier": 0.9917179183917054,
+ "nll": 2.0592524399269077,
+ "aurc": 0.5501051189999518,
+ "mean_confidence": 0.815483528194614,
+ "acc_at_50_coverage": 0.4066666666666667,
+ "acc_at_80_coverage": 0.375
+ },
+ "seconds": 4.2,
+ "dropped_questions": 0,
+ "questions_per_second": 71.4
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.6833333333333333,
+ "macro_f1": 0.6796582905834275,
+ "ece": 0.21653475758448035,
+ "brier": 0.5381491316014443,
+ "nll": 1.2558800480679206,
+ "aurc": 0.18656783659812115,
+ "mean_confidence": 0.8955365350700574,
+ "acc_at_50_coverage": 0.8133333333333334,
+ "acc_at_80_coverage": 0.7291666666666666
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.6833333333333333,
+ "macro_f1": 0.6796582905834275,
+ "ece": 0.21653475758448035,
+ "brier": 0.5381491316014443,
+ "nll": 1.2558800480679206,
+ "aurc": 0.18656783659812115,
+ "mean_confidence": 0.8955365350700574,
+ "acc_at_50_coverage": 0.8133333333333334,
+ "acc_at_80_coverage": 0.7291666666666666
+ },
+ "seconds": 1.17,
+ "dropped_questions": 0,
+ "questions_per_second": 256.3
+ }
+ },
+ "xnli.vi": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.51,
+ "macro_f1": 0.5048020259774798,
+ "ece": 0.19635322556599558,
+ "brier": 0.6321583135727612,
+ "nll": 1.0817933431023652,
+ "aurc": 0.35687656187199907,
+ "mean_confidence": 0.7063532255659957,
+ "acc_at_50_coverage": 0.6266666666666667,
+ "acc_at_80_coverage": 0.5541666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.51,
+ "macro_f1": 0.5048020259774798,
+ "ece": 0.31247173288896124,
+ "brier": 0.7290198204606503,
+ "nll": 1.4564757735641376,
+ "aurc": 0.35803749330841206,
+ "mean_confidence": 0.8224717328889614,
+ "acc_at_50_coverage": 0.6466666666666666,
+ "acc_at_80_coverage": 0.55
+ },
+ "seconds": 3.53,
+ "dropped_questions": 0,
+ "questions_per_second": 84.9
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.7233333333333334,
+ "macro_f1": 0.7163319041978768,
+ "ece": 0.20742967358049616,
+ "brier": 0.4767080414544521,
+ "nll": 1.1728364879399578,
+ "aurc": 0.14809157215092492,
+ "mean_confidence": 0.9307630069138296,
+ "acc_at_50_coverage": 0.8533333333333334,
+ "acc_at_80_coverage": 0.7625
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.7233333333333334,
+ "macro_f1": 0.7163319041978768,
+ "ece": 0.20742967358049616,
+ "brier": 0.4767080414544521,
+ "nll": 1.1728364879399578,
+ "aurc": 0.14809157215092492,
+ "mean_confidence": 0.9307630069138296,
+ "acc_at_50_coverage": 0.8533333333333334,
+ "acc_at_80_coverage": 0.7625
+ },
+ "seconds": 0.98,
+ "dropped_questions": 0,
+ "questions_per_second": 306.7
+ }
+ },
+ "xnli.th": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.4166666666666667,
+ "macro_f1": 0.41597547598264956,
+ "ece": 0.18541593372815865,
+ "brier": 0.7150968571448711,
+ "nll": 1.185035326280894,
+ "aurc": 0.49604564931203415,
+ "mean_confidence": 0.6020826003948253,
+ "acc_at_50_coverage": 0.4666666666666667,
+ "acc_at_80_coverage": 0.4375
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.4166666666666667,
+ "macro_f1": 0.41597547598264956,
+ "ece": 0.30600608264125995,
+ "brier": 0.8224991434988397,
+ "nll": 1.4917845671280279,
+ "aurc": 0.4956985674942147,
+ "mean_confidence": 0.7226727493079265,
+ "acc_at_50_coverage": 0.4533333333333333,
+ "acc_at_80_coverage": 0.43333333333333335
+ },
+ "seconds": 4.31,
+ "dropped_questions": 0,
+ "questions_per_second": 69.5
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.6966666666666667,
+ "macro_f1": 0.6869609915895595,
+ "ece": 0.21061457463296523,
+ "brier": 0.4997460636887329,
+ "nll": 1.1624382390052939,
+ "aurc": 0.15457771919077964,
+ "mean_confidence": 0.9071540187085024,
+ "acc_at_50_coverage": 0.82,
+ "acc_at_80_coverage": 0.7666666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.6966666666666667,
+ "macro_f1": 0.6869609915895595,
+ "ece": 0.21061457463296523,
+ "brier": 0.4997460636887329,
+ "nll": 1.1624382390052939,
+ "aurc": 0.15457771919077964,
+ "mean_confidence": 0.9071540187085024,
+ "acc_at_50_coverage": 0.82,
+ "acc_at_80_coverage": 0.7666666666666667
+ },
+ "seconds": 1.04,
+ "dropped_questions": 0,
+ "questions_per_second": 287.6
+ }
+ },
+ "xnli.el": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.5166666666666667,
+ "macro_f1": 0.5085754741061143,
+ "ece": 0.2158881142658567,
+ "brier": 0.6730261233634023,
+ "nll": 1.1811545983608287,
+ "aurc": 0.3619304897156812,
+ "mean_confidence": 0.7313205582371749,
+ "acc_at_50_coverage": 0.62,
+ "acc_at_80_coverage": 0.5625
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.5166666666666667,
+ "macro_f1": 0.5085754741061143,
+ "ece": 0.334197796750754,
+ "brier": 0.7690286317491565,
+ "nll": 1.6663537620258704,
+ "aurc": 0.3615853025660657,
+ "mean_confidence": 0.841674141964363,
+ "acc_at_50_coverage": 0.62,
+ "acc_at_80_coverage": 0.5666666666666667
+ },
+ "seconds": 3.37,
+ "dropped_questions": 0,
+ "questions_per_second": 89.0
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.7266666666666667,
+ "macro_f1": 0.7239553451902849,
+ "ece": 0.19141782798564486,
+ "brier": 0.4543850042303488,
+ "nll": 1.0568004891245892,
+ "aurc": 0.1251534218900087,
+ "mean_confidence": 0.9161183020059976,
+ "acc_at_50_coverage": 0.8933333333333333,
+ "acc_at_80_coverage": 0.7833333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.7266666666666667,
+ "macro_f1": 0.7239553451902849,
+ "ece": 0.19141782798564486,
+ "brier": 0.4543850042303488,
+ "nll": 1.0568004891245892,
+ "aurc": 0.1251534218900087,
+ "mean_confidence": 0.9161183020059976,
+ "acc_at_50_coverage": 0.8933333333333333,
+ "acc_at_80_coverage": 0.7833333333333333
+ },
+ "seconds": 1.28,
+ "dropped_questions": 0,
+ "questions_per_second": 235.1
+ }
+ },
+ "xnli.bg": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.5433333333333333,
+ "macro_f1": 0.5328651753682229,
+ "ece": 0.23137607365303675,
+ "brier": 0.6454032398069194,
+ "nll": 1.1365945276276332,
+ "aurc": 0.3029031135998286,
+ "mean_confidence": 0.7747094069863701,
+ "acc_at_50_coverage": 0.6933333333333334,
+ "acc_at_80_coverage": 0.6
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.5433333333333333,
+ "macro_f1": 0.5328651753682229,
+ "ece": 0.3365172510087523,
+ "brier": 0.7502977579481567,
+ "nll": 1.6499803540167166,
+ "aurc": 0.3027396258974141,
+ "mean_confidence": 0.8798505843420855,
+ "acc_at_50_coverage": 0.68,
+ "acc_at_80_coverage": 0.5958333333333333
+ },
+ "seconds": 3.2,
+ "dropped_questions": 0,
+ "questions_per_second": 93.9
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.7933333333333333,
+ "macro_f1": 0.791961144703296,
+ "ece": 0.13341324680488864,
+ "brier": 0.3506072942916916,
+ "nll": 0.7777559539036778,
+ "aurc": 0.0824887284269202,
+ "mean_confidence": 0.9240324494058397,
+ "acc_at_50_coverage": 0.9266666666666666,
+ "acc_at_80_coverage": 0.8583333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.7933333333333333,
+ "macro_f1": 0.791961144703296,
+ "ece": 0.13341324680488864,
+ "brier": 0.3506072942916916,
+ "nll": 0.7777559539036778,
+ "aurc": 0.0824887284269202,
+ "mean_confidence": 0.9240324494058397,
+ "acc_at_50_coverage": 0.9266666666666666,
+ "acc_at_80_coverage": 0.8583333333333333
+ },
+ "seconds": 1.05,
+ "dropped_questions": 0,
+ "questions_per_second": 284.8
+ }
+ },
+ "xnli.zh": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.6733333333333333,
+ "macro_f1": 0.6626749561613264,
+ "ece": 0.20887249624498655,
+ "brier": 0.5226839647931262,
+ "nll": 1.0026436713197937,
+ "aurc": 0.20776604839660678,
+ "mean_confidence": 0.8587428911489872,
+ "acc_at_50_coverage": 0.8066666666666666,
+ "acc_at_80_coverage": 0.7208333333333333
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.6733333333333333,
+ "macro_f1": 0.6626749561613264,
+ "ece": 0.27118436703316295,
+ "brier": 0.5795487241582471,
+ "nll": 1.5439941284041712,
+ "aurc": 0.20990279905613277,
+ "mean_confidence": 0.9333730582682843,
+ "acc_at_50_coverage": 0.8,
+ "acc_at_80_coverage": 0.7208333333333333
+ },
+ "seconds": 2.72,
+ "dropped_questions": 0,
+ "questions_per_second": 110.4
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.7566666666666667,
+ "macro_f1": 0.7538368278419627,
+ "ece": 0.16077182161775355,
+ "brier": 0.39710609854268974,
+ "nll": 0.8844138039352938,
+ "aurc": 0.11440906432455504,
+ "mean_confidence": 0.9092261280780669,
+ "acc_at_50_coverage": 0.9,
+ "acc_at_80_coverage": 0.8291666666666667
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.7566666666666667,
+ "macro_f1": 0.7538368278419627,
+ "ece": 0.16077182161775355,
+ "brier": 0.39710609854268974,
+ "nll": 0.8844138039352938,
+ "aurc": 0.11440906432455504,
+ "mean_confidence": 0.9092261280780669,
+ "acc_at_50_coverage": 0.9,
+ "acc_at_80_coverage": 0.8291666666666667
+ },
+ "seconds": 0.88,
+ "dropped_questions": 0,
+ "questions_per_second": 341.4
+ }
+ },
+ "xnli.sw": {
+ "laya": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.42333333333333334,
+ "macro_f1": 0.4207847707847708,
+ "ece": 0.29236211107175514,
+ "brier": 0.7784006863977958,
+ "nll": 1.3179917563408299,
+ "aurc": 0.5003573847218811,
+ "mean_confidence": 0.7024897047792208,
+ "acc_at_50_coverage": 0.47333333333333333,
+ "acc_at_80_coverage": 0.43333333333333335
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.42333333333333334,
+ "macro_f1": 0.4207847707847708,
+ "ece": 0.41450323613638784,
+ "brier": 0.919356157926492,
+ "nll": 1.8576785541576135,
+ "aurc": 0.5031777992926575,
+ "mean_confidence": 0.8296259550336644,
+ "acc_at_50_coverage": 0.4666666666666667,
+ "acc_at_80_coverage": 0.4375
+ },
+ "seconds": 2.75,
+ "dropped_questions": 0,
+ "questions_per_second": 109.1
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 300,
+ "accuracy": 0.6266666666666667,
+ "macro_f1": 0.6230840663744693,
+ "ece": 0.27059308880296506,
+ "brier": 0.5848817223991936,
+ "nll": 1.3826984215782445,
+ "aurc": 0.1918472633863418,
+ "mean_confidence": 0.8963267263770844,
+ "acc_at_50_coverage": 0.8266666666666667,
+ "acc_at_80_coverage": 0.725
+ },
+ "raw": {
+ "n": 300,
+ "accuracy": 0.6266666666666667,
+ "macro_f1": 0.6230840663744693,
+ "ece": 0.27059308880296506,
+ "brier": 0.5848817223991936,
+ "nll": 1.3826984215782445,
+ "aurc": 0.1918472633863418,
+ "mean_confidence": 0.8963267263770844,
+ "acc_at_50_coverage": 0.8266666666666667,
+ "acc_at_80_coverage": 0.725
+ },
+ "seconds": 1.02,
+ "dropped_questions": 0,
+ "questions_per_second": 293.7
+ }
+ },
+ "en.sst5": {
+ "laya": {
+ "calibrated": {
+ "n": 600,
+ "accuracy": 0.37166666666666665,
+ "macro_f1": 0.3288496250901994,
+ "ece": 0.2477510941191931,
+ "brier": 0.8172136446504027,
+ "nll": 1.7292404614651202,
+ "aurc": 0.5280437776692383,
+ "mean_confidence": 0.6194177607858598,
+ "acc_at_50_coverage": 0.45666666666666667,
+ "acc_at_80_coverage": 0.4041666666666667
+ },
+ "raw": {
+ "n": 600,
+ "accuracy": 0.37166666666666665,
+ "macro_f1": 0.3288496250901994,
+ "ece": 0.3005899997646033,
+ "brier": 0.8626452163129017,
+ "nll": 1.9530768413531427,
+ "aurc": 0.5350255397889842,
+ "mean_confidence": 0.67225666643127,
+ "acc_at_50_coverage": 0.46,
+ "acc_at_80_coverage": 0.39375
+ },
+ "seconds": 3.19,
+ "dropped_questions": 0,
+ "questions_per_second": 188.0,
+ "score_mae": 0.9045839813495934,
+ "within_1": 0.6316666666666667
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 600,
+ "accuracy": 0.2816666666666667,
+ "macro_f1": 0.20517704230754683,
+ "ece": 0.5233828299976504,
+ "brier": 1.1825193085471115,
+ "nll": 3.8489351560969514,
+ "aurc": 0.6864556453513037,
+ "mean_confidence": 0.804672039949155,
+ "acc_at_50_coverage": 0.2866666666666667,
+ "acc_at_80_coverage": 0.28541666666666665
+ },
+ "raw": {
+ "n": 600,
+ "accuracy": 0.2816666666666667,
+ "macro_f1": 0.20517704230754683,
+ "ece": 0.5233828299976504,
+ "brier": 1.1825193085471115,
+ "nll": 3.8489351560969514,
+ "aurc": 0.6864556453513037,
+ "mean_confidence": 0.804672039949155,
+ "acc_at_50_coverage": 0.2866666666666667,
+ "acc_at_80_coverage": 0.28541666666666665
+ },
+ "seconds": 1.53,
+ "dropped_questions": 0,
+ "questions_per_second": 391.0,
+ "score_mae": 1.3593954707085403,
+ "within_1": 0.45
+ }
+ },
+ "en.emotion": {
+ "laya": {
+ "calibrated": {
+ "n": 600,
+ "accuracy": 0.5733333333333334,
+ "macro_f1": 0.47631269173299584,
+ "ece": 0.32375073910645563,
+ "brier": 0.7305027390348787,
+ "nll": 2.144856609363522,
+ "aurc": 0.28845213093546057,
+ "mean_confidence": 0.8963017901835962,
+ "acc_at_50_coverage": 0.6866666666666666,
+ "acc_at_80_coverage": 0.6145833333333334
+ },
+ "raw": {
+ "n": 600,
+ "accuracy": 0.5733333333333334,
+ "macro_f1": 0.47631269173299584,
+ "ece": 0.3237525791408232,
+ "brier": 0.7305045932922586,
+ "nll": 2.1448862404580455,
+ "aurc": 0.28845213093546057,
+ "mean_confidence": 0.8963036515560666,
+ "acc_at_50_coverage": 0.6866666666666666,
+ "acc_at_80_coverage": 0.6145833333333334
+ },
+ "seconds": 2.46,
+ "dropped_questions": 0,
+ "questions_per_second": 243.9
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 600,
+ "accuracy": 0.5133333333333333,
+ "macro_f1": 0.42551959178754956,
+ "ece": 0.35195398913875425,
+ "brier": 0.8182858999827102,
+ "nll": 2.3635000382236657,
+ "aurc": 0.37013803812950735,
+ "mean_confidence": 0.8641186323647112,
+ "acc_at_50_coverage": 0.6066666666666667,
+ "acc_at_80_coverage": 0.5479166666666667
+ },
+ "raw": {
+ "n": 600,
+ "accuracy": 0.5133333333333333,
+ "macro_f1": 0.42551959178754956,
+ "ece": 0.35195398913875425,
+ "brier": 0.8182858999827102,
+ "nll": 2.3635000382236657,
+ "aurc": 0.37013803812950735,
+ "mean_confidence": 0.8641186323647112,
+ "acc_at_50_coverage": 0.6066666666666667,
+ "acc_at_80_coverage": 0.5479166666666667
+ },
+ "seconds": 1.12,
+ "dropped_questions": 0,
+ "questions_per_second": 535.6
+ }
+ },
+ "en.prompt_injections": {
+ "laya": {
+ "calibrated": {
+ "n": 116,
+ "accuracy": 0.6982758620689655,
+ "macro_f1": 0.6750700280112045,
+ "ece": 0.261929815155607,
+ "brier": 0.5258464578822987,
+ "nll": 1.6828205154616658,
+ "aurc": 0.10620459144410163,
+ "mean_confidence": 0.9602056772245726,
+ "acc_at_50_coverage": 0.9310344827586207,
+ "acc_at_80_coverage": 0.7717391304347826
+ },
+ "raw": {
+ "n": 116,
+ "accuracy": 0.6982758620689655,
+ "macro_f1": 0.6750700280112045,
+ "ece": 0.29121353183347143,
+ "brier": 0.5699061678546038,
+ "nll": 3.266111151889796,
+ "aurc": 0.10620459144410163,
+ "mean_confidence": 0.9826727110716177,
+ "acc_at_50_coverage": 0.9310344827586207,
+ "acc_at_80_coverage": 0.7717391304347826
+ },
+ "seconds": 1.23,
+ "dropped_questions": 0,
+ "questions_per_second": 94.6
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 116,
+ "accuracy": 0.5775862068965517,
+ "macro_f1": 0.5297426987672706,
+ "ece": 0.3622712910805021,
+ "brier": 0.7188281372535157,
+ "nll": 1.83371785008595,
+ "aurc": 0.2611079846570448,
+ "mean_confidence": 0.9398574979770538,
+ "acc_at_50_coverage": 0.7931034482758621,
+ "acc_at_80_coverage": 0.6413043478260869
+ },
+ "raw": {
+ "n": 116,
+ "accuracy": 0.5775862068965517,
+ "macro_f1": 0.5297426987672706,
+ "ece": 0.3622712910805021,
+ "brier": 0.7188281372535157,
+ "nll": 1.83371785008595,
+ "aurc": 0.2611079846570448,
+ "mean_confidence": 0.9398574979770538,
+ "acc_at_50_coverage": 0.7931034482758621,
+ "acc_at_80_coverage": 0.6413043478260869
+ },
+ "seconds": 0.56,
+ "dropped_questions": 0,
+ "questions_per_second": 206.6
+ }
+ },
+ "en.ag_news": {
+ "laya": {
+ "calibrated": {
+ "n": 600,
+ "accuracy": 0.9466666666666667,
+ "macro_f1": 0.9438349780251156,
+ "ece": 0.039366815217098554,
+ "brier": 0.08696698705958933,
+ "nll": 0.17319417764130313,
+ "aurc": 0.0068207233371325145,
+ "mean_confidence": 0.9214515425079212,
+ "acc_at_50_coverage": 1.0,
+ "acc_at_80_coverage": 0.9875
+ },
+ "raw": {
+ "n": 600,
+ "accuracy": 0.9466666666666667,
+ "macro_f1": 0.9438349780251156,
+ "ece": 0.033209929898696006,
+ "brier": 0.09122999613305266,
+ "nll": 0.17802561050018714,
+ "aurc": 0.0066994794605537205,
+ "mean_confidence": 0.9726261761504992,
+ "acc_at_50_coverage": 1.0,
+ "acc_at_80_coverage": 0.9895833333333334
+ },
+ "seconds": 4.66,
+ "dropped_questions": 0,
+ "questions_per_second": 128.8
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 600,
+ "accuracy": 0.9366666666666666,
+ "macro_f1": 0.9339174627621376,
+ "ece": 0.03790626053297799,
+ "brier": 0.09748790744422886,
+ "nll": 0.2233060249382501,
+ "aurc": 0.015727048809451808,
+ "mean_confidence": 0.9727651926995691,
+ "acc_at_50_coverage": 0.9866666666666667,
+ "acc_at_80_coverage": 0.9854166666666667
+ },
+ "raw": {
+ "n": 600,
+ "accuracy": 0.9366666666666666,
+ "macro_f1": 0.9339174627621376,
+ "ece": 0.03790626053297799,
+ "brier": 0.09748790744422886,
+ "nll": 0.2233060249382501,
+ "aurc": 0.015727048809451808,
+ "mean_confidence": 0.9727651926995691,
+ "acc_at_50_coverage": 0.9866666666666667,
+ "acc_at_80_coverage": 0.9854166666666667
+ },
+ "seconds": 2.13,
+ "dropped_questions": 0,
+ "questions_per_second": 281.3
+ }
+ },
+ "en.boolq": {
+ "laya": {
+ "calibrated": {
+ "n": 600,
+ "accuracy": 0.83,
+ "macro_f1": 0.8196095076400679,
+ "ece": 0.09610106838662799,
+ "brier": 0.2731780696217012,
+ "nll": 0.46556886296991934,
+ "aurc": 0.07288424964326473,
+ "mean_confidence": 0.9161033363989168,
+ "acc_at_50_coverage": 0.9366666666666666,
+ "acc_at_80_coverage": 0.8833333333333333
+ },
+ "raw": {
+ "n": 600,
+ "accuracy": 0.83,
+ "macro_f1": 0.8196095076400679,
+ "ece": 0.1497538899574864,
+ "brier": 0.3128681086108246,
+ "nll": 0.7678904569264094,
+ "aurc": 0.07288424964326473,
+ "mean_confidence": 0.9715743387844664,
+ "acc_at_50_coverage": 0.9366666666666666,
+ "acc_at_80_coverage": 0.8833333333333333
+ },
+ "seconds": 8.36,
+ "dropped_questions": 0,
+ "questions_per_second": 71.8
+ },
+ "laya-multilingual": {
+ "calibrated": {
+ "n": 600,
+ "accuracy": 0.7866666666666666,
+ "macro_f1": 0.7732399522859066,
+ "ece": 0.14529158506642545,
+ "brier": 0.35721659783636767,
+ "nll": 0.7001630668886732,
+ "aurc": 0.1102122813350894,
+ "mean_confidence": 0.9255312262359776,
+ "acc_at_50_coverage": 0.8933333333333333,
+ "acc_at_80_coverage": 0.8270833333333333
+ },
+ "raw": {
+ "n": 600,
+ "accuracy": 0.7866666666666666,
+ "macro_f1": 0.7732399522859066,
+ "ece": 0.14529158506642545,
+ "brier": 0.35721659783636767,
+ "nll": 0.7001630668886732,
+ "aurc": 0.1102122813350894,
+ "mean_confidence": 0.9255312262359776,
+ "acc_at_50_coverage": 0.8933333333333333,
+ "acc_at_80_coverage": 0.8270833333333333
+ },
+ "seconds": 3.98,
+ "dropped_questions": 0,
+ "questions_per_second": 150.8
+ }
+ }
+ },
+ "typed_decisions_matched_512": {
+ "laya": {
+ "n": 2000,
+ "accuracy": 0.362,
+ "macro_f1": 0.3273577076400338,
+ "ece": 0.17410090455022942,
+ "brier": 0.7496143221875959,
+ "nll": 1.3224634387781617,
+ "aurc": 0.5928492917684455,
+ "mean_confidence": 0.5361009045502294,
+ "acc_at_50_coverage": 0.427,
+ "acc_at_80_coverage": 0.386875
+ },
+ "laya-multilingual": {
+ "n": 2000,
+ "accuracy": 0.3495,
+ "macro_f1": 0.3481722969312257,
+ "ece": 0.3165801393685362,
+ "brier": 0.8917435315799593,
+ "nll": 1.8267597640548572,
+ "aurc": 0.5579990209884584,
+ "mean_confidence": 0.6624303215125597,
+ "acc_at_50_coverage": 0.414,
+ "acc_at_80_coverage": 0.36625
+ }
+ },
+ "latency": {
+ "laya": {
+ "1_questions": {
+ "p50_ms": 39.5,
+ "p95_ms": 44.8,
+ "ms_per_question": 39.5
+ },
+ "5_questions": {
+ "p50_ms": 84.5,
+ "p95_ms": 86.0,
+ "ms_per_question": 16.89
+ },
+ "10_questions": {
+ "p50_ms": 158.6,
+ "p95_ms": 160.1,
+ "ms_per_question": 15.86
+ },
+ "50_questions": {
+ "p50_ms": 771.3,
+ "p95_ms": 806.7,
+ "ms_per_question": 15.43
+ }
+ },
+ "laya-multilingual": {
+ "1_questions": {
+ "p50_ms": 32.8,
+ "p95_ms": 38.8,
+ "ms_per_question": 32.83
+ },
+ "5_questions": {
+ "p50_ms": 40.1,
+ "p95_ms": 43.5,
+ "ms_per_question": 8.02
+ },
+ "10_questions": {
+ "p50_ms": 72.3,
+ "p95_ms": 74.4,
+ "ms_per_question": 7.23
+ },
+ "50_questions": {
+ "p50_ms": 337.4,
+ "p95_ms": 693.7,
+ "ms_per_question": 6.75
+ }
+ }
+ },
+ "option_order_robustness": {
+ "laya": {
+ "massive_intent.en": {
+ "n": 200,
+ "flip_rate": 0.15
+ },
+ "en.emotion": {
+ "n": 200,
+ "flip_rate": 0.04
+ },
+ "xnli.en": {
+ "n": 200,
+ "flip_rate": 0.0
+ }
+ },
+ "laya-multilingual": {
+ "massive_intent.en": {
+ "n": 200,
+ "flip_rate": 0.23
+ },
+ "en.emotion": {
+ "n": 200,
+ "flip_rate": 0.09
+ },
+ "xnli.en": {
+ "n": 200,
+ "flip_rate": 0.015
+ }
+ }
+ },
+ "calibration_repair": {
+ "laya": {
+ "per_suite": {
+ "typed_decisions": {
+ "n_heldout": 1000,
+ "fitted_temperatures": {
+ "choice:3-5": 3.9261,
+ "noul:2": 2.0205,
+ "score:3-5": 3.0698
+ },
+ "ece_shipped": 0.2071110174317399,
+ "ece_refit": 0.12865648553068806,
+ "nll_shipped": 1.328208659782999,
+ "nll_refit": 1.2397667208426046,
+ "accuracy": 0.338
+ },
+ "massive_intent.en": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 1.5414
+ },
+ "ece_shipped": 0.2562159827237908,
+ "ece_refit": 0.11403699556212168,
+ "nll_shipped": 5.865505434392175,
+ "nll_refit": 1.0140600000067173,
+ "accuracy": 0.74
+ },
+ "massive_intent.de": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.3419
+ },
+ "ece_shipped": 0.5462625472109808,
+ "ece_refit": 0.09248203952235462,
+ "nll_shipped": 12.57482900519753,
+ "nll_refit": 1.9823033978431583,
+ "accuracy": 0.43333333333333335
+ },
+ "massive_intent.fr": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 1.9714
+ },
+ "ece_shipped": 0.5576747681888061,
+ "ece_refit": 0.11579583487750908,
+ "nll_shipped": 12.113473172991581,
+ "nll_refit": 1.6719642883309838,
+ "accuracy": 0.41333333333333333
+ },
+ "massive_intent.es": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.2294
+ },
+ "ece_shipped": 0.5322226422693049,
+ "ece_refit": 0.09213948384507138,
+ "nll_shipped": 12.050862077977984,
+ "nll_refit": 1.8740372427815475,
+ "accuracy": 0.4533333333333333
+ },
+ "massive_intent.pt": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.5841
+ },
+ "ece_shipped": 0.5235085119457203,
+ "ece_refit": 0.17951008108333782,
+ "nll_shipped": 12.221158727957615,
+ "nll_refit": 1.8423784639336334,
+ "accuracy": 0.4666666666666667
+ },
+ "massive_intent.ru": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.782
+ },
+ "ece_shipped": 0.6801221409141907,
+ "ece_refit": 0.06785126111487216,
+ "nll_shipped": 16.630473789135806,
+ "nll_refit": 2.4015938125011953,
+ "accuracy": 0.3
+ },
+ "massive_intent.tr": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 4.664
+ },
+ "ece_shipped": 0.7696493795726053,
+ "ece_refit": 0.06573726881689713,
+ "nll_shipped": 20.339742620957256,
+ "nll_refit": 2.7917378955701535,
+ "accuracy": 0.19333333333333333
+ },
+ "massive_intent.ar": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 5.4059
+ },
+ "ece_shipped": 0.7827356558020273,
+ "ece_refit": 0.043154654024976154,
+ "nll_shipped": 20.663786998267785,
+ "nll_refit": 2.881006518148397,
+ "accuracy": 0.15333333333333332
+ },
+ "massive_intent.hi": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 8.8425
+ },
+ "ece_shipped": 0.854953810628851,
+ "ece_refit": 0.037505765311847995,
+ "nll_shipped": 20.83134572134706,
+ "nll_refit": 2.943494345947451,
+ "accuracy": 0.09333333333333334
+ },
+ "massive_intent.ta": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 5.5405
+ },
+ "ece_shipped": 0.8039968607819044,
+ "ece_refit": 0.03421481533002084,
+ "nll_shipped": 19.612842933352653,
+ "nll_refit": 2.944986645721345,
+ "accuracy": 0.12
+ },
+ "massive_intent.zh-CN": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.0708
+ },
+ "ece_shipped": 0.40549573789270327,
+ "ece_refit": 0.08950865278434557,
+ "nll_shipped": 9.813177433742595,
+ "nll_refit": 1.5782438333241908,
+ "accuracy": 0.5933333333333334
+ },
+ "massive_intent.ja": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 1.8311
+ },
+ "ece_shipped": 0.42395100104886724,
+ "ece_refit": 0.1359537248702114,
+ "nll_shipped": 10.405710162642912,
+ "nll_refit": 1.7061709608589148,
+ "accuracy": 0.56
+ },
+ "massive_intent.ko": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 5.2745
+ },
+ "ece_shipped": 0.8638654472702805,
+ "ece_refit": 0.012764949526979565,
+ "nll_shipped": 21.82418989141709,
+ "nll_refit": 2.948163107719538,
+ "accuracy": 0.09333333333333334
+ },
+ "massive_intent.sw": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 6.7458
+ },
+ "ece_shipped": 0.8691068671314688,
+ "ece_refit": 0.0202221602304236,
+ "nll_shipped": 21.5339170347149,
+ "nll_refit": 2.9634040289307078,
+ "accuracy": 0.08
+ },
+ "massive_scenario.en": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.2294
+ },
+ "ece_shipped": 0.3815547151453086,
+ "ece_refit": 0.13146043085039616,
+ "nll_shipped": 10.22545791808144,
+ "nll_refit": 1.3921974143323663,
+ "accuracy": 0.62
+ },
+ "massive_scenario.de": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.8513
+ },
+ "ece_shipped": 0.6398830722839038,
+ "ece_refit": 0.09444310212762598,
+ "nll_shipped": 14.795613787775219,
+ "nll_refit": 2.1076492998547542,
+ "accuracy": 0.32
+ },
+ "massive_scenario.fr": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.1752
+ },
+ "ece_shipped": 0.483678749152581,
+ "ece_refit": 0.12165917488468315,
+ "nll_shipped": 11.072922732788207,
+ "nll_refit": 1.6632025303048328,
+ "accuracy": 0.5
+ },
+ "massive_scenario.es": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.5841
+ },
+ "ece_shipped": 0.5777992521171543,
+ "ece_refit": 0.1005032276156047,
+ "nll_shipped": 14.030134220612933,
+ "nll_refit": 2.044750694163988,
+ "accuracy": 0.4
+ },
+ "massive_scenario.pt": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.7144
+ },
+ "ece_shipped": 0.5417289841728091,
+ "ece_refit": 0.0973002974443076,
+ "nll_shipped": 13.572473995277965,
+ "nll_refit": 1.922970066535057,
+ "accuracy": 0.44666666666666666
+ },
+ "massive_scenario.ru": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.7144
+ },
+ "ece_shipped": 0.6569873694612363,
+ "ece_refit": 0.04551537841388301,
+ "nll_shipped": 16.419733500668183,
+ "nll_refit": 2.3359881701558023,
+ "accuracy": 0.32666666666666666
+ },
+ "massive_scenario.tr": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 4.2268
+ },
+ "ece_shipped": 0.7061936198618928,
+ "ece_refit": 0.06851599123705755,
+ "nll_shipped": 17.296401139792014,
+ "nll_refit": 2.5363076344706665,
+ "accuracy": 0.26
+ },
+ "massive_scenario.ar": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 4.7801
+ },
+ "ece_shipped": 0.8163170012715647,
+ "ece_refit": 0.0474423323304447,
+ "nll_shipped": 20.728188554378566,
+ "nll_refit": 2.7979794977740107,
+ "accuracy": 0.14666666666666667
+ },
+ "massive_scenario.hi": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 7.6289
+ },
+ "ece_shipped": 0.8660562191576803,
+ "ece_refit": 0.004584565442225005,
+ "nll_shipped": 21.782254412766893,
+ "nll_refit": 2.835531392620142,
+ "accuracy": 0.1
+ },
+ "massive_scenario.ta": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 6.1135
+ },
+ "ece_shipped": 0.8325140639998705,
+ "ece_refit": 0.07807608446121986,
+ "nll_shipped": 21.53368640920763,
+ "nll_refit": 2.8077670852380985,
+ "accuracy": 0.14
+ },
+ "massive_scenario.zh-CN": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.285
+ },
+ "ece_shipped": 0.46110257402566157,
+ "ece_refit": 0.09044395129929148,
+ "nll_shipped": 10.77292831665002,
+ "nll_refit": 1.6913117170360654,
+ "accuracy": 0.5266666666666666
+ },
+ "massive_scenario.ja": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.1752
+ },
+ "ece_shipped": 0.4881034787495095,
+ "ece_refit": 0.1119030215337075,
+ "nll_shipped": 11.324819142335816,
+ "nll_refit": 1.644154638119622,
+ "accuracy": 0.4866666666666667
+ },
+ "massive_scenario.ko": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 6.2658
+ },
+ "ece_shipped": 0.8350570565183021,
+ "ece_refit": 0.021565188964428522,
+ "nll_shipped": 21.331749553105603,
+ "nll_refit": 2.797665270418228,
+ "accuracy": 0.13333333333333333
+ },
+ "massive_scenario.sw": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 6.1135
+ },
+ "ece_shipped": 0.8135882896986645,
+ "ece_refit": 0.028498910373293804,
+ "nll_shipped": 20.22271430726738,
+ "nll_refit": 2.795054831143482,
+ "accuracy": 0.14
+ },
+ "xnli.en": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 2.4002
+ },
+ "ece_shipped": 0.07289677206460535,
+ "ece_refit": 0.08853303576022854,
+ "nll_shipped": 0.3345960511795842,
+ "nll_refit": 0.3437815148064849,
+ "accuracy": 0.88
+ },
+ "xnli.de": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 3.4716
+ },
+ "ece_shipped": 0.22490857541364537,
+ "ece_refit": 0.11641844789459176,
+ "nll_shipped": 0.9709805880887378,
+ "nll_refit": 0.8199119398248624,
+ "accuracy": 0.6333333333333333
+ },
+ "xnli.fr": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 3.6467
+ },
+ "ece_shipped": 0.16646567713583152,
+ "ece_refit": 0.06935764297795377,
+ "nll_shipped": 0.8149368257371297,
+ "nll_refit": 0.721784713245583,
+ "accuracy": 0.6933333333333334
+ },
+ "xnli.es": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 3.3049
+ },
+ "ece_shipped": 0.16094019064904574,
+ "ece_refit": 0.0710883876192177,
+ "nll_shipped": 0.8040389210247584,
+ "nll_refit": 0.7178860649223204,
+ "accuracy": 0.6866666666666666
+ },
+ "xnli.ru": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 3.9261
+ },
+ "ece_shipped": 0.2591002284575789,
+ "ece_refit": 0.09264246809537272,
+ "nll_shipped": 1.1705031954952403,
+ "nll_refit": 0.9191745262303057,
+ "accuracy": 0.5866666666666667
+ },
+ "xnli.tr": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 9.5198
+ },
+ "ece_shipped": 0.37022719319502834,
+ "ece_refit": 0.08047867025819397,
+ "nll_shipped": 1.4960379906771686,
+ "nll_refit": 1.0828012031471597,
+ "accuracy": 0.36666666666666664
+ },
+ "xnli.ar": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 8.6276
+ },
+ "ece_shipped": 0.29209360893026326,
+ "ece_refit": 0.06329832543949335,
+ "nll_shipped": 1.2892057415135658,
+ "nll_refit": 1.0539604556939484,
+ "accuracy": 0.4533333333333333
+ },
+ "xnli.hi": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 10.0
+ },
+ "ece_shipped": 0.2884204812865509,
+ "ece_refit": 0.062321224097440786,
+ "nll_shipped": 1.3478000099693062,
+ "nll_refit": 1.0813943966005897,
+ "accuracy": 0.41333333333333333
+ },
+ "xnli.ur": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 10.0
+ },
+ "ece_shipped": 0.2886956804091492,
+ "ece_refit": 0.031778266269963856,
+ "nll_shipped": 1.3295301273849722,
+ "nll_refit": 1.0728864514477925,
+ "accuracy": 0.4
+ },
+ "xnli.vi": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 4.1241
+ },
+ "ece_shipped": 0.15791339705634688,
+ "ece_refit": 0.06114141557091057,
+ "nll_shipped": 1.0300991074899297,
+ "nll_refit": 0.9465837284644331,
+ "accuracy": 0.5733333333333334
+ },
+ "xnli.th": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 5.6785
+ },
+ "ece_shipped": 0.19427425907207962,
+ "ece_refit": 0.043280618209652684,
+ "nll_shipped": 1.1883367978775616,
+ "nll_refit": 1.0611018511269417,
+ "accuracy": 0.42
+ },
+ "xnli.el": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 5.2745
+ },
+ "ece_shipped": 0.20863204752259085,
+ "ece_refit": 0.05938958684029479,
+ "nll_shipped": 1.1206284889886116,
+ "nll_refit": 0.9634498438091695,
+ "accuracy": 0.5666666666666667
+ },
+ "xnli.bg": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 3.7375
+ },
+ "ece_shipped": 0.25795700495898,
+ "ece_refit": 0.10558494899380169,
+ "nll_shipped": 1.2115903992573755,
+ "nll_refit": 0.9892777346276406,
+ "accuracy": 0.52
+ },
+ "xnli.zh": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 3.6467
+ },
+ "ece_shipped": 0.20970005695297053,
+ "ece_refit": 0.09349285486077741,
+ "nll_shipped": 1.004990307040937,
+ "nll_refit": 0.8142545372650651,
+ "accuracy": 0.6866666666666666
+ },
+ "xnli.sw": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 9.757
+ },
+ "ece_shipped": 0.27420298315497604,
+ "ece_refit": 0.048962289518859725,
+ "nll_shipped": 1.2961388406803456,
+ "nll_refit": 1.0525570434208937,
+ "accuracy": 0.4666666666666667
+ },
+ "en.sst5": {
+ "n_heldout": 300,
+ "fitted_temperatures": {
+ "score:3-5": 3.5581
+ },
+ "ece_shipped": 0.23766382422275603,
+ "ece_refit": 0.06455799128975741,
+ "nll_shipped": 1.6900878358865143,
+ "nll_refit": 1.3876706097407598,
+ "accuracy": 0.38
+ },
+ "en.emotion": {
+ "n_heldout": 300,
+ "fitted_temperatures": {
+ "choice:6-10": 2.8513
+ },
+ "ece_shipped": 0.3776578906246687,
+ "ece_refit": 0.13750650494005914,
+ "nll_shipped": 2.4668890351766506,
+ "nll_refit": 1.324234118429839,
+ "accuracy": 0.52
+ },
+ "en.prompt_injections": {
+ "n_heldout": 58,
+ "fitted_temperatures": {
+ "noul:2": 7.2626
+ },
+ "ece_shipped": 0.4421022956073672,
+ "ece_refit": 0.3291279870455534,
+ "nll_shipped": 2.792945084499727,
+ "nll_refit": 0.9516302345219299,
+ "accuracy": 0.5
+ },
+ "en.ag_news": {
+ "n_heldout": 300,
+ "fitted_temperatures": {
+ "choice:3-5": 1.266
+ },
+ "ece_shipped": 0.05126942440656791,
+ "ece_refit": 0.035514331323114996,
+ "nll_shipped": 0.1966735562695727,
+ "nll_refit": 0.1931097097550829,
+ "accuracy": 0.9333333333333333
+ },
+ "en.boolq": {
+ "n_heldout": 300,
+ "fitted_temperatures": {
+ "noul:2": 3.2246
+ },
+ "ece_shipped": 0.10215590522736365,
+ "ece_refit": 0.051566417833373256,
+ "nll_shipped": 0.5104874432163632,
+ "nll_refit": 0.43038591367218393,
+ "accuracy": 0.8266666666666667
+ }
+ },
+ "mean_ece_shipped": 0.4656,
+ "mean_ece_refit": 0.0812
+ },
+ "laya-multilingual": {
+ "per_suite": {
+ "typed_decisions": {
+ "n_heldout": 1000,
+ "fitted_temperatures": {
+ "choice:3-5": 7.2626,
+ "noul:2": 10.0,
+ "score:3-5": 6.4219
+ },
+ "ece_shipped": 0.31353688166253296,
+ "ece_refit": 0.05249973627123162,
+ "nll_shipped": 1.7826284099077552,
+ "nll_refit": 1.1529043184906664,
+ "accuracy": 0.351
+ },
+ "massive_intent.en": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.285
+ },
+ "ece_shipped": 0.28598488876984407,
+ "ece_refit": 0.11981541716793241,
+ "nll_shipped": 2.3166463541043396,
+ "nll_refit": 1.5109594902887804,
+ "accuracy": 0.6066666666666667
+ },
+ "massive_intent.de": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.4002
+ },
+ "ece_shipped": 0.39419152284441683,
+ "ece_refit": 0.11316109324129707,
+ "nll_shipped": 2.9830588533749482,
+ "nll_refit": 1.9610975344566464,
+ "accuracy": 0.43333333333333335
+ },
+ "massive_intent.fr": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.3419
+ },
+ "ece_shipped": 0.33182806315184016,
+ "ece_refit": 0.1650682537772552,
+ "nll_shipped": 2.8198802270685617,
+ "nll_refit": 1.8536174658295161,
+ "accuracy": 0.5
+ },
+ "massive_intent.es": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.0708
+ },
+ "ece_shipped": 0.3747860038944444,
+ "ece_refit": 0.16875478198848332,
+ "nll_shipped": 2.757769866366103,
+ "nll_refit": 1.883079560638871,
+ "accuracy": 0.43333333333333335
+ },
+ "massive_intent.pt": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.5841
+ },
+ "ece_shipped": 0.34120944724576563,
+ "ece_refit": 0.11388291274333108,
+ "nll_shipped": 2.8307898847131425,
+ "nll_refit": 1.8199963148617633,
+ "accuracy": 0.5133333333333333
+ },
+ "massive_intent.ru": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.3419
+ },
+ "ece_shipped": 0.3499191112180423,
+ "ece_refit": 0.12980454365283517,
+ "nll_shipped": 2.8491438768356323,
+ "nll_refit": 1.9214121580317896,
+ "accuracy": 0.4533333333333333
+ },
+ "massive_intent.tr": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.6484
+ },
+ "ece_shipped": 0.33391590405482796,
+ "ece_refit": 0.10349393836177559,
+ "nll_shipped": 2.8017043190240036,
+ "nll_refit": 1.9282719192318487,
+ "accuracy": 0.47333333333333333
+ },
+ "massive_intent.ar": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.46
+ },
+ "ece_shipped": 0.4099836963498909,
+ "ece_refit": 0.10917916558317786,
+ "nll_shipped": 3.1613960902759275,
+ "nll_refit": 2.25141683066659,
+ "accuracy": 0.3466666666666667
+ },
+ "massive_intent.hi": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.3419
+ },
+ "ece_shipped": 0.4378035227689349,
+ "ece_refit": 0.14973280309187975,
+ "nll_shipped": 3.5750849346908624,
+ "nll_refit": 2.342025405755958,
+ "accuracy": 0.34
+ },
+ "massive_intent.ta": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.9224
+ },
+ "ece_shipped": 0.5282864038131523,
+ "ece_refit": 0.1301923044510951,
+ "nll_shipped": 4.225495743306456,
+ "nll_refit": 2.7193019660496662,
+ "accuracy": 0.2
+ },
+ "massive_intent.zh-CN": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.0708
+ },
+ "ece_shipped": 0.2956479305219314,
+ "ece_refit": 0.11033723906598177,
+ "nll_shipped": 2.2249843788919748,
+ "nll_refit": 1.5650537999224634,
+ "accuracy": 0.58
+ },
+ "massive_intent.ja": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.0205
+ },
+ "ece_shipped": 0.30483003324052754,
+ "ece_refit": 0.14830564994188358,
+ "nll_shipped": 2.3891544641828086,
+ "nll_refit": 1.6551049648097012,
+ "accuracy": 0.5466666666666666
+ },
+ "massive_intent.ko": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.0708
+ },
+ "ece_shipped": 0.32002650648667197,
+ "ece_refit": 0.10080463213987816,
+ "nll_shipped": 2.6602817717599048,
+ "nll_refit": 1.9099858554110467,
+ "accuracy": 0.48
+ },
+ "massive_intent.sw": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 3.7375
+ },
+ "ece_shipped": 0.49570577568341406,
+ "ece_refit": 0.08349416187415526,
+ "nll_shipped": 4.279076139734234,
+ "nll_refit": 2.6662240908229258,
+ "accuracy": 0.2
+ },
+ "massive_scenario.en": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.4002
+ },
+ "ece_shipped": 0.3428270841005442,
+ "ece_refit": 0.10430091131513634,
+ "nll_shipped": 2.826442913358127,
+ "nll_refit": 1.7296360738235776,
+ "accuracy": 0.5133333333333333
+ },
+ "massive_scenario.de": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.46
+ },
+ "ece_shipped": 0.4098164059892938,
+ "ece_refit": 0.1103006818436626,
+ "nll_shipped": 3.0195190523299282,
+ "nll_refit": 1.883461519212016,
+ "accuracy": 0.44
+ },
+ "massive_scenario.fr": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.5213
+ },
+ "ece_shipped": 0.4442562009642703,
+ "ece_refit": 0.16025393217549574,
+ "nll_shipped": 3.3766059898271252,
+ "nll_refit": 1.9413440894718588,
+ "accuracy": 0.4266666666666667
+ },
+ "massive_scenario.es": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.6484
+ },
+ "ece_shipped": 0.4109456966606466,
+ "ece_refit": 0.10198578234795756,
+ "nll_shipped": 3.2636908758703864,
+ "nll_refit": 1.9252080039041213,
+ "accuracy": 0.44
+ },
+ "massive_scenario.pt": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.9224
+ },
+ "ece_shipped": 0.4105148693608891,
+ "ece_refit": 0.1138316207725879,
+ "nll_shipped": 3.3875343693016906,
+ "nll_refit": 1.9175783143767824,
+ "accuracy": 0.4666666666666667
+ },
+ "massive_scenario.ru": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.6484
+ },
+ "ece_shipped": 0.41903289757915385,
+ "ece_refit": 0.11771325878816861,
+ "nll_shipped": 3.4690396514901765,
+ "nll_refit": 2.02048805544728,
+ "accuracy": 0.44666666666666666
+ },
+ "massive_scenario.tr": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.46
+ },
+ "ece_shipped": 0.3588353003356555,
+ "ece_refit": 0.13829033714669553,
+ "nll_shipped": 2.432523043817323,
+ "nll_refit": 1.645665100066707,
+ "accuracy": 0.4866666666666667
+ },
+ "massive_scenario.ar": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.6484
+ },
+ "ece_shipped": 0.41920881717954284,
+ "ece_refit": 0.10673559257438134,
+ "nll_shipped": 3.45497223390683,
+ "nll_refit": 2.1575491605592583,
+ "accuracy": 0.3933333333333333
+ },
+ "massive_scenario.hi": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.4002
+ },
+ "ece_shipped": 0.41481688542538986,
+ "ece_refit": 0.15370426062587889,
+ "nll_shipped": 3.3578108561910596,
+ "nll_refit": 2.0777000340841267,
+ "accuracy": 0.41333333333333333
+ },
+ "massive_scenario.ta": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.5841
+ },
+ "ece_shipped": 0.48257092406411306,
+ "ece_refit": 0.1279809797192578,
+ "nll_shipped": 3.8504959505951137,
+ "nll_refit": 2.4452486759577177,
+ "accuracy": 0.26666666666666666
+ },
+ "massive_scenario.zh-CN": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.5213
+ },
+ "ece_shipped": 0.3981881387108269,
+ "ece_refit": 0.12326342022911907,
+ "nll_shipped": 2.927004837667629,
+ "nll_refit": 1.7716489027068876,
+ "accuracy": 0.4533333333333333
+ },
+ "massive_scenario.ja": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.5213
+ },
+ "ece_shipped": 0.3621028758403154,
+ "ece_refit": 0.13926267642528437,
+ "nll_shipped": 3.136840178283236,
+ "nll_refit": 1.8354044492411636,
+ "accuracy": 0.5066666666666667
+ },
+ "massive_scenario.ko": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 2.9224
+ },
+ "ece_shipped": 0.3768861443562832,
+ "ece_refit": 0.0977403817894932,
+ "nll_shipped": 2.9273708306086474,
+ "nll_refit": 1.9155954564982918,
+ "accuracy": 0.42
+ },
+ "massive_scenario.sw": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:11+": 3.5581
+ },
+ "ece_shipped": 0.506361296484783,
+ "ece_refit": 0.11207734739543539,
+ "nll_shipped": 4.700826623994613,
+ "nll_refit": 2.5552862898811015,
+ "accuracy": 0.2866666666666667
+ },
+ "xnli.en": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 1.8311
+ },
+ "ece_shipped": 0.09900583157195271,
+ "ece_refit": 0.06503906718116235,
+ "nll_shipped": 0.3925995455412425,
+ "nll_refit": 0.35876187759884653,
+ "accuracy": 0.8533333333333334
+ },
+ "xnli.de": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 2.46
+ },
+ "ece_shipped": 0.12594098727915568,
+ "ece_refit": 0.06500990874992986,
+ "nll_shipped": 0.653860650712282,
+ "nll_refit": 0.5067271285990049,
+ "accuracy": 0.8133333333333334
+ },
+ "xnli.fr": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 2.5841
+ },
+ "ece_shipped": 0.11460899279515438,
+ "ece_refit": 0.1072080948025079,
+ "nll_shipped": 0.6051545830743009,
+ "nll_refit": 0.5015551360343191,
+ "accuracy": 0.82
+ },
+ "xnli.es": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 2.285
+ },
+ "ece_shipped": 0.1176948222453516,
+ "ece_refit": 0.06262936231022864,
+ "nll_shipped": 0.5866791824036008,
+ "nll_refit": 0.49183324427122804,
+ "accuracy": 0.82
+ },
+ "xnli.ru": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 3.0698
+ },
+ "ece_shipped": 0.17366405837906923,
+ "ece_refit": 0.08228016201755169,
+ "nll_shipped": 0.9308139508033506,
+ "nll_refit": 0.6476032852724123,
+ "accuracy": 0.76
+ },
+ "xnli.tr": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 2.9951
+ },
+ "ece_shipped": 0.19836158969915463,
+ "ece_refit": 0.10254875033243725,
+ "nll_shipped": 0.9277086355768465,
+ "nll_refit": 0.6556320188131841,
+ "accuracy": 0.74
+ },
+ "xnli.ar": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 3.3049
+ },
+ "ece_shipped": 0.16700123516881432,
+ "ece_refit": 0.09902966264249458,
+ "nll_shipped": 0.8953173210214982,
+ "nll_refit": 0.6455067278444552,
+ "accuracy": 0.76
+ },
+ "xnli.hi": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 3.5581
+ },
+ "ece_shipped": 0.21483863590232802,
+ "ece_refit": 0.08204340573901349,
+ "nll_shipped": 1.1766358162380428,
+ "nll_refit": 0.7518036557992201,
+ "accuracy": 0.68
+ },
+ "xnli.ur": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 3.7375
+ },
+ "ece_shipped": 0.19644100874145254,
+ "ece_refit": 0.09763198576088242,
+ "nll_shipped": 1.0413509566733983,
+ "nll_refit": 0.7245230970472671,
+ "accuracy": 0.7333333333333333
+ },
+ "xnli.vi": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 3.3049
+ },
+ "ece_shipped": 0.17600979149888385,
+ "ece_refit": 0.09703177121512795,
+ "nll_shipped": 1.0306447465645896,
+ "nll_refit": 0.6622157369318552,
+ "accuracy": 0.76
+ },
+ "xnli.th": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 3.4716
+ },
+ "ece_shipped": 0.16614142465028775,
+ "ece_refit": 0.08809598765913614,
+ "nll_shipped": 0.9550613292832744,
+ "nll_refit": 0.6929707644114412,
+ "accuracy": 0.7333333333333333
+ },
+ "xnli.el": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 2.9951
+ },
+ "ece_shipped": 0.18505427976039845,
+ "ece_refit": 0.0619934233555169,
+ "nll_shipped": 0.9681318335182048,
+ "nll_refit": 0.6623250177254513,
+ "accuracy": 0.72
+ },
+ "xnli.bg": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 2.4002
+ },
+ "ece_shipped": 0.13546796723698223,
+ "ece_refit": 0.06411651174590313,
+ "nll_shipped": 0.7253311578700296,
+ "nll_refit": 0.5350767796618592,
+ "accuracy": 0.8133333333333334
+ },
+ "xnli.zh": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 2.5841
+ },
+ "ece_shipped": 0.14274233933256558,
+ "ece_refit": 0.083026544681253,
+ "nll_shipped": 0.8243758967604422,
+ "nll_refit": 0.612193238305549,
+ "accuracy": 0.7666666666666667
+ },
+ "xnli.sw": {
+ "n_heldout": 150,
+ "fitted_temperatures": {
+ "choice:3-5": 3.4716
+ },
+ "ece_shipped": 0.29730770690435115,
+ "ece_refit": 0.11350413359716122,
+ "nll_shipped": 1.4013281162840883,
+ "nll_refit": 0.8379049118827668,
+ "accuracy": 0.58
+ },
+ "en.sst5": {
+ "n_heldout": 300,
+ "fitted_temperatures": {
+ "score:3-5": 10.0
+ },
+ "ece_shipped": 0.5241645000058643,
+ "ece_refit": 0.03089034893874691,
+ "nll_shipped": 3.832620682023934,
+ "nll_refit": 1.5850353324296294,
+ "accuracy": 0.28
+ },
+ "en.emotion": {
+ "n_heldout": 300,
+ "fitted_temperatures": {
+ "choice:6-10": 3.1462
+ },
+ "ece_shipped": 0.40379864214963257,
+ "ece_refit": 0.08741204055524324,
+ "nll_shipped": 2.637519408908119,
+ "nll_refit": 1.4416251362588335,
+ "accuracy": 0.46
+ },
+ "en.prompt_injections": {
+ "n_heldout": 58,
+ "fitted_temperatures": {
+ "noul:2": 6.2658
+ },
+ "ece_shipped": 0.4514191955987473,
+ "ece_refit": 0.2407982144094497,
+ "nll_shipped": 2.251858103333174,
+ "nll_refit": 0.743234178474742,
+ "accuracy": 0.4827586206896552
+ },
+ "en.ag_news": {
+ "n_heldout": 300,
+ "fitted_temperatures": {
+ "choice:3-5": 1.5039
+ },
+ "ece_shipped": 0.04495656180857554,
+ "ece_refit": 0.010973389250756511,
+ "nll_shipped": 0.24864777119159517,
+ "nll_refit": 0.2083974803565066,
+ "accuracy": 0.9333333333333333
+ },
+ "en.boolq": {
+ "n_heldout": 300,
+ "fitted_temperatures": {
+ "noul:2": 2.8513
+ },
+ "ece_shipped": 0.15205137686074968,
+ "ece_refit": 0.04382109708993337,
+ "nll_shipped": 0.6707500082967005,
+ "nll_refit": 0.4659609286304744,
+ "accuracy": 0.7933333333333333
+ }
+ },
+ "mean_ece_shipped": 0.3135,
+ "mean_ece_refit": 0.1059
+ }
+ },
+ "caveats": {
+ "context_window": "laya max_len=512, laya-multilingual max_len=1024 (shipped configs). See typed_decisions_matched_512 for the equal-budget control.",
+ "shipped_calibration": "laya ships fitted temperatures; laya-multilingual ships all-1.0 (uncalibrated). See calibration_repair for the out-of-sample refit.",
+ "training_overlap": "ag_news and boolq were in Laya's training mix (retention, not generalisation). sst5, emotion, prompt_injections, banking77 were held out. Neither checkpoint trained on typed-decisions, MASSIVE or XNLI."
+ },
+ "summary": {
+ "typed_decisions": {
+ "laya": {
+ "n": 2000,
+ "accuracy": 0.362,
+ "macro_f1": 0.3273577076400338,
+ "ece": 0.17410090455022942,
+ "brier": 0.7496143221875959,
+ "nll": 1.3224634387781617,
+ "aurc": 0.5928492917684455,
+ "mean_confidence": 0.5361009045502294,
+ "acc_at_50_coverage": 0.427,
+ "acc_at_80_coverage": 0.386875
+ },
+ "laya-multilingual": {
+ "n": 2000,
+ "accuracy": 0.3515,
+ "macro_f1": 0.34975321170122525,
+ "ece": 0.31443177617478313,
+ "brier": 0.8904323210391285,
+ "nll": 1.8242712912901595,
+ "aurc": 0.5562863247537705,
+ "mean_confidence": 0.6629025476146312,
+ "acc_at_50_coverage": 0.414,
+ "acc_at_80_coverage": 0.368125
+ }
+ },
+ "massive_intent": {
+ "laya": {
+ "macro_accuracy": 0.3405,
+ "macro_ece": 0.6266,
+ "n_questions": 4200,
+ "n_suites": 14
+ },
+ "laya-multilingual": {
+ "macro_accuracy": 0.466,
+ "macro_ece": 0.3421,
+ "n_questions": 4200,
+ "n_suites": 14
+ }
+ },
+ "massive_scenario": {
+ "laya": {
+ "macro_accuracy": 0.3036,
+ "macro_ece": 0.6707,
+ "n_questions": 4200,
+ "n_suites": 14
+ },
+ "laya-multilingual": {
+ "macro_accuracy": 0.4474,
+ "macro_ece": 0.3857,
+ "n_questions": 4200,
+ "n_suites": 14
+ }
+ },
+ "xnli": {
+ "laya": {
+ "macro_accuracy": 0.5436,
+ "macro_ece": 0.2296,
+ "n_questions": 4500,
+ "n_suites": 15
+ },
+ "laya-multilingual": {
+ "macro_accuracy": 0.7384,
+ "macro_ece": 0.1813,
+ "n_questions": 4500,
+ "n_suites": 15
+ }
+ },
+ "english": {
+ "laya": {
+ "macro_accuracy": 0.684,
+ "macro_ece": 0.1938,
+ "n_questions": 2516,
+ "n_suites": 5
+ },
+ "laya-multilingual": {
+ "macro_accuracy": 0.6192,
+ "macro_ece": 0.2842,
+ "n_questions": 2516,
+ "n_suites": 5
+ }
+ },
+ "massive_intent_english_vs_rest": {
+ "english": {
+ "laya": 0.7833,
+ "laya-multilingual": 0.6567
+ },
+ "non_english": {
+ "laya": 0.3064,
+ "laya-multilingual": 0.4513
+ }
+ },
+ "massive_scenario_english_vs_rest": {
+ "english": {
+ "laya": 0.6033,
+ "laya-multilingual": 0.56
+ },
+ "non_english": {
+ "laya": 0.2805,
+ "laya-multilingual": 0.4387
+ }
+ },
+ "xnli_english_vs_rest": {
+ "english": {
+ "laya": 0.86,
+ "laya-multilingual": 0.8433
+ },
+ "non_english": {
+ "laya": 0.521,
+ "laya-multilingual": 0.731
+ }
+ }
+ }
+}
\ No newline at end of file
diff --git a/research/scripts/bench_apps.py b/research/scripts/bench_apps.py
new file mode 100644
index 00000000..c0b255ba
--- /dev/null
+++ b/research/scripts/bench_apps.py
@@ -0,0 +1,302 @@
+"""Application-workflow benchmark for the Laya checkpoints, plus the tasks where public
+Jev numbers exist so a like-for-like comparison is possible.
+
+Workflows (the demo Space's tabs), each on real labelled data:
+ 1 support triage banking77 (77-way intent) + customer-support-tickets queue routing
+ 2 email + phishing enron spam + phishing emails
+ 3 LLM guardrails lmsys/toxic-chat jailbreaking flag (held out of Laya training)
+ 4 RAG passage filter MS MARCO passage relevance
+ 5 moderation lmsys/toxic-chat toxicity flag (held out of Laya training)
+ 6 model routing domain classification over gsm8k / mbpp / writing / factual
+
+Jev-comparable tasks (AbdelStark/jev-benchmarks published Jev accuracy on these):
+ ag_news 0.910 | banking77 0.870 | dair emotion 0.480 (Brier 0.846, NLL 5.588)
+
+ USE_TF=0 python3 notebooks/bench_apps.py
+"""
+import gc
+import json
+import math
+import os
+import random
+import sys
+import time
+
+os.environ.setdefault("USE_TF", "0")
+os.environ.setdefault("USE_TORCH", "1")
+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
+
+import numpy as np # noqa: E402
+import torch # noqa: E402
+
+REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, os.path.join(REPO, "laya"))
+sys.path.insert(0, os.path.join(REPO, "notebooks"))
+
+import laya # noqa: E402
+from bench_local import load, metrics, score_cases, softmax_t, temp_for # noqa: E402
+
+OUT = os.path.join(REPO, "app_benchmark_results.json")
+SEED = 13
+N = int(os.environ.get("BENCH_N", "400"))
+
+# Published Jev numbers on identical public datasets (NOT measured here - no TypeSafe API access).
+JEV_PUBLISHED = {
+ "ag_news": {"accuracy": 0.910, "coverage_at_5pct_error": 0.830, "latency_p50_ms": 236,
+ "n": 100, "source": "AbdelStark/jev-benchmarks v0.1.0"},
+ "banking77": {"accuracy": 0.870, "coverage_at_5pct_error": 0.860, "latency_p50_ms": 246,
+ "n": 100, "labels": 72, "source": "AbdelStark/jev-benchmarks v0.1.0"},
+ "emotion": {"accuracy": 0.480, "brier": 0.846, "nll": 5.588, "coverage_at_5pct_error": 0.0,
+ "zero_prob_failures": 0.16, "n": 100, "source": "AbdelStark/jev-benchmarks v0.1.0"},
+ "typed_decisions": {"accuracy": 0.727, "soft_accuracy": 0.580, "brier": 0.148, "ece": 0.144,
+ "score_mae": 0.391, "ms_per_case": 710, "source": "laya repo comparison table"},
+ "_independent_": {"banking77_accuracy": 0.763, "sms_spam_accuracy": 0.930,
+ "permuted_accuracy": 0.767, "ece": 0.246, "latency_p50_ms": "264-276",
+ "option_order_flip_rate": 0.13,
+ "source": "nibzard/decision-model-benchmark"},
+}
+
+SUITES = {}
+
+
+def register(name, cases, gold, **meta):
+ SUITES[name] = {"cases": cases, "gold": gold, "meta": meta}
+ print(" %-26s %4d cases %s" % (name, len(cases), meta.get("note", "")), flush=True)
+
+
+def choice_q(qid, instructions, keys, gold_key):
+ crit = {k: None for k in keys}
+ return {qid: {"type": "choice", "instructions": instructions, "criteria": crit}}, keys.index(gold_key)
+
+
+def build():
+ from datasets import load_dataset
+ rng = random.Random(SEED)
+
+ # ---------------------------------------------------------- Jev-comparable: ag_news
+ try:
+ d = load_dataset("fancyzhx/ag_news", split="test")
+ crit = {"world": "world news and international politics", "sports": "sports",
+ "business": "business and economy", "sci_tech": "science and technology"}
+ keys = list(crit)
+ cases, gold = [], []
+ for r in list(d)[:N]:
+ cases.append(({"article": r["text"]},
+ {"topic": {"type": "choice", "instructions": "What is the topic of `article`?",
+ "criteria": dict(crit)}}))
+ gold.append(keys.index(keys[int(r["label"])]))
+ register("jev.ag_news", cases, gold, note="Jev 0.910", jev="ag_news", in_training=True)
+ except Exception as e:
+ print(" FAIL ag_news", str(e)[:80])
+
+ # ---------------------------------------------------------- Jev-comparable: emotion
+ try:
+ d = load_dataset("dair-ai/emotion", "split", split="test")
+ names = ["sadness", "joy", "love", "anger", "fear", "surprise"]
+ cases, gold = [], []
+ for r in list(d)[:N]:
+ cases.append(({"text": r["text"]},
+ {"emotion": {"type": "choice",
+ "instructions": "Which emotion is most strongly expressed in `text`?",
+ "criteria": {n: None for n in names}}}))
+ gold.append(int(r["label"]))
+ register("jev.emotion", cases, gold, note="Jev 0.480 (Brier 0.846)", jev="emotion",
+ in_training=False)
+ except Exception as e:
+ print(" FAIL emotion", str(e)[:80])
+
+ # ------------------------------------------- Jev-comparable + triage: banking77 (77 labels)
+ try:
+ d = load_dataset("mteb/banking77", split="test")
+ labels = sorted(set(d["label_text"]))
+ cases, gold = [], []
+ for r in list(d)[:N]:
+ qs, gi = choice_q("intent", "Which banking intent does `message` express?",
+ [x.replace("_", " ") for x in labels],
+ r["label_text"].replace("_", " "))
+ cases.append(({"message": r["text"]}, qs)); gold.append(gi)
+ register("jev.banking77_full", cases, gold,
+ note="%d labels at once | Jev 0.870 (72 labels)" % len(labels),
+ jev="banking77", in_training=False, n_labels=len(labels))
+ except Exception as e:
+ print(" FAIL banking77", str(e)[:80])
+
+ # ---------------------------------------------------------- 1. support triage (queues)
+ try:
+ d = load_dataset("Tobi-Bueck/customer-support-tickets", split="train")
+ QUEUES = {"Technical Support": "technical problems, bugs, outages, integrations",
+ "Product Support": "help using a product or feature",
+ "Customer Service": "general account or service questions",
+ "IT Support": "internal IT, devices, access, networks",
+ "Billing and Payments": "invoices, charges, refunds, payment methods",
+ "Returns and Exchanges": "returning or exchanging an item",
+ "Service Outages and Maintenance": "downtime, outages, scheduled maintenance",
+ "Sales and Pre-Sales": "pricing, quotes, buying",
+ "Human Resources": "employment, payroll, leave, hiring",
+ "General Inquiry": "anything else"}
+ keys = list(QUEUES)
+ cases, gold = [], []
+ for r in d:
+ if r.get("language") != "en" or r.get("queue") not in QUEUES or not r.get("body"):
+ continue
+ cases.append(({"subject": r["subject"] or "", "body": r["body"].replace("\\n", "\n")[:3000]},
+ {"queue": {"type": "choice",
+ "instructions": "Which support queue should handle this ticket?",
+ "criteria": dict(QUEUES)}}))
+ gold.append(keys.index(r["queue"]))
+ if len(cases) >= N:
+ break
+ register("app.support_triage", cases, gold, note="10-way queue routing", in_training=True)
+ except Exception as e:
+ print(" FAIL support_triage", str(e)[:80])
+
+ # ---------------------------------------------------------- 2. email + phishing
+ try:
+ d = load_dataset("SetFit/enron_spam", split="test")
+ cases, gold = [], []
+ for r in list(d)[:N]:
+ st = laya.email_state(r.get("subject") or "", (r.get("message") or "")[:3000])
+ cases.append((st, {"is_spam": {"type": "noul",
+ "instructions": "Is this email unsolicited spam or bulk marketing?"}}))
+ gold.append(int(r["label"]))
+ register("app.email_spam", cases, gold, note="enron spam", in_training=True)
+ except Exception as e:
+ print(" FAIL email_spam", str(e)[:80])
+
+ try:
+ d = load_dataset("zefang-liu/phishing-email-dataset", split="train")
+ rows = [r for r in list(d)[:6000]
+ if (r.get("Email Text") or "").strip() and r.get("Email Type") in ("Safe Email", "Phishing Email")]
+ rng.shuffle(rows)
+ cases, gold = [], []
+ for r in rows[:N]:
+ cases.append(({"email": r["Email Text"][:3000]},
+ {"is_phishing": {"type": "noul",
+ "instructions": "Is this email a phishing or scam attempt to steal money, credentials, or personal data?",
+ "criteria": {"true": "phishing, scam, or fraud",
+ "false": "a legitimate email (even if promotional)"}}}))
+ gold.append(int(r["Email Type"] == "Phishing Email"))
+ register("app.phishing", cases, gold, note="phishing emails", in_training=True)
+ except Exception as e:
+ print(" FAIL phishing", str(e)[:80])
+
+ # ------------------------------------- 3 & 5. guardrails + moderation (toxic-chat, HELD OUT)
+ try:
+ d = load_dataset("lmsys/toxic-chat", "toxicchat0124", split="test")
+ rows = [r for r in d if (r.get("user_input") or "").strip()]
+ jb = [r for r in rows if int(r.get("jailbreaking", 0)) == 1][:N // 2]
+ nj = [r for r in rows if int(r.get("jailbreaking", 0)) == 0][:N - len(jb)]
+ mix = jb + nj
+ rng.shuffle(mix)
+ cases, gold = [], []
+ for r in mix:
+ cases.append(({"prompt": r["user_input"][:3000]},
+ {"jailbreak": {"type": "noul",
+ "instructions": "Does `prompt` try to make an AI assistant ignore its rules, policies or system instructions?"}}))
+ gold.append(int(r["jailbreaking"]))
+ register("app.guardrails_jailbreak", cases, gold,
+ note="toxic-chat jailbreaking (HELD OUT)", in_training=False)
+
+ tox = [r for r in rows if int(r.get("toxicity", 0)) == 1][:N // 2]
+ ntox = [r for r in rows if int(r.get("toxicity", 0)) == 0][:N - len(tox)]
+ mix2 = tox + ntox
+ rng.shuffle(mix2)
+ cases, gold = [], []
+ for r in mix2:
+ cases.append(({"post": r["user_input"][:3000]},
+ {"toxic": {"type": "noul",
+ "instructions": "Is `post` toxic: rude, disrespectful or likely to make someone leave the discussion?"}}))
+ gold.append(int(r["toxicity"]))
+ register("app.moderation_toxicity", cases, gold,
+ note="toxic-chat toxicity (HELD OUT)", in_training=False)
+ except Exception as e:
+ print(" FAIL toxic-chat", str(e)[:80])
+
+ # ---------------------------------------------------------- 4. RAG passage filter
+ try:
+ d = load_dataset("microsoft/ms_marco", "v1.1", split="validation")
+ cases, gold = [], []
+ for r in d:
+ texts, sel = r["passages"]["passage_text"], r["passages"]["is_selected"]
+ pos = [t for t, s in zip(texts, sel) if s == 1]
+ neg = [t for t, s in zip(texts, sel) if s == 0]
+ if not pos or not neg:
+ continue
+ take_pos = len(cases) % 2 == 0
+ p = rng.choice(pos if take_pos else neg)
+ cases.append(({"query": r["query"], "passage": p},
+ {"relevant": {"type": "noul",
+ "instructions": "Does `passage` help answer `query`?"}}))
+ gold.append(1 if take_pos else 0)
+ if len(cases) >= N:
+ break
+ register("app.rag_relevance", cases, gold, note="MS MARCO relevance", in_training=True)
+ except Exception as e:
+ print(" FAIL rag", str(e)[:80])
+
+ # ---------------------------------------------------------- 6. model routing (domain)
+ try:
+ DOM = {"code": "software engineering, programming, refactoring, architecture, debugging",
+ "math_or_logic": "mathematics, logic puzzles, proofs, complex calculation",
+ "writing": "creative writing, essays, emails, blog posts, copywriting",
+ "factual_lookup": "facts, definitions, trivia, history",
+ "data_analysis": "statistics, SQL, data manipulation, metrics",
+ "chitchat": "casual conversation, greetings, small talk"}
+ keys = list(DOM)
+ pool = []
+ g = load_dataset("openai/gsm8k", "main", split="test")
+ pool += [(r["question"], "math_or_logic") for r in list(g)[:N // 3]]
+ m = load_dataset("google-research-datasets/mbpp", "full", split="test")
+ pool += [(r["text"], "code") for r in list(m)[:N // 3]]
+ t = load_dataset("fancyzhx/ag_news", split="test")
+ pool += [(r["text"][:400], "factual_lookup") for r in list(t)[:N // 3]]
+ rng.shuffle(pool)
+ cases, gold = [], []
+ for text, dom in pool[:N]:
+ cases.append(({"request": text},
+ {"domain": {"type": "choice",
+ "instructions": "What domain does `request` belong to?",
+ "criteria": dict(DOM)}}))
+ gold.append(keys.index(dom))
+ register("app.model_routing_domain", cases, gold,
+ note="gsm8k/mbpp/ag_news -> domain", in_training=False)
+ except Exception as e:
+ print(" FAIL routing", str(e)[:80])
+
+
+def main():
+ print("=== building suites (N=%d per task) ===\n" % N, flush=True)
+ build()
+ results = {"meta": {"timestamp": time.strftime("%Y-%m-%d %H:%M:%S"), "device": "cpu",
+ "n_per_task": N, "seed": SEED, "laya": laya.__version__},
+ "jev_published": JEV_PUBLISHED, "suites": {}}
+ for mname in ("english", "multilingual", "typed-decisions"):
+ print("\n=== %s ===" % mname, flush=True)
+ ag = load(mname)
+ for sname, S in SUITES.items():
+ try:
+ lgs, idx, secs, dropped = score_cases(ag, S["cases"], tag="%s/%s" % (mname, sname))
+ rows = [(S["gold"][ci], softmax_t(z, temp_for(ag, qt, k)) if z is not None else None)
+ for (ci, _, qt, k), z in zip(idx, lgs)]
+ m = metrics(rows)
+ m["seconds"] = round(secs, 1)
+ m["ms_per_case"] = round(1000 * secs / max(1, len(S["cases"])), 1)
+ m["dropped"] = dropped
+ m.update({k: v for k, v in S["meta"].items() if k != "note"})
+ results["suites"].setdefault(sname, {})[mname] = m
+ jev = JEV_PUBLISHED.get(S["meta"].get("jev") or "", {}).get("accuracy")
+ delta = (" vs Jev %.3f -> %+.3f" % (jev, m["accuracy"] - jev)) if jev else ""
+ print(" %-26s acc %.3f f1 %.3f ECE %.3f %5.1f ms/case%s"
+ % (sname, m["accuracy"], m["macro_f1"], m["ece"], m["ms_per_case"], delta),
+ flush=True)
+ except Exception as e:
+ import traceback; traceback.print_exc()
+ results["suites"].setdefault(sname, {})[mname] = {"error": str(e)[:200]}
+ del ag; gc.collect()
+ json.dump(results, open(OUT, "w"), indent=2)
+ json.dump(results, open(OUT, "w"), indent=2)
+ print("\nwrote %s" % OUT, flush=True)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/research/scripts/bench_latency.py b/research/scripts/bench_latency.py
new file mode 100644
index 00000000..e97ddbb4
--- /dev/null
+++ b/research/scripts/bench_latency.py
@@ -0,0 +1,169 @@
+"""Inference-speed benchmark, including what routing actually costs.
+
+Routing changes the latency picture in two ways:
+ * every call pays language detection (pure Python, no model)
+ * a call that switches checkpoint pays a model load, unless that model is already resident
+
+So there are two regimes -- hot (right model already loaded) and cold (swap) -- and the realistic
+number for a mixed-language workload sits between them, determined by `max_loaded`.
+
+ USE_TF=0 python3 notebooks/bench_latency.py
+"""
+import gc
+import json
+import os
+import statistics
+import sys
+import time
+
+os.environ.setdefault("USE_TF", "0")
+os.environ.setdefault("USE_TORCH", "1")
+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
+
+import numpy as np # noqa: E402
+import torch # noqa: E402
+
+REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, os.path.join(REPO, "laya"))
+
+import laya # noqa: E402
+from laya.lang import analyse # noqa: E402
+from laya.router import Router # noqa: E402
+
+ROOT = os.path.expanduser("~/laya_models")
+MODELS = {"english": os.path.join(ROOT, "laya"),
+ "multilingual": os.path.join(ROOT, "laya-multilingual"),
+ "typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}
+OUT = os.path.join(REPO, "latency_benchmark_results.json")
+
+STATE_EN = {"ticket": {"subject": "Payout failing", "messages": [{"from": "customer",
+ "text": "Hi, my Stripe payouts have failed for 3 days and I am losing sales. Please help ASAP. " * 6}]}}
+STATE_HI = {"ticket": {"subject": "भुगतान विफल", "messages": [{"from": "customer",
+ "text": "मेरा भुगतान तीन दिनों से विफल हो रहा है और मुझे नुकसान हो रहा है, कृपया तुरंत मदद करें। " * 6}]}}
+Q_NOUL = {"type": "noul", "instructions": "Does `ticket.messages[0].text` express urgency?"}
+Q_CHOICE = {"type": "choice", "instructions": "Which team should handle this?",
+ "criteria": {"billing": "payments", "technical": "bugs and integrations", "sales": "pricing"}}
+
+
+def qs(n):
+ return {("q%d" % i): (Q_NOUL if i % 2 else Q_CHOICE) for i in range(n)}
+
+
+def timed(fn, warmup=3, reps=15):
+ for _ in range(warmup):
+ fn()
+ ts = []
+ for _ in range(reps):
+ t = time.perf_counter()
+ fn()
+ ts.append((time.perf_counter() - t) * 1000)
+ return {"p50_ms": round(float(np.percentile(ts, 50)), 2),
+ "p95_ms": round(float(np.percentile(ts, 95)), 2),
+ "mean_ms": round(float(np.mean(ts)), 2),
+ "min_ms": round(float(np.min(ts)), 2)}
+
+
+def main():
+ res = {"meta": {"timestamp": time.strftime("%Y-%m-%d %H:%M:%S"), "device": "cpu",
+ "torch": torch.__version__, "threads": torch.get_num_threads(),
+ "laya": laya.__version__,
+ "note": "CPU numbers. GPU (T4) reference from the Colab run is in "
+ "laya_benchmark_results.json -> latency."}}
+
+ # ---------------------------------------------------------------- 1. detection overhead
+ print("=== 1. language detection overhead (no model) ===", flush=True)
+ det = {}
+ for label, st in (("english", STATE_EN), ("hindi", STATE_HI),
+ ("short english", {"m": "refund me"}),
+ ("large json", {"rows": [{"id": i, "text": "some ticket body here"} for i in range(200)]})):
+ det[label] = timed(lambda s=st: analyse(s), warmup=20, reps=200)
+ print(" %-16s p50 %7.3f ms p95 %7.3f ms" % (label, det[label]["p50_ms"], det[label]["p95_ms"]), flush=True)
+ res["detection_overhead"] = det
+
+ # ---------------------------------------------------------------- 2. per-model raw latency
+ print("\n=== 2. raw Agent.system_one latency (no routing) ===", flush=True)
+ raw, load_times = {}, {}
+ for mname, path in MODELS.items():
+ t = time.perf_counter()
+ ag = laya.load(path, device="cpu")
+ load_times[mname] = round((time.perf_counter() - t) * 1000, 1)
+ per = {}
+ for n in (1, 5, 10, 50):
+ q = qs(n)
+ r = timed(lambda: ag.system_one(STATE_EN, q), warmup=2, reps=10)
+ r["ms_per_question"] = round(r["p50_ms"] / n, 2)
+ per["%d_questions" % n] = r
+ print(" %-16s %2dq p50 %8.1f ms p95 %8.1f ms %6.2f ms/q"
+ % (mname, n, r["p50_ms"], r["p95_ms"], r["ms_per_question"]), flush=True)
+ raw[mname] = per
+ print(" %-16s cold load: %.0f ms" % (mname, load_times[mname]), flush=True)
+ del ag; gc.collect()
+ res["raw_latency"] = raw
+ res["cold_load_ms"] = load_times
+
+ # ---------------------------------------------------------------- 3. router hot path
+ print("\n=== 3. Router hot path (target model already resident) ===", flush=True)
+ r3 = Router(models=MODELS, device="cpu", max_loaded=3)
+ r3.predict(STATE_EN, qs(5)) # warm english
+ r3.predict(STATE_HI, qs(5)) # warm multilingual
+ hot = {}
+ for label, st in (("english", STATE_EN), ("hindi", STATE_HI)):
+ for n in (1, 10):
+ q = qs(n)
+ k = "%s_%dq" % (label, n)
+ hot[k] = timed(lambda s=st, qq=q: r3.predict(s, qq), warmup=2, reps=10)
+ hot[k]["ms_per_question"] = round(hot[k]["p50_ms"] / n, 2)
+ print(" %-10s %2dq p50 %8.1f ms (%.2f ms/q)" % (label, n, hot[k]["p50_ms"],
+ hot[k]["ms_per_question"]), flush=True)
+ res["router_hot"] = hot
+
+ # overhead vs raw, like for like
+ base_en = raw["english"]["10_questions"]["p50_ms"]
+ res["routing_overhead_hot_ms"] = round(hot["english_10q"]["p50_ms"] - base_en, 2)
+ print(" routing overhead when hot: %+.2f ms on a 10-question English call"
+ % res["routing_overhead_hot_ms"], flush=True)
+
+ # ---------------------------------------------------------------- 4. router cold path (swap)
+ print("\n=== 4. Router cold path (max_loaded=1 forces a swap on every language flip) ===", flush=True)
+ r1 = Router(models=MODELS, device="cpu", max_loaded=1)
+ r1.predict(STATE_EN, qs(5))
+ swap = []
+ for i in range(4):
+ st = STATE_HI if i % 2 == 0 else STATE_EN
+ t = time.perf_counter()
+ r1.predict(st, qs(10))
+ swap.append((time.perf_counter() - t) * 1000)
+ res["router_cold_swap"] = {"samples_ms": [round(x, 1) for x in swap],
+ "median_ms": round(statistics.median(swap), 1)}
+ print(" swap calls (10q): %s median %.0f ms"
+ % ([round(x) for x in swap], res["router_cold_swap"]["median_ms"]), flush=True)
+
+ # ---------------------------------------------------------------- 5. mixed workload
+ print("\n=== 5. mixed-language workload, 100 calls of 5 questions ===", flush=True)
+ import random
+ rng = random.Random(13)
+ mix = {}
+ for share in (0.0, 0.1, 0.3, 0.5):
+ stream = [(STATE_HI if rng.random() < share else STATE_EN) for _ in range(100)]
+ for cap, tag in ((1, "max_loaded=1"), (3, "max_loaded=3")):
+ rr = Router(models=MODELS, device="cpu", max_loaded=cap)
+ rr.predict(STATE_EN, qs(5)); rr.predict(STATE_HI, qs(5)) # warm what fits
+ q = qs(5)
+ t = time.perf_counter()
+ for st in stream:
+ rr.predict(st, q)
+ el = (time.perf_counter() - t)
+ key = "%d%%_non_english/%s" % (int(share * 100), tag)
+ mix[key] = {"total_s": round(el, 2), "mean_ms_per_call": round(el * 1000 / len(stream), 1),
+ "calls_per_s": round(len(stream) / el, 1)}
+ print(" %-32s %6.1f ms/call %5.1f calls/s"
+ % (key, mix[key]["mean_ms_per_call"], mix[key]["calls_per_s"]), flush=True)
+ rr.unload(); del rr; gc.collect()
+ res["mixed_workload"] = mix
+
+ json.dump(res, open(OUT, "w"), indent=2)
+ print("\nwrote %s" % OUT, flush=True)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/research/scripts/bench_local.py b/research/scripts/bench_local.py
new file mode 100644
index 00000000..1c77dfdf
--- /dev/null
+++ b/research/scripts/bench_local.py
@@ -0,0 +1,337 @@
+"""Local extensive benchmark of the three Laya checkpoints.
+
+Part A MASSIVE intent across every language the dataset ships (~51), 20-option choice.
+Part B typed-decisions (400 cases / 2,000 decisions) on all three checkpoints, so the
+ fine-tuned laya-typed-decisions can be compared with Jev's published 0.727 on the
+ same benchmark.
+
+Writes local_benchmark_results.json.
+
+ USE_TF=0 python3 notebooks/bench_local.py [--langs N] [--per-lang N] [--skip-a] [--skip-b]
+"""
+import argparse
+import gc
+import json
+import math
+import os
+import random
+import re
+import sys
+import time
+
+os.environ.setdefault("USE_TF", "0") # TensorFlow's abseil runtime deadlocks model build
+os.environ.setdefault("USE_TORCH", "1")
+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
+
+import numpy as np # noqa: E402
+import torch # noqa: E402
+
+REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, os.path.join(REPO, "laya"))
+
+import laya # noqa: E402
+from laya.common import QTYPES, build_sequence, collate_items, render_options, temp_bucket # noqa: E402
+
+ROOT = os.path.expanduser("~/laya_models")
+MODELS = {"english": os.path.join(ROOT, "laya"),
+ "multilingual": os.path.join(ROOT, "laya-multilingual"),
+ "typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}
+OUT = os.path.join(REPO, "local_benchmark_results.json")
+SEED, N_OPTS = 13, 20
+
+
+# ------------------------------------------------------------------ engine
+def to_internal(qdef):
+ t = qdef["type"]
+ crit = qdef.get("criteria")
+ if t == "choice" and isinstance(crit, list):
+ crit = {c: None for c in crit}
+ ins = qdef["instructions"]
+ return {"t": t, "ins": ins if isinstance(ins, str) else json.dumps(ins), "crit": crit}
+
+
+def score_cases(agent, cases, max_tokens=8192, max_seqs=64, tag=""):
+ max_len = agent.cfg.get("max_len", 512)
+ hml = agent.cfg.get("head_max_len", 192)
+ items, index, dropped = [], [], 0
+ for ci, (state, questions) in enumerate(cases):
+ for qid, qdef in questions.items():
+ q = to_internal(qdef)
+ try:
+ ids, mk = build_sequence(agent.tok, state, q, max_len, hml)
+ except Exception:
+ index.append((ci, qid, QTYPES[q["t"]], 0)); items.append(None); dropped += 1; continue
+ if len(mk) != len(render_options(q)):
+ index.append((ci, qid, QTYPES[q["t"]], 0)); items.append(None); dropped += 1; continue
+ items.append({"ids": ids, "markers": mk, "qtype": QTYPES[q["t"]]})
+ index.append((ci, qid, QTYPES[q["t"]], len(mk)))
+ order = sorted([i for i, it in enumerate(items) if it is not None],
+ key=lambda i: len(items[i]["ids"]))
+ out = [None] * len(items)
+ t0, done, i = time.time(), 0, 0
+ while i < len(order):
+ j, L = i, 0
+ while j < len(order) and j - i < max_seqs and \
+ max(L, len(items[order[j]]["ids"])) * (j - i + 1) <= max_tokens:
+ L = max(L, len(items[order[j]]["ids"])); j += 1
+ j = max(j, i + 1)
+ sel = [items[order[t]] for t in range(i, j)]
+ b = collate_items([sel], agent.tok.pad_token_id)
+ lg, _ = agent.model(b["input_ids"].to(agent.device), b["attention_mask"].to(agent.device),
+ b["marker_pos"].to(agent.device), b["marker_mask"].to(agent.device),
+ b["qtype"].to(agent.device))
+ lg = lg.float().cpu().numpy()
+ for r in range(j - i):
+ out[order[i + r]] = lg[r, :len(sel[r]["markers"])]
+ done += j - i
+ if tag and done % 500 < (j - i):
+ el = time.time() - t0
+ sys.stderr.write("\r [%s] %d/%d %.0f q/s ETA %ds "
+ % (tag, done, len(order), done / max(el, 1e-9),
+ int(el * (len(order) - done) / max(1, done))))
+ sys.stderr.flush()
+ i = j
+ if tag:
+ sys.stderr.write("\r" + " " * 70 + "\r")
+ return out, index, time.time() - t0, dropped
+
+
+def softmax_t(z, t=1.0):
+ z = np.asarray(z, float) / max(1e-3, float(t))
+ e = np.exp(z - z.max())
+ return e / e.sum()
+
+
+def temp_for(agent, qt, k):
+ return float(agent.temperature_by_options.get(temp_bucket(qt, k), agent.temperature[qt]))
+
+
+def ece_score(conf, corr, bins=15):
+ conf, corr = np.asarray(conf, float), np.asarray(corr, float)
+ if not len(conf):
+ return float("nan")
+ e, edges = 0.0, np.linspace(0, 1, bins + 1)
+ for lo, hi in zip(edges[:-1], edges[1:]):
+ s = (conf > lo) & (conf <= hi)
+ if s.any():
+ e += s.mean() * abs(conf[s].mean() - corr[s].mean())
+ return float(e)
+
+
+def macro_f1(g, p):
+ g, p = np.asarray(g), np.asarray(p)
+ f = []
+ for c in sorted(set(g.tolist()) | set(p.tolist())):
+ tp = int(((p == c) & (g == c)).sum()); fp = int(((p == c) & (g != c)).sum())
+ fn = int(((p != c) & (g == c)).sum())
+ f.append(2 * tp / max(1, 2 * tp + fp + fn))
+ return float(np.mean(f))
+
+
+def metrics(rows):
+ rows = [r for r in rows if r[1] is not None]
+ if not rows:
+ return {"n": 0}
+ g = np.array([x[0] for x in rows]); p = np.array([int(np.argmax(x[1])) for x in rows])
+ c = np.array([float(np.max(x[1])) for x in rows]); corr = (p == g).astype(float)
+ return {"n": len(rows), "accuracy": round(float(corr.mean()), 4),
+ "macro_f1": round(macro_f1(g, p), 4), "ece": round(ece_score(c, corr), 4),
+ "brier": round(float(np.mean([((np.asarray(x[1]) - np.eye(len(x[1]))[x[0]]) ** 2).sum()
+ for x in rows])), 4),
+ "nll": round(float(np.mean([-math.log(max(float(x[1][x[0]]), 1e-12)) for x in rows])), 4),
+ "mean_confidence": round(float(c.mean()), 4),
+ "acc_at_50_coverage": round(float(corr[np.argsort(-c)[:max(1, len(c)//2)]].mean()), 4)}
+
+
+def load(name):
+ ag = laya.load(MODELS[name], device="cpu")
+ ag.model.eval()
+ return ag
+
+
+# ------------------------------------------------------------------ part A
+def massive_languages():
+ from huggingface_hub import HfApi
+ files = [s.rfilename for s in (HfApi().dataset_info("mteb/amazon_massive_intent").siblings or [])]
+ return sorted({m.group(1) for f in files for m in [re.match(r"test/([A-Za-z\-]+)\.json", f)] if m})
+
+
+def build_massive(langs, per_lang):
+ from datasets import load_dataset
+ suites = {}
+ for lg in langs:
+ try:
+ d = load_dataset("mteb/amazon_massive_intent", lg, split="test")
+ labels = sorted(set(d["label_text"]))
+ rng = random.Random(SEED)
+ cases, gold = [], []
+ for r in list(d)[:per_lang]:
+ pool = [x for x in labels if x != r["label_text"]]
+ keys = [r["label_text"]] + rng.sample(pool, min(N_OPTS - 1, len(pool)))
+ rng.shuffle(keys)
+ cases.append(({"utterance": r["text"]},
+ {"intent": {"type": "choice",
+ "instructions": "What is the user asking for in `utterance`?",
+ "criteria": {k: k.replace("_", " ").replace(".", ": ") for k in keys}}}))
+ gold.append(keys.index(r["label_text"]))
+ suites[lg] = (cases, gold, len(labels))
+ print(" built %-8s %d cases (%d labels)" % (lg, len(cases), len(labels)), flush=True)
+ except Exception as e:
+ print(" FAIL %-8s %s" % (lg, str(e)[:70]), flush=True)
+ return suites
+
+
+def run_part_a(results, langs, per_lang):
+ print("\n=== PART A: MASSIVE intent, %d languages, %d cases each, %d options ===\n"
+ % (len(langs), per_lang, N_OPTS), flush=True)
+ suites = build_massive(langs, per_lang)
+ results["part_a"] = {"config": {"languages": sorted(suites), "per_lang": per_lang,
+ "n_options": N_OPTS, "seed": SEED}, "by_model": {}}
+ for mname in ("english", "multilingual"):
+ print("\n--- %s ---" % mname, flush=True)
+ ag = load(mname)
+ per = {}
+ for lg, (cases, gold, _) in sorted(suites.items()):
+ lgs, idx, secs, dropped = score_cases(ag, cases, tag="%s/%s" % (mname, lg))
+ rows = [(gold[ci], softmax_t(z, temp_for(ag, qt, k)) if z is not None else None)
+ for (ci, _, qt, k), z in zip(idx, lgs)]
+ m = metrics(rows); m["seconds"] = round(secs, 1); m["dropped"] = dropped
+ per[lg] = m
+ print(" %-8s acc %.3f f1 %.3f ECE %.3f conf %.3f (%.0f q/s)"
+ % (lg, m["accuracy"], m["macro_f1"], m["ece"], m["mean_confidence"],
+ m["n"] / max(secs, 1e-9)), flush=True)
+ accs = [v["accuracy"] for v in per.values()]
+ results["part_a"]["by_model"][mname] = {
+ "per_language": per,
+ "macro_accuracy": round(float(np.mean(accs)), 4),
+ "macro_ece": round(float(np.mean([v["ece"] for v in per.values()])), 4),
+ "n_languages": len(per),
+ "languages_above_random": int(sum(1 for a in accs if a > 3.0 / N_OPTS)),
+ }
+ print(" MACRO acc %.4f | ECE %.4f | %d/%d langs > 3x random"
+ % (results["part_a"]["by_model"][mname]["macro_accuracy"],
+ results["part_a"]["by_model"][mname]["macro_ece"],
+ results["part_a"]["by_model"][mname]["languages_above_random"], len(per)), flush=True)
+ del ag; gc.collect()
+ json.dump(results, open(OUT, "w"), indent=2)
+
+
+# ------------------------------------------------------------------ part B
+def build_typed_decisions():
+ import pandas as pd
+ p = os.path.join(REPO, "typed-decisions", "all", "test-00000-of-00001.parquet")
+ df = pd.read_parquet(p)
+ cases, gold, wfs = [], [], []
+ for _, r in df.iterrows():
+ qs = json.loads(r["questions"]); g = json.loads(r["gold"])
+ st = r["state"]
+ try:
+ st = json.loads(st)
+ except Exception:
+ pass
+ gm = {}
+ for qid, qd in qs.items():
+ gg = g[qid]
+ if qd["type"] == "choice":
+ keys = list(qd["criteria"].keys())
+ gm[qid] = {"idx": keys.index(str(gg["label"])),
+ "soft": [float(gg.get("probabilities", {}).get(k, 0.0)) for k in keys]}
+ elif qd["type"] == "noul":
+ pt = float(gg.get("probabilities", {}).get("true", gg.get("noul", 0.5)))
+ gm[qid] = {"idx": 1 if str(gg["label"]).lower() == "true" else 0, "soft": [1 - pt, pt]}
+ else:
+ n = len(qd["criteria"])
+ gm[qid] = {"idx": int(gg["label"]),
+ "soft": [float(gg.get("probabilities", {}).get(str(i), 0.0)) for i in range(n)],
+ "gold_score": float(gg.get("score", float(gg["label"])))}
+ cases.append((st, qs)); gold.append(gm); wfs.append(r["workflow"])
+ return cases, gold, wfs
+
+
+def run_part_b(results):
+ print("\n=== PART B: typed-decisions, 400 cases / 2,000 decisions, all 3 checkpoints ===\n",
+ flush=True)
+ cases, gold, wfs = build_typed_decisions()
+ # reference points that are NOT measured here - published / independently measured
+ results["part_b"] = {"reference_points": {
+ "jev_1.13.0_published": {"accuracy": 0.727, "soft_accuracy": 0.580, "brier": 0.148,
+ "ece": 0.144, "score_mae": 0.391, "ms_per_case": 710,
+ "source": "figure quoted in the laya repo's own comparison table"},
+ "teacher_self_agreement_ceiling": {"accuracy": 0.735},
+ "modernbert_base_specialist": {"accuracy": 0.646},
+ "random_guess": {"accuracy": 0.3175},
+ "per_question_majority_class": {"accuracy": 0.4610},
+ "note": "Jev was NOT run here - no TypeSafe API credential is available. These are "
+ "published numbers reproduced for context, not a measured head-to-head.",
+ }, "by_model": {}}
+ for mname in ("english", "multilingual", "typed-decisions"):
+ print("--- %s ---" % mname, flush=True)
+ ag = load(mname)
+ lgs, idx, secs, dropped = score_cases(ag, cases, tag=mname)
+ rows, soft, brier_s, mae, w1, by_wf, by_qt = [], [], [], [], [], {}, {}
+ for (ci, qid, qt, k), z in zip(idx, lgs):
+ g = gold[ci][qid]
+ if z is None:
+ rows.append((g["idx"], None)); continue
+ p = softmax_t(z, temp_for(ag, qt, k))
+ rows.append((g["idx"], p))
+ by_wf.setdefault(wfs[ci], []).append((g["idx"], p))
+ by_qt.setdefault({0: "choice", 1: "score", 2: "noul"}[qt], []).append((g["idx"], p))
+ gp = np.asarray(g["soft"], float)
+ if gp.sum() > 0:
+ gp = gp / gp.sum()
+ pp = p[:len(gp)] if len(p) >= len(gp) else np.pad(p, (0, len(gp) - len(p)))
+ pp = pp / max(pp.sum(), 1e-12)
+ soft.append(float((pp * gp).sum())); brier_s.append(float(((pp - gp) ** 2).sum()))
+ if "gold_score" in g:
+ exp = float((np.arange(len(p)) * p).sum())
+ mae.append(abs(exp - g["gold_score"])); w1.append(float(abs(exp - g["gold_score"]) <= 1))
+ m = metrics(rows)
+ m.update(soft_accuracy=round(float(np.mean(soft)), 4) if soft else None,
+ brier_vs_soft=round(float(np.mean(brier_s)), 4) if brier_s else None,
+ score_mae=round(float(np.mean(mae)), 4) if mae else None,
+ within_1_level=round(float(np.mean(w1)), 4) if w1 else None,
+ seconds=round(secs, 1), dropped=dropped,
+ ms_per_case=round(1000 * secs / len(cases), 1),
+ by_workflow={k: metrics(v) for k, v in sorted(by_wf.items())},
+ by_question_type={k: metrics(v) for k, v in sorted(by_qt.items())})
+ results["part_b"]["by_model"][mname] = m
+ print(" acc %.4f | soft %.4f | brier(soft) %s | ECE %.4f | MAE %s | %.0f ms/case"
+ % (m["accuracy"], m["soft_accuracy"] or 0, m["brier_vs_soft"], m["ece"],
+ m["score_mae"], m["ms_per_case"]), flush=True)
+ for wf, v in m["by_workflow"].items():
+ print(" %-28s acc %.3f (n=%d)" % (wf, v["accuracy"], v["n"]), flush=True)
+ del ag; gc.collect()
+ json.dump(results, open(OUT, "w"), indent=2)
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--langs", type=int, default=0, help="cap number of languages (0 = all)")
+ ap.add_argument("--per-lang", type=int, default=120)
+ ap.add_argument("--skip-a", action="store_true")
+ ap.add_argument("--skip-b", action="store_true")
+ a = ap.parse_args()
+
+ results = {"meta": {"timestamp": time.strftime("%Y-%m-%d %H:%M:%S"), "device": "cpu",
+ "torch": torch.__version__, "laya": laya.__version__,
+ "threads": torch.get_num_threads()}}
+ if os.path.exists(OUT):
+ try:
+ results = json.load(open(OUT)); results["meta"]["timestamp"] = time.strftime("%Y-%m-%d %H:%M:%S")
+ except Exception:
+ pass
+ if not a.skip_a:
+ langs = massive_languages()
+ if a.langs:
+ langs = langs[:a.langs]
+ run_part_a(results, langs, a.per_lang)
+ if not a.skip_b:
+ run_part_b(results)
+ json.dump(results, open(OUT, "w"), indent=2)
+ print("\nwrote %s" % OUT, flush=True)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/research/scripts/build_benchmark_nb.py b/research/scripts/build_benchmark_nb.py
new file mode 100644
index 00000000..8b131f2d
--- /dev/null
+++ b/research/scripts/build_benchmark_nb.py
@@ -0,0 +1,840 @@
+"""Emit laya_benchmark_colab.ipynb: an extensive head-to-head benchmark of
+convaiinnovations/laya (ModernBERT-large, English) vs convaiinnovations/laya-multilingual
+(mmBERT-base, 100+ languages), on a Colab T4.
+
+Writes one laya_benchmark_results.json that can be handed back for analysis.
+"""
+import json
+import os
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+
+
+def md(t):
+ return {"cell_type": "markdown", "metadata": {}, "source": t.strip("\n").splitlines(keepends=True)}
+
+
+def code(t):
+ return {"cell_type": "code", "execution_count": None, "metadata": {}, "outputs": [],
+ "source": t.strip("\n").splitlines(keepends=True)}
+
+
+CELLS = [
+ md("""
+# Laya vs Laya-Multilingual — extensive head-to-head benchmark
+
+Runs both open checkpoints over the same questions and writes a single
+`laya_benchmark_results.json`.
+
+| | `convaiinnovations/laya` | `convaiinnovations/laya-multilingual` |
+|---|---|---|
+| encoder | ModernBERT-large (English) | mmBERT-base (100+ languages) |
+| params | 421.3M (394.8M enc + 26.5M head) | 321.9M (306.9M enc + 15.0M head) |
+| context | 512 tokens | 1024 tokens |
+| shipped temperatures | fitted per type & option-count | **none (all 1.0)** |
+
+**What is measured**
+
+1. **typed-decisions** — 400 cases / 2,000 decisions, 4 workflows (the benchmark behind the published 0.766 claim). Zero-shot for both of these checkpoints.
+2. **Multilingual** — MASSIVE intent + scenario across 14 languages, XNLI across 15 languages.
+3. **English zero-shot** — SST-5, emotion, prompt-injections, plus banking77 as a 77-option stress test.
+4. **Latency** — p50/p95 at 1 / 5 / 10 / 50 questions per call.
+5. **Option-order robustness** — how often the answer flips when the options are permuted (the independent Jev benchmark measured 13% for Jev; worst LLM 37%).
+6. **Calibration repair** — ECE before and after refitting temperature, which matters because `laya-multilingual` shipped uncalibrated.
+
+**Setup:** Runtime → Change runtime type → **T4 GPU**. Then Run All (~25–40 min).
+Both models see byte-identical questions (fixed seed), so every difference is the model.
+"""),
+
+ md("## 1. GPU and environment"),
+ code("""
+import subprocess, sys, os, json, time
+print(subprocess.run(["nvidia-smi","--query-gpu=name,memory.total,compute_cap","--format=csv,noheader"],
+ capture_output=True, text=True).stdout.strip() or "NO GPU - set Runtime > Change runtime type > T4 GPU")
+import torch
+print("torch", torch.__version__, "| cuda", torch.cuda.is_available(),
+ "| device", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "cpu")
+assert torch.cuda.is_available(), "Enable the T4 GPU runtime before running."
+"""),
+
+ md("## 2. Dependencies"),
+ code("""
+!pip -q install -U "laya>=0.1.6" "transformers>=4.45" "datasets>=3.0" safetensors huggingface_hub 2>&1 | tail -3
+import laya, transformers, datasets
+print("laya", getattr(laya, "__version__", "?"), "| transformers", transformers.__version__, "| datasets", datasets.__version__)
+"""),
+
+ md("""## 3. Download both checkpoints
+
+`reference_compile` is forced off after loading: the `laya` package does not disable it (only
+`rl_agent_api.py` does), and torch.compile is a loss on small batches / few SMs like a T4."""),
+ code("""
+import torch, json, os, time
+from huggingface_hub import snapshot_download
+from safetensors import safe_open
+
+REPOS = {"laya": "convaiinnovations/laya", "laya-multilingual": "convaiinnovations/laya-multilingual"}
+PATHS, MODEL_META = {}, {}
+
+def patch_tokenizer_config(model_dir):
+ '''laya-multilingual ships extra_special_tokens as a LIST (inherited from the mmBERT/Gemma
+ tokenizer); transformers expects a dict and raises
+ AttributeError: list object has no attribute keys. Without this, AutoTokenizer -- and so
+ laya.load() -- fails outright on that repo. Also normalises tokenizer_class for 4.x/5.x.'''
+ p = os.path.join(model_dir, "tokenizer", "tokenizer_config.json")
+ if not os.path.exists(p):
+ return []
+ c = json.load(open(p))
+ changed = []
+ est = c.get("extra_special_tokens")
+ if isinstance(est, list):
+ c["extra_special_tokens"] = {("extra_%d" % i): t for i, t in enumerate(est)}
+ changed.append("extra_special_tokens list->dict")
+ if c.get("tokenizer_class") in (None, "TokenizersBackend"):
+ c["tokenizer_class"] = "PreTrainedTokenizerFast"
+ c.pop("backend", None); c.pop("is_local", None)
+ changed.append("tokenizer_class")
+ if changed:
+ json.dump(c, open(p, "w"), indent=2)
+ return changed
+
+for name, repo in REPOS.items():
+ t = time.time()
+ PATHS[name] = snapshot_download(repo)
+ fixed = patch_tokenizer_config(PATHS[name])
+ if fixed:
+ print(" patched tokenizer config for %s: %s" % (name, ", ".join(fixed)))
+ cfg = json.load(open(os.path.join(PATHS[name], "rl_agent_config.json")))
+ ecfg = json.load(open(os.path.join(PATHS[name], "encoder", "config.json")))
+ tot = enc = 0
+ with safe_open(os.path.join(PATHS[name], "model.safetensors"), "pt") as f:
+ for k in f.keys():
+ n = 1
+ for d in f.get_slice(k).get_shape():
+ n *= d
+ tot += n
+ enc += n if k.startswith("encoder.") else 0
+ MODEL_META[name] = {
+ "repo": repo, "params_total_m": round(tot/1e6, 2), "params_encoder_m": round(enc/1e6, 2),
+ "params_head_m": round((tot-enc)/1e6, 2), "hidden_size": ecfg.get("hidden_size"),
+ "num_layers": ecfg.get("num_hidden_layers"), "vocab_size": ecfg.get("vocab_size"),
+ "max_len": cfg.get("max_len"), "head_max_len": cfg.get("head_max_len"),
+ "temperature": cfg.get("temperature"), "temperature_by_options": cfg.get("temperature_by_options", {}),
+ "training": cfg.get("training", {}), "download_s": round(time.time()-t, 1),
+ "tokenizer_config_patched": fixed,
+ }
+ print(name, "->", PATHS[name])
+ print(" ", json.dumps({k: v for k, v in MODEL_META[name].items() if k != "temperature_by_options"}))
+
+def load_agent(name):
+ import laya
+ ag = laya.load(PATHS[name], device="cuda")
+ try:
+ ag.model.encoder.config.reference_compile = False # torch.compile hurts on T4-class GPUs
+ except Exception as e:
+ print("could not disable reference_compile:", e)
+ ag.model.eval()
+ return ag
+"""),
+
+ md("## 4. Benchmark engine — batched scoring and metrics"),
+ code(r"""
+%%writefile bench_engine.py
+import json, math, sys, time
+import numpy as np
+import torch
+from laya.common import QTYPES, build_sequence, collate_items, render_options, temp_bucket
+
+def to_internal(qdef):
+ t = qdef["type"]
+ crit = qdef.get("criteria")
+ if t == "choice" and isinstance(crit, list):
+ crit = {c: None for c in crit}
+ ins = qdef["instructions"]
+ if not isinstance(ins, str):
+ ins = json.dumps(ins)
+ return {"t": t, "ins": ins, "crit": crit}
+
+def score_cases(agent, cases, max_tokens=16384, max_seqs=128, progress=""):
+ '''cases: [(state, {qid: qdef})] -> (raw_logits per question, index, seconds, n_dropped).
+
+ Every question of every case is one sequence. Sequences are length-sorted and packed into
+ token-budgeted batches so padding stays small. Returns UNCALIBRATED logits; temperature is
+ applied later so raw/calibrated variants come from one forward pass.
+ '''
+ max_len = agent.cfg.get("max_len", 512)
+ head_max_len = agent.cfg.get("head_max_len", 192)
+ items, index, dropped = [], [], 0
+ for ci, (state, questions) in enumerate(cases):
+ for qid, qdef in questions.items():
+ q = to_internal(qdef)
+ try:
+ ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)
+ except Exception:
+ index.append((ci, qid, QTYPES[q["t"]], 0)); items.append(None); dropped += 1
+ continue
+ if len(markers) != len(render_options(q)):
+ index.append((ci, qid, QTYPES[q["t"]], 0)); items.append(None); dropped += 1
+ continue
+ items.append({"ids": ids, "markers": markers, "qtype": QTYPES[q["t"]]})
+ index.append((ci, qid, QTYPES[q["t"]], len(markers)))
+
+ valid = [i for i, it in enumerate(items) if it is not None]
+ order = sorted(valid, key=lambda i: len(items[i]["ids"]))
+ out = [None] * len(items)
+ t0, done, i = time.time(), 0, 0
+ while i < len(order):
+ j, L = i, 0
+ while j < len(order) and j - i < max_seqs and max(L, len(items[order[j]]["ids"])) * (j - i + 1) <= max_tokens:
+ L = max(L, len(items[order[j]]["ids"])); j += 1
+ j = max(j, i + 1)
+ sel = [items[order[t]] for t in range(i, j)]
+ b = collate_items([sel], agent.tok.pad_token_id)
+ with torch.autocast("cuda", dtype=agent.dtype, enabled=True):
+ logits, _ = agent.model(b["input_ids"].cuda(), b["attention_mask"].cuda(),
+ b["marker_pos"].cuda(), b["marker_mask"].cuda(), b["qtype"].cuda())
+ logits = logits.float().cpu().numpy()
+ for r in range(j - i):
+ out[order[i + r]] = logits[r, :len(sel[r]["markers"])]
+ done += j - i
+ if progress:
+ el = time.time() - t0
+ msg = "\r [%s] %d/%d seq | %.0f seq/s | ETA %ds " % (
+ progress, done, len(order), done/max(el, 1e-9),
+ int(el*(len(order)-done)/max(1, done)))
+ sys.stdout.write(msg)
+ sys.stdout.flush()
+ i = j
+ if progress:
+ print("\r [%s] %d sequences in %.1fs (%d dropped: options did not fit)%s" % (
+ progress, len(order), time.time()-t0, dropped, " "*20))
+ return out, index, time.time() - t0, dropped
+
+def softmax_t(z, t=1.0):
+ z = np.asarray(z, dtype=np.float64) / max(1e-3, float(t))
+ e = np.exp(z - z.max())
+ return e / e.sum()
+
+def temp_for(agent, qt, k, calibrated=True):
+ if not calibrated:
+ return 1.0
+ return float(agent.temperature_by_options.get(temp_bucket(qt, k), agent.temperature[qt]))
+
+# ------------------------------------------------------------------ metrics
+def ece_score(conf, correct, bins=15):
+ conf, correct = np.asarray(conf, float), np.asarray(correct, float)
+ if len(conf) == 0:
+ return float("nan")
+ edges = np.linspace(0, 1, bins + 1)
+ e = 0.0
+ for lo, hi in zip(edges[:-1], edges[1:]):
+ sel = (conf > lo) & (conf <= hi)
+ if sel.any():
+ e += sel.mean() * abs(conf[sel].mean() - correct[sel].mean())
+ return float(e)
+
+def macro_f1(gold, pred):
+ gold, pred = np.asarray(gold), np.asarray(pred)
+ f1 = []
+ for c in sorted(set(gold.tolist()) | set(pred.tolist())):
+ tp = int(((pred == c) & (gold == c)).sum())
+ fp = int(((pred == c) & (gold != c)).sum())
+ fn = int(((pred != c) & (gold == c)).sum())
+ f1.append(2*tp / max(1, 2*tp + fp + fn))
+ return float(np.mean(f1))
+
+def aurc(conf, correct):
+ conf, correct = np.asarray(conf, float), np.asarray(correct, float)
+ if len(conf) == 0:
+ return float("nan")
+ o = np.argsort(-conf)
+ return float((np.cumsum(1 - correct[o]) / np.arange(1, len(o)+1)).mean())
+
+def hard_metrics(rows):
+ '''rows: [(gold_idx, probs)] -> standard single-label classification + calibration metrics.'''
+ rows = [r for r in rows if r[1] is not None]
+ if not rows:
+ return {"n": 0}
+ gold = np.array([g for g, _ in rows])
+ pred = np.array([int(np.argmax(p)) for _, p in rows])
+ conf = np.array([float(np.max(p)) for _, p in rows])
+ corr = (pred == gold).astype(float)
+ m = {"n": len(rows),
+ "accuracy": float(corr.mean()),
+ "macro_f1": macro_f1(gold, pred),
+ "ece": ece_score(conf, corr),
+ "brier": float(np.mean([((np.asarray(p) - np.eye(len(p))[g])**2).sum() for g, p in rows])),
+ "nll": float(np.mean([-math.log(max(float(p[g]), 1e-12)) for g, p in rows])),
+ "aurc": aurc(conf, corr),
+ "mean_confidence": float(conf.mean())}
+ for cov in (0.5, 0.8):
+ k = max(1, int(len(conf)*cov))
+ m["acc_at_%d_coverage" % int(cov*100)] = float(corr[np.argsort(-conf)[:k]].mean())
+ return m
+
+def fit_temperature(pairs, lo=0.2, hi=10.0, steps=160):
+ '''pairs: [(logits, gold_idx)] -> single temperature minimising NLL (grid search, no autograd).'''
+ pairs = [(np.asarray(z, float), int(g)) for z, g in pairs if z is not None]
+ if len(pairs) < 25:
+ return 1.0
+ best_t, best = 1.0, float("inf")
+ for t in np.geomspace(lo, hi, steps):
+ tot = 0.0
+ for z, g in pairs:
+ zz = z/t
+ zz = zz - zz.max()
+ tot += -(zz[g] - math.log(np.exp(zz).sum()))
+ if tot < best:
+ best, best_t = tot, float(t)
+ return round(float(best_t), 4)
+"""),
+
+ md("""## 5. Task suites
+
+Every suite is built **once**, with a fixed seed, and reused for both models — so the two
+checkpoints answer byte-identical questions. Option sets for many-label tasks use
+`gold + N-1 distractors`, matching how the models were trained (the training recipe samples
+5–20 options per choice question rather than showing the full label space)."""),
+
+ code("""
+import random, json
+import numpy as np
+from datasets import load_dataset
+
+SEED = 13
+N_OPTS = 20 # options per many-label choice question (gold + 19 distractors)
+PER_LANG = 300 # cases per language
+SUITES = {} # name -> {"cases": [(state, questions)], "gold": [{qid: gold_idx}], "meta": {...}}
+
+def build_choice_questions(rng, gold, all_labels, n_opts, instructions, qid="label"):
+ pool = [x for x in all_labels if x != gold]
+ keys = [gold] + rng.sample(pool, min(n_opts-1, len(pool)))
+ rng.shuffle(keys)
+ crit = {k: k.replace("_", " ").replace(".", ": ") for k in keys}
+ return {qid: {"type": "choice", "instructions": instructions, "criteria": crit}}, keys.index(gold)
+
+def register(name, cases, golds, **meta):
+ SUITES[name] = {"cases": cases, "gold": golds, "meta": meta}
+ nq = sum(len(c[1]) for c in cases)
+ print(" %-34s %4d cases / %5d questions %s" % (name, len(cases), nq, meta.get("note", "")))
+"""),
+
+ md("### 5a. typed-decisions (English, 400 cases / 2,000 decisions)"),
+ code("""
+td_cases, td_golds, td_wf = [], [], []
+try:
+ td = load_dataset("LocalLLaMA/typed-decisions", "all", split="test")
+ for r in td:
+ questions = json.loads(r["questions"]) if isinstance(r["questions"], str) else r["questions"]
+ gold = json.loads(r["gold"]) if isinstance(r["gold"], str) else r["gold"]
+ state = r["state"]
+ try:
+ state = json.loads(state)
+ except Exception:
+ pass
+ gmap = {}
+ for qid, qdef in questions.items():
+ g = gold[qid]
+ if qdef["type"] == "choice":
+ keys = list(qdef["criteria"].keys())
+ gmap[qid] = {"idx": keys.index(str(g["label"])), "keys": keys,
+ "soft": [float(g.get("probabilities", {}).get(k, 0.0)) for k in keys]}
+ elif qdef["type"] == "noul":
+ idx = 1 if str(g["label"]).lower() == "true" else 0
+ pt = float(g.get("probabilities", {}).get("true", g.get("noul", 0.5)))
+ gmap[qid] = {"idx": idx, "keys": ["false", "true"], "soft": [1-pt, pt]}
+ else: # score
+ n = len(qdef["criteria"])
+ gmap[qid] = {"idx": int(g["label"]), "keys": [str(i) for i in range(n)],
+ "soft": [float(g.get("probabilities", {}).get(str(i), 0.0)) for i in range(n)],
+ "gold_score": float(g.get("score", float(g["label"])))}
+ td_cases.append((state, questions))
+ td_golds.append(gmap)
+ td_wf.append(r["workflow"])
+ register("typed_decisions", td_cases, td_golds, note="4 workflows, soft teacher labels",
+ workflows=td_wf, soft=True)
+except Exception as e:
+ print("typed-decisions FAILED:", type(e).__name__, e)
+"""),
+
+ md("### 5b. Multilingual — MASSIVE intent & scenario (14 languages), XNLI (15 languages)"),
+ code("""
+MASSIVE_LANGS = ["en","de","fr","es","pt","ru","tr","ar","hi","ta","zh-CN","ja","ko","sw"]
+XNLI_LANGS = ["en","de","fr","es","ru","tr","ar","hi","ur","vi","th","el","bg","zh","sw"]
+
+for task, short, instr in [("mteb/amazon_massive_intent", "massive_intent",
+ "What is the user asking for in `utterance`?"),
+ ("mteb/amazon_massive_scenario", "massive_scenario",
+ "Which domain does `utterance` belong to?")]:
+ for lg in MASSIVE_LANGS:
+ try:
+ d = load_dataset(task, lg, split="test")
+ labels = sorted(set(d["label_text"]))
+ rng = random.Random(SEED)
+ cases, golds = [], []
+ for r in list(d)[:PER_LANG]:
+ qs, gi = build_choice_questions(rng, r["label_text"], labels, N_OPTS, instr)
+ cases.append(({"utterance": r["text"]}, qs))
+ golds.append({"label": {"idx": gi}})
+ register("%s.%s" % (short, lg), cases, golds, lang=lg, n_options=N_OPTS,
+ label_space=len(labels), family=short)
+ except Exception as e:
+ print(" FAIL %s %s: %s" % (short, lg, str(e)[:90]))
+
+NLI_CRIT = {"entailment": "the premise implies the hypothesis is true",
+ "neutral": "the premise neither implies nor contradicts the hypothesis",
+ "contradiction": "the premise implies the hypothesis is false"}
+for lg in XNLI_LANGS:
+ try:
+ d = load_dataset("facebook/xnli", lg, split="test")
+ cases, golds = [], []
+ for r in list(d)[:PER_LANG]:
+ qs = {"relation": {"type": "choice",
+ "instructions": "What is the relationship between `premise` and `hypothesis`?",
+ "criteria": dict(NLI_CRIT)}}
+ cases.append(({"premise": r["premise"], "hypothesis": r["hypothesis"]}, qs))
+ golds.append({"relation": {"idx": int(r["label"])}})
+ register("xnli.%s" % lg, cases, golds, lang=lg, n_options=3, family="xnli")
+ except Exception as e:
+ print(" FAIL xnli %s: %s" % (lg, str(e)[:90]))
+"""),
+
+ md("""### 5c. English tasks
+
+`sst5`, `emotion` and `prompt_injections` were held out of Laya's training entirely (true
+zero-shot). `ag_news` and `boolq` were in the training mix, so they measure retention rather
+than generalisation. `banking77` is the many-option stress test — 77 options at once, versus
+Jev's documented hard cap at 255."""),
+ code("""
+rng = random.Random(SEED)
+
+# SST-5: ordinal score
+try:
+ d = load_dataset("SetFit/sst5", split="test")
+ crit = ["very negative", "negative", "neutral", "positive", "very positive"]
+ cases, golds = [], []
+ for r in list(d)[:600]:
+ cases.append(({"text": r["text"]},
+ {"sentiment": {"type": "score", "instructions": "How positive is the sentiment of `text`?",
+ "criteria": crit}}))
+ golds.append({"sentiment": {"idx": int(r["label"]), "gold_score": float(r["label"])}})
+ register("en.sst5", cases, golds, note="zero-shot, ordinal", family="english", held_out=True)
+except Exception as e: print(" FAIL sst5:", str(e)[:90])
+
+# emotion: 6-way choice
+try:
+ d = load_dataset("dair-ai/emotion", "split", split="test")
+ names = ["sadness","joy","love","anger","fear","surprise"]
+ cases, golds = [], []
+ for r in list(d)[:600]:
+ cases.append(({"text": r["text"]},
+ {"emotion": {"type": "choice",
+ "instructions": "Which emotion is most strongly expressed in `text`?",
+ "criteria": {n: None for n in names}}}))
+ golds.append({"emotion": {"idx": int(r["label"])}})
+ register("en.emotion", cases, golds, note="zero-shot", family="english", held_out=True)
+except Exception as e: print(" FAIL emotion:", str(e)[:90])
+
+# prompt injections: noul
+try:
+ d = load_dataset("deepset/prompt-injections", split="test")
+ cases, golds = [], []
+ for r in list(d):
+ cases.append(({"text": r["text"]},
+ {"injection": {"type": "noul",
+ "instructions": "Does `text` try to inject or override instructions given to an AI system?"}}))
+ golds.append({"injection": {"idx": int(r["label"])}})
+ register("en.prompt_injections", cases, golds, note="zero-shot", family="english", held_out=True)
+except Exception as e: print(" FAIL prompt_injections:", str(e)[:90])
+
+# banking77: 77-option stress test
+try:
+ d = load_dataset("PolyAI/banking77", split="test")
+ names = d.features["label"].names
+ cases, golds = [], []
+ for r in list(d)[:500]:
+ cases.append(({"message": r["text"]},
+ {"intent": {"type": "choice", "instructions": "Which banking intent does `message` express?",
+ "criteria": {n.replace("_", " "): None for n in names}}}))
+ golds.append({"intent": {"idx": int(r["label"])}})
+ register("en.banking77_full", cases, golds, note="77 options at once", family="english",
+ n_options=len(names), held_out=True)
+except Exception as e: print(" FAIL banking77:", str(e)[:90])
+
+# ag_news + boolq: in-task retention
+try:
+ d = load_dataset("fancyzhx/ag_news", split="test")
+ crit = {"world": "world news and international politics", "sports": "sports",
+ "business": "business and economy", "sci_tech": "science and technology"}
+ cases, golds = [], []
+ for r in list(d)[:600]:
+ cases.append(({"article": r["text"]},
+ {"topic": {"type": "choice", "instructions": "What is the topic of `article`?",
+ "criteria": dict(crit)}}))
+ golds.append({"topic": {"idx": int(r["label"])}})
+ register("en.ag_news", cases, golds, note="in-task (trained on)", family="english", held_out=False)
+except Exception as e: print(" FAIL ag_news:", str(e)[:90])
+
+try:
+ d = load_dataset("google/boolq", split="validation")
+ cases, golds = [], []
+ for r in list(d)[:600]:
+ cases.append(({"passage": r["passage"], "question": r["question"]},
+ {"answer": {"type": "noul",
+ "instructions": "Based on `passage`, is the answer to `question` yes?"}}))
+ golds.append({"answer": {"idx": int(bool(r["answer"]))}})
+ register("en.boolq", cases, golds, note="in-task (trained on)", family="english", held_out=False)
+except Exception as e: print(" FAIL boolq:", str(e)[:90])
+
+print()
+print("suites built:", len(SUITES), "| total questions:",
+ sum(sum(len(c[1]) for c in s["cases"]) for s in SUITES.values()))
+"""),
+
+ md("## 6. Run every suite on both models"),
+ code("""
+import bench_engine as BE
+import numpy as np, gc, time, json
+
+RESULTS = {"meta": {"seed": SEED, "n_opts": N_OPTS, "per_lang": PER_LANG,
+ "timestamp": time.strftime("%Y-%m-%d %H:%M:%S"),
+ "gpu": torch.cuda.get_device_name(0),
+ "torch": torch.__version__, "transformers": transformers.__version__,
+ "models": MODEL_META},
+ "suites": {}}
+RAW = {} # model -> suite -> (logits, index) kept in RAM for the calibration-repair section
+
+def evaluate_suite(agent, model_name, suite_name):
+ S = SUITES[suite_name]
+ logits, index, secs, dropped = BE.score_cases(agent, S["cases"], progress="%s/%s" % (model_name, suite_name))
+ RAW[model_name][suite_name] = (logits, index)
+ soft = S["meta"].get("soft", False)
+ rows_cal, rows_raw, extra = [], [], {"soft_acc": [], "brier_soft": [], "tv": [], "kl": [],
+ "score_mae": [], "within_1": []}
+ per_q_type = {}
+ for (ci, qid, qt, k), z in zip(index, logits):
+ g = S["gold"][ci].get(qid)
+ if g is None:
+ continue
+ if z is None:
+ rows_cal.append((g["idx"], None)); rows_raw.append((g["idx"], None)); continue
+ p_cal = BE.softmax_t(z, BE.temp_for(agent, qt, k, True))
+ p_raw = BE.softmax_t(z, 1.0)
+ rows_cal.append((g["idx"], p_cal)); rows_raw.append((g["idx"], p_raw))
+ per_q_type.setdefault(qt, []).append((g["idx"], p_cal))
+ if soft and "soft" in g:
+ gp = np.asarray(g["soft"], float)
+ if gp.sum() > 0:
+ gp = gp / gp.sum()
+ pp = p_cal[:len(gp)] if len(p_cal) >= len(gp) else np.pad(p_cal, (0, len(gp)-len(p_cal)))
+ pp = pp / max(pp.sum(), 1e-12)
+ extra["soft_acc"].append(float((pp*gp).sum()))
+ extra["brier_soft"].append(float(((pp-gp)**2).sum()))
+ extra["tv"].append(float(0.5*np.abs(pp-gp).sum()))
+ extra["kl"].append(float((gp*np.log(np.clip(gp/np.clip(pp,1e-12,None),1e-12,1e4))).sum()))
+ if "gold_score" in g:
+ exp = float((np.arange(len(p_cal))*p_cal).sum())
+ extra["score_mae"].append(abs(exp - g["gold_score"]))
+ extra["within_1"].append(float(abs(exp - g["gold_score"]) <= 1.0))
+
+ out = {"calibrated": BE.hard_metrics(rows_cal), "raw": BE.hard_metrics(rows_raw),
+ "seconds": round(secs, 2), "dropped_questions": dropped,
+ "questions_per_second": round(len([r for r in rows_cal if r[1] is not None])/max(secs,1e-9), 1)}
+ for k2, v in extra.items():
+ if v:
+ out[k2] = float(np.mean(v))
+ if len(per_q_type) > 1:
+ names = {0: "choice", 1: "score", 2: "noul"}
+ out["by_question_type"] = {names[qt]: BE.hard_metrics(rs) for qt, rs in sorted(per_q_type.items())}
+ return out
+
+for model_name in REPOS:
+ print("\\n" + "="*78 + "\\n %s\\n" % model_name + "="*78)
+ agent = load_agent(model_name)
+ RAW[model_name] = {}
+ for suite_name in SUITES:
+ try:
+ r = evaluate_suite(agent, model_name, suite_name)
+ RESULTS["suites"].setdefault(suite_name, {})[model_name] = r
+ c = r["calibrated"]
+ print(" %-32s acc %.3f f1 %.3f ECE %.3f n=%d" %
+ (suite_name, c.get("accuracy", float("nan")), c.get("macro_f1", float("nan")),
+ c.get("ece", float("nan")), c.get("n", 0)))
+ except Exception as e:
+ import traceback; traceback.print_exc()
+ RESULTS["suites"].setdefault(suite_name, {})[model_name] = {"error": "%s: %s" % (type(e).__name__, e)}
+ globals()["AGENT_" + model_name.replace("-", "_")] = agent
+ gc.collect(); torch.cuda.empty_cache()
+
+json.dump(RESULTS, open("laya_benchmark_results.json", "w"), indent=2)
+print("\\ncheckpoint saved")
+"""),
+
+ md("""### 6b. Control — matched context budget
+
+The two checkpoints ship different context windows (512 vs 1024 tokens), so on long states
+`laya-multilingual` sees more of the input. This re-runs typed-decisions with **both** models
+capped at 512 tokens, isolating the architecture from the context-window advantage."""),
+ code("""
+import bench_engine as BE
+import numpy as np
+
+RESULTS["typed_decisions_matched_512"] = {}
+if "typed_decisions" in SUITES:
+ S = SUITES["typed_decisions"]
+ for model_name in REPOS:
+ agent = globals()["AGENT_" + model_name.replace("-", "_")]
+ orig = agent.cfg.get("max_len")
+ agent.cfg["max_len"] = 512
+ try:
+ logits, index, secs, dropped = BE.score_cases(agent, S["cases"], progress="%s/td@512" % model_name)
+ rows = []
+ for (ci, qid, qt, k), z in zip(index, logits):
+ g = S["gold"][ci].get(qid)
+ if g is None or z is None:
+ continue
+ rows.append((g["idx"], BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))
+ RESULTS["typed_decisions_matched_512"][model_name] = BE.hard_metrics(rows)
+ print(" %-22s max_len 512 -> acc %.3f ECE %.3f n=%d"
+ % (model_name, RESULTS["typed_decisions_matched_512"][model_name]["accuracy"],
+ RESULTS["typed_decisions_matched_512"][model_name]["ece"],
+ RESULTS["typed_decisions_matched_512"][model_name]["n"]))
+ finally:
+ agent.cfg["max_len"] = orig
+"""),
+
+ md("## 7. Latency (p50 / p95 at 1, 5, 10, 50 questions per call)"),
+ code("""
+import numpy as np, time, torch
+
+LAT_STATE = {"ticket": {"subject": "Payout failing",
+ "messages": [{"from": "customer",
+ "text": "Hi, my Stripe payouts have failed for 3 days and I am losing sales. Please help ASAP. " * 6}]}}
+Q_NOUL = {"type": "noul", "instructions": "Does `ticket.messages[0].text` express urgency?"}
+Q_CHOICE = {"type": "choice", "instructions": "Which team should handle this?",
+ "criteria": {"billing": "payments", "technical": "bugs and integrations", "sales": "pricing"}}
+
+RESULTS["latency"] = {}
+for model_name in REPOS:
+ agent = globals()["AGENT_" + model_name.replace("-", "_")]
+ res = {}
+ for nq in (1, 5, 10, 50):
+ qs = {("q%d" % i): (Q_NOUL if i % 2 else Q_CHOICE) for i in range(nq)}
+ for _ in range(3):
+ agent.system_one(LAT_STATE, qs)
+ ts = []
+ for _ in range(20):
+ torch.cuda.synchronize(); t = time.perf_counter()
+ agent.system_one(LAT_STATE, qs)
+ torch.cuda.synchronize(); ts.append((time.perf_counter()-t)*1000)
+ res["%d_questions" % nq] = {"p50_ms": round(float(np.percentile(ts, 50)), 1),
+ "p95_ms": round(float(np.percentile(ts, 95)), 1),
+ "ms_per_question": round(float(np.percentile(ts, 50))/nq, 2)}
+ RESULTS["latency"][model_name] = res
+ print(model_name, json.dumps(res))
+"""),
+
+ md("""## 8. Option-order robustness
+
+Same question, options permuted. A well-behaved decision model should pick the same label.
+The independent Jev benchmark measured a **13% flip rate for Jev**, with the worst LLM at 37%."""),
+ code("""
+import random, numpy as np
+import bench_engine as BE
+
+PB_SUITES = [s for s in ("massive_intent.en", "en.emotion", "xnli.en") if s in SUITES]
+RESULTS["option_order_robustness"] = {}
+
+for model_name in REPOS:
+ agent = globals()["AGENT_" + model_name.replace("-", "_")]
+ out = {}
+ for sname in PB_SUITES:
+ S = SUITES[sname]
+ base = S["cases"][:200]
+ rng = random.Random(99)
+ perm_cases, perm_maps = [], []
+ for state, questions in base:
+ qid = list(questions)[0]
+ qdef = questions[qid]
+ keys = list(qdef["criteria"].keys())
+ order = list(range(len(keys))); rng.shuffle(order)
+ newcrit = {keys[i]: qdef["criteria"][keys[i]] for i in order}
+ perm_cases.append((state, {qid: {**qdef, "criteria": newcrit}}))
+ perm_maps.append((keys, [keys[i] for i in order]))
+ la, ia, _, _ = BE.score_cases(agent, base, progress="")
+ lb, ib, _, _ = BE.score_cases(agent, perm_cases, progress="")
+ flips = comp = 0
+ for (za, (_, _, _, ka)), (zb, (_, _, _, kb)), (keys, pkeys) in zip(zip(la, ia), zip(lb, ib), perm_maps):
+ if za is None or zb is None:
+ continue
+ comp += 1
+ if keys[int(np.argmax(za))] != pkeys[int(np.argmax(zb))]:
+ flips += 1
+ out[sname] = {"n": comp, "flip_rate": round(flips/max(1, comp), 4)}
+ print(" %-22s %-24s flip rate %.3f (n=%d)" % (model_name, sname, out[sname]["flip_rate"], comp))
+ RESULTS["option_order_robustness"][model_name] = out
+"""),
+
+ md("""## 9. Calibration repair
+
+`laya-multilingual` ships with `temperature = [1.0, 1.0, 1.0]` and no per-option-count buckets —
+it was never calibrated. This refits one temperature per (question type, option-count bucket) on
+**half** of each suite and reports ECE on the other half, so the improvement is measured
+out-of-sample. It shows how much of any calibration gap is a missing post-processing step rather
+than a property of the model."""),
+ code("""
+import numpy as np
+import bench_engine as BE
+from laya.common import temp_bucket
+
+RESULTS["calibration_repair"] = {}
+for model_name in REPOS:
+ agent = globals()["AGENT_" + model_name.replace("-", "_")]
+ per_suite, fitted_all = {}, {}
+ for sname, S in SUITES.items():
+ if sname not in RAW[model_name]:
+ continue
+ logits, index = RAW[model_name][sname]
+ pairs = []
+ for (ci, qid, qt, k), z in zip(index, logits):
+ g = S["gold"][ci].get(qid)
+ if g is not None and z is not None:
+ pairs.append((z, g["idx"], qt, k))
+ if len(pairs) < 60:
+ continue
+ half = len(pairs)//2
+ fit_set, held = pairs[:half], pairs[half:]
+ buckets = {}
+ for z, gi, qt, k in fit_set:
+ buckets.setdefault(temp_bucket(qt, k), []).append((z, gi))
+ fitted = {b: BE.fit_temperature(v) for b, v in buckets.items() if len(v) >= 25}
+ rows_ship, rows_fit = [], []
+ for z, gi, qt, k in held:
+ rows_ship.append((gi, BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))
+ rows_fit.append((gi, BE.softmax_t(z, fitted.get(temp_bucket(qt, k),
+ BE.temp_for(agent, qt, k, True)))))
+ ms, mf = BE.hard_metrics(rows_ship), BE.hard_metrics(rows_fit)
+ per_suite[sname] = {"n_heldout": ms["n"], "fitted_temperatures": fitted,
+ "ece_shipped": ms["ece"], "ece_refit": mf["ece"],
+ "nll_shipped": ms["nll"], "nll_refit": mf["nll"],
+ "accuracy": ms["accuracy"]}
+ fitted_all.update(fitted)
+ RESULTS["calibration_repair"][model_name] = {"per_suite": per_suite}
+ if per_suite:
+ a = float(np.mean([v["ece_shipped"] for v in per_suite.values()]))
+ b = float(np.mean([v["ece_refit"] for v in per_suite.values()]))
+ RESULTS["calibration_repair"][model_name]["mean_ece_shipped"] = round(a, 4)
+ RESULTS["calibration_repair"][model_name]["mean_ece_refit"] = round(b, 4)
+ print("%-22s mean ECE shipped %.4f -> refit %.4f" % (model_name, a, b))
+"""),
+
+ md("## 10. Aggregate, save and download"),
+ code("""
+import numpy as np, json
+
+def agg(prefix_or_family, key="family"):
+ out = {}
+ for m in REPOS:
+ accs, eces, ns = [], [], 0
+ for sname, S in SUITES.items():
+ if S["meta"].get(key) != prefix_or_family:
+ continue
+ r = RESULTS["suites"].get(sname, {}).get(m, {})
+ c = r.get("calibrated", {})
+ if c.get("n"):
+ accs.append(c["accuracy"]); eces.append(c["ece"]); ns += c["n"]
+ if accs:
+ out[m] = {"macro_accuracy": round(float(np.mean(accs)), 4),
+ "macro_ece": round(float(np.mean(eces)), 4),
+ "n_questions": ns, "n_suites": len(accs)}
+ return out
+
+RESULTS["caveats"] = {
+ "context_window": "laya max_len=512, laya-multilingual max_len=1024 (shipped configs). "
+ "See typed_decisions_matched_512 for the equal-budget control.",
+ "shipped_calibration": "laya ships fitted temperatures; laya-multilingual ships all-1.0 "
+ "(uncalibrated). See calibration_repair for the out-of-sample refit.",
+ "training_overlap": "ag_news and boolq were in Laya's training mix (retention, not generalisation). "
+ "sst5, emotion, prompt_injections, banking77 were held out. "
+ "Neither checkpoint trained on typed-decisions, MASSIVE or XNLI.",
+}
+
+RESULTS["summary"] = {
+ "typed_decisions": {m: RESULTS["suites"].get("typed_decisions", {}).get(m, {}).get("calibrated", {})
+ for m in REPOS},
+ "massive_intent": agg("massive_intent"),
+ "massive_scenario": agg("massive_scenario"),
+ "xnli": agg("xnli"),
+ "english": agg("english"),
+}
+
+# English vs non-English split inside each multilingual family
+for fam in ("massive_intent", "massive_scenario", "xnli"):
+ en, non = {}, {}
+ for m in REPOS:
+ ea, na = [], []
+ for sname, S in SUITES.items():
+ if S["meta"].get("family") != fam:
+ continue
+ c = RESULTS["suites"].get(sname, {}).get(m, {}).get("calibrated", {})
+ if not c.get("n"):
+ continue
+ (ea if S["meta"].get("lang") in ("en",) else na).append(c["accuracy"])
+ if ea: en[m] = round(float(np.mean(ea)), 4)
+ if na: non[m] = round(float(np.mean(na)), 4)
+ RESULTS["summary"][fam + "_english_vs_rest"] = {"english": en, "non_english": non}
+
+json.dump(RESULTS, open("laya_benchmark_results.json", "w"), indent=2)
+print(json.dumps(RESULTS["summary"], indent=2))
+print("\\nwrote laya_benchmark_results.json (%.1f KB)" % (os.path.getsize("laya_benchmark_results.json")/1024))
+"""),
+
+ code("""
+# Per-language table, printed for a quick eyeball before you download the JSON
+import numpy as np
+for fam in ("massive_intent", "massive_scenario", "xnli"):
+ langs = sorted({S["meta"]["lang"] for S in SUITES.values() if S["meta"].get("family") == fam})
+ if not langs:
+ continue
+ print("\\n=== %s : accuracy by language ===" % fam)
+ print("%-8s %12s %22s %10s" % ("lang", "laya", "laya-multilingual", "delta"))
+ for lg in langs:
+ sname = "%s.%s" % (fam, lg)
+ a = RESULTS["suites"].get(sname, {}).get("laya", {}).get("calibrated", {}).get("accuracy")
+ b = RESULTS["suites"].get(sname, {}).get("laya-multilingual", {}).get("calibrated", {}).get("accuracy")
+ if a is None or b is None:
+ continue
+ print("%-8s %12.3f %22.3f %+10.3f" % (lg, a, b, b-a))
+"""),
+
+ code("""
+try:
+ from google.colab import files
+ files.download("laya_benchmark_results.json")
+except Exception as e:
+ print("Not on Colab or download blocked (%s). Grab laya_benchmark_results.json from the file browser." % type(e).__name__)
+"""),
+]
+
+
+def main():
+ nb = {
+ "cells": CELLS,
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {"provenance": [], "gpuType": "T4", "toc_visible": True},
+ "kernelspec": {"display_name": "Python 3", "name": "python3"},
+ "language_info": {"name": "python"},
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ }
+ out = os.path.join(HERE, "laya_benchmark_colab.ipynb")
+ with open(out, "w") as f:
+ json.dump(nb, f, indent=1)
+ print("wrote %s (%d cells)" % (out, len(CELLS)))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/research/scripts/laya_benchmark_colab.ipynb b/research/scripts/laya_benchmark_colab.ipynb
new file mode 100644
index 00000000..2979c985
--- /dev/null
+++ b/research/scripts/laya_benchmark_colab.ipynb
@@ -0,0 +1,983 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Laya vs Laya-Multilingual \u2014 extensive head-to-head benchmark\n",
+ "\n",
+ "Runs both open checkpoints over the same questions and writes a single\n",
+ "`laya_benchmark_results.json`.\n",
+ "\n",
+ "| | `convaiinnovations/laya` | `convaiinnovations/laya-multilingual` |\n",
+ "|---|---|---|\n",
+ "| encoder | ModernBERT-large (English) | mmBERT-base (100+ languages) |\n",
+ "| params | 421.3M (394.8M enc + 26.5M head) | 321.9M (306.9M enc + 15.0M head) |\n",
+ "| context | 512 tokens | 1024 tokens |\n",
+ "| shipped temperatures | fitted per type & option-count | **none (all 1.0)** |\n",
+ "\n",
+ "**What is measured**\n",
+ "\n",
+ "1. **typed-decisions** \u2014 400 cases / 2,000 decisions, 4 workflows (the benchmark behind the published 0.766 claim). Zero-shot for both of these checkpoints.\n",
+ "2. **Multilingual** \u2014 MASSIVE intent + scenario across 14 languages, XNLI across 15 languages.\n",
+ "3. **English zero-shot** \u2014 SST-5, emotion, prompt-injections, plus banking77 as a 77-option stress test.\n",
+ "4. **Latency** \u2014 p50/p95 at 1 / 5 / 10 / 50 questions per call.\n",
+ "5. **Option-order robustness** \u2014 how often the answer flips when the options are permuted (the independent Jev benchmark measured 13% for Jev; worst LLM 37%).\n",
+ "6. **Calibration repair** \u2014 ECE before and after refitting temperature, which matters because `laya-multilingual` shipped uncalibrated.\n",
+ "\n",
+ "**Setup:** Runtime \u2192 Change runtime type \u2192 **T4 GPU**. Then Run All (~25\u201340 min).\n",
+ "Both models see byte-identical questions (fixed seed), so every difference is the model."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 1. GPU and environment"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import subprocess, sys, os, json, time\n",
+ "print(subprocess.run([\"nvidia-smi\",\"--query-gpu=name,memory.total,compute_cap\",\"--format=csv,noheader\"],\n",
+ " capture_output=True, text=True).stdout.strip() or \"NO GPU - set Runtime > Change runtime type > T4 GPU\")\n",
+ "import torch\n",
+ "print(\"torch\", torch.__version__, \"| cuda\", torch.cuda.is_available(),\n",
+ " \"| device\", torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"cpu\")\n",
+ "assert torch.cuda.is_available(), \"Enable the T4 GPU runtime before running.\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 2. Dependencies"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!pip -q install -U \"laya>=0.1.6\" \"transformers>=4.45\" \"datasets>=3.0\" safetensors huggingface_hub 2>&1 | tail -3\n",
+ "import laya, transformers, datasets\n",
+ "print(\"laya\", getattr(laya, \"__version__\", \"?\"), \"| transformers\", transformers.__version__, \"| datasets\", datasets.__version__)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 3. Download both checkpoints\n",
+ "\n",
+ "`reference_compile` is forced off after loading: the `laya` package does not disable it (only\n",
+ "`rl_agent_api.py` does), and torch.compile is a loss on small batches / few SMs like a T4."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import torch, json, os, time\n",
+ "from huggingface_hub import snapshot_download\n",
+ "from safetensors import safe_open\n",
+ "\n",
+ "REPOS = {\"laya\": \"convaiinnovations/laya\", \"laya-multilingual\": \"convaiinnovations/laya-multilingual\"}\n",
+ "PATHS, MODEL_META = {}, {}\n",
+ "\n",
+ "def patch_tokenizer_config(model_dir):\n",
+ " '''laya-multilingual ships extra_special_tokens as a LIST (inherited from the mmBERT/Gemma\n",
+ " tokenizer); transformers expects a dict and raises\n",
+ " AttributeError: list object has no attribute keys. Without this, AutoTokenizer -- and so\n",
+ " laya.load() -- fails outright on that repo. Also normalises tokenizer_class for 4.x/5.x.'''\n",
+ " p = os.path.join(model_dir, \"tokenizer\", \"tokenizer_config.json\")\n",
+ " if not os.path.exists(p):\n",
+ " return []\n",
+ " c = json.load(open(p))\n",
+ " changed = []\n",
+ " est = c.get(\"extra_special_tokens\")\n",
+ " if isinstance(est, list):\n",
+ " c[\"extra_special_tokens\"] = {(\"extra_%d\" % i): t for i, t in enumerate(est)}\n",
+ " changed.append(\"extra_special_tokens list->dict\")\n",
+ " if c.get(\"tokenizer_class\") in (None, \"TokenizersBackend\"):\n",
+ " c[\"tokenizer_class\"] = \"PreTrainedTokenizerFast\"\n",
+ " c.pop(\"backend\", None); c.pop(\"is_local\", None)\n",
+ " changed.append(\"tokenizer_class\")\n",
+ " if changed:\n",
+ " json.dump(c, open(p, \"w\"), indent=2)\n",
+ " return changed\n",
+ "\n",
+ "for name, repo in REPOS.items():\n",
+ " t = time.time()\n",
+ " PATHS[name] = snapshot_download(repo)\n",
+ " fixed = patch_tokenizer_config(PATHS[name])\n",
+ " if fixed:\n",
+ " print(\" patched tokenizer config for %s: %s\" % (name, \", \".join(fixed)))\n",
+ " cfg = json.load(open(os.path.join(PATHS[name], \"rl_agent_config.json\")))\n",
+ " ecfg = json.load(open(os.path.join(PATHS[name], \"encoder\", \"config.json\")))\n",
+ " tot = enc = 0\n",
+ " with safe_open(os.path.join(PATHS[name], \"model.safetensors\"), \"pt\") as f:\n",
+ " for k in f.keys():\n",
+ " n = 1\n",
+ " for d in f.get_slice(k).get_shape():\n",
+ " n *= d\n",
+ " tot += n\n",
+ " enc += n if k.startswith(\"encoder.\") else 0\n",
+ " MODEL_META[name] = {\n",
+ " \"repo\": repo, \"params_total_m\": round(tot/1e6, 2), \"params_encoder_m\": round(enc/1e6, 2),\n",
+ " \"params_head_m\": round((tot-enc)/1e6, 2), \"hidden_size\": ecfg.get(\"hidden_size\"),\n",
+ " \"num_layers\": ecfg.get(\"num_hidden_layers\"), \"vocab_size\": ecfg.get(\"vocab_size\"),\n",
+ " \"max_len\": cfg.get(\"max_len\"), \"head_max_len\": cfg.get(\"head_max_len\"),\n",
+ " \"temperature\": cfg.get(\"temperature\"), \"temperature_by_options\": cfg.get(\"temperature_by_options\", {}),\n",
+ " \"training\": cfg.get(\"training\", {}), \"download_s\": round(time.time()-t, 1),\n",
+ " \"tokenizer_config_patched\": fixed,\n",
+ " }\n",
+ " print(name, \"->\", PATHS[name])\n",
+ " print(\" \", json.dumps({k: v for k, v in MODEL_META[name].items() if k != \"temperature_by_options\"}))\n",
+ "\n",
+ "def load_agent(name):\n",
+ " import laya\n",
+ " ag = laya.load(PATHS[name], device=\"cuda\")\n",
+ " try:\n",
+ " ag.model.encoder.config.reference_compile = False # torch.compile hurts on T4-class GPUs\n",
+ " except Exception as e:\n",
+ " print(\"could not disable reference_compile:\", e)\n",
+ " ag.model.eval()\n",
+ " return ag"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 4. Benchmark engine \u2014 batched scoring and metrics"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%writefile bench_engine.py\n",
+ "import json, math, sys, time\n",
+ "import numpy as np\n",
+ "import torch\n",
+ "from laya.common import QTYPES, build_sequence, collate_items, render_options, temp_bucket\n",
+ "\n",
+ "def to_internal(qdef):\n",
+ " t = qdef[\"type\"]\n",
+ " crit = qdef.get(\"criteria\")\n",
+ " if t == \"choice\" and isinstance(crit, list):\n",
+ " crit = {c: None for c in crit}\n",
+ " ins = qdef[\"instructions\"]\n",
+ " if not isinstance(ins, str):\n",
+ " ins = json.dumps(ins)\n",
+ " return {\"t\": t, \"ins\": ins, \"crit\": crit}\n",
+ "\n",
+ "@torch.no_grad()\n",
+ "def score_cases(agent, cases, max_tokens=16384, max_seqs=128, progress=\"\"):\n",
+ " '''cases: [(state, {qid: qdef})] -> (raw_logits per question, index, seconds, n_dropped).\n",
+ "\n",
+ " Every question of every case is one sequence. Sequences are length-sorted and packed into\n",
+ " token-budgeted batches so padding stays small. Returns UNCALIBRATED logits; temperature is\n",
+ " applied later so raw/calibrated variants come from one forward pass.\n",
+ " '''\n",
+ " max_len = agent.cfg.get(\"max_len\", 512)\n",
+ " head_max_len = agent.cfg.get(\"head_max_len\", 192)\n",
+ " items, index, dropped = [], [], 0\n",
+ " for ci, (state, questions) in enumerate(cases):\n",
+ " for qid, qdef in questions.items():\n",
+ " q = to_internal(qdef)\n",
+ " try:\n",
+ " ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)\n",
+ " except Exception:\n",
+ " index.append((ci, qid, QTYPES[q[\"t\"]], 0)); items.append(None); dropped += 1\n",
+ " continue\n",
+ " if len(markers) != len(render_options(q)):\n",
+ " index.append((ci, qid, QTYPES[q[\"t\"]], 0)); items.append(None); dropped += 1\n",
+ " continue\n",
+ " items.append({\"ids\": ids, \"markers\": markers, \"qtype\": QTYPES[q[\"t\"]]})\n",
+ " index.append((ci, qid, QTYPES[q[\"t\"]], len(markers)))\n",
+ "\n",
+ " valid = [i for i, it in enumerate(items) if it is not None]\n",
+ " order = sorted(valid, key=lambda i: len(items[i][\"ids\"]))\n",
+ " out = [None] * len(items)\n",
+ " t0, done, i = time.time(), 0, 0\n",
+ " while i < len(order):\n",
+ " j, L = i, 0\n",
+ " while j < len(order) and j - i < max_seqs and max(L, len(items[order[j]][\"ids\"])) * (j - i + 1) <= max_tokens:\n",
+ " L = max(L, len(items[order[j]][\"ids\"])); j += 1\n",
+ " j = max(j, i + 1)\n",
+ " sel = [items[order[t]] for t in range(i, j)]\n",
+ " b = collate_items([sel], agent.tok.pad_token_id)\n",
+ " with torch.autocast(\"cuda\", dtype=agent.dtype, enabled=True):\n",
+ " logits, _ = agent.model(b[\"input_ids\"].cuda(), b[\"attention_mask\"].cuda(),\n",
+ " b[\"marker_pos\"].cuda(), b[\"marker_mask\"].cuda(), b[\"qtype\"].cuda())\n",
+ " logits = logits.float().cpu().numpy()\n",
+ " for r in range(j - i):\n",
+ " out[order[i + r]] = logits[r, :len(sel[r][\"markers\"])]\n",
+ " done += j - i\n",
+ " if progress:\n",
+ " el = time.time() - t0\n",
+ " msg = \"\\r [%s] %d/%d seq | %.0f seq/s | ETA %ds \" % (\n",
+ " progress, done, len(order), done/max(el, 1e-9),\n",
+ " int(el*(len(order)-done)/max(1, done)))\n",
+ " sys.stdout.write(msg)\n",
+ " sys.stdout.flush()\n",
+ " i = j\n",
+ " if progress:\n",
+ " print(\"\\r [%s] %d sequences in %.1fs (%d dropped: options did not fit)%s\" % (\n",
+ " progress, len(order), time.time()-t0, dropped, \" \"*20))\n",
+ " return out, index, time.time() - t0, dropped\n",
+ "\n",
+ "def softmax_t(z, t=1.0):\n",
+ " z = np.asarray(z, dtype=np.float64) / max(1e-3, float(t))\n",
+ " e = np.exp(z - z.max())\n",
+ " return e / e.sum()\n",
+ "\n",
+ "def temp_for(agent, qt, k, calibrated=True):\n",
+ " if not calibrated:\n",
+ " return 1.0\n",
+ " return float(agent.temperature_by_options.get(temp_bucket(qt, k), agent.temperature[qt]))\n",
+ "\n",
+ "# ------------------------------------------------------------------ metrics\n",
+ "def ece_score(conf, correct, bins=15):\n",
+ " conf, correct = np.asarray(conf, float), np.asarray(correct, float)\n",
+ " if len(conf) == 0:\n",
+ " return float(\"nan\")\n",
+ " edges = np.linspace(0, 1, bins + 1)\n",
+ " e = 0.0\n",
+ " for lo, hi in zip(edges[:-1], edges[1:]):\n",
+ " sel = (conf > lo) & (conf <= hi)\n",
+ " if sel.any():\n",
+ " e += sel.mean() * abs(conf[sel].mean() - correct[sel].mean())\n",
+ " return float(e)\n",
+ "\n",
+ "def macro_f1(gold, pred):\n",
+ " gold, pred = np.asarray(gold), np.asarray(pred)\n",
+ " f1 = []\n",
+ " for c in sorted(set(gold.tolist()) | set(pred.tolist())):\n",
+ " tp = int(((pred == c) & (gold == c)).sum())\n",
+ " fp = int(((pred == c) & (gold != c)).sum())\n",
+ " fn = int(((pred != c) & (gold == c)).sum())\n",
+ " f1.append(2*tp / max(1, 2*tp + fp + fn))\n",
+ " return float(np.mean(f1))\n",
+ "\n",
+ "def aurc(conf, correct):\n",
+ " conf, correct = np.asarray(conf, float), np.asarray(correct, float)\n",
+ " if len(conf) == 0:\n",
+ " return float(\"nan\")\n",
+ " o = np.argsort(-conf)\n",
+ " return float((np.cumsum(1 - correct[o]) / np.arange(1, len(o)+1)).mean())\n",
+ "\n",
+ "def hard_metrics(rows):\n",
+ " '''rows: [(gold_idx, probs)] -> standard single-label classification + calibration metrics.'''\n",
+ " rows = [r for r in rows if r[1] is not None]\n",
+ " if not rows:\n",
+ " return {\"n\": 0}\n",
+ " gold = np.array([g for g, _ in rows])\n",
+ " pred = np.array([int(np.argmax(p)) for _, p in rows])\n",
+ " conf = np.array([float(np.max(p)) for _, p in rows])\n",
+ " corr = (pred == gold).astype(float)\n",
+ " m = {\"n\": len(rows),\n",
+ " \"accuracy\": float(corr.mean()),\n",
+ " \"macro_f1\": macro_f1(gold, pred),\n",
+ " \"ece\": ece_score(conf, corr),\n",
+ " \"brier\": float(np.mean([((np.asarray(p) - np.eye(len(p))[g])**2).sum() for g, p in rows])),\n",
+ " \"nll\": float(np.mean([-math.log(max(float(p[g]), 1e-12)) for g, p in rows])),\n",
+ " \"aurc\": aurc(conf, corr),\n",
+ " \"mean_confidence\": float(conf.mean())}\n",
+ " for cov in (0.5, 0.8):\n",
+ " k = max(1, int(len(conf)*cov))\n",
+ " m[\"acc_at_%d_coverage\" % int(cov*100)] = float(corr[np.argsort(-conf)[:k]].mean())\n",
+ " return m\n",
+ "\n",
+ "def fit_temperature(pairs, lo=0.2, hi=10.0, steps=160):\n",
+ " '''pairs: [(logits, gold_idx)] -> single temperature minimising NLL (grid search, no autograd).'''\n",
+ " pairs = [(np.asarray(z, float), int(g)) for z, g in pairs if z is not None]\n",
+ " if len(pairs) < 25:\n",
+ " return 1.0\n",
+ " best_t, best = 1.0, float(\"inf\")\n",
+ " for t in np.geomspace(lo, hi, steps):\n",
+ " tot = 0.0\n",
+ " for z, g in pairs:\n",
+ " zz = z/t\n",
+ " zz = zz - zz.max()\n",
+ " tot += -(zz[g] - math.log(np.exp(zz).sum()))\n",
+ " if tot < best:\n",
+ " best, best_t = tot, float(t)\n",
+ " return round(float(best_t), 4)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 5. Task suites\n",
+ "\n",
+ "Every suite is built **once**, with a fixed seed, and reused for both models \u2014 so the two\n",
+ "checkpoints answer byte-identical questions. Option sets for many-label tasks use\n",
+ "`gold + N-1 distractors`, matching how the models were trained (the training recipe samples\n",
+ "5\u201320 options per choice question rather than showing the full label space)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import random, json\n",
+ "import numpy as np\n",
+ "from datasets import load_dataset\n",
+ "\n",
+ "SEED = 13\n",
+ "N_OPTS = 20 # options per many-label choice question (gold + 19 distractors)\n",
+ "PER_LANG = 300 # cases per language\n",
+ "SUITES = {} # name -> {\"cases\": [(state, questions)], \"gold\": [{qid: gold_idx}], \"meta\": {...}}\n",
+ "\n",
+ "def build_choice_questions(rng, gold, all_labels, n_opts, instructions, qid=\"label\"):\n",
+ " pool = [x for x in all_labels if x != gold]\n",
+ " keys = [gold] + rng.sample(pool, min(n_opts-1, len(pool)))\n",
+ " rng.shuffle(keys)\n",
+ " crit = {k: k.replace(\"_\", \" \").replace(\".\", \": \") for k in keys}\n",
+ " return {qid: {\"type\": \"choice\", \"instructions\": instructions, \"criteria\": crit}}, keys.index(gold)\n",
+ "\n",
+ "def register(name, cases, golds, **meta):\n",
+ " SUITES[name] = {\"cases\": cases, \"gold\": golds, \"meta\": meta}\n",
+ " nq = sum(len(c[1]) for c in cases)\n",
+ " print(\" %-34s %4d cases / %5d questions %s\" % (name, len(cases), nq, meta.get(\"note\", \"\")))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 5a. typed-decisions (English, 400 cases / 2,000 decisions)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "td_cases, td_golds, td_wf = [], [], []\n",
+ "try:\n",
+ " td = load_dataset(\"LocalLLaMA/typed-decisions\", \"all\", split=\"test\")\n",
+ " for r in td:\n",
+ " questions = json.loads(r[\"questions\"]) if isinstance(r[\"questions\"], str) else r[\"questions\"]\n",
+ " gold = json.loads(r[\"gold\"]) if isinstance(r[\"gold\"], str) else r[\"gold\"]\n",
+ " state = r[\"state\"]\n",
+ " try:\n",
+ " state = json.loads(state)\n",
+ " except Exception:\n",
+ " pass\n",
+ " gmap = {}\n",
+ " for qid, qdef in questions.items():\n",
+ " g = gold[qid]\n",
+ " if qdef[\"type\"] == \"choice\":\n",
+ " keys = list(qdef[\"criteria\"].keys())\n",
+ " gmap[qid] = {\"idx\": keys.index(str(g[\"label\"])), \"keys\": keys,\n",
+ " \"soft\": [float(g.get(\"probabilities\", {}).get(k, 0.0)) for k in keys]}\n",
+ " elif qdef[\"type\"] == \"noul\":\n",
+ " idx = 1 if str(g[\"label\"]).lower() == \"true\" else 0\n",
+ " pt = float(g.get(\"probabilities\", {}).get(\"true\", g.get(\"noul\", 0.5)))\n",
+ " gmap[qid] = {\"idx\": idx, \"keys\": [\"false\", \"true\"], \"soft\": [1-pt, pt]}\n",
+ " else: # score\n",
+ " n = len(qdef[\"criteria\"])\n",
+ " gmap[qid] = {\"idx\": int(g[\"label\"]), \"keys\": [str(i) for i in range(n)],\n",
+ " \"soft\": [float(g.get(\"probabilities\", {}).get(str(i), 0.0)) for i in range(n)],\n",
+ " \"gold_score\": float(g.get(\"score\", float(g[\"label\"])))}\n",
+ " td_cases.append((state, questions))\n",
+ " td_golds.append(gmap)\n",
+ " td_wf.append(r[\"workflow\"])\n",
+ " register(\"typed_decisions\", td_cases, td_golds, note=\"4 workflows, soft teacher labels\",\n",
+ " workflows=td_wf, soft=True)\n",
+ "except Exception as e:\n",
+ " print(\"typed-decisions FAILED:\", type(e).__name__, e)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 5b. Multilingual \u2014 MASSIVE intent & scenario (14 languages), XNLI (15 languages)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "MASSIVE_LANGS = [\"en\",\"de\",\"fr\",\"es\",\"pt\",\"ru\",\"tr\",\"ar\",\"hi\",\"ta\",\"zh-CN\",\"ja\",\"ko\",\"sw\"]\n",
+ "XNLI_LANGS = [\"en\",\"de\",\"fr\",\"es\",\"ru\",\"tr\",\"ar\",\"hi\",\"ur\",\"vi\",\"th\",\"el\",\"bg\",\"zh\",\"sw\"]\n",
+ "\n",
+ "for task, short, instr in [(\"mteb/amazon_massive_intent\", \"massive_intent\",\n",
+ " \"What is the user asking for in `utterance`?\"),\n",
+ " (\"mteb/amazon_massive_scenario\", \"massive_scenario\",\n",
+ " \"Which domain does `utterance` belong to?\")]:\n",
+ " for lg in MASSIVE_LANGS:\n",
+ " try:\n",
+ " d = load_dataset(task, lg, split=\"test\")\n",
+ " labels = sorted(set(d[\"label_text\"]))\n",
+ " rng = random.Random(SEED)\n",
+ " cases, golds = [], []\n",
+ " for r in list(d)[:PER_LANG]:\n",
+ " qs, gi = build_choice_questions(rng, r[\"label_text\"], labels, N_OPTS, instr)\n",
+ " cases.append(({\"utterance\": r[\"text\"]}, qs))\n",
+ " golds.append({\"label\": {\"idx\": gi}})\n",
+ " register(\"%s.%s\" % (short, lg), cases, golds, lang=lg, n_options=N_OPTS,\n",
+ " label_space=len(labels), family=short)\n",
+ " except Exception as e:\n",
+ " print(\" FAIL %s %s: %s\" % (short, lg, str(e)[:90]))\n",
+ "\n",
+ "NLI_CRIT = {\"entailment\": \"the premise implies the hypothesis is true\",\n",
+ " \"neutral\": \"the premise neither implies nor contradicts the hypothesis\",\n",
+ " \"contradiction\": \"the premise implies the hypothesis is false\"}\n",
+ "for lg in XNLI_LANGS:\n",
+ " try:\n",
+ " d = load_dataset(\"facebook/xnli\", lg, split=\"test\")\n",
+ " cases, golds = [], []\n",
+ " for r in list(d)[:PER_LANG]:\n",
+ " qs = {\"relation\": {\"type\": \"choice\",\n",
+ " \"instructions\": \"What is the relationship between `premise` and `hypothesis`?\",\n",
+ " \"criteria\": dict(NLI_CRIT)}}\n",
+ " cases.append(({\"premise\": r[\"premise\"], \"hypothesis\": r[\"hypothesis\"]}, qs))\n",
+ " golds.append({\"relation\": {\"idx\": int(r[\"label\"])}})\n",
+ " register(\"xnli.%s\" % lg, cases, golds, lang=lg, n_options=3, family=\"xnli\")\n",
+ " except Exception as e:\n",
+ " print(\" FAIL xnli %s: %s\" % (lg, str(e)[:90]))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 5c. English tasks\n",
+ "\n",
+ "`sst5`, `emotion` and `prompt_injections` were held out of Laya's training entirely (true\n",
+ "zero-shot). `ag_news` and `boolq` were in the training mix, so they measure retention rather\n",
+ "than generalisation. `banking77` is the many-option stress test \u2014 77 options at once, versus\n",
+ "Jev's documented hard cap at 255."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "rng = random.Random(SEED)\n",
+ "\n",
+ "# SST-5: ordinal score\n",
+ "try:\n",
+ " d = load_dataset(\"SetFit/sst5\", split=\"test\")\n",
+ " crit = [\"very negative\", \"negative\", \"neutral\", \"positive\", \"very positive\"]\n",
+ " cases, golds = [], []\n",
+ " for r in list(d)[:600]:\n",
+ " cases.append(({\"text\": r[\"text\"]},\n",
+ " {\"sentiment\": {\"type\": \"score\", \"instructions\": \"How positive is the sentiment of `text`?\",\n",
+ " \"criteria\": crit}}))\n",
+ " golds.append({\"sentiment\": {\"idx\": int(r[\"label\"]), \"gold_score\": float(r[\"label\"])}})\n",
+ " register(\"en.sst5\", cases, golds, note=\"zero-shot, ordinal\", family=\"english\", held_out=True)\n",
+ "except Exception as e: print(\" FAIL sst5:\", str(e)[:90])\n",
+ "\n",
+ "# emotion: 6-way choice\n",
+ "try:\n",
+ " d = load_dataset(\"dair-ai/emotion\", \"split\", split=\"test\")\n",
+ " names = [\"sadness\",\"joy\",\"love\",\"anger\",\"fear\",\"surprise\"]\n",
+ " cases, golds = [], []\n",
+ " for r in list(d)[:600]:\n",
+ " cases.append(({\"text\": r[\"text\"]},\n",
+ " {\"emotion\": {\"type\": \"choice\",\n",
+ " \"instructions\": \"Which emotion is most strongly expressed in `text`?\",\n",
+ " \"criteria\": {n: None for n in names}}}))\n",
+ " golds.append({\"emotion\": {\"idx\": int(r[\"label\"])}})\n",
+ " register(\"en.emotion\", cases, golds, note=\"zero-shot\", family=\"english\", held_out=True)\n",
+ "except Exception as e: print(\" FAIL emotion:\", str(e)[:90])\n",
+ "\n",
+ "# prompt injections: noul\n",
+ "try:\n",
+ " d = load_dataset(\"deepset/prompt-injections\", split=\"test\")\n",
+ " cases, golds = [], []\n",
+ " for r in list(d):\n",
+ " cases.append(({\"text\": r[\"text\"]},\n",
+ " {\"injection\": {\"type\": \"noul\",\n",
+ " \"instructions\": \"Does `text` try to inject or override instructions given to an AI system?\"}}))\n",
+ " golds.append({\"injection\": {\"idx\": int(r[\"label\"])}})\n",
+ " register(\"en.prompt_injections\", cases, golds, note=\"zero-shot\", family=\"english\", held_out=True)\n",
+ "except Exception as e: print(\" FAIL prompt_injections:\", str(e)[:90])\n",
+ "\n",
+ "# banking77: 77-option stress test\n",
+ "try:\n",
+ " d = load_dataset(\"PolyAI/banking77\", split=\"test\")\n",
+ " names = d.features[\"label\"].names\n",
+ " cases, golds = [], []\n",
+ " for r in list(d)[:500]:\n",
+ " cases.append(({\"message\": r[\"text\"]},\n",
+ " {\"intent\": {\"type\": \"choice\", \"instructions\": \"Which banking intent does `message` express?\",\n",
+ " \"criteria\": {n.replace(\"_\", \" \"): None for n in names}}}))\n",
+ " golds.append({\"intent\": {\"idx\": int(r[\"label\"])}})\n",
+ " register(\"en.banking77_full\", cases, golds, note=\"77 options at once\", family=\"english\",\n",
+ " n_options=len(names), held_out=True)\n",
+ "except Exception as e: print(\" FAIL banking77:\", str(e)[:90])\n",
+ "\n",
+ "# ag_news + boolq: in-task retention\n",
+ "try:\n",
+ " d = load_dataset(\"fancyzhx/ag_news\", split=\"test\")\n",
+ " crit = {\"world\": \"world news and international politics\", \"sports\": \"sports\",\n",
+ " \"business\": \"business and economy\", \"sci_tech\": \"science and technology\"}\n",
+ " cases, golds = [], []\n",
+ " for r in list(d)[:600]:\n",
+ " cases.append(({\"article\": r[\"text\"]},\n",
+ " {\"topic\": {\"type\": \"choice\", \"instructions\": \"What is the topic of `article`?\",\n",
+ " \"criteria\": dict(crit)}}))\n",
+ " golds.append({\"topic\": {\"idx\": int(r[\"label\"])}})\n",
+ " register(\"en.ag_news\", cases, golds, note=\"in-task (trained on)\", family=\"english\", held_out=False)\n",
+ "except Exception as e: print(\" FAIL ag_news:\", str(e)[:90])\n",
+ "\n",
+ "try:\n",
+ " d = load_dataset(\"google/boolq\", split=\"validation\")\n",
+ " cases, golds = [], []\n",
+ " for r in list(d)[:600]:\n",
+ " cases.append(({\"passage\": r[\"passage\"], \"question\": r[\"question\"]},\n",
+ " {\"answer\": {\"type\": \"noul\",\n",
+ " \"instructions\": \"Based on `passage`, is the answer to `question` yes?\"}}))\n",
+ " golds.append({\"answer\": {\"idx\": int(bool(r[\"answer\"]))}})\n",
+ " register(\"en.boolq\", cases, golds, note=\"in-task (trained on)\", family=\"english\", held_out=False)\n",
+ "except Exception as e: print(\" FAIL boolq:\", str(e)[:90])\n",
+ "\n",
+ "print()\n",
+ "print(\"suites built:\", len(SUITES), \"| total questions:\",\n",
+ " sum(sum(len(c[1]) for c in s[\"cases\"]) for s in SUITES.values()))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 6. Run every suite on both models"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import bench_engine as BE\n",
+ "import numpy as np, gc, time, json\n",
+ "\n",
+ "RESULTS = {\"meta\": {\"seed\": SEED, \"n_opts\": N_OPTS, \"per_lang\": PER_LANG,\n",
+ " \"timestamp\": time.strftime(\"%Y-%m-%d %H:%M:%S\"),\n",
+ " \"gpu\": torch.cuda.get_device_name(0),\n",
+ " \"torch\": torch.__version__, \"transformers\": transformers.__version__,\n",
+ " \"models\": MODEL_META},\n",
+ " \"suites\": {}}\n",
+ "RAW = {} # model -> suite -> (logits, index) kept in RAM for the calibration-repair section\n",
+ "\n",
+ "def evaluate_suite(agent, model_name, suite_name):\n",
+ " S = SUITES[suite_name]\n",
+ " logits, index, secs, dropped = BE.score_cases(agent, S[\"cases\"], progress=\"%s/%s\" % (model_name, suite_name))\n",
+ " RAW[model_name][suite_name] = (logits, index)\n",
+ " soft = S[\"meta\"].get(\"soft\", False)\n",
+ " rows_cal, rows_raw, extra = [], [], {\"soft_acc\": [], \"brier_soft\": [], \"tv\": [], \"kl\": [],\n",
+ " \"score_mae\": [], \"within_1\": []}\n",
+ " per_q_type = {}\n",
+ " for (ci, qid, qt, k), z in zip(index, logits):\n",
+ " g = S[\"gold\"][ci].get(qid)\n",
+ " if g is None:\n",
+ " continue\n",
+ " if z is None:\n",
+ " rows_cal.append((g[\"idx\"], None)); rows_raw.append((g[\"idx\"], None)); continue\n",
+ " p_cal = BE.softmax_t(z, BE.temp_for(agent, qt, k, True))\n",
+ " p_raw = BE.softmax_t(z, 1.0)\n",
+ " rows_cal.append((g[\"idx\"], p_cal)); rows_raw.append((g[\"idx\"], p_raw))\n",
+ " per_q_type.setdefault(qt, []).append((g[\"idx\"], p_cal))\n",
+ " if soft and \"soft\" in g:\n",
+ " gp = np.asarray(g[\"soft\"], float)\n",
+ " if gp.sum() > 0:\n",
+ " gp = gp / gp.sum()\n",
+ " pp = p_cal[:len(gp)] if len(p_cal) >= len(gp) else np.pad(p_cal, (0, len(gp)-len(p_cal)))\n",
+ " pp = pp / max(pp.sum(), 1e-12)\n",
+ " extra[\"soft_acc\"].append(float((pp*gp).sum()))\n",
+ " extra[\"brier_soft\"].append(float(((pp-gp)**2).sum()))\n",
+ " extra[\"tv\"].append(float(0.5*np.abs(pp-gp).sum()))\n",
+ " extra[\"kl\"].append(float((gp*np.log(np.clip(gp/np.clip(pp,1e-12,None),1e-12,1e4))).sum()))\n",
+ " if \"gold_score\" in g:\n",
+ " exp = float((np.arange(len(p_cal))*p_cal).sum())\n",
+ " extra[\"score_mae\"].append(abs(exp - g[\"gold_score\"]))\n",
+ " extra[\"within_1\"].append(float(abs(exp - g[\"gold_score\"]) <= 1.0))\n",
+ "\n",
+ " out = {\"calibrated\": BE.hard_metrics(rows_cal), \"raw\": BE.hard_metrics(rows_raw),\n",
+ " \"seconds\": round(secs, 2), \"dropped_questions\": dropped,\n",
+ " \"questions_per_second\": round(len([r for r in rows_cal if r[1] is not None])/max(secs,1e-9), 1)}\n",
+ " for k2, v in extra.items():\n",
+ " if v:\n",
+ " out[k2] = float(np.mean(v))\n",
+ " if len(per_q_type) > 1:\n",
+ " names = {0: \"choice\", 1: \"score\", 2: \"noul\"}\n",
+ " out[\"by_question_type\"] = {names[qt]: BE.hard_metrics(rs) for qt, rs in sorted(per_q_type.items())}\n",
+ " return out\n",
+ "\n",
+ "for model_name in REPOS:\n",
+ " print(\"\\n\" + \"=\"*78 + \"\\n %s\\n\" % model_name + \"=\"*78)\n",
+ " agent = load_agent(model_name)\n",
+ " RAW[model_name] = {}\n",
+ " for suite_name in SUITES:\n",
+ " try:\n",
+ " r = evaluate_suite(agent, model_name, suite_name)\n",
+ " RESULTS[\"suites\"].setdefault(suite_name, {})[model_name] = r\n",
+ " c = r[\"calibrated\"]\n",
+ " print(\" %-32s acc %.3f f1 %.3f ECE %.3f n=%d\" %\n",
+ " (suite_name, c.get(\"accuracy\", float(\"nan\")), c.get(\"macro_f1\", float(\"nan\")),\n",
+ " c.get(\"ece\", float(\"nan\")), c.get(\"n\", 0)))\n",
+ " except Exception as e:\n",
+ " import traceback; traceback.print_exc()\n",
+ " RESULTS[\"suites\"].setdefault(suite_name, {})[model_name] = {\"error\": \"%s: %s\" % (type(e).__name__, e)}\n",
+ " globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")] = agent\n",
+ " gc.collect(); torch.cuda.empty_cache()\n",
+ "\n",
+ "json.dump(RESULTS, open(\"laya_benchmark_results.json\", \"w\"), indent=2)\n",
+ "print(\"\\ncheckpoint saved\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 6b. Control \u2014 matched context budget\n",
+ "\n",
+ "The two checkpoints ship different context windows (512 vs 1024 tokens), so on long states\n",
+ "`laya-multilingual` sees more of the input. This re-runs typed-decisions with **both** models\n",
+ "capped at 512 tokens, isolating the architecture from the context-window advantage."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import bench_engine as BE\n",
+ "import numpy as np\n",
+ "\n",
+ "RESULTS[\"typed_decisions_matched_512\"] = {}\n",
+ "if \"typed_decisions\" in SUITES:\n",
+ " S = SUITES[\"typed_decisions\"]\n",
+ " for model_name in REPOS:\n",
+ " agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",
+ " orig = agent.cfg.get(\"max_len\")\n",
+ " agent.cfg[\"max_len\"] = 512\n",
+ " try:\n",
+ " logits, index, secs, dropped = BE.score_cases(agent, S[\"cases\"], progress=\"%s/td@512\" % model_name)\n",
+ " rows = []\n",
+ " for (ci, qid, qt, k), z in zip(index, logits):\n",
+ " g = S[\"gold\"][ci].get(qid)\n",
+ " if g is None or z is None:\n",
+ " continue\n",
+ " rows.append((g[\"idx\"], BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))\n",
+ " RESULTS[\"typed_decisions_matched_512\"][model_name] = BE.hard_metrics(rows)\n",
+ " print(\" %-22s max_len 512 -> acc %.3f ECE %.3f n=%d\"\n",
+ " % (model_name, RESULTS[\"typed_decisions_matched_512\"][model_name][\"accuracy\"],\n",
+ " RESULTS[\"typed_decisions_matched_512\"][model_name][\"ece\"],\n",
+ " RESULTS[\"typed_decisions_matched_512\"][model_name][\"n\"]))\n",
+ " finally:\n",
+ " agent.cfg[\"max_len\"] = orig"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 7. Latency (p50 / p95 at 1, 5, 10, 50 questions per call)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np, time, torch\n",
+ "\n",
+ "LAT_STATE = {\"ticket\": {\"subject\": \"Payout failing\",\n",
+ " \"messages\": [{\"from\": \"customer\",\n",
+ " \"text\": \"Hi, my Stripe payouts have failed for 3 days and I am losing sales. Please help ASAP. \" * 6}]}}\n",
+ "Q_NOUL = {\"type\": \"noul\", \"instructions\": \"Does `ticket.messages[0].text` express urgency?\"}\n",
+ "Q_CHOICE = {\"type\": \"choice\", \"instructions\": \"Which team should handle this?\",\n",
+ " \"criteria\": {\"billing\": \"payments\", \"technical\": \"bugs and integrations\", \"sales\": \"pricing\"}}\n",
+ "\n",
+ "RESULTS[\"latency\"] = {}\n",
+ "for model_name in REPOS:\n",
+ " agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",
+ " res = {}\n",
+ " for nq in (1, 5, 10, 50):\n",
+ " qs = {(\"q%d\" % i): (Q_NOUL if i % 2 else Q_CHOICE) for i in range(nq)}\n",
+ " for _ in range(3):\n",
+ " agent.system_one(LAT_STATE, qs)\n",
+ " ts = []\n",
+ " for _ in range(20):\n",
+ " torch.cuda.synchronize(); t = time.perf_counter()\n",
+ " agent.system_one(LAT_STATE, qs)\n",
+ " torch.cuda.synchronize(); ts.append((time.perf_counter()-t)*1000)\n",
+ " res[\"%d_questions\" % nq] = {\"p50_ms\": round(float(np.percentile(ts, 50)), 1),\n",
+ " \"p95_ms\": round(float(np.percentile(ts, 95)), 1),\n",
+ " \"ms_per_question\": round(float(np.percentile(ts, 50))/nq, 2)}\n",
+ " RESULTS[\"latency\"][model_name] = res\n",
+ " print(model_name, json.dumps(res))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 8. Option-order robustness\n",
+ "\n",
+ "Same question, options permuted. A well-behaved decision model should pick the same label.\n",
+ "The independent Jev benchmark measured a **13% flip rate for Jev**, with the worst LLM at 37%."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import random, numpy as np\n",
+ "import bench_engine as BE\n",
+ "\n",
+ "PB_SUITES = [s for s in (\"massive_intent.en\", \"en.emotion\", \"xnli.en\") if s in SUITES]\n",
+ "RESULTS[\"option_order_robustness\"] = {}\n",
+ "\n",
+ "for model_name in REPOS:\n",
+ " agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",
+ " out = {}\n",
+ " for sname in PB_SUITES:\n",
+ " S = SUITES[sname]\n",
+ " base = S[\"cases\"][:200]\n",
+ " rng = random.Random(99)\n",
+ " perm_cases, perm_maps = [], []\n",
+ " for state, questions in base:\n",
+ " qid = list(questions)[0]\n",
+ " qdef = questions[qid]\n",
+ " keys = list(qdef[\"criteria\"].keys())\n",
+ " order = list(range(len(keys))); rng.shuffle(order)\n",
+ " newcrit = {keys[i]: qdef[\"criteria\"][keys[i]] for i in order}\n",
+ " perm_cases.append((state, {qid: {**qdef, \"criteria\": newcrit}}))\n",
+ " perm_maps.append((keys, [keys[i] for i in order]))\n",
+ " la, ia, _, _ = BE.score_cases(agent, base, progress=\"\")\n",
+ " lb, ib, _, _ = BE.score_cases(agent, perm_cases, progress=\"\")\n",
+ " flips = comp = 0\n",
+ " for (za, (_, _, _, ka)), (zb, (_, _, _, kb)), (keys, pkeys) in zip(zip(la, ia), zip(lb, ib), perm_maps):\n",
+ " if za is None or zb is None:\n",
+ " continue\n",
+ " comp += 1\n",
+ " if keys[int(np.argmax(za))] != pkeys[int(np.argmax(zb))]:\n",
+ " flips += 1\n",
+ " out[sname] = {\"n\": comp, \"flip_rate\": round(flips/max(1, comp), 4)}\n",
+ " print(\" %-22s %-24s flip rate %.3f (n=%d)\" % (model_name, sname, out[sname][\"flip_rate\"], comp))\n",
+ " RESULTS[\"option_order_robustness\"][model_name] = out"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 9. Calibration repair\n",
+ "\n",
+ "`laya-multilingual` ships with `temperature = [1.0, 1.0, 1.0]` and no per-option-count buckets \u2014\n",
+ "it was never calibrated. This refits one temperature per (question type, option-count bucket) on\n",
+ "**half** of each suite and reports ECE on the other half, so the improvement is measured\n",
+ "out-of-sample. It shows how much of any calibration gap is a missing post-processing step rather\n",
+ "than a property of the model."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import bench_engine as BE\n",
+ "from laya.common import temp_bucket\n",
+ "\n",
+ "RESULTS[\"calibration_repair\"] = {}\n",
+ "for model_name in REPOS:\n",
+ " agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",
+ " per_suite, fitted_all = {}, {}\n",
+ " for sname, S in SUITES.items():\n",
+ " if sname not in RAW[model_name]:\n",
+ " continue\n",
+ " logits, index = RAW[model_name][sname]\n",
+ " pairs = []\n",
+ " for (ci, qid, qt, k), z in zip(index, logits):\n",
+ " g = S[\"gold\"][ci].get(qid)\n",
+ " if g is not None and z is not None:\n",
+ " pairs.append((z, g[\"idx\"], qt, k))\n",
+ " if len(pairs) < 60:\n",
+ " continue\n",
+ " half = len(pairs)//2\n",
+ " fit_set, held = pairs[:half], pairs[half:]\n",
+ " buckets = {}\n",
+ " for z, gi, qt, k in fit_set:\n",
+ " buckets.setdefault(temp_bucket(qt, k), []).append((z, gi))\n",
+ " fitted = {b: BE.fit_temperature(v) for b, v in buckets.items() if len(v) >= 25}\n",
+ " rows_ship, rows_fit = [], []\n",
+ " for z, gi, qt, k in held:\n",
+ " rows_ship.append((gi, BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))\n",
+ " rows_fit.append((gi, BE.softmax_t(z, fitted.get(temp_bucket(qt, k),\n",
+ " BE.temp_for(agent, qt, k, True)))))\n",
+ " ms, mf = BE.hard_metrics(rows_ship), BE.hard_metrics(rows_fit)\n",
+ " per_suite[sname] = {\"n_heldout\": ms[\"n\"], \"fitted_temperatures\": fitted,\n",
+ " \"ece_shipped\": ms[\"ece\"], \"ece_refit\": mf[\"ece\"],\n",
+ " \"nll_shipped\": ms[\"nll\"], \"nll_refit\": mf[\"nll\"],\n",
+ " \"accuracy\": ms[\"accuracy\"]}\n",
+ " fitted_all.update(fitted)\n",
+ " RESULTS[\"calibration_repair\"][model_name] = {\"per_suite\": per_suite}\n",
+ " if per_suite:\n",
+ " a = float(np.mean([v[\"ece_shipped\"] for v in per_suite.values()]))\n",
+ " b = float(np.mean([v[\"ece_refit\"] for v in per_suite.values()]))\n",
+ " RESULTS[\"calibration_repair\"][model_name][\"mean_ece_shipped\"] = round(a, 4)\n",
+ " RESULTS[\"calibration_repair\"][model_name][\"mean_ece_refit\"] = round(b, 4)\n",
+ " print(\"%-22s mean ECE shipped %.4f -> refit %.4f\" % (model_name, a, b))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 10. Aggregate, save and download"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np, json\n",
+ "\n",
+ "def agg(prefix_or_family, key=\"family\"):\n",
+ " out = {}\n",
+ " for m in REPOS:\n",
+ " accs, eces, ns = [], [], 0\n",
+ " for sname, S in SUITES.items():\n",
+ " if S[\"meta\"].get(key) != prefix_or_family:\n",
+ " continue\n",
+ " r = RESULTS[\"suites\"].get(sname, {}).get(m, {})\n",
+ " c = r.get(\"calibrated\", {})\n",
+ " if c.get(\"n\"):\n",
+ " accs.append(c[\"accuracy\"]); eces.append(c[\"ece\"]); ns += c[\"n\"]\n",
+ " if accs:\n",
+ " out[m] = {\"macro_accuracy\": round(float(np.mean(accs)), 4),\n",
+ " \"macro_ece\": round(float(np.mean(eces)), 4),\n",
+ " \"n_questions\": ns, \"n_suites\": len(accs)}\n",
+ " return out\n",
+ "\n",
+ "RESULTS[\"caveats\"] = {\n",
+ " \"context_window\": \"laya max_len=512, laya-multilingual max_len=1024 (shipped configs). \"\n",
+ " \"See typed_decisions_matched_512 for the equal-budget control.\",\n",
+ " \"shipped_calibration\": \"laya ships fitted temperatures; laya-multilingual ships all-1.0 \"\n",
+ " \"(uncalibrated). See calibration_repair for the out-of-sample refit.\",\n",
+ " \"training_overlap\": \"ag_news and boolq were in Laya's training mix (retention, not generalisation). \"\n",
+ " \"sst5, emotion, prompt_injections, banking77 were held out. \"\n",
+ " \"Neither checkpoint trained on typed-decisions, MASSIVE or XNLI.\",\n",
+ "}\n",
+ "\n",
+ "RESULTS[\"summary\"] = {\n",
+ " \"typed_decisions\": {m: RESULTS[\"suites\"].get(\"typed_decisions\", {}).get(m, {}).get(\"calibrated\", {})\n",
+ " for m in REPOS},\n",
+ " \"massive_intent\": agg(\"massive_intent\"),\n",
+ " \"massive_scenario\": agg(\"massive_scenario\"),\n",
+ " \"xnli\": agg(\"xnli\"),\n",
+ " \"english\": agg(\"english\"),\n",
+ "}\n",
+ "\n",
+ "# English vs non-English split inside each multilingual family\n",
+ "for fam in (\"massive_intent\", \"massive_scenario\", \"xnli\"):\n",
+ " en, non = {}, {}\n",
+ " for m in REPOS:\n",
+ " ea, na = [], []\n",
+ " for sname, S in SUITES.items():\n",
+ " if S[\"meta\"].get(\"family\") != fam:\n",
+ " continue\n",
+ " c = RESULTS[\"suites\"].get(sname, {}).get(m, {}).get(\"calibrated\", {})\n",
+ " if not c.get(\"n\"):\n",
+ " continue\n",
+ " (ea if S[\"meta\"].get(\"lang\") in (\"en\",) else na).append(c[\"accuracy\"])\n",
+ " if ea: en[m] = round(float(np.mean(ea)), 4)\n",
+ " if na: non[m] = round(float(np.mean(na)), 4)\n",
+ " RESULTS[\"summary\"][fam + \"_english_vs_rest\"] = {\"english\": en, \"non_english\": non}\n",
+ "\n",
+ "json.dump(RESULTS, open(\"laya_benchmark_results.json\", \"w\"), indent=2)\n",
+ "print(json.dumps(RESULTS[\"summary\"], indent=2))\n",
+ "print(\"\\nwrote laya_benchmark_results.json (%.1f KB)\" % (os.path.getsize(\"laya_benchmark_results.json\")/1024))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Per-language table, printed for a quick eyeball before you download the JSON\n",
+ "import numpy as np\n",
+ "for fam in (\"massive_intent\", \"massive_scenario\", \"xnli\"):\n",
+ " langs = sorted({S[\"meta\"][\"lang\"] for S in SUITES.values() if S[\"meta\"].get(\"family\") == fam})\n",
+ " if not langs:\n",
+ " continue\n",
+ " print(\"\\n=== %s : accuracy by language ===\" % fam)\n",
+ " print(\"%-8s %12s %22s %10s\" % (\"lang\", \"laya\", \"laya-multilingual\", \"delta\"))\n",
+ " for lg in langs:\n",
+ " sname = \"%s.%s\" % (fam, lg)\n",
+ " a = RESULTS[\"suites\"].get(sname, {}).get(\"laya\", {}).get(\"calibrated\", {}).get(\"accuracy\")\n",
+ " b = RESULTS[\"suites\"].get(sname, {}).get(\"laya-multilingual\", {}).get(\"calibrated\", {}).get(\"accuracy\")\n",
+ " if a is None or b is None:\n",
+ " continue\n",
+ " print(\"%-8s %12.3f %22.3f %+10.3f\" % (lg, a, b, b-a))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "try:\n",
+ " from google.colab import files\n",
+ " files.download(\"laya_benchmark_results.json\")\n",
+ "except Exception as e:\n",
+ " print(\"Not on Colab or download blocked (%s). Grab laya_benchmark_results.json from the file browser.\" % type(e).__name__)"
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "provenance": [],
+ "gpuType": "T4",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
diff --git a/research/scripts/make_plots.py b/research/scripts/make_plots.py
new file mode 100644
index 00000000..08a9e857
--- /dev/null
+++ b/research/scripts/make_plots.py
@@ -0,0 +1,154 @@
+"""Render the benchmark figure used in the GitHub and Hugging Face READMEs.
+
+ USE_TF=0 python3 notebooks/make_plots.py
+
+Reads local_benchmark_results.json (51-language sweep) and laya_benchmark_results.json
+(T4 latency + English suites); writes assets/laya_benchmark.png.
+"""
+import json
+import os
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt # noqa: E402
+import numpy as np # noqa: E402
+
+REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ASSETS = os.path.join(REPO, "assets")
+os.makedirs(ASSETS, exist_ok=True)
+
+# Validated categorical slots 1 and 2 (light mode) + ink tokens.
+BLUE, ORANGE = "#2a78d6", "#eb6834"
+SURFACE = "#fcfcfb"
+INK, INK2, INK3 = "#0b0b0b", "#52514e", "#8a8984"
+GRID = "#e4e3df"
+
+plt.rcParams.update({
+ "figure.facecolor": SURFACE, "axes.facecolor": SURFACE, "savefig.facecolor": SURFACE,
+ "font.family": "DejaVu Sans", "text.color": INK,
+ "axes.labelcolor": INK2, "xtick.color": INK2, "ytick.color": INK2,
+ "axes.edgecolor": GRID, "axes.linewidth": 1.0,
+ "xtick.major.size": 0, "ytick.major.size": 0,
+})
+
+
+def strip(ax, keep=("bottom",)):
+ for s in ("top", "right", "left", "bottom"):
+ ax.spines[s].set_visible(s in keep)
+
+
+def main():
+ loc = json.load(open(os.path.join(REPO, "local_benchmark_results.json")))
+ col = json.load(open(os.path.join(REPO, "laya_benchmark_results.json")))
+
+ en = loc["part_a"]["by_model"]["english"]["per_language"]
+ ml = loc["part_a"]["by_model"].get("multilingual", {}).get("per_language", {})
+ langs = sorted(set(en) & set(ml), key=lambda l: (ml[l]["accuracy"] - en[l]["accuracy"]))
+ if not langs:
+ raise SystemExit("multilingual sweep not finished yet")
+
+ fig = plt.figure(figsize=(16, max(9, 0.23 * len(langs) + 3.2)))
+ gs = fig.add_gridspec(3, 2, width_ratios=[1.35, 1], height_ratios=[1, 1, 1],
+ wspace=0.28, hspace=0.55)
+
+ # ---------------------------------------------------------------- A. dumbbell
+ axA = fig.add_subplot(gs[:, 0])
+ y = np.arange(len(langs))
+ e = np.array([en[l]["accuracy"] for l in langs])
+ m = np.array([ml[l]["accuracy"] for l in langs])
+ axA.hlines(y, e, m, color=GRID, lw=2.4, zorder=1)
+ axA.scatter(e, y, s=62, color=BLUE, zorder=3, label="laya (English checkpoint)",
+ edgecolors=SURFACE, linewidths=1.6)
+ axA.scatter(m, y, s=62, color=ORANGE, zorder=3, label="laya-multilingual",
+ edgecolors=SURFACE, linewidths=1.6)
+ axA.axvline(0.05, color=INK3, lw=1.4, ls=(0, (4, 3)), zorder=2)
+ axA.text(0.052, len(langs) - 0.4, "random guess (0.050)", color=INK3, fontsize=9, va="top")
+ axA.set_yticks(y); axA.set_yticklabels(langs, fontsize=9)
+ axA.set_ylim(-0.8, len(langs) - 0.2)
+ axA.set_xlim(-0.02, max(0.9, float(max(e.max(), m.max())) + 0.06))
+ axA.set_xlabel("accuracy · MASSIVE intent, 20 options", fontsize=10)
+ axA.set_title("Every language: which checkpoint can read it",
+ fontsize=14, fontweight="bold", color=INK, loc="left", pad=12)
+ axA.xaxis.grid(True, color=GRID, lw=0.9); axA.set_axisbelow(True)
+ strip(axA)
+ axA.legend(frameon=False, loc="lower right", fontsize=10, labelcolor=INK2)
+
+ # ---------------------------------------------------------------- B. latency
+ axB = fig.add_subplot(gs[0, 1])
+ lat = col["latency"]
+ ns = ["1_questions", "5_questions", "10_questions", "50_questions"]
+ lab = ["1", "5", "10", "50"]
+ a = [lat["laya"][n]["p50_ms"] for n in ns]
+ b = [lat["laya-multilingual"][n]["p50_ms"] for n in ns]
+ x = np.arange(len(ns)); w = 0.36
+ axB.bar(x - w / 2 - 0.01, a, w, color=BLUE, label="laya", zorder=3)
+ axB.bar(x + w / 2 + 0.01, b, w, color=ORANGE, label="laya-multilingual", zorder=3)
+ for xi, v in zip(x - w / 2 - 0.01, a):
+ axB.text(xi, v + 18, "%.0f" % v, ha="center", fontsize=9, color=INK2)
+ for xi, v in zip(x + w / 2 + 0.01, b):
+ axB.text(xi, v + 18, "%.0f" % v, ha="center", fontsize=9, color=INK2)
+ axB.axhline(256, color=INK3, lw=1.4, ls=(0, (4, 3)), zorder=2)
+ axB.text(3.42, 272, "Jev, measured\n236–276 ms", color=INK3, fontsize=8.5, ha="right")
+ axB.set_xticks(x); axB.set_xticklabels(lab)
+ axB.set_xlabel("questions per call", fontsize=10)
+ axB.set_ylabel("p50 latency (ms)", fontsize=10)
+ axB.set_ylim(0, max(a + b) * 1.18)
+ axB.set_title("Speed on one T4", fontsize=13, fontweight="bold", color=INK, loc="left", pad=10)
+ axB.yaxis.grid(True, color=GRID, lw=0.9); axB.set_axisbelow(True)
+ strip(axB); axB.legend(frameon=False, fontsize=9.5, loc="upper left", labelcolor=INK2)
+
+ # ---------------------------------------------------------------- C. vs Jev
+ axC = fig.add_subplot(gs[1, 1])
+ names = ["AG News\n(4 labels)", "DAIR Emotion\n(6 labels)", "typed-decisions\n(2,000 dec.)"]
+ jev = [0.910, 0.480, 0.727]
+ lay = [col["suites"]["en.ag_news"]["laya"]["calibrated"]["accuracy"],
+ col["suites"]["en.emotion"]["laya"]["calibrated"]["accuracy"], 0.766]
+ x = np.arange(3)
+ axC.bar(x - w / 2 - 0.01, jev, w, color=INK3, label="Jev (published)", zorder=3)
+ axC.bar(x + w / 2 + 0.01, lay, w, color=BLUE, label="Laya (measured)", zorder=3)
+ for xi, v in zip(x - w / 2 - 0.01, jev):
+ axC.text(xi, v + .02, "%.3f" % v, ha="center", fontsize=9, color=INK2)
+ for xi, v in zip(x + w / 2 + 0.01, lay):
+ axC.text(xi, v + .02, "%.3f" % v, ha="center", fontsize=9, color=INK2)
+ axC.set_xticks(x); axC.set_xticklabels(names, fontsize=9)
+ axC.set_ylim(0, 1.06); axC.set_ylabel("accuracy", fontsize=10)
+ axC.set_title("Against Jev, same public datasets", fontsize=13, fontweight="bold",
+ color=INK, loc="left", pad=10)
+ axC.yaxis.grid(True, color=GRID, lw=0.9); axC.set_axisbelow(True)
+ strip(axC); axC.legend(frameon=False, fontsize=9.5, loc="lower left", labelcolor=INK2)
+ axC.text(0, -0.34, "typed-decisions uses the fine-tuned checkpoint. Jev figures are "
+ "third-party published, not measured here.",
+ transform=axC.transAxes, fontsize=8, color=INK3)
+
+ # ---------------------------------------------------------------- D. calibration
+ axD = fig.add_subplot(gs[2, 1])
+ rep = col["calibration_repair"]
+ models = ["laya", "laya-multilingual"]
+ ship = [rep[m]["mean_ece_shipped"] for m in models]
+ refit = [rep[m]["mean_ece_refit"] for m in models]
+ x = np.arange(2)
+ axD.bar(x - w / 2 - 0.01, ship, w, color=ORANGE, label="as shipped", zorder=3)
+ axD.bar(x + w / 2 + 0.01, refit, w, color=BLUE, label="temperature refit", zorder=3)
+ for xi, v in zip(x - w / 2 - 0.01, ship):
+ axD.text(xi, v + .012, "%.3f" % v, ha="center", fontsize=9, color=INK2)
+ for xi, v in zip(x + w / 2 + 0.01, refit):
+ axD.text(xi, v + .012, "%.3f" % v, ha="center", fontsize=9, color=INK2)
+ axD.axhline(0.246, color=INK3, lw=1.4, ls=(0, (4, 3)), zorder=2)
+ axD.text(1.45, 0.253, "Jev ECE 0.246", color=INK3, fontsize=8.5, ha="right")
+ axD.set_xticks(x); axD.set_xticklabels(models, fontsize=9.5)
+ axD.set_ylabel("mean ECE (lower is better)", fontsize=10)
+ axD.set_ylim(0, max(ship) * 1.25)
+ axD.set_title("Calibration: fit your temperatures", fontsize=13, fontweight="bold",
+ color=INK, loc="left", pad=10)
+ axD.yaxis.grid(True, color=GRID, lw=0.9); axD.set_axisbelow(True)
+ strip(axD); axD.legend(frameon=False, fontsize=9.5, loc="upper right", labelcolor=INK2)
+
+ fig.suptitle("Laya benchmark · three checkpoints, identical questions, one T4",
+ fontsize=16, fontweight="bold", color=INK, x=0.012, ha="left", y=0.985)
+ out = os.path.join(ASSETS, "laya_benchmark.png")
+ fig.savefig(out, dpi=150, bbox_inches="tight")
+ print("wrote", out, "(%.0f KB)" % (os.path.getsize(out) / 1024))
+
+
+if __name__ == "__main__":
+ main()