Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
research: benchmark harnesses and raw results (#14)
Nandakishoree760389dc69
diff --git a/research/README.md b/research/README.mdnew file mode 100644index 00000000..7abd50c8--- /dev/null+++ b/research/README.md@@ -0,0 +1,60 @@+# Research++Benchmark harnesses and raw results for the Laya checkpoints. This branch is the evidence behind+the numbers quoted in the main README — nothing here is imported by the `laya` package.++## Scripts++| file | what it does |+|---|---|+| `scripts/laya_benchmark_colab.ipynb` | the full head-to-head on a Colab T4: typed-decisions, MASSIVE intent + scenario (14 languages), XNLI (15), English suites, latency, option-order robustness, calibration repair. Writes one JSON. |+| `scripts/build_benchmark_nb.py` | generator for that notebook (edit here, not the `.ipynb`) |+| `scripts/bench_local.py` | CPU sweep: MASSIVE intent across **all 51 languages**, plus typed-decisions on all three checkpoints |+| `scripts/bench_apps.py` | the six application workflows (support triage, email + phishing, guardrails, RAG relevance, moderation, model routing) plus the datasets where public Jev numbers exist |+| `scripts/bench_latency.py` | inference speed including what routing costs: detection overhead, hot path, cold-swap, mixed-language throughput at several `max_loaded` settings |+| `scripts/make_plots.py` | renders `assets/laya_benchmark.png` from the result JSONs |++Everything runs with `USE_TF=0` — `transformers` probes for TensorFlow at import, and when TF is+installed its abseil runtime can deadlock model construction on macOS/Python 3.9.++## Results++| file | contents |+|---|---|+| `results/t4_colab_benchmark.json` | 17,416 questions on one T4, both checkpoints, identical questions per model |+| `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options |++## Headline findings++**Routing takes Laya from 23 to 45 of 51 languages.** On MASSIVE intent (20 options, random =+0.050) the English checkpoint macro-averages 0.227 and clears 3x random on 23 of 51 languages;+the multilingual checkpoint reaches 0.366 and clears it on 45.++**The English checkpoint's confidence gives no warning when it cannot read the input.** Khmer:+0.000 accuracy at 0.952 mean confidence. Macro ECE 0.733 across 51 languages, with mean+confidence never dropping below 0.885 at any accuracy level. This is why routing has to happen+*before* the forward pass — confidence gating cannot catch it.++**Both checkpoints ship over-confident.** Refitting one temperature per (question type, option+count) on held-out data moves mean ECE 0.466 -> 0.081 (`laya`) and 0.314 -> 0.106+(`laya-multilingual`, which ships with no fitted temperatures at all).++**The base checkpoints are near chance on typed-decisions zero-shot** — 0.362 and 0.352 against+a 0.318 random baseline and a 0.461 majority-class baseline. The published 0.766 belongs to the+checkpoint fine-tuned on that benchmark's own training split.++**Speed.** 32.8 ms for one question and 7.2 ms/question at batch 10 on a T4; 103–332 questions/s+batched.++## On comparisons with Jev++There is no TypeSafe API credential in this project, so **Jev was never run here**. Every Jev+figure quoted is third-party published, with different sample sizes and prompts:++- [AbdelStark/jev-benchmarks](https://github.com/AbdelStark/jev-benchmarks) — AG News 0.910,+ Banking77 0.870, DAIR Emotion 0.480 (Brier 0.846, NLL 5.588, zero probability on the true label+ for 16% of examples)+- [nibzard/decision-model-benchmark](https://github.com/nibzard/decision-model-benchmark) — ECE+ 0.246 (worst in that study), banking77 0.763, option-order flip rate 13%, latency 264–276 ms p50++Treat those as indicative, not a controlled head-to-head.diff --git a/research/results/cpu_51_language_sweep.json b/research/results/cpu_51_language_sweep.jsonnew file mode 100644index 00000000..1cb7d5ed--- /dev/null+++ b/research/results/cpu_51_language_sweep.json@@ -0,0 +1,1430 @@+{+ "meta": {+ "timestamp": "2026-09-19 11:04:45",+ "device": "cpu",+ "torch": "2.8.0",+ "laya": "0.2.0",+ "threads": 4+ },+ "part_a": {+ "config": {+ "languages": [+ "af",+ "am",+ "ar",+ "az",+ "bn",+ "cy",+ "da",+ "de",+ "el",+ "en",+ "es",+ "fa",+ "fi",+ "fr",+ "he",+ "hi",+ "hu",+ "hy",+ "id",+ "is",+ "it",+ "ja",+ "jv",+ "ka",+ "km",+ "kn",+ "ko",+ "lv",+ "ml",+ "mn",+ "ms",+ "my",+ "nb",+ "nl",+ "pl",+ "pt",+ "ro",+ "ru",+ "sl",+ "sq",+ "sv",+ "sw",+ "ta",+ "te",+ "th",+ "tl",+ "tr",+ "ur",+ "vi",+ "zh-CN",+ "zh-TW"+ ],+ "per_lang": 100,+ "n_options": 20,+ "seed": 13+ },+ "by_model": {+ "english": {+ "per_language": {+ "af": {+ "n": 100,+ "accuracy": 0.29,+ "macro_f1": 0.2874,+ "ece": 0.6873,+ "brier": 1.3787,+ "nll": 16.3516,+ "mean_confidence": 0.9752,+ "acc_at_50_coverage": 0.42,+ "seconds": 11.6,+ "dropped": 0+ },+ "am": {+ "n": 100,+ "accuracy": 0.12,+ "macro_f1": 0.1122,+ "ece": 0.8252,+ "brier": 1.6644,+ "nll": 21.6026,+ "mean_confidence": 0.9452,+ "acc_at_50_coverage": 0.14,+ "seconds": 14.6,+ "dropped": 0+ },+ "ar": {+ "n": 100,+ "accuracy": 0.11,+ "macro_f1": 0.093,+ "ece": 0.8001,+ "brier": 1.6539,+ "nll": 20.5087,+ "mean_confidence": 0.8976,+ "acc_at_50_coverage": 0.12,+ "seconds": 11.8,+ "dropped": 0+ },+ "az": {+ "n": 100,+ "accuracy": 0.1,+ "macro_f1": 0.0629,+ "ece": 0.8247,+ "brier": 1.6966,+ "nll": 21.7232,+ "mean_confidence": 0.9247,+ "acc_at_50_coverage": 0.14,+ "seconds": 11.9,+ "dropped": 0+ },+ "bn": {+ "n": 100,+ "accuracy": 0.08,+ "macro_f1": 0.056,+ "ece": 0.8646,+ "brier": 1.7373,+ "nll": 20.8691,+ "mean_confidence": 0.9446,+ "acc_at_50_coverage": 0.1,+ "seconds": 13.9,+ "dropped": 0+ },+ "cy": {+ "n": 100,+ "accuracy": 0.12,+ "macro_f1": 0.0906,+ "ece": 0.8409,+ "brier": 1.7029,+ "nll": 21.2255,+ "mean_confidence": 0.9609,+ "acc_at_50_coverage": 0.14,+ "seconds": 11.8,+ "dropped": 0+ },+ "da": {+ "n": 100,+ "accuracy": 0.35,+ "macro_f1": 0.3082,+ "ece": 0.6263,+ "brier": 1.2633,+ "nll": 16.2409,+ "mean_confidence": 0.9763,+ "acc_at_50_coverage": 0.52,+ "seconds": 12.0,+ "dropped": 0+ },+ "de": {+ "n": 100,+ "accuracy": 0.42,+ "macro_f1": 0.387,+ "ece": 0.5584,+ "brier": 1.126,+ "nll": 13.7538,+ "mean_confidence": 0.9784,+ "acc_at_50_coverage": 0.66,+ "seconds": 14.1,+ "dropped": 0+ },+ "el": {+ "n": 100,+ "accuracy": 0.13,+ "macro_f1": 0.1,+ "ece": 0.8387,+ "brier": 1.6854,+ "nll": 21.1672,+ "mean_confidence": 0.9687,+ "acc_at_50_coverage": 0.16,+ "seconds": 15.1,+ "dropped": 0+ },+ "en": {+ "n": 100,+ "accuracy": 0.82,+ "macro_f1": 0.7876,+ "ece": 0.1789,+ "brier": 0.3563,+ "nll": 3.5193,+ "mean_confidence": 0.9989,+ "acc_at_50_coverage": 0.94,+ "seconds": 16.1,+ "dropped": 0+ },+ "es": {+ "n": 100,+ "accuracy": 0.51,+ "macro_f1": 0.4767,+ "ece": 0.4796,+ "brier": 0.9706,+ "nll": 11.846,+ "mean_confidence": 0.9896,+ "acc_at_50_coverage": 0.6,+ "seconds": 20.3,+ "dropped": 0+ },+ "fa": {+ "n": 100,+ "accuracy": 0.14,+ "macro_f1": 0.1197,+ "ece": 0.8196,+ "brier": 1.6664,+ "nll": 21.65,+ "mean_confidence": 0.9414,+ "acc_at_50_coverage": 0.16,+ "seconds": 24.5,+ "dropped": 0+ },+ "fi": {+ "n": 100,+ "accuracy": 0.13,+ "macro_f1": 0.1213,+ "ece": 0.8486,+ "brier": 1.7071,+ "nll": 21.923,+ "mean_confidence": 0.9786,+ "acc_at_50_coverage": 0.16,+ "seconds": 20.1,+ "dropped": 0+ },+ "fr": {+ "n": 100,+ "accuracy": 0.59,+ "macro_f1": 0.5822,+ "ece": 0.3882,+ "brier": 0.7935,+ "nll": 9.1132,+ "mean_confidence": 0.9782,+ "acc_at_50_coverage": 0.78,+ "seconds": 19.4,+ "dropped": 0+ },+ "he": {+ "n": 100,+ "accuracy": 0.06,+ "macro_f1": 0.039,+ "ece": 0.9111,+ "brier": 1.8321,+ "nll": 24.2157,+ "mean_confidence": 0.9643,+ "acc_at_50_coverage": 0.04,+ "seconds": 20.1,+ "dropped": 0+ },+ "hi": {+ "n": 100,+ "accuracy": 0.1,+ "macro_f1": 0.0758,+ "ece": 0.8504,+ "brier": 1.7169,+ "nll": 21.3541,+ "mean_confidence": 0.9408,+ "acc_at_50_coverage": 0.1,+ "seconds": 21.1,+ "dropped": 0+ },+ "hu": {+ "n": 100,+ "accuracy": 0.09,+ "macro_f1": 0.0813,+ "ece": 0.8571,+ "brier": 1.7503,+ "nll": 23.2612,+ "mean_confidence": 0.9471,+ "acc_at_50_coverage": 0.06,+ "seconds": 18.9,+ "dropped": 0+ },+ "hy": {+ "n": 100,+ "accuracy": 0.05,+ "macro_f1": 0.0355,+ "ece": 0.8348,+ "brier": 1.7324,+ "nll": 20.695,+ "mean_confidence": 0.8848,+ "acc_at_50_coverage": 0.08,+ "seconds": 22.5,+ "dropped": 0+ },+ "id": {+ "n": 100,+ "accuracy": 0.36,+ "macro_f1": 0.3283,+ "ece": 0.6133,+ "brier": 1.2484,+ "nll": 14.6851,+ "mean_confidence": 0.9619,+ "acc_at_50_coverage": 0.48,+ "seconds": 18.7,+ "dropped": 0+ },+ "is": {+ "n": 100,+ "accuracy": 0.11,+ "macro_f1": 0.0893,+ "ece": 0.8351,+ "brier": 1.6975,+ "nll": 21.9127,+ "mean_confidence": 0.9451,+ "acc_at_50_coverage": 0.16,+ "seconds": 19.2,+ "dropped": 0+ },+ "it": {+ "n": 100,+ "accuracy": 0.34,+ "macro_f1": 0.3101,+ "ece": 0.6474,+ "brier": 1.2866,+ "nll": 15.5383,+ "mean_confidence": 0.9735,+ "acc_at_50_coverage": 0.52,+ "seconds": 19.2,+ "dropped": 0+ },+ "ja": {+ "n": 100,+ "accuracy": 0.53,+ "macro_f1": 0.4942,+ "ece": 0.4599,+ "brier": 0.9247,+ "nll": 10.8681,+ "mean_confidence": 0.9801,+ "acc_at_50_coverage": 0.76,+ "seconds": 19.9,+ "dropped": 0+ },+ "jv": {+ "n": 100,+ "accuracy": 0.16,+ "macro_f1": 0.1537,+ "ece": 0.8035,+ "brier": 1.6288,+ "nll": 20.6859,+ "mean_confidence": 0.9635,+ "acc_at_50_coverage": 0.14,+ "seconds": 19.2,+ "dropped": 0+ },+ "ka": {+ "n": 100,+ "accuracy": 0.09,+ "macro_f1": 0.0742,+ "ece": 0.8448,+ "brier": 1.72,+ "nll": 21.2473,+ "mean_confidence": 0.9348,+ "acc_at_50_coverage": 0.1,+ "seconds": 22.0,+ "dropped": 0+ },+ "km": {+ "n": 100,+ "accuracy": 0.0,+ "macro_f1": 0.0,+ "ece": 0.9515,+ "brier": 1.9219,+ "nll": 23.2123,+ "mean_confidence": 0.9515,+ "acc_at_50_coverage": 0.0,+ "seconds": 25.4,+ "dropped": 0+ },+ "kn": {+ "n": 100,+ "accuracy": 0.11,+ "macro_f1": 0.0913,+ "ece": 0.8416,+ "brier": 1.7051,+ "nll": 20.4279,+ "mean_confidence": 0.9421,+ "acc_at_50_coverage": 0.06,+ "seconds": 24.3,+ "dropped": 0+ },+ "ko": {+ "n": 100,+ "accuracy": 0.11,+ "macro_f1": 0.0943,+ "ece": 0.8498,+ "brier": 1.7193,+ "nll": 22.1195,+ "mean_confidence": 0.9598,+ "acc_at_50_coverage": 0.14,+ "seconds": 20.1,+ "dropped": 0+ },+ "lv": {+ "n": 100,+ "accuracy": 0.1,+ "macro_f1": 0.0786,+ "ece": 0.8466,+ "brier": 1.7322,+ "nll": 21.4978,+ "mean_confidence": 0.9466,+ "acc_at_50_coverage": 0.1,+ "seconds": 19.4,+ "dropped": 0+ },+ "ml": {+ "n": 100,+ "accuracy": 0.07,+ "macro_f1": 0.0503,+ "ece": 0.8572,+ "brier": 1.7485,+ "nll": 20.0485,+ "mean_confidence": 0.9272,+ "acc_at_50_coverage": 0.1,+ "seconds": 24.6,+ "dropped": 0+ },+ "mn": {+ "n": 100,+ "accuracy": 0.13,+ "macro_f1": 0.1213,+ "ece": 0.837,+ "brier": 1.7109,+ "nll": 21.5421,+ "mean_confidence": 0.9591,+ "acc_at_50_coverage": 0.14,+ "seconds": 20.6,+ "dropped": 0+ },+ "ms": {+ "n": 100,+ "accuracy": 0.27,+ "macro_f1": 0.2372,+ "ece": 0.6877,+ "brier": 1.4027,+ "nll": 16.277,+ "mean_confidence": 0.9577,+ "acc_at_50_coverage": 0.42,+ "seconds": 19.2,+ "dropped": 0+ },+ "my": {+ "n": 100,+ "accuracy": 0.06,+ "macro_f1": 0.0455,+ "ece": 0.8605,+ "brier": 1.7635,+ "nll": 19.7847,+ "mean_confidence": 0.9205,+ "acc_at_50_coverage": 0.08,+ "seconds": 24.7,+ "dropped": 0+ },+ "nb": {+ "n": 100,+ "accuracy": 0.33,+ "macro_f1": 0.3025,+ "ece": 0.6484,+ "brier": 1.302,+ "nll": 16.4489,+ "mean_confidence": 0.9692,+ "acc_at_50_coverage": 0.5,+ "seconds": 19.2,+ "dropped": 0+ },+ "nl": {+ "n": 100,+ "accuracy": 0.39,+ "macro_f1": 0.3968,+ "ece": 0.5914,+ "brier": 1.1943,+ "nll": 14.4159,+ "mean_confidence": 0.9745,+ "acc_at_50_coverage": 0.54,+ "seconds": 20.7,+ "dropped": 0+ },+ "pl": {+ "n": 100,+ "accuracy": 0.24,+ "macro_f1": 0.2033,+ "ece": 0.7129,+ "brier": 1.4653,+ "nll": 18.4945,+ "mean_confidence": 0.9529,+ "acc_at_50_coverage": 0.34,+ "seconds": 20.8,+ "dropped": 0+ },+ "pt": {+ "n": 100,+ "accuracy": 0.47,+ "macro_f1": 0.4213,+ "ece": 0.5116,+ "brier": 1.0273,+ "nll": 12.7024,+ "mean_confidence": 0.9718,+ "acc_at_50_coverage": 0.66,+ "seconds": 20.8,+ "dropped": 0+ },+ "ro": {+ "n": 100,+ "accuracy": 0.33,+ "macro_f1": 0.2976,+ "ece": 0.6577,+ "brier": 1.3076,+ "nll": 15.9637,+ "mean_confidence": 0.9824,+ "acc_at_50_coverage": 0.48,+ "seconds": 20.0,+ "dropped": 0+ },+ "ru": {+ "n": 100,+ "accuracy": 0.31,+ "macro_f1": 0.3084,+ "ece": 0.668,+ "brier": 1.3452,+ "nll": 17.0527,+ "mean_confidence": 0.978,+ "acc_at_50_coverage": 0.42,+ "seconds": 20.5,+ "dropped": 0+ },+ "sl": {+ "n": 100,+ "accuracy": 0.2,+ "macro_f1": 0.1679,+ "ece": 0.7561,+ "brier": 1.5293,+ "nll": 19.7372,+ "mean_confidence": 0.9561,+ "acc_at_50_coverage": 0.26,+ "seconds": 20.2,+ "dropped": 0+ },+ "sq": {+ "n": 100,+ "accuracy": 0.21,+ "macro_f1": 0.1667,+ "ece": 0.7549,+ "brier": 1.5259,+ "nll": 19.5615,+ "mean_confidence": 0.9649,+ "acc_at_50_coverage": 0.28,+ "seconds": 20.5,+ "dropped": 0+ },+ "sv": {+ "n": 100,+ "accuracy": 0.38,+ "macro_f1": 0.3809,+ "ece": 0.5956,+ "brier": 1.2168,+ "nll": 14.3656,+ "mean_confidence": 0.9706,+ "acc_at_50_coverage": 0.5,+ "seconds": 19.9,+ "dropped": 0+ },+ "sw": {+ "n": 100,+ "accuracy": 0.13,+ "macro_f1": 0.1149,+ "ece": 0.8282,+ "brier": 1.6934,+ "nll": 21.2625,+ "mean_confidence": 0.9562,+ "acc_at_50_coverage": 0.14,+ "seconds": 21.2,+ "dropped": 0+ },+ "ta": {+ "n": 100,+ "accuracy": 0.12,+ "macro_f1": 0.0971,+ "ece": 0.8224,+ "brier": 1.6768,+ "nll": 19.6604,+ "mean_confidence": 0.9424,+ "acc_at_50_coverage": 0.12,+ "seconds": 25.6,+ "dropped": 0+ },+ "te": {+ "n": 100,+ "accuracy": 0.09,+ "macro_f1": 0.0722,+ "ece": 0.8575,+ "brier": 1.7384,+ "nll": 21.777,+ "mean_confidence": 0.9475,+ "acc_at_50_coverage": 0.06,+ "seconds": 25.2,+ "dropped": 0+ },+ "th": {+ "n": 100,+ "accuracy": 0.08,+ "macro_f1": 0.048,+ "ece": 0.8814,+ "brier": 1.7728,+ "nll": 22.4846,+ "mean_confidence": 0.9614,+ "acc_at_50_coverage": 0.1,+ "seconds": 22.9,+ "dropped": 0+ },+ "tl": {+ "n": 100,+ "accuracy": 0.29,+ "macro_f1": 0.249,+ "ece": 0.6756,+ "brier": 1.3666,+ "nll": 16.3214,+ "mean_confidence": 0.9656,+ "acc_at_50_coverage": 0.46,+ "seconds": 20.7,+ "dropped": 0+ },+ "tr": {+ "n": 100,+ "accuracy": 0.14,+ "macro_f1": 0.0967,+ "ece": 0.788,+ "brier": 1.6064,+ "nll": 19.6609,+ "mean_confidence": 0.928,+ "acc_at_50_coverage": 0.2,+ "seconds": 20.7,+ "dropped": 0+ },+ "ur": {+ "n": 100,+ "accuracy": 0.07,+ "macro_f1": 0.0645,+ "ece": 0.8826,+ "brier": 1.7822,+ "nll": 21.4571,+ "mean_confidence": 0.9526,+ "acc_at_50_coverage": 0.08,+ "seconds": 23.2,+ "dropped": 0+ },+ "vi": {+ "n": 100,+ "accuracy": 0.06,+ "macro_f1": 0.0498,+ "ece": 0.8914,+ "brier": 1.8055,+ "nll": 22.6885,+ "mean_confidence": 0.9514,+ "acc_at_50_coverage": 0.06,+ "seconds": 22.9,+ "dropped": 0+ },+ "zh-CN": {+ "n": 100,+ "accuracy": 0.62,+ "macro_f1": 0.6175,+ "ece": 0.376,+ "brier": 0.7521,+ "nll": 8.5938,+ "mean_confidence": 0.9885,+ "acc_at_50_coverage": 0.86,+ "seconds": 22.0,+ "dropped": 0+ },+ "zh-TW": {+ "n": 100,+ "accuracy": 0.46,+ "macro_f1": 0.4294,+ "ece": 0.5198,+ "brier": 1.0483,+ "nll": 12.3023,+ "mean_confidence": 0.9798,+ "acc_at_50_coverage": 0.74,+ "seconds": 21.1,+ "dropped": 0+ }+ },+ "macro_accuracy": 0.2269,+ "macro_ece": 0.7331,+ "n_languages": 51,+ "languages_above_random": 23+ },+ "multilingual": {+ "per_language": {+ "af": {+ "n": 100,+ "accuracy": 0.35,+ "macro_f1": 0.3133,+ "ece": 0.4845,+ "brier": 1.0466,+ "nll": 3.5729,+ "mean_confidence": 0.8028,+ "acc_at_50_coverage": 0.54,+ "seconds": 9.0,+ "dropped": 0+ },+ "am": {+ "n": 100,+ "accuracy": 0.11,+ "macro_f1": 0.1046,+ "ece": 0.4626,+ "brier": 1.2477,+ "nll": 4.265,+ "mean_confidence": 0.5706,+ "acc_at_50_coverage": 0.12,+ "seconds": 9.7,+ "dropped": 0+ },+ "ar": {+ "n": 100,+ "accuracy": 0.4,+ "macro_f1": 0.3926,+ "ece": 0.341,+ "brier": 0.8619,+ "nll": 2.697,+ "mean_confidence": 0.731,+ "acc_at_50_coverage": 0.6,+ "seconds": 8.8,+ "dropped": 0+ },+ "az": {+ "n": 100,+ "accuracy": 0.3,+ "macro_f1": 0.2693,+ "ece": 0.3685,+ "brier": 0.9758,+ "nll": 3.1016,+ "mean_confidence": 0.6685,+ "acc_at_50_coverage": 0.46,+ "seconds": 9.7,+ "dropped": 0+ },+ "bn": {+ "n": 100,+ "accuracy": 0.29,+ "macro_f1": 0.2654,+ "ece": 0.4085,+ "brier": 1.0428,+ "nll": 3.2654,+ "mean_confidence": 0.6944,+ "acc_at_50_coverage": 0.4,+ "seconds": 10.3,+ "dropped": 0+ },+ "cy": {+ "n": 100,+ "accuracy": 0.16,+ "macro_f1": 0.1432,+ "ece": 0.5905,+ "brier": 1.3315,+ "nll": 5.0927,+ "mean_confidence": 0.7149,+ "acc_at_50_coverage": 0.18,+ "seconds": 9.4,+ "dropped": 0+ },+ "da": {+ "n": 100,+ "accuracy": 0.5,+ "macro_f1": 0.4682,+ "ece": 0.2628,+ "brier": 0.7433,+ "nll": 2.0483,+ "mean_confidence": 0.7544,+ "acc_at_50_coverage": 0.68,+ "seconds": 9.3,+ "dropped": 0+ },+ "de": {+ "n": 100,+ "accuracy": 0.5,+ "macro_f1": 0.4847,+ "ece": 0.3014,+ "brier": 0.752,+ "nll": 2.4918,+ "mean_confidence": 0.784,+ "acc_at_50_coverage": 0.7,+ "seconds": 8.8,+ "dropped": 0+ },+ "el": {+ "n": 100,+ "accuracy": 0.38,+ "macro_f1": 0.3564,+ "ece": 0.3832,+ "brier": 0.9572,+ "nll": 3.235,+ "mean_confidence": 0.7507,+ "acc_at_50_coverage": 0.52,+ "seconds": 9.3,+ "dropped": 0+ },+ "en": {+ "n": 100,+ "accuracy": 0.68,+ "macro_f1": 0.6282,+ "ece": 0.209,+ "brier": 0.5052,+ "nll": 1.4497,+ "mean_confidence": 0.8719,+ "acc_at_50_coverage": 0.96,+ "seconds": 8.6,+ "dropped": 0+ },+ "es": {+ "n": 100,+ "accuracy": 0.53,+ "macro_f1": 0.4863,+ "ece": 0.2749,+ "brier": 0.6896,+ "nll": 2.0238,+ "mean_confidence": 0.8049,+ "acc_at_50_coverage": 0.76,+ "seconds": 9.0,+ "dropped": 0+ },+ "fa": {+ "n": 100,+ "accuracy": 0.39,+ "macro_f1": 0.3751,+ "ece": 0.3986,+ "brier": 0.9086,+ "nll": 2.7913,+ "mean_confidence": 0.7886,+ "acc_at_50_coverage": 0.6,+ "seconds": 9.0,+ "dropped": 0+ },+ "fi": {+ "n": 100,+ "accuracy": 0.29,+ "macro_f1": 0.2533,+ "ece": 0.4364,+ "brier": 1.0851,+ "nll": 3.7674,+ "mean_confidence": 0.7197,+ "acc_at_50_coverage": 0.38,+ "seconds": 9.2,+ "dropped": 0+ },+ "fr": {+ "n": 100,+ "accuracy": 0.54,+ "macro_f1": 0.5199,+ "ece": 0.277,+ "brier": 0.6613,+ "nll": 1.9862,+ "mean_confidence": 0.8074,+ "acc_at_50_coverage": 0.8,+ "seconds": 9.1,+ "dropped": 0+ },+ "he": {+ "n": 100,+ "accuracy": 0.4,+ "macro_f1": 0.3794,+ "ece": 0.3503,+ "brier": 0.9297,+ "nll": 2.6466,+ "mean_confidence": 0.7302,+ "acc_at_50_coverage": 0.5,+ "seconds": 8.9,+ "dropped": 0+ },+ "hi": {+ "n": 100,+ "accuracy": 0.43,+ "macro_f1": 0.397,+ "ece": 0.3206,+ "brier": 0.8304,+ "nll": 2.7293,+ "mean_confidence": 0.7506,+ "acc_at_50_coverage": 0.64,+ "seconds": 8.9,+ "dropped": 0+ },+ "hu": {+ "n": 100,+ "accuracy": 0.29,+ "macro_f1": 0.2654,+ "ece": 0.4221,+ "brier": 1.041,+ "nll": 3.5242,+ "mean_confidence": 0.7121,+ "acc_at_50_coverage": 0.44,+ "seconds": 8.6,+ "dropped": 0+ },+ "hy": {+ "n": 100,+ "accuracy": 0.15,+ "macro_f1": 0.1298,+ "ece": 0.5062,+ "brier": 1.2346,+ "nll": 4.0964,+ "mean_confidence": 0.6469,+ "acc_at_50_coverage": 0.22,+ "seconds": 9.2,+ "dropped": 0+ },+ "id": {+ "n": 100,+ "accuracy": 0.51,+ "macro_f1": 0.4955,+ "ece": 0.3048,+ "brier": 0.7362,+ "nll": 2.197,+ "mean_confidence": 0.8028,+ "acc_at_50_coverage": 0.72,+ "seconds": 8.8,+ "dropped": 0+ },+ "is": {+ "n": 100,+ "accuracy": 0.3,+ "macro_f1": 0.2831,+ "ece": 0.4692,+ "brier": 1.1123,+ "nll": 4.2693,+ "mean_confidence": 0.7551,+ "acc_at_50_coverage": 0.4,+ "seconds": 8.9,+ "dropped": 0+ },+ "it": {+ "n": 100,+ "accuracy": 0.5,+ "macro_f1": 0.4773,+ "ece": 0.302,+ "brier": 0.7537,+ "nll": 2.5765,+ "mean_confidence": 0.7825,+ "acc_at_50_coverage": 0.74,+ "seconds": 8.3,+ "dropped": 0+ },+ "ja": {+ "n": 100,+ "accuracy": 0.64,+ "macro_f1": 0.5923,+ "ece": 0.228,+ "brier": 0.5661,+ "nll": 1.5915,+ "mean_confidence": 0.8576,+ "acc_at_50_coverage": 0.84,+ "seconds": 8.1,+ "dropped": 0+ },+ "jv": {+ "n": 100,+ "accuracy": 0.27,+ "macro_f1": 0.2348,+ "ece": 0.5063,+ "brier": 1.1532,+ "nll": 3.9457,+ "mean_confidence": 0.7588,+ "acc_at_50_coverage": 0.34,+ "seconds": 8.7,+ "dropped": 0+ },+ "ka": {+ "n": 100,+ "accuracy": 0.11,+ "macro_f1": 0.1059,+ "ece": 0.5278,+ "brier": 1.3013,+ "nll": 4.4255,+ "mean_confidence": 0.6378,+ "acc_at_50_coverage": 0.14,+ "seconds": 9.2,+ "dropped": 0+ },+ "km": {+ "n": 100,+ "accuracy": 0.18,+ "macro_f1": 0.1745,+ "ece": 0.4123,+ "brier": 1.1383,+ "nll": 3.5727,+ "mean_confidence": 0.5762,+ "acc_at_50_coverage": 0.22,+ "seconds": 9.7,+ "dropped": 0+ },+ "kn": {+ "n": 100,+ "accuracy": 0.15,+ "macro_f1": 0.1388,+ "ece": 0.4371,+ "brier": 1.1512,+ "nll": 3.4801,+ "mean_confidence": 0.5804,+ "acc_at_50_coverage": 0.2,+ "seconds": 8.9,+ "dropped": 0+ },+ "ko": {+ "n": 100,+ "accuracy": 0.45,+ "macro_f1": 0.3927,+ "ece": 0.3293,+ "brier": 0.8278,+ "nll": 2.5605,+ "mean_confidence": 0.7297,+ "acc_at_50_coverage": 0.6,+ "seconds": 8.5,+ "dropped": 0+ },+ "lv": {+ "n": 100,+ "accuracy": 0.32,+ "macro_f1": 0.3023,+ "ece": 0.4804,+ "brier": 1.1257,+ "nll": 3.8958,+ "mean_confidence": 0.7779,+ "acc_at_50_coverage": 0.4,+ "seconds": 8.5,+ "dropped": 0+ },+ "ml": {+ "n": 100,+ "accuracy": 0.24,+ "macro_f1": 0.2188,+ "ece": 0.4142,+ "brier": 1.1209,+ "nll": 3.7403,+ "mean_confidence": 0.6486,+ "acc_at_50_coverage": 0.3,+ "seconds": 9.1,+ "dropped": 0+ },+ "mn": {+ "n": 100,+ "accuracy": 0.1,+ "macro_f1": 0.0735,+ "ece": 0.5581,+ "brier": 1.3302,+ "nll": 4.8227,+ "mean_confidence": 0.6562,+ "acc_at_50_coverage": 0.1,+ "seconds": 8.7,+ "dropped": 0+ },+ "ms": {+ "n": 100,+ "accuracy": 0.43,+ "macro_f1": 0.4032,+ "ece": 0.3924,+ "brier": 0.916,+ "nll": 2.9229,+ "mean_confidence": 0.8224,+ "acc_at_50_coverage": 0.62,+ "seconds": 8.4,+ "dropped": 0+ },+ "my": {+ "n": 100,+ "accuracy": 0.12,+ "macro_f1": 0.1185,+ "ece": 0.455,+ "brier": 1.2102,+ "nll": 3.8596,+ "mean_confidence": 0.575,+ "acc_at_50_coverage": 0.16,+ "seconds": 9.3,+ "dropped": 0+ },+ "nb": {+ "n": 100,+ "accuracy": 0.53,+ "macro_f1": 0.4796,+ "ece": 0.3273,+ "brier": 0.7368,+ "nll": 2.7576,+ "mean_confidence": 0.8573,+ "acc_at_50_coverage": 0.74,+ "seconds": 8.1,+ "dropped": 0+ },+ "nl": {+ "n": 100,+ "accuracy": 0.45,+ "macro_f1": 0.3861,+ "ece": 0.3775,+ "brier": 0.8735,+ "nll": 3.2134,+ "mean_confidence": 0.8137,+ "acc_at_50_coverage": 0.64,+ "seconds": 8.4,+ "dropped": 0+ },+ "pl": {+ "n": 100,+ "accuracy": 0.51,+ "macro_f1": 0.4859,+ "ece": 0.3501,+ "brier": 0.7774,+ "nll": 2.3803,+ "mean_confidence": 0.8356,+ "acc_at_50_coverage": 0.7,+ "seconds": 8.2,+ "dropped": 0+ },+ "pt": {+ "n": 100,+ "accuracy": 0.45,+ "macro_f1": 0.4108,+ "ece": 0.3419,+ "brier": 0.8515,+ "nll": 2.5313,+ "mean_confidence": 0.7863,+ "acc_at_50_coverage": 0.6,+ "seconds": 8.0,+ "dropped": 0+ },+ "ro": {+ "n": 100,+ "accuracy": 0.35,+ "macro_f1": 0.3269,+ "ece": 0.4035,+ "brier": 0.9823,+ "nll": 3.2401,+ "mean_confidence": 0.7501,+ "acc_at_50_coverage": 0.54,+ "seconds": 8.0,+ "dropped": 0+ },+ "ru": {+ "n": 100,+ "accuracy": 0.54,+ "macro_f1": 0.4872,+ "ece": 0.3163,+ "brier": 0.7624,+ "nll": 2.5212,+ "mean_confidence": 0.8405,+ "acc_at_50_coverage": 0.7,+ "seconds": 8.0,+ "dropped": 0+ },+ "sl": {+ "n": 100,+ "accuracy": 0.33,+ "macro_f1": 0.2981,+ "ece": 0.4335,+ "brier": 1.0212,+ "nll": 3.1538,+ "mean_confidence": 0.7635,+ "acc_at_50_coverage": 0.48,+ "seconds": 8.1,+ "dropped": 0+ },+ "sq": {+ "n": 100,+ "accuracy": 0.26,+ "macro_f1": 0.2275,+ "ece": 0.4764,+ "brier": 1.1099,+ "nll": 3.4562,+ "mean_confidence": 0.7348,+ "acc_at_50_coverage": 0.4,+ "seconds": 8.2,+ "dropped": 0+ },+ "sv": {+ "n": 100,+ "accuracy": 0.57,+ "macro_f1": 0.524,+ "ece": 0.2757,+ "brier": 0.7141,+ "nll": 2.3393,+ "mean_confidence": 0.8191,+ "acc_at_50_coverage": 0.74,+ "seconds": 7.8,+ "dropped": 0+ },+ "sw": {+ "n": 100,+ "accuracy": 0.18,+ "macro_f1": 0.1741,+ "ece": 0.5495,+ "brier": 1.265,+ "nll": 4.8324,+ "mean_confidence": 0.7295,+ "acc_at_50_coverage": 0.28,+ "seconds": 8.1,+ "dropped": 0+ },+ "ta": {+ "n": 100,+ "accuracy": 0.23,+ "macro_f1": 0.1945,+ "ece": 0.3968,+ "brier": 1.0924,+ "nll": 3.4722,+ "mean_confidence": 0.6268,+ "acc_at_50_coverage": 0.28,+ "seconds": 8.5,+ "dropped": 0+ },+ "te": {+ "n": 100,+ "accuracy": 0.22,+ "macro_f1": 0.1973,+ "ece": 0.3705,+ "brier": 1.0643,+ "nll": 3.348,+ "mean_confidence": 0.5905,+ "acc_at_50_coverage": 0.28,+ "seconds": 8.5,+ "dropped": 0+ },+ "th": {+ "n": 100,+ "accuracy": 0.48,+ "macro_f1": 0.4513,+ "ece": 0.3357,+ "brier": 0.8119,+ "nll": 2.5203,+ "mean_confidence": 0.7844,+ "acc_at_50_coverage": 0.62,+ "seconds": 8.1,+ "dropped": 0+ },+ "tl": {+ "n": 100,+ "accuracy": 0.36,+ "macro_f1": 0.332,+ "ece": 0.3738,+ "brier": 0.9662,+ "nll": 3.2804,+ "mean_confidence": 0.7338,+ "acc_at_50_coverage": 0.48,+ "seconds": 8.1,+ "dropped": 0+ },+ "tr": {+ "n": 100,+ "accuracy": 0.37,+ "macro_f1": 0.3398,+ "ece": 0.417,+ "brier": 0.9744,+ "nll": 3.1177,+ "mean_confidence": 0.7784,+ "acc_at_50_coverage": 0.5,+ "seconds": 7.8,+ "dropped": 0+ },+ "ur": {+ "n": 100,+ "accuracy": 0.4,+ "macro_f1": 0.3774,+ "ece": 0.3111,+ "brier": 0.8551,+ "nll": 2.7536,+ "mean_confidence": 0.7111,+ "acc_at_50_coverage": 0.58,+ "seconds": 8.0,+ "dropped": 0+ },+ "vi": {+ "n": 100,+ "accuracy": 0.26,+ "macro_f1": 0.2238,+ "ece": 0.5214,+ "brier": 1.145,+ "nll": 3.627,+ "mean_confidence": 0.7722,+ "acc_at_50_coverage": 0.4,+ "seconds": 7.9,+ "dropped": 0+ },+ "zh-CN": {+ "n": 100,+ "accuracy": 0.63,+ "macro_f1": 0.5936,+ "ece": 0.2123,+ "brier": 0.5221,+ "nll": 1.714,+ "mean_confidence": 0.8246,+ "acc_at_50_coverage": 0.88,+ "seconds": 7.7,+ "dropped": 0+ },+ "zh-TW": {+ "n": 100,+ "accuracy": 0.54,+ "macro_f1": 0.4956,+ "ece": 0.327,+ "brier": 0.7166,+ "nll": 2.0914,+ "mean_confidence": 0.83,+ "acc_at_50_coverage": 0.8,+ "seconds": 7.7,+ "dropped": 0+ }+ },+ "macro_accuracy": 0.3661,+ "macro_ece": 0.3869,+ "n_languages": 51,+ "languages_above_random": 45+ }+ }+ },+ "part_b": {+ "reference_points": {+ "jev_1.13.0_published": {+ "accuracy": 0.727,+ "soft_accuracy": 0.58,+ "brier": 0.148,+ "ece": 0.144,+ "score_mae": 0.391,+ "ms_per_case": 710,+ "source": "figure quoted in the laya repo's own comparison table"+ },+ "teacher_self_agreement_ceiling": {+ "accuracy": 0.735+ },+ "modernbert_base_specialist": {+ "accuracy": 0.646+ },+ "random_guess": {+ "accuracy": 0.3175+ },+ "per_question_majority_class": {+ "accuracy": 0.461+ },+ "note": "Jev was NOT run here - no TypeSafe API credential is available. These are published numbers reproduced for context, not a measured head-to-head."+ },+ "by_model": {+ "english": {+ "n": 2000,+ "accuracy": 0.3615,+ "macro_f1": 0.3266,+ "ece": 0.1747,+ "brier": 0.7497,+ "nll": 1.3227,+ "mean_confidence": 0.5362,+ "acc_at_50_coverage": 0.428,+ "soft_accuracy": 0.3315,+ "brier_vs_soft": 0.3155,+ "score_mae": 0.6937,+ "within_1_level": 0.7525,+ "seconds": 557.0,+ "dropped": 0,+ "ms_per_case": 1392.5,+ "by_workflow": {+ "agent_trace_observability": {+ "n": 500,+ "accuracy": 0.388,+ "macro_f1": 0.3315,+ "ece": 0.1999,+ "brier": 0.7477,+ "nll": 1.3386,+ "mean_confidence": 0.576,+ "acc_at_50_coverage": 0.448+ },+ "customer_service": {+ "n": 500,+ "accuracy": 0.382,+ "macro_f1": 0.3446,+ "ece": 0.1067,+ "brier": 0.7051,+ "nll": 1.295,+ "mean_confidence": 0.4783,+ "acc_at_50_coverage": 0.52+ },+ "invoice_processing": {+ "n": 500,+ "accuracy": 0.36,+ "macro_f1": 0.2176,+ "ece": 0.2281,+ "brier": 0.7889,+ "nll": 1.3281,+ "mean_confidence": 0.5458,+ "acc_at_50_coverage": 0.292+ },+ "security_incidents": {+ "n": 500,+ "accuracy": 0.316,+ "macro_f1": 0.2115,+ "ece": 0.2287,+ "brier": 0.757,+ "nll": 1.3292,+ "mean_confidence": 0.5445,+ "acc_at_50_coverage": 0.428+ }+ },+ "by_question_type": {+ "choice": {+ "n": 600,+ "accuracy": 0.2883,+ "macro_f1": 0.2958,+ "ece": 0.1681,+ "brier": 0.8079,+ "nll": 1.5603,+ "mean_confidence": 0.4527,+ "acc_at_50_coverage": 0.3867+ },+ "noul": {+ "n": 600,+ "accuracy": 0.4867,+ "macro_f1": 0.4681,+ "ece": 0.1747,+ "brier": 0.6068,+ "nll": 0.8473,+ "mean_confidence": 0.6454,+ "acc_at_50_coverage": 0.43+ },+ "score": {+ "n": 800,+ "accuracy": 0.3225,+ "macro_f1": 0.2413,+ "ece": 0.2065,+ "brier": 0.8131,+ "nll": 1.5011,+ "mean_confidence": 0.5168,+ "acc_at_50_coverage": 0.335+ }+ }+ }+ }+ }+}\ No newline at end of filediff --git a/research/results/t4_colab_benchmark.json b/research/results/t4_colab_benchmark.jsonnew file mode 100644index 00000000..ddc400a4--- /dev/null+++ b/research/results/t4_colab_benchmark.json@@ -0,0 +1,4425 @@+{+ "meta": {+ "seed": 13,+ "n_opts": 20,+ "per_lang": 300,+ "timestamp": "2026-09-19 04:45:43",+ "gpu": "Tesla T4",+ "torch": "2.11.0+cu128",+ "transformers": "5.17.0",+ "models": {+ "laya": {+ "repo": "convaiinnovations/laya",+ "params_total_m": 421.29,+ "params_encoder_m": 394.78,+ "params_head_m": 26.51,+ "hidden_size": 1024,+ "num_layers": 28,+ "vocab_size": 50368,+ "max_len": 512,+ "head_max_len": 192,+ "temperature": [+ 1.6369030475616455,+ 1.2514300346374512,+ 1.983399510383606+ ],+ "temperature_by_options": {+ "choice:3-5": 1.7601518630981445,+ "choice:6-10": 1.0000158548355103,+ "score:3-5": 1.2514300346374512,+ "noul:2": 1.983399510383606,+ "choice:11+": 0.10058280825614929,+ "choice:2": 1.9063563346862793+ },+ "training": {+ "updates": 7313,+ "epochs_completed": 1,+ "hours": 1.96,+ "world_size": 1,+ "fine_tuned_from_checkpoint": true+ },+ "download_s": 10.7,+ "tokenizer_config_patched": []+ },+ "laya-multilingual": {+ "repo": "convaiinnovations/laya-multilingual",+ "params_total_m": 321.91,+ "params_encoder_m": 306.94,+ "params_head_m": 14.97,+ "hidden_size": 768,+ "num_layers": 22,+ "vocab_size": 256000,+ "max_len": 1024,+ "head_max_len": 256,+ "temperature": [+ 1.0,+ 1.0,+ 1.0+ ],+ "temperature_by_options": {},+ "training": {+ "updates": 15987,+ "epochs_completed": 4,+ "hours": 4.97,+ "world_size": 1,+ "fine_tuned_from_checkpoint": false+ },+ "download_s": 8.3,+ "tokenizer_config_patched": [+ "extra_special_tokens list->dict"+ ]+ }+ }+ },+ "suites": {+ "typed_decisions": {+ "laya": {+ "calibrated": {+ "n": 2000,+ "accuracy": 0.362,+ "macro_f1": 0.3273577076400338,+ "ece": 0.17410090455022942,+ "brier": 0.7496143221875959,+ "nll": 1.3224634387781617,+ "aurc": 0.5928492917684455,+ "mean_confidence": 0.5361009045502294,+ "acc_at_50_coverage": 0.427,+ "acc_at_80_coverage": 0.386875+ },+ "raw": {+ "n": 2000,+ "accuracy": 0.362,+ "macro_f1": 0.3273577076400338,+ "ece": 0.2648801700275417,+ "brier": 0.8277820380915699,+ "nll": 1.53173039009466,+ "aurc": 0.5998599200683264,+ "mean_confidence": 0.6268801700275417,+ "acc_at_50_coverage": 0.425,+ "acc_at_80_coverage": 0.384375+ },+ "seconds": 36.2,+ "dropped_questions": 0,+ "questions_per_second": 55.2,+ "soft_acc": 0.3314854404264519,+ "brier_soft": 0.31543323975432846,+ "tv": 0.40975100160837324,+ "kl": 0.5667172541116379,+ "score_mae": 0.6936800032000978,+ "within_1": 0.7525,+ "by_question_type": {+ "choice": {+ "n": 600,+ "accuracy": 0.29,+ "macro_f1": 0.2976158635239495,+ "ece": 0.16639379710456426,+ "brier": 0.8078685799684945,+ "nll": 1.5601982379145576,+ "aurc": 0.6165177886981781,+ "mean_confidence": 0.45269924967840924,+ "acc_at_50_coverage": 0.38666666666666666,+ "acc_at_80_coverage": 0.32708333333333334+ },+ "score": {+ "n": 800,+ "accuracy": 0.3225,+ "macro_f1": 0.24129818605373515,+ "ece": 0.2047018946740388,+ "brier": 0.8130431610474794,+ "nll": 1.5009182409941182,+ "aurc": 0.6560815223692904,+ "mean_confidence": 0.5166723088589777,+ "acc_at_50_coverage": 0.335,+ "acc_at_80_coverage": 0.3109375+ },+ "noul": {+ "n": 600,+ "accuracy": 0.4866666666666667,+ "macro_f1": 0.46813409467673606,+ "ece": 0.17554133251857967,+ "brier": 0.6067882792601861,+ "nll": 0.8467889033538241,+ "aurc": 0.5565114121040284,+ "mean_confidence": 0.6454073536770518,+ "acc_at_50_coverage": 0.43,+ "acc_at_80_coverage": 0.49583333333333335+ }+ }+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 2000,+ "accuracy": 0.3515,+ "macro_f1": 0.34975321170122525,+ "ece": 0.31443177617478313,+ "brier": 0.8904323210391285,+ "nll": 1.8242712912901595,+ "aurc": 0.5562863247537705,+ "mean_confidence": 0.6629025476146312,+ "acc_at_50_coverage": 0.414,+ "acc_at_80_coverage": 0.368125+ },+ "raw": {+ "n": 2000,+ "accuracy": 0.3515,+ "macro_f1": 0.34975321170122525,+ "ece": 0.31443177617478313,+ "brier": 0.8904323210391285,+ "nll": 1.8242712912901595,+ "aurc": 0.5562863247537705,+ "mean_confidence": 0.6629025476146312,+ "acc_at_50_coverage": 0.414,+ "acc_at_80_coverage": 0.368125+ },+ "seconds": 19.35,+ "dropped_questions": 0,+ "questions_per_second": 103.4,+ "soft_acc": 0.3280210927253606,+ "brier_soft": 0.462641406109187,+ "tv": 0.46852647440176376,+ "kl": 1.0768631881828523,+ "score_mae": 0.7604419071688604,+ "within_1": 0.7,+ "by_question_type": {+ "choice": {+ "n": 600,+ "accuracy": 0.295,+ "macro_f1": 0.3242803546952691,+ "ece": 0.36240144724213347,+ "brier": 0.9885012541895022,+ "nll": 2.3854267310642197,+ "aurc": 0.5947836135918948,+ "mean_confidence": 0.6559877759584947,+ "acc_at_50_coverage": 0.32,+ "acc_at_80_coverage": 0.28541666666666665+ },+ "score": {+ "n": 800,+ "accuracy": 0.285,+ "macro_f1": 0.2545688462591573,+ "ece": 0.2324908307616596,+ "brier": 0.8672059845074727,+ "nll": 1.7060067133863015,+ "aurc": 0.7180285302053313,+ "mean_confidence": 0.5109780128240088,+ "acc_at_50_coverage": 0.26,+ "acc_at_80_coverage": 0.2765625+ },+ "noul": {+ "n": 600,+ "accuracy": 0.49666666666666665,+ "macro_f1": 0.45384837962962965,+ "ece": 0.37571669899159743,+ "brier": 0.8233318365976293,+ "nll": 1.4208019553879097,+ "aurc": 0.48235294849361254,+ "mean_confidence": 0.872383365658264,+ "acc_at_50_coverage": 0.47,+ "acc_at_80_coverage": 0.4979166666666667+ }+ }+ }+ },+ "massive_intent.en": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.7833333333333333,+ "macro_f1": 0.7859904974658811,+ "ece": 0.2142657822979579,+ "brier": 0.4285638518597769,+ "nll": 4.783053086201328,+ "aurc": 0.10066152754017459,+ "mean_confidence": 0.994229890548954,+ "acc_at_50_coverage": 0.9066666666666666,+ "acc_at_80_coverage": 0.8916666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.7833333333333333,+ "macro_f1": 0.7859904974658811,+ "ece": 0.12625541044799943,+ "brier": 0.3323165550840956,+ "nll": 1.040553004538537,+ "aurc": 0.0889005283504489,+ "mean_confidence": 0.9066535593761121,+ "acc_at_50_coverage": 0.94,+ "acc_at_80_coverage": 0.8875+ },+ "seconds": 3.45,+ "dropped_questions": 0,+ "questions_per_second": 87.1+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.6566666666666666,+ "macro_f1": 0.6613789797689034,+ "ece": 0.2366905012718295,+ "brier": 0.5754824338960343,+ "nll": 2.092763427064851,+ "aurc": 0.17723907538979836,+ "mean_confidence": 0.8847088454845109,+ "acc_at_50_coverage": 0.86,+ "acc_at_80_coverage": 0.7291666666666666+ },+ "raw": {+ "n": 300,+ "accuracy": 0.6566666666666666,+ "macro_f1": 0.6613789797689034,+ "ece": 0.2366905012718295,+ "brier": 0.5754824338960343,+ "nll": 2.092763427064851,+ "aurc": 0.17723907538979836,+ "mean_confidence": 0.8847088454845109,+ "acc_at_50_coverage": 0.86,+ "acc_at_80_coverage": 0.7291666666666666+ },+ "seconds": 1.95,+ "dropped_questions": 0,+ "questions_per_second": 153.9+ }+ },+ "massive_intent.de": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.42,+ "macro_f1": 0.41942609239230483,+ "ece": 0.5572355019418889,+ "brier": 1.1276064052483479,+ "nll": 13.457262315998419,+ "aurc": 0.40545585846660126,+ "mean_confidence": 0.9772355019418888,+ "acc_at_50_coverage": 0.62,+ "acc_at_80_coverage": 0.475+ },+ "raw": {+ "n": 300,+ "accuracy": 0.42,+ "macro_f1": 0.41942609239230483,+ "ece": 0.32121714127531525,+ "brier": 0.8544067217522678,+ "nll": 2.744549528377823,+ "aurc": 0.35925213513126003,+ "mean_confidence": 0.7358504676886946,+ "acc_at_50_coverage": 0.6266666666666667,+ "acc_at_80_coverage": 0.4791666666666667+ },+ "seconds": 3.61,+ "dropped_questions": 0,+ "questions_per_second": 83.2+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.4666666666666667,+ "macro_f1": 0.4664031848395659,+ "ece": 0.3671645464311559,+ "brier": 0.849063341128718,+ "nll": 2.793117284229578,+ "aurc": 0.3189612559140324,+ "mean_confidence": 0.8284870971670211,+ "acc_at_50_coverage": 0.68,+ "acc_at_80_coverage": 0.5291666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.4666666666666667,+ "macro_f1": 0.4664031848395659,+ "ece": 0.3671645464311559,+ "brier": 0.849063341128718,+ "nll": 2.793117284229578,+ "aurc": 0.3189612559140324,+ "mean_confidence": 0.8284870971670211,+ "acc_at_50_coverage": 0.68,+ "acc_at_80_coverage": 0.5291666666666667+ },+ "seconds": 2.0,+ "dropped_questions": 0,+ "questions_per_second": 150.1+ }+ },+ "massive_intent.fr": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.4866666666666667,+ "macro_f1": 0.49028575480881303,+ "ece": 0.4903791740210341,+ "brier": 0.9864560461649728,+ "nll": 11.056989529764705,+ "aurc": 0.3663541633047382,+ "mean_confidence": 0.9770458406877007,+ "acc_at_50_coverage": 0.6733333333333333,+ "acc_at_80_coverage": 0.5583333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.4866666666666667,+ "macro_f1": 0.49028575480881303,+ "ece": 0.2933044954853109,+ "brier": 0.7463862431004725,+ "nll": 2.0693345449320564,+ "aurc": 0.2894619278805127,+ "mean_confidence": 0.7750594706553864,+ "acc_at_50_coverage": 0.6866666666666666,+ "acc_at_80_coverage": 0.5625+ },+ "seconds": 3.64,+ "dropped_questions": 0,+ "questions_per_second": 82.4+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.5533333333333333,+ "macro_f1": 0.5512106277128809,+ "ece": 0.2782850354212724,+ "brier": 0.6978064231902851,+ "nll": 2.575724948009992,+ "aurc": 0.29284529217235095,+ "mean_confidence": 0.8273814788760117,+ "acc_at_50_coverage": 0.74,+ "acc_at_80_coverage": 0.6416666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.5533333333333333,+ "macro_f1": 0.5512106277128809,+ "ece": 0.2782850354212724,+ "brier": 0.6978064231902851,+ "nll": 2.575724948009992,+ "aurc": 0.29284529217235095,+ "mean_confidence": 0.8273814788760117,+ "acc_at_50_coverage": 0.74,+ "acc_at_80_coverage": 0.6416666666666667+ },+ "seconds": 2.03,+ "dropped_questions": 0,+ "questions_per_second": 148.0+ }+ },+ "massive_intent.es": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.48,+ "macro_f1": 0.47803545309485507,+ "ece": 0.5070839008158101,+ "brier": 1.0169852227862164,+ "nll": 12.023140744919116,+ "aurc": 0.41148437275049987,+ "mean_confidence": 0.97951395435739,+ "acc_at_50_coverage": 0.6333333333333333,+ "acc_at_80_coverage": 0.5291666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.48,+ "macro_f1": 0.47803545309485507,+ "ece": 0.3130614775019185,+ "brier": 0.8095850837679279,+ "nll": 2.5207261343558645,+ "aurc": 0.34546548311114267,+ "mean_confidence": 0.7758544205302613,+ "acc_at_50_coverage": 0.64,+ "acc_at_80_coverage": 0.525+ },+ "seconds": 3.59,+ "dropped_questions": 0,+ "questions_per_second": 83.5+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.5033333333333333,+ "macro_f1": 0.49673783269846206,+ "ece": 0.3057029621982169,+ "brier": 0.7731615819701524,+ "nll": 2.464950590788325,+ "aurc": 0.3298720500466257,+ "mean_confidence": 0.8090362955315501,+ "acc_at_50_coverage": 0.6933333333333334,+ "acc_at_80_coverage": 0.5583333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.5033333333333333,+ "macro_f1": 0.49673783269846206,+ "ece": 0.3057029621982169,+ "brier": 0.7731615819701524,+ "nll": 2.464950590788325,+ "aurc": 0.3298720500466257,+ "mean_confidence": 0.8090362955315501,+ "acc_at_50_coverage": 0.6933333333333334,+ "acc_at_80_coverage": 0.5583333333333333+ },+ "seconds": 2.02,+ "dropped_questions": 0,+ "questions_per_second": 148.2+ }+ },+ "massive_intent.pt": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.45666666666666667,+ "macro_f1": 0.4503225203161018,+ "ece": 0.5258154421900451,+ "brier": 1.0615125009722581,+ "nll": 12.57833633824028,+ "aurc": 0.38008688374084976,+ "mean_confidence": 0.9824821088567119,+ "acc_at_50_coverage": 0.6533333333333333,+ "acc_at_80_coverage": 0.5291666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.45666666666666667,+ "macro_f1": 0.4503225203161018,+ "ece": 0.3446569932959994,+ "brier": 0.8274315021487898,+ "nll": 2.762952285380389,+ "aurc": 0.345993489129298,+ "mean_confidence": 0.7959313580649547,+ "acc_at_50_coverage": 0.64,+ "acc_at_80_coverage": 0.5416666666666666+ },+ "seconds": 3.68,+ "dropped_questions": 0,+ "questions_per_second": 81.5+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.49333333333333335,+ "macro_f1": 0.4876915658908163,+ "ece": 0.34213246762505234,+ "brier": 0.8232388037791051,+ "nll": 2.8230107127485624,+ "aurc": 0.31534173225577855,+ "mean_confidence": 0.8248538216832458,+ "acc_at_50_coverage": 0.66,+ "acc_at_80_coverage": 0.5625+ },+ "raw": {+ "n": 300,+ "accuracy": 0.49333333333333335,+ "macro_f1": 0.4876915658908163,+ "ece": 0.34213246762505234,+ "brier": 0.8232388037791051,+ "nll": 2.8230107127485624,+ "aurc": 0.31534173225577855,+ "mean_confidence": 0.8248538216832458,+ "acc_at_50_coverage": 0.66,+ "acc_at_80_coverage": 0.5625+ },+ "seconds": 2.02,+ "dropped_questions": 0,+ "questions_per_second": 148.2+ }+ },+ "massive_intent.ru": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.29333333333333333,+ "macro_f1": 0.29312146555618207,+ "ece": 0.6849768215676151,+ "brier": 1.3819963575438161,+ "nll": 17.166320983364137,+ "aurc": 0.6050616643967031,+ "mean_confidence": 0.9783101549009484,+ "acc_at_50_coverage": 0.41333333333333333,+ "acc_at_80_coverage": 0.3458333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.29333333333333333,+ "macro_f1": 0.29312146555618207,+ "ece": 0.44761075495748537,+ "brier": 1.0753941894333248,+ "nll": 3.5739178816102837,+ "aurc": 0.5521122930922301,+ "mean_confidence": 0.7409440882908187,+ "acc_at_50_coverage": 0.41333333333333333,+ "acc_at_80_coverage": 0.35+ },+ "seconds": 3.89,+ "dropped_questions": 0,+ "questions_per_second": 77.1+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.5,+ "macro_f1": 0.5061202589086473,+ "ece": 0.3233476303888164,+ "brier": 0.7951925984574719,+ "nll": 2.6843260934306636,+ "aurc": 0.3438499231303233,+ "mean_confidence": 0.8233476303888164,+ "acc_at_50_coverage": 0.66,+ "acc_at_80_coverage": 0.5708333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.5,+ "macro_f1": 0.5061202589086473,+ "ece": 0.3233476303888164,+ "brier": 0.7951925984574719,+ "nll": 2.6843260934306636,+ "aurc": 0.3438499231303233,+ "mean_confidence": 0.8233476303888164,+ "acc_at_50_coverage": 0.66,+ "acc_at_80_coverage": 0.5708333333333333+ },+ "seconds": 2.02,+ "dropped_questions": 0,+ "questions_per_second": 148.6+ }+ },+ "massive_intent.tr": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.16,+ "macro_f1": 0.1486039563594193,+ "ece": 0.7897886355179764,+ "brier": 1.603631422859012,+ "nll": 20.466186122493088,+ "aurc": 0.746369008854861,+ "mean_confidence": 0.9497886355179764,+ "acc_at_50_coverage": 0.24,+ "acc_at_80_coverage": 0.1875+ },+ "raw": {+ "n": 300,+ "accuracy": 0.16,+ "macro_f1": 0.1486039563594193,+ "ece": 0.4083382750097803,+ "brier": 1.1535723588235036,+ "nll": 4.133708635893034,+ "aurc": 0.6850322055704485,+ "mean_confidence": 0.5658787393235634,+ "acc_at_50_coverage": 0.23333333333333334,+ "acc_at_80_coverage": 0.1875+ },+ "seconds": 3.87,+ "dropped_questions": 0,+ "questions_per_second": 77.5+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.43666666666666665,+ "macro_f1": 0.4349967052180575,+ "ece": 0.35706493438546416,+ "brier": 0.9040762205611305,+ "nll": 3.0042063806753494,+ "aurc": 0.3818364137902104,+ "mean_confidence": 0.793731601052131,+ "acc_at_50_coverage": 0.6,+ "acc_at_80_coverage": 0.49166666666666664+ },+ "raw": {+ "n": 300,+ "accuracy": 0.43666666666666665,+ "macro_f1": 0.4349967052180575,+ "ece": 0.35706493438546416,+ "brier": 0.9040762205611305,+ "nll": 3.0042063806753494,+ "aurc": 0.3818364137902104,+ "mean_confidence": 0.793731601052131,+ "acc_at_50_coverage": 0.6,+ "acc_at_80_coverage": 0.49166666666666664+ },+ "seconds": 2.0,+ "dropped_questions": 0,+ "questions_per_second": 150.1+ }+ },+ "massive_intent.ar": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.12666666666666668,+ "macro_f1": 0.1256503611446384,+ "ece": 0.798641770944787,+ "brier": 1.6462925546066582,+ "nll": 20.856901740883874,+ "aurc": 0.8392220838559886,+ "mean_confidence": 0.9253084376114538,+ "acc_at_50_coverage": 0.16666666666666666,+ "acc_at_80_coverage": 0.12916666666666668+ },+ "raw": {+ "n": 300,+ "accuracy": 0.12666666666666668,+ "macro_f1": 0.1256503611446384,+ "ece": 0.39476726769861925,+ "brier": 1.177780937942809,+ "nll": 4.090553130199672,+ "aurc": 0.8014559695763626,+ "mean_confidence": 0.5214339343652858,+ "acc_at_50_coverage": 0.16,+ "acc_at_80_coverage": 0.1375+ },+ "seconds": 3.99,+ "dropped_questions": 0,+ "questions_per_second": 75.1+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.38333333333333336,+ "macro_f1": 0.3773237673233435,+ "ece": 0.3670310695379155,+ "brier": 0.9436406092973436,+ "nll": 3.0610236443302123,+ "aurc": 0.4345891218357328,+ "mean_confidence": 0.7489822251667393,+ "acc_at_50_coverage": 0.5333333333333333,+ "acc_at_80_coverage": 0.44583333333333336+ },+ "raw": {+ "n": 300,+ "accuracy": 0.38333333333333336,+ "macro_f1": 0.3773237673233435,+ "ece": 0.3670310695379155,+ "brier": 0.9436406092973436,+ "nll": 3.0610236443302123,+ "aurc": 0.4345891218357328,+ "mean_confidence": 0.7489822251667393,+ "acc_at_50_coverage": 0.5333333333333333,+ "acc_at_80_coverage": 0.44583333333333336+ },+ "seconds": 1.96,+ "dropped_questions": 0,+ "questions_per_second": 153.1+ }+ },+ "massive_intent.hi": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.1,+ "macro_f1": 0.0953787060033515,+ "ece": 0.8431830079618857,+ "brier": 1.724777262836915,+ "nll": 21.299563769404223,+ "aurc": 0.904048278658199,+ "mean_confidence": 0.9431830079618855,+ "acc_at_50_coverage": 0.10666666666666667,+ "acc_at_80_coverage": 0.1125+ },+ "raw": {+ "n": 300,+ "accuracy": 0.1,+ "macro_f1": 0.0953787060033515,+ "ece": 0.4348448694638475,+ "brier": 1.2351111778950616,+ "nll": 4.272234684416419,+ "aurc": 0.8813871989083324,+ "mean_confidence": 0.5315443656017397,+ "acc_at_50_coverage": 0.1,+ "acc_at_80_coverage": 0.10833333333333334+ },+ "seconds": 4.34,+ "dropped_questions": 0,+ "questions_per_second": 69.2+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.38666666666666666,+ "macro_f1": 0.3799765136142003,+ "ece": 0.39705202860146693,+ "brier": 0.966182445618435,+ "nll": 3.1871501391728065,+ "aurc": 0.4557389301411769,+ "mean_confidence": 0.7817389664743709,+ "acc_at_50_coverage": 0.52,+ "acc_at_80_coverage": 0.44583333333333336+ },+ "raw": {+ "n": 300,+ "accuracy": 0.38666666666666666,+ "macro_f1": 0.3799765136142003,+ "ece": 0.39705202860146693,+ "brier": 0.966182445618435,+ "nll": 3.1871501391728065,+ "aurc": 0.4557389301411769,+ "mean_confidence": 0.7817389664743709,+ "acc_at_50_coverage": 0.52,+ "acc_at_80_coverage": 0.44583333333333336+ },+ "seconds": 1.98,+ "dropped_questions": 0,+ "questions_per_second": 151.2+ }+ },+ "massive_intent.ta": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.11333333333333333,+ "macro_f1": 0.10519330875322393,+ "ece": 0.8241359277747448,+ "brier": 1.6826711980977516,+ "nll": 19.965294983018907,+ "aurc": 0.8536231181038471,+ "mean_confidence": 0.935663685936003,+ "acc_at_50_coverage": 0.12666666666666668,+ "acc_at_80_coverage": 0.12916666666666668+ },+ "raw": {+ "n": 300,+ "accuracy": 0.11333333333333333,+ "macro_f1": 0.10519330875322393,+ "ece": 0.31129690461029336,+ "brier": 1.1128552391640942,+ "nll": 3.7176514878944316,+ "aurc": 0.8557360412466305,+ "mean_confidence": 0.4246302379436267,+ "acc_at_50_coverage": 0.12666666666666668,+ "acc_at_80_coverage": 0.12916666666666668+ },+ "seconds": 4.85,+ "dropped_questions": 0,+ "questions_per_second": 61.9+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.25,+ "macro_f1": 0.2475709870276944,+ "ece": 0.48455217697434594,+ "brier": 1.1784996530243754,+ "nll": 3.996754995032367,+ "aurc": 0.6777780939794569,+ "mean_confidence": 0.7345521769743459,+ "acc_at_50_coverage": 0.32,+ "acc_at_80_coverage": 0.275+ },+ "raw": {+ "n": 300,+ "accuracy": 0.25,+ "macro_f1": 0.2475709870276944,+ "ece": 0.48455217697434594,+ "brier": 1.1784996530243754,+ "nll": 3.996754995032367,+ "aurc": 0.6777780939794569,+ "mean_confidence": 0.7345521769743459,+ "acc_at_50_coverage": 0.32,+ "acc_at_80_coverage": 0.275+ },+ "seconds": 2.01,+ "dropped_questions": 0,+ "questions_per_second": 149.3+ }+ },+ "massive_intent.zh-CN": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.5933333333333334,+ "macro_f1": 0.5964552527791429,+ "ece": 0.39916859064752297,+ "brier": 0.80660057431863,+ "nll": 9.704434459218685,+ "aurc": 0.24511533386115847,+ "mean_confidence": 0.9889531214305413,+ "acc_at_50_coverage": 0.7733333333333333,+ "acc_at_80_coverage": 0.6666666666666666+ },+ "raw": {+ "n": 300,+ "accuracy": 0.5933333333333334,+ "macro_f1": 0.5964552527791429,+ "ece": 0.24586806288232513,+ "brier": 0.6523252570127208,+ "nll": 2.0586656002191677,+ "aurc": 0.2229554358525133,+ "mean_confidence": 0.8328217574294239,+ "acc_at_50_coverage": 0.7666666666666667,+ "acc_at_80_coverage": 0.6708333333333333+ },+ "seconds": 4.01,+ "dropped_questions": 0,+ "questions_per_second": 74.7+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.6066666666666667,+ "macro_f1": 0.6066977290299708,+ "ece": 0.2624488813566047,+ "brier": 0.6395126198165556,+ "nll": 2.057482622773751,+ "aurc": 0.20966586558217612,+ "mean_confidence": 0.8679142837146541,+ "acc_at_50_coverage": 0.84,+ "acc_at_80_coverage": 0.6958333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.6066666666666667,+ "macro_f1": 0.6066977290299708,+ "ece": 0.2624488813566047,+ "brier": 0.6395126198165556,+ "nll": 2.057482622773751,+ "aurc": 0.20966586558217612,+ "mean_confidence": 0.8679142837146541,+ "acc_at_50_coverage": 0.84,+ "acc_at_80_coverage": 0.6958333333333333+ },+ "seconds": 1.92,+ "dropped_questions": 0,+ "questions_per_second": 155.9+ }+ },+ "massive_intent.ja": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.5466666666666666,+ "macro_f1": 0.5505137504336817,+ "ece": 0.43745213971398433,+ "brier": 0.8910538194571062,+ "nll": 10.610208949690762,+ "aurc": 0.27361814455380085,+ "mean_confidence": 0.9841188063806509,+ "acc_at_50_coverage": 0.74,+ "acc_at_80_coverage": 0.6+ },+ "raw": {+ "n": 300,+ "accuracy": 0.5466666666666666,+ "macro_f1": 0.5505137504336817,+ "ece": 0.23469877865675798,+ "brier": 0.6962990571747073,+ "nll": 2.1285255990386305,+ "aurc": 0.26818040251979264,+ "mean_confidence": 0.7813654453234246,+ "acc_at_50_coverage": 0.7533333333333333,+ "acc_at_80_coverage": 0.625+ },+ "seconds": 4.08,+ "dropped_questions": 0,+ "questions_per_second": 73.5+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.5866666666666667,+ "macro_f1": 0.5789792898949173,+ "ece": 0.26880170740133336,+ "brier": 0.6641315368078033,+ "nll": 2.1563007642554584,+ "aurc": 0.2606575513014464,+ "mean_confidence": 0.8537674763577928,+ "acc_at_50_coverage": 0.7666666666666667,+ "acc_at_80_coverage": 0.6541666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.5866666666666667,+ "macro_f1": 0.5789792898949173,+ "ece": 0.26880170740133336,+ "brier": 0.6641315368078033,+ "nll": 2.1563007642554584,+ "aurc": 0.2606575513014464,+ "mean_confidence": 0.8537674763577928,+ "acc_at_50_coverage": 0.7666666666666667,+ "acc_at_80_coverage": 0.6541666666666667+ },+ "seconds": 1.91,+ "dropped_questions": 0,+ "questions_per_second": 157.0+ }+ },+ "massive_intent.ko": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.10333333333333333,+ "macro_f1": 0.09534820721043308,+ "ece": 0.8496262680408053,+ "brier": 1.718330722538468,+ "nll": 21.741823832218525,+ "aurc": 0.8696438316894232,+ "mean_confidence": 0.9529596013741385,+ "acc_at_50_coverage": 0.13333333333333333,+ "acc_at_80_coverage": 0.12083333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.10333333333333333,+ "macro_f1": 0.09534820721043308,+ "ece": 0.45471327863474253,+ "brier": 1.236188621010311,+ "nll": 4.2678925994892944,+ "aurc": 0.8530241092992829,+ "mean_confidence": 0.5580466119680759,+ "acc_at_50_coverage": 0.14,+ "acc_at_80_coverage": 0.12083333333333333+ },+ "seconds": 4.4,+ "dropped_questions": 0,+ "questions_per_second": 68.1+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.49,+ "macro_f1": 0.488708216378764,+ "ece": 0.3084237491405182,+ "brier": 0.7840216027433781,+ "nll": 2.48264831603104,+ "aurc": 0.3197043609942588,+ "mean_confidence": 0.7984237491405181,+ "acc_at_50_coverage": 0.68,+ "acc_at_80_coverage": 0.5833333333333334+ },+ "raw": {+ "n": 300,+ "accuracy": 0.49,+ "macro_f1": 0.488708216378764,+ "ece": 0.3084237491405182,+ "brier": 0.7840216027433781,+ "nll": 2.48264831603104,+ "aurc": 0.3197043609942588,+ "mean_confidence": 0.7984237491405181,+ "acc_at_50_coverage": 0.68,+ "acc_at_80_coverage": 0.5833333333333334+ },+ "seconds": 1.93,+ "dropped_questions": 0,+ "questions_per_second": 155.2+ }+ },+ "massive_intent.sw": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.10333333333333333,+ "macro_f1": 0.09566516861510858,+ "ece": 0.8512619986530077,+ "brier": 1.7306965908645546,+ "nll": 21.490747833569763,+ "aurc": 0.8578274070062489,+ "mean_confidence": 0.9539542441153954,+ "acc_at_50_coverage": 0.12666666666666668,+ "acc_at_80_coverage": 0.10833333333333334+ },+ "raw": {+ "n": 300,+ "accuracy": 0.10333333333333333,+ "macro_f1": 0.09566516861510858,+ "ece": 0.3882574858223093,+ "brier": 1.17321258150963,+ "nll": 4.096486897113687,+ "aurc": 0.8556569352727932,+ "mean_confidence": 0.49159081915564257,+ "acc_at_50_coverage": 0.12,+ "acc_at_80_coverage": 0.1125+ },+ "seconds": 4.2,+ "dropped_questions": 0,+ "questions_per_second": 71.5+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.21,+ "macro_f1": 0.21000860513059322,+ "ece": 0.4905093053428704,+ "brier": 1.1966822466692135,+ "nll": 4.229903642570186,+ "aurc": 0.6159026094360324,+ "mean_confidence": 0.7005093053428704,+ "acc_at_50_coverage": 0.30666666666666664,+ "acc_at_80_coverage": 0.24166666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.21,+ "macro_f1": 0.21000860513059322,+ "ece": 0.4905093053428704,+ "brier": 1.1966822466692135,+ "nll": 4.229903642570186,+ "aurc": 0.6159026094360324,+ "mean_confidence": 0.7005093053428704,+ "acc_at_50_coverage": 0.30666666666666664,+ "acc_at_80_coverage": 0.24166666666666667+ },+ "seconds": 1.92,+ "dropped_questions": 0,+ "questions_per_second": 155.9+ }+ },+ "massive_scenario.en": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.6033333333333334,+ "macro_f1": 0.5944362769884591,+ "ece": 0.3927976732517713,+ "brier": 0.7882183191086455,+ "nll": 9.781255719005527,+ "aurc": 0.31821612105676933,+ "mean_confidence": 0.9926578571022835,+ "acc_at_50_coverage": 0.7133333333333334,+ "acc_at_80_coverage": 0.6375+ },+ "raw": {+ "n": 300,+ "accuracy": 0.6033333333333334,+ "macro_f1": 0.5944362769884591,+ "ece": 0.2951358132155798,+ "brier": 0.6844123055005451,+ "nll": 2.0128865847342325,+ "aurc": 0.19393035010708404,+ "mean_confidence": 0.8966549050977662,+ "acc_at_50_coverage": 0.7933333333333333,+ "acc_at_80_coverage": 0.6541666666666667+ },+ "seconds": 2.26,+ "dropped_questions": 0,+ "questions_per_second": 132.6+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.56,+ "macro_f1": 0.5547673096787787,+ "ece": 0.3073935231585113,+ "brier": 0.7005044027556065,+ "nll": 2.445474960645575,+ "aurc": 0.17889742521042204,+ "mean_confidence": 0.8673935231585114,+ "acc_at_50_coverage": 0.8466666666666667,+ "acc_at_80_coverage": 0.6708333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.56,+ "macro_f1": 0.5547673096787787,+ "ece": 0.3073935231585113,+ "brier": 0.7005044027556065,+ "nll": 2.445474960645575,+ "aurc": 0.17889742521042204,+ "mean_confidence": 0.8673935231585114,+ "acc_at_50_coverage": 0.8466666666666667,+ "acc_at_80_coverage": 0.6708333333333333+ },+ "seconds": 0.93,+ "dropped_questions": 0,+ "questions_per_second": 322.3+ }+ },+ "massive_scenario.de": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.2966666666666667,+ "macro_f1": 0.2877606145118102,+ "ece": 0.6683327369921522,+ "brier": 1.3641775508951617,+ "nll": 16.096091375368577,+ "aurc": 0.5605746102500764,+ "mean_confidence": 0.9649994036588188,+ "acc_at_50_coverage": 0.41333333333333333,+ "acc_at_80_coverage": 0.32083333333333336+ },+ "raw": {+ "n": 300,+ "accuracy": 0.2966666666666667,+ "macro_f1": 0.2877606145118102,+ "ece": 0.42285314991027423,+ "brier": 1.0454422585082612,+ "nll": 3.1653200488989492,+ "aurc": 0.5145568812300169,+ "mean_confidence": 0.7195198165769409,+ "acc_at_50_coverage": 0.42,+ "acc_at_80_coverage": 0.32083333333333336+ },+ "seconds": 2.43,+ "dropped_questions": 0,+ "questions_per_second": 123.3+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.4766666666666667,+ "macro_f1": 0.4659465881614311,+ "ece": 0.3621706701744643,+ "brier": 0.8598210325231435,+ "nll": 2.7897092657552736,+ "aurc": 0.33911404853855776,+ "mean_confidence": 0.8388373368411309,+ "acc_at_50_coverage": 0.68,+ "acc_at_80_coverage": 0.5291666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.4766666666666667,+ "macro_f1": 0.4659465881614311,+ "ece": 0.3621706701744643,+ "brier": 0.8598210325231435,+ "nll": 2.7897092657552736,+ "aurc": 0.33911404853855776,+ "mean_confidence": 0.8388373368411309,+ "acc_at_50_coverage": 0.68,+ "acc_at_80_coverage": 0.5291666666666667+ },+ "seconds": 0.95,+ "dropped_questions": 0,+ "questions_per_second": 314.7+ }+ },+ "massive_scenario.fr": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.46,+ "macro_f1": 0.4519283547157385,+ "ece": 0.5235926580481265,+ "brier": 1.0568531962889085,+ "nll": 11.84809119688847,+ "aurc": 0.3935424667756691,+ "mean_confidence": 0.9835926580481263,+ "acc_at_50_coverage": 0.5733333333333334,+ "acc_at_80_coverage": 0.5041666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.46,+ "macro_f1": 0.4519283547157385,+ "ece": 0.327470752681133,+ "brier": 0.8270249223142718,+ "nll": 2.2711318154752536,+ "aurc": 0.3471555564698994,+ "mean_confidence": 0.7777987624031797,+ "acc_at_50_coverage": 0.5866666666666667,+ "acc_at_80_coverage": 0.4875+ },+ "seconds": 2.47,+ "dropped_questions": 0,+ "questions_per_second": 121.5+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.4533333333333333,+ "macro_f1": 0.45184505601719327,+ "ece": 0.41141516589821114,+ "brier": 0.9011266954092387,+ "nll": 2.969407019443928,+ "aurc": 0.31162388267444885,+ "mean_confidence": 0.8647484992315443,+ "acc_at_50_coverage": 0.6733333333333333,+ "acc_at_80_coverage": 0.5166666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.4533333333333333,+ "macro_f1": 0.45184505601719327,+ "ece": 0.41141516589821114,+ "brier": 0.9011266954092387,+ "nll": 2.969407019443928,+ "aurc": 0.31162388267444885,+ "mean_confidence": 0.8647484992315443,+ "acc_at_50_coverage": 0.6733333333333333,+ "acc_at_80_coverage": 0.5166666666666667+ },+ "seconds": 0.96,+ "dropped_questions": 0,+ "questions_per_second": 311.1+ }+ },+ "massive_scenario.es": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.38333333333333336,+ "macro_f1": 0.3808258971995832,+ "ece": 0.5943658153687433,+ "brier": 1.2053445709534798,+ "nll": 14.592647728493581,+ "aurc": 0.5137364183153083,+ "mean_confidence": 0.9776991487020765,+ "acc_at_50_coverage": 0.5,+ "acc_at_80_coverage": 0.42083333333333334+ },+ "raw": {+ "n": 300,+ "accuracy": 0.38333333333333336,+ "macro_f1": 0.3808258971995832,+ "ece": 0.4061125326810553,+ "brier": 0.974631055323556,+ "nll": 3.0268021168724615,+ "aurc": 0.449181477550745,+ "mean_confidence": 0.7872946293297955,+ "acc_at_50_coverage": 0.5266666666666666,+ "acc_at_80_coverage": 0.42916666666666664+ },+ "seconds": 2.42,+ "dropped_questions": 0,+ "questions_per_second": 123.9+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.44666666666666666,+ "macro_f1": 0.442167638299833,+ "ece": 0.4069640053563983,+ "brier": 0.8821714663948459,+ "nll": 3.0290174890601005,+ "aurc": 0.29734844288332196,+ "mean_confidence": 0.8479267644688439,+ "acc_at_50_coverage": 0.7133333333333334,+ "acc_at_80_coverage": 0.5291666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.44666666666666666,+ "macro_f1": 0.442167638299833,+ "ece": 0.4069640053563983,+ "brier": 0.8821714663948459,+ "nll": 3.0290174890601005,+ "aurc": 0.29734844288332196,+ "mean_confidence": 0.8479267644688439,+ "acc_at_50_coverage": 0.7133333333333334,+ "acc_at_80_coverage": 0.5291666666666667+ },+ "seconds": 0.96,+ "dropped_questions": 0,+ "questions_per_second": 313.0+ }+ },+ "massive_scenario.pt": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.42,+ "macro_f1": 0.41507318506283947,+ "ece": 0.5676626727748507,+ "brier": 1.142251765714677,+ "nll": 14.21340070224241,+ "aurc": 0.450532843481489,+ "mean_confidence": 0.9855712309838316,+ "acc_at_50_coverage": 0.5933333333333334,+ "acc_at_80_coverage": 0.45+ },+ "raw": {+ "n": 300,+ "accuracy": 0.42,+ "macro_f1": 0.41507318506283947,+ "ece": 0.37499435280399795,+ "brier": 0.9247850663292572,+ "nll": 2.9814680709745818,+ "aurc": 0.40034927925250896,+ "mean_confidence": 0.7906760763571875,+ "acc_at_50_coverage": 0.5866666666666667,+ "acc_at_80_coverage": 0.4791666666666667+ },+ "seconds": 2.48,+ "dropped_questions": 0,+ "questions_per_second": 120.7+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.4633333333333333,+ "macro_f1": 0.4583159082261799,+ "ece": 0.399902067877031,+ "brier": 0.9173335266917587,+ "nll": 3.300888761245623,+ "aurc": 0.3732814777085433,+ "mean_confidence": 0.861442229013749,+ "acc_at_50_coverage": 0.6133333333333333,+ "acc_at_80_coverage": 0.5125+ },+ "raw": {+ "n": 300,+ "accuracy": 0.4633333333333333,+ "macro_f1": 0.4583159082261799,+ "ece": 0.399902067877031,+ "brier": 0.9173335266917587,+ "nll": 3.300888761245623,+ "aurc": 0.3732814777085433,+ "mean_confidence": 0.861442229013749,+ "acc_at_50_coverage": 0.6133333333333333,+ "acc_at_80_coverage": 0.5125+ },+ "seconds": 0.95,+ "dropped_questions": 0,+ "questions_per_second": 316.5+ }+ },+ "massive_scenario.ru": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.34,+ "macro_f1": 0.3379920393729521,+ "ece": 0.6439844360330014,+ "brier": 1.292337140048209,+ "nll": 15.821192298551425,+ "aurc": 0.5736417959888973,+ "mean_confidence": 0.9775710458281587,+ "acc_at_50_coverage": 0.3933333333333333,+ "acc_at_80_coverage": 0.375+ },+ "raw": {+ "n": 300,+ "accuracy": 0.34,+ "macro_f1": 0.3379920393729521,+ "ece": 0.39153995940719355,+ "brier": 1.0187874517914235,+ "nll": 3.373532527817828,+ "aurc": 0.5612235825848404,+ "mean_confidence": 0.7278895131347629,+ "acc_at_50_coverage": 0.4266666666666667,+ "acc_at_80_coverage": 0.37916666666666665+ },+ "seconds": 2.62,+ "dropped_questions": 0,+ "questions_per_second": 114.6+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.47333333333333333,+ "macro_f1": 0.46948646596979554,+ "ece": 0.38306263750978925,+ "brier": 0.8780662503308873,+ "nll": 3.1476885318084227,+ "aurc": 0.3678024075289724,+ "mean_confidence": 0.8563959708431225,+ "acc_at_50_coverage": 0.6466666666666666,+ "acc_at_80_coverage": 0.55+ },+ "raw": {+ "n": 300,+ "accuracy": 0.47333333333333333,+ "macro_f1": 0.46948646596979554,+ "ece": 0.38306263750978925,+ "brier": 0.8780662503308873,+ "nll": 3.1476885318084227,+ "aurc": 0.3678024075289724,+ "mean_confidence": 0.8563959708431225,+ "acc_at_50_coverage": 0.6466666666666666,+ "acc_at_80_coverage": 0.55+ },+ "seconds": 0.94,+ "dropped_questions": 0,+ "questions_per_second": 319.5+ }+ },+ "massive_scenario.tr": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.22666666666666666,+ "macro_f1": 0.22930908041261722,+ "ece": 0.7344336039568069,+ "brier": 1.4942577998212168,+ "nll": 18.367193357732436,+ "aurc": 0.6448873480624936,+ "mean_confidence": 0.9611002706234736,+ "acc_at_50_coverage": 0.3333333333333333,+ "acc_at_80_coverage": 0.25416666666666665+ },+ "raw": {+ "n": 300,+ "accuracy": 0.22666666666666666,+ "macro_f1": 0.22930908041261722,+ "ece": 0.3714210891197136,+ "brier": 1.0763457483169783,+ "nll": 3.8042266690014293,+ "aurc": 0.5847453259269151,+ "mean_confidence": 0.597705349646592,+ "acc_at_50_coverage": 0.32666666666666666,+ "acc_at_80_coverage": 0.24583333333333332+ },+ "seconds": 2.55,+ "dropped_questions": 0,+ "questions_per_second": 117.5+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.48333333333333334,+ "macro_f1": 0.48279825510729374,+ "ece": 0.3510541120298612,+ "brier": 0.8351881304472127,+ "nll": 2.5433498577236873,+ "aurc": 0.3075129394182914,+ "mean_confidence": 0.8343874453631944,+ "acc_at_50_coverage": 0.6733333333333333,+ "acc_at_80_coverage": 0.55+ },+ "raw": {+ "n": 300,+ "accuracy": 0.48333333333333334,+ "macro_f1": 0.48279825510729374,+ "ece": 0.3510541120298612,+ "brier": 0.8351881304472127,+ "nll": 2.5433498577236873,+ "aurc": 0.3075129394182914,+ "mean_confidence": 0.8343874453631944,+ "acc_at_50_coverage": 0.6733333333333333,+ "acc_at_80_coverage": 0.55+ },+ "seconds": 0.94,+ "dropped_questions": 0,+ "questions_per_second": 320.1+ }+ },+ "massive_scenario.ar": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.14,+ "macro_f1": 0.1394507004001941,+ "ece": 0.8210527427159336,+ "brier": 1.6685223063858712,+ "nll": 20.83073025736417,+ "aurc": 0.8878838031394582,+ "mean_confidence": 0.9610527427159338,+ "acc_at_50_coverage": 0.10666666666666667,+ "acc_at_80_coverage": 0.1125+ },+ "raw": {+ "n": 300,+ "accuracy": 0.14,+ "macro_f1": 0.1394507004001941,+ "ece": 0.42293347604019144,+ "brier": 1.2073659505782506,+ "nll": 4.070635463836291,+ "aurc": 0.8709828618752677,+ "mean_confidence": 0.562436447123142,+ "acc_at_50_coverage": 0.12666666666666668,+ "acc_at_80_coverage": 0.11666666666666667+ },+ "seconds": 2.58,+ "dropped_questions": 0,+ "questions_per_second": 116.1+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.39666666666666667,+ "macro_f1": 0.3953884343454503,+ "ece": 0.41013940064528187,+ "brier": 0.9604693183837674,+ "nll": 3.244261200284413,+ "aurc": 0.41538539360629756,+ "mean_confidence": 0.8001018238229396,+ "acc_at_50_coverage": 0.56,+ "acc_at_80_coverage": 0.45416666666666666+ },+ "raw": {+ "n": 300,+ "accuracy": 0.39666666666666667,+ "macro_f1": 0.3953884343454503,+ "ece": 0.41013940064528187,+ "brier": 0.9604693183837674,+ "nll": 3.244261200284413,+ "aurc": 0.41538539360629756,+ "mean_confidence": 0.8001018238229396,+ "acc_at_50_coverage": 0.56,+ "acc_at_80_coverage": 0.45416666666666666+ },+ "seconds": 0.95,+ "dropped_questions": 0,+ "questions_per_second": 317.2+ }+ },+ "massive_scenario.hi": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.09666666666666666,+ "macro_f1": 0.09218738365400807,+ "ece": 0.8656490043549154,+ "brier": 1.750804430993744,+ "nll": 22.13358390954595,+ "aurc": 0.8898096195789185,+ "mean_confidence": 0.962315671021582,+ "acc_at_50_coverage": 0.10666666666666667,+ "acc_at_80_coverage": 0.1+ },+ "raw": {+ "n": 300,+ "accuracy": 0.09666666666666666,+ "macro_f1": 0.09218738365400807,+ "ece": 0.4678827983908522,+ "brier": 1.2401910045098414,+ "nll": 4.3450989701505796,+ "aurc": 0.8733878480349362,+ "mean_confidence": 0.5645494650575188,+ "acc_at_50_coverage": 0.10666666666666667,+ "acc_at_80_coverage": 0.10833333333333334+ },+ "seconds": 3.0,+ "dropped_questions": 0,+ "questions_per_second": 100.1+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.44666666666666666,+ "macro_f1": 0.44807505691508126,+ "ece": 0.3695063961757436,+ "brier": 0.9068063475197149,+ "nll": 3.0242455495430627,+ "aurc": 0.3385449178688652,+ "mean_confidence": 0.808062239575728,+ "acc_at_50_coverage": 0.5933333333333334,+ "acc_at_80_coverage": 0.4875+ },+ "raw": {+ "n": 300,+ "accuracy": 0.44666666666666666,+ "macro_f1": 0.44807505691508126,+ "ece": 0.3695063961757436,+ "brier": 0.9068063475197149,+ "nll": 3.0242455495430627,+ "aurc": 0.3385449178688652,+ "mean_confidence": 0.808062239575728,+ "acc_at_50_coverage": 0.5933333333333334,+ "acc_at_80_coverage": 0.4875+ },+ "seconds": 0.99,+ "dropped_questions": 0,+ "questions_per_second": 301.8+ }+ },+ "massive_scenario.ta": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.10333333333333333,+ "macro_f1": 0.10123211331251272,+ "ece": 0.874681700038002,+ "brier": 1.7602726601511418,+ "nll": 22.372137100940222,+ "aurc": 0.9231935952280065,+ "mean_confidence": 0.9780150333713353,+ "acc_at_50_coverage": 0.06666666666666667,+ "acc_at_80_coverage": 0.09583333333333334+ },+ "raw": {+ "n": 300,+ "accuracy": 0.10333333333333333,+ "macro_f1": 0.10123211331251272,+ "ece": 0.5694229234759011,+ "brier": 1.3300067700786005,+ "nll": 4.4484547544084645,+ "aurc": 0.9140831228744111,+ "mean_confidence": 0.6727562568092345,+ "acc_at_50_coverage": 0.06666666666666667,+ "acc_at_80_coverage": 0.10416666666666667+ },+ "seconds": 3.47,+ "dropped_questions": 0,+ "questions_per_second": 86.5+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.33666666666666667,+ "macro_f1": 0.3252572891630063,+ "ece": 0.42554588538467264,+ "brier": 1.0566269769548338,+ "nll": 3.5072212892827976,+ "aurc": 0.4944285972644023,+ "mean_confidence": 0.7619340369391754,+ "acc_at_50_coverage": 0.44666666666666666,+ "acc_at_80_coverage": 0.37083333333333335+ },+ "raw": {+ "n": 300,+ "accuracy": 0.33666666666666667,+ "macro_f1": 0.3252572891630063,+ "ece": 0.42554588538467264,+ "brier": 1.0566269769548338,+ "nll": 3.5072212892827976,+ "aurc": 0.4944285972644023,+ "mean_confidence": 0.7619340369391754,+ "acc_at_50_coverage": 0.44666666666666666,+ "acc_at_80_coverage": 0.37083333333333335+ },+ "seconds": 1.05,+ "dropped_questions": 0,+ "questions_per_second": 285.1+ }+ },+ "massive_scenario.zh-CN": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.4866666666666667,+ "macro_f1": 0.4818270425098754,+ "ece": 0.49165408019083584,+ "brier": 1.003183435429874,+ "nll": 11.86876499252355,+ "aurc": 0.37479171862970995,+ "mean_confidence": 0.9783207468575025,+ "acc_at_50_coverage": 0.62,+ "acc_at_80_coverage": 0.525+ },+ "raw": {+ "n": 300,+ "accuracy": 0.4866666666666667,+ "macro_f1": 0.4818270425098754,+ "ece": 0.32380128046122,+ "brier": 0.817820474064045,+ "nll": 2.3817626237262144,+ "aurc": 0.31927257748382765,+ "mean_confidence": 0.8055432886305594,+ "acc_at_50_coverage": 0.6266666666666667,+ "acc_at_80_coverage": 0.5375+ },+ "seconds": 2.44,+ "dropped_questions": 0,+ "questions_per_second": 123.1+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.49666666666666665,+ "macro_f1": 0.4946656595501705,+ "ece": 0.37089746453804967,+ "brier": 0.8238693890006955,+ "nll": 2.7291576173922643,+ "aurc": 0.26099321675377096,+ "mean_confidence": 0.8675641312047164,+ "acc_at_50_coverage": 0.72,+ "acc_at_80_coverage": 0.575+ },+ "raw": {+ "n": 300,+ "accuracy": 0.49666666666666665,+ "macro_f1": 0.4946656595501705,+ "ece": 0.37089746453804967,+ "brier": 0.8238693890006955,+ "nll": 2.7291576173922643,+ "aurc": 0.26099321675377096,+ "mean_confidence": 0.8675641312047164,+ "acc_at_50_coverage": 0.72,+ "acc_at_80_coverage": 0.575+ },+ "seconds": 0.91,+ "dropped_questions": 0,+ "questions_per_second": 329.1+ }+ },+ "massive_scenario.ja": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.4666666666666667,+ "macro_f1": 0.463871866192089,+ "ece": 0.5111735469099957,+ "brier": 1.0434086140512933,+ "nll": 11.8613739437226,+ "aurc": 0.36754816891195335,+ "mean_confidence": 0.9778402135766623,+ "acc_at_50_coverage": 0.64,+ "acc_at_80_coverage": 0.5208333333333334+ },+ "raw": {+ "n": 300,+ "accuracy": 0.4666666666666667,+ "macro_f1": 0.463871866192089,+ "ece": 0.31980380234551753,+ "brier": 0.808630329162578,+ "nll": 2.293172250253408,+ "aurc": 0.29887623886089426,+ "mean_confidence": 0.7864704690121842,+ "acc_at_50_coverage": 0.6533333333333333,+ "acc_at_80_coverage": 0.5166666666666667+ },+ "seconds": 2.49,+ "dropped_questions": 0,+ "questions_per_second": 120.7+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.5066666666666667,+ "macro_f1": 0.5055826834039463,+ "ece": 0.3361473942723203,+ "brier": 0.8127611007453828,+ "nll": 2.8748737144562644,+ "aurc": 0.2559812861674103,+ "mean_confidence": 0.8428140609389871,+ "acc_at_50_coverage": 0.74,+ "acc_at_80_coverage": 0.5708333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.5066666666666667,+ "macro_f1": 0.5055826834039463,+ "ece": 0.3361473942723203,+ "brier": 0.8127611007453828,+ "nll": 2.8748737144562644,+ "aurc": 0.2559812861674103,+ "mean_confidence": 0.8428140609389871,+ "acc_at_50_coverage": 0.74,+ "acc_at_80_coverage": 0.5708333333333333+ },+ "seconds": 0.92,+ "dropped_questions": 0,+ "questions_per_second": 324.7+ }+ },+ "massive_scenario.ko": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.12666666666666668,+ "macro_f1": 0.12095669847529399,+ "ece": 0.840051417707468,+ "brier": 1.6997114892267453,+ "nll": 21.805391671020374,+ "aurc": 0.8374933680329082,+ "mean_confidence": 0.9667180843741346,+ "acc_at_50_coverage": 0.13333333333333333,+ "acc_at_80_coverage": 0.125+ },+ "raw": {+ "n": 300,+ "accuracy": 0.12666666666666668,+ "macro_f1": 0.12095669847529399,+ "ece": 0.48904771206917413,+ "brier": 1.2507464283639262,+ "nll": 4.361151892016419,+ "aurc": 0.8100520077414101,+ "mean_confidence": 0.6114450666752915,+ "acc_at_50_coverage": 0.13333333333333333,+ "acc_at_80_coverage": 0.12916666666666668+ },+ "seconds": 2.62,+ "dropped_questions": 0,+ "questions_per_second": 114.5+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.42333333333333334,+ "macro_f1": 0.4132135713403089,+ "ece": 0.384623058336728,+ "brier": 0.9237245564942307,+ "nll": 3.1511491600254296,+ "aurc": 0.3293478211036618,+ "mean_confidence": 0.8079563916700613,+ "acc_at_50_coverage": 0.6133333333333333,+ "acc_at_80_coverage": 0.4875+ },+ "raw": {+ "n": 300,+ "accuracy": 0.42333333333333334,+ "macro_f1": 0.4132135713403089,+ "ece": 0.384623058336728,+ "brier": 0.9237245564942307,+ "nll": 3.1511491600254296,+ "aurc": 0.3293478211036618,+ "mean_confidence": 0.8079563916700613,+ "acc_at_50_coverage": 0.6133333333333333,+ "acc_at_80_coverage": 0.4875+ },+ "seconds": 0.97,+ "dropped_questions": 0,+ "questions_per_second": 309.4+ }+ },+ "massive_scenario.sw": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.1,+ "macro_f1": 0.0957338933732812,+ "ece": 0.8603138356126518,+ "brier": 1.7409367961677737,+ "nll": 20.656402774670834,+ "aurc": 0.8851119406852863,+ "mean_confidence": 0.9603138356126518,+ "acc_at_50_coverage": 0.10666666666666667,+ "acc_at_80_coverage": 0.0875+ },+ "raw": {+ "n": 300,+ "accuracy": 0.1,+ "macro_f1": 0.0957338933732812,+ "ece": 0.4739598046143587,+ "brier": 1.219804489946537,+ "nll": 4.306388937324703,+ "aurc": 0.88140076168849,+ "mean_confidence": 0.5739598046143588,+ "acc_at_50_coverage": 0.1,+ "acc_at_80_coverage": 0.09166666666666666+ },+ "seconds": 2.37,+ "dropped_questions": 0,+ "questions_per_second": 126.8+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.3,+ "macro_f1": 0.29610211511953605,+ "ece": 0.48034089961981297,+ "brier": 1.1406992541134244,+ "nll": 4.430710868073057,+ "aurc": 0.6298565574065912,+ "mean_confidence": 0.771683877944962,+ "acc_at_50_coverage": 0.38,+ "acc_at_80_coverage": 0.32916666666666666+ },+ "raw": {+ "n": 300,+ "accuracy": 0.3,+ "macro_f1": 0.29610211511953605,+ "ece": 0.48034089961981297,+ "brier": 1.1406992541134244,+ "nll": 4.430710868073057,+ "aurc": 0.6298565574065912,+ "mean_confidence": 0.771683877944962,+ "acc_at_50_coverage": 0.38,+ "acc_at_80_coverage": 0.32916666666666666+ },+ "seconds": 1.15,+ "dropped_questions": 0,+ "questions_per_second": 260.4+ }+ },+ "xnli.en": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.86,+ "macro_f1": 0.8611876188602533,+ "ece": 0.06851500868396258,+ "brier": 0.22050539483309456,+ "nll": 0.39876324404370733,+ "aurc": 0.03817391536388231,+ "mean_confidence": 0.920764748584588,+ "acc_at_50_coverage": 0.98,+ "acc_at_80_coverage": 0.9208333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.86,+ "macro_f1": 0.8611876188602533,+ "ece": 0.11742718550023877,+ "brier": 0.24798918341393375,+ "nll": 0.5741518550133503,+ "aurc": 0.03948252276018887,+ "mean_confidence": 0.9737337063751195,+ "acc_at_50_coverage": 0.98,+ "acc_at_80_coverage": 0.9166666666666666+ },+ "seconds": 2.14,+ "dropped_questions": 0,+ "questions_per_second": 139.9+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.8433333333333334,+ "macro_f1": 0.843059132192939,+ "ece": 0.08620842684402336,+ "brier": 0.23008455799877037,+ "nll": 0.4457800463487143,+ "aurc": 0.03935656889260041,+ "mean_confidence": 0.9294384876362892,+ "acc_at_50_coverage": 0.9733333333333334,+ "acc_at_80_coverage": 0.925+ },+ "raw": {+ "n": 300,+ "accuracy": 0.8433333333333334,+ "macro_f1": 0.843059132192939,+ "ece": 0.08620842684402336,+ "brier": 0.23008455799877037,+ "nll": 0.4457800463487143,+ "aurc": 0.03935656889260041,+ "mean_confidence": 0.9294384876362892,+ "acc_at_50_coverage": 0.9733333333333334,+ "acc_at_80_coverage": 0.925+ },+ "seconds": 0.9,+ "dropped_questions": 0,+ "questions_per_second": 331.5+ }+ },+ "xnli.de": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.64,+ "macro_f1": 0.6326035374850881,+ "ece": 0.21072499301639186,+ "brier": 0.5388959699858508,+ "nll": 0.9682222930794052,+ "aurc": 0.20711301581726796,+ "mean_confidence": 0.8426243028174698,+ "acc_at_50_coverage": 0.8066666666666666,+ "acc_at_80_coverage": 0.7166666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.64,+ "macro_f1": 0.6326035374850881,+ "ece": 0.29767984180382207,+ "brier": 0.6210199479399825,+ "nll": 1.454373830978189,+ "aurc": 0.20956478611441695,+ "mean_confidence": 0.9301385123819258,+ "acc_at_50_coverage": 0.8,+ "acc_at_80_coverage": 0.7166666666666667+ },+ "seconds": 2.56,+ "dropped_questions": 0,+ "questions_per_second": 117.4+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.7933333333333333,+ "macro_f1": 0.7929169579018676,+ "ece": 0.13507943353849292,+ "brier": 0.3391052729394329,+ "nll": 0.7604748461013707,+ "aurc": 0.08005420779482897,+ "mean_confidence": 0.9235221794766247,+ "acc_at_50_coverage": 0.9133333333333333,+ "acc_at_80_coverage": 0.8666666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.7933333333333333,+ "macro_f1": 0.7929169579018676,+ "ece": 0.13507943353849292,+ "brier": 0.3391052729394329,+ "nll": 0.7604748461013707,+ "aurc": 0.08005420779482897,+ "mean_confidence": 0.9235221794766247,+ "acc_at_50_coverage": 0.9133333333333333,+ "acc_at_80_coverage": 0.8666666666666667+ },+ "seconds": 0.93,+ "dropped_questions": 0,+ "questions_per_second": 321.9+ }+ },+ "xnli.fr": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.68,+ "macro_f1": 0.6767434402445008,+ "ece": 0.18792262418329336,+ "brier": 0.49136694862107977,+ "nll": 0.892734589072878,+ "aurc": 0.16347816642024618,+ "mean_confidence": 0.8642486447883189,+ "acc_at_50_coverage": 0.8266666666666667,+ "acc_at_80_coverage": 0.7375+ },+ "raw": {+ "n": 300,+ "accuracy": 0.68,+ "macro_f1": 0.6767434402445008,+ "ece": 0.258012379095395,+ "brier": 0.5539348541031027,+ "nll": 1.358595547094244,+ "aurc": 0.16696330194238168,+ "mean_confidence": 0.9380123790953951,+ "acc_at_50_coverage": 0.82,+ "acc_at_80_coverage": 0.7375+ },+ "seconds": 2.59,+ "dropped_questions": 0,+ "questions_per_second": 115.7+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.7666666666666667,+ "macro_f1": 0.7659139737754787,+ "ece": 0.16007730354280114,+ "brier": 0.36889637585192164,+ "nll": 0.7854969087097513,+ "aurc": 0.08175402414317233,+ "mean_confidence": 0.9220045466100965,+ "acc_at_50_coverage": 0.9333333333333333,+ "acc_at_80_coverage": 0.8583333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.7666666666666667,+ "macro_f1": 0.7659139737754787,+ "ece": 0.16007730354280114,+ "brier": 0.36889637585192164,+ "nll": 0.7854969087097513,+ "aurc": 0.08175402414317233,+ "mean_confidence": 0.9220045466100965,+ "acc_at_50_coverage": 0.9333333333333333,+ "acc_at_80_coverage": 0.8583333333333333+ },+ "seconds": 1.01,+ "dropped_questions": 0,+ "questions_per_second": 297.7+ }+ },+ "xnli.es": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.6833333333333333,+ "macro_f1": 0.6818501257866464,+ "ece": 0.17655950701994094,+ "brier": 0.47418149963116757,+ "nll": 0.8472352910712136,+ "aurc": 0.14227331792807743,+ "mean_confidence": 0.8598928403532743,+ "acc_at_50_coverage": 0.8666666666666667,+ "acc_at_80_coverage": 0.7583333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.6833333333333333,+ "macro_f1": 0.6818501257866464,+ "ece": 0.25490116584860123,+ "brier": 0.544935497456347,+ "nll": 1.2691842047169257,+ "aurc": 0.14385415213323569,+ "mean_confidence": 0.9382344991819344,+ "acc_at_50_coverage": 0.8666666666666667,+ "acc_at_80_coverage": 0.75+ },+ "seconds": 2.44,+ "dropped_questions": 0,+ "questions_per_second": 123.1+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.8,+ "macro_f1": 0.800149169688347,+ "ece": 0.13461852130061025,+ "brier": 0.32386057803816204,+ "nll": 0.6853079173514695,+ "aurc": 0.07282326021925968,+ "mean_confidence": 0.9188244095771106,+ "acc_at_50_coverage": 0.9533333333333334,+ "acc_at_80_coverage": 0.8708333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.8,+ "macro_f1": 0.800149169688347,+ "ece": 0.13461852130061025,+ "brier": 0.32386057803816204,+ "nll": 0.6853079173514695,+ "aurc": 0.07282326021925968,+ "mean_confidence": 0.9188244095771106,+ "acc_at_50_coverage": 0.9533333333333334,+ "acc_at_80_coverage": 0.8708333333333333+ },+ "seconds": 0.95,+ "dropped_questions": 0,+ "questions_per_second": 314.9+ }+ },+ "xnli.ru": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.61,+ "macro_f1": 0.6006412950587708,+ "ece": 0.2168477327377505,+ "brier": 0.6069292023023662,+ "nll": 1.1253612446761034,+ "aurc": 0.26181387796722144,+ "mean_confidence": 0.8262401584516464,+ "acc_at_50_coverage": 0.7066666666666667,+ "acc_at_80_coverage": 0.65+ },+ "raw": {+ "n": 300,+ "accuracy": 0.61,+ "macro_f1": 0.6006412950587708,+ "ece": 0.30976518905362826,+ "brier": 0.6837205620638928,+ "nll": 1.7063311206038916,+ "aurc": 0.26347910681159836,+ "mean_confidence": 0.9172463227137924,+ "acc_at_50_coverage": 0.7066666666666667,+ "acc_at_80_coverage": 0.65+ },+ "seconds": 3.11,+ "dropped_questions": 0,+ "questions_per_second": 96.5+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.74,+ "macro_f1": 0.734120855267125,+ "ece": 0.17782000732522102,+ "brier": 0.42816474330946314,+ "nll": 1.04445604970244,+ "aurc": 0.13628990016553602,+ "mean_confidence": 0.9120620401690492,+ "acc_at_50_coverage": 0.8733333333333333,+ "acc_at_80_coverage": 0.8083333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.74,+ "macro_f1": 0.734120855267125,+ "ece": 0.17782000732522102,+ "brier": 0.42816474330946314,+ "nll": 1.04445604970244,+ "aurc": 0.13628990016553602,+ "mean_confidence": 0.9120620401690492,+ "acc_at_50_coverage": 0.8733333333333333,+ "acc_at_80_coverage": 0.8083333333333333+ },+ "seconds": 0.99,+ "dropped_questions": 0,+ "questions_per_second": 302.3+ }+ },+ "xnli.tr": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.38333333333333336,+ "macro_f1": 0.3659172368849788,+ "ece": 0.3563651548153499,+ "brier": 0.8459639079784079,+ "nll": 1.4686604597336481,+ "aurc": 0.535768027732775,+ "mean_confidence": 0.7396984881486833,+ "acc_at_50_coverage": 0.4666666666666667,+ "acc_at_80_coverage": 0.4125+ },+ "raw": {+ "n": 300,+ "accuracy": 0.38333333333333336,+ "macro_f1": 0.3659172368849788,+ "ece": 0.4763532577379035,+ "brier": 0.9961971299685647,+ "nll": 2.178321712537001,+ "aurc": 0.5370329291137014,+ "mean_confidence": 0.859686591071237,+ "acc_at_50_coverage": 0.46,+ "acc_at_80_coverage": 0.4125+ },+ "seconds": 2.78,+ "dropped_questions": 0,+ "questions_per_second": 107.8+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.74,+ "macro_f1": 0.7375732440652106,+ "ece": 0.19369106633776462,+ "brier": 0.4476175522553707,+ "nll": 1.0262754243753656,+ "aurc": 0.137635591019981,+ "mean_confidence": 0.9240012000590464,+ "acc_at_50_coverage": 0.8933333333333333,+ "acc_at_80_coverage": 0.7916666666666666+ },+ "raw": {+ "n": 300,+ "accuracy": 0.74,+ "macro_f1": 0.7375732440652106,+ "ece": 0.19369106633776462,+ "brier": 0.4476175522553707,+ "nll": 1.0262754243753656,+ "aurc": 0.137635591019981,+ "mean_confidence": 0.9240012000590464,+ "acc_at_50_coverage": 0.8933333333333333,+ "acc_at_80_coverage": 0.7916666666666666+ },+ "seconds": 0.93,+ "dropped_questions": 0,+ "questions_per_second": 322.2+ }+ },+ "xnli.ar": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.44,+ "macro_f1": 0.4134423398629324,+ "ece": 0.28478987850626714,+ "brier": 0.7813091075197995,+ "nll": 1.3379267752927917,+ "aurc": 0.4768315716677202,+ "mean_confidence": 0.7147199743240977,+ "acc_at_50_coverage": 0.48,+ "acc_at_80_coverage": 0.45416666666666666+ },+ "raw": {+ "n": 300,+ "accuracy": 0.44,+ "macro_f1": 0.4134423398629324,+ "ece": 0.4083698884381125,+ "brier": 0.9167379504417611,+ "nll": 1.9116379283319231,+ "aurc": 0.47824095488177326,+ "mean_confidence": 0.8399337873849,+ "acc_at_50_coverage": 0.48,+ "acc_at_80_coverage": 0.45+ },+ "seconds": 3.32,+ "dropped_questions": 0,+ "questions_per_second": 90.5+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.7266666666666667,+ "macro_f1": 0.7261659274776471,+ "ece": 0.19939197696989705,+ "brier": 0.4625657093860309,+ "nll": 1.09160837839155,+ "aurc": 0.12492089138118848,+ "mean_confidence": 0.916744606923775,+ "acc_at_50_coverage": 0.8733333333333333,+ "acc_at_80_coverage": 0.775+ },+ "raw": {+ "n": 300,+ "accuracy": 0.7266666666666667,+ "macro_f1": 0.7261659274776471,+ "ece": 0.19939197696989705,+ "brier": 0.4625657093860309,+ "nll": 1.09160837839155,+ "aurc": 0.12492089138118848,+ "mean_confidence": 0.916744606923775,+ "acc_at_50_coverage": 0.8733333333333333,+ "acc_at_80_coverage": 0.775+ },+ "seconds": 0.98,+ "dropped_questions": 0,+ "questions_per_second": 307.2+ }+ },+ "xnli.hi": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.4,+ "macro_f1": 0.39253301861737616,+ "ece": 0.29034949231012497,+ "brier": 0.8294244668128908,+ "nll": 1.3949523678726539,+ "aurc": 0.5942668839361914,+ "mean_confidence": 0.6769456621737453,+ "acc_at_50_coverage": 0.36666666666666664,+ "acc_at_80_coverage": 0.37916666666666665+ },+ "raw": {+ "n": 300,+ "accuracy": 0.4,+ "macro_f1": 0.39253301861737616,+ "ece": 0.4142800361803396,+ "brier": 0.9722077032311629,+ "nll": 1.9630018094084252,+ "aurc": 0.6002955569729772,+ "mean_confidence": 0.8005386005987778,+ "acc_at_50_coverage": 0.36666666666666664,+ "acc_at_80_coverage": 0.3875+ },+ "seconds": 4.37,+ "dropped_questions": 0,+ "questions_per_second": 68.6+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.66,+ "macro_f1": 0.6568312214327381,+ "ece": 0.24207442048070107,+ "brier": 0.5353331393161586,+ "nll": 1.2911493476619622,+ "aurc": 0.1737586182765312,+ "mean_confidence": 0.9020744204807011,+ "acc_at_50_coverage": 0.8,+ "acc_at_80_coverage": 0.7333333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.66,+ "macro_f1": 0.6568312214327381,+ "ece": 0.24207442048070107,+ "brier": 0.5353331393161586,+ "nll": 1.2911493476619622,+ "aurc": 0.1737586182765312,+ "mean_confidence": 0.9020744204807011,+ "acc_at_50_coverage": 0.8,+ "acc_at_80_coverage": 0.7333333333333333+ },+ "seconds": 1.19,+ "dropped_questions": 0,+ "questions_per_second": 252.8+ }+ },+ "xnli.ur": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.37333333333333335,+ "macro_f1": 0.3356565779421737,+ "ece": 0.32197986258438044,+ "brier": 0.8422166660081521,+ "nll": 1.4332254995755682,+ "aurc": 0.549492087108305,+ "mean_confidence": 0.6953131959177138,+ "acc_at_50_coverage": 0.4066666666666667,+ "acc_at_80_coverage": 0.375+ },+ "raw": {+ "n": 300,+ "accuracy": 0.37333333333333335,+ "macro_f1": 0.3356565779421737,+ "ece": 0.4421501948612806,+ "brier": 0.9917179183917054,+ "nll": 2.0592524399269077,+ "aurc": 0.5501051189999518,+ "mean_confidence": 0.815483528194614,+ "acc_at_50_coverage": 0.4066666666666667,+ "acc_at_80_coverage": 0.375+ },+ "seconds": 4.2,+ "dropped_questions": 0,+ "questions_per_second": 71.4+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.6833333333333333,+ "macro_f1": 0.6796582905834275,+ "ece": 0.21653475758448035,+ "brier": 0.5381491316014443,+ "nll": 1.2558800480679206,+ "aurc": 0.18656783659812115,+ "mean_confidence": 0.8955365350700574,+ "acc_at_50_coverage": 0.8133333333333334,+ "acc_at_80_coverage": 0.7291666666666666+ },+ "raw": {+ "n": 300,+ "accuracy": 0.6833333333333333,+ "macro_f1": 0.6796582905834275,+ "ece": 0.21653475758448035,+ "brier": 0.5381491316014443,+ "nll": 1.2558800480679206,+ "aurc": 0.18656783659812115,+ "mean_confidence": 0.8955365350700574,+ "acc_at_50_coverage": 0.8133333333333334,+ "acc_at_80_coverage": 0.7291666666666666+ },+ "seconds": 1.17,+ "dropped_questions": 0,+ "questions_per_second": 256.3+ }+ },+ "xnli.vi": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.51,+ "macro_f1": 0.5048020259774798,+ "ece": 0.19635322556599558,+ "brier": 0.6321583135727612,+ "nll": 1.0817933431023652,+ "aurc": 0.35687656187199907,+ "mean_confidence": 0.7063532255659957,+ "acc_at_50_coverage": 0.6266666666666667,+ "acc_at_80_coverage": 0.5541666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.51,+ "macro_f1": 0.5048020259774798,+ "ece": 0.31247173288896124,+ "brier": 0.7290198204606503,+ "nll": 1.4564757735641376,+ "aurc": 0.35803749330841206,+ "mean_confidence": 0.8224717328889614,+ "acc_at_50_coverage": 0.6466666666666666,+ "acc_at_80_coverage": 0.55+ },+ "seconds": 3.53,+ "dropped_questions": 0,+ "questions_per_second": 84.9+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.7233333333333334,+ "macro_f1": 0.7163319041978768,+ "ece": 0.20742967358049616,+ "brier": 0.4767080414544521,+ "nll": 1.1728364879399578,+ "aurc": 0.14809157215092492,+ "mean_confidence": 0.9307630069138296,+ "acc_at_50_coverage": 0.8533333333333334,+ "acc_at_80_coverage": 0.7625+ },+ "raw": {+ "n": 300,+ "accuracy": 0.7233333333333334,+ "macro_f1": 0.7163319041978768,+ "ece": 0.20742967358049616,+ "brier": 0.4767080414544521,+ "nll": 1.1728364879399578,+ "aurc": 0.14809157215092492,+ "mean_confidence": 0.9307630069138296,+ "acc_at_50_coverage": 0.8533333333333334,+ "acc_at_80_coverage": 0.7625+ },+ "seconds": 0.98,+ "dropped_questions": 0,+ "questions_per_second": 306.7+ }+ },+ "xnli.th": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.4166666666666667,+ "macro_f1": 0.41597547598264956,+ "ece": 0.18541593372815865,+ "brier": 0.7150968571448711,+ "nll": 1.185035326280894,+ "aurc": 0.49604564931203415,+ "mean_confidence": 0.6020826003948253,+ "acc_at_50_coverage": 0.4666666666666667,+ "acc_at_80_coverage": 0.4375+ },+ "raw": {+ "n": 300,+ "accuracy": 0.4166666666666667,+ "macro_f1": 0.41597547598264956,+ "ece": 0.30600608264125995,+ "brier": 0.8224991434988397,+ "nll": 1.4917845671280279,+ "aurc": 0.4956985674942147,+ "mean_confidence": 0.7226727493079265,+ "acc_at_50_coverage": 0.4533333333333333,+ "acc_at_80_coverage": 0.43333333333333335+ },+ "seconds": 4.31,+ "dropped_questions": 0,+ "questions_per_second": 69.5+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.6966666666666667,+ "macro_f1": 0.6869609915895595,+ "ece": 0.21061457463296523,+ "brier": 0.4997460636887329,+ "nll": 1.1624382390052939,+ "aurc": 0.15457771919077964,+ "mean_confidence": 0.9071540187085024,+ "acc_at_50_coverage": 0.82,+ "acc_at_80_coverage": 0.7666666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.6966666666666667,+ "macro_f1": 0.6869609915895595,+ "ece": 0.21061457463296523,+ "brier": 0.4997460636887329,+ "nll": 1.1624382390052939,+ "aurc": 0.15457771919077964,+ "mean_confidence": 0.9071540187085024,+ "acc_at_50_coverage": 0.82,+ "acc_at_80_coverage": 0.7666666666666667+ },+ "seconds": 1.04,+ "dropped_questions": 0,+ "questions_per_second": 287.6+ }+ },+ "xnli.el": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.5166666666666667,+ "macro_f1": 0.5085754741061143,+ "ece": 0.2158881142658567,+ "brier": 0.6730261233634023,+ "nll": 1.1811545983608287,+ "aurc": 0.3619304897156812,+ "mean_confidence": 0.7313205582371749,+ "acc_at_50_coverage": 0.62,+ "acc_at_80_coverage": 0.5625+ },+ "raw": {+ "n": 300,+ "accuracy": 0.5166666666666667,+ "macro_f1": 0.5085754741061143,+ "ece": 0.334197796750754,+ "brier": 0.7690286317491565,+ "nll": 1.6663537620258704,+ "aurc": 0.3615853025660657,+ "mean_confidence": 0.841674141964363,+ "acc_at_50_coverage": 0.62,+ "acc_at_80_coverage": 0.5666666666666667+ },+ "seconds": 3.37,+ "dropped_questions": 0,+ "questions_per_second": 89.0+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.7266666666666667,+ "macro_f1": 0.7239553451902849,+ "ece": 0.19141782798564486,+ "brier": 0.4543850042303488,+ "nll": 1.0568004891245892,+ "aurc": 0.1251534218900087,+ "mean_confidence": 0.9161183020059976,+ "acc_at_50_coverage": 0.8933333333333333,+ "acc_at_80_coverage": 0.7833333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.7266666666666667,+ "macro_f1": 0.7239553451902849,+ "ece": 0.19141782798564486,+ "brier": 0.4543850042303488,+ "nll": 1.0568004891245892,+ "aurc": 0.1251534218900087,+ "mean_confidence": 0.9161183020059976,+ "acc_at_50_coverage": 0.8933333333333333,+ "acc_at_80_coverage": 0.7833333333333333+ },+ "seconds": 1.28,+ "dropped_questions": 0,+ "questions_per_second": 235.1+ }+ },+ "xnli.bg": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.5433333333333333,+ "macro_f1": 0.5328651753682229,+ "ece": 0.23137607365303675,+ "brier": 0.6454032398069194,+ "nll": 1.1365945276276332,+ "aurc": 0.3029031135998286,+ "mean_confidence": 0.7747094069863701,+ "acc_at_50_coverage": 0.6933333333333334,+ "acc_at_80_coverage": 0.6+ },+ "raw": {+ "n": 300,+ "accuracy": 0.5433333333333333,+ "macro_f1": 0.5328651753682229,+ "ece": 0.3365172510087523,+ "brier": 0.7502977579481567,+ "nll": 1.6499803540167166,+ "aurc": 0.3027396258974141,+ "mean_confidence": 0.8798505843420855,+ "acc_at_50_coverage": 0.68,+ "acc_at_80_coverage": 0.5958333333333333+ },+ "seconds": 3.2,+ "dropped_questions": 0,+ "questions_per_second": 93.9+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.7933333333333333,+ "macro_f1": 0.791961144703296,+ "ece": 0.13341324680488864,+ "brier": 0.3506072942916916,+ "nll": 0.7777559539036778,+ "aurc": 0.0824887284269202,+ "mean_confidence": 0.9240324494058397,+ "acc_at_50_coverage": 0.9266666666666666,+ "acc_at_80_coverage": 0.8583333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.7933333333333333,+ "macro_f1": 0.791961144703296,+ "ece": 0.13341324680488864,+ "brier": 0.3506072942916916,+ "nll": 0.7777559539036778,+ "aurc": 0.0824887284269202,+ "mean_confidence": 0.9240324494058397,+ "acc_at_50_coverage": 0.9266666666666666,+ "acc_at_80_coverage": 0.8583333333333333+ },+ "seconds": 1.05,+ "dropped_questions": 0,+ "questions_per_second": 284.8+ }+ },+ "xnli.zh": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.6733333333333333,+ "macro_f1": 0.6626749561613264,+ "ece": 0.20887249624498655,+ "brier": 0.5226839647931262,+ "nll": 1.0026436713197937,+ "aurc": 0.20776604839660678,+ "mean_confidence": 0.8587428911489872,+ "acc_at_50_coverage": 0.8066666666666666,+ "acc_at_80_coverage": 0.7208333333333333+ },+ "raw": {+ "n": 300,+ "accuracy": 0.6733333333333333,+ "macro_f1": 0.6626749561613264,+ "ece": 0.27118436703316295,+ "brier": 0.5795487241582471,+ "nll": 1.5439941284041712,+ "aurc": 0.20990279905613277,+ "mean_confidence": 0.9333730582682843,+ "acc_at_50_coverage": 0.8,+ "acc_at_80_coverage": 0.7208333333333333+ },+ "seconds": 2.72,+ "dropped_questions": 0,+ "questions_per_second": 110.4+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.7566666666666667,+ "macro_f1": 0.7538368278419627,+ "ece": 0.16077182161775355,+ "brier": 0.39710609854268974,+ "nll": 0.8844138039352938,+ "aurc": 0.11440906432455504,+ "mean_confidence": 0.9092261280780669,+ "acc_at_50_coverage": 0.9,+ "acc_at_80_coverage": 0.8291666666666667+ },+ "raw": {+ "n": 300,+ "accuracy": 0.7566666666666667,+ "macro_f1": 0.7538368278419627,+ "ece": 0.16077182161775355,+ "brier": 0.39710609854268974,+ "nll": 0.8844138039352938,+ "aurc": 0.11440906432455504,+ "mean_confidence": 0.9092261280780669,+ "acc_at_50_coverage": 0.9,+ "acc_at_80_coverage": 0.8291666666666667+ },+ "seconds": 0.88,+ "dropped_questions": 0,+ "questions_per_second": 341.4+ }+ },+ "xnli.sw": {+ "laya": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.42333333333333334,+ "macro_f1": 0.4207847707847708,+ "ece": 0.29236211107175514,+ "brier": 0.7784006863977958,+ "nll": 1.3179917563408299,+ "aurc": 0.5003573847218811,+ "mean_confidence": 0.7024897047792208,+ "acc_at_50_coverage": 0.47333333333333333,+ "acc_at_80_coverage": 0.43333333333333335+ },+ "raw": {+ "n": 300,+ "accuracy": 0.42333333333333334,+ "macro_f1": 0.4207847707847708,+ "ece": 0.41450323613638784,+ "brier": 0.919356157926492,+ "nll": 1.8576785541576135,+ "aurc": 0.5031777992926575,+ "mean_confidence": 0.8296259550336644,+ "acc_at_50_coverage": 0.4666666666666667,+ "acc_at_80_coverage": 0.4375+ },+ "seconds": 2.75,+ "dropped_questions": 0,+ "questions_per_second": 109.1+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 300,+ "accuracy": 0.6266666666666667,+ "macro_f1": 0.6230840663744693,+ "ece": 0.27059308880296506,+ "brier": 0.5848817223991936,+ "nll": 1.3826984215782445,+ "aurc": 0.1918472633863418,+ "mean_confidence": 0.8963267263770844,+ "acc_at_50_coverage": 0.8266666666666667,+ "acc_at_80_coverage": 0.725+ },+ "raw": {+ "n": 300,+ "accuracy": 0.6266666666666667,+ "macro_f1": 0.6230840663744693,+ "ece": 0.27059308880296506,+ "brier": 0.5848817223991936,+ "nll": 1.3826984215782445,+ "aurc": 0.1918472633863418,+ "mean_confidence": 0.8963267263770844,+ "acc_at_50_coverage": 0.8266666666666667,+ "acc_at_80_coverage": 0.725+ },+ "seconds": 1.02,+ "dropped_questions": 0,+ "questions_per_second": 293.7+ }+ },+ "en.sst5": {+ "laya": {+ "calibrated": {+ "n": 600,+ "accuracy": 0.37166666666666665,+ "macro_f1": 0.3288496250901994,+ "ece": 0.2477510941191931,+ "brier": 0.8172136446504027,+ "nll": 1.7292404614651202,+ "aurc": 0.5280437776692383,+ "mean_confidence": 0.6194177607858598,+ "acc_at_50_coverage": 0.45666666666666667,+ "acc_at_80_coverage": 0.4041666666666667+ },+ "raw": {+ "n": 600,+ "accuracy": 0.37166666666666665,+ "macro_f1": 0.3288496250901994,+ "ece": 0.3005899997646033,+ "brier": 0.8626452163129017,+ "nll": 1.9530768413531427,+ "aurc": 0.5350255397889842,+ "mean_confidence": 0.67225666643127,+ "acc_at_50_coverage": 0.46,+ "acc_at_80_coverage": 0.39375+ },+ "seconds": 3.19,+ "dropped_questions": 0,+ "questions_per_second": 188.0,+ "score_mae": 0.9045839813495934,+ "within_1": 0.6316666666666667+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 600,+ "accuracy": 0.2816666666666667,+ "macro_f1": 0.20517704230754683,+ "ece": 0.5233828299976504,+ "brier": 1.1825193085471115,+ "nll": 3.8489351560969514,+ "aurc": 0.6864556453513037,+ "mean_confidence": 0.804672039949155,+ "acc_at_50_coverage": 0.2866666666666667,+ "acc_at_80_coverage": 0.28541666666666665+ },+ "raw": {+ "n": 600,+ "accuracy": 0.2816666666666667,+ "macro_f1": 0.20517704230754683,+ "ece": 0.5233828299976504,+ "brier": 1.1825193085471115,+ "nll": 3.8489351560969514,+ "aurc": 0.6864556453513037,+ "mean_confidence": 0.804672039949155,+ "acc_at_50_coverage": 0.2866666666666667,+ "acc_at_80_coverage": 0.28541666666666665+ },+ "seconds": 1.53,+ "dropped_questions": 0,+ "questions_per_second": 391.0,+ "score_mae": 1.3593954707085403,+ "within_1": 0.45+ }+ },+ "en.emotion": {+ "laya": {+ "calibrated": {+ "n": 600,+ "accuracy": 0.5733333333333334,+ "macro_f1": 0.47631269173299584,+ "ece": 0.32375073910645563,+ "brier": 0.7305027390348787,+ "nll": 2.144856609363522,+ "aurc": 0.28845213093546057,+ "mean_confidence": 0.8963017901835962,+ "acc_at_50_coverage": 0.6866666666666666,+ "acc_at_80_coverage": 0.6145833333333334+ },+ "raw": {+ "n": 600,+ "accuracy": 0.5733333333333334,+ "macro_f1": 0.47631269173299584,+ "ece": 0.3237525791408232,+ "brier": 0.7305045932922586,+ "nll": 2.1448862404580455,+ "aurc": 0.28845213093546057,+ "mean_confidence": 0.8963036515560666,+ "acc_at_50_coverage": 0.6866666666666666,+ "acc_at_80_coverage": 0.6145833333333334+ },+ "seconds": 2.46,+ "dropped_questions": 0,+ "questions_per_second": 243.9+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 600,+ "accuracy": 0.5133333333333333,+ "macro_f1": 0.42551959178754956,+ "ece": 0.35195398913875425,+ "brier": 0.8182858999827102,+ "nll": 2.3635000382236657,+ "aurc": 0.37013803812950735,+ "mean_confidence": 0.8641186323647112,+ "acc_at_50_coverage": 0.6066666666666667,+ "acc_at_80_coverage": 0.5479166666666667+ },+ "raw": {+ "n": 600,+ "accuracy": 0.5133333333333333,+ "macro_f1": 0.42551959178754956,+ "ece": 0.35195398913875425,+ "brier": 0.8182858999827102,+ "nll": 2.3635000382236657,+ "aurc": 0.37013803812950735,+ "mean_confidence": 0.8641186323647112,+ "acc_at_50_coverage": 0.6066666666666667,+ "acc_at_80_coverage": 0.5479166666666667+ },+ "seconds": 1.12,+ "dropped_questions": 0,+ "questions_per_second": 535.6+ }+ },+ "en.prompt_injections": {+ "laya": {+ "calibrated": {+ "n": 116,+ "accuracy": 0.6982758620689655,+ "macro_f1": 0.6750700280112045,+ "ece": 0.261929815155607,+ "brier": 0.5258464578822987,+ "nll": 1.6828205154616658,+ "aurc": 0.10620459144410163,+ "mean_confidence": 0.9602056772245726,+ "acc_at_50_coverage": 0.9310344827586207,+ "acc_at_80_coverage": 0.7717391304347826+ },+ "raw": {+ "n": 116,+ "accuracy": 0.6982758620689655,+ "macro_f1": 0.6750700280112045,+ "ece": 0.29121353183347143,+ "brier": 0.5699061678546038,+ "nll": 3.266111151889796,+ "aurc": 0.10620459144410163,+ "mean_confidence": 0.9826727110716177,+ "acc_at_50_coverage": 0.9310344827586207,+ "acc_at_80_coverage": 0.7717391304347826+ },+ "seconds": 1.23,+ "dropped_questions": 0,+ "questions_per_second": 94.6+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 116,+ "accuracy": 0.5775862068965517,+ "macro_f1": 0.5297426987672706,+ "ece": 0.3622712910805021,+ "brier": 0.7188281372535157,+ "nll": 1.83371785008595,+ "aurc": 0.2611079846570448,+ "mean_confidence": 0.9398574979770538,+ "acc_at_50_coverage": 0.7931034482758621,+ "acc_at_80_coverage": 0.6413043478260869+ },+ "raw": {+ "n": 116,+ "accuracy": 0.5775862068965517,+ "macro_f1": 0.5297426987672706,+ "ece": 0.3622712910805021,+ "brier": 0.7188281372535157,+ "nll": 1.83371785008595,+ "aurc": 0.2611079846570448,+ "mean_confidence": 0.9398574979770538,+ "acc_at_50_coverage": 0.7931034482758621,+ "acc_at_80_coverage": 0.6413043478260869+ },+ "seconds": 0.56,+ "dropped_questions": 0,+ "questions_per_second": 206.6+ }+ },+ "en.ag_news": {+ "laya": {+ "calibrated": {+ "n": 600,+ "accuracy": 0.9466666666666667,+ "macro_f1": 0.9438349780251156,+ "ece": 0.039366815217098554,+ "brier": 0.08696698705958933,+ "nll": 0.17319417764130313,+ "aurc": 0.0068207233371325145,+ "mean_confidence": 0.9214515425079212,+ "acc_at_50_coverage": 1.0,+ "acc_at_80_coverage": 0.9875+ },+ "raw": {+ "n": 600,+ "accuracy": 0.9466666666666667,+ "macro_f1": 0.9438349780251156,+ "ece": 0.033209929898696006,+ "brier": 0.09122999613305266,+ "nll": 0.17802561050018714,+ "aurc": 0.0066994794605537205,+ "mean_confidence": 0.9726261761504992,+ "acc_at_50_coverage": 1.0,+ "acc_at_80_coverage": 0.9895833333333334+ },+ "seconds": 4.66,+ "dropped_questions": 0,+ "questions_per_second": 128.8+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 600,+ "accuracy": 0.9366666666666666,+ "macro_f1": 0.9339174627621376,+ "ece": 0.03790626053297799,+ "brier": 0.09748790744422886,+ "nll": 0.2233060249382501,+ "aurc": 0.015727048809451808,+ "mean_confidence": 0.9727651926995691,+ "acc_at_50_coverage": 0.9866666666666667,+ "acc_at_80_coverage": 0.9854166666666667+ },+ "raw": {+ "n": 600,+ "accuracy": 0.9366666666666666,+ "macro_f1": 0.9339174627621376,+ "ece": 0.03790626053297799,+ "brier": 0.09748790744422886,+ "nll": 0.2233060249382501,+ "aurc": 0.015727048809451808,+ "mean_confidence": 0.9727651926995691,+ "acc_at_50_coverage": 0.9866666666666667,+ "acc_at_80_coverage": 0.9854166666666667+ },+ "seconds": 2.13,+ "dropped_questions": 0,+ "questions_per_second": 281.3+ }+ },+ "en.boolq": {+ "laya": {+ "calibrated": {+ "n": 600,+ "accuracy": 0.83,+ "macro_f1": 0.8196095076400679,+ "ece": 0.09610106838662799,+ "brier": 0.2731780696217012,+ "nll": 0.46556886296991934,+ "aurc": 0.07288424964326473,+ "mean_confidence": 0.9161033363989168,+ "acc_at_50_coverage": 0.9366666666666666,+ "acc_at_80_coverage": 0.8833333333333333+ },+ "raw": {+ "n": 600,+ "accuracy": 0.83,+ "macro_f1": 0.8196095076400679,+ "ece": 0.1497538899574864,+ "brier": 0.3128681086108246,+ "nll": 0.7678904569264094,+ "aurc": 0.07288424964326473,+ "mean_confidence": 0.9715743387844664,+ "acc_at_50_coverage": 0.9366666666666666,+ "acc_at_80_coverage": 0.8833333333333333+ },+ "seconds": 8.36,+ "dropped_questions": 0,+ "questions_per_second": 71.8+ },+ "laya-multilingual": {+ "calibrated": {+ "n": 600,+ "accuracy": 0.7866666666666666,+ "macro_f1": 0.7732399522859066,+ "ece": 0.14529158506642545,+ "brier": 0.35721659783636767,+ "nll": 0.7001630668886732,+ "aurc": 0.1102122813350894,+ "mean_confidence": 0.9255312262359776,+ "acc_at_50_coverage": 0.8933333333333333,+ "acc_at_80_coverage": 0.8270833333333333+ },+ "raw": {+ "n": 600,+ "accuracy": 0.7866666666666666,+ "macro_f1": 0.7732399522859066,+ "ece": 0.14529158506642545,+ "brier": 0.35721659783636767,+ "nll": 0.7001630668886732,+ "aurc": 0.1102122813350894,+ "mean_confidence": 0.9255312262359776,+ "acc_at_50_coverage": 0.8933333333333333,+ "acc_at_80_coverage": 0.8270833333333333+ },+ "seconds": 3.98,+ "dropped_questions": 0,+ "questions_per_second": 150.8+ }+ }+ },+ "typed_decisions_matched_512": {+ "laya": {+ "n": 2000,+ "accuracy": 0.362,+ "macro_f1": 0.3273577076400338,+ "ece": 0.17410090455022942,+ "brier": 0.7496143221875959,+ "nll": 1.3224634387781617,+ "aurc": 0.5928492917684455,+ "mean_confidence": 0.5361009045502294,+ "acc_at_50_coverage": 0.427,+ "acc_at_80_coverage": 0.386875+ },+ "laya-multilingual": {+ "n": 2000,+ "accuracy": 0.3495,+ "macro_f1": 0.3481722969312257,+ "ece": 0.3165801393685362,+ "brier": 0.8917435315799593,+ "nll": 1.8267597640548572,+ "aurc": 0.5579990209884584,+ "mean_confidence": 0.6624303215125597,+ "acc_at_50_coverage": 0.414,+ "acc_at_80_coverage": 0.36625+ }+ },+ "latency": {+ "laya": {+ "1_questions": {+ "p50_ms": 39.5,+ "p95_ms": 44.8,+ "ms_per_question": 39.5+ },+ "5_questions": {+ "p50_ms": 84.5,+ "p95_ms": 86.0,+ "ms_per_question": 16.89+ },+ "10_questions": {+ "p50_ms": 158.6,+ "p95_ms": 160.1,+ "ms_per_question": 15.86+ },+ "50_questions": {+ "p50_ms": 771.3,+ "p95_ms": 806.7,+ "ms_per_question": 15.43+ }+ },+ "laya-multilingual": {+ "1_questions": {+ "p50_ms": 32.8,+ "p95_ms": 38.8,+ "ms_per_question": 32.83+ },+ "5_questions": {+ "p50_ms": 40.1,+ "p95_ms": 43.5,+ "ms_per_question": 8.02+ },+ "10_questions": {+ "p50_ms": 72.3,+ "p95_ms": 74.4,+ "ms_per_question": 7.23+ },+ "50_questions": {+ "p50_ms": 337.4,+ "p95_ms": 693.7,+ "ms_per_question": 6.75+ }+ }+ },+ "option_order_robustness": {+ "laya": {+ "massive_intent.en": {+ "n": 200,+ "flip_rate": 0.15+ },+ "en.emotion": {+ "n": 200,+ "flip_rate": 0.04+ },+ "xnli.en": {+ "n": 200,+ "flip_rate": 0.0+ }+ },+ "laya-multilingual": {+ "massive_intent.en": {+ "n": 200,+ "flip_rate": 0.23+ },+ "en.emotion": {+ "n": 200,+ "flip_rate": 0.09+ },+ "xnli.en": {+ "n": 200,+ "flip_rate": 0.015+ }+ }+ },+ "calibration_repair": {+ "laya": {+ "per_suite": {+ "typed_decisions": {+ "n_heldout": 1000,+ "fitted_temperatures": {+ "choice:3-5": 3.9261,+ "noul:2": 2.0205,+ "score:3-5": 3.0698+ },+ "ece_shipped": 0.2071110174317399,+ "ece_refit": 0.12865648553068806,+ "nll_shipped": 1.328208659782999,+ "nll_refit": 1.2397667208426046,+ "accuracy": 0.338+ },+ "massive_intent.en": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 1.5414+ },+ "ece_shipped": 0.2562159827237908,+ "ece_refit": 0.11403699556212168,+ "nll_shipped": 5.865505434392175,+ "nll_refit": 1.0140600000067173,+ "accuracy": 0.74+ },+ "massive_intent.de": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.3419+ },+ "ece_shipped": 0.5462625472109808,+ "ece_refit": 0.09248203952235462,+ "nll_shipped": 12.57482900519753,+ "nll_refit": 1.9823033978431583,+ "accuracy": 0.43333333333333335+ },+ "massive_intent.fr": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 1.9714+ },+ "ece_shipped": 0.5576747681888061,+ "ece_refit": 0.11579583487750908,+ "nll_shipped": 12.113473172991581,+ "nll_refit": 1.6719642883309838,+ "accuracy": 0.41333333333333333+ },+ "massive_intent.es": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.2294+ },+ "ece_shipped": 0.5322226422693049,+ "ece_refit": 0.09213948384507138,+ "nll_shipped": 12.050862077977984,+ "nll_refit": 1.8740372427815475,+ "accuracy": 0.4533333333333333+ },+ "massive_intent.pt": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.5841+ },+ "ece_shipped": 0.5235085119457203,+ "ece_refit": 0.17951008108333782,+ "nll_shipped": 12.221158727957615,+ "nll_refit": 1.8423784639336334,+ "accuracy": 0.4666666666666667+ },+ "massive_intent.ru": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.782+ },+ "ece_shipped": 0.6801221409141907,+ "ece_refit": 0.06785126111487216,+ "nll_shipped": 16.630473789135806,+ "nll_refit": 2.4015938125011953,+ "accuracy": 0.3+ },+ "massive_intent.tr": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 4.664+ },+ "ece_shipped": 0.7696493795726053,+ "ece_refit": 0.06573726881689713,+ "nll_shipped": 20.339742620957256,+ "nll_refit": 2.7917378955701535,+ "accuracy": 0.19333333333333333+ },+ "massive_intent.ar": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 5.4059+ },+ "ece_shipped": 0.7827356558020273,+ "ece_refit": 0.043154654024976154,+ "nll_shipped": 20.663786998267785,+ "nll_refit": 2.881006518148397,+ "accuracy": 0.15333333333333332+ },+ "massive_intent.hi": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 8.8425+ },+ "ece_shipped": 0.854953810628851,+ "ece_refit": 0.037505765311847995,+ "nll_shipped": 20.83134572134706,+ "nll_refit": 2.943494345947451,+ "accuracy": 0.09333333333333334+ },+ "massive_intent.ta": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 5.5405+ },+ "ece_shipped": 0.8039968607819044,+ "ece_refit": 0.03421481533002084,+ "nll_shipped": 19.612842933352653,+ "nll_refit": 2.944986645721345,+ "accuracy": 0.12+ },+ "massive_intent.zh-CN": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.0708+ },+ "ece_shipped": 0.40549573789270327,+ "ece_refit": 0.08950865278434557,+ "nll_shipped": 9.813177433742595,+ "nll_refit": 1.5782438333241908,+ "accuracy": 0.5933333333333334+ },+ "massive_intent.ja": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 1.8311+ },+ "ece_shipped": 0.42395100104886724,+ "ece_refit": 0.1359537248702114,+ "nll_shipped": 10.405710162642912,+ "nll_refit": 1.7061709608589148,+ "accuracy": 0.56+ },+ "massive_intent.ko": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 5.2745+ },+ "ece_shipped": 0.8638654472702805,+ "ece_refit": 0.012764949526979565,+ "nll_shipped": 21.82418989141709,+ "nll_refit": 2.948163107719538,+ "accuracy": 0.09333333333333334+ },+ "massive_intent.sw": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 6.7458+ },+ "ece_shipped": 0.8691068671314688,+ "ece_refit": 0.0202221602304236,+ "nll_shipped": 21.5339170347149,+ "nll_refit": 2.9634040289307078,+ "accuracy": 0.08+ },+ "massive_scenario.en": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.2294+ },+ "ece_shipped": 0.3815547151453086,+ "ece_refit": 0.13146043085039616,+ "nll_shipped": 10.22545791808144,+ "nll_refit": 1.3921974143323663,+ "accuracy": 0.62+ },+ "massive_scenario.de": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.8513+ },+ "ece_shipped": 0.6398830722839038,+ "ece_refit": 0.09444310212762598,+ "nll_shipped": 14.795613787775219,+ "nll_refit": 2.1076492998547542,+ "accuracy": 0.32+ },+ "massive_scenario.fr": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.1752+ },+ "ece_shipped": 0.483678749152581,+ "ece_refit": 0.12165917488468315,+ "nll_shipped": 11.072922732788207,+ "nll_refit": 1.6632025303048328,+ "accuracy": 0.5+ },+ "massive_scenario.es": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.5841+ },+ "ece_shipped": 0.5777992521171543,+ "ece_refit": 0.1005032276156047,+ "nll_shipped": 14.030134220612933,+ "nll_refit": 2.044750694163988,+ "accuracy": 0.4+ },+ "massive_scenario.pt": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.7144+ },+ "ece_shipped": 0.5417289841728091,+ "ece_refit": 0.0973002974443076,+ "nll_shipped": 13.572473995277965,+ "nll_refit": 1.922970066535057,+ "accuracy": 0.44666666666666666+ },+ "massive_scenario.ru": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.7144+ },+ "ece_shipped": 0.6569873694612363,+ "ece_refit": 0.04551537841388301,+ "nll_shipped": 16.419733500668183,+ "nll_refit": 2.3359881701558023,+ "accuracy": 0.32666666666666666+ },+ "massive_scenario.tr": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 4.2268+ },+ "ece_shipped": 0.7061936198618928,+ "ece_refit": 0.06851599123705755,+ "nll_shipped": 17.296401139792014,+ "nll_refit": 2.5363076344706665,+ "accuracy": 0.26+ },+ "massive_scenario.ar": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 4.7801+ },+ "ece_shipped": 0.8163170012715647,+ "ece_refit": 0.0474423323304447,+ "nll_shipped": 20.728188554378566,+ "nll_refit": 2.7979794977740107,+ "accuracy": 0.14666666666666667+ },+ "massive_scenario.hi": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 7.6289+ },+ "ece_shipped": 0.8660562191576803,+ "ece_refit": 0.004584565442225005,+ "nll_shipped": 21.782254412766893,+ "nll_refit": 2.835531392620142,+ "accuracy": 0.1+ },+ "massive_scenario.ta": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 6.1135+ },+ "ece_shipped": 0.8325140639998705,+ "ece_refit": 0.07807608446121986,+ "nll_shipped": 21.53368640920763,+ "nll_refit": 2.8077670852380985,+ "accuracy": 0.14+ },+ "massive_scenario.zh-CN": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.285+ },+ "ece_shipped": 0.46110257402566157,+ "ece_refit": 0.09044395129929148,+ "nll_shipped": 10.77292831665002,+ "nll_refit": 1.6913117170360654,+ "accuracy": 0.5266666666666666+ },+ "massive_scenario.ja": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.1752+ },+ "ece_shipped": 0.4881034787495095,+ "ece_refit": 0.1119030215337075,+ "nll_shipped": 11.324819142335816,+ "nll_refit": 1.644154638119622,+ "accuracy": 0.4866666666666667+ },+ "massive_scenario.ko": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 6.2658+ },+ "ece_shipped": 0.8350570565183021,+ "ece_refit": 0.021565188964428522,+ "nll_shipped": 21.331749553105603,+ "nll_refit": 2.797665270418228,+ "accuracy": 0.13333333333333333+ },+ "massive_scenario.sw": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 6.1135+ },+ "ece_shipped": 0.8135882896986645,+ "ece_refit": 0.028498910373293804,+ "nll_shipped": 20.22271430726738,+ "nll_refit": 2.795054831143482,+ "accuracy": 0.14+ },+ "xnli.en": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 2.4002+ },+ "ece_shipped": 0.07289677206460535,+ "ece_refit": 0.08853303576022854,+ "nll_shipped": 0.3345960511795842,+ "nll_refit": 0.3437815148064849,+ "accuracy": 0.88+ },+ "xnli.de": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 3.4716+ },+ "ece_shipped": 0.22490857541364537,+ "ece_refit": 0.11641844789459176,+ "nll_shipped": 0.9709805880887378,+ "nll_refit": 0.8199119398248624,+ "accuracy": 0.6333333333333333+ },+ "xnli.fr": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 3.6467+ },+ "ece_shipped": 0.16646567713583152,+ "ece_refit": 0.06935764297795377,+ "nll_shipped": 0.8149368257371297,+ "nll_refit": 0.721784713245583,+ "accuracy": 0.6933333333333334+ },+ "xnli.es": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 3.3049+ },+ "ece_shipped": 0.16094019064904574,+ "ece_refit": 0.0710883876192177,+ "nll_shipped": 0.8040389210247584,+ "nll_refit": 0.7178860649223204,+ "accuracy": 0.6866666666666666+ },+ "xnli.ru": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 3.9261+ },+ "ece_shipped": 0.2591002284575789,+ "ece_refit": 0.09264246809537272,+ "nll_shipped": 1.1705031954952403,+ "nll_refit": 0.9191745262303057,+ "accuracy": 0.5866666666666667+ },+ "xnli.tr": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 9.5198+ },+ "ece_shipped": 0.37022719319502834,+ "ece_refit": 0.08047867025819397,+ "nll_shipped": 1.4960379906771686,+ "nll_refit": 1.0828012031471597,+ "accuracy": 0.36666666666666664+ },+ "xnli.ar": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 8.6276+ },+ "ece_shipped": 0.29209360893026326,+ "ece_refit": 0.06329832543949335,+ "nll_shipped": 1.2892057415135658,+ "nll_refit": 1.0539604556939484,+ "accuracy": 0.4533333333333333+ },+ "xnli.hi": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 10.0+ },+ "ece_shipped": 0.2884204812865509,+ "ece_refit": 0.062321224097440786,+ "nll_shipped": 1.3478000099693062,+ "nll_refit": 1.0813943966005897,+ "accuracy": 0.41333333333333333+ },+ "xnli.ur": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 10.0+ },+ "ece_shipped": 0.2886956804091492,+ "ece_refit": 0.031778266269963856,+ "nll_shipped": 1.3295301273849722,+ "nll_refit": 1.0728864514477925,+ "accuracy": 0.4+ },+ "xnli.vi": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 4.1241+ },+ "ece_shipped": 0.15791339705634688,+ "ece_refit": 0.06114141557091057,+ "nll_shipped": 1.0300991074899297,+ "nll_refit": 0.9465837284644331,+ "accuracy": 0.5733333333333334+ },+ "xnli.th": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 5.6785+ },+ "ece_shipped": 0.19427425907207962,+ "ece_refit": 0.043280618209652684,+ "nll_shipped": 1.1883367978775616,+ "nll_refit": 1.0611018511269417,+ "accuracy": 0.42+ },+ "xnli.el": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 5.2745+ },+ "ece_shipped": 0.20863204752259085,+ "ece_refit": 0.05938958684029479,+ "nll_shipped": 1.1206284889886116,+ "nll_refit": 0.9634498438091695,+ "accuracy": 0.5666666666666667+ },+ "xnli.bg": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 3.7375+ },+ "ece_shipped": 0.25795700495898,+ "ece_refit": 0.10558494899380169,+ "nll_shipped": 1.2115903992573755,+ "nll_refit": 0.9892777346276406,+ "accuracy": 0.52+ },+ "xnli.zh": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 3.6467+ },+ "ece_shipped": 0.20970005695297053,+ "ece_refit": 0.09349285486077741,+ "nll_shipped": 1.004990307040937,+ "nll_refit": 0.8142545372650651,+ "accuracy": 0.6866666666666666+ },+ "xnli.sw": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 9.757+ },+ "ece_shipped": 0.27420298315497604,+ "ece_refit": 0.048962289518859725,+ "nll_shipped": 1.2961388406803456,+ "nll_refit": 1.0525570434208937,+ "accuracy": 0.4666666666666667+ },+ "en.sst5": {+ "n_heldout": 300,+ "fitted_temperatures": {+ "score:3-5": 3.5581+ },+ "ece_shipped": 0.23766382422275603,+ "ece_refit": 0.06455799128975741,+ "nll_shipped": 1.6900878358865143,+ "nll_refit": 1.3876706097407598,+ "accuracy": 0.38+ },+ "en.emotion": {+ "n_heldout": 300,+ "fitted_temperatures": {+ "choice:6-10": 2.8513+ },+ "ece_shipped": 0.3776578906246687,+ "ece_refit": 0.13750650494005914,+ "nll_shipped": 2.4668890351766506,+ "nll_refit": 1.324234118429839,+ "accuracy": 0.52+ },+ "en.prompt_injections": {+ "n_heldout": 58,+ "fitted_temperatures": {+ "noul:2": 7.2626+ },+ "ece_shipped": 0.4421022956073672,+ "ece_refit": 0.3291279870455534,+ "nll_shipped": 2.792945084499727,+ "nll_refit": 0.9516302345219299,+ "accuracy": 0.5+ },+ "en.ag_news": {+ "n_heldout": 300,+ "fitted_temperatures": {+ "choice:3-5": 1.266+ },+ "ece_shipped": 0.05126942440656791,+ "ece_refit": 0.035514331323114996,+ "nll_shipped": 0.1966735562695727,+ "nll_refit": 0.1931097097550829,+ "accuracy": 0.9333333333333333+ },+ "en.boolq": {+ "n_heldout": 300,+ "fitted_temperatures": {+ "noul:2": 3.2246+ },+ "ece_shipped": 0.10215590522736365,+ "ece_refit": 0.051566417833373256,+ "nll_shipped": 0.5104874432163632,+ "nll_refit": 0.43038591367218393,+ "accuracy": 0.8266666666666667+ }+ },+ "mean_ece_shipped": 0.4656,+ "mean_ece_refit": 0.0812+ },+ "laya-multilingual": {+ "per_suite": {+ "typed_decisions": {+ "n_heldout": 1000,+ "fitted_temperatures": {+ "choice:3-5": 7.2626,+ "noul:2": 10.0,+ "score:3-5": 6.4219+ },+ "ece_shipped": 0.31353688166253296,+ "ece_refit": 0.05249973627123162,+ "nll_shipped": 1.7826284099077552,+ "nll_refit": 1.1529043184906664,+ "accuracy": 0.351+ },+ "massive_intent.en": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.285+ },+ "ece_shipped": 0.28598488876984407,+ "ece_refit": 0.11981541716793241,+ "nll_shipped": 2.3166463541043396,+ "nll_refit": 1.5109594902887804,+ "accuracy": 0.6066666666666667+ },+ "massive_intent.de": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.4002+ },+ "ece_shipped": 0.39419152284441683,+ "ece_refit": 0.11316109324129707,+ "nll_shipped": 2.9830588533749482,+ "nll_refit": 1.9610975344566464,+ "accuracy": 0.43333333333333335+ },+ "massive_intent.fr": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.3419+ },+ "ece_shipped": 0.33182806315184016,+ "ece_refit": 0.1650682537772552,+ "nll_shipped": 2.8198802270685617,+ "nll_refit": 1.8536174658295161,+ "accuracy": 0.5+ },+ "massive_intent.es": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.0708+ },+ "ece_shipped": 0.3747860038944444,+ "ece_refit": 0.16875478198848332,+ "nll_shipped": 2.757769866366103,+ "nll_refit": 1.883079560638871,+ "accuracy": 0.43333333333333335+ },+ "massive_intent.pt": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.5841+ },+ "ece_shipped": 0.34120944724576563,+ "ece_refit": 0.11388291274333108,+ "nll_shipped": 2.8307898847131425,+ "nll_refit": 1.8199963148617633,+ "accuracy": 0.5133333333333333+ },+ "massive_intent.ru": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.3419+ },+ "ece_shipped": 0.3499191112180423,+ "ece_refit": 0.12980454365283517,+ "nll_shipped": 2.8491438768356323,+ "nll_refit": 1.9214121580317896,+ "accuracy": 0.4533333333333333+ },+ "massive_intent.tr": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.6484+ },+ "ece_shipped": 0.33391590405482796,+ "ece_refit": 0.10349393836177559,+ "nll_shipped": 2.8017043190240036,+ "nll_refit": 1.9282719192318487,+ "accuracy": 0.47333333333333333+ },+ "massive_intent.ar": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.46+ },+ "ece_shipped": 0.4099836963498909,+ "ece_refit": 0.10917916558317786,+ "nll_shipped": 3.1613960902759275,+ "nll_refit": 2.25141683066659,+ "accuracy": 0.3466666666666667+ },+ "massive_intent.hi": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.3419+ },+ "ece_shipped": 0.4378035227689349,+ "ece_refit": 0.14973280309187975,+ "nll_shipped": 3.5750849346908624,+ "nll_refit": 2.342025405755958,+ "accuracy": 0.34+ },+ "massive_intent.ta": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.9224+ },+ "ece_shipped": 0.5282864038131523,+ "ece_refit": 0.1301923044510951,+ "nll_shipped": 4.225495743306456,+ "nll_refit": 2.7193019660496662,+ "accuracy": 0.2+ },+ "massive_intent.zh-CN": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.0708+ },+ "ece_shipped": 0.2956479305219314,+ "ece_refit": 0.11033723906598177,+ "nll_shipped": 2.2249843788919748,+ "nll_refit": 1.5650537999224634,+ "accuracy": 0.58+ },+ "massive_intent.ja": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.0205+ },+ "ece_shipped": 0.30483003324052754,+ "ece_refit": 0.14830564994188358,+ "nll_shipped": 2.3891544641828086,+ "nll_refit": 1.6551049648097012,+ "accuracy": 0.5466666666666666+ },+ "massive_intent.ko": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.0708+ },+ "ece_shipped": 0.32002650648667197,+ "ece_refit": 0.10080463213987816,+ "nll_shipped": 2.6602817717599048,+ "nll_refit": 1.9099858554110467,+ "accuracy": 0.48+ },+ "massive_intent.sw": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 3.7375+ },+ "ece_shipped": 0.49570577568341406,+ "ece_refit": 0.08349416187415526,+ "nll_shipped": 4.279076139734234,+ "nll_refit": 2.6662240908229258,+ "accuracy": 0.2+ },+ "massive_scenario.en": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.4002+ },+ "ece_shipped": 0.3428270841005442,+ "ece_refit": 0.10430091131513634,+ "nll_shipped": 2.826442913358127,+ "nll_refit": 1.7296360738235776,+ "accuracy": 0.5133333333333333+ },+ "massive_scenario.de": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.46+ },+ "ece_shipped": 0.4098164059892938,+ "ece_refit": 0.1103006818436626,+ "nll_shipped": 3.0195190523299282,+ "nll_refit": 1.883461519212016,+ "accuracy": 0.44+ },+ "massive_scenario.fr": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.5213+ },+ "ece_shipped": 0.4442562009642703,+ "ece_refit": 0.16025393217549574,+ "nll_shipped": 3.3766059898271252,+ "nll_refit": 1.9413440894718588,+ "accuracy": 0.4266666666666667+ },+ "massive_scenario.es": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.6484+ },+ "ece_shipped": 0.4109456966606466,+ "ece_refit": 0.10198578234795756,+ "nll_shipped": 3.2636908758703864,+ "nll_refit": 1.9252080039041213,+ "accuracy": 0.44+ },+ "massive_scenario.pt": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.9224+ },+ "ece_shipped": 0.4105148693608891,+ "ece_refit": 0.1138316207725879,+ "nll_shipped": 3.3875343693016906,+ "nll_refit": 1.9175783143767824,+ "accuracy": 0.4666666666666667+ },+ "massive_scenario.ru": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.6484+ },+ "ece_shipped": 0.41903289757915385,+ "ece_refit": 0.11771325878816861,+ "nll_shipped": 3.4690396514901765,+ "nll_refit": 2.02048805544728,+ "accuracy": 0.44666666666666666+ },+ "massive_scenario.tr": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.46+ },+ "ece_shipped": 0.3588353003356555,+ "ece_refit": 0.13829033714669553,+ "nll_shipped": 2.432523043817323,+ "nll_refit": 1.645665100066707,+ "accuracy": 0.4866666666666667+ },+ "massive_scenario.ar": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.6484+ },+ "ece_shipped": 0.41920881717954284,+ "ece_refit": 0.10673559257438134,+ "nll_shipped": 3.45497223390683,+ "nll_refit": 2.1575491605592583,+ "accuracy": 0.3933333333333333+ },+ "massive_scenario.hi": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.4002+ },+ "ece_shipped": 0.41481688542538986,+ "ece_refit": 0.15370426062587889,+ "nll_shipped": 3.3578108561910596,+ "nll_refit": 2.0777000340841267,+ "accuracy": 0.41333333333333333+ },+ "massive_scenario.ta": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.5841+ },+ "ece_shipped": 0.48257092406411306,+ "ece_refit": 0.1279809797192578,+ "nll_shipped": 3.8504959505951137,+ "nll_refit": 2.4452486759577177,+ "accuracy": 0.26666666666666666+ },+ "massive_scenario.zh-CN": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.5213+ },+ "ece_shipped": 0.3981881387108269,+ "ece_refit": 0.12326342022911907,+ "nll_shipped": 2.927004837667629,+ "nll_refit": 1.7716489027068876,+ "accuracy": 0.4533333333333333+ },+ "massive_scenario.ja": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.5213+ },+ "ece_shipped": 0.3621028758403154,+ "ece_refit": 0.13926267642528437,+ "nll_shipped": 3.136840178283236,+ "nll_refit": 1.8354044492411636,+ "accuracy": 0.5066666666666667+ },+ "massive_scenario.ko": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 2.9224+ },+ "ece_shipped": 0.3768861443562832,+ "ece_refit": 0.0977403817894932,+ "nll_shipped": 2.9273708306086474,+ "nll_refit": 1.9155954564982918,+ "accuracy": 0.42+ },+ "massive_scenario.sw": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:11+": 3.5581+ },+ "ece_shipped": 0.506361296484783,+ "ece_refit": 0.11207734739543539,+ "nll_shipped": 4.700826623994613,+ "nll_refit": 2.5552862898811015,+ "accuracy": 0.2866666666666667+ },+ "xnli.en": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 1.8311+ },+ "ece_shipped": 0.09900583157195271,+ "ece_refit": 0.06503906718116235,+ "nll_shipped": 0.3925995455412425,+ "nll_refit": 0.35876187759884653,+ "accuracy": 0.8533333333333334+ },+ "xnli.de": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 2.46+ },+ "ece_shipped": 0.12594098727915568,+ "ece_refit": 0.06500990874992986,+ "nll_shipped": 0.653860650712282,+ "nll_refit": 0.5067271285990049,+ "accuracy": 0.8133333333333334+ },+ "xnli.fr": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 2.5841+ },+ "ece_shipped": 0.11460899279515438,+ "ece_refit": 0.1072080948025079,+ "nll_shipped": 0.6051545830743009,+ "nll_refit": 0.5015551360343191,+ "accuracy": 0.82+ },+ "xnli.es": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 2.285+ },+ "ece_shipped": 0.1176948222453516,+ "ece_refit": 0.06262936231022864,+ "nll_shipped": 0.5866791824036008,+ "nll_refit": 0.49183324427122804,+ "accuracy": 0.82+ },+ "xnli.ru": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 3.0698+ },+ "ece_shipped": 0.17366405837906923,+ "ece_refit": 0.08228016201755169,+ "nll_shipped": 0.9308139508033506,+ "nll_refit": 0.6476032852724123,+ "accuracy": 0.76+ },+ "xnli.tr": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 2.9951+ },+ "ece_shipped": 0.19836158969915463,+ "ece_refit": 0.10254875033243725,+ "nll_shipped": 0.9277086355768465,+ "nll_refit": 0.6556320188131841,+ "accuracy": 0.74+ },+ "xnli.ar": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 3.3049+ },+ "ece_shipped": 0.16700123516881432,+ "ece_refit": 0.09902966264249458,+ "nll_shipped": 0.8953173210214982,+ "nll_refit": 0.6455067278444552,+ "accuracy": 0.76+ },+ "xnli.hi": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 3.5581+ },+ "ece_shipped": 0.21483863590232802,+ "ece_refit": 0.08204340573901349,+ "nll_shipped": 1.1766358162380428,+ "nll_refit": 0.7518036557992201,+ "accuracy": 0.68+ },+ "xnli.ur": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 3.7375+ },+ "ece_shipped": 0.19644100874145254,+ "ece_refit": 0.09763198576088242,+ "nll_shipped": 1.0413509566733983,+ "nll_refit": 0.7245230970472671,+ "accuracy": 0.7333333333333333+ },+ "xnli.vi": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 3.3049+ },+ "ece_shipped": 0.17600979149888385,+ "ece_refit": 0.09703177121512795,+ "nll_shipped": 1.0306447465645896,+ "nll_refit": 0.6622157369318552,+ "accuracy": 0.76+ },+ "xnli.th": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 3.4716+ },+ "ece_shipped": 0.16614142465028775,+ "ece_refit": 0.08809598765913614,+ "nll_shipped": 0.9550613292832744,+ "nll_refit": 0.6929707644114412,+ "accuracy": 0.7333333333333333+ },+ "xnli.el": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 2.9951+ },+ "ece_shipped": 0.18505427976039845,+ "ece_refit": 0.0619934233555169,+ "nll_shipped": 0.9681318335182048,+ "nll_refit": 0.6623250177254513,+ "accuracy": 0.72+ },+ "xnli.bg": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 2.4002+ },+ "ece_shipped": 0.13546796723698223,+ "ece_refit": 0.06411651174590313,+ "nll_shipped": 0.7253311578700296,+ "nll_refit": 0.5350767796618592,+ "accuracy": 0.8133333333333334+ },+ "xnli.zh": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 2.5841+ },+ "ece_shipped": 0.14274233933256558,+ "ece_refit": 0.083026544681253,+ "nll_shipped": 0.8243758967604422,+ "nll_refit": 0.612193238305549,+ "accuracy": 0.7666666666666667+ },+ "xnli.sw": {+ "n_heldout": 150,+ "fitted_temperatures": {+ "choice:3-5": 3.4716+ },+ "ece_shipped": 0.29730770690435115,+ "ece_refit": 0.11350413359716122,+ "nll_shipped": 1.4013281162840883,+ "nll_refit": 0.8379049118827668,+ "accuracy": 0.58+ },+ "en.sst5": {+ "n_heldout": 300,+ "fitted_temperatures": {+ "score:3-5": 10.0+ },+ "ece_shipped": 0.5241645000058643,+ "ece_refit": 0.03089034893874691,+ "nll_shipped": 3.832620682023934,+ "nll_refit": 1.5850353324296294,+ "accuracy": 0.28+ },+ "en.emotion": {+ "n_heldout": 300,+ "fitted_temperatures": {+ "choice:6-10": 3.1462+ },+ "ece_shipped": 0.40379864214963257,+ "ece_refit": 0.08741204055524324,+ "nll_shipped": 2.637519408908119,+ "nll_refit": 1.4416251362588335,+ "accuracy": 0.46+ },+ "en.prompt_injections": {+ "n_heldout": 58,+ "fitted_temperatures": {+ "noul:2": 6.2658+ },+ "ece_shipped": 0.4514191955987473,+ "ece_refit": 0.2407982144094497,+ "nll_shipped": 2.251858103333174,+ "nll_refit": 0.743234178474742,+ "accuracy": 0.4827586206896552+ },+ "en.ag_news": {+ "n_heldout": 300,+ "fitted_temperatures": {+ "choice:3-5": 1.5039+ },+ "ece_shipped": 0.04495656180857554,+ "ece_refit": 0.010973389250756511,+ "nll_shipped": 0.24864777119159517,+ "nll_refit": 0.2083974803565066,+ "accuracy": 0.9333333333333333+ },+ "en.boolq": {+ "n_heldout": 300,+ "fitted_temperatures": {+ "noul:2": 2.8513+ },+ "ece_shipped": 0.15205137686074968,+ "ece_refit": 0.04382109708993337,+ "nll_shipped": 0.6707500082967005,+ "nll_refit": 0.4659609286304744,+ "accuracy": 0.7933333333333333+ }+ },+ "mean_ece_shipped": 0.3135,+ "mean_ece_refit": 0.1059+ }+ },+ "caveats": {+ "context_window": "laya max_len=512, laya-multilingual max_len=1024 (shipped configs). See typed_decisions_matched_512 for the equal-budget control.",+ "shipped_calibration": "laya ships fitted temperatures; laya-multilingual ships all-1.0 (uncalibrated). See calibration_repair for the out-of-sample refit.",+ "training_overlap": "ag_news and boolq were in Laya's training mix (retention, not generalisation). sst5, emotion, prompt_injections, banking77 were held out. Neither checkpoint trained on typed-decisions, MASSIVE or XNLI."+ },+ "summary": {+ "typed_decisions": {+ "laya": {+ "n": 2000,+ "accuracy": 0.362,+ "macro_f1": 0.3273577076400338,+ "ece": 0.17410090455022942,+ "brier": 0.7496143221875959,+ "nll": 1.3224634387781617,+ "aurc": 0.5928492917684455,+ "mean_confidence": 0.5361009045502294,+ "acc_at_50_coverage": 0.427,+ "acc_at_80_coverage": 0.386875+ },+ "laya-multilingual": {+ "n": 2000,+ "accuracy": 0.3515,+ "macro_f1": 0.34975321170122525,+ "ece": 0.31443177617478313,+ "brier": 0.8904323210391285,+ "nll": 1.8242712912901595,+ "aurc": 0.5562863247537705,+ "mean_confidence": 0.6629025476146312,+ "acc_at_50_coverage": 0.414,+ "acc_at_80_coverage": 0.368125+ }+ },+ "massive_intent": {+ "laya": {+ "macro_accuracy": 0.3405,+ "macro_ece": 0.6266,+ "n_questions": 4200,+ "n_suites": 14+ },+ "laya-multilingual": {+ "macro_accuracy": 0.466,+ "macro_ece": 0.3421,+ "n_questions": 4200,+ "n_suites": 14+ }+ },+ "massive_scenario": {+ "laya": {+ "macro_accuracy": 0.3036,+ "macro_ece": 0.6707,+ "n_questions": 4200,+ "n_suites": 14+ },+ "laya-multilingual": {+ "macro_accuracy": 0.4474,+ "macro_ece": 0.3857,+ "n_questions": 4200,+ "n_suites": 14+ }+ },+ "xnli": {+ "laya": {+ "macro_accuracy": 0.5436,+ "macro_ece": 0.2296,+ "n_questions": 4500,+ "n_suites": 15+ },+ "laya-multilingual": {+ "macro_accuracy": 0.7384,+ "macro_ece": 0.1813,+ "n_questions": 4500,+ "n_suites": 15+ }+ },+ "english": {+ "laya": {+ "macro_accuracy": 0.684,+ "macro_ece": 0.1938,+ "n_questions": 2516,+ "n_suites": 5+ },+ "laya-multilingual": {+ "macro_accuracy": 0.6192,+ "macro_ece": 0.2842,+ "n_questions": 2516,+ "n_suites": 5+ }+ },+ "massive_intent_english_vs_rest": {+ "english": {+ "laya": 0.7833,+ "laya-multilingual": 0.6567+ },+ "non_english": {+ "laya": 0.3064,+ "laya-multilingual": 0.4513+ }+ },+ "massive_scenario_english_vs_rest": {+ "english": {+ "laya": 0.6033,+ "laya-multilingual": 0.56+ },+ "non_english": {+ "laya": 0.2805,+ "laya-multilingual": 0.4387+ }+ },+ "xnli_english_vs_rest": {+ "english": {+ "laya": 0.86,+ "laya-multilingual": 0.8433+ },+ "non_english": {+ "laya": 0.521,+ "laya-multilingual": 0.731+ }+ }+ }+}\ No newline at end of filediff --git a/research/scripts/bench_apps.py b/research/scripts/bench_apps.pynew file mode 100644index 00000000..c0b255ba--- /dev/null+++ b/research/scripts/bench_apps.py@@ -0,0 +1,302 @@+"""Application-workflow benchmark for the Laya checkpoints, plus the tasks where public+Jev numbers exist so a like-for-like comparison is possible.++Workflows (the demo Space's tabs), each on real labelled data:+ 1 support triage banking77 (77-way intent) + customer-support-tickets queue routing+ 2 email + phishing enron spam + phishing emails+ 3 LLM guardrails lmsys/toxic-chat jailbreaking flag (held out of Laya training)+ 4 RAG passage filter MS MARCO passage relevance+ 5 moderation lmsys/toxic-chat toxicity flag (held out of Laya training)+ 6 model routing domain classification over gsm8k / mbpp / writing / factual++Jev-comparable tasks (AbdelStark/jev-benchmarks published Jev accuracy on these):+ ag_news 0.910 | banking77 0.870 | dair emotion 0.480 (Brier 0.846, NLL 5.588)++ USE_TF=0 python3 notebooks/bench_apps.py+"""+import gc+import json+import math+import os+import random+import sys+import time++os.environ.setdefault("USE_TF", "0")+os.environ.setdefault("USE_TORCH", "1")+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")++import numpy as np # noqa: E402+import torch # noqa: E402++REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+sys.path.insert(0, os.path.join(REPO, "laya"))+sys.path.insert(0, os.path.join(REPO, "notebooks"))++import laya # noqa: E402+from bench_local import load, metrics, score_cases, softmax_t, temp_for # noqa: E402++OUT = os.path.join(REPO, "app_benchmark_results.json")+SEED = 13+N = int(os.environ.get("BENCH_N", "400"))++# Published Jev numbers on identical public datasets (NOT measured here - no TypeSafe API access).+JEV_PUBLISHED = {+ "ag_news": {"accuracy": 0.910, "coverage_at_5pct_error": 0.830, "latency_p50_ms": 236,+ "n": 100, "source": "AbdelStark/jev-benchmarks v0.1.0"},+ "banking77": {"accuracy": 0.870, "coverage_at_5pct_error": 0.860, "latency_p50_ms": 246,+ "n": 100, "labels": 72, "source": "AbdelStark/jev-benchmarks v0.1.0"},+ "emotion": {"accuracy": 0.480, "brier": 0.846, "nll": 5.588, "coverage_at_5pct_error": 0.0,+ "zero_prob_failures": 0.16, "n": 100, "source": "AbdelStark/jev-benchmarks v0.1.0"},+ "typed_decisions": {"accuracy": 0.727, "soft_accuracy": 0.580, "brier": 0.148, "ece": 0.144,+ "score_mae": 0.391, "ms_per_case": 710, "source": "laya repo comparison table"},+ "_independent_": {"banking77_accuracy": 0.763, "sms_spam_accuracy": 0.930,+ "permuted_accuracy": 0.767, "ece": 0.246, "latency_p50_ms": "264-276",+ "option_order_flip_rate": 0.13,+ "source": "nibzard/decision-model-benchmark"},+}++SUITES = {}+++def register(name, cases, gold, **meta):+ SUITES[name] = {"cases": cases, "gold": gold, "meta": meta}+ print(" %-26s %4d cases %s" % (name, len(cases), meta.get("note", "")), flush=True)+++def choice_q(qid, instructions, keys, gold_key):+ crit = {k: None for k in keys}+ return {qid: {"type": "choice", "instructions": instructions, "criteria": crit}}, keys.index(gold_key)+++def build():+ from datasets import load_dataset+ rng = random.Random(SEED)++ # ---------------------------------------------------------- Jev-comparable: ag_news+ try:+ d = load_dataset("fancyzhx/ag_news", split="test")+ crit = {"world": "world news and international politics", "sports": "sports",+ "business": "business and economy", "sci_tech": "science and technology"}+ keys = list(crit)+ cases, gold = [], []+ for r in list(d)[:N]:+ cases.append(({"article": r["text"]},+ {"topic": {"type": "choice", "instructions": "What is the topic of `article`?",+ "criteria": dict(crit)}}))+ gold.append(keys.index(keys[int(r["label"])]))+ register("jev.ag_news", cases, gold, note="Jev 0.910", jev="ag_news", in_training=True)+ except Exception as e:+ print(" FAIL ag_news", str(e)[:80])++ # ---------------------------------------------------------- Jev-comparable: emotion+ try:+ d = load_dataset("dair-ai/emotion", "split", split="test")+ names = ["sadness", "joy", "love", "anger", "fear", "surprise"]+ cases, gold = [], []+ for r in list(d)[:N]:+ cases.append(({"text": r["text"]},+ {"emotion": {"type": "choice",+ "instructions": "Which emotion is most strongly expressed in `text`?",+ "criteria": {n: None for n in names}}}))+ gold.append(int(r["label"]))+ register("jev.emotion", cases, gold, note="Jev 0.480 (Brier 0.846)", jev="emotion",+ in_training=False)+ except Exception as e:+ print(" FAIL emotion", str(e)[:80])++ # ------------------------------------------- Jev-comparable + triage: banking77 (77 labels)+ try:+ d = load_dataset("mteb/banking77", split="test")+ labels = sorted(set(d["label_text"]))+ cases, gold = [], []+ for r in list(d)[:N]:+ qs, gi = choice_q("intent", "Which banking intent does `message` express?",+ [x.replace("_", " ") for x in labels],+ r["label_text"].replace("_", " "))+ cases.append(({"message": r["text"]}, qs)); gold.append(gi)+ register("jev.banking77_full", cases, gold,+ note="%d labels at once | Jev 0.870 (72 labels)" % len(labels),+ jev="banking77", in_training=False, n_labels=len(labels))+ except Exception as e:+ print(" FAIL banking77", str(e)[:80])++ # ---------------------------------------------------------- 1. support triage (queues)+ try:+ d = load_dataset("Tobi-Bueck/customer-support-tickets", split="train")+ QUEUES = {"Technical Support": "technical problems, bugs, outages, integrations",+ "Product Support": "help using a product or feature",+ "Customer Service": "general account or service questions",+ "IT Support": "internal IT, devices, access, networks",+ "Billing and Payments": "invoices, charges, refunds, payment methods",+ "Returns and Exchanges": "returning or exchanging an item",+ "Service Outages and Maintenance": "downtime, outages, scheduled maintenance",+ "Sales and Pre-Sales": "pricing, quotes, buying",+ "Human Resources": "employment, payroll, leave, hiring",+ "General Inquiry": "anything else"}+ keys = list(QUEUES)+ cases, gold = [], []+ for r in d:+ if r.get("language") != "en" or r.get("queue") not in QUEUES or not r.get("body"):+ continue+ cases.append(({"subject": r["subject"] or "", "body": r["body"].replace("\\n", "\n")[:3000]},+ {"queue": {"type": "choice",+ "instructions": "Which support queue should handle this ticket?",+ "criteria": dict(QUEUES)}}))+ gold.append(keys.index(r["queue"]))+ if len(cases) >= N:+ break+ register("app.support_triage", cases, gold, note="10-way queue routing", in_training=True)+ except Exception as e:+ print(" FAIL support_triage", str(e)[:80])++ # ---------------------------------------------------------- 2. email + phishing+ try:+ d = load_dataset("SetFit/enron_spam", split="test")+ cases, gold = [], []+ for r in list(d)[:N]:+ st = laya.email_state(r.get("subject") or "", (r.get("message") or "")[:3000])+ cases.append((st, {"is_spam": {"type": "noul",+ "instructions": "Is this email unsolicited spam or bulk marketing?"}}))+ gold.append(int(r["label"]))+ register("app.email_spam", cases, gold, note="enron spam", in_training=True)+ except Exception as e:+ print(" FAIL email_spam", str(e)[:80])++ try:+ d = load_dataset("zefang-liu/phishing-email-dataset", split="train")+ rows = [r for r in list(d)[:6000]+ if (r.get("Email Text") or "").strip() and r.get("Email Type") in ("Safe Email", "Phishing Email")]+ rng.shuffle(rows)+ cases, gold = [], []+ for r in rows[:N]:+ cases.append(({"email": r["Email Text"][:3000]},+ {"is_phishing": {"type": "noul",+ "instructions": "Is this email a phishing or scam attempt to steal money, credentials, or personal data?",+ "criteria": {"true": "phishing, scam, or fraud",+ "false": "a legitimate email (even if promotional)"}}}))+ gold.append(int(r["Email Type"] == "Phishing Email"))+ register("app.phishing", cases, gold, note="phishing emails", in_training=True)+ except Exception as e:+ print(" FAIL phishing", str(e)[:80])++ # ------------------------------------- 3 & 5. guardrails + moderation (toxic-chat, HELD OUT)+ try:+ d = load_dataset("lmsys/toxic-chat", "toxicchat0124", split="test")+ rows = [r for r in d if (r.get("user_input") or "").strip()]+ jb = [r for r in rows if int(r.get("jailbreaking", 0)) == 1][:N // 2]+ nj = [r for r in rows if int(r.get("jailbreaking", 0)) == 0][:N - len(jb)]+ mix = jb + nj+ rng.shuffle(mix)+ cases, gold = [], []+ for r in mix:+ cases.append(({"prompt": r["user_input"][:3000]},+ {"jailbreak": {"type": "noul",+ "instructions": "Does `prompt` try to make an AI assistant ignore its rules, policies or system instructions?"}}))+ gold.append(int(r["jailbreaking"]))+ register("app.guardrails_jailbreak", cases, gold,+ note="toxic-chat jailbreaking (HELD OUT)", in_training=False)++ tox = [r for r in rows if int(r.get("toxicity", 0)) == 1][:N // 2]+ ntox = [r for r in rows if int(r.get("toxicity", 0)) == 0][:N - len(tox)]+ mix2 = tox + ntox+ rng.shuffle(mix2)+ cases, gold = [], []+ for r in mix2:+ cases.append(({"post": r["user_input"][:3000]},+ {"toxic": {"type": "noul",+ "instructions": "Is `post` toxic: rude, disrespectful or likely to make someone leave the discussion?"}}))+ gold.append(int(r["toxicity"]))+ register("app.moderation_toxicity", cases, gold,+ note="toxic-chat toxicity (HELD OUT)", in_training=False)+ except Exception as e:+ print(" FAIL toxic-chat", str(e)[:80])++ # ---------------------------------------------------------- 4. RAG passage filter+ try:+ d = load_dataset("microsoft/ms_marco", "v1.1", split="validation")+ cases, gold = [], []+ for r in d:+ texts, sel = r["passages"]["passage_text"], r["passages"]["is_selected"]+ pos = [t for t, s in zip(texts, sel) if s == 1]+ neg = [t for t, s in zip(texts, sel) if s == 0]+ if not pos or not neg:+ continue+ take_pos = len(cases) % 2 == 0+ p = rng.choice(pos if take_pos else neg)+ cases.append(({"query": r["query"], "passage": p},+ {"relevant": {"type": "noul",+ "instructions": "Does `passage` help answer `query`?"}}))+ gold.append(1 if take_pos else 0)+ if len(cases) >= N:+ break+ register("app.rag_relevance", cases, gold, note="MS MARCO relevance", in_training=True)+ except Exception as e:+ print(" FAIL rag", str(e)[:80])++ # ---------------------------------------------------------- 6. model routing (domain)+ try:+ DOM = {"code": "software engineering, programming, refactoring, architecture, debugging",+ "math_or_logic": "mathematics, logic puzzles, proofs, complex calculation",+ "writing": "creative writing, essays, emails, blog posts, copywriting",+ "factual_lookup": "facts, definitions, trivia, history",+ "data_analysis": "statistics, SQL, data manipulation, metrics",+ "chitchat": "casual conversation, greetings, small talk"}+ keys = list(DOM)+ pool = []+ g = load_dataset("openai/gsm8k", "main", split="test")+ pool += [(r["question"], "math_or_logic") for r in list(g)[:N // 3]]+ m = load_dataset("google-research-datasets/mbpp", "full", split="test")+ pool += [(r["text"], "code") for r in list(m)[:N // 3]]+ t = load_dataset("fancyzhx/ag_news", split="test")+ pool += [(r["text"][:400], "factual_lookup") for r in list(t)[:N // 3]]+ rng.shuffle(pool)+ cases, gold = [], []+ for text, dom in pool[:N]:+ cases.append(({"request": text},+ {"domain": {"type": "choice",+ "instructions": "What domain does `request` belong to?",+ "criteria": dict(DOM)}}))+ gold.append(keys.index(dom))+ register("app.model_routing_domain", cases, gold,+ note="gsm8k/mbpp/ag_news -> domain", in_training=False)+ except Exception as e:+ print(" FAIL routing", str(e)[:80])+++def main():+ print("=== building suites (N=%d per task) ===\n" % N, flush=True)+ build()+ results = {"meta": {"timestamp": time.strftime("%Y-%m-%d %H:%M:%S"), "device": "cpu",+ "n_per_task": N, "seed": SEED, "laya": laya.__version__},+ "jev_published": JEV_PUBLISHED, "suites": {}}+ for mname in ("english", "multilingual", "typed-decisions"):+ print("\n=== %s ===" % mname, flush=True)+ ag = load(mname)+ for sname, S in SUITES.items():+ try:+ lgs, idx, secs, dropped = score_cases(ag, S["cases"], tag="%s/%s" % (mname, sname))+ rows = [(S["gold"][ci], softmax_t(z, temp_for(ag, qt, k)) if z is not None else None)+ for (ci, _, qt, k), z in zip(idx, lgs)]+ m = metrics(rows)+ m["seconds"] = round(secs, 1)+ m["ms_per_case"] = round(1000 * secs / max(1, len(S["cases"])), 1)+ m["dropped"] = dropped+ m.update({k: v for k, v in S["meta"].items() if k != "note"})+ results["suites"].setdefault(sname, {})[mname] = m+ jev = JEV_PUBLISHED.get(S["meta"].get("jev") or "", {}).get("accuracy")+ delta = (" vs Jev %.3f -> %+.3f" % (jev, m["accuracy"] - jev)) if jev else ""+ print(" %-26s acc %.3f f1 %.3f ECE %.3f %5.1f ms/case%s"+ % (sname, m["accuracy"], m["macro_f1"], m["ece"], m["ms_per_case"], delta),+ flush=True)+ except Exception as e:+ import traceback; traceback.print_exc()+ results["suites"].setdefault(sname, {})[mname] = {"error": str(e)[:200]}+ del ag; gc.collect()+ json.dump(results, open(OUT, "w"), indent=2)+ json.dump(results, open(OUT, "w"), indent=2)+ print("\nwrote %s" % OUT, flush=True)+++if __name__ == "__main__":+ main()diff --git a/research/scripts/bench_latency.py b/research/scripts/bench_latency.pynew file mode 100644index 00000000..e97ddbb4--- /dev/null+++ b/research/scripts/bench_latency.py@@ -0,0 +1,169 @@+"""Inference-speed benchmark, including what routing actually costs.++Routing changes the latency picture in two ways:+ * every call pays language detection (pure Python, no model)+ * a call that switches checkpoint pays a model load, unless that model is already resident++So there are two regimes -- hot (right model already loaded) and cold (swap) -- and the realistic+number for a mixed-language workload sits between them, determined by `max_loaded`.++ USE_TF=0 python3 notebooks/bench_latency.py+"""+import gc+import json+import os+import statistics+import sys+import time++os.environ.setdefault("USE_TF", "0")+os.environ.setdefault("USE_TORCH", "1")+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")++import numpy as np # noqa: E402+import torch # noqa: E402++REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+sys.path.insert(0, os.path.join(REPO, "laya"))++import laya # noqa: E402+from laya.lang import analyse # noqa: E402+from laya.router import Router # noqa: E402++ROOT = os.path.expanduser("~/laya_models")+MODELS = {"english": os.path.join(ROOT, "laya"),+ "multilingual": os.path.join(ROOT, "laya-multilingual"),+ "typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}+OUT = os.path.join(REPO, "latency_benchmark_results.json")++STATE_EN = {"ticket": {"subject": "Payout failing", "messages": [{"from": "customer",+ "text": "Hi, my Stripe payouts have failed for 3 days and I am losing sales. Please help ASAP. " * 6}]}}+STATE_HI = {"ticket": {"subject": "भुगतान विफल", "messages": [{"from": "customer",+ "text": "मेरा भुगतान तीन दिनों से विफल हो रहा है और मुझे नुकसान हो रहा है, कृपया तुरंत मदद करें। " * 6}]}}+Q_NOUL = {"type": "noul", "instructions": "Does `ticket.messages[0].text` express urgency?"}+Q_CHOICE = {"type": "choice", "instructions": "Which team should handle this?",+ "criteria": {"billing": "payments", "technical": "bugs and integrations", "sales": "pricing"}}+++def qs(n):+ return {("q%d" % i): (Q_NOUL if i % 2 else Q_CHOICE) for i in range(n)}+++def timed(fn, warmup=3, reps=15):+ for _ in range(warmup):+ fn()+ ts = []+ for _ in range(reps):+ t = time.perf_counter()+ fn()+ ts.append((time.perf_counter() - t) * 1000)+ return {"p50_ms": round(float(np.percentile(ts, 50)), 2),+ "p95_ms": round(float(np.percentile(ts, 95)), 2),+ "mean_ms": round(float(np.mean(ts)), 2),+ "min_ms": round(float(np.min(ts)), 2)}+++def main():+ res = {"meta": {"timestamp": time.strftime("%Y-%m-%d %H:%M:%S"), "device": "cpu",+ "torch": torch.__version__, "threads": torch.get_num_threads(),+ "laya": laya.__version__,+ "note": "CPU numbers. GPU (T4) reference from the Colab run is in "+ "laya_benchmark_results.json -> latency."}}++ # ---------------------------------------------------------------- 1. detection overhead+ print("=== 1. language detection overhead (no model) ===", flush=True)+ det = {}+ for label, st in (("english", STATE_EN), ("hindi", STATE_HI),+ ("short english", {"m": "refund me"}),+ ("large json", {"rows": [{"id": i, "text": "some ticket body here"} for i in range(200)]})):+ det[label] = timed(lambda s=st: analyse(s), warmup=20, reps=200)+ print(" %-16s p50 %7.3f ms p95 %7.3f ms" % (label, det[label]["p50_ms"], det[label]["p95_ms"]), flush=True)+ res["detection_overhead"] = det++ # ---------------------------------------------------------------- 2. per-model raw latency+ print("\n=== 2. raw Agent.system_one latency (no routing) ===", flush=True)+ raw, load_times = {}, {}+ for mname, path in MODELS.items():+ t = time.perf_counter()+ ag = laya.load(path, device="cpu")+ load_times[mname] = round((time.perf_counter() - t) * 1000, 1)+ per = {}+ for n in (1, 5, 10, 50):+ q = qs(n)+ r = timed(lambda: ag.system_one(STATE_EN, q), warmup=2, reps=10)+ r["ms_per_question"] = round(r["p50_ms"] / n, 2)+ per["%d_questions" % n] = r+ print(" %-16s %2dq p50 %8.1f ms p95 %8.1f ms %6.2f ms/q"+ % (mname, n, r["p50_ms"], r["p95_ms"], r["ms_per_question"]), flush=True)+ raw[mname] = per+ print(" %-16s cold load: %.0f ms" % (mname, load_times[mname]), flush=True)+ del ag; gc.collect()+ res["raw_latency"] = raw+ res["cold_load_ms"] = load_times++ # ---------------------------------------------------------------- 3. router hot path+ print("\n=== 3. Router hot path (target model already resident) ===", flush=True)+ r3 = Router(models=MODELS, device="cpu", max_loaded=3)+ r3.predict(STATE_EN, qs(5)) # warm english+ r3.predict(STATE_HI, qs(5)) # warm multilingual+ hot = {}+ for label, st in (("english", STATE_EN), ("hindi", STATE_HI)):+ for n in (1, 10):+ q = qs(n)+ k = "%s_%dq" % (label, n)+ hot[k] = timed(lambda s=st, qq=q: r3.predict(s, qq), warmup=2, reps=10)+ hot[k]["ms_per_question"] = round(hot[k]["p50_ms"] / n, 2)+ print(" %-10s %2dq p50 %8.1f ms (%.2f ms/q)" % (label, n, hot[k]["p50_ms"],+ hot[k]["ms_per_question"]), flush=True)+ res["router_hot"] = hot++ # overhead vs raw, like for like+ base_en = raw["english"]["10_questions"]["p50_ms"]+ res["routing_overhead_hot_ms"] = round(hot["english_10q"]["p50_ms"] - base_en, 2)+ print(" routing overhead when hot: %+.2f ms on a 10-question English call"+ % res["routing_overhead_hot_ms"], flush=True)++ # ---------------------------------------------------------------- 4. router cold path (swap)+ print("\n=== 4. Router cold path (max_loaded=1 forces a swap on every language flip) ===", flush=True)+ r1 = Router(models=MODELS, device="cpu", max_loaded=1)+ r1.predict(STATE_EN, qs(5))+ swap = []+ for i in range(4):+ st = STATE_HI if i % 2 == 0 else STATE_EN+ t = time.perf_counter()+ r1.predict(st, qs(10))+ swap.append((time.perf_counter() - t) * 1000)+ res["router_cold_swap"] = {"samples_ms": [round(x, 1) for x in swap],+ "median_ms": round(statistics.median(swap), 1)}+ print(" swap calls (10q): %s median %.0f ms"+ % ([round(x) for x in swap], res["router_cold_swap"]["median_ms"]), flush=True)++ # ---------------------------------------------------------------- 5. mixed workload+ print("\n=== 5. mixed-language workload, 100 calls of 5 questions ===", flush=True)+ import random+ rng = random.Random(13)+ mix = {}+ for share in (0.0, 0.1, 0.3, 0.5):+ stream = [(STATE_HI if rng.random() < share else STATE_EN) for _ in range(100)]+ for cap, tag in ((1, "max_loaded=1"), (3, "max_loaded=3")):+ rr = Router(models=MODELS, device="cpu", max_loaded=cap)+ rr.predict(STATE_EN, qs(5)); rr.predict(STATE_HI, qs(5)) # warm what fits+ q = qs(5)+ t = time.perf_counter()+ for st in stream:+ rr.predict(st, q)+ el = (time.perf_counter() - t)+ key = "%d%%_non_english/%s" % (int(share * 100), tag)+ mix[key] = {"total_s": round(el, 2), "mean_ms_per_call": round(el * 1000 / len(stream), 1),+ "calls_per_s": round(len(stream) / el, 1)}+ print(" %-32s %6.1f ms/call %5.1f calls/s"+ % (key, mix[key]["mean_ms_per_call"], mix[key]["calls_per_s"]), flush=True)+ rr.unload(); del rr; gc.collect()+ res["mixed_workload"] = mix++ json.dump(res, open(OUT, "w"), indent=2)+ print("\nwrote %s" % OUT, flush=True)+++if __name__ == "__main__":+ main()diff --git a/research/scripts/bench_local.py b/research/scripts/bench_local.pynew file mode 100644index 00000000..1c77dfdf--- /dev/null+++ b/research/scripts/bench_local.py@@ -0,0 +1,337 @@+"""Local extensive benchmark of the three Laya checkpoints.++Part A MASSIVE intent across every language the dataset ships (~51), 20-option choice.+Part B typed-decisions (400 cases / 2,000 decisions) on all three checkpoints, so the+ fine-tuned laya-typed-decisions can be compared with Jev's published 0.727 on the+ same benchmark.++Writes local_benchmark_results.json.++ USE_TF=0 python3 notebooks/bench_local.py [--langs N] [--per-lang N] [--skip-a] [--skip-b]+"""+import argparse+import gc+import json+import math+import os+import random+import re+import sys+import time++os.environ.setdefault("USE_TF", "0") # TensorFlow's abseil runtime deadlocks model build+os.environ.setdefault("USE_TORCH", "1")+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")++import numpy as np # noqa: E402+import torch # noqa: E402++REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+sys.path.insert(0, os.path.join(REPO, "laya"))++import laya # noqa: E402+from laya.common import QTYPES, build_sequence, collate_items, render_options, temp_bucket # noqa: E402++ROOT = os.path.expanduser("~/laya_models")+MODELS = {"english": os.path.join(ROOT, "laya"),+ "multilingual": os.path.join(ROOT, "laya-multilingual"),+ "typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}+OUT = os.path.join(REPO, "local_benchmark_results.json")+SEED, N_OPTS = 13, 20+++# ------------------------------------------------------------------ engine+def to_internal(qdef):+ t = qdef["type"]+ crit = qdef.get("criteria")+ if t == "choice" and isinstance(crit, list):+ crit = {c: None for c in crit}+ ins = qdef["instructions"]+ return {"t": t, "ins": ins if isinstance(ins, str) else json.dumps(ins), "crit": crit}++[email protected]_grad()+def score_cases(agent, cases, max_tokens=8192, max_seqs=64, tag=""):+ max_len = agent.cfg.get("max_len", 512)+ hml = agent.cfg.get("head_max_len", 192)+ items, index, dropped = [], [], 0+ for ci, (state, questions) in enumerate(cases):+ for qid, qdef in questions.items():+ q = to_internal(qdef)+ try:+ ids, mk = build_sequence(agent.tok, state, q, max_len, hml)+ except Exception:+ index.append((ci, qid, QTYPES[q["t"]], 0)); items.append(None); dropped += 1; continue+ if len(mk) != len(render_options(q)):+ index.append((ci, qid, QTYPES[q["t"]], 0)); items.append(None); dropped += 1; continue+ items.append({"ids": ids, "markers": mk, "qtype": QTYPES[q["t"]]})+ index.append((ci, qid, QTYPES[q["t"]], len(mk)))+ order = sorted([i for i, it in enumerate(items) if it is not None],+ key=lambda i: len(items[i]["ids"]))+ out = [None] * len(items)+ t0, done, i = time.time(), 0, 0+ while i < len(order):+ j, L = i, 0+ while j < len(order) and j - i < max_seqs and \+ max(L, len(items[order[j]]["ids"])) * (j - i + 1) <= max_tokens:+ L = max(L, len(items[order[j]]["ids"])); j += 1+ j = max(j, i + 1)+ sel = [items[order[t]] for t in range(i, j)]+ b = collate_items([sel], agent.tok.pad_token_id)+ lg, _ = agent.model(b["input_ids"].to(agent.device), b["attention_mask"].to(agent.device),+ b["marker_pos"].to(agent.device), b["marker_mask"].to(agent.device),+ b["qtype"].to(agent.device))+ lg = lg.float().cpu().numpy()+ for r in range(j - i):+ out[order[i + r]] = lg[r, :len(sel[r]["markers"])]+ done += j - i+ if tag and done % 500 < (j - i):+ el = time.time() - t0+ sys.stderr.write("\r [%s] %d/%d %.0f q/s ETA %ds "+ % (tag, done, len(order), done / max(el, 1e-9),+ int(el * (len(order) - done) / max(1, done))))+ sys.stderr.flush()+ i = j+ if tag:+ sys.stderr.write("\r" + " " * 70 + "\r")+ return out, index, time.time() - t0, dropped+++def softmax_t(z, t=1.0):+ z = np.asarray(z, float) / max(1e-3, float(t))+ e = np.exp(z - z.max())+ return e / e.sum()+++def temp_for(agent, qt, k):+ return float(agent.temperature_by_options.get(temp_bucket(qt, k), agent.temperature[qt]))+++def ece_score(conf, corr, bins=15):+ conf, corr = np.asarray(conf, float), np.asarray(corr, float)+ if not len(conf):+ return float("nan")+ e, edges = 0.0, np.linspace(0, 1, bins + 1)+ for lo, hi in zip(edges[:-1], edges[1:]):+ s = (conf > lo) & (conf <= hi)+ if s.any():+ e += s.mean() * abs(conf[s].mean() - corr[s].mean())+ return float(e)+++def macro_f1(g, p):+ g, p = np.asarray(g), np.asarray(p)+ f = []+ for c in sorted(set(g.tolist()) | set(p.tolist())):+ tp = int(((p == c) & (g == c)).sum()); fp = int(((p == c) & (g != c)).sum())+ fn = int(((p != c) & (g == c)).sum())+ f.append(2 * tp / max(1, 2 * tp + fp + fn))+ return float(np.mean(f))+++def metrics(rows):+ rows = [r for r in rows if r[1] is not None]+ if not rows:+ return {"n": 0}+ g = np.array([x[0] for x in rows]); p = np.array([int(np.argmax(x[1])) for x in rows])+ c = np.array([float(np.max(x[1])) for x in rows]); corr = (p == g).astype(float)+ return {"n": len(rows), "accuracy": round(float(corr.mean()), 4),+ "macro_f1": round(macro_f1(g, p), 4), "ece": round(ece_score(c, corr), 4),+ "brier": round(float(np.mean([((np.asarray(x[1]) - np.eye(len(x[1]))[x[0]]) ** 2).sum()+ for x in rows])), 4),+ "nll": round(float(np.mean([-math.log(max(float(x[1][x[0]]), 1e-12)) for x in rows])), 4),+ "mean_confidence": round(float(c.mean()), 4),+ "acc_at_50_coverage": round(float(corr[np.argsort(-c)[:max(1, len(c)//2)]].mean()), 4)}+++def load(name):+ ag = laya.load(MODELS[name], device="cpu")+ ag.model.eval()+ return ag+++# ------------------------------------------------------------------ part A+def massive_languages():+ from huggingface_hub import HfApi+ files = [s.rfilename for s in (HfApi().dataset_info("mteb/amazon_massive_intent").siblings or [])]+ return sorted({m.group(1) for f in files for m in [re.match(r"test/([A-Za-z\-]+)\.json", f)] if m})+++def build_massive(langs, per_lang):+ from datasets import load_dataset+ suites = {}+ for lg in langs:+ try:+ d = load_dataset("mteb/amazon_massive_intent", lg, split="test")+ labels = sorted(set(d["label_text"]))+ rng = random.Random(SEED)+ cases, gold = [], []+ for r in list(d)[:per_lang]:+ pool = [x for x in labels if x != r["label_text"]]+ keys = [r["label_text"]] + rng.sample(pool, min(N_OPTS - 1, len(pool)))+ rng.shuffle(keys)+ cases.append(({"utterance": r["text"]},+ {"intent": {"type": "choice",+ "instructions": "What is the user asking for in `utterance`?",+ "criteria": {k: k.replace("_", " ").replace(".", ": ") for k in keys}}}))+ gold.append(keys.index(r["label_text"]))+ suites[lg] = (cases, gold, len(labels))+ print(" built %-8s %d cases (%d labels)" % (lg, len(cases), len(labels)), flush=True)+ except Exception as e:+ print(" FAIL %-8s %s" % (lg, str(e)[:70]), flush=True)+ return suites+++def run_part_a(results, langs, per_lang):+ print("\n=== PART A: MASSIVE intent, %d languages, %d cases each, %d options ===\n"+ % (len(langs), per_lang, N_OPTS), flush=True)+ suites = build_massive(langs, per_lang)+ results["part_a"] = {"config": {"languages": sorted(suites), "per_lang": per_lang,+ "n_options": N_OPTS, "seed": SEED}, "by_model": {}}+ for mname in ("english", "multilingual"):+ print("\n--- %s ---" % mname, flush=True)+ ag = load(mname)+ per = {}+ for lg, (cases, gold, _) in sorted(suites.items()):+ lgs, idx, secs, dropped = score_cases(ag, cases, tag="%s/%s" % (mname, lg))+ rows = [(gold[ci], softmax_t(z, temp_for(ag, qt, k)) if z is not None else None)+ for (ci, _, qt, k), z in zip(idx, lgs)]+ m = metrics(rows); m["seconds"] = round(secs, 1); m["dropped"] = dropped+ per[lg] = m+ print(" %-8s acc %.3f f1 %.3f ECE %.3f conf %.3f (%.0f q/s)"+ % (lg, m["accuracy"], m["macro_f1"], m["ece"], m["mean_confidence"],+ m["n"] / max(secs, 1e-9)), flush=True)+ accs = [v["accuracy"] for v in per.values()]+ results["part_a"]["by_model"][mname] = {+ "per_language": per,+ "macro_accuracy": round(float(np.mean(accs)), 4),+ "macro_ece": round(float(np.mean([v["ece"] for v in per.values()])), 4),+ "n_languages": len(per),+ "languages_above_random": int(sum(1 for a in accs if a > 3.0 / N_OPTS)),+ }+ print(" MACRO acc %.4f | ECE %.4f | %d/%d langs > 3x random"+ % (results["part_a"]["by_model"][mname]["macro_accuracy"],+ results["part_a"]["by_model"][mname]["macro_ece"],+ results["part_a"]["by_model"][mname]["languages_above_random"], len(per)), flush=True)+ del ag; gc.collect()+ json.dump(results, open(OUT, "w"), indent=2)+++# ------------------------------------------------------------------ part B+def build_typed_decisions():+ import pandas as pd+ p = os.path.join(REPO, "typed-decisions", "all", "test-00000-of-00001.parquet")+ df = pd.read_parquet(p)+ cases, gold, wfs = [], [], []+ for _, r in df.iterrows():+ qs = json.loads(r["questions"]); g = json.loads(r["gold"])+ st = r["state"]+ try:+ st = json.loads(st)+ except Exception:+ pass+ gm = {}+ for qid, qd in qs.items():+ gg = g[qid]+ if qd["type"] == "choice":+ keys = list(qd["criteria"].keys())+ gm[qid] = {"idx": keys.index(str(gg["label"])),+ "soft": [float(gg.get("probabilities", {}).get(k, 0.0)) for k in keys]}+ elif qd["type"] == "noul":+ pt = float(gg.get("probabilities", {}).get("true", gg.get("noul", 0.5)))+ gm[qid] = {"idx": 1 if str(gg["label"]).lower() == "true" else 0, "soft": [1 - pt, pt]}+ else:+ n = len(qd["criteria"])+ gm[qid] = {"idx": int(gg["label"]),+ "soft": [float(gg.get("probabilities", {}).get(str(i), 0.0)) for i in range(n)],+ "gold_score": float(gg.get("score", float(gg["label"])))}+ cases.append((st, qs)); gold.append(gm); wfs.append(r["workflow"])+ return cases, gold, wfs+++def run_part_b(results):+ print("\n=== PART B: typed-decisions, 400 cases / 2,000 decisions, all 3 checkpoints ===\n",+ flush=True)+ cases, gold, wfs = build_typed_decisions()+ # reference points that are NOT measured here - published / independently measured+ results["part_b"] = {"reference_points": {+ "jev_1.13.0_published": {"accuracy": 0.727, "soft_accuracy": 0.580, "brier": 0.148,+ "ece": 0.144, "score_mae": 0.391, "ms_per_case": 710,+ "source": "figure quoted in the laya repo's own comparison table"},+ "teacher_self_agreement_ceiling": {"accuracy": 0.735},+ "modernbert_base_specialist": {"accuracy": 0.646},+ "random_guess": {"accuracy": 0.3175},+ "per_question_majority_class": {"accuracy": 0.4610},+ "note": "Jev was NOT run here - no TypeSafe API credential is available. These are "+ "published numbers reproduced for context, not a measured head-to-head.",+ }, "by_model": {}}+ for mname in ("english", "multilingual", "typed-decisions"):+ print("--- %s ---" % mname, flush=True)+ ag = load(mname)+ lgs, idx, secs, dropped = score_cases(ag, cases, tag=mname)+ rows, soft, brier_s, mae, w1, by_wf, by_qt = [], [], [], [], [], {}, {}+ for (ci, qid, qt, k), z in zip(idx, lgs):+ g = gold[ci][qid]+ if z is None:+ rows.append((g["idx"], None)); continue+ p = softmax_t(z, temp_for(ag, qt, k))+ rows.append((g["idx"], p))+ by_wf.setdefault(wfs[ci], []).append((g["idx"], p))+ by_qt.setdefault({0: "choice", 1: "score", 2: "noul"}[qt], []).append((g["idx"], p))+ gp = np.asarray(g["soft"], float)+ if gp.sum() > 0:+ gp = gp / gp.sum()+ pp = p[:len(gp)] if len(p) >= len(gp) else np.pad(p, (0, len(gp) - len(p)))+ pp = pp / max(pp.sum(), 1e-12)+ soft.append(float((pp * gp).sum())); brier_s.append(float(((pp - gp) ** 2).sum()))+ if "gold_score" in g:+ exp = float((np.arange(len(p)) * p).sum())+ mae.append(abs(exp - g["gold_score"])); w1.append(float(abs(exp - g["gold_score"]) <= 1))+ m = metrics(rows)+ m.update(soft_accuracy=round(float(np.mean(soft)), 4) if soft else None,+ brier_vs_soft=round(float(np.mean(brier_s)), 4) if brier_s else None,+ score_mae=round(float(np.mean(mae)), 4) if mae else None,+ within_1_level=round(float(np.mean(w1)), 4) if w1 else None,+ seconds=round(secs, 1), dropped=dropped,+ ms_per_case=round(1000 * secs / len(cases), 1),+ by_workflow={k: metrics(v) for k, v in sorted(by_wf.items())},+ by_question_type={k: metrics(v) for k, v in sorted(by_qt.items())})+ results["part_b"]["by_model"][mname] = m+ print(" acc %.4f | soft %.4f | brier(soft) %s | ECE %.4f | MAE %s | %.0f ms/case"+ % (m["accuracy"], m["soft_accuracy"] or 0, m["brier_vs_soft"], m["ece"],+ m["score_mae"], m["ms_per_case"]), flush=True)+ for wf, v in m["by_workflow"].items():+ print(" %-28s acc %.3f (n=%d)" % (wf, v["accuracy"], v["n"]), flush=True)+ del ag; gc.collect()+ json.dump(results, open(OUT, "w"), indent=2)+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--langs", type=int, default=0, help="cap number of languages (0 = all)")+ ap.add_argument("--per-lang", type=int, default=120)+ ap.add_argument("--skip-a", action="store_true")+ ap.add_argument("--skip-b", action="store_true")+ a = ap.parse_args()++ results = {"meta": {"timestamp": time.strftime("%Y-%m-%d %H:%M:%S"), "device": "cpu",+ "torch": torch.__version__, "laya": laya.__version__,+ "threads": torch.get_num_threads()}}+ if os.path.exists(OUT):+ try:+ results = json.load(open(OUT)); results["meta"]["timestamp"] = time.strftime("%Y-%m-%d %H:%M:%S")+ except Exception:+ pass+ if not a.skip_a:+ langs = massive_languages()+ if a.langs:+ langs = langs[:a.langs]+ run_part_a(results, langs, a.per_lang)+ if not a.skip_b:+ run_part_b(results)+ json.dump(results, open(OUT, "w"), indent=2)+ print("\nwrote %s" % OUT, flush=True)+++if __name__ == "__main__":+ main()diff --git a/research/scripts/build_benchmark_nb.py b/research/scripts/build_benchmark_nb.pynew file mode 100644index 00000000..8b131f2d--- /dev/null+++ b/research/scripts/build_benchmark_nb.py@@ -0,0 +1,840 @@+"""Emit laya_benchmark_colab.ipynb: an extensive head-to-head benchmark of+convaiinnovations/laya (ModernBERT-large, English) vs convaiinnovations/laya-multilingual+(mmBERT-base, 100+ languages), on a Colab T4.++Writes one laya_benchmark_results.json that can be handed back for analysis.+"""+import json+import os++HERE = os.path.dirname(os.path.abspath(__file__))+++def md(t):+ return {"cell_type": "markdown", "metadata": {}, "source": t.strip("\n").splitlines(keepends=True)}+++def code(t):+ return {"cell_type": "code", "execution_count": None, "metadata": {}, "outputs": [],+ "source": t.strip("\n").splitlines(keepends=True)}+++CELLS = [+ md("""+# Laya vs Laya-Multilingual — extensive head-to-head benchmark++Runs both open checkpoints over the same questions and writes a single+`laya_benchmark_results.json`.++| | `convaiinnovations/laya` | `convaiinnovations/laya-multilingual` |+|---|---|---|+| encoder | ModernBERT-large (English) | mmBERT-base (100+ languages) |+| params | 421.3M (394.8M enc + 26.5M head) | 321.9M (306.9M enc + 15.0M head) |+| context | 512 tokens | 1024 tokens |+| shipped temperatures | fitted per type & option-count | **none (all 1.0)** |++**What is measured**++1. **typed-decisions** — 400 cases / 2,000 decisions, 4 workflows (the benchmark behind the published 0.766 claim). Zero-shot for both of these checkpoints.+2. **Multilingual** — MASSIVE intent + scenario across 14 languages, XNLI across 15 languages.+3. **English zero-shot** — SST-5, emotion, prompt-injections, plus banking77 as a 77-option stress test.+4. **Latency** — p50/p95 at 1 / 5 / 10 / 50 questions per call.+5. **Option-order robustness** — how often the answer flips when the options are permuted (the independent Jev benchmark measured 13% for Jev; worst LLM 37%).+6. **Calibration repair** — ECE before and after refitting temperature, which matters because `laya-multilingual` shipped uncalibrated.++**Setup:** Runtime → Change runtime type → **T4 GPU**. Then Run All (~25–40 min).+Both models see byte-identical questions (fixed seed), so every difference is the model.+"""),++ md("## 1. GPU and environment"),+ code("""+import subprocess, sys, os, json, time+print(subprocess.run(["nvidia-smi","--query-gpu=name,memory.total,compute_cap","--format=csv,noheader"],+ capture_output=True, text=True).stdout.strip() or "NO GPU - set Runtime > Change runtime type > T4 GPU")+import torch+print("torch", torch.__version__, "| cuda", torch.cuda.is_available(),+ "| device", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "cpu")+assert torch.cuda.is_available(), "Enable the T4 GPU runtime before running."+"""),++ md("## 2. Dependencies"),+ code("""+!pip -q install -U "laya>=0.1.6" "transformers>=4.45" "datasets>=3.0" safetensors huggingface_hub 2>&1 | tail -3+import laya, transformers, datasets+print("laya", getattr(laya, "__version__", "?"), "| transformers", transformers.__version__, "| datasets", datasets.__version__)+"""),++ md("""## 3. Download both checkpoints++`reference_compile` is forced off after loading: the `laya` package does not disable it (only+`rl_agent_api.py` does), and torch.compile is a loss on small batches / few SMs like a T4."""),+ code("""+import torch, json, os, time+from huggingface_hub import snapshot_download+from safetensors import safe_open++REPOS = {"laya": "convaiinnovations/laya", "laya-multilingual": "convaiinnovations/laya-multilingual"}+PATHS, MODEL_META = {}, {}++def patch_tokenizer_config(model_dir):+ '''laya-multilingual ships extra_special_tokens as a LIST (inherited from the mmBERT/Gemma+ tokenizer); transformers expects a dict and raises+ AttributeError: list object has no attribute keys. Without this, AutoTokenizer -- and so+ laya.load() -- fails outright on that repo. Also normalises tokenizer_class for 4.x/5.x.'''+ p = os.path.join(model_dir, "tokenizer", "tokenizer_config.json")+ if not os.path.exists(p):+ return []+ c = json.load(open(p))+ changed = []+ est = c.get("extra_special_tokens")+ if isinstance(est, list):+ c["extra_special_tokens"] = {("extra_%d" % i): t for i, t in enumerate(est)}+ changed.append("extra_special_tokens list->dict")+ if c.get("tokenizer_class") in (None, "TokenizersBackend"):+ c["tokenizer_class"] = "PreTrainedTokenizerFast"+ c.pop("backend", None); c.pop("is_local", None)+ changed.append("tokenizer_class")+ if changed:+ json.dump(c, open(p, "w"), indent=2)+ return changed++for name, repo in REPOS.items():+ t = time.time()+ PATHS[name] = snapshot_download(repo)+ fixed = patch_tokenizer_config(PATHS[name])+ if fixed:+ print(" patched tokenizer config for %s: %s" % (name, ", ".join(fixed)))+ cfg = json.load(open(os.path.join(PATHS[name], "rl_agent_config.json")))+ ecfg = json.load(open(os.path.join(PATHS[name], "encoder", "config.json")))+ tot = enc = 0+ with safe_open(os.path.join(PATHS[name], "model.safetensors"), "pt") as f:+ for k in f.keys():+ n = 1+ for d in f.get_slice(k).get_shape():+ n *= d+ tot += n+ enc += n if k.startswith("encoder.") else 0+ MODEL_META[name] = {+ "repo": repo, "params_total_m": round(tot/1e6, 2), "params_encoder_m": round(enc/1e6, 2),+ "params_head_m": round((tot-enc)/1e6, 2), "hidden_size": ecfg.get("hidden_size"),+ "num_layers": ecfg.get("num_hidden_layers"), "vocab_size": ecfg.get("vocab_size"),+ "max_len": cfg.get("max_len"), "head_max_len": cfg.get("head_max_len"),+ "temperature": cfg.get("temperature"), "temperature_by_options": cfg.get("temperature_by_options", {}),+ "training": cfg.get("training", {}), "download_s": round(time.time()-t, 1),+ "tokenizer_config_patched": fixed,+ }+ print(name, "->", PATHS[name])+ print(" ", json.dumps({k: v for k, v in MODEL_META[name].items() if k != "temperature_by_options"}))++def load_agent(name):+ import laya+ ag = laya.load(PATHS[name], device="cuda")+ try:+ ag.model.encoder.config.reference_compile = False # torch.compile hurts on T4-class GPUs+ except Exception as e:+ print("could not disable reference_compile:", e)+ ag.model.eval()+ return ag+"""),++ md("## 4. Benchmark engine — batched scoring and metrics"),+ code(r"""+%%writefile bench_engine.py+import json, math, sys, time+import numpy as np+import torch+from laya.common import QTYPES, build_sequence, collate_items, render_options, temp_bucket++def to_internal(qdef):+ t = qdef["type"]+ crit = qdef.get("criteria")+ if t == "choice" and isinstance(crit, list):+ crit = {c: None for c in crit}+ ins = qdef["instructions"]+ if not isinstance(ins, str):+ ins = json.dumps(ins)+ return {"t": t, "ins": ins, "crit": crit}+[email protected]_grad()+def score_cases(agent, cases, max_tokens=16384, max_seqs=128, progress=""):+ '''cases: [(state, {qid: qdef})] -> (raw_logits per question, index, seconds, n_dropped).++ Every question of every case is one sequence. Sequences are length-sorted and packed into+ token-budgeted batches so padding stays small. Returns UNCALIBRATED logits; temperature is+ applied later so raw/calibrated variants come from one forward pass.+ '''+ max_len = agent.cfg.get("max_len", 512)+ head_max_len = agent.cfg.get("head_max_len", 192)+ items, index, dropped = [], [], 0+ for ci, (state, questions) in enumerate(cases):+ for qid, qdef in questions.items():+ q = to_internal(qdef)+ try:+ ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)+ except Exception:+ index.append((ci, qid, QTYPES[q["t"]], 0)); items.append(None); dropped += 1+ continue+ if len(markers) != len(render_options(q)):+ index.append((ci, qid, QTYPES[q["t"]], 0)); items.append(None); dropped += 1+ continue+ items.append({"ids": ids, "markers": markers, "qtype": QTYPES[q["t"]]})+ index.append((ci, qid, QTYPES[q["t"]], len(markers)))++ valid = [i for i, it in enumerate(items) if it is not None]+ order = sorted(valid, key=lambda i: len(items[i]["ids"]))+ out = [None] * len(items)+ t0, done, i = time.time(), 0, 0+ while i < len(order):+ j, L = i, 0+ while j < len(order) and j - i < max_seqs and max(L, len(items[order[j]]["ids"])) * (j - i + 1) <= max_tokens:+ L = max(L, len(items[order[j]]["ids"])); j += 1+ j = max(j, i + 1)+ sel = [items[order[t]] for t in range(i, j)]+ b = collate_items([sel], agent.tok.pad_token_id)+ with torch.autocast("cuda", dtype=agent.dtype, enabled=True):+ logits, _ = agent.model(b["input_ids"].cuda(), b["attention_mask"].cuda(),+ b["marker_pos"].cuda(), b["marker_mask"].cuda(), b["qtype"].cuda())+ logits = logits.float().cpu().numpy()+ for r in range(j - i):+ out[order[i + r]] = logits[r, :len(sel[r]["markers"])]+ done += j - i+ if progress:+ el = time.time() - t0+ msg = "\r [%s] %d/%d seq | %.0f seq/s | ETA %ds " % (+ progress, done, len(order), done/max(el, 1e-9),+ int(el*(len(order)-done)/max(1, done)))+ sys.stdout.write(msg)+ sys.stdout.flush()+ i = j+ if progress:+ print("\r [%s] %d sequences in %.1fs (%d dropped: options did not fit)%s" % (+ progress, len(order), time.time()-t0, dropped, " "*20))+ return out, index, time.time() - t0, dropped++def softmax_t(z, t=1.0):+ z = np.asarray(z, dtype=np.float64) / max(1e-3, float(t))+ e = np.exp(z - z.max())+ return e / e.sum()++def temp_for(agent, qt, k, calibrated=True):+ if not calibrated:+ return 1.0+ return float(agent.temperature_by_options.get(temp_bucket(qt, k), agent.temperature[qt]))++# ------------------------------------------------------------------ metrics+def ece_score(conf, correct, bins=15):+ conf, correct = np.asarray(conf, float), np.asarray(correct, float)+ if len(conf) == 0:+ return float("nan")+ edges = np.linspace(0, 1, bins + 1)+ e = 0.0+ for lo, hi in zip(edges[:-1], edges[1:]):+ sel = (conf > lo) & (conf <= hi)+ if sel.any():+ e += sel.mean() * abs(conf[sel].mean() - correct[sel].mean())+ return float(e)++def macro_f1(gold, pred):+ gold, pred = np.asarray(gold), np.asarray(pred)+ f1 = []+ for c in sorted(set(gold.tolist()) | set(pred.tolist())):+ tp = int(((pred == c) & (gold == c)).sum())+ fp = int(((pred == c) & (gold != c)).sum())+ fn = int(((pred != c) & (gold == c)).sum())+ f1.append(2*tp / max(1, 2*tp + fp + fn))+ return float(np.mean(f1))++def aurc(conf, correct):+ conf, correct = np.asarray(conf, float), np.asarray(correct, float)+ if len(conf) == 0:+ return float("nan")+ o = np.argsort(-conf)+ return float((np.cumsum(1 - correct[o]) / np.arange(1, len(o)+1)).mean())++def hard_metrics(rows):+ '''rows: [(gold_idx, probs)] -> standard single-label classification + calibration metrics.'''+ rows = [r for r in rows if r[1] is not None]+ if not rows:+ return {"n": 0}+ gold = np.array([g for g, _ in rows])+ pred = np.array([int(np.argmax(p)) for _, p in rows])+ conf = np.array([float(np.max(p)) for _, p in rows])+ corr = (pred == gold).astype(float)+ m = {"n": len(rows),+ "accuracy": float(corr.mean()),+ "macro_f1": macro_f1(gold, pred),+ "ece": ece_score(conf, corr),+ "brier": float(np.mean([((np.asarray(p) - np.eye(len(p))[g])**2).sum() for g, p in rows])),+ "nll": float(np.mean([-math.log(max(float(p[g]), 1e-12)) for g, p in rows])),+ "aurc": aurc(conf, corr),+ "mean_confidence": float(conf.mean())}+ for cov in (0.5, 0.8):+ k = max(1, int(len(conf)*cov))+ m["acc_at_%d_coverage" % int(cov*100)] = float(corr[np.argsort(-conf)[:k]].mean())+ return m++def fit_temperature(pairs, lo=0.2, hi=10.0, steps=160):+ '''pairs: [(logits, gold_idx)] -> single temperature minimising NLL (grid search, no autograd).'''+ pairs = [(np.asarray(z, float), int(g)) for z, g in pairs if z is not None]+ if len(pairs) < 25:+ return 1.0+ best_t, best = 1.0, float("inf")+ for t in np.geomspace(lo, hi, steps):+ tot = 0.0+ for z, g in pairs:+ zz = z/t+ zz = zz - zz.max()+ tot += -(zz[g] - math.log(np.exp(zz).sum()))+ if tot < best:+ best, best_t = tot, float(t)+ return round(float(best_t), 4)+"""),++ md("""## 5. Task suites++Every suite is built **once**, with a fixed seed, and reused for both models — so the two+checkpoints answer byte-identical questions. Option sets for many-label tasks use+`gold + N-1 distractors`, matching how the models were trained (the training recipe samples+5–20 options per choice question rather than showing the full label space)."""),++ code("""+import random, json+import numpy as np+from datasets import load_dataset++SEED = 13+N_OPTS = 20 # options per many-label choice question (gold + 19 distractors)+PER_LANG = 300 # cases per language+SUITES = {} # name -> {"cases": [(state, questions)], "gold": [{qid: gold_idx}], "meta": {...}}++def build_choice_questions(rng, gold, all_labels, n_opts, instructions, qid="label"):+ pool = [x for x in all_labels if x != gold]+ keys = [gold] + rng.sample(pool, min(n_opts-1, len(pool)))+ rng.shuffle(keys)+ crit = {k: k.replace("_", " ").replace(".", ": ") for k in keys}+ return {qid: {"type": "choice", "instructions": instructions, "criteria": crit}}, keys.index(gold)++def register(name, cases, golds, **meta):+ SUITES[name] = {"cases": cases, "gold": golds, "meta": meta}+ nq = sum(len(c[1]) for c in cases)+ print(" %-34s %4d cases / %5d questions %s" % (name, len(cases), nq, meta.get("note", "")))+"""),++ md("### 5a. typed-decisions (English, 400 cases / 2,000 decisions)"),+ code("""+td_cases, td_golds, td_wf = [], [], []+try:+ td = load_dataset("LocalLLaMA/typed-decisions", "all", split="test")+ for r in td:+ questions = json.loads(r["questions"]) if isinstance(r["questions"], str) else r["questions"]+ gold = json.loads(r["gold"]) if isinstance(r["gold"], str) else r["gold"]+ state = r["state"]+ try:+ state = json.loads(state)+ except Exception:+ pass+ gmap = {}+ for qid, qdef in questions.items():+ g = gold[qid]+ if qdef["type"] == "choice":+ keys = list(qdef["criteria"].keys())+ gmap[qid] = {"idx": keys.index(str(g["label"])), "keys": keys,+ "soft": [float(g.get("probabilities", {}).get(k, 0.0)) for k in keys]}+ elif qdef["type"] == "noul":+ idx = 1 if str(g["label"]).lower() == "true" else 0+ pt = float(g.get("probabilities", {}).get("true", g.get("noul", 0.5)))+ gmap[qid] = {"idx": idx, "keys": ["false", "true"], "soft": [1-pt, pt]}+ else: # score+ n = len(qdef["criteria"])+ gmap[qid] = {"idx": int(g["label"]), "keys": [str(i) for i in range(n)],+ "soft": [float(g.get("probabilities", {}).get(str(i), 0.0)) for i in range(n)],+ "gold_score": float(g.get("score", float(g["label"])))}+ td_cases.append((state, questions))+ td_golds.append(gmap)+ td_wf.append(r["workflow"])+ register("typed_decisions", td_cases, td_golds, note="4 workflows, soft teacher labels",+ workflows=td_wf, soft=True)+except Exception as e:+ print("typed-decisions FAILED:", type(e).__name__, e)+"""),++ md("### 5b. Multilingual — MASSIVE intent & scenario (14 languages), XNLI (15 languages)"),+ code("""+MASSIVE_LANGS = ["en","de","fr","es","pt","ru","tr","ar","hi","ta","zh-CN","ja","ko","sw"]+XNLI_LANGS = ["en","de","fr","es","ru","tr","ar","hi","ur","vi","th","el","bg","zh","sw"]++for task, short, instr in [("mteb/amazon_massive_intent", "massive_intent",+ "What is the user asking for in `utterance`?"),+ ("mteb/amazon_massive_scenario", "massive_scenario",+ "Which domain does `utterance` belong to?")]:+ for lg in MASSIVE_LANGS:+ try:+ d = load_dataset(task, lg, split="test")+ labels = sorted(set(d["label_text"]))+ rng = random.Random(SEED)+ cases, golds = [], []+ for r in list(d)[:PER_LANG]:+ qs, gi = build_choice_questions(rng, r["label_text"], labels, N_OPTS, instr)+ cases.append(({"utterance": r["text"]}, qs))+ golds.append({"label": {"idx": gi}})+ register("%s.%s" % (short, lg), cases, golds, lang=lg, n_options=N_OPTS,+ label_space=len(labels), family=short)+ except Exception as e:+ print(" FAIL %s %s: %s" % (short, lg, str(e)[:90]))++NLI_CRIT = {"entailment": "the premise implies the hypothesis is true",+ "neutral": "the premise neither implies nor contradicts the hypothesis",+ "contradiction": "the premise implies the hypothesis is false"}+for lg in XNLI_LANGS:+ try:+ d = load_dataset("facebook/xnli", lg, split="test")+ cases, golds = [], []+ for r in list(d)[:PER_LANG]:+ qs = {"relation": {"type": "choice",+ "instructions": "What is the relationship between `premise` and `hypothesis`?",+ "criteria": dict(NLI_CRIT)}}+ cases.append(({"premise": r["premise"], "hypothesis": r["hypothesis"]}, qs))+ golds.append({"relation": {"idx": int(r["label"])}})+ register("xnli.%s" % lg, cases, golds, lang=lg, n_options=3, family="xnli")+ except Exception as e:+ print(" FAIL xnli %s: %s" % (lg, str(e)[:90]))+"""),++ md("""### 5c. English tasks++`sst5`, `emotion` and `prompt_injections` were held out of Laya's training entirely (true+zero-shot). `ag_news` and `boolq` were in the training mix, so they measure retention rather+than generalisation. `banking77` is the many-option stress test — 77 options at once, versus+Jev's documented hard cap at 255."""),+ code("""+rng = random.Random(SEED)++# SST-5: ordinal score+try:+ d = load_dataset("SetFit/sst5", split="test")+ crit = ["very negative", "negative", "neutral", "positive", "very positive"]+ cases, golds = [], []+ for r in list(d)[:600]:+ cases.append(({"text": r["text"]},+ {"sentiment": {"type": "score", "instructions": "How positive is the sentiment of `text`?",+ "criteria": crit}}))+ golds.append({"sentiment": {"idx": int(r["label"]), "gold_score": float(r["label"])}})+ register("en.sst5", cases, golds, note="zero-shot, ordinal", family="english", held_out=True)+except Exception as e: print(" FAIL sst5:", str(e)[:90])++# emotion: 6-way choice+try:+ d = load_dataset("dair-ai/emotion", "split", split="test")+ names = ["sadness","joy","love","anger","fear","surprise"]+ cases, golds = [], []+ for r in list(d)[:600]:+ cases.append(({"text": r["text"]},+ {"emotion": {"type": "choice",+ "instructions": "Which emotion is most strongly expressed in `text`?",+ "criteria": {n: None for n in names}}}))+ golds.append({"emotion": {"idx": int(r["label"])}})+ register("en.emotion", cases, golds, note="zero-shot", family="english", held_out=True)+except Exception as e: print(" FAIL emotion:", str(e)[:90])++# prompt injections: noul+try:+ d = load_dataset("deepset/prompt-injections", split="test")+ cases, golds = [], []+ for r in list(d):+ cases.append(({"text": r["text"]},+ {"injection": {"type": "noul",+ "instructions": "Does `text` try to inject or override instructions given to an AI system?"}}))+ golds.append({"injection": {"idx": int(r["label"])}})+ register("en.prompt_injections", cases, golds, note="zero-shot", family="english", held_out=True)+except Exception as e: print(" FAIL prompt_injections:", str(e)[:90])++# banking77: 77-option stress test+try:+ d = load_dataset("PolyAI/banking77", split="test")+ names = d.features["label"].names+ cases, golds = [], []+ for r in list(d)[:500]:+ cases.append(({"message": r["text"]},+ {"intent": {"type": "choice", "instructions": "Which banking intent does `message` express?",+ "criteria": {n.replace("_", " "): None for n in names}}}))+ golds.append({"intent": {"idx": int(r["label"])}})+ register("en.banking77_full", cases, golds, note="77 options at once", family="english",+ n_options=len(names), held_out=True)+except Exception as e: print(" FAIL banking77:", str(e)[:90])++# ag_news + boolq: in-task retention+try:+ d = load_dataset("fancyzhx/ag_news", split="test")+ crit = {"world": "world news and international politics", "sports": "sports",+ "business": "business and economy", "sci_tech": "science and technology"}+ cases, golds = [], []+ for r in list(d)[:600]:+ cases.append(({"article": r["text"]},+ {"topic": {"type": "choice", "instructions": "What is the topic of `article`?",+ "criteria": dict(crit)}}))+ golds.append({"topic": {"idx": int(r["label"])}})+ register("en.ag_news", cases, golds, note="in-task (trained on)", family="english", held_out=False)+except Exception as e: print(" FAIL ag_news:", str(e)[:90])++try:+ d = load_dataset("google/boolq", split="validation")+ cases, golds = [], []+ for r in list(d)[:600]:+ cases.append(({"passage": r["passage"], "question": r["question"]},+ {"answer": {"type": "noul",+ "instructions": "Based on `passage`, is the answer to `question` yes?"}}))+ golds.append({"answer": {"idx": int(bool(r["answer"]))}})+ register("en.boolq", cases, golds, note="in-task (trained on)", family="english", held_out=False)+except Exception as e: print(" FAIL boolq:", str(e)[:90])++print()+print("suites built:", len(SUITES), "| total questions:",+ sum(sum(len(c[1]) for c in s["cases"]) for s in SUITES.values()))+"""),++ md("## 6. Run every suite on both models"),+ code("""+import bench_engine as BE+import numpy as np, gc, time, json++RESULTS = {"meta": {"seed": SEED, "n_opts": N_OPTS, "per_lang": PER_LANG,+ "timestamp": time.strftime("%Y-%m-%d %H:%M:%S"),+ "gpu": torch.cuda.get_device_name(0),+ "torch": torch.__version__, "transformers": transformers.__version__,+ "models": MODEL_META},+ "suites": {}}+RAW = {} # model -> suite -> (logits, index) kept in RAM for the calibration-repair section++def evaluate_suite(agent, model_name, suite_name):+ S = SUITES[suite_name]+ logits, index, secs, dropped = BE.score_cases(agent, S["cases"], progress="%s/%s" % (model_name, suite_name))+ RAW[model_name][suite_name] = (logits, index)+ soft = S["meta"].get("soft", False)+ rows_cal, rows_raw, extra = [], [], {"soft_acc": [], "brier_soft": [], "tv": [], "kl": [],+ "score_mae": [], "within_1": []}+ per_q_type = {}+ for (ci, qid, qt, k), z in zip(index, logits):+ g = S["gold"][ci].get(qid)+ if g is None:+ continue+ if z is None:+ rows_cal.append((g["idx"], None)); rows_raw.append((g["idx"], None)); continue+ p_cal = BE.softmax_t(z, BE.temp_for(agent, qt, k, True))+ p_raw = BE.softmax_t(z, 1.0)+ rows_cal.append((g["idx"], p_cal)); rows_raw.append((g["idx"], p_raw))+ per_q_type.setdefault(qt, []).append((g["idx"], p_cal))+ if soft and "soft" in g:+ gp = np.asarray(g["soft"], float)+ if gp.sum() > 0:+ gp = gp / gp.sum()+ pp = p_cal[:len(gp)] if len(p_cal) >= len(gp) else np.pad(p_cal, (0, len(gp)-len(p_cal)))+ pp = pp / max(pp.sum(), 1e-12)+ extra["soft_acc"].append(float((pp*gp).sum()))+ extra["brier_soft"].append(float(((pp-gp)**2).sum()))+ extra["tv"].append(float(0.5*np.abs(pp-gp).sum()))+ extra["kl"].append(float((gp*np.log(np.clip(gp/np.clip(pp,1e-12,None),1e-12,1e4))).sum()))+ if "gold_score" in g:+ exp = float((np.arange(len(p_cal))*p_cal).sum())+ extra["score_mae"].append(abs(exp - g["gold_score"]))+ extra["within_1"].append(float(abs(exp - g["gold_score"]) <= 1.0))++ out = {"calibrated": BE.hard_metrics(rows_cal), "raw": BE.hard_metrics(rows_raw),+ "seconds": round(secs, 2), "dropped_questions": dropped,+ "questions_per_second": round(len([r for r in rows_cal if r[1] is not None])/max(secs,1e-9), 1)}+ for k2, v in extra.items():+ if v:+ out[k2] = float(np.mean(v))+ if len(per_q_type) > 1:+ names = {0: "choice", 1: "score", 2: "noul"}+ out["by_question_type"] = {names[qt]: BE.hard_metrics(rs) for qt, rs in sorted(per_q_type.items())}+ return out++for model_name in REPOS:+ print("\\n" + "="*78 + "\\n %s\\n" % model_name + "="*78)+ agent = load_agent(model_name)+ RAW[model_name] = {}+ for suite_name in SUITES:+ try:+ r = evaluate_suite(agent, model_name, suite_name)+ RESULTS["suites"].setdefault(suite_name, {})[model_name] = r+ c = r["calibrated"]+ print(" %-32s acc %.3f f1 %.3f ECE %.3f n=%d" %+ (suite_name, c.get("accuracy", float("nan")), c.get("macro_f1", float("nan")),+ c.get("ece", float("nan")), c.get("n", 0)))+ except Exception as e:+ import traceback; traceback.print_exc()+ RESULTS["suites"].setdefault(suite_name, {})[model_name] = {"error": "%s: %s" % (type(e).__name__, e)}+ globals()["AGENT_" + model_name.replace("-", "_")] = agent+ gc.collect(); torch.cuda.empty_cache()++json.dump(RESULTS, open("laya_benchmark_results.json", "w"), indent=2)+print("\\ncheckpoint saved")+"""),++ md("""### 6b. Control — matched context budget++The two checkpoints ship different context windows (512 vs 1024 tokens), so on long states+`laya-multilingual` sees more of the input. This re-runs typed-decisions with **both** models+capped at 512 tokens, isolating the architecture from the context-window advantage."""),+ code("""+import bench_engine as BE+import numpy as np++RESULTS["typed_decisions_matched_512"] = {}+if "typed_decisions" in SUITES:+ S = SUITES["typed_decisions"]+ for model_name in REPOS:+ agent = globals()["AGENT_" + model_name.replace("-", "_")]+ orig = agent.cfg.get("max_len")+ agent.cfg["max_len"] = 512+ try:+ logits, index, secs, dropped = BE.score_cases(agent, S["cases"], progress="%s/td@512" % model_name)+ rows = []+ for (ci, qid, qt, k), z in zip(index, logits):+ g = S["gold"][ci].get(qid)+ if g is None or z is None:+ continue+ rows.append((g["idx"], BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))+ RESULTS["typed_decisions_matched_512"][model_name] = BE.hard_metrics(rows)+ print(" %-22s max_len 512 -> acc %.3f ECE %.3f n=%d"+ % (model_name, RESULTS["typed_decisions_matched_512"][model_name]["accuracy"],+ RESULTS["typed_decisions_matched_512"][model_name]["ece"],+ RESULTS["typed_decisions_matched_512"][model_name]["n"]))+ finally:+ agent.cfg["max_len"] = orig+"""),++ md("## 7. Latency (p50 / p95 at 1, 5, 10, 50 questions per call)"),+ code("""+import numpy as np, time, torch++LAT_STATE = {"ticket": {"subject": "Payout failing",+ "messages": [{"from": "customer",+ "text": "Hi, my Stripe payouts have failed for 3 days and I am losing sales. Please help ASAP. " * 6}]}}+Q_NOUL = {"type": "noul", "instructions": "Does `ticket.messages[0].text` express urgency?"}+Q_CHOICE = {"type": "choice", "instructions": "Which team should handle this?",+ "criteria": {"billing": "payments", "technical": "bugs and integrations", "sales": "pricing"}}++RESULTS["latency"] = {}+for model_name in REPOS:+ agent = globals()["AGENT_" + model_name.replace("-", "_")]+ res = {}+ for nq in (1, 5, 10, 50):+ qs = {("q%d" % i): (Q_NOUL if i % 2 else Q_CHOICE) for i in range(nq)}+ for _ in range(3):+ agent.system_one(LAT_STATE, qs)+ ts = []+ for _ in range(20):+ torch.cuda.synchronize(); t = time.perf_counter()+ agent.system_one(LAT_STATE, qs)+ torch.cuda.synchronize(); ts.append((time.perf_counter()-t)*1000)+ res["%d_questions" % nq] = {"p50_ms": round(float(np.percentile(ts, 50)), 1),+ "p95_ms": round(float(np.percentile(ts, 95)), 1),+ "ms_per_question": round(float(np.percentile(ts, 50))/nq, 2)}+ RESULTS["latency"][model_name] = res+ print(model_name, json.dumps(res))+"""),++ md("""## 8. Option-order robustness++Same question, options permuted. A well-behaved decision model should pick the same label.+The independent Jev benchmark measured a **13% flip rate for Jev**, with the worst LLM at 37%."""),+ code("""+import random, numpy as np+import bench_engine as BE++PB_SUITES = [s for s in ("massive_intent.en", "en.emotion", "xnli.en") if s in SUITES]+RESULTS["option_order_robustness"] = {}++for model_name in REPOS:+ agent = globals()["AGENT_" + model_name.replace("-", "_")]+ out = {}+ for sname in PB_SUITES:+ S = SUITES[sname]+ base = S["cases"][:200]+ rng = random.Random(99)+ perm_cases, perm_maps = [], []+ for state, questions in base:+ qid = list(questions)[0]+ qdef = questions[qid]+ keys = list(qdef["criteria"].keys())+ order = list(range(len(keys))); rng.shuffle(order)+ newcrit = {keys[i]: qdef["criteria"][keys[i]] for i in order}+ perm_cases.append((state, {qid: {**qdef, "criteria": newcrit}}))+ perm_maps.append((keys, [keys[i] for i in order]))+ la, ia, _, _ = BE.score_cases(agent, base, progress="")+ lb, ib, _, _ = BE.score_cases(agent, perm_cases, progress="")+ flips = comp = 0+ for (za, (_, _, _, ka)), (zb, (_, _, _, kb)), (keys, pkeys) in zip(zip(la, ia), zip(lb, ib), perm_maps):+ if za is None or zb is None:+ continue+ comp += 1+ if keys[int(np.argmax(za))] != pkeys[int(np.argmax(zb))]:+ flips += 1+ out[sname] = {"n": comp, "flip_rate": round(flips/max(1, comp), 4)}+ print(" %-22s %-24s flip rate %.3f (n=%d)" % (model_name, sname, out[sname]["flip_rate"], comp))+ RESULTS["option_order_robustness"][model_name] = out+"""),++ md("""## 9. Calibration repair++`laya-multilingual` ships with `temperature = [1.0, 1.0, 1.0]` and no per-option-count buckets —+it was never calibrated. This refits one temperature per (question type, option-count bucket) on+**half** of each suite and reports ECE on the other half, so the improvement is measured+out-of-sample. It shows how much of any calibration gap is a missing post-processing step rather+than a property of the model."""),+ code("""+import numpy as np+import bench_engine as BE+from laya.common import temp_bucket++RESULTS["calibration_repair"] = {}+for model_name in REPOS:+ agent = globals()["AGENT_" + model_name.replace("-", "_")]+ per_suite, fitted_all = {}, {}+ for sname, S in SUITES.items():+ if sname not in RAW[model_name]:+ continue+ logits, index = RAW[model_name][sname]+ pairs = []+ for (ci, qid, qt, k), z in zip(index, logits):+ g = S["gold"][ci].get(qid)+ if g is not None and z is not None:+ pairs.append((z, g["idx"], qt, k))+ if len(pairs) < 60:+ continue+ half = len(pairs)//2+ fit_set, held = pairs[:half], pairs[half:]+ buckets = {}+ for z, gi, qt, k in fit_set:+ buckets.setdefault(temp_bucket(qt, k), []).append((z, gi))+ fitted = {b: BE.fit_temperature(v) for b, v in buckets.items() if len(v) >= 25}+ rows_ship, rows_fit = [], []+ for z, gi, qt, k in held:+ rows_ship.append((gi, BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))+ rows_fit.append((gi, BE.softmax_t(z, fitted.get(temp_bucket(qt, k),+ BE.temp_for(agent, qt, k, True)))))+ ms, mf = BE.hard_metrics(rows_ship), BE.hard_metrics(rows_fit)+ per_suite[sname] = {"n_heldout": ms["n"], "fitted_temperatures": fitted,+ "ece_shipped": ms["ece"], "ece_refit": mf["ece"],+ "nll_shipped": ms["nll"], "nll_refit": mf["nll"],+ "accuracy": ms["accuracy"]}+ fitted_all.update(fitted)+ RESULTS["calibration_repair"][model_name] = {"per_suite": per_suite}+ if per_suite:+ a = float(np.mean([v["ece_shipped"] for v in per_suite.values()]))+ b = float(np.mean([v["ece_refit"] for v in per_suite.values()]))+ RESULTS["calibration_repair"][model_name]["mean_ece_shipped"] = round(a, 4)+ RESULTS["calibration_repair"][model_name]["mean_ece_refit"] = round(b, 4)+ print("%-22s mean ECE shipped %.4f -> refit %.4f" % (model_name, a, b))+"""),++ md("## 10. Aggregate, save and download"),+ code("""+import numpy as np, json++def agg(prefix_or_family, key="family"):+ out = {}+ for m in REPOS:+ accs, eces, ns = [], [], 0+ for sname, S in SUITES.items():+ if S["meta"].get(key) != prefix_or_family:+ continue+ r = RESULTS["suites"].get(sname, {}).get(m, {})+ c = r.get("calibrated", {})+ if c.get("n"):+ accs.append(c["accuracy"]); eces.append(c["ece"]); ns += c["n"]+ if accs:+ out[m] = {"macro_accuracy": round(float(np.mean(accs)), 4),+ "macro_ece": round(float(np.mean(eces)), 4),+ "n_questions": ns, "n_suites": len(accs)}+ return out++RESULTS["caveats"] = {+ "context_window": "laya max_len=512, laya-multilingual max_len=1024 (shipped configs). "+ "See typed_decisions_matched_512 for the equal-budget control.",+ "shipped_calibration": "laya ships fitted temperatures; laya-multilingual ships all-1.0 "+ "(uncalibrated). See calibration_repair for the out-of-sample refit.",+ "training_overlap": "ag_news and boolq were in Laya's training mix (retention, not generalisation). "+ "sst5, emotion, prompt_injections, banking77 were held out. "+ "Neither checkpoint trained on typed-decisions, MASSIVE or XNLI.",+}++RESULTS["summary"] = {+ "typed_decisions": {m: RESULTS["suites"].get("typed_decisions", {}).get(m, {}).get("calibrated", {})+ for m in REPOS},+ "massive_intent": agg("massive_intent"),+ "massive_scenario": agg("massive_scenario"),+ "xnli": agg("xnli"),+ "english": agg("english"),+}++# English vs non-English split inside each multilingual family+for fam in ("massive_intent", "massive_scenario", "xnli"):+ en, non = {}, {}+ for m in REPOS:+ ea, na = [], []+ for sname, S in SUITES.items():+ if S["meta"].get("family") != fam:+ continue+ c = RESULTS["suites"].get(sname, {}).get(m, {}).get("calibrated", {})+ if not c.get("n"):+ continue+ (ea if S["meta"].get("lang") in ("en",) else na).append(c["accuracy"])+ if ea: en[m] = round(float(np.mean(ea)), 4)+ if na: non[m] = round(float(np.mean(na)), 4)+ RESULTS["summary"][fam + "_english_vs_rest"] = {"english": en, "non_english": non}++json.dump(RESULTS, open("laya_benchmark_results.json", "w"), indent=2)+print(json.dumps(RESULTS["summary"], indent=2))+print("\\nwrote laya_benchmark_results.json (%.1f KB)" % (os.path.getsize("laya_benchmark_results.json")/1024))+"""),++ code("""+# Per-language table, printed for a quick eyeball before you download the JSON+import numpy as np+for fam in ("massive_intent", "massive_scenario", "xnli"):+ langs = sorted({S["meta"]["lang"] for S in SUITES.values() if S["meta"].get("family") == fam})+ if not langs:+ continue+ print("\\n=== %s : accuracy by language ===" % fam)+ print("%-8s %12s %22s %10s" % ("lang", "laya", "laya-multilingual", "delta"))+ for lg in langs:+ sname = "%s.%s" % (fam, lg)+ a = RESULTS["suites"].get(sname, {}).get("laya", {}).get("calibrated", {}).get("accuracy")+ b = RESULTS["suites"].get(sname, {}).get("laya-multilingual", {}).get("calibrated", {}).get("accuracy")+ if a is None or b is None:+ continue+ print("%-8s %12.3f %22.3f %+10.3f" % (lg, a, b, b-a))+"""),++ code("""+try:+ from google.colab import files+ files.download("laya_benchmark_results.json")+except Exception as e:+ print("Not on Colab or download blocked (%s). Grab laya_benchmark_results.json from the file browser." % type(e).__name__)+"""),+]+++def main():+ nb = {+ "cells": CELLS,+ "metadata": {+ "accelerator": "GPU",+ "colab": {"provenance": [], "gpuType": "T4", "toc_visible": True},+ "kernelspec": {"display_name": "Python 3", "name": "python3"},+ "language_info": {"name": "python"},+ },+ "nbformat": 4,+ "nbformat_minor": 0,+ }+ out = os.path.join(HERE, "laya_benchmark_colab.ipynb")+ with open(out, "w") as f:+ json.dump(nb, f, indent=1)+ print("wrote %s (%d cells)" % (out, len(CELLS)))+++if __name__ == "__main__":+ main()diff --git a/research/scripts/laya_benchmark_colab.ipynb b/research/scripts/laya_benchmark_colab.ipynbnew file mode 100644index 00000000..2979c985--- /dev/null+++ b/research/scripts/laya_benchmark_colab.ipynb@@ -0,0 +1,983 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "# Laya vs Laya-Multilingual \u2014 extensive head-to-head benchmark\n",+ "\n",+ "Runs both open checkpoints over the same questions and writes a single\n",+ "`laya_benchmark_results.json`.\n",+ "\n",+ "| | `convaiinnovations/laya` | `convaiinnovations/laya-multilingual` |\n",+ "|---|---|---|\n",+ "| encoder | ModernBERT-large (English) | mmBERT-base (100+ languages) |\n",+ "| params | 421.3M (394.8M enc + 26.5M head) | 321.9M (306.9M enc + 15.0M head) |\n",+ "| context | 512 tokens | 1024 tokens |\n",+ "| shipped temperatures | fitted per type & option-count | **none (all 1.0)** |\n",+ "\n",+ "**What is measured**\n",+ "\n",+ "1. **typed-decisions** \u2014 400 cases / 2,000 decisions, 4 workflows (the benchmark behind the published 0.766 claim). Zero-shot for both of these checkpoints.\n",+ "2. **Multilingual** \u2014 MASSIVE intent + scenario across 14 languages, XNLI across 15 languages.\n",+ "3. **English zero-shot** \u2014 SST-5, emotion, prompt-injections, plus banking77 as a 77-option stress test.\n",+ "4. **Latency** \u2014 p50/p95 at 1 / 5 / 10 / 50 questions per call.\n",+ "5. **Option-order robustness** \u2014 how often the answer flips when the options are permuted (the independent Jev benchmark measured 13% for Jev; worst LLM 37%).\n",+ "6. **Calibration repair** \u2014 ECE before and after refitting temperature, which matters because `laya-multilingual` shipped uncalibrated.\n",+ "\n",+ "**Setup:** Runtime \u2192 Change runtime type \u2192 **T4 GPU**. Then Run All (~25\u201340 min).\n",+ "Both models see byte-identical questions (fixed seed), so every difference is the model."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 1. GPU and environment"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import subprocess, sys, os, json, time\n",+ "print(subprocess.run([\"nvidia-smi\",\"--query-gpu=name,memory.total,compute_cap\",\"--format=csv,noheader\"],\n",+ " capture_output=True, text=True).stdout.strip() or \"NO GPU - set Runtime > Change runtime type > T4 GPU\")\n",+ "import torch\n",+ "print(\"torch\", torch.__version__, \"| cuda\", torch.cuda.is_available(),\n",+ " \"| device\", torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"cpu\")\n",+ "assert torch.cuda.is_available(), \"Enable the T4 GPU runtime before running.\""+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 2. Dependencies"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "!pip -q install -U \"laya>=0.1.6\" \"transformers>=4.45\" \"datasets>=3.0\" safetensors huggingface_hub 2>&1 | tail -3\n",+ "import laya, transformers, datasets\n",+ "print(\"laya\", getattr(laya, \"__version__\", \"?\"), \"| transformers\", transformers.__version__, \"| datasets\", datasets.__version__)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 3. Download both checkpoints\n",+ "\n",+ "`reference_compile` is forced off after loading: the `laya` package does not disable it (only\n",+ "`rl_agent_api.py` does), and torch.compile is a loss on small batches / few SMs like a T4."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import torch, json, os, time\n",+ "from huggingface_hub import snapshot_download\n",+ "from safetensors import safe_open\n",+ "\n",+ "REPOS = {\"laya\": \"convaiinnovations/laya\", \"laya-multilingual\": \"convaiinnovations/laya-multilingual\"}\n",+ "PATHS, MODEL_META = {}, {}\n",+ "\n",+ "def patch_tokenizer_config(model_dir):\n",+ " '''laya-multilingual ships extra_special_tokens as a LIST (inherited from the mmBERT/Gemma\n",+ " tokenizer); transformers expects a dict and raises\n",+ " AttributeError: list object has no attribute keys. Without this, AutoTokenizer -- and so\n",+ " laya.load() -- fails outright on that repo. Also normalises tokenizer_class for 4.x/5.x.'''\n",+ " p = os.path.join(model_dir, \"tokenizer\", \"tokenizer_config.json\")\n",+ " if not os.path.exists(p):\n",+ " return []\n",+ " c = json.load(open(p))\n",+ " changed = []\n",+ " est = c.get(\"extra_special_tokens\")\n",+ " if isinstance(est, list):\n",+ " c[\"extra_special_tokens\"] = {(\"extra_%d\" % i): t for i, t in enumerate(est)}\n",+ " changed.append(\"extra_special_tokens list->dict\")\n",+ " if c.get(\"tokenizer_class\") in (None, \"TokenizersBackend\"):\n",+ " c[\"tokenizer_class\"] = \"PreTrainedTokenizerFast\"\n",+ " c.pop(\"backend\", None); c.pop(\"is_local\", None)\n",+ " changed.append(\"tokenizer_class\")\n",+ " if changed:\n",+ " json.dump(c, open(p, \"w\"), indent=2)\n",+ " return changed\n",+ "\n",+ "for name, repo in REPOS.items():\n",+ " t = time.time()\n",+ " PATHS[name] = snapshot_download(repo)\n",+ " fixed = patch_tokenizer_config(PATHS[name])\n",+ " if fixed:\n",+ " print(\" patched tokenizer config for %s: %s\" % (name, \", \".join(fixed)))\n",+ " cfg = json.load(open(os.path.join(PATHS[name], \"rl_agent_config.json\")))\n",+ " ecfg = json.load(open(os.path.join(PATHS[name], \"encoder\", \"config.json\")))\n",+ " tot = enc = 0\n",+ " with safe_open(os.path.join(PATHS[name], \"model.safetensors\"), \"pt\") as f:\n",+ " for k in f.keys():\n",+ " n = 1\n",+ " for d in f.get_slice(k).get_shape():\n",+ " n *= d\n",+ " tot += n\n",+ " enc += n if k.startswith(\"encoder.\") else 0\n",+ " MODEL_META[name] = {\n",+ " \"repo\": repo, \"params_total_m\": round(tot/1e6, 2), \"params_encoder_m\": round(enc/1e6, 2),\n",+ " \"params_head_m\": round((tot-enc)/1e6, 2), \"hidden_size\": ecfg.get(\"hidden_size\"),\n",+ " \"num_layers\": ecfg.get(\"num_hidden_layers\"), \"vocab_size\": ecfg.get(\"vocab_size\"),\n",+ " \"max_len\": cfg.get(\"max_len\"), \"head_max_len\": cfg.get(\"head_max_len\"),\n",+ " \"temperature\": cfg.get(\"temperature\"), \"temperature_by_options\": cfg.get(\"temperature_by_options\", {}),\n",+ " \"training\": cfg.get(\"training\", {}), \"download_s\": round(time.time()-t, 1),\n",+ " \"tokenizer_config_patched\": fixed,\n",+ " }\n",+ " print(name, \"->\", PATHS[name])\n",+ " print(\" \", json.dumps({k: v for k, v in MODEL_META[name].items() if k != \"temperature_by_options\"}))\n",+ "\n",+ "def load_agent(name):\n",+ " import laya\n",+ " ag = laya.load(PATHS[name], device=\"cuda\")\n",+ " try:\n",+ " ag.model.encoder.config.reference_compile = False # torch.compile hurts on T4-class GPUs\n",+ " except Exception as e:\n",+ " print(\"could not disable reference_compile:\", e)\n",+ " ag.model.eval()\n",+ " return ag"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 4. Benchmark engine \u2014 batched scoring and metrics"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "%%writefile bench_engine.py\n",+ "import json, math, sys, time\n",+ "import numpy as np\n",+ "import torch\n",+ "from laya.common import QTYPES, build_sequence, collate_items, render_options, temp_bucket\n",+ "\n",+ "def to_internal(qdef):\n",+ " t = qdef[\"type\"]\n",+ " crit = qdef.get(\"criteria\")\n",+ " if t == \"choice\" and isinstance(crit, list):\n",+ " crit = {c: None for c in crit}\n",+ " ins = qdef[\"instructions\"]\n",+ " if not isinstance(ins, str):\n",+ " ins = json.dumps(ins)\n",+ " return {\"t\": t, \"ins\": ins, \"crit\": crit}\n",+ "\n",+ "@torch.no_grad()\n",+ "def score_cases(agent, cases, max_tokens=16384, max_seqs=128, progress=\"\"):\n",+ " '''cases: [(state, {qid: qdef})] -> (raw_logits per question, index, seconds, n_dropped).\n",+ "\n",+ " Every question of every case is one sequence. Sequences are length-sorted and packed into\n",+ " token-budgeted batches so padding stays small. Returns UNCALIBRATED logits; temperature is\n",+ " applied later so raw/calibrated variants come from one forward pass.\n",+ " '''\n",+ " max_len = agent.cfg.get(\"max_len\", 512)\n",+ " head_max_len = agent.cfg.get(\"head_max_len\", 192)\n",+ " items, index, dropped = [], [], 0\n",+ " for ci, (state, questions) in enumerate(cases):\n",+ " for qid, qdef in questions.items():\n",+ " q = to_internal(qdef)\n",+ " try:\n",+ " ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)\n",+ " except Exception:\n",+ " index.append((ci, qid, QTYPES[q[\"t\"]], 0)); items.append(None); dropped += 1\n",+ " continue\n",+ " if len(markers) != len(render_options(q)):\n",+ " index.append((ci, qid, QTYPES[q[\"t\"]], 0)); items.append(None); dropped += 1\n",+ " continue\n",+ " items.append({\"ids\": ids, \"markers\": markers, \"qtype\": QTYPES[q[\"t\"]]})\n",+ " index.append((ci, qid, QTYPES[q[\"t\"]], len(markers)))\n",+ "\n",+ " valid = [i for i, it in enumerate(items) if it is not None]\n",+ " order = sorted(valid, key=lambda i: len(items[i][\"ids\"]))\n",+ " out = [None] * len(items)\n",+ " t0, done, i = time.time(), 0, 0\n",+ " while i < len(order):\n",+ " j, L = i, 0\n",+ " while j < len(order) and j - i < max_seqs and max(L, len(items[order[j]][\"ids\"])) * (j - i + 1) <= max_tokens:\n",+ " L = max(L, len(items[order[j]][\"ids\"])); j += 1\n",+ " j = max(j, i + 1)\n",+ " sel = [items[order[t]] for t in range(i, j)]\n",+ " b = collate_items([sel], agent.tok.pad_token_id)\n",+ " with torch.autocast(\"cuda\", dtype=agent.dtype, enabled=True):\n",+ " logits, _ = agent.model(b[\"input_ids\"].cuda(), b[\"attention_mask\"].cuda(),\n",+ " b[\"marker_pos\"].cuda(), b[\"marker_mask\"].cuda(), b[\"qtype\"].cuda())\n",+ " logits = logits.float().cpu().numpy()\n",+ " for r in range(j - i):\n",+ " out[order[i + r]] = logits[r, :len(sel[r][\"markers\"])]\n",+ " done += j - i\n",+ " if progress:\n",+ " el = time.time() - t0\n",+ " msg = \"\\r [%s] %d/%d seq | %.0f seq/s | ETA %ds \" % (\n",+ " progress, done, len(order), done/max(el, 1e-9),\n",+ " int(el*(len(order)-done)/max(1, done)))\n",+ " sys.stdout.write(msg)\n",+ " sys.stdout.flush()\n",+ " i = j\n",+ " if progress:\n",+ " print(\"\\r [%s] %d sequences in %.1fs (%d dropped: options did not fit)%s\" % (\n",+ " progress, len(order), time.time()-t0, dropped, \" \"*20))\n",+ " return out, index, time.time() - t0, dropped\n",+ "\n",+ "def softmax_t(z, t=1.0):\n",+ " z = np.asarray(z, dtype=np.float64) / max(1e-3, float(t))\n",+ " e = np.exp(z - z.max())\n",+ " return e / e.sum()\n",+ "\n",+ "def temp_for(agent, qt, k, calibrated=True):\n",+ " if not calibrated:\n",+ " return 1.0\n",+ " return float(agent.temperature_by_options.get(temp_bucket(qt, k), agent.temperature[qt]))\n",+ "\n",+ "# ------------------------------------------------------------------ metrics\n",+ "def ece_score(conf, correct, bins=15):\n",+ " conf, correct = np.asarray(conf, float), np.asarray(correct, float)\n",+ " if len(conf) == 0:\n",+ " return float(\"nan\")\n",+ " edges = np.linspace(0, 1, bins + 1)\n",+ " e = 0.0\n",+ " for lo, hi in zip(edges[:-1], edges[1:]):\n",+ " sel = (conf > lo) & (conf <= hi)\n",+ " if sel.any():\n",+ " e += sel.mean() * abs(conf[sel].mean() - correct[sel].mean())\n",+ " return float(e)\n",+ "\n",+ "def macro_f1(gold, pred):\n",+ " gold, pred = np.asarray(gold), np.asarray(pred)\n",+ " f1 = []\n",+ " for c in sorted(set(gold.tolist()) | set(pred.tolist())):\n",+ " tp = int(((pred == c) & (gold == c)).sum())\n",+ " fp = int(((pred == c) & (gold != c)).sum())\n",+ " fn = int(((pred != c) & (gold == c)).sum())\n",+ " f1.append(2*tp / max(1, 2*tp + fp + fn))\n",+ " return float(np.mean(f1))\n",+ "\n",+ "def aurc(conf, correct):\n",+ " conf, correct = np.asarray(conf, float), np.asarray(correct, float)\n",+ " if len(conf) == 0:\n",+ " return float(\"nan\")\n",+ " o = np.argsort(-conf)\n",+ " return float((np.cumsum(1 - correct[o]) / np.arange(1, len(o)+1)).mean())\n",+ "\n",+ "def hard_metrics(rows):\n",+ " '''rows: [(gold_idx, probs)] -> standard single-label classification + calibration metrics.'''\n",+ " rows = [r for r in rows if r[1] is not None]\n",+ " if not rows:\n",+ " return {\"n\": 0}\n",+ " gold = np.array([g for g, _ in rows])\n",+ " pred = np.array([int(np.argmax(p)) for _, p in rows])\n",+ " conf = np.array([float(np.max(p)) for _, p in rows])\n",+ " corr = (pred == gold).astype(float)\n",+ " m = {\"n\": len(rows),\n",+ " \"accuracy\": float(corr.mean()),\n",+ " \"macro_f1\": macro_f1(gold, pred),\n",+ " \"ece\": ece_score(conf, corr),\n",+ " \"brier\": float(np.mean([((np.asarray(p) - np.eye(len(p))[g])**2).sum() for g, p in rows])),\n",+ " \"nll\": float(np.mean([-math.log(max(float(p[g]), 1e-12)) for g, p in rows])),\n",+ " \"aurc\": aurc(conf, corr),\n",+ " \"mean_confidence\": float(conf.mean())}\n",+ " for cov in (0.5, 0.8):\n",+ " k = max(1, int(len(conf)*cov))\n",+ " m[\"acc_at_%d_coverage\" % int(cov*100)] = float(corr[np.argsort(-conf)[:k]].mean())\n",+ " return m\n",+ "\n",+ "def fit_temperature(pairs, lo=0.2, hi=10.0, steps=160):\n",+ " '''pairs: [(logits, gold_idx)] -> single temperature minimising NLL (grid search, no autograd).'''\n",+ " pairs = [(np.asarray(z, float), int(g)) for z, g in pairs if z is not None]\n",+ " if len(pairs) < 25:\n",+ " return 1.0\n",+ " best_t, best = 1.0, float(\"inf\")\n",+ " for t in np.geomspace(lo, hi, steps):\n",+ " tot = 0.0\n",+ " for z, g in pairs:\n",+ " zz = z/t\n",+ " zz = zz - zz.max()\n",+ " tot += -(zz[g] - math.log(np.exp(zz).sum()))\n",+ " if tot < best:\n",+ " best, best_t = tot, float(t)\n",+ " return round(float(best_t), 4)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 5. Task suites\n",+ "\n",+ "Every suite is built **once**, with a fixed seed, and reused for both models \u2014 so the two\n",+ "checkpoints answer byte-identical questions. Option sets for many-label tasks use\n",+ "`gold + N-1 distractors`, matching how the models were trained (the training recipe samples\n",+ "5\u201320 options per choice question rather than showing the full label space)."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import random, json\n",+ "import numpy as np\n",+ "from datasets import load_dataset\n",+ "\n",+ "SEED = 13\n",+ "N_OPTS = 20 # options per many-label choice question (gold + 19 distractors)\n",+ "PER_LANG = 300 # cases per language\n",+ "SUITES = {} # name -> {\"cases\": [(state, questions)], \"gold\": [{qid: gold_idx}], \"meta\": {...}}\n",+ "\n",+ "def build_choice_questions(rng, gold, all_labels, n_opts, instructions, qid=\"label\"):\n",+ " pool = [x for x in all_labels if x != gold]\n",+ " keys = [gold] + rng.sample(pool, min(n_opts-1, len(pool)))\n",+ " rng.shuffle(keys)\n",+ " crit = {k: k.replace(\"_\", \" \").replace(\".\", \": \") for k in keys}\n",+ " return {qid: {\"type\": \"choice\", \"instructions\": instructions, \"criteria\": crit}}, keys.index(gold)\n",+ "\n",+ "def register(name, cases, golds, **meta):\n",+ " SUITES[name] = {\"cases\": cases, \"gold\": golds, \"meta\": meta}\n",+ " nq = sum(len(c[1]) for c in cases)\n",+ " print(\" %-34s %4d cases / %5d questions %s\" % (name, len(cases), nq, meta.get(\"note\", \"\")))"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 5a. typed-decisions (English, 400 cases / 2,000 decisions)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "td_cases, td_golds, td_wf = [], [], []\n",+ "try:\n",+ " td = load_dataset(\"LocalLLaMA/typed-decisions\", \"all\", split=\"test\")\n",+ " for r in td:\n",+ " questions = json.loads(r[\"questions\"]) if isinstance(r[\"questions\"], str) else r[\"questions\"]\n",+ " gold = json.loads(r[\"gold\"]) if isinstance(r[\"gold\"], str) else r[\"gold\"]\n",+ " state = r[\"state\"]\n",+ " try:\n",+ " state = json.loads(state)\n",+ " except Exception:\n",+ " pass\n",+ " gmap = {}\n",+ " for qid, qdef in questions.items():\n",+ " g = gold[qid]\n",+ " if qdef[\"type\"] == \"choice\":\n",+ " keys = list(qdef[\"criteria\"].keys())\n",+ " gmap[qid] = {\"idx\": keys.index(str(g[\"label\"])), \"keys\": keys,\n",+ " \"soft\": [float(g.get(\"probabilities\", {}).get(k, 0.0)) for k in keys]}\n",+ " elif qdef[\"type\"] == \"noul\":\n",+ " idx = 1 if str(g[\"label\"]).lower() == \"true\" else 0\n",+ " pt = float(g.get(\"probabilities\", {}).get(\"true\", g.get(\"noul\", 0.5)))\n",+ " gmap[qid] = {\"idx\": idx, \"keys\": [\"false\", \"true\"], \"soft\": [1-pt, pt]}\n",+ " else: # score\n",+ " n = len(qdef[\"criteria\"])\n",+ " gmap[qid] = {\"idx\": int(g[\"label\"]), \"keys\": [str(i) for i in range(n)],\n",+ " \"soft\": [float(g.get(\"probabilities\", {}).get(str(i), 0.0)) for i in range(n)],\n",+ " \"gold_score\": float(g.get(\"score\", float(g[\"label\"])))}\n",+ " td_cases.append((state, questions))\n",+ " td_golds.append(gmap)\n",+ " td_wf.append(r[\"workflow\"])\n",+ " register(\"typed_decisions\", td_cases, td_golds, note=\"4 workflows, soft teacher labels\",\n",+ " workflows=td_wf, soft=True)\n",+ "except Exception as e:\n",+ " print(\"typed-decisions FAILED:\", type(e).__name__, e)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 5b. Multilingual \u2014 MASSIVE intent & scenario (14 languages), XNLI (15 languages)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "MASSIVE_LANGS = [\"en\",\"de\",\"fr\",\"es\",\"pt\",\"ru\",\"tr\",\"ar\",\"hi\",\"ta\",\"zh-CN\",\"ja\",\"ko\",\"sw\"]\n",+ "XNLI_LANGS = [\"en\",\"de\",\"fr\",\"es\",\"ru\",\"tr\",\"ar\",\"hi\",\"ur\",\"vi\",\"th\",\"el\",\"bg\",\"zh\",\"sw\"]\n",+ "\n",+ "for task, short, instr in [(\"mteb/amazon_massive_intent\", \"massive_intent\",\n",+ " \"What is the user asking for in `utterance`?\"),\n",+ " (\"mteb/amazon_massive_scenario\", \"massive_scenario\",\n",+ " \"Which domain does `utterance` belong to?\")]:\n",+ " for lg in MASSIVE_LANGS:\n",+ " try:\n",+ " d = load_dataset(task, lg, split=\"test\")\n",+ " labels = sorted(set(d[\"label_text\"]))\n",+ " rng = random.Random(SEED)\n",+ " cases, golds = [], []\n",+ " for r in list(d)[:PER_LANG]:\n",+ " qs, gi = build_choice_questions(rng, r[\"label_text\"], labels, N_OPTS, instr)\n",+ " cases.append(({\"utterance\": r[\"text\"]}, qs))\n",+ " golds.append({\"label\": {\"idx\": gi}})\n",+ " register(\"%s.%s\" % (short, lg), cases, golds, lang=lg, n_options=N_OPTS,\n",+ " label_space=len(labels), family=short)\n",+ " except Exception as e:\n",+ " print(\" FAIL %s %s: %s\" % (short, lg, str(e)[:90]))\n",+ "\n",+ "NLI_CRIT = {\"entailment\": \"the premise implies the hypothesis is true\",\n",+ " \"neutral\": \"the premise neither implies nor contradicts the hypothesis\",\n",+ " \"contradiction\": \"the premise implies the hypothesis is false\"}\n",+ "for lg in XNLI_LANGS:\n",+ " try:\n",+ " d = load_dataset(\"facebook/xnli\", lg, split=\"test\")\n",+ " cases, golds = [], []\n",+ " for r in list(d)[:PER_LANG]:\n",+ " qs = {\"relation\": {\"type\": \"choice\",\n",+ " \"instructions\": \"What is the relationship between `premise` and `hypothesis`?\",\n",+ " \"criteria\": dict(NLI_CRIT)}}\n",+ " cases.append(({\"premise\": r[\"premise\"], \"hypothesis\": r[\"hypothesis\"]}, qs))\n",+ " golds.append({\"relation\": {\"idx\": int(r[\"label\"])}})\n",+ " register(\"xnli.%s\" % lg, cases, golds, lang=lg, n_options=3, family=\"xnli\")\n",+ " except Exception as e:\n",+ " print(\" FAIL xnli %s: %s\" % (lg, str(e)[:90]))"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 5c. English tasks\n",+ "\n",+ "`sst5`, `emotion` and `prompt_injections` were held out of Laya's training entirely (true\n",+ "zero-shot). `ag_news` and `boolq` were in the training mix, so they measure retention rather\n",+ "than generalisation. `banking77` is the many-option stress test \u2014 77 options at once, versus\n",+ "Jev's documented hard cap at 255."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "rng = random.Random(SEED)\n",+ "\n",+ "# SST-5: ordinal score\n",+ "try:\n",+ " d = load_dataset(\"SetFit/sst5\", split=\"test\")\n",+ " crit = [\"very negative\", \"negative\", \"neutral\", \"positive\", \"very positive\"]\n",+ " cases, golds = [], []\n",+ " for r in list(d)[:600]:\n",+ " cases.append(({\"text\": r[\"text\"]},\n",+ " {\"sentiment\": {\"type\": \"score\", \"instructions\": \"How positive is the sentiment of `text`?\",\n",+ " \"criteria\": crit}}))\n",+ " golds.append({\"sentiment\": {\"idx\": int(r[\"label\"]), \"gold_score\": float(r[\"label\"])}})\n",+ " register(\"en.sst5\", cases, golds, note=\"zero-shot, ordinal\", family=\"english\", held_out=True)\n",+ "except Exception as e: print(\" FAIL sst5:\", str(e)[:90])\n",+ "\n",+ "# emotion: 6-way choice\n",+ "try:\n",+ " d = load_dataset(\"dair-ai/emotion\", \"split\", split=\"test\")\n",+ " names = [\"sadness\",\"joy\",\"love\",\"anger\",\"fear\",\"surprise\"]\n",+ " cases, golds = [], []\n",+ " for r in list(d)[:600]:\n",+ " cases.append(({\"text\": r[\"text\"]},\n",+ " {\"emotion\": {\"type\": \"choice\",\n",+ " \"instructions\": \"Which emotion is most strongly expressed in `text`?\",\n",+ " \"criteria\": {n: None for n in names}}}))\n",+ " golds.append({\"emotion\": {\"idx\": int(r[\"label\"])}})\n",+ " register(\"en.emotion\", cases, golds, note=\"zero-shot\", family=\"english\", held_out=True)\n",+ "except Exception as e: print(\" FAIL emotion:\", str(e)[:90])\n",+ "\n",+ "# prompt injections: noul\n",+ "try:\n",+ " d = load_dataset(\"deepset/prompt-injections\", split=\"test\")\n",+ " cases, golds = [], []\n",+ " for r in list(d):\n",+ " cases.append(({\"text\": r[\"text\"]},\n",+ " {\"injection\": {\"type\": \"noul\",\n",+ " \"instructions\": \"Does `text` try to inject or override instructions given to an AI system?\"}}))\n",+ " golds.append({\"injection\": {\"idx\": int(r[\"label\"])}})\n",+ " register(\"en.prompt_injections\", cases, golds, note=\"zero-shot\", family=\"english\", held_out=True)\n",+ "except Exception as e: print(\" FAIL prompt_injections:\", str(e)[:90])\n",+ "\n",+ "# banking77: 77-option stress test\n",+ "try:\n",+ " d = load_dataset(\"PolyAI/banking77\", split=\"test\")\n",+ " names = d.features[\"label\"].names\n",+ " cases, golds = [], []\n",+ " for r in list(d)[:500]:\n",+ " cases.append(({\"message\": r[\"text\"]},\n",+ " {\"intent\": {\"type\": \"choice\", \"instructions\": \"Which banking intent does `message` express?\",\n",+ " \"criteria\": {n.replace(\"_\", \" \"): None for n in names}}}))\n",+ " golds.append({\"intent\": {\"idx\": int(r[\"label\"])}})\n",+ " register(\"en.banking77_full\", cases, golds, note=\"77 options at once\", family=\"english\",\n",+ " n_options=len(names), held_out=True)\n",+ "except Exception as e: print(\" FAIL banking77:\", str(e)[:90])\n",+ "\n",+ "# ag_news + boolq: in-task retention\n",+ "try:\n",+ " d = load_dataset(\"fancyzhx/ag_news\", split=\"test\")\n",+ " crit = {\"world\": \"world news and international politics\", \"sports\": \"sports\",\n",+ " \"business\": \"business and economy\", \"sci_tech\": \"science and technology\"}\n",+ " cases, golds = [], []\n",+ " for r in list(d)[:600]:\n",+ " cases.append(({\"article\": r[\"text\"]},\n",+ " {\"topic\": {\"type\": \"choice\", \"instructions\": \"What is the topic of `article`?\",\n",+ " \"criteria\": dict(crit)}}))\n",+ " golds.append({\"topic\": {\"idx\": int(r[\"label\"])}})\n",+ " register(\"en.ag_news\", cases, golds, note=\"in-task (trained on)\", family=\"english\", held_out=False)\n",+ "except Exception as e: print(\" FAIL ag_news:\", str(e)[:90])\n",+ "\n",+ "try:\n",+ " d = load_dataset(\"google/boolq\", split=\"validation\")\n",+ " cases, golds = [], []\n",+ " for r in list(d)[:600]:\n",+ " cases.append(({\"passage\": r[\"passage\"], \"question\": r[\"question\"]},\n",+ " {\"answer\": {\"type\": \"noul\",\n",+ " \"instructions\": \"Based on `passage`, is the answer to `question` yes?\"}}))\n",+ " golds.append({\"answer\": {\"idx\": int(bool(r[\"answer\"]))}})\n",+ " register(\"en.boolq\", cases, golds, note=\"in-task (trained on)\", family=\"english\", held_out=False)\n",+ "except Exception as e: print(\" FAIL boolq:\", str(e)[:90])\n",+ "\n",+ "print()\n",+ "print(\"suites built:\", len(SUITES), \"| total questions:\",\n",+ " sum(sum(len(c[1]) for c in s[\"cases\"]) for s in SUITES.values()))"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 6. Run every suite on both models"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import bench_engine as BE\n",+ "import numpy as np, gc, time, json\n",+ "\n",+ "RESULTS = {\"meta\": {\"seed\": SEED, \"n_opts\": N_OPTS, \"per_lang\": PER_LANG,\n",+ " \"timestamp\": time.strftime(\"%Y-%m-%d %H:%M:%S\"),\n",+ " \"gpu\": torch.cuda.get_device_name(0),\n",+ " \"torch\": torch.__version__, \"transformers\": transformers.__version__,\n",+ " \"models\": MODEL_META},\n",+ " \"suites\": {}}\n",+ "RAW = {} # model -> suite -> (logits, index) kept in RAM for the calibration-repair section\n",+ "\n",+ "def evaluate_suite(agent, model_name, suite_name):\n",+ " S = SUITES[suite_name]\n",+ " logits, index, secs, dropped = BE.score_cases(agent, S[\"cases\"], progress=\"%s/%s\" % (model_name, suite_name))\n",+ " RAW[model_name][suite_name] = (logits, index)\n",+ " soft = S[\"meta\"].get(\"soft\", False)\n",+ " rows_cal, rows_raw, extra = [], [], {\"soft_acc\": [], \"brier_soft\": [], \"tv\": [], \"kl\": [],\n",+ " \"score_mae\": [], \"within_1\": []}\n",+ " per_q_type = {}\n",+ " for (ci, qid, qt, k), z in zip(index, logits):\n",+ " g = S[\"gold\"][ci].get(qid)\n",+ " if g is None:\n",+ " continue\n",+ " if z is None:\n",+ " rows_cal.append((g[\"idx\"], None)); rows_raw.append((g[\"idx\"], None)); continue\n",+ " p_cal = BE.softmax_t(z, BE.temp_for(agent, qt, k, True))\n",+ " p_raw = BE.softmax_t(z, 1.0)\n",+ " rows_cal.append((g[\"idx\"], p_cal)); rows_raw.append((g[\"idx\"], p_raw))\n",+ " per_q_type.setdefault(qt, []).append((g[\"idx\"], p_cal))\n",+ " if soft and \"soft\" in g:\n",+ " gp = np.asarray(g[\"soft\"], float)\n",+ " if gp.sum() > 0:\n",+ " gp = gp / gp.sum()\n",+ " pp = p_cal[:len(gp)] if len(p_cal) >= len(gp) else np.pad(p_cal, (0, len(gp)-len(p_cal)))\n",+ " pp = pp / max(pp.sum(), 1e-12)\n",+ " extra[\"soft_acc\"].append(float((pp*gp).sum()))\n",+ " extra[\"brier_soft\"].append(float(((pp-gp)**2).sum()))\n",+ " extra[\"tv\"].append(float(0.5*np.abs(pp-gp).sum()))\n",+ " extra[\"kl\"].append(float((gp*np.log(np.clip(gp/np.clip(pp,1e-12,None),1e-12,1e4))).sum()))\n",+ " if \"gold_score\" in g:\n",+ " exp = float((np.arange(len(p_cal))*p_cal).sum())\n",+ " extra[\"score_mae\"].append(abs(exp - g[\"gold_score\"]))\n",+ " extra[\"within_1\"].append(float(abs(exp - g[\"gold_score\"]) <= 1.0))\n",+ "\n",+ " out = {\"calibrated\": BE.hard_metrics(rows_cal), \"raw\": BE.hard_metrics(rows_raw),\n",+ " \"seconds\": round(secs, 2), \"dropped_questions\": dropped,\n",+ " \"questions_per_second\": round(len([r for r in rows_cal if r[1] is not None])/max(secs,1e-9), 1)}\n",+ " for k2, v in extra.items():\n",+ " if v:\n",+ " out[k2] = float(np.mean(v))\n",+ " if len(per_q_type) > 1:\n",+ " names = {0: \"choice\", 1: \"score\", 2: \"noul\"}\n",+ " out[\"by_question_type\"] = {names[qt]: BE.hard_metrics(rs) for qt, rs in sorted(per_q_type.items())}\n",+ " return out\n",+ "\n",+ "for model_name in REPOS:\n",+ " print(\"\\n\" + \"=\"*78 + \"\\n %s\\n\" % model_name + \"=\"*78)\n",+ " agent = load_agent(model_name)\n",+ " RAW[model_name] = {}\n",+ " for suite_name in SUITES:\n",+ " try:\n",+ " r = evaluate_suite(agent, model_name, suite_name)\n",+ " RESULTS[\"suites\"].setdefault(suite_name, {})[model_name] = r\n",+ " c = r[\"calibrated\"]\n",+ " print(\" %-32s acc %.3f f1 %.3f ECE %.3f n=%d\" %\n",+ " (suite_name, c.get(\"accuracy\", float(\"nan\")), c.get(\"macro_f1\", float(\"nan\")),\n",+ " c.get(\"ece\", float(\"nan\")), c.get(\"n\", 0)))\n",+ " except Exception as e:\n",+ " import traceback; traceback.print_exc()\n",+ " RESULTS[\"suites\"].setdefault(suite_name, {})[model_name] = {\"error\": \"%s: %s\" % (type(e).__name__, e)}\n",+ " globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")] = agent\n",+ " gc.collect(); torch.cuda.empty_cache()\n",+ "\n",+ "json.dump(RESULTS, open(\"laya_benchmark_results.json\", \"w\"), indent=2)\n",+ "print(\"\\ncheckpoint saved\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 6b. Control \u2014 matched context budget\n",+ "\n",+ "The two checkpoints ship different context windows (512 vs 1024 tokens), so on long states\n",+ "`laya-multilingual` sees more of the input. This re-runs typed-decisions with **both** models\n",+ "capped at 512 tokens, isolating the architecture from the context-window advantage."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import bench_engine as BE\n",+ "import numpy as np\n",+ "\n",+ "RESULTS[\"typed_decisions_matched_512\"] = {}\n",+ "if \"typed_decisions\" in SUITES:\n",+ " S = SUITES[\"typed_decisions\"]\n",+ " for model_name in REPOS:\n",+ " agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",+ " orig = agent.cfg.get(\"max_len\")\n",+ " agent.cfg[\"max_len\"] = 512\n",+ " try:\n",+ " logits, index, secs, dropped = BE.score_cases(agent, S[\"cases\"], progress=\"%s/td@512\" % model_name)\n",+ " rows = []\n",+ " for (ci, qid, qt, k), z in zip(index, logits):\n",+ " g = S[\"gold\"][ci].get(qid)\n",+ " if g is None or z is None:\n",+ " continue\n",+ " rows.append((g[\"idx\"], BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))\n",+ " RESULTS[\"typed_decisions_matched_512\"][model_name] = BE.hard_metrics(rows)\n",+ " print(\" %-22s max_len 512 -> acc %.3f ECE %.3f n=%d\"\n",+ " % (model_name, RESULTS[\"typed_decisions_matched_512\"][model_name][\"accuracy\"],\n",+ " RESULTS[\"typed_decisions_matched_512\"][model_name][\"ece\"],\n",+ " RESULTS[\"typed_decisions_matched_512\"][model_name][\"n\"]))\n",+ " finally:\n",+ " agent.cfg[\"max_len\"] = orig"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 7. Latency (p50 / p95 at 1, 5, 10, 50 questions per call)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import numpy as np, time, torch\n",+ "\n",+ "LAT_STATE = {\"ticket\": {\"subject\": \"Payout failing\",\n",+ " \"messages\": [{\"from\": \"customer\",\n",+ " \"text\": \"Hi, my Stripe payouts have failed for 3 days and I am losing sales. Please help ASAP. \" * 6}]}}\n",+ "Q_NOUL = {\"type\": \"noul\", \"instructions\": \"Does `ticket.messages[0].text` express urgency?\"}\n",+ "Q_CHOICE = {\"type\": \"choice\", \"instructions\": \"Which team should handle this?\",\n",+ " \"criteria\": {\"billing\": \"payments\", \"technical\": \"bugs and integrations\", \"sales\": \"pricing\"}}\n",+ "\n",+ "RESULTS[\"latency\"] = {}\n",+ "for model_name in REPOS:\n",+ " agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",+ " res = {}\n",+ " for nq in (1, 5, 10, 50):\n",+ " qs = {(\"q%d\" % i): (Q_NOUL if i % 2 else Q_CHOICE) for i in range(nq)}\n",+ " for _ in range(3):\n",+ " agent.system_one(LAT_STATE, qs)\n",+ " ts = []\n",+ " for _ in range(20):\n",+ " torch.cuda.synchronize(); t = time.perf_counter()\n",+ " agent.system_one(LAT_STATE, qs)\n",+ " torch.cuda.synchronize(); ts.append((time.perf_counter()-t)*1000)\n",+ " res[\"%d_questions\" % nq] = {\"p50_ms\": round(float(np.percentile(ts, 50)), 1),\n",+ " \"p95_ms\": round(float(np.percentile(ts, 95)), 1),\n",+ " \"ms_per_question\": round(float(np.percentile(ts, 50))/nq, 2)}\n",+ " RESULTS[\"latency\"][model_name] = res\n",+ " print(model_name, json.dumps(res))"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 8. Option-order robustness\n",+ "\n",+ "Same question, options permuted. A well-behaved decision model should pick the same label.\n",+ "The independent Jev benchmark measured a **13% flip rate for Jev**, with the worst LLM at 37%."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import random, numpy as np\n",+ "import bench_engine as BE\n",+ "\n",+ "PB_SUITES = [s for s in (\"massive_intent.en\", \"en.emotion\", \"xnli.en\") if s in SUITES]\n",+ "RESULTS[\"option_order_robustness\"] = {}\n",+ "\n",+ "for model_name in REPOS:\n",+ " agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",+ " out = {}\n",+ " for sname in PB_SUITES:\n",+ " S = SUITES[sname]\n",+ " base = S[\"cases\"][:200]\n",+ " rng = random.Random(99)\n",+ " perm_cases, perm_maps = [], []\n",+ " for state, questions in base:\n",+ " qid = list(questions)[0]\n",+ " qdef = questions[qid]\n",+ " keys = list(qdef[\"criteria\"].keys())\n",+ " order = list(range(len(keys))); rng.shuffle(order)\n",+ " newcrit = {keys[i]: qdef[\"criteria\"][keys[i]] for i in order}\n",+ " perm_cases.append((state, {qid: {**qdef, \"criteria\": newcrit}}))\n",+ " perm_maps.append((keys, [keys[i] for i in order]))\n",+ " la, ia, _, _ = BE.score_cases(agent, base, progress=\"\")\n",+ " lb, ib, _, _ = BE.score_cases(agent, perm_cases, progress=\"\")\n",+ " flips = comp = 0\n",+ " for (za, (_, _, _, ka)), (zb, (_, _, _, kb)), (keys, pkeys) in zip(zip(la, ia), zip(lb, ib), perm_maps):\n",+ " if za is None or zb is None:\n",+ " continue\n",+ " comp += 1\n",+ " if keys[int(np.argmax(za))] != pkeys[int(np.argmax(zb))]:\n",+ " flips += 1\n",+ " out[sname] = {\"n\": comp, \"flip_rate\": round(flips/max(1, comp), 4)}\n",+ " print(\" %-22s %-24s flip rate %.3f (n=%d)\" % (model_name, sname, out[sname][\"flip_rate\"], comp))\n",+ " RESULTS[\"option_order_robustness\"][model_name] = out"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 9. Calibration repair\n",+ "\n",+ "`laya-multilingual` ships with `temperature = [1.0, 1.0, 1.0]` and no per-option-count buckets \u2014\n",+ "it was never calibrated. This refits one temperature per (question type, option-count bucket) on\n",+ "**half** of each suite and reports ECE on the other half, so the improvement is measured\n",+ "out-of-sample. It shows how much of any calibration gap is a missing post-processing step rather\n",+ "than a property of the model."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import numpy as np\n",+ "import bench_engine as BE\n",+ "from laya.common import temp_bucket\n",+ "\n",+ "RESULTS[\"calibration_repair\"] = {}\n",+ "for model_name in REPOS:\n",+ " agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n",+ " per_suite, fitted_all = {}, {}\n",+ " for sname, S in SUITES.items():\n",+ " if sname not in RAW[model_name]:\n",+ " continue\n",+ " logits, index = RAW[model_name][sname]\n",+ " pairs = []\n",+ " for (ci, qid, qt, k), z in zip(index, logits):\n",+ " g = S[\"gold\"][ci].get(qid)\n",+ " if g is not None and z is not None:\n",+ " pairs.append((z, g[\"idx\"], qt, k))\n",+ " if len(pairs) < 60:\n",+ " continue\n",+ " half = len(pairs)//2\n",+ " fit_set, held = pairs[:half], pairs[half:]\n",+ " buckets = {}\n",+ " for z, gi, qt, k in fit_set:\n",+ " buckets.setdefault(temp_bucket(qt, k), []).append((z, gi))\n",+ " fitted = {b: BE.fit_temperature(v) for b, v in buckets.items() if len(v) >= 25}\n",+ " rows_ship, rows_fit = [], []\n",+ " for z, gi, qt, k in held:\n",+ " rows_ship.append((gi, BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))\n",+ " rows_fit.append((gi, BE.softmax_t(z, fitted.get(temp_bucket(qt, k),\n",+ " BE.temp_for(agent, qt, k, True)))))\n",+ " ms, mf = BE.hard_metrics(rows_ship), BE.hard_metrics(rows_fit)\n",+ " per_suite[sname] = {\"n_heldout\": ms[\"n\"], \"fitted_temperatures\": fitted,\n",+ " \"ece_shipped\": ms[\"ece\"], \"ece_refit\": mf[\"ece\"],\n",+ " \"nll_shipped\": ms[\"nll\"], \"nll_refit\": mf[\"nll\"],\n",+ " \"accuracy\": ms[\"accuracy\"]}\n",+ " fitted_all.update(fitted)\n",+ " RESULTS[\"calibration_repair\"][model_name] = {\"per_suite\": per_suite}\n",+ " if per_suite:\n",+ " a = float(np.mean([v[\"ece_shipped\"] for v in per_suite.values()]))\n",+ " b = float(np.mean([v[\"ece_refit\"] for v in per_suite.values()]))\n",+ " RESULTS[\"calibration_repair\"][model_name][\"mean_ece_shipped\"] = round(a, 4)\n",+ " RESULTS[\"calibration_repair\"][model_name][\"mean_ece_refit\"] = round(b, 4)\n",+ " print(\"%-22s mean ECE shipped %.4f -> refit %.4f\" % (model_name, a, b))"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 10. Aggregate, save and download"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import numpy as np, json\n",+ "\n",+ "def agg(prefix_or_family, key=\"family\"):\n",+ " out = {}\n",+ " for m in REPOS:\n",+ " accs, eces, ns = [], [], 0\n",+ " for sname, S in SUITES.items():\n",+ " if S[\"meta\"].get(key) != prefix_or_family:\n",+ " continue\n",+ " r = RESULTS[\"suites\"].get(sname, {}).get(m, {})\n",+ " c = r.get(\"calibrated\", {})\n",+ " if c.get(\"n\"):\n",+ " accs.append(c[\"accuracy\"]); eces.append(c[\"ece\"]); ns += c[\"n\"]\n",+ " if accs:\n",+ " out[m] = {\"macro_accuracy\": round(float(np.mean(accs)), 4),\n",+ " \"macro_ece\": round(float(np.mean(eces)), 4),\n",+ " \"n_questions\": ns, \"n_suites\": len(accs)}\n",+ " return out\n",+ "\n",+ "RESULTS[\"caveats\"] = {\n",+ " \"context_window\": \"laya max_len=512, laya-multilingual max_len=1024 (shipped configs). \"\n",+ " \"See typed_decisions_matched_512 for the equal-budget control.\",\n",+ " \"shipped_calibration\": \"laya ships fitted temperatures; laya-multilingual ships all-1.0 \"\n",+ " \"(uncalibrated). See calibration_repair for the out-of-sample refit.\",\n",+ " \"training_overlap\": \"ag_news and boolq were in Laya's training mix (retention, not generalisation). \"\n",+ " \"sst5, emotion, prompt_injections, banking77 were held out. \"\n",+ " \"Neither checkpoint trained on typed-decisions, MASSIVE or XNLI.\",\n",+ "}\n",+ "\n",+ "RESULTS[\"summary\"] = {\n",+ " \"typed_decisions\": {m: RESULTS[\"suites\"].get(\"typed_decisions\", {}).get(m, {}).get(\"calibrated\", {})\n",+ " for m in REPOS},\n",+ " \"massive_intent\": agg(\"massive_intent\"),\n",+ " \"massive_scenario\": agg(\"massive_scenario\"),\n",+ " \"xnli\": agg(\"xnli\"),\n",+ " \"english\": agg(\"english\"),\n",+ "}\n",+ "\n",+ "# English vs non-English split inside each multilingual family\n",+ "for fam in (\"massive_intent\", \"massive_scenario\", \"xnli\"):\n",+ " en, non = {}, {}\n",+ " for m in REPOS:\n",+ " ea, na = [], []\n",+ " for sname, S in SUITES.items():\n",+ " if S[\"meta\"].get(\"family\") != fam:\n",+ " continue\n",+ " c = RESULTS[\"suites\"].get(sname, {}).get(m, {}).get(\"calibrated\", {})\n",+ " if not c.get(\"n\"):\n",+ " continue\n",+ " (ea if S[\"meta\"].get(\"lang\") in (\"en\",) else na).append(c[\"accuracy\"])\n",+ " if ea: en[m] = round(float(np.mean(ea)), 4)\n",+ " if na: non[m] = round(float(np.mean(na)), 4)\n",+ " RESULTS[\"summary\"][fam + \"_english_vs_rest\"] = {\"english\": en, \"non_english\": non}\n",+ "\n",+ "json.dump(RESULTS, open(\"laya_benchmark_results.json\", \"w\"), indent=2)\n",+ "print(json.dumps(RESULTS[\"summary\"], indent=2))\n",+ "print(\"\\nwrote laya_benchmark_results.json (%.1f KB)\" % (os.path.getsize(\"laya_benchmark_results.json\")/1024))"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# Per-language table, printed for a quick eyeball before you download the JSON\n",+ "import numpy as np\n",+ "for fam in (\"massive_intent\", \"massive_scenario\", \"xnli\"):\n",+ " langs = sorted({S[\"meta\"][\"lang\"] for S in SUITES.values() if S[\"meta\"].get(\"family\") == fam})\n",+ " if not langs:\n",+ " continue\n",+ " print(\"\\n=== %s : accuracy by language ===\" % fam)\n",+ " print(\"%-8s %12s %22s %10s\" % (\"lang\", \"laya\", \"laya-multilingual\", \"delta\"))\n",+ " for lg in langs:\n",+ " sname = \"%s.%s\" % (fam, lg)\n",+ " a = RESULTS[\"suites\"].get(sname, {}).get(\"laya\", {}).get(\"calibrated\", {}).get(\"accuracy\")\n",+ " b = RESULTS[\"suites\"].get(sname, {}).get(\"laya-multilingual\", {}).get(\"calibrated\", {}).get(\"accuracy\")\n",+ " if a is None or b is None:\n",+ " continue\n",+ " print(\"%-8s %12.3f %22.3f %+10.3f\" % (lg, a, b, b-a))"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "try:\n",+ " from google.colab import files\n",+ " files.download(\"laya_benchmark_results.json\")\n",+ "except Exception as e:\n",+ " print(\"Not on Colab or download blocked (%s). Grab laya_benchmark_results.json from the file browser.\" % type(e).__name__)"+ ]+ }+ ],+ "metadata": {+ "accelerator": "GPU",+ "colab": {+ "provenance": [],+ "gpuType": "T4",+ "toc_visible": true+ },+ "kernelspec": {+ "display_name": "Python 3",+ "name": "python3"+ },+ "language_info": {+ "name": "python"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 0+}\ No newline at end of filediff --git a/research/scripts/make_plots.py b/research/scripts/make_plots.pynew file mode 100644index 00000000..08a9e857--- /dev/null+++ b/research/scripts/make_plots.py@@ -0,0 +1,154 @@+"""Render the benchmark figure used in the GitHub and Hugging Face READMEs.++ USE_TF=0 python3 notebooks/make_plots.py++Reads local_benchmark_results.json (51-language sweep) and laya_benchmark_results.json+(T4 latency + English suites); writes assets/laya_benchmark.png.+"""+import json+import os++import matplotlib+matplotlib.use("Agg")+import matplotlib.pyplot as plt # noqa: E402+import numpy as np # noqa: E402++REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+ASSETS = os.path.join(REPO, "assets")+os.makedirs(ASSETS, exist_ok=True)++# Validated categorical slots 1 and 2 (light mode) + ink tokens.+BLUE, ORANGE = "#2a78d6", "#eb6834"+SURFACE = "#fcfcfb"+INK, INK2, INK3 = "#0b0b0b", "#52514e", "#8a8984"+GRID = "#e4e3df"++plt.rcParams.update({+ "figure.facecolor": SURFACE, "axes.facecolor": SURFACE, "savefig.facecolor": SURFACE,+ "font.family": "DejaVu Sans", "text.color": INK,+ "axes.labelcolor": INK2, "xtick.color": INK2, "ytick.color": INK2,+ "axes.edgecolor": GRID, "axes.linewidth": 1.0,+ "xtick.major.size": 0, "ytick.major.size": 0,+})+++def strip(ax, keep=("bottom",)):+ for s in ("top", "right", "left", "bottom"):+ ax.spines[s].set_visible(s in keep)+++def main():+ loc = json.load(open(os.path.join(REPO, "local_benchmark_results.json")))+ col = json.load(open(os.path.join(REPO, "laya_benchmark_results.json")))++ en = loc["part_a"]["by_model"]["english"]["per_language"]+ ml = loc["part_a"]["by_model"].get("multilingual", {}).get("per_language", {})+ langs = sorted(set(en) & set(ml), key=lambda l: (ml[l]["accuracy"] - en[l]["accuracy"]))+ if not langs:+ raise SystemExit("multilingual sweep not finished yet")++ fig = plt.figure(figsize=(16, max(9, 0.23 * len(langs) + 3.2)))+ gs = fig.add_gridspec(3, 2, width_ratios=[1.35, 1], height_ratios=[1, 1, 1],+ wspace=0.28, hspace=0.55)++ # ---------------------------------------------------------------- A. dumbbell+ axA = fig.add_subplot(gs[:, 0])+ y = np.arange(len(langs))+ e = np.array([en[l]["accuracy"] for l in langs])+ m = np.array([ml[l]["accuracy"] for l in langs])+ axA.hlines(y, e, m, color=GRID, lw=2.4, zorder=1)+ axA.scatter(e, y, s=62, color=BLUE, zorder=3, label="laya (English checkpoint)",+ edgecolors=SURFACE, linewidths=1.6)+ axA.scatter(m, y, s=62, color=ORANGE, zorder=3, label="laya-multilingual",+ edgecolors=SURFACE, linewidths=1.6)+ axA.axvline(0.05, color=INK3, lw=1.4, ls=(0, (4, 3)), zorder=2)+ axA.text(0.052, len(langs) - 0.4, "random guess (0.050)", color=INK3, fontsize=9, va="top")+ axA.set_yticks(y); axA.set_yticklabels(langs, fontsize=9)+ axA.set_ylim(-0.8, len(langs) - 0.2)+ axA.set_xlim(-0.02, max(0.9, float(max(e.max(), m.max())) + 0.06))+ axA.set_xlabel("accuracy · MASSIVE intent, 20 options", fontsize=10)+ axA.set_title("Every language: which checkpoint can read it",+ fontsize=14, fontweight="bold", color=INK, loc="left", pad=12)+ axA.xaxis.grid(True, color=GRID, lw=0.9); axA.set_axisbelow(True)+ strip(axA)+ axA.legend(frameon=False, loc="lower right", fontsize=10, labelcolor=INK2)++ # ---------------------------------------------------------------- B. latency+ axB = fig.add_subplot(gs[0, 1])+ lat = col["latency"]+ ns = ["1_questions", "5_questions", "10_questions", "50_questions"]+ lab = ["1", "5", "10", "50"]+ a = [lat["laya"][n]["p50_ms"] for n in ns]+ b = [lat["laya-multilingual"][n]["p50_ms"] for n in ns]+ x = np.arange(len(ns)); w = 0.36+ axB.bar(x - w / 2 - 0.01, a, w, color=BLUE, label="laya", zorder=3)+ axB.bar(x + w / 2 + 0.01, b, w, color=ORANGE, label="laya-multilingual", zorder=3)+ for xi, v in zip(x - w / 2 - 0.01, a):+ axB.text(xi, v + 18, "%.0f" % v, ha="center", fontsize=9, color=INK2)+ for xi, v in zip(x + w / 2 + 0.01, b):+ axB.text(xi, v + 18, "%.0f" % v, ha="center", fontsize=9, color=INK2)+ axB.axhline(256, color=INK3, lw=1.4, ls=(0, (4, 3)), zorder=2)+ axB.text(3.42, 272, "Jev, measured\n236–276 ms", color=INK3, fontsize=8.5, ha="right")+ axB.set_xticks(x); axB.set_xticklabels(lab)+ axB.set_xlabel("questions per call", fontsize=10)+ axB.set_ylabel("p50 latency (ms)", fontsize=10)+ axB.set_ylim(0, max(a + b) * 1.18)+ axB.set_title("Speed on one T4", fontsize=13, fontweight="bold", color=INK, loc="left", pad=10)+ axB.yaxis.grid(True, color=GRID, lw=0.9); axB.set_axisbelow(True)+ strip(axB); axB.legend(frameon=False, fontsize=9.5, loc="upper left", labelcolor=INK2)++ # ---------------------------------------------------------------- C. vs Jev+ axC = fig.add_subplot(gs[1, 1])+ names = ["AG News\n(4 labels)", "DAIR Emotion\n(6 labels)", "typed-decisions\n(2,000 dec.)"]+ jev = [0.910, 0.480, 0.727]+ lay = [col["suites"]["en.ag_news"]["laya"]["calibrated"]["accuracy"],+ col["suites"]["en.emotion"]["laya"]["calibrated"]["accuracy"], 0.766]+ x = np.arange(3)+ axC.bar(x - w / 2 - 0.01, jev, w, color=INK3, label="Jev (published)", zorder=3)+ axC.bar(x + w / 2 + 0.01, lay, w, color=BLUE, label="Laya (measured)", zorder=3)+ for xi, v in zip(x - w / 2 - 0.01, jev):+ axC.text(xi, v + .02, "%.3f" % v, ha="center", fontsize=9, color=INK2)+ for xi, v in zip(x + w / 2 + 0.01, lay):+ axC.text(xi, v + .02, "%.3f" % v, ha="center", fontsize=9, color=INK2)+ axC.set_xticks(x); axC.set_xticklabels(names, fontsize=9)+ axC.set_ylim(0, 1.06); axC.set_ylabel("accuracy", fontsize=10)+ axC.set_title("Against Jev, same public datasets", fontsize=13, fontweight="bold",+ color=INK, loc="left", pad=10)+ axC.yaxis.grid(True, color=GRID, lw=0.9); axC.set_axisbelow(True)+ strip(axC); axC.legend(frameon=False, fontsize=9.5, loc="lower left", labelcolor=INK2)+ axC.text(0, -0.34, "typed-decisions uses the fine-tuned checkpoint. Jev figures are "+ "third-party published, not measured here.",+ transform=axC.transAxes, fontsize=8, color=INK3)++ # ---------------------------------------------------------------- D. calibration+ axD = fig.add_subplot(gs[2, 1])+ rep = col["calibration_repair"]+ models = ["laya", "laya-multilingual"]+ ship = [rep[m]["mean_ece_shipped"] for m in models]+ refit = [rep[m]["mean_ece_refit"] for m in models]+ x = np.arange(2)+ axD.bar(x - w / 2 - 0.01, ship, w, color=ORANGE, label="as shipped", zorder=3)+ axD.bar(x + w / 2 + 0.01, refit, w, color=BLUE, label="temperature refit", zorder=3)+ for xi, v in zip(x - w / 2 - 0.01, ship):+ axD.text(xi, v + .012, "%.3f" % v, ha="center", fontsize=9, color=INK2)+ for xi, v in zip(x + w / 2 + 0.01, refit):+ axD.text(xi, v + .012, "%.3f" % v, ha="center", fontsize=9, color=INK2)+ axD.axhline(0.246, color=INK3, lw=1.4, ls=(0, (4, 3)), zorder=2)+ axD.text(1.45, 0.253, "Jev ECE 0.246", color=INK3, fontsize=8.5, ha="right")+ axD.set_xticks(x); axD.set_xticklabels(models, fontsize=9.5)+ axD.set_ylabel("mean ECE (lower is better)", fontsize=10)+ axD.set_ylim(0, max(ship) * 1.25)+ axD.set_title("Calibration: fit your temperatures", fontsize=13, fontweight="bold",+ color=INK, loc="left", pad=10)+ axD.yaxis.grid(True, color=GRID, lw=0.9); axD.set_axisbelow(True)+ strip(axD); axD.legend(frameon=False, fontsize=9.5, loc="upper right", labelcolor=INK2)++ fig.suptitle("Laya benchmark · three checkpoints, identical questions, one T4",+ fontsize=16, fontweight="bold", color=INK, x=0.012, ha="left", y=0.985)+ out = os.path.join(ASSETS, "laya_benchmark.png")+ fig.savefig(out, dpi=150, bbox_inches="tight")+ print("wrote", out, "(%.0f KB)" % (os.path.getsize(out) / 1024))+++if __name__ == "__main__":+ main()