Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
research: refresh the 51-language multilingual columns from the re-run (#208)
PerryLinkad28be47ccfc
diff --git a/BENCHMARKS.md b/BENCHMARKS.mdindex 8a9ce586..79d8e228 100644--- a/BENCHMARKS.md+++ b/BENCHMARKS.md@@ -23,6 +23,16 @@ The same 51 languages and 5,100 cases have now been re-run after the temperatureThe raw-temperature column reproduces the committed file, so the only variable left is the clamp. `choice:11+` is the sole bucket it moves, and every case in this sweep is a 20-option question, so the clamp applies to all 5,100 — and lowers ECE in all 51 languages. `acc_at_50_coverage` is the one rank-quality column that uses the confidence values: macro 0.3004 → 0.3020, and `en` 0.94 → 0.98, so the flatter distribution selects a slightly better half rather than a worse one.+**The multilingual columns below are the re-run, not the committed file.** The committed `cpu_51_language_sweep.json` was measured at laya 0.2.0, and its `laya-multilingual` half does not reproduce on current code — 51 languages and 5,100 cases re-run under laya 0.2.0, 0.3.7 and 0.3.20, on x86 CPU and on an Apple M5 under both CPU and MPS, all give **0.4008** macro accuracy and match the committed file on only **6 of 51** languages, while the `laya` half reproduces exactly on all of them. Three machines agreeing settled it ([#208](https://github.com/NandhaKishorM/laya/issues/208)). The table therefore prints the refreshed numbers for both checkpoints, from `research/results/cpu_51_language_sweep_refreshed.json`, which records the environment and keeps the superseded committed columns beside the new ones:++| | committed (`laya-multilingual`) | refreshed |+|---|---|---|+| macro accuracy | 0.3661 | **0.4008** |+| macro ECE | 0.3869 | **0.3911** |+| languages clearing 3× random | 45 / 51 | **48 / 51** |++The direction is consistent rather than noise: of the 16 languages that move by 0.05 or more, **every one moves up** (`bn` 0.29 → 0.45, `kn` 0.15 → 0.30, `hy` 0.15 → 0.25), and none move down by that much. `laya`'s columns are unchanged by the refresh — they reproduce the committed file to the last stored digit — so the accuracy spread between the two checkpoints is wider than the committed table suggested, and `en` remains the one language where `laya` wins (0.820 against 0.710).+---## Headline@@ -43,65 +53,65 @@ The raw-temperature column reproduces the committed file, so the only variable l| | laya | laya-multilingual ||---|---|---|-| macro accuracy | 0.2269 | **0.3661** |-| macro ECE *(lower better)* | 0.7331 | **0.3869** |-| languages clearing 3× random | 23 / 51 | **45 / 51** |+| macro accuracy | 0.2269 | **0.4008** |+| macro ECE *(lower better)* | 0.5709 | **0.3911** |+| languages clearing 3x random | 23 / 51 | **48 / 51** |<details><summary><b>Per language (51)</b> — sorted by how much routing gains</summary>| lang | laya | laya-multilingual | Δ | laya ECE | multilingual ECE ||---|---|---|---|---|---|-| `th` | 0.080 | 0.480 | +0.400 | 0.881 | 0.336 |-| `ko` | 0.110 | 0.450 | +0.340 | 0.850 | 0.329 |-| `he` | 0.060 | 0.400 | +0.340 | 0.911 | 0.350 |-| `ur` | 0.070 | 0.400 | +0.330 | 0.883 | 0.311 |-| `hi` | 0.100 | 0.430 | +0.330 | 0.850 | 0.321 |-| `ar` | 0.110 | 0.400 | +0.290 | 0.800 | 0.341 |-| `pl` | 0.240 | 0.510 | +0.270 | 0.713 | 0.350 |-| `el` | 0.130 | 0.380 | +0.250 | 0.839 | 0.383 |-| `fa` | 0.140 | 0.390 | +0.250 | 0.820 | 0.399 |-| `ru` | 0.310 | 0.540 | +0.230 | 0.668 | 0.316 |-| `tr` | 0.140 | 0.370 | +0.230 | 0.788 | 0.417 |-| `lv` | 0.100 | 0.320 | +0.220 | 0.847 | 0.480 |-| `bn` | 0.080 | 0.290 | +0.210 | 0.865 | 0.408 |-| `nb` | 0.330 | 0.530 | +0.200 | 0.648 | 0.327 |-| `vi` | 0.060 | 0.260 | +0.200 | 0.891 | 0.521 |-| `az` | 0.100 | 0.300 | +0.200 | 0.825 | 0.368 |-| `hu` | 0.090 | 0.290 | +0.200 | 0.857 | 0.422 |-| `is` | 0.110 | 0.300 | +0.190 | 0.835 | 0.469 |-| `sv` | 0.380 | 0.570 | +0.190 | 0.596 | 0.276 |-| `km` | 0.000 | 0.180 | +0.180 | 0.952 | 0.412 |-| `ml` | 0.070 | 0.240 | +0.170 | 0.857 | 0.414 |-| `it` | 0.340 | 0.500 | +0.160 | 0.647 | 0.302 |-| `fi` | 0.130 | 0.290 | +0.160 | 0.849 | 0.436 |-| `ms` | 0.270 | 0.430 | +0.160 | 0.688 | 0.392 |-| `da` | 0.350 | 0.500 | +0.150 | 0.626 | 0.263 |-| `id` | 0.360 | 0.510 | +0.150 | 0.613 | 0.305 |-| `te` | 0.090 | 0.220 | +0.130 | 0.858 | 0.370 |-| `sl` | 0.200 | 0.330 | +0.130 | 0.756 | 0.433 |-| `jv` | 0.160 | 0.270 | +0.110 | 0.803 | 0.506 |-| `ta` | 0.120 | 0.230 | +0.110 | 0.822 | 0.397 |-| `ja` | 0.530 | 0.640 | +0.110 | 0.460 | 0.228 |-| `hy` | 0.050 | 0.150 | +0.100 | 0.835 | 0.506 |-| `zh-TW` | 0.460 | 0.540 | +0.080 | 0.520 | 0.327 |-| `de` | 0.420 | 0.500 | +0.080 | 0.558 | 0.301 |-| `tl` | 0.290 | 0.360 | +0.070 | 0.676 | 0.374 |-| `nl` | 0.390 | 0.450 | +0.060 | 0.591 | 0.378 |-| `af` | 0.290 | 0.350 | +0.060 | 0.687 | 0.484 |-| `my` | 0.060 | 0.120 | +0.060 | 0.861 | 0.455 |-| `sq` | 0.210 | 0.260 | +0.050 | 0.755 | 0.476 |-| `sw` | 0.130 | 0.180 | +0.050 | 0.828 | 0.549 |-| `cy` | 0.120 | 0.160 | +0.040 | 0.841 | 0.591 |-| `kn` | 0.110 | 0.150 | +0.040 | 0.842 | 0.437 |-| `es` | 0.510 | 0.530 | +0.020 | 0.480 | 0.275 |-| `ka` | 0.090 | 0.110 | +0.020 | 0.845 | 0.528 |-| `ro` | 0.330 | 0.350 | +0.020 | 0.658 | 0.404 |-| `zh-CN` | 0.620 | 0.630 | +0.010 | 0.376 | 0.212 |-| `am` | 0.120 | 0.110 | -0.010 | 0.825 | 0.463 |-| `pt` | 0.470 | 0.450 | -0.020 | 0.512 | 0.342 |-| `mn` | 0.130 | 0.100 | -0.030 | 0.837 | 0.558 |-| `fr` | 0.590 | 0.540 | -0.050 | 0.388 | 0.277 |-| `en` | 0.820 | 0.680 | -0.140 | 0.179 | 0.209 |+| `th` | 0.080 | 0.480 | +0.400 | 0.718 | 0.356 |+| `bn` | 0.080 | 0.450 | +0.370 | 0.627 | 0.354 |+| `fa` | 0.140 | 0.510 | +0.370 | 0.633 | 0.298 |+| `hi` | 0.100 | 0.460 | +0.360 | 0.642 | 0.368 |+| `ko` | 0.110 | 0.470 | +0.360 | 0.680 | 0.311 |+| `ar` | 0.110 | 0.460 | +0.350 | 0.580 | 0.315 |+| `ur` | 0.070 | 0.420 | +0.350 | 0.643 | 0.336 |+| `el` | 0.130 | 0.440 | +0.310 | 0.679 | 0.430 |+| `he` | 0.060 | 0.370 | +0.310 | 0.738 | 0.419 |+| `vi` | 0.060 | 0.340 | +0.280 | 0.690 | 0.464 |+| `hu` | 0.090 | 0.360 | +0.270 | 0.678 | 0.401 |+| `az` | 0.100 | 0.360 | +0.260 | 0.598 | 0.381 |+| `pl` | 0.240 | 0.500 | +0.260 | 0.570 | 0.353 |+| `tr` | 0.140 | 0.400 | +0.260 | 0.587 | 0.400 |+| `ru` | 0.310 | 0.570 | +0.260 | 0.595 | 0.296 |+| `is` | 0.110 | 0.350 | +0.240 | 0.664 | 0.452 |+| `nb` | 0.330 | 0.560 | +0.230 | 0.518 | 0.294 |+| `fi` | 0.130 | 0.340 | +0.210 | 0.719 | 0.392 |+| `ml` | 0.070 | 0.280 | +0.210 | 0.601 | 0.443 |+| `lv` | 0.100 | 0.310 | +0.210 | 0.666 | 0.472 |+| `hy` | 0.050 | 0.250 | +0.200 | 0.571 | 0.513 |+| `km` | 0.000 | 0.200 | +0.200 | 0.705 | 0.445 |+| `kn` | 0.110 | 0.300 | +0.190 | 0.590 | 0.388 |+| `ta` | 0.120 | 0.310 | +0.190 | 0.554 | 0.448 |+| `it` | 0.340 | 0.520 | +0.180 | 0.516 | 0.343 |+| `da` | 0.350 | 0.520 | +0.170 | 0.526 | 0.315 |+| `sl` | 0.200 | 0.370 | +0.170 | 0.599 | 0.434 |+| `id` | 0.360 | 0.510 | +0.150 | 0.487 | 0.323 |+| `zh-TW` | 0.460 | 0.610 | +0.150 | 0.434 | 0.266 |+| `jv` | 0.160 | 0.300 | +0.140 | 0.670 | 0.484 |+| `ms` | 0.270 | 0.410 | +0.140 | 0.506 | 0.436 |+| `te` | 0.090 | 0.220 | +0.130 | 0.669 | 0.497 |+| `ja` | 0.530 | 0.640 | +0.110 | 0.366 | 0.234 |+| `sv` | 0.380 | 0.490 | +0.110 | 0.487 | 0.341 |+| `my` | 0.060 | 0.160 | +0.100 | 0.580 | 0.477 |+| `sw` | 0.130 | 0.230 | +0.100 | 0.632 | 0.492 |+| `sq` | 0.210 | 0.300 | +0.090 | 0.603 | 0.487 |+| `de` | 0.420 | 0.500 | +0.080 | 0.452 | 0.336 |+| `nl` | 0.390 | 0.470 | +0.080 | 0.502 | 0.397 |+| `es` | 0.510 | 0.580 | +0.070 | 0.422 | 0.258 |+| `af` | 0.290 | 0.350 | +0.060 | 0.559 | 0.439 |+| `ka` | 0.090 | 0.150 | +0.060 | 0.610 | 0.542 |+| `tl` | 0.290 | 0.350 | +0.060 | 0.512 | 0.430 |+| `ro` | 0.330 | 0.370 | +0.040 | 0.550 | 0.397 |+| `pt` | 0.470 | 0.500 | +0.030 | 0.432 | 0.341 |+| `zh-CN` | 0.620 | 0.650 | +0.030 | 0.320 | 0.219 |+| `am` | 0.120 | 0.150 | +0.030 | 0.680 | 0.484 |+| `mn` | 0.130 | 0.160 | +0.030 | 0.670 | 0.577 |+| `cy` | 0.120 | 0.130 | +0.010 | 0.645 | 0.587 |+| `fr` | 0.590 | 0.600 | +0.010 | 0.299 | 0.248 |+| `en` | 0.820 | 0.710 | -0.110 | 0.138 | 0.233 |</details>@@ -116,7 +126,7 @@ The raw-temperature column reproduces the committed file, so the only variable l| XNLI — English | **0.860** | 0.843 || XNLI — other languages | 0.521 | **0.731** |-The English checkpoint does not degrade gracefully outside English — it collapses, and stays confident doing so. Khmer: **0.000 accuracy at 0.952 confidence**. Its mean confidence never drops below 0.885 at any accuracy level, so confidence gating cannot catch it — which is why routing happens *before* the forward pass.+The English checkpoint does not degrade gracefully outside English — it collapses, and stays confident doing so. Khmer: **0.000 accuracy at 0.952 confidence** raw, **0.705** as served after the clamp. Its mean confidence never drops below 0.885 (raw; 0.621 as served) at any accuracy level, so confidence gating cannot catch it — which is why routing happens *before* the forward pass.---diff --git a/README.md b/README.mdindex 54c20cf9..bc4c0447 100644--- a/README.md+++ b/README.md@@ -364,11 +364,11 @@ On a shared benchmark (17,416 questions, one T4 GPU, identical questions per mod| MASSIVE intent, 13 other languages | 0.306 | **0.451** | **0.451** || XNLI, English | **0.860** | 0.843 | **0.860** || XNLI, 14 other languages | 0.521 | **0.731** | **0.731** |-| Languages usable (>3x random) | 23 / 51 | 45 / 51 | **45 / 51** |+| Languages usable (>3x random) | 23 / 51 | 48 / 51 | **48 / 51** || Latency, 1 question (T4 GPU) | 39.5 ms | **32.8 ms** | **32.8 ms** || Latency, 10 questions batched | 158.6 ms | **72.3 ms** | **72.3 ms** |-The English checkpoint collapses on non-Latin scripts (Khmer scores **0.000 accuracy at 0.952 confidence**). Because the model stays confident while being wrong, confidence gating cannot save you. `Router` detects the script in <0.5 ms pure Python before the forward pass.+The English checkpoint collapses on non-Latin scripts (Khmer scores **0.000 accuracy at 0.952 confidence**, the raw-temperature figure; **0.705** as served after the clamp). Because the model stays confident while being wrong, confidence gating cannot save you. `Router` detects the script in <0.5 ms pure Python before the forward pass.### Production Preload & Memory@@ -882,7 +882,7 @@ published, never measured here** (no TypeSafe API access), so sample sizes and p| Banking77 (72 vs 77 labels) | **0.870** | 0.425 | Jev leads on >20 options || ECE *(lower better)* | 0.246 | **0.081** | 3× better (post-temperature) || p50 latency, 1 question | 236–276 ms | **32.8 ms** | 7.8× faster |-| Languages usable | *no published benchmark* | **45 of 51** | — |+| Languages usable | *no published benchmark* | **48 of 51** | — || Weights | closed API | **Apache 2.0** | — || Cost | $0.042 / 1M tokens | **$0 self-hosted** | — |@@ -933,10 +933,13 @@ All of the capability on this benchmark comes from fine-tuning.| XNLI, 14 other languages | 0.521 | **0.731** |Across all 51 languages the English checkpoint macro-averages **0.227** with macro ECE-**0.733**, and only 23 of 51 languages clear 3x random. Khmer scores **0.000 at 95.2%-confidence**. This is why [`Router`](#quickstart-route-mode-recommended) exists: the-model's own confidence gives no warning, so the routing decision has to be made before the-forward pass.+**0.733** *(0.571 as served, after the temperature clamp)*, and only 23 of 51 languages clear+3x random. `laya-multilingual` reaches **0.401** on the same 5,100 cases, clearing 3x random in+48 of 51 — see [BENCHMARKS.md](BENCHMARKS.md) for why those multilingual numbers are a re-run+rather than the committed file. Khmer scores **0.000 at 95.2% confidence** raw, **70.5%** as+served, on the English checkpoint. This is why+[`Router`](#quickstart-route-mode-recommended) exists: the model's own+confidence gives no warning, so the routing decision has to be made before the forward pass.### English tasksdiff --git a/research/README.md b/research/README.mdindex 9ce68d7c..4e879b54 100644--- a/research/README.md+++ b/research/README.md@@ -78,18 +78,22 @@ python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-si| `results/t4_colab_benchmark.json` | 17,416 questions on one T4, both checkpoints, identical questions per model || `results/long_context_multilingual.json` | the long-document run behind `assets/long_context_8192.png`: every prediction, with device and library versions || `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options |-| `results/cpu_51_language_sweep_clamped.json` | the same 51 languages and 5,100 cases re-run after the temperature clamp, raw temperatures and served temperatures side by side ([#208](https://github.com/NandhaKishorM/laya/issues/208)) |+| `results/cpu_51_language_sweep_clamped.json` | the English half of the 51 languages re-run after the temperature clamp, raw temperatures and served temperatures side by side ([#222](https://github.com/NandhaKishorM/laya/pull/222)) |+| `results/cpu_51_language_sweep_refreshed.json` | both checkpoints with the environment recorded, the superseded committed columns kept beside the new ones. The multilingual half of the committed sweep does not reproduce on current code, so this is what the table in `BENCHMARKS.md` prints ([#208](https://github.com/NandhaKishorM/laya/issues/208)) |## Headline findings-**Routing takes Laya from 23 to 45 of 51 languages.** On MASSIVE intent (20 options, random =+**Routing takes Laya from 23 to 48 of 51 languages.** On MASSIVE intent (20 options, random =0.050) the English checkpoint macro-averages 0.227 and clears 3x random on 23 of 51 languages;-the multilingual checkpoint reaches 0.366 and clears it on 45.+the multilingual checkpoint reaches 0.401 and clears it on 48. The multilingual figures are the+re-run, not the committed sweep — see `results/cpu_51_language_sweep_refreshed.json` below.**The English checkpoint's confidence gives no warning when it cannot read the input.** Khmer:-0.000 accuracy at 0.952 mean confidence. Macro ECE 0.733 across 51 languages, with mean-confidence never dropping below 0.885 at any accuracy level. This is why routing has to happen-*before* the forward pass — confidence gating cannot catch it.+0.000 accuracy at 0.952 mean confidence — that is the *raw* temperature; as served, after the+#42 clamp, it is 0.705 at the same 0.000 accuracy. Macro ECE 0.733 across 51 languages as+committed (0.571 as served), with mean confidence never dropping below 0.885 raw, or 0.621 as+served, at any accuracy level. This is why routing has to happen *before* the forward pass — confidence gating cannot+catch it.**Both checkpoints ship over-confident.** Refitting one temperature per (question type, optioncount) on held-out data moves mean ECE 0.466 -> 0.081 (`laya`) and 0.314 -> 0.106diff --git a/research/results/cpu_51_language_sweep_refreshed.json b/research/results/cpu_51_language_sweep_refreshed.jsonnew file mode 100644index 00000000..8fb0e7f8--- /dev/null+++ b/research/results/cpu_51_language_sweep_refreshed.json@@ -0,0 +1,3153 @@+{+ "meta": {+ "timestamp": "2026-09-24 19:04:32",+ "device": "cpu",+ "threads": 20,+ "cpu": "Intel64 Family 6 Model 198 Stepping 2, GenuineIntel",+ "platform": "Windows-11-10.0.26200-SP0",+ "python": "3.12.14",+ "laya": "0.3.20",+ "torch": "2.14.0+cpu",+ "transformers": "5.17.0",+ "harness": "research/eval/laya_eval.py",+ "seed": 13,+ "n_options": 20,+ "per_lang": 100,+ "dataset": "mteb/amazon_massive_intent",+ "split": "test",+ "instructions": "What is the user asking for in `utterance`?",+ "why": "The multilingual columns of cpu_51_language_sweep.json were measured at laya 0.2.0 and do not reproduce on current code; the English columns do. See issue #208. Both halves of this file are one run at one version; `provenance` records it."+ },+ "provenance": {+ "english": {+ "source": "a fresh run on the machine described in `meta`",+ "laya_version": "0.3.20",+ "max_len": 512,+ "head_max_len": 192,+ "freshly_run_for_this_file": true,+ "note": "bit-identical on all 51 languages to the #222 re-run at laya 0.3.7, clamped and unclamped, which is why the `laya` column matches the committed sweep exactly"+ },+ "multilingual": {+ "source": "a fresh run on the machine described in `meta`",+ "laya_version": "0.3.20",+ "max_len": 1024,+ "head_max_len": 256,+ "freshly_run_for_this_file": true,+ "note": "the committed multilingual columns do not reproduce; four runs on three machines and three laya versions agree on these numbers"+ }+ },+ "by_model": {+ "english": {+ "summary": {+ "committed": {+ "macro_accuracy": 0.2269,+ "macro_ece": 0.7331,+ "n_languages": 51,+ "languages_above_random": 23+ },+ "refreshed_clamped": {+ "macro_accuracy": 0.2269,+ "macro_ece": 0.5709,+ "macro_f1": 0.2053,+ "seconds": 2022.3,+ "n_languages": 51+ },+ "refreshed_unclamped": {+ "macro_accuracy": 0.2269,+ "macro_ece": 0.7331,+ "macro_f1": 0.2053,+ "seconds": 1924.0,+ "n_languages": 51+ }+ },+ "per_language": {+ "af": {+ "committed": {+ "n": 100,+ "accuracy": 0.29,+ "macro_f1": 0.2874,+ "ece": 0.6873,+ "brier": 1.3787,+ "nll": 16.3516,+ "mean_confidence": 0.9752,+ "acc_at_50_coverage": 0.42,+ "seconds": 11.6,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.29,+ "macro_f1": 0.2874,+ "ece": 0.5588,+ "mean_confidence": 0.8488,+ "acc_at_50_coverage": 0.42,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.29,+ "macro_f1": 0.2874,+ "ece": 0.6873,+ "mean_confidence": 0.9752,+ "acc_at_50_coverage": 0.42,+ "n": 100+ }+ },+ "am": {+ "committed": {+ "n": 100,+ "accuracy": 0.12,+ "macro_f1": 0.1122,+ "ece": 0.8252,+ "brier": 1.6644,+ "nll": 21.6026,+ "mean_confidence": 0.9452,+ "acc_at_50_coverage": 0.14,+ "seconds": 14.6,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.12,+ "macro_f1": 0.1122,+ "ece": 0.6804,+ "mean_confidence": 0.7873,+ "acc_at_50_coverage": 0.12,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.12,+ "macro_f1": 0.1122,+ "ece": 0.8253,+ "mean_confidence": 0.9453,+ "acc_at_50_coverage": 0.14,+ "n": 100+ }+ },+ "ar": {+ "committed": {+ "n": 100,+ "accuracy": 0.11,+ "macro_f1": 0.093,+ "ece": 0.8001,+ "brier": 1.6539,+ "nll": 20.5087,+ "mean_confidence": 0.8976,+ "acc_at_50_coverage": 0.12,+ "seconds": 11.8,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.11,+ "macro_f1": 0.093,+ "ece": 0.5795,+ "mean_confidence": 0.6833,+ "acc_at_50_coverage": 0.12,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.11,+ "macro_f1": 0.093,+ "ece": 0.8001,+ "mean_confidence": 0.8976,+ "acc_at_50_coverage": 0.12,+ "n": 100+ }+ },+ "az": {+ "committed": {+ "n": 100,+ "accuracy": 0.1,+ "macro_f1": 0.0629,+ "ece": 0.8247,+ "brier": 1.6966,+ "nll": 21.7232,+ "mean_confidence": 0.9247,+ "acc_at_50_coverage": 0.14,+ "seconds": 11.9,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.1,+ "macro_f1": 0.0629,+ "ece": 0.5983,+ "mean_confidence": 0.6983,+ "acc_at_50_coverage": 0.14,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.1,+ "macro_f1": 0.0629,+ "ece": 0.8247,+ "mean_confidence": 0.9247,+ "acc_at_50_coverage": 0.14,+ "n": 100+ }+ },+ "bn": {+ "committed": {+ "n": 100,+ "accuracy": 0.08,+ "macro_f1": 0.056,+ "ece": 0.8646,+ "brier": 1.7373,+ "nll": 20.8691,+ "mean_confidence": 0.9446,+ "acc_at_50_coverage": 0.1,+ "seconds": 13.9,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.08,+ "macro_f1": 0.056,+ "ece": 0.6267,+ "mean_confidence": 0.7067,+ "acc_at_50_coverage": 0.12,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.08,+ "macro_f1": 0.056,+ "ece": 0.8646,+ "mean_confidence": 0.9446,+ "acc_at_50_coverage": 0.1,+ "n": 100+ }+ },+ "cy": {+ "committed": {+ "n": 100,+ "accuracy": 0.12,+ "macro_f1": 0.0906,+ "ece": 0.8409,+ "brier": 1.7029,+ "nll": 21.2255,+ "mean_confidence": 0.9609,+ "acc_at_50_coverage": 0.14,+ "seconds": 11.8,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.12,+ "macro_f1": 0.0906,+ "ece": 0.6452,+ "mean_confidence": 0.7652,+ "acc_at_50_coverage": 0.14,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.12,+ "macro_f1": 0.0906,+ "ece": 0.8409,+ "mean_confidence": 0.9609,+ "acc_at_50_coverage": 0.14,+ "n": 100+ }+ },+ "da": {+ "committed": {+ "n": 100,+ "accuracy": 0.35,+ "macro_f1": 0.3082,+ "ece": 0.6263,+ "brier": 1.2633,+ "nll": 16.2409,+ "mean_confidence": 0.9763,+ "acc_at_50_coverage": 0.52,+ "seconds": 12.0,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.35,+ "macro_f1": 0.3082,+ "ece": 0.526,+ "mean_confidence": 0.876,+ "acc_at_50_coverage": 0.52,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.35,+ "macro_f1": 0.3082,+ "ece": 0.6263,+ "mean_confidence": 0.9763,+ "acc_at_50_coverage": 0.52,+ "n": 100+ }+ },+ "de": {+ "committed": {+ "n": 100,+ "accuracy": 0.42,+ "macro_f1": 0.387,+ "ece": 0.5584,+ "brier": 1.126,+ "nll": 13.7538,+ "mean_confidence": 0.9784,+ "acc_at_50_coverage": 0.66,+ "seconds": 14.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.42,+ "macro_f1": 0.387,+ "ece": 0.452,+ "mean_confidence": 0.872,+ "acc_at_50_coverage": 0.68,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.42,+ "macro_f1": 0.387,+ "ece": 0.5584,+ "mean_confidence": 0.9784,+ "acc_at_50_coverage": 0.66,+ "n": 100+ }+ },+ "el": {+ "committed": {+ "n": 100,+ "accuracy": 0.13,+ "macro_f1": 0.1,+ "ece": 0.8387,+ "brier": 1.6854,+ "nll": 21.1672,+ "mean_confidence": 0.9687,+ "acc_at_50_coverage": 0.16,+ "seconds": 15.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.13,+ "macro_f1": 0.1,+ "ece": 0.6789,+ "mean_confidence": 0.8089,+ "acc_at_50_coverage": 0.2,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.13,+ "macro_f1": 0.1,+ "ece": 0.8388,+ "mean_confidence": 0.9688,+ "acc_at_50_coverage": 0.16,+ "n": 100+ }+ },+ "en": {+ "committed": {+ "n": 100,+ "accuracy": 0.82,+ "macro_f1": 0.7876,+ "ece": 0.1789,+ "brier": 0.3563,+ "nll": 3.5193,+ "mean_confidence": 0.9989,+ "acc_at_50_coverage": 0.94,+ "seconds": 16.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.82,+ "macro_f1": 0.7876,+ "ece": 0.1382,+ "mean_confidence": 0.9582,+ "acc_at_50_coverage": 0.98,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.82,+ "macro_f1": 0.7876,+ "ece": 0.1789,+ "mean_confidence": 0.9989,+ "acc_at_50_coverage": 0.94,+ "n": 100+ }+ },+ "es": {+ "committed": {+ "n": 100,+ "accuracy": 0.51,+ "macro_f1": 0.4767,+ "ece": 0.4796,+ "brier": 0.9706,+ "nll": 11.846,+ "mean_confidence": 0.9896,+ "acc_at_50_coverage": 0.6,+ "seconds": 20.3,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.51,+ "macro_f1": 0.4767,+ "ece": 0.4225,+ "mean_confidence": 0.9245,+ "acc_at_50_coverage": 0.62,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.51,+ "macro_f1": 0.4767,+ "ece": 0.4796,+ "mean_confidence": 0.9896,+ "acc_at_50_coverage": 0.6,+ "n": 100+ }+ },+ "fa": {+ "committed": {+ "n": 100,+ "accuracy": 0.14,+ "macro_f1": 0.1197,+ "ece": 0.8196,+ "brier": 1.6664,+ "nll": 21.65,+ "mean_confidence": 0.9414,+ "acc_at_50_coverage": 0.16,+ "seconds": 24.5,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.14,+ "macro_f1": 0.1197,+ "ece": 0.6331,+ "mean_confidence": 0.7731,+ "acc_at_50_coverage": 0.16,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.14,+ "macro_f1": 0.1197,+ "ece": 0.8196,+ "mean_confidence": 0.9414,+ "acc_at_50_coverage": 0.16,+ "n": 100+ }+ },+ "fi": {+ "committed": {+ "n": 100,+ "accuracy": 0.13,+ "macro_f1": 0.1213,+ "ece": 0.8486,+ "brier": 1.7071,+ "nll": 21.923,+ "mean_confidence": 0.9786,+ "acc_at_50_coverage": 0.16,+ "seconds": 20.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.13,+ "macro_f1": 0.1213,+ "ece": 0.7189,+ "mean_confidence": 0.8489,+ "acc_at_50_coverage": 0.16,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.13,+ "macro_f1": 0.1213,+ "ece": 0.8486,+ "mean_confidence": 0.9786,+ "acc_at_50_coverage": 0.16,+ "n": 100+ }+ },+ "fr": {+ "committed": {+ "n": 100,+ "accuracy": 0.59,+ "macro_f1": 0.5822,+ "ece": 0.3882,+ "brier": 0.7935,+ "nll": 9.1132,+ "mean_confidence": 0.9782,+ "acc_at_50_coverage": 0.78,+ "seconds": 19.4,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.59,+ "macro_f1": 0.5822,+ "ece": 0.2995,+ "mean_confidence": 0.8895,+ "acc_at_50_coverage": 0.78,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.59,+ "macro_f1": 0.5822,+ "ece": 0.3882,+ "mean_confidence": 0.9782,+ "acc_at_50_coverage": 0.78,+ "n": 100+ }+ },+ "he": {+ "committed": {+ "n": 100,+ "accuracy": 0.06,+ "macro_f1": 0.039,+ "ece": 0.9111,+ "brier": 1.8321,+ "nll": 24.2157,+ "mean_confidence": 0.9643,+ "acc_at_50_coverage": 0.04,+ "seconds": 20.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.06,+ "macro_f1": 0.039,+ "ece": 0.7379,+ "mean_confidence": 0.7842,+ "acc_at_50_coverage": 0.02,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.06,+ "macro_f1": 0.039,+ "ece": 0.9111,+ "mean_confidence": 0.9643,+ "acc_at_50_coverage": 0.04,+ "n": 100+ }+ },+ "hi": {+ "committed": {+ "n": 100,+ "accuracy": 0.1,+ "macro_f1": 0.0758,+ "ece": 0.8504,+ "brier": 1.7169,+ "nll": 21.3541,+ "mean_confidence": 0.9408,+ "acc_at_50_coverage": 0.1,+ "seconds": 21.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.1,+ "macro_f1": 0.0758,+ "ece": 0.6416,+ "mean_confidence": 0.7416,+ "acc_at_50_coverage": 0.08,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.1,+ "macro_f1": 0.0758,+ "ece": 0.8504,+ "mean_confidence": 0.9408,+ "acc_at_50_coverage": 0.1,+ "n": 100+ }+ },+ "hu": {+ "committed": {+ "n": 100,+ "accuracy": 0.09,+ "macro_f1": 0.0813,+ "ece": 0.8571,+ "brier": 1.7503,+ "nll": 23.2612,+ "mean_confidence": 0.9471,+ "acc_at_50_coverage": 0.06,+ "seconds": 18.9,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.09,+ "macro_f1": 0.0813,+ "ece": 0.6783,+ "mean_confidence": 0.7683,+ "acc_at_50_coverage": 0.06,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.09,+ "macro_f1": 0.0813,+ "ece": 0.8571,+ "mean_confidence": 0.9471,+ "acc_at_50_coverage": 0.06,+ "n": 100+ }+ },+ "hy": {+ "committed": {+ "n": 100,+ "accuracy": 0.05,+ "macro_f1": 0.0355,+ "ece": 0.8348,+ "brier": 1.7324,+ "nll": 20.695,+ "mean_confidence": 0.8848,+ "acc_at_50_coverage": 0.08,+ "seconds": 22.5,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.05,+ "macro_f1": 0.0355,+ "ece": 0.5709,+ "mean_confidence": 0.6209,+ "acc_at_50_coverage": 0.04,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.05,+ "macro_f1": 0.0355,+ "ece": 0.8348,+ "mean_confidence": 0.8848,+ "acc_at_50_coverage": 0.08,+ "n": 100+ }+ },+ "id": {+ "committed": {+ "n": 100,+ "accuracy": 0.36,+ "macro_f1": 0.3283,+ "ece": 0.6133,+ "brier": 1.2484,+ "nll": 14.6851,+ "mean_confidence": 0.9619,+ "acc_at_50_coverage": 0.48,+ "seconds": 18.7,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.36,+ "macro_f1": 0.3283,+ "ece": 0.487,+ "mean_confidence": 0.8304,+ "acc_at_50_coverage": 0.48,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.36,+ "macro_f1": 0.3283,+ "ece": 0.6133,+ "mean_confidence": 0.9619,+ "acc_at_50_coverage": 0.48,+ "n": 100+ }+ },+ "is": {+ "committed": {+ "n": 100,+ "accuracy": 0.11,+ "macro_f1": 0.0893,+ "ece": 0.8351,+ "brier": 1.6975,+ "nll": 21.9127,+ "mean_confidence": 0.9451,+ "acc_at_50_coverage": 0.16,+ "seconds": 19.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.11,+ "macro_f1": 0.0893,+ "ece": 0.6641,+ "mean_confidence": 0.7741,+ "acc_at_50_coverage": 0.16,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.11,+ "macro_f1": 0.0893,+ "ece": 0.8351,+ "mean_confidence": 0.9451,+ "acc_at_50_coverage": 0.16,+ "n": 100+ }+ },+ "it": {+ "committed": {+ "n": 100,+ "accuracy": 0.34,+ "macro_f1": 0.3101,+ "ece": 0.6474,+ "brier": 1.2866,+ "nll": 15.5383,+ "mean_confidence": 0.9735,+ "acc_at_50_coverage": 0.52,+ "seconds": 19.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.34,+ "macro_f1": 0.3101,+ "ece": 0.5156,+ "mean_confidence": 0.8534,+ "acc_at_50_coverage": 0.52,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.34,+ "macro_f1": 0.3101,+ "ece": 0.6474,+ "mean_confidence": 0.9735,+ "acc_at_50_coverage": 0.52,+ "n": 100+ }+ },+ "ja": {+ "committed": {+ "n": 100,+ "accuracy": 0.53,+ "macro_f1": 0.4942,+ "ece": 0.4599,+ "brier": 0.9247,+ "nll": 10.8681,+ "mean_confidence": 0.9801,+ "acc_at_50_coverage": 0.76,+ "seconds": 19.9,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.53,+ "macro_f1": 0.4942,+ "ece": 0.3657,+ "mean_confidence": 0.8957,+ "acc_at_50_coverage": 0.76,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.53,+ "macro_f1": 0.4942,+ "ece": 0.4599,+ "mean_confidence": 0.9801,+ "acc_at_50_coverage": 0.76,+ "n": 100+ }+ },+ "jv": {+ "committed": {+ "n": 100,+ "accuracy": 0.16,+ "macro_f1": 0.1537,+ "ece": 0.8035,+ "brier": 1.6288,+ "nll": 20.6859,+ "mean_confidence": 0.9635,+ "acc_at_50_coverage": 0.14,+ "seconds": 19.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.16,+ "macro_f1": 0.1537,+ "ece": 0.67,+ "mean_confidence": 0.83,+ "acc_at_50_coverage": 0.16,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.16,+ "macro_f1": 0.1537,+ "ece": 0.8035,+ "mean_confidence": 0.9635,+ "acc_at_50_coverage": 0.14,+ "n": 100+ }+ },+ "ka": {+ "committed": {+ "n": 100,+ "accuracy": 0.09,+ "macro_f1": 0.0742,+ "ece": 0.8448,+ "brier": 1.72,+ "nll": 21.2473,+ "mean_confidence": 0.9348,+ "acc_at_50_coverage": 0.1,+ "seconds": 22.0,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.09,+ "macro_f1": 0.0742,+ "ece": 0.6102,+ "mean_confidence": 0.7002,+ "acc_at_50_coverage": 0.1,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.09,+ "macro_f1": 0.0742,+ "ece": 0.8448,+ "mean_confidence": 0.9348,+ "acc_at_50_coverage": 0.1,+ "n": 100+ }+ },+ "km": {+ "committed": {+ "n": 100,+ "accuracy": 0.0,+ "macro_f1": 0.0,+ "ece": 0.9515,+ "brier": 1.9219,+ "nll": 23.2123,+ "mean_confidence": 0.9515,+ "acc_at_50_coverage": 0.0,+ "seconds": 25.4,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.0,+ "macro_f1": 0.0,+ "ece": 0.7051,+ "mean_confidence": 0.7051,+ "acc_at_50_coverage": 0.0,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.0,+ "macro_f1": 0.0,+ "ece": 0.9515,+ "mean_confidence": 0.9515,+ "acc_at_50_coverage": 0.0,+ "n": 100+ }+ },+ "kn": {+ "committed": {+ "n": 100,+ "accuracy": 0.11,+ "macro_f1": 0.0913,+ "ece": 0.8416,+ "brier": 1.7051,+ "nll": 20.4279,+ "mean_confidence": 0.9421,+ "acc_at_50_coverage": 0.06,+ "seconds": 24.3,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.11,+ "macro_f1": 0.0913,+ "ece": 0.5903,+ "mean_confidence": 0.695,+ "acc_at_50_coverage": 0.08,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.11,+ "macro_f1": 0.0913,+ "ece": 0.8416,+ "mean_confidence": 0.9421,+ "acc_at_50_coverage": 0.06,+ "n": 100+ }+ },+ "ko": {+ "committed": {+ "n": 100,+ "accuracy": 0.11,+ "macro_f1": 0.0943,+ "ece": 0.8498,+ "brier": 1.7193,+ "nll": 22.1195,+ "mean_confidence": 0.9598,+ "acc_at_50_coverage": 0.14,+ "seconds": 20.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.11,+ "macro_f1": 0.0943,+ "ece": 0.6804,+ "mean_confidence": 0.7904,+ "acc_at_50_coverage": 0.14,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.11,+ "macro_f1": 0.0943,+ "ece": 0.8498,+ "mean_confidence": 0.9598,+ "acc_at_50_coverage": 0.14,+ "n": 100+ }+ },+ "lv": {+ "committed": {+ "n": 100,+ "accuracy": 0.1,+ "macro_f1": 0.0786,+ "ece": 0.8466,+ "brier": 1.7322,+ "nll": 21.4978,+ "mean_confidence": 0.9466,+ "acc_at_50_coverage": 0.1,+ "seconds": 19.4,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.1,+ "macro_f1": 0.0786,+ "ece": 0.6659,+ "mean_confidence": 0.7659,+ "acc_at_50_coverage": 0.1,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.1,+ "macro_f1": 0.0786,+ "ece": 0.8466,+ "mean_confidence": 0.9466,+ "acc_at_50_coverage": 0.1,+ "n": 100+ }+ },+ "ml": {+ "committed": {+ "n": 100,+ "accuracy": 0.07,+ "macro_f1": 0.0503,+ "ece": 0.8572,+ "brier": 1.7485,+ "nll": 20.0485,+ "mean_confidence": 0.9272,+ "acc_at_50_coverage": 0.1,+ "seconds": 24.6,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.07,+ "macro_f1": 0.0503,+ "ece": 0.601,+ "mean_confidence": 0.671,+ "acc_at_50_coverage": 0.1,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.07,+ "macro_f1": 0.0503,+ "ece": 0.8572,+ "mean_confidence": 0.9272,+ "acc_at_50_coverage": 0.1,+ "n": 100+ }+ },+ "mn": {+ "committed": {+ "n": 100,+ "accuracy": 0.13,+ "macro_f1": 0.1213,+ "ece": 0.837,+ "brier": 1.7109,+ "nll": 21.5421,+ "mean_confidence": 0.9591,+ "acc_at_50_coverage": 0.14,+ "seconds": 20.6,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.13,+ "macro_f1": 0.1213,+ "ece": 0.6704,+ "mean_confidence": 0.7977,+ "acc_at_50_coverage": 0.14,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.13,+ "macro_f1": 0.1213,+ "ece": 0.837,+ "mean_confidence": 0.9591,+ "acc_at_50_coverage": 0.14,+ "n": 100+ }+ },+ "ms": {+ "committed": {+ "n": 100,+ "accuracy": 0.27,+ "macro_f1": 0.2372,+ "ece": 0.6877,+ "brier": 1.4027,+ "nll": 16.277,+ "mean_confidence": 0.9577,+ "acc_at_50_coverage": 0.42,+ "seconds": 19.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.27,+ "macro_f1": 0.2372,+ "ece": 0.5064,+ "mean_confidence": 0.7764,+ "acc_at_50_coverage": 0.4,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.27,+ "macro_f1": 0.2372,+ "ece": 0.6877,+ "mean_confidence": 0.9577,+ "acc_at_50_coverage": 0.42,+ "n": 100+ }+ },+ "my": {+ "committed": {+ "n": 100,+ "accuracy": 0.06,+ "macro_f1": 0.0455,+ "ece": 0.8605,+ "brier": 1.7635,+ "nll": 19.7847,+ "mean_confidence": 0.9205,+ "acc_at_50_coverage": 0.08,+ "seconds": 24.7,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.06,+ "macro_f1": 0.0455,+ "ece": 0.5797,+ "mean_confidence": 0.6397,+ "acc_at_50_coverage": 0.08,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.06,+ "macro_f1": 0.0455,+ "ece": 0.8605,+ "mean_confidence": 0.9205,+ "acc_at_50_coverage": 0.08,+ "n": 100+ }+ },+ "nb": {+ "committed": {+ "n": 100,+ "accuracy": 0.33,+ "macro_f1": 0.3025,+ "ece": 0.6484,+ "brier": 1.302,+ "nll": 16.4489,+ "mean_confidence": 0.9692,+ "acc_at_50_coverage": 0.5,+ "seconds": 19.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.33,+ "macro_f1": 0.3025,+ "ece": 0.5179,+ "mean_confidence": 0.8479,+ "acc_at_50_coverage": 0.52,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.33,+ "macro_f1": 0.3025,+ "ece": 0.6484,+ "mean_confidence": 0.9692,+ "acc_at_50_coverage": 0.5,+ "n": 100+ }+ },+ "nl": {+ "committed": {+ "n": 100,+ "accuracy": 0.39,+ "macro_f1": 0.3968,+ "ece": 0.5914,+ "brier": 1.1943,+ "nll": 14.4159,+ "mean_confidence": 0.9745,+ "acc_at_50_coverage": 0.54,+ "seconds": 20.7,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.39,+ "macro_f1": 0.3968,+ "ece": 0.5025,+ "mean_confidence": 0.8831,+ "acc_at_50_coverage": 0.56,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.39,+ "macro_f1": 0.3968,+ "ece": 0.5914,+ "mean_confidence": 0.9745,+ "acc_at_50_coverage": 0.54,+ "n": 100+ }+ },+ "pl": {+ "committed": {+ "n": 100,+ "accuracy": 0.24,+ "macro_f1": 0.2033,+ "ece": 0.7129,+ "brier": 1.4653,+ "nll": 18.4945,+ "mean_confidence": 0.9529,+ "acc_at_50_coverage": 0.34,+ "seconds": 20.8,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.24,+ "macro_f1": 0.2033,+ "ece": 0.5695,+ "mean_confidence": 0.7951,+ "acc_at_50_coverage": 0.36,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.24,+ "macro_f1": 0.2033,+ "ece": 0.7129,+ "mean_confidence": 0.9529,+ "acc_at_50_coverage": 0.34,+ "n": 100+ }+ },+ "pt": {+ "committed": {+ "n": 100,+ "accuracy": 0.47,+ "macro_f1": 0.4213,+ "ece": 0.5116,+ "brier": 1.0273,+ "nll": 12.7024,+ "mean_confidence": 0.9718,+ "acc_at_50_coverage": 0.66,+ "seconds": 20.8,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.47,+ "macro_f1": 0.4213,+ "ece": 0.4317,+ "mean_confidence": 0.8894,+ "acc_at_50_coverage": 0.66,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.47,+ "macro_f1": 0.4213,+ "ece": 0.5116,+ "mean_confidence": 0.9718,+ "acc_at_50_coverage": 0.66,+ "n": 100+ }+ },+ "ro": {+ "committed": {+ "n": 100,+ "accuracy": 0.33,+ "macro_f1": 0.2976,+ "ece": 0.6577,+ "brier": 1.3076,+ "nll": 15.9637,+ "mean_confidence": 0.9824,+ "acc_at_50_coverage": 0.48,+ "seconds": 20.0,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.33,+ "macro_f1": 0.2976,+ "ece": 0.5499,+ "mean_confidence": 0.8668,+ "acc_at_50_coverage": 0.48,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.33,+ "macro_f1": 0.2976,+ "ece": 0.6576,+ "mean_confidence": 0.9824,+ "acc_at_50_coverage": 0.48,+ "n": 100+ }+ },+ "ru": {+ "committed": {+ "n": 100,+ "accuracy": 0.31,+ "macro_f1": 0.3084,+ "ece": 0.668,+ "brier": 1.3452,+ "nll": 17.0527,+ "mean_confidence": 0.978,+ "acc_at_50_coverage": 0.42,+ "seconds": 20.5,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.31,+ "macro_f1": 0.3084,+ "ece": 0.5946,+ "mean_confidence": 0.9046,+ "acc_at_50_coverage": 0.42,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.31,+ "macro_f1": 0.3084,+ "ece": 0.668,+ "mean_confidence": 0.978,+ "acc_at_50_coverage": 0.42,+ "n": 100+ }+ },+ "sl": {+ "committed": {+ "n": 100,+ "accuracy": 0.2,+ "macro_f1": 0.1679,+ "ece": 0.7561,+ "brier": 1.5293,+ "nll": 19.7372,+ "mean_confidence": 0.9561,+ "acc_at_50_coverage": 0.26,+ "seconds": 20.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.2,+ "macro_f1": 0.1679,+ "ece": 0.5994,+ "mean_confidence": 0.7994,+ "acc_at_50_coverage": 0.26,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.2,+ "macro_f1": 0.1679,+ "ece": 0.7561,+ "mean_confidence": 0.9561,+ "acc_at_50_coverage": 0.26,+ "n": 100+ }+ },+ "sq": {+ "committed": {+ "n": 100,+ "accuracy": 0.21,+ "macro_f1": 0.1667,+ "ece": 0.7549,+ "brier": 1.5259,+ "nll": 19.5615,+ "mean_confidence": 0.9649,+ "acc_at_50_coverage": 0.28,+ "seconds": 20.5,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.21,+ "macro_f1": 0.1667,+ "ece": 0.6031,+ "mean_confidence": 0.8131,+ "acc_at_50_coverage": 0.28,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.21,+ "macro_f1": 0.1667,+ "ece": 0.7549,+ "mean_confidence": 0.9649,+ "acc_at_50_coverage": 0.28,+ "n": 100+ }+ },+ "sv": {+ "committed": {+ "n": 100,+ "accuracy": 0.38,+ "macro_f1": 0.3809,+ "ece": 0.5956,+ "brier": 1.2168,+ "nll": 14.3656,+ "mean_confidence": 0.9706,+ "acc_at_50_coverage": 0.5,+ "seconds": 19.9,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.38,+ "macro_f1": 0.3809,+ "ece": 0.4874,+ "mean_confidence": 0.8501,+ "acc_at_50_coverage": 0.5,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.38,+ "macro_f1": 0.3809,+ "ece": 0.5956,+ "mean_confidence": 0.9706,+ "acc_at_50_coverage": 0.5,+ "n": 100+ }+ },+ "sw": {+ "committed": {+ "n": 100,+ "accuracy": 0.13,+ "macro_f1": 0.1149,+ "ece": 0.8282,+ "brier": 1.6934,+ "nll": 21.2625,+ "mean_confidence": 0.9562,+ "acc_at_50_coverage": 0.14,+ "seconds": 21.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.13,+ "macro_f1": 0.1149,+ "ece": 0.6318,+ "mean_confidence": 0.7511,+ "acc_at_50_coverage": 0.14,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.13,+ "macro_f1": 0.1149,+ "ece": 0.8282,+ "mean_confidence": 0.9562,+ "acc_at_50_coverage": 0.14,+ "n": 100+ }+ },+ "ta": {+ "committed": {+ "n": 100,+ "accuracy": 0.12,+ "macro_f1": 0.0971,+ "ece": 0.8224,+ "brier": 1.6768,+ "nll": 19.6604,+ "mean_confidence": 0.9424,+ "acc_at_50_coverage": 0.12,+ "seconds": 25.6,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.12,+ "macro_f1": 0.0971,+ "ece": 0.5541,+ "mean_confidence": 0.6741,+ "acc_at_50_coverage": 0.12,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.12,+ "macro_f1": 0.0971,+ "ece": 0.8224,+ "mean_confidence": 0.9424,+ "acc_at_50_coverage": 0.12,+ "n": 100+ }+ },+ "te": {+ "committed": {+ "n": 100,+ "accuracy": 0.09,+ "macro_f1": 0.0722,+ "ece": 0.8575,+ "brier": 1.7384,+ "nll": 21.777,+ "mean_confidence": 0.9475,+ "acc_at_50_coverage": 0.06,+ "seconds": 25.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.09,+ "macro_f1": 0.0722,+ "ece": 0.6689,+ "mean_confidence": 0.7568,+ "acc_at_50_coverage": 0.06,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.09,+ "macro_f1": 0.0722,+ "ece": 0.8575,+ "mean_confidence": 0.9475,+ "acc_at_50_coverage": 0.06,+ "n": 100+ }+ },+ "th": {+ "committed": {+ "n": 100,+ "accuracy": 0.08,+ "macro_f1": 0.048,+ "ece": 0.8814,+ "brier": 1.7728,+ "nll": 22.4846,+ "mean_confidence": 0.9614,+ "acc_at_50_coverage": 0.1,+ "seconds": 22.9,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.08,+ "macro_f1": 0.048,+ "ece": 0.7179,+ "mean_confidence": 0.7979,+ "acc_at_50_coverage": 0.1,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.08,+ "macro_f1": 0.048,+ "ece": 0.8814,+ "mean_confidence": 0.9614,+ "acc_at_50_coverage": 0.1,+ "n": 100+ }+ },+ "tl": {+ "committed": {+ "n": 100,+ "accuracy": 0.29,+ "macro_f1": 0.249,+ "ece": 0.6756,+ "brier": 1.3666,+ "nll": 16.3214,+ "mean_confidence": 0.9656,+ "acc_at_50_coverage": 0.46,+ "seconds": 20.7,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.29,+ "macro_f1": 0.249,+ "ece": 0.5121,+ "mean_confidence": 0.8021,+ "acc_at_50_coverage": 0.46,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.29,+ "macro_f1": 0.249,+ "ece": 0.6756,+ "mean_confidence": 0.9656,+ "acc_at_50_coverage": 0.46,+ "n": 100+ }+ },+ "tr": {+ "committed": {+ "n": 100,+ "accuracy": 0.14,+ "macro_f1": 0.0967,+ "ece": 0.788,+ "brier": 1.6064,+ "nll": 19.6609,+ "mean_confidence": 0.928,+ "acc_at_50_coverage": 0.2,+ "seconds": 20.7,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.14,+ "macro_f1": 0.0967,+ "ece": 0.5874,+ "mean_confidence": 0.7274,+ "acc_at_50_coverage": 0.2,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.14,+ "macro_f1": 0.0967,+ "ece": 0.788,+ "mean_confidence": 0.928,+ "acc_at_50_coverage": 0.2,+ "n": 100+ }+ },+ "ur": {+ "committed": {+ "n": 100,+ "accuracy": 0.07,+ "macro_f1": 0.0645,+ "ece": 0.8826,+ "brier": 1.7822,+ "nll": 21.4571,+ "mean_confidence": 0.9526,+ "acc_at_50_coverage": 0.08,+ "seconds": 23.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.07,+ "macro_f1": 0.0645,+ "ece": 0.6428,+ "mean_confidence": 0.7128,+ "acc_at_50_coverage": 0.08,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.07,+ "macro_f1": 0.0645,+ "ece": 0.8826,+ "mean_confidence": 0.9526,+ "acc_at_50_coverage": 0.08,+ "n": 100+ }+ },+ "vi": {+ "committed": {+ "n": 100,+ "accuracy": 0.06,+ "macro_f1": 0.0498,+ "ece": 0.8914,+ "brier": 1.8055,+ "nll": 22.6885,+ "mean_confidence": 0.9514,+ "acc_at_50_coverage": 0.06,+ "seconds": 22.9,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.06,+ "macro_f1": 0.0498,+ "ece": 0.69,+ "mean_confidence": 0.75,+ "acc_at_50_coverage": 0.06,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.06,+ "macro_f1": 0.0498,+ "ece": 0.8914,+ "mean_confidence": 0.9514,+ "acc_at_50_coverage": 0.06,+ "n": 100+ }+ },+ "zh-CN": {+ "committed": {+ "n": 100,+ "accuracy": 0.62,+ "macro_f1": 0.6175,+ "ece": 0.376,+ "brier": 0.7521,+ "nll": 8.5938,+ "mean_confidence": 0.9885,+ "acc_at_50_coverage": 0.86,+ "seconds": 22.0,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.62,+ "macro_f1": 0.6175,+ "ece": 0.3198,+ "mean_confidence": 0.9244,+ "acc_at_50_coverage": 0.84,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.62,+ "macro_f1": 0.6175,+ "ece": 0.376,+ "mean_confidence": 0.9885,+ "acc_at_50_coverage": 0.86,+ "n": 100+ }+ },+ "zh-TW": {+ "committed": {+ "n": 100,+ "accuracy": 0.46,+ "macro_f1": 0.4294,+ "ece": 0.5198,+ "brier": 1.0483,+ "nll": 12.3023,+ "mean_confidence": 0.9798,+ "acc_at_50_coverage": 0.74,+ "seconds": 21.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.46,+ "macro_f1": 0.4294,+ "ece": 0.4344,+ "mean_confidence": 0.8944,+ "acc_at_50_coverage": 0.72,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.46,+ "macro_f1": 0.4294,+ "ece": 0.5198,+ "mean_confidence": 0.9798,+ "acc_at_50_coverage": 0.74,+ "n": 100+ }+ }+ }+ },+ "multilingual": {+ "summary": {+ "committed": {+ "macro_accuracy": 0.3661,+ "macro_ece": 0.3869,+ "n_languages": 51,+ "languages_above_random": 45+ },+ "refreshed_clamped": {+ "macro_accuracy": 0.4008,+ "macro_ece": 0.3911,+ "macro_f1": 0.3737,+ "seconds": 744.2,+ "n_languages": 51+ },+ "refreshed_unclamped": {+ "macro_accuracy": 0.4008,+ "macro_ece": 0.3911,+ "macro_f1": 0.3737,+ "seconds": 731.8,+ "n_languages": 51+ }+ },+ "per_language": {+ "af": {+ "committed": {+ "n": 100,+ "accuracy": 0.35,+ "macro_f1": 0.3133,+ "ece": 0.4845,+ "brier": 1.0466,+ "nll": 3.5729,+ "mean_confidence": 0.8028,+ "acc_at_50_coverage": 0.54,+ "seconds": 9.0,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.35,+ "macro_f1": 0.3411,+ "ece": 0.4394,+ "mean_confidence": 0.7894,+ "acc_at_50_coverage": 0.58,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.35,+ "macro_f1": 0.3411,+ "ece": 0.4394,+ "mean_confidence": 0.7894,+ "acc_at_50_coverage": 0.58,+ "n": 100+ }+ },+ "am": {+ "committed": {+ "n": 100,+ "accuracy": 0.11,+ "macro_f1": 0.1046,+ "ece": 0.4626,+ "brier": 1.2477,+ "nll": 4.265,+ "mean_confidence": 0.5706,+ "acc_at_50_coverage": 0.12,+ "seconds": 9.7,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.15,+ "macro_f1": 0.1307,+ "ece": 0.4839,+ "mean_confidence": 0.6339,+ "acc_at_50_coverage": 0.26,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.15,+ "macro_f1": 0.1307,+ "ece": 0.4839,+ "mean_confidence": 0.6339,+ "acc_at_50_coverage": 0.26,+ "n": 100+ }+ },+ "ar": {+ "committed": {+ "n": 100,+ "accuracy": 0.4,+ "macro_f1": 0.3926,+ "ece": 0.341,+ "brier": 0.8619,+ "nll": 2.697,+ "mean_confidence": 0.731,+ "acc_at_50_coverage": 0.6,+ "seconds": 8.8,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.46,+ "macro_f1": 0.4325,+ "ece": 0.315,+ "mean_confidence": 0.775,+ "acc_at_50_coverage": 0.6,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.46,+ "macro_f1": 0.4325,+ "ece": 0.315,+ "mean_confidence": 0.775,+ "acc_at_50_coverage": 0.6,+ "n": 100+ }+ },+ "az": {+ "committed": {+ "n": 100,+ "accuracy": 0.3,+ "macro_f1": 0.2693,+ "ece": 0.3685,+ "brier": 0.9758,+ "nll": 3.1016,+ "mean_confidence": 0.6685,+ "acc_at_50_coverage": 0.46,+ "seconds": 9.7,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.36,+ "macro_f1": 0.3361,+ "ece": 0.381,+ "mean_confidence": 0.7211,+ "acc_at_50_coverage": 0.54,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.36,+ "macro_f1": 0.3361,+ "ece": 0.381,+ "mean_confidence": 0.7211,+ "acc_at_50_coverage": 0.54,+ "n": 100+ }+ },+ "bn": {+ "committed": {+ "n": 100,+ "accuracy": 0.29,+ "macro_f1": 0.2654,+ "ece": 0.4085,+ "brier": 1.0428,+ "nll": 3.2654,+ "mean_confidence": 0.6944,+ "acc_at_50_coverage": 0.4,+ "seconds": 10.3,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.45,+ "macro_f1": 0.44,+ "ece": 0.354,+ "mean_confidence": 0.7788,+ "acc_at_50_coverage": 0.54,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.45,+ "macro_f1": 0.44,+ "ece": 0.354,+ "mean_confidence": 0.7788,+ "acc_at_50_coverage": 0.54,+ "n": 100+ }+ },+ "cy": {+ "committed": {+ "n": 100,+ "accuracy": 0.16,+ "macro_f1": 0.1432,+ "ece": 0.5905,+ "brier": 1.3315,+ "nll": 5.0927,+ "mean_confidence": 0.7149,+ "acc_at_50_coverage": 0.18,+ "seconds": 9.4,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.13,+ "macro_f1": 0.1217,+ "ece": 0.5873,+ "mean_confidence": 0.7173,+ "acc_at_50_coverage": 0.22,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.13,+ "macro_f1": 0.1217,+ "ece": 0.5873,+ "mean_confidence": 0.7173,+ "acc_at_50_coverage": 0.22,+ "n": 100+ }+ },+ "da": {+ "committed": {+ "n": 100,+ "accuracy": 0.5,+ "macro_f1": 0.4682,+ "ece": 0.2628,+ "brier": 0.7433,+ "nll": 2.0483,+ "mean_confidence": 0.7544,+ "acc_at_50_coverage": 0.68,+ "seconds": 9.3,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.52,+ "macro_f1": 0.4956,+ "ece": 0.315,+ "mean_confidence": 0.8062,+ "acc_at_50_coverage": 0.7,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.52,+ "macro_f1": 0.4956,+ "ece": 0.315,+ "mean_confidence": 0.8062,+ "acc_at_50_coverage": 0.7,+ "n": 100+ }+ },+ "de": {+ "committed": {+ "n": 100,+ "accuracy": 0.5,+ "macro_f1": 0.4847,+ "ece": 0.3014,+ "brier": 0.752,+ "nll": 2.4918,+ "mean_confidence": 0.784,+ "acc_at_50_coverage": 0.7,+ "seconds": 8.8,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.5,+ "macro_f1": 0.4768,+ "ece": 0.3358,+ "mean_confidence": 0.8131,+ "acc_at_50_coverage": 0.76,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.5,+ "macro_f1": 0.4768,+ "ece": 0.3358,+ "mean_confidence": 0.8131,+ "acc_at_50_coverage": 0.76,+ "n": 100+ }+ },+ "el": {+ "committed": {+ "n": 100,+ "accuracy": 0.38,+ "macro_f1": 0.3564,+ "ece": 0.3832,+ "brier": 0.9572,+ "nll": 3.235,+ "mean_confidence": 0.7507,+ "acc_at_50_coverage": 0.52,+ "seconds": 9.3,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.44,+ "macro_f1": 0.4148,+ "ece": 0.4297,+ "mean_confidence": 0.801,+ "acc_at_50_coverage": 0.58,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.44,+ "macro_f1": 0.4148,+ "ece": 0.4297,+ "mean_confidence": 0.801,+ "acc_at_50_coverage": 0.58,+ "n": 100+ }+ },+ "en": {+ "committed": {+ "n": 100,+ "accuracy": 0.68,+ "macro_f1": 0.6282,+ "ece": 0.209,+ "brier": 0.5052,+ "nll": 1.4497,+ "mean_confidence": 0.8719,+ "acc_at_50_coverage": 0.96,+ "seconds": 8.6,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.71,+ "macro_f1": 0.6689,+ "ece": 0.2333,+ "mean_confidence": 0.8876,+ "acc_at_50_coverage": 0.92,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.71,+ "macro_f1": 0.6689,+ "ece": 0.2333,+ "mean_confidence": 0.8876,+ "acc_at_50_coverage": 0.92,+ "n": 100+ }+ },+ "es": {+ "committed": {+ "n": 100,+ "accuracy": 0.53,+ "macro_f1": 0.4863,+ "ece": 0.2749,+ "brier": 0.6896,+ "nll": 2.0238,+ "mean_confidence": 0.8049,+ "acc_at_50_coverage": 0.76,+ "seconds": 9.0,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.58,+ "macro_f1": 0.5213,+ "ece": 0.2577,+ "mean_confidence": 0.796,+ "acc_at_50_coverage": 0.7,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.58,+ "macro_f1": 0.5213,+ "ece": 0.2577,+ "mean_confidence": 0.796,+ "acc_at_50_coverage": 0.7,+ "n": 100+ }+ },+ "fa": {+ "committed": {+ "n": 100,+ "accuracy": 0.39,+ "macro_f1": 0.3751,+ "ece": 0.3986,+ "brier": 0.9086,+ "nll": 2.7913,+ "mean_confidence": 0.7886,+ "acc_at_50_coverage": 0.6,+ "seconds": 9.0,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.51,+ "macro_f1": 0.4728,+ "ece": 0.2985,+ "mean_confidence": 0.7957,+ "acc_at_50_coverage": 0.6,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.51,+ "macro_f1": 0.4728,+ "ece": 0.2985,+ "mean_confidence": 0.7957,+ "acc_at_50_coverage": 0.6,+ "n": 100+ }+ },+ "fi": {+ "committed": {+ "n": 100,+ "accuracy": 0.29,+ "macro_f1": 0.2533,+ "ece": 0.4364,+ "brier": 1.0851,+ "nll": 3.7674,+ "mean_confidence": 0.7197,+ "acc_at_50_coverage": 0.38,+ "seconds": 9.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.34,+ "macro_f1": 0.3282,+ "ece": 0.3918,+ "mean_confidence": 0.7316,+ "acc_at_50_coverage": 0.46,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.34,+ "macro_f1": 0.3282,+ "ece": 0.3918,+ "mean_confidence": 0.7316,+ "acc_at_50_coverage": 0.46,+ "n": 100+ }+ },+ "fr": {+ "committed": {+ "n": 100,+ "accuracy": 0.54,+ "macro_f1": 0.5199,+ "ece": 0.277,+ "brier": 0.6613,+ "nll": 1.9862,+ "mean_confidence": 0.8074,+ "acc_at_50_coverage": 0.8,+ "seconds": 9.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.6,+ "macro_f1": 0.5733,+ "ece": 0.2484,+ "mean_confidence": 0.8403,+ "acc_at_50_coverage": 0.8,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.6,+ "macro_f1": 0.5733,+ "ece": 0.2484,+ "mean_confidence": 0.8403,+ "acc_at_50_coverage": 0.8,+ "n": 100+ }+ },+ "he": {+ "committed": {+ "n": 100,+ "accuracy": 0.4,+ "macro_f1": 0.3794,+ "ece": 0.3503,+ "brier": 0.9297,+ "nll": 2.6466,+ "mean_confidence": 0.7302,+ "acc_at_50_coverage": 0.5,+ "seconds": 8.9,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.37,+ "macro_f1": 0.3581,+ "ece": 0.4186,+ "mean_confidence": 0.787,+ "acc_at_50_coverage": 0.58,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.37,+ "macro_f1": 0.3581,+ "ece": 0.4186,+ "mean_confidence": 0.787,+ "acc_at_50_coverage": 0.58,+ "n": 100+ }+ },+ "hi": {+ "committed": {+ "n": 100,+ "accuracy": 0.43,+ "macro_f1": 0.397,+ "ece": 0.3206,+ "brier": 0.8304,+ "nll": 2.7293,+ "mean_confidence": 0.7506,+ "acc_at_50_coverage": 0.64,+ "seconds": 8.9,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.46,+ "macro_f1": 0.409,+ "ece": 0.3685,+ "mean_confidence": 0.8175,+ "acc_at_50_coverage": 0.58,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.46,+ "macro_f1": 0.409,+ "ece": 0.3685,+ "mean_confidence": 0.8175,+ "acc_at_50_coverage": 0.58,+ "n": 100+ }+ },+ "hu": {+ "committed": {+ "n": 100,+ "accuracy": 0.29,+ "macro_f1": 0.2654,+ "ece": 0.4221,+ "brier": 1.041,+ "nll": 3.5242,+ "mean_confidence": 0.7121,+ "acc_at_50_coverage": 0.44,+ "seconds": 8.6,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.36,+ "macro_f1": 0.3339,+ "ece": 0.4005,+ "mean_confidence": 0.7457,+ "acc_at_50_coverage": 0.46,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.36,+ "macro_f1": 0.3339,+ "ece": 0.4005,+ "mean_confidence": 0.7457,+ "acc_at_50_coverage": 0.46,+ "n": 100+ }+ },+ "hy": {+ "committed": {+ "n": 100,+ "accuracy": 0.15,+ "macro_f1": 0.1298,+ "ece": 0.5062,+ "brier": 1.2346,+ "nll": 4.0964,+ "mean_confidence": 0.6469,+ "acc_at_50_coverage": 0.22,+ "seconds": 9.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.25,+ "macro_f1": 0.2364,+ "ece": 0.5127,+ "mean_confidence": 0.7627,+ "acc_at_50_coverage": 0.3,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.25,+ "macro_f1": 0.2364,+ "ece": 0.5127,+ "mean_confidence": 0.7627,+ "acc_at_50_coverage": 0.3,+ "n": 100+ }+ },+ "id": {+ "committed": {+ "n": 100,+ "accuracy": 0.51,+ "macro_f1": 0.4955,+ "ece": 0.3048,+ "brier": 0.7362,+ "nll": 2.197,+ "mean_confidence": 0.8028,+ "acc_at_50_coverage": 0.72,+ "seconds": 8.8,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.51,+ "macro_f1": 0.4973,+ "ece": 0.3225,+ "mean_confidence": 0.8281,+ "acc_at_50_coverage": 0.76,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.51,+ "macro_f1": 0.4973,+ "ece": 0.3225,+ "mean_confidence": 0.8281,+ "acc_at_50_coverage": 0.76,+ "n": 100+ }+ },+ "is": {+ "committed": {+ "n": 100,+ "accuracy": 0.3,+ "macro_f1": 0.2831,+ "ece": 0.4692,+ "brier": 1.1123,+ "nll": 4.2693,+ "mean_confidence": 0.7551,+ "acc_at_50_coverage": 0.4,+ "seconds": 8.9,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.35,+ "macro_f1": 0.3355,+ "ece": 0.4518,+ "mean_confidence": 0.7949,+ "acc_at_50_coverage": 0.46,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.35,+ "macro_f1": 0.3355,+ "ece": 0.4518,+ "mean_confidence": 0.7949,+ "acc_at_50_coverage": 0.46,+ "n": 100+ }+ },+ "it": {+ "committed": {+ "n": 100,+ "accuracy": 0.5,+ "macro_f1": 0.4773,+ "ece": 0.302,+ "brier": 0.7537,+ "nll": 2.5765,+ "mean_confidence": 0.7825,+ "acc_at_50_coverage": 0.74,+ "seconds": 8.3,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.52,+ "macro_f1": 0.5034,+ "ece": 0.3428,+ "mean_confidence": 0.819,+ "acc_at_50_coverage": 0.68,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.52,+ "macro_f1": 0.5034,+ "ece": 0.3428,+ "mean_confidence": 0.819,+ "acc_at_50_coverage": 0.68,+ "n": 100+ }+ },+ "ja": {+ "committed": {+ "n": 100,+ "accuracy": 0.64,+ "macro_f1": 0.5923,+ "ece": 0.228,+ "brier": 0.5661,+ "nll": 1.5915,+ "mean_confidence": 0.8576,+ "acc_at_50_coverage": 0.84,+ "seconds": 8.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.64,+ "macro_f1": 0.5896,+ "ece": 0.2337,+ "mean_confidence": 0.8737,+ "acc_at_50_coverage": 0.84,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.64,+ "macro_f1": 0.5896,+ "ece": 0.2337,+ "mean_confidence": 0.8737,+ "acc_at_50_coverage": 0.84,+ "n": 100+ }+ },+ "jv": {+ "committed": {+ "n": 100,+ "accuracy": 0.27,+ "macro_f1": 0.2348,+ "ece": 0.5063,+ "brier": 1.1532,+ "nll": 3.9457,+ "mean_confidence": 0.7588,+ "acc_at_50_coverage": 0.34,+ "seconds": 8.7,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.3,+ "macro_f1": 0.2714,+ "ece": 0.4842,+ "mean_confidence": 0.7676,+ "acc_at_50_coverage": 0.4,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.3,+ "macro_f1": 0.2714,+ "ece": 0.4842,+ "mean_confidence": 0.7676,+ "acc_at_50_coverage": 0.4,+ "n": 100+ }+ },+ "ka": {+ "committed": {+ "n": 100,+ "accuracy": 0.11,+ "macro_f1": 0.1059,+ "ece": 0.5278,+ "brier": 1.3013,+ "nll": 4.4255,+ "mean_confidence": 0.6378,+ "acc_at_50_coverage": 0.14,+ "seconds": 9.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.15,+ "macro_f1": 0.1449,+ "ece": 0.5422,+ "mean_confidence": 0.6835,+ "acc_at_50_coverage": 0.22,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.15,+ "macro_f1": 0.1449,+ "ece": 0.5422,+ "mean_confidence": 0.6835,+ "acc_at_50_coverage": 0.22,+ "n": 100+ }+ },+ "km": {+ "committed": {+ "n": 100,+ "accuracy": 0.18,+ "macro_f1": 0.1745,+ "ece": 0.4123,+ "brier": 1.1383,+ "nll": 3.5727,+ "mean_confidence": 0.5762,+ "acc_at_50_coverage": 0.22,+ "seconds": 9.7,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.2,+ "macro_f1": 0.188,+ "ece": 0.4454,+ "mean_confidence": 0.6454,+ "acc_at_50_coverage": 0.22,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.2,+ "macro_f1": 0.188,+ "ece": 0.4454,+ "mean_confidence": 0.6454,+ "acc_at_50_coverage": 0.22,+ "n": 100+ }+ },+ "kn": {+ "committed": {+ "n": 100,+ "accuracy": 0.15,+ "macro_f1": 0.1388,+ "ece": 0.4371,+ "brier": 1.1512,+ "nll": 3.4801,+ "mean_confidence": 0.5804,+ "acc_at_50_coverage": 0.2,+ "seconds": 8.9,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.3,+ "macro_f1": 0.2465,+ "ece": 0.388,+ "mean_confidence": 0.6578,+ "acc_at_50_coverage": 0.36,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.3,+ "macro_f1": 0.2465,+ "ece": 0.388,+ "mean_confidence": 0.6578,+ "acc_at_50_coverage": 0.36,+ "n": 100+ }+ },+ "ko": {+ "committed": {+ "n": 100,+ "accuracy": 0.45,+ "macro_f1": 0.3927,+ "ece": 0.3293,+ "brier": 0.8278,+ "nll": 2.5605,+ "mean_confidence": 0.7297,+ "acc_at_50_coverage": 0.6,+ "seconds": 8.5,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.47,+ "macro_f1": 0.4428,+ "ece": 0.3108,+ "mean_confidence": 0.7773,+ "acc_at_50_coverage": 0.68,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.47,+ "macro_f1": 0.4428,+ "ece": 0.3108,+ "mean_confidence": 0.7773,+ "acc_at_50_coverage": 0.68,+ "n": 100+ }+ },+ "lv": {+ "committed": {+ "n": 100,+ "accuracy": 0.32,+ "macro_f1": 0.3023,+ "ece": 0.4804,+ "brier": 1.1257,+ "nll": 3.8958,+ "mean_confidence": 0.7779,+ "acc_at_50_coverage": 0.4,+ "seconds": 8.5,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.31,+ "macro_f1": 0.3022,+ "ece": 0.472,+ "mean_confidence": 0.782,+ "acc_at_50_coverage": 0.46,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.31,+ "macro_f1": 0.3022,+ "ece": 0.472,+ "mean_confidence": 0.782,+ "acc_at_50_coverage": 0.46,+ "n": 100+ }+ },+ "ml": {+ "committed": {+ "n": 100,+ "accuracy": 0.24,+ "macro_f1": 0.2188,+ "ece": 0.4142,+ "brier": 1.1209,+ "nll": 3.7403,+ "mean_confidence": 0.6486,+ "acc_at_50_coverage": 0.3,+ "seconds": 9.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.28,+ "macro_f1": 0.2712,+ "ece": 0.4433,+ "mean_confidence": 0.7072,+ "acc_at_50_coverage": 0.32,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.28,+ "macro_f1": 0.2712,+ "ece": 0.4433,+ "mean_confidence": 0.7072,+ "acc_at_50_coverage": 0.32,+ "n": 100+ }+ },+ "mn": {+ "committed": {+ "n": 100,+ "accuracy": 0.1,+ "macro_f1": 0.0735,+ "ece": 0.5581,+ "brier": 1.3302,+ "nll": 4.8227,+ "mean_confidence": 0.6562,+ "acc_at_50_coverage": 0.1,+ "seconds": 8.7,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.16,+ "macro_f1": 0.1422,+ "ece": 0.577,+ "mean_confidence": 0.737,+ "acc_at_50_coverage": 0.22,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.16,+ "macro_f1": 0.1422,+ "ece": 0.577,+ "mean_confidence": 0.737,+ "acc_at_50_coverage": 0.22,+ "n": 100+ }+ },+ "ms": {+ "committed": {+ "n": 100,+ "accuracy": 0.43,+ "macro_f1": 0.4032,+ "ece": 0.3924,+ "brier": 0.916,+ "nll": 2.9229,+ "mean_confidence": 0.8224,+ "acc_at_50_coverage": 0.62,+ "seconds": 8.4,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.41,+ "macro_f1": 0.3787,+ "ece": 0.4364,+ "mean_confidence": 0.8269,+ "acc_at_50_coverage": 0.58,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.41,+ "macro_f1": 0.3787,+ "ece": 0.4364,+ "mean_confidence": 0.8269,+ "acc_at_50_coverage": 0.58,+ "n": 100+ }+ },+ "my": {+ "committed": {+ "n": 100,+ "accuracy": 0.12,+ "macro_f1": 0.1185,+ "ece": 0.455,+ "brier": 1.2102,+ "nll": 3.8596,+ "mean_confidence": 0.575,+ "acc_at_50_coverage": 0.16,+ "seconds": 9.3,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.16,+ "macro_f1": 0.1424,+ "ece": 0.4766,+ "mean_confidence": 0.6219,+ "acc_at_50_coverage": 0.1,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.16,+ "macro_f1": 0.1424,+ "ece": 0.4766,+ "mean_confidence": 0.6219,+ "acc_at_50_coverage": 0.1,+ "n": 100+ }+ },+ "nb": {+ "committed": {+ "n": 100,+ "accuracy": 0.53,+ "macro_f1": 0.4796,+ "ece": 0.3273,+ "brier": 0.7368,+ "nll": 2.7576,+ "mean_confidence": 0.8573,+ "acc_at_50_coverage": 0.74,+ "seconds": 8.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.56,+ "macro_f1": 0.5174,+ "ece": 0.2937,+ "mean_confidence": 0.8537,+ "acc_at_50_coverage": 0.74,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.56,+ "macro_f1": 0.5174,+ "ece": 0.2937,+ "mean_confidence": 0.8537,+ "acc_at_50_coverage": 0.74,+ "n": 100+ }+ },+ "nl": {+ "committed": {+ "n": 100,+ "accuracy": 0.45,+ "macro_f1": 0.3861,+ "ece": 0.3775,+ "brier": 0.8735,+ "nll": 3.2134,+ "mean_confidence": 0.8137,+ "acc_at_50_coverage": 0.64,+ "seconds": 8.4,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.47,+ "macro_f1": 0.4069,+ "ece": 0.397,+ "mean_confidence": 0.8557,+ "acc_at_50_coverage": 0.68,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.47,+ "macro_f1": 0.4069,+ "ece": 0.397,+ "mean_confidence": 0.8557,+ "acc_at_50_coverage": 0.68,+ "n": 100+ }+ },+ "pl": {+ "committed": {+ "n": 100,+ "accuracy": 0.51,+ "macro_f1": 0.4859,+ "ece": 0.3501,+ "brier": 0.7774,+ "nll": 2.3803,+ "mean_confidence": 0.8356,+ "acc_at_50_coverage": 0.7,+ "seconds": 8.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.5,+ "macro_f1": 0.4781,+ "ece": 0.3529,+ "mean_confidence": 0.8407,+ "acc_at_50_coverage": 0.76,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.5,+ "macro_f1": 0.4781,+ "ece": 0.3529,+ "mean_confidence": 0.8407,+ "acc_at_50_coverage": 0.76,+ "n": 100+ }+ },+ "pt": {+ "committed": {+ "n": 100,+ "accuracy": 0.45,+ "macro_f1": 0.4108,+ "ece": 0.3419,+ "brier": 0.8515,+ "nll": 2.5313,+ "mean_confidence": 0.7863,+ "acc_at_50_coverage": 0.6,+ "seconds": 8.0,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.5,+ "macro_f1": 0.4598,+ "ece": 0.3405,+ "mean_confidence": 0.8136,+ "acc_at_50_coverage": 0.72,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.5,+ "macro_f1": 0.4598,+ "ece": 0.3405,+ "mean_confidence": 0.8136,+ "acc_at_50_coverage": 0.72,+ "n": 100+ }+ },+ "ro": {+ "committed": {+ "n": 100,+ "accuracy": 0.35,+ "macro_f1": 0.3269,+ "ece": 0.4035,+ "brier": 0.9823,+ "nll": 3.2401,+ "mean_confidence": 0.7501,+ "acc_at_50_coverage": 0.54,+ "seconds": 8.0,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.37,+ "macro_f1": 0.3426,+ "ece": 0.3974,+ "mean_confidence": 0.7674,+ "acc_at_50_coverage": 0.56,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.37,+ "macro_f1": 0.3426,+ "ece": 0.3974,+ "mean_confidence": 0.7674,+ "acc_at_50_coverage": 0.56,+ "n": 100+ }+ },+ "ru": {+ "committed": {+ "n": 100,+ "accuracy": 0.54,+ "macro_f1": 0.4872,+ "ece": 0.3163,+ "brier": 0.7624,+ "nll": 2.5212,+ "mean_confidence": 0.8405,+ "acc_at_50_coverage": 0.7,+ "seconds": 8.0,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.57,+ "macro_f1": 0.5342,+ "ece": 0.2959,+ "mean_confidence": 0.8594,+ "acc_at_50_coverage": 0.7,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.57,+ "macro_f1": 0.5342,+ "ece": 0.2959,+ "mean_confidence": 0.8594,+ "acc_at_50_coverage": 0.7,+ "n": 100+ }+ },+ "sl": {+ "committed": {+ "n": 100,+ "accuracy": 0.33,+ "macro_f1": 0.2981,+ "ece": 0.4335,+ "brier": 1.0212,+ "nll": 3.1538,+ "mean_confidence": 0.7635,+ "acc_at_50_coverage": 0.48,+ "seconds": 8.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.37,+ "macro_f1": 0.348,+ "ece": 0.4338,+ "mean_confidence": 0.7952,+ "acc_at_50_coverage": 0.48,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.37,+ "macro_f1": 0.348,+ "ece": 0.4338,+ "mean_confidence": 0.7952,+ "acc_at_50_coverage": 0.48,+ "n": 100+ }+ },+ "sq": {+ "committed": {+ "n": 100,+ "accuracy": 0.26,+ "macro_f1": 0.2275,+ "ece": 0.4764,+ "brier": 1.1099,+ "nll": 3.4562,+ "mean_confidence": 0.7348,+ "acc_at_50_coverage": 0.4,+ "seconds": 8.2,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.3,+ "macro_f1": 0.2679,+ "ece": 0.4869,+ "mean_confidence": 0.7795,+ "acc_at_50_coverage": 0.4,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.3,+ "macro_f1": 0.2679,+ "ece": 0.4869,+ "mean_confidence": 0.7795,+ "acc_at_50_coverage": 0.4,+ "n": 100+ }+ },+ "sv": {+ "committed": {+ "n": 100,+ "accuracy": 0.57,+ "macro_f1": 0.524,+ "ece": 0.2757,+ "brier": 0.7141,+ "nll": 2.3393,+ "mean_confidence": 0.8191,+ "acc_at_50_coverage": 0.74,+ "seconds": 7.8,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.49,+ "macro_f1": 0.4496,+ "ece": 0.341,+ "mean_confidence": 0.8215,+ "acc_at_50_coverage": 0.74,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.49,+ "macro_f1": 0.4496,+ "ece": 0.341,+ "mean_confidence": 0.8215,+ "acc_at_50_coverage": 0.74,+ "n": 100+ }+ },+ "sw": {+ "committed": {+ "n": 100,+ "accuracy": 0.18,+ "macro_f1": 0.1741,+ "ece": 0.5495,+ "brier": 1.265,+ "nll": 4.8324,+ "mean_confidence": 0.7295,+ "acc_at_50_coverage": 0.28,+ "seconds": 8.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.23,+ "macro_f1": 0.2259,+ "ece": 0.4922,+ "mean_confidence": 0.7118,+ "acc_at_50_coverage": 0.3,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.23,+ "macro_f1": 0.2259,+ "ece": 0.4922,+ "mean_confidence": 0.7118,+ "acc_at_50_coverage": 0.3,+ "n": 100+ }+ },+ "ta": {+ "committed": {+ "n": 100,+ "accuracy": 0.23,+ "macro_f1": 0.1945,+ "ece": 0.3968,+ "brier": 1.0924,+ "nll": 3.4722,+ "mean_confidence": 0.6268,+ "acc_at_50_coverage": 0.28,+ "seconds": 8.5,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.31,+ "macro_f1": 0.2842,+ "ece": 0.4479,+ "mean_confidence": 0.7371,+ "acc_at_50_coverage": 0.34,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.31,+ "macro_f1": 0.2842,+ "ece": 0.4479,+ "mean_confidence": 0.7371,+ "acc_at_50_coverage": 0.34,+ "n": 100+ }+ },+ "te": {+ "committed": {+ "n": 100,+ "accuracy": 0.22,+ "macro_f1": 0.1973,+ "ece": 0.3705,+ "brier": 1.0643,+ "nll": 3.348,+ "mean_confidence": 0.5905,+ "acc_at_50_coverage": 0.28,+ "seconds": 8.5,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.22,+ "macro_f1": 0.1935,+ "ece": 0.4969,+ "mean_confidence": 0.7169,+ "acc_at_50_coverage": 0.3,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.22,+ "macro_f1": 0.1935,+ "ece": 0.4969,+ "mean_confidence": 0.7169,+ "acc_at_50_coverage": 0.3,+ "n": 100+ }+ },+ "th": {+ "committed": {+ "n": 100,+ "accuracy": 0.48,+ "macro_f1": 0.4513,+ "ece": 0.3357,+ "brier": 0.8119,+ "nll": 2.5203,+ "mean_confidence": 0.7844,+ "acc_at_50_coverage": 0.62,+ "seconds": 8.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.48,+ "macro_f1": 0.4524,+ "ece": 0.3563,+ "mean_confidence": 0.8259,+ "acc_at_50_coverage": 0.62,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.48,+ "macro_f1": 0.4524,+ "ece": 0.3563,+ "mean_confidence": 0.8259,+ "acc_at_50_coverage": 0.62,+ "n": 100+ }+ },+ "tl": {+ "committed": {+ "n": 100,+ "accuracy": 0.36,+ "macro_f1": 0.332,+ "ece": 0.3738,+ "brier": 0.9662,+ "nll": 3.2804,+ "mean_confidence": 0.7338,+ "acc_at_50_coverage": 0.48,+ "seconds": 8.1,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.35,+ "macro_f1": 0.3033,+ "ece": 0.4297,+ "mean_confidence": 0.7797,+ "acc_at_50_coverage": 0.54,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.35,+ "macro_f1": 0.3033,+ "ece": 0.4297,+ "mean_confidence": 0.7797,+ "acc_at_50_coverage": 0.54,+ "n": 100+ }+ },+ "tr": {+ "committed": {+ "n": 100,+ "accuracy": 0.37,+ "macro_f1": 0.3398,+ "ece": 0.417,+ "brier": 0.9744,+ "nll": 3.1177,+ "mean_confidence": 0.7784,+ "acc_at_50_coverage": 0.5,+ "seconds": 7.8,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.4,+ "macro_f1": 0.3846,+ "ece": 0.4004,+ "mean_confidence": 0.8004,+ "acc_at_50_coverage": 0.6,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.4,+ "macro_f1": 0.3846,+ "ece": 0.4004,+ "mean_confidence": 0.8004,+ "acc_at_50_coverage": 0.6,+ "n": 100+ }+ },+ "ur": {+ "committed": {+ "n": 100,+ "accuracy": 0.4,+ "macro_f1": 0.3774,+ "ece": 0.3111,+ "brier": 0.8551,+ "nll": 2.7536,+ "mean_confidence": 0.7111,+ "acc_at_50_coverage": 0.58,+ "seconds": 8.0,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.42,+ "macro_f1": 0.3947,+ "ece": 0.3355,+ "mean_confidence": 0.7555,+ "acc_at_50_coverage": 0.6,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.42,+ "macro_f1": 0.3947,+ "ece": 0.3355,+ "mean_confidence": 0.7555,+ "acc_at_50_coverage": 0.6,+ "n": 100+ }+ },+ "vi": {+ "committed": {+ "n": 100,+ "accuracy": 0.26,+ "macro_f1": 0.2238,+ "ece": 0.5214,+ "brier": 1.145,+ "nll": 3.627,+ "mean_confidence": 0.7722,+ "acc_at_50_coverage": 0.4,+ "seconds": 7.9,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.34,+ "macro_f1": 0.3059,+ "ece": 0.4636,+ "mean_confidence": 0.778,+ "acc_at_50_coverage": 0.46,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.34,+ "macro_f1": 0.3059,+ "ece": 0.4636,+ "mean_confidence": 0.778,+ "acc_at_50_coverage": 0.46,+ "n": 100+ }+ },+ "zh-CN": {+ "committed": {+ "n": 100,+ "accuracy": 0.63,+ "macro_f1": 0.5936,+ "ece": 0.2123,+ "brier": 0.5221,+ "nll": 1.714,+ "mean_confidence": 0.8246,+ "acc_at_50_coverage": 0.88,+ "seconds": 7.7,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.65,+ "macro_f1": 0.605,+ "ece": 0.2192,+ "mean_confidence": 0.859,+ "acc_at_50_coverage": 0.84,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.65,+ "macro_f1": 0.605,+ "ece": 0.2192,+ "mean_confidence": 0.859,+ "acc_at_50_coverage": 0.84,+ "n": 100+ }+ },+ "zh-TW": {+ "committed": {+ "n": 100,+ "accuracy": 0.54,+ "macro_f1": 0.4956,+ "ece": 0.327,+ "brier": 0.7166,+ "nll": 2.0914,+ "mean_confidence": 0.83,+ "acc_at_50_coverage": 0.8,+ "seconds": 7.7,+ "dropped": 0+ },+ "refreshed_clamped": {+ "accuracy": 0.61,+ "macro_f1": 0.5565,+ "ece": 0.2662,+ "mean_confidence": 0.8558,+ "acc_at_50_coverage": 0.78,+ "n": 100+ },+ "refreshed_unclamped": {+ "accuracy": 0.61,+ "macro_f1": 0.5565,+ "ece": 0.2662,+ "mean_confidence": 0.8558,+ "acc_at_50_coverage": 0.78,+ "n": 100+ }+ }+ }+ }+ }+}diff --git a/tests/test_doc_tables.py b/tests/test_doc_tables.pyindex 58a48b4d..9b5ca167 100644--- a/tests/test_doc_tables.py+++ b/tests/test_doc_tables.py@@ -195,6 +195,60 @@ def main() -> int:rounded(max(v["d_fast_stock"] for v in fp16_rows)), "0.009")check("fp16 prose/README's 'within 0.009 of fp32'",rounded(max(v["d_fast_fp32"] for v in fp16_rows)), "0.009")+ # ------------------------------------- the 51-language table (#208)+ # The multilingual half of the committed sweep does not reproduce on current code, so the+ # table prints the refreshed re-run instead. That makes `cpu_51_language_sweep_refreshed.json`+ # the artifact behind every cell, and it is checkable here: the `laya` column must still+ # equal the committed file (it reproduces exactly, which is what makes the multilingual+ # divergence interesting rather than a harness change), and the `laya-multilingual` column+ # must equal the refresh.+ REFRESH = os.path.join("research", "results", "cpu_51_language_sweep_refreshed.json")+ if os.path.exists(REFRESH):+ with open(REFRESH, encoding="utf-8") as fh:+ refreshed = json.load(fh)+ committed_sweep = json.loads(read(os.path.join("research", "results",+ "cpu_51_language_sweep.json")))+ en = refreshed["by_model"]["english"]+ ml = refreshed["by_model"]["multilingual"]++ # the refreshed `laya` column is the committed one, per language+ same = [lg for lg in en["per_language"]+ if abs(en["per_language"][lg]["refreshed_clamped"]["accuracy"]+ - committed_sweep["part_a"]["by_model"]["english"]["per_language"][lg]["accuracy"]) > 5e-5]+ check("51-language/laya reproduces the committed file exactly", same, [])++ # ...and the multilingual one does not, which is why the refresh exists+ differ = [lg for lg in ml["per_language"]+ if abs(ml["per_language"][lg]["refreshed_clamped"]["accuracy"]+ - ml["per_language"][lg]["committed"]["accuracy"]) > 5e-5]+ check_true("51-language/multilingual differs from committed in most languages",+ len(differ) > 40, "%d of 51 differ" % len(differ))++ bench = read(BENCHMARKS)+ header = "| lang | laya | laya-multilingual | \u0394 | laya ECE | multilingual ECE |"+ start = bench.find(header)+ check_true("51-language/table found in BENCHMARKS.md", start != -1, "header not found")+ if start != -1:+ body = bench[start:bench.find("</details>", start)]+ rows = [l for l in body.splitlines() if l.startswith("| `")]+ check("51-language/table has one row per language", len(rows), 51)+ mismatched = []+ for row in rows:+ cells = [c.strip() for c in row.strip("|").split("|")]+ lg = cells[0].strip("`")+ e = en["per_language"].get(lg, {}).get("refreshed_clamped")+ m = ml["per_language"].get(lg, {}).get("refreshed_clamped")+ if e is None or m is None:+ mismatched.append("%s: not in the artifact" % lg)+ continue+ want = ["%.3f" % e["accuracy"], "%.3f" % m["accuracy"],+ "%+.3f" % (m["accuracy"] - e["accuracy"]),+ "%.3f" % e["ece"], "%.3f" % m["ece"]]+ if cells[1:] != want:+ mismatched.append("%s: %s != %s" % (lg, cells[1:], want))+ check("51-language/every cell matches the refreshed artifact", mismatched, [])+ else:+ FAIL.append("51-language/%s is missing, so the table is unbacked" % REFRESH)# ------------------------------------------------- what this cannot checkunbacked = [