Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

research: re-run the 51-language sweep in both temperature regimes (the #208 ask) (#222)

PerryLinkfcf1d7fe5d02
diff --git a/BENCHMARKS.md b/BENCHMARKS.md
index 823fccc1..cb5f5101 100644
--- a/BENCHMARKS.md
+++ b/BENCHMARKS.md
@@ -9,7 +9,19 @@ Every checkpoint answered **byte-identical questions** in each run (fixed seed).
| Applications | the seven workflow themes + the datasets where Jev numbers exist, all three checkpoints (laya 0.2.1, CPU, 400 cases per task, seed 13, 2026-09-19) | `research/results/app_benchmark_results.json` |
-**Calibration columns in the CPU sweep predate the temperature clamp.** The 51-language ECE and mean-confidence figures were produced before #42 clamped temperatures to `[0.5, 5]`, so today's package reports different confidence for the affected buckets (`choice:11+` is now served at 0.5, not 0.1006). Accuracy columns are unaffected. A re-run with the current package is tracked in #208.
+**Calibration columns in the CPU sweep predate the temperature clamp.** The 51-language ECE and mean-confidence figures were produced before #42 clamped temperatures to `[0.5, 5]`, so today's package reports different confidence for the affected buckets. Accuracy columns are unaffected, because a temperature-scaled softmax has the same argmax at every positive temperature.
+
+The same 51 languages and 5,100 cases have now been re-run with 0.3.7 in both regimes (`research/results/cpu_51_language_sweep_clamped.json`, [#208](https://github.com/NandhaKishorM/laya/issues/208)). Macro accuracy reproduces at **0.2269** exactly, and macro ECE moves **0.7331 → 0.5709**:
+
+| | committed | re-run, raw temperatures | re-run, as served |
+|---|---|---|---|
+| macro accuracy | 0.2269 | 0.2269 | 0.2269 |
+| macro ECE | 0.7331 | **0.7331** | 0.5709 |
+| macro F1 | 0.2053 | 0.2053 | 0.2053 |
+| mean confidence, `en` | 0.9989 | **0.9989** | 0.9582 |
+| ECE, `en` | 0.1789 | **0.1789** | 0.1382 |
+
+The raw-temperature column reproduces the committed file, so the only variable left is the clamp. `choice:11+` is the sole bucket it moves, and every case in this sweep is a 20-option question, so the clamp applies to all 5,100 — and lowers ECE in all 51 languages. `acc_at_50_coverage` is the one rank-quality column that uses the confidence values: macro 0.3004 → 0.3020, and `en` 0.94 → 0.98, so the flatter distribution selects a slightly better half rather than a worse one.
---
diff --git a/research/README.md b/research/README.md
index b0fcf57e..4afac5b5 100644
--- a/research/README.md
+++ b/research/README.md
@@ -27,6 +27,7 @@ installed its abseil runtime can deadlock model construction on macOS/Python 3.9
|---|---|
| `results/t4_colab_benchmark.json` | 17,416 questions on one T4, both checkpoints, identical questions per model |
| `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options |
+| `results/cpu_51_language_sweep_clamped.json` | the same 51 languages and 5,100 cases re-run with 0.3.7, raw temperatures and served temperatures side by side ([#208](https://github.com/NandhaKishorM/laya/issues/208)) |
## Headline findings
diff --git a/research/eval/README.md b/research/eval/README.md
index 7e011310..6470bbda 100644
--- a/research/eval/README.md
+++ b/research/eval/README.md
@@ -137,8 +137,24 @@ having an empty `temperature_by_options`, so the clamp cannot explain it.
Ruled out: the option sets (identical digest to the english run), the weights
(bundled and standalone multilingual are byte-identical, all 170 tensors
`torch.equal`), the dataset (revision `940fd47a`, last modified 2026-02-24), and
-`build_sequence` (unchanged since `v0.2.0`). It is in the multilingual inference path
-between `laya 0.2.0` and `0.3.6` and is **not** reconciled. Flagged rather than hidden.
+`build_sequence` (unchanged since `v0.2.0`). Also ruled out, on re-measurement:
+
+* **the shipped `head_max_len`**, which matters here because this checkpoint ships
+ `256` and english ships `192`. The harness reads it from the checkpoint's own
+ config and the run's `config` block records `head_max_len: 256, max_len: 1024`, so
+ the multilingual numbers above were not taken at english's budget. Re-running with
+ the value read from config gives the same `0.4008`, and `6/51` again.
+* **which of the two multilingual copies was measured.** The bundled `multilingual/`
+ subfolder and the standalone `convaiinnovations/laya-multilingual` repo were each
+ run end to end over all 51 languages and both give `macro_accuracy 0.4008`,
+ `macro_ece 0.3911`, `6/51`.
+* **a checkpoint change since the committed sweep.** `multilingual/model.safetensors`
+ is `643835514` bytes at `sha256 b99c8bea…` and `multilingual/rl_agent_config.json`
+ is `472` bytes at `sha256 00e35f88…` at every revision from the sweep's timestamp to
+ today; the Hub commits in that window are model-card `docs:`/`assets:` only.
+
+It is in the multilingual inference path between `laya 0.2.0` and `0.3.6` and is
+**not** reconciled. Flagged rather than hidden.
Related: **`head_max_len` is load-bearing for accuracy**, not just for option
truncation. The english checkpoint at its shipped `head_max_len=192` scores 0.82;
@@ -150,7 +166,7 @@ forcing 256 or 512 drops it to 0.79.
checkpoint, no network:
```bash
-python research/eval/test_laya_eval.py # 49 passed, 0 failed
+python research/eval/test_laya_eval.py # 64 passed, 0 failed
```
It pins the upstream constants (seed 13, 20 options, the exact instruction string),
@@ -176,3 +192,25 @@ not just against this harness's own arithmetic.
binned differently. It now matches `laya.common.ece_score`,
`research/scripts/bench_local.py` and `research/scripts/build_benchmark_nb.py`, and
`test_laya_eval.py` asserts that agreement.
+
+### The temperature clamp, measured both ways
+
+`research/results/cpu_51_language_sweep_clamped.json` carries the same re-run twice, once per
+regime, against the committed columns. Macro accuracy reproduces the committed file exactly
+and macro ECE is the only macro figure that moves:
+
+| | committed | `--unclamped` | default |
+|---|---|---|---|
+| `macro_accuracy` | 0.2269 | **0.2269** | 0.2269 |
+| `macro_ece` | 0.7331 | **0.7331** | 0.5709 |
+| `macro_f1` | 0.2053 | **0.2053** | 0.2053 |
+
+Per language, the unclamped run agrees with the committed file on `accuracy` and `macro_f1`
+in **51/51**, on `ece` in **48/51** and on `mean_confidence` in **49/51**. The handful that
+differ do so by `0.0001`, the last stored digit: the committed run used torch 2.8.0 and this
+one 2.14.0. The clamped run differs from the committed file on `ece` and `mean_confidence` in
+**51/51**, every one of them lower, because it is the only column the clamp can move.
+
+`accuracy`, `macro_f1` and `n` are identical in all three columns by construction: scaling
+logits by any positive temperature does not change the argmax. That is why a re-run can settle
+the calibration question without reopening the accuracy numbers.
diff --git a/research/eval/test_laya_eval.py b/research/eval/test_laya_eval.py
index 2e730a32..ccdecd87 100644
--- a/research/eval/test_laya_eval.py
+++ b/research/eval/test_laya_eval.py
@@ -160,6 +160,90 @@ check("const/instructions match bench_local.py",
INSTRUCTIONS, "What is the user asking for in `utterance`?")
+# ------------------------------------------- the #208 before/after re-run file
+# research/results/cpu_51_language_sweep_clamped.json records the committed sweep,
+# the pre-clamp re-run and the served-temperature re-run side by side. It is only
+# useful if it still agrees with the committed file, so that agreement is a test.
+import json # noqa: E402
+
+_RESULTS = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))),
+ "research", "results")
+
+
+def _load(name):
+ with open(os.path.join(_RESULTS, name), encoding="utf-8") as fh:
+ return json.load(fh)
+
+
+_rerun = _load("cpu_51_language_sweep_clamped.json")
+_sweep = _load("cpu_51_language_sweep.json")["part_a"]["by_model"]["english"]
+_langs = _sweep["per_language"]
+
+# Both files round each per-language figure to 4 decimals (bench_local.py:138-144), so
+# agreement has to be judged at that resolution: two files can disagree by 1 in the
+# last stored digit for reasons that have nothing to do with the temperatures, and
+# three languages do (am, el, ro) because the committed run used torch 2.8.0 and this
+# one 2.14.0. The macro figures match exactly, which is why that is the headline.
+_QUANT = 1.5e-4
+
+
+def _close(a, b, tol=_QUANT):
+ return abs(a - b) <= tol
+
+check("208/one entry per language", len(_rerun["per_language"]), len(_langs))
+check("208/case count is languages x per_lang",
+ _rerun["config"]["n_cases"],
+ _rerun["config"]["languages"] * _rerun["config"]["per_lang"])
+check("208/macro accuracy copied from the committed sweep",
+ _rerun["macro"]["committed"]["accuracy"], _sweep["macro_accuracy"])
+check("208/macro ece copied from the committed sweep",
+ _rerun["macro"]["committed"]["ece"], _sweep["macro_ece"])
+
+# accuracy is argmax of a temperature-scaled softmax, so it cannot move with T
+check_true("208/accuracy identical in all three regimes",
+ all(len({_rerun["per_language"][lg][r]["accuracy"] for r in
+ ("committed", "unclamped_rerun", "clamped_rerun")}) == 1
+ for lg in _langs))
+check_true("208/macro_f1 identical in all three regimes",
+ all(len({_rerun["per_language"][lg][r]["macro_f1"] for r in
+ ("committed", "unclamped_rerun", "clamped_rerun")}) == 1
+ for lg in _langs))
+
+# the committed calibration columns must match the unclamped re-run, and only the
+# clamped re-run may differ -- that is the whole claim
+check_true("208/ece matches the committed file in the unclamped re-run",
+ all(_close(_rerun["per_language"][lg]["unclamped_rerun"]["ece"],
+ _langs[lg]["ece"]) for lg in _langs))
+check_true("208/ece differs from the committed file in the clamped re-run",
+ all(not _close(_rerun["per_language"][lg]["clamped_rerun"]["ece"],
+ _langs[lg]["ece"]) for lg in _langs))
+check_true("208/mean_confidence matches in the unclamped re-run",
+ all(_close(_rerun["per_language"][lg]["unclamped_rerun"]["mean_confidence"],
+ _langs[lg]["mean_confidence"]) for lg in _langs))
+check_true("208/mean_confidence differs in the clamped re-run",
+ all(not _close(_rerun["per_language"][lg]["clamped_rerun"]["mean_confidence"],
+ _langs[lg]["mean_confidence"]) for lg in _langs))
+
+# the unclamped re-run is a reproduction, not an approximation: name the tolerance
+# so a future change that widens it has to say so
+check_true("208/unclamped reproduction agrees in at least 48 of 51 languages",
+ sum(1 for lg in _langs
+ if _close(_rerun["per_language"][lg]["unclamped_rerun"]["ece"], _langs[lg]["ece"])) >= 48)
+
+# the clamp can lower ECE everywhere without lowering rank quality, so this is a
+# guard against the misleading "the clamp makes it worse" reading
+check_true("208/the clamp lowers macro ece",
+ _rerun["macro"]["clamped_rerun"]["macro_ece"]
+ < _rerun["macro"]["unclamped_rerun"]["macro_ece"])
+check_true("208/every per-language delta is reported",
+ all("delta_ece" in v and "delta_mean_confidence" in v
+ for v in _rerun["per_language"].values()))
+check("208/only choice:11+ is the clamped bucket",
+ _rerun["temperature_choice_11_plus"]["unclamped_rerun"], 0.10058280825614929)
+check("208/the served bucket is 0.5",
+ _rerun["temperature_choice_11_plus"]["clamped_rerun"], 0.5)
+
+
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
print(" FAIL " + f)
diff --git a/research/results/cpu_51_language_sweep_clamped.json b/research/results/cpu_51_language_sweep_clamped.json
new file mode 100644
index 00000000..40d60c50
--- /dev/null
+++ b/research/results/cpu_51_language_sweep_clamped.json
@@ -0,0 +1,1433 @@
+{
+ "why": "#208: the committed 51-language sweep's `ece` and `mean_confidence` columns were produced before #42 clamped temperatures to [0.5, 5]. This file re-runs the same 51 languages and the same 5,100 cases with the current package, in both regimes, so the two can be compared like for like. `accuracy` and `macro_f1` are unchanged, because a temperature-scaled softmax has the same argmax for every positive T.",
+ "how": [
+ "research/eval/laya_eval.py --model convaiinnovations/laya --langs all --per-lang 100 --n-opts 20 --device cpu --unclamped",
+ "research/eval/laya_eval.py --model convaiinnovations/laya --langs all --per-lang 100 --n-opts 20 --device cpu"
+ ],
+ "config": {
+ "model": "convaiinnovations/laya",
+ "subfolder": null,
+ "dataset": "mteb/amazon_massive_intent",
+ "split": "test",
+ "languages": 51,
+ "per_lang": 100,
+ "n_opts": 20,
+ "seed": 13,
+ "device": "cpu",
+ "head_max_len": 192,
+ "max_len": 512,
+ "instructions": "What is the user asking for in `utterance`?",
+ "laya_version": "0.3.7",
+ "n_cases": 5100
+ },
+ "temperature_choice_11_plus": {
+ "committed_sweep": "0.2.0",
+ "unclamped_rerun": 0.10058280825614929,
+ "clamped_rerun": 0.5,
+ "note": "a 20-option question is bucket `choice:11+`; it is the only bucket the clamp moves, and every case in this sweep is a 20-option question"
+ },
+ "macro": {
+ "committed": {
+ "accuracy": 0.2269,
+ "ece": 0.7331,
+ "languages": 51
+ },
+ "unclamped_rerun": {
+ "languages": 51,
+ "macro_accuracy": 0.2269,
+ "macro_ece": 0.7331,
+ "macro_f1": 0.2053,
+ "seconds": 1444.9
+ },
+ "clamped_rerun": {
+ "languages": 51,
+ "macro_accuracy": 0.2269,
+ "macro_ece": 0.5709,
+ "macro_f1": 0.2053,
+ "seconds": 1360.1
+ }
+ },
+ "per_language": {
+ "af": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.29,
+ "macro_f1": 0.2874,
+ "ece": 0.6873,
+ "mean_confidence": 0.9752,
+ "acc_at_50_coverage": 0.42
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.29,
+ "macro_f1": 0.2874,
+ "ece": 0.6873,
+ "mean_confidence": 0.9752,
+ "acc_at_50_coverage": 0.42
+ },
+ "clamped_rerun": {
+ "accuracy": 0.29,
+ "macro_f1": 0.2874,
+ "ece": 0.5588,
+ "mean_confidence": 0.8488,
+ "acc_at_50_coverage": 0.42
+ },
+ "delta_ece": -0.1285,
+ "delta_mean_confidence": -0.1264
+ },
+ "am": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.12,
+ "macro_f1": 0.1122,
+ "ece": 0.8252,
+ "mean_confidence": 0.9452,
+ "acc_at_50_coverage": 0.14
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.12,
+ "macro_f1": 0.1122,
+ "ece": 0.8253,
+ "mean_confidence": 0.9453,
+ "acc_at_50_coverage": 0.14
+ },
+ "clamped_rerun": {
+ "accuracy": 0.12,
+ "macro_f1": 0.1122,
+ "ece": 0.6804,
+ "mean_confidence": 0.7873,
+ "acc_at_50_coverage": 0.12
+ },
+ "delta_ece": -0.1448,
+ "delta_mean_confidence": -0.1579
+ },
+ "ar": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.11,
+ "macro_f1": 0.093,
+ "ece": 0.8001,
+ "mean_confidence": 0.8976,
+ "acc_at_50_coverage": 0.12
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.11,
+ "macro_f1": 0.093,
+ "ece": 0.8001,
+ "mean_confidence": 0.8976,
+ "acc_at_50_coverage": 0.12
+ },
+ "clamped_rerun": {
+ "accuracy": 0.11,
+ "macro_f1": 0.093,
+ "ece": 0.5795,
+ "mean_confidence": 0.6833,
+ "acc_at_50_coverage": 0.12
+ },
+ "delta_ece": -0.2206,
+ "delta_mean_confidence": -0.2143
+ },
+ "az": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.1,
+ "macro_f1": 0.0629,
+ "ece": 0.8247,
+ "mean_confidence": 0.9247,
+ "acc_at_50_coverage": 0.14
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.1,
+ "macro_f1": 0.0629,
+ "ece": 0.8247,
+ "mean_confidence": 0.9247,
+ "acc_at_50_coverage": 0.14
+ },
+ "clamped_rerun": {
+ "accuracy": 0.1,
+ "macro_f1": 0.0629,
+ "ece": 0.5983,
+ "mean_confidence": 0.6983,
+ "acc_at_50_coverage": 0.14
+ },
+ "delta_ece": -0.2264,
+ "delta_mean_confidence": -0.2264
+ },
+ "bn": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.08,
+ "macro_f1": 0.056,
+ "ece": 0.8646,
+ "mean_confidence": 0.9446,
+ "acc_at_50_coverage": 0.1
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.08,
+ "macro_f1": 0.056,
+ "ece": 0.8646,
+ "mean_confidence": 0.9446,
+ "acc_at_50_coverage": 0.1
+ },
+ "clamped_rerun": {
+ "accuracy": 0.08,
+ "macro_f1": 0.056,
+ "ece": 0.6267,
+ "mean_confidence": 0.7067,
+ "acc_at_50_coverage": 0.12
+ },
+ "delta_ece": -0.2379,
+ "delta_mean_confidence": -0.2379
+ },
+ "cy": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.12,
+ "macro_f1": 0.0906,
+ "ece": 0.8409,
+ "mean_confidence": 0.9609,
+ "acc_at_50_coverage": 0.14
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.12,
+ "macro_f1": 0.0906,
+ "ece": 0.8409,
+ "mean_confidence": 0.9609,
+ "acc_at_50_coverage": 0.14
+ },
+ "clamped_rerun": {
+ "accuracy": 0.12,
+ "macro_f1": 0.0906,
+ "ece": 0.6452,
+ "mean_confidence": 0.7652,
+ "acc_at_50_coverage": 0.14
+ },
+ "delta_ece": -0.1957,
+ "delta_mean_confidence": -0.1957
+ },
+ "da": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.35,
+ "macro_f1": 0.3082,
+ "ece": 0.6263,
+ "mean_confidence": 0.9763,
+ "acc_at_50_coverage": 0.52
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.35,
+ "macro_f1": 0.3082,
+ "ece": 0.6263,
+ "mean_confidence": 0.9763,
+ "acc_at_50_coverage": 0.52
+ },
+ "clamped_rerun": {
+ "accuracy": 0.35,
+ "macro_f1": 0.3082,
+ "ece": 0.526,
+ "mean_confidence": 0.876,
+ "acc_at_50_coverage": 0.52
+ },
+ "delta_ece": -0.1003,
+ "delta_mean_confidence": -0.1003
+ },
+ "de": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.42,
+ "macro_f1": 0.387,
+ "ece": 0.5584,
+ "mean_confidence": 0.9784,
+ "acc_at_50_coverage": 0.66
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.42,
+ "macro_f1": 0.387,
+ "ece": 0.5584,
+ "mean_confidence": 0.9784,
+ "acc_at_50_coverage": 0.66
+ },
+ "clamped_rerun": {
+ "accuracy": 0.42,
+ "macro_f1": 0.387,
+ "ece": 0.452,
+ "mean_confidence": 0.872,
+ "acc_at_50_coverage": 0.68
+ },
+ "delta_ece": -0.1064,
+ "delta_mean_confidence": -0.1064
+ },
+ "el": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.13,
+ "macro_f1": 0.1,
+ "ece": 0.8387,
+ "mean_confidence": 0.9687,
+ "acc_at_50_coverage": 0.16
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.13,
+ "macro_f1": 0.1,
+ "ece": 0.8388,
+ "mean_confidence": 0.9688,
+ "acc_at_50_coverage": 0.16
+ },
+ "clamped_rerun": {
+ "accuracy": 0.13,
+ "macro_f1": 0.1,
+ "ece": 0.6789,
+ "mean_confidence": 0.8089,
+ "acc_at_50_coverage": 0.2
+ },
+ "delta_ece": -0.1598,
+ "delta_mean_confidence": -0.1598
+ },
+ "en": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.82,
+ "macro_f1": 0.7876,
+ "ece": 0.1789,
+ "mean_confidence": 0.9989,
+ "acc_at_50_coverage": 0.94
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.82,
+ "macro_f1": 0.7876,
+ "ece": 0.1789,
+ "mean_confidence": 0.9989,
+ "acc_at_50_coverage": 0.94
+ },
+ "clamped_rerun": {
+ "accuracy": 0.82,
+ "macro_f1": 0.7876,
+ "ece": 0.1382,
+ "mean_confidence": 0.9582,
+ "acc_at_50_coverage": 0.98
+ },
+ "delta_ece": -0.0407,
+ "delta_mean_confidence": -0.0407
+ },
+ "es": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.51,
+ "macro_f1": 0.4767,
+ "ece": 0.4796,
+ "mean_confidence": 0.9896,
+ "acc_at_50_coverage": 0.6
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.51,
+ "macro_f1": 0.4767,
+ "ece": 0.4796,
+ "mean_confidence": 0.9896,
+ "acc_at_50_coverage": 0.6
+ },
+ "clamped_rerun": {
+ "accuracy": 0.51,
+ "macro_f1": 0.4767,
+ "ece": 0.4225,
+ "mean_confidence": 0.9245,
+ "acc_at_50_coverage": 0.62
+ },
+ "delta_ece": -0.0571,
+ "delta_mean_confidence": -0.0651
+ },
+ "fa": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.14,
+ "macro_f1": 0.1197,
+ "ece": 0.8196,
+ "mean_confidence": 0.9414,
+ "acc_at_50_coverage": 0.16
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.14,
+ "macro_f1": 0.1197,
+ "ece": 0.8196,
+ "mean_confidence": 0.9414,
+ "acc_at_50_coverage": 0.16
+ },
+ "clamped_rerun": {
+ "accuracy": 0.14,
+ "macro_f1": 0.1197,
+ "ece": 0.6331,
+ "mean_confidence": 0.7731,
+ "acc_at_50_coverage": 0.16
+ },
+ "delta_ece": -0.1865,
+ "delta_mean_confidence": -0.1683
+ },
+ "fi": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.13,
+ "macro_f1": 0.1213,
+ "ece": 0.8486,
+ "mean_confidence": 0.9786,
+ "acc_at_50_coverage": 0.16
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.13,
+ "macro_f1": 0.1213,
+ "ece": 0.8486,
+ "mean_confidence": 0.9786,
+ "acc_at_50_coverage": 0.16
+ },
+ "clamped_rerun": {
+ "accuracy": 0.13,
+ "macro_f1": 0.1213,
+ "ece": 0.7189,
+ "mean_confidence": 0.8489,
+ "acc_at_50_coverage": 0.16
+ },
+ "delta_ece": -0.1297,
+ "delta_mean_confidence": -0.1297
+ },
+ "fr": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.59,
+ "macro_f1": 0.5822,
+ "ece": 0.3882,
+ "mean_confidence": 0.9782,
+ "acc_at_50_coverage": 0.78
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.59,
+ "macro_f1": 0.5822,
+ "ece": 0.3882,
+ "mean_confidence": 0.9782,
+ "acc_at_50_coverage": 0.78
+ },
+ "clamped_rerun": {
+ "accuracy": 0.59,
+ "macro_f1": 0.5822,
+ "ece": 0.2995,
+ "mean_confidence": 0.8895,
+ "acc_at_50_coverage": 0.78
+ },
+ "delta_ece": -0.0887,
+ "delta_mean_confidence": -0.0887
+ },
+ "he": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.06,
+ "macro_f1": 0.039,
+ "ece": 0.9111,
+ "mean_confidence": 0.9643,
+ "acc_at_50_coverage": 0.04
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.06,
+ "macro_f1": 0.039,
+ "ece": 0.9111,
+ "mean_confidence": 0.9643,
+ "acc_at_50_coverage": 0.04
+ },
+ "clamped_rerun": {
+ "accuracy": 0.06,
+ "macro_f1": 0.039,
+ "ece": 0.7379,
+ "mean_confidence": 0.7842,
+ "acc_at_50_coverage": 0.02
+ },
+ "delta_ece": -0.1732,
+ "delta_mean_confidence": -0.1801
+ },
+ "hi": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.1,
+ "macro_f1": 0.0758,
+ "ece": 0.8504,
+ "mean_confidence": 0.9408,
+ "acc_at_50_coverage": 0.1
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.1,
+ "macro_f1": 0.0758,
+ "ece": 0.8504,
+ "mean_confidence": 0.9408,
+ "acc_at_50_coverage": 0.1
+ },
+ "clamped_rerun": {
+ "accuracy": 0.1,
+ "macro_f1": 0.0758,
+ "ece": 0.6416,
+ "mean_confidence": 0.7416,
+ "acc_at_50_coverage": 0.08
+ },
+ "delta_ece": -0.2088,
+ "delta_mean_confidence": -0.1992
+ },
+ "hu": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.09,
+ "macro_f1": 0.0813,
+ "ece": 0.8571,
+ "mean_confidence": 0.9471,
+ "acc_at_50_coverage": 0.06
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.09,
+ "macro_f1": 0.0813,
+ "ece": 0.8571,
+ "mean_confidence": 0.9471,
+ "acc_at_50_coverage": 0.06
+ },
+ "clamped_rerun": {
+ "accuracy": 0.09,
+ "macro_f1": 0.0813,
+ "ece": 0.6783,
+ "mean_confidence": 0.7683,
+ "acc_at_50_coverage": 0.06
+ },
+ "delta_ece": -0.1788,
+ "delta_mean_confidence": -0.1788
+ },
+ "hy": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.05,
+ "macro_f1": 0.0355,
+ "ece": 0.8348,
+ "mean_confidence": 0.8848,
+ "acc_at_50_coverage": 0.08
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.05,
+ "macro_f1": 0.0355,
+ "ece": 0.8348,
+ "mean_confidence": 0.8848,
+ "acc_at_50_coverage": 0.08
+ },
+ "clamped_rerun": {
+ "accuracy": 0.05,
+ "macro_f1": 0.0355,
+ "ece": 0.5709,
+ "mean_confidence": 0.6209,
+ "acc_at_50_coverage": 0.04
+ },
+ "delta_ece": -0.2639,
+ "delta_mean_confidence": -0.2639
+ },
+ "id": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.36,
+ "macro_f1": 0.3283,
+ "ece": 0.6133,
+ "mean_confidence": 0.9619,
+ "acc_at_50_coverage": 0.48
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.36,
+ "macro_f1": 0.3283,
+ "ece": 0.6133,
+ "mean_confidence": 0.9619,
+ "acc_at_50_coverage": 0.48
+ },
+ "clamped_rerun": {
+ "accuracy": 0.36,
+ "macro_f1": 0.3283,
+ "ece": 0.487,
+ "mean_confidence": 0.8304,
+ "acc_at_50_coverage": 0.48
+ },
+ "delta_ece": -0.1263,
+ "delta_mean_confidence": -0.1315
+ },
+ "is": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.11,
+ "macro_f1": 0.0893,
+ "ece": 0.8351,
+ "mean_confidence": 0.9451,
+ "acc_at_50_coverage": 0.16
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.11,
+ "macro_f1": 0.0893,
+ "ece": 0.8351,
+ "mean_confidence": 0.9451,
+ "acc_at_50_coverage": 0.16
+ },
+ "clamped_rerun": {
+ "accuracy": 0.11,
+ "macro_f1": 0.0893,
+ "ece": 0.6641,
+ "mean_confidence": 0.7741,
+ "acc_at_50_coverage": 0.16
+ },
+ "delta_ece": -0.171,
+ "delta_mean_confidence": -0.171
+ },
+ "it": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.34,
+ "macro_f1": 0.3101,
+ "ece": 0.6474,
+ "mean_confidence": 0.9735,
+ "acc_at_50_coverage": 0.52
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.34,
+ "macro_f1": 0.3101,
+ "ece": 0.6474,
+ "mean_confidence": 0.9735,
+ "acc_at_50_coverage": 0.52
+ },
+ "clamped_rerun": {
+ "accuracy": 0.34,
+ "macro_f1": 0.3101,
+ "ece": 0.5156,
+ "mean_confidence": 0.8534,
+ "acc_at_50_coverage": 0.52
+ },
+ "delta_ece": -0.1318,
+ "delta_mean_confidence": -0.1201
+ },
+ "ja": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.53,
+ "macro_f1": 0.4942,
+ "ece": 0.4599,
+ "mean_confidence": 0.9801,
+ "acc_at_50_coverage": 0.76
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.53,
+ "macro_f1": 0.4942,
+ "ece": 0.4599,
+ "mean_confidence": 0.9801,
+ "acc_at_50_coverage": 0.76
+ },
+ "clamped_rerun": {
+ "accuracy": 0.53,
+ "macro_f1": 0.4942,
+ "ece": 0.3657,
+ "mean_confidence": 0.8957,
+ "acc_at_50_coverage": 0.76
+ },
+ "delta_ece": -0.0942,
+ "delta_mean_confidence": -0.0844
+ },
+ "jv": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.16,
+ "macro_f1": 0.1537,
+ "ece": 0.8035,
+ "mean_confidence": 0.9635,
+ "acc_at_50_coverage": 0.14
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.16,
+ "macro_f1": 0.1537,
+ "ece": 0.8035,
+ "mean_confidence": 0.9635,
+ "acc_at_50_coverage": 0.14
+ },
+ "clamped_rerun": {
+ "accuracy": 0.16,
+ "macro_f1": 0.1537,
+ "ece": 0.67,
+ "mean_confidence": 0.83,
+ "acc_at_50_coverage": 0.16
+ },
+ "delta_ece": -0.1335,
+ "delta_mean_confidence": -0.1335
+ },
+ "ka": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.09,
+ "macro_f1": 0.0742,
+ "ece": 0.8448,
+ "mean_confidence": 0.9348,
+ "acc_at_50_coverage": 0.1
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.09,
+ "macro_f1": 0.0742,
+ "ece": 0.8448,
+ "mean_confidence": 0.9348,
+ "acc_at_50_coverage": 0.1
+ },
+ "clamped_rerun": {
+ "accuracy": 0.09,
+ "macro_f1": 0.0742,
+ "ece": 0.6102,
+ "mean_confidence": 0.7002,
+ "acc_at_50_coverage": 0.1
+ },
+ "delta_ece": -0.2346,
+ "delta_mean_confidence": -0.2346
+ },
+ "km": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.0,
+ "macro_f1": 0.0,
+ "ece": 0.9515,
+ "mean_confidence": 0.9515,
+ "acc_at_50_coverage": 0.0
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.0,
+ "macro_f1": 0.0,
+ "ece": 0.9515,
+ "mean_confidence": 0.9515,
+ "acc_at_50_coverage": 0.0
+ },
+ "clamped_rerun": {
+ "accuracy": 0.0,
+ "macro_f1": 0.0,
+ "ece": 0.7051,
+ "mean_confidence": 0.7051,
+ "acc_at_50_coverage": 0.0
+ },
+ "delta_ece": -0.2464,
+ "delta_mean_confidence": -0.2464
+ },
+ "kn": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.11,
+ "macro_f1": 0.0913,
+ "ece": 0.8416,
+ "mean_confidence": 0.9421,
+ "acc_at_50_coverage": 0.06
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.11,
+ "macro_f1": 0.0913,
+ "ece": 0.8416,
+ "mean_confidence": 0.9421,
+ "acc_at_50_coverage": 0.06
+ },
+ "clamped_rerun": {
+ "accuracy": 0.11,
+ "macro_f1": 0.0913,
+ "ece": 0.5903,
+ "mean_confidence": 0.695,
+ "acc_at_50_coverage": 0.08
+ },
+ "delta_ece": -0.2513,
+ "delta_mean_confidence": -0.2471
+ },
+ "ko": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.11,
+ "macro_f1": 0.0943,
+ "ece": 0.8498,
+ "mean_confidence": 0.9598,
+ "acc_at_50_coverage": 0.14
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.11,
+ "macro_f1": 0.0943,
+ "ece": 0.8498,
+ "mean_confidence": 0.9598,
+ "acc_at_50_coverage": 0.14
+ },
+ "clamped_rerun": {
+ "accuracy": 0.11,
+ "macro_f1": 0.0943,
+ "ece": 0.6804,
+ "mean_confidence": 0.7904,
+ "acc_at_50_coverage": 0.14
+ },
+ "delta_ece": -0.1694,
+ "delta_mean_confidence": -0.1694
+ },
+ "lv": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.1,
+ "macro_f1": 0.0786,
+ "ece": 0.8466,
+ "mean_confidence": 0.9466,
+ "acc_at_50_coverage": 0.1
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.1,
+ "macro_f1": 0.0786,
+ "ece": 0.8466,
+ "mean_confidence": 0.9466,
+ "acc_at_50_coverage": 0.1
+ },
+ "clamped_rerun": {
+ "accuracy": 0.1,
+ "macro_f1": 0.0786,
+ "ece": 0.6659,
+ "mean_confidence": 0.7659,
+ "acc_at_50_coverage": 0.1
+ },
+ "delta_ece": -0.1807,
+ "delta_mean_confidence": -0.1807
+ },
+ "ml": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.07,
+ "macro_f1": 0.0503,
+ "ece": 0.8572,
+ "mean_confidence": 0.9272,
+ "acc_at_50_coverage": 0.1
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.07,
+ "macro_f1": 0.0503,
+ "ece": 0.8572,
+ "mean_confidence": 0.9272,
+ "acc_at_50_coverage": 0.1
+ },
+ "clamped_rerun": {
+ "accuracy": 0.07,
+ "macro_f1": 0.0503,
+ "ece": 0.601,
+ "mean_confidence": 0.671,
+ "acc_at_50_coverage": 0.1
+ },
+ "delta_ece": -0.2562,
+ "delta_mean_confidence": -0.2562
+ },
+ "mn": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.13,
+ "macro_f1": 0.1213,
+ "ece": 0.837,
+ "mean_confidence": 0.9591,
+ "acc_at_50_coverage": 0.14
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.13,
+ "macro_f1": 0.1213,
+ "ece": 0.837,
+ "mean_confidence": 0.9591,
+ "acc_at_50_coverage": 0.14
+ },
+ "clamped_rerun": {
+ "accuracy": 0.13,
+ "macro_f1": 0.1213,
+ "ece": 0.6704,
+ "mean_confidence": 0.7977,
+ "acc_at_50_coverage": 0.14
+ },
+ "delta_ece": -0.1666,
+ "delta_mean_confidence": -0.1614
+ },
+ "ms": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.27,
+ "macro_f1": 0.2372,
+ "ece": 0.6877,
+ "mean_confidence": 0.9577,
+ "acc_at_50_coverage": 0.42
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.27,
+ "macro_f1": 0.2372,
+ "ece": 0.6877,
+ "mean_confidence": 0.9577,
+ "acc_at_50_coverage": 0.42
+ },
+ "clamped_rerun": {
+ "accuracy": 0.27,
+ "macro_f1": 0.2372,
+ "ece": 0.5064,
+ "mean_confidence": 0.7764,
+ "acc_at_50_coverage": 0.4
+ },
+ "delta_ece": -0.1813,
+ "delta_mean_confidence": -0.1813
+ },
+ "my": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.06,
+ "macro_f1": 0.0455,
+ "ece": 0.8605,
+ "mean_confidence": 0.9205,
+ "acc_at_50_coverage": 0.08
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.06,
+ "macro_f1": 0.0455,
+ "ece": 0.8605,
+ "mean_confidence": 0.9205,
+ "acc_at_50_coverage": 0.08
+ },
+ "clamped_rerun": {
+ "accuracy": 0.06,
+ "macro_f1": 0.0455,
+ "ece": 0.5797,
+ "mean_confidence": 0.6397,
+ "acc_at_50_coverage": 0.08
+ },
+ "delta_ece": -0.2808,
+ "delta_mean_confidence": -0.2808
+ },
+ "nb": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.33,
+ "macro_f1": 0.3025,
+ "ece": 0.6484,
+ "mean_confidence": 0.9692,
+ "acc_at_50_coverage": 0.5
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.33,
+ "macro_f1": 0.3025,
+ "ece": 0.6484,
+ "mean_confidence": 0.9692,
+ "acc_at_50_coverage": 0.5
+ },
+ "clamped_rerun": {
+ "accuracy": 0.33,
+ "macro_f1": 0.3025,
+ "ece": 0.5179,
+ "mean_confidence": 0.8479,
+ "acc_at_50_coverage": 0.52
+ },
+ "delta_ece": -0.1305,
+ "delta_mean_confidence": -0.1213
+ },
+ "nl": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.39,
+ "macro_f1": 0.3968,
+ "ece": 0.5914,
+ "mean_confidence": 0.9745,
+ "acc_at_50_coverage": 0.54
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.39,
+ "macro_f1": 0.3968,
+ "ece": 0.5914,
+ "mean_confidence": 0.9745,
+ "acc_at_50_coverage": 0.54
+ },
+ "clamped_rerun": {
+ "accuracy": 0.39,
+ "macro_f1": 0.3968,
+ "ece": 0.5025,
+ "mean_confidence": 0.8831,
+ "acc_at_50_coverage": 0.56
+ },
+ "delta_ece": -0.0889,
+ "delta_mean_confidence": -0.0914
+ },
+ "pl": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.24,
+ "macro_f1": 0.2033,
+ "ece": 0.7129,
+ "mean_confidence": 0.9529,
+ "acc_at_50_coverage": 0.34
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.24,
+ "macro_f1": 0.2033,
+ "ece": 0.7129,
+ "mean_confidence": 0.9529,
+ "acc_at_50_coverage": 0.34
+ },
+ "clamped_rerun": {
+ "accuracy": 0.24,
+ "macro_f1": 0.2033,
+ "ece": 0.5695,
+ "mean_confidence": 0.7951,
+ "acc_at_50_coverage": 0.36
+ },
+ "delta_ece": -0.1434,
+ "delta_mean_confidence": -0.1578
+ },
+ "pt": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.47,
+ "macro_f1": 0.4213,
+ "ece": 0.5116,
+ "mean_confidence": 0.9718,
+ "acc_at_50_coverage": 0.66
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.47,
+ "macro_f1": 0.4213,
+ "ece": 0.5116,
+ "mean_confidence": 0.9718,
+ "acc_at_50_coverage": 0.66
+ },
+ "clamped_rerun": {
+ "accuracy": 0.47,
+ "macro_f1": 0.4213,
+ "ece": 0.4317,
+ "mean_confidence": 0.8894,
+ "acc_at_50_coverage": 0.66
+ },
+ "delta_ece": -0.0799,
+ "delta_mean_confidence": -0.0824
+ },
+ "ro": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.33,
+ "macro_f1": 0.2976,
+ "ece": 0.6577,
+ "mean_confidence": 0.9824,
+ "acc_at_50_coverage": 0.48
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.33,
+ "macro_f1": 0.2976,
+ "ece": 0.6576,
+ "mean_confidence": 0.9824,
+ "acc_at_50_coverage": 0.48
+ },
+ "clamped_rerun": {
+ "accuracy": 0.33,
+ "macro_f1": 0.2976,
+ "ece": 0.5499,
+ "mean_confidence": 0.8668,
+ "acc_at_50_coverage": 0.48
+ },
+ "delta_ece": -0.1078,
+ "delta_mean_confidence": -0.1156
+ },
+ "ru": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.31,
+ "macro_f1": 0.3084,
+ "ece": 0.668,
+ "mean_confidence": 0.978,
+ "acc_at_50_coverage": 0.42
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.31,
+ "macro_f1": 0.3084,
+ "ece": 0.668,
+ "mean_confidence": 0.978,
+ "acc_at_50_coverage": 0.42
+ },
+ "clamped_rerun": {
+ "accuracy": 0.31,
+ "macro_f1": 0.3084,
+ "ece": 0.5946,
+ "mean_confidence": 0.9046,
+ "acc_at_50_coverage": 0.42
+ },
+ "delta_ece": -0.0734,
+ "delta_mean_confidence": -0.0734
+ },
+ "sl": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.2,
+ "macro_f1": 0.1679,
+ "ece": 0.7561,
+ "mean_confidence": 0.9561,
+ "acc_at_50_coverage": 0.26
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.2,
+ "macro_f1": 0.1679,
+ "ece": 0.7561,
+ "mean_confidence": 0.9561,
+ "acc_at_50_coverage": 0.26
+ },
+ "clamped_rerun": {
+ "accuracy": 0.2,
+ "macro_f1": 0.1679,
+ "ece": 0.5994,
+ "mean_confidence": 0.7994,
+ "acc_at_50_coverage": 0.26
+ },
+ "delta_ece": -0.1567,
+ "delta_mean_confidence": -0.1567
+ },
+ "sq": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.21,
+ "macro_f1": 0.1667,
+ "ece": 0.7549,
+ "mean_confidence": 0.9649,
+ "acc_at_50_coverage": 0.28
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.21,
+ "macro_f1": 0.1667,
+ "ece": 0.7549,
+ "mean_confidence": 0.9649,
+ "acc_at_50_coverage": 0.28
+ },
+ "clamped_rerun": {
+ "accuracy": 0.21,
+ "macro_f1": 0.1667,
+ "ece": 0.6031,
+ "mean_confidence": 0.8131,
+ "acc_at_50_coverage": 0.28
+ },
+ "delta_ece": -0.1518,
+ "delta_mean_confidence": -0.1518
+ },
+ "sv": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.38,
+ "macro_f1": 0.3809,
+ "ece": 0.5956,
+ "mean_confidence": 0.9706,
+ "acc_at_50_coverage": 0.5
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.38,
+ "macro_f1": 0.3809,
+ "ece": 0.5956,
+ "mean_confidence": 0.9706,
+ "acc_at_50_coverage": 0.5
+ },
+ "clamped_rerun": {
+ "accuracy": 0.38,
+ "macro_f1": 0.3809,
+ "ece": 0.4874,
+ "mean_confidence": 0.8501,
+ "acc_at_50_coverage": 0.5
+ },
+ "delta_ece": -0.1082,
+ "delta_mean_confidence": -0.1205
+ },
+ "sw": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.13,
+ "macro_f1": 0.1149,
+ "ece": 0.8282,
+ "mean_confidence": 0.9562,
+ "acc_at_50_coverage": 0.14
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.13,
+ "macro_f1": 0.1149,
+ "ece": 0.8282,
+ "mean_confidence": 0.9562,
+ "acc_at_50_coverage": 0.14
+ },
+ "clamped_rerun": {
+ "accuracy": 0.13,
+ "macro_f1": 0.1149,
+ "ece": 0.6318,
+ "mean_confidence": 0.7511,
+ "acc_at_50_coverage": 0.14
+ },
+ "delta_ece": -0.1964,
+ "delta_mean_confidence": -0.2051
+ },
+ "ta": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.12,
+ "macro_f1": 0.0971,
+ "ece": 0.8224,
+ "mean_confidence": 0.9424,
+ "acc_at_50_coverage": 0.12
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.12,
+ "macro_f1": 0.0971,
+ "ece": 0.8224,
+ "mean_confidence": 0.9424,
+ "acc_at_50_coverage": 0.12
+ },
+ "clamped_rerun": {
+ "accuracy": 0.12,
+ "macro_f1": 0.0971,
+ "ece": 0.5541,
+ "mean_confidence": 0.6741,
+ "acc_at_50_coverage": 0.12
+ },
+ "delta_ece": -0.2683,
+ "delta_mean_confidence": -0.2683
+ },
+ "te": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.09,
+ "macro_f1": 0.0722,
+ "ece": 0.8575,
+ "mean_confidence": 0.9475,
+ "acc_at_50_coverage": 0.06
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.09,
+ "macro_f1": 0.0722,
+ "ece": 0.8575,
+ "mean_confidence": 0.9475,
+ "acc_at_50_coverage": 0.06
+ },
+ "clamped_rerun": {
+ "accuracy": 0.09,
+ "macro_f1": 0.0722,
+ "ece": 0.6689,
+ "mean_confidence": 0.7568,
+ "acc_at_50_coverage": 0.06
+ },
+ "delta_ece": -0.1886,
+ "delta_mean_confidence": -0.1907
+ },
+ "th": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.08,
+ "macro_f1": 0.048,
+ "ece": 0.8814,
+ "mean_confidence": 0.9614,
+ "acc_at_50_coverage": 0.1
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.08,
+ "macro_f1": 0.048,
+ "ece": 0.8814,
+ "mean_confidence": 0.9614,
+ "acc_at_50_coverage": 0.1
+ },
+ "clamped_rerun": {
+ "accuracy": 0.08,
+ "macro_f1": 0.048,
+ "ece": 0.7179,
+ "mean_confidence": 0.7979,
+ "acc_at_50_coverage": 0.1
+ },
+ "delta_ece": -0.1635,
+ "delta_mean_confidence": -0.1635
+ },
+ "tl": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.29,
+ "macro_f1": 0.249,
+ "ece": 0.6756,
+ "mean_confidence": 0.9656,
+ "acc_at_50_coverage": 0.46
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.29,
+ "macro_f1": 0.249,
+ "ece": 0.6756,
+ "mean_confidence": 0.9656,
+ "acc_at_50_coverage": 0.46
+ },
+ "clamped_rerun": {
+ "accuracy": 0.29,
+ "macro_f1": 0.249,
+ "ece": 0.5121,
+ "mean_confidence": 0.8021,
+ "acc_at_50_coverage": 0.46
+ },
+ "delta_ece": -0.1635,
+ "delta_mean_confidence": -0.1635
+ },
+ "tr": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.14,
+ "macro_f1": 0.0967,
+ "ece": 0.788,
+ "mean_confidence": 0.928,
+ "acc_at_50_coverage": 0.2
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.14,
+ "macro_f1": 0.0967,
+ "ece": 0.788,
+ "mean_confidence": 0.928,
+ "acc_at_50_coverage": 0.2
+ },
+ "clamped_rerun": {
+ "accuracy": 0.14,
+ "macro_f1": 0.0967,
+ "ece": 0.5874,
+ "mean_confidence": 0.7274,
+ "acc_at_50_coverage": 0.2
+ },
+ "delta_ece": -0.2006,
+ "delta_mean_confidence": -0.2006
+ },
+ "ur": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.07,
+ "macro_f1": 0.0645,
+ "ece": 0.8826,
+ "mean_confidence": 0.9526,
+ "acc_at_50_coverage": 0.08
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.07,
+ "macro_f1": 0.0645,
+ "ece": 0.8826,
+ "mean_confidence": 0.9526,
+ "acc_at_50_coverage": 0.08
+ },
+ "clamped_rerun": {
+ "accuracy": 0.07,
+ "macro_f1": 0.0645,
+ "ece": 0.6428,
+ "mean_confidence": 0.7128,
+ "acc_at_50_coverage": 0.08
+ },
+ "delta_ece": -0.2398,
+ "delta_mean_confidence": -0.2398
+ },
+ "vi": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.06,
+ "macro_f1": 0.0498,
+ "ece": 0.8914,
+ "mean_confidence": 0.9514,
+ "acc_at_50_coverage": 0.06
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.06,
+ "macro_f1": 0.0498,
+ "ece": 0.8914,
+ "mean_confidence": 0.9514,
+ "acc_at_50_coverage": 0.06
+ },
+ "clamped_rerun": {
+ "accuracy": 0.06,
+ "macro_f1": 0.0498,
+ "ece": 0.69,
+ "mean_confidence": 0.75,
+ "acc_at_50_coverage": 0.06
+ },
+ "delta_ece": -0.2014,
+ "delta_mean_confidence": -0.2014
+ },
+ "zh-CN": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.62,
+ "macro_f1": 0.6175,
+ "ece": 0.376,
+ "mean_confidence": 0.9885,
+ "acc_at_50_coverage": 0.86
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.62,
+ "macro_f1": 0.6175,
+ "ece": 0.376,
+ "mean_confidence": 0.9885,
+ "acc_at_50_coverage": 0.86
+ },
+ "clamped_rerun": {
+ "accuracy": 0.62,
+ "macro_f1": 0.6175,
+ "ece": 0.3198,
+ "mean_confidence": 0.9244,
+ "acc_at_50_coverage": 0.84
+ },
+ "delta_ece": -0.0562,
+ "delta_mean_confidence": -0.0641
+ },
+ "zh-TW": {
+ "n": 100,
+ "committed": {
+ "accuracy": 0.46,
+ "macro_f1": 0.4294,
+ "ece": 0.5198,
+ "mean_confidence": 0.9798,
+ "acc_at_50_coverage": 0.74
+ },
+ "unclamped_rerun": {
+ "accuracy": 0.46,
+ "macro_f1": 0.4294,
+ "ece": 0.5198,
+ "mean_confidence": 0.9798,
+ "acc_at_50_coverage": 0.74
+ },
+ "clamped_rerun": {
+ "accuracy": 0.46,
+ "macro_f1": 0.4294,
+ "ece": 0.4344,
+ "mean_confidence": 0.8944,
+ "acc_at_50_coverage": 0.72
+ },
+ "delta_ece": -0.0854,
+ "delta_mean_confidence": -0.0854
+ }
+ },
+ "spread": {
+ "largest_ece_change": [
+ {
+ "lang": "my",
+ "delta_ece": -0.2808
+ },
+ {
+ "lang": "ta",
+ "delta_ece": -0.2683
+ },
+ {
+ "lang": "hy",
+ "delta_ece": -0.2639
+ },
+ {
+ "lang": "ml",
+ "delta_ece": -0.2562
+ },
+ {
+ "lang": "kn",
+ "delta_ece": -0.2513
+ }
+ ],
+ "smallest_ece_change": [
+ {
+ "lang": "pt",
+ "delta_ece": -0.0799
+ },
+ {
+ "lang": "ru",
+ "delta_ece": -0.0734
+ },
+ {
+ "lang": "es",
+ "delta_ece": -0.0571
+ },
+ {
+ "lang": "zh-CN",
+ "delta_ece": -0.0562
+ },
+ {
+ "lang": "en",
+ "delta_ece": -0.0407
+ }
+ ]
+ },
+ "fields_the_harness_does_not_compute": {
+ "committed_only": [
+ "brier",
+ "nll",
+ "seconds",
+ "dropped"
+ ],
+ "note": "the committed file stores these and research/eval/laya_eval.py does not, so this re-run covers the six comparable columns rather than replacing the file"
+ }
+}