Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

release: 0.3.7

Nandakishor010bacef009c
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 02649ee2..9a06e70e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -62,6 +62,14 @@ jobs:
python tests/test_decision_model.py
python tests/test_packaging.py
python tests/test_lang_guess.py
+ python tests/test_calibration_persistence.py
+ python tests/test_context_manager.py
+ python tests/test_criteria_normalization.py
+ python tests/test_docker_entrypoint.py
+ python tests/test_empty_questions.py
+ python tests/test_router_memory.py
+ python tests/test_shortlist_cosine.py
+ python tests/test_temperature_loading.py
- name: Email cleaning tests
run: python tests/test_email.py
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 13faadb6..40ce7392 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -45,6 +45,14 @@ jobs:
python tests/test_decision_model.py
python tests/test_packaging.py
python tests/test_lang_guess.py
+ python tests/test_calibration_persistence.py
+ python tests/test_context_manager.py
+ python tests/test_criteria_normalization.py
+ python tests/test_docker_entrypoint.py
+ python tests/test_empty_questions.py
+ python tests/test_router_memory.py
+ python tests/test_shortlist_cosine.py
+ python tests/test_temperature_loading.py
- name: Build
run: python -m build
diff --git a/BENCHMARKS.md b/BENCHMARKS.md
index f92912c1..9e67c7c3 100644
--- a/BENCHMARKS.md
+++ b/BENCHMARKS.md
@@ -5,8 +5,11 @@ Every checkpoint answered **byte-identical questions** in each run (fixed seed).
| run | what | where |
|---|---|---|
| T4 Colab | typed-decisions, MASSIVE (14 langs), XNLI (15 langs), English suites, latency, option-order robustness, calibration repair | `research/results/t4_colab_benchmark.json` |
-| CPU sweep | MASSIVE intent across **all 51 languages**, typed-decisions on all three checkpoints | `research/results/cpu_51_language_sweep.json` |
-| Applications | the six workflow themes + the datasets where Jev numbers exist, all three checkpoints | `research/results/app_benchmark.json` |
+| CPU sweep | MASSIVE intent across **all 51 languages**; its typed-decisions part (`part_b`) covers the English checkpoint only | `research/results/cpu_51_language_sweep.json` |
+| Applications | the seven workflow themes + the datasets where Jev numbers exist, all three checkpoints (laya 0.2.1, CPU, 400 cases per task, seed 13, 2026-09-19) | `research/results/app_benchmark_results.json` |
+
+
+**Calibration columns in the CPU sweep predate the temperature clamp.** The 51-language ECE and mean-confidence figures were produced before #42 clamped temperatures to `[0.5, 5]`, so today's package reports different confidence for the affected buckets (`choice:11+` is now served at 0.5, not 0.1006). Accuracy columns are unaffected. A re-run with the current package is tracked in #208.
---
diff --git a/README.md b/README.md
index 2c6ee88f..a11bdaec 100644
--- a/README.md
+++ b/README.md
@@ -35,6 +35,17 @@ Three checkpoints, and a `Router` that picks between them per request:
| [`laya-multilingual`](https://huggingface.co/convaiinnovations/laya-multilingual) | mmBERT-base | 322M | 1024 | 100+ languages, 2x faster |
| [`laya-typed-decisions`](https://huggingface.co/convaiinnovations/laya-typed-decisions) | ModernBERT-large | 421M | 1024 | the typed-decisions workflows |
+### What's new in 0.3.7
+
+* **About 10x faster loading.** Checkpoints are built without the throwaway random weight initialisation, so `laya.load()` drops from about 22 s to about 2 s on CPU with bit-identical answers. This also skips the pass that crashed on Windows with Python 3.14 (#123).
+* **Better routing for non-English Latin text.** Plain-ASCII Spanish, Italian, Portuguese and French (accents stripped by mail clients and ticket systems) and Brazilian Portuguese support text now reach the multilingual checkpoint. Letters in scripts the router has no range for no longer fall through to English, and URLs, e-mail addresses and dotted names no longer count as words. Checked on 20,000 English texts, with no English prose moved.
+* **A Jev-compatible HTTP server.** `pip install "laya[serve]"`, then `laya-serve`, speaks `POST /v1/systemone`, so existing TypeSafe clients work by changing `baseUrl`. See [Self-Hosting](#self-hosting-http-server-jev-compatible).
+* **Router defaults and hooks.** `Router()` keeps two checkpoints resident, so alternating languages no longer reload a model on every request. You can pass your own language guess with `lang_guess=` and send undecided text to `Router(default=...)`. `Router` and `Agent` also work as context managers, and evicted models free their memory.
+* **Clearer errors and safer edge cases.** A malformed question is rejected with a message naming the question and what to fix. An empty question set returns an empty answer, and invalid temperatures in a checkpoint no longer stop it loading.
+* **E-mail cleaning for Portuguese and Spanish** replies, signatures and footers.
+* **Fine-tuning notebook fixes.** Calibration is now fitted on a held-out slice rather than on training data (#186), and stale temperature overrides are cleared before a refit.
+* **Docker quickstart** under `docs/docker.md`.
+
---
## Installation
@@ -596,6 +607,14 @@ result["shortlist"]["intent"]["labels"] # the top 20 labels sent to the model
[Issue #102](https://github.com/NandhaKishorM/laya/issues/102) reports that a top-20 zero-shot shortlist moved a BANKING77 run from 54.3% to 60.8% on the reporter's setup. Those figures are the reporter's; this repository has not remeasured them.
* Ordinal `score` questions are the weakest primitive (SST-5 0.372).
+* **`noul` can follow its option labels instead of the state, most strongly on `laya` (English).** `noul` renders its two options as `false:` / `true:`, and on the English checkpoint that label pair can dominate the answer, returning a confident "no" for clearly positive input (#156). Until a retrained checkpoint lands, check `noul` answers on your own data. If they look stuck, ask the same question as a two-option `choice` with neutral keys and your yes/no wording as the descriptions:
+
+ ```python
+ {"type": "choice", "instructions": "Is this review positive?",
+ "criteria": {"A": "yes, the review is positive", "B": "no, the review is negative"}}
+ ```
+* **`laya-multilingual` has a position bias on `score` questions** (#131): it rarely picks the first-listed level, in any language. For English score questions, route to `model="english"`, and for other languages validate score outputs on your own data before relying on them.
+* **`action.act_probability` carries no usable signal yet** (#185). It reads 1.0 for almost every input, and its raw logits run against correctness (AUROC 0.30 on 396 labelled decisions). Gate on `confidence` instead, which reaches an AUROC of 0.77 on the same items.
* `laya` collapses outside English; `laya-multilingual` is weaker on English. Route, or pick
deliberately.
@@ -603,6 +622,7 @@ result["shortlist"]["intent"]["labels"] # the top 20 labels sent to the model
## Community Tools
+* **[omp-laya-judge](https://github.com/F0Rextasy/omp-laya-judge)**: an [oh-my-pi](https://github.com/can1357/oh-my-pi) plugin with a local System-1 judge MCP server and skill (`choice`/`bool`/`score`, 0 tokens, about 0.3 s on CPU), confidence-gated escalation, and reproducible quiz and Snake demos.
* [laya-adk-toolkit](https://github.com/Ashfaqbs/laya-adk-toolkit) — [Google ADK](https://google.github.io/adk-docs/) tools that let an agent call Laya's `classify`/`score`/`detect` typed decisions directly as tools, instead of asking an LLM to guess at structured output.
---
@@ -615,10 +635,6 @@ result["shortlist"]["intent"]["labels"] # the top 20 labels sent to the model
---
-## Community integrations
-
-* **[omp-laya-judge](https://github.com/F0Rextasy/omp-laya-judge)** — [oh-my-pi](https://github.com/can1357/oh-my-pi) plugin: local System-1 judge MCP server + skill (`choice`/`bool`/`score`, 0 tokens, ~0.3 s on CPU), session-start banner, auto-use rule with confidence-gated escalation, plus reproducible quiz (6/8) and planner-assisted Snake (score 12, 300 moves, safety shield) demos.
-
## Fine-Tuning
Fine-tune Laya on your own domain data. The notebook runs on Kaggle's free 2xT4 GPUs and does
diff --git a/laya/__init__.py b/laya/__init__.py
index 5ea9d431..f2a46ecc 100644
--- a/laya/__init__.py
+++ b/laya/__init__.py
@@ -23,7 +23,7 @@ from .presets import (
from .router import DEFAULT_MODELS, RouteDecision, Router
from .shortlist import embed_fn_from_agent, predict_shortlist, shortlist_choice
-__version__ = "0.3.6"
+__version__ = "0.3.7"
__all__ = [
"Agent",
"RLAgent",
diff --git a/pyproject.toml b/pyproject.toml
index ad0e9a14..f519fc39 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "laya"
-version = "0.3.6"
+version = "0.3.7"
description = "Fast, non-autoregressive System 1 decision engine with calibrated probabilities"
readme = "README.md"
requires-python = ">=3.10"
diff --git a/research/results/app_benchmark_results.json b/research/results/app_benchmark_results.json
new file mode 100644
index 00000000..5f259ff2
--- /dev/null
+++ b/research/results/app_benchmark_results.json
@@ -0,0 +1,507 @@
+{
+ "meta": {
+ "timestamp": "2026-09-19 11:57:16",
+ "device": "cpu",
+ "n_per_task": 400,
+ "seed": 13,
+ "laya": "0.2.1"
+ },
+ "jev_published": {
+ "ag_news": {
+ "accuracy": 0.91,
+ "coverage_at_5pct_error": 0.83,
+ "latency_p50_ms": 236,
+ "n": 100,
+ "source": "AbdelStark/jev-benchmarks v0.1.0"
+ },
+ "banking77": {
+ "accuracy": 0.87,
+ "coverage_at_5pct_error": 0.86,
+ "latency_p50_ms": 246,
+ "n": 100,
+ "labels": 72,
+ "source": "AbdelStark/jev-benchmarks v0.1.0"
+ },
+ "emotion": {
+ "accuracy": 0.48,
+ "brier": 0.846,
+ "nll": 5.588,
+ "coverage_at_5pct_error": 0.0,
+ "zero_prob_failures": 0.16,
+ "n": 100,
+ "source": "AbdelStark/jev-benchmarks v0.1.0"
+ },
+ "typed_decisions": {
+ "accuracy": 0.727,
+ "soft_accuracy": 0.58,
+ "brier": 0.148,
+ "ece": 0.144,
+ "score_mae": 0.391,
+ "ms_per_case": 710,
+ "source": "laya repo comparison table"
+ },
+ "_independent_": {
+ "banking77_accuracy": 0.763,
+ "sms_spam_accuracy": 0.93,
+ "permuted_accuracy": 0.767,
+ "ece": 0.246,
+ "latency_p50_ms": "264-276",
+ "option_order_flip_rate": 0.13,
+ "source": "nibzard/decision-model-benchmark"
+ }
+ },
+ "suites": {
+ "jev.ag_news": {
+ "english": {
+ "n": 400,
+ "accuracy": 0.95,
+ "macro_f1": 0.9439,
+ "ece": 0.0316,
+ "brier": 0.0808,
+ "nll": 0.1637,
+ "mean_confidence": 0.922,
+ "acc_at_50_coverage": 1.0,
+ "seconds": 25.9,
+ "ms_per_case": 64.8,
+ "dropped": 0,
+ "jev": "ag_news",
+ "in_training": true
+ },
+ "multilingual": {
+ "n": 400,
+ "accuracy": 0.93,
+ "macro_f1": 0.925,
+ "ece": 0.042,
+ "brier": 0.1092,
+ "nll": 0.2329,
+ "mean_confidence": 0.967,
+ "acc_at_50_coverage": 0.995,
+ "seconds": 14.3,
+ "ms_per_case": 35.8,
+ "dropped": 0,
+ "jev": "ag_news",
+ "in_training": true
+ },
+ "typed-decisions": {
+ "n": 400,
+ "accuracy": 0.9525,
+ "macro_f1": 0.9468,
+ "ece": 0.155,
+ "brier": 0.1137,
+ "nll": 0.2801,
+ "mean_confidence": 0.7975,
+ "acc_at_50_coverage": 1.0,
+ "seconds": 41.4,
+ "ms_per_case": 103.6,
+ "dropped": 0,
+ "jev": "ag_news",
+ "in_training": true
+ }
+ },
+ "jev.emotion": {
+ "english": {
+ "n": 400,
+ "accuracy": 0.595,
+ "macro_f1": 0.4752,
+ "ece": 0.3063,
+ "brier": 0.6963,
+ "nll": 2.0194,
+ "mean_confidence": 0.9013,
+ "acc_at_50_coverage": 0.715,
+ "seconds": 13.2,
+ "ms_per_case": 33.0,
+ "dropped": 0,
+ "jev": "emotion",
+ "in_training": false
+ },
+ "multilingual": {
+ "n": 400,
+ "accuracy": 0.53,
+ "macro_f1": 0.4441,
+ "ece": 0.3412,
+ "brier": 0.7659,
+ "nll": 2.2797,
+ "mean_confidence": 0.8712,
+ "acc_at_50_coverage": 0.635,
+ "seconds": 7.0,
+ "ms_per_case": 17.6,
+ "dropped": 0,
+ "jev": "emotion",
+ "in_training": false
+ },
+ "typed-decisions": {
+ "n": 400,
+ "accuracy": 0.6,
+ "macro_f1": 0.4741,
+ "ece": 0.2005,
+ "brier": 0.6113,
+ "nll": 1.3242,
+ "mean_confidence": 0.7928,
+ "acc_at_50_coverage": 0.71,
+ "seconds": 19.9,
+ "ms_per_case": 49.7,
+ "dropped": 0,
+ "jev": "emotion",
+ "in_training": false
+ }
+ },
+ "jev.banking77_full": {
+ "english": {
+ "n": 400,
+ "accuracy": 0.425,
+ "macro_f1": 0.1121,
+ "ece": 0.5397,
+ "brier": 1.0968,
+ "nll": 12.0952,
+ "mean_confidence": 0.9616,
+ "acc_at_50_coverage": 0.59,
+ "seconds": 93.9,
+ "ms_per_case": 234.8,
+ "dropped": 0,
+ "jev": "banking77",
+ "in_training": false,
+ "n_labels": 77
+ },
+ "multilingual": {
+ "n": 400,
+ "accuracy": 0.425,
+ "macro_f1": 0.1125,
+ "ece": 0.4037,
+ "brier": 0.9455,
+ "nll": 4.5134,
+ "mean_confidence": 0.8229,
+ "acc_at_50_coverage": 0.61,
+ "seconds": 52.8,
+ "ms_per_case": 132.0,
+ "dropped": 0,
+ "jev": "banking77",
+ "in_training": false,
+ "n_labels": 77
+ },
+ "typed-decisions": {
+ "n": 400,
+ "accuracy": 0.4925,
+ "macro_f1": 0.1307,
+ "ece": 0.4173,
+ "brier": 0.8894,
+ "nll": 5.4993,
+ "mean_confidence": 0.9061,
+ "acc_at_50_coverage": 0.685,
+ "seconds": 117.4,
+ "ms_per_case": 293.6,
+ "dropped": 0,
+ "jev": "banking77",
+ "in_training": false,
+ "n_labels": 77
+ }
+ },
+ "app.support_triage": {
+ "english": {
+ "n": 400,
+ "accuracy": 0.5025,
+ "macro_f1": 0.4802,
+ "ece": 0.0907,
+ "brier": 0.6438,
+ "nll": 1.4237,
+ "mean_confidence": 0.5065,
+ "acc_at_50_coverage": 0.595,
+ "seconds": 90.9,
+ "ms_per_case": 227.2,
+ "dropped": 0,
+ "in_training": true
+ },
+ "multilingual": {
+ "n": 400,
+ "accuracy": 0.5225,
+ "macro_f1": 0.5159,
+ "ece": 0.0724,
+ "brier": 0.6034,
+ "nll": 1.2902,
+ "mean_confidence": 0.5949,
+ "acc_at_50_coverage": 0.685,
+ "seconds": 35.7,
+ "ms_per_case": 89.3,
+ "dropped": 0,
+ "in_training": true
+ },
+ "typed-decisions": {
+ "n": 400,
+ "accuracy": 0.505,
+ "macro_f1": 0.4699,
+ "ece": 0.1149,
+ "brier": 0.6603,
+ "nll": 1.4739,
+ "mean_confidence": 0.4165,
+ "acc_at_50_coverage": 0.595,
+ "seconds": 90.3,
+ "ms_per_case": 225.7,
+ "dropped": 0,
+ "in_training": true
+ }
+ },
+ "app.email_spam": {
+ "english": {
+ "n": 400,
+ "accuracy": 0.9925,
+ "macro_f1": 0.9925,
+ "ece": 0.0133,
+ "brier": 0.0119,
+ "nll": 0.023,
+ "mean_confidence": 0.9831,
+ "acc_at_50_coverage": 1.0,
+ "seconds": 108.8,
+ "ms_per_case": 272.0,
+ "dropped": 0,
+ "in_training": true
+ },
+ "multilingual": {
+ "n": 400,
+ "accuracy": 0.9925,
+ "macro_f1": 0.9925,
+ "ece": 0.0079,
+ "brier": 0.0148,
+ "nll": 0.0351,
+ "mean_confidence": 0.9976,
+ "acc_at_50_coverage": 1.0,
+ "seconds": 63.4,
+ "ms_per_case": 158.6,
+ "dropped": 0,
+ "in_training": true
+ },
+ "typed-decisions": {
+ "n": 400,
+ "accuracy": 0.9575,
+ "macro_f1": 0.9575,
+ "ece": 0.1085,
+ "brier": 0.0979,
+ "nll": 0.1927,
+ "mean_confidence": 0.849,
+ "acc_at_50_coverage": 1.0,
+ "seconds": 142.4,
+ "ms_per_case": 355.9,
+ "dropped": 0,
+ "in_training": true
+ }
+ },
+ "app.phishing": {
+ "english": {
+ "n": 400,
+ "accuracy": 0.98,
+ "macro_f1": 0.9789,
+ "ece": 0.0121,
+ "brier": 0.0292,
+ "nll": 0.048,
+ "mean_confidence": 0.9738,
+ "acc_at_50_coverage": 1.0,
+ "seconds": 118.8,
+ "ms_per_case": 297.0,
+ "dropped": 0,
+ "in_training": true
+ },
+ "multilingual": {
+ "n": 400,
+ "accuracy": 0.9925,
+ "macro_f1": 0.9921,
+ "ece": 0.0032,
+ "brier": 0.0121,
+ "nll": 0.0188,
+ "mean_confidence": 0.9949,
+ "acc_at_50_coverage": 1.0,
+ "seconds": 68.2,
+ "ms_per_case": 170.6,
+ "dropped": 0,
+ "in_training": true
+ },
+ "typed-decisions": {
+ "n": 400,
+ "accuracy": 0.94,
+ "macro_f1": 0.9353,
+ "ece": 0.0762,
+ "brier": 0.1062,
+ "nll": 0.2039,
+ "mean_confidence": 0.865,
+ "acc_at_50_coverage": 0.99,
+ "seconds": 144.0,
+ "ms_per_case": 360.1,
+ "dropped": 0,
+ "in_training": true
+ }
+ },
+ "app.guardrails_jailbreak": {
+ "english": {
+ "n": 400,
+ "accuracy": 0.7075,
+ "macro_f1": 0.684,
+ "ece": 0.2591,
+ "brier": 0.5389,
+ "nll": 1.8576,
+ "mean_confidence": 0.9657,
+ "acc_at_50_coverage": 0.83,
+ "seconds": 50.7,
+ "ms_per_case": 126.7,
+ "dropped": 0,
+ "in_training": false
+ },
+ "multilingual": {
+ "n": 400,
+ "accuracy": 0.755,
+ "macro_f1": 0.7208,
+ "ece": 0.1974,
+ "brier": 0.425,
+ "nll": 1.22,
+ "mean_confidence": 0.9454,
+ "acc_at_50_coverage": 0.84,
+ "seconds": 23.1,
+ "ms_per_case": 57.8,
+ "dropped": 0,
+ "in_training": false
+ },
+ "typed-decisions": {
+ "n": 400,
+ "accuracy": 0.7625,
+ "macro_f1": 0.733,
+ "ece": 0.0869,
+ "brier": 0.3365,
+ "nll": 0.5061,
+ "mean_confidence": 0.841,
+ "acc_at_50_coverage": 0.885,
+ "seconds": 49.5,
+ "ms_per_case": 123.8,
+ "dropped": 0,
+ "in_training": false
+ }
+ },
+ "app.moderation_toxicity": {
+ "english": {
+ "n": 400,
+ "accuracy": 0.53,
+ "macro_f1": 0.4003,
+ "ece": 0.2961,
+ "brier": 0.6015,
+ "nll": 0.8862,
+ "mean_confidence": 0.8242,
+ "acc_at_50_coverage": 0.735,
+ "seconds": 45.5,
+ "ms_per_case": 113.8,
+ "dropped": 0,
+ "in_training": false
+ },
+ "multilingual": {
+ "n": 400,
+ "accuracy": 0.525,
+ "macro_f1": 0.394,
+ "ece": 0.3809,
+ "brier": 0.7348,
+ "nll": 1.1981,
+ "mean_confidence": 0.9059,
+ "acc_at_50_coverage": 0.725,
+ "seconds": 19.1,
+ "ms_per_case": 47.8,
+ "dropped": 0,
+ "in_training": false
+ },
+ "typed-decisions": {
+ "n": 400,
+ "accuracy": 0.53,
+ "macro_f1": 0.3967,
+ "ece": 0.244,
+ "brier": 0.5315,
+ "nll": 0.7305,
+ "mean_confidence": 0.7553,
+ "acc_at_50_coverage": 0.72,
+ "seconds": 43.6,
+ "ms_per_case": 109.1,
+ "dropped": 0,
+ "in_training": false
+ }
+ },
+ "app.rag_relevance": {
+ "english": {
+ "n": 400,
+ "accuracy": 0.625,
+ "macro_f1": 0.6234,
+ "ece": 0.1175,
+ "brier": 0.4656,
+ "nll": 0.671,
+ "mean_confidence": 0.7425,
+ "acc_at_50_coverage": 0.72,
+ "seconds": 52.0,
+ "ms_per_case": 130.0,
+ "dropped": 0,
+ "in_training": true
+ },
+ "multilingual": {
+ "n": 400,
+ "accuracy": 0.6575,
+ "macro_f1": 0.6567,
+ "ece": 0.1718,
+ "brier": 0.489,
+ "nll": 0.7659,
+ "mean_confidence": 0.8293,
+ "acc_at_50_coverage": 0.73,
+ "seconds": 22.3,
+ "ms_per_case": 55.7,
+ "dropped": 0,
+ "in_training": true
+ },
+ "typed-decisions": {
+ "n": 400,
+ "accuracy": 0.625,
+ "macro_f1": 0.6239,
+ "ece": 0.0419,
+ "brier": 0.4458,
+ "nll": 0.6367,
+ "mean_confidence": 0.6433,
+ "acc_at_50_coverage": 0.73,
+ "seconds": 52.8,
+ "ms_per_case": 132.0,
+ "dropped": 0,
+ "in_training": true
+ }
+ },
+ "app.model_routing_domain": {
+ "english": {
+ "n": 399,
+ "accuracy": 0.6391,
+ "macro_f1": 0.3199,
+ "ece": 0.0889,
+ "brier": 0.4938,
+ "nll": 1.089,
+ "mean_confidence": 0.7121,
+ "acc_at_50_coverage": 0.8744,
+ "seconds": 50.8,
+ "ms_per_case": 127.4,
+ "dropped": 0,
+ "in_training": false
+ },
+ "multilingual": {
+ "n": 399,
+ "accuracy": 0.1228,
+ "macro_f1": 0.0912,
+ "ece": 0.4898,
+ "brier": 1.1396,
+ "nll": 2.3047,
+ "mean_confidence": 0.6126,
+ "acc_at_50_coverage": 0.0452,
+ "seconds": 21.9,
+ "ms_per_case": 54.9,
+ "dropped": 0,
+ "in_training": false
+ },
+ "typed-decisions": {
+ "n": 399,
+ "accuracy": 0.6591,
+ "macro_f1": 0.3426,
+ "ece": 0.1058,
+ "brier": 0.503,
+ "nll": 1.0623,
+ "mean_confidence": 0.5909,
+ "acc_at_50_coverage": 0.8342,
+ "seconds": 53.3,
+ "ms_per_case": 133.6,
+ "dropped": 0,
+ "in_training": false
+ }
+ }
+ }
+}
\ No newline at end of file