Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(benchmarks): align the laya-multilingual typed-decisions row with the committed JSON (#300) (#313)
kusef7a68458211c
diff --git a/BENCHMARKS.md b/BENCHMARKS.mdindex ea7fa1c4..93ad27c6 100644--- a/BENCHMARKS.md+++ b/BENCHMARKS.md@@ -156,7 +156,7 @@ banking77 is the one clear loss, and it is architectural: a choice question's op|---|---|---|---|---|---|| `laya-typed-decisions` | **0.766** | 0.471 | 0.061 | 0.213 | 0.242 || `laya` | 0.361 | 0.332 | 0.316 | 0.175 | 0.694 |-| `laya-multilingual` | 0.342 | 0.326 | 0.439 | 0.285 | 0.687 |+| `laya-multilingual` | 0.352 | 0.328 | 0.463 | 0.314 | 0.760 || *Jev 1.13.0 (published)* | *0.727* | *0.580* | *0.148* | *0.144* | *0.391* || *teacher ceiling* | *0.735* | *—* | *—* | *—* | *—* || *majority class* | *0.461* | *—* | *—* | *—* | *—* |@@ -169,7 +169,7 @@ banking77 is the one clear loss, and it is architectural: a choice question's op| invoice processing | 0.804 || security incidents | 0.766 |-**The base checkpoints sit below the majority-class baseline** (0.362 and 0.342 against 0.461). All of the capability on this benchmark comes from fine-tuning.+**The base checkpoints sit below the majority-class baseline** (0.362 and 0.352 against 0.461). All of the capability on this benchmark comes from fine-tuning.---diff --git a/README.md b/README.mdindex 691b4347..4b408e54 100644--- a/README.md+++ b/README.md@@ -726,7 +726,7 @@ Full detail, including every workflow and all 51 languages: **[`BENCHMARKS.md`](|---|---|---|---|---|---|| **`laya-typed-decisions`** | **0.766** | 0.471 | **0.062** | 0.213 | **0.242** || `laya` | 0.362 | 0.332 | 0.316 | 0.175 | 0.694 |-| `laya-multilingual` | 0.342 | 0.326 | 0.439 | 0.285 | 0.687 |+| `laya-multilingual` | 0.352 | 0.328 | 0.463 | 0.314 | 0.760 || *Jev 1.13.0 (published)* | *0.727* | *0.580* | *0.148* | *0.144* | *0.391* || *teacher self-agreement ceiling* | *0.735* | | | | || *per-question majority class* | *0.461* | | | | |@@ -741,7 +741,7 @@ Two places it still trails Jev: **soft accuracy** (0.471 vs 0.580 — its argmaxits distributions match the teacher less well) and **ECE** (0.213 vs 0.144), which temperaturefitting addresses.-**The base checkpoints sit below the majority-class baseline** (0.362 and 0.342 against 0.461).+**The base checkpoints sit below the majority-class baseline** (0.362 and 0.352 against 0.461).All of the capability on this benchmark comes from fine-tuning.### Multilingual (51 languages, MASSIVE intent, 20 options, random = 0.050)