Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(benchmarks): align the laya-multilingual typed-decisions row with the committed JSON (#300) (#313)

kusef7a68458211c
diff --git a/BENCHMARKS.md b/BENCHMARKS.md
index ea7fa1c4..93ad27c6 100644
--- a/BENCHMARKS.md
+++ b/BENCHMARKS.md
@@ -156,7 +156,7 @@ banking77 is the one clear loss, and it is architectural: a choice question's op
|---|---|---|---|---|---|
| `laya-typed-decisions` | **0.766** | 0.471 | 0.061 | 0.213 | 0.242 |
| `laya` | 0.361 | 0.332 | 0.316 | 0.175 | 0.694 |
-| `laya-multilingual` | 0.342 | 0.326 | 0.439 | 0.285 | 0.687 |
+| `laya-multilingual` | 0.352 | 0.328 | 0.463 | 0.314 | 0.760 |
| *Jev 1.13.0 (published)* | *0.727* | *0.580* | *0.148* | *0.144* | *0.391* |
| *teacher ceiling* | *0.735* | *—* | *—* | *—* | *—* |
| *majority class* | *0.461* | *—* | *—* | *—* | *—* |
@@ -169,7 +169,7 @@ banking77 is the one clear loss, and it is architectural: a choice question's op
| invoice processing | 0.804 |
| security incidents | 0.766 |
-**The base checkpoints sit below the majority-class baseline** (0.362 and 0.342 against 0.461). All of the capability on this benchmark comes from fine-tuning.
+**The base checkpoints sit below the majority-class baseline** (0.362 and 0.352 against 0.461). All of the capability on this benchmark comes from fine-tuning.
---
diff --git a/README.md b/README.md
index 691b4347..4b408e54 100644
--- a/README.md
+++ b/README.md
@@ -726,7 +726,7 @@ Full detail, including every workflow and all 51 languages: **[`BENCHMARKS.md`](
|---|---|---|---|---|---|
| **`laya-typed-decisions`** | **0.766** | 0.471 | **0.062** | 0.213 | **0.242** |
| `laya` | 0.362 | 0.332 | 0.316 | 0.175 | 0.694 |
-| `laya-multilingual` | 0.342 | 0.326 | 0.439 | 0.285 | 0.687 |
+| `laya-multilingual` | 0.352 | 0.328 | 0.463 | 0.314 | 0.760 |
| *Jev 1.13.0 (published)* | *0.727* | *0.580* | *0.148* | *0.144* | *0.391* |
| *teacher self-agreement ceiling* | *0.735* | | | | |
| *per-question majority class* | *0.461* | | | | |
@@ -741,7 +741,7 @@ Two places it still trails Jev: **soft accuracy** (0.471 vs 0.580 — its argmax
its distributions match the teacher less well) and **ECE** (0.213 vs 0.144), which temperature
fitting addresses.
-**The base checkpoints sit below the majority-class baseline** (0.362 and 0.342 against 0.461).
+**The base checkpoints sit below the majority-class baseline** (0.362 and 0.352 against 0.461).
All of the capability on this benchmark comes from fine-tuning.
### Multilingual (51 languages, MASSIVE intent, 20 options, random = 0.050)