Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs: consolidated benchmark report and figure

Nandakishor5b75dc2cb48e
diff --git a/BENCHMARKS.md b/BENCHMARKS.md
new file mode 100644
index 00000000..2d448bec
--- /dev/null
+++ b/BENCHMARKS.md
@@ -0,0 +1,202 @@
+# Laya benchmarks
+
+Every checkpoint answered **byte-identical questions** in each run (fixed seed). Jev figures are **third-party published, never measured here** — no TypeSafe API access — so sample sizes and prompts differ; treat them as indicative.
+
+| run | what | where |
+|---|---|---|
+| T4 Colab | typed-decisions, MASSIVE (14 langs), XNLI (15 langs), English suites, latency, option-order robustness, calibration repair | `research/results/t4_colab_benchmark.json` |
+| CPU sweep | MASSIVE intent across **all 51 languages**, typed-decisions on all three checkpoints | `research/results/cpu_51_language_sweep.json` |
+| Applications | the six workflow themes + the datasets where Jev numbers exist, all three checkpoints | `research/results/app_benchmark.json` |
+
+---
+
+## Headline
+
+| | Laya | Jev (published) |
+|---|---|---|
+| typed-decisions (2,000 decisions) | **0.766** | 0.727 |
+| AG News (4 labels) | **0.953** | 0.910 |
+| DAIR Emotion (6 labels) | **0.600** | 0.480 |
+| ECE after temperature fitting | **0.081** | 0.246 |
+| p50 latency, 1 question (T4) | **32.8 ms** | 236-276 ms |
+
+---
+
+## Languages
+
+### All 51 MASSIVE languages — intent, 20 options (random = 0.050)
+
+| | laya | laya-multilingual |
+|---|---|---|
+| macro accuracy | 0.2269 | **0.3661** |
+| macro ECE *(lower better)* | 0.7331 | **0.3869** |
+| languages clearing 3× random | 23 / 51 | **45 / 51** |
+
+<details><summary><b>Per language (51)</b> — sorted by how much routing gains</summary>
+
+| lang | laya | laya-multilingual | Δ | laya ECE | multilingual ECE |
+|---|---|---|---|---|---|
+| `th` | 0.080 | 0.480 | +0.400 | 0.881 | 0.336 |
+| `ko` | 0.110 | 0.450 | +0.340 | 0.850 | 0.329 |
+| `he` | 0.060 | 0.400 | +0.340 | 0.911 | 0.350 |
+| `ur` | 0.070 | 0.400 | +0.330 | 0.883 | 0.311 |
+| `hi` | 0.100 | 0.430 | +0.330 | 0.850 | 0.321 |
+| `ar` | 0.110 | 0.400 | +0.290 | 0.800 | 0.341 |
+| `pl` | 0.240 | 0.510 | +0.270 | 0.713 | 0.350 |
+| `el` | 0.130 | 0.380 | +0.250 | 0.839 | 0.383 |
+| `fa` | 0.140 | 0.390 | +0.250 | 0.820 | 0.399 |
+| `ru` | 0.310 | 0.540 | +0.230 | 0.668 | 0.316 |
+| `tr` | 0.140 | 0.370 | +0.230 | 0.788 | 0.417 |
+| `lv` | 0.100 | 0.320 | +0.220 | 0.847 | 0.480 |
+| `bn` | 0.080 | 0.290 | +0.210 | 0.865 | 0.408 |
+| `nb` | 0.330 | 0.530 | +0.200 | 0.648 | 0.327 |
+| `vi` | 0.060 | 0.260 | +0.200 | 0.891 | 0.521 |
+| `az` | 0.100 | 0.300 | +0.200 | 0.825 | 0.368 |
+| `hu` | 0.090 | 0.290 | +0.200 | 0.857 | 0.422 |
+| `is` | 0.110 | 0.300 | +0.190 | 0.835 | 0.469 |
+| `sv` | 0.380 | 0.570 | +0.190 | 0.596 | 0.276 |
+| `km` | 0.000 | 0.180 | +0.180 | 0.952 | 0.412 |
+| `ml` | 0.070 | 0.240 | +0.170 | 0.857 | 0.414 |
+| `it` | 0.340 | 0.500 | +0.160 | 0.647 | 0.302 |
+| `fi` | 0.130 | 0.290 | +0.160 | 0.849 | 0.436 |
+| `ms` | 0.270 | 0.430 | +0.160 | 0.688 | 0.392 |
+| `da` | 0.350 | 0.500 | +0.150 | 0.626 | 0.263 |
+| `id` | 0.360 | 0.510 | +0.150 | 0.613 | 0.305 |
+| `te` | 0.090 | 0.220 | +0.130 | 0.858 | 0.370 |
+| `sl` | 0.200 | 0.330 | +0.130 | 0.756 | 0.433 |
+| `jv` | 0.160 | 0.270 | +0.110 | 0.803 | 0.506 |
+| `ta` | 0.120 | 0.230 | +0.110 | 0.822 | 0.397 |
+| `ja` | 0.530 | 0.640 | +0.110 | 0.460 | 0.228 |
+| `hy` | 0.050 | 0.150 | +0.100 | 0.835 | 0.506 |
+| `zh-TW` | 0.460 | 0.540 | +0.080 | 0.520 | 0.327 |
+| `de` | 0.420 | 0.500 | +0.080 | 0.558 | 0.301 |
+| `tl` | 0.290 | 0.360 | +0.070 | 0.676 | 0.374 |
+| `nl` | 0.390 | 0.450 | +0.060 | 0.591 | 0.378 |
+| `af` | 0.290 | 0.350 | +0.060 | 0.687 | 0.484 |
+| `my` | 0.060 | 0.120 | +0.060 | 0.861 | 0.455 |
+| `sq` | 0.210 | 0.260 | +0.050 | 0.755 | 0.476 |
+| `sw` | 0.130 | 0.180 | +0.050 | 0.828 | 0.549 |
+| `cy` | 0.120 | 0.160 | +0.040 | 0.841 | 0.591 |
+| `kn` | 0.110 | 0.150 | +0.040 | 0.842 | 0.437 |
+| `es` | 0.510 | 0.530 | +0.020 | 0.480 | 0.275 |
+| `ka` | 0.090 | 0.110 | +0.020 | 0.845 | 0.528 |
+| `ro` | 0.330 | 0.350 | +0.020 | 0.658 | 0.404 |
+| `zh-CN` | 0.620 | 0.630 | +0.010 | 0.376 | 0.212 |
+| `am` | 0.120 | 0.110 | -0.010 | 0.825 | 0.463 |
+| `pt` | 0.470 | 0.450 | -0.020 | 0.512 | 0.342 |
+| `mn` | 0.130 | 0.100 | -0.030 | 0.837 | 0.558 |
+| `fr` | 0.590 | 0.540 | -0.050 | 0.388 | 0.277 |
+| `en` | 0.820 | 0.680 | -0.140 | 0.179 | 0.209 |
+
+</details>
+
+### English vs the rest
+
+| task | | laya | laya-multilingual |
+|---|---|---|---|
+| MASSIVE intent — English | **0.783** | 0.657 |
+| MASSIVE intent — other languages | 0.306 | **0.451** |
+| MASSIVE scenario — English | **0.603** | 0.560 |
+| MASSIVE scenario — other languages | 0.281 | **0.439** |
+| XNLI — English | **0.860** | 0.843 |
+| XNLI — other languages | 0.521 | **0.731** |
+
+The English checkpoint does not degrade gracefully outside English — it collapses, and stays confident doing so. Khmer: **0.000 accuracy at 0.952 confidence**. Its mean confidence never drops below 0.885 at any accuracy level, so confidence gating cannot catch it — which is why routing happens *before* the forward pass.
+
+---
+
+## Themes — the application workflows
+
+Each is real labelled data, 400 cases, all three checkpoints. *held out* means the source was **not** in Laya's training mix.
+
+| theme | laya | laya-multilingual | laya-typed-decisions | data |
+|---|---|---|---|---|
+| Email spam | **0.993** | 0.993 | 0.958 | in training |
+| Phishing | 0.980 | **0.993** | 0.940 | in training |
+| LLM guardrails (jailbreak) | 0.708 | 0.755 | **0.762** | **held out** |
+| Moderation (toxicity) | **0.530** | 0.525 | 0.530 | **held out** |
+| RAG passage relevance | 0.625 | **0.657** | 0.625 | in training |
+| Support triage (10-way queue) | 0.502 | **0.522** | 0.505 | in training |
+| Model routing (domain) | 0.639 | 0.123 | **0.659** | held out |
+
+**Where it is strong:** email spam 0.993 and phishing 0.993, both with ECE around 0.01 — production-grade, though both were in the training mix.
+
+**Where it is weak:** moderation on held-out toxic-chat is 0.530 with macro-F1 0.400 — barely above chance on a balanced split. The demo Space has a Moderation tab; hand-picked examples work, real traffic does not. Guardrails at 0.708–0.762 is the honest jailbreak-detection number, consistent across two unrelated datasets (deepset prompt-injections measured 0.698 separately).
+
+### On the public datasets where Jev numbers exist
+
+| dataset | laya | laya-multilingual | laya-typed-decisions | Jev (published) |
+|---|---|---|---|---|
+| AG News (4 labels) | 0.950 | 0.930 | **0.953** | 0.910 |
+| DAIR Emotion (6 labels) | 0.595 | 0.530 | **0.600** | 0.480 |
+| banking77 (77 labels) | 0.425 | 0.425 | **0.492** | 0.870 |
+
+banking77 is the one clear loss, and it is architectural: a choice question's options share a fixed `head_max_len` budget, so 77 labels get roughly 4 tokens each and stop being distinguishable. Both checkpoints score **exactly 0.425**, which is what you would expect from a budget ceiling rather than a capability gap. Keep choice questions under ~20 options.
+
+---
+
+## typed-decisions — 400 cases, 2,000 decisions
+
+| model | accuracy | soft acc | Brier | ECE | score MAE |
+|---|---|---|---|---|---|
+| `laya-typed-decisions` | **0.766** | 0.471 | 0.061 | 0.213 | 0.242 |
+| `laya` | 0.361 | 0.332 | 0.316 | 0.175 | 0.694 |
+| `laya-multilingual` | 0.342 | 0.326 | 0.439 | 0.285 | 0.687 |
+| *Jev 1.13.0 (published)* | *0.727* | *0.580* | *0.148* | *0.144* | *0.391* |
+| *teacher ceiling* | *0.735* | *—* | *—* | *—* | *—* |
+| *majority class* | *0.461* | *—* | *—* | *—* | *—* |
+| *random guess* | *0.318* | *—* | *—* | *—* | *—* |
+
+| workflow | laya-typed-decisions |
+|---|---|
+| agent trace observability | 0.730 |
+| customer service | 0.764 |
+| invoice processing | 0.804 |
+| security incidents | 0.766 |
+
+**The base checkpoints sit below the majority-class baseline** (0.362 and 0.342 against 0.461). All of the capability on this benchmark comes from fine-tuning.
+
+---
+
+## Speed (Tesla T4)
+
+| questions per call | laya | laya-multilingual |
+|---|---|---|
+| 1 | 39.5 ms | **32.8 ms** |
+| 5 | 84.5 ms | **40.1 ms** |
+| 10 | 158.6 ms | **72.3 ms** |
+| 50 | 771.3 ms | **337.4 ms** |
+
+103–332 questions/sec batched. Jev independently measured at 236-276 ms p50, so Laya answers one question roughly **6–7× faster**.
+
+### Calibration
+
+| | as shipped | temperature refit |
+|---|---|---|
+| `laya` | 0.466 | **0.081** |
+| `laya-multilingual` | 0.314 | **0.106** |
+
+Both ship over-confident; `laya-multilingual` ships with no fitted temperatures at all. Refitting one temperature per (question type, option count) on held-out data is the single highest-value fix available, and takes ECE below Jev's measured 0.246.
+
+### Option-order robustness
+
+How often the answer changes when the options are permuted. Jev measured at 0.13.
+
+| suite | laya | laya-multilingual |
+|---|---|---|
+| massive_intent.en | 0.150 | 0.230 |
+| en.emotion | 0.040 | 0.090 |
+| xnli.en | 0.000 | 0.015 |
+
+At 20 options both are less order-stable than Jev — worth fixing with more aggressive option-order shuffling during training.
+
+---
+
+## Limits, stated plainly
+
+- **Near chance on typed-decisions zero-shot** — the 0.766 belongs to the fine-tuned checkpoint, on that benchmark's own training split.
+- **Moderation does not hold up on held-out data** (0.530, macro-F1 0.400).
+- **Keep `choice` questions under ~20 options.**
+- **Both checkpoints ship over-confident.** Fit temperatures on your own data.
+- **Ordinal `score` is the weakest primitive** (SST-5 0.372).
+- `laya` collapses outside English; `laya-multilingual` is weaker on English. Route.
diff --git a/README.md b/README.md
index 0be79fa3..30f44d69 100644
--- a/README.md
+++ b/README.md
@@ -12,7 +12,7 @@
[![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://opensource.org/licenses/Apache-2.0)
<p align="center">
- <img src="assets/laya_benchmark.png" alt="Laya benchmark: 51-language coverage, speed on a T4, comparison with TypeSafe Jev, and calibration before and after temperature fitting" width="100%" />
+ <img src="assets/laya_benchmark_common.png" alt="Laya consolidated benchmark: application workflows, English vs other languages, speed on a T4, batching, and typed-decisions against Jev" width="100%" />
</p>
Laya evaluates typed questions (`choice`, `score`, `noul`) over any state (text, email, ticket or JSON document) in **a single forward pass** — 33 ms for one question, 7.2 ms/question batched, measured on a T4. No text generation, so nothing to parse and nothing to hallucinate.
@@ -241,6 +241,12 @@ router.loaded # ['multilingual']
## Benchmarks
+**Full report: [`BENCHMARKS.md`](BENCHMARKS.md)** — every run consolidated, languages and themes, with per-language detail for all 51 languages.
+
+<p align="center">
+ <img src="assets/laya_benchmark.png" alt="Per-language accuracy for both checkpoints across 51 languages" width="100%" />
+</p>
+
All Laya numbers below are measured. Every model answered byte-identical questions
(fixed seed) in the same run. Reproduce with
[`notebooks/laya_benchmark_colab.ipynb`](https://github.com/NandhaKishorM/laya) on a T4.
diff --git a/assets/laya_benchmark_common.png b/assets/laya_benchmark_common.png
new file mode 100644
index 00000000..1a05395c
Binary files /dev/null and b/assets/laya_benchmark_common.png differ