Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(benchmarks): trace every table to its run, align script outputs with cited paths (#134)

Nafeel00539d320287139
diff --git a/BENCHMARKS.md b/BENCHMARKS.md
index 93ad27c6..93255c19 100644
--- a/BENCHMARKS.md
+++ b/BENCHMARKS.md
@@ -35,6 +35,8 @@ The raw-temperature column reproduces the committed file, so the only variable l
| ECE after temperature fitting | **0.081** | 0.246 |
| p50 latency, 1 question (T4) | **32.8 ms** | 236-276 ms |
+Headline Laya cells for AG News and DAIR Emotion are the Applications-run numbers (`research/results/app_benchmark_results.json`); typed-decisions 0.766 is the fine-tuned checkpoint (no committed result file behind it yet). The committed T4 English suites give 0.947 (AG News) and 0.573 (DAIR Emotion) for `laya`.
+
---
## Languages
@@ -122,7 +124,7 @@ The English checkpoint does not degrade gracefully outside English — it collap
## Themes — the application workflows
-Each is real labelled data, 400 cases, all three checkpoints. *held out* means the source was **not** in Laya's training mix.
+Each is real labelled data, 400 cases, all three checkpoints. *held out* means the source was **not** in Laya's training mix. Source: the Applications run (`research/scripts/bench_apps.py`, N=400 per task) — `research/results/app_benchmark_results.json`.
| theme | laya | laya-multilingual | laya-typed-decisions | data |
|---|---|---|---|---|
@@ -140,6 +142,8 @@ Each is real labelled data, 400 cases, all three checkpoints. *held out* means t
### On the public datasets where Jev numbers exist
+Laya columns are from the same Applications run (`research/results/app_benchmark_results.json`, N=400 per task), so AG News / DAIR Emotion differ slightly from the committed T4 English suites below (N=600: 0.947 / 0.573 for `laya`).
+
| dataset | laya | laya-multilingual | laya-typed-decisions | Jev (published) |
|---|---|---|---|---|
| AG News (4 labels) | 0.950 | 0.930 | **0.953** | 0.910 |
@@ -155,7 +159,7 @@ banking77 is the one clear loss, and it is architectural: a choice question's op
| model | accuracy | soft acc | Brier | ECE | score MAE |
|---|---|---|---|---|---|
| `laya-typed-decisions` | **0.766** | 0.471 | 0.061 | 0.213 | 0.242 |
-| `laya` | 0.361 | 0.332 | 0.316 | 0.175 | 0.694 |
+| `laya` | 0.362 | 0.332 | 0.316 | 0.175 | 0.694 |
| `laya-multilingual` | 0.352 | 0.328 | 0.463 | 0.314 | 0.760 |
| *Jev 1.13.0 (published)* | *0.727* | *0.580* | *0.148* | *0.144* | *0.391* |
| *teacher ceiling* | *0.735* | *—* | *—* | *—* | *—* |
@@ -169,7 +173,7 @@ banking77 is the one clear loss, and it is architectural: a choice question's op
| invoice processing | 0.804 |
| security incidents | 0.766 |
-**The base checkpoints sit below the majority-class baseline** (0.362 and 0.352 against 0.461). All of the capability on this benchmark comes from fine-tuning.
+**The base checkpoints sit below the majority-class baseline** (0.362 and 0.352 against 0.461). All of the capability on this benchmark comes from fine-tuning. Base rows: `suites.typed_decisions` in `research/results/t4_colab_benchmark.json` (0.3620 / 0.3515; CPU sweep `part_b` gives 0.3615 for English). The `laya-typed-decisions` row and the four per-workflow scores are from the fine-tuning run and have no committed result file behind them yet.
---
diff --git a/README.md b/README.md
index 507e230e..c2c6eec0 100644
--- a/README.md
+++ b/README.md
@@ -792,6 +792,7 @@ question roughly **6-7x faster**.
Every Laya figure is what `Router().predict(...)` actually returns — the checkpoint the router
selects for that input, not a hand-picked best of three. Jev figures are **third-party
published, never measured here** (no TypeSafe API access), so sample sizes and prompts differ.
+The Laya column here comes from the Applications run (`research/scripts/bench_apps.py`, N=400 per task; `research/results/app_benchmark_results.json`), so AG News / DAIR Emotion read 0.950 / 0.595 here versus 0.947 / 0.573 in the committed T4 English suites (N=600) below.
| | Jev 1.13.0 | Laya (routed) | |
|---|---|---|---|
@@ -840,7 +841,7 @@ its distributions match the teacher less well) and **ECE** (0.213 vs 0.144), whi
fitting addresses.
**The base checkpoints sit below the majority-class baseline** (0.362 and 0.352 against 0.461).
-All of the capability on this benchmark comes from fine-tuning.
+All of the capability on this benchmark comes from fine-tuning. Base rows match the committed runs (`suites.typed_decisions` in `research/results/t4_colab_benchmark.json`: 0.3620 / 0.3515; CPU sweep `part_b`: 0.3615 for English). The `laya-typed-decisions` row and its four per-workflow scores are from the fine-tuning run and have no committed result file behind them yet.
### Multilingual (51 languages, MASSIVE intent, 20 options, random = 0.050)
diff --git a/research/README.md b/research/README.md
index 15783662..17caeea5 100644
--- a/research/README.md
+++ b/research/README.md
@@ -69,8 +69,10 @@ python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-si
|---|---|
| `results/t4_colab_benchmark.json` | 17,416 questions on one T4, both checkpoints, identical questions per model |
| `results/long_context_multilingual.json` | the long-document run behind `assets/long_context_8192.png`: every prediction, with device and library versions |
-| `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options |
+| `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options (committed `part_b` covers the English checkpoint only; rerun `research/scripts/bench_local.py` for all three) |
| `results/cpu_51_language_sweep_clamped.json` | the same 51 languages and 5,100 cases re-run after the temperature clamp, raw temperatures and served temperatures side by side ([#208](https://github.com/NandhaKishorM/laya/issues/208)) |
+| `results/app_benchmark_results.json` | Applications run (400 cases per task, seed 13, 2026-09-19); source for the BENCHMARKS Themes / Jev-dataset tables and the README routed AG News / DAIR Emotion cells |
+| `results/latency_benchmark.json` | **not yet committed** — generate with `research/scripts/bench_latency.py`; CPU routing-cost regimes behind the README preload table |
## Headline findings
diff --git a/research/scripts/bench_apps.py b/research/scripts/bench_apps.py
index 1d83a90e..cc526244 100644
--- a/research/scripts/bench_apps.py
+++ b/research/scripts/bench_apps.py
@@ -29,14 +29,15 @@ os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
import numpy as np # noqa: E402
import torch # noqa: E402
-REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-sys.path.insert(0, os.path.dirname(REPO))
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # bench_local lives here
+REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+RESULTS_DIR = os.path.join(REPO, "research", "results")
+sys.path.insert(0, REPO) # `import laya` works without an installed package
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import laya # noqa: E402
from bench_local import load, metrics, score_cases, softmax_t, temp_for # noqa: E402
-OUT = os.path.join(REPO, "app_benchmark_results.json")
+OUT = os.path.join(RESULTS_DIR, "app_benchmark_results.json")
SEED = 13
N = int(os.environ.get("BENCH_N", "400"))
diff --git a/research/scripts/bench_latency.py b/research/scripts/bench_latency.py
index 43dec911..105cf501 100644
--- a/research/scripts/bench_latency.py
+++ b/research/scripts/bench_latency.py
@@ -23,8 +23,9 @@ os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
import numpy as np # noqa: E402
import torch # noqa: E402
-REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-sys.path.insert(0, os.path.dirname(REPO))
+REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+RESULTS_DIR = os.path.join(REPO, "research", "results")
+sys.path.insert(0, REPO) # `import laya` works without an installed package
import laya # noqa: E402
from laya.lang import analyse # noqa: E402
@@ -34,7 +35,7 @@ ROOT = os.path.expanduser("~/laya_models")
MODELS = {"english": os.path.join(ROOT, "laya"),
"multilingual": os.path.join(ROOT, "laya-multilingual"),
"typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}
-OUT = os.path.join(REPO, "latency_benchmark_results.json")
+OUT = os.path.join(RESULTS_DIR, "latency_benchmark.json")
STATE_EN = {"ticket": {"subject": "Payout failing", "messages": [{"from": "customer",
"text": "Hi, my Stripe payouts have failed for 3 days and I am losing sales. Please help ASAP. " * 6}]}}
@@ -68,7 +69,7 @@ def main():
"torch": torch.__version__, "threads": torch.get_num_threads(),
"laya": laya.__version__,
"note": "CPU numbers. GPU (T4) reference from the Colab run is in "
- "laya_benchmark_results.json -> latency."}}
+ "research/results/t4_colab_benchmark.json -> latency."}}
# ---------------------------------------------------------------- 1. detection overhead
print("=== 1. language detection overhead (no model) ===", flush=True)
diff --git a/research/scripts/bench_local.py b/research/scripts/bench_local.py
index 3a3958fe..a39ee872 100644
--- a/research/scripts/bench_local.py
+++ b/research/scripts/bench_local.py
@@ -5,7 +5,7 @@ Part B typed-decisions (400 cases / 2,000 decisions) on all three checkpoints,
fine-tuned laya-typed-decisions can be compared with Jev's published 0.727 on the
same benchmark.
-Writes local_benchmark_results.json.
+Writes research/results/cpu_51_language_sweep.json.
USE_TF=0 python3 research/scripts/bench_local.py [--langs N] [--per-lang N] [--skip-a] [--skip-b]
"""
@@ -26,8 +26,9 @@ os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
import numpy as np # noqa: E402
import torch # noqa: E402
-REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-sys.path.insert(0, os.path.dirname(REPO))
+REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+RESULTS_DIR = os.path.join(REPO, "research", "results")
+sys.path.insert(0, REPO) # `import laya` works without an installed package
import laya # noqa: E402
from laya.common import QTYPES, build_sequence, collate_items, render_options, temp_bucket # noqa: E402
@@ -36,7 +37,7 @@ ROOT = os.path.expanduser("~/laya_models")
MODELS = {"english": os.path.join(ROOT, "laya"),
"multilingual": os.path.join(ROOT, "laya-multilingual"),
"typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}
-OUT = os.path.join(REPO, "local_benchmark_results.json")
+OUT = os.path.join(RESULTS_DIR, "cpu_51_language_sweep.json")
SEED, N_OPTS = 13, 20
diff --git a/research/scripts/build_benchmark_nb.py b/research/scripts/build_benchmark_nb.py
index 5c859035..aa792e79 100644
--- a/research/scripts/build_benchmark_nb.py
+++ b/research/scripts/build_benchmark_nb.py
@@ -2,7 +2,8 @@
convaiinnovations/laya (ModernBERT-large, English) vs convaiinnovations/laya-multilingual
(mmBERT-base, 100+ languages), on a Colab T4.
-Writes one laya_benchmark_results.json that can be handed back for analysis.
+Writes one laya_benchmark_results.json on Colab that is committed as
+research/results/t4_colab_benchmark.json.
"""
import json
import os
@@ -24,7 +25,8 @@ CELLS = [
# Laya vs Laya-Multilingual — extensive head-to-head benchmark
Runs both open checkpoints over the same questions and writes a single
-`laya_benchmark_results.json`.
+`laya_benchmark_results.json` on Colab (committed as
+`research/results/t4_colab_benchmark.json`).
| | `convaiinnovations/laya` | `convaiinnovations/laya-multilingual` |
|---|---|---|
diff --git a/research/scripts/make_plots.py b/research/scripts/make_plots.py
index 2a944e7f..460d2684 100644
--- a/research/scripts/make_plots.py
+++ b/research/scripts/make_plots.py
@@ -103,7 +103,8 @@ def main():
names = ["AG News\n(4 labels)", "DAIR Emotion\n(6 labels)", "typed-decisions\n(2,000 dec.)"]
jev = [0.910, 0.480, 0.727]
lay = [col["suites"]["en.ag_news"]["laya"]["calibrated"]["accuracy"],
- col["suites"]["en.emotion"]["laya"]["calibrated"]["accuracy"], 0.766]
+ col["suites"]["en.emotion"]["laya"]["calibrated"]["accuracy"],
+ 0.766] # fine-tuned checkpoint; uncommitted run (no committed result file yet)
x = np.arange(3)
axC.bar(x - w / 2 - 0.01, jev, w, color=INK3, label="Jev (published)", zorder=3)
axC.bar(x + w / 2 + 0.01, lay, w, color=BLUE, label="Laya (measured)", zorder=3)