Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(benchmarks): trace every table to its run, align script outputs with cited paths (#134)
Nafeel00539d320287139
diff --git a/BENCHMARKS.md b/BENCHMARKS.mdindex 93ad27c6..93255c19 100644--- a/BENCHMARKS.md+++ b/BENCHMARKS.md@@ -35,6 +35,8 @@ The raw-temperature column reproduces the committed file, so the only variable l| ECE after temperature fitting | **0.081** | 0.246 || p50 latency, 1 question (T4) | **32.8 ms** | 236-276 ms |+Headline Laya cells for AG News and DAIR Emotion are the Applications-run numbers (`research/results/app_benchmark_results.json`); typed-decisions 0.766 is the fine-tuned checkpoint (no committed result file behind it yet). The committed T4 English suites give 0.947 (AG News) and 0.573 (DAIR Emotion) for `laya`.+---## Languages@@ -122,7 +124,7 @@ The English checkpoint does not degrade gracefully outside English — it collap## Themes — the application workflows-Each is real labelled data, 400 cases, all three checkpoints. *held out* means the source was **not** in Laya's training mix.+Each is real labelled data, 400 cases, all three checkpoints. *held out* means the source was **not** in Laya's training mix. Source: the Applications run (`research/scripts/bench_apps.py`, N=400 per task) — `research/results/app_benchmark_results.json`.| theme | laya | laya-multilingual | laya-typed-decisions | data ||---|---|---|---|---|@@ -140,6 +142,8 @@ Each is real labelled data, 400 cases, all three checkpoints. *held out* means t### On the public datasets where Jev numbers exist+Laya columns are from the same Applications run (`research/results/app_benchmark_results.json`, N=400 per task), so AG News / DAIR Emotion differ slightly from the committed T4 English suites below (N=600: 0.947 / 0.573 for `laya`).+| dataset | laya | laya-multilingual | laya-typed-decisions | Jev (published) ||---|---|---|---|---|| AG News (4 labels) | 0.950 | 0.930 | **0.953** | 0.910 |@@ -155,7 +159,7 @@ banking77 is the one clear loss, and it is architectural: a choice question's op| model | accuracy | soft acc | Brier | ECE | score MAE ||---|---|---|---|---|---|| `laya-typed-decisions` | **0.766** | 0.471 | 0.061 | 0.213 | 0.242 |-| `laya` | 0.361 | 0.332 | 0.316 | 0.175 | 0.694 |+| `laya` | 0.362 | 0.332 | 0.316 | 0.175 | 0.694 || `laya-multilingual` | 0.352 | 0.328 | 0.463 | 0.314 | 0.760 || *Jev 1.13.0 (published)* | *0.727* | *0.580* | *0.148* | *0.144* | *0.391* || *teacher ceiling* | *0.735* | *—* | *—* | *—* | *—* |@@ -169,7 +173,7 @@ banking77 is the one clear loss, and it is architectural: a choice question's op| invoice processing | 0.804 || security incidents | 0.766 |-**The base checkpoints sit below the majority-class baseline** (0.362 and 0.352 against 0.461). All of the capability on this benchmark comes from fine-tuning.+**The base checkpoints sit below the majority-class baseline** (0.362 and 0.352 against 0.461). All of the capability on this benchmark comes from fine-tuning. Base rows: `suites.typed_decisions` in `research/results/t4_colab_benchmark.json` (0.3620 / 0.3515; CPU sweep `part_b` gives 0.3615 for English). The `laya-typed-decisions` row and the four per-workflow scores are from the fine-tuning run and have no committed result file behind them yet.---diff --git a/README.md b/README.mdindex 507e230e..c2c6eec0 100644--- a/README.md+++ b/README.md@@ -792,6 +792,7 @@ question roughly **6-7x faster**.Every Laya figure is what `Router().predict(...)` actually returns — the checkpoint the routerselects for that input, not a hand-picked best of three. Jev figures are **third-partypublished, never measured here** (no TypeSafe API access), so sample sizes and prompts differ.+The Laya column here comes from the Applications run (`research/scripts/bench_apps.py`, N=400 per task; `research/results/app_benchmark_results.json`), so AG News / DAIR Emotion read 0.950 / 0.595 here versus 0.947 / 0.573 in the committed T4 English suites (N=600) below.| | Jev 1.13.0 | Laya (routed) | ||---|---|---|---|@@ -840,7 +841,7 @@ its distributions match the teacher less well) and **ECE** (0.213 vs 0.144), whifitting addresses.**The base checkpoints sit below the majority-class baseline** (0.362 and 0.352 against 0.461).-All of the capability on this benchmark comes from fine-tuning.+All of the capability on this benchmark comes from fine-tuning. Base rows match the committed runs (`suites.typed_decisions` in `research/results/t4_colab_benchmark.json`: 0.3620 / 0.3515; CPU sweep `part_b`: 0.3615 for English). The `laya-typed-decisions` row and its four per-workflow scores are from the fine-tuning run and have no committed result file behind them yet.### Multilingual (51 languages, MASSIVE intent, 20 options, random = 0.050)diff --git a/research/README.md b/research/README.mdindex 15783662..17caeea5 100644--- a/research/README.md+++ b/research/README.md@@ -69,8 +69,10 @@ python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-si|---|---|| `results/t4_colab_benchmark.json` | 17,416 questions on one T4, both checkpoints, identical questions per model || `results/long_context_multilingual.json` | the long-document run behind `assets/long_context_8192.png`: every prediction, with device and library versions |-| `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options |+| `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options (committed `part_b` covers the English checkpoint only; rerun `research/scripts/bench_local.py` for all three) || `results/cpu_51_language_sweep_clamped.json` | the same 51 languages and 5,100 cases re-run after the temperature clamp, raw temperatures and served temperatures side by side ([#208](https://github.com/NandhaKishorM/laya/issues/208)) |+| `results/app_benchmark_results.json` | Applications run (400 cases per task, seed 13, 2026-09-19); source for the BENCHMARKS Themes / Jev-dataset tables and the README routed AG News / DAIR Emotion cells |+| `results/latency_benchmark.json` | **not yet committed** — generate with `research/scripts/bench_latency.py`; CPU routing-cost regimes behind the README preload table |## Headline findingsdiff --git a/research/scripts/bench_apps.py b/research/scripts/bench_apps.pyindex 1d83a90e..cc526244 100644--- a/research/scripts/bench_apps.py+++ b/research/scripts/bench_apps.py@@ -29,14 +29,15 @@ os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")import numpy as np # noqa: E402import torch # noqa: E402-REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))-sys.path.insert(0, os.path.dirname(REPO))-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # bench_local lives here+REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+RESULTS_DIR = os.path.join(REPO, "research", "results")+sys.path.insert(0, REPO) # `import laya` works without an installed package+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))import laya # noqa: E402from bench_local import load, metrics, score_cases, softmax_t, temp_for # noqa: E402-OUT = os.path.join(REPO, "app_benchmark_results.json")+OUT = os.path.join(RESULTS_DIR, "app_benchmark_results.json")SEED = 13N = int(os.environ.get("BENCH_N", "400"))diff --git a/research/scripts/bench_latency.py b/research/scripts/bench_latency.pyindex 43dec911..105cf501 100644--- a/research/scripts/bench_latency.py+++ b/research/scripts/bench_latency.py@@ -23,8 +23,9 @@ os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")import numpy as np # noqa: E402import torch # noqa: E402-REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))-sys.path.insert(0, os.path.dirname(REPO))+REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+RESULTS_DIR = os.path.join(REPO, "research", "results")+sys.path.insert(0, REPO) # `import laya` works without an installed packageimport laya # noqa: E402from laya.lang import analyse # noqa: E402@@ -34,7 +35,7 @@ ROOT = os.path.expanduser("~/laya_models")MODELS = {"english": os.path.join(ROOT, "laya"),"multilingual": os.path.join(ROOT, "laya-multilingual"),"typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}-OUT = os.path.join(REPO, "latency_benchmark_results.json")+OUT = os.path.join(RESULTS_DIR, "latency_benchmark.json")STATE_EN = {"ticket": {"subject": "Payout failing", "messages": [{"from": "customer","text": "Hi, my Stripe payouts have failed for 3 days and I am losing sales. Please help ASAP. " * 6}]}}@@ -68,7 +69,7 @@ def main():"torch": torch.__version__, "threads": torch.get_num_threads(),"laya": laya.__version__,"note": "CPU numbers. GPU (T4) reference from the Colab run is in "- "laya_benchmark_results.json -> latency."}}+ "research/results/t4_colab_benchmark.json -> latency."}}# ---------------------------------------------------------------- 1. detection overheadprint("=== 1. language detection overhead (no model) ===", flush=True)diff --git a/research/scripts/bench_local.py b/research/scripts/bench_local.pyindex 3a3958fe..a39ee872 100644--- a/research/scripts/bench_local.py+++ b/research/scripts/bench_local.py@@ -5,7 +5,7 @@ Part B typed-decisions (400 cases / 2,000 decisions) on all three checkpoints,fine-tuned laya-typed-decisions can be compared with Jev's published 0.727 on thesame benchmark.-Writes local_benchmark_results.json.+Writes research/results/cpu_51_language_sweep.json.USE_TF=0 python3 research/scripts/bench_local.py [--langs N] [--per-lang N] [--skip-a] [--skip-b]"""@@ -26,8 +26,9 @@ os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")import numpy as np # noqa: E402import torch # noqa: E402-REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))-sys.path.insert(0, os.path.dirname(REPO))+REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+RESULTS_DIR = os.path.join(REPO, "research", "results")+sys.path.insert(0, REPO) # `import laya` works without an installed packageimport laya # noqa: E402from laya.common import QTYPES, build_sequence, collate_items, render_options, temp_bucket # noqa: E402@@ -36,7 +37,7 @@ ROOT = os.path.expanduser("~/laya_models")MODELS = {"english": os.path.join(ROOT, "laya"),"multilingual": os.path.join(ROOT, "laya-multilingual"),"typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}-OUT = os.path.join(REPO, "local_benchmark_results.json")+OUT = os.path.join(RESULTS_DIR, "cpu_51_language_sweep.json")SEED, N_OPTS = 13, 20diff --git a/research/scripts/build_benchmark_nb.py b/research/scripts/build_benchmark_nb.pyindex 5c859035..aa792e79 100644--- a/research/scripts/build_benchmark_nb.py+++ b/research/scripts/build_benchmark_nb.py@@ -2,7 +2,8 @@convaiinnovations/laya (ModernBERT-large, English) vs convaiinnovations/laya-multilingual(mmBERT-base, 100+ languages), on a Colab T4.-Writes one laya_benchmark_results.json that can be handed back for analysis.+Writes one laya_benchmark_results.json on Colab that is committed as+research/results/t4_colab_benchmark.json."""import jsonimport os@@ -24,7 +25,8 @@ CELLS = [# Laya vs Laya-Multilingual — extensive head-to-head benchmarkRuns both open checkpoints over the same questions and writes a single-`laya_benchmark_results.json`.+`laya_benchmark_results.json` on Colab (committed as+`research/results/t4_colab_benchmark.json`).| | `convaiinnovations/laya` | `convaiinnovations/laya-multilingual` ||---|---|---|diff --git a/research/scripts/make_plots.py b/research/scripts/make_plots.pyindex 2a944e7f..460d2684 100644--- a/research/scripts/make_plots.py+++ b/research/scripts/make_plots.py@@ -103,7 +103,8 @@ def main():names = ["AG News\n(4 labels)", "DAIR Emotion\n(6 labels)", "typed-decisions\n(2,000 dec.)"]jev = [0.910, 0.480, 0.727]lay = [col["suites"]["en.ag_news"]["laya"]["calibrated"]["accuracy"],- col["suites"]["en.emotion"]["laya"]["calibrated"]["accuracy"], 0.766]+ col["suites"]["en.emotion"]["laya"]["calibrated"]["accuracy"],+ 0.766] # fine-tuned checkpoint; uncommitted run (no committed result file yet)x = np.arange(3)axC.bar(x - w / 2 - 0.01, jev, w, color=INK3, label="Jev (published)", zorder=3)axC.bar(x + w / 2 + 0.01, lay, w, color=BLUE, label="Laya (measured)", zorder=3)