Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(benchmarks): keep bench defaults on gitignored fresh-run paths (#134)
Nafeel005314cd5bafcd1
diff --git a/research/README.md b/research/README.mdindex 17caeea5..a820a860 100644--- a/research/README.md+++ b/research/README.md@@ -69,10 +69,10 @@ python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-si|---|---|| `results/t4_colab_benchmark.json` | 17,416 questions on one T4, both checkpoints, identical questions per model || `results/long_context_multilingual.json` | the long-document run behind `assets/long_context_8192.png`: every prediction, with device and library versions |-| `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options (committed `part_b` covers the English checkpoint only; rerun `research/scripts/bench_local.py` for all three) |+| `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options (committed `part_b` covers the English checkpoint only; a rerun of `research/scripts/bench_local.py` writes the gitignored `research/local_benchmark_results.json` and leaves this file untouched) || `results/cpu_51_language_sweep_clamped.json` | the same 51 languages and 5,100 cases re-run after the temperature clamp, raw temperatures and served temperatures side by side ([#208](https://github.com/NandhaKishorM/laya/issues/208)) |-| `results/app_benchmark_results.json` | Applications run (400 cases per task, seed 13, 2026-09-19); source for the BENCHMARKS Themes / Jev-dataset tables and the README routed AG News / DAIR Emotion cells |-| `results/latency_benchmark.json` | **not yet committed** — generate with `research/scripts/bench_latency.py`; CPU routing-cost regimes behind the README preload table |+| `results/app_benchmark_results.json` | Applications run (400 cases per task, seed 13, 2026-09-19); source for the BENCHMARKS Themes / Jev-dataset tables and the README routed AG News / DAIR Emotion cells (a rerun of `research/scripts/bench_apps.py` writes the gitignored `research/app_benchmark_results.json` instead) |+| `research/latency_benchmark_results.json` | **not committed** (gitignored) — generate with `research/scripts/bench_latency.py`; CPU routing-cost regimes behind the README preload table |## Headline findingsdiff --git a/research/scripts/bench_apps.py b/research/scripts/bench_apps.pyindex cc526244..c54d2746 100644--- a/research/scripts/bench_apps.py+++ b/research/scripts/bench_apps.py@@ -12,6 +12,12 @@ Workflows (the demo Space's tabs), each on real labelled data:Jev-comparable tasks (AbdelStark/jev-benchmarks published Jev accuracy on these):ag_news 0.910 | banking77 0.870 | dair emotion 0.480 (Brier 0.846, NLL 5.588)+Writes research/app_benchmark_results.json (fresh run, gitignored).++The committed Applications run behind the BENCHMARKS Themes / Jev-dataset+tables is research/results/app_benchmark_results.json — this script must not+overwrite it, so the default output stays on the gitignored name.+USE_TF=0 python3 research/scripts/bench_apps.py"""import gc@@ -30,14 +36,13 @@ import numpy as np # noqa: E402import torch # noqa: E402REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))-RESULTS_DIR = os.path.join(REPO, "research", "results")sys.path.insert(0, REPO) # `import laya` works without an installed packagesys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))import laya # noqa: E402from bench_local import load, metrics, score_cases, softmax_t, temp_for # noqa: E402-OUT = os.path.join(RESULTS_DIR, "app_benchmark_results.json")+OUT = os.path.join(REPO, "research", "app_benchmark_results.json")SEED = 13N = int(os.environ.get("BENCH_N", "400"))diff --git a/research/scripts/bench_latency.py b/research/scripts/bench_latency.pyindex 105cf501..7bd7d98a 100644--- a/research/scripts/bench_latency.py+++ b/research/scripts/bench_latency.py@@ -7,6 +7,9 @@ Routing changes the latency picture in two ways:So there are two regimes -- hot (right model already loaded) and cold (swap) -- and the realisticnumber for a mixed-language workload sits between them, determined by `max_loaded`.+Writes research/latency_benchmark_results.json (fresh run, gitignored;+the name tests/test_doc_tables.py expects).+USE_TF=0 python3 research/scripts/bench_latency.py"""import gc@@ -24,7 +27,6 @@ import numpy as np # noqa: E402import torch # noqa: E402REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))-RESULTS_DIR = os.path.join(REPO, "research", "results")sys.path.insert(0, REPO) # `import laya` works without an installed packageimport laya # noqa: E402@@ -35,7 +37,7 @@ ROOT = os.path.expanduser("~/laya_models")MODELS = {"english": os.path.join(ROOT, "laya"),"multilingual": os.path.join(ROOT, "laya-multilingual"),"typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}-OUT = os.path.join(RESULTS_DIR, "latency_benchmark.json")+OUT = os.path.join(REPO, "research", "latency_benchmark_results.json")STATE_EN = {"ticket": {"subject": "Payout failing", "messages": [{"from": "customer","text": "Hi, my Stripe payouts have failed for 3 days and I am losing sales. Please help ASAP. " * 6}]}}diff --git a/research/scripts/bench_local.py b/research/scripts/bench_local.pyindex a39ee872..356af50f 100644--- a/research/scripts/bench_local.py+++ b/research/scripts/bench_local.py@@ -5,7 +5,11 @@ Part B typed-decisions (400 cases / 2,000 decisions) on all three checkpoints,fine-tuned laya-typed-decisions can be compared with Jev's published 0.727 on thesame benchmark.-Writes research/results/cpu_51_language_sweep.json.+Writes research/local_benchmark_results.json (fresh run, gitignored).++The committed CPU sweep behind the BENCHMARKS tables is+research/results/cpu_51_language_sweep.json — this script must not+overwrite it, so the default output stays on the gitignored name.USE_TF=0 python3 research/scripts/bench_local.py [--langs N] [--per-lang N] [--skip-a] [--skip-b]"""@@ -27,7 +31,6 @@ import numpy as np # noqa: E402import torch # noqa: E402REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))-RESULTS_DIR = os.path.join(REPO, "research", "results")sys.path.insert(0, REPO) # `import laya` works without an installed packageimport laya # noqa: E402@@ -37,7 +40,7 @@ ROOT = os.path.expanduser("~/laya_models")MODELS = {"english": os.path.join(ROOT, "laya"),"multilingual": os.path.join(ROOT, "laya-multilingual"),"typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}-OUT = os.path.join(RESULTS_DIR, "cpu_51_language_sweep.json")+OUT = os.path.join(REPO, "research", "local_benchmark_results.json")SEED, N_OPTS = 13, 20