Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(benchmarks): keep bench defaults on gitignored fresh-run paths (#134)

Nafeel005314cd5bafcd1
diff --git a/research/README.md b/research/README.md
index 17caeea5..a820a860 100644
--- a/research/README.md
+++ b/research/README.md
@@ -69,10 +69,10 @@ python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-si
|---|---|
| `results/t4_colab_benchmark.json` | 17,416 questions on one T4, both checkpoints, identical questions per model |
| `results/long_context_multilingual.json` | the long-document run behind `assets/long_context_8192.png`: every prediction, with device and library versions |
-| `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options (committed `part_b` covers the English checkpoint only; rerun `research/scripts/bench_local.py` for all three) |
+| `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options (committed `part_b` covers the English checkpoint only; a rerun of `research/scripts/bench_local.py` writes the gitignored `research/local_benchmark_results.json` and leaves this file untouched) |
| `results/cpu_51_language_sweep_clamped.json` | the same 51 languages and 5,100 cases re-run after the temperature clamp, raw temperatures and served temperatures side by side ([#208](https://github.com/NandhaKishorM/laya/issues/208)) |
-| `results/app_benchmark_results.json` | Applications run (400 cases per task, seed 13, 2026-09-19); source for the BENCHMARKS Themes / Jev-dataset tables and the README routed AG News / DAIR Emotion cells |
-| `results/latency_benchmark.json` | **not yet committed** — generate with `research/scripts/bench_latency.py`; CPU routing-cost regimes behind the README preload table |
+| `results/app_benchmark_results.json` | Applications run (400 cases per task, seed 13, 2026-09-19); source for the BENCHMARKS Themes / Jev-dataset tables and the README routed AG News / DAIR Emotion cells (a rerun of `research/scripts/bench_apps.py` writes the gitignored `research/app_benchmark_results.json` instead) |
+| `research/latency_benchmark_results.json` | **not committed** (gitignored) — generate with `research/scripts/bench_latency.py`; CPU routing-cost regimes behind the README preload table |
## Headline findings
diff --git a/research/scripts/bench_apps.py b/research/scripts/bench_apps.py
index cc526244..c54d2746 100644
--- a/research/scripts/bench_apps.py
+++ b/research/scripts/bench_apps.py
@@ -12,6 +12,12 @@ Workflows (the demo Space's tabs), each on real labelled data:
Jev-comparable tasks (AbdelStark/jev-benchmarks published Jev accuracy on these):
ag_news 0.910 | banking77 0.870 | dair emotion 0.480 (Brier 0.846, NLL 5.588)
+Writes research/app_benchmark_results.json (fresh run, gitignored).
+
+The committed Applications run behind the BENCHMARKS Themes / Jev-dataset
+tables is research/results/app_benchmark_results.json — this script must not
+overwrite it, so the default output stays on the gitignored name.
+
USE_TF=0 python3 research/scripts/bench_apps.py
"""
import gc
@@ -30,14 +36,13 @@ import numpy as np # noqa: E402
import torch # noqa: E402
REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-RESULTS_DIR = os.path.join(REPO, "research", "results")
sys.path.insert(0, REPO) # `import laya` works without an installed package
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import laya # noqa: E402
from bench_local import load, metrics, score_cases, softmax_t, temp_for # noqa: E402
-OUT = os.path.join(RESULTS_DIR, "app_benchmark_results.json")
+OUT = os.path.join(REPO, "research", "app_benchmark_results.json")
SEED = 13
N = int(os.environ.get("BENCH_N", "400"))
diff --git a/research/scripts/bench_latency.py b/research/scripts/bench_latency.py
index 105cf501..7bd7d98a 100644
--- a/research/scripts/bench_latency.py
+++ b/research/scripts/bench_latency.py
@@ -7,6 +7,9 @@ Routing changes the latency picture in two ways:
So there are two regimes -- hot (right model already loaded) and cold (swap) -- and the realistic
number for a mixed-language workload sits between them, determined by `max_loaded`.
+Writes research/latency_benchmark_results.json (fresh run, gitignored;
+the name tests/test_doc_tables.py expects).
+
USE_TF=0 python3 research/scripts/bench_latency.py
"""
import gc
@@ -24,7 +27,6 @@ import numpy as np # noqa: E402
import torch # noqa: E402
REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-RESULTS_DIR = os.path.join(REPO, "research", "results")
sys.path.insert(0, REPO) # `import laya` works without an installed package
import laya # noqa: E402
@@ -35,7 +37,7 @@ ROOT = os.path.expanduser("~/laya_models")
MODELS = {"english": os.path.join(ROOT, "laya"),
"multilingual": os.path.join(ROOT, "laya-multilingual"),
"typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}
-OUT = os.path.join(RESULTS_DIR, "latency_benchmark.json")
+OUT = os.path.join(REPO, "research", "latency_benchmark_results.json")
STATE_EN = {"ticket": {"subject": "Payout failing", "messages": [{"from": "customer",
"text": "Hi, my Stripe payouts have failed for 3 days and I am losing sales. Please help ASAP. " * 6}]}}
diff --git a/research/scripts/bench_local.py b/research/scripts/bench_local.py
index a39ee872..356af50f 100644
--- a/research/scripts/bench_local.py
+++ b/research/scripts/bench_local.py
@@ -5,7 +5,11 @@ Part B typed-decisions (400 cases / 2,000 decisions) on all three checkpoints,
fine-tuned laya-typed-decisions can be compared with Jev's published 0.727 on the
same benchmark.
-Writes research/results/cpu_51_language_sweep.json.
+Writes research/local_benchmark_results.json (fresh run, gitignored).
+
+The committed CPU sweep behind the BENCHMARKS tables is
+research/results/cpu_51_language_sweep.json — this script must not
+overwrite it, so the default output stays on the gitignored name.
USE_TF=0 python3 research/scripts/bench_local.py [--langs N] [--per-lang N] [--skip-a] [--skip-b]
"""
@@ -27,7 +31,6 @@ import numpy as np # noqa: E402
import torch # noqa: E402
REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-RESULTS_DIR = os.path.join(REPO, "research", "results")
sys.path.insert(0, REPO) # `import laya` works without an installed package
import laya # noqa: E402
@@ -37,7 +40,7 @@ ROOT = os.path.expanduser("~/laya_models")
MODELS = {"english": os.path.join(ROOT, "laya"),
"multilingual": os.path.join(ROOT, "laya-multilingual"),
"typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}
-OUT = os.path.join(RESULTS_DIR, "cpu_51_language_sweep.json")
+OUT = os.path.join(REPO, "research", "local_benchmark_results.json")
SEED, N_OPTS = 13, 20