Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(research): let the bench harnesses take their model root from LAYA_MODELS

thread3dc7958dbf1602
diff --git a/research/README.md b/research/README.md
index 9ce68d7c..7b8cbcf3 100644
--- a/research/README.md
+++ b/research/README.md
@@ -32,6 +32,33 @@ the numbers quoted in the main README — nothing here is imported by the `laya`
Everything runs with `USE_TF=0` — `transformers` probes for TensorFlow at import, and when TF is
installed its abseil runtime can deadlock model construction on macOS/Python 3.9.
+## Running the harnesses
+
+The checkpoints come from `LAYA_MODELS` (default `~/laya_models`, a directory holding `laya/`,
+`laya-multilingual/` and `laya-typed-decisions/`), so a checkout can point them at a local
+`models/` copy:
+
+```bash
+# from the repository root
+LAYA_MODELS=$PWD/models python research/scripts/bench_local.py --langs 10 --per-lang 60 --skip-b
+LAYA_MODELS=$PWD/models BENCH_N=80 python research/scripts/bench_apps.py
+LAYA_MODELS=$PWD/models python research/scripts/bench_latency.py
+```
+
+| option | effect |
+|---|---|
+| `bench_local.py --langs N` | caps the language list (sorted, so `10` is the first ten codes; `0` = all 51) |
+| `bench_local.py --per-lang N` | cases per language (default 120) |
+| `bench_local.py --skip-a` / `--skip-b` | run only the typed-decisions half, or only the MASSIVE half |
+| `bench_apps.py` with `BENCH_N` | cases per application suite (default 400) |
+
+`bench_local.py` and `bench_apps.py` write `research/*_benchmark_results.json`, which is
+gitignored — the curated runs live in `results/`. Set `HF_HOME` inside the checkout to keep the
+dataset cache local.
+
+`bench_local.py` and `bench_latency.py` take the model root from `LAYA_MODELS` rather than assuming
+`~/laya_models`, so the harnesses run against a checkout that keeps its weights under `models/`.
+
## Length batching
[Local CPU measurements](results/length_batching_cpu_20260924.json) compare the original
diff --git a/research/scripts/bench_latency.py b/research/scripts/bench_latency.py
index 43dec911..b2207d65 100644
--- a/research/scripts/bench_latency.py
+++ b/research/scripts/bench_latency.py
@@ -30,7 +30,7 @@ import laya # noqa: E402
from laya.lang import analyse # noqa: E402
from laya.router import Router # noqa: E402
-ROOT = os.path.expanduser("~/laya_models")
+ROOT = os.environ.get("LAYA_MODELS", os.path.expanduser("~/laya_models"))
MODELS = {"english": os.path.join(ROOT, "laya"),
"multilingual": os.path.join(ROOT, "laya-multilingual"),
"typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}
diff --git a/research/scripts/bench_local.py b/research/scripts/bench_local.py
index 3a3958fe..7babd5a4 100644
--- a/research/scripts/bench_local.py
+++ b/research/scripts/bench_local.py
@@ -32,7 +32,7 @@ sys.path.insert(0, os.path.dirname(REPO))
import laya # noqa: E402
from laya.common import QTYPES, build_sequence, collate_items, render_options, temp_bucket # noqa: E402
-ROOT = os.path.expanduser("~/laya_models")
+ROOT = os.environ.get("LAYA_MODELS", os.path.expanduser("~/laya_models"))
MODELS = {"english": os.path.join(ROOT, "laya"),
"multilingual": os.path.join(ROOT, "laya-multilingual"),
"typed-decisions": os.path.join(ROOT, "laya-typed-decisions")}