Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

13.3 KB
Raw Blame

Research

Benchmark harnesses and raw results for the Laya checkpoints. This branch is the evidence behind the numbers quoted in the main README — nothing here is imported by the laya package.

Evaluation

  • eval/: the independent per-language MASSIVE harness (laya_eval.py), the per-case report behind the published numbers, plus the metamorphic and presentation checks.
  • evals/: the labelled datasets, thresholds and regression gate consumed by laya.evals / laya-evals and the scheduled evals workflow.

Community diagnostics

  • Chinese workplace decisions (Feishu-style) — 64 synthetic scenarios, paired recorded Laya/Jev responses, English/Chinese cards, and a model-free audit. 中文入口. Start with python research/benchmarks/feishu_zh/audit.py; no downloads or API keys required. This is a contributed historical snapshot, separate from the upstream sweeps below.
  • Chinese short-command routing — 18 frozen Chinese voice commands, a seven-rung ablation of the documented prompt guidance on both the six-way choice path and the four-question noul path, and every per-case decision archived. 中文入口. Start with python research/benchmarks/zh_short_commands/audit.py; the audit needs no downloads and the archive records which weights produced the numbers.

Scripts

file what it does
scripts/laya_benchmark_colab.ipynb the full head-to-head on a Colab T4: typed-decisions, MASSIVE intent + scenario (14 languages), XNLI (15), English suites, latency, option-order robustness, calibration repair. Writes one JSON.
scripts/build_benchmark_nb.py generator for that notebook (edit here, not the .ipynb)
scripts/bench_local.py CPU sweep: MASSIVE intent across all 51 languages, plus typed-decisions on all three checkpoints
scripts/bench_apps.py the six application workflows (support triage, email + phishing, guardrails, RAG relevance, moderation, model routing) plus the datasets where public Jev numbers exist
scripts/bench_latency.py inference speed including what routing costs: detection overhead, hot path, cold-swap, mixed-language throughput at several max_loaded settings
scripts/bench_length_batching.py compare upstream contiguous batches with optional length sorting on synthetic tickets, including output consistency and optional fresh-process memory profiles (psutil required for memory mode)
scripts/bench_nvidia_capacity.py laya as a dynamically batched HTTP service on one NVIDIA GPU: open-loop Poisson sweep to the highest rate meeting a p99 SLO, 24-hour curve replay, and the ONNX export the TensorRT rows need (TensorRT via ONNX Runtime, not a laya backend). Results: results/nvidia_capacity_20260925.json
scripts/make_plots.py renders assets/laya_benchmark.png from the result JSONs
scripts/bench_long_context.py laya-multilingual on long documents: 20 support requests in 8 languages, each placed after 0 to 7,000 tokens of unrelated text, scored at the default limit and at max_len=8192
scripts/bench_position_sensitivity.py the same 20 requests, filler and question, with the request swept through a fixed-length document instead of only appended to it, so the effect of evidence position is separable from document length
scripts/plot_long_context.py renders assets/long_context_8192.png from results/long_context_multilingual.json

Everything runs with USE_TF=0 — transformers probes for TensorFlow at import, and when TF is installed its abseil runtime can deadlock model construction on macOS/Python 3.9.

Running the harnesses

The checkpoints come from LAYA_MODELS (default ~/laya_models, a directory holding laya/, laya-multilingual/ and laya-typed-decisions/), so a checkout can point them at a local models/ copy:

# from the repository root
LAYA_MODELS=$PWD/models python research/scripts/bench_local.py --langs 10 --per-lang 60 --skip-b
LAYA_MODELS=$PWD/models BENCH_N=80 python research/scripts/bench_apps.py
LAYA_MODELS=$PWD/models python research/scripts/bench_latency.py
option effect
bench_local.py --langs N caps the language list (sorted, so 10 is the first ten codes; 0 = all 51)
bench_local.py --per-lang N cases per language (default 120)
bench_local.py --skip-a / --skip-b run only the typed-decisions half, or only the MASSIVE half
bench_apps.py with BENCH_N cases per application suite (default 400)

bench_local.py and bench_apps.py write research/*_benchmark_results.json, which is gitignored — the curated runs live in results/. Set HF_HOME inside the checkout to keep the dataset cache local.

bench_local.py and bench_latency.py take the model root from LAYA_MODELS rather than assuming ~/laya_models, so the harnesses run against a checkout that keeps its weights under models/.

Length batching

Local CPU measurements compare the original predict_batch method at 1e28ac20c0896b1c37a744cd11f740eb98f8b178 with sort_by_length=True. On 10,000 synthetic English support tickets, the multilingual checkpoint took 1775.71 s before and 825.09 s after (2.15x), including tokenization, collation, inference and decoding. There were zero choice/action decision changes, zero usage mismatches, and a maximum returned numeric difference of 0.0001. This is a throughput workload, not a labeled accuracy test.

The machine was a Ryzen 9 5950X with 64 GiB RAM, Windows 11, CPU float32 and 16 Torch threads. An unrelated adaptation experiment was running on the GPU. Small tests alternate execution order over repeated rounds; the 1,000 and 10,000 input tests each have one full timing pair. The similar-length control showed no regression. GPU, MPS, compiled and TileLang performance were not measured. Results depend on input lengths, checkpoint, batch size and hardware.

To reproduce, use a fresh Python 3.12 environment and the pinned model snapshot:

python -m pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cpu
python -m pip install transformers==4.57.1 huggingface_hub==0.36.2 numpy==2.3.5 psutil==7.2.2
python -c "from huggingface_hub import snapshot_download; print(snapshot_download('convaiinnovations/laya', revision='aa8c91ca088ec597df95a0d1c76b3063cb2ae5e8', allow_patterns=['multilingual/*']) + '/multilingual')"

Replace MODEL_DIR below with the printed checkpoint directory. Start with --count 16 --rounds 2 --questions 3, then increase the count. The full CPU comparison takes tens of minutes:

python research/scripts/bench_length_batching.py MODEL_DIR --count 10000 --batch-size 8 --threads 16 --rounds 1 --output length-10000.json

Memory measurements run each mode in a separate process after model warm-up. They sample process RSS every 10 ms during 64-input inference, excluding model-load transients; short allocation peaks can be missed. They do not measure the peak memory of the 10,000-input run:

python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-size 8 --threads 16 --memory-mode original --output memory-original.json
python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-size 8 --threads 16 --memory-mode grouped --output memory-grouped.json

Results

file contents
results/t4_colab_benchmark.json 17,416 questions on one T4, both checkpoints, identical questions per model
results/long_context_multilingual.json the long-document run behind assets/long_context_8192.png: every prediction, with device and library versions
results/cpu_51_language_sweep.json 51 languages x 2 checkpoints, MASSIVE intent, 20 options (committed part_b covers the English checkpoint only; a rerun of research/scripts/bench_local.py writes the gitignored research/local_benchmark_results.json and leaves this file untouched)
results/cpu_51_language_sweep_clamped.json the same 51 languages and 5,100 cases re-run after the temperature clamp, raw temperatures and served temperatures side by side (#208)
results/cpu_51_language_sweep_refreshed.json both checkpoints with the environment recorded, the superseded committed columns kept beside the new ones. The multilingual half of the committed sweep does not reproduce on current code, so this is what the table in BENCHMARKS.md prints (#208)
results/app_benchmark_results.json Applications run (400 cases per task, seed 13, 2026-09-19); source for the BENCHMARKS Themes / Jev-dataset tables and the README routed AG News / DAIR Emotion cells (a rerun of research/scripts/bench_apps.py writes the gitignored research/app_benchmark_results.json instead)
research/latency_benchmark_results.json not committed (gitignored) — generate with research/scripts/bench_latency.py; CPU routing-cost regimes behind the README preload table
results/position_sensitivity_multilingual.json accuracy against evidence position at fixed document length, n=20 per cell, every prediction kept

Headline findings

Position matters as much as length, and bench_long_context.py samples exactly one position. That script always appends the request to the end of the filler, so it measures a single point on the position axis. Sweeping the request through a fixed-length document instead (bench_position_sensitivity.py, the same 20 requests and the same construction, so position=1.0 is the cell bench_long_context.py reports) gives, at max_len=8192 with nothing truncated:

filler tokens pos 0.00 pos 0.25 pos 0.50 pos 0.75 pos 1.00 spread
4,000 0.850 0.550 0.650 0.600 0.950 0.400
7,000 0.850 0.550 0.500 0.650 0.400 0.450

The majority-class rate of this item set is 0.450 (billing 9, technical 7, sales 4, other 0) — so the interior of the document sits at the floor, where a model that always answered billing would score the same. A request placed next to the option markers (pos 0.00) or, at 4,000 tokens, at the very end (pos 1.00), is answered at 0.85–0.95.

Which end is best depends on length, so a pad-only sweep cannot see the shape at all: pos 1.00 is the best cell at 4,000 tokens (0.950) and the worst at 7,000 (0.400). Paired on the same 20 items, the interior is significantly worse than an endpoint in both cases — pos 1.00 vs pos 0.25 at 4,000 tokens is 0 wrong→right / 8 right→wrong (exact McNemar p=0.0078), and pos 0.00 vs pos 0.25 at 7,000 is 0 / 6 (p=0.031).

Caveat: n=20 per cell, so each cell carries a ±0.22 95 % interval and the ordering of the interior cells is not resolvable. The endpoint-versus-interior gap is well outside that, and the paired tests above are on the same items. Read this as "position is a large effect that the existing sweep does not sample", not as a per-cell ranking.

Routing takes Laya from 23 to 48 of 51 languages. On MASSIVE intent (20 options, random = 0.050) the English checkpoint macro-averages 0.227 and clears 3x random on 23 of 51 languages; the multilingual checkpoint reaches 0.401 and clears it on 48. The multilingual figures are the re-run, not the committed sweep — see results/cpu_51_language_sweep_refreshed.json above.

The English checkpoint's confidence gives no warning when it cannot read the input. Khmer: 0.000 accuracy at 0.952 mean confidence — that is the raw temperature; as served, after the #42 clamp, it is 0.705 at the same 0.000 accuracy. Macro ECE 0.733 across 51 languages as committed (0.571 as served), with mean confidence never dropping below 0.885 raw, or 0.621 as served, at any accuracy level. This is why routing has to happen before the forward pass — confidence gating cannot catch it.

Both checkpoints ship over-confident. Refitting one temperature per (question type, option count) on held-out data moves mean ECE 0.466 -> 0.081 (laya) and 0.314 -> 0.106 (laya-multilingual, which ships with no fitted temperatures at all).

The base checkpoints are near chance on typed-decisions zero-shot — 0.362 and 0.352 against a 0.318 random baseline and a 0.461 majority-class baseline. The published 0.766 belongs to the checkpoint fine-tuned on that benchmark's own training split.

Speed. 32.8 ms for one question and 7.2 ms/question at batch 10 on a T4; 103–332 questions/s batched.

On comparisons with Jev

For the original upstream suites listed above, Jev was not run directly. Their Jev figures are third-party published, with different sample sizes and prompts. The separate community diagnostic linked above includes paired API responses and documents its own limitations:

  • AbdelStark/jev-benchmarks — AG News 0.910, Banking77 0.870, DAIR Emotion 0.480 (Brier 0.846, NLL 5.588, zero probability on the true label for 16% of examples)
  • nibzard/decision-model-benchmark — ECE 0.246 (worst in that study), banking77 0.763, option-order flip rate 13%, latency 264–276 ms p50

Treat those as indicative, not a controlled head-to-head.