Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

35.3 KB
Raw Blame

Laya benchmarks

Every checkpoint answered byte-identical questions in each run (fixed seed). Jev figures are third-party published, never measured here — no TypeSafe API access — so sample sizes and prompts differ; treat them as indicative.

run what where
T4 Colab typed-decisions, MASSIVE (14 langs), XNLI (15 langs), English suites, latency, option-order robustness, calibration repair research/results/t4_colab_benchmark.json
CPU sweep MASSIVE intent across all 51 languages; its typed-decisions part (part_b) covers the English checkpoint only research/results/cpu_51_language_sweep.json
Applications the seven workflow themes + the datasets where Jev numbers exist, all three checkpoints (laya 0.2.1, CPU, 400 cases per task, seed 13, 2026-09-19) research/results/app_benchmark_results.json
NVIDIA capacity laya served with dynamic batching under p99 SLOs on RTX PRO 5000, RTX PRO 6000 and H100 NVL (eager FP16, TensorRT FP16 via ONNX Runtime), backend parity, 24-hour replay research/results/nvidia_capacity_20260925.json

Calibration columns in the CPU sweep predate the temperature clamp. The 51-language ECE and mean-confidence figures were produced before #42 clamped temperatures to [0.5, 5], so today's package reports different confidence for the affected buckets. Accuracy columns are unaffected, because a temperature-scaled softmax has the same argmax at every positive temperature.

The same 51 languages and 5,100 cases have now been re-run after the temperature clamp, in both regimes (research/results/cpu_51_language_sweep_clamped.json, #208). Macro accuracy reproduces at 0.2269 exactly, and macro ECE moves 0.7331 → 0.5709:

committed re-run, raw temperatures re-run, as served
macro accuracy 0.2269 0.2269 0.2269
macro ECE 0.7331 0.7331 0.5709
macro F1 0.2053 0.2053 0.2053
mean confidence, en 0.9989 0.9989 0.9582
ECE, en 0.1789 0.1789 0.1382

The raw-temperature column reproduces the committed file, so the only variable left is the clamp. choice:11+ is the sole bucket it moves, and every case in this sweep is a 20-option question, so the clamp applies to all 5,100 — and lowers ECE in all 51 languages. acc_at_50_coverage is the one rank-quality column that uses the confidence values: macro 0.3004 → 0.3020, and en 0.94 → 0.98, so the flatter distribution selects a slightly better half rather than a worse one.

The multilingual columns below are the re-run, not the committed file. The committed cpu_51_language_sweep.json was measured at laya 0.2.0, and its laya-multilingual half does not reproduce on current code — 51 languages and 5,100 cases re-run under laya 0.2.0, 0.3.7 and 0.3.20, on x86 CPU and on an Apple M5 under both CPU and MPS, all give 0.4008 macro accuracy and match the committed file on only 6 of 51 languages, while the laya half reproduces exactly on all of them. Three machines agreeing settled it (#208). The table therefore prints the refreshed numbers for both checkpoints, from research/results/cpu_51_language_sweep_refreshed.json, which records the environment and keeps the superseded committed columns beside the new ones:

committed (laya-multilingual) refreshed
macro accuracy 0.3661 0.4008
macro ECE 0.3869 0.3911
languages clearing 3× random 45 / 51 48 / 51

The direction is consistent rather than noise: of the 16 languages that move by 0.05 or more, 15 move up (bn 0.29 → 0.45, kn 0.15 → 0.30, hy 0.15 → 0.25) and sv is the one that moves down (0.57 → 0.49). laya's columns are unchanged by the refresh — they reproduce the committed file to the last stored digit — so the accuracy spread between the two checkpoints is wider than the committed table suggested, and en remains the one language where laya wins (0.820 against 0.710).


Headline

Laya Jev (published)
typed-decisions (2,000 decisions) 0.766 0.727
AG News (4 labels) 0.953 0.910
DAIR Emotion (6 labels) 0.600 0.480
ECE after temperature fitting 0.081 0.246
p50 latency, 1 question (T4) 32.8 ms 236-276 ms

Headline Laya cells for AG News and DAIR Emotion are the Applications-run numbers (research/results/app_benchmark_results.json); typed-decisions 0.766 is the fine-tuned checkpoint (no committed result file behind it yet). The committed T4 English suites give 0.947 (AG News) and 0.573 (DAIR Emotion) for laya.


Languages

All 51 MASSIVE languages — intent, 20 options (random = 0.050)

laya laya-multilingual
macro accuracy 0.2269 0.4008
macro ECE (lower better) 0.5709 0.3911
languages clearing 3x random 23 / 51 48 / 51
Per language (51) — sorted by how much routing gains
lang laya laya-multilingual Δ laya ECE multilingual ECE
th 0.080 0.480 +0.400 0.718 0.356
bn 0.080 0.450 +0.370 0.627 0.354
fa 0.140 0.510 +0.370 0.633 0.298
hi 0.100 0.460 +0.360 0.642 0.368
ko 0.110 0.470 +0.360 0.680 0.311
ar 0.110 0.460 +0.350 0.580 0.315
ur 0.070 0.420 +0.350 0.643 0.336
el 0.130 0.440 +0.310 0.679 0.430
he 0.060 0.370 +0.310 0.738 0.419
vi 0.060 0.340 +0.280 0.690 0.464
hu 0.090 0.360 +0.270 0.678 0.401
az 0.100 0.360 +0.260 0.598 0.381
pl 0.240 0.500 +0.260 0.570 0.353
tr 0.140 0.400 +0.260 0.587 0.400
ru 0.310 0.570 +0.260 0.595 0.296
is 0.110 0.350 +0.240 0.664 0.452
nb 0.330 0.560 +0.230 0.518 0.294
fi 0.130 0.340 +0.210 0.719 0.392
ml 0.070 0.280 +0.210 0.601 0.443
lv 0.100 0.310 +0.210 0.666 0.472
hy 0.050 0.250 +0.200 0.571 0.513
km 0.000 0.200 +0.200 0.705 0.445
kn 0.110 0.300 +0.190 0.590 0.388
ta 0.120 0.310 +0.190 0.554 0.448
it 0.340 0.520 +0.180 0.516 0.343
da 0.350 0.520 +0.170 0.526 0.315
sl 0.200 0.370 +0.170 0.599 0.434
id 0.360 0.510 +0.150 0.487 0.323
zh-TW 0.460 0.610 +0.150 0.434 0.266
jv 0.160 0.300 +0.140 0.670 0.484
ms 0.270 0.410 +0.140 0.506 0.436
te 0.090 0.220 +0.130 0.669 0.497
ja 0.530 0.640 +0.110 0.366 0.234
sv 0.380 0.490 +0.110 0.487 0.341
my 0.060 0.160 +0.100 0.580 0.477
sw 0.130 0.230 +0.100 0.632 0.492
sq 0.210 0.300 +0.090 0.603 0.487
de 0.420 0.500 +0.080 0.452 0.336
nl 0.390 0.470 +0.080 0.502 0.397
es 0.510 0.580 +0.070 0.422 0.258
af 0.290 0.350 +0.060 0.559 0.439
ka 0.090 0.150 +0.060 0.610 0.542
tl 0.290 0.350 +0.060 0.512 0.430
ro 0.330 0.370 +0.040 0.550 0.397
pt 0.470 0.500 +0.030 0.432 0.341
zh-CN 0.620 0.650 +0.030 0.320 0.219
am 0.120 0.150 +0.030 0.680 0.484
mn 0.130 0.160 +0.030 0.670 0.577
cy 0.120 0.130 +0.010 0.645 0.587
fr 0.590 0.600 +0.010 0.299 0.248
en 0.820 0.710 -0.110 0.138 0.233

English vs the rest

Source: the T4 Colab run (research/scripts/laya_benchmark_colab.ipynb, seed 13, 20 options per choice question, the first 300 test rows per language) — research/results/t4_colab_benchmark.json. "Other languages" is the average over the 13 non-English MASSIVE languages and the 14 non-English XNLI languages; each language contributed exactly 300 questions, so averaging the languages and pooling the questions give the same figure. tests/test_doc_tables.py checks every cell of this table, and both copies in README.md, against the artifact.

task laya laya-multilingual
MASSIVE intent — English 0.783 0.657
MASSIVE intent — other languages 0.306 0.451
MASSIVE scenario — English 0.603 0.560
MASSIVE scenario — other languages 0.281 0.439
XNLI — English 0.860 0.843
XNLI — other languages 0.521 0.731

The English checkpoint does not degrade gracefully outside English — it collapses, and stays confident doing so. Khmer: 0.000 accuracy at 0.952 confidence raw, 0.705 as served after the clamp. Its mean confidence never drops below 0.885 (raw; 0.621 as served) at any accuracy level, so confidence gating cannot catch it — which is why routing happens before the forward pass.


Themes — the application workflows

Each is real labelled data, 400 cases, all three checkpoints. held out means the source was not in Laya's training mix. Source: the Applications run (research/scripts/bench_apps.py, N=400 per task) — research/results/app_benchmark_results.json.

theme laya laya-multilingual laya-typed-decisions data
Email spam 0.993 0.993 0.958 in training
Phishing 0.980 0.993 0.940 in training
LLM guardrails (jailbreak) 0.708 0.755 0.762 held out
Moderation (toxicity) 0.530 0.525 0.530 held out
RAG passage relevance 0.625 0.657 0.625 in training
Support triage (10-way queue) 0.502 0.522 0.505 in training
Model routing (domain) 0.639 0.123 0.659 held out

Where it is strong: email spam 0.993 and phishing 0.993, both with ECE around 0.01 — production-grade, though both were in the training mix.

Where it is weak: moderation on held-out toxic-chat is 0.530 with macro-F1 0.400 — barely above chance on a balanced split. The demo Space has a Moderation tab; hand-picked examples work, real traffic does not. Guardrails at 0.708–0.762 is the honest jailbreak-detection number, consistent across two unrelated datasets (deepset prompt-injections measured 0.698 separately).

On the public datasets where Jev numbers exist

Laya columns are from the same Applications run (research/results/app_benchmark_results.json, N=400 per task), so AG News / DAIR Emotion differ slightly from the committed T4 English suites above (N=600: 0.947 / 0.573 for laya).

dataset laya laya-multilingual laya-typed-decisions Jev (published)
AG News (4 labels) 0.950 0.930 0.953 0.910
DAIR Emotion (6 labels) 0.595 0.530 0.600 0.480
banking77 (77 labels) 0.425 0.425 0.492 0.870

banking77 is the one clear loss, and it is architectural: a choice question's options share a fixed head_max_len budget, so 77 labels get roughly 4 tokens each and stop being distinguishable. Both checkpoints score exactly 0.425, which is what you would expect from a budget ceiling rather than a capability gap. Keep choice questions under ~20 options.


typed-decisions — 400 cases, 2,000 decisions

model accuracy soft acc Brier ECE score MAE
laya-typed-decisions 0.766 0.471 0.061 0.213 0.242
laya 0.362 0.331 0.315 0.174 0.694
laya-multilingual 0.352 0.328 0.463 0.314 0.760
Jev 1.13.0 (published) 0.727 0.580 0.148 0.144 0.391
teacher ceiling 0.735 — — — —
majority class 0.461 — — — —
random guess 0.318 — — — —
workflow laya-typed-decisions
agent trace observability 0.730
customer service 0.764
invoice processing 0.804
security incidents 0.766

The base checkpoints sit below the majority-class baseline (0.362 and 0.352 against 0.461). All of the capability on this benchmark comes from fine-tuning. Base rows: suites.typed_decisions in research/results/t4_colab_benchmark.json (0.3620 / 0.3515; CPU sweep part_b gives 0.3615 for English). The laya-typed-decisions row and the four per-workflow scores are from the fine-tuning run and have no committed result file behind them yet.


Speed (Tesla T4)

questions per call laya laya-multilingual
1 39.5 ms 32.8 ms
5 84.5 ms 40.1 ms
10 158.6 ms 72.3 ms
50 771.3 ms 337.4 ms

103–332 questions/sec batched. Jev independently measured at 236-276 ms p50, so Laya answers one question roughly 6–7× faster.

Calibration

as shipped temperature refit
laya 0.466 0.081
laya-multilingual 0.314 0.106

Both ship over-confident; laya-multilingual ships with no fitted temperatures at all. Refitting one temperature per (question type, option count) on held-out data is the single highest-value fix available, and takes ECE below Jev's measured 0.246.

Option-order robustness

How often the answer changes when the options are permuted. Jev measured at 0.13.

suite laya laya-multilingual
massive_intent.en 0.150 0.230
en.emotion 0.040 0.090
xnli.en 0.000 0.015

At 20 options both are less order-stable than Jev — worth fixing with more aggressive option-order shuffling during training.

Other hardware: GB10, a laptop CPU, and an Intel Arc

Contributed measurements from a router deployment (laya 0.3.5). They were taken through a small HTTP server wrapping Agent.system_one, not in-process, so every figure includes one HTTP round trip.

NVIDIA GB10 (DGX Spark, aarch64), CUDA

typed-decisions checkpoint (1024 ctx), default dtype, torch 2.14.0+cu130. The GPU was shared with a resident 73 GB SGLang server and a whisper server. Each question is a 3-option choice, with 40 calls per row after warm-up. Loopback round trip to /health was 0.6 ms, so network is not in these numbers.

questions per call p50 p95
1 100.2 ms 169.3 ms
5 137.7 ms 162.4 ms
10 159.3 ms 243.0 ms
50 443.1 ms 464.6 ms

Each extra question costs about 7.0 ms, half the T4's ~14.9 ms. But one question is slower than the T4's 39.5 ms, because roughly 93 ms per call is fixed overhead that the GPU does not remove. We have not isolated where that overhead goes. On a GB10, batching questions into one call is where the speedup is.

On laya_router's 180 labelled requests (one tier question), accuracy on CUDA matched CPU to within one row per wording (0.700 vs 0.694, 0.656 vs 0.656, 0.611 vs 0.606). That is backend floating-point noise, not a change in behaviour.

Setup note for aarch64 without root: Triton JIT-compiles a CUDA shim with gcc on the first CUDA call, which fails with Python.h: No such file or directory if python3-dev is absent. Fetch the headers with apt-get download libpython3.12-dev python3.12-dev, unpack with dpkg-deb -x into a directory, and set CPATH to both usr/include and usr/include/python3.12 under it.

Laptop CPU (Ryzen 9 6900HX, avx2 only, WSL2)

Pin inter-op threads to 1. system_one runs one forward pass per call, so there is nothing for inter-op parallelism to overlap. On a three-question call over HTTP, on a busy host, torch's defaults (10 intra-op, 5 inter-op on 10 vCPUs) gave p50 9,396 ms. torch.set_num_threads(8) plus torch.set_num_interop_threads(1) brought it to 783 ms, 12x faster with no code change.

With inter-op pinned, one question in-process on a quieter host:

intra-op threads p50 p95
1 910 ms 1,023 ms
4 374 ms 552 ms
8 329 ms 378 ms
10 (every vCPU) 388 ms 708 ms

The best setting is the physical core count plus a little, not one thread per vCPU. SMT siblings contend.

Intel Arc B390 (torch 2.14.0+xpu), XPU — before/after vs CPU

english checkpoint (421M, ModernBERT-large), in-process agent.predict(), one 2-option choice question (~90 tokens), 40 calls per row after 5 warm-ups. XPU row at the default bf16 with autocast (XPU autocast supports bf16/fp16 only); CPU row fp32, pinned as recommended above (intra-op 8, inter-op 1). Rows measured on the same laptop; CPU rows are stable across sessions (p95 within ~10% of p50).

questions per call CPU p50 XPU p50 XPU p95 speedup (p50)
1 288.2 ms 29.7 ms 30.4 ms 9.7x
3 730.6 ms 45.4 ms 46.8 ms 16.1x
10 2608.7 ms 96.9 ms 102.5 ms 26.9x

CPU scales roughly linearly with question count (288.2 -> 2608.7 ms, 9.1x for 10x the questions), while XPU scales sub-linearly (29.7 -> 96.9 ms, 3.3x), so the speedup widens from ~10x to ~27x. The XPU p95 stays within ~6% of its p50 on every row (30.4, 46.8, 102.5). At one question the Arc B390 is slightly faster than the T4's 32.8 ms p50 above.

Apple M1 Pro, MPS — fp16 autocast against fp32

On MPS, Agent autocasts a forward to fp16 at or above mps_amp_min_rows question rows (default 5, LAYA_MPS_AMP_MIN_ROWS). The default came from one M5 (#109), where fp16 won from four rows. On an M1 Pro (16 GB, macOS 26.1, torch 2.14.0, transformers 5.17.0, laya 0.3.26) it loses almost everywhere. benchmarks/bench_mps_autocast.py runs every request once in fp32 and once in fp16 on the same loaded agent, back to back with the order alternating, so both modes see the same load (the machine was not idle: 1-min load 6 to 10). 72 pairs per row: 24 states, three passes. A short state is one message, a long one a thread of six. Results in benchmarks/results/mps_autocast_*_m1pro.json.

english (ModernBERT-large), median ms, and fp16 minus fp32 per request:

state rows fp32 fp16 fp16 − fp32 max probability change
short 1 58.2 79.4 +21.0 0.0011
short 4 147.6 170.4 +23.6 0.0061
short 5 190.4 213.6 +22.9 0.0061
short 8 290.6 316.4 +28.6 0.0061
long 1 133.7 157.3 +24.2 0.0081
long 4 458.1 492.6 +23.8 0.0081
long 5 394.4 403.3 +9.1 0.0081
long 8 585.5 566.6 −13.7 0.0081

multilingual (mmBERT-base):

state rows fp32 fp16 fp16 − fp32 max probability change
short 1 25.5 34.7 +9.8 0.0038
short 4 45.9 55.7 +9.9 0.0038
short 5 56.4 66.5 +10.3 0.0038
short 8 78.1 90.9 +11.8 0.0038
long 1 44.0 53.7 +9.3 0.0107
long 4 145.4 157.1 +11.7 0.0106
long 5 183.6 195.1 +12.9 0.0106
long 8 267.6 283.6 +17.2 0.0206

fp16 costs about 20 to 30 ms per request on english and about 10 ms on multilingual, whatever the row count; it only wins on long english states with eight rows. So at the default of 5 an M1 Pro pays for autocast on every request of five or more rows. LAYA_MPS_AMP_MIN_ROWS=1000000 keeps fp32 throughout.

Of 1,248 decisions per checkpoint, fp16 changed none on english and one on multilingual (a noul next to 0.5). A decision near the boundary can flip.

Apple M5 Pro, MPS: fp16 autocast against fp32

The same script, unchanged, on an Apple M5 Pro (24 GB, macOS 27.0.1, Python 3.12, torch 2.14.1, transformers 5.19.0, laya 0.4.6). Same protocol: 72 pairs per row, 24 states, three passes, fp32 and fp16 back to back on one loaded agent with the order alternating. The machine was lightly loaded (1-min load 4.3 during english, 0.8 during multilingual). Results in benchmarks/results/mps_autocast_*_m5pro.json.

Here the gate goes the way #109 measured on its M5, and further: fp16 wins from two rows on short states and from one row on long states, on both checkpoints.

english (ModernBERT-large), median ms, and fp16 minus fp32 per request:

state rows fp32 fp16 fp16 - fp32 max probability change
short 1 25.6 25.5 -0.3 0.0018
short 2 36.6 30.3 -5.8 0.0018
short 4 64.4 40.2 -23.7 0.0031
short 5 74.3 44.7 -29.4 0.0031
short 6 84.5 49.1 -35.3 0.0031
short 8 110.3 59.0 -51.1 0.0126
long 1 48.8 34.3 -15.0 0.0012
long 2 86.6 50.8 -36.7 0.0012
long 4 165.1 79.0 -86.5 0.0014
long 5 203.5 93.3 -110.6 0.0014
long 6 241.4 108.1 -133.7 0.0014
long 8 312.6 135.7 -177.4 0.0021

multilingual (mmBERT-base):

state rows fp32 fp16 fp16 - fp32 max probability change
short 1 12.1 13.4 +1.3 0.0022
short 2 17.2 16.3 -1.1 0.0022
short 4 25.5 19.7 -5.8 0.0022
short 5 30.7 22.2 -8.4 0.0022
short 6 34.9 24.5 -10.5 0.0022
short 8 44.3 29.6 -14.6 0.0022
long 1 21.5 17.3 -4.3 0.0020
long 2 36.4 25.0 -11.7 0.0021
long 4 69.0 40.4 -27.8 0.0021
long 5 79.8 45.0 -35.0 0.0021
long 6 95.1 52.3 -43.0 0.0050
long 8 123.3 65.2 -58.7 0.0050

The saving grows with the work in the request: on english long states fp16 is 2.3x faster at eight rows (312.6 to 135.7 ms), and the only cell where fp32 wins is a single short multilingual row, by 1.3 ms. So at the default of 5 an M5 Pro runs rows 2 to 4 (and every long single-row request) in fp32 when fp16 would have been faster, while an M1 Pro pays for fp16 from row 5 up. The two tables bracket the current default; which way it should move, or whether it should depend on the chip, is a separate decision. LAYA_MPS_AMP_MIN_ROWS=1 takes fp16 everywhere on an M5 Pro.

Of 1,248 decisions per checkpoint, fp16 changed one on english (a short state at eight rows, where the largest probability change was 0.0126) and none on multilingual.

Calibration on a routing task runs the other way

On laya_router's 180 requests (zero-shot, one 3-tier choice), nearly every configuration we measured was under-confident (the few exceptions were +0.01 to +0.06, and among the least accurate). Mean P(chosen) (the chosen option's probability, not the entropy-based confidence field) sat below accuracy, by −0.18 on the root checkpoint with example-led tier descriptions (0.562 vs 0.744) and by −0.19 on typed-decisions (0.501 vs 0.694). This is one task and one set of labels, so it does not contradict the over-confidence reported above. It does mean the direction of the miscalibration depends on the task, and a temperature fit on your own data is the right fix either way.

Server CPU: AMD EPYC 9R14, 4 cores, Linux

research/scripts/bench_latency.py ran in-process and unchanged at v0.3.20 on an AWS m7a.xlarge: 4 physical cores (no SMT), 16 GiB RAM. The run used OMP_NUM_THREADS=4, device="cpu", fp32, torch 2.14.0, transformers 5.17.0, Python 3.14.4, and checkpoints at revision 55cf4c4. Questions alternate a 3-option choice and a noul. Each row is 10 timed calls after 2 warm-up calls. Raw results: research/results/latency_cpu_m7a_xlarge_20260924.json.

checkpoint 1 question 5 10 50 cold load
english 580 ms 3,072 ms 6,244 ms 35,969 ms 4.4 s
multilingual 193 ms 912 ms 1,842 ms 11,157 ms 2.5 s
typed-decisions 584 ms 2,819 ms 6,031 ms 35,653 ms 0.5 s

Values are p50. p95 is within 2% of p50 on every row. Up to 10 questions, each question costs about 600 ms on english and typed-decisions and about 185 ms on multilingual. At 50 questions, the cost per question rises by 15–20% on all three. Batching questions saves little on CPU, unlike the GB10 above. Cold load depends on the OS file cache, so treat that column as approximate. Peak memory for the whole script, with up to five checkpoints loaded at once, was 9.3 GiB (maximum RSS).

Independent NVIDIA CUDA capacity study

An independent study measured laya as a dynamically batched service on a frozen mix of 1,000 public SAM.gov notices. Each request asked the same three typed questions, so the table reports decisions per second (three decisions per request) at the highest tested load that met both the latency SLO and the achieved-rate gate. The study pinned Laya to 6a58191 and recorded the checkpoint hashes, software versions and GPU environments. Sweep rows, parity counts, the replay summary and the environments: research/results/nvidia_capacity_20260925.json. Sweep and replay script: research/scripts/bench_nvidia_capacity.py.

GPU and serving backend p99 ≤ 50 ms p99 ≤ 130 ms
RTX PRO 5000 Blackwell, TensorRT FP16 15 decisions/s 42 decisions/s
RTX PRO 6000 Blackwell, TensorRT FP16 not measured 146 decisions/s
H100 NVL, TensorRT FP16 105 decisions/s 175 decisions/s
H100 NVL, 7 × MIG 1g.12gb, eager FP16 not met not met reliably[^mig-rate]

TensorRT is not a laya backend. It is an external runtime: these rows ran ONNX Runtime's TensorRT execution provider (FP16, LayerNorm kept in FP32) over an ONNX export of the checkpoint. The eager rows are laya's own PyTorch model under FP16 autocast.

Every tested backend and precision reproduced the upstream FP32 answers on all 63 parity questions: 74 of 74 backend-and-device rows passed across four GPUs. Dynamic serving depended on the architecture: TensorRT raised the H100's 130 ms capacity from 93 to 175 decisions/s, while eager FP16 and TensorRT both reached 146 decisions/s on the RTX PRO 6000.

A compressed replay of a 24-hour, 10-million-decision curve on one RTX PRO 6000 (eager FP16) completed 138,863 requests with zero errors and 111 ms overall p99. Two peak-hour segments reached 132 and 144 ms, so deployments that must hold 130 ms continuously need about 25% headroom at that volume.

These are capacity results for one English federal-procurement workload, not general latency guarantees. Server sweeps and replays used one run per configuration; the RTX PRO 6000 sweep did not search below 50 requests/s; and torch.compile was not tested as a serving backend. Raw per-request laya results: sweeps for the RTX PRO 5000, RTX PRO 6000 and H100 NVL, and the day replay.

[^mig-rate]: Seven concurrent slices reached about 49 decisions/s at p99 127 ms at the lowest load, but the achieved request rate fell below the study's 90% gate. Higher loads missed the 130 ms SLO.


Limits, stated plainly

  • Near chance on typed-decisions zero-shot — the 0.766 belongs to the fine-tuned checkpoint, on that benchmark's own training split.
  • Moderation does not hold up on held-out data (0.530, macro-F1 0.400).
  • Keep choice questions under ~20 options.
  • Both checkpoints ship over-confident. Fit temperatures on your own data.
  • Ordinal score is the weakest primitive (SST-5 0.372).
  • laya collapses outside English; laya-multilingual is weaker on English. Route.

GPU fast path

pip install laya[fast] + laya.load(..., fast=True) replaces the encoder/head forward with fused TileLang kernels (GEMM+epilogue, GEMM+GEGLU, residual+LayerNorm, in-place RoPE, sliding-window flash attention over the packed QKV buffer), bf16-resident weights and one CUDA graph per (batch, length) bucket. Measured with benchmarks/bench_fast.py --eval 1000 on an RTX 4070 Ti SUPER, torch 2.11 + CUDA 13, tilelang 0.1.14; raw numbers in benchmarks/results/.

Same answers

benchmarks/parity_fast.py answers a fixed, deterministic set of 60 states x up to 8 questions (the five presets over 12 texts in six languages, short and long) with the stock bf16-autocast forward, the fast path, and an fp32 forward as the reference; every per-option probability from all three is in benchmarks/results/parity_*.json, so the comparison can be re-checked without a GPU.

checkpoint type n max |p_fast - p_stock| max |p_fast - p_fp32| max |p_stock - p_fp32| argmax fast = stock fast = fp32
laya choice 48 0.031 0.022 0.024 47/48 47/48
laya noul 180 0.076 0.043 0.058 180/180 180/180
laya score 60 0.015 0.011 0.017 59/60 60/60
laya-multilingual choice 48 0.049 0.015 0.039 47/48 47/48
laya-multilingual noul 180 0.037 0.045 0.045 180/180 179/180
laya-multilingual score 60 0.010 0.009 0.009 59/60 59/60

The fast path stays close to the fp32 reference on every row — at most 0.046 away, against 0.058 for the stock path on the same row — and no row is more than 0.076 from stock. The residual stream stays in fp32 in both, and the two bf16 paths differ from each other only by bf16 accumulation order; the few argmax disagreements are near-tie options, and on every one of them the fast path agrees with fp32. One row is the exception to the stronger reading that used to be printed here: on laya-multilingual noul the stock bf16 path is marginally closer to fp32 than the fast path is (0.0446 against 0.0455), so this table does not show that the fast path is never further from fp32. Dataset accuracy / ECE (AG News, dair-ai emotion, 1,000 samples each) are identical within noise; see benchmarks/bench_fast.py --eval 1000.

fp16

The fast path runs in the agent's autocast dtype when accelerate() is called, so an agent set to fp16 (agent.dtype = torch.float16, or the CUDA autocast override proposed for #443) gets fp16 kernels and fp16 weights; the residual stream and every accumulation stay fp32 in both dtypes. Same fixed set, parity_fast.py --dtype fp16 | bf16, RTX 4070 Ti SUPER; per-option probabilities in benchmarks/results/parity_*_rtx4070.json (the bf16 columns are the table above, plus a bf16 run of laya-typed-decisions):

checkpoint type n max |p_fast - p_fp32| bf16 max |p_fast - p_fp32| fp16 argmax fast = fp32, bf16 fp16
laya choice 48 0.022 0.004 47/48 48/48
laya noul 180 0.043 0.005 180/180 180/180
laya score 60 0.011 0.003 60/60 60/60
laya-multilingual choice 48 0.015 0.002 47/48 48/48
laya-multilingual noul 180 0.045 0.009 179/180 180/180
laya-multilingual score 60 0.009 0.001 59/60 60/60
laya-typed-decisions choice 48 0.019 0.002 47/48 47/48
laya-typed-decisions noul 180 0.023 0.005 180/180 180/180
laya-typed-decisions score 60 0.009 0.001 60/60 60/60

In fp16 the fast path is 3-10x closer to fp32 than in bf16 and agrees with the fp16 stock path on every argmax (864/864; the most it moves a probability against fp16 stock is 0.009). The one fp16 disagreement with fp32 is a laya-typed-decisions choice question whose top two options are 0.001 apart in fp32; the fp16 stock path flips it too. agent.predict() latency shows no consistent difference between the dtypes: on every case of the table below, on both checkpoints, fp16 and bf16 are within 10% of each other in both directions (single runs of 50 iterations), for stock and fast alike.

Latency, agent.predict() end to end (ms, incl. tokenization)

checkpoint case stock fast speedup
laya (ModernBERT-large) 1 question, 72 tok 17.7 4.6 3.8×
3 questions, 72 tok 18.9 6.6 2.9×
30 questions, 72 tok 43.2 35.7 1.2×
30 questions, 512 tok 327.5 232.1 1.4×
laya-multilingual (mmBERT-base) 1 question, 72 tok 14.1 2.8 5.1×
3 questions, 72 tok 15.0 3.9 3.9×
30 questions, 72 tok 22.2 17.8 1.2×
30 questions, 966 tok 320.7 187.6 1.7×
laya-multilingual, AG News eval loop 1 question / sample 14.9 3.2 4.7×

Small requests are launch-overhead bound in the stock path (≈200 kernels from Python per call); the CUDA graph removes that. Large batches are GEMM bound; the fused kernels sit at ~80 TFLOPS there, on par with cuBLAS, so the gain comes from the fused epilogues and the sliding-window attention (16× faster than SDPA with a dense mask at L=1024). First use of a new length bucket compiles kernels (a few seconds, cached on disk); inputs ≤256 tokens share one dynamic-shape kernel and never recompile.

Community evaluation: zh-CN / zh-TW (external, 2026-09-25)

From @CodyQin's zh-decision-bench (dataset CC BY 4.0, raw predictions published). Three parts:

1. Same-methodology rerun of Part A (zh only, current package)

bench_local.py Part A rerun on laya 0.3.20 (CUDA), public checkpoints, --per-lang 100, seed 13 — result file: research/results/zh_rerun_part_a.json. The english checkpoint's accuracies replicate the sweep above exactly; multilingual zh-TW improves under 0.3.20 + the temperature clamp.

checkpoint zh-CN acc / ECE zh-TW acc / ECE vs sweep above (0.2.0, pre-clamp)
english 0.620 / 0.320 0.460 / 0.434 accuracy identical (0.62 / 0.46); ECE lower post-clamp
multilingual 0.650 / 0.219 0.610 / 0.266 zh-CN within noise; zh-TW 0.54 -> 0.61

2. zh-decision-bench: business-scenario eval set (laya-evals format)

219 items / 284 questions: MASSIVE zh-CN dev (quality-filtered, 6-domain routing) plus human-adjudicated synthetic e-commerce CS and content-moderation items. Ships in this harness's format as research/evals/zh_decision_bench.jsonl:

laya-evals run research/evals/zh_decision_bench.jsonl --model multilingual --slice tag

Results (laya 0.3.20, CUDA; bootstrap CIs and raw predictions in the source repo):

checkpoint voice routing (n=179) CS routing (n=25) urgency (n=25) scam/promo (n=15) escalate (n=40)
multilingual 0.883 / 0.061 0.640 / 0.293 0.560 / 0.091 0.667 / 0.311 0.550 / 0.230
english 0.754 / 0.281 0.520 / 0.184 0.520 / 0.207 0.667 / 0.321 0.575 / 0.269

Also measured: option-order flip rate 28% (CS) / 10.6% (voice); zh-CN to native zh-TW parallel-utterance flip rate 12.8%. Task framing differs from the sweep above (6-domain routing vs 20-way intent), so these complement rather than compare.

3. Temperature refit for zh (official temp_bucket convention)

Refit on zh-decision-bench (multilingual checkpoint; 50/50 fit/test split by item hash), clamped to [0.5, 5]:

bucket T raw T clamped test ECE before -> after
choice:6-10 1.52 1.52 0.062 -> 0.093
choice:3-5 2.61 2.61 0.329 -> 0.352
score:3-5 1.33 1.33 0.118 -> 0.126
noul:2 10.23 5.00 0.180 -> 0.098

The noul:2 raw fit (10.2) exceeds TEMP_MAX: Chinese binary-judgment over-confidence outruns the shipped clamp's correction range (NLL still improves, 0.79 -> 0.55).