Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(benchmarks): add independent NVIDIA capacity results
bd-bkinge118e919de466
diff --git a/BENCHMARKS.md b/BENCHMARKS.mdindex 8a9ce586..5a6e0cbc 100644--- a/BENCHMARKS.md+++ b/BENCHMARKS.md@@ -270,6 +270,41 @@ On laya_router's 180 requests (zero-shot, one 3-tier `choice`), nearly every conValues are p50. p95 is within 2% of p50 on every row. Up to 10 questions, each question costs about 600 ms on `english` and `typed-decisions` and about 185 ms on `multilingual`. At 50 questions, the cost per question rises by 15–20% on all three. Batching questions saves little on CPU, unlike the GB10 above. Cold load depends on the OS file cache, so treat that column as approximate. Peak memory for the whole script, with up to five checkpoints loaded at once, was 9.3 GiB (maximum RSS).+## Independent NVIDIA CUDA capacity study++An independent study measured `laya` as a dynamically batched service on a frozen mix of 1,000 public+SAM.gov notices. Each request asked the same three typed questions, so the table reports decisions per second+(three decisions per request) at the highest tested load that met both the latency SLO and the achieved-rate gate.+The study pinned Laya to+[`6a58191`](https://github.com/NandhaKishorM/laya/commit/6a5819129eb220570792e417e49723d697efd76f)+and recorded the checkpoint hashes, software versions and GPU environments.++| GPU and serving backend | p99 ≤ 50 ms | p99 ≤ 130 ms |+|---|---:|---:|+| RTX PRO 5000 Blackwell, TensorRT FP16 | 15 decisions/s | 42 decisions/s |+| RTX PRO 6000 Blackwell, TensorRT FP16 | not measured | 146 decisions/s |+| H100 NVL, TensorRT FP16 | 105 decisions/s | 175 decisions/s |+| H100 NVL, 7 × MIG 1g.12gb, eager FP16 | not met | not met reliably[^mig-rate] |++Every tested backend and precision reproduced the upstream FP32 answers on all 63 parity questions: 74 of 74+backend-and-device rows passed across four GPUs. In fixed-shape throughput tests,+`torch.compile(mode="max-autotune")` FP16 was 1.3–1.7× faster than eager FP16 at batch 256. Dynamic serving+depended on the architecture: TensorRT raised the H100's 130 ms capacity from 93 to 175 decisions/s, while+eager FP16 and TensorRT both reached 146 decisions/s on the RTX PRO 6000.++A compressed replay of a 24-hour, 10-million-decision curve on one RTX PRO 6000 completed 138,863 requests+with zero errors and 111 ms overall p99. Two peak-hour segments reached 132 and 143 ms, so deployments that+must hold 130 ms continuously need about 25% headroom at that volume.++These are capacity results for one English federal-procurement workload, not general latency guarantees. Server+sweeps and replays used one run per configuration; the RTX PRO 6000 sweep did not search below 50 requests/s;+and `torch.compile` was not tested as a serving backend. The+[full report](https://github.com/bhushankinge/laya-cuda-bench/blob/6cf4148ef4d148273eae83604815fb63ca2254de/REPORT.md),+[raw results and harness](https://github.com/bhushankinge/laya-cuda-bench/tree/6cf4148ef4d148273eae83604815fb63ca2254de) are public.++[^mig-rate]: Seven concurrent slices reached about 49 decisions/s at p99 127 ms at the lowest load, but the+ achieved request rate fell below the study's 90% gate. Higher loads missed the 130 ms SLO.+---## Limits, stated plainly