Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(benchmarks): back the NVIDIA capacity section with a committed results file and script
Bhushan Kinge721cf9f682b9
diff --git a/BENCHMARKS.md b/BENCHMARKS.mdindex 5a6e0cbc..2a856763 100644--- a/BENCHMARKS.md+++ b/BENCHMARKS.md@@ -7,6 +7,7 @@ Every checkpoint answered **byte-identical questions** in each run (fixed seed).| T4 Colab | typed-decisions, MASSIVE (14 langs), XNLI (15 langs), English suites, latency, option-order robustness, calibration repair | `research/results/t4_colab_benchmark.json` || CPU sweep | MASSIVE intent across **all 51 languages**; its typed-decisions part (`part_b`) covers the English checkpoint only | `research/results/cpu_51_language_sweep.json` || Applications | the seven workflow themes + the datasets where Jev numbers exist, all three checkpoints (laya 0.2.1, CPU, 400 cases per task, seed 13, 2026-09-19) | `research/results/app_benchmark_results.json` |+| NVIDIA capacity | `laya` served with dynamic batching under p99 SLOs on RTX PRO 5000, RTX PRO 6000 and H100 NVL (eager FP16, TensorRT FP16 via ONNX Runtime), backend parity, 24-hour replay | `research/results/nvidia_capacity_20260925.json` |**Calibration columns in the CPU sweep predate the temperature clamp.** The 51-language ECE and mean-confidence figures were produced before #42 clamped temperatures to `[0.5, 5]`, so today's package reports different confidence for the affected buckets. Accuracy columns are unaffected, because a temperature-scaled softmax has the same argmax at every positive temperature.@@ -277,7 +278,9 @@ SAM.gov notices. Each request asked the same three typed questions, so the table(three decisions per request) at the highest tested load that met both the latency SLO and the achieved-rate gate.The study pinned Laya to[`6a58191`](https://github.com/NandhaKishorM/laya/commit/6a5819129eb220570792e417e49723d697efd76f)-and recorded the checkpoint hashes, software versions and GPU environments.+and recorded the checkpoint hashes, software versions and GPU environments. Sweep rows, parity counts, the replay+summary and the environments: `research/results/nvidia_capacity_20260925.json`. Sweep and replay script:+`research/scripts/bench_nvidia_capacity.py`.| GPU and serving backend | p99 ≤ 50 ms | p99 ≤ 130 ms ||---|---:|---:|@@ -286,21 +289,24 @@ and recorded the checkpoint hashes, software versions and GPU environments.| H100 NVL, TensorRT FP16 | 105 decisions/s | 175 decisions/s || H100 NVL, 7 × MIG 1g.12gb, eager FP16 | not met | not met reliably[^mig-rate] |+TensorRT is not a laya backend. It is an external runtime: these rows ran ONNX Runtime's TensorRT execution+provider (FP16, LayerNorm kept in FP32) over an ONNX export of the checkpoint. The eager rows are laya's own+PyTorch model under FP16 autocast.+Every tested backend and precision reproduced the upstream FP32 answers on all 63 parity questions: 74 of 74-backend-and-device rows passed across four GPUs. In fixed-shape throughput tests,-`torch.compile(mode="max-autotune")` FP16 was 1.3–1.7× faster than eager FP16 at batch 256. Dynamic serving-depended on the architecture: TensorRT raised the H100's 130 ms capacity from 93 to 175 decisions/s, while-eager FP16 and TensorRT both reached 146 decisions/s on the RTX PRO 6000.+backend-and-device rows passed across four GPUs. Dynamic serving depended on the architecture: TensorRT raised+the H100's 130 ms capacity from 93 to 175 decisions/s, while eager FP16 and TensorRT both reached 146 decisions/s+on the RTX PRO 6000.-A compressed replay of a 24-hour, 10-million-decision curve on one RTX PRO 6000 completed 138,863 requests-with zero errors and 111 ms overall p99. Two peak-hour segments reached 132 and 143 ms, so deployments that-must hold 130 ms continuously need about 25% headroom at that volume.+A compressed replay of a 24-hour, 10-million-decision curve on one RTX PRO 6000 (eager FP16) completed 138,863+requests with zero errors and 111 ms overall p99. Two peak-hour segments reached 132 and 144 ms, so deployments+that must hold 130 ms continuously need about 25% headroom at that volume.These are capacity results for one English federal-procurement workload, not general latency guarantees. Serversweeps and replays used one run per configuration; the RTX PRO 6000 sweep did not search below 50 requests/s;-and `torch.compile` was not tested as a serving backend. The-[full report](https://github.com/bhushankinge/laya-cuda-bench/blob/6cf4148ef4d148273eae83604815fb63ca2254de/REPORT.md),-[raw results and harness](https://github.com/bhushankinge/laya-cuda-bench/tree/6cf4148ef4d148273eae83604815fb63ca2254de) are public.+and `torch.compile` was not tested as a serving backend. Raw per-request laya results: sweeps for the+[RTX PRO 5000](https://github.com/bhushankinge/laya-cuda-bench/tree/6cf4148ef4d148273eae83604815fb63ca2254de/results/zbook-rtxpro5000/server), [RTX PRO 6000](https://github.com/bhushankinge/laya-cuda-bench/tree/6cf4148ef4d148273eae83604815fb63ca2254de/results/rtxpro6000-ws/server) and+[H100 NVL](https://github.com/bhushankinge/laya-cuda-bench/tree/6cf4148ef4d148273eae83604815fb63ca2254de/results/h100nvl/server), and the [day replay](https://github.com/bhushankinge/laya-cuda-bench/tree/6cf4148ef4d148273eae83604815fb63ca2254de/results/rtxpro6000-ws/replay).[^mig-rate]: Seven concurrent slices reached about 49 decisions/s at p99 127 ms at the lowest load, but theachieved request rate fell below the study's 90% gate. Higher loads missed the 130 ms SLO.diff --git a/research/README.md b/research/README.mdindex 9ce68d7c..a43df803 100644--- a/research/README.md+++ b/research/README.md@@ -25,6 +25,7 @@ the numbers quoted in the main README — nothing here is imported by the `laya`| `scripts/bench_apps.py` | the six application workflows (support triage, email + phishing, guardrails, RAG relevance, moderation, model routing) plus the datasets where public Jev numbers exist || `scripts/bench_latency.py` | inference speed including what routing costs: detection overhead, hot path, cold-swap, mixed-language throughput at several `max_loaded` settings || `scripts/bench_length_batching.py` | compare upstream contiguous batches with optional length sorting on synthetic tickets, including output consistency and optional fresh-process memory profiles (`psutil` required for memory mode) |+| `scripts/bench_nvidia_capacity.py` | `laya` as a dynamically batched HTTP service on one NVIDIA GPU: open-loop Poisson sweep to the highest rate meeting a p99 SLO, 24-hour curve replay, and the ONNX export the TensorRT rows need (TensorRT via ONNX Runtime, not a laya backend). Results: `results/nvidia_capacity_20260925.json` || `scripts/make_plots.py` | renders `assets/laya_benchmark.png` from the result JSONs || `scripts/bench_long_context.py` | `laya-multilingual` on long documents: 20 support requests in 8 languages, each placed after 0 to 7,000 tokens of unrelated text, scored at the default limit and at `max_len=8192` || `scripts/plot_long_context.py` | renders `assets/long_context_8192.png` from `results/long_context_multilingual.json` |diff --git a/research/results/nvidia_capacity_20260925.json b/research/results/nvidia_capacity_20260925.jsonnew file mode 100644index 00000000..8f17544c--- /dev/null+++ b/research/results/nvidia_capacity_20260925.json@@ -0,0 +1,1610 @@+{+ "description": "Independent NVIDIA CUDA capacity study of laya as a dynamically batched HTTP service (trimmed, laya-only). Workload: 1,000 public SAM.gov notices, natural length mix (<= 472 laya tokens); each request asks the same 3 typed questions (choice, score, noul), so decisions/s = 3 x requests/s. A sustained point needs p99 <= SLO, achieved >= 90% of the offered rate and latency trend <= 1.5 over a 60 s step.",+ "source": {+ "repo": "https://github.com/bhushankinge/laya-cuda-bench",+ "commit": "6cf4148ef4d148273eae83604815fb63ca2254de",+ "file_paths_relative_to": "https://github.com/bhushankinge/laya-cuda-bench/blob/6cf4148ef4d148273eae83604815fb63ca2254de/"+ },+ "script": "research/scripts/bench_nvidia_capacity.py",+ "laya": {+ "commit": "6a5819129eb220570792e417e49723d697efd76f",+ "checkpoint": "convaiinnovations/laya",+ "checkpoint_revision": "c5d78730f3493e4fe16d61507ef4b78eef7318cf",+ "model_safetensors_sha256": "891102d372688fc2a094dac56a384bc537b87c63f21f9f3dac0be2b7cbc8d86c"+ },+ "tensorrt_note": "TensorRT rows run ONNX Runtime's TensorRT execution provider over an ONNX export of the checkpoint (FP16, LayerNorm kept in FP32). TensorRT is an external runtime, not a laya-shipped backend.",+ "measured": "2026-09-22 to 2026-09-23, one run per configuration",+ "environment": {+ "zbook-rtxpro5000": {+ "box": "zbook-rtxpro5000",+ "collected_at": "2026-09-22T19:32:27.811144+00:00",+ "os": "Linux-6.17.0-35-generic-x86_64-with-glibc2.39",+ "python": "3.12.3",+ "cpu_count": 24,+ "ram_gb": 66.8,+ "torch_disable_native_jit": "1",+ "laya_trt_max_batch": "256",+ "packages": {+ "torch": "2.14.0+cu130",+ "transformers": "5.17.0",+ "onnxruntime-gpu": "1.30.0",+ "onnxruntime": null,+ "tensorrt-cu13": "10.16.1.11",+ "tensorrt": null,+ "laya": "0.3.3",+ "nvidia-ml-py": "13.610.43",+ "numpy": "2.5.3",+ "huggingface_hub": "1.32.0",+ "typesafe-sdk": "0.7.1"+ },+ "torch_cuda": "13.0",+ "gpu": {+ "name": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "capability": "12.0",+ "total_mem_gb": 25.1+ },+ "driver": "595.71.05",+ "mig_mode": "[N/A]",+ "power_limit_w": "[N/A]"+ },+ "rtxpro6000-ws": {+ "box": "rtxpro6000-ws",+ "collected_at": "2026-09-23T03:17:22.051199+00:00",+ "os": "Linux-6.17.0-40-generic-x86_64-with-glibc2.39",+ "python": "3.12.3",+ "cpu_count": 24,+ "ram_gb": 100.6,+ "torch_disable_native_jit": "1",+ "laya_trt_max_batch": "256",+ "packages": {+ "torch": "2.14.0+cu130",+ "transformers": "5.17.0",+ "onnxruntime-gpu": "1.30.0",+ "onnxruntime": null,+ "tensorrt-cu13": "10.16.1.11",+ "tensorrt": null,+ "laya": "0.3.3",+ "nvidia-ml-py": "13.610.43",+ "numpy": "2.5.3",+ "huggingface_hub": "1.32.0",+ "typesafe-sdk": "0.7.1"+ },+ "torch_cuda": "13.0",+ "gpu": {+ "name": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "capability": "12.0",+ "total_mem_gb": 102.0+ },+ "driver": "595.84",+ "mig_mode": "[N/A]",+ "power_limit_w": "600.00 W"+ },+ "h100nvl": {+ "box": "h100nvl",+ "collected_at": "2026-09-23T19:12:58.800605+00:00",+ "os": "Linux-4.18.0-553.104.1.el8_10.x86_64-x86_64-with-glibc2.39",+ "python": "3.12.3",+ "cpu_count": 128,+ "ram_gb": 540.2,+ "torch_disable_native_jit": "1",+ "laya_trt_max_batch": "256",+ "packages": {+ "torch": "2.14.0+cu130",+ "transformers": "5.17.0",+ "onnxruntime-gpu": "1.30.0",+ "onnxruntime": null,+ "tensorrt-cu13": "10.16.1.11",+ "tensorrt": null,+ "laya": "0.3.3",+ "nvidia-ml-py": "13.610.43",+ "numpy": "2.5.2",+ "huggingface_hub": "1.32.0",+ "typesafe-sdk": "0.7.1"+ },+ "torch_cuda": "13.0",+ "gpu": {+ "name": "NVIDIA H100 NVL",+ "capability": "9.0",+ "total_mem_gb": 99.9+ },+ "driver": "580.95.05",+ "mig_mode": "Disabled",+ "power_limit_w": "400.00 W"+ },+ "laptop-rtx2000ada": {+ "box": "laptop-rtx2000ada",+ "collected_at": "2026-09-22T16:03:52.795468+00:00",+ "os": "Linux-6.17.0-1032-oem-x86_64-with-glibc2.39",+ "python": "3.12.3",+ "cpu_count": 28,+ "ram_gb": 33.2,+ "torch_disable_native_jit": null,+ "laya_trt_max_batch": null,+ "packages": {+ "torch": "2.14.0+cu130",+ "transformers": "5.17.0",+ "onnxruntime-gpu": null,+ "onnxruntime": null,+ "tensorrt-cu13": null,+ "tensorrt": null,+ "laya": "0.3.3",+ "nvidia-ml-py": "13.610.43",+ "numpy": "2.5.3",+ "huggingface_hub": "1.32.0",+ "typesafe-sdk": "0.7.1"+ },+ "torch_cuda": "13.0",+ "gpu": {+ "name": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "capability": "8.9",+ "total_mem_gb": 8.2+ },+ "driver": "590.48.01",+ "mig_mode": "[N/A]",+ "power_limit_w": "[N/A]"+ },+ "h100nvl-mig-1g.12gb": {+ "box": "h100nvl",+ "collected_at": "2026-09-23T22:44:12.234270+00:00",+ "os": "Linux-4.18.0-553.104.1.el8_10.x86_64-x86_64-with-glibc2.39",+ "python": "3.12.3",+ "cpu_count": 128,+ "ram_gb": 540.2,+ "torch_disable_native_jit": "1",+ "laya_trt_max_batch": "256",+ "packages": {+ "torch": "2.14.0+cu130",+ "transformers": "5.17.0",+ "onnxruntime-gpu": "1.30.0",+ "onnxruntime": null,+ "tensorrt-cu13": "10.16.1.11",+ "tensorrt": null,+ "laya": "0.3.3",+ "nvidia-ml-py": "13.610.43",+ "numpy": "2.5.2",+ "huggingface_hub": "1.32.0",+ "typesafe-sdk": "0.7.1"+ },+ "torch_cuda": "13.0",+ "gpu": {+ "name": "NVIDIA H100 NVL MIG 1g.12gb",+ "capability": "9.0",+ "total_mem_gb": 11.5+ },+ "driver": "580.95.05",+ "mig_mode": "Enabled",+ "power_limit_w": "400.00 W"+ }+ },+ "slo_sweeps": [+ {+ "gpu": "RTX PRO 5000 Blackwell Laptop 24 GB",+ "box": "zbook-rtxpro5000",+ "file": "results/zbook-rtxpro5000/server/laya.ort-trt-fp16.sweep.json",+ "backend": "ort-trt-fp16-maxb256",+ "max_batch": 16,+ "max_delay_ms": 2.0,+ "warmup_s": 20.0,+ "step_s": 60.0,+ "sustained": {+ "50": {+ "target_qps": 5.0,+ "achieved_qps": 5.006,+ "decisions_per_s": 15.019,+ "p99_ms": 45.331,+ "p50_ms": 18.913+ },+ "130": {+ "target_qps": 13.72,+ "achieved_qps": 13.93,+ "decisions_per_s": 41.791,+ "p99_ms": 113.126,+ "p50_ms": 20.084+ }+ },+ "steps": [+ {+ "target_qps": 5.0,+ "achieved_qps": 5.006,+ "requests": 300,+ "errors": 0,+ "decisions_per_s": 15.019,+ "p50_ms": 18.913,+ "p95_ms": 33.36,+ "p99_ms": 45.331,+ "max_ms": 59.726,+ "mean_served_batch": 3.14,+ "latency_trend": 1.096,+ "saturated": false+ },+ {+ "target_qps": 7.0,+ "achieved_qps": 6.67,+ "requests": 402,+ "errors": 0,+ "decisions_per_s": 20.01,+ "p50_ms": 19.044,+ "p95_ms": 33.996,+ "p99_ms": 60.609,+ "max_ms": 71.188,+ "mean_served_batch": 3.149,+ "latency_trend": 0.947,+ "saturated": false+ },+ {+ "target_qps": 9.8,+ "achieved_qps": 9.764,+ "requests": 586,+ "errors": 0,+ "decisions_per_s": 29.292,+ "p50_ms": 19.668,+ "p95_ms": 40.148,+ "p99_ms": 61.07,+ "max_ms": 76.422,+ "mean_served_batch": 3.205,+ "latency_trend": 1.047,+ "saturated": false+ },+ {+ "target_qps": 13.72,+ "achieved_qps": 13.93,+ "requests": 836,+ "errors": 0,+ "decisions_per_s": 41.791,+ "p50_ms": 20.084,+ "p95_ms": 55.122,+ "p99_ms": 113.126,+ "max_ms": 154.412,+ "mean_served_batch": 3.459,+ "latency_trend": 1.049,+ "saturated": false+ },+ {+ "target_qps": 19.208,+ "achieved_qps": 18.811,+ "requests": 1129,+ "errors": 0,+ "decisions_per_s": 56.433,+ "p50_ms": 22.796,+ "p95_ms": 83.012,+ "p99_ms": 135.269,+ "max_ms": 208.868,+ "mean_served_batch": 3.702,+ "latency_trend": 0.944,+ "saturated": false+ },+ {+ "target_qps": 26.891,+ "achieved_qps": 27.824,+ "requests": 1673,+ "errors": 0,+ "decisions_per_s": 83.472,+ "p50_ms": 30.928,+ "p95_ms": 361.955,+ "p99_ms": 628.47,+ "max_ms": 775.337,+ "mean_served_batch": 5.733,+ "latency_trend": 0.888,+ "saturated": true+ }+ ]+ },+ {+ "gpu": "RTX PRO 5000 Blackwell Laptop 24 GB",+ "box": "zbook-rtxpro5000",+ "file": "results/zbook-rtxpro5000/server/laya.eager-fp16.sweep.json",+ "backend": "torch-eager-fp16",+ "max_batch": 16,+ "max_delay_ms": 2.0,+ "warmup_s": 20.0,+ "step_s": 60.0,+ "sustained": {+ "50": null,+ "130": {+ "target_qps": 13.72,+ "achieved_qps": 13.93,+ "decisions_per_s": 41.79,+ "p99_ms": 118.241,+ "p50_ms": 25.027+ }+ },+ "steps": [+ {+ "target_qps": 5.0,+ "achieved_qps": 5.006,+ "requests": 300,+ "errors": 0,+ "decisions_per_s": 15.018,+ "p50_ms": 24.487,+ "p95_ms": 42.937,+ "p99_ms": 55.583,+ "max_ms": 75.993,+ "mean_served_batch": 3.16,+ "latency_trend": 1.043,+ "saturated": false+ },+ {+ "target_qps": 7.0,+ "achieved_qps": 6.669,+ "requests": 402,+ "errors": 0,+ "decisions_per_s": 20.007,+ "p50_ms": 23.649,+ "p95_ms": 44.621,+ "p99_ms": 61.364,+ "max_ms": 86.137,+ "mean_served_batch": 3.179,+ "latency_trend": 0.952,+ "saturated": false+ },+ {+ "target_qps": 9.8,+ "achieved_qps": 9.763,+ "requests": 586,+ "errors": 0,+ "decisions_per_s": 29.289,+ "p50_ms": 23.822,+ "p95_ms": 54.293,+ "p99_ms": 72.834,+ "max_ms": 84.366,+ "mean_served_batch": 3.256,+ "latency_trend": 0.993,+ "saturated": false+ },+ {+ "target_qps": 13.72,+ "achieved_qps": 13.93,+ "requests": 836,+ "errors": 0,+ "decisions_per_s": 41.79,+ "p50_ms": 25.027,+ "p95_ms": 72.338,+ "p99_ms": 118.241,+ "max_ms": 202.486,+ "mean_served_batch": 3.574,+ "latency_trend": 1.062,+ "saturated": false+ },+ {+ "target_qps": 19.208,+ "achieved_qps": 18.807,+ "requests": 1129,+ "errors": 0,+ "decisions_per_s": 56.422,+ "p50_ms": 29.164,+ "p95_ms": 90.721,+ "p99_ms": 143.14,+ "max_ms": 193.706,+ "mean_served_batch": 3.909,+ "latency_trend": 1.03,+ "saturated": false+ },+ {+ "target_qps": 26.891,+ "achieved_qps": 27.822,+ "requests": 1673,+ "errors": 0,+ "decisions_per_s": 83.465,+ "p50_ms": 52.473,+ "p95_ms": 759.726,+ "p99_ms": 947.794,+ "max_ms": 1063.501,+ "mean_served_batch": 8.025,+ "latency_trend": 0.816,+ "saturated": true+ }+ ]+ },+ {+ "gpu": "RTX PRO 6000 Blackwell Workstation 96 GB",+ "box": "rtxpro6000-ws",+ "file": "results/rtxpro6000-ws/server/laya.ort-trt-fp16.sweep.json",+ "backend": "ort-trt-fp16-maxb256",+ "max_batch": 128,+ "max_delay_ms": 2.0,+ "warmup_s": 20.0,+ "step_s": 60.0,+ "sustained": {+ "50": null,+ "130": {+ "target_qps": 50.0,+ "achieved_qps": 48.561,+ "decisions_per_s": 145.682,+ "p99_ms": 67.54,+ "p50_ms": 19.6+ }+ },+ "steps": [+ {+ "target_qps": 50.0,+ "achieved_qps": 48.561,+ "requests": 2914,+ "errors": 0,+ "decisions_per_s": 145.682,+ "p50_ms": 19.6,+ "p95_ms": 50.224,+ "p99_ms": 67.54,+ "max_ms": 103.343,+ "mean_served_batch": 4.792,+ "latency_trend": 0.941,+ "saturated": false+ },+ {+ "target_qps": 70.0,+ "achieved_qps": 69.589,+ "requests": 4175,+ "errors": 0,+ "decisions_per_s": 208.766,+ "p50_ms": 29.465,+ "p95_ms": 116.168,+ "p99_ms": 176.576,+ "max_ms": 250.543,+ "mean_served_batch": 8.238,+ "latency_trend": 1.063,+ "saturated": false+ },+ {+ "target_qps": 98.0,+ "achieved_qps": 88.605,+ "requests": 5881,+ "errors": 0,+ "decisions_per_s": 265.814,+ "p50_ms": 4280.761,+ "p95_ms": 6263.83,+ "p99_ms": 6660.771,+ "max_ms": 6853.595,+ "mean_served_batch": 128.793,+ "latency_trend": 2.043,+ "saturated": true+ }+ ]+ },+ {+ "gpu": "RTX PRO 6000 Blackwell Workstation 96 GB",+ "box": "rtxpro6000-ws",+ "file": "results/rtxpro6000-ws/server/laya.eager-fp16.sweep.json",+ "backend": "torch-eager-fp16",+ "max_batch": 128,+ "max_delay_ms": 2.0,+ "warmup_s": 20.0,+ "step_s": 60.0,+ "sustained": {+ "50": null,+ "130": {+ "target_qps": 50.0,+ "achieved_qps": 48.541,+ "decisions_per_s": 145.623,+ "p99_ms": 86.466,+ "p50_ms": 50.549+ }+ },+ "steps": [+ {+ "target_qps": 50.0,+ "achieved_qps": 48.541,+ "requests": 2914,+ "errors": 0,+ "decisions_per_s": 145.623,+ "p50_ms": 50.549,+ "p95_ms": 74.713,+ "p99_ms": 86.466,+ "max_ms": 121.705,+ "mean_served_batch": 7.465,+ "latency_trend": 0.992,+ "saturated": false+ },+ {+ "target_qps": 70.0,+ "achieved_qps": 69.568,+ "requests": 4175,+ "errors": 0,+ "decisions_per_s": 208.704,+ "p50_ms": 61.617,+ "p95_ms": 140.363,+ "p99_ms": 194.784,+ "max_ms": 230.976,+ "mean_served_batch": 12.364,+ "latency_trend": 1.008,+ "saturated": false+ },+ {+ "target_qps": 98.0,+ "achieved_qps": 62.866,+ "requests": 5881,+ "errors": 1046,+ "decisions_per_s": 188.598,+ "p50_ms": 12311.677,+ "p95_ms": 16984.734,+ "p99_ms": 17193.178,+ "max_ms": 19999.371,+ "mean_served_batch": 128.932,+ "latency_trend": 2.278,+ "saturated": true+ }+ ]+ },+ {+ "gpu": "H100 NVL 94 GB",+ "box": "h100nvl",+ "file": "results/h100nvl/server/laya.ort-trt-fp16.sweep.json",+ "backend": "ort-trt-fp16-maxb256",+ "max_batch": 128,+ "max_delay_ms": 2.0,+ "warmup_s": 20.0,+ "step_s": 60.0,+ "sustained": {+ "50": {+ "target_qps": 35.714,+ "achieved_qps": 34.902,+ "decisions_per_s": 104.706,+ "p99_ms": 46.062,+ "p50_ms": 14.515+ },+ "130": {+ "target_qps": 59.161,+ "achieved_qps": 58.235,+ "decisions_per_s": 174.706,+ "p99_ms": 91.136,+ "p50_ms": 18.792+ }+ },+ "steps": [+ {+ "target_qps": 35.714,+ "achieved_qps": 34.902,+ "requests": 2094,+ "errors": 0,+ "decisions_per_s": 104.706,+ "p50_ms": 14.515,+ "p95_ms": 32.2,+ "p99_ms": 46.062,+ "max_ms": 81.516,+ "mean_served_batch": 3.731,+ "latency_trend": 1.005,+ "saturated": false+ },+ {+ "target_qps": 42.258,+ "achieved_qps": 43.7,+ "requests": 2621,+ "errors": 0,+ "decisions_per_s": 131.1,+ "p50_ms": 15.844,+ "p95_ms": 39.572,+ "p99_ms": 52.629,+ "max_ms": 75.421,+ "mean_served_batch": 4.165,+ "latency_trend": 1.088,+ "saturated": false+ },+ {+ "target_qps": 50.0,+ "achieved_qps": 48.563,+ "requests": 2914,+ "errors": 0,+ "decisions_per_s": 145.689,+ "p50_ms": 16.98,+ "p95_ms": 45.198,+ "p99_ms": 63.734,+ "max_ms": 98.071,+ "mean_served_batch": 4.566,+ "latency_trend": 0.97,+ "saturated": false+ },+ {+ "target_qps": 59.161,+ "achieved_qps": 58.235,+ "requests": 3498,+ "errors": 0,+ "decisions_per_s": 174.706,+ "p50_ms": 18.792,+ "p95_ms": 58.097,+ "p99_ms": 91.136,+ "max_ms": 164.121,+ "mean_served_batch": 5.403,+ "latency_trend": 0.988,+ "saturated": false+ },+ {+ "target_qps": 70.0,+ "achieved_qps": 69.593,+ "requests": 4175,+ "errors": 0,+ "decisions_per_s": 208.779,+ "p50_ms": 23.847,+ "p95_ms": 88.346,+ "p99_ms": 135.857,+ "max_ms": 212.034,+ "mean_served_batch": 7.242,+ "latency_trend": 1.054,+ "saturated": false+ },+ {+ "target_qps": 98.0,+ "achieved_qps": 97.749,+ "requests": 5881,+ "errors": 0,+ "decisions_per_s": 293.248,+ "p50_ms": 170.396,+ "p95_ms": 475.441,+ "p99_ms": 706.513,+ "max_ms": 884.172,+ "mean_served_batch": 41.265,+ "latency_trend": 1.574,+ "saturated": true+ }+ ]+ },+ {+ "gpu": "H100 NVL 94 GB",+ "box": "h100nvl",+ "file": "results/h100nvl/server/laya.eager-fp16.sweep.json",+ "backend": "torch-eager-fp16",+ "max_batch": 128,+ "max_delay_ms": 2.0,+ "warmup_s": 20.0,+ "step_s": 60.0,+ "sustained": {+ "50": {+ "target_qps": 13.015,+ "achieved_qps": 13.051,+ "decisions_per_s": 39.154,+ "p99_ms": 38.84,+ "p50_ms": 17.763+ },+ "130": {+ "target_qps": 30.184,+ "achieved_qps": 31.092,+ "decisions_per_s": 93.275,+ "p99_ms": 100.294,+ "p50_ms": 19.305+ }+ },+ "steps": [+ {+ "target_qps": 13.015,+ "achieved_qps": 13.051,+ "requests": 783,+ "errors": 0,+ "decisions_per_s": 39.154,+ "p50_ms": 17.763,+ "p95_ms": 30.516,+ "p99_ms": 38.84,+ "max_ms": 96.222,+ "mean_served_batch": 3.383,+ "latency_trend": 1.022,+ "saturated": false+ },+ {+ "target_qps": 15.4,+ "achieved_qps": 15.947,+ "requests": 956,+ "errors": 0,+ "decisions_per_s": 47.842,+ "p50_ms": 17.869,+ "p95_ms": 31.729,+ "p99_ms": 88.714,+ "max_ms": 126.199,+ "mean_served_batch": 3.496,+ "latency_trend": 1.002,+ "saturated": false+ },+ {+ "target_qps": 18.222,+ "achieved_qps": 19.637,+ "requests": 1178,+ "errors": 0,+ "decisions_per_s": 58.911,+ "p50_ms": 18.076,+ "p95_ms": 34.952,+ "p99_ms": 95.025,+ "max_ms": 117.211,+ "mean_served_batch": 3.576,+ "latency_trend": 0.977,+ "saturated": false+ },+ {+ "target_qps": 25.51,+ "achieved_qps": 25.434,+ "requests": 1529,+ "errors": 0,+ "decisions_per_s": 76.303,+ "p50_ms": 18.744,+ "p95_ms": 50.522,+ "p99_ms": 124.008,+ "max_ms": 176.933,+ "mean_served_batch": 3.836,+ "latency_trend": 0.969,+ "saturated": false+ },+ {+ "target_qps": 30.184,+ "achieved_qps": 31.092,+ "requests": 1866,+ "errors": 0,+ "decisions_per_s": 93.275,+ "p50_ms": 19.305,+ "p95_ms": 40.97,+ "p99_ms": 100.294,+ "max_ms": 143.284,+ "mean_served_batch": 4.055,+ "latency_trend": 1.012,+ "saturated": false+ },+ {+ "target_qps": 35.714,+ "achieved_qps": 34.899,+ "requests": 2094,+ "errors": 0,+ "decisions_per_s": 104.696,+ "p50_ms": 22.107,+ "p95_ms": 102.874,+ "p99_ms": 135.348,+ "max_ms": 178.08,+ "mean_served_batch": 4.908,+ "latency_trend": 0.826,+ "saturated": false+ },+ {+ "target_qps": 50.0,+ "achieved_qps": 48.559,+ "requests": 2914,+ "errors": 0,+ "decisions_per_s": 145.678,+ "p50_ms": 100.56,+ "p95_ms": 164.806,+ "p99_ms": 199.552,+ "max_ms": 248.792,+ "mean_served_batch": 12.986,+ "latency_trend": 0.854,+ "saturated": false+ },+ {+ "target_qps": 70.0,+ "achieved_qps": 69.589,+ "requests": 4175,+ "errors": 0,+ "decisions_per_s": 208.767,+ "p50_ms": 106.299,+ "p95_ms": 229.279,+ "p99_ms": 327.726,+ "max_ms": 491.874,+ "mean_served_batch": 18.97,+ "latency_trend": 0.837,+ "saturated": false+ },+ {+ "target_qps": 98.0,+ "achieved_qps": 97.346,+ "requests": 5881,+ "errors": 0,+ "decisions_per_s": 292.037,+ "p50_ms": 401.076,+ "p95_ms": 717.645,+ "p99_ms": 842.341,+ "max_ms": 960.329,+ "mean_served_batch": 82.195,+ "latency_trend": 0.908,+ "saturated": true+ }+ ]+ },+ {+ "gpu": "H100 NVL, 7 x MIG 1g.12gb, concurrent",+ "box": "h100nvl",+ "backend": "torch-eager-fp16",+ "max_batch": 64,+ "files": "results/h100nvl/server/laya.eager-fp16.mig-1g.12gb-slice{0..6}.sweep.json",+ "sustained": {+ "50": null,+ "130": null+ },+ "steps": [+ {+ "target_qps_per_slice": 2.603,+ "decisions_per_s_total": 48.61,+ "p99_ms_min": 125.918,+ "p99_ms_max": 127.171,+ "achieved_over_target_min": 0.889,+ "errors": 0,+ "slices_meeting_130ms": 0,+ "slices_meeting_50ms": 0+ },+ {+ "target_qps_per_slice": 3.644,+ "decisions_per_s_total": 72.699,+ "p99_ms_min": 188.392,+ "p99_ms_max": 456.968,+ "achieved_over_target_min": 0.949,+ "errors": 0,+ "slices_meeting_130ms": 0,+ "slices_meeting_50ms": 0+ },+ {+ "target_qps_per_slice": 5.102,+ "decisions_per_s_total": 115.614,+ "p99_ms_min": 152.471,+ "p99_ms_max": 285.66,+ "achieved_over_target_min": 1.079,+ "errors": 0,+ "slices_meeting_130ms": 0,+ "slices_meeting_50ms": 0+ },+ {+ "target_qps_per_slice": 7.143,+ "decisions_per_s_total": 149.68,+ "p99_ms_min": 222.046,+ "p99_ms_max": 246.109,+ "achieved_over_target_min": 0.998,+ "errors": 0,+ "slices_meeting_130ms": 0,+ "slices_meeting_50ms": 0+ },+ {+ "target_qps_per_slice": 10.0,+ "decisions_per_s_total": 217.081,+ "p99_ms_min": 363.269,+ "p99_ms_max": 410.228,+ "achieved_over_target_min": 1.034,+ "errors": 0,+ "slices_meeting_130ms": 0,+ "slices_meeting_50ms": 0+ },+ {+ "target_qps_per_slice": 14.0,+ "decisions_per_s_total": 283.667,+ "p99_ms_min": 583.931,+ "p99_ms_max": 1110.721,+ "achieved_over_target_min": 0.962,+ "errors": 0,+ "slices_meeting_130ms": 0,+ "slices_meeting_50ms": 0+ }+ ]+ }+ ],+ "parity": {+ "reference": "upstream FP32 forward at the pinned laya commit",+ "cases": 16,+ "questions_per_row": 63,+ "rows": 74,+ "rows_all_63_agree": 74,+ "gpus": 4,+ "files": "results/*/parity*.json",+ "by_row": [+ {+ "gpu": "NVIDIA H100 NVL",+ "model": "laya",+ "backend": "torch-eager-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00673+ },+ {+ "gpu": "NVIDIA H100 NVL",+ "model": "laya",+ "backend": "torch-eager-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0+ },+ {+ "gpu": "NVIDIA H100 NVL",+ "model": "laya",+ "backend": "ort-cuda-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0063+ },+ {+ "gpu": "NVIDIA H100 NVL",+ "model": "laya",+ "backend": "ort-trt-fp16-maxb256",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0149+ },+ {+ "gpu": "NVIDIA H100 NVL",+ "model": "laya-multilingual",+ "backend": "torch-eager-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0011+ },+ {+ "gpu": "NVIDIA H100 NVL",+ "model": "laya-multilingual",+ "backend": "torch-eager-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0+ },+ {+ "gpu": "NVIDIA H100 NVL",+ "model": "laya-multilingual",+ "backend": "ort-cuda-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.000977+ },+ {+ "gpu": "NVIDIA H100 NVL",+ "model": "laya-multilingual",+ "backend": "ort-trt-fp16-maxb256",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00189+ },+ {+ "gpu": "NVIDIA H100 NVL",+ "model": "laya",+ "backend": "torch-compile-max-autotune-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00561+ },+ {+ "gpu": "NVIDIA H100 NVL",+ "model": "laya-multilingual",+ "backend": "torch-compile-max-autotune-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.000728+ },+ {+ "gpu": "NVIDIA H100 NVL MIG 1g.12gb",+ "model": "laya",+ "backend": "torch-eager-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00672+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya",+ "backend": "torch-eager-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya",+ "backend": "torch-eager-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00305+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya",+ "backend": "torch-eager-bf16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0146+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya",+ "backend": "ort-cuda-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 3.2e-06+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya",+ "backend": "ort-cuda-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00752+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya",+ "backend": "torch-compile-max-autotune-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00331+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya-multilingual",+ "backend": "torch-eager-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya-multilingual",+ "backend": "torch-eager-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.000773+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya-multilingual",+ "backend": "torch-eager-bf16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00437+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya-multilingual",+ "backend": "ort-cuda-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 5.99e-07+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya-multilingual",+ "backend": "ort-cuda-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00151+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya-multilingual",+ "backend": "torch-compile-max-autotune-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.000679+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya-typed-decisions",+ "backend": "torch-eager-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya-typed-decisions",+ "backend": "torch-eager-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00129+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya-typed-decisions",+ "backend": "torch-eager-bf16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00692+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya-typed-decisions",+ "backend": "ort-cuda-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 1.43e-06+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya-typed-decisions",+ "backend": "ort-cuda-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00242+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya-typed-decisions",+ "backend": "torch-compile-max-autotune-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00137+ },+ {+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",+ "model": "laya",+ "backend": "ort-trt-fp16-maxb32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0168+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya",+ "backend": "torch-eager-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya",+ "backend": "torch-eager-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00545+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya",+ "backend": "torch-eager-bf16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0164+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya",+ "backend": "ort-cuda-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 1.43e-06+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya",+ "backend": "ort-cuda-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0101+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya",+ "backend": "torch-compile-max-autotune-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00605+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya",+ "backend": "ort-trt-fp16-maxb128",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00573+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-multilingual",+ "backend": "torch-eager-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-multilingual",+ "backend": "torch-eager-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.000531+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-multilingual",+ "backend": "torch-eager-bf16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00606+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-multilingual",+ "backend": "ort-cuda-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 7.09e-07+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-multilingual",+ "backend": "ort-cuda-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.000995+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-multilingual",+ "backend": "torch-compile-max-autotune-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.000867+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-multilingual",+ "backend": "ort-trt-fp16-maxb128",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00173+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-typed-decisions",+ "backend": "torch-eager-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-typed-decisions",+ "backend": "torch-eager-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00122+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-typed-decisions",+ "backend": "torch-eager-bf16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0148+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-typed-decisions",+ "backend": "ort-cuda-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 1.71e-06+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-typed-decisions",+ "backend": "ort-cuda-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00217+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-typed-decisions",+ "backend": "torch-compile-max-autotune-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0012+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-typed-decisions",+ "backend": "ort-trt-fp16-maxb128",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00772+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya",+ "backend": "ort-trt-fp16-maxb256",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0188+ },+ {+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",+ "model": "laya-multilingual",+ "backend": "ort-trt-fp16-maxb256",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00173+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya",+ "backend": "torch-eager-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya",+ "backend": "torch-eager-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00529+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya",+ "backend": "torch-eager-bf16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00897+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya",+ "backend": "ort-cuda-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 6.53e-06+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya",+ "backend": "ort-cuda-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00316+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya",+ "backend": "torch-compile-max-autotune-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00953+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya",+ "backend": "ort-trt-fp16-maxb64",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0115+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-multilingual",+ "backend": "torch-eager-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-multilingual",+ "backend": "torch-eager-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.000539+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-multilingual",+ "backend": "torch-eager-bf16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00621+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-multilingual",+ "backend": "ort-cuda-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 4.06e-07+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-multilingual",+ "backend": "ort-cuda-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00101+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-multilingual",+ "backend": "torch-compile-max-autotune-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.000671+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-multilingual",+ "backend": "ort-trt-fp16-maxb64",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00236+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-typed-decisions",+ "backend": "torch-eager-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.0+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-typed-decisions",+ "backend": "torch-eager-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00233+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-typed-decisions",+ "backend": "torch-eager-bf16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00786+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-typed-decisions",+ "backend": "ort-cuda-fp32",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 1.62e-06+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-typed-decisions",+ "backend": "ort-cuda-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00301+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-typed-decisions",+ "backend": "torch-compile-max-autotune-fp16",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00146+ },+ {+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",+ "model": "laya-typed-decisions",+ "backend": "ort-trt-fp16-maxb64",+ "argmax_agreements": 63,+ "questions": 63,+ "probability_max_abs_error": 0.00479+ }+ ]+ },+ "replay": {+ "gpu": "RTX PRO 6000 Blackwell Workstation 96 GB",+ "box": "rtxpro6000-ws",+ "backend": "torch-eager-fp16",+ "max_batch": 128,+ "file": "results/rtxpro6000-ws/replay/laya.eager-fp16.10M-day.jsonl",+ "curve": "fixtures/workload/diurnal.csv",+ "total_decisions_per_day": 10000000,+ "compress": 24,+ "hour_replayed_s": 150,+ "requests": 138863,+ "errors": 0,+ "p50_ms": 47.372,+ "p90_ms": 70.068,+ "p99_ms": 110.725,+ "max_ms": 250.599,+ "decisions_per_s_mean": 115.716,+ "mean_served_batch": 7.287,+ "per_hour": [+ {+ "hour": 0,+ "decisions_per_s": 64.5,+ "p99_ms": 65.8+ },+ {+ "hour": 1,+ "decisions_per_s": 54.7,+ "p99_ms": 64.4+ },+ {+ "hour": 2,+ "decisions_per_s": 49.4,+ "p99_ms": 64.0+ },+ {+ "hour": 3,+ "decisions_per_s": 48.2,+ "p99_ms": 65.9+ },+ {+ "hour": 4,+ "decisions_per_s": 49.1,+ "p99_ms": 65.3+ },+ {+ "hour": 5,+ "decisions_per_s": 57.4,+ "p99_ms": 64.8+ },+ {+ "hour": 6,+ "decisions_per_s": 81.4,+ "p99_ms": 71.1+ },+ {+ "hour": 7,+ "decisions_per_s": 118.7,+ "p99_ms": 85.6+ },+ {+ "hour": 8,+ "decisions_per_s": 147.5,+ "p99_ms": 92.8+ },+ {+ "hour": 9,+ "decisions_per_s": 177.4,+ "p99_ms": 124.5+ },+ {+ "hour": 10,+ "decisions_per_s": 187.4,+ "p99_ms": 132.2+ },+ {+ "hour": 11,+ "decisions_per_s": 182.8,+ "p99_ms": 126.0+ },+ {+ "hour": 12,+ "decisions_per_s": 161.3,+ "p99_ms": 110.8+ },+ {+ "hour": 13,+ "decisions_per_s": 170.7,+ "p99_ms": 125.5+ },+ {+ "hour": 14,+ "decisions_per_s": 185.4,+ "p99_ms": 143.7+ },+ {+ "hour": 15,+ "decisions_per_s": 177.3,+ "p99_ms": 109.4+ },+ {+ "hour": 16,+ "decisions_per_s": 163.6,+ "p99_ms": 115.8+ },+ {+ "hour": 17,+ "decisions_per_s": 140.4,+ "p99_ms": 90.0+ },+ {+ "hour": 18,+ "decisions_per_s": 115.3,+ "p99_ms": 85.9+ },+ {+ "hour": 19,+ "decisions_per_s": 105.0,+ "p99_ms": 79.8+ },+ {+ "hour": 20,+ "decisions_per_s": 98.7,+ "p99_ms": 77.7+ },+ {+ "hour": 21,+ "decisions_per_s": 89.4,+ "p99_ms": 69.9+ },+ {+ "hour": 22,+ "decisions_per_s": 80.3,+ "p99_ms": 68.3+ },+ {+ "hour": 23,+ "decisions_per_s": 71.7,+ "p99_ms": 65.8+ }+ ]+ }+}diff --git a/research/scripts/bench_nvidia_capacity.py b/research/scripts/bench_nvidia_capacity.pynew file mode 100644index 00000000..9a23a82c--- /dev/null+++ b/research/scripts/bench_nvidia_capacity.py@@ -0,0 +1,421 @@+"""Capacity of one NVIDIA GPU serving `laya` under a p99 SLO: dynamic-batch HTTP server + open-loop load.++The laya-only part of the independent study in BENCHMARKS.md ("Independent NVIDIA CUDA capacity study"),+condensed from https://github.com/bhushankinge/laya-cuda-bench at 6cf4148 (harness/server.py, loadgen.py,+sweep.py, export_onnx.py). Result: research/results/nvidia_capacity_20260925.json.++Workload: the frozen SAM.gov mix and the 3-question bundle from that repo's fixtures/workload/ at 6cf4148+(states.jsonl, questions.json, diurnal.csv). Every request = one notice + the same three questions.++ USE_TF=0 python research/scripts/bench_nvidia_capacity.py sweep --model-dir ~/laya_models/laya \+ --workload fixtures/workload --backend eager-fp16 --max-batch 128 --start-qps 50 --out sweep.json+ USE_TF=0 python research/scripts/bench_nvidia_capacity.py export --model-dir ~/laya_models/laya+ LAYA_TRT_MAX_BATCH=256 USE_TF=0 python research/scripts/bench_nvidia_capacity.py sweep ... --backend trt-fp16+ USE_TF=0 python research/scripts/bench_nvidia_capacity.py replay ... --curve fixtures/workload/diurnal.csv \+ --total-decisions 10000000 --compress 24 --out day.jsonl++Backends: `eager-fp16` (torch autocast FP16) and `trt-fp16`. TensorRT is not a laya backend: `trt-fp16` runs+ONNX Runtime's TensorRT execution provider (onnxruntime-gpu + tensorrt, installed separately) over an FP16+ONNX export made by `export`, with LayerNorm kept in FP32.++Sweep: geometric steps from --start-qps (Poisson arrivals, --warmup s excluded, then --duration s measured),+up while the server keeps up, down first if the first rate is already saturated. Saturated = errors, p99 > 4x+the loosest SLO, achieved < 80% of target, or latency trend > 2. It then extends downward until the tightest+SLO is met, and adds one step per SLO at the geometric mean of the best passing and lowest failing rate.+Sustained = achieved >= 90% of target, p99 <= SLO and latency trend <= 1.5; decisions/s = 3 x requests/s.+Latency = completion minus *scheduled* send time, so client lag counts against the server.+"""+import argparse+import asyncio+import json+import math+import os+import random+import subprocess+import sys+import time+import urllib.request+from concurrent.futures import ThreadPoolExecutor+from pathlib import Path++os.environ.setdefault("USE_TF", "0")+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")+# "eager" means stock aten kernels: torch 2.14 otherwise routes some eager ops to Triton via torch._native+os.environ.setdefault("TORCH_DISABLE_NATIVE_JIT", "1")+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))++import numpy as np # noqa: E402++INPUTS = ["input_ids", "attention_mask", "marker_pos", "marker_mask", "qtype"]+++# ---- workload ---------------------------------------------------------------------------------------------++def load_workload(d):+ """Notices at most 472 `laya` tokens (749 of the 1,000), plus the 3-question bundle."""+ d = Path(d)+ states = [json.loads(l) for l in (d / "states.jsonl").read_text().splitlines() if l.strip()]+ return [s["text"] for s in states if s["tokens"]["laya"] <= 472], json.loads((d / "questions.json").read_text())+++# ---- server -----------------------------------------------------------------------------------------------++def make_forward(agent, backend, model_dir):+ """fn(batch of host tensors) -> (logits[n,k], act_probs[n,2]) as float32 numpy."""+ import torch+ if backend == "eager-fp16":+ agent.model.to(agent.device)++ @torch.inference_mode()+ def fn(b):+ b = {k: v.to(agent.device, non_blocking=True) for k, v in b.items()}+ with torch.autocast(device_type="cuda", dtype=torch.float16):+ logits, act = agent.model(*(b[k] for k in INPUTS))+ return logits.float().cpu().numpy(), torch.softmax(act.float(), -1).cpu().numpy()+ return fn+ if backend == "trt-fp16":+ import ctypes+ import importlib.util++ import onnxruntime as ort+ agent.model.to("cpu") # ORT holds its own weights+ libdir = Path(importlib.util.find_spec("tensorrt_libs").origin).parent # pip TensorRT is not on the loader path+ for lib in ("libnvinfer.so.10", "libnvinfer_plugin.so.10", "libnvonnxparser.so.10"):+ ctypes.CDLL(str(libdir / lib), mode=ctypes.RTLD_GLOBAL)+ mb, L = int(os.environ.get("LAYA_TRT_MAX_BATCH", 256)), agent.cfg.get("max_len", 512)+ shp = lambda b, l, k: f"input_ids:{b}x{l},attention_mask:{b}x{l},marker_pos:{b}x{k},marker_mask:{b}x{k},qtype:{b}"+ so = ort.SessionOptions()+ so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL+ opts = {"device_id": 0, "trt_fp16_enable": True, "trt_layer_norm_fp32_fallback": True,+ "trt_engine_cache_enable": True, "trt_engine_cache_path": str(Path(model_dir) / "trt_cache"),+ "trt_timing_cache_enable": True, "trt_profile_min_shapes": shp(1, 8, 2),+ "trt_profile_opt_shapes": shp(16, 256, 4), "trt_profile_max_shapes": shp(mb, L, 20)}+ sess = ort.InferenceSession(str(Path(model_dir) / "onnx" / "laya-fp16.onnx"), so,+ providers=[("TensorrtExecutionProvider", opts), ("CUDAExecutionProvider", {"device_id": 0})])+ if sess.get_providers()[0] != "TensorrtExecutionProvider": # ORT falls back silently; never mislabel a row+ raise RuntimeError(f"TensorRT EP unavailable, session runs on {sess.get_providers()}")++ def fn(b):+ logits, act = sess.run(None, {k: b[k].numpy() for k in INPUTS})+ return np.asarray(logits, np.float32), np.asarray(act, np.float32)+ return fn+ raise ValueError(backend)+++def forward_requests(agent, fn, reqs):+ """Every (state, question) pair of every request becomes one row, built by upstream build_sequence;+ calibration and answer schema mirror Agent.predict, so only the forward pass differs."""+ from laya.common import (+ QTYPES,+ build_sequence,+ collate_items,+ confidence_from_probs,+ render_options,+ temp_bucket,+ )+ max_len, head_max_len = agent.cfg.get("max_len", 512), agent.cfg.get("head_max_len", 192)+ items, meta = [], []+ for r, (state, questions) in enumerate(reqs):+ for qid, qdef in questions.items():+ q = agent._to_internal(qdef)+ ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)+ if len(markers) != len(render_options(q)):+ raise ValueError("question %r options exceed head_max_len=%d" % (qid, head_max_len))+ items.append({"ids": ids, "markers": markers, "qtype": QTYPES[q["t"]]})+ meta.append((r, qid, q, len(markers), QTYPES[q["t"]]))+ b = collate_items([items], agent.tok.pad_token_id)+ logits, act = fn({k: b[k] for k in INPUTS})+ answers, tokens = [dict() for _ in reqs], [0] * len(reqs)+ for i, ((r, qid, q, k, qt), t) in enumerate(zip(meta, b["attention_mask"].sum(1).tolist())):+ z = np.asarray(logits[i, :k], np.float64) / max(1e-3, float(agent.temperature_by_options.get(temp_bucket(qt, k), agent.temperature[qt])))+ p = np.exp(z - z.max())+ p /= p.sum()+ conf, ext = round(confidence_from_probs(p, k), 4), {"act_probability": round(float(act[i, 0]), 4)}+ if q["t"] == "choice":+ keys = list(q["crit"].keys())+ a = {"type": "choice", "choice": keys[int(p.argmax())], "probabilities": {kk: round(float(v), 4) for kk, v in zip(keys, p)},+ "confidence": conf, "action": ext}+ elif q["t"] == "score":+ a = {"type": "score", "score": round(float((np.arange(k) * p).sum()), 4),+ "legend": {str(j): c for j, c in enumerate(q["crit"])},+ "probabilities": {str(j): round(float(v), 4) for j, v in enumerate(p)}, "confidence": conf, "action": ext}+ else:+ a = {"type": "noul", "noul": round(float(p[1]), 4), "confidence": round(max(float(p[1]), 1.0 - float(p[1])), 4), "action": ext}+ answers[r][qid] = a+ tokens[r] += int(t)+ return answers, tokens+++def serve(args):+ """Triton-style dynamic batcher: take the first waiting request, collect until max_batch rows or+ max_delay, run one forward on a single worker thread, answer everyone."""+ from aiohttp import web++ from laya.agent import Agent++ agent = Agent(args.model_dir, device="cuda")+ fn = make_forward(agent, args.backend, args.model_dir)+ states, questions = load_workload(args.workload)+ for n in (1, min(16, args.max_batch), args.max_batch): # compile / engine build before accepting traffic+ forward_requests(agent, fn, [(states[i % len(states)], questions) for i in range(max(1, n // 3))])+ q, pool = asyncio.Queue(), ThreadPoolExecutor(max_workers=1)++ async def batcher():+ loop = asyncio.get_running_loop()+ while True:+ pending = [await q.get()]+ rows, deadline = len(pending[0][0][1]), loop.time() + args.max_delay_ms / 1e3+ while rows < args.max_batch and (timeout := deadline - loop.time()) > 0:+ try:+ pending.append(await asyncio.wait_for(q.get(), timeout))+ except asyncio.TimeoutError:+ break+ rows += len(pending[-1][0][1])+ try:+ answers, toks = await loop.run_in_executor(pool, forward_requests, agent, fn, [p[0] for p in pending])+ for (_, fut), a, t in zip(pending, answers, toks):+ if not fut.done():+ fut.set_result({"answers": a, "usage": {"input_tokens": t}, "served_batch": rows})+ except Exception as e:+ for _, fut in pending:+ if not fut.done():+ fut.set_exception(e)++ async def predict(request):+ body = await request.json()+ if not isinstance(body.get("questions"), dict):+ raise web.HTTPBadRequest(text="need {state, questions}")+ fut = asyncio.get_running_loop().create_future()+ await q.put(((body.get("state", ""), body["questions"]), fut))+ try:+ return web.json_response(await fut)+ except ValueError as e:+ raise web.HTTPBadRequest(text=str(e))++ async def healthz(_):+ return web.json_response({"ok": True})++ async def start(app):+ app["task"] = asyncio.create_task(batcher())++ app = web.Application(client_max_size=4 * 2**20)+ app.on_startup.append(start)+ app.router.add_post("/predict", predict)+ app.router.add_get("/healthz", healthz)+ web.run_app(app, host="127.0.0.1", port=args.port, print=None)+++def start_server(args, timeout=900):+ cmd = [sys.executable, __file__, "serve", "--model-dir", args.model_dir, "--workload", args.workload,+ "--backend", args.backend, "--max-batch", str(args.max_batch), "--max-delay-ms", str(args.max_delay_ms), "--port", str(args.port)]+ proc, t0 = subprocess.Popen(cmd), time.time()+ while time.time() - t0 < timeout:+ try:+ urllib.request.urlopen(f"http://127.0.0.1:{args.port}/healthz", timeout=2)+ return proc+ except Exception:+ if proc.poll() is not None:+ raise RuntimeError("server exited early")+ time.sleep(1)+ proc.kill()+ raise TimeoutError("server did not become healthy")+++# ---- open-loop load ---------------------------------------------------------------------------------------++def schedule_poisson(qps, duration, rng):+ t, out = 0.0, []+ while t < duration:+ t += rng.expovariate(qps)+ out.append(t)+ return out+++def schedule_curve(curve_csv, total_decisions, compress, rng):+ """curve_csv: hour,weight (24 rows); total_decisions is the day's volume. Each hour is replayed for+ 3600/compress s at that hour's real arrival rate, so the run walks the day's shape without inflating load."""+ weights = [float(l.split(",")[1]) for l in Path(curve_csv).read_text().splitlines()[1:] if l.strip()]+ total_req, hour_s, out = total_decisions / 3 / compress, 3600.0 / compress, []+ for h, w in enumerate(weights):+ qps, t = total_req * w / sum(weights) / hour_s, h * hour_s+ while t < (h + 1) * hour_s:+ t += rng.expovariate(qps)+ out.append(t)+ return out+++def summarize(rows):+ ok = [r for r in rows if r["status"] == 200]+ if not ok:+ return {"requests": len(rows), "ok": 0}+ lat = sorted(r["latency_ms"] for r in ok)+ span = max(r["t_done"] for r in ok) - min(r["t_sched"] for r in ok)+ pct = lambda p: lat[min(len(lat) - 1, int(math.ceil(p * len(lat))) - 1)]+ by_time, med = sorted(ok, key=lambda r: r["t_sched"]), lambda rs: sorted(r["latency_ms"] for r in rs)[len(rs) // 2]+ third = max(1, len(by_time) // 3)+ return {"requests": len(rows), "ok": len(ok), "errors": len(rows) - len(ok), "achieved_qps": len(ok) / span,+ "decisions_per_s": sum(r["n_decisions"] for r in ok) / span, "p50_ms": pct(0.5), "p95_ms": pct(0.95),+ "p99_ms": pct(0.99), "max_ms": lat[-1], "mean_served_batch": sum(r["served_batch"] for r in ok) / len(ok),+ # queue growth inside a step: the last third is slower than the first+ "latency_trend": med(by_time[-third:]) / max(1e-9, med(by_time[:third]))}+++async def run_load(url, states, questions, times, warmup_s, seed, out_path=None):+ import aiohttp+ rng, rows = random.Random(seed), []+ async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(limit=0), timeout=aiohttp.ClientTimeout(total=30)) as sess:+ start = time.perf_counter()++ async def one(t_sched):+ body, now = {"state": rng.choice(states), "questions": questions}, time.perf_counter() - start+ try:+ async with sess.post(url, json=body) as resp:+ js, status = (await resp.json() if resp.status == 200 else {}), resp.status+ except Exception:+ js, status = {}, 599+ done = time.perf_counter() - start+ rows.append({"t_sched": t_sched, "t_send": now, "t_done": done, "latency_ms": (done - t_sched) * 1e3,+ "status": status, "served_batch": js.get("served_batch", 0), "n_decisions": len(js.get("answers", {}))})++ tasks = []+ for t in times: # open loop: send on schedule regardless of responses, so queueing shows up as latency+ if (delay := t - (time.perf_counter() - start)) > 0:+ await asyncio.sleep(delay)+ tasks.append(asyncio.create_task(one(t)))+ await asyncio.gather(*tasks)+ if out_path:+ Path(out_path).write_text("".join(json.dumps(r) + "\n" for r in rows))+ return summarize([r for r in rows if r["t_sched"] >= warmup_s])+++# ---- commands ---------------------------------------------------------------------------------------------++def meets(s, slo):+ return bool(s.get("ok")) and s["p99_ms"] <= slo and s["latency_trend"] <= 1.5 and s["achieved_qps"] >= 0.9 * s["target_qps"]+++def sweep(args):+ states, questions = load_workload(args.workload)+ url, steps, qps = f"http://127.0.0.1:{args.port}/predict", [], args.start_qps+ proc = start_server(args)+ try:+ def step(qps):+ times = schedule_poisson(qps, args.duration + args.warmup, random.Random(len(steps)))+ s = asyncio.run(run_load(url, states, questions, times, args.warmup, len(steps)))+ s["target_qps"] = qps+ s["saturated"] = s.get("ok", 0) == 0 or s.get("errors", 0) > 0 or s.get("p99_ms", 1e9) > 4 * max(args.slo_ms) \+ or s.get("achieved_qps", 0) < 0.8 * qps or s.get("latency_trend", 9) > 2.0+ steps.append(s)+ print(f"target {qps:7.1f} q/s -> achieved {s.get('achieved_qps', 0):7.1f} p50 {s.get('p50_ms', 0):6.1f} "+ f"p99 {s.get('p99_ms', 0):7.1f} ms{' [saturated]' if s['saturated'] else ''}", flush=True)+ return s++ for _ in range(args.max_steps):+ if step(qps)["saturated"]:+ if any(not x["saturated"] for x in steps[:-1]):+ break+ qps /= args.factor+ else:+ qps *= args.factor+ for _ in range(4):+ if any(meets(s, min(args.slo_ms)) for s in steps):+ break+ step(min(s["target_qps"] for s in steps) / args.factor)+ for slo in args.slo_ms:+ good = [s["target_qps"] for s in steps if meets(s, slo)]+ bad = [s["target_qps"] for s in steps if good and s["target_qps"] > max(good) and not meets(s, slo)]+ if good and bad:+ step((max(good) * min(bad)) ** 0.5)+ finally:+ proc.terminate()+ proc.wait(timeout=30)+ sustained = {}+ for slo in args.slo_ms:+ good = [s for s in steps if meets(s, slo)]+ sustained[str(int(slo))] = max(good, key=lambda s: s["target_qps"]) if good else None+ print(f"p99 <= {slo:g} ms: " + (f"{sustained[str(int(slo))]['decisions_per_s']:.0f} decisions/s" if good else "not met"))+ report = {"backend": args.backend, "max_batch": args.max_batch, "max_delay_ms": args.max_delay_ms, "warmup_s": args.warmup,+ "step_s": args.duration, "slo_ms": args.slo_ms, "sustained": sustained,+ "steps": sorted(steps, key=lambda s: s["target_qps"])}+ Path(args.out).write_text(json.dumps(report, indent=1) + "\n")+ print("wrote", args.out)+++def replay(args):+ states, questions = load_workload(args.workload)+ times = schedule_curve(args.curve, args.total_decisions, args.compress, random.Random(0))+ proc = start_server(args)+ try:+ s = asyncio.run(run_load(f"http://127.0.0.1:{args.port}/predict", states, questions, times, 0, 0, args.out))+ finally:+ proc.terminate()+ proc.wait(timeout=30)+ print(json.dumps(s, indent=1))+++def export(args):+ """FP32 ONNX (dynamo exporter, opset 18, dynamic batch/sequence/options) and an FP16 copy for TensorRT."""+ import onnx+ import torch+ from onnxruntime.transformers.float16 import convert_float_to_float16++ from laya.agent import Agent++ class Wrapper(torch.nn.Module):+ def __init__(self, m):+ super().__init__()+ self.m = m++ def forward(self, input_ids, attention_mask, marker_pos, marker_mask, qtype):+ logits, act = self.m(input_ids, attention_mask, marker_pos, marker_mask, qtype)+ return logits, torch.softmax(act.float(), -1)++ w = Wrapper(Agent(args.model_dir, device="cpu").model.float().eval())+ out = Path(args.model_dir) / "onnx"+ out.mkdir(exist_ok=True)+ ex = (torch.randint(5, 1000, (2, 40)), torch.ones(2, 40, dtype=torch.long), torch.tensor([[3, 9, 15, 21], [3, 9, 0, 0]]),+ torch.tensor([[True] * 4, [True, True, False, False]]), torch.tensor([0, 2]))+ ex[1][1, 30:] = 0+ batch, seq, opts = torch.export.Dim("batch"), torch.export.Dim("seq", min=8), torch.export.Dim("options", min=2)+ # grad stays enabled: nn.TransformerEncoderLayer's fused fast path (not exportable) is only taken under no_grad+ prog = torch.onnx.export(w, ex, opset_version=18, dynamo=True, optimize=True, input_names=INPUTS, output_names=["logits", "act_probs"],+ dynamic_shapes={"input_ids": {0: batch, 1: seq}, "attention_mask": {0: batch, 1: seq},+ "marker_pos": {0: batch, 1: opts}, "marker_mask": {0: batch, 1: opts}, "qtype": {0: batch}})+ prog.save(str(out / "laya-fp32.onnx"), external_data=False)+ onnx.save(convert_float_to_float16(onnx.load(str(out / "laya-fp32.onnx")), keep_io_types=True), str(out / "laya-fp16.onnx"))+ print("wrote", out / "laya-fp16.onnx")+++def main():+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ sub = ap.add_subparsers(dest="cmd", required=True)+ for name in ("sweep", "replay", "serve", "export"):+ p = sub.add_parser(name)+ p.add_argument("--model-dir", required=True, help="laya checkpoint directory (the English `laya` checkpoint)")+ if name == "export":+ continue+ p.add_argument("--workload", required=True, help="directory with states.jsonl and questions.json")+ p.add_argument("--backend", choices=["eager-fp16", "trt-fp16"], default="eager-fp16")+ p.add_argument("--max-batch", type=int, default=64, help="rows per forward (requests x 3 questions)")+ p.add_argument("--max-delay-ms", type=float, default=2.0)+ p.add_argument("--port", type=int, default=8080)+ p.add_argument("--out")+ if name == "sweep":+ p.add_argument("--start-qps", type=float, default=5)+ p.add_argument("--factor", type=float, default=1.5)+ p.add_argument("--max-steps", type=int, default=20)+ p.add_argument("--warmup", type=float, default=20)+ p.add_argument("--duration", type=float, default=60)+ p.add_argument("--slo-ms", nargs="+", type=float, default=[50, 130])+ if name == "replay":+ p.add_argument("--curve", required=True)+ p.add_argument("--total-decisions", type=float, required=True)+ p.add_argument("--compress", type=float, default=24)+ args = ap.parse_args()+ if args.cmd == "sweep" and not args.out:+ ap.error("sweep needs --out")+ {"sweep": sweep, "replay": replay, "serve": serve, "export": export}[args.cmd](args)+++if __name__ == "__main__":+ main()