Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(benchmarks): back the NVIDIA capacity section with a committed results file and script

Bhushan Kinge721cf9f682b9
diff --git a/BENCHMARKS.md b/BENCHMARKS.md
index 5a6e0cbc..2a856763 100644
--- a/BENCHMARKS.md
+++ b/BENCHMARKS.md
@@ -7,6 +7,7 @@ Every checkpoint answered **byte-identical questions** in each run (fixed seed).
| T4 Colab | typed-decisions, MASSIVE (14 langs), XNLI (15 langs), English suites, latency, option-order robustness, calibration repair | `research/results/t4_colab_benchmark.json` |
| CPU sweep | MASSIVE intent across **all 51 languages**; its typed-decisions part (`part_b`) covers the English checkpoint only | `research/results/cpu_51_language_sweep.json` |
| Applications | the seven workflow themes + the datasets where Jev numbers exist, all three checkpoints (laya 0.2.1, CPU, 400 cases per task, seed 13, 2026-09-19) | `research/results/app_benchmark_results.json` |
+| NVIDIA capacity | `laya` served with dynamic batching under p99 SLOs on RTX PRO 5000, RTX PRO 6000 and H100 NVL (eager FP16, TensorRT FP16 via ONNX Runtime), backend parity, 24-hour replay | `research/results/nvidia_capacity_20260925.json` |
**Calibration columns in the CPU sweep predate the temperature clamp.** The 51-language ECE and mean-confidence figures were produced before #42 clamped temperatures to `[0.5, 5]`, so today's package reports different confidence for the affected buckets. Accuracy columns are unaffected, because a temperature-scaled softmax has the same argmax at every positive temperature.
@@ -277,7 +278,9 @@ SAM.gov notices. Each request asked the same three typed questions, so the table
(three decisions per request) at the highest tested load that met both the latency SLO and the achieved-rate gate.
The study pinned Laya to
[`6a58191`](https://github.com/NandhaKishorM/laya/commit/6a5819129eb220570792e417e49723d697efd76f)
-and recorded the checkpoint hashes, software versions and GPU environments.
+and recorded the checkpoint hashes, software versions and GPU environments. Sweep rows, parity counts, the replay
+summary and the environments: `research/results/nvidia_capacity_20260925.json`. Sweep and replay script:
+`research/scripts/bench_nvidia_capacity.py`.
| GPU and serving backend | p99 ≤ 50 ms | p99 ≤ 130 ms |
|---|---:|---:|
@@ -286,21 +289,24 @@ and recorded the checkpoint hashes, software versions and GPU environments.
| H100 NVL, TensorRT FP16 | 105 decisions/s | 175 decisions/s |
| H100 NVL, 7 × MIG 1g.12gb, eager FP16 | not met | not met reliably[^mig-rate] |
+TensorRT is not a laya backend. It is an external runtime: these rows ran ONNX Runtime's TensorRT execution
+provider (FP16, LayerNorm kept in FP32) over an ONNX export of the checkpoint. The eager rows are laya's own
+PyTorch model under FP16 autocast.
+
Every tested backend and precision reproduced the upstream FP32 answers on all 63 parity questions: 74 of 74
-backend-and-device rows passed across four GPUs. In fixed-shape throughput tests,
-`torch.compile(mode="max-autotune")` FP16 was 1.3–1.7× faster than eager FP16 at batch 256. Dynamic serving
-depended on the architecture: TensorRT raised the H100's 130 ms capacity from 93 to 175 decisions/s, while
-eager FP16 and TensorRT both reached 146 decisions/s on the RTX PRO 6000.
+backend-and-device rows passed across four GPUs. Dynamic serving depended on the architecture: TensorRT raised
+the H100's 130 ms capacity from 93 to 175 decisions/s, while eager FP16 and TensorRT both reached 146 decisions/s
+on the RTX PRO 6000.
-A compressed replay of a 24-hour, 10-million-decision curve on one RTX PRO 6000 completed 138,863 requests
-with zero errors and 111 ms overall p99. Two peak-hour segments reached 132 and 143 ms, so deployments that
-must hold 130 ms continuously need about 25% headroom at that volume.
+A compressed replay of a 24-hour, 10-million-decision curve on one RTX PRO 6000 (eager FP16) completed 138,863
+requests with zero errors and 111 ms overall p99. Two peak-hour segments reached 132 and 144 ms, so deployments
+that must hold 130 ms continuously need about 25% headroom at that volume.
These are capacity results for one English federal-procurement workload, not general latency guarantees. Server
sweeps and replays used one run per configuration; the RTX PRO 6000 sweep did not search below 50 requests/s;
-and `torch.compile` was not tested as a serving backend. The
-[full report](https://github.com/bhushankinge/laya-cuda-bench/blob/6cf4148ef4d148273eae83604815fb63ca2254de/REPORT.md),
-[raw results and harness](https://github.com/bhushankinge/laya-cuda-bench/tree/6cf4148ef4d148273eae83604815fb63ca2254de) are public.
+and `torch.compile` was not tested as a serving backend. Raw per-request laya results: sweeps for the
+[RTX PRO 5000](https://github.com/bhushankinge/laya-cuda-bench/tree/6cf4148ef4d148273eae83604815fb63ca2254de/results/zbook-rtxpro5000/server), [RTX PRO 6000](https://github.com/bhushankinge/laya-cuda-bench/tree/6cf4148ef4d148273eae83604815fb63ca2254de/results/rtxpro6000-ws/server) and
+[H100 NVL](https://github.com/bhushankinge/laya-cuda-bench/tree/6cf4148ef4d148273eae83604815fb63ca2254de/results/h100nvl/server), and the [day replay](https://github.com/bhushankinge/laya-cuda-bench/tree/6cf4148ef4d148273eae83604815fb63ca2254de/results/rtxpro6000-ws/replay).
[^mig-rate]: Seven concurrent slices reached about 49 decisions/s at p99 127 ms at the lowest load, but the
achieved request rate fell below the study's 90% gate. Higher loads missed the 130 ms SLO.
diff --git a/research/README.md b/research/README.md
index 9ce68d7c..a43df803 100644
--- a/research/README.md
+++ b/research/README.md
@@ -25,6 +25,7 @@ the numbers quoted in the main README — nothing here is imported by the `laya`
| `scripts/bench_apps.py` | the six application workflows (support triage, email + phishing, guardrails, RAG relevance, moderation, model routing) plus the datasets where public Jev numbers exist |
| `scripts/bench_latency.py` | inference speed including what routing costs: detection overhead, hot path, cold-swap, mixed-language throughput at several `max_loaded` settings |
| `scripts/bench_length_batching.py` | compare upstream contiguous batches with optional length sorting on synthetic tickets, including output consistency and optional fresh-process memory profiles (`psutil` required for memory mode) |
+| `scripts/bench_nvidia_capacity.py` | `laya` as a dynamically batched HTTP service on one NVIDIA GPU: open-loop Poisson sweep to the highest rate meeting a p99 SLO, 24-hour curve replay, and the ONNX export the TensorRT rows need (TensorRT via ONNX Runtime, not a laya backend). Results: `results/nvidia_capacity_20260925.json` |
| `scripts/make_plots.py` | renders `assets/laya_benchmark.png` from the result JSONs |
| `scripts/bench_long_context.py` | `laya-multilingual` on long documents: 20 support requests in 8 languages, each placed after 0 to 7,000 tokens of unrelated text, scored at the default limit and at `max_len=8192` |
| `scripts/plot_long_context.py` | renders `assets/long_context_8192.png` from `results/long_context_multilingual.json` |
diff --git a/research/results/nvidia_capacity_20260925.json b/research/results/nvidia_capacity_20260925.json
new file mode 100644
index 00000000..8f17544c
--- /dev/null
+++ b/research/results/nvidia_capacity_20260925.json
@@ -0,0 +1,1610 @@
+{
+ "description": "Independent NVIDIA CUDA capacity study of laya as a dynamically batched HTTP service (trimmed, laya-only). Workload: 1,000 public SAM.gov notices, natural length mix (<= 472 laya tokens); each request asks the same 3 typed questions (choice, score, noul), so decisions/s = 3 x requests/s. A sustained point needs p99 <= SLO, achieved >= 90% of the offered rate and latency trend <= 1.5 over a 60 s step.",
+ "source": {
+ "repo": "https://github.com/bhushankinge/laya-cuda-bench",
+ "commit": "6cf4148ef4d148273eae83604815fb63ca2254de",
+ "file_paths_relative_to": "https://github.com/bhushankinge/laya-cuda-bench/blob/6cf4148ef4d148273eae83604815fb63ca2254de/"
+ },
+ "script": "research/scripts/bench_nvidia_capacity.py",
+ "laya": {
+ "commit": "6a5819129eb220570792e417e49723d697efd76f",
+ "checkpoint": "convaiinnovations/laya",
+ "checkpoint_revision": "c5d78730f3493e4fe16d61507ef4b78eef7318cf",
+ "model_safetensors_sha256": "891102d372688fc2a094dac56a384bc537b87c63f21f9f3dac0be2b7cbc8d86c"
+ },
+ "tensorrt_note": "TensorRT rows run ONNX Runtime's TensorRT execution provider over an ONNX export of the checkpoint (FP16, LayerNorm kept in FP32). TensorRT is an external runtime, not a laya-shipped backend.",
+ "measured": "2026-09-22 to 2026-09-23, one run per configuration",
+ "environment": {
+ "zbook-rtxpro5000": {
+ "box": "zbook-rtxpro5000",
+ "collected_at": "2026-09-22T19:32:27.811144+00:00",
+ "os": "Linux-6.17.0-35-generic-x86_64-with-glibc2.39",
+ "python": "3.12.3",
+ "cpu_count": 24,
+ "ram_gb": 66.8,
+ "torch_disable_native_jit": "1",
+ "laya_trt_max_batch": "256",
+ "packages": {
+ "torch": "2.14.0+cu130",
+ "transformers": "5.17.0",
+ "onnxruntime-gpu": "1.30.0",
+ "onnxruntime": null,
+ "tensorrt-cu13": "10.16.1.11",
+ "tensorrt": null,
+ "laya": "0.3.3",
+ "nvidia-ml-py": "13.610.43",
+ "numpy": "2.5.3",
+ "huggingface_hub": "1.32.0",
+ "typesafe-sdk": "0.7.1"
+ },
+ "torch_cuda": "13.0",
+ "gpu": {
+ "name": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "capability": "12.0",
+ "total_mem_gb": 25.1
+ },
+ "driver": "595.71.05",
+ "mig_mode": "[N/A]",
+ "power_limit_w": "[N/A]"
+ },
+ "rtxpro6000-ws": {
+ "box": "rtxpro6000-ws",
+ "collected_at": "2026-09-23T03:17:22.051199+00:00",
+ "os": "Linux-6.17.0-40-generic-x86_64-with-glibc2.39",
+ "python": "3.12.3",
+ "cpu_count": 24,
+ "ram_gb": 100.6,
+ "torch_disable_native_jit": "1",
+ "laya_trt_max_batch": "256",
+ "packages": {
+ "torch": "2.14.0+cu130",
+ "transformers": "5.17.0",
+ "onnxruntime-gpu": "1.30.0",
+ "onnxruntime": null,
+ "tensorrt-cu13": "10.16.1.11",
+ "tensorrt": null,
+ "laya": "0.3.3",
+ "nvidia-ml-py": "13.610.43",
+ "numpy": "2.5.3",
+ "huggingface_hub": "1.32.0",
+ "typesafe-sdk": "0.7.1"
+ },
+ "torch_cuda": "13.0",
+ "gpu": {
+ "name": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "capability": "12.0",
+ "total_mem_gb": 102.0
+ },
+ "driver": "595.84",
+ "mig_mode": "[N/A]",
+ "power_limit_w": "600.00 W"
+ },
+ "h100nvl": {
+ "box": "h100nvl",
+ "collected_at": "2026-09-23T19:12:58.800605+00:00",
+ "os": "Linux-4.18.0-553.104.1.el8_10.x86_64-x86_64-with-glibc2.39",
+ "python": "3.12.3",
+ "cpu_count": 128,
+ "ram_gb": 540.2,
+ "torch_disable_native_jit": "1",
+ "laya_trt_max_batch": "256",
+ "packages": {
+ "torch": "2.14.0+cu130",
+ "transformers": "5.17.0",
+ "onnxruntime-gpu": "1.30.0",
+ "onnxruntime": null,
+ "tensorrt-cu13": "10.16.1.11",
+ "tensorrt": null,
+ "laya": "0.3.3",
+ "nvidia-ml-py": "13.610.43",
+ "numpy": "2.5.2",
+ "huggingface_hub": "1.32.0",
+ "typesafe-sdk": "0.7.1"
+ },
+ "torch_cuda": "13.0",
+ "gpu": {
+ "name": "NVIDIA H100 NVL",
+ "capability": "9.0",
+ "total_mem_gb": 99.9
+ },
+ "driver": "580.95.05",
+ "mig_mode": "Disabled",
+ "power_limit_w": "400.00 W"
+ },
+ "laptop-rtx2000ada": {
+ "box": "laptop-rtx2000ada",
+ "collected_at": "2026-09-22T16:03:52.795468+00:00",
+ "os": "Linux-6.17.0-1032-oem-x86_64-with-glibc2.39",
+ "python": "3.12.3",
+ "cpu_count": 28,
+ "ram_gb": 33.2,
+ "torch_disable_native_jit": null,
+ "laya_trt_max_batch": null,
+ "packages": {
+ "torch": "2.14.0+cu130",
+ "transformers": "5.17.0",
+ "onnxruntime-gpu": null,
+ "onnxruntime": null,
+ "tensorrt-cu13": null,
+ "tensorrt": null,
+ "laya": "0.3.3",
+ "nvidia-ml-py": "13.610.43",
+ "numpy": "2.5.3",
+ "huggingface_hub": "1.32.0",
+ "typesafe-sdk": "0.7.1"
+ },
+ "torch_cuda": "13.0",
+ "gpu": {
+ "name": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "capability": "8.9",
+ "total_mem_gb": 8.2
+ },
+ "driver": "590.48.01",
+ "mig_mode": "[N/A]",
+ "power_limit_w": "[N/A]"
+ },
+ "h100nvl-mig-1g.12gb": {
+ "box": "h100nvl",
+ "collected_at": "2026-09-23T22:44:12.234270+00:00",
+ "os": "Linux-4.18.0-553.104.1.el8_10.x86_64-x86_64-with-glibc2.39",
+ "python": "3.12.3",
+ "cpu_count": 128,
+ "ram_gb": 540.2,
+ "torch_disable_native_jit": "1",
+ "laya_trt_max_batch": "256",
+ "packages": {
+ "torch": "2.14.0+cu130",
+ "transformers": "5.17.0",
+ "onnxruntime-gpu": "1.30.0",
+ "onnxruntime": null,
+ "tensorrt-cu13": "10.16.1.11",
+ "tensorrt": null,
+ "laya": "0.3.3",
+ "nvidia-ml-py": "13.610.43",
+ "numpy": "2.5.2",
+ "huggingface_hub": "1.32.0",
+ "typesafe-sdk": "0.7.1"
+ },
+ "torch_cuda": "13.0",
+ "gpu": {
+ "name": "NVIDIA H100 NVL MIG 1g.12gb",
+ "capability": "9.0",
+ "total_mem_gb": 11.5
+ },
+ "driver": "580.95.05",
+ "mig_mode": "Enabled",
+ "power_limit_w": "400.00 W"
+ }
+ },
+ "slo_sweeps": [
+ {
+ "gpu": "RTX PRO 5000 Blackwell Laptop 24 GB",
+ "box": "zbook-rtxpro5000",
+ "file": "results/zbook-rtxpro5000/server/laya.ort-trt-fp16.sweep.json",
+ "backend": "ort-trt-fp16-maxb256",
+ "max_batch": 16,
+ "max_delay_ms": 2.0,
+ "warmup_s": 20.0,
+ "step_s": 60.0,
+ "sustained": {
+ "50": {
+ "target_qps": 5.0,
+ "achieved_qps": 5.006,
+ "decisions_per_s": 15.019,
+ "p99_ms": 45.331,
+ "p50_ms": 18.913
+ },
+ "130": {
+ "target_qps": 13.72,
+ "achieved_qps": 13.93,
+ "decisions_per_s": 41.791,
+ "p99_ms": 113.126,
+ "p50_ms": 20.084
+ }
+ },
+ "steps": [
+ {
+ "target_qps": 5.0,
+ "achieved_qps": 5.006,
+ "requests": 300,
+ "errors": 0,
+ "decisions_per_s": 15.019,
+ "p50_ms": 18.913,
+ "p95_ms": 33.36,
+ "p99_ms": 45.331,
+ "max_ms": 59.726,
+ "mean_served_batch": 3.14,
+ "latency_trend": 1.096,
+ "saturated": false
+ },
+ {
+ "target_qps": 7.0,
+ "achieved_qps": 6.67,
+ "requests": 402,
+ "errors": 0,
+ "decisions_per_s": 20.01,
+ "p50_ms": 19.044,
+ "p95_ms": 33.996,
+ "p99_ms": 60.609,
+ "max_ms": 71.188,
+ "mean_served_batch": 3.149,
+ "latency_trend": 0.947,
+ "saturated": false
+ },
+ {
+ "target_qps": 9.8,
+ "achieved_qps": 9.764,
+ "requests": 586,
+ "errors": 0,
+ "decisions_per_s": 29.292,
+ "p50_ms": 19.668,
+ "p95_ms": 40.148,
+ "p99_ms": 61.07,
+ "max_ms": 76.422,
+ "mean_served_batch": 3.205,
+ "latency_trend": 1.047,
+ "saturated": false
+ },
+ {
+ "target_qps": 13.72,
+ "achieved_qps": 13.93,
+ "requests": 836,
+ "errors": 0,
+ "decisions_per_s": 41.791,
+ "p50_ms": 20.084,
+ "p95_ms": 55.122,
+ "p99_ms": 113.126,
+ "max_ms": 154.412,
+ "mean_served_batch": 3.459,
+ "latency_trend": 1.049,
+ "saturated": false
+ },
+ {
+ "target_qps": 19.208,
+ "achieved_qps": 18.811,
+ "requests": 1129,
+ "errors": 0,
+ "decisions_per_s": 56.433,
+ "p50_ms": 22.796,
+ "p95_ms": 83.012,
+ "p99_ms": 135.269,
+ "max_ms": 208.868,
+ "mean_served_batch": 3.702,
+ "latency_trend": 0.944,
+ "saturated": false
+ },
+ {
+ "target_qps": 26.891,
+ "achieved_qps": 27.824,
+ "requests": 1673,
+ "errors": 0,
+ "decisions_per_s": 83.472,
+ "p50_ms": 30.928,
+ "p95_ms": 361.955,
+ "p99_ms": 628.47,
+ "max_ms": 775.337,
+ "mean_served_batch": 5.733,
+ "latency_trend": 0.888,
+ "saturated": true
+ }
+ ]
+ },
+ {
+ "gpu": "RTX PRO 5000 Blackwell Laptop 24 GB",
+ "box": "zbook-rtxpro5000",
+ "file": "results/zbook-rtxpro5000/server/laya.eager-fp16.sweep.json",
+ "backend": "torch-eager-fp16",
+ "max_batch": 16,
+ "max_delay_ms": 2.0,
+ "warmup_s": 20.0,
+ "step_s": 60.0,
+ "sustained": {
+ "50": null,
+ "130": {
+ "target_qps": 13.72,
+ "achieved_qps": 13.93,
+ "decisions_per_s": 41.79,
+ "p99_ms": 118.241,
+ "p50_ms": 25.027
+ }
+ },
+ "steps": [
+ {
+ "target_qps": 5.0,
+ "achieved_qps": 5.006,
+ "requests": 300,
+ "errors": 0,
+ "decisions_per_s": 15.018,
+ "p50_ms": 24.487,
+ "p95_ms": 42.937,
+ "p99_ms": 55.583,
+ "max_ms": 75.993,
+ "mean_served_batch": 3.16,
+ "latency_trend": 1.043,
+ "saturated": false
+ },
+ {
+ "target_qps": 7.0,
+ "achieved_qps": 6.669,
+ "requests": 402,
+ "errors": 0,
+ "decisions_per_s": 20.007,
+ "p50_ms": 23.649,
+ "p95_ms": 44.621,
+ "p99_ms": 61.364,
+ "max_ms": 86.137,
+ "mean_served_batch": 3.179,
+ "latency_trend": 0.952,
+ "saturated": false
+ },
+ {
+ "target_qps": 9.8,
+ "achieved_qps": 9.763,
+ "requests": 586,
+ "errors": 0,
+ "decisions_per_s": 29.289,
+ "p50_ms": 23.822,
+ "p95_ms": 54.293,
+ "p99_ms": 72.834,
+ "max_ms": 84.366,
+ "mean_served_batch": 3.256,
+ "latency_trend": 0.993,
+ "saturated": false
+ },
+ {
+ "target_qps": 13.72,
+ "achieved_qps": 13.93,
+ "requests": 836,
+ "errors": 0,
+ "decisions_per_s": 41.79,
+ "p50_ms": 25.027,
+ "p95_ms": 72.338,
+ "p99_ms": 118.241,
+ "max_ms": 202.486,
+ "mean_served_batch": 3.574,
+ "latency_trend": 1.062,
+ "saturated": false
+ },
+ {
+ "target_qps": 19.208,
+ "achieved_qps": 18.807,
+ "requests": 1129,
+ "errors": 0,
+ "decisions_per_s": 56.422,
+ "p50_ms": 29.164,
+ "p95_ms": 90.721,
+ "p99_ms": 143.14,
+ "max_ms": 193.706,
+ "mean_served_batch": 3.909,
+ "latency_trend": 1.03,
+ "saturated": false
+ },
+ {
+ "target_qps": 26.891,
+ "achieved_qps": 27.822,
+ "requests": 1673,
+ "errors": 0,
+ "decisions_per_s": 83.465,
+ "p50_ms": 52.473,
+ "p95_ms": 759.726,
+ "p99_ms": 947.794,
+ "max_ms": 1063.501,
+ "mean_served_batch": 8.025,
+ "latency_trend": 0.816,
+ "saturated": true
+ }
+ ]
+ },
+ {
+ "gpu": "RTX PRO 6000 Blackwell Workstation 96 GB",
+ "box": "rtxpro6000-ws",
+ "file": "results/rtxpro6000-ws/server/laya.ort-trt-fp16.sweep.json",
+ "backend": "ort-trt-fp16-maxb256",
+ "max_batch": 128,
+ "max_delay_ms": 2.0,
+ "warmup_s": 20.0,
+ "step_s": 60.0,
+ "sustained": {
+ "50": null,
+ "130": {
+ "target_qps": 50.0,
+ "achieved_qps": 48.561,
+ "decisions_per_s": 145.682,
+ "p99_ms": 67.54,
+ "p50_ms": 19.6
+ }
+ },
+ "steps": [
+ {
+ "target_qps": 50.0,
+ "achieved_qps": 48.561,
+ "requests": 2914,
+ "errors": 0,
+ "decisions_per_s": 145.682,
+ "p50_ms": 19.6,
+ "p95_ms": 50.224,
+ "p99_ms": 67.54,
+ "max_ms": 103.343,
+ "mean_served_batch": 4.792,
+ "latency_trend": 0.941,
+ "saturated": false
+ },
+ {
+ "target_qps": 70.0,
+ "achieved_qps": 69.589,
+ "requests": 4175,
+ "errors": 0,
+ "decisions_per_s": 208.766,
+ "p50_ms": 29.465,
+ "p95_ms": 116.168,
+ "p99_ms": 176.576,
+ "max_ms": 250.543,
+ "mean_served_batch": 8.238,
+ "latency_trend": 1.063,
+ "saturated": false
+ },
+ {
+ "target_qps": 98.0,
+ "achieved_qps": 88.605,
+ "requests": 5881,
+ "errors": 0,
+ "decisions_per_s": 265.814,
+ "p50_ms": 4280.761,
+ "p95_ms": 6263.83,
+ "p99_ms": 6660.771,
+ "max_ms": 6853.595,
+ "mean_served_batch": 128.793,
+ "latency_trend": 2.043,
+ "saturated": true
+ }
+ ]
+ },
+ {
+ "gpu": "RTX PRO 6000 Blackwell Workstation 96 GB",
+ "box": "rtxpro6000-ws",
+ "file": "results/rtxpro6000-ws/server/laya.eager-fp16.sweep.json",
+ "backend": "torch-eager-fp16",
+ "max_batch": 128,
+ "max_delay_ms": 2.0,
+ "warmup_s": 20.0,
+ "step_s": 60.0,
+ "sustained": {
+ "50": null,
+ "130": {
+ "target_qps": 50.0,
+ "achieved_qps": 48.541,
+ "decisions_per_s": 145.623,
+ "p99_ms": 86.466,
+ "p50_ms": 50.549
+ }
+ },
+ "steps": [
+ {
+ "target_qps": 50.0,
+ "achieved_qps": 48.541,
+ "requests": 2914,
+ "errors": 0,
+ "decisions_per_s": 145.623,
+ "p50_ms": 50.549,
+ "p95_ms": 74.713,
+ "p99_ms": 86.466,
+ "max_ms": 121.705,
+ "mean_served_batch": 7.465,
+ "latency_trend": 0.992,
+ "saturated": false
+ },
+ {
+ "target_qps": 70.0,
+ "achieved_qps": 69.568,
+ "requests": 4175,
+ "errors": 0,
+ "decisions_per_s": 208.704,
+ "p50_ms": 61.617,
+ "p95_ms": 140.363,
+ "p99_ms": 194.784,
+ "max_ms": 230.976,
+ "mean_served_batch": 12.364,
+ "latency_trend": 1.008,
+ "saturated": false
+ },
+ {
+ "target_qps": 98.0,
+ "achieved_qps": 62.866,
+ "requests": 5881,
+ "errors": 1046,
+ "decisions_per_s": 188.598,
+ "p50_ms": 12311.677,
+ "p95_ms": 16984.734,
+ "p99_ms": 17193.178,
+ "max_ms": 19999.371,
+ "mean_served_batch": 128.932,
+ "latency_trend": 2.278,
+ "saturated": true
+ }
+ ]
+ },
+ {
+ "gpu": "H100 NVL 94 GB",
+ "box": "h100nvl",
+ "file": "results/h100nvl/server/laya.ort-trt-fp16.sweep.json",
+ "backend": "ort-trt-fp16-maxb256",
+ "max_batch": 128,
+ "max_delay_ms": 2.0,
+ "warmup_s": 20.0,
+ "step_s": 60.0,
+ "sustained": {
+ "50": {
+ "target_qps": 35.714,
+ "achieved_qps": 34.902,
+ "decisions_per_s": 104.706,
+ "p99_ms": 46.062,
+ "p50_ms": 14.515
+ },
+ "130": {
+ "target_qps": 59.161,
+ "achieved_qps": 58.235,
+ "decisions_per_s": 174.706,
+ "p99_ms": 91.136,
+ "p50_ms": 18.792
+ }
+ },
+ "steps": [
+ {
+ "target_qps": 35.714,
+ "achieved_qps": 34.902,
+ "requests": 2094,
+ "errors": 0,
+ "decisions_per_s": 104.706,
+ "p50_ms": 14.515,
+ "p95_ms": 32.2,
+ "p99_ms": 46.062,
+ "max_ms": 81.516,
+ "mean_served_batch": 3.731,
+ "latency_trend": 1.005,
+ "saturated": false
+ },
+ {
+ "target_qps": 42.258,
+ "achieved_qps": 43.7,
+ "requests": 2621,
+ "errors": 0,
+ "decisions_per_s": 131.1,
+ "p50_ms": 15.844,
+ "p95_ms": 39.572,
+ "p99_ms": 52.629,
+ "max_ms": 75.421,
+ "mean_served_batch": 4.165,
+ "latency_trend": 1.088,
+ "saturated": false
+ },
+ {
+ "target_qps": 50.0,
+ "achieved_qps": 48.563,
+ "requests": 2914,
+ "errors": 0,
+ "decisions_per_s": 145.689,
+ "p50_ms": 16.98,
+ "p95_ms": 45.198,
+ "p99_ms": 63.734,
+ "max_ms": 98.071,
+ "mean_served_batch": 4.566,
+ "latency_trend": 0.97,
+ "saturated": false
+ },
+ {
+ "target_qps": 59.161,
+ "achieved_qps": 58.235,
+ "requests": 3498,
+ "errors": 0,
+ "decisions_per_s": 174.706,
+ "p50_ms": 18.792,
+ "p95_ms": 58.097,
+ "p99_ms": 91.136,
+ "max_ms": 164.121,
+ "mean_served_batch": 5.403,
+ "latency_trend": 0.988,
+ "saturated": false
+ },
+ {
+ "target_qps": 70.0,
+ "achieved_qps": 69.593,
+ "requests": 4175,
+ "errors": 0,
+ "decisions_per_s": 208.779,
+ "p50_ms": 23.847,
+ "p95_ms": 88.346,
+ "p99_ms": 135.857,
+ "max_ms": 212.034,
+ "mean_served_batch": 7.242,
+ "latency_trend": 1.054,
+ "saturated": false
+ },
+ {
+ "target_qps": 98.0,
+ "achieved_qps": 97.749,
+ "requests": 5881,
+ "errors": 0,
+ "decisions_per_s": 293.248,
+ "p50_ms": 170.396,
+ "p95_ms": 475.441,
+ "p99_ms": 706.513,
+ "max_ms": 884.172,
+ "mean_served_batch": 41.265,
+ "latency_trend": 1.574,
+ "saturated": true
+ }
+ ]
+ },
+ {
+ "gpu": "H100 NVL 94 GB",
+ "box": "h100nvl",
+ "file": "results/h100nvl/server/laya.eager-fp16.sweep.json",
+ "backend": "torch-eager-fp16",
+ "max_batch": 128,
+ "max_delay_ms": 2.0,
+ "warmup_s": 20.0,
+ "step_s": 60.0,
+ "sustained": {
+ "50": {
+ "target_qps": 13.015,
+ "achieved_qps": 13.051,
+ "decisions_per_s": 39.154,
+ "p99_ms": 38.84,
+ "p50_ms": 17.763
+ },
+ "130": {
+ "target_qps": 30.184,
+ "achieved_qps": 31.092,
+ "decisions_per_s": 93.275,
+ "p99_ms": 100.294,
+ "p50_ms": 19.305
+ }
+ },
+ "steps": [
+ {
+ "target_qps": 13.015,
+ "achieved_qps": 13.051,
+ "requests": 783,
+ "errors": 0,
+ "decisions_per_s": 39.154,
+ "p50_ms": 17.763,
+ "p95_ms": 30.516,
+ "p99_ms": 38.84,
+ "max_ms": 96.222,
+ "mean_served_batch": 3.383,
+ "latency_trend": 1.022,
+ "saturated": false
+ },
+ {
+ "target_qps": 15.4,
+ "achieved_qps": 15.947,
+ "requests": 956,
+ "errors": 0,
+ "decisions_per_s": 47.842,
+ "p50_ms": 17.869,
+ "p95_ms": 31.729,
+ "p99_ms": 88.714,
+ "max_ms": 126.199,
+ "mean_served_batch": 3.496,
+ "latency_trend": 1.002,
+ "saturated": false
+ },
+ {
+ "target_qps": 18.222,
+ "achieved_qps": 19.637,
+ "requests": 1178,
+ "errors": 0,
+ "decisions_per_s": 58.911,
+ "p50_ms": 18.076,
+ "p95_ms": 34.952,
+ "p99_ms": 95.025,
+ "max_ms": 117.211,
+ "mean_served_batch": 3.576,
+ "latency_trend": 0.977,
+ "saturated": false
+ },
+ {
+ "target_qps": 25.51,
+ "achieved_qps": 25.434,
+ "requests": 1529,
+ "errors": 0,
+ "decisions_per_s": 76.303,
+ "p50_ms": 18.744,
+ "p95_ms": 50.522,
+ "p99_ms": 124.008,
+ "max_ms": 176.933,
+ "mean_served_batch": 3.836,
+ "latency_trend": 0.969,
+ "saturated": false
+ },
+ {
+ "target_qps": 30.184,
+ "achieved_qps": 31.092,
+ "requests": 1866,
+ "errors": 0,
+ "decisions_per_s": 93.275,
+ "p50_ms": 19.305,
+ "p95_ms": 40.97,
+ "p99_ms": 100.294,
+ "max_ms": 143.284,
+ "mean_served_batch": 4.055,
+ "latency_trend": 1.012,
+ "saturated": false
+ },
+ {
+ "target_qps": 35.714,
+ "achieved_qps": 34.899,
+ "requests": 2094,
+ "errors": 0,
+ "decisions_per_s": 104.696,
+ "p50_ms": 22.107,
+ "p95_ms": 102.874,
+ "p99_ms": 135.348,
+ "max_ms": 178.08,
+ "mean_served_batch": 4.908,
+ "latency_trend": 0.826,
+ "saturated": false
+ },
+ {
+ "target_qps": 50.0,
+ "achieved_qps": 48.559,
+ "requests": 2914,
+ "errors": 0,
+ "decisions_per_s": 145.678,
+ "p50_ms": 100.56,
+ "p95_ms": 164.806,
+ "p99_ms": 199.552,
+ "max_ms": 248.792,
+ "mean_served_batch": 12.986,
+ "latency_trend": 0.854,
+ "saturated": false
+ },
+ {
+ "target_qps": 70.0,
+ "achieved_qps": 69.589,
+ "requests": 4175,
+ "errors": 0,
+ "decisions_per_s": 208.767,
+ "p50_ms": 106.299,
+ "p95_ms": 229.279,
+ "p99_ms": 327.726,
+ "max_ms": 491.874,
+ "mean_served_batch": 18.97,
+ "latency_trend": 0.837,
+ "saturated": false
+ },
+ {
+ "target_qps": 98.0,
+ "achieved_qps": 97.346,
+ "requests": 5881,
+ "errors": 0,
+ "decisions_per_s": 292.037,
+ "p50_ms": 401.076,
+ "p95_ms": 717.645,
+ "p99_ms": 842.341,
+ "max_ms": 960.329,
+ "mean_served_batch": 82.195,
+ "latency_trend": 0.908,
+ "saturated": true
+ }
+ ]
+ },
+ {
+ "gpu": "H100 NVL, 7 x MIG 1g.12gb, concurrent",
+ "box": "h100nvl",
+ "backend": "torch-eager-fp16",
+ "max_batch": 64,
+ "files": "results/h100nvl/server/laya.eager-fp16.mig-1g.12gb-slice{0..6}.sweep.json",
+ "sustained": {
+ "50": null,
+ "130": null
+ },
+ "steps": [
+ {
+ "target_qps_per_slice": 2.603,
+ "decisions_per_s_total": 48.61,
+ "p99_ms_min": 125.918,
+ "p99_ms_max": 127.171,
+ "achieved_over_target_min": 0.889,
+ "errors": 0,
+ "slices_meeting_130ms": 0,
+ "slices_meeting_50ms": 0
+ },
+ {
+ "target_qps_per_slice": 3.644,
+ "decisions_per_s_total": 72.699,
+ "p99_ms_min": 188.392,
+ "p99_ms_max": 456.968,
+ "achieved_over_target_min": 0.949,
+ "errors": 0,
+ "slices_meeting_130ms": 0,
+ "slices_meeting_50ms": 0
+ },
+ {
+ "target_qps_per_slice": 5.102,
+ "decisions_per_s_total": 115.614,
+ "p99_ms_min": 152.471,
+ "p99_ms_max": 285.66,
+ "achieved_over_target_min": 1.079,
+ "errors": 0,
+ "slices_meeting_130ms": 0,
+ "slices_meeting_50ms": 0
+ },
+ {
+ "target_qps_per_slice": 7.143,
+ "decisions_per_s_total": 149.68,
+ "p99_ms_min": 222.046,
+ "p99_ms_max": 246.109,
+ "achieved_over_target_min": 0.998,
+ "errors": 0,
+ "slices_meeting_130ms": 0,
+ "slices_meeting_50ms": 0
+ },
+ {
+ "target_qps_per_slice": 10.0,
+ "decisions_per_s_total": 217.081,
+ "p99_ms_min": 363.269,
+ "p99_ms_max": 410.228,
+ "achieved_over_target_min": 1.034,
+ "errors": 0,
+ "slices_meeting_130ms": 0,
+ "slices_meeting_50ms": 0
+ },
+ {
+ "target_qps_per_slice": 14.0,
+ "decisions_per_s_total": 283.667,
+ "p99_ms_min": 583.931,
+ "p99_ms_max": 1110.721,
+ "achieved_over_target_min": 0.962,
+ "errors": 0,
+ "slices_meeting_130ms": 0,
+ "slices_meeting_50ms": 0
+ }
+ ]
+ }
+ ],
+ "parity": {
+ "reference": "upstream FP32 forward at the pinned laya commit",
+ "cases": 16,
+ "questions_per_row": 63,
+ "rows": 74,
+ "rows_all_63_agree": 74,
+ "gpus": 4,
+ "files": "results/*/parity*.json",
+ "by_row": [
+ {
+ "gpu": "NVIDIA H100 NVL",
+ "model": "laya",
+ "backend": "torch-eager-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00673
+ },
+ {
+ "gpu": "NVIDIA H100 NVL",
+ "model": "laya",
+ "backend": "torch-eager-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0
+ },
+ {
+ "gpu": "NVIDIA H100 NVL",
+ "model": "laya",
+ "backend": "ort-cuda-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0063
+ },
+ {
+ "gpu": "NVIDIA H100 NVL",
+ "model": "laya",
+ "backend": "ort-trt-fp16-maxb256",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0149
+ },
+ {
+ "gpu": "NVIDIA H100 NVL",
+ "model": "laya-multilingual",
+ "backend": "torch-eager-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0011
+ },
+ {
+ "gpu": "NVIDIA H100 NVL",
+ "model": "laya-multilingual",
+ "backend": "torch-eager-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0
+ },
+ {
+ "gpu": "NVIDIA H100 NVL",
+ "model": "laya-multilingual",
+ "backend": "ort-cuda-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.000977
+ },
+ {
+ "gpu": "NVIDIA H100 NVL",
+ "model": "laya-multilingual",
+ "backend": "ort-trt-fp16-maxb256",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00189
+ },
+ {
+ "gpu": "NVIDIA H100 NVL",
+ "model": "laya",
+ "backend": "torch-compile-max-autotune-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00561
+ },
+ {
+ "gpu": "NVIDIA H100 NVL",
+ "model": "laya-multilingual",
+ "backend": "torch-compile-max-autotune-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.000728
+ },
+ {
+ "gpu": "NVIDIA H100 NVL MIG 1g.12gb",
+ "model": "laya",
+ "backend": "torch-eager-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00672
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya",
+ "backend": "torch-eager-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya",
+ "backend": "torch-eager-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00305
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya",
+ "backend": "torch-eager-bf16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0146
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya",
+ "backend": "ort-cuda-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 3.2e-06
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya",
+ "backend": "ort-cuda-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00752
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya",
+ "backend": "torch-compile-max-autotune-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00331
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya-multilingual",
+ "backend": "torch-eager-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya-multilingual",
+ "backend": "torch-eager-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.000773
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya-multilingual",
+ "backend": "torch-eager-bf16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00437
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya-multilingual",
+ "backend": "ort-cuda-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 5.99e-07
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya-multilingual",
+ "backend": "ort-cuda-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00151
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya-multilingual",
+ "backend": "torch-compile-max-autotune-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.000679
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya-typed-decisions",
+ "backend": "torch-eager-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya-typed-decisions",
+ "backend": "torch-eager-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00129
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya-typed-decisions",
+ "backend": "torch-eager-bf16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00692
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya-typed-decisions",
+ "backend": "ort-cuda-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 1.43e-06
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya-typed-decisions",
+ "backend": "ort-cuda-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00242
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya-typed-decisions",
+ "backend": "torch-compile-max-autotune-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00137
+ },
+ {
+ "gpu": "NVIDIA RTX 2000 Ada Generation Laptop GPU",
+ "model": "laya",
+ "backend": "ort-trt-fp16-maxb32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0168
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya",
+ "backend": "torch-eager-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya",
+ "backend": "torch-eager-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00545
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya",
+ "backend": "torch-eager-bf16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0164
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya",
+ "backend": "ort-cuda-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 1.43e-06
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya",
+ "backend": "ort-cuda-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0101
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya",
+ "backend": "torch-compile-max-autotune-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00605
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya",
+ "backend": "ort-trt-fp16-maxb128",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00573
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-multilingual",
+ "backend": "torch-eager-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-multilingual",
+ "backend": "torch-eager-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.000531
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-multilingual",
+ "backend": "torch-eager-bf16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00606
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-multilingual",
+ "backend": "ort-cuda-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 7.09e-07
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-multilingual",
+ "backend": "ort-cuda-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.000995
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-multilingual",
+ "backend": "torch-compile-max-autotune-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.000867
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-multilingual",
+ "backend": "ort-trt-fp16-maxb128",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00173
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-typed-decisions",
+ "backend": "torch-eager-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-typed-decisions",
+ "backend": "torch-eager-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00122
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-typed-decisions",
+ "backend": "torch-eager-bf16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0148
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-typed-decisions",
+ "backend": "ort-cuda-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 1.71e-06
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-typed-decisions",
+ "backend": "ort-cuda-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00217
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-typed-decisions",
+ "backend": "torch-compile-max-autotune-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0012
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-typed-decisions",
+ "backend": "ort-trt-fp16-maxb128",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00772
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya",
+ "backend": "ort-trt-fp16-maxb256",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0188
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+ "model": "laya-multilingual",
+ "backend": "ort-trt-fp16-maxb256",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00173
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya",
+ "backend": "torch-eager-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya",
+ "backend": "torch-eager-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00529
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya",
+ "backend": "torch-eager-bf16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00897
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya",
+ "backend": "ort-cuda-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 6.53e-06
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya",
+ "backend": "ort-cuda-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00316
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya",
+ "backend": "torch-compile-max-autotune-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00953
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya",
+ "backend": "ort-trt-fp16-maxb64",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0115
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-multilingual",
+ "backend": "torch-eager-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-multilingual",
+ "backend": "torch-eager-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.000539
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-multilingual",
+ "backend": "torch-eager-bf16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00621
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-multilingual",
+ "backend": "ort-cuda-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 4.06e-07
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-multilingual",
+ "backend": "ort-cuda-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00101
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-multilingual",
+ "backend": "torch-compile-max-autotune-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.000671
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-multilingual",
+ "backend": "ort-trt-fp16-maxb64",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00236
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-typed-decisions",
+ "backend": "torch-eager-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.0
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-typed-decisions",
+ "backend": "torch-eager-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00233
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-typed-decisions",
+ "backend": "torch-eager-bf16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00786
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-typed-decisions",
+ "backend": "ort-cuda-fp32",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 1.62e-06
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-typed-decisions",
+ "backend": "ort-cuda-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00301
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-typed-decisions",
+ "backend": "torch-compile-max-autotune-fp16",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00146
+ },
+ {
+ "gpu": "NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU",
+ "model": "laya-typed-decisions",
+ "backend": "ort-trt-fp16-maxb64",
+ "argmax_agreements": 63,
+ "questions": 63,
+ "probability_max_abs_error": 0.00479
+ }
+ ]
+ },
+ "replay": {
+ "gpu": "RTX PRO 6000 Blackwell Workstation 96 GB",
+ "box": "rtxpro6000-ws",
+ "backend": "torch-eager-fp16",
+ "max_batch": 128,
+ "file": "results/rtxpro6000-ws/replay/laya.eager-fp16.10M-day.jsonl",
+ "curve": "fixtures/workload/diurnal.csv",
+ "total_decisions_per_day": 10000000,
+ "compress": 24,
+ "hour_replayed_s": 150,
+ "requests": 138863,
+ "errors": 0,
+ "p50_ms": 47.372,
+ "p90_ms": 70.068,
+ "p99_ms": 110.725,
+ "max_ms": 250.599,
+ "decisions_per_s_mean": 115.716,
+ "mean_served_batch": 7.287,
+ "per_hour": [
+ {
+ "hour": 0,
+ "decisions_per_s": 64.5,
+ "p99_ms": 65.8
+ },
+ {
+ "hour": 1,
+ "decisions_per_s": 54.7,
+ "p99_ms": 64.4
+ },
+ {
+ "hour": 2,
+ "decisions_per_s": 49.4,
+ "p99_ms": 64.0
+ },
+ {
+ "hour": 3,
+ "decisions_per_s": 48.2,
+ "p99_ms": 65.9
+ },
+ {
+ "hour": 4,
+ "decisions_per_s": 49.1,
+ "p99_ms": 65.3
+ },
+ {
+ "hour": 5,
+ "decisions_per_s": 57.4,
+ "p99_ms": 64.8
+ },
+ {
+ "hour": 6,
+ "decisions_per_s": 81.4,
+ "p99_ms": 71.1
+ },
+ {
+ "hour": 7,
+ "decisions_per_s": 118.7,
+ "p99_ms": 85.6
+ },
+ {
+ "hour": 8,
+ "decisions_per_s": 147.5,
+ "p99_ms": 92.8
+ },
+ {
+ "hour": 9,
+ "decisions_per_s": 177.4,
+ "p99_ms": 124.5
+ },
+ {
+ "hour": 10,
+ "decisions_per_s": 187.4,
+ "p99_ms": 132.2
+ },
+ {
+ "hour": 11,
+ "decisions_per_s": 182.8,
+ "p99_ms": 126.0
+ },
+ {
+ "hour": 12,
+ "decisions_per_s": 161.3,
+ "p99_ms": 110.8
+ },
+ {
+ "hour": 13,
+ "decisions_per_s": 170.7,
+ "p99_ms": 125.5
+ },
+ {
+ "hour": 14,
+ "decisions_per_s": 185.4,
+ "p99_ms": 143.7
+ },
+ {
+ "hour": 15,
+ "decisions_per_s": 177.3,
+ "p99_ms": 109.4
+ },
+ {
+ "hour": 16,
+ "decisions_per_s": 163.6,
+ "p99_ms": 115.8
+ },
+ {
+ "hour": 17,
+ "decisions_per_s": 140.4,
+ "p99_ms": 90.0
+ },
+ {
+ "hour": 18,
+ "decisions_per_s": 115.3,
+ "p99_ms": 85.9
+ },
+ {
+ "hour": 19,
+ "decisions_per_s": 105.0,
+ "p99_ms": 79.8
+ },
+ {
+ "hour": 20,
+ "decisions_per_s": 98.7,
+ "p99_ms": 77.7
+ },
+ {
+ "hour": 21,
+ "decisions_per_s": 89.4,
+ "p99_ms": 69.9
+ },
+ {
+ "hour": 22,
+ "decisions_per_s": 80.3,
+ "p99_ms": 68.3
+ },
+ {
+ "hour": 23,
+ "decisions_per_s": 71.7,
+ "p99_ms": 65.8
+ }
+ ]
+ }
+}
diff --git a/research/scripts/bench_nvidia_capacity.py b/research/scripts/bench_nvidia_capacity.py
new file mode 100644
index 00000000..9a23a82c
--- /dev/null
+++ b/research/scripts/bench_nvidia_capacity.py
@@ -0,0 +1,421 @@
+"""Capacity of one NVIDIA GPU serving `laya` under a p99 SLO: dynamic-batch HTTP server + open-loop load.
+
+The laya-only part of the independent study in BENCHMARKS.md ("Independent NVIDIA CUDA capacity study"),
+condensed from https://github.com/bhushankinge/laya-cuda-bench at 6cf4148 (harness/server.py, loadgen.py,
+sweep.py, export_onnx.py). Result: research/results/nvidia_capacity_20260925.json.
+
+Workload: the frozen SAM.gov mix and the 3-question bundle from that repo's fixtures/workload/ at 6cf4148
+(states.jsonl, questions.json, diurnal.csv). Every request = one notice + the same three questions.
+
+ USE_TF=0 python research/scripts/bench_nvidia_capacity.py sweep --model-dir ~/laya_models/laya \
+ --workload fixtures/workload --backend eager-fp16 --max-batch 128 --start-qps 50 --out sweep.json
+ USE_TF=0 python research/scripts/bench_nvidia_capacity.py export --model-dir ~/laya_models/laya
+ LAYA_TRT_MAX_BATCH=256 USE_TF=0 python research/scripts/bench_nvidia_capacity.py sweep ... --backend trt-fp16
+ USE_TF=0 python research/scripts/bench_nvidia_capacity.py replay ... --curve fixtures/workload/diurnal.csv \
+ --total-decisions 10000000 --compress 24 --out day.jsonl
+
+Backends: `eager-fp16` (torch autocast FP16) and `trt-fp16`. TensorRT is not a laya backend: `trt-fp16` runs
+ONNX Runtime's TensorRT execution provider (onnxruntime-gpu + tensorrt, installed separately) over an FP16
+ONNX export made by `export`, with LayerNorm kept in FP32.
+
+Sweep: geometric steps from --start-qps (Poisson arrivals, --warmup s excluded, then --duration s measured),
+up while the server keeps up, down first if the first rate is already saturated. Saturated = errors, p99 > 4x
+the loosest SLO, achieved < 80% of target, or latency trend > 2. It then extends downward until the tightest
+SLO is met, and adds one step per SLO at the geometric mean of the best passing and lowest failing rate.
+Sustained = achieved >= 90% of target, p99 <= SLO and latency trend <= 1.5; decisions/s = 3 x requests/s.
+Latency = completion minus *scheduled* send time, so client lag counts against the server.
+"""
+import argparse
+import asyncio
+import json
+import math
+import os
+import random
+import subprocess
+import sys
+import time
+import urllib.request
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+
+os.environ.setdefault("USE_TF", "0")
+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
+# "eager" means stock aten kernels: torch 2.14 otherwise routes some eager ops to Triton via torch._native
+os.environ.setdefault("TORCH_DISABLE_NATIVE_JIT", "1")
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import numpy as np # noqa: E402
+
+INPUTS = ["input_ids", "attention_mask", "marker_pos", "marker_mask", "qtype"]
+
+
+# ---- workload ---------------------------------------------------------------------------------------------
+
+def load_workload(d):
+ """Notices at most 472 `laya` tokens (749 of the 1,000), plus the 3-question bundle."""
+ d = Path(d)
+ states = [json.loads(l) for l in (d / "states.jsonl").read_text().splitlines() if l.strip()]
+ return [s["text"] for s in states if s["tokens"]["laya"] <= 472], json.loads((d / "questions.json").read_text())
+
+
+# ---- server -----------------------------------------------------------------------------------------------
+
+def make_forward(agent, backend, model_dir):
+ """fn(batch of host tensors) -> (logits[n,k], act_probs[n,2]) as float32 numpy."""
+ import torch
+ if backend == "eager-fp16":
+ agent.model.to(agent.device)
+
+ @torch.inference_mode()
+ def fn(b):
+ b = {k: v.to(agent.device, non_blocking=True) for k, v in b.items()}
+ with torch.autocast(device_type="cuda", dtype=torch.float16):
+ logits, act = agent.model(*(b[k] for k in INPUTS))
+ return logits.float().cpu().numpy(), torch.softmax(act.float(), -1).cpu().numpy()
+ return fn
+ if backend == "trt-fp16":
+ import ctypes
+ import importlib.util
+
+ import onnxruntime as ort
+ agent.model.to("cpu") # ORT holds its own weights
+ libdir = Path(importlib.util.find_spec("tensorrt_libs").origin).parent # pip TensorRT is not on the loader path
+ for lib in ("libnvinfer.so.10", "libnvinfer_plugin.so.10", "libnvonnxparser.so.10"):
+ ctypes.CDLL(str(libdir / lib), mode=ctypes.RTLD_GLOBAL)
+ mb, L = int(os.environ.get("LAYA_TRT_MAX_BATCH", 256)), agent.cfg.get("max_len", 512)
+ shp = lambda b, l, k: f"input_ids:{b}x{l},attention_mask:{b}x{l},marker_pos:{b}x{k},marker_mask:{b}x{k},qtype:{b}"
+ so = ort.SessionOptions()
+ so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
+ opts = {"device_id": 0, "trt_fp16_enable": True, "trt_layer_norm_fp32_fallback": True,
+ "trt_engine_cache_enable": True, "trt_engine_cache_path": str(Path(model_dir) / "trt_cache"),
+ "trt_timing_cache_enable": True, "trt_profile_min_shapes": shp(1, 8, 2),
+ "trt_profile_opt_shapes": shp(16, 256, 4), "trt_profile_max_shapes": shp(mb, L, 20)}
+ sess = ort.InferenceSession(str(Path(model_dir) / "onnx" / "laya-fp16.onnx"), so,
+ providers=[("TensorrtExecutionProvider", opts), ("CUDAExecutionProvider", {"device_id": 0})])
+ if sess.get_providers()[0] != "TensorrtExecutionProvider": # ORT falls back silently; never mislabel a row
+ raise RuntimeError(f"TensorRT EP unavailable, session runs on {sess.get_providers()}")
+
+ def fn(b):
+ logits, act = sess.run(None, {k: b[k].numpy() for k in INPUTS})
+ return np.asarray(logits, np.float32), np.asarray(act, np.float32)
+ return fn
+ raise ValueError(backend)
+
+
+def forward_requests(agent, fn, reqs):
+ """Every (state, question) pair of every request becomes one row, built by upstream build_sequence;
+ calibration and answer schema mirror Agent.predict, so only the forward pass differs."""
+ from laya.common import (
+ QTYPES,
+ build_sequence,
+ collate_items,
+ confidence_from_probs,
+ render_options,
+ temp_bucket,
+ )
+ max_len, head_max_len = agent.cfg.get("max_len", 512), agent.cfg.get("head_max_len", 192)
+ items, meta = [], []
+ for r, (state, questions) in enumerate(reqs):
+ for qid, qdef in questions.items():
+ q = agent._to_internal(qdef)
+ ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)
+ if len(markers) != len(render_options(q)):
+ raise ValueError("question %r options exceed head_max_len=%d" % (qid, head_max_len))
+ items.append({"ids": ids, "markers": markers, "qtype": QTYPES[q["t"]]})
+ meta.append((r, qid, q, len(markers), QTYPES[q["t"]]))
+ b = collate_items([items], agent.tok.pad_token_id)
+ logits, act = fn({k: b[k] for k in INPUTS})
+ answers, tokens = [dict() for _ in reqs], [0] * len(reqs)
+ for i, ((r, qid, q, k, qt), t) in enumerate(zip(meta, b["attention_mask"].sum(1).tolist())):
+ z = np.asarray(logits[i, :k], np.float64) / max(1e-3, float(agent.temperature_by_options.get(temp_bucket(qt, k), agent.temperature[qt])))
+ p = np.exp(z - z.max())
+ p /= p.sum()
+ conf, ext = round(confidence_from_probs(p, k), 4), {"act_probability": round(float(act[i, 0]), 4)}
+ if q["t"] == "choice":
+ keys = list(q["crit"].keys())
+ a = {"type": "choice", "choice": keys[int(p.argmax())], "probabilities": {kk: round(float(v), 4) for kk, v in zip(keys, p)},
+ "confidence": conf, "action": ext}
+ elif q["t"] == "score":
+ a = {"type": "score", "score": round(float((np.arange(k) * p).sum()), 4),
+ "legend": {str(j): c for j, c in enumerate(q["crit"])},
+ "probabilities": {str(j): round(float(v), 4) for j, v in enumerate(p)}, "confidence": conf, "action": ext}
+ else:
+ a = {"type": "noul", "noul": round(float(p[1]), 4), "confidence": round(max(float(p[1]), 1.0 - float(p[1])), 4), "action": ext}
+ answers[r][qid] = a
+ tokens[r] += int(t)
+ return answers, tokens
+
+
+def serve(args):
+ """Triton-style dynamic batcher: take the first waiting request, collect until max_batch rows or
+ max_delay, run one forward on a single worker thread, answer everyone."""
+ from aiohttp import web
+
+ from laya.agent import Agent
+
+ agent = Agent(args.model_dir, device="cuda")
+ fn = make_forward(agent, args.backend, args.model_dir)
+ states, questions = load_workload(args.workload)
+ for n in (1, min(16, args.max_batch), args.max_batch): # compile / engine build before accepting traffic
+ forward_requests(agent, fn, [(states[i % len(states)], questions) for i in range(max(1, n // 3))])
+ q, pool = asyncio.Queue(), ThreadPoolExecutor(max_workers=1)
+
+ async def batcher():
+ loop = asyncio.get_running_loop()
+ while True:
+ pending = [await q.get()]
+ rows, deadline = len(pending[0][0][1]), loop.time() + args.max_delay_ms / 1e3
+ while rows < args.max_batch and (timeout := deadline - loop.time()) > 0:
+ try:
+ pending.append(await asyncio.wait_for(q.get(), timeout))
+ except asyncio.TimeoutError:
+ break
+ rows += len(pending[-1][0][1])
+ try:
+ answers, toks = await loop.run_in_executor(pool, forward_requests, agent, fn, [p[0] for p in pending])
+ for (_, fut), a, t in zip(pending, answers, toks):
+ if not fut.done():
+ fut.set_result({"answers": a, "usage": {"input_tokens": t}, "served_batch": rows})
+ except Exception as e:
+ for _, fut in pending:
+ if not fut.done():
+ fut.set_exception(e)
+
+ async def predict(request):
+ body = await request.json()
+ if not isinstance(body.get("questions"), dict):
+ raise web.HTTPBadRequest(text="need {state, questions}")
+ fut = asyncio.get_running_loop().create_future()
+ await q.put(((body.get("state", ""), body["questions"]), fut))
+ try:
+ return web.json_response(await fut)
+ except ValueError as e:
+ raise web.HTTPBadRequest(text=str(e))
+
+ async def healthz(_):
+ return web.json_response({"ok": True})
+
+ async def start(app):
+ app["task"] = asyncio.create_task(batcher())
+
+ app = web.Application(client_max_size=4 * 2**20)
+ app.on_startup.append(start)
+ app.router.add_post("/predict", predict)
+ app.router.add_get("/healthz", healthz)
+ web.run_app(app, host="127.0.0.1", port=args.port, print=None)
+
+
+def start_server(args, timeout=900):
+ cmd = [sys.executable, __file__, "serve", "--model-dir", args.model_dir, "--workload", args.workload,
+ "--backend", args.backend, "--max-batch", str(args.max_batch), "--max-delay-ms", str(args.max_delay_ms), "--port", str(args.port)]
+ proc, t0 = subprocess.Popen(cmd), time.time()
+ while time.time() - t0 < timeout:
+ try:
+ urllib.request.urlopen(f"http://127.0.0.1:{args.port}/healthz", timeout=2)
+ return proc
+ except Exception:
+ if proc.poll() is not None:
+ raise RuntimeError("server exited early")
+ time.sleep(1)
+ proc.kill()
+ raise TimeoutError("server did not become healthy")
+
+
+# ---- open-loop load ---------------------------------------------------------------------------------------
+
+def schedule_poisson(qps, duration, rng):
+ t, out = 0.0, []
+ while t < duration:
+ t += rng.expovariate(qps)
+ out.append(t)
+ return out
+
+
+def schedule_curve(curve_csv, total_decisions, compress, rng):
+ """curve_csv: hour,weight (24 rows); total_decisions is the day's volume. Each hour is replayed for
+ 3600/compress s at that hour's real arrival rate, so the run walks the day's shape without inflating load."""
+ weights = [float(l.split(",")[1]) for l in Path(curve_csv).read_text().splitlines()[1:] if l.strip()]
+ total_req, hour_s, out = total_decisions / 3 / compress, 3600.0 / compress, []
+ for h, w in enumerate(weights):
+ qps, t = total_req * w / sum(weights) / hour_s, h * hour_s
+ while t < (h + 1) * hour_s:
+ t += rng.expovariate(qps)
+ out.append(t)
+ return out
+
+
+def summarize(rows):
+ ok = [r for r in rows if r["status"] == 200]
+ if not ok:
+ return {"requests": len(rows), "ok": 0}
+ lat = sorted(r["latency_ms"] for r in ok)
+ span = max(r["t_done"] for r in ok) - min(r["t_sched"] for r in ok)
+ pct = lambda p: lat[min(len(lat) - 1, int(math.ceil(p * len(lat))) - 1)]
+ by_time, med = sorted(ok, key=lambda r: r["t_sched"]), lambda rs: sorted(r["latency_ms"] for r in rs)[len(rs) // 2]
+ third = max(1, len(by_time) // 3)
+ return {"requests": len(rows), "ok": len(ok), "errors": len(rows) - len(ok), "achieved_qps": len(ok) / span,
+ "decisions_per_s": sum(r["n_decisions"] for r in ok) / span, "p50_ms": pct(0.5), "p95_ms": pct(0.95),
+ "p99_ms": pct(0.99), "max_ms": lat[-1], "mean_served_batch": sum(r["served_batch"] for r in ok) / len(ok),
+ # queue growth inside a step: the last third is slower than the first
+ "latency_trend": med(by_time[-third:]) / max(1e-9, med(by_time[:third]))}
+
+
+async def run_load(url, states, questions, times, warmup_s, seed, out_path=None):
+ import aiohttp
+ rng, rows = random.Random(seed), []
+ async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(limit=0), timeout=aiohttp.ClientTimeout(total=30)) as sess:
+ start = time.perf_counter()
+
+ async def one(t_sched):
+ body, now = {"state": rng.choice(states), "questions": questions}, time.perf_counter() - start
+ try:
+ async with sess.post(url, json=body) as resp:
+ js, status = (await resp.json() if resp.status == 200 else {}), resp.status
+ except Exception:
+ js, status = {}, 599
+ done = time.perf_counter() - start
+ rows.append({"t_sched": t_sched, "t_send": now, "t_done": done, "latency_ms": (done - t_sched) * 1e3,
+ "status": status, "served_batch": js.get("served_batch", 0), "n_decisions": len(js.get("answers", {}))})
+
+ tasks = []
+ for t in times: # open loop: send on schedule regardless of responses, so queueing shows up as latency
+ if (delay := t - (time.perf_counter() - start)) > 0:
+ await asyncio.sleep(delay)
+ tasks.append(asyncio.create_task(one(t)))
+ await asyncio.gather(*tasks)
+ if out_path:
+ Path(out_path).write_text("".join(json.dumps(r) + "\n" for r in rows))
+ return summarize([r for r in rows if r["t_sched"] >= warmup_s])
+
+
+# ---- commands ---------------------------------------------------------------------------------------------
+
+def meets(s, slo):
+ return bool(s.get("ok")) and s["p99_ms"] <= slo and s["latency_trend"] <= 1.5 and s["achieved_qps"] >= 0.9 * s["target_qps"]
+
+
+def sweep(args):
+ states, questions = load_workload(args.workload)
+ url, steps, qps = f"http://127.0.0.1:{args.port}/predict", [], args.start_qps
+ proc = start_server(args)
+ try:
+ def step(qps):
+ times = schedule_poisson(qps, args.duration + args.warmup, random.Random(len(steps)))
+ s = asyncio.run(run_load(url, states, questions, times, args.warmup, len(steps)))
+ s["target_qps"] = qps
+ s["saturated"] = s.get("ok", 0) == 0 or s.get("errors", 0) > 0 or s.get("p99_ms", 1e9) > 4 * max(args.slo_ms) \
+ or s.get("achieved_qps", 0) < 0.8 * qps or s.get("latency_trend", 9) > 2.0
+ steps.append(s)
+ print(f"target {qps:7.1f} q/s -> achieved {s.get('achieved_qps', 0):7.1f} p50 {s.get('p50_ms', 0):6.1f} "
+ f"p99 {s.get('p99_ms', 0):7.1f} ms{' [saturated]' if s['saturated'] else ''}", flush=True)
+ return s
+
+ for _ in range(args.max_steps):
+ if step(qps)["saturated"]:
+ if any(not x["saturated"] for x in steps[:-1]):
+ break
+ qps /= args.factor
+ else:
+ qps *= args.factor
+ for _ in range(4):
+ if any(meets(s, min(args.slo_ms)) for s in steps):
+ break
+ step(min(s["target_qps"] for s in steps) / args.factor)
+ for slo in args.slo_ms:
+ good = [s["target_qps"] for s in steps if meets(s, slo)]
+ bad = [s["target_qps"] for s in steps if good and s["target_qps"] > max(good) and not meets(s, slo)]
+ if good and bad:
+ step((max(good) * min(bad)) ** 0.5)
+ finally:
+ proc.terminate()
+ proc.wait(timeout=30)
+ sustained = {}
+ for slo in args.slo_ms:
+ good = [s for s in steps if meets(s, slo)]
+ sustained[str(int(slo))] = max(good, key=lambda s: s["target_qps"]) if good else None
+ print(f"p99 <= {slo:g} ms: " + (f"{sustained[str(int(slo))]['decisions_per_s']:.0f} decisions/s" if good else "not met"))
+ report = {"backend": args.backend, "max_batch": args.max_batch, "max_delay_ms": args.max_delay_ms, "warmup_s": args.warmup,
+ "step_s": args.duration, "slo_ms": args.slo_ms, "sustained": sustained,
+ "steps": sorted(steps, key=lambda s: s["target_qps"])}
+ Path(args.out).write_text(json.dumps(report, indent=1) + "\n")
+ print("wrote", args.out)
+
+
+def replay(args):
+ states, questions = load_workload(args.workload)
+ times = schedule_curve(args.curve, args.total_decisions, args.compress, random.Random(0))
+ proc = start_server(args)
+ try:
+ s = asyncio.run(run_load(f"http://127.0.0.1:{args.port}/predict", states, questions, times, 0, 0, args.out))
+ finally:
+ proc.terminate()
+ proc.wait(timeout=30)
+ print(json.dumps(s, indent=1))
+
+
+def export(args):
+ """FP32 ONNX (dynamo exporter, opset 18, dynamic batch/sequence/options) and an FP16 copy for TensorRT."""
+ import onnx
+ import torch
+ from onnxruntime.transformers.float16 import convert_float_to_float16
+
+ from laya.agent import Agent
+
+ class Wrapper(torch.nn.Module):
+ def __init__(self, m):
+ super().__init__()
+ self.m = m
+
+ def forward(self, input_ids, attention_mask, marker_pos, marker_mask, qtype):
+ logits, act = self.m(input_ids, attention_mask, marker_pos, marker_mask, qtype)
+ return logits, torch.softmax(act.float(), -1)
+
+ w = Wrapper(Agent(args.model_dir, device="cpu").model.float().eval())
+ out = Path(args.model_dir) / "onnx"
+ out.mkdir(exist_ok=True)
+ ex = (torch.randint(5, 1000, (2, 40)), torch.ones(2, 40, dtype=torch.long), torch.tensor([[3, 9, 15, 21], [3, 9, 0, 0]]),
+ torch.tensor([[True] * 4, [True, True, False, False]]), torch.tensor([0, 2]))
+ ex[1][1, 30:] = 0
+ batch, seq, opts = torch.export.Dim("batch"), torch.export.Dim("seq", min=8), torch.export.Dim("options", min=2)
+ # grad stays enabled: nn.TransformerEncoderLayer's fused fast path (not exportable) is only taken under no_grad
+ prog = torch.onnx.export(w, ex, opset_version=18, dynamo=True, optimize=True, input_names=INPUTS, output_names=["logits", "act_probs"],
+ dynamic_shapes={"input_ids": {0: batch, 1: seq}, "attention_mask": {0: batch, 1: seq},
+ "marker_pos": {0: batch, 1: opts}, "marker_mask": {0: batch, 1: opts}, "qtype": {0: batch}})
+ prog.save(str(out / "laya-fp32.onnx"), external_data=False)
+ onnx.save(convert_float_to_float16(onnx.load(str(out / "laya-fp32.onnx")), keep_io_types=True), str(out / "laya-fp16.onnx"))
+ print("wrote", out / "laya-fp16.onnx")
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ sub = ap.add_subparsers(dest="cmd", required=True)
+ for name in ("sweep", "replay", "serve", "export"):
+ p = sub.add_parser(name)
+ p.add_argument("--model-dir", required=True, help="laya checkpoint directory (the English `laya` checkpoint)")
+ if name == "export":
+ continue
+ p.add_argument("--workload", required=True, help="directory with states.jsonl and questions.json")
+ p.add_argument("--backend", choices=["eager-fp16", "trt-fp16"], default="eager-fp16")
+ p.add_argument("--max-batch", type=int, default=64, help="rows per forward (requests x 3 questions)")
+ p.add_argument("--max-delay-ms", type=float, default=2.0)
+ p.add_argument("--port", type=int, default=8080)
+ p.add_argument("--out")
+ if name == "sweep":
+ p.add_argument("--start-qps", type=float, default=5)
+ p.add_argument("--factor", type=float, default=1.5)
+ p.add_argument("--max-steps", type=int, default=20)
+ p.add_argument("--warmup", type=float, default=20)
+ p.add_argument("--duration", type=float, default=60)
+ p.add_argument("--slo-ms", nargs="+", type=float, default=[50, 130])
+ if name == "replay":
+ p.add_argument("--curve", required=True)
+ p.add_argument("--total-decisions", type=float, required=True)
+ p.add_argument("--compress", type=float, default=24)
+ args = ap.parse_args()
+ if args.cmd == "sweep" and not args.out:
+ ap.error("sweep needs --out")
+ {"sweep": sweep, "replay": replay, "serve": serve, "export": export}[args.cmd](args)
+
+
+if __name__ == "__main__":
+ main()