Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(reference): correct the INT8 quantization section (per-tensor default, honest accuracy caveat) (#790)

AlKor13b1831c77d3b7
diff --git a/docs/reference/agent.md b/docs/reference/agent.md
index fe396b3d..75e95484 100644
--- a/docs/reference/agent.md
+++ b/docs/reference/agent.md
@@ -13,15 +13,20 @@ exported ONNX model on CPU; import it from `laya.onnx_agent`.
## Quantized export
`scripts/export_onnx.py --quantize` writes an INT8 weight-only quantized copy beside the fp32
-export (`laya.onnx` also produces `laya.int8.onnx`). Dynamic per-channel quantization converts
-the `MatMul` weights to int8 with activations left in fp32, so no calibration dataset is needed,
-and `ONNXAgent` loads the result by pointing `onnx_path` at it. On the English checkpoint, CPU
-(M-series, 20 support-ticket states x choice/noul/score): model file 1.6 GB -> 581 MB, p50
-per-state latency ~340 ms -> ~250 ms (~1.35x), and zero decision changes versus fp32 (largest single
-probability drift 0.09). Per-tensor scales instead of per-channel flipped 3 of 20 states with
-drift up to 0.29, which is why the exporter uses per-channel. The int8 graph is CPU-only: ONNX
-Runtime has no INT8 MatMul kernel on the CUDAExecutionProvider, and a GPU provider silently
-falls back per node.
+export (`laya.onnx` also produces `laya.int8.onnx`). Dynamic quantization converts the `MatMul`
+weights to int8 with the activation scale computed per input at run time, so no calibration
+dataset is needed, and `ONNXAgent` loads the result by pointing `onnx_path` at it. On CPU it is
+roughly 2x faster than the eager model and ~1.8x faster than the fp32 ONNX graph, and 1.4-2.8x
+smaller depending on the checkpoint.
+
+INT8 trades real accuracy, so it is a size/latency option, not a free one — do not use it where
+the calibrated probability or confidence matters. Scales are **per-tensor** by default; `--per-channel`
+opts into per-channel weights but on the dynamic path that collapses the decision model (agreement
+with the eager model dropped to ~32% on the English checkpoint and ~40% on the multilingual one,
+vs ~67% / ~83% per-tensor; see issue #790). Even per-tensor drifts noticeably on the larger
+checkpoint; accuracy-safe int8 would need QAT or SmoothQuant-style outlier handling. The int8
+graph is CPU-only: ONNX Runtime has no INT8 MatMul kernel on the CUDAExecutionProvider, and a GPU
+provider silently falls back per node.
```bash
python scripts/export_onnx.py --model convaiinnovations/laya --output laya.onnx --quantize