Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(reference): correct the INT8 quantization section (per-tensor default, honest accuracy caveat) (#790)
AlKor13b1831c77d3b7
diff --git a/docs/reference/agent.md b/docs/reference/agent.mdindex fe396b3d..75e95484 100644--- a/docs/reference/agent.md+++ b/docs/reference/agent.md@@ -13,15 +13,20 @@ exported ONNX model on CPU; import it from `laya.onnx_agent`.## Quantized export`scripts/export_onnx.py --quantize` writes an INT8 weight-only quantized copy beside the fp32-export (`laya.onnx` also produces `laya.int8.onnx`). Dynamic per-channel quantization converts-the `MatMul` weights to int8 with activations left in fp32, so no calibration dataset is needed,-and `ONNXAgent` loads the result by pointing `onnx_path` at it. On the English checkpoint, CPU-(M-series, 20 support-ticket states x choice/noul/score): model file 1.6 GB -> 581 MB, p50-per-state latency ~340 ms -> ~250 ms (~1.35x), and zero decision changes versus fp32 (largest single-probability drift 0.09). Per-tensor scales instead of per-channel flipped 3 of 20 states with-drift up to 0.29, which is why the exporter uses per-channel. The int8 graph is CPU-only: ONNX-Runtime has no INT8 MatMul kernel on the CUDAExecutionProvider, and a GPU provider silently-falls back per node.+export (`laya.onnx` also produces `laya.int8.onnx`). Dynamic quantization converts the `MatMul`+weights to int8 with the activation scale computed per input at run time, so no calibration+dataset is needed, and `ONNXAgent` loads the result by pointing `onnx_path` at it. On CPU it is+roughly 2x faster than the eager model and ~1.8x faster than the fp32 ONNX graph, and 1.4-2.8x+smaller depending on the checkpoint.++INT8 trades real accuracy, so it is a size/latency option, not a free one — do not use it where+the calibrated probability or confidence matters. Scales are **per-tensor** by default; `--per-channel`+opts into per-channel weights but on the dynamic path that collapses the decision model (agreement+with the eager model dropped to ~32% on the English checkpoint and ~40% on the multilingual one,+vs ~67% / ~83% per-tensor; see issue #790). Even per-tensor drifts noticeably on the larger+checkpoint; accuracy-safe int8 would need QAT or SmoothQuant-style outlier handling. The int8+graph is CPU-only: ONNX Runtime has no INT8 MatMul kernel on the CUDAExecutionProvider, and a GPU+provider silently falls back per node.```bashpython scripts/export_onnx.py --model convaiinnovations/laya --output laya.onnx --quantize