Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(ci): the Windows fault is bf16 CPU autocast, so qualify it in a child
Nandakishor390d0eeb9b43
diff --git a/docs/reference/agent.md b/docs/reference/agent.mdindex 34ab5aed..90337a38 100644--- a/docs/reference/agent.md+++ b/docs/reference/agent.md@@ -35,6 +35,42 @@ provider silently falls back per node.python scripts/export_onnx.py --model convaiinnovations/laya --output laya.onnx --quantize```+## `LAYA_CPU_AMP=bf16` needs native BF16 instructions++The CPU forward runs in fp32 unless `LAYA_CPU_AMP` is set to `bf16` or `bfloat16`+(`laya.agent._cpu_amp_dtype`). It is opt-in because bf16 is only worth asking for on hardware+with native BF16 instructions, and because on hardware without them the failure is not an+exception. oneDNN selects a kernel for an instruction set the CPU does not have, and the process+takes an illegal instruction: SIGILL on POSIX, `0xC000001D` (`STATUS_ILLEGAL_INSTRUCTION`) on+Windows. `Agent._infer` retries a failed autocast in fp32, but it catches `RuntimeError`, and a+fault is not raised to Python, so that fallback cannot help here.++Measured on this project's own Windows CI, where the bf16 autocast forward in+`tests/test_runtime_fixes.py` faulted on roughly three runs in thirty-five, on one runner label,+because a hosted fleet is not one CPU model. The same class of failure is+[#1065](https://github.com/NandhaKishorM/laya/issues/1065) below.++So check the host before setting it, and check it where the server will run rather than where you+develop:++```python+import torch+print(torch.backends.cpu.get_cpu_capability()) # "AVX512" is the one that implies BF16 support+```++A capability of `"DEFAULT"` or `"NO AVX"` is a host to leave in fp32. The safe way to qualify an+unfamiliar one is a child process, because a fault in the parent cannot be caught:++```bash+python -c "import torch, torch.nn as nn; x = torch.ones(1, 16); m = nn.Linear(16, 16)+with torch.no_grad(), torch.autocast(device_type='cpu', dtype=torch.bfloat16):+ m(x)+print('bf16 CPU autocast runs here')"+```++A non-zero exit with no traceback is the fault, not a failed assertion. `tests/test_runtime_fixes.py`+runs exactly this probe before it asserts anything about the autocast path.+## In-process dynamic quantizationThe quantized export above is the int8 path `ONNXAgent` loads. `torch.ao.quantization.quantize_dynamic`diff --git a/laya/agent.py b/laya/agent.pyindex a89fded3..df2cae50 100644--- a/laya/agent.py+++ b/laya/agent.py@@ -313,7 +313,13 @@ def _cpu_amp_dtype() -> Optional[torch.dtype]:"""Autocast dtype on CPU: bf16 when LAYA_CPU_AMP=bf16|bfloat16, None to leave the forward infp32. No fp16 spelling is offered here, because CPU autocast has no fp16 fast path that beatsfp32 -- bf16 is the only reduced precision worth asking for, and only on hardware with native- BF16 instructions, which is why it stays opt-in. Anything else is ignored."""+ BF16 instructions, which is why it stays opt-in. Anything else is ignored.++ On hardware without those instructions the failure is a native fault, not an exception:+ oneDNN picks a kernel the CPU cannot execute and the process dies with SIGILL, or+ `0xC000001D` on Windows. `_infer`'s autocast retry catches `RuntimeError`, so it cannot+ see that and cannot fall back. Qualify the host first; `docs/reference/agent.md` has the+ one-line child-process probe and the `get_cpu_capability()` reading to check."""if os.environ.get("LAYA_CPU_AMP", "").lower() in ("bf16", "bfloat16"):return torch.bfloat16return Nonediff --git a/tests/test_runtime_fixes.py b/tests/test_runtime_fixes.pyindex e7d3e2e0..31f85254 100644--- a/tests/test_runtime_fixes.py+++ b/tests/test_runtime_fixes.py@@ -6,6 +6,7 @@ Covers:* the autocast path works and degrades to full precision instead of failing"""import os+import subprocessimport sysfrom contextlib import nullcontextfrom types import SimpleNamespace@@ -150,13 +151,48 @@ am = torch.ones_like(iid)mp = torch.tensor([[1, 3]])mm = torch.ones_like(mp, dtype=torch.bool)qt = torch.tensor([0])-with torch.no_grad():- logits32, act32 = model(iid, am, mp, mm, qt)- with torch.autocast(device_type="cpu", dtype=torch.bfloat16, enabled=True):- logits16, act16 = model(iid, am, mp, mm, qt)-check("autocast/shapes match", tuple(logits16.shape), tuple(logits32.shape))-check("autocast/finite", bool(torch.isfinite(logits16).all() and torch.isfinite(act16).all()), True)-check("autocast/close to fp32", float((logits32 - logits16).abs().max()) < 0.5, True)+++# Whether this CPU can execute a bf16 autocast forward at all, asked in a child process, because+# the answer on some machines is a native fault that no `except` can see. `tests (windows)` has+# killed this suite with `0xC000001D` (STATUS_ILLEGAL_INSTRUCTION) about three runs in+# thirty-five, and the stage markers added for it narrowed the fault to this section: the lane+# printed `[stage] state reused once` and `[stage] autocast` and then died. It is the same class+# as issue #1065, where the fbgemm quantization engine raised SIGILL on a CPU with SSE4.2 and no+# AVX2 -- oneDNN picks a kernel for an instruction set the CPU does not have. It is intermittent+# on one runner label because the hosted fleet is not one CPU model.+#+# So the three checks below run only where the forward is executable, and where it is not the+# suite says so with the capability torch reports instead of taking the process down. laya's own+# behaviour is unchanged either way: `LAYA_CPU_AMP` is opt-in, and `_cpu_amp_dtype` documents that+# it is for hardware with native BF16 instructions.+_AMP_PROBE = (+ "import torch, torch.nn as nn\n"+ "m = nn.Linear(16, 16)\n"+ "x = torch.ones(1, 16)\n"+ "with torch.no_grad(), torch.autocast(device_type='cpu', dtype=torch.bfloat16, enabled=True):\n"+ " m(x)\n"+ "print('ok')\n"+)+_probe = subprocess.run([sys.executable, "-c", _AMP_PROBE], capture_output=True, text=True)+_amp_runs = _probe.returncode == 0 and _probe.stdout.strip() == "ok"+if not _amp_runs:+ try:+ _capability = torch.backends.cpu.get_cpu_capability()+ except Exception as exc: # pragma: no cover+ _capability = "unknown (%s)" % type(exc).__name__+ stage("autocast: SKIPPED, this CPU cannot execute a bf16 autocast forward "+ "(capability %s, child exit %s)" % (_capability, _probe.returncode))+ PASS.append("autocast/skipped: bf16 CPU autocast is not executable here")+else:+ with torch.no_grad():+ logits32, act32 = model(iid, am, mp, mm, qt)+ with torch.autocast(device_type="cpu", dtype=torch.bfloat16, enabled=True):+ logits16, act16 = model(iid, am, mp, mm, qt)+ check("autocast/shapes match", tuple(logits16.shape), tuple(logits32.shape))+ check("autocast/finite",+ bool(torch.isfinite(logits16).all() and torch.isfinite(act16).all()), True)+ check("autocast/close to fp32", float((logits32 - logits16).abs().max()) < 0.5, True)calls = {"n": 0}