Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(ci): the Windows fault is bf16 CPU autocast, so qualify it in a child

Nandakishor390d0eeb9b43
diff --git a/docs/reference/agent.md b/docs/reference/agent.md
index 34ab5aed..90337a38 100644
--- a/docs/reference/agent.md
+++ b/docs/reference/agent.md
@@ -35,6 +35,42 @@ provider silently falls back per node.
python scripts/export_onnx.py --model convaiinnovations/laya --output laya.onnx --quantize
```
+## `LAYA_CPU_AMP=bf16` needs native BF16 instructions
+
+The CPU forward runs in fp32 unless `LAYA_CPU_AMP` is set to `bf16` or `bfloat16`
+(`laya.agent._cpu_amp_dtype`). It is opt-in because bf16 is only worth asking for on hardware
+with native BF16 instructions, and because on hardware without them the failure is not an
+exception. oneDNN selects a kernel for an instruction set the CPU does not have, and the process
+takes an illegal instruction: SIGILL on POSIX, `0xC000001D` (`STATUS_ILLEGAL_INSTRUCTION`) on
+Windows. `Agent._infer` retries a failed autocast in fp32, but it catches `RuntimeError`, and a
+fault is not raised to Python, so that fallback cannot help here.
+
+Measured on this project's own Windows CI, where the bf16 autocast forward in
+`tests/test_runtime_fixes.py` faulted on roughly three runs in thirty-five, on one runner label,
+because a hosted fleet is not one CPU model. The same class of failure is
+[#1065](https://github.com/NandhaKishorM/laya/issues/1065) below.
+
+So check the host before setting it, and check it where the server will run rather than where you
+develop:
+
+```python
+import torch
+print(torch.backends.cpu.get_cpu_capability()) # "AVX512" is the one that implies BF16 support
+```
+
+A capability of `"DEFAULT"` or `"NO AVX"` is a host to leave in fp32. The safe way to qualify an
+unfamiliar one is a child process, because a fault in the parent cannot be caught:
+
+```bash
+python -c "import torch, torch.nn as nn; x = torch.ones(1, 16); m = nn.Linear(16, 16)
+with torch.no_grad(), torch.autocast(device_type='cpu', dtype=torch.bfloat16):
+ m(x)
+print('bf16 CPU autocast runs here')"
+```
+
+A non-zero exit with no traceback is the fault, not a failed assertion. `tests/test_runtime_fixes.py`
+runs exactly this probe before it asserts anything about the autocast path.
+
## In-process dynamic quantization
The quantized export above is the int8 path `ONNXAgent` loads. `torch.ao.quantization.quantize_dynamic`
diff --git a/laya/agent.py b/laya/agent.py
index a89fded3..df2cae50 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -313,7 +313,13 @@ def _cpu_amp_dtype() -> Optional[torch.dtype]:
"""Autocast dtype on CPU: bf16 when LAYA_CPU_AMP=bf16|bfloat16, None to leave the forward in
fp32. No fp16 spelling is offered here, because CPU autocast has no fp16 fast path that beats
fp32 -- bf16 is the only reduced precision worth asking for, and only on hardware with native
- BF16 instructions, which is why it stays opt-in. Anything else is ignored."""
+ BF16 instructions, which is why it stays opt-in. Anything else is ignored.
+
+ On hardware without those instructions the failure is a native fault, not an exception:
+ oneDNN picks a kernel the CPU cannot execute and the process dies with SIGILL, or
+ `0xC000001D` on Windows. `_infer`'s autocast retry catches `RuntimeError`, so it cannot
+ see that and cannot fall back. Qualify the host first; `docs/reference/agent.md` has the
+ one-line child-process probe and the `get_cpu_capability()` reading to check."""
if os.environ.get("LAYA_CPU_AMP", "").lower() in ("bf16", "bfloat16"):
return torch.bfloat16
return None
diff --git a/tests/test_runtime_fixes.py b/tests/test_runtime_fixes.py
index e7d3e2e0..31f85254 100644
--- a/tests/test_runtime_fixes.py
+++ b/tests/test_runtime_fixes.py
@@ -6,6 +6,7 @@ Covers:
* the autocast path works and degrades to full precision instead of failing
"""
import os
+import subprocess
import sys
from contextlib import nullcontext
from types import SimpleNamespace
@@ -150,13 +151,48 @@ am = torch.ones_like(iid)
mp = torch.tensor([[1, 3]])
mm = torch.ones_like(mp, dtype=torch.bool)
qt = torch.tensor([0])
-with torch.no_grad():
- logits32, act32 = model(iid, am, mp, mm, qt)
- with torch.autocast(device_type="cpu", dtype=torch.bfloat16, enabled=True):
- logits16, act16 = model(iid, am, mp, mm, qt)
-check("autocast/shapes match", tuple(logits16.shape), tuple(logits32.shape))
-check("autocast/finite", bool(torch.isfinite(logits16).all() and torch.isfinite(act16).all()), True)
-check("autocast/close to fp32", float((logits32 - logits16).abs().max()) < 0.5, True)
+
+
+# Whether this CPU can execute a bf16 autocast forward at all, asked in a child process, because
+# the answer on some machines is a native fault that no `except` can see. `tests (windows)` has
+# killed this suite with `0xC000001D` (STATUS_ILLEGAL_INSTRUCTION) about three runs in
+# thirty-five, and the stage markers added for it narrowed the fault to this section: the lane
+# printed `[stage] state reused once` and `[stage] autocast` and then died. It is the same class
+# as issue #1065, where the fbgemm quantization engine raised SIGILL on a CPU with SSE4.2 and no
+# AVX2 -- oneDNN picks a kernel for an instruction set the CPU does not have. It is intermittent
+# on one runner label because the hosted fleet is not one CPU model.
+#
+# So the three checks below run only where the forward is executable, and where it is not the
+# suite says so with the capability torch reports instead of taking the process down. laya's own
+# behaviour is unchanged either way: `LAYA_CPU_AMP` is opt-in, and `_cpu_amp_dtype` documents that
+# it is for hardware with native BF16 instructions.
+_AMP_PROBE = (
+ "import torch, torch.nn as nn\n"
+ "m = nn.Linear(16, 16)\n"
+ "x = torch.ones(1, 16)\n"
+ "with torch.no_grad(), torch.autocast(device_type='cpu', dtype=torch.bfloat16, enabled=True):\n"
+ " m(x)\n"
+ "print('ok')\n"
+)
+_probe = subprocess.run([sys.executable, "-c", _AMP_PROBE], capture_output=True, text=True)
+_amp_runs = _probe.returncode == 0 and _probe.stdout.strip() == "ok"
+if not _amp_runs:
+ try:
+ _capability = torch.backends.cpu.get_cpu_capability()
+ except Exception as exc: # pragma: no cover
+ _capability = "unknown (%s)" % type(exc).__name__
+ stage("autocast: SKIPPED, this CPU cannot execute a bf16 autocast forward "
+ "(capability %s, child exit %s)" % (_capability, _probe.returncode))
+ PASS.append("autocast/skipped: bf16 CPU autocast is not executable here")
+else:
+ with torch.no_grad():
+ logits32, act32 = model(iid, am, mp, mm, qt)
+ with torch.autocast(device_type="cpu", dtype=torch.bfloat16, enabled=True):
+ logits16, act16 = model(iid, am, mp, mm, qt)
+ check("autocast/shapes match", tuple(logits16.shape), tuple(logits32.shape))
+ check("autocast/finite",
+ bool(torch.isfinite(logits16).all() and torch.isfinite(act16).all()), True)
+ check("autocast/close to fp32", float((logits32 - logits16).abs().max()) < 0.5, True)
calls = {"n": 0}