Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(agent): reject unparseable LAYA_CUDA_AMP and turn autocast off for fp32
Som Tripathi46509b91b4ef
diff --git a/README.md b/README.mdindex c3bfebd0..92f39d48 100644--- a/README.md+++ b/README.md@@ -1035,7 +1035,7 @@ else:A threshold is a policy you choose from measured accuracy at that coverage on your data, not a property of the model. Both checkpoints are over-confident as shipped and `laya-multilingual` has no fitted temperatures at all, so fit them before relying on these numbers — see [Calibration](#calibration) below, and the [fine-tuning notebook](https://github.com/NandhaKishorM/laya/blob/main/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb) for the fitting loop itself. Then pick the point where the errors you accept are ones you can live with. Confidence orders decisions; it does not establish that a decision is correct.-A threshold also depends on the autocast dtype. On CUDA at compute capability 8 or above the runtime uses the checkpoint's `amp_dtype`, which is bf16 for all three shipped checkpoints. On the fixed set from `benchmarks/parity_fast.py` (60 states, 288 questions per checkpoint, RTX 2000 Ada) bf16 moves a probability by up to 0.073 against the fp32 forward and flips 3 of 864 argmaxes across the three checkpoints; fp16 stays within 0.019 and flips none, at the same latency. `LAYA_CUDA_AMP=fp16` selects fp16 and `LAYA_CUDA_AMP=bf16` selects bf16 (`LAYA_CPU_AMP=bf16` is the CPU counterpart). MPS autocasts in fp16 too, but the overhead dominates on a single small row, so there it engages only once a call reaches `mps_amp_min_rows` rows -- 5 by default, `LAYA_MPS_AMP_MIN_ROWS` to move it; a value that does not parse falls back to 5 and anything below 1 is clamped to 1. On MPS, `agent.dtype` is the autocast target, so it says float16 even when every call stays below the gate and runs in fp32; `agent.dtype_for(rows)` returns the precision a call with that many rows runs in. Fit and measure a threshold in the dtype you serve with.+A threshold also depends on the autocast dtype. On CUDA at compute capability 8 or above the runtime uses the checkpoint's `amp_dtype`, which is bf16 for all three shipped checkpoints. On the fixed set from `benchmarks/parity_fast.py` (60 states, 288 questions per checkpoint, RTX 2000 Ada) bf16 moves a probability by up to 0.073 against the fp32 forward and flips 3 of 864 argmaxes across the three checkpoints; fp16 stays within 0.019 and flips none, at the same latency. `LAYA_CUDA_AMP=fp16` selects fp16, `LAYA_CUDA_AMP=bf16` selects bf16 and `LAYA_CUDA_AMP=fp32` turns autocast off for the full-precision forward those numbers are measured against (`LAYA_CPU_AMP=bf16` is the CPU counterpart). MPS autocasts in fp16 too, but the overhead dominates on a single small row, so there it engages only once a call reaches `mps_amp_min_rows` rows -- 5 by default, `LAYA_MPS_AMP_MIN_ROWS` to move it; a value that does not parse falls back to 5 and anything below 1 is clamped to 1. On MPS, `agent.dtype` is the autocast target, so it says float16 even when every call stays below the gate and runs in fp32; `agent.dtype_for(rows)` returns the precision a call with that many rows runs in. Fit and measure a threshold in the dtype you serve with.### Opt-in abstention: `min_confidence`diff --git a/docs/docker.md b/docs/docker.mdindex 27f34c6f..5a480570 100644--- a/docs/docker.md+++ b/docs/docker.md@@ -68,7 +68,7 @@ work with `docker run -e`; Compose-only settings are identified below.| Variable | Default | Purpose || --- | --- | --- || `LAYA_DEVICE` | `cpu` / `cuda` | Device selected by the base / GPU configuration |-| `LAYA_CUDA_AMP` | unset (checkpoint's `amp_dtype`) | `fp16`/`float16` or `bf16`/`bfloat16` for the CUDA forward; anything else is ignored. Not cosmetic: the README's threshold section measures bf16 flipping 3 of 864 argmaxes on the parity set where fp16 flips none |+| `LAYA_CUDA_AMP` | unset (checkpoint's `amp_dtype`) | `fp16`/`float16` or `bf16`/`bfloat16` for the CUDA forward, or `fp32`/`float32` to turn autocast off; anything else fails the load with a `ValueError`. GPUs below compute capability 8 run fp16 for either half-precision spelling. Not cosmetic: the README's threshold section measures bf16 flipping 3 of 864 argmaxes on the parity set where fp16 flips none || `LAYA_CPU_AMP` | unset | `bf16` or `bfloat16` opts the CPU forward into bf16; anything else leaves it fp32. No fp16 spelling turns it on either: CPU autocast has no fp16 fast path that beats fp32, so bf16 is the only reduced precision core offers on this device || `LAYA_MODEL` | `auto` | Router alias: `auto`, `english`, `multilingual`, `typed-decisions` || `LAYA_MODEL_PATH` | unset | Compatible checkpoint path inside the container |diff --git a/laya/agent.py b/laya/agent.pyindex 8ebf2c06..dec08ac0 100644--- a/laya/agent.py+++ b/laya/agent.py@@ -296,16 +296,26 @@ def _mps_amp_min_rows() -> int:def _cuda_amp_dtype(checkpoint_default: Optional[str]) -> torch.dtype:- """Autocast dtype on CUDA at compute capability >= 8: the checkpoint's `amp_dtype` (bf16 for the- shipped checkpoints), or LAYA_CUDA_AMP=fp16|float16|bf16|bfloat16 when set. fp16 stays 2-10x- closer to the fp32 forward than bf16 on every shipped checkpoint at the same speed; anything- else is ignored. The long spellings count because they are what `agent.dtype` reports back, so- the precision a user reads off one response is one they can ask for on the next."""- raw = os.environ.get("LAYA_CUDA_AMP", "").lower()+ """Precision asked for on CUDA: the checkpoint's `amp_dtype` (bf16 for the shipped checkpoints),+ or LAYA_CUDA_AMP=fp16|float16|bf16|bfloat16|fp32|float32 when set. fp16 stays 2-10x closer to+ the fp32 forward than bf16 on every shipped checkpoint at the same speed. float32 means no+ autocast at all, and the caller turns it off rather than autocasting to float32. Below compute+ capability 8 the caller runs fp16 for either half-precision ask. The long spellings count+ because they are what `agent.dtype` reports back, so the precision a user reads off one+ response is one they can ask for on the next. The value is trimmed and lower-cased, an empty+ one means unset, and anything else raises ValueError: a typo fails the load instead of+ silently serving the checkpoint's choice."""+ raw = os.environ.get("LAYA_CUDA_AMP", "").strip().lower()if raw in ("fp16", "float16"):return torch.float16if raw in ("bf16", "bfloat16"):return torch.bfloat16+ if raw in ("fp32", "float32"):+ return torch.float32+ if raw:+ raise ValueError("LAYA_CUDA_AMP=%r is not a CUDA precision; use fp16, float16, bf16, bfloat16, "+ "fp32 or float32, or leave it unset for the checkpoint's own amp_dtype"+ % os.environ["LAYA_CUDA_AMP"])return amp_dtype(checkpoint_default)@@ -796,20 +806,26 @@ class Agent(HookRegistry):self.mps_amp_min_rows = _mps_amp_min_rows()if self.device.type == "cuda":self.amp_enabled = True- try:- major = torch.cuda.get_device_capability(self.device)[0]- except (RuntimeError, AttributeError):- # A driver that reports a GPU but cannot answer the capability query- # cannot be trusted with bf16 either; fp16 is the safe default and- # placement below still decides whether the device is usable at all.- warnings.warn("Warning: could not query CUDA capability; assuming fp16 defaults.",- RuntimeWarning)- self.dtype = torch.float16+ # Read on every CUDA device, so a value that does not parse fails here whatever the+ # GPU, and fp32 means full precision on all of them (#443).+ requested = _cuda_amp_dtype(self.cfg.get("amp_dtype", "fp16"))+ if requested == torch.float32:+ self.amp_enabled = Falseelse:- if major < 8:+ try:+ major = torch.cuda.get_device_capability(self.device)[0]+ except (RuntimeError, AttributeError):+ # A driver that reports a GPU but cannot answer the capability query+ # cannot be trusted with bf16 either; fp16 is the safe default and+ # placement below still decides whether the device is usable at all.+ warnings.warn("Warning: could not query CUDA capability; assuming fp16 defaults.",+ RuntimeWarning)self.dtype = torch.float16else:- self.dtype = _cuda_amp_dtype(self.cfg.get("amp_dtype", "fp16"))+ if major < 8:+ self.dtype = torch.float16+ else:+ self.dtype = requestedelif self.device.type == "mps":self.amp_enabled = Trueself.dtype = torch.float16@@ -907,6 +923,8 @@ class Agent(HookRegistry):matches the stock forward it replaces within rounding (see benchmarks/parity_fast.py). Afterchanging `agent.dtype`, call `deaccelerate()` then `accelerate()` to rebuild it. Returns True ifenabled. With `strict=False` any failure (no CUDA, tilelang missing) leaves the stock path in place.+ An fp32 agent (`LAYA_CUDA_AMP=fp32`) is such a failure: its kernels only run in 16 bits, so it+ warns and keeps the stock forward, or raises with `strict=True`."""if self._fast is not None:return True@@ -914,6 +932,13 @@ class Agent(HookRegistry):if strict:raise RuntimeError("laya fast path needs a CUDA device")return False+ if self.dtype not in (torch.bfloat16, torch.float16):+ msg = ("laya fast path runs in bf16 or fp16 only and this agent runs %s; "+ "keeping the stock forward" % self.dtype)+ if strict:+ raise RuntimeError(msg)+ warnings.warn("Warning: " + msg, RuntimeWarning)+ return Falseif self._backend is not None:self.deaccelerate()last = Nonediff --git a/nix/laya-serve.nix b/nix/laya-serve.nixindex a7ef1c1d..3f564dc3 100644--- a/nix/laya-serve.nix+++ b/nix/laya-serve.nix@@ -72,13 +72,13 @@ in`laya.agent`, not here: `tests/test_packaging.py` reads them out of theruntime and fails if this text drifts from it, which is why there is no`enum` on this option. The recognised spellings are `fp16`, `float16`,- `bf16` and `bfloat16` — the first two name one dtype and the last two- another. Anything else is ignored rather than an error, and the- checkpoint's own choice stays in force.+ `bf16`, `bfloat16`, `fp32` and `float32` — each pair names one dtype,+ and `fp32`/`float32` turn autocast off. Anything else is an error: the+ service fails at load instead of serving the checkpoint's own choice.- The type allows any single token but refuses whitespace, which is not- style: the runtime lower-cases the value and then compares it without- trimming, so a trailing space keeps full precision and says nothing.+ The type allows any single token but refuses whitespace. The runtime+ trims and lower-cases the value, so a stray space would change nothing;+ the type keeps what this module writes identical to what it reads.null leaves every CUDA checkpoint on its own `amp_dtype`.'';};diff --git a/tests/test_confidence.py b/tests/test_confidence.pyindex afe10b9b..770d2b3a 100644--- a/tests/test_confidence.py+++ b/tests/test_confidence.py@@ -2240,7 +2240,7 @@ for _fn in (test_tl_kernels_drops_the_wrong_row_claim,# --------------------------------------- DecisionResult's docstring must split confidence by question type# `laya/structured.py`'s `DecisionResult` closed with "confidence keeps the **normalized-entropy# value** it has always had, because that is a different quantity on a scale that depends on the-# label count". `Agent._decode_answers` (`laya/agent.py:1369-1398`) uses two different formulas:+# label count". `Agent._decode_answers` (`laya/agent.py:1394-1423`) uses two different formulas:# * `choice` and `score` -> `confidence_from_probs(p, k)` = `1 - H(p) / log(k)`# * `noul` -> `max(p_true, 1 - p_true)` (its own comment: "identical here:# over two options `max(p_true, 1 - p_true)` is `max(p)`")diff --git a/tests/test_env_docs.py b/tests/test_env_docs.pyindex 10d90d4b..1c2144b6 100644--- a/tests/test_env_docs.py+++ b/tests/test_env_docs.py@@ -74,11 +74,13 @@ SKIP_DIRS = {".git", ".github", ".cache", ".venv", "site", "__pycache__", "node_# and cannot become a permanent hole in the sweep.DEFERRED: dict = {}-# The half-precision spellings a document could offer for the two AMP names. Kept as the closed-# universe of things a prose site can mean rather than as an accepted list: the accepted list is-# derived from laya/agent.py below, and a spelling outside this set (`fp32`, `int8`) is not a-# value anyone offers -- it is what the forward falls back to, and pages name it for that.-DTYPE_SPELLINGS = ("fp16", "float16", "bf16", "bfloat16")+# The dtype spellings a document could offer for the two AMP names. Kept as the closed universe of+# things a prose site can mean rather than as an accepted list: the accepted list is derived from+# laya/agent.py below. `fp32`/`float32` are in it because `LAYA_CUDA_AMP` takes them (#443); a page+# that offers them for `LAYA_CPU_AMP`, where full precision is what an unset value already means,+# offers a value the CPU comparison does not read. A spelling outside this set (`int8`) is not a+# value anyone offers.+DTYPE_SPELLINGS = ("fp16", "float16", "bf16", "bfloat16", "fp32", "float32")AMP_VARS = ("LAYA_CUDA_AMP", "LAYA_CPU_AMP")BACKTICK = re.compile(r"`([^`\n]+)`")# A sentence that claims its own list is the whole vocabulary. Only these phrases make a docdiff --git a/tests/test_load_errors.py b/tests/test_load_errors.pyindex 57a0e687..8932e64c 100644--- a/tests/test_load_errors.py+++ b/tests/test_load_errors.py@@ -22,8 +22,9 @@ import reimport shutilimport sysimport tempfile+import typesimport warnings-from contextlib import redirect_stdout+from contextlib import nullcontext, redirect_stdoutfrom pathlib import Pathfrom unittest.mock import patch@@ -39,6 +40,7 @@ from tokenizers.pre_tokenizers import Whitespace # noqa: E402from transformers import BertConfig, BertModel, PreTrainedTokenizerFast # noqa: E402from laya import load # noqa: E402+from laya.agent import _amp_context # noqa: E402from laya.common import DecisionModel # noqa: E402from laya.router import Router # noqa: E402@@ -497,6 +499,96 @@ check_true("unprobed GPU/warns about the probe",del _no_probe+# ------------------------------------------- 10. LAYA_CUDA_AMP on a CUDA device (#443)+# The override is read on every CUDA device, so a value it cannot parse fails the load whatever the+# GPU, and `fp32` turns autocast off instead of entering `torch.autocast(dtype=float32)`. Below+# compute capability 8 the two half-precision asks still run fp16. Loaded through the real+# `Agent.__init__` on a faked device, the way section 9 does it, because the device policy is the+# thing under test and CI has no GPU.+def load_on_fake_gpu(amp, capability=(8, 6), device="cuda:0", **kw):+ probe = {"side_effect": capability} if isinstance(capability, Exception) else {"return_value": capability}+ with patch.dict(os.environ, {"LAYA_CUDA_AMP": amp}), \+ patch("torch.cuda.is_available", return_value=True), \+ patch("torch.cuda.device_count", return_value=1), \+ patch("torch.cuda.get_device_capability", **probe), \+ patch.object(torch.nn.Module, "to", lambda self, *a, **k: self):+ return load(str(REPO), device=device, **kw)+++def amp_state(agent):+ return agent.dtype, agent.amp_enabled, agent.dtype_for(1)+++def load_raises(amp, capability):+ try:+ load_on_fake_gpu(amp, capability)+ except ValueError as exc:+ return str(exc)+ return None+++# The compose files forward `${LAYA_CUDA_AMP:-}`, so an unset variable reaches the load as "".+check("cuda amp/an empty value keeps the checkpoint's choice",+ amp_state(load_on_fake_gpu("")), (torch.float16, True, torch.float16))+check("cuda amp/bf16 on cc 8.6 autocasts in bf16",+ amp_state(load_on_fake_gpu("bf16")), (torch.bfloat16, True, torch.bfloat16))+check("cuda amp/bf16 below cc 8 still runs fp16",+ amp_state(load_on_fake_gpu("bf16", (7, 5))), (torch.float16, True, torch.float16))+_fp32 = load_on_fake_gpu("fp32")+check("cuda amp/fp32 turns autocast off", amp_state(_fp32), (torch.float32, False, torch.float32))+check_true("cuda amp/fp32 forward enters no autocast",+ isinstance(_amp_context(_fp32.device, _fp32.dtype, _fp32._amp_enabled_for(1)), nullcontext))+check("cuda amp/float32 below cc 8 is full precision too",+ amp_state(load_on_fake_gpu("float32", (7, 5))), (torch.float32, False, torch.float32))+check("cuda amp/fp32 needs no capability probe",+ amp_state(load_on_fake_gpu("fp32", RuntimeError("no driver"))), (torch.float32, False, torch.float32))+for _cap in ((8, 6), (7, 5)):+ _err = load_raises("int8", _cap)+ check_true("cuda amp/an unparseable value fails the load on cc %d.%d" % _cap,+ _err is not None and "'int8'" in _err and "LAYA_CUDA_AMP" in _err, repr(_err))+# CPU never reads the variable: compose.yaml forwards it to the base service as well, so a stale+# value on a CPU host must not stop that host from loading.+with patch.dict(os.environ, {"LAYA_CUDA_AMP": "int8"}):+ _cpu = load(str(REPO), device="cpu")+check("cuda amp/cpu ignores the variable", amp_state(_cpu), (torch.float32, False, torch.float32))+del _cpu++# The fused fast path only runs in 16 bits and used to map every other dtype to bf16, so+# `fast=True` with fp32 served bf16 without a word. A stub stands in for `laya.fast`, because+# tilelang is not installed in CI; it records the dtype each build asked for.+class _StubFast:+ built = []++ def __init__(self, model, max_len, use_graphs, dtype):+ _StubFast.built.append(dtype)+ self.max_len = max_len++ def forward(self, *args, **kwargs):+ raise AssertionError("the stub is never run")+++_stub = types.ModuleType("laya.fast")+_stub.FastLaya = _StubFast+with patch.dict(sys.modules, {"laya.fast": _stub}):+ check("fast/bf16 builds the fused path in bf16",+ (load_on_fake_gpu("bf16").accelerate(), _StubFast.built), (True, [torch.bfloat16]))+ with warnings.catch_warnings(record=True) as caught:+ warnings.simplefilter("always")+ _fast32 = load_on_fake_gpu("fp32", fast=True)+ check("fast/fp32 keeps the stock forward", (_fast32._fast, _StubFast.built), (None, [torch.bfloat16]))+ check_true("fast/fp32 says why",+ any(issubclass(w.category, RuntimeWarning) and "float32" in str(w.message) for w in caught),+ str([str(w.message) for w in caught]))+ try:+ _fast32.accelerate(strict=True)+ _strict = None+ except RuntimeError as exc:+ _strict = str(exc)+ check_true("fast/fp32 with strict=True raises", _strict is not None and "float32" in _strict, repr(_strict))+ check("fast/strict refusal builds nothing", _StubFast.built, [torch.bfloat16])+del _fp32, _fast32++TMP.cleanup()print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))diff --git a/tests/test_packaging.py b/tests/test_packaging.pyindex 9cd9b62b..40110aae 100644--- a/tests/test_packaging.py+++ b/tests/test_packaging.py@@ -1057,7 +1057,7 @@ for _opt, _var in sorted(AMP_MEANING.items()):all("`%s`" % t in described for t in accepted),"core compares %s against %s; the option text says: %s"% (_var, accepted, described.strip()[:160]))- offered = [t for t in ("fp16", "float16", "bf16", "bfloat16")+ offered = [t for t in ("fp16", "float16", "bf16", "bfloat16", "fp32", "float32")if "`%s`" % t in described and t not in accepted]check_true("nix/%s offers no dtype core ignores for it" % _opt, not offered,"%s is not in the set laya.agent compares %s against" % (offered, _var))@@ -1070,8 +1070,8 @@ check_true("nix/mpsAmpMinRows's type refuses what the runtime would clamp",% option_type("mpsAmpMinRows"))# Both AMP options tell the operator that the transport refuses whitespace, and that is a claim-# about the type, not the prose: the runtime lower-cases but does not trim, so `"bf16 "` is a-# silently inert setting. Checked where the claim is made.+# about the type, not the prose: a single token is the value the service reads, whatever each+# runtime comparison does with surrounding space. Checked where the claim is made.for _opt in ("cudaAmp", "cpuAmp"):_t = option_type(_opt)check_true("nix/%s's type carries its own no-whitespace claim" % _opt,diff --git a/tests/test_runtime_fixes.py b/tests/test_runtime_fixes.pyindex 70ed549a..b1f75cac 100644--- a/tests/test_runtime_fixes.py+++ b/tests/test_runtime_fixes.py@@ -327,8 +327,26 @@ os.environ["LAYA_CUDA_AMP"] = "fp16"check("cuda-amp/env fp16 overrides a bf16 checkpoint", _cuda_amp_dtype("bf16"), torch.float16)os.environ["LAYA_CUDA_AMP"] = "BF16"check("cuda-amp/env bf16 overrides an fp16 checkpoint", _cuda_amp_dtype("fp16"), torch.bfloat16)+# A value the function cannot parse used to fall back to the checkpoint without a word, so a typo+# served bf16 to a deployment that believed it had asked for something else (#443). It raises now.os.environ["LAYA_CUDA_AMP"] = "int8"-check("cuda-amp/env invalid falls back to the checkpoint", _cuda_amp_dtype("bf16"), torch.bfloat16)+try:+ _cuda_amp_dtype("bf16")+ invalid = None+except ValueError as e:+ invalid = str(e)+check("cuda-amp/env invalid raises and names the value", invalid is not None and "'int8'" in invalid, True)+# fp32 is the precision the parity numbers are measured against; the caller turns autocast off for+# it rather than entering an autocast region at float32.+os.environ["LAYA_CUDA_AMP"] = "fp32"+check("cuda-amp/env fp32 asks for full precision", _cuda_amp_dtype("bf16"), torch.float32)+os.environ["LAYA_CUDA_AMP"] = "Float32"+check("cuda-amp/env float32 is the same ask as fp32", _cuda_amp_dtype("bf16"), torch.float32)+os.environ["LAYA_CUDA_AMP"] = " bf16\n"+check("cuda-amp/env value is trimmed", _cuda_amp_dtype("fp16"), torch.bfloat16)+# The compose files forward `${LAYA_CUDA_AMP:-}`, so an unset variable arrives as "".+os.environ["LAYA_CUDA_AMP"] = ""+check("cuda-amp/env empty means unset", _cuda_amp_dtype("bf16"), torch.bfloat16)# `agent.dtype` reports float16 and bfloat16, so those are the spellings a caller can read off one# response and ask for on the next. laya/agent.py has accepted both from the start; both prose# sites that listed the vocabulary described them as inert, and tests/test_env_docs.py now holds