Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(agent): reject unparseable LAYA_CUDA_AMP and turn autocast off for fp32

Som Tripathi46509b91b4ef
diff --git a/README.md b/README.md
index c3bfebd0..92f39d48 100644
--- a/README.md
+++ b/README.md
@@ -1035,7 +1035,7 @@ else:
A threshold is a policy you choose from measured accuracy at that coverage on your data, not a property of the model. Both checkpoints are over-confident as shipped and `laya-multilingual` has no fitted temperatures at all, so fit them before relying on these numbers — see [Calibration](#calibration) below, and the [fine-tuning notebook](https://github.com/NandhaKishorM/laya/blob/main/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb) for the fitting loop itself. Then pick the point where the errors you accept are ones you can live with. Confidence orders decisions; it does not establish that a decision is correct.
-A threshold also depends on the autocast dtype. On CUDA at compute capability 8 or above the runtime uses the checkpoint's `amp_dtype`, which is bf16 for all three shipped checkpoints. On the fixed set from `benchmarks/parity_fast.py` (60 states, 288 questions per checkpoint, RTX 2000 Ada) bf16 moves a probability by up to 0.073 against the fp32 forward and flips 3 of 864 argmaxes across the three checkpoints; fp16 stays within 0.019 and flips none, at the same latency. `LAYA_CUDA_AMP=fp16` selects fp16 and `LAYA_CUDA_AMP=bf16` selects bf16 (`LAYA_CPU_AMP=bf16` is the CPU counterpart). MPS autocasts in fp16 too, but the overhead dominates on a single small row, so there it engages only once a call reaches `mps_amp_min_rows` rows -- 5 by default, `LAYA_MPS_AMP_MIN_ROWS` to move it; a value that does not parse falls back to 5 and anything below 1 is clamped to 1. On MPS, `agent.dtype` is the autocast target, so it says float16 even when every call stays below the gate and runs in fp32; `agent.dtype_for(rows)` returns the precision a call with that many rows runs in. Fit and measure a threshold in the dtype you serve with.
+A threshold also depends on the autocast dtype. On CUDA at compute capability 8 or above the runtime uses the checkpoint's `amp_dtype`, which is bf16 for all three shipped checkpoints. On the fixed set from `benchmarks/parity_fast.py` (60 states, 288 questions per checkpoint, RTX 2000 Ada) bf16 moves a probability by up to 0.073 against the fp32 forward and flips 3 of 864 argmaxes across the three checkpoints; fp16 stays within 0.019 and flips none, at the same latency. `LAYA_CUDA_AMP=fp16` selects fp16, `LAYA_CUDA_AMP=bf16` selects bf16 and `LAYA_CUDA_AMP=fp32` turns autocast off for the full-precision forward those numbers are measured against (`LAYA_CPU_AMP=bf16` is the CPU counterpart). MPS autocasts in fp16 too, but the overhead dominates on a single small row, so there it engages only once a call reaches `mps_amp_min_rows` rows -- 5 by default, `LAYA_MPS_AMP_MIN_ROWS` to move it; a value that does not parse falls back to 5 and anything below 1 is clamped to 1. On MPS, `agent.dtype` is the autocast target, so it says float16 even when every call stays below the gate and runs in fp32; `agent.dtype_for(rows)` returns the precision a call with that many rows runs in. Fit and measure a threshold in the dtype you serve with.
### Opt-in abstention: `min_confidence`
diff --git a/docs/docker.md b/docs/docker.md
index 27f34c6f..5a480570 100644
--- a/docs/docker.md
+++ b/docs/docker.md
@@ -68,7 +68,7 @@ work with `docker run -e`; Compose-only settings are identified below.
| Variable | Default | Purpose |
| --- | --- | --- |
| `LAYA_DEVICE` | `cpu` / `cuda` | Device selected by the base / GPU configuration |
-| `LAYA_CUDA_AMP` | unset (checkpoint's `amp_dtype`) | `fp16`/`float16` or `bf16`/`bfloat16` for the CUDA forward; anything else is ignored. Not cosmetic: the README's threshold section measures bf16 flipping 3 of 864 argmaxes on the parity set where fp16 flips none |
+| `LAYA_CUDA_AMP` | unset (checkpoint's `amp_dtype`) | `fp16`/`float16` or `bf16`/`bfloat16` for the CUDA forward, or `fp32`/`float32` to turn autocast off; anything else fails the load with a `ValueError`. GPUs below compute capability 8 run fp16 for either half-precision spelling. Not cosmetic: the README's threshold section measures bf16 flipping 3 of 864 argmaxes on the parity set where fp16 flips none |
| `LAYA_CPU_AMP` | unset | `bf16` or `bfloat16` opts the CPU forward into bf16; anything else leaves it fp32. No fp16 spelling turns it on either: CPU autocast has no fp16 fast path that beats fp32, so bf16 is the only reduced precision core offers on this device |
| `LAYA_MODEL` | `auto` | Router alias: `auto`, `english`, `multilingual`, `typed-decisions` |
| `LAYA_MODEL_PATH` | unset | Compatible checkpoint path inside the container |
diff --git a/laya/agent.py b/laya/agent.py
index 8ebf2c06..dec08ac0 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -296,16 +296,26 @@ def _mps_amp_min_rows() -> int:
def _cuda_amp_dtype(checkpoint_default: Optional[str]) -> torch.dtype:
- """Autocast dtype on CUDA at compute capability >= 8: the checkpoint's `amp_dtype` (bf16 for the
- shipped checkpoints), or LAYA_CUDA_AMP=fp16|float16|bf16|bfloat16 when set. fp16 stays 2-10x
- closer to the fp32 forward than bf16 on every shipped checkpoint at the same speed; anything
- else is ignored. The long spellings count because they are what `agent.dtype` reports back, so
- the precision a user reads off one response is one they can ask for on the next."""
- raw = os.environ.get("LAYA_CUDA_AMP", "").lower()
+ """Precision asked for on CUDA: the checkpoint's `amp_dtype` (bf16 for the shipped checkpoints),
+ or LAYA_CUDA_AMP=fp16|float16|bf16|bfloat16|fp32|float32 when set. fp16 stays 2-10x closer to
+ the fp32 forward than bf16 on every shipped checkpoint at the same speed. float32 means no
+ autocast at all, and the caller turns it off rather than autocasting to float32. Below compute
+ capability 8 the caller runs fp16 for either half-precision ask. The long spellings count
+ because they are what `agent.dtype` reports back, so the precision a user reads off one
+ response is one they can ask for on the next. The value is trimmed and lower-cased, an empty
+ one means unset, and anything else raises ValueError: a typo fails the load instead of
+ silently serving the checkpoint's choice."""
+ raw = os.environ.get("LAYA_CUDA_AMP", "").strip().lower()
if raw in ("fp16", "float16"):
return torch.float16
if raw in ("bf16", "bfloat16"):
return torch.bfloat16
+ if raw in ("fp32", "float32"):
+ return torch.float32
+ if raw:
+ raise ValueError("LAYA_CUDA_AMP=%r is not a CUDA precision; use fp16, float16, bf16, bfloat16, "
+ "fp32 or float32, or leave it unset for the checkpoint's own amp_dtype"
+ % os.environ["LAYA_CUDA_AMP"])
return amp_dtype(checkpoint_default)
@@ -796,20 +806,26 @@ class Agent(HookRegistry):
self.mps_amp_min_rows = _mps_amp_min_rows()
if self.device.type == "cuda":
self.amp_enabled = True
- try:
- major = torch.cuda.get_device_capability(self.device)[0]
- except (RuntimeError, AttributeError):
- # A driver that reports a GPU but cannot answer the capability query
- # cannot be trusted with bf16 either; fp16 is the safe default and
- # placement below still decides whether the device is usable at all.
- warnings.warn("Warning: could not query CUDA capability; assuming fp16 defaults.",
- RuntimeWarning)
- self.dtype = torch.float16
+ # Read on every CUDA device, so a value that does not parse fails here whatever the
+ # GPU, and fp32 means full precision on all of them (#443).
+ requested = _cuda_amp_dtype(self.cfg.get("amp_dtype", "fp16"))
+ if requested == torch.float32:
+ self.amp_enabled = False
else:
- if major < 8:
+ try:
+ major = torch.cuda.get_device_capability(self.device)[0]
+ except (RuntimeError, AttributeError):
+ # A driver that reports a GPU but cannot answer the capability query
+ # cannot be trusted with bf16 either; fp16 is the safe default and
+ # placement below still decides whether the device is usable at all.
+ warnings.warn("Warning: could not query CUDA capability; assuming fp16 defaults.",
+ RuntimeWarning)
self.dtype = torch.float16
else:
- self.dtype = _cuda_amp_dtype(self.cfg.get("amp_dtype", "fp16"))
+ if major < 8:
+ self.dtype = torch.float16
+ else:
+ self.dtype = requested
elif self.device.type == "mps":
self.amp_enabled = True
self.dtype = torch.float16
@@ -907,6 +923,8 @@ class Agent(HookRegistry):
matches the stock forward it replaces within rounding (see benchmarks/parity_fast.py). After
changing `agent.dtype`, call `deaccelerate()` then `accelerate()` to rebuild it. Returns True if
enabled. With `strict=False` any failure (no CUDA, tilelang missing) leaves the stock path in place.
+ An fp32 agent (`LAYA_CUDA_AMP=fp32`) is such a failure: its kernels only run in 16 bits, so it
+ warns and keeps the stock forward, or raises with `strict=True`.
"""
if self._fast is not None:
return True
@@ -914,6 +932,13 @@ class Agent(HookRegistry):
if strict:
raise RuntimeError("laya fast path needs a CUDA device")
return False
+ if self.dtype not in (torch.bfloat16, torch.float16):
+ msg = ("laya fast path runs in bf16 or fp16 only and this agent runs %s; "
+ "keeping the stock forward" % self.dtype)
+ if strict:
+ raise RuntimeError(msg)
+ warnings.warn("Warning: " + msg, RuntimeWarning)
+ return False
if self._backend is not None:
self.deaccelerate()
last = None
diff --git a/nix/laya-serve.nix b/nix/laya-serve.nix
index a7ef1c1d..3f564dc3 100644
--- a/nix/laya-serve.nix
+++ b/nix/laya-serve.nix
@@ -72,13 +72,13 @@ in
`laya.agent`, not here: `tests/test_packaging.py` reads them out of the
runtime and fails if this text drifts from it, which is why there is no
`enum` on this option. The recognised spellings are `fp16`, `float16`,
- `bf16` and `bfloat16` — the first two name one dtype and the last two
- another. Anything else is ignored rather than an error, and the
- checkpoint's own choice stays in force.
+ `bf16`, `bfloat16`, `fp32` and `float32` — each pair names one dtype,
+ and `fp32`/`float32` turn autocast off. Anything else is an error: the
+ service fails at load instead of serving the checkpoint's own choice.
- The type allows any single token but refuses whitespace, which is not
- style: the runtime lower-cases the value and then compares it without
- trimming, so a trailing space keeps full precision and says nothing.
+ The type allows any single token but refuses whitespace. The runtime
+ trims and lower-cases the value, so a stray space would change nothing;
+ the type keeps what this module writes identical to what it reads.
null leaves every CUDA checkpoint on its own `amp_dtype`.
'';
};
diff --git a/tests/test_confidence.py b/tests/test_confidence.py
index afe10b9b..770d2b3a 100644
--- a/tests/test_confidence.py
+++ b/tests/test_confidence.py
@@ -2240,7 +2240,7 @@ for _fn in (test_tl_kernels_drops_the_wrong_row_claim,
# --------------------------------------- DecisionResult's docstring must split confidence by question type
# `laya/structured.py`'s `DecisionResult` closed with "confidence keeps the **normalized-entropy
# value** it has always had, because that is a different quantity on a scale that depends on the
-# label count". `Agent._decode_answers` (`laya/agent.py:1369-1398`) uses two different formulas:
+# label count". `Agent._decode_answers` (`laya/agent.py:1394-1423`) uses two different formulas:
# * `choice` and `score` -> `confidence_from_probs(p, k)` = `1 - H(p) / log(k)`
# * `noul` -> `max(p_true, 1 - p_true)` (its own comment: "identical here:
# over two options `max(p_true, 1 - p_true)` is `max(p)`")
diff --git a/tests/test_env_docs.py b/tests/test_env_docs.py
index 10d90d4b..1c2144b6 100644
--- a/tests/test_env_docs.py
+++ b/tests/test_env_docs.py
@@ -74,11 +74,13 @@ SKIP_DIRS = {".git", ".github", ".cache", ".venv", "site", "__pycache__", "node_
# and cannot become a permanent hole in the sweep.
DEFERRED: dict = {}
-# The half-precision spellings a document could offer for the two AMP names. Kept as the closed
-# universe of things a prose site can mean rather than as an accepted list: the accepted list is
-# derived from laya/agent.py below, and a spelling outside this set (`fp32`, `int8`) is not a
-# value anyone offers -- it is what the forward falls back to, and pages name it for that.
-DTYPE_SPELLINGS = ("fp16", "float16", "bf16", "bfloat16")
+# The dtype spellings a document could offer for the two AMP names. Kept as the closed universe of
+# things a prose site can mean rather than as an accepted list: the accepted list is derived from
+# laya/agent.py below. `fp32`/`float32` are in it because `LAYA_CUDA_AMP` takes them (#443); a page
+# that offers them for `LAYA_CPU_AMP`, where full precision is what an unset value already means,
+# offers a value the CPU comparison does not read. A spelling outside this set (`int8`) is not a
+# value anyone offers.
+DTYPE_SPELLINGS = ("fp16", "float16", "bf16", "bfloat16", "fp32", "float32")
AMP_VARS = ("LAYA_CUDA_AMP", "LAYA_CPU_AMP")
BACKTICK = re.compile(r"`([^`\n]+)`")
# A sentence that claims its own list is the whole vocabulary. Only these phrases make a doc
diff --git a/tests/test_load_errors.py b/tests/test_load_errors.py
index 57a0e687..8932e64c 100644
--- a/tests/test_load_errors.py
+++ b/tests/test_load_errors.py
@@ -22,8 +22,9 @@ import re
import shutil
import sys
import tempfile
+import types
import warnings
-from contextlib import redirect_stdout
+from contextlib import nullcontext, redirect_stdout
from pathlib import Path
from unittest.mock import patch
@@ -39,6 +40,7 @@ from tokenizers.pre_tokenizers import Whitespace # noqa: E402
from transformers import BertConfig, BertModel, PreTrainedTokenizerFast # noqa: E402
from laya import load # noqa: E402
+from laya.agent import _amp_context # noqa: E402
from laya.common import DecisionModel # noqa: E402
from laya.router import Router # noqa: E402
@@ -497,6 +499,96 @@ check_true("unprobed GPU/warns about the probe",
del _no_probe
+# ------------------------------------------- 10. LAYA_CUDA_AMP on a CUDA device (#443)
+# The override is read on every CUDA device, so a value it cannot parse fails the load whatever the
+# GPU, and `fp32` turns autocast off instead of entering `torch.autocast(dtype=float32)`. Below
+# compute capability 8 the two half-precision asks still run fp16. Loaded through the real
+# `Agent.__init__` on a faked device, the way section 9 does it, because the device policy is the
+# thing under test and CI has no GPU.
+def load_on_fake_gpu(amp, capability=(8, 6), device="cuda:0", **kw):
+ probe = {"side_effect": capability} if isinstance(capability, Exception) else {"return_value": capability}
+ with patch.dict(os.environ, {"LAYA_CUDA_AMP": amp}), \
+ patch("torch.cuda.is_available", return_value=True), \
+ patch("torch.cuda.device_count", return_value=1), \
+ patch("torch.cuda.get_device_capability", **probe), \
+ patch.object(torch.nn.Module, "to", lambda self, *a, **k: self):
+ return load(str(REPO), device=device, **kw)
+
+
+def amp_state(agent):
+ return agent.dtype, agent.amp_enabled, agent.dtype_for(1)
+
+
+def load_raises(amp, capability):
+ try:
+ load_on_fake_gpu(amp, capability)
+ except ValueError as exc:
+ return str(exc)
+ return None
+
+
+# The compose files forward `${LAYA_CUDA_AMP:-}`, so an unset variable reaches the load as "".
+check("cuda amp/an empty value keeps the checkpoint's choice",
+ amp_state(load_on_fake_gpu("")), (torch.float16, True, torch.float16))
+check("cuda amp/bf16 on cc 8.6 autocasts in bf16",
+ amp_state(load_on_fake_gpu("bf16")), (torch.bfloat16, True, torch.bfloat16))
+check("cuda amp/bf16 below cc 8 still runs fp16",
+ amp_state(load_on_fake_gpu("bf16", (7, 5))), (torch.float16, True, torch.float16))
+_fp32 = load_on_fake_gpu("fp32")
+check("cuda amp/fp32 turns autocast off", amp_state(_fp32), (torch.float32, False, torch.float32))
+check_true("cuda amp/fp32 forward enters no autocast",
+ isinstance(_amp_context(_fp32.device, _fp32.dtype, _fp32._amp_enabled_for(1)), nullcontext))
+check("cuda amp/float32 below cc 8 is full precision too",
+ amp_state(load_on_fake_gpu("float32", (7, 5))), (torch.float32, False, torch.float32))
+check("cuda amp/fp32 needs no capability probe",
+ amp_state(load_on_fake_gpu("fp32", RuntimeError("no driver"))), (torch.float32, False, torch.float32))
+for _cap in ((8, 6), (7, 5)):
+ _err = load_raises("int8", _cap)
+ check_true("cuda amp/an unparseable value fails the load on cc %d.%d" % _cap,
+ _err is not None and "'int8'" in _err and "LAYA_CUDA_AMP" in _err, repr(_err))
+# CPU never reads the variable: compose.yaml forwards it to the base service as well, so a stale
+# value on a CPU host must not stop that host from loading.
+with patch.dict(os.environ, {"LAYA_CUDA_AMP": "int8"}):
+ _cpu = load(str(REPO), device="cpu")
+check("cuda amp/cpu ignores the variable", amp_state(_cpu), (torch.float32, False, torch.float32))
+del _cpu
+
+# The fused fast path only runs in 16 bits and used to map every other dtype to bf16, so
+# `fast=True` with fp32 served bf16 without a word. A stub stands in for `laya.fast`, because
+# tilelang is not installed in CI; it records the dtype each build asked for.
+class _StubFast:
+ built = []
+
+ def __init__(self, model, max_len, use_graphs, dtype):
+ _StubFast.built.append(dtype)
+ self.max_len = max_len
+
+ def forward(self, *args, **kwargs):
+ raise AssertionError("the stub is never run")
+
+
+_stub = types.ModuleType("laya.fast")
+_stub.FastLaya = _StubFast
+with patch.dict(sys.modules, {"laya.fast": _stub}):
+ check("fast/bf16 builds the fused path in bf16",
+ (load_on_fake_gpu("bf16").accelerate(), _StubFast.built), (True, [torch.bfloat16]))
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ _fast32 = load_on_fake_gpu("fp32", fast=True)
+ check("fast/fp32 keeps the stock forward", (_fast32._fast, _StubFast.built), (None, [torch.bfloat16]))
+ check_true("fast/fp32 says why",
+ any(issubclass(w.category, RuntimeWarning) and "float32" in str(w.message) for w in caught),
+ str([str(w.message) for w in caught]))
+ try:
+ _fast32.accelerate(strict=True)
+ _strict = None
+ except RuntimeError as exc:
+ _strict = str(exc)
+ check_true("fast/fp32 with strict=True raises", _strict is not None and "float32" in _strict, repr(_strict))
+ check("fast/strict refusal builds nothing", _StubFast.built, [torch.bfloat16])
+del _fp32, _fast32
+
+
TMP.cleanup()
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
diff --git a/tests/test_packaging.py b/tests/test_packaging.py
index 9cd9b62b..40110aae 100644
--- a/tests/test_packaging.py
+++ b/tests/test_packaging.py
@@ -1057,7 +1057,7 @@ for _opt, _var in sorted(AMP_MEANING.items()):
all("`%s`" % t in described for t in accepted),
"core compares %s against %s; the option text says: %s"
% (_var, accepted, described.strip()[:160]))
- offered = [t for t in ("fp16", "float16", "bf16", "bfloat16")
+ offered = [t for t in ("fp16", "float16", "bf16", "bfloat16", "fp32", "float32")
if "`%s`" % t in described and t not in accepted]
check_true("nix/%s offers no dtype core ignores for it" % _opt, not offered,
"%s is not in the set laya.agent compares %s against" % (offered, _var))
@@ -1070,8 +1070,8 @@ check_true("nix/mpsAmpMinRows's type refuses what the runtime would clamp",
% option_type("mpsAmpMinRows"))
# Both AMP options tell the operator that the transport refuses whitespace, and that is a claim
-# about the type, not the prose: the runtime lower-cases but does not trim, so `"bf16 "` is a
-# silently inert setting. Checked where the claim is made.
+# about the type, not the prose: a single token is the value the service reads, whatever each
+# runtime comparison does with surrounding space. Checked where the claim is made.
for _opt in ("cudaAmp", "cpuAmp"):
_t = option_type(_opt)
check_true("nix/%s's type carries its own no-whitespace claim" % _opt,
diff --git a/tests/test_runtime_fixes.py b/tests/test_runtime_fixes.py
index 70ed549a..b1f75cac 100644
--- a/tests/test_runtime_fixes.py
+++ b/tests/test_runtime_fixes.py
@@ -327,8 +327,26 @@ os.environ["LAYA_CUDA_AMP"] = "fp16"
check("cuda-amp/env fp16 overrides a bf16 checkpoint", _cuda_amp_dtype("bf16"), torch.float16)
os.environ["LAYA_CUDA_AMP"] = "BF16"
check("cuda-amp/env bf16 overrides an fp16 checkpoint", _cuda_amp_dtype("fp16"), torch.bfloat16)
+# A value the function cannot parse used to fall back to the checkpoint without a word, so a typo
+# served bf16 to a deployment that believed it had asked for something else (#443). It raises now.
os.environ["LAYA_CUDA_AMP"] = "int8"
-check("cuda-amp/env invalid falls back to the checkpoint", _cuda_amp_dtype("bf16"), torch.bfloat16)
+try:
+ _cuda_amp_dtype("bf16")
+ invalid = None
+except ValueError as e:
+ invalid = str(e)
+check("cuda-amp/env invalid raises and names the value", invalid is not None and "'int8'" in invalid, True)
+# fp32 is the precision the parity numbers are measured against; the caller turns autocast off for
+# it rather than entering an autocast region at float32.
+os.environ["LAYA_CUDA_AMP"] = "fp32"
+check("cuda-amp/env fp32 asks for full precision", _cuda_amp_dtype("bf16"), torch.float32)
+os.environ["LAYA_CUDA_AMP"] = "Float32"
+check("cuda-amp/env float32 is the same ask as fp32", _cuda_amp_dtype("bf16"), torch.float32)
+os.environ["LAYA_CUDA_AMP"] = " bf16\n"
+check("cuda-amp/env value is trimmed", _cuda_amp_dtype("fp16"), torch.bfloat16)
+# The compose files forward `${LAYA_CUDA_AMP:-}`, so an unset variable arrives as "".
+os.environ["LAYA_CUDA_AMP"] = ""
+check("cuda-amp/env empty means unset", _cuda_amp_dtype("bf16"), torch.bfloat16)
# `agent.dtype` reports float16 and bfloat16, so those are the spellings a caller can read off one
# response and ask for on the next. laya/agent.py has accepted both from the start; both prose
# sites that listed the vocabulary described them as inert, and tests/test_env_docs.py now holds