Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(agent): say what precision a call runs in, not only the autocast target (#621)

mchlcsf9ec87d42d8f
diff --git a/README.md b/README.md
index 4aadb0c4..48a9c1d1 100644
--- a/README.md
+++ b/README.md
@@ -701,7 +701,7 @@ else:
A threshold is a policy you choose from measured accuracy at that coverage on your data, not a property of the model. Both checkpoints are over-confident as shipped and `laya-multilingual` has no fitted temperatures at all, so fit them before relying on these numbers — see [Calibration](#calibration) above, and the [fine-tuning notebook](notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb) for the fitting loop itself. Then pick the point where the errors you accept are ones you can live with. Confidence orders decisions; it does not establish that a decision is correct.
-A threshold also depends on the autocast dtype. On CUDA at compute capability 8 or above the runtime uses the checkpoint's `amp_dtype`, which is bf16 for all three shipped checkpoints. On the fixed set from `benchmarks/parity_fast.py` (60 states, 288 questions per checkpoint, RTX 2000 Ada) bf16 moves a probability by up to 0.073 against the fp32 forward and flips 3 of 864 argmaxes across the three checkpoints; fp16 stays within 0.019 and flips none, at the same latency. `LAYA_CUDA_AMP=fp16` selects fp16 and `LAYA_CUDA_AMP=bf16` selects bf16 (`LAYA_CPU_AMP=bf16` is the CPU counterpart). MPS autocasts in fp16 too, but the overhead dominates on a single small row, so there it engages only once a call reaches `mps_amp_min_rows` rows -- 5 by default, `LAYA_MPS_AMP_MIN_ROWS` to move it; a value that does not parse falls back to 5 and anything below 1 is clamped to 1. Fit and measure a threshold in the dtype you serve with.
+A threshold also depends on the autocast dtype. On CUDA at compute capability 8 or above the runtime uses the checkpoint's `amp_dtype`, which is bf16 for all three shipped checkpoints. On the fixed set from `benchmarks/parity_fast.py` (60 states, 288 questions per checkpoint, RTX 2000 Ada) bf16 moves a probability by up to 0.073 against the fp32 forward and flips 3 of 864 argmaxes across the three checkpoints; fp16 stays within 0.019 and flips none, at the same latency. `LAYA_CUDA_AMP=fp16` selects fp16 and `LAYA_CUDA_AMP=bf16` selects bf16 (`LAYA_CPU_AMP=bf16` is the CPU counterpart). MPS autocasts in fp16 too, but the overhead dominates on a single small row, so there it engages only once a call reaches `mps_amp_min_rows` rows -- 5 by default, `LAYA_MPS_AMP_MIN_ROWS` to move it; a value that does not parse falls back to 5 and anything below 1 is clamped to 1. On MPS, `agent.dtype` is the autocast target, so it says float16 even when every call stays below the gate and runs in fp32; `agent.dtype_for(rows)` returns the precision a call with that many rows runs in. Fit and measure a threshold in the dtype you serve with.
### Opt-in abstention: `min_confidence`
diff --git a/laya/agent.py b/laya/agent.py
index 5ca3861c..d6c275a3 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -256,7 +256,12 @@ def _with_start_probe(hook_kwargs, probe):
class Agent(HookRegistry):
- """System 1 decision model runtime: fast, non-autoregressive, calibrated decisions."""
+ """System 1 decision model runtime: fast, non-autoregressive, calibrated decisions.
+
+ `dtype` is the autocast target, not the precision of every call. On MPS a call
+ autocasts only at or above `mps_amp_min_rows` rows, so `dtype` can say float16 while
+ a call runs in float32. `dtype_for(rows)` returns the precision of a call with `rows` rows.
+ """
# Hooks are opt-in. `hooks`/`_hooks_mutex` defaults come from HookRegistry; the rest keep a
# hand-built instance (`Agent.__new__` in tests) working and make an unset hook a no-op.
@@ -794,6 +799,16 @@ class Agent(HookRegistry):
return False
return True
+ def dtype_for(self, rows: int) -> torch.dtype:
+ """Precision that a forward pass with `rows` question rows runs in.
+
+ `dtype` is the autocast target, set once at load time. Whether a forward autocasts is
+ decided per call: on MPS only at or above `mps_amp_min_rows` rows. This returns `dtype`
+ when a forward with `rows` rows autocasts, and `torch.float32` when it does not. A
+ `predict` call runs one row per question.
+ """
+ return self.dtype if self._amp_enabled_for(rows) else torch.float32
+
def _infer(self, b: Dict):
"""Run the forward pass under autocast, degrading gracefully on OOM or unsupported autocast."""
use_amp = self._amp_enabled_for(b["input_ids"].shape[0])
diff --git a/tests/test_runtime_fixes.py b/tests/test_runtime_fixes.py
index a89e0890..1a441ebb 100644
--- a/tests/test_runtime_fixes.py
+++ b/tests/test_runtime_fixes.py
@@ -203,6 +203,17 @@ check("mps-gate/amp disabled stays off", _mps_agent(amp=False)._amp_enabled_for(
cpu = _bare_agent(FakeModel(), dtype=torch.bfloat16, amp=True)
check("cpu-gate/not gated by rows", cpu._amp_enabled_for(1), True)
+# `dtype` is the autocast target; `dtype_for(rows)` is the precision a forward with `rows` rows
+# runs in (#621). Below the MPS gate that is fp32, even though `dtype` still says fp16.
+a = _mps_agent()
+check("dtype_for/mps below threshold is fp32", a.dtype_for(a.mps_amp_min_rows - 1), torch.float32)
+check("dtype_for/mps at threshold is the target", a.dtype_for(a.mps_amp_min_rows), torch.float16)
+check("dtype_for/mps huge threshold stays fp32", _mps_agent(min_rows=10 ** 9).dtype_for(10), torch.float32)
+check("dtype_for/target unchanged", a.dtype, torch.float16)
+check("dtype_for/amp disabled is fp32", _mps_agent(amp=False).dtype_for(100), torch.float32)
+check("dtype_for/cpu bf16 not gated by rows", cpu.dtype_for(1), torch.bfloat16)
+check("dtype_for/plain cpu is fp32", _bare_agent(FakeModel()).dtype_for(1), torch.float32)
+
os.environ["LAYA_MPS_AMP_MIN_ROWS"] = "2"
check("mps-gate/env override", _mps_amp_min_rows(), 2)
os.environ["LAYA_MPS_AMP_MIN_ROWS"] = "nonsense"