Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(agent): say what precision a call runs in, not only the autocast target (#621)
mchlcsf9ec87d42d8f
diff --git a/README.md b/README.mdindex 4aadb0c4..48a9c1d1 100644--- a/README.md+++ b/README.md@@ -701,7 +701,7 @@ else:A threshold is a policy you choose from measured accuracy at that coverage on your data, not a property of the model. Both checkpoints are over-confident as shipped and `laya-multilingual` has no fitted temperatures at all, so fit them before relying on these numbers — see [Calibration](#calibration) above, and the [fine-tuning notebook](notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb) for the fitting loop itself. Then pick the point where the errors you accept are ones you can live with. Confidence orders decisions; it does not establish that a decision is correct.-A threshold also depends on the autocast dtype. On CUDA at compute capability 8 or above the runtime uses the checkpoint's `amp_dtype`, which is bf16 for all three shipped checkpoints. On the fixed set from `benchmarks/parity_fast.py` (60 states, 288 questions per checkpoint, RTX 2000 Ada) bf16 moves a probability by up to 0.073 against the fp32 forward and flips 3 of 864 argmaxes across the three checkpoints; fp16 stays within 0.019 and flips none, at the same latency. `LAYA_CUDA_AMP=fp16` selects fp16 and `LAYA_CUDA_AMP=bf16` selects bf16 (`LAYA_CPU_AMP=bf16` is the CPU counterpart). MPS autocasts in fp16 too, but the overhead dominates on a single small row, so there it engages only once a call reaches `mps_amp_min_rows` rows -- 5 by default, `LAYA_MPS_AMP_MIN_ROWS` to move it; a value that does not parse falls back to 5 and anything below 1 is clamped to 1. Fit and measure a threshold in the dtype you serve with.+A threshold also depends on the autocast dtype. On CUDA at compute capability 8 or above the runtime uses the checkpoint's `amp_dtype`, which is bf16 for all three shipped checkpoints. On the fixed set from `benchmarks/parity_fast.py` (60 states, 288 questions per checkpoint, RTX 2000 Ada) bf16 moves a probability by up to 0.073 against the fp32 forward and flips 3 of 864 argmaxes across the three checkpoints; fp16 stays within 0.019 and flips none, at the same latency. `LAYA_CUDA_AMP=fp16` selects fp16 and `LAYA_CUDA_AMP=bf16` selects bf16 (`LAYA_CPU_AMP=bf16` is the CPU counterpart). MPS autocasts in fp16 too, but the overhead dominates on a single small row, so there it engages only once a call reaches `mps_amp_min_rows` rows -- 5 by default, `LAYA_MPS_AMP_MIN_ROWS` to move it; a value that does not parse falls back to 5 and anything below 1 is clamped to 1. On MPS, `agent.dtype` is the autocast target, so it says float16 even when every call stays below the gate and runs in fp32; `agent.dtype_for(rows)` returns the precision a call with that many rows runs in. Fit and measure a threshold in the dtype you serve with.### Opt-in abstention: `min_confidence`diff --git a/laya/agent.py b/laya/agent.pyindex 5ca3861c..d6c275a3 100644--- a/laya/agent.py+++ b/laya/agent.py@@ -256,7 +256,12 @@ def _with_start_probe(hook_kwargs, probe):class Agent(HookRegistry):- """System 1 decision model runtime: fast, non-autoregressive, calibrated decisions."""+ """System 1 decision model runtime: fast, non-autoregressive, calibrated decisions.++ `dtype` is the autocast target, not the precision of every call. On MPS a call+ autocasts only at or above `mps_amp_min_rows` rows, so `dtype` can say float16 while+ a call runs in float32. `dtype_for(rows)` returns the precision of a call with `rows` rows.+ """# Hooks are opt-in. `hooks`/`_hooks_mutex` defaults come from HookRegistry; the rest keep a# hand-built instance (`Agent.__new__` in tests) working and make an unset hook a no-op.@@ -794,6 +799,16 @@ class Agent(HookRegistry):return Falsereturn True+ def dtype_for(self, rows: int) -> torch.dtype:+ """Precision that a forward pass with `rows` question rows runs in.++ `dtype` is the autocast target, set once at load time. Whether a forward autocasts is+ decided per call: on MPS only at or above `mps_amp_min_rows` rows. This returns `dtype`+ when a forward with `rows` rows autocasts, and `torch.float32` when it does not. A+ `predict` call runs one row per question.+ """+ return self.dtype if self._amp_enabled_for(rows) else torch.float32+def _infer(self, b: Dict):"""Run the forward pass under autocast, degrading gracefully on OOM or unsupported autocast."""use_amp = self._amp_enabled_for(b["input_ids"].shape[0])diff --git a/tests/test_runtime_fixes.py b/tests/test_runtime_fixes.pyindex a89e0890..1a441ebb 100644--- a/tests/test_runtime_fixes.py+++ b/tests/test_runtime_fixes.py@@ -203,6 +203,17 @@ check("mps-gate/amp disabled stays off", _mps_agent(amp=False)._amp_enabled_for(cpu = _bare_agent(FakeModel(), dtype=torch.bfloat16, amp=True)check("cpu-gate/not gated by rows", cpu._amp_enabled_for(1), True)+# `dtype` is the autocast target; `dtype_for(rows)` is the precision a forward with `rows` rows+# runs in (#621). Below the MPS gate that is fp32, even though `dtype` still says fp16.+a = _mps_agent()+check("dtype_for/mps below threshold is fp32", a.dtype_for(a.mps_amp_min_rows - 1), torch.float32)+check("dtype_for/mps at threshold is the target", a.dtype_for(a.mps_amp_min_rows), torch.float16)+check("dtype_for/mps huge threshold stays fp32", _mps_agent(min_rows=10 ** 9).dtype_for(10), torch.float32)+check("dtype_for/target unchanged", a.dtype, torch.float16)+check("dtype_for/amp disabled is fp32", _mps_agent(amp=False).dtype_for(100), torch.float32)+check("dtype_for/cpu bf16 not gated by rows", cpu.dtype_for(1), torch.bfloat16)+check("dtype_for/plain cpu is fp32", _bare_agent(FakeModel()).dtype_for(1), torch.float32)+os.environ["LAYA_MPS_AMP_MIN_ROWS"] = "2"check("mps-gate/env override", _mps_amp_min_rows(), 2)os.environ["LAYA_MPS_AMP_MIN_ROWS"] = "nonsense"