Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(examples): example 35 must teach the autocast policy the runtime sets
Aashishb6b19d2b0da4
diff --git a/examples/35_device_selection_and_latency.py b/examples/35_device_selection_and_latency.pyindex 0a24944d..859292ba 100644--- a/examples/35_device_selection_and_latency.py+++ b/examples/35_device_selection_and_latency.py@@ -1,23 +1,32 @@-"""Example 35 -- device selection, fp32 off CUDA, and device latency.+"""Example 35 -- device selection, the per-call autocast policy, and device latency.Laya picks CUDA, then MPS, then CPU when `device=None`. This example times the multilingual-checkpoint on CPU and on MPS (if present), prints median ms and ms/question, and checks that the-two devices return the same answers.+checkpoint on CPU and on MPS (if present), prints median ms and ms/question, checks that the two+devices return the same answers, and prints -- then measures -- the precision each forward really+runs in."""import jsonimport torch+import torch.nn as nn-from _common import STATE_HI, banner, describe, load, timed+from _common import STATE_HI, banner, describe, heading, load, timedbanner("35", "Device selection and latency", """Device resolution in `Agent.__init__`: an explicit `device=` wins (with a warning and a- fallback to CPU if it is unavailable), otherwise CUDA, then MPS, then CPU. Precision follows- the device -- mixed precision (bf16/fp16) is a CUDA win, while CPU and MPS run **fp32**;- `torch.autocast` has no MPS backend here, so Laya only wraps the forward pass on CUDA.+ fallback to CPU if it is unavailable), otherwise CUDA, then MPS, then CPU. Mixed precision is+ chosen per device at load time and per call at inference time, and it is not CUDA-only: MPS+ gets `amp_enabled=True` with an autocast `dtype` of float16, XPU gets bfloat16, and CPU gets+ bfloat16 only when `LAYA_CPU_AMP=bf16` asks for it. What one call actually runs in is+ `dtype_for(rows)`, and on MPS that is gated by row count: below `mps_amp_min_rows` (5 rows by+ default, override with `LAYA_MPS_AMP_MIN_ROWS`) an MPS forward stays fp32, because fp16+ autocast loses on a single small row and only wins once the batch has several. So `dtype` is+ the target and `dtype_for(rows)` is the precision; this example prints both and then measures+ the same MPS batch under each.For explicit control pass `device="cpu"`, `"mps"` or `"cuda"`. Laya does not read an env var- itself, but a common application pattern is to read one (`LAYA_DEVICE`) and pass it through.+ for the device itself, but a common application pattern is to read one (`LAYA_DEVICE`) and+ pass it through.`timed(fn, repeat=3)` discards one warm-up call, so the first MPS call's ~13 s of Metalkernel compilation does not pollute the median. Both legs answer the same question set.@@ -55,6 +64,9 @@ def measure(device):result, median_ms = timed(lambda: agent.predict(STATE_HI, QUESTIONS), repeat=3)print(" %-4s -> loaded on %-4s dtype=%-8s median %7.1f ms/call %.1f ms/question"% (device, agent.device, agent.dtype, median_ms, median_ms / N_QUESTIONS))+ print(" `amp_enabled`=%s `dtype_for(%d rows)`=%s `mps_amp_min_rows`=%s"+ " (MPS only)"+ % (agent.amp_enabled, N_QUESTIONS, agent.dtype_for(N_QUESTIONS), agent.mps_amp_min_rows))return agent, result, median_ms@@ -84,5 +96,68 @@ if mps_result is not None:print("\n device line and one answer from the CPU leg:")describe(cpu_result["answers"])-print(" both legs used fp32 (device.type in ('cpu','mps') -> torch.float32);")-print(" mixed precision would only be enabled on CUDA.")+++# ------------------------------------------ which precision the forward really ran in, measured+heading("which precision a forward really runs in")+++def observed(agent, questions):+ """(`torch.is_autocast_enabled()`, the first Linear layer's output dtype) inside a real forward."""+ seen = {}+ linear = next((m for m in agent.model.modules() if isinstance(m, nn.Linear)), None)++ def hook(module, args, output):+ seen.setdefault("autocast", torch.is_autocast_enabled())+ seen.setdefault("dtype", str(output.dtype))+ handle = None if linear is None else linear.register_forward_hook(hook)+ try:+ agent.predict(STATE_HI, questions)+ finally:+ if handle is not None:+ handle.remove()+ return seen.get("autocast"), seen.get("dtype")+++ONE = {"tone": {"type": "noul", "instructions": "Is the customer's tone angry?"}}+FIVE = dict(ONE)+FIVE.update({+ "language": {"type": "noul", "instructions": "Is the message written in Hindi?"},+ "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},+ "leaving": {"type": "noul", "instructions": "Does the customer threaten to leave?"},+ "duplicate": {"type": "noul", "instructions": "Is a duplicate charge mentioned?"},+})++print(" `dtype` is the autocast target picked once at load; `dtype_for(rows)` is the precision a")+print(" call with `rows` rows really runs in, and `predict` spends one row per question. The two")+print(" legs above asked %d questions, so the MPS leg ran fp32 because %d is under"+ " `mps_amp_min_rows` --" % (N_QUESTIONS, N_QUESTIONS))+print(" the row gate decided that, not the device. Three batch sizes, three thresholds each:")++if mps_agent is None:+ print(" these arms need an MPS device, and this machine skipped that leg, so there is nothing")+ print(" to measure here. On Apple Silicon the table shows the same agent autocast on or off.")+else:+ gate = mps_agent.mps_amp_min_rows+ for questions in (ONE, QUESTIONS, FIVE):+ label = "%d question%s" % (len(questions), "" if len(questions) == 1 else "s")+ arms = (("as shipped", gate), ("fp16 forced on", 1), ("fp32 forced off", 10 ** 9))+ medians = {}+ for arm, min_rows in arms:+ mps_agent.mps_amp_min_rows = min_rows+ autocast, dtype = observed(mps_agent, questions)+ _, median_ms = timed(lambda: mps_agent.predict(STATE_HI, questions), repeat=5)+ medians[arm] = median_ms+ print(" %-12s %-15s `mps_amp_min_rows`=%-10s autocast=%-5s %-16s %5.1f ms"+ % (label, arm, min_rows, autocast, dtype, median_ms))+ print(" %-12s this machine, fp16 over fp32 at this batch size: %.2fx"+ % ("", medians["fp16 forced on"] / medians["fp32 forced off"]))+ mps_agent.mps_amp_min_rows = gate+ print("""+ `torch.is_autocast_enabled()` is set inside the MPS forward and the first Linear layer emits+ its autocast dtype, so MPS mixed precision is real on this torch build. `amp_enabled` and+ `dtype` are what the load-time policy picked for the device; `dtype_for(rows)` and the flag+ observed above are what a call actually ran in. `mps_amp_min_rows` is where the library puts+ the crossover on MPS, and `LAYA_MPS_AMP_MIN_ROWS` moves it -- the two forced arms print this+ machine's ratio at each batch size, so the threshold can be checked against the hardware a call+ is really running on rather than taken on faith.""")