Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(backends): auto resolves to eager for a parallel-layout checkpoint, on CUDA as well

Aashishd18091e19009
diff --git a/docs/compile-and-fast-path.md b/docs/compile-and-fast-path.md
index cf76b5b0..09162b6a 100644
--- a/docs/compile-and-fast-path.md
+++ b/docs/compile-and-fast-path.md
@@ -19,7 +19,12 @@ and `fast`; omitting it preserves both flags' existing behaviour.
to choose its cache directory (default `~/.cache/laya/inductor`).
- `tilelang`: an adapter around the current fast path, using the agent's bf16 or fp16 dtype.
- `auto`: TileLang on CUDA with a supported ModernBERT encoder and dtype when TileLang is
- installed, otherwise compile on CUDA; eager on other devices.
+ installed, otherwise compile on CUDA; eager on other devices. One check comes before the device:
+ a checkpoint on the parallel option layout (`Agent.parallel_options`, set by `option_layout` in
+ its config) is eager on CUDA too, because the accelerated forwards do not take the parallel
+ masks. Nothing flags the choice — `auto` answered with a backend, just not the one the clause
+ above implies — so `agent.backend` is what says which arm ran, and `laya.backends.auto_policy` is
+ the authority on the order of those checks.
- `onnx`: `laya.load(..., backend="onnx", onnx_path="model.onnx")` returns the existing
`ONNXAgent`. Without `onnx_path`, it uses `laya.onnx`.
diff --git a/laya/backends/__init__.py b/laya/backends/__init__.py
index 5b16aa36..2e66b169 100644
--- a/laya/backends/__init__.py
+++ b/laya/backends/__init__.py
@@ -9,7 +9,10 @@
| `tilelang` | fused TileLang kernels, 16-bit weights, one CUDA graph per bucket | CUDA, `laya[fast]`, bf16/fp16 |
| `onnx` | ONNX Runtime via `laya.onnx_agent.ONNXAgent` (the explicit CPU option, its own class) | `laya[onnx]`, an exported model |
-`auto` picks per device: CUDA -> `tilelang` when tilelang imports and the encoder is a
+`auto` resolves through `auto_policy`, which answers `eager` before it looks at the device whenever
+the accelerated forwards cannot take the agent's input layout -- `Agent.parallel_options`, a
+checkpoint whose config sets `option_layout: "parallel"`, whose masks only the stock forward reads.
+Otherwise it picks per device: CUDA -> `tilelang` when tilelang imports and the encoder is a
ModernBERT-family model (the shipped checkpoints), else `compile`; CPU and MPS -> `eager`.
Every backend honours `agent.dtype`, and one that cannot run falls back to `eager` with one
`RuntimeWarning`; `strict=True` raises instead. See `laya.backends.base` for the contract and