Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

Merge pull request #1139 from aashish254/docs-tl-cpu-target

Nandakishorcbac8760a3e9
diff --cc docs/compile-and-fast-path.md
index 9baab2f8,6d5640a1..ecf7dfd6
--- a/docs/compile-and-fast-path.md
+++ b/docs/compile-and-fast-path.md
@@@ -117,25 -115,18 +117,25 @@@ a reader of these notes needs to not wa
- **Export works.** `torch.export` of `DecisionModel` succeeds, in about 5 s, with dynamic rows,
markers and tokens. Tokens must be declared as a multiple of 16 (`16 * Dim(...)`); a plain range
fails the exporter's own `L % 8` alignment guard. This is the same mask alignment as above.
-- **Packaging fails.** How it fails depends on how the program was exported:
- - **Under autocast**, the program carries dtype asserts that AOTI trips outside autocast:
- `Tensor dtype mismatch! Expected: torch.bfloat16, Got: torch.float32`.
- - **From a bf16 copy without autocast**, tracing fails inside the forward:
- `mat1 and mat2 must have the same dtype`. `DecisionModel.forward` upcasts the pooled state
- and the confidence features to fp32 before the action head, and autocast normally
- reconciles that.
-
-The artifact route therefore needs a dtype-explicit action head: either cast its input to the
-head's dtype, or run the head in fp32.
+- **The dtype failure these notes measured is fixed.** Packaging from a bf16 copy exported without
+ autocast used to fail inside the forward with `mat1 and mat2 must have the same dtype`:
+ `DecisionModel.forward` keeps the pooled state and the confidence features in fp32 even when the
+ weights are bf16, and with no autocast to reconcile that the first action linear received fp32
+ against bf16 weights. The forward now casts the concatenated input to the head's weight dtype
+ **only when no autocast is active** (`laya/common.py:653-657`), so the AMP and fp32 paths keep
+ their numerics and the decision logits stay fp32.
+ `tests/test_decision_model.py::test_action_head_explicit_dtype_and_autocast` pins the input each
+ head sees for every combination of fp32/bf16 weights with no-AMP/bf16-AMP/fp16-AMP.
+- **Autocast is still the wrong export context.** A program traced under autocast carries dtype
+ asserts that AOTI trips outside autocast:
+ `Tensor dtype mismatch! Expected: torch.bfloat16, Got: torch.float32`. Export a separate
+ evaluation copy in an explicit dtype instead — that is why the cast above is conditional rather
+ than unconditional.
+
+`python scripts/check_aoti.py` runs the whole route on a tiny randomly initialised encoder with no
+network access, and prints `SKIP: CUDA and a C++ compiler are required` where it cannot package.
- ## TileLang portability: the kernels are CUDA-only
+ ## TileLang portability: the GPU specialisation is CUDA-only
tilelang registers targets for CUDA, HIP, Metal, WebGPU and a C backend. Without AMD or Apple
hardware, the answerable question was whether `laya/tl_kernels.py` lowers for the CPU at all.