Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

Merge pull request #897 from cklxx/fix-aoti-dtype

Nandakishor0042f58f7dc6
diff --cc docs/fast-backends.md
index fe062bc4,cd0eb2c3..0eb431e5
--- a/docs/fast-backends.md
+++ b/docs/fast-backends.md
@@@ -1,172 -1,120 +1,292 @@@
-# Fast inference: AOTInductor
+# Fast backends
+
+## TileLang portability
+
+All five kernels in `laya/tl_kernels.py` can execute on TileLang's CPU (`c`)
+target through `compile_cpu`. This is an explicit, scalar fp32 specialization,
+not a CPU acceleration backend for `Agent.accelerate`. It requires TileLang and
+a local C++ compiler. It introduces no runtime service dependency.
+
+```python
+import torch
+from laya import tl_kernels as K
+
+A = torch.randn(17, 67)
+W = torch.randn(70, 67)
+b = torch.randn(70)
+C = torch.empty(17, 70)
+kernel = K.compile_cpu(K.gemm_kernel, 70, 67, bias=True, act="gelu")
+kernel(A, W, b, C)
+```
+
+`compile_cpu(factory, *args, **kwargs)` takes the same shape and operation options
+as the five GPU factories. It selects `cpu=True`, `dtype="float32"`, target `c`,
+and disables vectorization. All floating inputs and outputs must be contiguous
+CPU float32 tensors; attention lengths remain int32. Convert 16-bit activations
+with `.cpu().float().contiguous()` before calling it. LayerNorm still updates its
+residual tensor in place, and RoPE still updates the Q/K columns in place.
+Retain the compiled kernel to reuse its dynamic dimensions.
+
+CPU specialization replaces fragment and shared allocations with local buffers,
+uses serial `T.grid` loops, and omits attention's GPU swizzle annotation. GEMMs
+use TileLang's scalar CPU implementation and reductions use local buffers.
+The original tiled algorithms, padding predicates, attention masks, and online
+softmax remain shared with the GPU implementation. CPU intermediates stay fp32,
+including attention probabilities; GPU intermediates retain their original dtype.
+No CPU speedup or full-model CPU backend is claimed.
+
+### Re-probe on TileLang 0.1.14
+
+Observed on Linux, Python 3.12.13, torch 2.11.0+cu130, TileLang 0.1.14,
+and an RTX 4070 Ti SUPER. The earlier portability concern applies to compiling
+the **unchanged GPU specialization**, not to the possibility of a CPU lowering.
+At base commit `fa9a2a7`, this documentation page was absent from the checkout.
+
+The test compiles `factory.get_tir(...)` with `target="c"` and the GPU's `FAST`
+pass configuration. These are the exact diagnostic texts (source locations and
+stack traces omitted). Both bf16 and fp16 were probed for every kernel.
+
+| Kernel and probe dimensions | First bf16 failure | First fp16 failure |
+| --- | --- | --- |
+| `gemm_kernel(128, 64)` | `Check failed: layout_map.count(buffer) != 0 (0 vs. 0) : The layout for fragment C_l can not be inferred correctly.` | Same |
+| `gemm_geglu_kernel(64, 64)` | ``CPU fill only supports local and global buffers, but got dst scope `local.fragment`.`` | Same |
+| `add_ln_kernel(128)` | ``CPU reduce only supports local src and local/local.var dst buffers, got src scope `local.fragment` and dst scope `local.fragment`.`` | Same |
+| `rope_kernel(2, 64)` | `Cannot convert type bfloat16 to C type` | C++ compiler: `error: no matching function for call to ‘vec_type<float, 4>::vec_type(half4&)’` |
+| `attn_kernel(1, 64, 2, 64)` | `Check failed: layout_map.count(buffer) != 0 (0 vs. 0) : The layout for fragment s_c can not be inferred correctly.` | Same |
+
+The first four fragment/reduction failures are `tvm.error.InternalError`.
+The bf16 codegen failure is also an `InternalError`. FP16 RoPE raises
+`RuntimeError: Compilation Failed!` followed by the compiler invocation and
+source; the diagnostic above is emitted on stderr. Its generated vector casts
+also fail converting float vectors back to half vectors.
+
+To isolate dtype support from fragment support, the tests compile **each** kernel
+again with `cpu=True` (local buffers and serial loops), retaining bf16 and disabling
+vectorization. All five then fail with exactly:
+
+```text
+Cannot convert type bfloat16 to C type
+```
+
+Thus fragments are the first blocker for GEMM, GEGLU and attention, fragment
+reductions for LayerNorm, and bf16 is independently a blocker for all five.
+RoPE has no fragment allocation or reduction; GEMM and GEGLU have no explicit
+`T.reduce_*` operation. Attention's reductions are initially masked by its layout
+failure. Separate fp32-only fragment probes reproduce the fill error and the
+reduction error for both `reduce_sum` and `reduce_max`, without any GEMM or bf16.
+Replacing scopes alone also produced this semantic error for the GEMM
+buffer (the other affected buffers were `Ci`, `x`, and `s`):
+
+```text
+[Tilelang Semantic Check] Local buffer `C_l` is indexed by T.Parallel loop variable `i`. Local buffers are thread-private and do not participate in parallel layout inference. Use T.serial/T.vectorized/T.unroll for per-thread local indexing, or T.alloc_fragment when the indexed dimension should be distributed across threads.
+```
+
+The serial fp32 specialization removes these blockers. Diagnostic assertions are
+version-pinned to 0.1.14 and skip on another version, where the errors should be
+re-probed. Numerical CPU tests continue to run on other versions.
+
+### Numerical checks
+
+Run `python -m pytest tests/test_fast_cpu.py -q -s`. On the environment above:
+**58 passed**. CPU-only checks do not require CUDA; only the GPU comparisons skip
+without it. Coverage includes uneven GEMM M/N/K tiles, all GEMM epilogues, GEGLU,
+all residual/bias combinations, large residual values, RoPE position wraparound
+and untouched V columns, static/dynamic attention shapes, sliding windows,
+partial tiles, unequal lengths, empty sequences, and finite padding outputs.
+
+The seed is 1234. GPU comparisons use identical input values rounded to bf16 or
+fp16, then promoted to fp32 for CPU execution. These are absolute tolerances for
+the bounded fixtures, not a guarantee for arbitrary magnitudes or model depth.
+Attention comparisons use valid query rows, as in `tests/test_fast.py`.
+
+| Kernel | Max CPU vs fp32 reference | Max CPU vs GPU (both dtypes) | CPU/GPU tolerance |
+| --- | ---: | ---: | ---: |
+| GEMM | 2.38419e-7 | 0.00770831 | 0.05 |
+| GEGLU | 2.98023e-8 | 0.000208303 | 0.05 |
+| LayerNorm | 1.07288e-6 | 0.0156183 | 0.05 |
+| RoPE | 0 | 0.0130053 | 0.05 |
+| Attention | 5.96046e-7 | 0.00377572 | 0.02 |
+
+CPU/reference tolerance is 2e-5 (2e-6 for RoPE). Residual stream updates are
+exactly equal, including the no-residual case.
+
+### GPU preservation evidence
+
+The `cpu` keyword defaults to false. Existing bf16/fp16 defaults, GPU allocation
+scopes, parallel loops, swizzles and `FAST` options are unchanged. Default fp32
+GPU calls still raise `ValueError`.
+
+Before/after runs used the original module extracted with
+`git show fa9a2a7:laya/tl_kernels.py` and the changed module, respectively.
+The original was loaded as `laya.tl_kernels` via `importlib` for baseline runs;
+all other Laya code and the Python environment stayed the same.
+
+- `python -m pytest tests/test_fast.py -q`, with the full-forward tests configured
+ to load the cached English checkpoint identified below:
+ **13 passed before; 13 passed after**. Initially without a checkpoint it was
+ 11 passed / 2 skipped. Both full runs emitted the checkpoint's existing
+ temperature-clamping warning (`choice:11+=0.10058280825614929 -> 0.5`).
+- Seeded capture of the existing kernel tests: **24 output tensors bit-identical**,
+ including residual updates; maximum before/after difference **0**.
+- Generated CUDA source for all five probe shapes above, in both dtypes:
+ **10/10 byte-identical**. This also covers RoPE, absent from the original fast suite.
+- `python benchmarks/parity_fast.py --model "$MODEL" --dtype bf16 --json ...`
+ and the equivalent fp16 command: **288 questions over 60 states per dtype**.
+ All before/after JSON records (fp32, stock, and fast probabilities) compare
+ exactly equal; maximum before/after probability difference **0**.
+
+`MODEL` was the cached `convaiinnovations/laya` English snapshot
+`55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`; runs used `HF_HUB_OFFLINE=1`.
+No multilingual or typed-decisions checkpoint comparison is claimed here.
+
+| dtype | type | n | Max fast-stock, before = after | Fast/stock argmax agreement, before = after |
+| --- | --- | ---: | ---: | ---: |
+| bf16 | choice | 48 | 0.0310 | 47/48 |
+| bf16 | noul | 180 | 0.0756 | 180/180 |
+| bf16 | score | 60 | 0.0152 | 60/60 |
+| fp16 | choice | 48 | 0.0069 | 48/48 |
+| fp16 | noul | 180 | 0.0092 | 180/180 |
+| fp16 | score | 60 | 0.0040 | 60/60 |
+
+
+### Repository checks
+
+All commands used `/home/ckl/projects/S/laya/.venv/bin/python`; `ruff` and
+`zensical` came from that virtualenv's `bin` directory.
+
+| Command | Result |
+| --- | --- |
+| `ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120` | `All checks passed!` |
+| `python -m compileall -q laya/ tests/` | Exit 0, no output |
+| `python tests/test_router.py` | 703 passed, 0 failed |
+| `python tests/test_criteria.py` | 198 passed, 0 failed |
+| `python tests/test_hooks.py` | 240 passed, 0 failed |
+| `python tests/test_hooks_api.py` | 415 passed, 0 failed |
+| `python tests/test_packaging.py` | 131 passed, 0 failed |
+| `uv pip install --python /home/ckl/projects/S/laya/.venv/bin/python -r requirements-docs.txt` | Checked 3 packages (already installed); virtualenv has no pip |
+| `zensical build --strict --clean` | `No issues found`; no `griffe:` lines |
+
+The new suite is registered as requiring the optional TileLang extra and a C++
+compiler in the packaging test's existing exemptions. The API contract test pins
+the additive keyword-only CPU selector, unchanged GPU dtype default, and
+`compile_cpu` signature without importing TileLang into the base CI environment.
++
+
+ ## AOTInductor
+
+ `DecisionModel` can be exported, compiled into a `.pt2` package, and loaded with
+ `torch._inductor.aoti_load_package`. The package returns both decision logits and action logits.
+ Tokenization, padding, temperature calibration and answer formatting remain the caller's job;
+ this does not add an `Agent` backend or change its default execution path.
+
+ The [closed PR #472](https://github.com/NandhaKishorM/laya/pull/472) documented the earlier dtype
+ blocker. The action head's pooled state and confidence features are computed in fp32, even
+ when a model's weights are explicitly bf16. Without autocast, the first action linear therefore
+ received an fp32 input and bf16 weights. The forward now casts the concatenated input to the
+ head's weight dtype **only outside autocast**. Softmax, entropy and the returned decision logits
+ retain their fp32 calculations. Existing fp32-parameter eager inference, including fp16/bf16
+ AMP, keeps its numerics; mixed weight/autocast dtypes also retain autocast's original conversion.
+
+ Export a separate evaluation copy converted to bf16, outside autocast. Declare the token
+ axis as `16 * Dim("tokens16", ...)` to satisfy the attention alignment guards. Rows and marker
+ counts can be dynamic independently. Do not assume an export captured under autocast can be
+ packaged outside that context: use explicit weight dtypes for this recipe.
+
-## Reproduce offline
++### Reproduce offline
+
+ The assert-based check uses a tiny randomly initialized ModernBERT by default, without network
+ access. Pass a local checkpoint directory for a real-model measurement. Missing CUDA,
+ AOTInductor APIs or a C++ compiler produces an explicit `SKIP`; compilation and parity failures
+ on a supported installation fail the check.
+
+ ```bash
+ python scripts/check_aoti.py --output-dir /tmp/laya-aoti-smoke
+ HF_HUB_OFFLINE=1 TORCHINDUCTOR_CACHE_DIR=/tmp/laya-aoti/cache \
+ python scripts/check_aoti.py --model /path/to/local/multilingual \
+ --output-dir /tmp/laya-aoti
+ ```
+
+ The output directory holds `decision.pt2`, `results.json`, `inputs.pt` and `eager.pt`. The JSON
+ includes full `nvidia-smi` snapshots, export/package/load times, package bytes, latency and
+ maximum absolute logit/probability deltas for both heads. Binary artifacts and compiler caches
+ are deliberately kept out of the repository.
+
-## Measured on RTX 4070 Ti SUPER
++### Measured on RTX 4070 Ti SUPER
+
+ 2026-10-03, Linux x86-64, Python 3.12.13, torch 2.11.0+cu130, transformers 5.17.0,
+ NVIDIA driver 615.71.09, 16,376 MiB VRAM. Checkpoint: `convaiinnovations/laya`,
+ `multilingual` subdirectory at revision `1c5edc17a7acd8701df6fc341c0d179f1c62c982`.
+ The baseline is main `fa9a2a7`. Full machine snapshots and unrounded measurements are in
+ [`aoti_multilingual_rtx4070.json`](https://github.com/NandhaKishorM/laya/blob/main/benchmarks/results/aoti_multilingual_rtx4070.json).
+
+ | Measurement | Before | After |
+ |---|---:|---:|
+ | Export | 5.00 s | 3.89 s |
+ | Packaging | failed after 16.59 s | 60.02 s |
+ | Package size | no artifact | 645,729,621 bytes (615.82 MiB) |
+ | Load in the already initialized process | unavailable | 0.447 s |
+ | Load in a fresh process with an empty cache | unavailable | 4.867 s |
+
+ The reproduced failure is at packaging, after successful export:
+ `mat1 and mat2 must have the same dtype, but got Float and BFloat16`.
+ Each run used a separate initially empty Inductor cache; AMP compilation ran before packaging,
+ so the package timing is not a fresh-interpreter cold-start measurement.
+ The fresh-process check loaded only the package and saved inputs (no checkpoint), with
+ `torch.compile` and packaging entry points blocked. It reproduced both heads' answers.
+ PyTorch still compiled a small CPU AVX capability probe into the initially empty cache;
+ no model graph or CUDA kernel was compiled. A blanket "no compiler activity at load" claim
+ would therefore be inaccurate on this runtime.
+
+ The same saved inputs contain seven decision rows across three batches, with real tokenized
+ billing/refund text, choice/score/noul questions, padding and unequal valid marker counts.
+ Each latency is the median of five groups of 30 forwards after ten warmups, using synchronized
+ wall-clock timing. `torch.compile(dynamic=True)` uses the default mode. No explicit CUDA graphs,
+ tokenization, export or compilation are included in these latency numbers.
+
+ | Rows × tokens × marker slots | AMP eager before → after (ms) | AMP compiled before → after (ms) | Explicit bf16 eager (ms) | Explicit bf16 compiled (ms) | AOTI bf16 (ms) |
+ |---|---:|---:|---:|---:|---:|
+ | 2 × 128 × 3 | 15.1173 → 14.1200 | 6.7117 → 6.9076 | 13.9635 | 4.8835 | 2.2674 |
+ | 1 × 256 × 3 | 15.4687 → 14.4156 | 6.7274 → 5.4772 | 14.7994 | 4.7250 | 2.4154 |
+ | 4 × 160 × 5 | 14.8452 → 14.5384 | 7.3872 → 5.5934 | 14.7115 | 5.1393 | 3.6018 |
+
+ AMP here means fp32 parameters with bf16 autocast. Explicit bf16 means weights and the residual
+ stream are bf16, without autocast; use those columns for the closest execution comparison to
+ the package. Explicit bf16 eager/compiled and AOTI were unavailable before this fix.
+
+ | Comparison, maximum over all seven rows | Decision logits | Decision probabilities | Action logits | Action probabilities |
+ |---|---:|---:|---:|---:|
+ | Existing eager before vs after, fp32 and fp16/bf16 AMP | 0 | 0 | 0 | 0 |
+ | AOTI vs explicit bf16 eager | 0.5625 | 0.00659859 | 0 | 0 |
+ | AOTI vs existing bf16 AMP eager | 0.4375 | 0.00769910 | 8.0 | 0 |
+
+ Both decision and action argmax agree on **7/7 rows** for both AOTI comparisons. Probabilities
+ are raw softmax outputs, without calibration. The action distribution is saturated on these
+ inputs, so its zero probability delta does not imply identical underlying logits against AMP.
+ The explicit bf16 compiled forward also differs from explicit bf16 eager (maximum decision
+ logit delta 0.5, probability delta 0.00659859). Reduced-precision compilation is not bit-exact.
+
+ The GPU was shared with desktop applications and other Python jobs; CPU compilation was also
+ shared, and clocks were not locked. These are observed timings, not an isolated speedup claim.
+ `nvidia-smi` snapshots bracketing the runs:
+
+ | Run / snapshot | GPU utilization | Used VRAM | Power | Temperature / state |
+ |---|---:|---:|---:|---|
+ | Before / start | 28% | 2,817 MiB | 12 W | 33°C / P8 |
+ | Before / end | 10% | 8,560 MiB | 23 W | 36°C / P3 |
+ | After / start | 0% | 2,634 MiB | 12 W | 33°C / P8 |
+ | After / end | 73% | 6,476 MiB | 160 W | 42°C / P2 |
+
-## Remaining limits
++### Remaining limits
+
+ - The artifact is specific to this checkpoint, precision, PyTorch/runtime stack and GPU target;
+ portability to other hardware or PyTorch versions was not tested.
+ - This check exports rows 1–8, tokens 32–512 in multiples of 16, and marker slots 2–8. It exercises
+ three shapes, including shapes different from the export example, rather than every point
+ in those ranges. One valid option can use padded marker slots; a true one-slot tensor takes
+ the separate single-option branch and needs a separate export. Arbitrary token lengths and
+ a production bucketing/dispatch layer are outside this change.
+ - Seven rows verify the packaging regression, not broad checkpoint accuracy or calibrated
+ confidence parity. Casting an export copy to bf16 differs from retaining fp32 weights under
+ autocast; the existing eager path itself remains unchanged.
+ - The script requires a CUDA-capable PyTorch build and local compiler toolchain to create the
+ artifact. The `.pt2` embeds the model and CUDA kernels; no hosted service is involved.