Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(fast): add fp32 TileLang CPU lowering

cklxxb7abae7cb9c2
diff --git a/docs/fast-backends.md b/docs/fast-backends.md
new file mode 100644
index 00000000..3588cd2a
--- /dev/null
+++ b/docs/fast-backends.md
@@ -0,0 +1,171 @@
+# Fast backends
+
+## TileLang portability
+
+All five kernels in `laya/tl_kernels.py` can execute on TileLang's CPU (`c`)
+target through `compile_cpu`. This is an explicit, scalar fp32 specialization,
+not a CPU acceleration backend for `Agent.accelerate`. It requires TileLang and
+a local C++ compiler. It introduces no runtime service dependency.
+
+```python
+import torch
+from laya import tl_kernels as K
+
+A = torch.randn(17, 67)
+W = torch.randn(70, 67)
+b = torch.randn(70)
+C = torch.empty(17, 70)
+kernel = K.compile_cpu(K.gemm_kernel, 70, 67, bias=True, act="gelu")
+kernel(A, W, b, C)
+```
+
+`compile_cpu(factory, *args, **kwargs)` takes the same shape and operation options
+as the five GPU factories. It selects `cpu=True`, `dtype="float32"`, target `c`,
+and disables vectorization. All floating inputs and outputs must be contiguous
+CPU float32 tensors; attention lengths remain int32. Convert 16-bit activations
+with `.cpu().float().contiguous()` before calling it. LayerNorm still updates its
+residual tensor in place, and RoPE still updates the Q/K columns in place.
+Retain the compiled kernel to reuse its dynamic dimensions.
+
+CPU specialization replaces fragment and shared allocations with local buffers,
+uses serial `T.grid` loops, and omits attention's GPU swizzle annotation. GEMMs
+use TileLang's scalar CPU implementation and reductions use local buffers.
+The original tiled algorithms, padding predicates, attention masks, and online
+softmax remain shared with the GPU implementation. CPU intermediates stay fp32,
+including attention probabilities; GPU intermediates retain their original dtype.
+No CPU speedup or full-model CPU backend is claimed.
+
+### Re-probe on TileLang 0.1.14
+
+Observed on Linux, Python 3.12.13, torch 2.11.0+cu130, TileLang 0.1.14,
+and an RTX 4070 Ti SUPER. The earlier portability concern applies to compiling
+the **unchanged GPU specialization**, not to the possibility of a CPU lowering.
+At base commit `fa9a2a7`, this documentation page was absent from the checkout.
+
+The test compiles `factory.get_tir(...)` with `target="c"` and the GPU's `FAST`
+pass configuration. These are the exact diagnostic texts (source locations and
+stack traces omitted). Both bf16 and fp16 were probed for every kernel.
+
+| Kernel and probe dimensions | First bf16 failure | First fp16 failure |
+| --- | --- | --- |
+| `gemm_kernel(128, 64)` | `Check failed: layout_map.count(buffer) != 0 (0 vs. 0) : The layout for fragment C_l can not be inferred correctly.` | Same |
+| `gemm_geglu_kernel(64, 64)` | ``CPU fill only supports local and global buffers, but got dst scope `local.fragment`.`` | Same |
+| `add_ln_kernel(128)` | ``CPU reduce only supports local src and local/local.var dst buffers, got src scope `local.fragment` and dst scope `local.fragment`.`` | Same |
+| `rope_kernel(2, 64)` | `Cannot convert type bfloat16 to C type` | C++ compiler: `error: no matching function for call to ‘vec_type<float, 4>::vec_type(half4&)’` |
+| `attn_kernel(1, 64, 2, 64)` | `Check failed: layout_map.count(buffer) != 0 (0 vs. 0) : The layout for fragment s_c can not be inferred correctly.` | Same |
+
+The first four fragment/reduction failures are `tvm.error.InternalError`.
+The bf16 codegen failure is also an `InternalError`. FP16 RoPE raises
+`RuntimeError: Compilation Failed!` followed by the compiler invocation and
+source; the diagnostic above is emitted on stderr. Its generated vector casts
+also fail converting float vectors back to half vectors.
+
+To isolate dtype support from fragment support, the tests compile **each** kernel
+again with `cpu=True` (local buffers and serial loops), retaining bf16 and disabling
+vectorization. All five then fail with exactly:
+
+```text
+Cannot convert type bfloat16 to C type
+```
+
+Thus fragments are the first blocker for GEMM, GEGLU and attention, fragment
+reductions for LayerNorm, and bf16 is independently a blocker for all five.
+RoPE has no fragment allocation or reduction; GEMM and GEGLU have no explicit
+`T.reduce_*` operation. Attention's reductions are initially masked by its layout
+failure. Separate fp32-only fragment probes reproduce the fill error and the
+reduction error for both `reduce_sum` and `reduce_max`, without any GEMM or bf16.
+Replacing scopes alone also produced this semantic error for the GEMM
+buffer (the other affected buffers were `Ci`, `x`, and `s`):
+
+```text
+[Tilelang Semantic Check] Local buffer `C_l` is indexed by T.Parallel loop variable `i`. Local buffers are thread-private and do not participate in parallel layout inference. Use T.serial/T.vectorized/T.unroll for per-thread local indexing, or T.alloc_fragment when the indexed dimension should be distributed across threads.
+```
+
+The serial fp32 specialization removes these blockers. Diagnostic assertions are
+version-pinned to 0.1.14 and skip on another version, where the errors should be
+re-probed. Numerical CPU tests continue to run on other versions.
+
+### Numerical checks
+
+Run `python -m pytest tests/test_fast_cpu.py -q -s`. On the environment above:
+**58 passed**. CPU-only checks do not require CUDA; only the GPU comparisons skip
+without it. Coverage includes uneven GEMM M/N/K tiles, all GEMM epilogues, GEGLU,
+all residual/bias combinations, large residual values, RoPE position wraparound
+and untouched V columns, static/dynamic attention shapes, sliding windows,
+partial tiles, unequal lengths, empty sequences, and finite padding outputs.
+
+The seed is 1234. GPU comparisons use identical input values rounded to bf16 or
+fp16, then promoted to fp32 for CPU execution. These are absolute tolerances for
+the bounded fixtures, not a guarantee for arbitrary magnitudes or model depth.
+Attention comparisons use valid query rows, as in `tests/test_fast.py`.
+
+| Kernel | Max CPU vs fp32 reference | Max CPU vs GPU (both dtypes) | CPU/GPU tolerance |
+| --- | ---: | ---: | ---: |
+| GEMM | 2.38419e-7 | 0.00770831 | 0.05 |
+| GEGLU | 2.98023e-8 | 0.000208303 | 0.05 |
+| LayerNorm | 1.07288e-6 | 0.0156183 | 0.05 |
+| RoPE | 0 | 0.0130053 | 0.05 |
+| Attention | 5.96046e-7 | 0.00377572 | 0.02 |
+
+CPU/reference tolerance is 2e-5 (2e-6 for RoPE). Residual stream updates are
+exactly equal, including the no-residual case.
+
+### GPU preservation evidence
+
+The `cpu` keyword defaults to false. Existing bf16/fp16 defaults, GPU allocation
+scopes, parallel loops, swizzles and `FAST` options are unchanged. Default fp32
+GPU calls still raise `ValueError`.
+
+Before/after runs used the original module extracted with
+`git show fa9a2a7:laya/tl_kernels.py` and the changed module, respectively.
+The original was loaded as `laya.tl_kernels` via `importlib` for baseline runs;
+all other Laya code and the Python environment stayed the same.
+
+- `python -m pytest tests/test_fast.py -q`, with `LAYA_TEST_MODEL` set:
+ **13 passed before; 13 passed after**. Initially without that variable it was
+ 11 passed / 2 skipped. Both full runs emitted the checkpoint's existing
+ temperature-clamping warning (`choice:11+=0.10058280825614929 -> 0.5`).
+- Seeded capture of the existing kernel tests: **24 output tensors bit-identical**,
+ including residual updates; maximum before/after difference **0**.
+- Generated CUDA source for all five probe shapes above, in both dtypes:
+ **10/10 byte-identical**. This also covers RoPE, absent from the original fast suite.
+- `python benchmarks/parity_fast.py --model "$MODEL" --dtype bf16 --json ...`
+ and the equivalent fp16 command: **288 questions over 60 states per dtype**.
+ All before/after JSON records (fp32, stock, and fast probabilities) compare
+ exactly equal; maximum before/after probability difference **0**.
+
+`MODEL` was the cached `convaiinnovations/laya` English snapshot
+`55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`; runs used `HF_HUB_OFFLINE=1`.
+No multilingual or typed-decisions checkpoint comparison is claimed here.
+
+| dtype | type | n | Max fast-stock, before = after | Fast/stock argmax agreement, before = after |
+| --- | --- | ---: | ---: | ---: |
+| bf16 | choice | 48 | 0.0310 | 47/48 |
+| bf16 | noul | 180 | 0.0756 | 180/180 |
+| bf16 | score | 60 | 0.0152 | 60/60 |
+| fp16 | choice | 48 | 0.0069 | 48/48 |
+| fp16 | noul | 180 | 0.0092 | 180/180 |
+| fp16 | score | 60 | 0.0040 | 60/60 |
+
+
+### Repository checks
+
+All commands used `/home/ckl/projects/S/laya/.venv/bin/python`; `ruff` and
+`zensical` came from that virtualenv's `bin` directory.
+
+| Command | Result |
+| --- | --- |
+| `ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120` | `All checks passed!` |
+| `python -m compileall -q laya/ tests/` | Exit 0, no output |
+| `python tests/test_router.py` | 703 passed, 0 failed |
+| `python tests/test_criteria.py` | 198 passed, 0 failed |
+| `python tests/test_hooks.py` | 240 passed, 0 failed |
+| `python tests/test_hooks_api.py` | 415 passed, 0 failed |
+| `python tests/test_packaging.py` | 131 passed, 0 failed |
+| `uv pip install --python /home/ckl/projects/S/laya/.venv/bin/python -r requirements-docs.txt` | Checked 3 packages (already installed); virtualenv has no pip |
+| `zensical build --strict --clean` | `No issues found`; no `griffe:` lines |
+
+The new suite is registered as requiring the optional TileLang extra and a C++
+compiler in the packaging test's existing exemptions. The API contract test pins
+the additive keyword-only CPU selector, unchanged GPU dtype default, and
+`compile_cpu` signature without importing TileLang into the base CI environment.
diff --git a/laya/tl_kernels.py b/laya/tl_kernels.py
index b5c9ee61..5121fbf8 100644
--- a/laya/tl_kernels.py
+++ b/laya/tl_kernels.py
@@ -1,8 +1,9 @@
"""TileLang kernels for the Laya (ModernBERT + decision head) encoder.
-All kernels take 16-bit activations (bf16 by default, fp16 with dtype="float16"), accumulate in fp32. Row count M is a runtime
+GPU kernels take 16-bit activations (bf16 by default, fp16 with dtype="float16"), accumulate in fp32. Row count M is a runtime
symbol so one compiled kernel serves every batch/sequence bucket; M must be a
multiple of 16 (the caller pads); out-of-bounds rows are predicated by TileLang.
+Use compile_cpu for an explicit fp32 CPU specialization; the GPU defaults are unchanged.
"""
import tilelang
import tilelang.language as T
@@ -12,7 +13,9 @@ DTYPES = ("bfloat16", "float16")
FAST = {tilelang.PassConfigKey.TL_ENABLE_FAST_MATH: True}
-def _dt(dtype):
+def _dt(dtype, cpu=False):
+ if cpu and dtype == "float32":
+ return dtype
if dtype not in DTYPES:
raise ValueError("tl_kernels dtype must be one of %s, got %r" % (DTYPES, dtype))
return dtype
@@ -28,17 +31,20 @@ def _act(x, kind):
# ----------------------------------------------------------------------------- GEMM
@tilelang.jit(pass_configs=FAST)
-def gemm_kernel(N, K, bias=False, act="none", bm=64, bn=128, bk=64, stages=3, threads=128, dtype="bfloat16"):
+def gemm_kernel(N, K, bias=False, act="none", bm=64, bn=128, bk=64, stages=3, threads=128, dtype="bfloat16", *, cpu=False):
"""C[M,N] = act(A[M,K] @ W[N,K]^T + b)."""
- DT = _dt(dtype)
+ DT = _dt(dtype, cpu)
+ alloc = T.alloc_local if cpu else T.alloc_fragment
+ shared = T.alloc_local if cpu else T.alloc_shared
+ loop = T.grid if cpu else T.Parallel
M = T.dynamic("M")
@T.prim_func
def main(A: T.Tensor((M, K), DT), W: T.Tensor((N, K), DT), Bv: T.Tensor((N,), ACC), C: T.Tensor((M, N), DT)):
with T.Kernel(T.ceildiv(N, bn), T.ceildiv(M, bm), threads=threads) as (bx, by):
- A_s = T.alloc_shared((bm, bk), DT)
- W_s = T.alloc_shared((bn, bk), DT)
- C_l = T.alloc_fragment((bm, bn), ACC)
+ A_s = shared((bm, bk), DT)
+ W_s = shared((bn, bk), DT)
+ C_l = alloc((bm, bn), ACC)
T.clear(C_l)
for k in T.Pipelined(T.ceildiv(K, bk), num_stages=stages):
T.copy(A[by * bm, k * bk], A_s)
@@ -46,29 +52,32 @@ def gemm_kernel(N, K, bias=False, act="none", bm=64, bn=128, bk=64, stages=3, th
T.gemm(A_s, W_s, C_l, transpose_B=True)
# `bias` is a Python-level flag, so each branch is one expression (no re-bound TileLang value)
if bias:
- for i, j in T.Parallel(bm, bn):
+ for i, j in loop(bm, bn):
C_l[i, j] = _act(C_l[i, j] + Bv[bx * bn + j], act)
else:
- for i, j in T.Parallel(bm, bn):
+ for i, j in loop(bm, bn):
C_l[i, j] = _act(C_l[i, j], act)
T.copy(C_l, C[by * bm, bx * bn])
return main
@tilelang.jit(pass_configs=FAST)
-def gemm_geglu_kernel(F, K, bm=64, bn=64, bk=64, stages=3, threads=128, dtype="bfloat16"):
+def gemm_geglu_kernel(F, K, bm=64, bn=64, bk=64, stages=3, threads=128, dtype="bfloat16", *, cpu=False):
"""ModernBERT GLU MLP up-projection, fused: C[M,F] = gelu(A @ Wi[:F]^T) * (A @ Wi[F:]^T)."""
- DT = _dt(dtype)
+ DT = _dt(dtype, cpu)
+ alloc = T.alloc_local if cpu else T.alloc_fragment
+ shared = T.alloc_local if cpu else T.alloc_shared
+ loop = T.grid if cpu else T.Parallel
M = T.dynamic("M")
@T.prim_func
def main(A: T.Tensor((M, K), DT), W: T.Tensor((2 * F, K), DT), C: T.Tensor((M, F), DT)):
with T.Kernel(T.ceildiv(F, bn), T.ceildiv(M, bm), threads=threads) as (bx, by):
- A_s = T.alloc_shared((bm, bk), DT)
- Wi_s = T.alloc_shared((bn, bk), DT)
- Wg_s = T.alloc_shared((bn, bk), DT)
- Ci = T.alloc_fragment((bm, bn), ACC)
- Cg = T.alloc_fragment((bm, bn), ACC)
+ A_s = shared((bm, bk), DT)
+ Wi_s = shared((bn, bk), DT)
+ Wg_s = shared((bn, bk), DT)
+ Ci = alloc((bm, bn), ACC)
+ Cg = alloc((bm, bn), ACC)
T.clear(Ci); T.clear(Cg)
for k in T.Pipelined(T.ceildiv(K, bk), num_stages=stages):
T.copy(A[by * bm, k * bk], A_s)
@@ -76,7 +85,7 @@ def gemm_geglu_kernel(F, K, bm=64, bn=64, bk=64, stages=3, threads=128, dtype="b
T.copy(W[F + bx * bn, k * bk], Wg_s)
T.gemm(A_s, Wi_s, Ci, transpose_B=True)
T.gemm(A_s, Wg_s, Cg, transpose_B=True)
- for i, j in T.Parallel(bm, bn):
+ for i, j in loop(bm, bn):
Ci[i, j] = _act(Ci[i, j], "gelu") * Cg[i, j]
T.copy(Ci, C[by * bm, bx * bn])
return main
@@ -84,47 +93,50 @@ def gemm_geglu_kernel(F, K, bm=64, bn=64, bk=64, stages=3, threads=128, dtype="b
# ----------------------------------------------------------------------------- LayerNorm (+residual)
@tilelang.jit(pass_configs=FAST)
-def add_ln_kernel(D, residual=True, bias=False, eps=1e-5, bm=4, threads=32, dtype="bfloat16"):
+def add_ln_kernel(D, residual=True, bias=False, eps=1e-5, bm=4, threads=32, dtype="bfloat16", *, cpu=False):
"""X (fp32 residual stream) += R (16-bit branch output, if residual); Y (16-bit) = LN(X) * w (+ b).
The residual stream stays in fp32 exactly like the stock autocast path: ModernBERT-large's residual
activations reach ~3e4, where bf16's 8-bit mantissa would lose ~100 units per add and drift layer by layer."""
- DT = _dt(dtype)
+ DT = _dt(dtype, cpu)
+ alloc = T.alloc_local if cpu else T.alloc_fragment
+ shared = T.alloc_local if cpu else T.alloc_shared
+ loop = T.grid if cpu else T.Parallel
M = T.dynamic("M")
@T.prim_func
def main(X: T.Tensor((M, D), ACC), R: T.Tensor((M, D), DT), Wv: T.Tensor((D,), ACC), Bv: T.Tensor((D,), ACC),
Y: T.Tensor((M, D), DT)):
with T.Kernel(T.ceildiv(M, bm), threads=threads) as bx:
- x = T.alloc_fragment((bm, D), ACC)
- xs = T.alloc_fragment((bm, D), ACC)
- mean = T.alloc_fragment((bm,), ACC)
- var = T.alloc_fragment((bm,), ACC)
- Xb = T.alloc_shared((bm, D), ACC)
- Rb = T.alloc_shared((bm, D), DT)
- Yb = T.alloc_shared((bm, D), DT)
+ x = alloc((bm, D), ACC)
+ xs = alloc((bm, D), ACC)
+ mean = alloc((bm,), ACC)
+ var = alloc((bm,), ACC)
+ Xb = shared((bm, D), ACC)
+ Rb = shared((bm, D), DT)
+ Yb = shared((bm, D), DT)
T.copy(X[bx * bm, 0], Xb)
T.copy(Xb, x)
if residual:
T.copy(R[bx * bm, 0], Rb)
T.copy(Rb, xs)
- for i, j in T.Parallel(bm, D):
+ for i, j in loop(bm, D):
x[i, j] = x[i, j] + xs[i, j]
T.copy(x, Xb)
T.copy(Xb, X[bx * bm, 0])
T.reduce_sum(x, mean, dim=1)
- for i in T.Parallel(bm):
+ for i in loop(bm):
mean[i] = mean[i] / D
- for i, j in T.Parallel(bm, D):
+ for i, j in loop(bm, D):
xs[i, j] = (x[i, j] - mean[i]) * (x[i, j] - mean[i])
T.reduce_sum(xs, var, dim=1)
- for i in T.Parallel(bm):
+ for i in loop(bm):
var[i] = T.rsqrt(var[i] / D + eps)
if bias:
- for i, j in T.Parallel(bm, D):
+ for i, j in loop(bm, D):
xs[i, j] = (x[i, j] - mean[i]) * var[i] * Wv[j] + Bv[j]
else:
- for i, j in T.Parallel(bm, D):
+ for i, j in loop(bm, D):
xs[i, j] = (x[i, j] - mean[i]) * var[i] * Wv[j]
T.copy(xs, Yb)
T.copy(Yb, Y[bx * bm, 0])
@@ -133,10 +145,11 @@ def add_ln_kernel(D, residual=True, bias=False, eps=1e-5, bm=4, threads=32, dtyp
# ----------------------------------------------------------------------------- RoPE (in place on packed qkv)
@tilelang.jit(pass_configs=FAST)
-def rope_kernel(H, Dh, bm=32, threads=128, dtype="bfloat16"):
+def rope_kernel(H, Dh, bm=32, threads=128, dtype="bfloat16", *, cpu=False):
"""QKV[M, 3*H*Dh] packed as (q|k|v)(h)(d). Rotates q and k in place (rotate-half convention, fp32 math).
cos/sin: [L, Dh/2]. Row r has position r % L. M and L are runtime symbols."""
- DT = _dt(dtype)
+ DT = _dt(dtype, cpu)
+ loop = T.grid if cpu else T.Parallel
M, L = T.dynamic("M"), T.dynamic("L")
half = Dh // 2
W = 2 * H * Dh # q and k columns
@@ -144,7 +157,7 @@ def rope_kernel(H, Dh, bm=32, threads=128, dtype="bfloat16"):
@T.prim_func
def main(QKV: T.Tensor((M, 3 * H * Dh), DT), Cos: T.Tensor((L, half), ACC), Sin: T.Tensor((L, half), ACC)):
with T.Kernel(T.ceildiv(M, bm), threads=threads) as bx:
- for i, c in T.Parallel(bm, W // 2):
+ for i, c in loop(bm, W // 2):
r = bx * bm + i
pos = r % L
hh = c // half # which (q|k, head)
@@ -162,14 +175,17 @@ def rope_kernel(H, Dh, bm=32, threads=128, dtype="bfloat16"):
# ----------------------------------------------------------------------------- flash attention (padding mask + sliding window)
@tilelang.jit(pass_configs=FAST)
-def attn_kernel(B, L, H, Dh, window=0, bm=64, bn=64, stages=1, threads=128, dtype="bfloat16"):
+def attn_kernel(B, L, H, Dh, window=0, bm=64, bn=64, stages=1, threads=128, dtype="bfloat16", *, cpu=False):
"""QKV: [B, L, 3, H, Dh] 16-bit (a view of the packed [M, 3*H*Dh] buffer). Lens: [B] int32 valid length.
O: [B, L, H*Dh]. window>0 => bidirectional sliding window |i-j| <= window. Masked scores use a large
finite negative so fully-masked (padding) rows stay finite.
B and/or L may be None: they then become runtime symbols (one compile serves every shape, at the
cost of predicated loads -- ~4x slower for full attention at L=1024, free for short inputs)."""
- DT = _dt(dtype)
+ DT = _dt(dtype, cpu)
+ alloc = T.alloc_local if cpu else T.alloc_fragment
+ shared = T.alloc_local if cpu else T.alloc_shared
+ loop = T.grid if cpu else T.Parallel
scale = (1.0 / Dh) ** 0.5 * 1.44269504 # log2(e)
if B is None:
B = T.dynamic("B")
@@ -180,19 +196,20 @@ def attn_kernel(B, L, H, Dh, window=0, bm=64, bn=64, stages=1, threads=128, dtyp
@T.prim_func
def main(QKV: T.Tensor((B, L, 3, H, Dh), DT), Lens: T.Tensor((B,), "int32"), O: T.Tensor((B, L, H * Dh), DT)):
with T.Kernel(T.ceildiv(L, bm), H, B, threads=threads) as (bx, by, bz):
- Q_s = T.alloc_shared((bm, Dh), DT)
- K_s = T.alloc_shared((bn, Dh), DT)
- V_s = T.alloc_shared((bn, Dh), DT)
- O_s = T.alloc_shared((bm, Dh), DT)
- s = T.alloc_fragment((bm, bn), ACC)
- s_c = T.alloc_fragment((bm, bn), DT)
- o = T.alloc_fragment((bm, Dh), ACC)
- m = T.alloc_fragment((bm,), ACC)
- m_prev = T.alloc_fragment((bm,), ACC)
- sc = T.alloc_fragment((bm,), ACC)
- rs = T.alloc_fragment((bm,), ACC)
- l = T.alloc_fragment((bm,), ACC)
- T.annotate_layout({Q_s: tilelang.layout.make_swizzled_layout(Q_s)})
+ Q_s = shared((bm, Dh), DT)
+ K_s = shared((bn, Dh), DT)
+ V_s = shared((bn, Dh), DT)
+ O_s = shared((bm, Dh), DT)
+ s = alloc((bm, bn), ACC)
+ s_c = alloc((bm, bn), DT)
+ o = alloc((bm, Dh), ACC)
+ m = alloc((bm,), ACC)
+ m_prev = alloc((bm,), ACC)
+ sc = alloc((bm,), ACC)
+ rs = alloc((bm,), ACC)
+ l = alloc((bm,), ACC)
+ if not cpu:
+ T.annotate_layout({Q_s: tilelang.layout.make_swizzled_layout(Q_s)})
T.copy(QKV[bz, bx * bm:(bx + 1) * bm, 0, by, :], Q_s)
T.fill(o, 0); T.fill(l, 0); T.fill(m, NEG)
n = Lens[bz]
@@ -204,7 +221,7 @@ def attn_kernel(B, L, H, Dh, window=0, bm=64, bn=64, stages=1, threads=128, dtyp
k_hi = T.ceildiv(n, bn)
for k in T.Pipelined(k_lo, k_hi, num_stages=stages):
T.copy(QKV[bz, k * bn:(k + 1) * bn, 1, by, :], K_s)
- for i, j in T.Parallel(bm, bn):
+ for i, j in loop(bm, bn):
qi = bx * bm + i
kj = k * bn + j
if window > 0:
@@ -216,19 +233,32 @@ def attn_kernel(B, L, H, Dh, window=0, bm=64, bn=64, stages=1, threads=128, dtyp
T.copy(QKV[bz, k * bn:(k + 1) * bn, 2, by, :], V_s)
T.copy(m, m_prev)
T.reduce_max(s, m, dim=1, clear=False)
- for i in T.Parallel(bm):
+ for i in loop(bm):
sc[i] = T.exp2(m_prev[i] * scale - m[i] * scale)
- for i, j in T.Parallel(bm, bn):
+ for i, j in loop(bm, bn):
s[i, j] = T.exp2(s[i, j] * scale - m[i] * scale)
T.reduce_sum(s, rs, dim=1)
- for i in T.Parallel(bm):
+ for i in loop(bm):
l[i] = l[i] * sc[i] + rs[i]
T.copy(s, s_c)
- for i, j in T.Parallel(bm, Dh):
+ for i, j in loop(bm, Dh):
o[i, j] = o[i, j] * sc[i]
T.gemm(s_c, V_s, o, policy=T.GemmWarpPolicy.FullRow)
- for i, j in T.Parallel(bm, Dh):
+ for i, j in loop(bm, Dh):
o[i, j] = o[i, j] / T.max(l[i], 1e-30)
T.copy(o, O_s)
T.copy(O_s, O[bz, bx * bm:(bx + 1) * bm, by * Dh:(by + 1) * Dh])
return main
+
+
+def compile_cpu(kernel, *args, **kwargs):
+ """Compile a kernel factory for CPU with fp32 tensors and local scratch buffers.
+
+ This is a scalar correctness path, not an Agent.accelerate backend. Convert all
+ floating inputs to contiguous CPU float32 tensors; integer lengths stay int32.
+ Output buffers (including the in-place residual and QKV) must also be on CPU.
+ """
+ kwargs["dtype"] = "float32"
+ kwargs["cpu"] = True
+ return tilelang.compile(kernel.get_tir(*args, **kwargs), target="c",
+ pass_configs={tilelang.PassConfigKey.TIR_DISABLE_VECTORIZE: True})
diff --git a/tests/test_fast_cpu.py b/tests/test_fast_cpu.py
new file mode 100644
index 00000000..7ca60833
--- /dev/null
+++ b/tests/test_fast_cpu.py
@@ -0,0 +1,205 @@
+"""TileLang C target: original limitations, fp32 CPU execution, and CUDA parity.
+
+Run: python -m pytest tests/test_fast_cpu.py -q -s
+CPU checks need torch, tilelang and a C++ compiler; CUDA parity skips without CUDA.
+"""
+import os, sys
+import pytest
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+torch = pytest.importorskip("torch")
+tilelang = pytest.importorskip("tilelang")
+from laya import tl_kernels as K # noqa: E402
+
+
[email protected](params=[torch.float32, torch.bfloat16, torch.float16], ids=["cpu", "bf16", "fp16"])
+def dtype(request):
+ if request.param != torch.float32 and not torch.cuda.is_available():
+ pytest.skip("GPU parity needs CUDA")
+ torch.manual_seed(1234)
+ return request.param
+
+
+def rand(*shape, dtype=torch.float32, scale=1):
+ return (torch.randn(*shape) * scale).to(dtype).float()
+
+
+def check(name, actual, expected, tol):
+ delta = (actual.float() - expected.float()).abs().max().item()
+ print("%s max_abs=%.9g tolerance=%g" % (name, delta, tol))
+ assert torch.isfinite(actual).all()
+ assert delta <= tol
+
+
+def gpu(dtype, factory, args, tensors, floating, **kwargs):
+ dt = {torch.bfloat16: "bfloat16", torch.float16: "float16"}[dtype]
+ values = [x.cuda().to(dtype) if i in floating else x.cuda() for i, x in enumerate(tensors)]
+ factory(*args, dtype=dt, **kwargs)(*values)
+ return [x.cpu() for x in values]
+
+
[email protected]("bias,act", [(False, "none"), (True, "gelu"), (True, "relu")])
+def test_gemm(dtype, bias, act):
+ M, N, D = 17, 70, 67
+ a, w, b = rand(M, D, dtype=dtype), rand(N, D, dtype=dtype, scale=0.02), rand(N)
+ out = torch.empty(M, N)
+ opts = dict(bias=bias, act=act)
+ K.compile_cpu(K.gemm_kernel, N, D, **opts)(a, w, b, out)
+ ref = a @ w.T + (b if bias else 0)
+ ref = {"none": ref, "gelu": torch.nn.functional.gelu(ref), "relu": torch.relu(ref)}[act]
+ check("gemm/cpu-reference", out, ref, 2e-5)
+ if dtype != torch.float32:
+ other = gpu(dtype, K.gemm_kernel, (N,D), [a,w,b,torch.empty_like(out)], (0,1,3), **opts)[-1]
+ check("gemm/cpu-gpu", out, other, 0.05)
+
+
+def test_geglu(dtype):
+ M, F, D = 17, 70, 67
+ a, w = rand(M,D,dtype=dtype), rand(2*F,D,dtype=dtype,scale=0.02)
+ out = torch.empty(M,F)
+ K.compile_cpu(K.gemm_geglu_kernel,F,D)(a,w,out)
+ x = a @ w.T
+ check("geglu/cpu-reference", out, torch.nn.functional.gelu(x[:,:F]) * x[:,F:], 2e-5)
+ if dtype != torch.float32:
+ other = gpu(dtype,K.gemm_geglu_kernel,(F,D),[a,w,torch.empty_like(out)],(0,1,2))[-1]
+ check("geglu/cpu-gpu", out, other, 0.05)
+
+
[email protected]("residual,bias", [(False,False),(False,True),(True,False),(True,True)])
+def test_layernorm(dtype, residual, bias):
+ M, D = 17, 128
+ x, r = rand(M,D,scale=3000), rand(M,D,dtype=dtype,scale=50)
+ w, b = torch.rand(D)+0.5, rand(D)
+ updated, out = x.clone(), torch.empty_like(x)
+ opts = dict(residual=residual,bias=bias)
+ K.compile_cpu(K.add_ln_kernel,D,**opts)(updated,r,w,b,out)
+ refx = x + r if residual else x
+ assert torch.equal(updated,refx)
+ check("ln/cpu-reference",out,torch.nn.functional.layer_norm(refx,(D,),w,b if bias else None,1e-5),2e-5)
+ if dtype != torch.float32:
+ other = gpu(dtype,K.add_ln_kernel,(D,),[x.clone(),r,w,b,torch.empty_like(out)],(1,4),**opts)
+ assert torch.equal(updated,other[0])
+ check("ln/cpu-gpu",out,other[-1],0.05)
+
+
+def test_rope(dtype):
+ M, L, H, D = 35, 7, 2, 64
+ qkv = rand(M,3*H*D,dtype=dtype)
+ angles = rand(L,D//2)
+ cos, sin = angles.cos(), angles.sin()
+ out = qkv.clone()
+ K.compile_cpu(K.rope_kernel,H,D)(out,cos,sin)
+ ref = qkv.reshape(M,3,H,D).clone()
+ x0, x1 = ref[:,:2,:,:D//2].clone(),ref[:,:2,:,D//2:].clone()
+ cs,sn = [v[torch.arange(M)%L,None,None,:] for v in (cos,sin)]
+ ref[:,:2,:,:D//2],ref[:,:2,:,D//2:] = x0*cs-x1*sn,x1*cs+x0*sn
+ check("rope/cpu-reference",out,ref.reshape_as(out),2e-6)
+ assert torch.equal(out[:,2*H*D:],qkv[:,2*H*D:])
+ if dtype != torch.float32:
+ other = gpu(dtype,K.rope_kernel,(H,D),[qkv.clone(),cos,sin],(0,))[0]
+ check("rope/cpu-gpu",out,other,0.05)
+
+
[email protected]("window,dynamic", [(0,False),(0,True),(17,False),(17,True)])
+def test_attention(dtype, window, dynamic):
+ B,L,H,D = 3, 80, 2, 32
+ qkv = rand(B,L,3,H,D,dtype=dtype)
+ lens = torch.tensor([L,39,0],dtype=torch.int32)
+ out = torch.empty(B,L,H*D)
+ args = (None if dynamic else B,None if dynamic else L,H,D)
+ K.compile_cpu(K.attn_kernel,*args,window=window)(qkv,lens,out)
+ q,k,v = [qkv[:,:,i].transpose(1,2) for i in range(3)]
+ idx = torch.arange(L)
+ mask = (idx[None,:]<lens[:,None])[:,None,None,:].expand(B,1,L,L)
+ if window:
+ mask = mask & ((idx[:,None]-idx[None,:]).abs()<=window)[None,None]
+ ref = torch.nn.functional.scaled_dot_product_attention(q,k,v,attn_mask=mask).transpose(1,2).reshape_as(out)
+ valid = idx[None,:]<lens[:,None]
+ assert torch.isfinite(out).all()
+ assert torch.count_nonzero(out[2]) == 0
+ check("attn/cpu-reference",out[valid],ref[valid],2e-5)
+ if dtype != torch.float32:
+ other = gpu(dtype,K.attn_kernel,args,[qkv,lens,torch.empty_like(out)],(0,2),window=window)[-1]
+ assert torch.isfinite(other).all()
+ check("attn/cpu-gpu",out[valid],other[valid],0.02)
+
+
+# These are the first failures of the original GPU specialization, not a claim
+# that the math is impossible on CPU. Fail on changed diagnostics after upgrades.
+PROBES = [
+ (K.gemm_kernel,(128,64),"The layout for fragment C_l can not be inferred correctly."),
+ (K.gemm_geglu_kernel,(64,64),"CPU fill only supports local and global buffers, but got dst scope `local.fragment`."),
+ (K.add_ln_kernel,(128,),"CPU reduce only supports local src and local/local.var dst buffers, got src scope `local.fragment` and dst scope `local.fragment`."),
+ (K.rope_kernel,(2,64),"Cannot convert type bfloat16 to C type"),
+ (K.attn_kernel,(1,64,2,64),"The layout for fragment s_c can not be inferred correctly."),
+]
+
+
[email protected]("factory,args,message",PROBES,ids=[p[0].func.__name__ for p in PROBES])
[email protected]("dtype",K.DTYPES)
+def test_original_cpu_failure(factory,args,message,dtype,capfd):
+ if tilelang.__version__ != "0.1.14":
+ pytest.skip("diagnostics pinned to TileLang 0.1.14; re-probe on upgrade")
+ if factory is K.rope_kernel and dtype == "float16":
+ with pytest.raises(RuntimeError,match="Compilation Failed!"):
+ tilelang.compile(factory.get_tir(*args,dtype=dtype),target="c",pass_configs=K.FAST)
+ captured = capfd.readouterr()
+ # Pin the type mismatch, not the full compiler sentence: the wording differs
+ # between runs of the same compiler ("no matching function for call to
+ # 'vec_type<float, 4>::vec_type(half4&)'" vs "no known conversion for argument 1
+ # from 'vec_type<float, 4>' to 'vec_type<half_float::half, 4>&&'").
+ assert "vec_type<float, 4>" in captured.err
+ assert ("no matching function for call to" in captured.err
+ or "no known conversion for argument 1" in captured.err)
+ print("rope_kernel float16: vec_type<float, 4> / half4 conversion is unavailable")
+ else:
+ with pytest.raises(tilelang.tvm.error.InternalError) as exc:
+ tilelang.compile(factory.get_tir(*args,dtype=dtype),target="c",pass_configs=K.FAST)
+ assert message in str(exc.value)
+ print(factory.func.__name__ + " " + dtype + ": " + message)
+
+
[email protected]("factory,args,message",PROBES,ids=[p[0].func.__name__ for p in PROBES])
+def test_bf16_after_removing_fragments(factory,args,message):
+ if tilelang.__version__ != "0.1.14":
+ pytest.skip("diagnostics pinned to TileLang 0.1.14; re-probe on upgrade")
+ with pytest.raises(tilelang.tvm.error.InternalError) as exc:
+ tilelang.compile(factory.get_tir(*args,cpu=True),target="c",
+ pass_configs={tilelang.PassConfigKey.TIR_DISABLE_VECTORIZE: True})
+ assert "Cannot convert type bfloat16 to C type" in str(exc.value)
+
+
+def test_gpu_still_rejects_fp32():
+ with pytest.raises(ValueError):
+ K.gemm_kernel.get_tir(128,64,dtype="float32")
+
+
[email protected]("operation",["fill","sum","max"])
+def test_fragment_operations_without_bf16(operation):
+ # Isolate GEGLU's fill and LN/attention's reductions from GEMM layout and dtype.
+ if tilelang.__version__ != "0.1.14":
+ pytest.skip("diagnostics pinned to TileLang 0.1.14; re-probe on upgrade")
+ import tilelang.language as T
+
+ @T.prim_func
+ def probe(X: T.Tensor((4,128),"float32"),Y: T.Tensor((4,),"float32")):
+ with T.Kernel(1,threads=32):
+ x = T.alloc_fragment((4,128),"float32")
+ y = T.alloc_fragment((4,),"float32")
+ for i,j in T.Parallel(4,128):
+ x[i,j] = X[i,j]
+ if operation == "fill":
+ T.fill(y,0)
+ elif operation == "sum":
+ T.reduce_sum(x,y,dim=1)
+ else:
+ T.reduce_max(x,y,dim=1)
+ for i in T.Parallel(4):
+ Y[i] = y[i]
+
+ message = ("CPU fill only supports local and global buffers, but got dst scope `local.fragment`."
+ if operation == "fill" else
+ "CPU reduce only supports local src and local/local.var dst buffers, got src scope `local.fragment` and dst scope `local.fragment`.")
+ with pytest.raises(tilelang.tvm.error.InternalError) as exc:
+ tilelang.compile(probe,target="c")
+ assert message in str(exc.value)
+ print(operation + ": " + message)
diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.py
index a8aa4957..fcbf6f24 100644
--- a/tests/test_hooks_api.py
+++ b/tests/test_hooks_api.py
@@ -905,6 +905,22 @@ for param in ("checkpoint_id", "embedder_id"):
check_param("evaluate_shortlist", evaluate_shortlist, param, inspect.Parameter.empty)
+# Pin the optional TileLang entry points without importing the fast extra in CI.
+import ast # noqa: E402
+
+with open(os.path.join(os.path.dirname(os.path.dirname(__file__)), "laya", "tl_kernels.py")) as f:
+ _tl_defs = {node.name: node for node in ast.parse(f.read()).body if isinstance(node, ast.FunctionDef)}
+for _name in ("gemm_kernel", "gemm_geglu_kernel", "add_ln_kernel", "rope_kernel", "attn_kernel"):
+ _args = _tl_defs[_name].args
+ check("%s/cpu keyword-only" % _name, [arg.arg for arg in _args.kwonlyargs], ["cpu"])
+ check("%s/cpu default" % _name, ast.literal_eval(_args.kw_defaults[0]), False)
+ check("%s/GPU dtype default" % _name, ast.literal_eval(_args.defaults[-1]), "bfloat16")
+_args = _tl_defs["compile_cpu"].args
+check("compile_cpu/arguments", [arg.arg for arg in _args.args], ["kernel"])
+check("compile_cpu/varargs", _args.vararg.arg, "args")
+check("compile_cpu/kwargs", _args.kwarg.arg, "kwargs")
+
+
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
print(" FAIL", f)
diff --git a/tests/test_packaging.py b/tests/test_packaging.py
index bd7097b9..48fff76c 100644
--- a/tests/test_packaging.py
+++ b/tests/test_packaging.py
@@ -206,6 +206,7 @@ EXEMPT_TEST_SUITES = {
"test_mcp_local_e2e.py": "Requires local checkpoints under ~/laya_models (AGENTS.md)",
"test_onnx.py": "Requires onnx extra; skip-guarded on lane without it (AGENTS.md)",
"test_fast.py": "Requires CUDA and tilelang extra (AGENTS.md)",
+ "test_fast_cpu.py": "Requires tilelang extra and a C++ compiler; optional CUDA parity",
"test_server_example.py": "Requires cached or downloaded weights for examples/server.py",
}