Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(fast): add fp32 TileLang CPU lowering
cklxxb7abae7cb9c2
diff --git a/docs/fast-backends.md b/docs/fast-backends.mdnew file mode 100644index 00000000..3588cd2a--- /dev/null+++ b/docs/fast-backends.md@@ -0,0 +1,171 @@+# Fast backends++## TileLang portability++All five kernels in `laya/tl_kernels.py` can execute on TileLang's CPU (`c`)+target through `compile_cpu`. This is an explicit, scalar fp32 specialization,+not a CPU acceleration backend for `Agent.accelerate`. It requires TileLang and+a local C++ compiler. It introduces no runtime service dependency.++```python+import torch+from laya import tl_kernels as K++A = torch.randn(17, 67)+W = torch.randn(70, 67)+b = torch.randn(70)+C = torch.empty(17, 70)+kernel = K.compile_cpu(K.gemm_kernel, 70, 67, bias=True, act="gelu")+kernel(A, W, b, C)+```++`compile_cpu(factory, *args, **kwargs)` takes the same shape and operation options+as the five GPU factories. It selects `cpu=True`, `dtype="float32"`, target `c`,+and disables vectorization. All floating inputs and outputs must be contiguous+CPU float32 tensors; attention lengths remain int32. Convert 16-bit activations+with `.cpu().float().contiguous()` before calling it. LayerNorm still updates its+residual tensor in place, and RoPE still updates the Q/K columns in place.+Retain the compiled kernel to reuse its dynamic dimensions.++CPU specialization replaces fragment and shared allocations with local buffers,+uses serial `T.grid` loops, and omits attention's GPU swizzle annotation. GEMMs+use TileLang's scalar CPU implementation and reductions use local buffers.+The original tiled algorithms, padding predicates, attention masks, and online+softmax remain shared with the GPU implementation. CPU intermediates stay fp32,+including attention probabilities; GPU intermediates retain their original dtype.+No CPU speedup or full-model CPU backend is claimed.++### Re-probe on TileLang 0.1.14++Observed on Linux, Python 3.12.13, torch 2.11.0+cu130, TileLang 0.1.14,+and an RTX 4070 Ti SUPER. The earlier portability concern applies to compiling+the **unchanged GPU specialization**, not to the possibility of a CPU lowering.+At base commit `fa9a2a7`, this documentation page was absent from the checkout.++The test compiles `factory.get_tir(...)` with `target="c"` and the GPU's `FAST`+pass configuration. These are the exact diagnostic texts (source locations and+stack traces omitted). Both bf16 and fp16 were probed for every kernel.++| Kernel and probe dimensions | First bf16 failure | First fp16 failure |+| --- | --- | --- |+| `gemm_kernel(128, 64)` | `Check failed: layout_map.count(buffer) != 0 (0 vs. 0) : The layout for fragment C_l can not be inferred correctly.` | Same |+| `gemm_geglu_kernel(64, 64)` | ``CPU fill only supports local and global buffers, but got dst scope `local.fragment`.`` | Same |+| `add_ln_kernel(128)` | ``CPU reduce only supports local src and local/local.var dst buffers, got src scope `local.fragment` and dst scope `local.fragment`.`` | Same |+| `rope_kernel(2, 64)` | `Cannot convert type bfloat16 to C type` | C++ compiler: `error: no matching function for call to ‘vec_type<float, 4>::vec_type(half4&)’` |+| `attn_kernel(1, 64, 2, 64)` | `Check failed: layout_map.count(buffer) != 0 (0 vs. 0) : The layout for fragment s_c can not be inferred correctly.` | Same |++The first four fragment/reduction failures are `tvm.error.InternalError`.+The bf16 codegen failure is also an `InternalError`. FP16 RoPE raises+`RuntimeError: Compilation Failed!` followed by the compiler invocation and+source; the diagnostic above is emitted on stderr. Its generated vector casts+also fail converting float vectors back to half vectors.++To isolate dtype support from fragment support, the tests compile **each** kernel+again with `cpu=True` (local buffers and serial loops), retaining bf16 and disabling+vectorization. All five then fail with exactly:++```text+Cannot convert type bfloat16 to C type+```++Thus fragments are the first blocker for GEMM, GEGLU and attention, fragment+reductions for LayerNorm, and bf16 is independently a blocker for all five.+RoPE has no fragment allocation or reduction; GEMM and GEGLU have no explicit+`T.reduce_*` operation. Attention's reductions are initially masked by its layout+failure. Separate fp32-only fragment probes reproduce the fill error and the+reduction error for both `reduce_sum` and `reduce_max`, without any GEMM or bf16.+Replacing scopes alone also produced this semantic error for the GEMM+buffer (the other affected buffers were `Ci`, `x`, and `s`):++```text+[Tilelang Semantic Check] Local buffer `C_l` is indexed by T.Parallel loop variable `i`. Local buffers are thread-private and do not participate in parallel layout inference. Use T.serial/T.vectorized/T.unroll for per-thread local indexing, or T.alloc_fragment when the indexed dimension should be distributed across threads.+```++The serial fp32 specialization removes these blockers. Diagnostic assertions are+version-pinned to 0.1.14 and skip on another version, where the errors should be+re-probed. Numerical CPU tests continue to run on other versions.++### Numerical checks++Run `python -m pytest tests/test_fast_cpu.py -q -s`. On the environment above:+**58 passed**. CPU-only checks do not require CUDA; only the GPU comparisons skip+without it. Coverage includes uneven GEMM M/N/K tiles, all GEMM epilogues, GEGLU,+all residual/bias combinations, large residual values, RoPE position wraparound+and untouched V columns, static/dynamic attention shapes, sliding windows,+partial tiles, unequal lengths, empty sequences, and finite padding outputs.++The seed is 1234. GPU comparisons use identical input values rounded to bf16 or+fp16, then promoted to fp32 for CPU execution. These are absolute tolerances for+the bounded fixtures, not a guarantee for arbitrary magnitudes or model depth.+Attention comparisons use valid query rows, as in `tests/test_fast.py`.++| Kernel | Max CPU vs fp32 reference | Max CPU vs GPU (both dtypes) | CPU/GPU tolerance |+| --- | ---: | ---: | ---: |+| GEMM | 2.38419e-7 | 0.00770831 | 0.05 |+| GEGLU | 2.98023e-8 | 0.000208303 | 0.05 |+| LayerNorm | 1.07288e-6 | 0.0156183 | 0.05 |+| RoPE | 0 | 0.0130053 | 0.05 |+| Attention | 5.96046e-7 | 0.00377572 | 0.02 |++CPU/reference tolerance is 2e-5 (2e-6 for RoPE). Residual stream updates are+exactly equal, including the no-residual case.++### GPU preservation evidence++The `cpu` keyword defaults to false. Existing bf16/fp16 defaults, GPU allocation+scopes, parallel loops, swizzles and `FAST` options are unchanged. Default fp32+GPU calls still raise `ValueError`.++Before/after runs used the original module extracted with+`git show fa9a2a7:laya/tl_kernels.py` and the changed module, respectively.+The original was loaded as `laya.tl_kernels` via `importlib` for baseline runs;+all other Laya code and the Python environment stayed the same.++- `python -m pytest tests/test_fast.py -q`, with `LAYA_TEST_MODEL` set:+ **13 passed before; 13 passed after**. Initially without that variable it was+ 11 passed / 2 skipped. Both full runs emitted the checkpoint's existing+ temperature-clamping warning (`choice:11+=0.10058280825614929 -> 0.5`).+- Seeded capture of the existing kernel tests: **24 output tensors bit-identical**,+ including residual updates; maximum before/after difference **0**.+- Generated CUDA source for all five probe shapes above, in both dtypes:+ **10/10 byte-identical**. This also covers RoPE, absent from the original fast suite.+- `python benchmarks/parity_fast.py --model "$MODEL" --dtype bf16 --json ...`+ and the equivalent fp16 command: **288 questions over 60 states per dtype**.+ All before/after JSON records (fp32, stock, and fast probabilities) compare+ exactly equal; maximum before/after probability difference **0**.++`MODEL` was the cached `convaiinnovations/laya` English snapshot+`55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`; runs used `HF_HUB_OFFLINE=1`.+No multilingual or typed-decisions checkpoint comparison is claimed here.++| dtype | type | n | Max fast-stock, before = after | Fast/stock argmax agreement, before = after |+| --- | --- | ---: | ---: | ---: |+| bf16 | choice | 48 | 0.0310 | 47/48 |+| bf16 | noul | 180 | 0.0756 | 180/180 |+| bf16 | score | 60 | 0.0152 | 60/60 |+| fp16 | choice | 48 | 0.0069 | 48/48 |+| fp16 | noul | 180 | 0.0092 | 180/180 |+| fp16 | score | 60 | 0.0040 | 60/60 |+++### Repository checks++All commands used `/home/ckl/projects/S/laya/.venv/bin/python`; `ruff` and+`zensical` came from that virtualenv's `bin` directory.++| Command | Result |+| --- | --- |+| `ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120` | `All checks passed!` |+| `python -m compileall -q laya/ tests/` | Exit 0, no output |+| `python tests/test_router.py` | 703 passed, 0 failed |+| `python tests/test_criteria.py` | 198 passed, 0 failed |+| `python tests/test_hooks.py` | 240 passed, 0 failed |+| `python tests/test_hooks_api.py` | 415 passed, 0 failed |+| `python tests/test_packaging.py` | 131 passed, 0 failed |+| `uv pip install --python /home/ckl/projects/S/laya/.venv/bin/python -r requirements-docs.txt` | Checked 3 packages (already installed); virtualenv has no pip |+| `zensical build --strict --clean` | `No issues found`; no `griffe:` lines |++The new suite is registered as requiring the optional TileLang extra and a C+++compiler in the packaging test's existing exemptions. The API contract test pins+the additive keyword-only CPU selector, unchanged GPU dtype default, and+`compile_cpu` signature without importing TileLang into the base CI environment.diff --git a/laya/tl_kernels.py b/laya/tl_kernels.pyindex b5c9ee61..5121fbf8 100644--- a/laya/tl_kernels.py+++ b/laya/tl_kernels.py@@ -1,8 +1,9 @@"""TileLang kernels for the Laya (ModernBERT + decision head) encoder.-All kernels take 16-bit activations (bf16 by default, fp16 with dtype="float16"), accumulate in fp32. Row count M is a runtime+GPU kernels take 16-bit activations (bf16 by default, fp16 with dtype="float16"), accumulate in fp32. Row count M is a runtimesymbol so one compiled kernel serves every batch/sequence bucket; M must be amultiple of 16 (the caller pads); out-of-bounds rows are predicated by TileLang.+Use compile_cpu for an explicit fp32 CPU specialization; the GPU defaults are unchanged."""import tilelangimport tilelang.language as T@@ -12,7 +13,9 @@ DTYPES = ("bfloat16", "float16")FAST = {tilelang.PassConfigKey.TL_ENABLE_FAST_MATH: True}-def _dt(dtype):+def _dt(dtype, cpu=False):+ if cpu and dtype == "float32":+ return dtypeif dtype not in DTYPES:raise ValueError("tl_kernels dtype must be one of %s, got %r" % (DTYPES, dtype))return dtype@@ -28,17 +31,20 @@ def _act(x, kind):# ----------------------------------------------------------------------------- GEMM@tilelang.jit(pass_configs=FAST)-def gemm_kernel(N, K, bias=False, act="none", bm=64, bn=128, bk=64, stages=3, threads=128, dtype="bfloat16"):+def gemm_kernel(N, K, bias=False, act="none", bm=64, bn=128, bk=64, stages=3, threads=128, dtype="bfloat16", *, cpu=False):"""C[M,N] = act(A[M,K] @ W[N,K]^T + b)."""- DT = _dt(dtype)+ DT = _dt(dtype, cpu)+ alloc = T.alloc_local if cpu else T.alloc_fragment+ shared = T.alloc_local if cpu else T.alloc_shared+ loop = T.grid if cpu else T.ParallelM = T.dynamic("M")@T.prim_funcdef main(A: T.Tensor((M, K), DT), W: T.Tensor((N, K), DT), Bv: T.Tensor((N,), ACC), C: T.Tensor((M, N), DT)):with T.Kernel(T.ceildiv(N, bn), T.ceildiv(M, bm), threads=threads) as (bx, by):- A_s = T.alloc_shared((bm, bk), DT)- W_s = T.alloc_shared((bn, bk), DT)- C_l = T.alloc_fragment((bm, bn), ACC)+ A_s = shared((bm, bk), DT)+ W_s = shared((bn, bk), DT)+ C_l = alloc((bm, bn), ACC)T.clear(C_l)for k in T.Pipelined(T.ceildiv(K, bk), num_stages=stages):T.copy(A[by * bm, k * bk], A_s)@@ -46,29 +52,32 @@ def gemm_kernel(N, K, bias=False, act="none", bm=64, bn=128, bk=64, stages=3, thT.gemm(A_s, W_s, C_l, transpose_B=True)# `bias` is a Python-level flag, so each branch is one expression (no re-bound TileLang value)if bias:- for i, j in T.Parallel(bm, bn):+ for i, j in loop(bm, bn):C_l[i, j] = _act(C_l[i, j] + Bv[bx * bn + j], act)else:- for i, j in T.Parallel(bm, bn):+ for i, j in loop(bm, bn):C_l[i, j] = _act(C_l[i, j], act)T.copy(C_l, C[by * bm, bx * bn])return main@tilelang.jit(pass_configs=FAST)-def gemm_geglu_kernel(F, K, bm=64, bn=64, bk=64, stages=3, threads=128, dtype="bfloat16"):+def gemm_geglu_kernel(F, K, bm=64, bn=64, bk=64, stages=3, threads=128, dtype="bfloat16", *, cpu=False):"""ModernBERT GLU MLP up-projection, fused: C[M,F] = gelu(A @ Wi[:F]^T) * (A @ Wi[F:]^T)."""- DT = _dt(dtype)+ DT = _dt(dtype, cpu)+ alloc = T.alloc_local if cpu else T.alloc_fragment+ shared = T.alloc_local if cpu else T.alloc_shared+ loop = T.grid if cpu else T.ParallelM = T.dynamic("M")@T.prim_funcdef main(A: T.Tensor((M, K), DT), W: T.Tensor((2 * F, K), DT), C: T.Tensor((M, F), DT)):with T.Kernel(T.ceildiv(F, bn), T.ceildiv(M, bm), threads=threads) as (bx, by):- A_s = T.alloc_shared((bm, bk), DT)- Wi_s = T.alloc_shared((bn, bk), DT)- Wg_s = T.alloc_shared((bn, bk), DT)- Ci = T.alloc_fragment((bm, bn), ACC)- Cg = T.alloc_fragment((bm, bn), ACC)+ A_s = shared((bm, bk), DT)+ Wi_s = shared((bn, bk), DT)+ Wg_s = shared((bn, bk), DT)+ Ci = alloc((bm, bn), ACC)+ Cg = alloc((bm, bn), ACC)T.clear(Ci); T.clear(Cg)for k in T.Pipelined(T.ceildiv(K, bk), num_stages=stages):T.copy(A[by * bm, k * bk], A_s)@@ -76,7 +85,7 @@ def gemm_geglu_kernel(F, K, bm=64, bn=64, bk=64, stages=3, threads=128, dtype="bT.copy(W[F + bx * bn, k * bk], Wg_s)T.gemm(A_s, Wi_s, Ci, transpose_B=True)T.gemm(A_s, Wg_s, Cg, transpose_B=True)- for i, j in T.Parallel(bm, bn):+ for i, j in loop(bm, bn):Ci[i, j] = _act(Ci[i, j], "gelu") * Cg[i, j]T.copy(Ci, C[by * bm, bx * bn])return main@@ -84,47 +93,50 @@ def gemm_geglu_kernel(F, K, bm=64, bn=64, bk=64, stages=3, threads=128, dtype="b# ----------------------------------------------------------------------------- LayerNorm (+residual)@tilelang.jit(pass_configs=FAST)-def add_ln_kernel(D, residual=True, bias=False, eps=1e-5, bm=4, threads=32, dtype="bfloat16"):+def add_ln_kernel(D, residual=True, bias=False, eps=1e-5, bm=4, threads=32, dtype="bfloat16", *, cpu=False):"""X (fp32 residual stream) += R (16-bit branch output, if residual); Y (16-bit) = LN(X) * w (+ b).The residual stream stays in fp32 exactly like the stock autocast path: ModernBERT-large's residualactivations reach ~3e4, where bf16's 8-bit mantissa would lose ~100 units per add and drift layer by layer."""- DT = _dt(dtype)+ DT = _dt(dtype, cpu)+ alloc = T.alloc_local if cpu else T.alloc_fragment+ shared = T.alloc_local if cpu else T.alloc_shared+ loop = T.grid if cpu else T.ParallelM = T.dynamic("M")@T.prim_funcdef main(X: T.Tensor((M, D), ACC), R: T.Tensor((M, D), DT), Wv: T.Tensor((D,), ACC), Bv: T.Tensor((D,), ACC),Y: T.Tensor((M, D), DT)):with T.Kernel(T.ceildiv(M, bm), threads=threads) as bx:- x = T.alloc_fragment((bm, D), ACC)- xs = T.alloc_fragment((bm, D), ACC)- mean = T.alloc_fragment((bm,), ACC)- var = T.alloc_fragment((bm,), ACC)- Xb = T.alloc_shared((bm, D), ACC)- Rb = T.alloc_shared((bm, D), DT)- Yb = T.alloc_shared((bm, D), DT)+ x = alloc((bm, D), ACC)+ xs = alloc((bm, D), ACC)+ mean = alloc((bm,), ACC)+ var = alloc((bm,), ACC)+ Xb = shared((bm, D), ACC)+ Rb = shared((bm, D), DT)+ Yb = shared((bm, D), DT)T.copy(X[bx * bm, 0], Xb)T.copy(Xb, x)if residual:T.copy(R[bx * bm, 0], Rb)T.copy(Rb, xs)- for i, j in T.Parallel(bm, D):+ for i, j in loop(bm, D):x[i, j] = x[i, j] + xs[i, j]T.copy(x, Xb)T.copy(Xb, X[bx * bm, 0])T.reduce_sum(x, mean, dim=1)- for i in T.Parallel(bm):+ for i in loop(bm):mean[i] = mean[i] / D- for i, j in T.Parallel(bm, D):+ for i, j in loop(bm, D):xs[i, j] = (x[i, j] - mean[i]) * (x[i, j] - mean[i])T.reduce_sum(xs, var, dim=1)- for i in T.Parallel(bm):+ for i in loop(bm):var[i] = T.rsqrt(var[i] / D + eps)if bias:- for i, j in T.Parallel(bm, D):+ for i, j in loop(bm, D):xs[i, j] = (x[i, j] - mean[i]) * var[i] * Wv[j] + Bv[j]else:- for i, j in T.Parallel(bm, D):+ for i, j in loop(bm, D):xs[i, j] = (x[i, j] - mean[i]) * var[i] * Wv[j]T.copy(xs, Yb)T.copy(Yb, Y[bx * bm, 0])@@ -133,10 +145,11 @@ def add_ln_kernel(D, residual=True, bias=False, eps=1e-5, bm=4, threads=32, dtyp# ----------------------------------------------------------------------------- RoPE (in place on packed qkv)@tilelang.jit(pass_configs=FAST)-def rope_kernel(H, Dh, bm=32, threads=128, dtype="bfloat16"):+def rope_kernel(H, Dh, bm=32, threads=128, dtype="bfloat16", *, cpu=False):"""QKV[M, 3*H*Dh] packed as (q|k|v)(h)(d). Rotates q and k in place (rotate-half convention, fp32 math).cos/sin: [L, Dh/2]. Row r has position r % L. M and L are runtime symbols."""- DT = _dt(dtype)+ DT = _dt(dtype, cpu)+ loop = T.grid if cpu else T.ParallelM, L = T.dynamic("M"), T.dynamic("L")half = Dh // 2W = 2 * H * Dh # q and k columns@@ -144,7 +157,7 @@ def rope_kernel(H, Dh, bm=32, threads=128, dtype="bfloat16"):@T.prim_funcdef main(QKV: T.Tensor((M, 3 * H * Dh), DT), Cos: T.Tensor((L, half), ACC), Sin: T.Tensor((L, half), ACC)):with T.Kernel(T.ceildiv(M, bm), threads=threads) as bx:- for i, c in T.Parallel(bm, W // 2):+ for i, c in loop(bm, W // 2):r = bx * bm + ipos = r % Lhh = c // half # which (q|k, head)@@ -162,14 +175,17 @@ def rope_kernel(H, Dh, bm=32, threads=128, dtype="bfloat16"):# ----------------------------------------------------------------------------- flash attention (padding mask + sliding window)@tilelang.jit(pass_configs=FAST)-def attn_kernel(B, L, H, Dh, window=0, bm=64, bn=64, stages=1, threads=128, dtype="bfloat16"):+def attn_kernel(B, L, H, Dh, window=0, bm=64, bn=64, stages=1, threads=128, dtype="bfloat16", *, cpu=False):"""QKV: [B, L, 3, H, Dh] 16-bit (a view of the packed [M, 3*H*Dh] buffer). Lens: [B] int32 valid length.O: [B, L, H*Dh]. window>0 => bidirectional sliding window |i-j| <= window. Masked scores use a largefinite negative so fully-masked (padding) rows stay finite.B and/or L may be None: they then become runtime symbols (one compile serves every shape, at thecost of predicated loads -- ~4x slower for full attention at L=1024, free for short inputs)."""- DT = _dt(dtype)+ DT = _dt(dtype, cpu)+ alloc = T.alloc_local if cpu else T.alloc_fragment+ shared = T.alloc_local if cpu else T.alloc_shared+ loop = T.grid if cpu else T.Parallelscale = (1.0 / Dh) ** 0.5 * 1.44269504 # log2(e)if B is None:B = T.dynamic("B")@@ -180,19 +196,20 @@ def attn_kernel(B, L, H, Dh, window=0, bm=64, bn=64, stages=1, threads=128, dtyp@T.prim_funcdef main(QKV: T.Tensor((B, L, 3, H, Dh), DT), Lens: T.Tensor((B,), "int32"), O: T.Tensor((B, L, H * Dh), DT)):with T.Kernel(T.ceildiv(L, bm), H, B, threads=threads) as (bx, by, bz):- Q_s = T.alloc_shared((bm, Dh), DT)- K_s = T.alloc_shared((bn, Dh), DT)- V_s = T.alloc_shared((bn, Dh), DT)- O_s = T.alloc_shared((bm, Dh), DT)- s = T.alloc_fragment((bm, bn), ACC)- s_c = T.alloc_fragment((bm, bn), DT)- o = T.alloc_fragment((bm, Dh), ACC)- m = T.alloc_fragment((bm,), ACC)- m_prev = T.alloc_fragment((bm,), ACC)- sc = T.alloc_fragment((bm,), ACC)- rs = T.alloc_fragment((bm,), ACC)- l = T.alloc_fragment((bm,), ACC)- T.annotate_layout({Q_s: tilelang.layout.make_swizzled_layout(Q_s)})+ Q_s = shared((bm, Dh), DT)+ K_s = shared((bn, Dh), DT)+ V_s = shared((bn, Dh), DT)+ O_s = shared((bm, Dh), DT)+ s = alloc((bm, bn), ACC)+ s_c = alloc((bm, bn), DT)+ o = alloc((bm, Dh), ACC)+ m = alloc((bm,), ACC)+ m_prev = alloc((bm,), ACC)+ sc = alloc((bm,), ACC)+ rs = alloc((bm,), ACC)+ l = alloc((bm,), ACC)+ if not cpu:+ T.annotate_layout({Q_s: tilelang.layout.make_swizzled_layout(Q_s)})T.copy(QKV[bz, bx * bm:(bx + 1) * bm, 0, by, :], Q_s)T.fill(o, 0); T.fill(l, 0); T.fill(m, NEG)n = Lens[bz]@@ -204,7 +221,7 @@ def attn_kernel(B, L, H, Dh, window=0, bm=64, bn=64, stages=1, threads=128, dtypk_hi = T.ceildiv(n, bn)for k in T.Pipelined(k_lo, k_hi, num_stages=stages):T.copy(QKV[bz, k * bn:(k + 1) * bn, 1, by, :], K_s)- for i, j in T.Parallel(bm, bn):+ for i, j in loop(bm, bn):qi = bx * bm + ikj = k * bn + jif window > 0:@@ -216,19 +233,32 @@ def attn_kernel(B, L, H, Dh, window=0, bm=64, bn=64, stages=1, threads=128, dtypT.copy(QKV[bz, k * bn:(k + 1) * bn, 2, by, :], V_s)T.copy(m, m_prev)T.reduce_max(s, m, dim=1, clear=False)- for i in T.Parallel(bm):+ for i in loop(bm):sc[i] = T.exp2(m_prev[i] * scale - m[i] * scale)- for i, j in T.Parallel(bm, bn):+ for i, j in loop(bm, bn):s[i, j] = T.exp2(s[i, j] * scale - m[i] * scale)T.reduce_sum(s, rs, dim=1)- for i in T.Parallel(bm):+ for i in loop(bm):l[i] = l[i] * sc[i] + rs[i]T.copy(s, s_c)- for i, j in T.Parallel(bm, Dh):+ for i, j in loop(bm, Dh):o[i, j] = o[i, j] * sc[i]T.gemm(s_c, V_s, o, policy=T.GemmWarpPolicy.FullRow)- for i, j in T.Parallel(bm, Dh):+ for i, j in loop(bm, Dh):o[i, j] = o[i, j] / T.max(l[i], 1e-30)T.copy(o, O_s)T.copy(O_s, O[bz, bx * bm:(bx + 1) * bm, by * Dh:(by + 1) * Dh])return main+++def compile_cpu(kernel, *args, **kwargs):+ """Compile a kernel factory for CPU with fp32 tensors and local scratch buffers.++ This is a scalar correctness path, not an Agent.accelerate backend. Convert all+ floating inputs to contiguous CPU float32 tensors; integer lengths stay int32.+ Output buffers (including the in-place residual and QKV) must also be on CPU.+ """+ kwargs["dtype"] = "float32"+ kwargs["cpu"] = True+ return tilelang.compile(kernel.get_tir(*args, **kwargs), target="c",+ pass_configs={tilelang.PassConfigKey.TIR_DISABLE_VECTORIZE: True})diff --git a/tests/test_fast_cpu.py b/tests/test_fast_cpu.pynew file mode 100644index 00000000..7ca60833--- /dev/null+++ b/tests/test_fast_cpu.py@@ -0,0 +1,205 @@+"""TileLang C target: original limitations, fp32 CPU execution, and CUDA parity.++Run: python -m pytest tests/test_fast_cpu.py -q -s+CPU checks need torch, tilelang and a C++ compiler; CUDA parity skips without CUDA.+"""+import os, sys+import pytest+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+torch = pytest.importorskip("torch")+tilelang = pytest.importorskip("tilelang")+from laya import tl_kernels as K # noqa: E402++[email protected](params=[torch.float32, torch.bfloat16, torch.float16], ids=["cpu", "bf16", "fp16"])+def dtype(request):+ if request.param != torch.float32 and not torch.cuda.is_available():+ pytest.skip("GPU parity needs CUDA")+ torch.manual_seed(1234)+ return request.param+++def rand(*shape, dtype=torch.float32, scale=1):+ return (torch.randn(*shape) * scale).to(dtype).float()+++def check(name, actual, expected, tol):+ delta = (actual.float() - expected.float()).abs().max().item()+ print("%s max_abs=%.9g tolerance=%g" % (name, delta, tol))+ assert torch.isfinite(actual).all()+ assert delta <= tol+++def gpu(dtype, factory, args, tensors, floating, **kwargs):+ dt = {torch.bfloat16: "bfloat16", torch.float16: "float16"}[dtype]+ values = [x.cuda().to(dtype) if i in floating else x.cuda() for i, x in enumerate(tensors)]+ factory(*args, dtype=dt, **kwargs)(*values)+ return [x.cpu() for x in values]++[email protected]("bias,act", [(False, "none"), (True, "gelu"), (True, "relu")])+def test_gemm(dtype, bias, act):+ M, N, D = 17, 70, 67+ a, w, b = rand(M, D, dtype=dtype), rand(N, D, dtype=dtype, scale=0.02), rand(N)+ out = torch.empty(M, N)+ opts = dict(bias=bias, act=act)+ K.compile_cpu(K.gemm_kernel, N, D, **opts)(a, w, b, out)+ ref = a @ w.T + (b if bias else 0)+ ref = {"none": ref, "gelu": torch.nn.functional.gelu(ref), "relu": torch.relu(ref)}[act]+ check("gemm/cpu-reference", out, ref, 2e-5)+ if dtype != torch.float32:+ other = gpu(dtype, K.gemm_kernel, (N,D), [a,w,b,torch.empty_like(out)], (0,1,3), **opts)[-1]+ check("gemm/cpu-gpu", out, other, 0.05)+++def test_geglu(dtype):+ M, F, D = 17, 70, 67+ a, w = rand(M,D,dtype=dtype), rand(2*F,D,dtype=dtype,scale=0.02)+ out = torch.empty(M,F)+ K.compile_cpu(K.gemm_geglu_kernel,F,D)(a,w,out)+ x = a @ w.T+ check("geglu/cpu-reference", out, torch.nn.functional.gelu(x[:,:F]) * x[:,F:], 2e-5)+ if dtype != torch.float32:+ other = gpu(dtype,K.gemm_geglu_kernel,(F,D),[a,w,torch.empty_like(out)],(0,1,2))[-1]+ check("geglu/cpu-gpu", out, other, 0.05)++[email protected]("residual,bias", [(False,False),(False,True),(True,False),(True,True)])+def test_layernorm(dtype, residual, bias):+ M, D = 17, 128+ x, r = rand(M,D,scale=3000), rand(M,D,dtype=dtype,scale=50)+ w, b = torch.rand(D)+0.5, rand(D)+ updated, out = x.clone(), torch.empty_like(x)+ opts = dict(residual=residual,bias=bias)+ K.compile_cpu(K.add_ln_kernel,D,**opts)(updated,r,w,b,out)+ refx = x + r if residual else x+ assert torch.equal(updated,refx)+ check("ln/cpu-reference",out,torch.nn.functional.layer_norm(refx,(D,),w,b if bias else None,1e-5),2e-5)+ if dtype != torch.float32:+ other = gpu(dtype,K.add_ln_kernel,(D,),[x.clone(),r,w,b,torch.empty_like(out)],(1,4),**opts)+ assert torch.equal(updated,other[0])+ check("ln/cpu-gpu",out,other[-1],0.05)+++def test_rope(dtype):+ M, L, H, D = 35, 7, 2, 64+ qkv = rand(M,3*H*D,dtype=dtype)+ angles = rand(L,D//2)+ cos, sin = angles.cos(), angles.sin()+ out = qkv.clone()+ K.compile_cpu(K.rope_kernel,H,D)(out,cos,sin)+ ref = qkv.reshape(M,3,H,D).clone()+ x0, x1 = ref[:,:2,:,:D//2].clone(),ref[:,:2,:,D//2:].clone()+ cs,sn = [v[torch.arange(M)%L,None,None,:] for v in (cos,sin)]+ ref[:,:2,:,:D//2],ref[:,:2,:,D//2:] = x0*cs-x1*sn,x1*cs+x0*sn+ check("rope/cpu-reference",out,ref.reshape_as(out),2e-6)+ assert torch.equal(out[:,2*H*D:],qkv[:,2*H*D:])+ if dtype != torch.float32:+ other = gpu(dtype,K.rope_kernel,(H,D),[qkv.clone(),cos,sin],(0,))[0]+ check("rope/cpu-gpu",out,other,0.05)++[email protected]("window,dynamic", [(0,False),(0,True),(17,False),(17,True)])+def test_attention(dtype, window, dynamic):+ B,L,H,D = 3, 80, 2, 32+ qkv = rand(B,L,3,H,D,dtype=dtype)+ lens = torch.tensor([L,39,0],dtype=torch.int32)+ out = torch.empty(B,L,H*D)+ args = (None if dynamic else B,None if dynamic else L,H,D)+ K.compile_cpu(K.attn_kernel,*args,window=window)(qkv,lens,out)+ q,k,v = [qkv[:,:,i].transpose(1,2) for i in range(3)]+ idx = torch.arange(L)+ mask = (idx[None,:]<lens[:,None])[:,None,None,:].expand(B,1,L,L)+ if window:+ mask = mask & ((idx[:,None]-idx[None,:]).abs()<=window)[None,None]+ ref = torch.nn.functional.scaled_dot_product_attention(q,k,v,attn_mask=mask).transpose(1,2).reshape_as(out)+ valid = idx[None,:]<lens[:,None]+ assert torch.isfinite(out).all()+ assert torch.count_nonzero(out[2]) == 0+ check("attn/cpu-reference",out[valid],ref[valid],2e-5)+ if dtype != torch.float32:+ other = gpu(dtype,K.attn_kernel,args,[qkv,lens,torch.empty_like(out)],(0,2),window=window)[-1]+ assert torch.isfinite(other).all()+ check("attn/cpu-gpu",out[valid],other[valid],0.02)+++# These are the first failures of the original GPU specialization, not a claim+# that the math is impossible on CPU. Fail on changed diagnostics after upgrades.+PROBES = [+ (K.gemm_kernel,(128,64),"The layout for fragment C_l can not be inferred correctly."),+ (K.gemm_geglu_kernel,(64,64),"CPU fill only supports local and global buffers, but got dst scope `local.fragment`."),+ (K.add_ln_kernel,(128,),"CPU reduce only supports local src and local/local.var dst buffers, got src scope `local.fragment` and dst scope `local.fragment`."),+ (K.rope_kernel,(2,64),"Cannot convert type bfloat16 to C type"),+ (K.attn_kernel,(1,64,2,64),"The layout for fragment s_c can not be inferred correctly."),+]++[email protected]("factory,args,message",PROBES,ids=[p[0].func.__name__ for p in PROBES])[email protected]("dtype",K.DTYPES)+def test_original_cpu_failure(factory,args,message,dtype,capfd):+ if tilelang.__version__ != "0.1.14":+ pytest.skip("diagnostics pinned to TileLang 0.1.14; re-probe on upgrade")+ if factory is K.rope_kernel and dtype == "float16":+ with pytest.raises(RuntimeError,match="Compilation Failed!"):+ tilelang.compile(factory.get_tir(*args,dtype=dtype),target="c",pass_configs=K.FAST)+ captured = capfd.readouterr()+ # Pin the type mismatch, not the full compiler sentence: the wording differs+ # between runs of the same compiler ("no matching function for call to+ # 'vec_type<float, 4>::vec_type(half4&)'" vs "no known conversion for argument 1+ # from 'vec_type<float, 4>' to 'vec_type<half_float::half, 4>&&'").+ assert "vec_type<float, 4>" in captured.err+ assert ("no matching function for call to" in captured.err+ or "no known conversion for argument 1" in captured.err)+ print("rope_kernel float16: vec_type<float, 4> / half4 conversion is unavailable")+ else:+ with pytest.raises(tilelang.tvm.error.InternalError) as exc:+ tilelang.compile(factory.get_tir(*args,dtype=dtype),target="c",pass_configs=K.FAST)+ assert message in str(exc.value)+ print(factory.func.__name__ + " " + dtype + ": " + message)++[email protected]("factory,args,message",PROBES,ids=[p[0].func.__name__ for p in PROBES])+def test_bf16_after_removing_fragments(factory,args,message):+ if tilelang.__version__ != "0.1.14":+ pytest.skip("diagnostics pinned to TileLang 0.1.14; re-probe on upgrade")+ with pytest.raises(tilelang.tvm.error.InternalError) as exc:+ tilelang.compile(factory.get_tir(*args,cpu=True),target="c",+ pass_configs={tilelang.PassConfigKey.TIR_DISABLE_VECTORIZE: True})+ assert "Cannot convert type bfloat16 to C type" in str(exc.value)+++def test_gpu_still_rejects_fp32():+ with pytest.raises(ValueError):+ K.gemm_kernel.get_tir(128,64,dtype="float32")++[email protected]("operation",["fill","sum","max"])+def test_fragment_operations_without_bf16(operation):+ # Isolate GEGLU's fill and LN/attention's reductions from GEMM layout and dtype.+ if tilelang.__version__ != "0.1.14":+ pytest.skip("diagnostics pinned to TileLang 0.1.14; re-probe on upgrade")+ import tilelang.language as T++ @T.prim_func+ def probe(X: T.Tensor((4,128),"float32"),Y: T.Tensor((4,),"float32")):+ with T.Kernel(1,threads=32):+ x = T.alloc_fragment((4,128),"float32")+ y = T.alloc_fragment((4,),"float32")+ for i,j in T.Parallel(4,128):+ x[i,j] = X[i,j]+ if operation == "fill":+ T.fill(y,0)+ elif operation == "sum":+ T.reduce_sum(x,y,dim=1)+ else:+ T.reduce_max(x,y,dim=1)+ for i in T.Parallel(4):+ Y[i] = y[i]++ message = ("CPU fill only supports local and global buffers, but got dst scope `local.fragment`."+ if operation == "fill" else+ "CPU reduce only supports local src and local/local.var dst buffers, got src scope `local.fragment` and dst scope `local.fragment`.")+ with pytest.raises(tilelang.tvm.error.InternalError) as exc:+ tilelang.compile(probe,target="c")+ assert message in str(exc.value)+ print(operation + ": " + message)diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.pyindex a8aa4957..fcbf6f24 100644--- a/tests/test_hooks_api.py+++ b/tests/test_hooks_api.py@@ -905,6 +905,22 @@ for param in ("checkpoint_id", "embedder_id"):check_param("evaluate_shortlist", evaluate_shortlist, param, inspect.Parameter.empty)+# Pin the optional TileLang entry points without importing the fast extra in CI.+import ast # noqa: E402++with open(os.path.join(os.path.dirname(os.path.dirname(__file__)), "laya", "tl_kernels.py")) as f:+ _tl_defs = {node.name: node for node in ast.parse(f.read()).body if isinstance(node, ast.FunctionDef)}+for _name in ("gemm_kernel", "gemm_geglu_kernel", "add_ln_kernel", "rope_kernel", "attn_kernel"):+ _args = _tl_defs[_name].args+ check("%s/cpu keyword-only" % _name, [arg.arg for arg in _args.kwonlyargs], ["cpu"])+ check("%s/cpu default" % _name, ast.literal_eval(_args.kw_defaults[0]), False)+ check("%s/GPU dtype default" % _name, ast.literal_eval(_args.defaults[-1]), "bfloat16")+_args = _tl_defs["compile_cpu"].args+check("compile_cpu/arguments", [arg.arg for arg in _args.args], ["kernel"])+check("compile_cpu/varargs", _args.vararg.arg, "args")+check("compile_cpu/kwargs", _args.kwarg.arg, "kwargs")++print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))for f in FAIL:print(" FAIL", f)diff --git a/tests/test_packaging.py b/tests/test_packaging.pyindex bd7097b9..48fff76c 100644--- a/tests/test_packaging.py+++ b/tests/test_packaging.py@@ -206,6 +206,7 @@ EXEMPT_TEST_SUITES = {"test_mcp_local_e2e.py": "Requires local checkpoints under ~/laya_models (AGENTS.md)","test_onnx.py": "Requires onnx extra; skip-guarded on lane without it (AGENTS.md)","test_fast.py": "Requires CUDA and tilelang extra (AGENTS.md)",+ "test_fast_cpu.py": "Requires tilelang extra and a C++ compiler; optional CUDA parity","test_server_example.py": "Requires cached or downloaded weights for examples/server.py",}