Skip to content

Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.

QK norm + RoPE backward: pin the dX contraction so compiled fp16 matches eager on ROCm (#15506)

Daniel Handdd983813e78
diff --git a/unsloth/kernels/qk_norm_rope.py b/unsloth/kernels/qk_norm_rope.py
index 0b9dc9bc65c..60b0541ab01 100644
--- a/unsloth/kernels/qk_norm_rope.py
+++ b/unsloth/kernels/qk_norm_rope.py
@@ -188,8 +188,10 @@ def _norm_rope_bwd_block(
rowsum = tl.sum(g0 * n0, axis = 1) + tl.sum(g1 * n1, axis = 1)
scale = (inv / HEAD_DIM)[:, None]
dx_ptr = dX + (row * n_heads + hoff) * HEAD_DIM + cols[None, :]
- tl.store(dx_ptr, scale * (HEAD_DIM * g0 - n0 * rowsum[:, None]), mask = mask)
- tl.store(dx_ptr + HALF, scale * (HEAD_DIM * g1 - n1 * rowsum[:, None]), mask = mask)
+ # HEAD_DIM * g - n * rowsum with the contraction pinned (HEAD_DIM * g is exact): left to the backend
+ # the eager and the Inductor build contract it differently on ROCm, and fp16 dX lands 1 ulp apart.
+ tl.store(dx_ptr, scale * tl.fma(-n0, rowsum[:, None], HEAD_DIM * g0), mask = mask)
+ tl.store(dx_ptr + HALF, scale * tl.fma(-n1, rowsum[:, None], HEAD_DIM * g1), mask = mask)
@triton.jit