Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

prefill: say why the FP16 path rotates H in place in fp16

Laurent Zuijdwijk86faf035b67d
diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp
index 5ebfc931c..0f68a3002 100644
--- a/src/prefill/prefill.cpp
+++ b/src/prefill/prefill.cpp
@@ -3724,6 +3724,7 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
} else {
gather_rows16(m.mixed_h, m.src_dev, m.Xs, T * K, N, m.cs);
// APR: Hadamard-folded gate/up take H (s * x); Xs is this layer's private gathered copy
+ // (rotated in fp16 storage, fp32 arithmetic: see the down projection's note below)
if (had_gu) strata::kernels::hadamard_rows_f16(m.Xs, T * K, N, had_block, had_sx, m.cs);
}
// multi-GPU: the peer's share, enqueued before the primary's own experts so both cards work at once
@@ -4257,6 +4258,14 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
pt.mark(kPfGemmGU, cs);
m.gemm.f16(m.Xs + o0 * N, m.dq_gu[q], m.GU + o0 * 1280, ne, 1280, N);
swiglu_interleaved(m.GU + o0 * 1280, m.Hh + o0 * 640, ne, m.cs);
+ // APR, deliberately in place on the fp16 H: hadamard_rows_f16 reads fp16, runs the
+ // butterflies in fp32 and rounds once on the store, so against an fp32 rotation the only
+ // extra error is the one fp16 rounding of the SwiGLU output before it (~2^-11 relative per
+ // value; the GEMM reads fp16 either way). An fp32 rotation would need a T*K*640 float
+ // buffer (~200 MiB at an 8k chunk) or a fused SwiGLU + rotation kernel, for the fallback
+ // of types and builds without MMQ (the MMQ path rotates fp32 H). native_expert_parity
+ // bounds the fp16 rotation against the float one at relative L1 error < 2e-3. The
+ // gate/up input (Xs above) is handled the same way.
if (had_d) strata::kernels::hadamard_rows_f16(m.Hh + o0 * 640, ne, 640, had_block, had_sh, m.cs);
pt.mark(kPfGemmD, cs);
m.gemm.f16(m.Hh + o0 * 640, m.dq_d[q], m.Dm + o0 * N, ne, N, 640);