Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Trellis numerics: say which codebook each path reads and what is bitwise

Laurent Zuijdwijk390da5d905fe
diff --git a/docs/GYRO.md b/docs/GYRO.md
index 5b73b0a52..04dd89eea 100644
--- a/docs/GYRO.md
+++ b/docs/GYRO.md
@@ -173,6 +173,19 @@ checked. A real chat at 128k context worked through the web interface.
Single RTX 4090 smoke run (Gyro-S, MTP draft, greedy): code 107.5, prose 84.3, JSON 76.3 tokens/s.
+## Numerics
+
+- **Decode** (Strata's own expert kernels) reads the trellis weights through an int8 copy of the codebook with one
+ global scale (`tq_lut_i8`, relative RMS error 0.85 % of the codebook's standard deviation, far below the
+ quantization error itself), against q8_1 activations.
+- **Prompt processing** decodes them otherwise: the FP16 path through the fp16 codebook (`tq_lut_f16`, what GGML's
+ own dequantizer reads); the MMQ path through the llama.cpp fork's kernels, which read the int8 codebook as well
+ on the fork's current `main`.
+- So decode and prompt results agree within tolerance, not bit for bit. `native_expert_parity` checks one expert
+ on the decode kernels against GGML's float dequantizer and a float SwiGLU: relative L1 error below 3e-2, the
+ bound every expert type gets there. Its "bitwise equal" lines compare Strata's decode-once and per-entry decode
+ kernels with each other, not with a float reference.
+
## Known limits
- **Not measured yet:** token parity against llama.cpp.
diff --git a/src/kernels/cuda/iq_kernels.cu b/src/kernels/cuda/iq_kernels.cu
index cc3378235..91e608da1 100644
--- a/src/kernels/cuda/iq_kernels.cu
+++ b/src/kernels/cuda/iq_kernels.cu
@@ -650,12 +650,23 @@ __device__ __forceinline__ void tq_states4(const uint8_t* qs, int l, uint32_t s[
}
// lane iqs/4 of block kbx against the q8_1 blocks aligned with the block's start (4 per trellis block). Split in
// two, like the IQ types' decode-once kernels below: tq_load decodes the lane's 16 codebook values (weight side only),
-// tq_apply does the activation side. vec_dot_tq_q8_1 is apply(load()), so the per-entry and the decode-once kernels
-// do the same float operations in the same order: bitwise equal.
-// The lane's 16 codebook values as 4 ints of packed int8 (the int8 codebook tq_lut_i8, one global scale
-// TQ_LUT_I8_SCALE; relative RMS error 0.85% of the codebook std, far below the trellis error): a trellis step's
-// 4 weights are two 2-byte entries = one int, so the apply side is 4 dp4a instead of 16 int8->float + 16 FMAs, and
-// a decoded lane takes 5 registers instead of 17. load/apply split as before: bitwise equal by construction.
+// tq_apply does the activation side. vec_dot_tq_q8_1 is apply(load()), and the decode-once kernels call tq_load once
+// per weight part and tq_apply per entry: the same integer and float operations in the same order, so the two kernel
+// families are bitwise equal to EACH OTHER (native_expert_parity: "decode-once vs per-entry kernels"). Neither is
+// bitwise equal to a float decode of the weights, see Numerics.
+// The lane's 16 codebook values as 4 ints of packed int8: a trellis step's 4 weights are two 2-byte entries = one
+// int, so the apply side is 4 dp4a instead of 16 int8->float + 16 FMAs, and a decoded lane takes 5 registers
+// instead of 17.
+//
+// Numerics. Decode (these kernels) reads the int8 codebook tq_lut_i8: one global scale TQ_LUT_I8_SCALE, relative
+// RMS error 0.85% of the codebook std, far below the trellis quantization error. The prompt path decodes the
+// trellis otherwise: the FP16 path (dq_tq below) through the fp16 codebook tq_lut_f16, which is what ggml's to_float
+// reads, and the MMQ path through the GGML fork's tile loader (the int8 codebook too on the fork's main since
+// 2026-10-03; the fp16 codebook requantized to int8 per 16 weights before that and with GGML_CUDA_TQ_F32_DOT).
+// Decode and prompt therefore agree within tolerance, not bitwise. native_expert_parity bounds this path: one
+// expert (int8 codebook, q8_1 activations) against ggml's float dequantizer and a float SwiGLU, relative L1 error
+// sum|gpu - ref| / sum|ref| < 3e-2 (the bound every expert type's CPU and GPU paths get there); the prompt path's
+// fp16 dequantizers against to_float at < 1e-6, their fp16 output equal to the fp16 of the float one.
struct TqLane { int q[4]; float d; };
template<int TY>
__device__ __forceinline__ TqLane tq_load(const void* __restrict__ vbq, int kbx, int iqs) {
diff --git a/src/kernels/native_expert_parity.cpp b/src/kernels/native_expert_parity.cpp
index d6a17484a..714fabba0 100644
--- a/src/kernels/native_expert_parity.cpp
+++ b/src/kernels/native_expert_parity.cpp
@@ -6,8 +6,10 @@
//
// (a) float reference: ggml's own dequantizer (`to_float`) and a float SwiGLU expert, (b) the CPU path
// (ggml-cpu vec_dot with its quantized activations), (c) the GPU path (`native_expert_grouped`, q8_1
-// activations). (b) and (c) each differ from (a) by their activation rounding only (a few 1e-3 relative). (c) runs
-// twice, with the kernels that decode a weight part once for all entries and with the per-entry ones: bitwise equal.
+// activations). (b) and (c) each differ from (a) by their activation rounding (a few 1e-3 relative) and, for the
+// trellis types on the GPU, by its int8 codebook (tq_lut_i8, see iq_kernels.cu "Numerics"); both must stay within
+// relative L1 error 3e-2 of (a). (c) runs twice, with the kernels that decode a weight part once for all entries and
+// with the per-entry ones: those two are bitwise equal to each other.
// (d) the GPU dequantizers of the prompt path and the embedding against `to_float` (Q8_0: bit for bit).
// The synthetic mode and (d) follow eddoursul/Strata 8029fa9.
#include "strata/artifact/gguf_reader.hpp"