Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Trellis numerics: say which codebook each path reads and what is bitwise
Laurent Zuijdwijk390da5d905fe
diff --git a/docs/GYRO.md b/docs/GYRO.mdindex 5b73b0a52..04dd89eea 100644--- a/docs/GYRO.md+++ b/docs/GYRO.md@@ -173,6 +173,19 @@ checked. A real chat at 128k context worked through the web interface.Single RTX 4090 smoke run (Gyro-S, MTP draft, greedy): code 107.5, prose 84.3, JSON 76.3 tokens/s.+## Numerics++- **Decode** (Strata's own expert kernels) reads the trellis weights through an int8 copy of the codebook with one+ global scale (`tq_lut_i8`, relative RMS error 0.85 % of the codebook's standard deviation, far below the+ quantization error itself), against q8_1 activations.+- **Prompt processing** decodes them otherwise: the FP16 path through the fp16 codebook (`tq_lut_f16`, what GGML's+ own dequantizer reads); the MMQ path through the llama.cpp fork's kernels, which read the int8 codebook as well+ on the fork's current `main`.+- So decode and prompt results agree within tolerance, not bit for bit. `native_expert_parity` checks one expert+ on the decode kernels against GGML's float dequantizer and a float SwiGLU: relative L1 error below 3e-2, the+ bound every expert type gets there. Its "bitwise equal" lines compare Strata's decode-once and per-entry decode+ kernels with each other, not with a float reference.+## Known limits- **Not measured yet:** token parity against llama.cpp.diff --git a/src/kernels/cuda/iq_kernels.cu b/src/kernels/cuda/iq_kernels.cuindex cc3378235..91e608da1 100644--- a/src/kernels/cuda/iq_kernels.cu+++ b/src/kernels/cuda/iq_kernels.cu@@ -650,12 +650,23 @@ __device__ __forceinline__ void tq_states4(const uint8_t* qs, int l, uint32_t s[}// lane iqs/4 of block kbx against the q8_1 blocks aligned with the block's start (4 per trellis block). Split in// two, like the IQ types' decode-once kernels below: tq_load decodes the lane's 16 codebook values (weight side only),-// tq_apply does the activation side. vec_dot_tq_q8_1 is apply(load()), so the per-entry and the decode-once kernels-// do the same float operations in the same order: bitwise equal.-// The lane's 16 codebook values as 4 ints of packed int8 (the int8 codebook tq_lut_i8, one global scale-// TQ_LUT_I8_SCALE; relative RMS error 0.85% of the codebook std, far below the trellis error): a trellis step's-// 4 weights are two 2-byte entries = one int, so the apply side is 4 dp4a instead of 16 int8->float + 16 FMAs, and-// a decoded lane takes 5 registers instead of 17. load/apply split as before: bitwise equal by construction.+// tq_apply does the activation side. vec_dot_tq_q8_1 is apply(load()), and the decode-once kernels call tq_load once+// per weight part and tq_apply per entry: the same integer and float operations in the same order, so the two kernel+// families are bitwise equal to EACH OTHER (native_expert_parity: "decode-once vs per-entry kernels"). Neither is+// bitwise equal to a float decode of the weights, see Numerics.+// The lane's 16 codebook values as 4 ints of packed int8: a trellis step's 4 weights are two 2-byte entries = one+// int, so the apply side is 4 dp4a instead of 16 int8->float + 16 FMAs, and a decoded lane takes 5 registers+// instead of 17.+//+// Numerics. Decode (these kernels) reads the int8 codebook tq_lut_i8: one global scale TQ_LUT_I8_SCALE, relative+// RMS error 0.85% of the codebook std, far below the trellis quantization error. The prompt path decodes the+// trellis otherwise: the FP16 path (dq_tq below) through the fp16 codebook tq_lut_f16, which is what ggml's to_float+// reads, and the MMQ path through the GGML fork's tile loader (the int8 codebook too on the fork's main since+// 2026-10-03; the fp16 codebook requantized to int8 per 16 weights before that and with GGML_CUDA_TQ_F32_DOT).+// Decode and prompt therefore agree within tolerance, not bitwise. native_expert_parity bounds this path: one+// expert (int8 codebook, q8_1 activations) against ggml's float dequantizer and a float SwiGLU, relative L1 error+// sum|gpu - ref| / sum|ref| < 3e-2 (the bound every expert type's CPU and GPU paths get there); the prompt path's+// fp16 dequantizers against to_float at < 1e-6, their fp16 output equal to the fp16 of the float one.struct TqLane { int q[4]; float d; };template<int TY>__device__ __forceinline__ TqLane tq_load(const void* __restrict__ vbq, int kbx, int iqs) {diff --git a/src/kernels/native_expert_parity.cpp b/src/kernels/native_expert_parity.cppindex d6a17484a..714fabba0 100644--- a/src/kernels/native_expert_parity.cpp+++ b/src/kernels/native_expert_parity.cpp@@ -6,8 +6,10 @@//// (a) float reference: ggml's own dequantizer (`to_float`) and a float SwiGLU expert, (b) the CPU path// (ggml-cpu vec_dot with its quantized activations), (c) the GPU path (`native_expert_grouped`, q8_1-// activations). (b) and (c) each differ from (a) by their activation rounding only (a few 1e-3 relative). (c) runs-// twice, with the kernels that decode a weight part once for all entries and with the per-entry ones: bitwise equal.+// activations). (b) and (c) each differ from (a) by their activation rounding (a few 1e-3 relative) and, for the+// trellis types on the GPU, by its int8 codebook (tq_lut_i8, see iq_kernels.cu "Numerics"); both must stay within+// relative L1 error 3e-2 of (a). (c) runs twice, with the kernels that decode a weight part once for all entries and+// with the per-entry ones: those two are bitwise equal to each other.// (d) the GPU dequantizers of the prompt path and the embedding against `to_float` (Q8_0: bit for bit).// The synthetic mode and (d) follow eddoursul/Strata 8029fa9.#include "strata/artifact/gguf_reader.hpp"