37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Develop 2026-10-14 (5): the ChangeLog with the MXFP4_MOE file's speed against the Q4_K_M
camauri2bfaf2d4b268
diff --git a/ChangeLog.md b/ChangeLog.mdindex 373508b..9ef8ceb 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -9,7 +9,7 @@## [2026-10-14] - MXFP4 experts, and output heads in BF16-- **The "MXFP4_MOE" files run.** Their routed experts are MXFP4 (OCP's microscaling format: 32 four-bit values and a power of two a block), which gguf2jns refused and the engine could not read. The engine has MXFP4 products now - IQ4_NL's kernel with the block's power of two for a scale, the same bits on AVX2 and AVX-VNNI, for one vector or eight, and the same bits as the exact Q8_0 rewrite of the same blocks - and gguf2jns keeps such experts as they are. noctrex's Qwen3.6-35B-A3B-MXFP4_MOE against llama.cpp on the same file: 199 of 200 most likely tokens agree, mean logit difference 0.105 (on Italian prose 63 of 64, 0.085), closer than bartowski's Q4_K_M to its own llama.cpp (196, 0.146; 60, 0.135). The file itself, against llama.cpp's Q8_0 of the same model, is further than that Q4_K_M (378 against 387 of 400 most likely tokens, 364 against 370 on Italian prose) and 9% smaller (20.3 GB against 22.3). On 12 threads of the development laptop the MXFP4 product reads 66 GB/s where IQ4_NL reads 73: 4% fewer weights a second, a block of it being 17 bytes to IQ4_NL's 18 (its scales come from a table: computed in the kernel, they took registers the unpacking needs and it ran 14% slower).+- **The "MXFP4_MOE" files run.** Their routed experts are MXFP4 (OCP's microscaling format: 32 four-bit values and a power of two a block), which gguf2jns refused and the engine could not read. The engine has MXFP4 products now - IQ4_NL's kernel with the block's power of two for a scale, the same bits on AVX2 and AVX-VNNI, for one vector or eight, and the same bits as the exact Q8_0 rewrite of the same blocks - and gguf2jns keeps such experts as they are. noctrex's Qwen3.6-35B-A3B-MXFP4_MOE against llama.cpp on the same file: 199 of 200 most likely tokens agree, mean logit difference 0.105 (on Italian prose 63 of 64, 0.085), closer than bartowski's Q4_K_M to its own llama.cpp (196, 0.146; 60, 0.135). The file itself, against llama.cpp's Q8_0 of the same model, is further than that Q4_K_M (378 against 387 of 400 most likely tokens, 364 against 370 on Italian prose) and 9% smaller (20.3 GB against 22.3). It is also slower on the development laptop (janas-bench, two runs each in alternated order): a prompt at 85 tokens/s against 96, a reply at 25.3 against 27.0, and without drafts, where the Q4_K_M has its MTP block (38 tokens/s): its weights outside the experts are Q8_0 and the head BF16, 3.1 GB in the file where the Q4_K_M's are 1.9. On 12 threads of the development laptop the MXFP4 product reads 66 GB/s where IQ4_NL reads 73: 4% fewer weights a second, a block of it being 17 bytes to IQ4_NL's 18 (its scales come from a table: computed in the kernel, they took registers the unpacking needs and it ran 14% slower).- **An output head in BF16 or F16 is read as Q8_0.** The "MXFP4_MOE" files keep theirs in BF16 (1,017 MB on Qwen3.6-35B-A3B), which the engine refused. It is rewritten once, as the model opens, the way llama.cpp's quantization makes Q8_0, into 540 MB - the head is read whole at every token - and the model's description says so; the file keeps the BF16. A token embedding table in BF16 or F16 is rewritten the same way.