37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-10-09 (2): the Q3_K product a quarter faster, the three-bit Qwen3-30B-A3B as fast as the four-bit one held in memory
camauri945eec60ff63
diff --git a/ChangeLog.md b/ChangeLog.mdindex 9e2a917..d8e9569 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -4,7 +4,8 @@## [2026-10-09] - Qwen3-30B-A3B in three bits-- **`qwen3-30b-a3b-q3`: the same model in unsloth's dynamic three-bit mix, smaller and closer to the original.** Measured against Qwen's own Q8_0 (400 tokens of llama.cpp's logits), unsloth's UD-Q3_K_XL agrees on the most likely token more often than Qwen's Q4_K_M that the catalog already had: 386 against 376 on code, 360 against 335 on Italian prose, with 27% fewer bytes of experts (12.8 GB against 17.6). With little memory it is also faster: with the expert cache held at 2 GiB, 19.8 tokens/s against 13.6. Held whole in memory it is 5% slower, since its Q3_K product is slower than Q4_K's. `qwen3-30b-a3b` is unchanged. The same trade does not pay on Qwen3.6-35B-A3B, whose Q4_K_M is already made with an importance matrix: bartowski's Q3_K_M lost 26 positions of 400 on Italian for 2-12% more speed, so it is not in the catalog.+- **`qwen3-30b-a3b-q3`: the same model in unsloth's dynamic three-bit mix, smaller and closer to the original.** Measured against Qwen's own Q8_0 (400 tokens of llama.cpp's logits), unsloth's UD-Q3_K_XL agrees on the most likely token more often than Qwen's Q4_K_M that the catalog already had: 386 against 376 on code, 360 against 335 on Italian prose, with 27% fewer bytes of experts (12.8 GB against 17.6). With little memory it is also faster: with the expert cache held at 2 GiB, 19.8 tokens/s against 13.6. Held whole in memory it is as fast: 32.0-32.2 tokens/s against 32.1-32.2, once the Q3_K product was made faster (below). `qwen3-30b-a3b` is unchanged.+- **The Q3_K product is a quarter faster, with the same results to the bit.** It unpacked the eight registers of weights of a block and their eight registers of scales before using any: sixteen, all that AVX2 has, so they went through the stack (perf counted more loads than Q4_K for half the bytes). Made where each is used, one core reads 16.4 GB/s instead of 11.9, twelve threads 60.8 instead of 48.9, and the three-bit Qwen3-30B-A3B went from 29.3 tokens/s to 32.1 held in memory. Every unsloth or bartowski file with Q3_K in it gains as much. The same trade does not pay on Qwen3.6-35B-A3B, whose Q4_K_M is already made with an importance matrix: bartowski's Q3_K_M lost 26 positions of 400 on Italian for 2-12% more speed, so it is not in the catalog.## [2026-10-08] - Light experts not in memory left out, not the lightest of alldiff --git a/README.md b/README.mdindex bf35a6b..32544d9 100644--- a/README.md+++ b/README.md@@ -68,7 +68,7 @@ Everything the engine does is chosen from what it measures, so another machine w- **Optional GPU** (Vulkan 1.3, integrated or discrete), used only where the engine measures that it helps, for every quantized type the CPU reads. Results are identical with and without it.- **Experts read ahead** when the expert cache is too small for the model: the next layer's router is applied early and the experts it picks that are not in memory are read from disk while the current layer computes. It turns itself on only when the cache misses many experts a token (Qwen3-Next-80B with a 4 GiB cache: +6%) and changes no result. A long prompt is read the same way, block by block: the next layer's router is applied to the whole block and the experts most of its tokens will want are read while the current layer computes; on after a block that missed many experts. Qwen3-Next-80B reading 4,096 tokens: 36.8 to 42.0 tokens/s with a 4 GiB cache, 48.5 to 50.0 with 16 GiB (the 256 tokens written after it 1.4% slower, the cache holding other experts); Qwen3-30B-A3B, whose experts all fit, even. `JANAS_PREFETCH_PREFILL=0` turns it off.- **Light experts not in memory are left out** when the expert cache holds less than a fifth of the experts: an expert the router weighs below 0.8/k of a token (0.95/k under a tenth), and that would have to be read from disk, is not read and not computed, and the token's other experts take its share — the cache-conditional routing of Skliar et al. (TMLR 2025). Until 8 October 2026 the engine dropped the lightest experts whether in memory or not (six of ten, or eight). Qwen3-Next-80B, four bits, 400 tokens against llama.cpp, most likely token agreeing and tokens/s: with a 2 GiB cache all ten 393 at 11.0, six 383 at 20.9, now 380 at 26.4; with 4 GiB 393 at 20.7, six 383 at 27.6, now 384 at 27.7, with the mean logit difference 0.36 against 0.53; on Italian prose with 8 GiB 382 at 18.7, eight 361 at 21.4, now 372 at 23.7. On Qwen3-30B-A3B and Qwen3.6-35B-A3B with caches of 9% and 18% of their experts (1.5-3.5 GiB), on code and Italian prose, it was faster than the old way every time, by 9-37% (Qwen3-30B-A3B with 3 GiB: 28.7 tokens/s against 24.4, 381 tokens agreeing of 400 against 377), with a mean logit difference as low or lower in seven of the eight; it agreed on fewer most likely tokens in two (Italian, 64 tokens: 47 against 54 on the 30B with 3 GiB). `/experts n` (or `0`) asks for a number of experts and turns it off.-- **Every weight type near the memory's speed**, and the same results to the bit on every path (AVX2, AVX-VNNI, GPU). `tests/bench_kernels` measures each type's product on its own: writing a token, on 12 threads of the development laptop, Q8_0 84 GB/s, Q6_K 73, Q4_K 69, the other 4- and 5-bit types 59-68, Q3_K 48, IQ3_S 25, against the 81 GB/s the memory gives; and a token of Qwen3-Next-80B-A3B, Qwen3.6-35B-A3B, Qwen3-30B-A3B or a 27B, phase by phase, reads its weights at 59-62 GB/s on average. Until 6 October 2026 the Q3_K, IQ4_XS and IQ3_S products were six to thirteen times slower than that (their kernels converted a scale by calling a function), which made unsloth's UD files, which mix those types in, write at half their speed: Qwen3.8-27B UD-Q4_K_XL went from 1.7 tokens/s to 3.6.+- **Every weight type near the memory's speed**, and the same results to the bit on every path (AVX2, AVX-VNNI, GPU). `tests/bench_kernels` measures each type's product on its own: writing a token, on 12 threads of the development laptop, Q8_0 84 GB/s, Q6_K 73, Q4_K 69, the other 4- and 5-bit types 59-68, Q3_K 61 (48 until 9 October 2026, when its unpacking went through the stack), IQ3_S 25, against the 81 GB/s the memory gives; and a token of Qwen3-Next-80B-A3B, Qwen3.6-35B-A3B, Qwen3-30B-A3B or a 27B, phase by phase, reads its weights at 59-62 GB/s on average. Until 6 October 2026 the Q3_K, IQ4_XS and IQ3_S products were six to thirteen times slower than that (their kernels converted a scale by calling a function), which made unsloth's UD files, which mix those types in, write at half their speed: Qwen3.8-27B UD-Q4_K_XL went from 1.7 tokens/s to 3.6.On the machine above, with nothing else running, `janas-bench` reports this (measured before the changes of 6-7 October 2026: on Qwen3-30B-A3B in memory, `tests/bench_long` found them worth 4% reading a 1,900-token prompt and 1.5% writing after it):@@ -134,7 +134,7 @@ Janas reads its own format, `.jns`, converted from a **Q4_K_M** GGUF (Q3_K, IQ3_| **Qwen3-Coder-Next** | [Qwen](https://huggingface.co/Qwen/Qwen3-Coder-Next-GGUF) | 48.4 GB | 48.4 GB | 32 GB || **Qwen3-Coder-30B-A3B-Instruct** — for code completion in the editor ([docs/code-completion.md](docs/code-completion.md)) | [unsloth](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF) | 18.6 GB | 18.6 GB | 32 GB, to keep it whole in memory || **Qwen3-30B-A3B** | [Qwen](https://huggingface.co/Qwen/Qwen3-30B-A3B-GGUF) | 18.6 GB | 18.6 GB | 16 GB |-| **Qwen3-30B-A3B in three bits** (`qwen3-30b-a3b-q3`) — unsloth's dynamic UD-Q3_K_XL: against Qwen's own Q8_0 it agrees more often than the Q4_K_M above (400 tokens: code 386 against 376, Italian 360 against 335), with 27% fewer bytes of experts; with the expert cache held at 2 GiB on this 32 GB machine, 19.8 tokens/s against 13.6; held whole in memory, 5% slower (its Q3_K product is slower than Q4_K's) | [unsloth](https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF) | 13.8 GB | 13.8 GB | 8-16 GB |+| **Qwen3-30B-A3B in three bits** (`qwen3-30b-a3b-q3`) — unsloth's dynamic UD-Q3_K_XL: against Qwen's own Q8_0 it agrees more often than the Q4_K_M above (400 tokens: code 386 against 376, Italian 360 against 335), with 27% fewer bytes of experts; with the expert cache held at 2 GiB on this 32 GB machine, 19.8 tokens/s against 13.6; held whole in memory, as fast (32.0-32.2 tokens/s against 32.1-32.2) | [unsloth](https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF) | 13.8 GB | 13.8 GB | 8-16 GB || **Qwen3.5-9B** — dense; its prediction block from the checkpoint ([MODELS.md](MODELS.md#the-fingerprints)) | [unsloth](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF) | 5.7 GB | 5.7 GB | 16 GB || **Qwen3.5-2B** — dense, small and quick; a prediction block too | [unsloth](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) | 1.3 GB | 1.3 GB | 8 GB || **Qwen3-4B** — dense, and the smallest here that answers well | [Qwen](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | 2.5 GB | 2.5 GB | 8 GB |diff --git a/src/common/catalog.c b/src/common/catalog.cindex b6ca211..3d15350 100644--- a/src/common/catalog.c+++ b/src/common/catalog.c@@ -118,8 +118,8 @@ const struct janas_model_entry janas_catalog[] = {importance matrix: nearer Qwen's own Q8_0 than Qwen's Q4_K_M (400tokens, most likely token agreeing: code 386 against 376, Italian360 against 335) with 27% fewer bytes of experts; with a cache of- 2 GiB 19.8 tokens/s against 13.6, held whole in memory 5% slower- (9 Oct 2026). No fast profile measured yet. */+ 2 GiB 19.8 tokens/s against 13.6, held whole in memory as fast+ (32.1, 9 Oct 2026). No fast profile measured yet. */{.name = "qwen3-30b-a3b-q3",.what = "Qwen3-30B-A3B, unsloth's 3-bit mix: smaller, as good",.repo = "unsloth/Qwen3-30B-A3B-GGUF",diff --git a/src/llm/quant_dotn.h b/src/llm/quant_dotn.hindex a351865..6fbdbfa 100644--- a/src/llm/quant_dotn.h+++ b/src/llm/quant_dotn.h@@ -1361,24 +1361,6 @@ DOTN(q3k_dotn_body)(const struct janas_block_q3k *w,for (size_t b = 0; b < nb; b++) {if (nv <= 2)DOTN(prefetch_ahead)(&w[b], sizeof(w[b]));- __m256i u[8], sc16[8];- __m256i hm = _mm256_loadu_si256((const __m256i *)w[b].hmask);- _Pragma("GCC unroll 2") for (int n = 0; n < 2; n++)- {- __m256i qs =- _mm256_loadu_si256((const __m256i *)(w[b].qs + 32 * n));- _Pragma("GCC unroll 4") for (int j = 0; j < 4; j++)- {- __m256i lo =- _mm256_and_si256(_mm256_srli_epi16(qs, 2 * j), three);- /* bit j + 4n of hmask moved to bit 2, within its byte */- const int sh = j + 4 * n - 2;- __m256i hs = sh > 0 ? _mm256_srli_epi16(hm, sh)- : sh < 0 ? _mm256_slli_epi16(hm, -sh)- : hm;- u[4 * n + j] = _mm256_or_si256(lo, _mm256_and_si256(hs, four));- }- }/* register k holds groups 2k (low lane) and 2k + 1 (high): thescales as int16, the even ones in the low lane and the odd ones inthe high, then word k of each lane spread over it */@@ -1386,25 +1368,45 @@ DOTN(q3k_dotn_body)(const struct janas_block_q3k *w,__m256i eo = _mm256_cvtepi8_epi16(_mm_shuffle_epi8(s8, _mm_setr_epi8(0, 2, 4, 6, 8, 10, 12, 14, 1, 3,5, 7, 9, 11, 13, 15)));- _Pragma("GCC unroll 8") for (int k = 0; k < 8; k++) sc16[k] =- _mm256_shuffle_epi8(- eo, _mm256_set1_epi16((short)((2 * k + 1) << 8 | 2 * k)));__m256i scall = _mm256_cvtepi8_epi16(s8);float d = _cvtsh_ss(w[b].d);- _Pragma("GCC unroll 8") for (int v = 0; v < nv; v++)+ __m256i hm = _mm256_loadu_si256((const __m256i *)w[b].hmask);+ __m256i qs[2] = {_mm256_loadu_si256((const __m256i *)w[b].qs),+ _mm256_loadu_si256((const __m256i *)(w[b].qs + 32))};+ /* each register of weights and its scales made where it is used,+ for every vector at once: made all first, eight of each, they+ took all sixteen registers and went through the stack (more+ loads than Q4_K for half the bytes, 9 Oct 2026) */+ __m256i sum[8];+ _Pragma("GCC unroll 8") for (int v = 0; v < nv; v++) sum[v] =+ _mm256_setzero_si256();+ _Pragma("GCC unroll 8") for (int k = 0; k < 8; k++){- const struct janas_block_q8k *xb = x + (size_t)v * xs + b;- __m256i sum = _mm256_setzero_si256();- _Pragma("GCC unroll 8") for (int k = 0; k < 8; k++)+ const int n = k / 4, j = k % 4;+ __m256i lo =+ _mm256_and_si256(_mm256_srli_epi16(qs[n], 2 * j), three);+ /* bit j + 4n of hmask moved to bit 2, within its byte */+ const int sh = j + 4 * n - 2;+ __m256i hs = sh > 0 ? _mm256_srli_epi16(hm, sh)+ : sh < 0 ? _mm256_slli_epi16(hm, -sh)+ : hm;+ __m256i u = _mm256_or_si256(lo, _mm256_and_si256(hs, four));+ __m256i sc = _mm256_shuffle_epi8(+ eo, _mm256_set1_epi16((short)((2 * k + 1) << 8 | 2 * k)));+ _Pragma("GCC unroll 8") for (int v = 0; v < nv; v++){- __m256i a =- _mm256_loadu_si256((const __m256i *)(xb->qs + 32 * k));- sum = DOTN_ACC(sum, _mm256_maddubs_epi16(u[k], a), sc16[k]);+ __m256i a = _mm256_loadu_si256(+ (const __m256i *)(x[(size_t)v * xs + b].qs + 32 * k));+ sum[v] = DOTN_ACC(sum[v], _mm256_maddubs_epi16(u, a), sc);}+ }+ _Pragma("GCC unroll 8") for (int v = 0; v < nv; v++)+ {+ const struct janas_block_q8k *xb = x + (size_t)v * xs + b;__m256i corr = _mm256_madd_epi16(_mm256_loadu_si256((const __m256i *)xb->bsums), scall);int32_t sumi = DOTN(hsum_epi32)(- _mm256_sub_epi32(sum, _mm256_slli_epi32(corr, 2)));+ _mm256_sub_epi32(sum[v], _mm256_slli_epi32(corr, 2)));acc[v] = fmaf(d * xb->d, (float)sumi, acc[v]);}}