37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-10-09 (3): Qwen3-4B and Qwen3-30B-A3B also with an importance matrix; fast profiles for the new versions
camauri4d2d719265eb
diff --git a/ChangeLog.md b/ChangeLog.mdindex d8e9569..286229b 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -2,6 +2,10 @@> Curated, user-facing summary of completed work, newest first.+## [2026-10-09] - Qwen3-4B and Qwen3-30B-A3B also with an importance matrix++- **`qwen3-4b-imatrix` and `qwen3-30b-a3b-imatrix`: bartowski's Q4_K_M beside Qwen's own, which stay the default.** Qwen's Q4_K_M files are quantized without an importance matrix, bartowski's with one. Against Qwen's own Q8_0 (400 tokens of llama.cpp's logits, most likely token agreeing): Qwen3-4B 381 against 364 on code and 351 against 329 on Italian, at the same size and speed; Qwen3-30B-A3B 388 against 376 and 367 against 335, the nearest of the three versions now in the catalog, 8-10% slower held in memory since it keeps more matrices in Q6_K. Each has its fast profile measured: `--head 4 --attention fast` (+5.7%) and `--head 4` (+3.2%). The three-bit Qwen3-30B-A3B also has one now: `--head 4` (+3.8%).+## [2026-10-09] - Qwen3-30B-A3B in three bits- **`qwen3-30b-a3b-q3`: the same model in unsloth's dynamic three-bit mix, smaller and closer to the original.** Measured against Qwen's own Q8_0 (400 tokens of llama.cpp's logits), unsloth's UD-Q3_K_XL agrees on the most likely token more often than Qwen's Q4_K_M that the catalog already had: 386 against 376 on code, 360 against 335 on Italian prose, with 27% fewer bytes of experts (12.8 GB against 17.6). With little memory it is also faster: with the expert cache held at 2 GiB, 19.8 tokens/s against 13.6. Held whole in memory it is as fast: 32.0-32.2 tokens/s against 32.1-32.2, once the Q3_K product was made faster (below). `qwen3-30b-a3b` is unchanged.diff --git a/MODELS.md b/MODELS.mdindex 3f45211..12ec880 100644--- a/MODELS.md+++ b/MODELS.md@@ -94,6 +94,14 @@ e8ed44bdd3c612ce3ec1204a4b9db671115efab4a2ada384c70cf6492d430291 qwen3-4b-q4km-1cbd8ebfdf14aee05777f5278e655382f040cdedc21d282e93806fc8cf279fc5 qwen3-4b-q4km.jns (jns_planes)```+**Qwen3-4B with an importance matrix** (`qwen3-4b-imatrix`), from [`bartowski/Qwen_Qwen3-4B-GGUF`](https://huggingface.co/bartowski/Qwen_Qwen3-4B-GGUF): the same weight types as Qwen's Q4_K_M, quantized with an importance matrix.++```+fbe1d5edd4ce802ae3ae7c7e4ab7d09789d697fdac1fc7929f8df4ca3c41bae3 Qwen_Qwen3-4B-Q4_K_M.gguf+aa4f0bc6033a206ba5ac51dcdd799299d369019d5614dc08a6baa0bda6123098 qwen3-4b-bq4km-flat.jns (gguf2jns)+134ae0afbbd3e45b3cdbfd36fd205fb98c91df62a129541048e29370a42b835e qwen3-4b-bq4km.jns (jns_planes)+```+**Qwen3.5-0.8B, Qwen3.5-2B and Qwen3.5-9B**, dense, from [`unsloth/Qwen3.5-0.8B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF), [`unsloth/Qwen3.5-2B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) and [`unsloth/Qwen3.5-9B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF), GGUF conversions of the Qwen team's checkpoints. Converted in one step, as they are kept here: `jns_planes` could cut planes from their Q6_K `down` matrices, but a model this size has no need to read fewer bits.```@@ -113,6 +121,14 @@ e40ec6c58ef2390c7a4ce8c9b7b6b384ed4e55c9980f646d1c98e83e2c018115 qwen3.5-9b-q4k3fa0185b7eecd43e222d420f68aae28a86b3b90392fd7e4052b5cd337318e895 qwen3-30b-a3b-q4km.jns (jns_planes)```+**Qwen3-30B-A3B with an importance matrix** (`qwen3-30b-a3b-imatrix`), from [`bartowski/Qwen_Qwen3-30B-A3B-GGUF`](https://huggingface.co/bartowski/Qwen_Qwen3-30B-A3B-GGUF), bartowski's Q4_K_M.++```+a015794bfb1d69cb03dbb86b185fb2b9b339f757df5f8f9dd9ebdab8f6ed5d32 Qwen_Qwen3-30B-A3B-Q4_K_M.gguf+c2089d293f10d2e0fdabbb7ea6f17d6834f3027480de987fb6dd718e301eaa04 qwen3-30b-a3b-bq4km-flat.jns (gguf2jns)+bee2aa5a9b5a33f19d9c153d2a0a7e8ac493a670b2d0084c195ce977cf3c83ea qwen3-30b-a3b-bq4km.jns (jns_planes)+```+**Qwen3-30B-A3B in three bits** (`qwen3-30b-a3b-q3`), from [`unsloth/Qwen3-30B-A3B-GGUF`](https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF), unsloth's dynamic UD-Q3_K_XL. Converted in one step: its down matrices are Q3_K and Q4_K, so there are no planes to cut.```diff --git a/README.md b/README.mdindex 32544d9..0b5148a 100644--- a/README.md+++ b/README.md@@ -134,10 +134,12 @@ Janas reads its own format, `.jns`, converted from a **Q4_K_M** GGUF (Q3_K, IQ3_| **Qwen3-Coder-Next** | [Qwen](https://huggingface.co/Qwen/Qwen3-Coder-Next-GGUF) | 48.4 GB | 48.4 GB | 32 GB || **Qwen3-Coder-30B-A3B-Instruct** — for code completion in the editor ([docs/code-completion.md](docs/code-completion.md)) | [unsloth](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF) | 18.6 GB | 18.6 GB | 32 GB, to keep it whole in memory || **Qwen3-30B-A3B** | [Qwen](https://huggingface.co/Qwen/Qwen3-30B-A3B-GGUF) | 18.6 GB | 18.6 GB | 16 GB |+| **Qwen3-30B-A3B with an importance matrix** (`qwen3-30b-a3b-imatrix`) — bartowski's Q4_K_M: the nearest to Qwen's own Q8_0 of the three here (400 tokens: code 388, Italian 367, against 376 and 335 for the Q4_K_M above), 8-10% slower held in memory: it keeps more matrices in Q6_K | [bartowski](https://huggingface.co/bartowski/Qwen_Qwen3-30B-A3B-GGUF) | 18.6 GB | 18.6 GB | 16 GB || **Qwen3-30B-A3B in three bits** (`qwen3-30b-a3b-q3`) — unsloth's dynamic UD-Q3_K_XL: against Qwen's own Q8_0 it agrees more often than the Q4_K_M above (400 tokens: code 386 against 376, Italian 360 against 335), with 27% fewer bytes of experts; with the expert cache held at 2 GiB on this 32 GB machine, 19.8 tokens/s against 13.6; held whole in memory, as fast (32.0-32.2 tokens/s against 32.1-32.2) | [unsloth](https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF) | 13.8 GB | 13.8 GB | 8-16 GB || **Qwen3.5-9B** — dense; its prediction block from the checkpoint ([MODELS.md](MODELS.md#the-fingerprints)) | [unsloth](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF) | 5.7 GB | 5.7 GB | 16 GB || **Qwen3.5-2B** — dense, small and quick; a prediction block too | [unsloth](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) | 1.3 GB | 1.3 GB | 8 GB || **Qwen3-4B** — dense, and the smallest here that answers well | [Qwen](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | 2.5 GB | 2.5 GB | 8 GB |+| **Qwen3-4B with an importance matrix** (`qwen3-4b-imatrix`) — bartowski's Q4_K_M, the same weight types and size as Qwen's: against Qwen's own Q8_0 (400 tokens) code 381 against 364, Italian 351 against 329, at the same speed | [bartowski](https://huggingface.co/bartowski/Qwen_Qwen3-4B-GGUF) | 2.5 GB | 2.5 GB | 8 GB || **Qwen3-0.6B** — not to talk to: to draft for a dense one (`--draft`) | [unsloth](https://huggingface.co/unsloth/Qwen3-0.6B-GGUF) | 0.40 GB | 0.40 GB | with the model above || **Qwen3-Embedding-0.6B** — not to talk to: for embeddings (`janas-server --embedding-model`); published in Q8_0 and converted as it is | [Qwen](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B-GGUF) | 0.64 GB | 0.64 GB | beside the chat model |@@ -247,6 +249,9 @@ The settings come in layers, each over the one before: the model's profile in th| Qwen3.5-2B | `--head 4` | 10.2% | 376 → 352 / 388 → 375 || Qwen3.5-9B | `--head 4` | 5.1% | 383 → 372 / 393 → 391 || Qwen3-30B-A3B | `--head 4 --attention fast --bits 4` | 10.0% | 57 → 55 of 64 / 392 → 385 |+| Qwen3-4B with an importance matrix | `--head 4 --attention fast` (9 October) | 5.7% | 351 → 350 / 381 → 376, against Qwen's Q8_0 |+| Qwen3-30B-A3B with an importance matrix | `--head 4` (9 October; `--bits 4` added 1.6%, `--attention fast` 0.6%) | 3.2% | 367 → 354 / 388 → 387, against Qwen's Q8_0 |+| Qwen3-30B-A3B in three bits | `--head 4` (measured on 9 October; `--attention fast` added 1.3%) | 3.8% | 360 → 348 / 386 → 382, against Qwen's Q8_0 || Qwen3.6-35B-A3B | `--head 4 --bits 4` | 6.5% | 60 → 55 of 64 / 196 → 194 of 200 || Qwen3-Next-80B-A3B | `--head 4 --bits 4` | 8.6% | 386 → 372 / 397 → 392 || Qwen3-Coder-Next | `--head 4 --bits 4` | 10.5% | — / 193 → 196 of 200 |diff --git a/src/common/catalog.c b/src/common/catalog.cindex 3d15350..dd5269d 100644--- a/src/common/catalog.c+++ b/src/common/catalog.c@@ -51,6 +51,25 @@ const struct janas_model_entry janas_catalog[] = {.ram_gb = 8,.draft = "qwen3-0.6b",.fast = "--head 4 --attention fast"},+ /* the same weights types and size from bartowski, quantized with an+ importance matrix: against Qwen's own Q8_0 (400 tokens, most likely+ token agreeing) code 381 against 364, Italian 351 against 329, at the+ same speed (28.1 tokens/s against 27.6, 9 Oct 2026) */+ {.name = "qwen3-4b-imatrix",+ .what = "Qwen3-4B, bartowski's Q4_K_M: same size, nearer the original",+ .repo = "bartowski/Qwen_Qwen3-4B-GGUF",+ .parts = {"Qwen_Qwen3-4B-Q4_K_M.gguf"},+ .part_sha =+ {"fbe1d5edd4ce802ae3ae7c7e4ab7d09789d697fdac1fc7929f8df4ca3c41bae3"},+ .gguf_gb = 2.5,+ .out = "qwen3-4b-bq4km.jns",+ .flat_sha =+ "aa4f0bc6033a206ba5ac51dcdd799299d369019d5614dc08a6baa0bda6123098",+ .jns_sha =+ "134ae0afbbd3e45b3cdbfd36fd205fb98c91df62a129541048e29370a42b835e",+ .ram_gb = 8,+ .draft = "qwen3-0.6b",+ .fast = "--head 4 --attention fast"},{.name = "qwen3.5-0.8b",.what = "Qwen3.5-0.8B, dense - a test model of the Qwen3.5 architecture, ""not for chatting",@@ -119,7 +138,8 @@ const struct janas_model_entry janas_catalog[] = {tokens, most likely token agreeing: code 386 against 376, Italian360 against 335) with 27% fewer bytes of experts; with a cache of2 GiB 19.8 tokens/s against 13.6, held whole in memory as fast- (32.1, 9 Oct 2026). No fast profile measured yet. */+ (32.1, 9 Oct 2026). Fast profile: --head 4, 3.8% (bench_long+ 1900 + 128); --attention fast added 1.3%, under the 2% a lever needs. */{.name = "qwen3-30b-a3b-q3",.what = "Qwen3-30B-A3B, unsloth's 3-bit mix: smaller, as good",.repo = "unsloth/Qwen3-30B-A3B-GGUF",@@ -130,7 +150,27 @@ const struct janas_model_entry janas_catalog[] = {.out = "qwen3-30b-a3b-udq3kxl.jns",.flat_sha ="7a77e62a3d6b1df321f455472e0b970a09eb3534ec76982a6e169087d4897656",- .ram_gb = 8},+ .ram_gb = 8,+ .fast = "--head 4"},+ /* bartowski's Q4_K_M, quantized with an importance matrix: the nearest+ to Qwen's own Q8_0 of the three (400 tokens: code 388, Italian 367,+ against 376 and 335 for Qwen's Q4_K_M and 386 and 360 for the+ three-bit mix), 8-10% slower than Qwen's held in memory, since it+ keeps more matrices in Q6_K (9 Oct 2026) */+ {.name = "qwen3-30b-a3b-imatrix",+ .what = "Qwen3-30B-A3B, bartowski's Q4_K_M, nearest to the Q8_0",+ .repo = "bartowski/Qwen_Qwen3-30B-A3B-GGUF",+ .parts = {"Qwen_Qwen3-30B-A3B-Q4_K_M.gguf"},+ .part_sha =+ {"a015794bfb1d69cb03dbb86b185fb2b9b339f757df5f8f9dd9ebdab8f6ed5d32"},+ .gguf_gb = 18.6,+ .out = "qwen3-30b-a3b-bq4km.jns",+ .flat_sha =+ "c2089d293f10d2e0fdabbb7ea6f17d6834f3027480de987fb6dd718e301eaa04",+ .jns_sha =+ "bee2aa5a9b5a33f19d9c153d2a0a7e8ac493a670b2d0084c195ce977cf3c83ea",+ .ram_gb = 16,+ .fast = "--head 4"},{.name = "qwen3.6-35b-a3b",.what = "Qwen3.6-35B-A3B, mixture of experts, MTP inside",.repo = "bartowski/Qwen_Qwen3.6-35B-A3B-GGUF",