Skip to content

armgit / janas

Public Imported from GitHub
0

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Release 2026-10-09 (3): Qwen3-4B and Qwen3-30B-A3B also with an importance matrix; fast profiles for the new versions

camauri4d2d719265eb
diff --git a/ChangeLog.md b/ChangeLog.md
index d8e9569..286229b 100644
--- a/ChangeLog.md
+++ b/ChangeLog.md
@@ -2,6 +2,10 @@
> Curated, user-facing summary of completed work, newest first.
+## [2026-10-09] - Qwen3-4B and Qwen3-30B-A3B also with an importance matrix
+
+- **`qwen3-4b-imatrix` and `qwen3-30b-a3b-imatrix`: bartowski's Q4_K_M beside Qwen's own, which stay the default.** Qwen's Q4_K_M files are quantized without an importance matrix, bartowski's with one. Against Qwen's own Q8_0 (400 tokens of llama.cpp's logits, most likely token agreeing): Qwen3-4B 381 against 364 on code and 351 against 329 on Italian, at the same size and speed; Qwen3-30B-A3B 388 against 376 and 367 against 335, the nearest of the three versions now in the catalog, 8-10% slower held in memory since it keeps more matrices in Q6_K. Each has its fast profile measured: `--head 4 --attention fast` (+5.7%) and `--head 4` (+3.2%). The three-bit Qwen3-30B-A3B also has one now: `--head 4` (+3.8%).
+
## [2026-10-09] - Qwen3-30B-A3B in three bits
- **`qwen3-30b-a3b-q3`: the same model in unsloth's dynamic three-bit mix, smaller and closer to the original.** Measured against Qwen's own Q8_0 (400 tokens of llama.cpp's logits), unsloth's UD-Q3_K_XL agrees on the most likely token more often than Qwen's Q4_K_M that the catalog already had: 386 against 376 on code, 360 against 335 on Italian prose, with 27% fewer bytes of experts (12.8 GB against 17.6). With little memory it is also faster: with the expert cache held at 2 GiB, 19.8 tokens/s against 13.6. Held whole in memory it is as fast: 32.0-32.2 tokens/s against 32.1-32.2, once the Q3_K product was made faster (below). `qwen3-30b-a3b` is unchanged.
diff --git a/MODELS.md b/MODELS.md
index 3f45211..12ec880 100644
--- a/MODELS.md
+++ b/MODELS.md
@@ -94,6 +94,14 @@ e8ed44bdd3c612ce3ec1204a4b9db671115efab4a2ada384c70cf6492d430291 qwen3-4b-q4km-
1cbd8ebfdf14aee05777f5278e655382f040cdedc21d282e93806fc8cf279fc5 qwen3-4b-q4km.jns (jns_planes)
```
+**Qwen3-4B with an importance matrix** (`qwen3-4b-imatrix`), from [`bartowski/Qwen_Qwen3-4B-GGUF`](https://huggingface.co/bartowski/Qwen_Qwen3-4B-GGUF): the same weight types as Qwen's Q4_K_M, quantized with an importance matrix.
+
+```
+fbe1d5edd4ce802ae3ae7c7e4ab7d09789d697fdac1fc7929f8df4ca3c41bae3 Qwen_Qwen3-4B-Q4_K_M.gguf
+aa4f0bc6033a206ba5ac51dcdd799299d369019d5614dc08a6baa0bda6123098 qwen3-4b-bq4km-flat.jns (gguf2jns)
+134ae0afbbd3e45b3cdbfd36fd205fb98c91df62a129541048e29370a42b835e qwen3-4b-bq4km.jns (jns_planes)
+```
+
**Qwen3.5-0.8B, Qwen3.5-2B and Qwen3.5-9B**, dense, from [`unsloth/Qwen3.5-0.8B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF), [`unsloth/Qwen3.5-2B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) and [`unsloth/Qwen3.5-9B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF), GGUF conversions of the Qwen team's checkpoints. Converted in one step, as they are kept here: `jns_planes` could cut planes from their Q6_K `down` matrices, but a model this size has no need to read fewer bits.
```
@@ -113,6 +121,14 @@ e40ec6c58ef2390c7a4ce8c9b7b6b384ed4e55c9980f646d1c98e83e2c018115 qwen3.5-9b-q4k
3fa0185b7eecd43e222d420f68aae28a86b3b90392fd7e4052b5cd337318e895 qwen3-30b-a3b-q4km.jns (jns_planes)
```
+**Qwen3-30B-A3B with an importance matrix** (`qwen3-30b-a3b-imatrix`), from [`bartowski/Qwen_Qwen3-30B-A3B-GGUF`](https://huggingface.co/bartowski/Qwen_Qwen3-30B-A3B-GGUF), bartowski's Q4_K_M.
+
+```
+a015794bfb1d69cb03dbb86b185fb2b9b339f757df5f8f9dd9ebdab8f6ed5d32 Qwen_Qwen3-30B-A3B-Q4_K_M.gguf
+c2089d293f10d2e0fdabbb7ea6f17d6834f3027480de987fb6dd718e301eaa04 qwen3-30b-a3b-bq4km-flat.jns (gguf2jns)
+bee2aa5a9b5a33f19d9c153d2a0a7e8ac493a670b2d0084c195ce977cf3c83ea qwen3-30b-a3b-bq4km.jns (jns_planes)
+```
+
**Qwen3-30B-A3B in three bits** (`qwen3-30b-a3b-q3`), from [`unsloth/Qwen3-30B-A3B-GGUF`](https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF), unsloth's dynamic UD-Q3_K_XL. Converted in one step: its down matrices are Q3_K and Q4_K, so there are no planes to cut.
```
diff --git a/README.md b/README.md
index 32544d9..0b5148a 100644
--- a/README.md
+++ b/README.md
@@ -134,10 +134,12 @@ Janas reads its own format, `.jns`, converted from a **Q4_K_M** GGUF (Q3_K, IQ3_
| **Qwen3-Coder-Next** | [Qwen](https://huggingface.co/Qwen/Qwen3-Coder-Next-GGUF) | 48.4 GB | 48.4 GB | 32 GB |
| **Qwen3-Coder-30B-A3B-Instruct** — for code completion in the editor ([docs/code-completion.md](docs/code-completion.md)) | [unsloth](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF) | 18.6 GB | 18.6 GB | 32 GB, to keep it whole in memory |
| **Qwen3-30B-A3B** | [Qwen](https://huggingface.co/Qwen/Qwen3-30B-A3B-GGUF) | 18.6 GB | 18.6 GB | 16 GB |
+| **Qwen3-30B-A3B with an importance matrix** (`qwen3-30b-a3b-imatrix`) — bartowski's Q4_K_M: the nearest to Qwen's own Q8_0 of the three here (400 tokens: code 388, Italian 367, against 376 and 335 for the Q4_K_M above), 8-10% slower held in memory: it keeps more matrices in Q6_K | [bartowski](https://huggingface.co/bartowski/Qwen_Qwen3-30B-A3B-GGUF) | 18.6 GB | 18.6 GB | 16 GB |
| **Qwen3-30B-A3B in three bits** (`qwen3-30b-a3b-q3`) — unsloth's dynamic UD-Q3_K_XL: against Qwen's own Q8_0 it agrees more often than the Q4_K_M above (400 tokens: code 386 against 376, Italian 360 against 335), with 27% fewer bytes of experts; with the expert cache held at 2 GiB on this 32 GB machine, 19.8 tokens/s against 13.6; held whole in memory, as fast (32.0-32.2 tokens/s against 32.1-32.2) | [unsloth](https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF) | 13.8 GB | 13.8 GB | 8-16 GB |
| **Qwen3.5-9B** — dense; its prediction block from the checkpoint ([MODELS.md](MODELS.md#the-fingerprints)) | [unsloth](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF) | 5.7 GB | 5.7 GB | 16 GB |
| **Qwen3.5-2B** — dense, small and quick; a prediction block too | [unsloth](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) | 1.3 GB | 1.3 GB | 8 GB |
| **Qwen3-4B** — dense, and the smallest here that answers well | [Qwen](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | 2.5 GB | 2.5 GB | 8 GB |
+| **Qwen3-4B with an importance matrix** (`qwen3-4b-imatrix`) — bartowski's Q4_K_M, the same weight types and size as Qwen's: against Qwen's own Q8_0 (400 tokens) code 381 against 364, Italian 351 against 329, at the same speed | [bartowski](https://huggingface.co/bartowski/Qwen_Qwen3-4B-GGUF) | 2.5 GB | 2.5 GB | 8 GB |
| **Qwen3-0.6B** — not to talk to: to draft for a dense one (`--draft`) | [unsloth](https://huggingface.co/unsloth/Qwen3-0.6B-GGUF) | 0.40 GB | 0.40 GB | with the model above |
| **Qwen3-Embedding-0.6B** — not to talk to: for embeddings (`janas-server --embedding-model`); published in Q8_0 and converted as it is | [Qwen](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B-GGUF) | 0.64 GB | 0.64 GB | beside the chat model |
@@ -247,6 +249,9 @@ The settings come in layers, each over the one before: the model's profile in th
| Qwen3.5-2B | `--head 4` | 10.2% | 376 → 352 / 388 → 375 |
| Qwen3.5-9B | `--head 4` | 5.1% | 383 → 372 / 393 → 391 |
| Qwen3-30B-A3B | `--head 4 --attention fast --bits 4` | 10.0% | 57 → 55 of 64 / 392 → 385 |
+| Qwen3-4B with an importance matrix | `--head 4 --attention fast` (9 October) | 5.7% | 351 → 350 / 381 → 376, against Qwen's Q8_0 |
+| Qwen3-30B-A3B with an importance matrix | `--head 4` (9 October; `--bits 4` added 1.6%, `--attention fast` 0.6%) | 3.2% | 367 → 354 / 388 → 387, against Qwen's Q8_0 |
+| Qwen3-30B-A3B in three bits | `--head 4` (measured on 9 October; `--attention fast` added 1.3%) | 3.8% | 360 → 348 / 386 → 382, against Qwen's Q8_0 |
| Qwen3.6-35B-A3B | `--head 4 --bits 4` | 6.5% | 60 → 55 of 64 / 196 → 194 of 200 |
| Qwen3-Next-80B-A3B | `--head 4 --bits 4` | 8.6% | 386 → 372 / 397 → 392 |
| Qwen3-Coder-Next | `--head 4 --bits 4` | 10.5% | — / 193 → 196 of 200 |
diff --git a/src/common/catalog.c b/src/common/catalog.c
index 3d15350..dd5269d 100644
--- a/src/common/catalog.c
+++ b/src/common/catalog.c
@@ -51,6 +51,25 @@ const struct janas_model_entry janas_catalog[] = {
.ram_gb = 8,
.draft = "qwen3-0.6b",
.fast = "--head 4 --attention fast"},
+ /* the same weights types and size from bartowski, quantized with an
+ importance matrix: against Qwen's own Q8_0 (400 tokens, most likely
+ token agreeing) code 381 against 364, Italian 351 against 329, at the
+ same speed (28.1 tokens/s against 27.6, 9 Oct 2026) */
+ {.name = "qwen3-4b-imatrix",
+ .what = "Qwen3-4B, bartowski's Q4_K_M: same size, nearer the original",
+ .repo = "bartowski/Qwen_Qwen3-4B-GGUF",
+ .parts = {"Qwen_Qwen3-4B-Q4_K_M.gguf"},
+ .part_sha =
+ {"fbe1d5edd4ce802ae3ae7c7e4ab7d09789d697fdac1fc7929f8df4ca3c41bae3"},
+ .gguf_gb = 2.5,
+ .out = "qwen3-4b-bq4km.jns",
+ .flat_sha =
+ "aa4f0bc6033a206ba5ac51dcdd799299d369019d5614dc08a6baa0bda6123098",
+ .jns_sha =
+ "134ae0afbbd3e45b3cdbfd36fd205fb98c91df62a129541048e29370a42b835e",
+ .ram_gb = 8,
+ .draft = "qwen3-0.6b",
+ .fast = "--head 4 --attention fast"},
{.name = "qwen3.5-0.8b",
.what = "Qwen3.5-0.8B, dense - a test model of the Qwen3.5 architecture, "
"not for chatting",
@@ -119,7 +138,8 @@ const struct janas_model_entry janas_catalog[] = {
tokens, most likely token agreeing: code 386 against 376, Italian
360 against 335) with 27% fewer bytes of experts; with a cache of
2 GiB 19.8 tokens/s against 13.6, held whole in memory as fast
- (32.1, 9 Oct 2026). No fast profile measured yet. */
+ (32.1, 9 Oct 2026). Fast profile: --head 4, 3.8% (bench_long
+ 1900 + 128); --attention fast added 1.3%, under the 2% a lever needs. */
{.name = "qwen3-30b-a3b-q3",
.what = "Qwen3-30B-A3B, unsloth's 3-bit mix: smaller, as good",
.repo = "unsloth/Qwen3-30B-A3B-GGUF",
@@ -130,7 +150,27 @@ const struct janas_model_entry janas_catalog[] = {
.out = "qwen3-30b-a3b-udq3kxl.jns",
.flat_sha =
"7a77e62a3d6b1df321f455472e0b970a09eb3534ec76982a6e169087d4897656",
- .ram_gb = 8},
+ .ram_gb = 8,
+ .fast = "--head 4"},
+ /* bartowski's Q4_K_M, quantized with an importance matrix: the nearest
+ to Qwen's own Q8_0 of the three (400 tokens: code 388, Italian 367,
+ against 376 and 335 for Qwen's Q4_K_M and 386 and 360 for the
+ three-bit mix), 8-10% slower than Qwen's held in memory, since it
+ keeps more matrices in Q6_K (9 Oct 2026) */
+ {.name = "qwen3-30b-a3b-imatrix",
+ .what = "Qwen3-30B-A3B, bartowski's Q4_K_M, nearest to the Q8_0",
+ .repo = "bartowski/Qwen_Qwen3-30B-A3B-GGUF",
+ .parts = {"Qwen_Qwen3-30B-A3B-Q4_K_M.gguf"},
+ .part_sha =
+ {"a015794bfb1d69cb03dbb86b185fb2b9b339f757df5f8f9dd9ebdab8f6ed5d32"},
+ .gguf_gb = 18.6,
+ .out = "qwen3-30b-a3b-bq4km.jns",
+ .flat_sha =
+ "c2089d293f10d2e0fdabbb7ea6f17d6834f3027480de987fb6dd718e301eaa04",
+ .jns_sha =
+ "bee2aa5a9b5a33f19d9c153d2a0a7e8ac493a670b2d0084c195ce977cf3c83ea",
+ .ram_gb = 16,
+ .fast = "--head 4"},
{.name = "qwen3.6-35b-a3b",
.what = "Qwen3.6-35B-A3B, mixture of experts, MTP inside",
.repo = "bartowski/Qwen_Qwen3.6-35B-A3B-GGUF",