37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-10-11 (6): the measures of 11 October - Qwen3-Coder-Next 66 against 14 tok/s reading, Qwen3.6-35B-A3B 104 against 79 - and the test machine's operating system
camauri3d6481665619
diff --git a/README.md b/README.mdindex 3488f65..4d15097 100644--- a/README.md+++ b/README.md@@ -2,12 +2,13 @@[]() []() [](https://www.linkedin.com/in/maurizio-cammalleri-80a89a11/) [](https://cammalleri.substack.com/)-**37 tok/s from Qwen3-Next-80B (a 48 GB model) on a laptop with 32 GB of memory, 24 without drafts, where llama.cpp manages 2.8; reading a prompt, 62 tok/s against 16.** Faster than llama.cpp on every model compared on the same laptop (Intel Core Ultra 9 185H, its integrated Arc GPU, 32 GB, an NVMe SSD; 9-10 October 2026):+**37 tok/s from Qwen3-Next-80B (a 48 GB model) on a laptop with 32 GB of memory, 24 without drafts, where llama.cpp manages 2.8; reading a prompt, 62 tok/s against 16.** Janas-LLM was faster than llama.cpp on every model compared on the same laptop (Debian 13 Linux, Intel Core Ultra 9 185H, its integrated Arc GPU, 32 GB, an NVMe SSD; 9-11 October 2026; the whole machine [below](#the-machine-it-grew-on)):-| Model | reading 1,024 tokens: Janas | llama.cpp | writing: Janas | llama.cpp |+| Model | reading 1,024 tokens: Janas-LLM | llama.cpp | writing: Janas-LLM | llama.cpp ||---|---:|---:|---:|---:|| Qwen3-Next-80B-A3B (48.4 GB, more than the memory) | 62 tok/s | 16 tok/s | 24 tok/s | 2.8 tok/s |-| Qwen3.6-35B-A3B | 88 tok/s | 78 tok/s | 20 tok/s | 11 tok/s |+| Qwen3-Coder-Next (48.4 GB, more than the memory) | 66 tok/s | 14 tok/s | 22 tok/s | 2.2 tok/s |+| Qwen3.6-35B-A3B | 104 tok/s | 79 tok/s | 19 tok/s | 12 tok/s || Qwen3-30B-A3B | 135 tok/s | 122 tok/s | 32 tok/s | 15 tok/s || Gemma-4-26B-A4B-it | 84 tok/s | 76 tok/s | 18 tok/s | 12 tok/s || Gemma-4-12B-it | 65 tok/s | 46 tok/s | 9.9 tok/s | 7.1 tok/s |@@ -16,7 +17,18 @@| Qwen3-4B | 193 tok/s | 164 tok/s | 27 tok/s | 18 tok/s || Gemma-4-E2B-it | 191 tok/s | 150 tok/s | 38 tok/s | 19 tok/s || Qwen3.5-2B | 441 tok/s | 323 tok/s | 52 tok/s | 28 tok/s |-| Qwen3-0.6B (one round) | 938 tok/s | 687 tok/s | 131 tok/s | 44 tok/s |+| Qwen3.5-0.8B | 799 tok/s | 519 tok/s | 109 tok/s | 32 tok/s |+| Qwen3-0.6B | 939 tok/s | 670 tok/s | 130 tok/s | 45 tok/s |++The same models from other hands, in the catalog beside them:++| Model | reading 1,024 tokens: Janas-LLM | llama.cpp | writing: Janas-LLM | llama.cpp |+|---|---:|---:|---:|---:|+| Qwen3-30B-A3B, bartowski's Q4_K_M (importance matrix) | 131 tok/s | 122 tok/s | 32 tok/s | 16 tok/s |+| Qwen3-30B-A3B, unsloth's UD-Q3_K_XL (3 bits) | 100 tok/s | **112 tok/s** | 36 tok/s | 11 tok/s |+| Qwen3-4B, bartowski's Q4_K_M (importance matrix) | 191 tok/s | 160 tok/s | 26 tok/s | 18 tok/s |++One exception: the 3-bit Qwen3-30B reads a prompt slower than llama.cpp, since its Q3_K weights do not go to the GPU's fast prompt yet; it writes three times as fast.How they were measured, and the speeds with the drafts: [What works now](#what-works-now). On another machine the figures will be other ones: `janas-bench` measures yours.@@ -64,7 +76,7 @@ If something goes wrong, `Ctrl-C` stops a reply and `Ctrl-D` leaves the chat; `-Everything in this repository was written and measured on:-- **Linux Debian 13**, x86-64+- **Linux Debian 13** (trixie), x86-64, kernel 6.12.107; GPU driver Mesa 26.1.6 (Vulkan); built with GCC 14.2- **Intel Core Ultra 9 185H** (6 performance cores, 8 efficiency, 2 low-power; AVX2, FMA, F16C, AVX-VNNI; integrated Arc GPU)- **32 GB** of DDR5 RAM- **1 TB NVMe SSD** (about 5.7 GB/s reading)@@ -120,7 +132,7 @@ The last column is a draft guessing the next tokens and the model checking them,**This column was wrong until 26 September 2026**, and higher: it measured the block continuing a paragraph that repeated itself, which a model copies and the block guesses almost every time - 107 tok/s for Qwen3.5-2B, 26 for Qwen3.5-9B. The MoE figures of the time (29 and 30) happened to be lower than today's; the ChangeLog has the details.-The comparison with llama.cpp at the top of this page was measured so: llama.cpp (build `2b18470` of 18 September 2026) with its Vulkan backend on the same integrated GPU, everything on it (`llama-bench -p 1024 -n 16 -ngl 99 -t 12`; the largest model otherwise, below), against `janas-bench --prompt 1024 --gen 16` on the same files the same days (9 and 10 October 2026; llama.cpp's best configuration, Janas's own choice; Qwen3-0.6B in one round).+The comparison with llama.cpp at the top of this page was measured so: llama.cpp (build `2b18470` of 18 September 2026) with its Vulkan backend on the same integrated GPU, everything on it (`llama-bench -p 1024 -n 16 -ngl 99 -t 12`; the largest model otherwise, below), against `janas-bench --prompt 1024 --gen 16` on the same files the same days (9-11 October 2026; llama.cpp's best configuration, Janas-LLM's own choice; on 11 October three rounds each, `llama-bench -r 3` and `janas-bench --rounds 3`, for Qwen3-0.6B, Qwen3.5-0.8B, Qwen3.6-35B-A3B and the models from other hands; Qwen3-Coder-Next as Qwen3-Next-80B, below).On Qwen3-Next-80B-A3B, whose file is larger than this laptop's 30 GB, llama.cpp with its usual build ran out of memory as soon as the GPU took part, even with the experts left on the CPU (`-ngl 99 -ncmoe 48`): its CPU backend copies the weights into memory of its own to repack them, and the Vulkan backend copies those it leaves to the CPU into buffers the system cannot reclaim. It ran with a build without repacking (`GGML_CPU_REPACK=OFF`) and two options more, `--no-host 1 -lm mmap`, which leave the experts in the file, read through the page cache: 16 tok/s reading with the GPU, and 2.8 writing on the CPU alone (with the GPU, 2.0); these are its two figures in the table. Janas ran the same model with no option of its own, the experts it uses most in a cache of about 19 GB and the rest read from the disk. Janas's figures are one round each; the writing column is without drafts on both sides. The two tools do not measure in exactly the same way (`llama-bench` writes without a prompt before it; `janas-bench` after the 1,024 tokens), and it is one machine. llama.cpp's own CPU backend did worse on this laptop than its Vulkan one at reading (Qwen3-4B 70 tok/s on 20 threads) and as well at writing.diff --git a/dist/README.md b/dist/README.mdindex 0a4e5f2..e2a232a 100644--- a/dist/README.md+++ b/dist/README.md@@ -1,24 +1,26 @@# Janas for Linux x86-64-**Qwen3-Next-80B, a 48 GB model, on a laptop with 32 GB of memory: 37 tok/s with its own draft block, 24 without, where llama.cpp manages 2.8 on the same laptop.** Reading a prompt of 1,024 tokens: 62 tok/s against 16. Janas was faster than llama.cpp on every model compared on that laptop (Intel Core Ultra 9 185H, its integrated Arc GPU, 32 GB of memory, an NVMe SSD), measured on 9 and 10 October 2026:+**Qwen3-Next-80B, a 48 GB model, on a laptop with 32 GB of memory: 37 tok/s with its own draft block, 24 without, where llama.cpp manages 2.8 on the same laptop.** Reading a prompt of 1,024 tokens: 62 tok/s against 16. Janas-LLM was faster than llama.cpp on every model compared on that laptop (Debian 13 Linux with kernel 6.12.107, Intel Core Ultra 9 185H, its integrated Arc GPU with Mesa 26.1.6, 32 GB of DDR5-5600 memory, an NVMe SSD), measured on 9-11 October 2026:-| Model | reading a prompt: Janas | llama.cpp | writing: Janas | llama.cpp |+| Model | reading 1,024 tokens: Janas-LLM | llama.cpp | writing: Janas-LLM | llama.cpp ||---|---:|---:|---:|---:|-| Qwen3-0.6B | 938 tok/s | 687 tok/s | 131 tok/s | 44 tok/s |-| Qwen3.5-2B | 441 tok/s | 323 tok/s | 52 tok/s | 28 tok/s |-| Qwen3-4B | 193 tok/s | 164 tok/s | 27 tok/s | 18 tok/s |-| Qwen3.5-9B | 111 tok/s | 78 tok/s | 14 tok/s | 8.7 tok/s |-| Qwen3-30B-A3B | 135 tok/s | 122 tok/s | 32 tok/s | 15 tok/s |-| Qwen3.6-35B-A3B | 88 tok/s | 78 tok/s | 20 tok/s | 11 tok/s || Qwen3-Next-80B-A3B (48.4 GB, more than the memory) | 62 tok/s | 16 tok/s | 24 tok/s | 2.8 tok/s |-| Gemma-4-E2B-it | 191 tok/s | 150 tok/s | 38 tok/s | 19 tok/s |-| Gemma-4-E4B-it | 122 tok/s | 109 tok/s | 20 tok/s | 15 tok/s |-| Gemma-4-12B-it | 65 tok/s | 46 tok/s | 9.9 tok/s | 7.1 tok/s |+| Qwen3-Coder-Next (48.4 GB, more than the memory) | 66 tok/s | 14 tok/s | 22 tok/s | 2.2 tok/s |+| Qwen3.6-35B-A3B | 104 tok/s | 79 tok/s | 19 tok/s | 12 tok/s |+| Qwen3-30B-A3B | 135 tok/s | 122 tok/s | 32 tok/s | 15 tok/s || Gemma-4-26B-A4B-it | 84 tok/s | 76 tok/s | 18 tok/s | 12 tok/s |+| Gemma-4-12B-it | 65 tok/s | 46 tok/s | 9.9 tok/s | 7.1 tok/s |+| Qwen3.5-9B | 111 tok/s | 78 tok/s | 14 tok/s | 8.7 tok/s |+| Gemma-4-E4B-it | 122 tok/s | 109 tok/s | 20 tok/s | 15 tok/s |+| Qwen3-4B | 193 tok/s | 164 tok/s | 27 tok/s | 18 tok/s |+| Gemma-4-E2B-it | 191 tok/s | 150 tok/s | 38 tok/s | 19 tok/s |+| Qwen3.5-2B | 441 tok/s | 323 tok/s | 52 tok/s | 28 tok/s |+| Qwen3.5-0.8B | 799 tok/s | 519 tok/s | 109 tok/s | 32 tok/s |+| Qwen3-0.6B | 939 tok/s | 670 tok/s | 130 tok/s | 45 tok/s |-Qwen3-0.6B in one round, the others as the project's README gives them. On another machine the figures will be other ones: `janas-bench` measures yours (below). How they were taken, and everything else: <https://github.com/prabanta-dev/janas>.+The 3-bit Qwen3-30B of the catalog is the one exception: it reads a prompt slower than llama.cpp (100 tok/s against 112), its Q3_K weights not on the GPU's fast prompt yet. On another machine the figures will be other ones: `janas-bench` measures yours (below). How they were taken, and everything else: <https://github.com/prabanta-dev/janas>.-Janas runs large language models on ordinary computers, with or without a GPU, mixtures of experts larger than the memory among them: the weights stay on disk, the experts each token needs are read from the SSD and the ones that keep coming back stay in memory. It is written in C and needs nothing but the system's C library; a GPU is used through its Vulkan driver when there is one.+Janas-LLM, the inference engine of Janas, runs large language models on ordinary computers, with or without a GPU, mixtures of experts larger than the memory among them: the weights stay on disk, the experts each token needs are read from the SSD and the ones that keep coming back stay in memory. It is written in C and needs nothing but the system's C library; a GPU is used through its Vulkan driver when there is one.## Start