37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-10-21 (2): the drafts in the chat as in janas-bench - Qwen3-Next-80B-A3B's reply of 64 tokens with the cache warm 31.2 tok/s with 32 drafts, where it came at 24.7 with 4 (five seeds 30.9-35.3, the median 31.4): the drafts' planner no longer learns from the configurations the tuner tries, nor from the disk's waits; the drafts' 37 tok/s of 10 October with its conditions
camauri8728fbfe747f
diff --git a/ChangeLog.md b/ChangeLog.mdindex bf016e0..62a8f34 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -14,10 +14,11 @@- **A mixture's Q3_K experts go to the GPU** in the fast prompt (`q3k_mmid.comp`, one launch for a layer's experts, as Q4_K's, Q6_K's and MXFP4's), and with a whole token on the GPU (`--reply-on gpu`). Unsloth's 3-bit Qwen3-30B-A3B read a prompt of 1,024 tokens at 119.5 tok/s, where it read 98.5 with its experts on the CPU and llama.cpp reads 114.3: the one model of the README's table on which llama.cpp read faster no longer is. Given the CPU's quantized activations the new shader agrees with the CPU's products to their rounding: against llama.cpp's scores of Qwen's Q8_0 the most likely token agreed at 220 and 249 of 256 positions (Italian prose, code), against the exact way's 221 and 245.- **An expert cache reaches the GPU in regions below 4 GiB**: Mesa's Intel driver passes the size of an imported allocation on to the kernel in 32 bits, and a region of 4 GiB exactly - the second of unsloth's 3-bit Qwen3-30B-A3B, whose slots of 2,236,416 bytes fit 1,920 times 1 MB short of it - went as 0 bytes and was refused: the experts stayed on the CPU, with nothing said. None of the other mixtures measured had such a region. `JANAS_GPU_VERBOSE=1` now says why an import is refused, and where.- **The fast prompt's distance from the exact way, measured again**, on Qwen3-30B-A3B against llama.cpp's scores of Qwen's Q8_0, Italian prose and code: the most likely token within 5 of 256 positions either way (unsloth's 3-bit file 216 and 246 against the exact way's 221 and 245, Qwen's Q4_K_M 210 and 239 against 207 and 244), the mean difference of the scores within 3%. `docs/features.md` said within 2-3 positions and 2%, from the models and texts measured on 10 October; on these two texts 8 to 27 of the 256 positions change their most likely token with the rounding, and they fell as far as 5 either side.+- **The drafts in the chat as in `janas-bench`.** The planner that chooses how many of the MTP block's drafts to verify learned what a pass costs from every pass, and in a new conversation the tuner tries each configuration in turn on the passes of 2 to 16 tokens, those that verify drafts: from the slow ones it tried (279-357 ms a pass where its choice takes 50) the planner learned that the drafts did not pay, and drafted one pass in sixteen. It now learns only from the passes run on the tuner's choice, without what they waited for the disk, and learns the costs again at each turn. Qwen3-Next-80B-A3B in the chat, a reply of 64 tokens with the cache warm: 31.2 tok/s with 32 drafts, where it came at 24.7 with 4; the first reply of a session 19.3 against 16.0; over five seeds 30.9-35.3 tok/s, the median 31.4.- **The thread that keeps the NPU awake sleeps while it is not needed**: it woke every half millisecond for as long as the program ran, 2,000 times a second beside the cores writing a reply.- **`janas_aligned_alloc` asks for a size the alignment divides**, as C11 wants: ThreadSanitizer refused `test_aligned`'s one byte on eight.- **Two research probes, merged though not final**, both off unless their variables are set: Ana-Net (the experts' down matrices by neuron, `JANAS_Q6_K_PN` from `tools/jns_neurons.c`, each token's neurons read at 6, 4 or 2 bits or not at all, `JANAS_NEURON_LEVELS`; records for the predictor's tests, `JANAS_ANA_*`), and Janet's detectors in place of a model's feed-forward, layer by layer (`model_janet.c`, `JANAS_JANET_*`).-- **The README**: the banner back at the top; one table of the speeds, each engine at its best and the configuration that gave Janas-LLM its figure; the NPU, how it works, and llama.cpp's OpenVINO backend on it; the options that choose what runs where; the measurement notes of 21 October.+- **The README**: the banner back at the top; one table of the speeds, each engine at its best and the configuration that gave Janas-LLM its figure; the NPU, how it works, and llama.cpp's OpenVINO backend on it; the options that choose what runs where; the measurement notes of 21 October, and the conditions of the drafts' figures of 10 October beside the chat's of 21 October.## [2026-10-18] - Qwen3.6-35B-A3B's expert cache holds what it has room fordiff --git a/README.md b/README.mdindex e66ef04..b83b598 100644--- a/README.md+++ b/README.md@@ -35,7 +35,7 @@| Qwen3.5-0.8B | 1,403 tok/s | NPU, 6 threads | 514 tok/s | 110 tok/s | CPU, 12 threads | 31.1 tok/s || Qwen3-0.6B | 1,421 tok/s | NPU, 6 threads | 710 tok/s | 133 tok/s | CPU, 20 threads | 46.0 tok/s |-Janas-LLM's figures are the best of the configurations `janas-bench` measures, and "how" names it: the CPU's threads, with the GPU reading a prompt's blocks (its fast prompt) or the NPU reading them; a reply is written by the CPU. The configuration the engine chose by itself, as a chat runs, was within 1.5% of the best reading and within 2.6% writing, on every model. llama.cpp's are its best on this laptop, with its Vulkan backend on the same GPU; it does not use the NPU, and its OpenVINO backend, which does, is compared under [Intel's NPU](#intels-npu). † llama.cpp's figures of 11 October: the file it read is no longer on this machine. The drafts, which bring a reply sooner (37 tok/s from Qwen3-Next-80B-A3B on 10 October), and how every figure was taken: [How the figures were measured](#how-the-figures-were-measured). On another machine they will be other ones: `janas-bench` measures yours.+Janas-LLM's figures are the best of the configurations `janas-bench` measures, and "how" names it: the CPU's threads, with the GPU reading a prompt's blocks (its fast prompt) or the NPU reading them; a reply is written by the CPU. The configuration the engine chose by itself, as a chat runs, was within 1.5% of the best reading and within 2.6% writing, on every model. llama.cpp's are its best on this laptop, with its Vulkan backend on the same GPU; it does not use the NPU, and its OpenVINO backend, which does, is compared under [Intel's NPU](#intels-npu). † llama.cpp's figures of 11 October: the file it read is no longer on this machine. The drafts, which bring a reply sooner (37 tok/s from Qwen3-Next-80B-A3B in `janas-bench` on 10 October, the expert cache warm; 31-35 in the chat on 21 October), and how every figure was taken: [How the figures were measured](#how-the-figures-were-measured). On another machine they will be other ones: `janas-bench` measures yours.**New: Intel's NPU reads the prompts.** On the same laptop's NPU Janas-LLM reads 1,024 tokens of Qwen3.5-2B at 729 tok/s, of Gemma-4-E2B at 508 and of Qwen3-4B at 333, 1.5 to 2.6 times as fast as without it, with answers as close to llama.cpp's as the CPU's. It takes the NPU by itself where it measures faster, and three options choose by hand: [Intel's NPU](#intels-npu), [Choosing what runs where](#choosing-what-runs-where).@@ -148,7 +148,7 @@ The table at the top, on 21 October 2026: `janas-bench --prompt 1024 --gen 16 --Qwen3-Next-80B-A3B and Qwen3-Coder-Next are larger than the laptop's memory. llama.cpp with its usual build ran out of memory as soon as the GPU took part, even with the experts left on the CPU (`-ngl 99 -ncmoe 48`): its CPU backend copies the weights into memory of its own to repack them, and the Vulkan backend copies those it leaves to the CPU into buffers the system cannot reclaim. It ran with a build without repacking (`GGML_CPU_REPACK=OFF`) and `--no-host 1 -lm mmap`, which leave the experts in the file, read through the page cache: 17.7 tok/s reading with the GPU, 3.0 writing on the CPU alone (1.8 with the GPU; its CPU backend read at 12.6). Janas-LLM ran them with no option of its own, the experts it uses most in a cache of 21 GiB and the rest read from the disk.-**The drafts** are tokens guessed ahead that the model checks, all at once: the same text, sooner. They come from the model's own prediction block (the Qwen3.5 and Qwen3.6 models, Qwen3-Next) or from Gemma 4's assistant, a small drafting model of its own; the whole of it is under [`--mtp`](docs/chat.md). Measured on 10 October 2026 on a chat reply - an ordinary question in English ("explain how bread is made at home"), 64 tokens of the answer, sampled as the chat samples - the reply came at 37 tok/s from Qwen3-Next-80B-A3B, 34 from Qwen3.6-35B-A3B, 26 from Gemma-4-26B-A4B, 14 from Gemma-4-12B, 24 from Qwen3.5-9B, 32 from Gemma-4-E4B, 52 from Gemma-4-E2B and 67 from Qwen3.5-2B. What the drafts are worth depends on the text: there 74-89% of them were kept on the Qwen models and on Gemma's two larger ones, 63-72% on E2B and E4B, while on Italian prose Qwen3.5-2B kept about a third and gained next to nothing; and one round of 64 tokens is noisy, a tenth either way. Qwen3-30B-A3B and the dense Qwen3 have no such block (Qwen3-4B can take Qwen3-0.6B as its draft, `--draft`); the dense Qwen3.5 have one in their original checkpoint, which `hf2jns_mtp` converts ([MODELS.md](MODELS.md#the-fingerprints)).+**The drafts** are tokens guessed ahead that the model checks, all at once: the same text, sooner. They come from the model's own prediction block (the Qwen3.5 and Qwen3.6 models, Qwen3-Next) or from Gemma 4's assistant, a small drafting model of its own; the whole of it is under [`--mtp`](docs/chat.md). Measured on 10 October 2026 by `janas-bench` on a chat reply - an ordinary question in English ("explain how bread is made at home"), 64 tokens of the answer, sampled as the chat samples with one fixed seed, after a prompt of 256 tokens had warmed the expert cache, in the configuration the engine chose (on Qwen3-Next-80B-A3B 20 threads for every pass) - the reply came at 37 tok/s from Qwen3-Next-80B-A3B, 34 from Qwen3.6-35B-A3B, 26 from Gemma-4-26B-A4B, 14 from Gemma-4-12B, 24 from Qwen3.5-9B, 32 from Gemma-4-E4B, 52 from Gemma-4-E2B and 67 from Qwen3.5-2B. In the chat, on 21 October, the same question brought Qwen3-Next-80B-A3B's reply at 31-35 tok/s with the cache warm (31.4 the median of five seeds, its single tokens on the 12 threads the engine chose there) and at 16-20 tok/s as the first reply of a session, its experts still coming from the disk. What the drafts are worth depends on the text: there 74-89% of them were kept on the Qwen models and on Gemma's two larger ones, 63-72% on E2B and E4B, while on Italian prose Qwen3.5-2B kept about a third and gained next to nothing; and one round of 64 tokens is noisy, a tenth either way. Qwen3-30B-A3B and the dense Qwen3 have no such block (Qwen3-4B can take Qwen3-0.6B as its draft, `--draft`); the dense Qwen3.5 have one in their original checkpoint, which `hf2jns_mtp` converts ([MODELS.md](MODELS.md#the-fingerprints)).**The threads.** On the first passes of each kind the engine tries every core, one thread per performance core and the performance cores' threads, keeps the fastest, and tries the others now and then, so that a machine that warms up can change its verdict. Here, on 21 October 2026, it settled on all 20 usable threads for a reply's tokens on 12 of the 17 files and on the performance cores' 12 on the other five, and on 6 to 20 threads for a prompt, beside the GPU or the NPU; on an i9-14900HX every thread replied at less than half the speed of the performance cores' 16.diff --git a/src/llm/generate.c b/src/llm/generate.cindex 92ffc4c..9460aeb 100644--- a/src/llm/generate.c+++ b/src/llm/generate.c@@ -702,6 +702,10 @@ int janas_llm_session_append(struct janas_llm_session *s, const int32_t *tokens,return 0;if (n > s->cap + 1 - s->n)return -1;+ /* a new turn: the passes' costs learned again, on the cache and the+ context as they are now (the drafts' odds are kept) */+ memset(s->pl.t, 0, sizeof(s->pl.t));+ s->pl.chain = 0;memcpy(s->hist + s->n, tokens, n * sizeof(int32_t));s->n += n;s->ret = s->n;@@ -956,6 +960,17 @@ int janas_llm_session_next(struct janas_llm_session *s, int32_t *token)1) != 0)return -1;double dt = now() - tp;+ /* what the planner and spec_auto learn: the pass without its waits for+ the disk, and nothing from a pass the tuner ran on an alternative it+ was trying. In a new conversation it tries each configuration in turn+ on the passes of 2 to 16 tokens, those that verify the drafts, and+ learned from the slow ones (279-357 ms a pass where its choice takes+ 50), the planner refused the drafts: Qwen3-Next-80B-A3B wrote a reply+ of 64 tokens with 4 drafts at 24.7 token/s, where janas-bench, its+ configuration forced, wrote with drafts at 34.8 (21 Oct 2026). The+ statistics keep the time as it was. */+ double cost = janas_llm_model_pass_seconds(m);+ int learn = cost > 0;s->call_start = pos;s->st.passes++;/* accept the drafts the model would have produced, then its token; a@@ -987,9 +1002,9 @@ int janas_llm_session_next(struct janas_llm_session *s, int32_t *token)s->st.pass_count[nd + 1]++;s->st.pass_seconds[nd + 1] += dt;}- if ((plan || copies) && !settling_now) {+ if ((plan || copies) && !settling_now && learn) {double *tt = &s->pl.t[nd + 1];- *tt = *tt > 0 ? 0.8 * *tt + 0.2 * dt : dt;+ *tt = *tt > 0 ? 0.8 * *tt + 0.2 * cost : cost;}if (o->spec_mode == JANAS_SPEC_DRAFT && nd > 0)/* the drafts the target took were the drafting model's own, so its@@ -1020,8 +1035,9 @@ int janas_llm_session_next(struct janas_llm_session *s, int32_t *token)s->st.spec_passes++;s->st.drafted += (uint32_t)nd;s->st.accepted += (uint32_t)acc;- if (!settling) {- s->spec_cost = s->spec_cost ? 0.8 * s->spec_cost + 0.2 * dt : dt;+ if (!settling && learn) {+ s->spec_cost =+ s->spec_cost ? 0.8 * s->spec_cost + 0.2 * cost : cost;s->gain = s->gain ? 0.8 * s->gain + 0.2 * (acc + 1) : acc + 1;s->spec_seen++;}@@ -1042,10 +1058,11 @@ int janas_llm_session_next(struct janas_llm_session *s, int32_t *token)s->retry = AUTO_RETRY_MIN;}}- } else if (settling) {- /* nothing learned while the cache fills */+ } else if (settling || !learn) {+ /* nothing learned while the cache fills, nor from a try */} else {- s->single_cost = s->single_cost ? 0.8 * s->single_cost + 0.2 * dt : dt;+ s->single_cost =+ s->single_cost ? 0.8 * s->single_cost + 0.2 * cost : cost;s->single_at = s->st.passes;}/* the MTP rows of the new tokens always (the block's KV cache must staydiff --git a/src/llm/model.c b/src/llm/model.cindex c87e400..25ffed6 100644--- a/src/llm/model.c+++ b/src/llm/model.c@@ -402,6 +402,7 @@ int janas_llm_model_forward(struct janas_llm_model *m, const int32_t *tokens,dt -= c1.wait_seconds - c0.wait_seconds;/* nor the NPU's first compilation of a matrix: once a session */dt -= janas_llm_model_npu_compile_seconds(m) - k0;+ m->last_pass_s = dt > 0 && !m->tuner.trying ? dt : 0;/* while the cache is still filling in the background, passes are notrepresentative: neither the tuner nor the draft planner may learnfrom them, or a slow start decides the whole session */@@ -415,6 +416,11 @@ int janas_llm_model_forward(struct janas_llm_model *m, const int32_t *tokens,return r;}+double janas_llm_model_pass_seconds(const struct janas_llm_model *m)+{+ return m->last_pass_s;+}+static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,uint32_t pos0, float *logits, int all_logits){diff --git a/src/llm/model.h b/src/llm/model.hindex 7463af8..57a16a6 100644--- a/src/llm/model.h+++ b/src/llm/model.h@@ -124,6 +124,13 @@ uint64_t janas_llm_model_warm_done(const struct janas_llm_model *m);costs then says nothing about the machine, so nothing is learned from it. */int janas_llm_model_settling(const struct janas_llm_model *m);+/* The last janas_llm_model_forward's time without what it waited for+ experts read from the disk, nor the NPU's first compilations: what the+ pass costs with the experts in memory, as the tuner learns it. 0 when+ the tuner ran the pass on an alternative it was trying, not on its+ choice: what that costs says nothing of the passes to come. */+double janas_llm_model_pass_seconds(const struct janas_llm_model *m);+/* The block attention and the recurrent log work in: a pass that can betaken back (a draft) is at most this long. */#define JANAS_LLM_MAX_BLOCK 64diff --git a/src/llm/model_impl.h b/src/llm/model_impl.hindex 4fa0cab..623d9fc 100644--- a/src/llm/model_impl.h+++ b/src/llm/model_impl.h@@ -81,6 +81,7 @@ struct janas_llm_model {(and whether the GPU helps) for each kind of pass */int p_cores, p_threads, all_threads;struct janas_tuner tuner;+ double last_pass_s; /* janas_llm_model_pass_seconds */char tune_key[768];int gpu_use; /* the current pass may use the GPU */struct janas_gpu *gpu; /* optional, for products over blocks */diff --git a/src/llm/tuner.c b/src/llm/tuner.cindex 6465929..4998b17 100644--- a/src/llm/tuner.c+++ b/src/llm/tuner.c@@ -182,6 +182,7 @@ int janas_tuner_best(const struct janas_tuner *t, int cls)int janas_tuner_pick(struct janas_tuner *t, int cls){+ t->trying = 0;if (t->forced >= 0) {t->cls[cls].passes++;return t->forced;@@ -198,8 +199,10 @@ int janas_tuner_pick(struct janas_tuner *t, int cls)if (usable(t, cls, i) &&(least < 0 || t->cls[cls].tried[i] < t->cls[cls].tried[least]))least = i;- if (least >= 0 && t->cls[cls].tried[least] < TRIES)+ if (least >= 0 && t->cls[cls].tried[least] < TRIES) {+ t->trying = 1;return least;+ }/** Exploit, trying the longest untried alternative now and then, to* follow the machine's state. A try is a pass run slower on purpose, so@@ -226,6 +229,7 @@ int janas_tuner_pick(struct janas_tuner *t, int cls)if (old >= 0) {t->cls[cls].retry_best = best;t->cls[cls].retry_cand = old;+ t->trying = 1;return old;}}diff --git a/src/llm/tuner.h b/src/llm/tuner.hindex 195a71e..b6bb60b 100644--- a/src/llm/tuner.h+++ b/src/llm/tuner.h@@ -47,6 +47,8 @@ struct janas_tuner {int allow_gpu; /* candidates with the GPU may be chosen */int gpu_one; /* ... for one token too (the GPU takes single tokens) */int forced; /* >= 0: every pass uses this candidate (and is measured) */+ int trying; /* the last pick an alternative tried (exploring, or now+ and then), not the class's choice *//* the combinations of devices a class may use (common/devices.h: bitgpu | npu << 1), the user's choice; 0 any */int allowed[JANAS_TUNE_CLASSES];