37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Develop 2026-10-14 (4): an integrated GPU no longer kept awake for a reply's single tokens (issue #16)
camaurie4691cd13bf0
diff --git a/ChangeLog.md b/ChangeLog.mdindex e8937bb..373508b 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -2,6 +2,11 @@> Curated, user-facing summary of completed work, newest first.+## [2026-10-14] - An integrated GPU no longer kept awake for single tokens++- **The GPU is kept awake only for the passes it takes.** On an integrated GPU a reply's single tokens stay on the CPU, but a configuration with the GPU kept it awake for them too - an empty dispatch every half millisecond - and on a laptop the awake GPU takes the power the cores need: janas-bench's "+ GPU" rows wrote a reply slower than the CPU alone, 10.5 tokens/s against 17.3 on a Radeon 780M (issue #16, reported by dlahoza), 2-5% slower on the development laptop's Arc. Now they write as fast (Qwen3-4B, 20, 12 and 6 threads: 28.5/28.1, 28.0/28.0 and 26.3/26.3 tokens/s with "+ GPU", 28.5/28.3, 28.1/28.0 and 26.3/26.3 without, two alternated runs). A chat's own choices were not affected: the tuner gives single tokens to the GPU only where it takes them, a discrete GPU. janas-bench's note says that the single tokens are the CPU's in every row where so.++## [2026-10-14] - MXFP4 experts, and output heads in BF16- **The "MXFP4_MOE" files run.** Their routed experts are MXFP4 (OCP's microscaling format: 32 four-bit values and a power of two a block), which gguf2jns refused and the engine could not read. The engine has MXFP4 products now - IQ4_NL's kernel with the block's power of two for a scale, the same bits on AVX2 and AVX-VNNI, for one vector or eight, and the same bits as the exact Q8_0 rewrite of the same blocks - and gguf2jns keeps such experts as they are. noctrex's Qwen3.6-35B-A3B-MXFP4_MOE against llama.cpp on the same file: 199 of 200 most likely tokens agree, mean logit difference 0.105 (on Italian prose 63 of 64, 0.085), closer than bartowski's Q4_K_M to its own llama.cpp (196, 0.146; 60, 0.135). The file itself, against llama.cpp's Q8_0 of the same model, is further than that Q4_K_M (378 against 387 of 400 most likely tokens, 364 against 370 on Italian prose) and 9% smaller (20.3 GB against 22.3). On 12 threads of the development laptop the MXFP4 product reads 66 GB/s where IQ4_NL reads 73: 4% fewer weights a second, a block of it being 17 bytes to IQ4_NL's 18 (its scales come from a table: computed in the kernel, they took registers the unpacking needs and it ran 14% slower).diff --git a/docs/hardware.md b/docs/hardware.mdindex 5f12493..fb0dc84 100644--- a/docs/hardware.md+++ b/docs/hardware.md@@ -47,7 +47,7 @@ The [hardware report form](https://github.com/prabanta-dev/janas/issues/new?temp5. **Windows on other machines.** The Windows build has run on one laptop, the development one (Windows 11, build 26300), where its tests pass and it is as fast as on Linux; `janas-bench`'s report from any other is welcome.6. **Other models** of the supported families. A file that will not convert, or converts and then answers nonsense, is a useful bug.-**What testers found so far.** On a Radeon 780M (Ryzen 7 255, issue #16) the GPU read Qwen3-4B's prompt 5.2 times faster than the CPU alone (477 against 92 tokens/s, the second with `JANAS_GPU_FAST_PROMPT=0`), and subgroups of 32 (`JANAS_GPU_SG=32`) instead of the 64 the engine chose cost 14% (405 against 470 tokens/s): the default is right there. On a Quadro RTX 5000 (issue #17) the defect above.+**What testers found so far.** On a Radeon 780M (Ryzen 7 255, issue #16) the GPU read Qwen3-4B's prompt 5.2 times faster than the CPU alone (477 against 92 tokens/s, the second with `JANAS_GPU_FAST_PROMPT=0`), and subgroups of 32 (`JANAS_GPU_SG=32`) instead of the 64 the engine chose cost 14% (405 against 470 tokens/s): the default is right there. Its "+ GPU" rows wrote a reply at 10.5 tokens/s against 17.3 on the CPU alone: the GPU was kept awake for the single tokens of a reply, which an integrated GPU does not take, and awake it only took the power the cores need. Since 14 October 2026 it is kept awake only for the passes it takes; on the development laptop's Arc the "+ GPU" rows wrote 2-5% slower than the CPU alone before, and as fast after (Qwen3-4B, two alternated runs each). On a Quadro RTX 5000 (issue #17) the defect above.**What not to bother with yet:** macOS (not supported), and models outside the Qwen3, Qwen3.5, Qwen3.6, Qwen3-Next and Gemma 4 families (they are refused with a message).diff --git a/src/bench/main.c b/src/bench/main.cindex e31f3da..58fbc83 100644--- a/src/bench/main.c+++ b/src/bench/main.c@@ -448,8 +448,12 @@ int main(int argc, char **argv)if (cpu_rows && janas_llm_model_gpu_prompts(m))out("note: in every row the GPU reads a prompt's blocks of %d ""tokens or more (the fast prompt); \"+ GPU\" also gives it a "- "part in the shorter passes\n",- JANAS_GPU_FAST_N);+ "part in the shorter passes%s\n",+ JANAS_GPU_FAST_N,+ gpu && janas_gpu_min_n(gpu) > 1+ ? ", from 2 tokens on: a reply's single tokens are the "+ "CPU's in every row"+ : "");janas_llm_model_force(m, -1);char tune[512];int32_t tn;diff --git a/src/llm/model.c b/src/llm/model.cindex 9fbe267..ae8a8b7 100644--- a/src/llm/model.c+++ b/src/llm/model.c@@ -389,7 +389,7 @@ int janas_llm_model_forward(struct janas_llm_model *m, const int32_t *tokens,/* each pass measured for the tuner: the configuration never changesthe results */int cls = janas_tune_class(n), cand = janas_tuner_pick(&m->tuner, cls);- janas_m_apply_candidate(m, cand);+ janas_m_apply_candidate(m, cand, n);/* measured without the time spent waiting for experts read from disk:it depends on the cache, not on the configuration */struct janas_expert_cache_stats c0, c1;@@ -677,8 +677,8 @@ int janas_llm_mtp_draft(struct janas_llm_model *m, const int32_t *tokens,}if (janas_llm_mtp_forward(m, tokens, hidden, n, pos0, NULL, 0) != 0)return -1;- janas_m_apply_candidate(m,- janas_tuner_best(&m->tuner, 0)); /* one-row head */+ janas_m_apply_candidate(m, janas_tuner_best(&m->tuner, 0),+ 1); /* one-row head */double td = now();size_t nbd = m->d_model / JANAS_QK;struct janas_matvec_task t = {.type = (int)m->output->type,@@ -721,8 +721,8 @@ int janas_llm_mtp_forward(struct janas_llm_model *m, const int32_t *tokens,for (uint32_t j = 0; j < n; j++)if (tokens[j] < 0 || (uint32_t)tokens[j] >= m->n_vocab)return -1;- janas_m_apply_candidate(m,- janas_tuner_best(&m->tuner, janas_tune_class(n)));+ janas_m_apply_candidate(m, janas_tuner_best(&m->tuner, janas_tune_class(n)),+ n);uint32_t dm = m->d_model;size_t nbd = dm / JANAS_QK, nbc = 2 * dm / JANAS_QK;const struct layer *ly = &m->mtp_layer;diff --git a/src/llm/model_gpu.c b/src/llm/model_gpu.cindex 0ca9bce..5d6871a 100644--- a/src/llm/model_gpu.c+++ b/src/llm/model_gpu.c@@ -486,6 +486,16 @@ uint32_t janas_llm_model_pass_cap(const struct janas_llm_model *m)return (uint32_t)janas_gpu_few_max(m->gpu);}+int janas_m_gpu_token_asked(void)+{+ static int asked = -1;+ if (asked < 0) {+ const char *e = getenv("JANAS_GPU_TOKEN");+ asked = e && atoi(e) > 0;+ }+ return asked;+}+int janas_m_gpu_token(struct janas_llm_model *m, uint32_t pos, uint32_t n,int all, float *logits){diff --git a/src/llm/model_impl.h b/src/llm/model_impl.hindex bfcabcd..da044b8 100644--- a/src/llm/model_impl.h+++ b/src/llm/model_impl.h@@ -396,7 +396,9 @@ void janas_m_arena_to_gpu(struct janas_llm_model *m);the GPU failed after replaying on the base in place, the state lost. */int janas_m_gpu_token(struct janas_llm_model *m, uint32_t pos, uint32_t n,int all, float *logits);-void janas_m_apply_candidate(struct janas_llm_model *m, int cand);+void janas_m_apply_candidate(struct janas_llm_model *m, int cand, uint32_t nv);+/* Whether the whole token is asked of the GPU (JANAS_GPU_TOKEN=1). */+int janas_m_gpu_token_asked(void);uint32_t *janas_m_profile_read(const struct janas_llm_model *m, size_t *used);void janas_m_profile_write(const struct janas_llm_model *m);void janas_m_norm_quant(struct janas_llm_model *m, const float *w,diff --git a/src/llm/model_tune.c b/src/llm/model_tune.cindex e5918bf..4727b01 100644--- a/src/llm/model_tune.c+++ b/src/llm/model_tune.c@@ -116,8 +116,14 @@ void janas_m_init_tuner(struct janas_llm_model *m)janas_tuner_load(&m->tuner, m->tune_key);}-/* Threads and GPU of a candidate, for the passes that follow. */-void janas_m_apply_candidate(struct janas_llm_model *m, int cand)+/* Threads and GPU of a candidate, for the passes that follow, of nv+ vectors. The GPU is kept awake only where it takes such a pass: on an+ integrated GPU a single token's products stay on the CPU+ (janas_gpu_min_n 2) unless the whole token is asked of it+ (JANAS_GPU_TOKEN=1), and keeping it awake there only took the power the+ cores need - a Radeon 780M wrote at 10.5 tokens/s in janas-bench's+ "+ GPU" rows against 17.3 on the CPU alone (issue #16). */+void janas_m_apply_candidate(struct janas_llm_model *m, int cand, uint32_t nv){/* a borrowed pool belongs to the model that made it: a drafting modelsetting its own thread count would set the target's too */@@ -125,7 +131,9 @@ void janas_m_apply_candidate(struct janas_llm_model *m, int cand)janas_pool_set_active(m->compute, m->tuner.cand[cand].threads);m->gpu_use = m->tuner.cand[cand].gpu;if (m->gpu) {- janas_gpu_keep_awake(m->gpu, m->gpu_use);+ janas_gpu_keep_awake(m->gpu,+ m->gpu_use && (nv >= janas_gpu_min_n(m->gpu) ||+ janas_m_gpu_token_asked()));janas_gpu_set_only_fast(m->gpu, !m->gpu_use);}}