37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-09-27: the threads chosen per machine, a steadier prompt ETA
camauri5d2388368419
diff --git a/ChangeLog.md b/ChangeLog.mdindex e6939ed..ff9b826 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -2,6 +2,27 @@> Curated, user-facing summary of completed work, newest first.+## [2026-09-27] - The threads chosen per machine++- **The time left to read a long prompt no longer jumps at its start.**+ The fitted line of issue #9 rested on two blocks when first used, and+ their noise could tilt it: an estimate many times too long at 1,024+ tokens. It now moves from the latest speed to the fitted line over the+ first sixteen blocks.++- **The engine chooses its threads again, once per machine and model.**+ Since 26 September it always used every usable thread, which was the+ fastest on the development laptop; on an i9-14900HX (issue #13) every+ thread replied at 9.9 tokens/s where the performance cores' 16 gave+ 24.7, and on a 9950X3D one thread per core was 4-15% ahead (#15). The+ first passes of each kind now try every thread, one per performance core+ and the performance cores' threads; the threads of the fastest are kept+ for good, saved with the machine's profile, and only whether the GPU+ takes part keeps being tried. Tried again later, at a longer context,+ the thread counts had lost unfairly to costs measured at a short one,+ which is why they had been fixed. `JANAS_TUNE_THREADS=0` keeps every+ thread.+## [2026-09-27] - Gemma 4's assistant drafts- **Gemma 4's assistant is taken as the MTP file** (`--mtp`, converted bydiff --git a/INSTALL.md b/INSTALL.mdindex 7f33108..bd58c3f 100644--- a/INSTALL.md+++ b/INSTALL.md@@ -87,4 +87,5 @@ It takes a few minutes: it measures prompt processing and generation speed for e- `JANAS_GPU=0` turns the GPU off; `JANAS_GPU_DEVICE=integrated` or `discrete` picks one when there are both.- `JANAS_KERNELS=avx2` uses the AVX2 kernels even where AVX-VNNI is available.- `JANAS_TUNE=0` turns self-tuning off (fixed defaults); delete `~/.cache/janas/tuning.txt` to start tuning from scratch.+- `JANAS_TUNE_THREADS=0` keeps every usable thread instead of letting the first passes choose between every thread, one per performance core and the performance cores' threads; the choice, once made, is kept for that machine and model.- The expert cache takes the free memory, leaving a fifth of the machine's memory to the rest of the system; `--cache <GiB>` in `janas-chat` sets it.diff --git a/README.md b/README.mdindex 0b3c966..0400815 100644--- a/README.md+++ b/README.md@@ -144,9 +144,13 @@ On the machine above, with nothing else running, `janas-bench` reports this:Every figure is the **mean of three rounds**, not a best round, of theconfiguration the engine itself chose for each kind of pass - the one a chat-runs: all the usable cores (performance and efficiency cores, not the-low-power ones), with the integrated GPU only where it helped, which was-reading prompts on the two largest mixtures. The rounds run in alternating+runs: here all the usable cores (performance and efficiency cores, not+the low-power ones), with the integrated GPU only where it helped, which+was reading prompts on the two largest mixtures. On another machine the+engine may settle on fewer threads: the first passes of each kind try+every core, one thread per performance core and the performance cores'+threads, and keep the fastest (on an i9-14900HX every thread replied at+less than half the speed of the performance cores' 16). The rounds run in alternatingorder so that the machine warming up and the cache filling weigh on eachsetting alike. Expect a few per cent either way between runs, and ratherless than these on a first run, while the cache is still filling.diff --git a/src/llm/api_reply.c b/src/llm/api_reply.cindex 0143805..ffbeb72 100644--- a/src/llm/api_reply.c+++ b/src/llm/api_reply.c@@ -52,8 +52,12 @@ static double cpu_now(void)* 260,954-token prompt of Qwen3-Next-80B the elapsed time followed a n +* b n^2 to R^2 0.99999 (issue #9), where the latest speed promised 2.5* times too little at 32K. Never below what the last block's speed says:- * the cost does not fall as the context grows.+ * the cost does not fall as the context grows. A line through the first+ * few blocks tilts with their noise (an ETA many times too long at 1,024+ * tokens, issue #9), so the estimate moves from the last block's speed to+ * the fitted line over the first ETA_BLOCKS blocks fitted.*/+#define ETA_BLOCKS 16static double input_eta(const janas_llm_chat *c, uint32_t left){if (left == 0)@@ -68,7 +72,10 @@ static double input_eta(const janas_llm_chat *c, uint32_t left)double a = (c->eta_sc - b * c->eta_sx) / w;double p0 = janas_llm_session_computed(c->s), p1 = p0 + left;double fit = a * (p1 - p0) + 0.5 * b * (p1 * p1 - p0 * p0);- return fit > flat ? fit : flat;+ if (fit <= flat)+ return flat;+ double k = (double)(c->eta_steps - 2) / ETA_BLOCKS;+ return flat + (k < 1 ? k : 1) * (fit - flat);}/* One block of the prompt read: n tokens from position p0, in dt seconds.diff --git a/src/llm/model_tune.c b/src/llm/model_tune.cindex 4989cdc..4a78445 100644--- a/src/llm/model_tune.c+++ b/src/llm/model_tune.c@@ -29,18 +29,19 @@/** The tuner's candidates: every usable core - the performance cores with* all their threads and the efficiency cores, not the low-power island,- * which the layout leaves out - with and without the GPU if there is one.- * Measured on 26 Sep 2026 on six models (Qwen3.5-2B to Qwen3-Next-80B),- * that was the fastest or level in every kind of pass; one thread per- * performance core, or their threads alone, lost 5-20% writing. Trying- * them in use only made a chat run on fewer cores than janas-bench had- * reported, whenever a stretch of long context had tipped the averages:- * the same machine, two speeds. So the threads are fixed, and the tuner- * only decides whether the GPU helps. JANAS_TUNE_THREADS=1 brings back- * those candidates (one per performance core, all their threads), to- * experiment with. JANAS_TUNE=0 turns tuning off; JANAS_BLOCK_THREADS fixes- * the threads for blocks (and turns it off too); JANAS_GPU_ALWAYS=1 makes- * the GPU variants the defaults for several tokens.+ * which the layout leaves out - one thread per performance core, and the+ * performance cores' threads alone; each with and without the GPU if there+ * is one. On the development laptop (6 performance cores, 8 efficiency)+ * every core was the fastest or level on six models; on an i9-14900HX (8+ * and 16) every thread replied at 9.9 tokens/s where 16 gave 24.7 (issue+ * #13), and on a 9950X3D one thread per core was 4-15% ahead (#15). So the+ * first passes of each kind try them all and the tuner keeps the threads+ * of the fastest (tuner.h: settled once, never tried again at a longer+ * context, which had moved chats onto fewer cores than they could use).+ * JANAS_TUNE_THREADS=0 keeps every core without trying the others;+ * JANAS_TUNE=0 turns tuning off; JANAS_BLOCK_THREADS fixes the threads for+ * blocks (and turns it off too); JANAS_GPU_ALWAYS=1 makes the GPU variants+ * the defaults for several tokens.*/void janas_m_init_tuner(struct janas_llm_model *m){@@ -48,7 +49,7 @@ void janas_m_init_tuner(struct janas_llm_model *m)int n = 0, opts[3] = {m->all_threads, m->p_cores, m->p_threads};const char *bt = getenv("JANAS_BLOCK_THREADS"), *tn = getenv("JANAS_TUNE"),*tt = getenv("JANAS_TUNE_THREADS");- int n_opts = tt && atoi(tt) > 0 ? 3 : 1;+ int n_opts = tt && atoi(tt) == 0 ? 1 : 3;int block = bt && atoi(bt) > 0 ? atoi(bt) : m->all_threads;if (block > m->all_threads)block = m->all_threads;diff --git a/src/llm/tuner.c b/src/llm/tuner.cindex e844c9e..858864b 100644--- a/src/llm/tuner.c+++ b/src/llm/tuner.c@@ -45,7 +45,32 @@ void janas_tuner_init(struct janas_tuner *t, const struct janas_tune_cand *c,power its keep-alive costs) */static int usable(const struct janas_tuner *t, int cls, int i){- return !t->cand[i].gpu || (t->allow_gpu && cls > 0);+ return (!t->cand[i].gpu || (t->allow_gpu && cls > 0)) &&+ (!t->cls[cls].threads || t->cand[i].threads == t->cls[cls].threads);+}++/* The default of a class: its own, or once the threads are settled the+ candidate with those threads and no GPU */+static int class_def(const struct janas_tuner *t, int cls)+{+ int def = t->cls[cls].def;+ if (!t->cls[cls].threads || t->cand[def].threads == t->cls[cls].threads)+ return def;+ for (int i = 0; i < t->n_cand; i++)+ if (!t->cand[i].gpu && t->cand[i].threads == t->cls[cls].threads)+ return i;+ return def;+}++/* Once every candidate has its tries, the threads of the best are kept */+static void settle_threads(struct janas_tuner *t, int cls)+{+ if (t->cls[cls].threads)+ return;+ for (int i = 0; i < t->n_cand; i++)+ if (usable(t, cls, i) && t->cls[cls].tried[i] < TRIES)+ return;+ t->cls[cls].threads = t->cand[janas_tuner_best(t, cls)].threads;}void janas_tuner_force(struct janas_tuner *t, int cand)@@ -62,7 +87,7 @@ int janas_tuner_best(const struct janas_tuner *t, int cls){/* the default unless an alternative is clearly faster: measurementsare noisy, a near tie is not worth a change */- int def = t->cls[cls].def, best = def;+ int def = class_def(t, cls), best = def;double bar =t->cls[cls].tried[def] >= TRIES ? t->cls[cls].cost[def] * MARGIN : 1e30;for (int i = 0; i < t->n_cand; i++)@@ -84,7 +109,8 @@ int janas_tuner_pick(struct janas_tuner *t, int cls)return t->cls[cls].def;uint32_t p = t->cls[cls].passes++;if (p < WARM_PASSES)- return t->cls[cls].def;+ return class_def(t, cls);+ settle_threads(t, cls);/* explore: the least tried candidate until each has TRIES */int least = -1;for (int i = 0; i < t->n_cand; i++)@@ -205,6 +231,23 @@ void janas_tuner_load(struct janas_tuner *t, const char *key)t->cls[cls].tried[cand] = tries;}fclose(f);+ /* the threads settled: a line of their own, candidate -1 */+ if (!(f = fopen(path, "r")))+ return;+ while (fgets(line, sizeof(line), f)) {+ int cls, cand;+ double threads;+ unsigned zero;+ if (strncmp(line, key, kl) != 0 || line[kl] != '\t' ||+ sscanf(line + kl + 1, "%d\t%d\t%lf\t%u", &cls, &cand, &threads,+ &zero) != 4 ||+ cls < 0 || cls >= JANAS_TUNE_CLASSES || cand != -1)+ continue;+ for (int i = 0; i < t->n_cand; i++)+ if (t->cand[i].threads == (int)threads)+ t->cls[cls].threads = (int)threads;+ }+ fclose(f);}void janas_tuner_save(const struct janas_tuner *t, const char *key)@@ -229,6 +272,9 @@ void janas_tuner_save(const struct janas_tuner *t, const char *key)if (t->cls[k].tried[i] > 0)fprintf(out, "%s\t%d\t%d\t%.9g\t%u\n", key, k, i,t->cls[k].cost[i], t->cls[k].tried[i]);+ for (int k = 0; k < JANAS_TUNE_CLASSES; k++)+ if (t->cls[k].threads)+ fprintf(out, "%s\t%d\t-1\t%d\t0\n", key, k, t->cls[k].threads);if (fclose(out) == 0)rename(tmp, path);elsediff --git a/src/llm/tuner.h b/src/llm/tuner.hindex c336ff8..2852a3e 100644--- a/src/llm/tuner.h+++ b/src/llm/tuner.h@@ -8,8 +8,12 @@* runs anyway: for each kind of pass (one token, a few, a block) it tries* each candidate a few times, keeps the fastest, and now and then tries the* others again to follow the machine's state (heat, power source). The- * choices are saved per machine and model, so the next start begins from- * them.+ * number of threads is settled once, by the first tries, and then kept:+ * a pass costs more as the context grows, so a candidate tried again at+ * long context against costs measured at short context loses unfairly -+ * which moved chats onto fewer cores than they could use. Only whether the+ * GPU takes part keeps being tried. The choices are saved per machine and+ * model, so the next start begins from them.*/#ifndef JANAS_LLM_TUNER_H#define JANAS_LLM_TUNER_H@@ -42,6 +46,7 @@ struct janas_tuner {pending): a try that changes nothing doubles the gap */uint32_t retry_at, retry_gap;int retry_best, retry_cand; /* and the candidate tried */+ int threads; /* settled by the first tries (0: not yet) */} cls[JANAS_TUNE_CLASSES];};diff --git a/tests/test_tuner.c b/tests/test_tuner.cindex 40f1250..d2ff40f 100644--- a/tests/test_tuner.c+++ b/tests/test_tuner.c@@ -96,6 +96,8 @@ int main(void)janas_tuner_load(&u, "test|key");CHECK(janas_tuner_best(&u, 2) == 3, "loaded best %d, want 3",janas_tuner_best(&u, 2));+ CHECK(u.cls[2].threads == 20 && u.cls[2].threads == t.cls[2].threads,+ "loaded threads %d, want 20", u.cls[2].threads);struct janas_tuner v;janas_tuner_init(&v, cand, 4, def, 0);janas_tuner_load(&v, "other|key");@@ -122,15 +124,33 @@ int main(void)}CHECK(tries >= 5 && tries <= 30, "%d tries in 20000 passes", tries);tries = 0;- for (int i = 0; i < 6000; i++) { /* the machine changes: 1 is best now */+ for (int i = 0; i < 6000; i++) { /* the machine changes: the GPU wins */int c = janas_tuner_pick(&t, 2);- tries += c == 1;- janas_tuner_record(&t, 2, c, (const double[]){1.0, 0.3, 0.8, 0.9}[c]);+ tries += c == 3;+ janas_tuner_record(&t, 2, c, (const double[]){1.0, 1.2, 0.8, 0.3}[c]);}- CHECK(janas_tuner_best(&t, 2) == 1, "after the change best %d, want 1",+ CHECK(janas_tuner_best(&t, 2) == 3, "after the change best %d, want 3",janas_tuner_best(&t, 2));CHECK(tries > 2000, "the new best taken %d times in 6000", tries);+ /* the threads are settled by the first tries and kept: a pass costing+ more as the context grows must not hand the choice to a candidate+ whose cost was measured at short context (a chat drifting onto+ fewer cores). 20 threads win the first tries; then every pass of+ theirs costs more, the others are never measured again */+ janas_tuner_init(&t, cand, 4, def, 0);+ janas_tuner_allow_gpu(&t, 0);+ run(&t, (const double[]){1.0, 1.2, 0.8, 0.9}, 40);+ CHECK(janas_tuner_best(&t, 2) == 2, "settled best %d, want 2",+ janas_tuner_best(&t, 2));+ for (int i = 0; i < 5000; i++) {+ int c = janas_tuner_pick(&t, 2);+ CHECK(t.cand[c].threads == 20, "pass on %d threads after settling",+ t.cand[c].threads);+ janas_tuner_record(&t, 2, c, 0.8 + i * 0.001);+ }+ CHECK(janas_tuner_best(&t, 2) == 2, "drifted: best %d, want 2",+ janas_tuner_best(&t, 2));/* forced: always that candidate */janas_tuner_force(&v, 1);for (int i = 0; i < 10; i++)