Skip to content

armgit / janas

Public Imported from GitHub
0

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Release 2026-09-27: the threads chosen per machine, a steadier prompt ETA

camauri5d2388368419
diff --git a/ChangeLog.md b/ChangeLog.md
index e6939ed..ff9b826 100644
--- a/ChangeLog.md
+++ b/ChangeLog.md
@@ -2,6 +2,27 @@
> Curated, user-facing summary of completed work, newest first.
+## [2026-09-27] - The threads chosen per machine
+
+- **The time left to read a long prompt no longer jumps at its start.**
+ The fitted line of issue #9 rested on two blocks when first used, and
+ their noise could tilt it: an estimate many times too long at 1,024
+ tokens. It now moves from the latest speed to the fitted line over the
+ first sixteen blocks.
+
+- **The engine chooses its threads again, once per machine and model.**
+ Since 26 September it always used every usable thread, which was the
+ fastest on the development laptop; on an i9-14900HX (issue #13) every
+ thread replied at 9.9 tokens/s where the performance cores' 16 gave
+ 24.7, and on a 9950X3D one thread per core was 4-15% ahead (#15). The
+ first passes of each kind now try every thread, one per performance core
+ and the performance cores' threads; the threads of the fastest are kept
+ for good, saved with the machine's profile, and only whether the GPU
+ takes part keeps being tried. Tried again later, at a longer context,
+ the thread counts had lost unfairly to costs measured at a short one,
+ which is why they had been fixed. `JANAS_TUNE_THREADS=0` keeps every
+ thread.
+
## [2026-09-27] - Gemma 4's assistant drafts
- **Gemma 4's assistant is taken as the MTP file** (`--mtp`, converted by
diff --git a/INSTALL.md b/INSTALL.md
index 7f33108..bd58c3f 100644
--- a/INSTALL.md
+++ b/INSTALL.md
@@ -87,4 +87,5 @@ It takes a few minutes: it measures prompt processing and generation speed for e
- `JANAS_GPU=0` turns the GPU off; `JANAS_GPU_DEVICE=integrated` or `discrete` picks one when there are both.
- `JANAS_KERNELS=avx2` uses the AVX2 kernels even where AVX-VNNI is available.
- `JANAS_TUNE=0` turns self-tuning off (fixed defaults); delete `~/.cache/janas/tuning.txt` to start tuning from scratch.
+- `JANAS_TUNE_THREADS=0` keeps every usable thread instead of letting the first passes choose between every thread, one per performance core and the performance cores' threads; the choice, once made, is kept for that machine and model.
- The expert cache takes the free memory, leaving a fifth of the machine's memory to the rest of the system; `--cache <GiB>` in `janas-chat` sets it.
diff --git a/README.md b/README.md
index 0b3c966..0400815 100644
--- a/README.md
+++ b/README.md
@@ -144,9 +144,13 @@ On the machine above, with nothing else running, `janas-bench` reports this:
Every figure is the **mean of three rounds**, not a best round, of the
configuration the engine itself chose for each kind of pass - the one a chat
-runs: all the usable cores (performance and efficiency cores, not the
-low-power ones), with the integrated GPU only where it helped, which was
-reading prompts on the two largest mixtures. The rounds run in alternating
+runs: here all the usable cores (performance and efficiency cores, not
+the low-power ones), with the integrated GPU only where it helped, which
+was reading prompts on the two largest mixtures. On another machine the
+engine may settle on fewer threads: the first passes of each kind try
+every core, one thread per performance core and the performance cores'
+threads, and keep the fastest (on an i9-14900HX every thread replied at
+less than half the speed of the performance cores' 16). The rounds run in alternating
order so that the machine warming up and the cache filling weigh on each
setting alike. Expect a few per cent either way between runs, and rather
less than these on a first run, while the cache is still filling.
diff --git a/src/llm/api_reply.c b/src/llm/api_reply.c
index 0143805..ffbeb72 100644
--- a/src/llm/api_reply.c
+++ b/src/llm/api_reply.c
@@ -52,8 +52,12 @@ static double cpu_now(void)
* 260,954-token prompt of Qwen3-Next-80B the elapsed time followed a n +
* b n^2 to R^2 0.99999 (issue #9), where the latest speed promised 2.5
* times too little at 32K. Never below what the last block's speed says:
- * the cost does not fall as the context grows.
+ * the cost does not fall as the context grows. A line through the first
+ * few blocks tilts with their noise (an ETA many times too long at 1,024
+ * tokens, issue #9), so the estimate moves from the last block's speed to
+ * the fitted line over the first ETA_BLOCKS blocks fitted.
*/
+#define ETA_BLOCKS 16
static double input_eta(const janas_llm_chat *c, uint32_t left)
{
if (left == 0)
@@ -68,7 +72,10 @@ static double input_eta(const janas_llm_chat *c, uint32_t left)
double a = (c->eta_sc - b * c->eta_sx) / w;
double p0 = janas_llm_session_computed(c->s), p1 = p0 + left;
double fit = a * (p1 - p0) + 0.5 * b * (p1 * p1 - p0 * p0);
- return fit > flat ? fit : flat;
+ if (fit <= flat)
+ return flat;
+ double k = (double)(c->eta_steps - 2) / ETA_BLOCKS;
+ return flat + (k < 1 ? k : 1) * (fit - flat);
}
/* One block of the prompt read: n tokens from position p0, in dt seconds.
diff --git a/src/llm/model_tune.c b/src/llm/model_tune.c
index 4989cdc..4a78445 100644
--- a/src/llm/model_tune.c
+++ b/src/llm/model_tune.c
@@ -29,18 +29,19 @@
/*
* The tuner's candidates: every usable core - the performance cores with
* all their threads and the efficiency cores, not the low-power island,
- * which the layout leaves out - with and without the GPU if there is one.
- * Measured on 26 Sep 2026 on six models (Qwen3.5-2B to Qwen3-Next-80B),
- * that was the fastest or level in every kind of pass; one thread per
- * performance core, or their threads alone, lost 5-20% writing. Trying
- * them in use only made a chat run on fewer cores than janas-bench had
- * reported, whenever a stretch of long context had tipped the averages:
- * the same machine, two speeds. So the threads are fixed, and the tuner
- * only decides whether the GPU helps. JANAS_TUNE_THREADS=1 brings back
- * those candidates (one per performance core, all their threads), to
- * experiment with. JANAS_TUNE=0 turns tuning off; JANAS_BLOCK_THREADS fixes
- * the threads for blocks (and turns it off too); JANAS_GPU_ALWAYS=1 makes
- * the GPU variants the defaults for several tokens.
+ * which the layout leaves out - one thread per performance core, and the
+ * performance cores' threads alone; each with and without the GPU if there
+ * is one. On the development laptop (6 performance cores, 8 efficiency)
+ * every core was the fastest or level on six models; on an i9-14900HX (8
+ * and 16) every thread replied at 9.9 tokens/s where 16 gave 24.7 (issue
+ * #13), and on a 9950X3D one thread per core was 4-15% ahead (#15). So the
+ * first passes of each kind try them all and the tuner keeps the threads
+ * of the fastest (tuner.h: settled once, never tried again at a longer
+ * context, which had moved chats onto fewer cores than they could use).
+ * JANAS_TUNE_THREADS=0 keeps every core without trying the others;
+ * JANAS_TUNE=0 turns tuning off; JANAS_BLOCK_THREADS fixes the threads for
+ * blocks (and turns it off too); JANAS_GPU_ALWAYS=1 makes the GPU variants
+ * the defaults for several tokens.
*/
void janas_m_init_tuner(struct janas_llm_model *m)
{
@@ -48,7 +49,7 @@ void janas_m_init_tuner(struct janas_llm_model *m)
int n = 0, opts[3] = {m->all_threads, m->p_cores, m->p_threads};
const char *bt = getenv("JANAS_BLOCK_THREADS"), *tn = getenv("JANAS_TUNE"),
*tt = getenv("JANAS_TUNE_THREADS");
- int n_opts = tt && atoi(tt) > 0 ? 3 : 1;
+ int n_opts = tt && atoi(tt) == 0 ? 1 : 3;
int block = bt && atoi(bt) > 0 ? atoi(bt) : m->all_threads;
if (block > m->all_threads)
block = m->all_threads;
diff --git a/src/llm/tuner.c b/src/llm/tuner.c
index e844c9e..858864b 100644
--- a/src/llm/tuner.c
+++ b/src/llm/tuner.c
@@ -45,7 +45,32 @@ void janas_tuner_init(struct janas_tuner *t, const struct janas_tune_cand *c,
power its keep-alive costs) */
static int usable(const struct janas_tuner *t, int cls, int i)
{
- return !t->cand[i].gpu || (t->allow_gpu && cls > 0);
+ return (!t->cand[i].gpu || (t->allow_gpu && cls > 0)) &&
+ (!t->cls[cls].threads || t->cand[i].threads == t->cls[cls].threads);
+}
+
+/* The default of a class: its own, or once the threads are settled the
+ candidate with those threads and no GPU */
+static int class_def(const struct janas_tuner *t, int cls)
+{
+ int def = t->cls[cls].def;
+ if (!t->cls[cls].threads || t->cand[def].threads == t->cls[cls].threads)
+ return def;
+ for (int i = 0; i < t->n_cand; i++)
+ if (!t->cand[i].gpu && t->cand[i].threads == t->cls[cls].threads)
+ return i;
+ return def;
+}
+
+/* Once every candidate has its tries, the threads of the best are kept */
+static void settle_threads(struct janas_tuner *t, int cls)
+{
+ if (t->cls[cls].threads)
+ return;
+ for (int i = 0; i < t->n_cand; i++)
+ if (usable(t, cls, i) && t->cls[cls].tried[i] < TRIES)
+ return;
+ t->cls[cls].threads = t->cand[janas_tuner_best(t, cls)].threads;
}
void janas_tuner_force(struct janas_tuner *t, int cand)
@@ -62,7 +87,7 @@ int janas_tuner_best(const struct janas_tuner *t, int cls)
{
/* the default unless an alternative is clearly faster: measurements
are noisy, a near tie is not worth a change */
- int def = t->cls[cls].def, best = def;
+ int def = class_def(t, cls), best = def;
double bar =
t->cls[cls].tried[def] >= TRIES ? t->cls[cls].cost[def] * MARGIN : 1e30;
for (int i = 0; i < t->n_cand; i++)
@@ -84,7 +109,8 @@ int janas_tuner_pick(struct janas_tuner *t, int cls)
return t->cls[cls].def;
uint32_t p = t->cls[cls].passes++;
if (p < WARM_PASSES)
- return t->cls[cls].def;
+ return class_def(t, cls);
+ settle_threads(t, cls);
/* explore: the least tried candidate until each has TRIES */
int least = -1;
for (int i = 0; i < t->n_cand; i++)
@@ -205,6 +231,23 @@ void janas_tuner_load(struct janas_tuner *t, const char *key)
t->cls[cls].tried[cand] = tries;
}
fclose(f);
+ /* the threads settled: a line of their own, candidate -1 */
+ if (!(f = fopen(path, "r")))
+ return;
+ while (fgets(line, sizeof(line), f)) {
+ int cls, cand;
+ double threads;
+ unsigned zero;
+ if (strncmp(line, key, kl) != 0 || line[kl] != '\t' ||
+ sscanf(line + kl + 1, "%d\t%d\t%lf\t%u", &cls, &cand, &threads,
+ &zero) != 4 ||
+ cls < 0 || cls >= JANAS_TUNE_CLASSES || cand != -1)
+ continue;
+ for (int i = 0; i < t->n_cand; i++)
+ if (t->cand[i].threads == (int)threads)
+ t->cls[cls].threads = (int)threads;
+ }
+ fclose(f);
}
void janas_tuner_save(const struct janas_tuner *t, const char *key)
@@ -229,6 +272,9 @@ void janas_tuner_save(const struct janas_tuner *t, const char *key)
if (t->cls[k].tried[i] > 0)
fprintf(out, "%s\t%d\t%d\t%.9g\t%u\n", key, k, i,
t->cls[k].cost[i], t->cls[k].tried[i]);
+ for (int k = 0; k < JANAS_TUNE_CLASSES; k++)
+ if (t->cls[k].threads)
+ fprintf(out, "%s\t%d\t-1\t%d\t0\n", key, k, t->cls[k].threads);
if (fclose(out) == 0)
rename(tmp, path);
else
diff --git a/src/llm/tuner.h b/src/llm/tuner.h
index c336ff8..2852a3e 100644
--- a/src/llm/tuner.h
+++ b/src/llm/tuner.h
@@ -8,8 +8,12 @@
* runs anyway: for each kind of pass (one token, a few, a block) it tries
* each candidate a few times, keeps the fastest, and now and then tries the
* others again to follow the machine's state (heat, power source). The
- * choices are saved per machine and model, so the next start begins from
- * them.
+ * number of threads is settled once, by the first tries, and then kept:
+ * a pass costs more as the context grows, so a candidate tried again at
+ * long context against costs measured at short context loses unfairly -
+ * which moved chats onto fewer cores than they could use. Only whether the
+ * GPU takes part keeps being tried. The choices are saved per machine and
+ * model, so the next start begins from them.
*/
#ifndef JANAS_LLM_TUNER_H
#define JANAS_LLM_TUNER_H
@@ -42,6 +46,7 @@ struct janas_tuner {
pending): a try that changes nothing doubles the gap */
uint32_t retry_at, retry_gap;
int retry_best, retry_cand; /* and the candidate tried */
+ int threads; /* settled by the first tries (0: not yet) */
} cls[JANAS_TUNE_CLASSES];
};
diff --git a/tests/test_tuner.c b/tests/test_tuner.c
index 40f1250..d2ff40f 100644
--- a/tests/test_tuner.c
+++ b/tests/test_tuner.c
@@ -96,6 +96,8 @@ int main(void)
janas_tuner_load(&u, "test|key");
CHECK(janas_tuner_best(&u, 2) == 3, "loaded best %d, want 3",
janas_tuner_best(&u, 2));
+ CHECK(u.cls[2].threads == 20 && u.cls[2].threads == t.cls[2].threads,
+ "loaded threads %d, want 20", u.cls[2].threads);
struct janas_tuner v;
janas_tuner_init(&v, cand, 4, def, 0);
janas_tuner_load(&v, "other|key");
@@ -122,15 +124,33 @@ int main(void)
}
CHECK(tries >= 5 && tries <= 30, "%d tries in 20000 passes", tries);
tries = 0;
- for (int i = 0; i < 6000; i++) { /* the machine changes: 1 is best now */
+ for (int i = 0; i < 6000; i++) { /* the machine changes: the GPU wins */
int c = janas_tuner_pick(&t, 2);
- tries += c == 1;
- janas_tuner_record(&t, 2, c, (const double[]){1.0, 0.3, 0.8, 0.9}[c]);
+ tries += c == 3;
+ janas_tuner_record(&t, 2, c, (const double[]){1.0, 1.2, 0.8, 0.3}[c]);
}
- CHECK(janas_tuner_best(&t, 2) == 1, "after the change best %d, want 1",
+ CHECK(janas_tuner_best(&t, 2) == 3, "after the change best %d, want 3",
janas_tuner_best(&t, 2));
CHECK(tries > 2000, "the new best taken %d times in 6000", tries);
+ /* the threads are settled by the first tries and kept: a pass costing
+ more as the context grows must not hand the choice to a candidate
+ whose cost was measured at short context (a chat drifting onto
+ fewer cores). 20 threads win the first tries; then every pass of
+ theirs costs more, the others are never measured again */
+ janas_tuner_init(&t, cand, 4, def, 0);
+ janas_tuner_allow_gpu(&t, 0);
+ run(&t, (const double[]){1.0, 1.2, 0.8, 0.9}, 40);
+ CHECK(janas_tuner_best(&t, 2) == 2, "settled best %d, want 2",
+ janas_tuner_best(&t, 2));
+ for (int i = 0; i < 5000; i++) {
+ int c = janas_tuner_pick(&t, 2);
+ CHECK(t.cand[c].threads == 20, "pass on %d threads after settling",
+ t.cand[c].threads);
+ janas_tuner_record(&t, 2, c, 0.8 + i * 0.001);
+ }
+ CHECK(janas_tuner_best(&t, 2) == 2, "drifted: best %d, want 2",
+ janas_tuner_best(&t, 2));
/* forced: always that candidate */
janas_tuner_force(&v, 1);
for (int i = 0; i < 10; i++)