Skip to content

armgit / janas

Public Imported from GitHub
0

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Release 2026-10-08 (4): light experts not in memory left out instead of fewer experts per token; the machine's measure without a head start

camauri1a8d6f765e36
diff --git a/ChangeLog.md b/ChangeLog.md
index 4dd00f2..f55a6d8 100644
--- a/ChangeLog.md
+++ b/ChangeLog.md
@@ -2,6 +2,14 @@
> Curated, user-facing summary of completed work, newest first.
+## [2026-10-08] - Light experts not in memory left out, not the lightest of all
+
+- **With a small expert cache, only what would cost a read is given up.** When the cache holds less than a fifth of a model's experts, the engine used to drop the lightest experts of every token, six of ten or eight, whether they were in memory or not. Now it keeps every expert in memory and leaves out only the light ones it would have to read from disk (below 0.8/k of a token's weight, 0.95/k under a tenth), the other experts taking their share. Qwen3-Next-80B against llama.cpp, 400 tokens: with a 2 GiB cache 26.4 tokens/s instead of 20.9 at about the same agreement (380 against 383 most likely tokens); with 4 GiB the same speed (27.7) with a mean logit difference of 0.36 instead of 0.53; on Italian prose with 8 GiB 372 tokens agreeing instead of 361, and faster (23.7 against 21.4 tokens/s). On Qwen3-30B-A3B and Qwen3.6-35B-A3B with 1.5-3.5 GiB of cache it was 9-37% faster than the old way on code and on Italian prose, as close to llama.cpp or closer in most cases. `/experts n` asks for an exact number again and turns it off; `JANAS_MISS_SKIP` sets the weight for measures.
+
+## [2026-10-08] - The machine's measure without a head start
+
+- **The measure on this machine no longer favours whichever way goes second.** It warmed the model on the prose prompt alone and then measured drafting first, so the code and the list ran on cold experts with the drafts and on warm ones without: Qwen3-30B-A3B looked 10% slower drafting from the conversation (31.4 tokens/s against 34.8) and was set to `--no-spec`; with the order swapped it was 9% faster. Now every prompt warms the model, and drafting and not drafting are measured prompt by prompt, taking turns at going first: 33.5 and 34.3, twice alike, under the 3% a change needs, so the 30B keeps its drafts. Run `janas-chat <model> --tune` to measure again a model tuned before.
+
## [2026-10-08] - Qwen3.5-0.8B is a test model
- **The catalog says what each small model is for.** Qwen3.5-0.8B is a test model of the Qwen3.5 architecture, not for chatting: asked for twenty actions, it called the right tool 7 times out of 15 and made facts up ("Bologna founded in 1177"). janas-chat leaves it out of its list and opens it by name with a warning; `janas-get list` says so. Qwen3.5-2B, which called the right tool 14 times out of 15, is described as the small one that calls tools well; Qwen3-0.6B stays the draft of Qwen3-4B.
diff --git a/README.md b/README.md
index 607147f..1f53d9d 100644
--- a/README.md
+++ b/README.md
@@ -67,6 +67,7 @@ Everything the engine does is chosen from what it measures, so another machine w
- **Faster replies** from drafts the model verifies, so the text is exactly the one it would have written: from the model's own multi-token prediction block where it has one, from the conversation where it does not, or from a small model of the same family given with `--draft`.
- **Optional GPU** (Vulkan 1.3, integrated or discrete), used only where the engine measures that it helps, for every quantized type the CPU reads. Results are identical with and without it.
- **Experts read ahead** when the expert cache is too small for the model: the next layer's router is applied early and the experts it picks that are not in memory are read from disk while the current layer computes. It turns itself on only when the cache misses many experts a token (Qwen3-Next-80B with a 4 GiB cache: +6%) and changes no result. A long prompt is read the same way, block by block: the next layer's router is applied to the whole block and the experts most of its tokens will want are read while the current layer computes; on after a block that missed many experts. Qwen3-Next-80B reading 4,096 tokens: 36.8 to 42.0 tokens/s with a 4 GiB cache, 48.5 to 50.0 with 16 GiB (the 256 tokens written after it 1.4% slower, the cache holding other experts); Qwen3-30B-A3B, whose experts all fit, even. `JANAS_PREFETCH_PREFILL=0` turns it off.
+- **Light experts not in memory are left out** when the expert cache holds less than a fifth of the experts: an expert the router weighs below 0.8/k of a token (0.95/k under a tenth), and that would have to be read from disk, is not read and not computed, and the token's other experts take its share — the cache-conditional routing of Skliar et al. (TMLR 2025). Until 8 October 2026 the engine dropped the lightest experts whether in memory or not (six of ten, or eight). Qwen3-Next-80B, four bits, 400 tokens against llama.cpp, most likely token agreeing and tokens/s: with a 2 GiB cache all ten 393 at 11.0, six 383 at 20.9, now 380 at 26.4; with 4 GiB 393 at 20.7, six 383 at 27.6, now 384 at 27.7, with the mean logit difference 0.36 against 0.53; on Italian prose with 8 GiB 382 at 18.7, eight 361 at 21.4, now 372 at 23.7. On Qwen3-30B-A3B and Qwen3.6-35B-A3B with caches of 9% and 18% of their experts (1.5-3.5 GiB), on code and Italian prose, it was faster than the old way every time, by 9-37% (Qwen3-30B-A3B with 3 GiB: 28.7 tokens/s against 24.4, 381 tokens agreeing of 400 against 377), with a mean logit difference as low or lower in seven of the eight; it agreed on fewer most likely tokens in two (Italian, 64 tokens: 47 against 54 on the 30B with 3 GiB). `/experts n` (or `0`) asks for a number of experts and turns it off.
- **Every weight type near the memory's speed**, and the same results to the bit on every path (AVX2, AVX-VNNI, GPU). `tests/bench_kernels` measures each type's product on its own: writing a token, on 12 threads of the development laptop, Q8_0 84 GB/s, Q6_K 73, Q4_K 69, the other 4- and 5-bit types 59-68, Q3_K 48, IQ3_S 25, against the 81 GB/s the memory gives; and a token of Qwen3-Next-80B-A3B, Qwen3.6-35B-A3B, Qwen3-30B-A3B or a 27B, phase by phase, reads its weights at 59-62 GB/s on average. Until 6 October 2026 the Q3_K, IQ4_XS and IQ3_S products were six to thirteen times slower than that (their kernels converted a scale by calling a function), which made unsloth's UD files, which mix those types in, write at half their speed: Qwen3.8-27B UD-Q4_K_XL went from 1.7 tokens/s to 3.6.
On the machine above, with nothing else running, `janas-bench` reports this (measured before the changes of 6-7 October 2026: on Qwen3-30B-A3B in memory, `tests/bench_long` found them worth 4% reading a 1,900-token prompt and 1.5% writing after it):
@@ -251,7 +252,7 @@ The settings come in layers, each over the one before: the model's profile in th
Qwen3.5-0.8B does not take `--head 4`: with it, it answered "3" to "2+3". Gemma 4 has no fast profile: its keys are sixteen-bit, where `--attention fast` has nothing to do, its heads are not six-bit and its experts are not in planes. `--attention fast` pays where attention is a larger share of a token at 1,900 positions (Qwen3-0.6B +7.5%, Qwen3-30B-A3B +4.0%); on the hybrid Qwen3.5, Qwen3.6 and Qwen3-Next it adds 2% or less.
-**The measure on this machine.** What makes a model fastest depends on the machine too, so the first time a model starts in a terminal the chat offers to measure it: the model opened each way that can differ — drafts from its prediction file or draft model, from the conversation, or none; in the `fast` profile also the output head as in the file — one reply to warm it and the same reply measured at temperature 0, so every way writes the same text. A way other than the one the model starts with has to be at least 3% faster to be kept (one reply is measured, and a couple of per cent is its noise). The result goes to `~/.cache/janas/tune-<name>-<profile>.conf`, with the speeds as a comment, and is used for the levers you did not set yourself; a "no" is kept too, so the question comes once. `--tune` measures again, `--no-tune` neither asks nor uses it. On the development laptop: Qwen3-4B 33.1 tokens/s with Qwen3-0.6B's drafts, 29.2 without drafts, 29.5 with drafts from the conversation; Qwen3.5-2B 77.4 with its MTP block, 41.9 without drafts, 60.6 from the conversation, and with the head at four bits 77.4 against 75.6 at six; Qwen3-0.6B within 1% either way, so it starts as it would have.
+**The measure on this machine.** What makes a model fastest depends on the machine too, so the first time a model starts in a terminal the chat offers to measure it: the model opened each way that can differ — drafts from its prediction file or draft model, from the conversation, or none; in the `fast` profile also the output head as in the file — a reply to each of three prompts (prose, code, a list) to warm it, then the same three measured at temperature 0, so every way writes the same text; drafting and not drafting are measured prompt by prompt, taking turns at going first, since a mixture of experts is still warming while it is measured (until 8 October 2026 the drafts went first and warmed only on the prose, and Qwen3-30B-A3B looked 10% slower with them: with the order swapped, 9% faster). A way other than the one the model starts with has to be at least 3% faster to be kept (a couple of per cent is the noise of three replies). The result goes to `~/.cache/janas/tune-<name>-<profile>.conf`, with the speeds as a comment, and is used for the levers you did not set yourself; a "no" is kept too, so the question comes once. `--tune` measures again, `--no-tune` neither asks nor uses it. On the development laptop: Qwen3-4B 33.1 tokens/s with Qwen3-0.6B's drafts, 29.2 without drafts, 29.5 with drafts from the conversation; Qwen3.5-2B 77.4 with its MTP block, 41.9 without drafts, 60.6 from the conversation, and with the head at four bits 77.4 against 75.6 at six; Qwen3-0.6B within 1% either way, so it starts as it would have.
| Option | What it does |
|---|---|
@@ -302,7 +303,7 @@ In the chat, a line starting with `/` is a command, and it turns grey-blue as so
| `/system [text\|off]` | show or change the system message |
| `/temp <t>`, `/spec`, `/think` | sampling, drafts, reasoning |
| `/mode [auto\|eco\|max]` | power mode, and what the engine chose |
-| `/experts [n]` | experts per token: fewer is faster and a little less accurate |
+| `/experts [n]` | experts per token: fewer is faster and a little less accurate; any number, `0` included, turns off the leaving out of light experts not in memory |
| `/markdown [on\|off]` | read the model's marks, or print them |
| `/gpu [on\|off]` | whether the GPU may be given work |
| `/save-config`, `/load-config` | keep these settings for the next start, or read them back; both ask first where something would be lost |
@@ -611,7 +612,7 @@ The [hardware report form](https://github.com/prabanta-dev/janas/issues/new?temp
1. **CPUs without AVX-VNNI**, and **AMD** — the arithmetic has a path for them and it is the least exercised of all.
2. **Discrete GPUs** (NVIDIA, AMD, Intel Arc). The GPU support is correct, bit for bit, and it pays where it was measured: on the development laptop's integrated GPU a dense model whose weights all fit in memory reads a prompt about 29% faster (Qwen3-4B, 83 to 107 tokens/s), and on an RTX 5080 (issue #15) Qwen3-4B read 13% faster and, with the GPU on single tokens too, replied 11-16% faster. Every other card is unknown. **Careful:** a GPU driver reset takes the desktop with it. Start with `janas-bench`, not with a long chat.
-3. **Machines with 16 GB or less.** The engine is supposed to trade quality for memory on its own — fewer bits per weight, fewer experts per token — and the thresholds for that were measured on 32 GB.
+3. **Machines with 16 GB or less.** The engine is supposed to trade quality for memory on its own — fewer bits per weight, light experts not in memory left out — and the thresholds for that were measured on 32 GB.
4. **Slower disks.** Everything assumes an NVMe SSD; on a SATA disk the engine should still work and simply wait more, but nobody has watched it do so.
5. **Other models** of the supported families. A file that will not convert, or converts and then answers nonsense, is a useful bug.
diff --git a/include/janas/llm.h b/include/janas/llm.h
index ca4f8c4..284f53f 100644
--- a/include/janas/llm.h
+++ b/include/janas/llm.h
@@ -193,7 +193,9 @@ JANAS_LLM_API int32_t janas_llm_tuning(const janas_llm *llm, char *buf,
* out are those the router weights least. With Qwen3-Next-80B-A3B, eight of
* its ten keep the most likely token in 387 positions of 400 against 394 for
* all ten. n of 0 restores the model's own count; the change applies from the
- * next reply.
+ * next reply. Where the cache holds less than a fifth of the experts, the
+ * engine leaves out on its own the light experts that are not in RAM rather
+ * than read them; setting a number here, 0 included, turns that off.
*/
JANAS_LLM_API int32_t janas_llm_experts(const janas_llm *llm, int32_t *used,
int32_t *most);
diff --git a/src/chat/tune.c b/src/chat/tune.c
index 6d0958b..f5bac1e 100644
--- a/src/chat/tune.c
+++ b/src/chat/tune.c
@@ -4,10 +4,11 @@
* tune.c - janas-chat's measure of a model on this machine (see tune.h).
*
* Each way is a model opened anew - the prediction file and the head's bits
- * are taken when it opens - one reply to warm its caches, then three
- * replies of different kinds measured together: output tokens over output
- * seconds, at temperature 0, so every way writes the same text. Drafting off is measured on the first
- * opening, since it only takes a parameter of the chat.
+ * are taken when it opens - a reply to each prompt to warm its caches, then
+ * three replies of different kinds measured together: output tokens over
+ * output seconds, at temperature 0, so every way writes the same text.
+ * Drafting off is measured on the first opening, since it only takes a
+ * parameter of the chat, one prompt at a time beside drafting on.
*/
#include "tune.h"
@@ -19,7 +20,7 @@
#define TUNE_REPLY 128
/* a way other than the one the model starts with has to be this much
- faster: one reply is measured, and a couple of per cent is its noise
+ faster: a couple of per cent is the noise of the measure
(Qwen3-0.6B: 154.6 and 154.7 drafting, 157.2 and 156.3 without) */
#define TUNE_MARGIN 1.03
/* three kinds of reply - prose, code, a list - since how far ahead the
@@ -80,6 +81,56 @@ static double reply(janas_llm_chat *c)
return tok / sec;
}
+/*
+ * Every prompt once, unmeasured. The first reply to a kind of text is slow
+ * on a mixture of experts - its experts are touched for the first time -
+ * and warming on the first prompt alone left the code and the list cold for
+ * whichever way came first: Qwen3-30B-A3B wrote the code at 27.9 tokens/s
+ * the first time and 34.5 the second, drafting or not, and since drafting
+ * always came first the tuning chose --no-spec on a difference of 10% that
+ * was all order (8 Oct 2026: 31.4 drafting and 34.4 without, 34.5 and 31.9
+ * with the order swapped).
+ */
+static int warm(janas_llm_chat *c)
+{
+ double tok = 0, sec = 0;
+ for (size_t i = 0; i < N_PROMPTS; i++)
+ if (reply1(c, PROMPTS[i], &tok, &sec) != 0)
+ return -1;
+ return 0;
+}
+
+/* One reply to prompt with drafting as spec says, added to tok and sec. */
+static int reply_way(janas_llm_chat *c, struct janas_llm_chat_params *q,
+ int spec, const char *prompt, double *tok, double *sec)
+{
+ q->speculate = spec;
+ if (janas_llm_chat_set_params(c, q) != JANAS_LLM_OK)
+ return -1;
+ return reply1(c, prompt, tok, sec);
+}
+
+/*
+ * Drafting on and off, prompt by prompt, which goes first taking turns:
+ * the caches go on warming while a model is measured - most of all on a
+ * mixture larger than its expert cache - and two series one after the
+ * other gave the second the warmer machine.
+ */
+static int both(janas_llm_chat *c, struct janas_llm_chat_params *q, double *on,
+ double *off)
+{
+ double tok[2] = {0, 0}, sec[2] = {0, 0}; /* without, with drafts */
+ for (size_t i = 0; i < N_PROMPTS; i++)
+ for (int j = 0; j < 2; j++) {
+ int spec = (int)((i + (size_t)j) % 2 == 0);
+ if (reply_way(c, q, spec, PROMPTS[i], &tok[spec], &sec[spec]))
+ return -1;
+ }
+ *on = tok[1] / sec[1];
+ *off = tok[0] / sec[0];
+ return 0;
+}
+
/* One way: opened, warmed, measured drafting (*on) and, if off is given,
without drafting. 0, or -1. */
static int measure(const char *path, const struct janas_llm_params *mp,
@@ -102,16 +153,11 @@ static int measure(const char *path, const struct janas_llm_params *mp,
if (!gpu)
janas_llm_set_gpu(llm, 0);
int rc = -1;
- double wt = 0, ws = 0; /* the first prompt once, to warm the caches */
- if (janas_llm_chat_create(llm, &q, &c) == JANAS_LLM_OK &&
- reply1(c, PROMPTS[0], &wt, &ws) == 0 && (*on = reply(c)) > 0) {
- rc = 0;
- if (off) {
- q.speculate = 0;
- if (janas_llm_chat_set_params(c, &q) != JANAS_LLM_OK ||
- (*off = reply(c)) <= 0)
- rc = -1;
- }
+ if (janas_llm_chat_create(llm, &q, &c) == JANAS_LLM_OK && warm(c) == 0) {
+ if (off)
+ rc = both(c, &q, on, off);
+ else
+ rc = (*on = reply(c)) > 0 ? 0 : -1;
}
janas_llm_chat_destroy(c);
janas_llm_close(llm);
@@ -189,11 +235,12 @@ int chat_tune_run(const char *path, const struct janas_llm_params *mp,
char when[32];
time_t t = time(NULL);
strftime(when, sizeof(when), "%Y-%m-%d %H:%M", localtime(&t));
- fprintf(f,
- "# janas-chat: measured on this machine, %s, %zu replies of up "
- "to %d tokens\n# at temperature 0 each way (tokens/s): drafting from "
- "%s %.1f",
- when, N_PROMPTS, TUNE_REPLY, src, on);
+ fprintf(
+ f,
+ "# janas-chat: measured on this machine, %s, %zu replies of up "
+ "to %d tokens\n# at temperature 0 each way (tokens/s): drafting from "
+ "%s %.1f",
+ when, N_PROMPTS, TUNE_REPLY, src, on);
if (dims.spec)
fprintf(f, ", without drafts %.1f", off);
if (alt)
diff --git a/src/chat/tune.h b/src/chat/tune.h
index f74c8de..b3dd5be 100644
--- a/src/chat/tune.h
+++ b/src/chat/tune.h
@@ -3,8 +3,8 @@
/*
* tune.h - janas-chat's measure of a model on this machine: the levers that
* depend on it (where the drafts come from, whether to draft at all, the
- * output head's bits in the fast profile) tried on one fixed reply each, the
- * fastest kept in a file of its own - a layer between the catalog's profile
+ * output head's bits in the fast profile) tried on three fixed replies each,
+ * the fastest kept in a file of its own - a layer between the catalog's profile
* and the user's settings, never written by /save-config.
*/
#ifndef JANAS_CHAT_TUNE_H
diff --git a/src/common/env_names.c b/src/common/env_names.c
index 913643d..1f63a6a 100644
--- a/src/common/env_names.c
+++ b/src/common/env_names.c
@@ -24,6 +24,7 @@ const char *const JANAS_ENV_NAMES[] = {"JANAS_ALPHAVANTAGE_KEY",
"JANAS_DRAFT_VOCAB",
"JANAS_EXPERT_BITS",
"JANAS_EXPERTS",
+ "JANAS_MISS_SKIP",
"JANAS_EXPTRACE",
"JANAS_FLIGHTS_CONFIG",
"JANAS_GPU",
diff --git a/src/llm/api.c b/src/llm/api.c
index 16b8117..253bea9 100644
--- a/src/llm/api.c
+++ b/src/llm/api.c
@@ -494,18 +494,22 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,
if (warm_n)
snprintf(warm, sizeof(warm), ", %llu experts loading in the background",
(unsigned long long)warm_n);
- char bits[128] = "";
+ char bits[192] = "";
int nb = janas_llm_model_expert_bits(llm->m);
- uint32_t used = janas_llm_model_experts(llm->m, NULL);
+ uint32_t most = 0, used = janas_llm_model_experts(llm->m, &most);
int w = 0;
if (nb != 6)
w = snprintf(bits, sizeof(bits), ", down matrix at %d bits%s", nb,
janas_llm_model_expert_bits_auto(llm->m) ? " (memory)"
: "");
- if (janas_llm_model_experts_auto(llm->m) && w >= 0 &&
+ if (used < most && w >= 0 && (size_t)w < sizeof(bits))
+ w += snprintf(bits + w, sizeof(bits) - (size_t)w,
+ ", %u experts per token", used);
+ if (janas_llm_model_miss_skip(llm->m) > 0 && w >= 0 &&
(size_t)w < sizeof(bits))
w += snprintf(bits + w, sizeof(bits) - (size_t)w,
- ", %u experts per token (memory)", used);
+ ", light experts not in RAM left out%s",
+ janas_llm_model_experts_auto(llm->m) ? " (memory)" : "");
uint64_t d0, d1;
janas_llm_model_head_requant(llm->m, &d0, &d1);
if (d0 && w >= 0 && (size_t)w < sizeof(bits))
diff --git a/src/llm/chat.h b/src/llm/chat.h
index 58f84d3..857c2f2 100644
--- a/src/llm/chat.h
+++ b/src/llm/chat.h
@@ -57,7 +57,7 @@ struct janas_llm {
whether the GPU is given work is a thing that changes */
uint64_t cache_bytes, reserve_bytes;
int n_compute;
- char arch[32], bits[128], warm[64];
+ char arch[32], bits[192], warm[64];
char name[96];
int32_t mode;
int gpu_wanted; /* the caller's say, on top of what the mode decides */
diff --git a/src/llm/expert_cache.c b/src/llm/expert_cache.c
index 2da0cf0..a000171 100644
--- a/src/llm/expert_cache.c
+++ b/src/llm/expert_cache.c
@@ -1019,6 +1019,19 @@ int janas_expert_cache_begin(struct janas_expert_cache *c, uint32_t layer,
return queued;
}
+void janas_expert_cache_present(struct janas_expert_cache *c, uint32_t layer,
+ const uint32_t *ids, size_t n, uint8_t *in)
+{
+ uint32_t ne = c->j->h.n_expert;
+ size_t base = (size_t)layer * ne;
+ /* the background filling changes the map under this lock */
+ pthread_mutex_lock(&c->api_lock);
+ for (size_t i = 0; i < n; i++)
+ in[i] = layer < c->j->h.n_layer && ids[i] < ne &&
+ c->where[base + ids[i]] != NONE;
+ pthread_mutex_unlock(&c->api_lock);
+}
+
int janas_expert_cache_finish(struct janas_expert_cache *c)
{
double t0 = now();
diff --git a/src/llm/expert_cache.h b/src/llm/expert_cache.h
index 7456d10..b3aa9b2 100644
--- a/src/llm/expert_cache.h
+++ b/src/llm/expert_cache.h
@@ -151,6 +151,14 @@ int janas_expert_cache_prefetch(struct janas_expert_cache *c, uint32_t layer,
/* Whether this cache can read ahead at all (the kernel has io_uring). */
int janas_expert_cache_can_prefetch(const struct janas_expert_cache *c);
+/*
+ * in[i] = 1 where expert ids[i] of the layer is in the cache (or being read
+ * into it), 0 where a request would read it. Nothing is touched: a question,
+ * not a request. Between requests, not between begin and finish.
+ */
+void janas_expert_cache_present(struct janas_expert_cache *c, uint32_t layer,
+ const uint32_t *ids, size_t n, uint8_t *in);
+
void janas_expert_cache_stats(const struct janas_expert_cache *c,
struct janas_expert_cache_stats *s);
void janas_expert_cache_reset_stats(struct janas_expert_cache *c);
diff --git a/src/llm/model.h b/src/llm/model.h
index 54b118e..74551c7 100644
--- a/src/llm/model.h
+++ b/src/llm/model.h
@@ -92,9 +92,12 @@ void janas_llm_model_head_requant(const struct janas_llm_model *m,
int janas_llm_model_attn_scores(const struct janas_llm_model *m);
int janas_llm_model_attn_scores_auto(const struct janas_llm_model *m);
-/* Whether the engine also chose how many experts a token uses, because the
- cache holds too small a share of the model to use them all at speed. */
+/* Whether the engine also chose to leave out the light experts that are not
+ in RAM, because the cache holds too small a share of the model to read
+ them all at speed; and the weight below which it does (0: never). Setting
+ the number of experts (janas_llm_model_set_experts) turns it off. */
int janas_llm_model_experts_auto(const struct janas_llm_model *m);
+float janas_llm_model_miss_skip(const struct janas_llm_model *m);
/*
* The experts the engine means to preload when the model is opened (from the
diff --git a/src/llm/model_ffn.c b/src/llm/model_ffn.c
index efd9955..71275d6 100644
--- a/src/llm/model_ffn.c
+++ b/src/llm/model_ffn.c
@@ -417,6 +417,72 @@ static void axpy_worker(void *arg, int tid, int n_threads)
}
}
+/*
+ * Cache-conditional routing (Skliar et al., "Mixture of Cache-Conditional
+ * Experts", TMLR 2025): an expert that is not in RAM and that every token
+ * of the block weighs below m->miss_skip is left out instead of read, and
+ * each token's other weights are renormalized, as with fewer experts per
+ * token. Unlike fewer experts per token, which drops the lightest whether
+ * they are in RAM or not, only what would cost a read goes. Simulated on
+ * Qwen3-Next-80B's routing over seven kinds of text (8 Oct 2026): below the
+ * lightest quarter of the weights (0.06 of ten), a third fewer experts read
+ * with a cache of 10-30% of them, for 1-4% of what the experts add.
+ * Returns the pairs left, compacted in place, token by token.
+ */
+static uint32_t skip_misses(struct janas_llm_model *m,
+ struct janas_expert_cache *cache, uint32_t l,
+ uint32_t n, uint32_t k)
+{
+ uint32_t np = n * k;
+ float th = m->miss_skip, wmax[4096];
+ uint8_t mark[4096]; /* 1: light, asked of the cache; 2: light and out */
+ for (uint32_t p = 0; p < np; p++) {
+ wmax[m->pair_exp[p]] = 0.0f;
+ mark[m->pair_exp[p]] = 0;
+ }
+ for (uint32_t p = 0; p < np; p++)
+ if (m->pair_w[p] > wmax[m->pair_exp[p]])
+ wmax[m->pair_exp[p]] = m->pair_w[p];
+ uint32_t light[256];
+ uint8_t in[256];
+ size_t nl = 0;
+ for (uint32_t p = 0; p <= np; p++) {
+ if (p < np) {
+ uint32_t e = m->pair_exp[p];
+ if (wmax[e] >= th || mark[e])
+ continue;
+ mark[e] = 1;
+ light[nl++] = e;
+ }
+ if (nl && (nl == 256 || p == np)) {
+ janas_expert_cache_present(cache, l, light, nl, in);
+ for (size_t i = 0; i < nl; i++)
+ if (!in[i])
+ mark[light[i]] = 2;
+ nl = 0;
+ }
+ }
+ uint32_t q = 0;
+ for (uint32_t j = 0; j < n; j++) {
+ float kept = 0.0f;
+ for (uint32_t i = 0; i < k; i++)
+ if (mark[m->pair_exp[j * k + i]] != 2)
+ kept += m->pair_w[j * k + i];
+ for (uint32_t i = 0; i < k; i++) {
+ uint32_t p = j * k + i;
+ if (mark[m->pair_exp[p]] == 2)
+ continue;
+ m->pair_tok[q] = j;
+ m->pair_exp[q] = m->pair_exp[p];
+ m->pair_w[q] = m->pair_w[p] / kept;
+ q++;
+ }
+ }
+ m->skip_asked += np;
+ m->skip_left += q;
+ return q;
+}
+
/* Routed (and shared) experts of a block, added to x. The routed experts
are layer l of the file behind the cache (the main model or the MTP). */
int janas_m_moe_block(struct janas_llm_model *m, const struct layer *ly,
@@ -427,6 +493,7 @@ int janas_m_moe_block(struct janas_llm_model *m, const struct layer *ly,
double t0 = now();
uint32_t dm = m->d_model, ne = m->n_expert;
uint32_t k = part == MOE_SHARED ? 0 : m->k_use;
+ uint32_t np = n * k;
if (k) {
struct router_job rj = {m, ly->router16, n};
janas_pool_run(m->compute, router_worker, &rj);
@@ -443,9 +510,11 @@ int janas_m_moe_block(struct janas_llm_model *m, const struct layer *ly,
m->pair_w[p] *= ly->exp_scale[m->pair_exp[p]];
if (m->route_fn && cache == m->cache)
m->route_fn(m->route_ctx, l, n, k, m->pair_exp);
+ /* the weights must be the router's own, summing to one a token */
+ if (m->miss_skip > 0 && cache && !ly->exp_scale)
+ np = skip_misses(m, cache, l, n, k);
}
/* counting sort of the pairs by expert */
- uint32_t np = n * k;
memset(m->count, 0, (ne + 1) * sizeof(uint32_t));
for (uint32_t p = 0; p < np; p++)
m->count[m->pair_exp[p] + 1]++;
diff --git a/src/llm/model_impl.h b/src/llm/model_impl.h
index 7d7d8c9..affdaa3 100644
--- a/src/llm/model_impl.h
+++ b/src/llm/model_impl.h
@@ -264,7 +264,11 @@ struct janas_llm_model {
int exp_planes; /* some layer keeps its down matrix in planes */
int exp_auto; /* the level was chosen from the memory available */
int k_auto; /* and so was the number of experts per token */
- int warm_use; /* preload the most used experts, and count them */
+ /* an expert not in RAM that every token of the block weighs below this
+ is left out, not read (see skip_misses); 0: never. And how many were */
+ float miss_skip;
+ uint64_t skip_asked, skip_left;
+ int warm_use; /* preload the most used experts, and count them */
uint32_t *use_count; /* experts used this session, n_layer * n_expert */
uint64_t head_before, head_after; /* output head requantized (head_bits) */
/* the caller's CPUs before the load pinned it (a cpu_set_t), given
diff --git a/src/llm/model_load.c b/src/llm/model_load.c
index 1ff5607..25a9ddd 100644
--- a/src/llm/model_load.c
+++ b/src/llm/model_load.c
@@ -960,32 +960,39 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
}
m->k_use = m->k;
/*
- * How many experts a token uses: all of them while the cache holds a
- * fair share of the model, fewer only where it does not and the machine
- * would wait for the disk at every token. Measured on 20 Sep 2026 with
- * Qwen3-Next (400 tokens of C, four bits): with a cache of 26% of the
- * experts, using six instead of ten gains 14%; with 13%, 24%; with 6%,
- * 46%. The quality given up is the same everywhere (in Italian prose,
- * 22 most-likely tokens in 400 with eight experts and 55 with six), so
- * it is only worth it where the gain is large.
+ * Where the cache holds too small a share of the experts and the machine
+ * would wait for the disk at every token, the light experts that are not
+ * in RAM are left out (skip_misses): below 0.95/k of a token's weight
+ * with a cache of less than a tenth of the experts, below 0.8/k with
+ * less than a fifth. Until 8 Oct 2026 the lightest experts were left out
+ * whether in RAM or not - six of ten, or eight - which gave up more for
+ * less. Qwen3-Next-80B, four bits, 400 tokens against llama.cpp (most
+ * likely token agreeing, mean |logit difference|, tokens/s):
+ * C, 2 GiB all ten 393 0.17 11.0, six 383 0.53 20.9, skip 380
+ * 0.44 26.4 C, 4 GiB all ten 393 0.17 20.7, six 383 0.53 27.6, skip 384
+ * 0.36 27.7 C, 8 GiB all ten 393 0.17 26.0, eight 385 0.32 28.1, skip
+ * 391 0.25 27.1 Italian, 4 all ten 382 0.14 13.7, six 333 0.46 18.9,
+ * (0.8/k) 359 0.23 19.4 Italian, 8 all ten 382 0.14 18.7, eight 361
+ * 0.26 21.4, skip 372 0.23 23.7
*/
if (o->cache_bytes && m->j.h.experts_bytes) {
double cover = (double)o->cache_bytes / (double)m->j.h.experts_bytes;
- uint32_t k = m->k;
- if (cover < 0.10)
- k = m->k * 6 / 10;
- else if (cover < 0.20)
- k = m->k * 8 / 10;
- if (k < 1)
- k = 1;
- if (k != m->k) {
- janas_llm_model_set_experts(m, k);
+ if (cover < 0.20) {
+ m->miss_skip = (cover < 0.10 ? 0.95f : 0.8f) / (float)m->k;
m->k_auto = 1;
}
}
- if (getenv("JANAS_EXPERTS")) { /* fewer experts per token, for measures */
+ if (getenv("JANAS_EXPERTS")) /* experts per token, for measures */
janas_llm_model_set_experts(m, (uint32_t)atoi(getenv("JANAS_EXPERTS")));
- m->k_auto = 0;
+ {
+ /* experts not in RAM left out below this weight, for measures (0:
+ never); at least one expert a token always stays, since the
+ heaviest weighs 1/k or more */
+ const char *ms = getenv("JANAS_MISS_SKIP");
+ if (ms)
+ m->miss_skip = (float)atof(ms);
+ if (m->miss_skip >= 1.0f / (float)m->k)
+ m->miss_skip = 0.0f;
}
{
const char *vm = getenv("JANAS_VMAP");
@@ -1187,6 +1194,15 @@ void janas_llm_model_free(struct janas_llm_model *m)
{
if (janas_m_exp_tracing())
janas_m_exp_report();
+ if (m && m->skip_asked && (getenv("JANAS_MISS_SKIP") || m->trace))
+ fprintf(stderr,
+ "janas: experts not in RAM below %.3f left out: %llu of %llu "
+ "routed (%.1f%%)\n",
+ m->miss_skip,
+ (unsigned long long)(m->skip_asked - m->skip_left),
+ (unsigned long long)m->skip_asked,
+ 100.0 * (double)(m->skip_asked - m->skip_left) /
+ (double)m->skip_asked);
if (!m)
return;
if (m->caller_pinned)
@@ -1305,6 +1321,11 @@ int janas_llm_model_set_experts(struct janas_llm_model *m, uint32_t n)
if (n > m->k)
return 0;
m->k_use = n;
+ /* asked for a number: that many, none left out by the engine */
+ if (m->k_auto) {
+ m->miss_skip = 0.0f;
+ m->k_auto = 0;
+ }
return 1;
}
@@ -1381,6 +1402,11 @@ int janas_llm_model_experts_auto(const struct janas_llm_model *m)
return m->k_auto;
}
+float janas_llm_model_miss_skip(const struct janas_llm_model *m)
+{
+ return m->miss_skip;
+}
+
uint64_t janas_llm_model_warm(const struct janas_llm_model *m, double *seconds)
{
if (seconds)
diff --git a/tests/llm_eval.c b/tests/llm_eval.c
index 1ccca64..4bfafb1 100644
--- a/tests/llm_eval.c
+++ b/tests/llm_eval.c
@@ -156,8 +156,8 @@ int main(int argc, char **argv)
janas_llm_model_decode(m, tokens[p], (uint32_t)p, seq + p * nv);
double pb0[JANAS_PH_COUNT], pb1[JANAS_PH_COUNT];
janas_llm_model_phases(m, pb0);
- double tb = now(), maxd = 0;
- size_t same = 0;
+ double tb = now(), maxd = 0, sumd = 0;
+ size_t same = 0, top = 0;
for (size_t p0 = 0; p0 < nn; p0 += (size_t)block) {
uint32_t b =
(uint32_t)(nn - p0 < (size_t)block ? nn - p0 : (size_t)block);
@@ -173,9 +173,11 @@ int main(int argc, char **argv)
double d = fabs((double)a[i] - c[i]);
if (d > maxd)
maxd = d;
+ sumd += d;
eq &= a[i] == c[i];
}
same += eq;
+ top += argmax(a, nv) == argmax(c, nv);
}
}
double tbs = now() - tb;
@@ -185,6 +187,9 @@ int main(int argc, char **argv)
"blocks of %d: %zu/%zu positions bit-identical to token by token, "
"max |diff| %g; %.1f ms/token in blocks (%.1f token/s)\n",
block, same, nn, maxd, tbs / nn * 1e3, nn / tbs);
+ printf("blocks of %d: most likely token as token by token at %zu/%zu, "
+ "mean |diff| %.4f\n",
+ block, top, nn, sumd / ((double)nn * nv));
free(seq);
free(blk);
}