37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-10-06: the prefill reads the next layer's experts ahead
camauri4bde67ce4027
diff --git a/ChangeLog.md b/ChangeLog.mdindex 9687284..50ad4ed 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -2,6 +2,13 @@> Curated, user-facing summary of completed work, newest first.+## [2026-10-06] - A long prompt reads the next layer's experts ahead++- **The prefill reads ahead too.** Block by block, the next layer's router is applied to the whole block before the current layer's experts, and the experts most of its tokens will want are read from the disk while the current layer computes; until now each layer waited for the disk chunk by chunk. On after a block whose layers missed more than 32 experts each, off where the experts fit. Qwen3-Next-80B-A3B, 4,096 tokens, every bit: 36.8 to 42.0 tokens/s with a 4 GiB cache (time waited for the disk 40.8 to 20.7 s), 48.5 to 50.0 with 16 GiB, the 256 tokens written after it 21.8 to 21.5 tokens/s; Qwen3-30B-A3B in a cache that holds it, even. The cost is the router applied once more, about 1.1 ms a token. `JANAS_PREFETCH_PREFILL=0` never, `=2` always. The idea is Strata's (github.com/Niko1221/Strata), which streams the next layer's experts to its graphics card.+- **The experts read ahead are kept until their layer is done.** A prefill block asks the cache for its experts 64 at a time, and the first request let go of every expert read ahead for the layer, so those read for the later chunks were thrown out before they were asked for: 42% of the reads ahead were used, 82-89% now.+- `tests/bench_long` counts the experts read ahead as read from the disk, as the chat's statistics do; it counted them as found in memory.+- **Tried and left out**: drafts copied from the context weighed against the MTP block's, as Strata does. On Qwen3.5-9B the MTP drafts already kept 95-99% where a reply copies its prompt (a function renamed, a table), and the copies, kept 20% of the time, only took their place.+## [2026-10-03] - Strict settings- **A word not known is refused, with the one nearest to it.** `janas-chat` (command line and `chat.conf`), `janas-server`, `janas-mcp` and `janas-bench` refuse an option mistyped ("--tmep is not an option (did you mean --temp?)") and a value that is not one: numbers are read whole and within their limits (`--ctx 32k` was 32 tokens, `--temp 0,7` was 0, `--port 70000` wrapped), choices are one of theirs (`--think of` turned reasoning on, `--mode ecco` was auto). `janas-bench` left an unknown option and a last one without its value aside, and measured something else than asked. The chat's file said "a word it does not know" without saying which, and went on.diff --git a/README.md b/README.mdindex 58aaf8e..c43a248 100644--- a/README.md+++ b/README.md@@ -66,7 +66,7 @@ Everything the engine does is chosen from what it measures, so another machine w- **`janas-bench`:** measures the machine, fills its profile and writes a report.- **Faster replies** from drafts the model verifies, so the text is exactly the one it would have written: from the model's own multi-token prediction block where it has one, from the conversation where it does not, or from a small model of the same family given with `--draft`.- **Optional GPU** (Vulkan 1.3, integrated or discrete), used only where the engine measures that it helps, for every quantized type the CPU reads. Results are identical with and without it.-- **Experts read ahead** when the expert cache is too small for the model: the next layer's router is applied early and the experts it picks that are not in memory are read from disk while the current layer computes. It turns itself on only when the cache misses many experts a token (Qwen3-Next-80B with a 4 GiB cache: +6%) and changes no result.+- **Experts read ahead** when the expert cache is too small for the model: the next layer's router is applied early and the experts it picks that are not in memory are read from disk while the current layer computes. It turns itself on only when the cache misses many experts a token (Qwen3-Next-80B with a 4 GiB cache: +6%) and changes no result. A long prompt is read the same way, block by block: the next layer's router is applied to the whole block and the experts most of its tokens will want are read while the current layer computes; on after a block that missed many experts. Qwen3-Next-80B reading 4,096 tokens: 36.8 to 42.0 tokens/s with a 4 GiB cache, 48.5 to 50.0 with 16 GiB (the 256 tokens written after it 1.4% slower, the cache holding other experts); Qwen3-30B-A3B, whose experts all fit, even. `JANAS_PREFETCH_PREFILL=0` turns it off.On the machine above, with nothing else running, `janas-bench` reports this:diff --git a/src/common/env_names.c b/src/common/env_names.cindex fd8ee94..fbe5afb 100644--- a/src/common/env_names.c+++ b/src/common/env_names.c@@ -58,6 +58,7 @@ const char *const JANAS_ENV_NAMES[] = {"JANAS_ALPHAVANTAGE_KEY","JANAS_PHOTON_URL","JANAS_POOLTRACE","JANAS_PREFETCH",+ "JANAS_PREFETCH_PREFILL","JANAS_PREFILL_BLOCK","JANAS_PROGRESS","JANAS_SHA256_C",diff --git a/src/llm/expert_cache.c b/src/llm/expert_cache.cindex 9c25edd..2da0cf0 100644--- a/src/llm/expert_cache.c+++ b/src/llm/expert_cache.c@@ -27,8 +27,10 @@#define URING_BATCH 4/* and again from this many times the threads of the I/O pool on */#define URING_DEEP_TIMES 4-/* reads ahead at most, for one layer */-#define PF_MAX 64+/* reads ahead at most, for one layer (a prefill block's), and never more+ than an eighth of the cache: kept until their layer asks, they must not+ leave the layer before too few slots to work with */+#define PF_MAX 256struct read_job {uint64_t offset;@@ -887,6 +889,8 @@ int janas_expert_cache_prefetch(struct janas_expert_cache *c, uint32_t layer,pf_settle(c, p); /* a set left over (its layer never asked) */size_t base = (size_t)layer * c->j->h.n_expert, count = 0;uint64_t bytes = c->j->layers[layer].level_bytes[c->level - 1];+ if (max > c->n_slots / 8)+ max = c->n_slots / 8;for (size_t i = 0; i < n && count < max && count < PF_MAX; i++) {if (ids[i] >= c->j->h.n_expert || c->where[base + ids[i]] != NONE)continue;@@ -955,8 +959,12 @@ int janas_expert_cache_begin(struct janas_expert_cache *c, uint32_t layer,c->cur_layer = layer;c->pass_asked += n;/* reads ahead for this layer (or for one it went past): wait for them;- counted as used where this request asks for them, then settled. The- set of a later layer stays as it is, still in flight. */+ counted as used where a request asks for them. This layer's set is+ kept until the layer is done: a prefill block asks for its experts+ 64 at a time, and settling the set at the first request left the+ slots read for the later chunks free to go before those asked (6 Oct+ 2026: 42% of the reads ahead used, the cache poorer for the decoding+ after). The set of a later layer stays as it is, still in flight. */int pf_here[2];for (int p = 0; p < 2; p++) {pf_here[p] = c->pf_n[p] && layer >= c->pf_layer[p];@@ -974,18 +982,21 @@ int janas_expert_cache_begin(struct janas_expert_cache *c, uint32_t layer,s[i] = c->where[base + ids[i]];ready[i] = s[i] != NONE;if (s[i] != NONE) {- /* a read ahead for this very layer: its set is settled here */+ /* a read ahead for this very layer: counted once */if (c->pf_flag[s[i]] == 1 && pf_here[layer & 1] &&c->pf_layer[layer & 1] == layer) {c->st.prefetch_used++;- c->pf_flag[s[i]] = 2; /* counted once */+ c->pf_flag[s[i]] = 2;}touch(c, s[i]);c->st.hits++;}}+ /* settled too: a set more than one layer ahead, which is of the pass+ before (the last layer's, when this pass reads nothing ahead) */for (int p = 0; p < 2; p++)- if (pf_here[p])+ if (c->pf_n[p] &&+ (c->pf_layer[p] < layer || c->pf_layer[p] > layer + 1))pf_settle(c, p); /* the unused ones: last in line, free to go */int queued = 0;for (size_t i = 0; i < n; i++)diff --git a/src/llm/expert_cache.h b/src/llm/expert_cache.hindex c15843c..7456d10 100644--- a/src/llm/expert_cache.h+++ b/src/llm/expert_cache.h@@ -134,7 +134,8 @@ int janas_expert_cache_finish(struct janas_expert_cache *c);/** Reads ahead up to max of the experts ids of the layer that are not in the- * cache, in the order given (the most likely first), while the caller goes+ * cache (256 at most, and an eighth of the cache's slots), in the order+ * given (the most likely first), while the caller goes* on: for a layer that will ask soon. They take the least recently used* slots and stay last in line, and until that layer's next request they are* never taken back (a read in flight must not land in a slot given todiff --git a/src/llm/model.c b/src/llm/model.cindex bb9769c..8b67360 100644--- a/src/llm/model.c+++ b/src/llm/model.c@@ -344,6 +344,27 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,#define PF_ON 60#define PF_OFF 40+/*+ * Reading ahead in prefill blocks (unless JANAS_PREFETCH_PREFILL=0 or 2):+ * on after a block whose layers would have missed more than PFB_ON experts+ * each, as a rule (the misses and those the reads ahead spared); only for+ * blocks of PFB_MIN_N tokens or more, the passes that verify drafts being+ * decoding. It costs layer l + 1's router once more on every token (1.1+ * ms a token on Qwen3-Next-80B) and spares half the time waited for the+ * disk where the cache is small for the model (6 Oct 2026).+ */+#define PFB_MIN_N 32+#define PFB_ON 32++static void prefetch_block_decide(struct janas_llm_model *m,+ const struct janas_expert_cache_stats *a,+ const struct janas_expert_cache_stats *b)+{+ uint64_t would =+ (b->misses - a->misses) + (b->prefetch_used - a->prefetch_used);+ m->pfb_on = would > (uint64_t)PFB_ON * m->n_layer;+}+static void prefetch_decide(struct janas_llm_model *m,const struct janas_expert_cache_stats *a,const struct janas_expert_cache_stats *b)@@ -385,6 +406,8 @@ int janas_llm_model_forward(struct janas_llm_model *m, const int32_t *tokens,janas_tuner_record(&m->tuner, cls, cand, dt / n);if (r == 0 && n == 1 && m->pf_mode == 1)prefetch_decide(m, &c0, &c1);+ if (r == 0 && n >= PFB_MIN_N && m->pfb_mode == 1)+ prefetch_block_decide(m, &c0, &c1);janas_m_arena_to_gpu(m);return r;}@@ -456,6 +479,8 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,return -1;if (m->pf_on && n == 1 && l + 1 < m->n_layer)janas_m_prefetch(m, l);+ else if (m->pfb_on && n >= PFB_MIN_N && l + 1 < m->n_layer)+ janas_m_prefetch_block(m, l, n);janas_m_norm_quant(m, f32(m, ly->ffn_norm), 0, n);trace_row(m, "post_norm", m->xn + (size_t)(n - 1) * dm, dm);if (ly->post_ffn) {diff --git a/src/llm/model_ffn.c b/src/llm/model_ffn.cindex 85bd5fe..efd9955 100644--- a/src/llm/model_ffn.c+++ b/src/llm/model_ffn.c@@ -319,6 +319,43 @@ void janas_m_prefetch(struct janas_llm_model *m, uint32_t l)m->phase[JANAS_PH_ROUTER] += now() - t0;}+static int by_votes(const void *a, const void *b)+{+ uint64_t x = *(const uint64_t *)a, y = *(const uint64_t *)b;+ return (x < y) - (x > y); /* descending */+}++/*+ * Reads ahead for layer l + 1 of a prefill block, as Strata does for its+ * graphics card (github.com/Niko1221/Strata): layer l + 1's router on the+ * whole block, before layer l's experts, and the experts most of its tokens+ * would choose read first, while layer l's experts are computed. Without+ * it a layer whose experts are not in the cache waits for the disk chunk by+ * chunk, with only the experts already in to compute meanwhile.+ */+void janas_m_prefetch_block(struct janas_llm_model *m, uint32_t l, uint32_t n)+{+ double t0 = now();+ uint32_t k = m->k_use, ne = m->n_expert;+ uint32_t *ids = malloc((size_t)n * k * sizeof(*ids));+ if (ids && janas_m_route_rank(m, l + 1, n, k, ids) == 0) {+ uint32_t votes[4096] = {0}; /* n_expert <= 4096, checked at load */+ for (size_t p = 0; p < (size_t)n * k; p++)+ votes[ids[p]]++;+ uint64_t key[4096];+ uint32_t nk = 0;+ for (uint32_t e = 0; e < ne; e++)+ if (votes[e])+ key[nk++] = (uint64_t)votes[e] << 32 | e;+ qsort(key, nk, sizeof(*key), by_votes);+ for (uint32_t i = 0; i < nk; i++)+ ids[i] = (uint32_t)key[i];+ janas_expert_cache_prefetch(m->cache, l + 1, ids, nk, nk);+ }+ free(ids);+ m->phase[JANAS_PH_ROUTER] += now() - t0;+}+/* The router's choice for token j: softmax, top-k, normalized weights. */static void route_tok(struct janas_llm_model *m, uint32_t j){diff --git a/src/llm/model_impl.h b/src/llm/model_impl.hindex 2772b11..9b4b23e 100644--- a/src/llm/model_impl.h+++ b/src/llm/model_impl.h@@ -281,6 +281,10 @@ struct janas_llm_model {int pf_mode, pf_on;uint32_t pf_tokens;uint64_t pf_would;+ /* reads ahead in a prefill block (janas_m_prefetch_block): pfb_mode 0+ never, 1 after a block that missed many experts (pfb_on), 2 always+ (JANAS_PREFETCH_PREFILL=0, unset, =2) */+ int pfb_mode, pfb_on;uint64_t warm_experts; /* experts preloaded at open from the profile */double warm_seconds;@@ -409,6 +413,8 @@ int janas_m_route_rank(struct janas_llm_model *m, uint32_t l, uint32_t n,uint32_t k, uint32_t *ids);/* Reads ahead for layer l + 1 of a single token (JANAS_PREFETCH). */void janas_m_prefetch(struct janas_llm_model *m, uint32_t l);+/* The same for a prefill block of n tokens (JANAS_PREFETCH_PREFILL). */+void janas_m_prefetch_block(struct janas_llm_model *m, uint32_t l, uint32_t n);int janas_m_moe_block(struct janas_llm_model *m, const struct layer *ly,struct janas_expert_cache *cache,const struct janas_jns_layer *jl, uint32_t l, uint32_t n,diff --git a/src/llm/model_load.c b/src/llm/model_load.cindex 0bfd15d..4dcbc9a 100644--- a/src/llm/model_load.c+++ b/src/llm/model_load.c@@ -1018,6 +1018,15 @@ struct janas_llm_model *janas_llm_model_load(const char *path,m->pf_mode = 2;m->pf_on = 1;}+ /* and in prefill blocks, where they pay (model.c): Qwen3-Next-80B+ with a 4 GiB cache read 4096 tokens at 41.0 token/s against 34.6+ without (6 Oct 2026); JANAS_PREFETCH_PREFILL=0 never, =2 always */+ const char *pfb = getenv("JANAS_PREFETCH_PREFILL");+ m->pfb_mode = 1;+ if (pfb && strcmp(pfb, "0") == 0)+ m->pfb_mode = 0;+ else if (pfb && strcmp(pfb, "2") == 0)+ m->pfb_mode = m->pfb_on = 2;}/* fill the cache with the experts this machine used most, before thefirst reply has to find them one token at a time. A caller that doesdiff --git a/tests/bench_long.c b/tests/bench_long.cindex 71565c4..ca5add1 100644--- a/tests/bench_long.c+++ b/tests/bench_long.c@@ -178,10 +178,19 @@ int main(int argc, char **argv)long long d1 = disk_busy_ms(argv[1]);janas_llm_model_phases(m, ph[1]);janas_expert_cache_stats(c, &cs);+ /* the experts read ahead were read from the disk too, though the+ requests then find them: counted as read, not as in RAM */printf("prefill %u tokens: %.1f s (%.1f token/s); experts %.1f%% from ""RAM, %.2f GB read, %.1f s waited\n",- np, t1 - t0, np / (t1 - t0), 100.0 * cs.hits / cs.requests,- cs.bytes_read / 1e9, cs.wait_seconds);+ np, t1 - t0, np / (t1 - t0),+ 100.0 * (cs.hits - cs.prefetch_used) / cs.requests,+ (cs.bytes_read + cs.prefetch_bytes) / 1e9, cs.wait_seconds);+ if (cs.prefetch_reads)+ printf("prefill, read ahead: %llu experts, %llu of them asked for "+ "(%.0f%%)\n",+ (unsigned long long)cs.prefetch_reads,+ (unsigned long long)cs.prefetch_used,+ 100.0 * cs.prefetch_used / cs.prefetch_reads);if (d0 >= 0 && d1 >= 0)printf("prefill, disk busy: %.2f s (%.0f%%)\n", (d1 - d0) / 1e3,100.0 * (d1 - d0) / 1e3 / (t1 - t0));@@ -209,8 +218,10 @@ int main(int argc, char **argv)janas_expert_cache_stats(c, &cs);printf("decode %u tokens after %u: %.2f s (%.2f token/s); experts ""%.1f%% from RAM, %.2f GB read (%.1f MB/token), %.2f s waited\n",- nn, np, t2 - t1, nn / (t2 - t1), 100.0 * cs.hits / cs.requests,- cs.bytes_read / 1e9, cs.bytes_read / 1e6 / nn, cs.wait_seconds);+ nn, np, t2 - t1, nn / (t2 - t1),+ 100.0 * (cs.hits - cs.prefetch_used) / cs.requests,+ (cs.bytes_read + cs.prefetch_bytes) / 1e9,+ (cs.bytes_read + cs.prefetch_bytes) / 1e6 / nn, cs.wait_seconds);if (d1 >= 0 && d2 >= 0)printf("decode, disk busy: %.2f s (%.0f%%)\n", (d2 - d1) / 1e3,100.0 * (d2 - d1) / 1e3 / (t2 - t1));