37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-09-27: Gemma 4's assistants, about 1.5x faster replies
camauri2afc5296bdac
diff --git a/ChangeLog.md b/ChangeLog.mdindex 77530e4..e6939ed 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -2,6 +2,29 @@> Curated, user-facing summary of completed work, newest first.+## [2026-09-27] - Gemma 4's assistant drafts++- **Gemma 4's assistant is taken as the MTP file** (`--mtp`, converted by+ `gguf2jns` from `mtp-gemma-4-*.gguf`): a small model of its own - 256+ wide, four layers with queries only - reading the main model's keys and+ values, fed the main model's embedding of the last token and its final+ state, drafting at that token's position and chaining further drafts at+ the same one, as it was trained. A draft costs about a millisecond.+- An Italian chat reply of 110 tokens, greedy, the same tokens with and+ without it (Core Ultra 9 185H, tokens/s): **E2B 34.7 -> 51.0 (1.47x),+ E4B 18.3 -> 26.3 (1.44x), 12B 8.5 -> 12.9 (1.51x), 26B 16.9 -> 24.5+ (1.44x)**; 64-83% of the drafts the planner chose to verify were+ accepted (on E4B llama.cpp with the same assistant accepts 50% of its+ drafts). The 12B's and 26B's assistants, 1024 wide, give a KV head+ count per layer, which is read as such.+- Its drafts are picked, as the MTP blocks' are, from a head of 32768+ token ids plus those met in the conversation, instead of all 262144: 8+ MB read a draft instead of 67, for as many accepted.+- E2B's and E4B's per-layer branch reads its f32 matrices (220 MB a token+ on E4B) once for all the tokens of a pass, with a vector dot product:+ passes over a few tokens 17% faster, single tokens 4%.+ `JANAS_DRAFT_VOCAB` sets the draft head's size, to experiment.+## [2026-09-27] - janas-try reports only this run- **A test report carries only the speeds this run measured.** Thediff --git a/MODELS.md b/MODELS.mdindex db5ca9a..a800260 100644--- a/MODELS.md+++ b/MODELS.md@@ -46,9 +46,9 @@ fingerprint of what the recipe produces — not forty gigabytes. Convert your owcopy, compare, and you know you are running the file the numbers in thisrepository were measured on.-Not a claim, a measurement: each of the four models below was converted again-from its GGUF and compared, byte for byte, with the file this project runs.-All four matched.+Not a claim, a measurement: four of the Qwen models below were converted+again from their GGUF and compared, byte for byte, with the files this+project runs, and so were the eight Gemma 4 files. All of them matched.### The recipe@@ -149,6 +149,36 @@ cut. Two conversions give the same bytes.1e8eb8cbeba7a1a657354d259c5730bb79f6087160a9ca8da901bd08bdbe943f qwen3-embedding-0.6b-q8.jns (gguf2jns)```+**Gemma 4: E2B, E4B, 12B and 26B-A4B, and their assistants**, from+[`unsloth/gemma-4-E2B-it-GGUF`](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF),+[`unsloth/gemma-4-E4B-it-GGUF`](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF),+[`unsloth/gemma-4-12b-it-GGUF`](https://huggingface.co/unsloth/gemma-4-12b-it-GGUF)+and [`unsloth/gemma-4-26B-A4B-it-GGUF`](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF).+Each converted in one step, as it is kept here, and each converted twice on+27 September 2026: the same bytes. The assistant, Gemma's drafting model+(the `mtp-` file, given to the chat with `--mtp`), converts with+`gguf2jns` like the model. The 26B's feed-forwards are widened with zeros+to a multiple of 256, which is why its file is larger than its GGUF.++```+740185b21d22ceb83a11c3aa62ad5842ef32c70f6096d756bbee85a1e4ec34b8 gemma-4-E2B-it-Q4_K_M.gguf+1f18e08878df41f377d563b38048d41841c7973ddea845cd9672fcf23cbd864b gemma-4-e2b-it-q4km.jns (gguf2jns)+9eba819938efccfd6044f8af84e3bbfddc639a2bcf32ebc36420e6a649191919 mtp-gemma-4-E2B-it.gguf+79df1f3c817542b3867a0940b1a5b7330d478bddb4102ec90ff817334877cd62 gemma-4-e2b-it-mtp.jns (gguf2jns)+85a896a047553e842f25297ee5b031d64ff30147d9c4af17b1e4b394cd1fab87 gemma-4-E4B-it-Q4_K_M.gguf+7ec828e9bdc1ada2eda907e93b87072f3180f2e8cba5ccedcd360d1a53858675 gemma-4-e4b-it-q4km.jns (gguf2jns)+b6a723115efa510d3b3215db1e26790dae84cd08c2134a764f3d194f1f0c3376 mtp-gemma-4-E4B-it.gguf+7c6fdb74ba1eb11f9d47b15097a09cce0142aea380a41f1986ba7b8ec4785b53 gemma-4-e4b-it-mtp.jns (gguf2jns)+0a270ec9fe6b34f4a0d33992b6135117b484ebc4766ab76b51d4ae8c457e4c42 gemma-4-12b-it-Q4_K_M.gguf+cae09e04598891dea6e9cd93fb496fbde08d822ac35de4ff45dbfab1ea25c4dd gemma-4-12b-it-q4km.jns (gguf2jns)+145db9094bc0f85f1701e255a2ed216dcc9800fc8bc8631ad00905b456bd451b mtp-gemma-4-12b-it.gguf+35e640f20dd74e862df9c1d949ad4edca9f58beef0cb29e99cccc6c99297826f gemma-4-12b-it-mtp.jns (gguf2jns)+f2c28b3dc4776931ac6f879e11f203dec637ea0f14267a86ec8f6165f63f293f gemma-4-26B-A4B-it-UD-Q4_K_M.gguf+35c3845bbc21de52bf630d3c7455b4bb5379c34487b5925e020ec2e573931666 gemma-4-26b-a4b-it-udq4km.jns (gguf2jns)+6326fb9f5e487aa8dcdd313a091e3c67724cb2a666ec3b7d2895b5b26d93ed1b mtp-gemma-4-26B-A4B-it.gguf+6ba24ee91fe6fe904b29919acbb654a66d9caa0aba67db1df9f921a76e5bcedf gemma-4-26b-it-mtp.jns (gguf2jns)+```+The multi-token prediction block is not converted from a GGUF — GGUF filesleave it out — but from the original checkpoint's last shard, quantized toQ4_K by `tools/hf2jns_mtp`, which also copies the metadata of the converteddiff --git a/README.md b/README.mdindex a0d70d9..0b3c966 100644--- a/README.md+++ b/README.md@@ -89,7 +89,12 @@ report from `janas-bench` is the single most useful thing you can send back**chat and tools: their per-layer embeddings - a table as large as half ofE2B's file - are read from the file a row at a time and never loaded, soE2B ran here in 1.5 GB of memory (a 3.1 GB file) and E4B in 3.1 GB (5.0- GB). Against llama.cpp on 200 tokens of prose the most likely token+ GB). Each of the four has an **assistant**, a small drafting model of+ its own that reads the main model's keys and values, taken as its+ `--mtp` file: on an Italian chat reply of 110 tokens, greedy, it took+ E2B from 34.7 tokens/s to 51.0, E4B from 18.3 to 26.3, the 12B from 8.5+ to 12.9 and the 26B from 16.9 to 24.5 on the development laptop, the+ same tokens with and without it. Against llama.cpp on 200 tokens of prose the most likely tokenagrees at 160 positions on the 12B (mean logit difference 0.84) and at145 on the 26B (1.43); on another 200-token text at 191 on E2B (0.45) and195 on E4B (0.28). The larger two turn small numerical differences into@@ -312,11 +317,33 @@ a minute. It also takes the shards themselves, when you have them on disk(`hf2jns_mtp <model.jns> <shard>... <out.jns>`). Qwen3.6-35B-A3B needs none ofthis: its GGUF keeps the block.+**Gemma 4** comes from unsloth's GGUF files, converted in one step, and+so does its assistant (the `mtp-gemma-4-*.gguf` file beside the model,+under 0.5 GB), which serves as the multi-token prediction file:++```sh+huggingface-cli download unsloth/gemma-4-E4B-it-GGUF \+ --include "gemma-4-E4B-it-Q4_K_M.gguf" "mtp-gemma-4-E4B-it.gguf" \+ --local-dir models/gguf+bin/x86_64-linux/gguf2jns models/gguf/gemma-4-E4B-it-Q4_K_M.gguf \+ models/gemma-4-e4b.jns+bin/x86_64-linux/gguf2jns models/gguf/mtp-gemma-4-E4B-it.gguf \+ models/gemma-4-e4b-mtp.jns+```++The same for E2B, the 12B (`gemma-4-12b-it-Q4_K_M.gguf`) and the 26B+(`gemma-4-26B-A4B-it-UD-Q4_K_M.gguf`); the fingerprints are in+[MODELS.md](MODELS.md#the-fingerprints). E2B's per-layer table, half of its+file, is read from the file a token at a time and never loaded: E2B ran+here in 1.5 GB of memory, E4B in 3.1 GB.+**And chat:**```shbin/x86_64-linux/janas-chat models/qwen3-next.jns \--mtp models/qwen3-next-mtp.jns --stats+bin/x86_64-linux/janas-chat models/gemma-4-e4b.jns \+ --mtp models/gemma-4-e4b-mtp.jns --stats```The first start reads the model's resident weights (a couple of GB) and then@@ -364,8 +391,11 @@ So no wrong guess can survive into the text, and with the same seed the replyis the one you would have got without any of it - checked on every run of thetests, 256 tokens with drafts and 256 without, from one seed, identical. Onthe development laptop `--mtp` takes Qwen3-Next-80B-A3B from 23 tok/s to 36-and Qwen3.5-9B from 13 to 22 on an English explanation; how much it gives-depends on the text, and on Italian prose a small model gains little.+and Qwen3.5-9B from 13 to 22 on an English explanation, and Gemma 4's+assistants take its four models about one and a half times faster on an+Italian chat reply (E4B from 18 tok/s to 26, the 12B from 8.5 to 12.9);+how much it gives depends on the text, and on Italian prose a small Qwen+gains little.`--no-spec` and `/spec off` turn it off; they buy nothing but time, and arethere to measure with.diff --git a/src/llm/generate.c b/src/llm/generate.cindex 827ac38..2f16ad9 100644--- a/src/llm/generate.c+++ b/src/llm/generate.c@@ -232,8 +232,9 @@ static int mtp_step(struct janas_llm_model *m, const int32_t *hist,if (pl && plan_drafts(pl, survs, got) < got)break;double tc = now();- if (janas_llm_mtp_draft(m, &draft[got - 1], hrow, 1,- pos + acc + (uint32_t)got, &draft[got],+ uint32_t at = janas_llm_mtp_chain_fixed(m) ? pos + acc+ : pos + acc + (uint32_t)got;+ if (janas_llm_mtp_draft(m, &draft[got - 1], hrow, 1, at, &draft[got],&conf) != 0 ||(surv *= conf) < min_conf)break; /* past the context, or not sure enough */diff --git a/src/llm/model.c b/src/llm/model.cindex d6a9ea8..a48f089 100644--- a/src/llm/model.c+++ b/src/llm/model.c@@ -83,6 +83,11 @@ static void embed(struct janas_llm_model *m, int32_t token, float *x)}}+void janas_m_embed(struct janas_llm_model *m, int32_t token, float *x)+{+ embed(m, token, x);+}+/* RoPE angles of positions pos0 .. pos0 + n - 1 for n_rot dimensions, thesame for every head; ff: a divisor of each pair's angle, or NULL. */static void rope_table(float *cs, float *sn, uint32_t n, uint32_t pos0,@@ -457,8 +462,9 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,double td = now();uint32_t first = all_logits ? 0 : n - 1;/* with an MTP block, every token's final hidden state is kept for it */- janas_m_norm_quant(m, f32(m, m->out_norm), m->mtp ? 0 : first, n);- if (m->mtp)+ int keep = m->mtp || m->asst;+ janas_m_norm_quant(m, f32(m, m->out_norm), keep ? 0 : first, n);+ if (keep)memcpy(m->hid, m->xn, (size_t)n * dm * sizeof(float));struct janas_matvec_task to = {.type = (int)m->output->type,.w = data(m, m->output),@@ -496,7 +502,12 @@ uint32_t janas_llm_model_max_block(const struct janas_llm_model *m)int janas_llm_model_has_mtp(const struct janas_llm_model *m){- return m->mtp;+ return m->mtp || m->asst;+}++int janas_llm_mtp_chain_fixed(const struct janas_llm_model *m)+{+ return m->asst != NULL;}int janas_llm_model_recurrent(const struct janas_llm_model *m)@@ -534,6 +545,19 @@ exp_sum_avx2(const float *l, uint32_t n, float mx, uint32_t *done)}#endif+double janas_m_exp_sum(const float *l, uint32_t n, float mx)+{+ double sum = 0;+ uint32_t i = 0;+#if defined(__x86_64__)+ if (__builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma"))+ sum = exp_sum_avx2(l, n, mx, &i);+#endif+ for (; i < n; i++)+ sum += expf(l[i] - mx);+ return sum;+}+void janas_llm_mtp_note_tokens(struct janas_llm_model *m, const int32_t *tokens,uint32_t n){@@ -545,8 +569,7 @@ void janas_llm_mtp_note_tokens(struct janas_llm_model *m, const int32_t *tokens,m->dh_n == m->dh_cap)continue;memcpy(m->dh_w + (size_t)m->dh_n * m->dh_row,- (const uint8_t *)data(m, m->output) + (size_t)t * m->dh_row,- m->dh_row);+ m->dh_src + (size_t)t * m->dh_row, m->dh_row);m->dh_ids[m->dh_n++] = t;m->dh_in[t] = 1;}@@ -556,6 +579,14 @@ int janas_llm_mtp_draft(struct janas_llm_model *m, const int32_t *tokens,const float *hidden, uint32_t n, uint32_t pos0,int32_t *token, float *conf){+ if (m->asst) { /* Gemma 4's assistant: the last row alone */+ uint32_t dm = m->d_model;+ if (n < 1 || n > m->blk)+ return -1;+ return janas_m_assist_draft(+ m, tokens[n - 1], hidden + (size_t)(n - 1) * dm, pos0 + n - 1,+ m->mtp_hid + (size_t)(n - 1) * dm, token, conf);+ }if (janas_llm_mtp_forward(m, tokens, hidden, n, pos0, NULL, 0) != 0)return -1;janas_m_apply_candidate(m,@@ -596,6 +627,8 @@ int janas_llm_mtp_forward(struct janas_llm_model *m, const int32_t *tokens,const float *hidden, uint32_t n, uint32_t pos0,float *logits, int all_logits){+ if (m->asst) /* no keys and values of its own: nothing to fill */+ return logits || n < 1 || n > m->blk ? -1 : 0;if (!m->mtp || n < 1 || n > m->blk || pos0 + n > m->n_ctx)return -1;for (uint32_t j = 0; j < n; j++)diff --git a/src/llm/model.h b/src/llm/model.hindex 06f298e..0b42726 100644--- a/src/llm/model.h+++ b/src/llm/model.h@@ -184,6 +184,10 @@ int janas_llm_model_decode(struct janas_llm_model *m, int32_t token,* guesses are chained.*/int janas_llm_model_has_mtp(const struct janas_llm_model *m);+/* 1 when chained drafts keep the position of the first (Gemma 4's+ assistant, which reads the main model's cache and has none of its+ own); 0 when each is one position further (an MTP block) */+int janas_llm_mtp_chain_fixed(const struct janas_llm_model *m);/* 1 when the model carries a recurrent state (qwen3next, qwen35moe): it cango back no further than the start of its last forward call. */int janas_llm_model_recurrent(const struct janas_llm_model *m);diff --git a/src/llm/model_assist.c b/src/llm/model_assist.cnew file mode 100644index 0000000..3472398--- /dev/null+++ b/src/llm/model_assist.c@@ -0,0 +1,451 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * model_assist.c - Gemma 4's assistant: the drafts of a small model of its+ * own (gemma4-assistant, a JNS file given as the MTP block).+ *+ * The assistant is narrow (256 wide for E2B and E4B) and has a few layers+ * (four) with queries only: it keeps no keys and values, and reads the main+ * model's - its sliding-window layers the main model's last sliding layer's+ * cache, its full layer the last full one's. Its input is the main model's+ * embedding of the last token beside the main model's final state (after+ * the final norm), projected to its width; its output, logits over the+ * whole vocabulary and a state projected back to the main model's width,+ * on which the next draft is chained.+ *+ * A draft of token t_{p+1} (Janas's row: the token with the main model's+ * state h_p, "at" p) runs at t_{p+1}'s own position, p + 1, and attends+ * the main model's cache up to p: the token itself is not in it yet. The+ * drafts chained after it keep that same position, as the model was+ * trained (Hugging Face's gemma4_assistant), and so read the same cache.+ */+#define _GNU_SOURCE+#include <math.h>+#include <stdlib.h>+#include <string.h>+#include <unistd.h>++#include "matvec.h"+#include "vmath.h"++#include "model_impl.h"++#define ASST_MAX_LAYERS 8++struct asst_layer {+ int swa;+ uint32_t hd, n_rot, slot; /* slot: the main model's KV it reads */+ const struct janas_jns_tensor *attn_norm, *wq, *q_norm, *wo, *post_attn,+ *ffn_norm, *post_ffn;+ float out_scale;+ const uint8_t *ffn[3]; /* gate, up, down: in the slot as read */+ struct janas_jns_matrix mx[3];+};++struct janas_assist {+ struct janas_jns j;+ int open;+ uint8_t *res, *slots;+ uint32_t n_layer, d, n_head, ff, hd_max;+ float eps;+ const float *rope_freqs;+ float rope_base, rope_base_swa;+ const struct janas_jns_tensor *embd, *out_norm, *pre, *post;+ struct asst_layer L[ASST_MAX_LAYERS];+ struct janas_attn *attn, *attn_swa;+ /* one row's scratch */+ float *cat, *x, *xn, *q, *att, *g, *u, *logits, *cs, *sn, *cs_swa, *sn_swa;+ struct janas_block_q8k *catq, *xq, *attq, *hq;+};++static const void *adata(const struct janas_assist *a,+ const struct janas_jns_tensor *t)+{+ return a->res + (t->offset - a->j.h.resident_offset);+}++static int ameta_u32(const struct janas_assist *a, const char *suffix,+ uint32_t *v)+{+ char key[128];+ snprintf(key, sizeof(key), "gemma4-assistant.%s", suffix);+ return janas_jns_meta_u32(&a->j, key, v);+}++static int ameta_f32(const struct janas_assist *a, const char *suffix, float *v)+{+ char key[128];+ snprintf(key, sizeof(key), "gemma4-assistant.%s", suffix);+ return janas_jns_meta_f32(&a->j, key, v);+}++/* A tensor of the assistant's file, of the expected shape (d1 0: any);+ f32: of type f32, else of a type the products know. */+static const struct janas_jns_tensor *aneed(const struct janas_assist *a,+ const char *name, int f32,+ uint64_t d0, uint64_t d1)+{+ const struct janas_jns_tensor *t = janas_jns_tensor(&a->j, name);+ if (!t || t->dims[0] != d0 || (d1 && t->dims[1] != d1))+ return NULL;+ if (f32 ? t->type != 0 : janas_qtype_block_size((int)t->type) == 0)+ return NULL;+ return t;+}++int janas_m_assist_is(const char *path)+{+ struct janas_jns j;+ char err[256];+ if (janas_jns_open(path, &j, err, sizeof(err)) != 0)+ return 0;+ int is = strncmp(j.h.arch, "gemma4-assistant", sizeof(j.h.arch)) == 0;+ janas_jns_close(&j);+ return is;+}++void janas_m_assist_free(struct janas_llm_model *m)+{+ struct janas_assist *a = m->asst;+ if (!a)+ return;+ if (a->open)+ janas_jns_close(&a->j);+ janas_attn_destroy(a->attn);+ janas_attn_destroy(a->attn_swa);+ free(a->res);+ free(a->slots);+ free(a->cat);+ free(a->x);+ free(a->xn);+ free(a->q);+ free(a->att);+ free(a->g);+ free(a->u);+ free(a->logits);+ free(a->cs);+ free(a->sn);+ free(a->cs_swa);+ free(a->sn_swa);+ free(a->catq);+ free(a->xq);+ free(a->attq);+ free(a->hq);+ free(a);+ m->asst = NULL;+}++static int fail(char *err, size_t err_len, const char *path, const char *why)+{+ snprintf(err, err_len, "assistant %s: %s", path, why);+ return -1;+}++int janas_m_assist_load(struct janas_llm_model *m, const char *path,+ int attn_scores, char *err, size_t err_len)+{+ struct janas_assist *a = calloc(1, sizeof(*a));+ if (!a)+ return fail(err, err_len, path, "out of memory");+ m->asst = a;+ char why[256];+ if (janas_jns_open(path, &a->j, why, sizeof(why)) != 0)+ return fail(err, err_len, path, why);+ a->open = 1;+ const struct janas_jns_header *h = &a->j.h;+ uint32_t dm = m->d_model, out = 0, kl = 0, kl_swa = 0, shared = 0;+ const uint8_t *pat = NULL;+ uint64_t np = 0;+ a->n_layer = h->n_layer;+ a->d = h->d_model;+ if (!m->a->swa || m->n_layer < 2 || a->n_layer == 0 ||+ a->n_layer > ASST_MAX_LAYERS ||+ ameta_u32(a, "embedding_length_out", &out) || out != dm ||+ ameta_u32(a, "attention.head_count", &a->n_head) ||+ ameta_u32(a, "attention.key_length", &kl) ||+ ameta_u32(a, "attention.key_length_swa", &kl_swa) ||+ ameta_u32(a, "attention.shared_kv_layers", &shared) ||+ ameta_f32(a, "attention.layer_norm_rms_epsilon", &a->eps) ||+ ameta_f32(a, "rope.freq_base", &a->rope_base) ||+ ameta_f32(a, "rope.freq_base_swa", &a->rope_base_swa) ||+ janas_jns_meta_bytes(+ &a->j, "gemma4-assistant.attention.sliding_window_pattern", &np,+ &pat) ||+ np < a->n_layer || shared != a->n_layer || kl != m->head_dim ||+ kl_swa != m->head_swa || a->d % JANAS_QK || a->n_head == 0 ||+ a->n_head % m->n_head_kv || a->n_head % m->n_kv_swa)+ return fail(err, err_len, path, "not an assistant of this model");+ /* its KV heads, one count or one per layer (12B, 26B): those of the+ main model's caches it reads */+ uint32_t kv_one = 0;+ uint64_t n_kv_arr = 0;+ const int32_t *kv_arr = NULL;+ if (janas_jns_meta_ints(&a->j, "gemma4-assistant.attention.head_count_kv",+ &n_kv_arr, &kv_arr) != 0 ||+ n_kv_arr < a->n_layer) {+ kv_arr = NULL;+ if (ameta_u32(a, "attention.head_count_kv", &kv_one))+ return fail(err, err_len, path, "no KV head count");+ }+ a->ff = a->j.layers[0].m[JANAS_JNS_GATE].rows;+ a->hd_max = kl > kl_swa ? kl : kl_swa;+ /* the resident tensors, then each layer's feed-forward (its one slot) */+ a->res = aligned_alloc(4096, h->resident_bytes);+ uint64_t sb = 0;+ for (uint32_t l = 0; l < a->n_layer; l++)+ if (a->j.layers[l].slot_bytes > sb)+ sb = a->j.layers[l].slot_bytes;+ a->slots = aligned_alloc(4096, sb * a->n_layer);+ if (!a->res || !a->slots)+ return fail(err, err_len, path, "out of memory");+ struct {+ uint8_t *dst;+ uint64_t off, len;+ } rd[1 + ASST_MAX_LAYERS];+ rd[0].dst = a->res;+ rd[0].off = h->resident_offset;+ rd[0].len = h->resident_bytes;+ for (uint32_t l = 0; l < a->n_layer; l++) {+ rd[1 + l].dst = a->slots + (size_t)l * sb;+ rd[1 + l].off = janas_jns_expert_offset(&a->j, l, 0);+ rd[1 + l].len = a->j.layers[l].slot_bytes;+ }+ for (uint32_t i = 0; i <= a->n_layer; i++)+ for (uint64_t done = 0; done < rd[i].len;) {+ ssize_t r = pread(a->j.fd, rd[i].dst + done, rd[i].len - done,+ (off_t)(rd[i].off + done));+ if (r <= 0)+ return fail(err, err_len, path, "cannot read it");+ done += (uint64_t)r;+ }+ uint32_t d = a->d;+ if (!(a->embd = aneed(a, "token_embd.weight", 0, d, m->n_vocab)) ||+ !(a->out_norm = aneed(a, "output_norm.weight", 1, d, 0)) ||+ !(a->pre = aneed(a, "nextn.pre_projection.weight", 0, 2 * dm, d)) ||+ !(a->post = aneed(a, "nextn.post_projection.weight", 0, d, dm)))+ return fail(err, err_len, path, "tensors missing or unexpected");+ const struct janas_jns_tensor *rf =+ janas_jns_tensor(&a->j, "rope_freqs.weight");+ if (rf && (rf->type != 0 || rf->dims[0] != m->n_rot / 2))+ return fail(err, err_len, path, "rope_freqs unexpected");+ a->rope_freqs = rf ? adata(a, rf) : NULL;+ for (uint32_t l = 0; l < a->n_layer; l++) {+ struct asst_layer *L = &a->L[l];+ L->swa = pat[l] != 0;+ L->hd = L->swa ? m->head_swa : m->head_dim;+ L->n_rot = L->swa ? m->n_rot_swa : m->n_rot;+ /* the main model's last layer of the same kind */+ const struct layer *src = &m->layers[m->n_layer - (L->swa ? 2 : 1)];+ int32_t kvh = (int32_t)kv_one;+ if (kv_arr)+ memcpy(&kvh, kv_arr + l, sizeof(kvh));+ if (src->swa != L->swa || kvh <= 0 || src->n_kv != (uint32_t)kvh)+ return fail(err, err_len, path, "its layers do not match");+ L->slot = src->slot;+ uint32_t qd = a->n_head * L->hd;+ char name[64];+#define T(field, suffix, f32, d0, d1) \+ snprintf(name, sizeof(name), "blk.%u." suffix, l); \+ if (!(L->field = aneed(a, name, f32, d0, d1))) \+ return fail(err, err_len, path, "a layer's tensors");+ T(attn_norm, "attn_norm.weight", 1, d, 0)+ T(wq, "attn_q.weight", 0, d, qd)+ T(q_norm, "attn_q_norm.weight", 1, L->hd, 0)+ T(wo, "attn_output.weight", 0, qd, d)+ T(post_attn, "post_attention_norm.weight", 1, d, 0)+ T(ffn_norm, "ffn_norm.weight", 1, d, 0)+ T(post_ffn, "post_ffw_norm.weight", 1, d, 0)+#undef T+ snprintf(name, sizeof(name), "blk.%u.layer_output_scale.weight", l);+ const struct janas_jns_tensor *os = janas_jns_tensor(&a->j, name);+ L->out_scale = os && os->type == 0 && os->dims[0] == 1+ ? *(const float *)adata(a, os)+ : 1.0f;+ for (int p = 0; p < 3; p++) {+ L->mx[p] = a->j.layers[l].m[p];+ L->ffn[p] = a->slots + (size_t)l * sb + L->mx[p].offset;+ }+ if (L->mx[JANAS_JNS_GATE].rows != a->ff ||+ L->mx[JANAS_JNS_GATE].cols != d ||+ L->mx[JANAS_JNS_DOWN].rows != d || a->ff % JANAS_QK)+ return fail(err, err_len, path, "a feed-forward's shape");+ }+ /* attention over the main model's caches with the assistant's heads.+ Its query stands at p + 1 but, the token not being in the cache,+ is attended as p: one position less of window keeps what it sees+ p + 1 - window + 1 .. p, as for the main model at p + 1 */+ uint32_t s_full = m->layers[m->n_layer - 1].slot,+ s_swa = m->layers[m->n_layer - 2].slot;+ int n_thr = janas_pool_size(m->compute);+ a->attn = janas_attn_create(a->n_head, m->n_head_kv, m->head_dim,+ m->kv_cap[s_full], n_thr, attn_scores);+ a->attn_swa = janas_attn_create(a->n_head, m->n_kv_swa, m->head_swa,+ m->kv_cap[s_swa], n_thr, attn_scores);+ if (!a->attn || !a->attn_swa)+ return fail(err, err_len, path, "out of memory");+ janas_attn_set_kv16(a->attn, m->kv16);+ janas_attn_set_scale(a->attn, 1.0f);+ janas_attn_set_ring(a->attn, m->kv_ring[s_full]);+ janas_attn_set_kv16(a->attn_swa, m->kv16);+ janas_attn_set_scale(a->attn_swa, 1.0f);+ janas_attn_set_ring(a->attn_swa, m->kv_ring[s_swa]);+ janas_attn_set_window(a->attn_swa, m->window > 1 ? m->window - 1 : 1);+ size_t qd = (size_t)a->n_head * a->hd_max;+ a->cat = malloc(2 * dm * sizeof(float));+ a->x = malloc(d * sizeof(float));+ a->xn = malloc((d > dm ? d : dm) * sizeof(float));+ a->q = malloc(qd * sizeof(float));+ a->att = malloc(qd * sizeof(float));+ a->g = malloc(a->ff * sizeof(float));+ a->u = malloc(a->ff * sizeof(float));+ a->logits = malloc((size_t)m->n_vocab * sizeof(float));+ a->cs = malloc(m->n_rot / 2 * sizeof(float));+ a->sn = malloc(m->n_rot / 2 * sizeof(float));+ a->cs_swa = malloc(m->n_rot_swa / 2 * sizeof(float));+ a->sn_swa = malloc(m->n_rot_swa / 2 * sizeof(float));+ a->catq = malloc(2 * dm / JANAS_QK * sizeof(*a->catq));+ a->xq = malloc(d / JANAS_QK * sizeof(*a->xq));+ a->attq = malloc(qd / JANAS_QK * sizeof(*a->attq));+ a->hq = malloc(a->ff / JANAS_QK * sizeof(*a->hq));+ if (!m->hid)+ m->hid = malloc((size_t)m->blk * dm * sizeof(float));+ if (!m->mtp_hid)+ m->mtp_hid = malloc((size_t)m->blk * dm * sizeof(float));+ if (!a->cat || !a->x || !a->xn || !a->q || !a->att || !a->g || !a->u ||+ !a->logits || !a->cs || !a->sn || !a->cs_swa || !a->sn_swa ||+ !a->catq || !a->xq || !a->attq || !a->hq || !m->hid || !m->mtp_hid)+ return fail(err, err_len, path, "out of memory");+ return 0;+}++/*+ * The draft head, as the MTP block's (model_load.c): the assistant's own+ * output rows for the base lowest token ids - SentencePiece's most frequent+ * pieces come first - and every token met in the conversation. Its full+ * head is the whole vocabulary, 67 MB read for every draft on E4B; 32768+ * rows are 8 MB. base >= the vocabulary keeps it whole.+ */+int janas_m_assist_head(struct janas_llm_model *m, uint32_t base)+{+ const struct janas_assist *a = m->asst;+ return janas_m_init_draft_head(+ m, a->res + (a->embd->offset - a->j.h.resident_offset),+ a->d / JANAS_QK * janas_qtype_block_size((int)a->embd->type), base);+}++/* y = W x, x in Q8_K, on the compute pool */+static void mv(struct janas_llm_model *m, int type, const void *w,+ const struct janas_block_q8k *x, float *y, uint32_t rows,+ uint32_t cols)+{+ struct janas_matvec_task t = {.type = type,+ .w = w,+ .x = x,+ .y = y,+ .rows = rows,+ .cols = cols,+ .n_vec = 1};+ janas_gpu_matvec_group(NULL, m->compute, &t, 1);+}++static void mvt(struct janas_llm_model *m, const struct janas_assist *a,+ const struct janas_jns_tensor *t,+ const struct janas_block_q8k *x, float *y)+{+ mv(m, (int)t->type, adata(a, t), x, y, (uint32_t)t->dims[1],+ (uint32_t)t->dims[0]);+}++static void rope_at(float *cs, float *sn, uint32_t pos, uint32_t n_rot,+ float base, const float *ff)+{+ for (uint32_t i = 0; i < n_rot / 2; i++) {+ float theta = (float)pos * powf(base, -2.0f * (float)i / (float)n_rot);+ if (ff)+ theta /= ff[i];+ cs[i] = cosf(theta);+ sn[i] = sinf(theta);+ }+}++static void rope_neox(float *v, uint32_t n, const float *cs, const float *sn)+{+ uint32_t half = n / 2;+ for (uint32_t i = 0; i < half; i++) {+ float x0 = v[i], x1 = v[i + half];+ v[i] = x0 * cs[i] - x1 * sn[i];+ v[i + half] = x0 * sn[i] + x1 * cs[i];+ }+}++/*+ * The draft after token (whose state from the main model, or from the+ * assistant's previous draft, is h) for position p + 1, the main model's+ * cache holding up to p: the likeliest token, its probability, and the+ * state to chain the next draft on (hnext, d_model wide).+ */+int janas_m_assist_draft(struct janas_llm_model *m, int32_t token,+ const float *h, uint32_t p, float *hnext, int32_t *out,+ float *conf)+{+ struct janas_assist *a = m->asst;+ uint32_t dm = m->d_model, d = a->d;+ if (p + 1 >= m->n_ctx || token < 0 || (uint32_t)token >= m->n_vocab)+ return -1;+ double t0 = now();+ /* [embedding of the token, as the main model scales it; h] */+ janas_m_embed(m, token, a->cat);+ memcpy(a->cat + dm, h, dm * sizeof(float));+ janas_q8k_quantize(a->cat, a->catq, 2 * dm);+ mvt(m, a, a->pre, a->catq, a->x);+ rope_at(a->cs, a->sn, p + 1, m->n_rot, a->rope_base, a->rope_freqs);+ rope_at(a->cs_swa, a->sn_swa, p + 1, m->n_rot_swa, a->rope_base_swa, NULL);+ for (uint32_t l = 0; l < a->n_layer; l++) {+ const struct asst_layer *L = &a->L[l];+ uint32_t qd = a->n_head * L->hd;+ rms_norm(a->xn, a->x, adata(a, L->attn_norm), d, a->eps);+ janas_q8k_quantize(a->xn, a->xq, d);+ mvt(m, a, L->wq, a->xq, a->q);+ for (uint32_t hh = 0; hh < a->n_head; hh++) {+ float *v = a->q + (size_t)hh * L->hd;+ rms_norm(v, v, adata(a, L->q_norm), L->hd, a->eps);+ rope_neox(v, L->n_rot, L->swa ? a->cs_swa : a->cs,+ L->swa ? a->sn_swa : a->sn);+ }+ janas_attn_run(L->swa ? a->attn_swa : a->attn, m->compute, a->q,+ m->kcache + m->kv_off[L->slot],+ m->kscale + m->ks_off[L->slot],+ m->vcache + m->kv_off[L->slot],+ m->vscale + m->ks_off[L->slot], 1, p, a->att);+ janas_q8k_quantize(a->att, a->attq, qd);+ mvt(m, a, L->wo, a->attq, a->xn);+ rms_norm(a->xn, a->xn, adata(a, L->post_attn), d, a->eps);+ for (uint32_t i = 0; i < d; i++)+ a->x[i] += a->xn[i];+ rms_norm(a->xn, a->x, adata(a, L->ffn_norm), d, a->eps);+ janas_q8k_quantize(a->xn, a->xq, d);+ mv(m, (int)L->mx[0].type, L->ffn[0], a->xq, a->g, a->ff, d);+ mv(m, (int)L->mx[1].type, L->ffn[1], a->xq, a->u, a->ff, d);+ for (uint32_t i = 0; i < a->ff; i++)+ a->g[i] = janas_gelu_mul_det(a->g[i], a->u[i]);+ janas_q8k_quantize(a->g, a->hq, a->ff);+ mv(m, (int)L->mx[2].type, L->ffn[2], a->hq, a->xn, d, a->ff);+ rms_norm(a->xn, a->xn, adata(a, L->post_ffn), d, a->eps);+ for (uint32_t i = 0; i < d; i++)+ a->x[i] = (a->x[i] + a->xn[i]) * L->out_scale;+ }+ rms_norm(a->xn, a->x, adata(a, a->out_norm), d, a->eps);+ janas_q8k_quantize(a->xn, a->xq, d);+ mv(m, (int)a->embd->type, m->dh_w, a->xq, a->logits, m->dh_n, d);+ mvt(m, a, a->post, a->xq, hnext);+ /* the likeliest token of the head and its probability within it */+ const float *lg = a->logits;+ uint32_t best = 0;+ for (uint32_t i = 1; i < m->dh_n; i++)+ if (lg[i] > lg[best])+ best = i;+ *out = m->dh_ids[best];+ *conf = (float)(1.0 / janas_m_exp_sum(lg, m->dh_n, lg[best]));+ m->phase[JANAS_PH_OUTPUT] += now() - t0;+ return 0;+}diff --git a/src/llm/model_impl.h b/src/llm/model_impl.hindex dc84a19..571a745 100644--- a/src/llm/model_impl.h+++ b/src/llm/model_impl.h@@ -203,6 +203,8 @@ struct janas_llm_model {* call, mtp_hid the MTP's own (to chain guesses).*/struct janas_jns mtp_j;+ /* Gemma 4's assistant, in place of an MTP block (model_assist.c) */+ struct janas_assist *asst;uint8_t *mtp_res;struct layer mtp_layer;struct janas_expert_cache *mtp_cache;@@ -222,6 +224,7 @@ struct janas_llm_model {* token seen in the context, at a fraction of the full head's reads.*/uint8_t *dh_w, *dh_in; /* rows; per token id: 1 if in the head */+ const uint8_t *dh_src; /* the full head the rows come from */int32_t *dh_ids;uint32_t dh_n, dh_cap;size_t dh_row;@@ -332,6 +335,20 @@ void janas_m_attn_layer(struct janas_llm_model *m, const struct layer *ly,uint32_t n, uint32_t pos0);void janas_m_rec_layer(struct janas_llm_model *m, const struct layer *ly,uint32_t n);+void janas_m_embed(struct janas_llm_model *m, int32_t token, float *x);+/* sum of exp(l[i] - mx), i < n, vectorized where the CPU can */+double janas_m_exp_sum(const float *l, uint32_t n, float mx);+/* the draft head: the first base rows of the head src (rows of row_bytes) */+int janas_m_init_draft_head(struct janas_llm_model *m, const uint8_t *src,+ size_t row_bytes, uint32_t base);+int janas_m_assist_is(const char *path);+int janas_m_assist_load(struct janas_llm_model *m, const char *path,+ int attn_scores, char *err, size_t err_len);+void janas_m_assist_free(struct janas_llm_model *m);+int janas_m_assist_head(struct janas_llm_model *m, uint32_t base);+int janas_m_assist_draft(struct janas_llm_model *m, int32_t token,+ const float *h, uint32_t p, float *hnext, int32_t *out,+ float *conf);int janas_m_ple_inputs(struct janas_llm_model *m, const int32_t *tokens,uint32_t n);void janas_m_ple_layer(struct janas_llm_model *m, const struct layer *ly,diff --git a/src/llm/model_load.c b/src/llm/model_load.cindex 1ef2c20..0d093d0 100644--- a/src/llm/model_load.c+++ b/src/llm/model_load.c@@ -460,11 +460,20 @@ static int load_mtp_internal(struct janas_llm_model *m, char *err,/* The draft head with the base lowest ids, room for 8192 more tokens. */static int init_draft_head(struct janas_llm_model *m, uint32_t base)+{+ return janas_m_init_draft_head(+ m, data(m, m->output),+ m->d_model / JANAS_QK * janas_qtype_block_size((int)m->output->type),+ base);+}++int janas_m_init_draft_head(struct janas_llm_model *m, const uint8_t *src,+ size_t row_bytes, uint32_t base){if (base > m->n_vocab)base = m->n_vocab;- m->dh_row =- m->d_model / JANAS_QK * janas_qtype_block_size((int)m->output->type);+ m->dh_src = src;+ m->dh_row = row_bytes;m->dh_cap = base + 8192 < m->n_vocab ? base + 8192 : m->n_vocab;m->dh_w = malloc((size_t)m->dh_cap * m->dh_row);m->dh_ids = malloc(m->dh_cap * sizeof(int32_t));@@ -472,7 +481,7 @@ static int init_draft_head(struct janas_llm_model *m, uint32_t base)m->dh_logits = malloc(m->dh_cap * sizeof(float));if (!m->dh_w || !m->dh_ids || !m->dh_in || !m->dh_logits)return -1;- memcpy(m->dh_w, data(m, m->output), (size_t)base * m->dh_row);+ memcpy(m->dh_w, src, (size_t)base * m->dh_row);for (uint32_t i = 0; i < base; i++) {m->dh_ids[i] = (int32_t)i;m->dh_in[i] = 1;@@ -679,7 +688,14 @@ struct janas_llm_model *janas_llm_model_load(const char *path,m->n_ctx = o->n_ctx;if (m->n_ctx_train && m->n_ctx > m->n_ctx_train)m->n_ctx = m->n_ctx_train;- if (o->mtp_path && load_mtp(m, o->mtp_path, err, err_len) != 0)+ /* Gemma 4's assistant is opened last, over the caches it reads */+ int assist = o->mtp_path && janas_m_assist_is(o->mtp_path);+ /* the draft head's lowest token ids: the caller's, JANAS_DRAFT_VOCAB+ (to experiment), or 32768 */+ uint32_t dvocab = o->draft_vocab ? o->draft_vocab : 32768;+ if (getenv("JANAS_DRAFT_VOCAB") && atol(getenv("JANAS_DRAFT_VOCAB")) > 0)+ dvocab = (uint32_t)atol(getenv("JANAS_DRAFT_VOCAB"));+ if (o->mtp_path && !assist && load_mtp(m, o->mtp_path, err, err_len) != 0)goto fail;if (!m->mtp && m->j.h.n_layer > m->n_layer &&load_mtp_internal(m, err, err_len) != 0)@@ -963,6 +979,10 @@ struct janas_llm_model *janas_llm_model_load(const char *path,snprintf(err, err_len, "out of memory");goto fail;}+ if (assist && (janas_m_assist_load(m, o->mtp_path, o->attn_scores, err,+ err_len) != 0 ||+ janas_m_assist_head(m, dvocab) != 0))+ goto fail;if (pin >= 0)janas_pin_current_thread(pin);m->cache = janas_expert_cache_create(&m->j, o->cache_bytes, m->exp_level,@@ -1009,7 +1029,7 @@ struct janas_llm_model *janas_llm_model_load(const char *path,m->exp_level, m->io, err, err_len): m->cache;if (!m->hid || !m->mtp_hid || !m->cat || !m->catq || !m->mtp_cache ||- init_draft_head(m, o->draft_vocab ? o->draft_vocab : 32768) != 0)+ init_draft_head(m, dvocab) != 0)goto fail;}if (m->gpu) {@@ -1109,6 +1129,7 @@ void janas_llm_model_free(struct janas_llm_model *m)free(m->tok_first);free(m->tok_fill);free(m->tok_list);+ janas_m_assist_free(m);if (m->mtp_open)janas_jns_close(&m->mtp_j);free(m->mtp_res);diff --git a/src/llm/model_ple.c b/src/llm/model_ple.cindex a9c8c73..1e44bfb 100644--- a/src/llm/model_ple.c+++ b/src/llm/model_ple.c@@ -101,18 +101,51 @@ struct f32_job {uint32_t rows, cols, n;};+#if defined(__x86_64__)+__attribute__((target("avx2,fma"))) static float+dot_f32_avx2(const float *w, const float *x, uint32_t n)+{+ __m256 a0 = _mm256_setzero_ps(), a1 = _mm256_setzero_ps();+ uint32_t i = 0;+ for (; i + 16 <= n; i += 16) {+ a0 =+ _mm256_fmadd_ps(_mm256_loadu_ps(w + i), _mm256_loadu_ps(x + i), a0);+ a1 = _mm256_fmadd_ps(_mm256_loadu_ps(w + i + 8),+ _mm256_loadu_ps(x + i + 8), a1);+ }+ float r = janas_hsum8(_mm256_add_ps(a0, a1));+ for (; i < n; i++)+ r += w[i] * x[i];+ return r;+}+#endif++/* one dot product, the same for a row and a token whatever the pass */+static float dot_f32(const float *w, const float *x, uint32_t n)+{+#if defined(__x86_64__)+ if (__builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma"))+ return dot_f32_avx2(w, x, n);+#endif+ float r = 0.0f;+ for (uint32_t i = 0; i < n; i++)+ r += w[i] * x[i];+ return r;+}++/* rows in equal runs, each read once for all the tokens: the f32+ matrices of E2B's and E4B's branches are 2.6 MB a layer */static void f32_worker(void *arg, int tid, int n_threads){const struct f32_job *b = arg;- size_t total = (size_t)b->rows * b->n;- for (size_t i = (size_t)tid; i < total; i += (size_t)n_threads) {- uint32_t j = (uint32_t)(i / b->rows), r = (uint32_t)(i % b->rows);+ uint32_t per = (b->rows + (uint32_t)n_threads - 1) / (uint32_t)n_threads;+ uint32_t r0 = (uint32_t)tid * per;+ uint32_t r1 = r0 + per < b->rows ? r0 + per : b->rows;+ for (uint32_t r = r0; r < r1; r++) {const float *w = b->w + (size_t)r * b->cols;- const float *x = b->x + (size_t)j * b->cols;- float s = 0.0f;- for (uint32_t c = 0; c < b->cols; c++)- s += w[c] * x[c];- b->y[(size_t)j * b->rows + r] = s;+ for (uint32_t j = 0; j < b->n; j++)+ b->y[(size_t)j * b->rows + r] =+ dot_f32(w, b->x + (size_t)j * b->cols, b->cols);}}