37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-09-27: Gemma 4 (12B, 26B, E2B, E4B), why a build has no GPU
camauri86771d821d3f
diff --git a/ChangeLog.md b/ChangeLog.mdindex 98c0868..05189b7 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -2,6 +2,98 @@> Curated, user-facing summary of completed work, newest first.+## [2026-09-27] - Why a build has no GPU++- **A build without GPU support says what was missing.** The GPU needs+ both `glslangValidator` and the Vulkan headers when Janas is built; with+ the first and not the second (CachyOS without `vulkan-headers`) the build+ left the GPU out without a word, and `janas-try.sh` reported "built with+ GPU support". Now `build.sh` prints which piece is missing, the engine's+ "GPU: none usable" names it, `janas-try.sh` checks for the headers too,+ and the README gives the Arch packages.++## [2026-09-27] - Gemma 4: E2B and E4B++- **Gemma-4-E2B-it and E4B-it run and chat**, tools included. Each token+ brings a few numbers for every layer (per-layer embeddings): a row of a+ table of their own plus a projection of the embedding (bf16), which open+ a branch of the residual after each feed-forward. The table, half of+ E2B's file, is never loaded: a token's row, a few kilobytes, is read from+ the file when it is needed, so E2B runs in 1.5 GB of memory from a 3.1 GB+ file, E4B in 3.1 GB from 5.0 GB.+- **The last layers share keys and values**: E2B's last 20 layers and+ E4B's last 18 compute queries only, and read the cache of the last+ earlier layer of their kind. E2B's feed-forward widens from 6,144 to+ 12,288 halfway; the engine takes a width per layer.+- Against llama.cpp, 200 tokens of text: E2B agrees on the most likely+ token at 191 positions, E4B at 195; prompt blocks give the same bits as+ token by token.+- **Without reasoning, E2B and E4B are no longer handed an empty thought+ channel**: their template does not write one (the 12B's and 26B's do),+ and given one they reasoned inside the reply. The chat template is now+ read whole: Gemma 4's is 19 KB, and the part after 16 KB was lost.++## [2026-09-27] - Gemma 4: tools++- **Gemma 4 calls tools**, in its own notation: the functions declared in+ the system turn (`<|tool>declaration:name{...}<tool|>`, parameters written+ as its template writes them, keys sorted), calls `call:name{key:value}`+ read into JSON, held by a grammar to a declared name and a well-formed+ value, and the tools' answers written inside the model's own turn, which+ it then continues. Checked token for token against the template rendered+ with Jinja: the first prompt with its declarations, a conversation loaded+ with a call and its answer (as `janas-server` gives it), and the answer+ appended live. Gemma-4-12B-it called a weather function with the right+ argument and used its answer.++## [2026-09-27] - Gemma 4: the 26B mixture, a sliding context, rings++- **Gemma-4-26B-A4B-it runs and chats**: 128 experts (eight a token)+ beside a dense feed-forward, the two as branches of their own - each+ normalized on the way in and out, then summed - a router reading its own+ input, and a scale per expert, folded into its weight. `gguf2jns` splits+ gate and up, which the file keeps fused, and widens with zeros the+ feed-forwards that are not a multiple of 256 wide (704 and 2,112 here):+ every weight type reads zero bytes as zero weights, so the products are+ unchanged. Files that need no widening convert to the same bytes as+ before.+- **The sliding-window layers keep a ring** of the positions they can see+ - the window and a prefill block, 1,536 for Gemma 4 - instead of the whole+ context, and attention skips the chunks before the window without reading+ them. The logits are the same to the bit with and without the ring+ (`JANAS_SWA_RING=0`), 1,500 tokens past the window; kept conversations,+ on disk too, come back with the same replies.+- **A full context slides on Gemma 4 too**: each layer's keys turned back+ with its own RoPE (two kinds, the full layers with their frequency+ factors), sixteen-bit keys, the rings. Qwen models slide as before, to+ the bit.++## [2026-09-26] - Gemma 4 in the chat++- **Gemma-4-12B-it chats**: Gemma's format - `<bos>`, turns opened by+ `<|turn>` and closed by `<turn|>`, the assistant called "model" - with+ its reasoning on or off: on, `<|think|>` heads the system turn and the+ model writes its thought channel, shown as reasoning; off, the channel is+ written empty, as the template does. The prompts were checked token for+ token against the model's own template rendered with Jinja and tokenized+ by llama.cpp, with and without reasoning. No tools yet, and a full+ context stops the chat instead of sliding.++## [2026-09-26] - Gemma 4: keys and values at sixteen bits++- **Gemma 4 keeps its keys and values at sixteen bits** (integers with the+ same scale per position and head, some 250 times finer than eight bits);+ the attention reads them through kernels of their own, the scores over+ the float query. Qwen models stay at eight bits, bit for bit as before;+ `JANAS_KV=8` or `16` overrides the choice. Against llama.cpp on 200+ tokens of prose: 160 most likely tokens in agreement and a mean logit+ difference of 0.84, against 143 and 1.53 at eight bits.+- **A trap in the reference**: llama.cpp's logits of Gemma 4 depend on its+ batch size - with batches of 64 tokens or more, the early positions see+ later tokens. The comparisons above use batches of 8, which agree with a+ prompt of two tokens run alone. The earlier figures (175 of 200) were+ measured against the larger batches.+## [2026-09-26] - Why there is no GPU- **"GPU: none usable" now says why**: built without GPU supportdiff --git a/README.md b/README.mdindex 4ffb4fa..a0d70d9 100644--- a/README.md+++ b/README.md@@ -74,13 +74,31 @@ report from `janas-bench` is the single most useful thing you can send back**the one the engine was shaped around, and the one whose numbers are quotedbelow. The others have been converted, checked against a reference and used,but far less.-- **In progress, not yet usable: Gemma 4** (`gemma4`). Gemma-4-12B-it- converts, its tokenizer gives llama.cpp's tokens on a mixed text, and- its forward pass runs; but the chat does not know Gemma's format yet, and- with the engine's eight-bit KV cache its logits drift from llama.cpp's- (the most likely token agrees at 175 of 200 positions; llama.cpp itself- agrees at 169 when its own cache goes from f16 to eight bits). A finer- cache for Gemma comes first.+- **Gemma 4** (`gemma4`): **Gemma-4-12B-it** (dense) and+ **Gemma-4-26B-A4B-it** (a mixture of 128 experts beside a dense+ feed-forward) convert and chat, with their reasoning on or off+ (`--think`); the prompts are, token for token, the ones the model's own+ chat template writes, and a full context slides as it does for the other+ models. Its sliding-window layers keep only the last positions they can+ see, so a long context costs mostly the full layers: for the 12B about 16+ KB a token and half a gigabyte fixed, instead of 175 KB a token. Keys and+ values are kept at sixteen bits (Gemma is far more sensitive to them than+ Qwen). Tools work in Gemma's own notation - declarations, calls and the+ tools' answers inside the model's turn, token for token as its template+ writes them. The small **Gemma-4-E2B-it** and **E4B-it** run too, with+ chat and tools: their per-layer embeddings - a table as large as half of+ E2B's file - are read from the file a row at a time and never loaded, so+ E2B ran here in 1.5 GB of memory (a 3.1 GB file) and E4B in 3.1 GB (5.0+ GB). Against llama.cpp on 200 tokens of prose the most likely token+ agrees at 160 positions on the 12B (mean logit difference 0.84) and at+ 145 on the 26B (1.43); on another 200-token text at 191 on E2B (0.45) and+ 195 on E4B (0.28). The larger two turn small numerical differences into+ large ones, and llama.cpp is not a steady reference for them: against+ itself with an eight-bit cache it agrees at 153 positions on the 12B, and+ with a 32-bit cache and no flash attention, at 112. The+ reference has to be computed in batches of at most 32 tokens: with larger+ ones llama.cpp's logits for the early positions change (it looks ahead),+ for a reason not yet found.- **Reasoning models:** the reasoning is shown apart, in grey, and can be turned off.- **`janas-chat`:** a terminal chat with a line editor of its own, history,colours, a status line and a context that slides instead of ending.@@ -175,6 +193,7 @@ the machine to itself.```shsudo apt install build-essential # Debian/Ubuntusudo apt install libvulkan-dev glslang-tools vulkan-tools # optional, GPU+sudo pacman -S vulkan-headers glslang vulkan-tools # the same, Arch./build.sh # everything into bin/x86_64-linux/./build.sh release test # and run the testsdiff --git a/build.sh b/build.shindex 365c54a..1830248 100755--- a/build.sh+++ b/build.sh@@ -177,9 +177,17 @@ esacmkdir -p "$OBJ" "$BIN"# GPU shaders (src/*/shaders/*.comp) as SPIR-V in C arrays; without-# glslangValidator the library is built without GPU support+# glslangValidator, or without the Vulkan headers, the library is built+# without GPU support - and says which is missing, here and at run timeSPV="$OBJ/spv"mkdir -p "$SPV"+GPU_MISSING=""+command -v glslangValidator >/dev/null 2>&1 ||+ GPU_MISSING="glslangValidator (glslang-tools, glslang)"+echo '#include <vulkan/vulkan.h>' | "$CC" -E -x c - >/dev/null 2>&1 ||+ GPU_MISSING="${GPU_MISSING:+$GPU_MISSING and }the Vulkan headers (libvulkan-dev, vulkan-headers)"+[ -z "$GPU_MISSING" ] ||+ echo "note: building without GPU support: $GPU_MISSING not found"if command -v glslangValidator >/dev/null 2>&1; thenfor comp in src/*/shaders/*.comp; do[ -f "$comp" ] || continuediff --git a/src/llm/api.c b/src/llm/api.cindex edc066b..292f0f9 100644--- a/src/llm/api.c+++ b/src/llm/api.c@@ -92,17 +92,57 @@ static uint64_t file_size(const char *path)return stat(path, &st) == 0 ? (uint64_t)st.st_size : 0;}-/* Recognizes the chat format; returns 0 or an error code. */-static int32_t find_format(janas_llm *llm, const struct janas_jns *j)+/* Recognizes the chat format from the template in tmpl; returns 0 or an+ error code. */+static int32_t find_format_in(janas_llm *llm, const struct janas_jns *j,+ const char *tmpl){- char tmpl[16384];- if (janas_jns_meta_str(j, "tokenizer.chat_template", tmpl, sizeof(tmpl)) !=- 0)- tmpl[0] = 0;llm->im_start = janas_tokenizer_find(llm->tok, "<|im_start|>");llm->im_end = janas_tokenizer_find(llm->tok, "<|im_end|>");- if (llm->im_start >= 0 && llm->im_end >= 0 &&- (tmpl[0] == 0 || strstr(tmpl, "<|im_start|>"))) {+ llm->asst_role = "assistant";+ llm->bos = llm->think_on = -1;+ llm->quote = llm->decl_open = llm->decl_close = -1;+ int32_t turn_open = janas_tokenizer_find(llm->tok, "<|turn>");+ int32_t turn_close = janas_tokenizer_find(llm->tok, "<turn|>");+ if (turn_open >= 0 && turn_close >= 0 && strstr(tmpl, "<|turn>")) {+ /* Gemma 4: no tools yet (its calls have a syntax of their own) */+ llm->format = FORMAT_GEMMA;+ llm->im_start = turn_open;+ llm->im_end = turn_close;+ llm->asst_role = "model";+ llm->bos = janas_tokenizer_find(llm->tok, "<bos>");+ llm->stop[llm->n_stop++] = turn_close;+ int32_t eos = janas_tokenizer_find(llm->tok, "<eos>");+ if (eos >= 0)+ llm->stop[llm->n_stop++] = eos;+ llm->think_open = janas_tokenizer_find(llm->tok, "<|channel>");+ llm->think_close = janas_tokenizer_find(llm->tok, "<channel|>");+ llm->think_on = janas_tokenizer_find(llm->tok, "<|think|>");+ llm->thinker = strstr(tmpl, "enable_thinking") &&+ llm->think_open >= 0 && llm->think_close >= 0 &&+ llm->think_on >= 0;+ /* without reasoning the 12B and 26B are given an empty channel;+ E2B and E4B are not, and given one they reason in the reply */+ llm->empty_thought =+ strstr(tmpl, "'<|channel>thought\\n<channel|>'") != NULL;+ /* tools: its own notation (tools.c); a call ends the reply at+ <|tool_response>, which the tools' answers then follow */+ llm->call_open = janas_tokenizer_find(llm->tok, "<|tool_call>");+ llm->call_close = janas_tokenizer_find(llm->tok, "<tool_call|>");+ llm->resp_open = janas_tokenizer_find(llm->tok, "<|tool_response>");+ llm->resp_close = janas_tokenizer_find(llm->tok, "<tool_response|>");+ llm->quote = janas_tokenizer_find(llm->tok, "<|\"|>");+ llm->decl_open = janas_tokenizer_find(llm->tok, "<|tool>");+ llm->decl_close = janas_tokenizer_find(llm->tok, "<tool|>");+ llm->tools = janas_tools_dialect(tmpl);+ if (llm->call_open < 0 || llm->call_close < 0 || llm->resp_open < 0 ||+ llm->resp_close < 0 || llm->quote < 0 || llm->decl_open < 0 ||+ llm->decl_close < 0)+ llm->tools = JANAS_TOOLS_NONE;+ if (llm->tools != JANAS_TOOLS_NONE)+ llm->stop[llm->n_stop++] = llm->resp_open;+ } else if (llm->im_start >= 0 && llm->im_end >= 0 &&+ (tmpl[0] == 0 || strstr(tmpl, "<|im_start|>"))) {llm->format = FORMAT_CHATML;llm->stop[llm->n_stop++] = llm->im_end;llm->think_open = janas_tokenizer_find(llm->tok, "<think>");@@ -136,7 +176,7 @@ static int32_t find_format(janas_llm *llm, const struct janas_jns *j)if (janas_jns_meta_u32(j, "tokenizer.ggml.eos_token_id", &eos) == 0)llm->eos_id = (int32_t)eos;if (janas_jns_meta_u32(j, "tokenizer.ggml.eos_token_id", &eos) == 0 &&- (int32_t)eos != llm->stop[0])+ !janas_api_is_stop(llm, (int32_t)eos))llm->stop[llm->n_stop++] = (int32_t)eos;int32_t eot = janas_tokenizer_find(llm->tok, "<|endoftext|>");if (eot >= 0 && eot != llm->stop[0] &&@@ -145,6 +185,20 @@ static int32_t find_format(janas_llm *llm, const struct janas_jns *j)return 0;}+/* The same, reading the template (Gemma 4's: 19 KB) */+static int32_t find_format(janas_llm *llm, const struct janas_jns *j)+{+ size_t cap = (size_t)1 << 17;+ char *tmpl = malloc(cap);+ if (!tmpl)+ return janas_api_fail(JANAS_LLM_ENOMEM, "out of memory");+ if (janas_jns_meta_str(j, "tokenizer.chat_template", tmpl, cap) != 0)+ tmpl[0] = 0;+ int32_t rc = find_format_in(llm, j, tmpl);+ free(tmpl);+ return rc;+}+int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,janas_llm **out){@@ -180,8 +234,11 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,janas_jns_meta_str(&j, "general.name", name, sizeof(name));janas_jns_meta_str(&j, "general.architecture", arch, sizeof(arch));int32_t rc = 0;- if (strcmp(kind, "gpt2") != 0 ||- (strcmp(pre, "qwen2") != 0 && strcmp(pre, "qwen35") != 0))+ /* byte-level BPE with Qwen's pre-tokenizers, or Gemma 4's */+ int known = (strcmp(kind, "gpt2") == 0 &&+ (strcmp(pre, "qwen2") == 0 || strcmp(pre, "qwen35") == 0)) ||+ strcmp(kind, "gemma4") == 0;+ if (!known)rc = janas_api_fail(JANAS_LLM_EMODEL, "tokenizer %s/%s not supported",kind, pre);else if (!(llm->tok = janas_tokenizer_create(&j, err, sizeof(err))))@@ -189,6 +246,12 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,elserc = find_format(llm, &j);uint64_t resident = j.h.resident_bytes;+ /* less Gemma 4's per-layer embeddings, read from the file a row at a+ time (model_ple.c) */+ const struct janas_jns_tensor *ple =+ janas_jns_tensor(&j, "per_layer_token_embd.weight");+ if (ple && ple->bytes < resident)+ resident -= ple->bytes;/* what the cache would need to hold every slot at once. Not the size ofthe expert region: the cache's slots are all as big as the largest, sowhere the layers differ (a down matrix in Q6_K in one and Q4_K in the@@ -236,22 +299,50 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,/* Gemma 4: sliding-window layers with a head size of their own, and anattention (its partial results) for each kind of layer */const uint8_t *pat = NULL;- uint32_t len_swa = 0, kv_swa = 0;+ uint32_t len_swa = 0, kv_swa = 0, window = 0;+ /* a sliding-window layer keeps a ring of positions (model_load.c): a+ fixed amount, counted apart from the tokens */+ uint64_t ring_bytes = 0;+ uint32_t ring = 0;+ snprintf(key, sizeof(key), "%.32s.attention.sliding_window", arch);+ if (janas_jns_meta_u32(&j, key, &window) == 0 && window) {+ const char *pb = getenv("JANAS_PREFILL_BLOCK"),+ *rg = getenv("JANAS_SWA_RING");+ long blk = pb ? atol(pb) : 256;+ ring = janas_attn_ring_positions(window, blk < 64 ? 64+ : blk > 4096 ? 4096+ : (uint32_t)blk);+ if (ring >= p.n_ctx || (rg && atoi(rg) == 0))+ ring = 0;+ }snprintf(key, sizeof(key), "%.32s.attention.sliding_window_pattern", arch);if (janas_jns_meta_bytes(&j, key, &n_pat, &pat) != 0 || n_pat != n_layer)pat = NULL;snprintf(key, sizeof(key), "%.32s.attention.key_length_swa", arch);janas_jns_meta_u32(&j, key, &len_swa);+ /* E2B and E4B: one count of KV heads for both kinds, and the last+ shared_kv_layers layers reading earlier layers' keys and values */+ uint32_t shared = 0;+ snprintf(key, sizeof(key), "%.32s.attention.shared_kv_layers", arch);+ janas_jns_meta_u32(&j, key, &shared);snprintf(key, sizeof(key), "%.32s.attention.head_count_kv", arch);- if (janas_jns_meta_ints(&j, key, &n_arr, &per_layer) == 0 && per_layer &&- n_arr == n_layer) {+ int arr = janas_jns_meta_ints(&j, key, &n_arr, &per_layer) == 0 &&+ per_layer && n_arr == n_layer;+ if (arr || (pat && kv_heads)) {+ uint32_t one = kv_heads;kv_token = 0;- for (uint64_t i = 0; i < n_arr; i++) {- int32_t h;- memcpy(&h, per_layer + i, sizeof(h));+ for (uint64_t i = 0; i + shared < n_layer; i++) {+ int32_t h = (int32_t)one;+ if (arr)+ memcpy(&h, per_layer + i, sizeof(h));uint32_t len = pat && pat[i] && len_swa ? len_swa : kv_len;- if (h > 0)- kv_token += (uint64_t)h * (len * 2 + 8);+ /* keys and values of one byte a number, or two (Gemma 4) */+ uint64_t per =+ h > 0 ? (uint64_t)h * (len * (a && a->kv16 ? 4 : 2) + 8) : 0;+ if (pat && pat[i] && ring)+ ring_bytes += per * ring;+ else+ kv_token += per;if (h > 0 && pat && pat[i])kv_swa = (uint32_t)h;else if (h > 0 && pat)@@ -271,7 +362,9 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,? janas_attn_part_bytes(n_head, kv_heads, kv_len, p.n_ctx): 0;if (pat && n_head && kv_swa && len_swa)- attn_part += janas_attn_part_bytes(n_head, kv_swa, len_swa, p.n_ctx);+ attn_part += janas_attn_part_bytes(n_head, kv_swa, len_swa,+ ring ? ring : p.n_ctx) ++ ring_bytes;/* and the scratch of a pass, which grows with the prefill block(janas_llm_model_max_block): per token, the rows of its experts (gate,up, their product, down) and a few rows of the model's width. An@@ -282,7 +375,16 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,snprintf(key, sizeof(key), "%.32s.expert_feed_forward_length", arch);if (janas_jns_meta_u32(&j, key, &ff) != 0) {snprintf(key, sizeof(key), "%.32s.feed_forward_length", arch);- janas_jns_meta_u32(&j, key, &ff);+ const int32_t *ffs = NULL;+ uint64_t n_ff = 0;+ if (janas_jns_meta_u32(&j, key, &ff) != 0 &&+ janas_jns_meta_ints(&j, key, &n_ff, &ffs) == 0 && ffs)+ for (uint64_t i = 0; i < n_ff; i++) { /* E2B: the widest */+ int32_t v;+ memcpy(&v, ffs + i, sizeof(v));+ if (v > 0 && (uint32_t)v > ff)+ ff = (uint32_t)v;+ }}snprintf(key, sizeof(key), "%.32s.expert_used_count", arch);janas_jns_meta_u32(&j, key, &k_used);@@ -406,6 +508,8 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,llm->cache_bytes = cache;llm->reserve_bytes = reserve;llm->kv_token = kv_token;+ llm->kv_ring = ring;+ llm->kv_ring_token = ring ? ring_bytes / ring : 0;llm->n_compute = mo.n_compute;snprintf(llm->arch, sizeof(llm->arch), "%s", arch);snprintf(llm->bits, sizeof(llm->bits), "%s", bits);diff --git a/src/llm/api_chat.c b/src/llm/api_chat.cindex 07f2cc2..762a6a9 100644--- a/src/llm/api_chat.c+++ b/src/llm/api_chat.c@@ -256,10 +256,35 @@ int janas_api_add_message(janas_llm_chat *c, const char *role, const char *text,}}+/* The start of the sequence: Gemma's <bos>, nothing for ChatML. */+int janas_api_begin_sequence(janas_llm_chat *c)+{+ return c->llm->bos >= 0 ? janas_api_add_special(c, c->llm->bos) : 0;+}+int janas_api_start_reply(janas_llm_chat *c){const janas_llm *llm = c->llm;switch (llm->format) {+ case FORMAT_GEMMA:+ if (c->in_turn) { /* after the tools' answers: the model goes on */+ c->in_turn = 0;+ c->in_think = c->chan_label = 0;+ return janas_api_flush_text(c);+ }+ if (janas_api_add_special(c, llm->im_start) ||+ janas_api_add_text(c, "model\n", 6))+ return -1;+ /* reasoning asked for (<|think|> in the system turn): the model+ opens its channel itself; otherwise the channel is written empty+ where the template does (not E2B and E4B's) */+ c->in_think = c->chan_label = 0;+ if ((llm->thinker && c->p.thinking != 0) || !llm->empty_thought)+ return janas_api_flush_text(c);+ return janas_api_add_special(c, llm->think_open) ||+ janas_api_add_text(c, "thought\n", 8) ||+ janas_api_add_special(c, llm->think_close) ||+ janas_api_flush_text(c);case FORMAT_CHATML:default:if (janas_api_add_special(c, llm->im_start) ||@@ -292,6 +317,7 @@ int janas_api_start_reply(janas_llm_chat *c)int janas_api_add_markup(janas_llm_chat *c, const char *s, size_t n){const janas_llm *llm = c->llm;+ int gemma = llm->format == FORMAT_GEMMA;const struct {const char *text;int32_t id;@@ -300,7 +326,15 @@ int janas_api_add_markup(janas_llm_chat *c, const char *s, size_t n){"<tool_response>", llm->resp_open},{"</tool_response>", llm->resp_close},{"<think>", llm->think_open},- {"</think>", llm->think_close}};+ {"</think>", llm->think_close},+ /* Gemma 4 */+ {"<|tool_call>", gemma ? llm->call_open : -1},+ {"<tool_call|>", gemma ? llm->call_close : -1},+ {"<|tool_response>", gemma ? llm->resp_open : -1},+ {"<tool_response|>", gemma ? llm->resp_close : -1},+ {"<|\"|>", llm->quote},+ {"<|tool>", llm->decl_open},+ {"<tool|>", llm->decl_close}};size_t from = 0;for (size_t i = 0; i < n; i++) {if (s[i] != '<')@@ -326,6 +360,26 @@ int janas_api_add_system(janas_llm_chat *c, const char *sys, size_t n){const janas_llm *llm = c->llm;c->tools_tokens = 0;+ if (llm->format == FORMAT_GEMMA) {+ /* a system turn only when there is something to say: the message,+ or <|think|> at its top when reasoning is asked for */+ int think = llm->thinker && c->p.thinking != 0;+ int tools = c->tools && llm->tools == JANAS_TOOLS_GEMMA;+ if (!think && !tools && (!sys || n == 0))+ return 0;+ struct janas_buf b = {0};+ janas_tools_system(&b, tools ? JANAS_TOOLS_GEMMA : JANAS_TOOLS_NONE,+ c->tools, sys, n);+ int err = b.oom || janas_api_add_special(c, llm->im_start) ||+ janas_api_add_text(c, "system\n", 7) ||+ (think && (janas_api_add_special(c, llm->think_on) ||+ janas_api_add_text(c, "\n", 1))) ||+ janas_api_add_markup(c, b.p, b.n) ||+ janas_api_add_special(c, llm->im_end) ||+ janas_api_add_text(c, "\n", 1);+ janas_buf_free(&b);+ return err ? -1 : 0;+ }if (!c->tools || llm->tools == JANAS_TOOLS_NONE)return janas_api_add_message(c, "system", sys ? sys : "", n);/* how many of its tokens the tools take: the message without them is@@ -626,22 +680,46 @@ static int32_t send_turn(janas_llm_chat *c, const char *text, size_t n,const int32_t *lens){janas_api_prep_start(c);+ /* Gemma: the tools answer inside the model's turn, each answer named+ after its call, which the reply's calls say (cleared next) */+ int gemma_answers = texts && c->llm->format == FORMAT_GEMMA &&+ c->llm->tools == JANAS_TOOLS_GEMMA &&+ janas_llm_session_length(c->s) > 0;+ struct janas_buf ans = {0};+ for (int32_t i = 0; gemma_answers && i < n_res; i++) {+ const char *nm = i < (int32_t)c->n_calls ? c->calls[i].name : "tool";+ if (i > 0 || !c->closed)+ janas_buf_puts(&ans, "<|tool_response>");+ janas_tools_gemma_response(+ &ans, nm, strlen(nm), texts[i],+ janas_api_text_len(texts[i], lens ? lens[i] : -1));+ }janas_api_reply_clear(c);c->n_ids = c->n_text = 0;- int err = 0;- if (janas_llm_session_length(c->s) > 0) {+ int err = ans.oom;+ /* (ans is freed once written, below) */+ if (gemma_answers) {+ err |= janas_api_add_markup(c, ans.p, ans.n);+ c->in_turn = 1;+ } else if (janas_llm_session_length(c->s) > 0) {/* close the previous reply: its stop token is in the sequence, orit was cut and gets one now; then the newline after it */if (!c->closed)err |= janas_api_add_special(c, c->llm->im_end);err |= janas_api_add_text(c, "\n", 1);- } else if (c->system || c->tools) {- err |= janas_api_add_system(c, c->system,- c->system ? strlen(c->system) : 0);+ } else {+ /* the start: <bos> where the format has one, the system message */+ err |= janas_api_begin_sequence(c);+ if (c->system || c->tools || c->llm->format == FORMAT_GEMMA)+ err |= janas_api_add_system(c, c->system,+ c->system ? strlen(c->system) : 0);c->sys_tokens = (uint32_t)c->n_ids;}err |= janas_api_note_turn(c, (uint32_t)janas_llm_session_length(c->s));- if (texts) {+ janas_buf_free(&ans);+ if (gemma_answers) {+ /* written above */+ } else if (texts) {err |= janas_api_add_special(c, c->llm->im_start) ||janas_api_add_text(c, "user", 4);for (int32_t i = 0; i < n_res; i++) {diff --git a/src/llm/api_load.c b/src/llm/api_load.cindex 709a8fe..2ee2068 100644--- a/src/llm/api_load.c+++ b/src/llm/api_load.c@@ -145,8 +145,9 @@ static int add_assistant(janas_llm_chat *c, const struct cmsg *m, int after_q)int err = janas_api_add_special(c, llm->im_start);/* Qwen3-Coder writes the calls right after the role */int bare = d == JANAS_TOOLS_XML_CODER && m->n_calls;- err |= janas_api_add_text(c, bare ? "assistant" : "assistant\n",- bare ? 9 : 10);+ err |= janas_api_add_text(c, llm->asst_role, strlen(llm->asst_role));+ if (!bare)+ err |= janas_api_add_text(c, "\n", 1);if (d == JANAS_TOOLS_XML && llm->thinker && after_q)err |= janas_api_add_markup(c, "<think>\n\n</think>\n\n", 19);if (m->n_calls && d != JANAS_TOOLS_NONE) {@@ -162,6 +163,43 @@ static int add_assistant(janas_llm_chat *c, const struct cmsg *m, int after_q)return err ? -1 : 0;}+/*+ * Gemma 4's model turn with tools: its calls, the answers of the tool+ * messages after it (they belong to the turn), then its text; the turn+ * stays open when the model speaks next (the next message is its own, or+ * the reply is to follow the answers). Returns the index of the last+ * message used; *open whether the turn was left open.+ */+static size_t add_gemma_turn(janas_llm_chat *c, size_t i, int reopen, int *open)+{+ const janas_llm *llm = c->llm;+ const struct cmsg *m = &c->msgs[i];+ int err = 0;+ if (!reopen)+ err |= janas_api_add_special(c, llm->im_start) ||+ janas_api_add_text(c, "model\n", 6);+ struct janas_buf b = {0};+ janas_tools_assistant(&b, JANAS_TOOLS_GEMMA, NULL, 0, m->calls, m->n_calls);+ size_t j = i + 1, k = 0;+ for (; j < c->n_msgs && c->msgs[j].role == JANAS_LLM_ROLE_TOOL; j++, k++) {+ const char *nm = k < m->n_calls ? m->calls[k].name : "tool";+ size_t nn = k < m->n_calls ? m->calls[k].name_n : 4;+ janas_buf_puts(&b, "<|tool_response>");+ janas_tools_gemma_response(&b, nm, nn, c->msgs[j].text, c->msgs[j].n);+ }+ if (m->text)+ janas_buf_put(&b, m->text, m->n);+ err |= b.oom || janas_api_add_markup(c, b.p, b.n);+ janas_buf_free(&b);+ int next_model =+ j < c->n_msgs && c->msgs[j].role == JANAS_LLM_ROLE_ASSISTANT;+ *open = next_model || (j == c->n_msgs && k > 0);+ if (!*open)+ err |= janas_api_add_special(c, llm->im_end) ||+ janas_api_add_text(c, "\n", 1);+ return err ? (size_t)-1 : j - 1;+}+/* The messages built, in the model's format, into c->ids; the reply's turnstarted. *reply_off gets where it starts. */static int render(janas_llm_chat *c, uint32_t *reply_off)@@ -171,13 +209,17 @@ static int render(janas_llm_chat *c, uint32_t *reply_off)for (size_t i = 0; i < c->n_msgs; i++)if (c->msgs[i].role == JANAS_LLM_ROLE_USER)last_q = i;- int err = 0;+ int err = janas_api_begin_sequence(c);+ if (llm->format == FORMAT_GEMMA && !c->tools &&+ c->msgs[0].role != JANAS_LLM_ROLE_SYSTEM)+ err |= janas_api_add_system(c, NULL, 0); /* <|think|>, if asked */if (c->tools && c->msgs[0].role != JANAS_LLM_ROLE_SYSTEM) {/* with tools there is a system message whatever the caller says */err |= janas_api_add_system(c, NULL, 0);err |= janas_api_flush_text(c);c->sys_tokens = (uint32_t)c->n_ids;}+ int gemma_open = 0; /* Gemma: the model's turn left open */for (size_t i = 0; i < c->n_msgs && !err; i++) {const struct cmsg *m = &c->msgs[i];switch (m->role) {@@ -193,6 +235,19 @@ static int render(janas_llm_chat *c, uint32_t *reply_off)err |= janas_api_add_message(c, "user", m->text, m->n);break;case JANAS_LLM_ROLE_ASSISTANT: {+ if (llm->tools == JANAS_TOOLS_GEMMA && (m->n_calls || gemma_open)) {+ int open = 0;+ size_t last = add_gemma_turn(c, i, gemma_open, &open);+ if (last == (size_t)-1) {+ err = 1;+ break;+ }+ i = last;+ gemma_open = open;+ if (open && i + 1 == c->n_msgs)+ c->in_turn = 1; /* the reply follows the answers */+ break;+ }const struct memo *mm = memo_find(c, m);if (mm) { /* one of the last replies: as the model wrote it */err |= add_ids(c, mm->ids, mm->n);diff --git a/src/llm/api_reply.c b/src/llm/api_reply.cindex 87bfc15..0143805 100644--- a/src/llm/api_reply.c+++ b/src/llm/api_reply.c@@ -121,7 +121,11 @@ void janas_api_update_stats(janas_llm_chat *c)s->input_cpu_seconds = c->input_cpu;s->input_threads = janas_llm_model_threads(c->llm->m, 2);s->output_threads = janas_llm_model_threads(c->llm->m, 0);- s->context_bytes = c->llm->kv_token * s->context_used;+ s->context_bytes =+ c->llm->kv_token * s->context_used ++ c->llm->kv_ring_token * (s->context_used < c->llm->kv_ring+ ? s->context_used+ : c->llm->kv_ring);struct rusage ru;if (getrusage(RUSAGE_SELF, &ru) == 0)s->peak_rss_bytes = (uint64_t)ru.ru_maxrss * 1024;@@ -194,9 +198,22 @@ static void take(janas_llm_chat *c, int32_t t)c->answer_tokens++; /* text or a call: what max_answer counts */if (t == llm->think_open || t == llm->think_close) {c->in_think = t == llm->think_open;+ /* Gemma names its channel before the reasoning: "thought\n" */+ c->chan_label = c->in_think && llm->format == FORMAT_GEMMA;janas_api_record(c, t, JANAS_LLM_PART_MARK, "", 0);return;}+ if (c->chan_label) {+ char lab[32];+ size_t ln = janas_tokenizer_decode(llm->tok, t, lab, sizeof(lab));+ if (ln < sizeof(lab) && ((ln == 7 && !memcmp(lab, "thought", 7)) ||+ (ln == 1 && *lab == '\n'))) {+ c->chan_label = lab[0] != '\n';+ janas_api_record(c, t, JANAS_LLM_PART_MARK, "", 0);+ return;+ }+ c->chan_label = 0;+ }if (c->tools && t == llm->call_open && !c->in_think && !c->in_call) {c->in_call = 1;c->body.n = 0;diff --git a/src/llm/api_tools.c b/src/llm/api_tools.cindex efbf2a2..bb7c75f 100644--- a/src/llm/api_tools.c+++ b/src/llm/api_tools.c@@ -178,11 +178,13 @@ int janas_api_reply_setup(janas_llm_chat *c)return -1;}uint32_t call = JANAS_NO_RULE, answer = JANAS_NO_RULE;+ int only = choice == JANAS_LLM_TOOLS_FUNCTION ? c->tool_only : -1;if (tools && choice != JANAS_LLM_TOOLS_NONE)- call = janas_tools_rule(- b, j, llm->tools, c->tools,- choice == JANAS_LLM_TOOLS_FUNCTION ? c->tool_only : -1,- llm->call_close);+ call = llm->tools == JANAS_TOOLS_GEMMA+ ? janas_tools_rule_gemma(b, c->tools, only, llm->call_close,+ llm->quote)+ : janas_tools_rule(b, j, llm->tools, c->tools, only,+ llm->call_close);if (must) {answer = calls_rule(b, llm, call, c->parallel);} else if (c->format != JANAS_LLM_FORMAT_TEXT) {diff --git a/src/llm/arch.c b/src/llm/arch.cindex be7c9bb..1e4beb0 100644--- a/src/llm/arch.c+++ b/src/llm/arch.c@@ -29,11 +29,15 @@ static const struct janas_arch archs[] = {.post_attn_norm = 1,.vmap_mod = 1,.nextn_in_file = 1},+ /* its scores are not scaled down, and an eight-bit key moves them: at+ eight bits llama.cpp agrees with its own f16 cache at 169 of 200 most+ likely tokens */{.name = "gemma4",.rec = JANAS_REC_NONE,.gelu = 1,.sandwich = 1,- .swa = 1},+ .swa = 1,+ .kv16 = 1},};const struct janas_arch *janas_arch_find(const char *name, unsigned len)diff --git a/src/llm/arch.h b/src/llm/arch.hindex 3878d1d..8965b35 100644--- a/src/llm/arch.h+++ b/src/llm/arch.h@@ -35,6 +35,7 @@ struct janas_arch {its head size, KV heads and RoPE; V normalized withoutweights, V = K where attn_v is missing, scores notscaled (the q and k norms do it) */+ int kv16; /* keys and values kept at sixteen bits, not eight */};/* The architecture called name, or NULL if Janas does not know it. */diff --git a/src/llm/attention.c b/src/llm/attention.cindex 489390b..416ec0f 100644--- a/src/llm/attention.c+++ b/src/llm/attention.c@@ -40,6 +40,8 @@ struct janas_attn {int scores; /* JANAS_ATTN_FLOAT or JANAS_ATTN_INT16, resolved */int scores_auto; /* 1: the engine chose it */float scale; /* of the scores: 1 / sqrt(head_dim) unless set */+ int kv16; /* keys and values of sixteen bits, not eight */+ int ring; /* position p in row p mod n_ctx (with a window) */uint32_t window; /* a token sees the last window positions; 0: all */};@@ -52,6 +54,7 @@ struct attn_job {const float *ks, *vs; /* one scale per position and KV head */float *out;uint32_t n, pos0, n_chunks, split;+ uint32_t c0; /* the first chunk any token of the call sees */atomic_uint next; /* next item to claim */};@@ -62,7 +65,9 @@ static float *att_partial(const struct janas_attn *A, uint32_t kvh, uint32_t c,{uint32_t group = A->n_head / A->n_kv;return A->part +- ((((size_t)kvh * A->chunks + c) * JANAS_ATTN_MAX_BLOCK + j) * group ++ ((((size_t)kvh * A->chunks + c % A->chunks) * JANAS_ATTN_MAX_BLOCK ++ j) *+ group +g) *(A->hd + 2);}@@ -118,6 +123,33 @@ static void att_values_ref(const float *const *w, const int8_t *V, uint32_t nt,}}+/* The same over keys and values of sixteen bits. */+static void att_scores_f32_ref16(const float *const *q, const int16_t *K,+ const float *ks, uint32_t nt, uint32_t hd,+ float scale, float *const *sc, int ns)+{+ for (uint32_t t = 0; t < nt; t++)+ for (int s = 0; s < ns; s++) {+ float sum = 0.0f;+ for (uint32_t i = 0; i < hd; i++)+ sum += q[s][i] * (float)K[(size_t)t * hd + i];+ sc[s][t] = sum * scale * ks[t];+ }+}++static void att_values_ref16(const float *const *w, const int16_t *V,+ uint32_t nt, uint32_t hd, float *const *out,+ int ns)+{+ for (int s = 0; s < ns; s++)+ for (uint32_t i = 0; i < hd; i++) {+ float acc = 0.0f;+ for (uint32_t t = 0; t < nt; t++)+ acc += w[s][t] * (float)V[(size_t)t * hd + i];+ out[s][i] = acc;+ }+}+#if defined(__x86_64__)#define AVX2 __attribute__((target("avx2,fma,f16c")))@@ -232,6 +264,103 @@ static AVX2 void att_values_avx2(const float *const *w, const int8_t *V,values_body(w, V, nt, hd, out, 1);}}++/*+ * Keys and values of sixteen bits (Gemma 4, whose unscaled scores feel an+ * eight-bit key: llama.cpp itself, its cache taken from f16 to q8_0, agreed+ * with itself at 169 of 200 most likely tokens). The same kernels, the rows+ * read as int16; the scores always over the float query, since an integer+ * sum of sixteen-bit products would not fit in 32 bits.+ */+static inline AVX2 __attribute__((always_inline)) void+f32_body16(const float *const *q, const int16_t *K, const float *ks,+ uint32_t nt, uint32_t hd, float scale, float *const *sc,+ const int16_t *pf, const int ns)+{+ for (uint32_t t = 0; t < nt; t++) {+ const int16_t *k = K + (size_t)t * hd;+ if (pf)+ for (uint32_t i = 0; i < hd; i += 32)+ _mm_prefetch((const char *)(pf + (size_t)t * hd + i),+ _MM_HINT_T1);+ __m256 acc[8];+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++) acc[s] =+ _mm256_setzero_ps();+ for (uint32_t i = 0; i < hd; i += 8) {+ __m256 kv = _mm256_cvtepi32_ps(_mm256_cvtepi16_epi32(+ _mm_loadu_si128((const __m128i *)(k + i))));+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++) acc[s] =+ _mm256_fmadd_ps(_mm256_loadu_ps(q[s] + i), kv, acc[s]);+ }+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++)+ {+ __m128 r = _mm_add_ps(_mm256_castps256_ps128(acc[s]),+ _mm256_extractf128_ps(acc[s], 1));+ r = _mm_add_ps(r, _mm_movehl_ps(r, r));+ r = _mm_add_ss(r, _mm_movehdup_ps(r));+ sc[s][t] = _mm_cvtss_f32(r) * scale * ks[t];+ }+ }+}++static AVX2 void att_scores_f32_avx2_16(const float *const *q, const int16_t *K,+ const float *ks, uint32_t nt,+ uint32_t hd, float scale,+ float *const *sc, const int16_t *pf,+ int ns)+{+ switch (ns) {+ case 8:+ f32_body16(q, K, ks, nt, hd, scale, sc, pf, 8);+ break;+ case 4:+ f32_body16(q, K, ks, nt, hd, scale, sc, pf, 4);+ break;+ case 2:+ f32_body16(q, K, ks, nt, hd, scale, sc, pf, 2);+ break;+ default:+ f32_body16(q, K, ks, nt, hd, scale, sc, pf, 1);+ }+}++static inline AVX2 __attribute__((always_inline)) void+values_body16(const float *const *w, const int16_t *V, uint32_t nt, uint32_t hd,+ float *const *out, const int ns)+{+ for (uint32_t i = 0; i < hd; i += 8) {+ __m256 acc[8];+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++) acc[s] =+ _mm256_setzero_ps();+ for (uint32_t t = 0; t < nt; t++) {+ __m256 v = _mm256_cvtepi32_ps(_mm256_cvtepi16_epi32(+ _mm_loadu_si128((const __m128i *)(V + (size_t)t * hd + i))));+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++) acc[s] =+ _mm256_fmadd_ps(_mm256_broadcast_ss(w[s] + t), v, acc[s]);+ }+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++)+ _mm256_storeu_ps(out[s] + i, acc[s]);+ }+}++static AVX2 void att_values_avx2_16(const float *const *w, const int16_t *V,+ uint32_t nt, uint32_t hd, float *const *out,+ int ns)+{+ switch (ns) {+ case 8:+ values_body16(w, V, nt, hd, out, 8);+ break;+ case 4:+ values_body16(w, V, nt, hd, out, 4);+ break;+ case 2:+ values_body16(w, V, nt, hd, out, 2);+ break;+ default:+ values_body16(w, V, nt, hd, out, 1);+ }+}#endif/*@@ -305,7 +434,8 @@ static void attn_worker(void *arg, int tid, int n_threads)struct janas_attn *A = a->A;uint32_t hd = A->hd;uint32_t group = A->n_head / A->n_kv, gs = group / a->split;- uint32_t items = A->n_kv * a->n_chunks * a->split;+ uint32_t span = a->n_chunks - a->c0; /* chunks c0 .. n_chunks - 1 */+ uint32_t items = A->n_kv * span * a->split;float scale = A->scale;float *sc = A->scr + (size_t)tid * JANAS_ATTN_MAX_BLOCK * group * ATT_CHUNK;#if defined(__x86_64__)@@ -319,16 +449,20 @@ static void attn_worker(void *arg, int tid, int n_threads)atomic_fetch_add_explicit(&a->next, 1, memory_order_relaxed);if (it >= items)break;- uint32_t sub = it % a->split, c = it / a->split % a->n_chunks;- uint32_t kvh = it / a->split / a->n_chunks, g0 = sub * gs;+ uint32_t sub = it % a->split, c = a->c0 + it / a->split % span;+ uint32_t kvh = it / a->split / span, g0 = sub * gs;uint32_t t0 = c * ATT_CHUNK, t1 = t0 + ATT_CHUNK;if (t1 > a->pos0 + a->n)t1 = a->pos0 + a->n;uint32_t nt = t1 - t0;- const int8_t *K = a->k + ((size_t)kvh * A->n_ctx + t0) * hd;- const int8_t *V = a->v + ((size_t)kvh * A->n_ctx + t0) * hd;- const float *ks = a->ks + (size_t)kvh * A->n_ctx + t0;- const float *vs = a->vs + (size_t)kvh * A->n_ctx + t0;+ /* rows of one or two bytes a number; in a ring, the chunk's rows+ are contiguous all the same (n_ctx is a multiple of a chunk) */+ size_t r0 = (size_t)kvh * A->n_ctx + (A->ring ? t0 % A->n_ctx : t0);+ size_t row0 = (r0 * hd) << A->kv16;+ const int8_t *K = a->k + row0;+ const int8_t *V = a->v + row0;+ const float *ks = a->ks + r0;+ const float *vs = a->vs + r0;/* token j sees positions up to pos0 + j: tokens before jlo seenothing of this chunk */uint32_t jlo = t0 > a->pos0 ? t0 - a->pos0 : 0;@@ -348,7 +482,7 @@ static void attn_worker(void *arg, int tid, int n_threads)const int16_t *qi[8];const float *qf[8];float qsc[8], *srow[8];- int fast = A->scores == JANAS_ATTN_INT16;+ int fast = A->scores == JANAS_ATTN_INT16 && !A->kv16;for (uint32_t s = 0; s < step; s++) {uint32_t sl = s0 + s, j = jlo + sl / gs, g = g0 + sl % gs;size_t h = (size_t)j * A->n_head + kvh * group + g;@@ -361,7 +495,17 @@ static void attn_worker(void *arg, int tid, int n_threads)srow[s] = sc + (size_t)sl * ATT_CHUNK;}const int8_t *pf = s0 ? NULL : V;- if (fast) {+ if (A->kv16) {+#if defined(__x86_64__)+ if (avx)+ att_scores_f32_avx2_16(qf, (const int16_t *)K, ks, nt, hd,+ scale, srow, (const int16_t *)pf,+ (int)step);+ else+#endif+ att_scores_f32_ref16(qf, (const int16_t *)K, ks, nt, hd,+ scale, srow, (int)step);+ } else if (fast) {#if defined(__x86_64__)if (vnni)att_scores_i16_vnni(qi, K, ks, nt, hd, qsc, scale, srow, pf,@@ -429,10 +573,16 @@ static void attn_worker(void *arg, int tid, int n_threads)att_partial(A, kvh, c, jlo + sl / gs, g0 + sl % gs) + 2;}#if defined(__x86_64__)- if (avx)+ if (avx && A->kv16)+ att_values_avx2_16(w, (const int16_t *)V, nt, hd, out,+ (int)step);+ else if (avx)att_values_avx2(w, V, nt, hd, out, (int)step);else#endif+ if (A->kv16)+ att_values_ref16(w, (const int16_t *)V, nt, hd, out, (int)step);+ elseatt_values_ref(w, V, nt, hd, out, (int)step);}}@@ -507,14 +657,14 @@ static void merge_worker(void *arg, int tid, int n_threads)uint32_t kvh = h / group, g = h % group;uint32_t nc = (a->pos0 + j + ATT_CHUNK) / ATT_CHUNK;float mx = -INFINITY, sum = 0.0f;- for (uint32_t c = 0; c < nc; c++) {+ for (uint32_t c = a->c0; c < nc; c++) {float v = att_partial(A, kvh, c, j, g)[0];if (v > mx)mx = v;}if (skip_stat > 0) {uint64_t k[3] = {0};- for (uint32_t c = 0; c < nc; c++) {+ for (uint32_t c = a->c0; c < nc; c++) {float v = att_partial(A, kvh, c, j, g)[0];k[0] += v < mx - 8.0f;k[1] += v < mx - 16.0f;@@ -530,7 +680,7 @@ static void merge_worker(void *arg, int tid, int n_threads)}float *out = a->out + (size_t)j * qd + (size_t)h * hd;memset(out, 0, hd * sizeof(float));- for (uint32_t c = 0; c < nc; c++) {+ for (uint32_t c = a->c0; c < nc; c++) {const float *pm = att_partial(A, kvh, c, j, g);float w = expf(pm[0] - mx);sum += w * pm[1];@@ -625,6 +775,26 @@ void janas_attn_set_window(struct janas_attn *a, uint32_t window)a->window = window;}+uint32_t janas_attn_ring_positions(uint32_t window, uint32_t n)+{+ /* the window before the first token, the tokens, and a chunk more for+ the first one's partial alignment, in whole chunks */+ uint32_t need = window + n + ATT_CHUNK;+ return (need + ATT_CHUNK - 1) / ATT_CHUNK * ATT_CHUNK;+}++void janas_attn_set_ring(struct janas_attn *a, int on)+{+ a->ring = on != 0;+}++void janas_attn_set_kv16(struct janas_attn *a, int on)+{+ a->kv16 = on != 0;+ if (a->kv16 && a->scores == JANAS_ATTN_INT16)+ a->scores = JANAS_ATTN_FLOAT; /* no integer path over 16-bit keys */+}+int janas_attn_scores_mode(const struct janas_attn *a){return a ? a->scores : JANAS_ATTN_FLOAT;@@ -654,7 +824,18 @@ void janas_attn_run(struct janas_attn *A, struct janas_pool *pool,/* enough items for every thread: split the query heads of a group */uint32_t n_chunks = (pos0 + n + ATT_CHUNK - 1) / ATT_CHUNK, split = 1;uint32_t group = A->n_head / A->n_kv;- while (A->n_kv * n_chunks * split < 2 * (uint32_t)A->n_threads &&+ /* with a window, the chunks wholly before the first token's are never+ read: at a long context most of them */+ uint32_t c0 = A->window && pos0 + 1 > A->window+ ? (pos0 + 1 - A->window) / ATT_CHUNK+ : 0;+ if (A->ring && n_chunks - c0 > A->chunks) {+ /* the ring is too small for this call: a caller's error */+ fprintf(stderr, "janas_attn_run: %u chunks for a ring of %u\n",+ n_chunks - c0, A->chunks);+ abort();+ }+ while (A->n_kv * (n_chunks - c0) * split < 2 * (uint32_t)A->n_threads &&group % (split * 2) == 0)split *= 2;struct attn_job job = {.A = A,@@ -667,6 +848,7 @@ void janas_attn_run(struct janas_attn *A, struct janas_pool *pool,.n = n,.pos0 = pos0,.n_chunks = n_chunks,+ .c0 = c0,.split = split};job.qi = A->qi;job.qs = A->qs;@@ -677,7 +859,7 @@ void janas_attn_run(struct janas_attn *A, struct janas_pool *pool,if (skip_stat)atexit(skip_report);}- if (A->scores == JANAS_ATTN_INT16)+ if (A->scores == JANAS_ATTN_INT16 && !A->kv16)janas_pool_run(pool, quant_worker, &job);janas_pool_run(pool, attn_worker, &job);janas_pool_run(pool, merge_worker, &job);diff --git a/src/llm/attention.h b/src/llm/attention.hindex 06092fa..7872075 100644--- a/src/llm/attention.h+++ b/src/llm/attention.h@@ -67,6 +67,21 @@ void janas_attn_set_scale(struct janas_attn *a, float scale);/* A sliding window: token p sees positions p - window + 1 .. p only; 0 (thedefault) for all. The cache still holds every position. */void janas_attn_set_window(struct janas_attn *a, uint32_t window);+/* Keys and values of sixteen bits (int16 with the same scale per position+ and KV head) instead of eight; the scores are then computed over the float+ query (FLOAT). Twice the bytes, some 250 times finer: Gemma 4 needs it. */+void janas_attn_set_kv16(struct janas_attn *a, int on);+/*+ * A ring (with a window): the cache holds n_ctx positions, n_ctx a multiple+ * of 256, position p in row p mod n_ctx; a call must fit in it, window and+ * the call's tokens together with room for the chunk boundaries+ * (janas_attn_ring_positions). The partial results are kept for that many+ * positions only.+ */+void janas_attn_set_ring(struct janas_attn *a, int on);+/* The positions a ring needs for a window and calls of up to n tokens+ whose keys and values are written before any of them is attended. */+uint32_t janas_attn_ring_positions(uint32_t window, uint32_t n);/* What the scores are actually computed with: FLOAT or INT16. */int janas_attn_scores_mode(const struct janas_attn *a);diff --git a/src/llm/chat.h b/src/llm/chat.hindex ca746b0..915d632 100644--- a/src/llm/chat.h+++ b/src/llm/chat.h@@ -22,6 +22,10 @@enum chat_format {FORMAT_CHATML, /* <|im_start|>role\ntext<|im_end|>\n */+ /* <bos> once, then <|turn>role\ntext<turn|>\n, the assistant called+ "model", the reasoning in <|channel>thought\n...<channel|>:+ ChatML's shape, so im_start and im_end hold <|turn> and <turn|> */+ FORMAT_GEMMA,};struct janas_llm {@@ -29,11 +33,16 @@ struct janas_llm {struct janas_tokenizer *tok;int format;int32_t im_start, im_end;- int32_t think_open, think_close; /* reasoning markers, or -1 */- int thinker; /* the template knows enable_thinking */- enum janas_tool_dialect tools; /* how its template writes tool calls */- int32_t call_open, call_close; /* <tool_call>, </tool_call>, or -1 */- int32_t resp_open, resp_close; /* <tool_response>, </tool_response> */+ const char *asst_role; /* "assistant", or Gemma's "model" */+ int32_t bos; /* written once at the start, or -1 */+ int32_t think_on; /* Gemma's <|think|>: reasoning asked for, or -1 */+ int32_t quote, decl_open, decl_close; /* Gemma's <|"|>, <|tool>, <tool|> */+ int32_t think_open, think_close; /* reasoning markers, or -1 */+ int thinker; /* the template knows enable_thinking */+ int empty_thought; /* Gemma 4: an empty channel when not reasoning */+ enum janas_tool_dialect tools; /* how its template writes tool calls */+ int32_t call_open, call_close; /* <tool_call>, </tool_call>, or -1 */+ int32_t resp_open, resp_close; /* <tool_response>, </tool_response> */uint32_t pooling; /* embeddings: 1 mean, 2 first, 3 last; 0 none */int add_eos; /* the tokenizer ends every text with eos_id */int32_t eos_id;@@ -53,6 +62,10 @@ struct janas_llm {int32_t mode;int gpu_wanted; /* the caller's say, on top of what the mode decides */uint64_t kv_token; /* bytes of attention's keys and values per token */+ /* and of the sliding-window layers' rings, per position up to ring+ positions (Gemma 4) */+ uint64_t kv_ring_token;+ uint32_t kv_ring;};struct janas_llm_chat {@@ -60,10 +73,13 @@ struct janas_llm_chat {struct janas_llm_session *s;struct janas_llm_chat_params p;char *system;- int replying; /* a reply is being read */- int closed; /* the last reply ended with a stop token */- int ended; /* the reply is over: the next call says DONE */- int in_think; /* the reply is in its reasoning part */+ int replying; /* a reply is being read */+ int closed; /* the last reply ended with a stop token */+ int ended; /* the reply is over: the next call says DONE */+ int in_think; /* the reply is in its reasoning part */+ int in_turn; /* Gemma: the reply continues the model's turn (after the+ tools' answers), no new turn to open */+ int chan_label; /* Gemma: the channel's name ("thought\n") to skip */uint32_t reply_tokens;uint32_t answer_tokens; /* of them, outside the reasoning */char pend[1024]; /* reply bytes not yet returned */@@ -217,6 +233,7 @@ int janas_api_add_message(janas_llm_chat *c, const char *role, const char *text,size_t n);int janas_api_add_system(janas_llm_chat *c, const char *sys, size_t n);int janas_api_start_reply(janas_llm_chat *c);+int janas_api_begin_sequence(janas_llm_chat *c);int janas_api_note_turn(janas_llm_chat *c, uint32_t at);/* The reply starts: reply_at is where its turn begins in the sequence(UINT32_MAX for raw text), total the prompt's length, cached how much ofdiff --git a/src/llm/gpu.c b/src/llm/gpu.cindex c4a98db..5313038 100644--- a/src/llm/gpu.c+++ b/src/llm/gpu.c@@ -1438,7 +1438,16 @@ struct janas_gpu {struct janas_gpu *janas_gpu_create(char *err, size_t err_len){- snprintf(err, err_len, "built without GPU support");+#if !defined(JANAS_GPU_SPV)+ snprintf(err, err_len,+ "built without GPU support: glslangValidator was "+ "not installed when Janas was built");+#else+ snprintf(err, err_len,+ "built without GPU support: the Vulkan headers "+ "(vulkan/vulkan.h) were not installed when Janas "+ "was built");+#endifreturn NULL;}diff --git a/src/llm/model.c b/src/llm/model.cindex b32c069..d6a9ea8 100644--- a/src/llm/model.c+++ b/src/llm/model.c@@ -120,11 +120,74 @@ static void sandwich_worker(void *arg, int tid, int n_threads)float *x = m->x + (size_t)j * dm;const float *r = m->xres + (size_t)j * dm;rms_norm(x, x, t->w, dm, m->eps);+ /* with per-layer embeddings the scale comes after their branch */+ float s = m->ple_dim ? 1.0f : t->ly->out_scale;for (uint32_t i = 0; i < dm; i++)- x[i] = (r[i] + x[i]) * t->ly->out_scale;+ x[i] = (r[i] + x[i]) * s;}}+/* Gemma 4's mixture, token by token: the dense branch's output normalized+ aside (xbr), then the experts' input and the router's from the residual,+ then the experts' output normalized and the two summed */+static void branch1_worker(void *arg, int tid, int n_threads)+{+ struct tok_job *t = arg;+ struct janas_llm_model *m = t->m;+ uint32_t dm = m->d_model;+ for (uint32_t j = (uint32_t)tid; j < t->n; j += (uint32_t)n_threads)+ rms_norm(m->xbr + (size_t)j * dm, m->x + (size_t)j * dm,+ f32(m, t->ly->post_norm1), dm, m->eps);+}++static void prep2_worker(void *arg, int tid, int n_threads)+{+ struct tok_job *t = arg;+ struct janas_llm_model *m = t->m;+ uint32_t dm = m->d_model;+ const float *rs = t->ly->router_scale;+ float sd = 1.0f / sqrtf((float)dm);+ for (uint32_t j = (uint32_t)tid; j < t->n; j += (uint32_t)n_threads) {+ const float *r = m->xres + (size_t)j * dm;+ float *xn = m->xn + (size_t)j * dm;+ /* the experts' input, to Q8_K through xn */+ rms_norm(xn, r, f32(m, t->ly->pre_norm2), dm, m->eps);+ janas_q8k_quantize(xn, m->xq + j * (dm / JANAS_QK), dm);+ /* the router's: the residual normalized without weights, scaled+ by 1 / sqrt(d_model) and per dimension */+ double ss = 0.0;+ for (uint32_t i = 0; i < dm; i++)+ ss += (double)r[i] * r[i];+ float sc = 1.0f / sqrtf((float)(ss / dm) + m->eps);+ for (uint32_t i = 0; i < dm; i++)+ xn[i] = r[i] * sc * sd * rs[i];+ }+}++static void branch2_worker(void *arg, int tid, int n_threads)+{+ struct tok_job *t = arg;+ struct janas_llm_model *m = t->m;+ uint32_t dm = m->d_model;+ for (uint32_t j = (uint32_t)tid; j < t->n; j += (uint32_t)n_threads) {+ float *x = m->x + (size_t)j * dm;+ const float *b = m->xbr + (size_t)j * dm;+ rms_norm(x, x, f32(m, t->ly->post_norm2), dm, m->eps);+ for (uint32_t i = 0; i < dm; i++)+ x[i] += b[i];+ }+}++static void run_tok(struct janas_llm_model *m, const struct layer *ly,+ uint32_t n, void (*fn)(void *, int, int))+{+ struct tok_job t = {.m = m, .ly = ly, .n = n};+ if (n < 2)+ fn(&t, 0, 1);+ else+ janas_pool_run(m->compute, fn, &t);+}+/* logits soft-capped: c * tanh(l / c) */static void softcap_worker(void *arg, int tid, int n_threads){@@ -137,6 +200,35 @@ static void softcap_worker(void *arg, int tid, int n_threads)l[i] = c * tanhf(l[i] / c);}+/* A key row turned back by the angles in cs, sn (half pairs, NeoX) and put+ in its bits again: a rotation can take a pair past the largest number of+ the row, so the scale is found anew. row: int8 or int16 (wide). */+static void unrotate_row(void *row, float *scale, uint32_t hd, uint32_t half,+ const float *cs, const float *sn, int wide)+{+ float tmp[512], mx = 0.0f, z = *scale;+ int8_t *r8 = row;+ int16_t *r16 = row;+ for (uint32_t i = 0; i < hd; i++)+ tmp[i] = (wide ? (float)r16[i] : (float)r8[i]) * z;+ for (uint32_t i = 0; i < half; i++) {+ float a = tmp[i], b = tmp[i + half];+ tmp[i] = a * cs[i] + b * sn[i];+ tmp[i + half] = b * cs[i] - a * sn[i];+ }+ for (uint32_t i = 0; i < hd; i++)+ if (fabsf(tmp[i]) > mx)+ mx = fabsf(tmp[i]);+ float top = wide ? 32767.0f : 127.0f, inv = mx > 0 ? top / mx : 0.0f;+ for (uint32_t i = 0; i < hd; i++) {+ if (wide)+ r16[i] = (int16_t)lrintf(tmp[i] * inv);+ else+ r8[i] = (int8_t)lrintf(tmp[i] * inv);+ }+ *scale = mx / top;+}+/** Forgets the positions keep .. keep + drop of the attention layers: what* follows them moves down and its keys are turned back by drop positions,@@ -144,66 +236,83 @@ static void softcap_worker(void *arg, int tid, int n_threads)* state is left alone: it is a summary of everything seen, and in a hybrid* model that is what carries the older text once the window has passed it.* The first `keep` positions stay where they are, which keeps the system- * message - and the anchor it gives attention - in place.+ * message - and the anchor it gives attention - in place. Each slot with its+ * own geometry and RoPE (Gemma 4: sliding-window layers and full ones); a+ * ring (a sliding-window layer's) keeps only its last positions, and those+ * are the ones that move.*/int janas_llm_model_shift(struct janas_llm_model *m, uint32_t keep,uint32_t drop, uint32_t n_kv){- /* (Gemma 4: two RoPEs and a window, not done yet) */- if (!drop || n_kv > m->n_ctx || keep + drop > n_kv || m->head_dim > 512 ||- m->a->swa)- return -1;- uint32_t hd = m->head_dim, half = m->n_rot / 2;- uint32_t move = n_kv - keep - drop;- float *cs = malloc(2 * (size_t)half * sizeof(float));- if (!cs)+ if (!drop || n_kv > m->n_ctx || keep + drop > n_kv || m->hd_max > 512)return -1;- float *sn = cs + half;- for (uint32_t i = 0; i < half; i++) {- float theta = (float)drop *- powf(m->rope_base, -2.0f * (float)i / (float)m->n_rot);- cs[i] = cosf(theta);- sn[i] = sinf(theta);- }- uint32_t slots = m->n_attn + (m->mtp ? 1u : 0u);- for (uint32_t sl = 0; sl < slots; sl++)- for (uint32_t h = 0; h < m->n_head_kv; h++) {- size_t base = ((size_t)sl * m->n_head_kv + h) * m->n_ctx * hd;- size_t sbase = ((size_t)sl * m->n_head_kv + h) * m->n_ctx;- int8_t *k = m->kcache + base, *v = m->vcache + base;- float *kz = m->kscale + sbase, *vz = m->vscale + sbase;- if (move) {- memmove(k + (size_t)keep * hd, k + (size_t)(keep + drop) * hd,- (size_t)move * hd);- memmove(v + (size_t)keep * hd, v + (size_t)(keep + drop) * hd,- (size_t)move * hd);- memmove(kz + keep, kz + keep + drop, move * sizeof(float));- memmove(vz + keep, vz + keep + drop, move * sizeof(float));- }- for (uint32_t p = keep; p < keep + move; p++) {- int8_t *row = k + (size_t)p * hd;- /* turned back and put in eight bits again: a rotation can- take a pair past the largest number of the head, so the- scale is found anew */- float tmp[512], mx = 0.0f;- for (uint32_t i = 0; i < half; i++) {- float a = (float)row[i] * kz[p];- float b = (float)row[i + half] * kz[p];- tmp[i] = a * cs[i] + b * sn[i];- tmp[i + half] = b * cs[i] - a * sn[i];+ uint32_t move = n_kv - keep - drop, slots = m->n_attn + (m->mtp ? 1u : 0u);+ size_t elt = (size_t)1 << m->kv16;+ float cs[256], sn[256];+ uint8_t *ring_tmp = NULL;+ for (uint32_t sl = 0; sl < slots; sl++) {+ uint32_t hd = m->kv_hd[sl], cap = m->kv_cap[sl];+ int swa = m->kv_swa[sl];+ uint32_t n_rot = swa ? m->n_rot_swa : m->n_rot, half = n_rot / 2;+ float base = swa ? m->rope_base_swa : m->rope_base;+ const float *ff = swa ? NULL : m->rope_freqs;+ for (uint32_t i = 0; i < half; i++) {+ float theta =+ (float)drop * powf(base, -2.0f * (float)i / (float)n_rot);+ if (ff)+ theta /= ff[i];+ cs[i] = cosf(theta);+ sn[i] = sinf(theta);+ }+ size_t row = hd * elt;+ for (uint32_t h = 0; h < m->kv_heads[sl]; h++) {+ uint8_t *k = (uint8_t *)m->kcache + m->kv_off[sl] + h * cap * row;+ uint8_t *v = (uint8_t *)m->vcache + m->kv_off[sl] + h * cap * row;+ float *kz = m->kscale + m->ks_off[sl] + (size_t)h * cap;+ float *vz = m->vscale + m->ks_off[sl] + (size_t)h * cap;+ if (!m->kv_ring[sl]) {+ if (move) {+ memmove(k + keep * row, k + (keep + drop) * row,+ move * row);+ memmove(v + keep * row, v + (keep + drop) * row,+ move * row);+ memmove(kz + keep, kz + keep + drop, move * sizeof(float));+ memmove(vz + keep, vz + keep + drop, move * sizeof(float));}- for (uint32_t i = 2 * half; i < hd; i++)- tmp[i] = (float)row[i] * kz[p];- for (uint32_t i = 0; i < hd; i++)- if (fabsf(tmp[i]) > mx)- mx = fabsf(tmp[i]);- float inv = mx > 0 ? 127.0f / mx : 0.0f;- for (uint32_t i = 0; i < hd; i++)- row[i] = (int8_t)lrintf(tmp[i] * inv);- kz[p] = mx / 127.0f;+ for (uint32_t p = keep; p < keep + move; p++)+ unrotate_row(k + p * row, kz + p, hd, half, cs, sn,+ m->kv16);+ continue;+ }+ /* a ring: the positions it still has, from a copy of it, each+ to its new row (they cannot land on a kept one: see the+ bound on lo) */+ if (!ring_tmp &&+ !(ring_tmp = malloc(2 * (size_t)cap * (row + sizeof(float)))))+ return -1;+ uint8_t *tk = ring_tmp, *tv = tk + cap * row;+ float *tkz = (float *)(void *)(tv + cap * row), *tvz = tkz + cap;+ memcpy(tk, k, cap * row);+ memcpy(tv, v, cap * row);+ memcpy(tkz, kz, cap * sizeof(float));+ memcpy(tvz, vz, cap * sizeof(float));+ uint32_t lo = keep + drop;+ if (n_kv > cap && n_kv - cap > lo)+ lo = n_kv - cap;+ for (uint32_t p = lo; p < n_kv; p++) {+ uint32_t src = p % cap, dst = (p - drop) % cap;+ memcpy(k + dst * row, tk + src * row, row);+ memcpy(v + dst * row, tv + src * row, row);+ kz[dst] = tkz[src];+ vz[dst] = tvz[src];+ unrotate_row(k + dst * row, kz + dst, hd, half, cs, sn,+ m->kv16);}}- free(cs);+ }+ free(ring_tmp);+ if (m->ring_hi > keep + drop)+ m->ring_hi -= drop;/* the recurrent state holds what it has seen, dropped tokens included -that is what carries the older text - but the position it stands atmoves with everything else */@@ -284,6 +393,8 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,fprintf(stderr, "\n");trace_row(m, "embed", m->x + (size_t)(n - 1) * dm, dm);}+ if (m->ple_dim && janas_m_ple_inputs(m, tokens, n) != 0)+ return -1;set_rope(m, n, pos0);for (uint32_t l = 0; l < m->n_layer; l++) {const struct layer *ly = &m->layers[l];@@ -301,8 +412,26 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,memcpy(m->xres, m->x, (size_t)n * dm * sizeof(float));memset(m->x, 0, (size_t)n * dm * sizeof(float));}- if (janas_m_moe_block(m, ly, m->cache, &m->j.layers[l], l, n) != 0)+ if (ly->pre_norm2) {+ /* Gemma 4's mixture: the dense branch on xn / xq as normed+ above, then the experts' branch */+ size_t bytes = (size_t)n * dm * sizeof(float);+ if (janas_m_moe_block(m, ly, m->cache, &m->j.layers[l], l, n,+ MOE_SHARED) != 0)+ return -1;+ run_tok(m, ly, n, branch1_worker);+ trace_row(m, "ffn_mlp", m->xbr + (size_t)(n - 1) * dm, dm);+ run_tok(m, ly, n, prep2_worker);+ memset(m->x, 0, bytes);+ if (janas_m_moe_block(m, ly, m->cache, &m->j.layers[l], l, n,+ MOE_ROUTED) != 0)+ return -1;+ run_tok(m, ly, n, branch2_worker);+ trace_row(m, "ffn_moe_comb", m->x + (size_t)(n - 1) * dm, dm);+ } else if (janas_m_moe_block(m, ly, m->cache, &m->j.layers[l], l, n,+ MOE_ALL) != 0) {return -1;+ }if (ly->post_ffn) {struct tok_job st = {.m = m, .ly = ly, .w = f32(m, ly->post_ffn), .n = n};@@ -311,6 +440,8 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,elsejanas_pool_run(m->compute, sandwich_worker, &st);}+ if (m->ple_dim)+ janas_m_ple_layer(m, ly, l, n);if (m->trace) {/* JANAS_TRACE=1: the state after each layer, last token of theblock, to compare with another implementation */@@ -322,6 +453,7 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,l, sqrt(nn), v[0], v[1], v[2], v[3]);}}+ m->ring_hi = pos0 + n; /* the rings hold up to here */double td = now();uint32_t first = all_logits ? 0 : n - 1;/* with an MTP block, every token's final hidden state is kept for it */@@ -511,7 +643,8 @@ int janas_llm_mtp_forward(struct janas_llm_model *m, const int32_t *tokens,f32(m, ly->ffn_norm), dm, m->eps);janas_q8k_quantize(m->xn + (size_t)j * dm, m->xq + j * nbd, dm);}- if (janas_m_moe_block(m, ly, m->mtp_cache, m->mtp_jl, m->mtp_cl, n) != 0)+ if (janas_m_moe_block(m, ly, m->mtp_cache, m->mtp_jl, m->mtp_cl, n,+ MOE_ALL) != 0)return -1;double td = now();uint32_t first = all_logits ? 0 : n - 1;diff --git a/src/llm/model_attn.c b/src/llm/model_attn.cindex 1812496..809d994 100644--- a/src/llm/model_attn.c+++ b/src/llm/model_attn.c@@ -98,8 +98,9 @@ static void qkv_item(const struct tok_job *t, uint32_t j, uint32_t h)float *kv = m->kk + (size_t)j * kvd + (size_t)h * hd;rms_norm(kv, kv, f32(m, ly->k_norm), hd, m->eps);rope_neox(kv, ly->n_rot, cs, sn);- size_t at = ((size_t)h * m->n_ctx + t->pos0 + j) * hd;- size_t as = (size_t)h * m->n_ctx + t->pos0 + j;+ uint32_t p = t->pos0 + j, row = t->ring ? p % t->cap : p;+ size_t at = ((size_t)h * t->cap + row) * hd;+ size_t as = (size_t)h * t->cap + row;float *vr = m->vv + (size_t)j * kvd + (size_t)h * hd;if (m->a->swa) { /* Gemma 4: V normalized, without weights */double ss = 0.0;@@ -120,6 +121,18 @@ static void qkv_item(const struct tok_job *t, uint32_t j, uint32_t h)if (b > mv)mv = b;}+ if (m->kv16) { /* sixteen bits: the largest becomes 32767 */+ t->kc_scale[as] = mk / 32767.0f;+ t->vc_scale[as] = mv / 32767.0f;+ float ik = mk > 0 ? 32767.0f / mk : 0.0f;+ float iv = mv > 0 ? 32767.0f / mv : 0.0f;+ int16_t *kc = (int16_t *)(void *)t->kc, *vc = (int16_t *)(void *)t->vc;+ for (uint32_t i = 0; i < hd; i++) {+ kc[at + i] = (int16_t)lrintf(kv[i] * ik);+ vc[at + i] = (int16_t)lrintf(vr[i] * iv);+ }+ return;+ }t->kc_scale[as] = mk / 127.0f;t->vc_scale[as] = mv / 127.0f;float ik = mk > 0 ? 127.0f / mk : 0.0f;@@ -133,7 +146,8 @@ static void qkv_item(const struct tok_job *t, uint32_t j, uint32_t h)static void qkv_worker(void *arg, int tid, int n_threads){struct tok_job *t = arg;- uint32_t nh = t->m->n_head + t->ly->n_kv;+ /* a layer without keys and values of its own: its queries alone */+ uint32_t nh = t->m->n_head + (t->ly->wk ? t->ly->n_kv : 0);for (uint32_t i = (uint32_t)tid; i < t->n * nh; i += (uint32_t)n_threads)qkv_item(t, i / nh, i % nh);}@@ -212,6 +226,8 @@ void janas_m_attn_layer(struct janas_llm_model *m, const struct layer *ly,double ta = now();/* qwen3next: each query head is followed by its output gate */float *qout = m->a->attn_gate ? m->big : m->q;+ /* no k: the keys and values of an earlier layer, already cached */+ int nt = !ly->wk ? 1 : ly->wv ? 3 : 2;struct janas_matvec_task t[3] = {{.type = (int)ly->wq->type,.w = data(m, ly->wq),.x = m->xq,@@ -219,8 +235,8 @@ void janas_m_attn_layer(struct janas_llm_model *m, const struct layer *ly,.rows = m->a->attn_gate ? 2 * qd : qd,.cols = dm,.n_vec = n},- {.type = (int)ly->wk->type,- .w = data(m, ly->wk),+ {.type = ly->wk ? (int)ly->wk->type : 0,+ .w = ly->wk ? data(m, ly->wk) : NULL,.x = m->xq,.y = m->kk,.rows = kvd,@@ -234,8 +250,8 @@ void janas_m_attn_layer(struct janas_llm_model *m, const struct layer *ly,.cols = dm,.n_vec = n}};janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,- m->compute, t, ly->wv ? 3 : 2);- if (!ly->wv) /* Gemma 4's full layers: V is K, before K's norm */+ m->compute, t, (size_t)nt);+ if (nt == 2) /* Gemma 4's full layers: V is K, before K's norm */memcpy(m->vv, m->kk, (size_t)n * kvd * sizeof(float));int8_t *kc = m->kcache + m->kv_off[ly->slot];int8_t *vc = m->vcache + m->kv_off[ly->slot];@@ -248,7 +264,9 @@ void janas_m_attn_layer(struct janas_llm_model *m, const struct layer *ly,.kc_scale = kcs,.vc_scale = vcs,.kc = kc,- .vc = vc};+ .vc = vc,+ .cap = m->kv_cap[ly->slot],+ .ring = m->kv_ring[ly->slot]};janas_pool_run(m->compute, qkv_worker, &tj);/* the last token's last head of each, as llama.cpp's graph names them */trace_row(m, "Qcur_pos", m->q + (size_t)(n - 1) * qd + (m->n_head - 1) * hd,diff --git a/src/llm/model_ffn.c b/src/llm/model_ffn.cindex 8dd7a7e..9995ef9 100644--- a/src/llm/model_ffn.c+++ b/src/llm/model_ffn.c@@ -99,11 +99,12 @@ static void act_worker(void *arg, int tid, int n_threads){struct act_job *a = arg;struct janas_llm_model *m = a->m;- uint32_t ff = m->d_ff;+ uint32_t stride = m->ff_buf;for (uint32_t i = (uint32_t)tid; i < a->n; i += (uint32_t)n_threads) {size_t g = m->glist[i];- const float *gt = m->gate + g * ff, *u = m->up + g * ff;- float *h = m->h + g * ff;+ uint32_t ff = m->hlen[g];+ const float *gt = m->gate + g * stride, *u = m->up + g * stride;+ float *h = m->h + g * stride;/* deterministic operations only, so that a GPU computing anexpert gives the same bits */uint32_t r = 0;@@ -122,7 +123,7 @@ static void act_worker(void *arg, int tid, int n_threads)for (; r < ff; r++)h[r] = janas_silu_mul_det(gt[r], u[r]);}- janas_q8k_quantize_det(h, m->hq + g * ff / JANAS_QK, ff);+ janas_q8k_quantize_det(h, m->hq + g * stride / JANAS_QK, ff);}}@@ -170,8 +171,8 @@ static void router_worker(void *arg, int tid, int n_threads)uint32_t ne = m->n_expert, dm = m->d_model;for (uint32_t e = (uint32_t)tid; e < ne; e += (uint32_t)n_threads)for (uint32_t j = 0; j < r->n; j++) {- const uint16_t *w = r->w + (size_t)e * dm;const float *x = m->xn + (size_t)j * dm;+ const uint16_t *w = r->w + (size_t)e * dm;#if defined(__x86_64__)if (m->avx2 && m->f16c) {m->router[(size_t)j * ne + e] = dot_f16_avx2(w, x, dm);@@ -211,7 +212,7 @@ static void run_experts(struct janas_llm_model *m,const struct expert_run *runs, uint32_t n_runs){double t_enter = janas_m_exp_tracing() ? now() : 0;- uint32_t dm = m->d_model, ff = m->d_ff;+ uint32_t dm = m->d_model, ff = m->ff_buf;size_t nbd = dm / JANAS_QK, nbf = ff / JANAS_QK;struct janas_matvec_task t1[2 * MAX_RUNS], t2[MAX_RUNS];uint32_t ng = 0;@@ -222,6 +223,7 @@ static void run_experts(struct janas_llm_model *m,m->xq + (size_t)m->pair_tok[m->order[g]] * nbd,nbd * sizeof(*m->xg));m->glist[ng++] = g;+ m->hlen[g] = runs[r].mx[0].rows;}for (int p = 0; p < 2; p++) {const struct janas_jns_matrix *mx = &runs[r].mx[p];@@ -326,19 +328,27 @@ static void axpy_worker(void *arg, int tid, int n_threads)are layer l of the file behind the cache (the main model or the MTP). */int janas_m_moe_block(struct janas_llm_model *m, const struct layer *ly,struct janas_expert_cache *cache,- const struct janas_jns_layer *jl, uint32_t l, uint32_t n)+ const struct janas_jns_layer *jl, uint32_t l, uint32_t n,+ int part){double t0 = now();- uint32_t dm = m->d_model, ne = m->n_expert, k = m->k_use;- struct router_job rj = {m, ly->router16, n};- janas_pool_run(m->compute, router_worker, &rj);+ uint32_t dm = m->d_model, ne = m->n_expert;+ uint32_t k = part == MOE_SHARED ? 0 : m->k_use;+ if (k) {+ struct router_job rj = {m, ly->router16, n};+ janas_pool_run(m->compute, router_worker, &rj);- /* per token: softmax, top-k, normalized weights */- struct tok_job tj = {.m = m, .n = n};- if (n < 2)- route_tok(m, 0);- else- janas_pool_run(m->compute, route_worker, &tj);+ /* per token: softmax, top-k, normalized weights */+ struct tok_job tj = {.m = m, .n = n};+ if (n < 2)+ route_tok(m, 0);+ else+ janas_pool_run(m->compute, route_worker, &tj);+ /* an expert's own output scale, folded into its weight (Gemma 4) */+ if (ly->exp_scale)+ for (uint32_t p = 0; p < n * k; p++)+ m->pair_w[p] *= ly->exp_scale[m->pair_exp[p]];+ }/* counting sort of the pairs by expert */uint32_t np = n * k;memset(m->count, 0, (ne + 1) * sizeof(uint32_t));@@ -359,11 +369,14 @@ int janas_m_moe_block(struct janas_llm_model *m, const struct layer *ly,}/* the shared expert: one more pair per token, after the routed ones */uint32_t nt = np;- if (m->a->shared_expert) {+ if (ly->sh_w[0] && part != MOE_ROUTED) {for (uint32_t j = 0; j < n; j++, nt++) {m->pair_tok[nt] = j;+ /* its gate, or none: a branch of its own (Gemma 4) */m->pair_w[nt] =- sigmoid(janas_dot_f32(ly->sh_gate, m->xn + (size_t)j * dm, dm));+ ly->sh_gate ? sigmoid(janas_dot_f32(ly->sh_gate,+ m->xn + (size_t)j * dm, dm))+ : 1.0f;m->order[nt] = nt;}}diff --git a/src/llm/model_impl.h b/src/llm/model_impl.hindex 15a25ea..dc84a19 100644--- a/src/llm/model_impl.h+++ b/src/llm/model_impl.h@@ -40,6 +40,16 @@ struct layer {output scale (1 if none) */const struct janas_jns_tensor *post_attn, *post_ffn;float out_scale;+ /* Gemma 4's mixture: the dense feed-forward (the shared expert, weight+ 1, no gate) and the experts are two branches, each normalized on the+ way in and out; the router reads its own input, scaled per dimension,+ and each expert's output has a scale of its own */+ const struct janas_jns_tensor *pre_norm2, *post_norm1, *post_norm2;+ const float *router_scale, *exp_scale;+ /* Gemma 4's per-layer embeddings (E2B, E4B): after the FFN, the layer's+ own input of ple_dim numbers per token gates a branch of the+ residual, gelu(gate x) * input, projected back, normalized, added */+ const struct janas_jns_tensor *ple_gate, *ple_proj, *ple_post;/* Gated DeltaNet */const struct janas_jns_tensor *ssm_in, *ssm_ba, *conv, *dt_bias, *ssm_a,*ssm_norm, *ssm_out;@@ -80,6 +90,8 @@ struct janas_llm_model {const struct janas_arch *a; /* what this architecture has (arch.h) */uint32_t n_layer, d_model, n_head, n_head_kv, head_dim, n_rot, n_expert, k,n_vocab, n_ctx, d_ff, n_attn, n_rec;+ uint32_t ff_buf; /* rows of a pair's buffers: d_ff, or a wider shared+ feed-forward's (Gemma 4's 26B) */float eps, rope_base;/* Gemma 4: the sliding-window layers' geometry (head size and RoPE inhead_swa, n_rot_swa, rope_base_swa; n_head_kv, head_dim and n_rot@@ -89,11 +101,32 @@ struct janas_llm_model {uint32_t head_swa, n_rot_swa, n_kv_swa, window, hd_max, kvd_max;float rope_base_swa, softcap;const float *rope_freqs;+ /* Gemma 4's per-layer embeddings (0: none): ple_dim numbers per layer+ and token, from a table of their own read a row at a time from the+ file (ple_tok, never in memory) and a projection of the embedding+ (ple_model, bf16), normalized by ple_norm; ple holds the block's,+ token after token, layer after layer, ple_g a branch's rows,+ ple_row a row of the table as floats, then as read (room for twice+ its numbers in floats) */+ uint32_t ple_dim;+ const struct janas_jns_tensor *ple_tok, *ple_model, *ple_norm;+ float *ple, *ple_g, *ple_row;+ struct janas_block_q8k *ple_q;+ /* layers from n_kv_layers on keep no keys and values of their own:+ they read those of the last earlier layer of their kind */+ uint32_t n_kv_layers;/* the KV cache of slot s (attention layer, then the MTP block's) startsat kv_off[s] bytes and its scales at ks_off[s] floats; kv_hd andkv_heads its geometry. Uniform unless the layers differ (Gemma 4) */size_t *kv_off, *ks_off;uint32_t *kv_hd, *kv_heads;+ int kv16; /* keys and values of 16 bits (arch, or JANAS_KV=8|16) */+ /* positions a slot holds per KV head: n_ctx, or for a sliding-window+ layer a ring of fewer (kv_ring), position p in row p mod kv_cap;+ ring_hi: one past the last position written */+ uint32_t *kv_cap;+ uint8_t *kv_ring, *kv_swa; /* and whether its layer is a sliding one */+ uint32_t ring_hi;/* Gated DeltaNet geometry: ds = head size (keys and values), n_kh keyheads, n_vh value heads (n_vh / n_kh per key head), conv over theq|k|v channels */@@ -157,7 +190,7 @@ struct janas_llm_model {struct janas_attn *attn;/* Gemma 4: the sliding-window layers' RoPE table and attention, and theresidual kept aside while the FFN's output is normalized */- float *rope_cos_swa, *rope_sin_swa, *xres;+ float *rope_cos_swa, *rope_sin_swa, *xres, *xbr; /* xbr: a branch */struct janas_attn *attn_swa;/*@@ -199,6 +232,7 @@ struct janas_llm_model {then one pair per token for the shared expert */uint32_t *pair_tok, *pair_exp, *order, *count, *glist;float *pair_w, *gate, *up, *h, *dout;+ uint32_t *hlen; /* per pair row: the rows of its expert's gate */struct janas_block_q8k *xg, *hq;uint32_t k_use; /* experts used per token, at most k (see set_experts) */int exp_level; /* planes of the experts' down matrix read: 1, 2 or 3 */@@ -282,7 +316,8 @@ struct tok_job {uint32_t first, n, pos0;float *kc_scale, *vc_scale;int8_t *kc, *vc;- float *out; /* rows the job rewrites (the soft-capped logits) */+ uint32_t cap, ring; /* the slot's positions per head, and whether a ring */+ float *out; /* rows the job rewrites (the soft-capped logits) */};void janas_m_init_tuner(struct janas_llm_model *m);@@ -297,10 +332,17 @@ void janas_m_attn_layer(struct janas_llm_model *m, const struct layer *ly,uint32_t n, uint32_t pos0);void janas_m_rec_layer(struct janas_llm_model *m, const struct layer *ly,uint32_t n);+int janas_m_ple_inputs(struct janas_llm_model *m, const int32_t *tokens,+ uint32_t n);+void janas_m_ple_layer(struct janas_llm_model *m, const struct layer *ly,+ uint32_t l, uint32_t n);int janas_m_exp_tracing(void);void janas_m_exp_report(void);+/* which experts a moe_block call runs */+enum { MOE_ALL, MOE_SHARED, MOE_ROUTED };int janas_m_moe_block(struct janas_llm_model *m, const struct layer *ly,struct janas_expert_cache *cache,- const struct janas_jns_layer *jl, uint32_t l, uint32_t n);+ const struct janas_jns_layer *jl, uint32_t l, uint32_t n,+ int part);#endifdiff --git a/src/llm/model_load.c b/src/llm/model_load.cindex f9d7989..1ef2c20 100644--- a/src/llm/model_load.c+++ b/src/llm/model_load.c@@ -101,12 +101,18 @@ static int load_swa_params(struct janas_llm_model *m, const uint8_t **pat)if (janas_jns_meta_bytes(&m->j, key, &np, pat) || np < m->n_layer)return -1;snprintf(key, sizeof(key), "%.32s.attention.head_count_kv", m->j.h.arch);- if (janas_jns_meta_ints(&m->j, key, &nk, &kv) || nk < m->n_layer)- return -1;+ /* E2B and E4B give one count for both kinds */+ uint32_t one = 0;+ if (janas_jns_meta_ints(&m->j, key, &nk, &kv) || nk < m->n_layer) {+ if (janas_jns_meta_u32(&m->j, key, &one) || one == 0)+ return -1;+ kv = NULL;+ }m->n_head_kv = m->n_kv_swa = 0;for (uint32_t l = 0; l < m->n_layer; l++) {- int32_t h;- memcpy(&h, kv + l, 4);+ int32_t h = (int32_t)one;+ if (kv)+ memcpy(&h, kv + l, 4);uint32_t *want = (*pat)[l] ? &m->n_kv_swa : &m->n_head_kv;if (h <= 0 || (*want && *want != (uint32_t)h))return -1;@@ -182,6 +188,10 @@ static int load_layer(struct janas_llm_model *m, const struct janas_jns *j,snprintf(name, sizeof(name), "blk.%u." suffix, l); \if (!(ly->field = need(j, name, kind, d0, d1, err, err_len))) \return -1;+#define NEED_T(var, suffix, d0) \+ snprintf(name, sizeof(name), "blk.%u." suffix, l); \+ if (!(var = need(j, name, T_F32, d0, 0, err, err_len))) \+ return -1;NEED(attn_norm, "attn_norm.weight", T_F32, dm, 0)if (ly->rec) {uint32_t s = m->ds;@@ -211,14 +221,36 @@ static int load_layer(struct janas_llm_model *m, const struct janas_jns *j,NEED(ssm_out, "ssm_out.weight", T_QUANT, m->d_inner, dm)} else {NEED(wq, "attn_q.weight", T_QUANT, dm, qd * (m->a->attn_gate ? 2 : 1))- NEED(wk, "attn_k.weight", T_QUANT, dm, kvd)- snprintf(name, sizeof(name), "blk.%u.attn_v.weight", l);- if (!m->a->swa || janas_jns_tensor(j, name)) {- NEED(wv, "attn_v.weight", T_QUANT, dm, kvd)- }NEED(wo, "attn_output.weight", T_QUANT, qd, dm)NEED(q_norm, "attn_q_norm.weight", T_F32, ly->hd, 0)- NEED(k_norm, "attn_k_norm.weight", T_F32, ly->hd, 0)+ /* a layer reading an earlier one's keys and values has no use for+ its k and v, even where the file has them */+ if (!m->n_kv_layers || l < m->n_kv_layers) {+ NEED(wk, "attn_k.weight", T_QUANT, dm, kvd)+ snprintf(name, sizeof(name), "blk.%u.attn_v.weight", l);+ if (!m->a->swa || janas_jns_tensor(j, name)) {+ NEED(wv, "attn_v.weight", T_QUANT, dm, kvd)+ }+ NEED(k_norm, "attn_k_norm.weight", T_F32, ly->hd, 0)+ }+ }+ if (m->ple_dim) {+ const struct janas_jns_tensor *pt[2];+ static const char *const pn[2] = {"inp_gate.weight", "proj.weight"};+ for (int p = 0; p < 2; p++) {+ snprintf(name, sizeof(name), "blk.%u.%s", l, pn[p]);+ pt[p] = janas_jns_tensor(j, name);+ uint64_t d0 = p ? m->ple_dim : dm, d1 = p ? dm : m->ple_dim;+ if (!pt[p] || pt[p]->dims[0] != d0 || pt[p]->dims[1] != d1 ||+ (pt[p]->type != 0 &&+ janas_qtype_block_size((int)pt[p]->type) == 0)) {+ snprintf(err, err_len, "tensor %s missing or unexpected", name);+ return -1;+ }+ }+ ly->ple_gate = pt[0];+ ly->ple_proj = pt[1];+ NEED(ple_post, "post_norm.weight", T_F32, dm, 0)}ly->out_scale = 1.0f;if (m->a->sandwich) {@@ -264,6 +296,41 @@ static int load_layer(struct janas_llm_model *m, const struct janas_jns *j,}ly->router16 = h16;}+ if (m->a->sandwich && ly->gate_inp) {+ /* Gemma 4's mixture: the dense feed-forward beside the experts (as+ wide as its file says, widened by gguf2jns), its norms, and the+ router's and the experts' scales */+ static const char *const dn[3] = {"ffn_gate.weight", "ffn_up.weight",+ "ffn_down.weight"};+ const struct janas_jns_tensor *sw[3];+ for (int p = 0; p < 3; p++) {+ snprintf(name, sizeof(name), "blk.%u.%s", l, dn[p]);+ if (!(sw[p] = need(j, name, T_QUANT, p == 2 ? sw[0]->dims[1] : dm,+ p == 2 ? dm : 0, err, err_len)))+ return -1;+ }+ if (sw[0]->dims[1] != sw[1]->dims[1] || sw[0]->dims[1] % JANAS_QK ||+ sw[2]->dims[0] != sw[0]->dims[1] || sw[2]->dims[1] != dm) {+ snprintf(err, err_len, "layer %u: dense feed-forward shapes", l);+ return -1;+ }+ for (int p = 0; p < 3; p++) {+ ly->sh_w[p] = data(m, sw[p]);+ ly->sh_mx[p] =+ (struct janas_jns_matrix){.type = sw[p]->type,+ .rows = (uint32_t)sw[p]->dims[1],+ .cols = (uint32_t)sw[p]->dims[0],+ .bytes = sw[p]->bytes};+ }+ NEED(pre_norm2, "pre_ffw_norm_2.weight", T_F32, dm, 0)+ NEED(post_norm1, "post_ffw_norm_1.weight", T_F32, dm, 0)+ NEED(post_norm2, "post_ffw_norm_2.weight", T_F32, dm, 0)+ const struct janas_jns_tensor *rs, *es;+ NEED_T(rs, "ffn_gate_inp.scale", dm)+ NEED_T(es, "ffn_down_exps.scale", m->n_expert)+ ly->router_scale = f32(m, rs);+ ly->exp_scale = f32(m, es);+ }if (m->a->shared_expert) {const struct janas_jns_tensor *sg, *sw[3];static const char *const sh_names[3] = {"ffn_gate_shexp.weight",@@ -306,6 +373,7 @@ static int load_layer(struct janas_llm_model *m, const struct janas_jns *j,}}#undef NEED+#undef NEED_Treturn 0;}@@ -465,12 +533,48 @@ struct janas_llm_model *janas_llm_model_load(const char *path,m->kvd_max = m->n_head_kv * m->head_dim;if (m->n_kv_swa * m->head_swa > m->kvd_max)m->kvd_max = m->n_kv_swa * m->head_swa;- for (uint32_t l = 0; l < m->n_layer; l++)- if (m->j.layers[l].m[JANAS_JNS_GATE].rows != m->d_ff ||+ /* the experts of a mixture are alike; a dense model's feed-forward+ may widen from one layer to the next (Gemma 4's E2B: 6144, then+ 12288): the buffers take the widest */+ for (uint32_t l = 0; l < m->n_layer; l++) {+ uint32_t r = m->j.layers[l].m[JANAS_JNS_GATE].rows;+ if ((r != m->d_ff && m->n_expert > 1) || r % JANAS_QK ||m->j.layers[l].m[JANAS_JNS_DOWN].rows != m->d_model) {snprintf(err, err_len, "layer %u: expert shapes differ", l);goto fail;}+ if (r > m->d_ff)+ m->d_ff = r;+ }+ /* Gemma 4's E2B and E4B: per-layer embeddings, and the last layers+ reading the keys and values of earlier ones */+ if (m->a->swa) {+ uint32_t shared = 0;+ meta_u32(m, "embedding_length_per_layer_input", &m->ple_dim);+ meta_u32(m, "attention.shared_kv_layers", &shared);+ if (shared && (shared + 2 > m->n_layer || !swa_pat)) {+ snprintf(err, err_len, "inconsistent shared_kv_layers");+ goto fail;+ }+ m->n_kv_layers = shared ? m->n_layer - shared : 0;+ if (m->ple_dim &&+ (m->ple_dim % JANAS_QK ||+ !(m->ple_tok =+ janas_jns_tensor(&m->j, "per_layer_token_embd.weight")) ||+ m->ple_tok->dims[0] != (uint64_t)m->n_layer * m->ple_dim ||+ janas_qtype_block_size((int)m->ple_tok->type) == 0 ||+ !(m->ple_model =+ janas_jns_tensor(&m->j, "per_layer_model_proj.weight")) ||+ m->ple_model->type != 30 || m->ple_model->dims[0] != m->d_model ||+ m->ple_model->dims[1] != (uint64_t)m->n_layer * m->ple_dim ||+ !(m->ple_norm =+ janas_jns_tensor(&m->j, "per_layer_proj_norm.weight")) ||+ m->ple_norm->type != 0 || m->ple_norm->dims[0] != m->ple_dim)) {+ snprintf(err, err_len,+ "per-layer embeddings missing or unexpected");+ goto fail;+ }+ }/* resident region into RAM; with the GPU, memory from its driver(JANAS_GPU_ALLOC) or imported at the end */@@ -494,10 +598,25 @@ struct janas_llm_model *janas_llm_model_load(const char *path,snprintf(err, err_len, "out of memory for the resident region");goto fail;}+ /* all of it but the per-layer embeddings' table, read a row at a time+ where it is (model_ple.c): its pages, never touched, take no memory */+ uint64_t skip_a = m->j.h.resident_bytes, skip_b = skip_a;+ if (m->ple_tok && m->ple_tok->offset >= m->j.h.resident_offset) {+ skip_a = (m->ple_tok->offset - m->j.h.resident_offset) & ~4095ull;+ skip_b =+ (m->ple_tok->offset - m->j.h.resident_offset + m->ple_tok->bytes) &+ ~4095ull;+ if (skip_b < skip_a)+ skip_b = skip_a;+ }for (uint64_t done = 0; done < m->j.h.resident_bytes;) {- ssize_t r =- pread(m->j.fd, m->resident + done, m->j.h.resident_bytes - done,- (off_t)(m->j.h.resident_offset + done));+ if (done == skip_a && skip_b > skip_a) {+ done = skip_b;+ continue;+ }+ uint64_t end = done < skip_a ? skip_a : m->j.h.resident_bytes;+ ssize_t r = pread(m->j.fd, m->resident + done, end - done,+ (off_t)(m->j.h.resident_offset + done));if (r <= 0) {snprintf(err, err_len, "cannot read the resident region");goto fail;@@ -529,13 +648,25 @@ struct janas_llm_model *janas_llm_model_load(const char *path,for (uint32_t l = 0; l < m->n_layer; l++) {struct layer *ly = &m->layers[l];ly->rec = m->a->rec && (l + 1) % interval != 0;- ly->slot = ly->rec ? m->n_rec++ : m->n_attn++;if (swa_pat) {ly->swa = swa_pat[l] != 0;ly->hd = ly->swa ? m->head_swa : m->head_dim;ly->n_kv = ly->swa ? m->n_kv_swa : m->n_head_kv;ly->n_rot = ly->swa ? m->n_rot_swa : m->n_rot;}+ if (m->n_kv_layers && l >= m->n_kv_layers) {+ /* the slot of the last layer of its kind that has one, as+ llama.cpp takes it: the one before last, or the last */+ uint32_t src = m->n_kv_layers - (ly->swa ? 2 : 1);+ if (m->layers[src].swa != ly->swa) {+ snprintf(err, err_len, "layer %u: no keys and values to share",+ l);+ goto fail;+ }+ ly->slot = m->layers[src].slot;+ } else {+ ly->slot = ly->rec ? m->n_rec++ : m->n_attn++;+ }if (load_layer(m, &m->j, l, ly, err, err_len) != 0)goto fail;}@@ -564,13 +695,39 @@ struct janas_llm_model *janas_llm_model_load(const char *path,}m->rope_freqs = rf ? f32(m, rf) : NULL;}- /* the KV cache, slot after slot, each with its own geometry */+ /* the KV cache, slot after slot, each with its own geometry, at eight+ bits a number or sixteen */+ {+ const char *kv = getenv("JANAS_KV");+ m->kv16 = kv ? atoi(kv) == 16 : m->a->kv16;+ }+ {+ const char *pb = getenv("JANAS_PREFILL_BLOCK");+ long v = pb ? atol(pb) : 256;+ m->blk = v < JANAS_LLM_MAX_BLOCK ? JANAS_LLM_MAX_BLOCK+ : v > 4096 ? 4096+ : (uint32_t)v;+ }+ /* the sliding-window layers keep a ring of positions, not the whole+ context: the window and a prefill block (JANAS_SWA_RING=0 keeps+ every position, as the full layers do) */+ uint32_t ring_cap = 0;+ if (m->a->swa) {+ const char *rg = getenv("JANAS_SWA_RING");+ ring_cap = janas_attn_ring_positions(m->window, m->blk);+ if ((rg && atoi(rg) == 0) || ring_cap >= m->n_ctx)+ ring_cap = 0;+ }uint32_t n_slots = m->n_attn + (m->mtp ? 1 : 0);+ m->kv_cap = calloc(n_slots + 1, sizeof(uint32_t));+ m->kv_ring = calloc(n_slots + 1, 1);+ m->kv_swa = calloc(n_slots + 1, 1);m->kv_off = malloc((n_slots + 1) * sizeof(size_t));m->ks_off = malloc((n_slots + 1) * sizeof(size_t));m->kv_hd = calloc(n_slots + 1, sizeof(uint32_t));m->kv_heads = calloc(n_slots + 1, sizeof(uint32_t));- if (!m->kv_off || !m->ks_off || !m->kv_hd || !m->kv_heads) {+ if (!m->kv_off || !m->ks_off || !m->kv_hd || !m->kv_heads || !m->kv_cap ||+ !m->kv_ring || !m->kv_swa) {snprintf(err, err_len, "out of memory");goto fail;}@@ -580,13 +737,17 @@ struct janas_llm_model *janas_llm_model_load(const char *path,continue;m->kv_hd[ly->slot] = ly->hd;m->kv_heads[ly->slot] = ly->n_kv;+ m->kv_swa[ly->slot] = (uint8_t)ly->swa;+ m->kv_ring[ly->slot] = ly->swa && ring_cap;+ m->kv_cap[ly->slot] = m->kv_ring[ly->slot] ? ring_cap : m->n_ctx;}size_t kv_len = 0, ks_len = 0;for (uint32_t s = 0; s < n_slots; s++) {m->kv_off[s] = kv_len;m->ks_off[s] = ks_len;- kv_len += (size_t)m->n_ctx * m->kv_heads[s] * m->kv_hd[s];- ks_len += (size_t)m->n_ctx * m->kv_heads[s];+ kv_len += ((size_t)m->kv_cap[s] * m->kv_heads[s] * m->kv_hd[s])+ << m->kv16;+ ks_len += (size_t)m->kv_cap[s] * m->kv_heads[s];}m->kv_off[n_slots] = kv_len;m->ks_off[n_slots] = ks_len;@@ -594,13 +755,12 @@ struct janas_llm_model *janas_llm_model_load(const char *path,m->vcache = malloc(kv_len ? kv_len : 1);m->kscale = malloc((ks_len ? ks_len : 1) * sizeof(float));m->vscale = malloc((ks_len ? ks_len : 1) * sizeof(float));- {- const char *pb = getenv("JANAS_PREFILL_BLOCK");- long v = pb ? atol(pb) : 256;- m->blk = v < JANAS_LLM_MAX_BLOCK ? JANAS_LLM_MAX_BLOCK- : v > 4096 ? 4096- : (uint32_t)v;- }+ /* a pair's rows: the experts', or a wider shared feed-forward's */+ m->ff_buf = m->d_ff;+ for (uint32_t l = 0; l < m->n_layer; l++)+ if (m->layers[l].sh_w[0] && m->layers[l].sh_mx[0].rows > m->ff_buf)+ m->ff_buf = m->layers[l].sh_mx[0].rows;+ ff = m->ff_buf;size_t B = m->blk, P = B * (m->k + 1);size_t inner = m->d_inner > qd ? m->d_inner : qd;size_t bigw = m->qkvz_dim > 2 * qd ? m->qkvz_dim : 2 * qd;@@ -617,7 +777,19 @@ struct janas_llm_model *janas_llm_model_load(const char *path,m->rope_cos_swa = malloc(B * m->n_rot_swa / 2 * sizeof(float));m->rope_sin_swa = malloc(B * m->n_rot_swa / 2 * sizeof(float));m->xres = malloc(B * dm * sizeof(float));- if (!m->rope_cos_swa || !m->rope_sin_swa || !m->xres) {+ m->xbr = malloc(B * dm * sizeof(float));+ if (!m->rope_cos_swa || !m->rope_sin_swa || !m->xres || !m->xbr) {+ snprintf(err, err_len, "out of memory");+ goto fail;+ }+ }+ if (m->ple_dim) {+ size_t W = (size_t)m->n_layer * m->ple_dim;+ m->ple = malloc(B * W * sizeof(float));+ m->ple_g = malloc(B * m->ple_dim * sizeof(float));+ m->ple_row = malloc(2 * W * sizeof(float));+ m->ple_q = malloc(B * m->ple_dim / JANAS_QK * sizeof(*m->ple_q));+ if (!m->ple || !m->ple_g || !m->ple_row || !m->ple_q) {snprintf(err, err_len, "out of memory");goto fail;}@@ -636,6 +808,7 @@ struct janas_llm_model *janas_llm_model_load(const char *path,m->dout = malloc(P * dm * sizeof(float));m->xg = malloc(P * dm / JANAS_QK * sizeof(*m->xg));m->hq = malloc(P * ff / JANAS_QK * sizeof(*m->hq));+ m->hlen = malloc(P * sizeof(uint32_t));m->tok_first = malloc((B + 1) * sizeof(uint32_t));m->tok_fill = malloc(B * sizeof(uint32_t));m->tok_list = malloc(P * sizeof(uint32_t));@@ -644,7 +817,8 @@ struct janas_llm_model *janas_llm_model_load(const char *path,!m->kk || !m->vv || !m->att || !m->router || !m->rope_cos ||!m->rope_sin || !m->xq || !m->attq || !m->pair_tok || !m->pair_exp ||!m->order || !m->glist || !m->count || !m->pair_w || !m->gate ||- !m->up || !m->h || !m->dout || !m->xg || !m->hq || m->n_expert > 4096) {+ !m->up || !m->h || !m->dout || !m->xg || !m->hq || !m->hlen ||+ m->n_expert > 4096) {snprintf(err, err_len, "out of memory");goto fail;}@@ -769,14 +943,18 @@ struct janas_llm_model *janas_llm_model_load(const char *path,janas_pool_set_spin(m->io, 0);m->attn = janas_attn_create(m->n_head, m->n_head_kv, m->head_dim, m->n_ctx,janas_pool_size(m->compute), o->attn_scores);+ if (m->attn)+ janas_attn_set_kv16(m->attn, m->kv16);if (m->attn && m->a->swa) {/* Gemma 4: the q and k norms scale the scores; the sliding-windowlayers have an attention of their own */janas_attn_set_scale(m->attn, 1.0f);- m->attn_swa =- janas_attn_create(m->n_head, m->n_kv_swa, m->head_swa, m->n_ctx,- janas_pool_size(m->compute), o->attn_scores);+ m->attn_swa = janas_attn_create(+ m->n_head, m->n_kv_swa, m->head_swa, ring_cap ? ring_cap : m->n_ctx,+ janas_pool_size(m->compute), o->attn_scores);if (m->attn_swa) {+ janas_attn_set_ring(m->attn_swa, ring_cap != 0);+ janas_attn_set_kv16(m->attn_swa, m->kv16);janas_attn_set_scale(m->attn_swa, 1.0f);janas_attn_set_window(m->attn_swa, m->window);}@@ -902,11 +1080,20 @@ void janas_llm_model_free(struct janas_llm_model *m)free(m->vscale);free(m->kv_off);free(m->ks_off);+ free(m->kv_cap);+ free(m->kv_ring);+ free(m->kv_swa);free(m->kv_hd);free(m->kv_heads);free(m->rope_cos_swa);+ free(m->ple);+ free(m->ple_g);+ free(m->ple_row);+ free(m->ple_q);free(m->rope_sin_swa);free(m->xres);+ free(m->xbr);+ free(m->hlen);free(m->kcache);free(m->vcache);free(m->ssm_buf);diff --git a/src/llm/model_ple.c b/src/llm/model_ple.cnew file mode 100644index 0000000..a9c8c73--- /dev/null+++ b/src/llm/model_ple.c@@ -0,0 +1,222 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * model_ple.c - Gemma 4's per-layer embeddings (E2B, E4B).+ *+ * Every token brings, besides its embedding, a few numbers for each layer+ * (ple_dim, 256): a row of a table of their own (n_layer x ple_dim numbers+ * per token, as large as half of E2B's file) plus a projection of the+ * embedding, normalized layer by layer. After its feed-forward, each layer+ * opens a branch of the residual: gelu(gate x) times its own numbers,+ * projected back to the model's width, normalized, added.+ *+ * The table is read a row at a time from the file, when a token needs it:+ * a few kilobytes a token, never gigabytes in memory.+ */+#define _GNU_SOURCE+#include <math.h>+#include <stdlib.h>+#include <string.h>+#include <unistd.h>++#include "matvec.h"+#include "vmath.h"++#include "model_impl.h"++/* f32 rounded to bf16 as ggml does it (to nearest, ties to even), kept as+ f32: the embedding meets the bf16 projection at bf16, as in llama.cpp */+static float round_bf16(float f)+{+ uint32_t u = janas_f2u(f);+ if ((u & 0x7fffffffu) > 0x7f800000u) /* NaN */+ return f;+ u = (u + (0x7fffu + ((u >> 16) & 1u))) & 0xffff0000u;+ return janas_u2f(u);+}++struct bf16_job {+ const uint16_t *w;+ const float *x; /* n rows of cols, rounded to bf16 */+ float *y; /* n rows of rows */+ uint32_t rows, cols, n;+ float scale;+};++#if defined(__x86_64__)+#include <immintrin.h>++__attribute__((target("avx2,fma"))) static float+dot_bf16_avx2(const uint16_t *w, const float *x, uint32_t n)+{+ __m256 acc = _mm256_setzero_ps();+ uint32_t i = 0;+ for (; i + 8 <= n; i += 8) {+ __m128i h = _mm_loadu_si128((const __m128i *)(const void *)(w + i));+ __m256 wf = _mm256_castsi256_ps(+ _mm256_slli_epi32(_mm256_cvtepu16_epi32(h), 16));+ acc = _mm256_fmadd_ps(wf, _mm256_loadu_ps(x + i), acc);+ }+ __m128 s =+ _mm_add_ps(_mm256_castps256_ps128(acc), _mm256_extractf128_ps(acc, 1));+ s = _mm_add_ps(s, _mm_movehl_ps(s, s));+ s = _mm_add_ss(s, _mm_movehdup_ps(s));+ float r = _mm_cvtss_f32(s);+ for (; i < n; i++)+ r += janas_u2f((uint32_t)w[i] << 16) * x[i];+ return r;+}+#endif++static float dot_bf16(const uint16_t *w, const float *x, uint32_t n)+{+#if defined(__x86_64__)+ if (__builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma"))+ return dot_bf16_avx2(w, x, n);+#endif+ float r = 0.0f;+ for (uint32_t i = 0; i < n; i++)+ r += janas_u2f((uint32_t)w[i] << 16) * x[i];+ return r;+}++/* rows split in equal runs, each row read once for all the tokens */+static void bf16_worker(void *arg, int tid, int n_threads)+{+ const struct bf16_job *b = arg;+ uint32_t per = (b->rows + (uint32_t)n_threads - 1) / (uint32_t)n_threads;+ uint32_t r0 = (uint32_t)tid * per;+ uint32_t r1 = r0 + per < b->rows ? r0 + per : b->rows;+ for (uint32_t r = r0; r < r1; r++) {+ const uint16_t *w = b->w + (size_t)r * b->cols;+ for (uint32_t j = 0; j < b->n; j++)+ b->y[(size_t)j * b->rows + r] =+ dot_bf16(w, b->x + (size_t)j * b->cols, b->cols) * b->scale;+ }+}++struct f32_job {+ const float *w, *x;+ float *y;+ uint32_t rows, cols, n;+};++static void f32_worker(void *arg, int tid, int n_threads)+{+ const struct f32_job *b = arg;+ size_t total = (size_t)b->rows * b->n;+ for (size_t i = (size_t)tid; i < total; i += (size_t)n_threads) {+ uint32_t j = (uint32_t)(i / b->rows), r = (uint32_t)(i % b->rows);+ const float *w = b->w + (size_t)r * b->cols;+ const float *x = b->x + (size_t)j * b->cols;+ float s = 0.0f;+ for (uint32_t c = 0; c < b->cols; c++)+ s += w[c] * x[c];+ b->y[(size_t)j * b->rows + r] = s;+ }+}++/* y (n rows of t's rows) = t x (n rows of cols): f32 weights as they are,+ quantized ones on x put in Q8_K (q, n * cols / 256 blocks) */+static void product(struct janas_llm_model *m, const struct janas_jns_tensor *t,+ const float *x, struct janas_block_q8k *q, float *y,+ uint32_t n)+{+ uint32_t cols = (uint32_t)t->dims[0], rows = (uint32_t)t->dims[1];+ if (t->type == 0) {+ struct f32_job b = {+ .w = f32(m, t), .x = x, .y = y, .rows = rows, .cols = cols, .n = n};+ janas_pool_run(m->compute, f32_worker, &b);+ return;+ }+ size_t nb = cols / JANAS_QK;+ for (uint32_t j = 0; j < n; j++)+ janas_q8k_quantize(x + (size_t)j * cols, q + j * nb, cols);+ struct janas_matvec_task tk = {.type = (int)t->type,+ .w = data(m, t),+ .x = q,+ .y = y,+ .rows = rows,+ .cols = cols,+ .n_vec = n};+ janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,+ m->compute, &tk, 1);+}++/*+ * The block's per-layer inputs, from the embeddings in x (already scaled by+ * sqrt(d_model)): the projection over sqrt(d_model), each layer's part+ * normalized, plus the table's row times sqrt(ple_dim), all over sqrt(2).+ */+int janas_m_ple_inputs(struct janas_llm_model *m, const int32_t *tokens,+ uint32_t n)+{+ uint32_t dm = m->d_model, P = m->ple_dim, W = m->n_layer * P;+ float *xb = m->xn; /* the embeddings at bf16 */+ for (size_t i = 0; i < (size_t)n * dm; i++)+ xb[i] = round_bf16(m->x[i]);+ struct bf16_job b = {.w = data(m, m->ple_model),+ .x = xb,+ .y = m->ple,+ .rows = W,+ .cols = dm,+ .n = n,+ .scale = 1.0f / sqrtf((float)dm)};+ if (n < 2)+ bf16_worker(&b, 0, 1);+ else+ janas_pool_run(m->compute, bf16_worker, &b);+ const float *nw = f32(m, m->ple_norm);+ size_t row_bytes =+ W / JANAS_QK * janas_qtype_block_size((int)m->ple_tok->type);+ uint8_t *raw = (uint8_t *)(m->ple_row + W); /* the row as read */+ float es = sqrtf((float)P), is2 = 1.0f / sqrtf(2.0f);+ for (uint32_t j = 0; j < n; j++) {+ off_t at =+ (off_t)(m->ple_tok->offset + (uint64_t)tokens[j] * row_bytes);+ for (size_t done = 0; done < row_bytes;) {+ ssize_t r =+ pread(m->j.fd, raw + done, row_bytes - done, at + (off_t)done);+ if (r <= 0)+ return -1;+ done += (size_t)r;+ }+ janas_dequantize((int)m->ple_tok->type, raw, m->ple_row, W);+ float *p = m->ple + (size_t)j * W;+ for (uint32_t l = 0; l < m->n_layer; l++) {+ float *pl = p + (size_t)l * P;+ const float *el = m->ple_row + (size_t)l * P;+ rms_norm(pl, pl, nw, P, m->eps);+ for (uint32_t i = 0; i < P; i++)+ pl[i] = (pl[i] + el[i] * es) * is2;+ }+ }+ trace_row(m, "inp_per_layer", m->ple + (size_t)(n - 1) * W, P);+ return 0;+}++/*+ * Layer l's branch, on the residual in x after the feed-forward, then the+ * layer's output scale: x = (x + norm(proj(gelu(gate x) * input))) * s.+ */+void janas_m_ple_layer(struct janas_llm_model *m, const struct layer *ly,+ uint32_t l, uint32_t n)+{+ uint32_t dm = m->d_model, P = m->ple_dim, W = m->n_layer * P;+ product(m, ly->ple_gate, m->x, m->xq, m->ple_g, n);+ for (uint32_t j = 0; j < n; j++) {+ float *g = m->ple_g + (size_t)j * P;+ const float *in = m->ple + (size_t)j * W + (size_t)l * P;+ for (uint32_t i = 0; i < P; i++)+ g[i] = janas_gelu_mul_det(g[i], in[i]);+ }+ product(m, ly->ple_proj, m->ple_g, m->ple_q, m->xn, n);+ const float *pw = f32(m, ly->ple_post);+ for (uint32_t j = 0; j < n; j++) {+ float *x = m->x + (size_t)j * dm, *y = m->xn + (size_t)j * dm;+ rms_norm(y, y, pw, dm, m->eps);+ for (uint32_t i = 0; i < dm; i++)+ x[i] = (x[i] + y[i]) * ly->out_scale;+ }+ trace_row(m, "per_layer_out", m->x + (size_t)(n - 1) * dm, dm);+}diff --git a/src/llm/model_state.c b/src/llm/model_state.cindex cee1457..64ee241 100644--- a/src/llm/model_state.c+++ b/src/llm/model_state.c@@ -74,6 +74,13 @@ size_t janas_llm_state_bytes(const struct janas_llm_state *st)return st->bytes;}+/* The positions a slot keeps of a state of n: all, or for a ring the last+ window of them - all that the next token can see. */+static size_t slot_rows(const struct janas_llm_model *m, uint32_t sl, size_t n)+{+ return m->kv_ring[sl] && n > m->window ? m->window : n;+}+/* The bytes of keys (or values) and the number of scales of n positionsof the first n_slots slots. */static void kv_lengths(const struct janas_llm_model *m, uint32_t n_slots,@@ -81,8 +88,9 @@ static void kv_lengths(const struct janas_llm_model *m, uint32_t n_slots,{*kv = *rows = 0;for (uint32_t sl = 0; sl < n_slots; sl++) {- *rows += (size_t)m->kv_heads[sl] * n;- *kv += (size_t)m->kv_heads[sl] * n * m->kv_hd[sl];+ size_t r = slot_rows(m, sl, n);+ *rows += (size_t)m->kv_heads[sl] * r;+ *kv += ((size_t)m->kv_heads[sl] * r * m->kv_hd[sl]) << m->kv16;}}@@ -90,13 +98,34 @@ static void kv_lengths(const struct janas_llm_model *m, uint32_t n_slots,static void copy_kv(struct janas_llm_model *m, struct janas_llm_state *st,int out){- size_t n = st->n, sb = 0, ss = 0;- for (uint32_t sl = 0; sl < st->n_slots; sl++)- for (uint32_t h = 0; h < m->kv_heads[sl];- h++, sb += n * m->kv_hd[sl], ss += n) {- size_t hd = m->kv_hd[sl];- size_t cb = m->kv_off[sl] + h * m->n_ctx * hd;- size_t cs = m->ks_off[sl] + h * m->n_ctx;+ size_t sb = 0, ss = 0;+ for (uint32_t sl = 0; sl < st->n_slots; sl++) {+ size_t n = slot_rows(m, sl, st->n), cap = m->kv_cap[sl];+ size_t hd = (size_t)m->kv_hd[sl] << m->kv16; /* bytes a row */+ if (m->kv_ring[sl]) {+ /* the last n positions, wherever the ring has them */+ for (uint32_t h = 0; h < m->kv_heads[sl]; h++)+ for (size_t p = st->n - n; p < st->n; p++, sb += hd, ss++) {+ size_t r = (size_t)h * cap + p % cap;+ size_t cb = m->kv_off[sl] + r * hd;+ size_t cs = m->ks_off[sl] + r;+ if (out) {+ memcpy(st->k + sb, m->kcache + cb, hd);+ memcpy(st->v + sb, m->vcache + cb, hd);+ st->ks[ss] = m->kscale[cs];+ st->vs[ss] = m->vscale[cs];+ } else {+ memcpy(m->kcache + cb, st->k + sb, hd);+ memcpy(m->vcache + cb, st->v + sb, hd);+ m->kscale[cs] = st->ks[ss];+ m->vscale[cs] = st->vs[ss];+ }+ }+ continue;+ }+ for (uint32_t h = 0; h < m->kv_heads[sl]; h++, sb += n * hd, ss += n) {+ size_t cb = m->kv_off[sl] + h * cap * hd;+ size_t cs = m->ks_off[sl] + h * cap;if (out) {memcpy(st->k + sb, m->kcache + cb, n * hd);memcpy(st->v + sb, m->vcache + cb, n * hd);@@ -109,6 +138,7 @@ static void copy_kv(struct janas_llm_model *m, struct janas_llm_state *st,memcpy(m->vscale + cs, st->vs + ss, n * sizeof(float));}}+ }}static void *dup_mem(const void *p, size_t len)@@ -124,6 +154,11 @@ struct janas_llm_state *janas_llm_model_state_save(struct janas_llm_model *m,{if (n > m->n_ctx)return NULL;+ /* a ring keeps the window before n only while nothing past n + (its+ capacity - the window) has been written over it */+ for (uint32_t sl = 0; sl < m->n_attn; sl++)+ if (m->kv_ring[sl] && m->ring_hi > n + m->kv_cap[sl] - m->window)+ return NULL;int nbuf = 0;if (m->a->rec) {/* the state must stand at n, or be able to get there by replay */@@ -195,6 +230,7 @@ int janas_llm_model_state_load(struct janas_llm_model *m,(m->a->rec != 0) != (st->nbuf != 0))return -1;copy_kv(m, (struct janas_llm_state *)st, 0);+ m->ring_hi = st->n;if (st->nbuf == 1) {/* into the base, standing at n with nothing logged: the next callstarts from it */@@ -256,7 +292,9 @@ static void shape(const struct janas_llm_model *m, uint32_t *h)h[H_VERSION] = STATE_VERSION;h[H_SLOTS] = m->n_attn + (m->mtp ? 1u : 0u);h[H_HEADS] = m->n_head_kv;- h[H_HD] = m->head_dim;+ /* the bytes of a row: a state kept at eight bits does not load into a+ cache of sixteen (JANAS_KV) */+ h[H_HD] = m->head_dim << m->kv16;h[H_REC] = m->n_rec;h[H_VH] = m->n_vh;h[H_DS] = m->ds;diff --git a/src/llm/tools.c b/src/llm/tools.cindex a1ee367..80db70c 100644--- a/src/llm/tools.c+++ b/src/llm/tools.c@@ -20,6 +20,8 @@ struct janas_toolset {enum janas_tool_dialect janas_tools_dialect(const char *tmpl){+ if (tmpl && strstr(tmpl, "<|tool_call>"))+ return JANAS_TOOLS_GEMMA;if (!tmpl || !strstr(tmpl, "<tool_call>"))return JANAS_TOOLS_NONE;if (!strstr(tmpl, "<function="))@@ -172,6 +174,360 @@ static void extra_keys(struct janas_buf *out, const struct janas_json *obj,}}+/* ---- Gemma 4's notation ---- */++#define GQ "<|\"|>" /* its string quote, a token of its own */++/* Jinja's dictsort: an object's members by key, case-insensitive. */+static size_t sorted(const struct janas_json *obj,+ const struct janas_json **out, size_t cap)+{+ size_t n = 0;+ for (const struct janas_json *k = obj ? obj->child : NULL; k && n < cap;+ k = k->next) {+ size_t i = n++;+ for (; i > 0; i--) {+ const struct janas_json *p = out[i - 1];+ size_t m = p->key_n < k->key_n ? p->key_n : k->key_n;+ int c = 0;+ for (size_t x = 0; x < m && !c; x++) {+ int a = (unsigned char)p->key[x], b = (unsigned char)k->key[x];+ a = a >= 'A' && a <= 'Z' ? a + 32 : a;+ b = b >= 'A' && b <= 'Z' ? b + 32 : b;+ c = a - b;+ }+ if (c < 0 || (c == 0 && p->key_n <= k->key_n))+ break;+ out[i] = p;+ }+ out[i] = k;+ }+ return n;+}++/* The template's format_argument: strings quoted, keys quoted or not. */+static void g_arg(struct janas_buf *out, const struct janas_json *v, int ek)+{+ switch (v->type) {+ case JANAS_JSON_NULL:+ janas_buf_puts(out, "null");+ break;+ case JANAS_JSON_TRUE:+ janas_buf_puts(out, "true");+ break;+ case JANAS_JSON_FALSE:+ janas_buf_puts(out, "false");+ break;+ case JANAS_JSON_STRING:+ janas_buf_puts(out, GQ);+ janas_buf_put(out, v->s, v->n);+ janas_buf_puts(out, GQ);+ break;+ case JANAS_JSON_OBJECT: {+ const struct janas_json *k[256];+ size_t n = sorted(v, k, 256);+ janas_buf_puts(out, "{");+ for (size_t i = 0; i < n; i++) {+ if (i)+ janas_buf_puts(out, ",");+ if (ek)+ janas_buf_puts(out, GQ);+ janas_buf_put(out, k[i]->key, k[i]->key_n);+ if (ek)+ janas_buf_puts(out, GQ);+ janas_buf_puts(out, ":");+ g_arg(out, k[i], ek);+ }+ janas_buf_puts(out, "}");+ break;+ }+ case JANAS_JSON_ARRAY:+ janas_buf_puts(out, "[");+ for (const struct janas_json *e = v->child; e; e = e->next) {+ g_arg(out, e, ek);+ if (e->next)+ janas_buf_puts(out, ",");+ }+ janas_buf_puts(out, "]");+ break;+ default:+ janas_buf_put(out, v->s, v->n);+ }+}++/* A JSON string upper-cased, as Jinja's "| upper". */+static void g_upper(struct janas_buf *out, const struct janas_json *v)+{+ for (size_t i = 0; v && v->type == JANAS_JSON_STRING && i < v->n; i++) {+ char c = v->s[i];+ janas_buf_put(out, &(char){c >= 'a' && c <= 'z' ? c - 32 : c}, 1);+ }+}++static int g_type_is(const struct janas_json *v, const char *up)+{+ struct janas_buf b = {0};+ g_upper(&b, janas_json_get(v, "type"));+ int r = b.n == strlen(up) && b.p && memcmp(b.p, up, b.n) == 0;+ janas_buf_free(&b);+ return r;+}++static void g_required(struct janas_buf *out, const struct janas_json *req)+{+ janas_buf_puts(out, "required:[");+ for (const struct janas_json *e = req ? req->child : NULL; e; e = e->next) {+ janas_buf_puts(out, GQ);+ janas_buf_put(out, e->s, e->n);+ janas_buf_puts(out, GQ);+ if (e->next)+ janas_buf_puts(out, ",");+ }+ janas_buf_puts(out, "]");+}++/* The template's format_parameters. */+static void g_params(struct janas_buf *out, const struct janas_json *props,+ int filter)+{+ static const char *const std[] = {"description", "type", "properties",+ "required", "nullable"};+ const struct janas_json *k[256];+ size_t n = sorted(props, k, 256);+ int first = 1;+ for (size_t i = 0; i < n; i++) {+ const struct janas_json *v = k[i];+ int skip = 0;+ for (size_t s = 0; filter && s < 5; s++)+ skip |=+ strlen(std[s]) == v->key_n && !memcmp(std[s], v->key, v->key_n);+ if (skip)+ continue;+ if (!first)+ janas_buf_puts(out, ",");+ first = 0;+ janas_buf_put(out, v->key, v->key_n);+ janas_buf_puts(out, ":{");+ int comma = 0;+ const struct janas_json *d = janas_json_get(v, "description");+ if (d && d->type == JANAS_JSON_STRING && d->n) {+ janas_buf_puts(out, "description:" GQ);+ janas_buf_put(out, d->s, d->n);+ janas_buf_puts(out, GQ);+ comma = 1;+ }+ const struct janas_json *en = janas_json_get(v, "enum");+ const struct janas_json *it = janas_json_get(v, "items");+ if (g_type_is(v, "STRING") && en && en->n) {+ janas_buf_puts(out, comma ? ",enum:" : "enum:");+ comma = 1;+ g_arg(out, en, 1);+ } else if (g_type_is(v, "ARRAY") && it &&+ it->type == JANAS_JSON_OBJECT && it->n) {+ janas_buf_puts(out, comma ? ",items:{" : "items:{");+ comma = 1;+ const struct janas_json *ik[64];+ size_t in = sorted(it, ik, 64);+ int f2 = 1;+ for (size_t x = 0; x < in; x++) {+ const struct janas_json *iv = ik[x];+ if (iv->type == JANAS_JSON_NULL)+ continue;+ if (!f2)+ janas_buf_puts(out, ",");+ f2 = 0;+ if (iv->key_n == 10 && !memcmp(iv->key, "properties", 10)) {+ janas_buf_puts(out, "properties:{");+ if (iv->type == JANAS_JSON_OBJECT)+ g_params(out, iv, 0);+ janas_buf_puts(out, "}");+ } else if (iv->key_n == 8 && !memcmp(iv->key, "required", 8)) {+ g_required(out, iv);+ } else if (iv->key_n == 4 && !memcmp(iv->key, "type", 4)) {+ janas_buf_puts(out, "type:");+ if (iv->type == JANAS_JSON_STRING) {+ janas_buf_puts(out, GQ);+ g_upper(out, iv);+ janas_buf_puts(out, GQ);+ } else {+ janas_buf_puts(out, "[");+ for (const struct janas_json *e = iv->child; e;+ e = e->next) {+ janas_buf_puts(out, GQ);+ g_upper(out, e);+ janas_buf_puts(out, GQ);+ if (e->next)+ janas_buf_puts(out, ",");+ }+ janas_buf_puts(out, "]");+ }+ } else {+ janas_buf_put(out, iv->key, iv->key_n);+ janas_buf_puts(out, ":");+ g_arg(out, iv, 1);+ }+ }+ janas_buf_puts(out, "}");+ }+ const struct janas_json *nl = janas_json_get(v, "nullable");+ if (nl && nl->type == JANAS_JSON_TRUE) {+ janas_buf_puts(out, comma ? ",nullable:true" : "nullable:true");+ comma = 1;+ }+ if (g_type_is(v, "OBJECT")) {+ const struct janas_json *pp = janas_json_get(v, "properties");+ janas_buf_puts(out, comma ? ",properties:{" : "properties:{");+ comma = 1;+ if (pp && pp->type == JANAS_JSON_OBJECT)+ g_params(out, pp, 0);+ else+ g_params(out, v, 1);+ janas_buf_puts(out, "}");+ const struct janas_json *rq = janas_json_get(v, "required");+ if (rq && rq->n) {+ janas_buf_puts(out, ",");+ g_required(out, rq);+ }+ }+ janas_buf_puts(out, comma ? ",type:" GQ : "type:" GQ);+ g_upper(out, janas_json_get(v, "type"));+ janas_buf_puts(out, GQ "}");+ }+}++/* The template's format_function_declaration, trimmed. */+static void g_function(struct janas_buf *out, const struct janas_json *f)+{+ const struct janas_json *nm = janas_json_get(f, "name");+ const struct janas_json *ds = janas_json_get(f, "description");+ const struct janas_json *pa = janas_json_get(f, "parameters");+ janas_buf_puts(out, "declaration:");+ if (nm)+ janas_buf_put(out, nm->s, nm->n);+ janas_buf_puts(out, "{description:" GQ);+ if (ds && ds->type == JANAS_JSON_STRING)+ janas_buf_put(out, ds->s, ds->n);+ janas_buf_puts(out, GQ);+ if (pa && pa->type == JANAS_JSON_OBJECT && pa->n) {+ janas_buf_puts(out, ",parameters:{");+ const struct janas_json *pp = janas_json_get(pa, "properties");+ if (pp && pp->type == JANAS_JSON_OBJECT && pp->n) {+ janas_buf_puts(out, "properties:{");+ g_params(out, pp, 0);+ janas_buf_puts(out, "},");+ }+ const struct janas_json *rq = janas_json_get(pa, "required");+ if (rq && rq->n) {+ g_required(out, rq);+ janas_buf_puts(out, ",");+ }+ if (janas_json_get(pa, "type")) {+ janas_buf_puts(out, "type:" GQ);+ g_upper(out, janas_json_get(pa, "type"));+ janas_buf_puts(out, GQ "}");+ }+ }+ janas_buf_puts(out, "}");+}++void janas_tools_gemma_response(struct janas_buf *out, const char *name,+ size_t name_n, const char *text, size_t n)+{+ /* a message's content is text, JSON or not: the template quotes it */+ janas_buf_puts(out, "response:");+ janas_buf_put(out, name, name_n);+ janas_buf_puts(out, "{value:" GQ);+ janas_buf_put(out, text, n);+ janas_buf_puts(out, GQ "}<tool_response|>");+}++/* Gemma's notation to JSON, one value from *p; 0, or -1 when it is not. */+static int g_to_json(const char **p, const char *end, struct janas_buf *out)+{+ const char *s = *p;+ size_t ql = strlen(GQ);+ while (s < end && (*s == ' ' || *s == '\n'))+ s++;+ if (s >= end)+ return -1;+ if ((size_t)(end - s) >= ql && !memcmp(s, GQ, ql)) {+ const char *b = s + ql, *e = b;+ while (e + ql <= end && memcmp(e, GQ, ql))+ e++;+ if (e + ql > end)+ return -1;+ janas_json_write_str(out, b, (size_t)(e - b));+ *p = e + ql;+ return 0;+ }+ if (*s == '{' || *s == '[') {+ char close = *s == '{' ? '}' : ']';+ int obj = *s == '{';+ janas_buf_put(out, s, 1);+ s++;+ for (int first = 1;; first = 0) {+ while (s < end && (*s == ' ' || *s == '\n'))+ s++;+ if (s < end && *s == close)+ break;+ if (!first) {+ if (s >= end || *s != ',')+ return -1;+ janas_buf_puts(out, ",");+ s++;+ }+ if (obj) { /* a key, bare or quoted, then ':' */+ while (s < end && *s == ' ')+ s++;+ const char *k = s;+ if ((size_t)(end - s) >= ql && !memcmp(s, GQ, ql)) {+ k = s += ql;+ while (s + ql <= end && memcmp(s, GQ, ql))+ s++;+ janas_json_write_str(out, k, (size_t)(s - k));+ s += ql;+ } else {+ while (s < end && *s != ':')+ s++;+ janas_json_write_str(out, k, (size_t)(s - k));+ }+ if (s >= end || *s != ':')+ return -1;+ janas_buf_puts(out, ":");+ s++;+ }+ if (g_to_json(&s, end, out))+ return -1;+ }+ janas_buf_put(out, &close, 1);+ *p = s + 1;+ return 0;+ }+ const char *b = s;+ while (s < end && *s != ',' && *s != '}' && *s != ']')+ s++;+ size_t n = (size_t)(s - b);+ while (n && b[n - 1] == ' ')+ n--;+ if (!n)+ return -1;+ int bare = (n == 4 && (!memcmp(b, "true", 4) || !memcmp(b, "null", 4))) ||+ (n == 5 && !memcmp(b, "false", 5));+ char *x = NULL;+ char tmp[64];+ if (!bare && n < sizeof(tmp)) {+ memcpy(tmp, b, n);+ tmp[n] = 0;+ strtod(tmp, &x);+ }+ if (bare || (x && *x == 0))+ janas_buf_put(out, b, n);+ else /* a bare word: taken as a string */+ janas_json_write_str(out, b, n);+ *p = s;+ return 0;+}+/* One function, as Qwen3-Coder's template describes it. */static void coder_function(struct janas_buf *out, const struct janas_json *f){@@ -263,6 +619,18 @@ void janas_tools_system(struct janas_buf *out, enum janas_tool_dialect d,}break;}+ case JANAS_TOOLS_GEMMA: {+ size_t n = sys_n;+ const char *s = sys ? trim(sys, &n) : NULL;+ if (s && n)+ janas_buf_put(out, s, n);+ for (uint32_t i = 0; i < t->n; i++) {+ janas_buf_puts(out, "<|tool>");+ g_function(out, t->fn[i]);+ janas_buf_puts(out, "<tool|>");+ }+ break;+ }case JANAS_TOOLS_XML_CODER:if (sys)janas_buf_put(out, sys, sys_n);@@ -284,6 +652,30 @@ void janas_tools_system(struct janas_buf *out, enum janas_tool_dialect d,}}+static uint32_t g_value_rule(struct janas_gbuild *b, int32_t quote);++/* The rule of a Gemma call: call:name{members} then the closing token. */+uint32_t janas_tools_rule_gemma(struct janas_gbuild *b,+ const struct janas_toolset *t, int only,+ int32_t close, int32_t quote)+{+ uint32_t fns = janas_gb_rule(b);+ for (uint32_t i = 0; i < t->n; i++) {+ if (only >= 0 && (uint32_t)only != i)+ continue;+ const struct janas_json *nm = janas_json_get(t->fn[i], "name");+ janas_gb_lit(b, janas_gb_alt(b, fns), nm->s, nm->n);+ }+ uint32_t val = g_value_rule(b, quote);+ uint32_t r = janas_gb_rule(b);+ uint32_t a = janas_gb_alt(b, r);+ janas_gb_lit(b, a, "call:", 5);+ janas_gb_ref(b, a, fns);+ janas_gb_ref(b, a, val);+ janas_gb_token(b, a, close);+ return r;+}+/* ---- calls of earlier turns ---- *//* The arguments of an XML call, one <parameter=...> block per key. */@@ -308,6 +700,31 @@ void janas_tools_assistant(struct janas_buf *out, enum janas_tool_dialect d,{size_t tn = n;const char *tc = content ? trim(content, &tn) : "";+ if (d == JANAS_TOOLS_GEMMA) { /* the calls first, then the text */+ for (size_t i = 0; i < n_calls; i++) {+ janas_buf_puts(out, "<|tool_call>call:");+ janas_buf_put(out, calls[i].name, calls[i].name_n);+ struct janas_json_doc *doc =+ janas_json_parse(calls[i].args, calls[i].args_n, NULL, 0);+ const struct janas_json *r = janas_json_root(doc);+ const struct janas_json *k[256];+ size_t m =+ r && r->type == JANAS_JSON_OBJECT ? sorted(r, k, 256) : 0;+ janas_buf_puts(out, "{");+ for (size_t x = 0; x < m; x++) {+ if (x)+ janas_buf_puts(out, ",");+ janas_buf_put(out, k[x]->key, k[x]->key_n);+ janas_buf_puts(out, ":");+ g_arg(out, k[x], 0);+ }+ janas_buf_puts(out, "}<tool_call|>");+ janas_json_free(doc);+ }+ if (content)+ janas_buf_put(out, content, n);+ return;+ }if (d == JANAS_TOOLS_XML_CODER) {if (tn) {janas_buf_puts(out, "\n");@@ -442,6 +859,21 @@ int janas_tools_parse(enum janas_tool_dialect d, const struct janas_toolset *t,const char *body, size_t n, struct janas_buf *name,struct janas_buf *args){+ if (d == JANAS_TOOLS_GEMMA) { /* call:name{...} */+ const char *s = body, *end = body + n;+ while (s < end && (*s == ' ' || *s == '\n'))+ s++;+ if ((size_t)(end - s) < 5 || memcmp(s, "call:", 5))+ return -1;+ s += 5;+ const char *nm = s;+ while (s < end && *s != '{')+ s++;+ if (s == end || janas_toolset_find(t, nm, (size_t)(s - nm)) < 0)+ return -1;+ janas_buf_put(name, nm, (size_t)(s - nm));+ return g_to_json(&s, end, args);+ }if (d != JANAS_TOOLS_JSON)return parse_xml(t, body, n, name, args);struct janas_json_doc *doc = janas_json_parse(body, n, NULL, 0);@@ -524,6 +956,83 @@ static uint32_t xml_params_rule(struct janas_gbuild *b, struct janas_jsg *j,return first;}+/*+ * Gemma's notation, any value: strings between quote tokens, numbers and+ * words, objects of bare keys, lists. Not held to the function's schema -+ * the name is - but always a value the parser reads.+ */+static uint32_t g_value_rule(struct janas_gbuild *b, int32_t quote)+{+ uint8_t any[32], key[32], word[32];+ memset(any, 0xff, sizeof(any));+ memset(key, 0, sizeof(key));+ memset(word, 0, sizeof(word));+ for (int c = 0; c < 256; c++) {+ int k = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') ||+ (c >= '0' && c <= '9') || c == '_';+ int w = k || c == '-' || c == '+' || c == '.';+ if (k)+ key[c / 8] |= (uint8_t)(1 << (c % 8));+ if (w)+ word[c / 8] |= (uint8_t)(1 << (c % 8));+ }+ uint32_t text = janas_gb_rule(b); /* any bytes */+ janas_gb_alt(b, text);+ uint32_t a = janas_gb_alt(b, text);+ janas_gb_set(b, a, any);+ janas_gb_ref(b, a, text);+ uint32_t keyr = janas_gb_rule(b), wordr = janas_gb_rule(b);+ a = janas_gb_alt(b, keyr);+ janas_gb_set(b, a, key);+ a = janas_gb_alt(b, keyr);+ janas_gb_set(b, a, key);+ janas_gb_ref(b, a, keyr);+ a = janas_gb_alt(b, wordr);+ janas_gb_set(b, a, word);+ a = janas_gb_alt(b, wordr);+ janas_gb_set(b, a, word);+ janas_gb_ref(b, a, wordr);+ uint32_t val = janas_gb_rule(b), members = janas_gb_rule(b),+ elems = janas_gb_rule(b);+ a = janas_gb_alt(b, val); /* a string */+ janas_gb_token(b, a, quote);+ janas_gb_ref(b, a, text);+ janas_gb_token(b, a, quote);+ a = janas_gb_alt(b, val); /* a number or a word */+ janas_gb_ref(b, a, wordr);+ a = janas_gb_alt(b, val); /* an object */+ janas_gb_byte(b, a, '{');+ janas_gb_byte(b, a, '}');+ a = janas_gb_alt(b, val);+ janas_gb_byte(b, a, '{');+ janas_gb_ref(b, a, members);+ janas_gb_byte(b, a, '}');+ a = janas_gb_alt(b, val); /* a list */+ janas_gb_byte(b, a, '[');+ janas_gb_byte(b, a, ']');+ a = janas_gb_alt(b, val);+ janas_gb_byte(b, a, '[');+ janas_gb_ref(b, a, elems);+ janas_gb_byte(b, a, ']');+ for (int more = 0; more < 2; more++) {+ a = janas_gb_alt(b, members);+ janas_gb_ref(b, a, keyr);+ janas_gb_byte(b, a, ':');+ janas_gb_ref(b, a, val);+ if (more) {+ janas_gb_byte(b, a, ',');+ janas_gb_ref(b, a, members);+ }+ a = janas_gb_alt(b, elems);+ janas_gb_ref(b, a, val);+ if (more) {+ janas_gb_byte(b, a, ',');+ janas_gb_ref(b, a, elems);+ }+ }+ return val;+}+uint32_t janas_tools_rule(struct janas_gbuild *b, struct janas_jsg *j,enum janas_tool_dialect d,const struct janas_toolset *t, int only,diff --git a/src/llm/tools.h b/src/llm/tools.hindex 95a57d8..3a49455 100644--- a/src/llm/tools.h+++ b/src/llm/tools.h@@ -31,6 +31,11 @@ enum janas_tool_dialect {JANAS_TOOLS_JSON,JANAS_TOOLS_XML,JANAS_TOOLS_XML_CODER,+ /* Gemma 4: <|tool>declaration:f{...}<tool|> in the system turn, calls+ <|tool_call>call:f{key:value,...}<tool_call|>, strings between <|"|>+ tokens, answers <|tool_response>response:f{value:...}<tool_response|>+ inside the model's own turn */+ JANAS_TOOLS_GEMMA,};/* The dialect of a chat template (NONE when it has no tools). */@@ -80,6 +85,19 @@ int janas_tools_parse(enum janas_tool_dialect d, const struct janas_toolset *t,* functions (only < 0) or function `only`, with arguments valid against its* parameters.*/+/* Gemma 4's call rule: the quote token <|"|> too. */+uint32_t janas_tools_rule_gemma(struct janas_gbuild *b,+ const struct janas_toolset *t, int only,+ int32_t close, int32_t quote);++/* Gemma 4: a tool's answer as its template writes it, without the opening+ <|tool_response> (the model's stop token, already in the sequence, or+ written by the caller): response:name{value:<|"|>text<|"|>}<tool_response|>+ - the text as it is, JSON or not, as the template quotes a message's+ content. */+void janas_tools_gemma_response(struct janas_buf *out, const char *name,+ size_t name_n, const char *text, size_t n);+uint32_t janas_tools_rule(struct janas_gbuild *b, struct janas_jsg *j,enum janas_tool_dialect d,const struct janas_toolset *t, int only,diff --git a/tests/test_attention.c b/tests/test_attention.cindex aca35e9..7aad910 100644--- a/tests/test_attention.c+++ b/tests/test_attention.c@@ -155,6 +155,99 @@ static void run(struct janas_pool *pool, uint32_t n_head, uint32_t n_kv,janas_attn_destroy(a);}+/*+ * Gemma 4's settings: keys and values of sixteen bits, a sliding window and+ * a score scale of 1, against a double-precision reference that sees the+ * same positions. Not fed to the checksum, which stays that of the+ * eight-bit runs.+ */+static void run16(struct janas_pool *pool, uint32_t n_head, uint32_t n_kv,+ uint32_t hd, uint32_t n_ctx, uint32_t pos0, uint32_t n,+ uint32_t window)+{+ struct janas_attn *a =+ janas_attn_create(n_head, n_kv, hd, n_ctx, janas_pool_size(pool),+ JANAS_ATTN_INT16); /* the cache forces FLOAT */+ janas_attn_set_kv16(a, 1);+ janas_attn_set_window(a, window);+ janas_attn_set_scale(a, 0.02f);+ size_t qd = (size_t)n_head * hd, kv = (size_t)n_kv * n_ctx * hd;+ int16_t *K = malloc(kv * 2), *V = malloc(kv * 2);+ size_t nsc = (size_t)n_kv * n_ctx;+ float *ks = malloc(nsc * sizeof(float)), *vs = malloc(nsc * sizeof(float));+ float *q = malloc(n * qd * sizeof(float));+ float *blk = malloc(n * qd * sizeof(float));+ float *one = malloc(qd * sizeof(float));+ for (size_t i = 0; i < kv; i++) {+ K[i] = (int16_t)(int)(rndf(-32767, 32767));+ V[i] = (int16_t)(int)(rndf(-32767, 32767));+ }+ for (size_t i = 0; i < nsc; i++) {+ ks[i] = rndf(0.001f, 0.02f) / 258.0f;+ vs[i] = rndf(0.001f, 0.02f) / 258.0f;+ }+ for (size_t i = 0; i < n * qd; i++)+ q[i] = rndf(-3, 3);+ janas_attn_run(a, pool, q, (const int8_t *)K, ks, (const int8_t *)V, vs, n,+ pos0, blk);+ double worst = 0;+ int same = 1;+ uint32_t group = n_head / n_kv;+ for (uint32_t j = 0; j < n; j++) {+ janas_attn_run(a, pool, q + j * qd, (const int8_t *)K, ks,+ (const int8_t *)V, vs, 1, pos0 + j, one);+ if (memcmp(one, blk + j * qd, qd * sizeof(float)) != 0)+ same = 0;+ uint32_t p = pos0 + j,+ lo = window && p + 1 > window ? p + 1 - window : 0;+ for (uint32_t h = 0; h < n_head; h++) {+ size_t base = (size_t)(h / group) * n_ctx * hd;+ size_t sb = (size_t)(h / group) * n_ctx;+ const float *qh = q + j * qd + (size_t)h * hd;+ double mx = -INFINITY, sum = 0, *sc = malloc((p + 1) * 8);+ double out[512] = {0};+ for (uint32_t t = lo; t <= p; t++) {+ double d = 0;+ for (uint32_t i = 0; i < hd; i++)+ d += (double)qh[i] * K[base + (size_t)t * hd + i] *+ ks[sb + t];+ sc[t] = d * 0.02;+ mx = sc[t] > mx ? sc[t] : mx;+ }+ for (uint32_t t = lo; t <= p; t++) {+ double w = exp(sc[t] - mx);+ sum += w;+ for (uint32_t i = 0; i < hd; i++)+ out[i] += w * V[base + (size_t)t * hd + i] * vs[sb + t];+ }+ for (uint32_t i = 0; i < hd; i++) {+ double e =+ fabs(out[i] / sum - blk[j * qd + (size_t)h * hd + i]);+ worst = e > worst ? e : worst;+ }+ free(sc);+ }+ }+ CHECK(same, "16-bit, window %u, pos0 %u, n %u: block differs", window, pos0,+ n);+ CHECK(janas_attn_scores_mode(a) == JANAS_ATTN_FLOAT,+ "16-bit keys with integer scores");+ CHECK(worst < 1e-5, "16-bit, window %u, pos0 %u, n %u: error %g", window,+ pos0, n, worst);+ printf("heads %2u/%u dim %3u, positions %4u..%4u, 16-bit, window %4u: "+ "error %.2e, blocks %s\n",+ n_head, n_kv, hd, pos0, pos0 + n - 1, window, worst,+ same ? "bit-identical" : "DIFFER");+ free(K);+ free(V);+ free(ks);+ free(vs);+ free(q);+ free(blk);+ free(one);+ janas_attn_destroy(a);+}+int main(void){/* a test that deadlocks must fail, not hold the suite for ever */@@ -169,6 +262,10 @@ int main(void)run(pool, 32, 4, 128, 1024, 300, 64, sc); /* qwen3moe geometry */run(pool, 12, 4, 64, 600, 511, 3, sc); /* 3 heads per group */}+ run16(pool, 16, 8, 256, 1024, 0, 64, 0);+ run16(pool, 16, 8, 256, 2048, 700, 64, 256); /* a window of 256 */+ run16(pool, 16, 1, 512, 1024, 250, 17, 0); /* Gemma 4's full layers */+ run16(pool, 16, 8, 256, 2048, 1030, 5, 1024);janas_pool_destroy(pool);if (failures) {fprintf(stderr, "%d failures\n", failures);diff --git a/tools/gguf2jns.c b/tools/gguf2jns.cindex 34724d8..23bf739 100644--- a/tools/gguf2jns.c+++ b/tools/gguf2jns.c@@ -10,6 +10,13 @@* resident region. A dense model has no experts to route: its feed-forward* becomes the single slot of the layer, written in the same way.*+ * A feed-forward whose width is not a multiple of 256 (Gemma 4's 26B: 704+ * and 2112) is widened with zeros to the next one - rows of zero bytes in+ * gate and up, a zero tail on each row of down: every weight type reads zero+ * bytes as zero weights, so the products are the same, and the engine's+ * activations, quantized 256 at a time, fit. Gate and up fused in one+ * tensor (ffn_gate_up_exps) are split into the two matrices of the slot.+ ** A model split into several GGUF files (name-00001-of-0000N.gguf) is read* from all of them: give the first. Data is copied in streaming; the model* is never held in memory.@@ -54,6 +61,16 @@ struct tensor {uint64_t base; /* the part's data start */int expert; /* one of the experts' matrices */uint64_t new_offset;+ /* a resident matrix widened with zeros: its rows, then as written */+ uint64_t rows, rb, new_rows, new_rb, new_dims[2], new_bytes;+};++/* A matrix of a slot: from which tensor, where in an expert's part of it,+ its rows and row bytes, and as written. */+struct src_matrix {+ struct tensor *t;+ uint64_t off, per; /* in an expert's bytes of t, and those bytes */+ uint64_t rows, rb, new_rows, new_rb;};struct part {@@ -162,6 +179,49 @@ static uint32_t crc_zeros(uint32_t crc, uint64_t n)return crc;}+static uint8_t *obuf; /* rows widened, before they are written */++/*+ * rows rows of rb bytes at src of t, to dst as new_rows rows of new_rb+ * bytes: each row's tail and the extra rows zero (the output is created+ * zero-filled, so only the CRC has to count them).+ */+static uint32_t copy_rows(const struct tensor *t, uint64_t src, uint64_t rows,+ uint64_t rb, uint64_t new_rows, uint64_t new_rb,+ uint64_t dst, uint32_t crc, int with_crc)+{+ if (new_rb == rb) {+ crc = copy(t, src, rows * rb, dst, crc, with_crc);+ } else {+ uint64_t per = COPY_CHUNK / new_rb ? COPY_CHUNK / new_rb : 1;+ int fd = parts[t->part].fd;+ for (uint64_t r0 = 0; r0 < rows; r0 += per) {+ uint64_t k = rows - r0 < per ? rows - r0 : per;+ if (pread(fd, buf, k * rb, (off_t)(t->base + src + r0 * rb)) !=+ (ssize_t)(k * rb))+ die("%s", "unexpected end of the GGUF file");+ memset(obuf, 0, k * new_rb);+ for (uint64_t r = 0; r < k; r++)+ memcpy(obuf + r * new_rb, buf + r * rb, rb);+ if (with_crc)+ crc = janas_crc32(crc, obuf, k * new_rb);+ if (pwrite(out_fd, obuf, k * new_rb, (off_t)(dst + r0 * new_rb)) !=+ (ssize_t)(k * new_rb))+ die("write failed: %s", strerror(errno));+ copied += k * rb;+ }+ }+ if (with_crc)+ crc = crc_zeros(crc, (new_rows - rows) * new_rb);+ return crc;+}++/* A width rounded up to a multiple of 256 (the engine's activations). */+static uint64_t wide(uint64_t n)+{+ return (n + 255) / 256 * 256;+}+static int by_place(const void *a, const void *b){const struct tensor *x = *(const struct tensor *const *)a;@@ -232,7 +292,7 @@ int main(int argc, char **argv)/* expert geometry per layer */struct janas_jns_layer_v2 *layers = calloc(n_layer, sizeof(*layers));- struct tensor *(*mats)[3] = calloc(n_layer, sizeof(*mats));+ struct src_matrix(*mats)[3] = calloc(n_layer, sizeof(*mats));if (!layers || !mats)die("%s", "out of memory");static const char *const part_name[3] = {"gate", "up", "down"};@@ -244,6 +304,13 @@ int main(int argc, char **argv): "blk.%llu.ffn_%s_exps.weight",(unsigned long long)l, part_name[p]);struct tensor *t = find(key);+ int half = 0; /* gate and up fused: this one's half of the rows */+ if (!t && !dense && p < 2) {+ snprintf(key, sizeof(key), "blk.%llu.ffn_gate_up_exps.weight",+ (unsigned long long)l);+ t = find(key);+ half = 1;+ }if (!t)die("%s missing: only models with routed experts in every ""layer, or dense ones, are supported",@@ -251,19 +318,38 @@ int main(int argc, char **argv)uint64_t ne = t->t->n_dims > 2 ? t->t->dims[2] : 1;if (ne != n_expert || t->t->bytes % n_expert ||t->t->dims[0] > UINT32_MAX || t->t->dims[1] > UINT32_MAX ||- t->t->dims[3] != 1)+ t->t->dims[3] != 1 || (half && t->t->dims[1] % 2))die("%s: unexpected shape", key);uint64_t per = t->t->bytes / n_expert;+ uint64_t rows = t->t->dims[1] / (half ? 2 : 1);+ uint64_t cols = t->t->dims[0];+ if (per % t->t->dims[1])+ die("%s: rows of unequal size", key);+ uint64_t rb = per / t->t->dims[1];+ /* widened to a multiple of 256: the rows of gate and up, the+ row of down */+ uint64_t new_rows = p < 2 ? wide(rows) : rows;+ uint64_t new_cols = p < 2 ? cols : wide(cols);+ if (new_cols != cols && (rb * new_cols) % cols)+ die("%s: cannot widen this type", key);+ uint64_t new_rb = rb * new_cols / cols;off = align_up(off, MATRIX_ALIGN);layers[l].m[p] =(struct janas_jns_matrix){.type = t->t->type,- .rows = (uint32_t)t->t->dims[1],- .cols = (uint32_t)t->t->dims[0],+ .rows = (uint32_t)new_rows,+ .cols = (uint32_t)new_cols,.offset = off,- .bytes = per};- off += per;+ .bytes = new_rows * new_rb};+ off += new_rows * new_rb;t->expert = 1;- mats[l][p] = t;+ mats[l][p] =+ (struct src_matrix){.t = t,+ .off = p == 1 && half ? rows * rb : 0,+ .per = per,+ .rows = rows,+ .rb = rb,+ .new_rows = new_rows,+ .new_rb = new_rb};}layers[l].slot_bytes = align_up(off, ALIGN);}@@ -283,6 +369,34 @@ int main(int argc, char **argv)resident[n_res++] = &tensors[i];}qsort(resident, n_res, sizeof(*resident), by_place);+ /* a dense feed-forward beside the experts (Gemma 4's 26B), widened as+ theirs */+ for (size_t i = 0; i < n_res; i++) {+ struct tensor *t = resident[i];+ char *name = str_dup(&t->t->name);+ size_t nl = strlen(name);+ int gu = nl > 16 && (!strcmp(name + nl - 16, ".ffn_gate.weight") ||+ !strcmp(name + nl - 14, ".ffn_up.weight"));+ int dn = nl > 16 && !strcmp(name + nl - 16, ".ffn_down.weight");+ free(name);+ t->new_bytes = t->t->bytes;+ t->new_dims[0] = t->t->dims[0];+ t->new_dims[1] = t->t->dims[1];+ if (t->t->n_dims != 2 || !(gu || dn))+ continue;+ uint64_t rows = t->t->dims[1], cols = t->t->dims[0];+ if (t->t->bytes % rows)+ die("%s", "a feed-forward matrix with rows of unequal size");+ t->rows = rows;+ t->rb = t->t->bytes / rows;+ t->new_rows = gu ? wide(rows) : rows;+ t->new_dims[1] = t->new_rows;+ t->new_dims[0] = gu ? cols : wide(cols);+ if ((t->rb * t->new_dims[0]) % cols)+ die("%s", "a feed-forward matrix whose type cannot be widened");+ t->new_rb = t->rb * t->new_dims[0] / cols;+ t->new_bytes = t->new_rows * t->new_rb;+ }/* layout */uint64_t metadata_bytes = 8 + (g0->kv_end - g0->kv_start);@@ -298,7 +412,7 @@ int main(int argc, char **argv)uint64_t pos = resident_offset;for (size_t i = 0; i < n_res; i++) {resident[i]->new_offset = pos;- pos = align_up(pos + resident[i]->t->bytes, ALIGN);+ pos = align_up(pos + resident[i]->new_bytes, ALIGN);}uint64_t resident_bytes = pos - resident_offset;uint64_t experts_offset = pos;@@ -329,12 +443,18 @@ int main(int argc, char **argv)if (out_fd < 0 || ftruncate(out_fd, (off_t)file_bytes) != 0)die("%s: cannot write", dst);buf = malloc(COPY_CHUNK);- if (!buf)+ obuf = malloc(COPY_CHUNK * 2);+ if (!buf || !obuf)die("%s", "out of memory");- for (size_t i = 0; i < n_res; i++)- copy(resident[i], resident[i]->t->offset, resident[i]->t->bytes,- resident[i]->new_offset, 0, 0);+ for (size_t i = 0; i < n_res; i++) {+ const struct tensor *t = resident[i];+ if (t->new_bytes != t->t->bytes)+ copy_rows(t, t->t->offset, t->rows, t->rb, t->new_rows, t->new_rb,+ t->new_offset, 0, 0);+ else+ copy(t, t->t->offset, t->t->bytes, t->new_offset, 0, 0);+ }for (uint64_t l = 0; l < n_layer; l++) {for (uint64_t e = 0; e < n_expert; e++) {@@ -345,8 +465,10 @@ int main(int argc, char **argv)const struct janas_jns_matrix *m = &layers[l].m[p];if (m->offset > end) /* padding inside the slot is zero */crc = crc_zeros(crc, m->offset - end);- crc = copy(mats[l][p], mats[l][p]->t->offset + e * m->bytes,- m->bytes, slot + m->offset, crc, 1);+ const struct src_matrix *s = &mats[l][p];+ crc = copy_rows(s->t, s->t->t->offset + e * s->per + s->off,+ s->rows, s->rb, s->new_rows, s->new_rb,+ slot + m->offset, crc, 1);end = m->offset + m->bytes;}crc = crc_zeros(crc, layers[l].slot_bytes - end);@@ -379,8 +501,10 @@ int main(int argc, char **argv)d.type = t->type;d.n_dims = t->n_dims;memcpy(d.dims, t->dims, sizeof(d.dims));+ d.dims[0] = resident[i]->new_dims[0];+ d.dims[1] = resident[i]->new_dims[1];d.offset = resident[i]->new_offset;- d.bytes = t->bytes;+ d.bytes = resident[i]->new_bytes;memcpy(tbl + (tensor_dir - layer_table) + i * sizeof(d), &d, sizeof(d));}uint8_t *meta = malloc(metadata_bytes);@@ -425,6 +549,7 @@ int main(int argc, char **argv)free(meta);free(tbl);free(buf);+ free(obuf);free(slot_off);free(sums);free(resident);diff --git a/tools/janas-try.sh b/tools/janas-try.shindex 040165d..9916634 100755--- a/tools/janas-try.sh+++ b/tools/janas-try.sh@@ -172,10 +172,13 @@ machine()grep -qi microsoft /proc/version 2>/dev/null && DISTRO="$DISTRO (WSL)"# the GPU: whether the build can have it (the shaders need# glslangValidator) and what Vulkan sees - name, kind, version- if command -v glslangValidator >/dev/null; then- GPU_BUILD="with GPU support"- else+ if ! command -v glslangValidator >/dev/null; thenGPU_BUILD="without GPU support (glslangValidator not installed)"+ elif ! echo '#include <vulkan/vulkan.h>' |+ ${CC:-gcc} -E -x c - >/dev/null 2>&1; then+ GPU_BUILD="without GPU support (Vulkan headers not installed)"+ else+ GPU_BUILD="with GPU support"fiif command -v vulkaninfo >/dev/null; thenGPU_VK=$(vulkaninfo --summary 2>/dev/null | awk '