Skip to content

armgit / janas

Public Imported from GitHub
0

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Release 2026-09-27: Gemma 4 (12B, 26B, E2B, E4B), why a build has no GPU

camauri86771d821d3f
diff --git a/ChangeLog.md b/ChangeLog.md
index 98c0868..05189b7 100644
--- a/ChangeLog.md
+++ b/ChangeLog.md
@@ -2,6 +2,98 @@
> Curated, user-facing summary of completed work, newest first.
+## [2026-09-27] - Why a build has no GPU
+
+- **A build without GPU support says what was missing.** The GPU needs
+ both `glslangValidator` and the Vulkan headers when Janas is built; with
+ the first and not the second (CachyOS without `vulkan-headers`) the build
+ left the GPU out without a word, and `janas-try.sh` reported "built with
+ GPU support". Now `build.sh` prints which piece is missing, the engine's
+ "GPU: none usable" names it, `janas-try.sh` checks for the headers too,
+ and the README gives the Arch packages.
+
+## [2026-09-27] - Gemma 4: E2B and E4B
+
+- **Gemma-4-E2B-it and E4B-it run and chat**, tools included. Each token
+ brings a few numbers for every layer (per-layer embeddings): a row of a
+ table of their own plus a projection of the embedding (bf16), which open
+ a branch of the residual after each feed-forward. The table, half of
+ E2B's file, is never loaded: a token's row, a few kilobytes, is read from
+ the file when it is needed, so E2B runs in 1.5 GB of memory from a 3.1 GB
+ file, E4B in 3.1 GB from 5.0 GB.
+- **The last layers share keys and values**: E2B's last 20 layers and
+ E4B's last 18 compute queries only, and read the cache of the last
+ earlier layer of their kind. E2B's feed-forward widens from 6,144 to
+ 12,288 halfway; the engine takes a width per layer.
+- Against llama.cpp, 200 tokens of text: E2B agrees on the most likely
+ token at 191 positions, E4B at 195; prompt blocks give the same bits as
+ token by token.
+- **Without reasoning, E2B and E4B are no longer handed an empty thought
+ channel**: their template does not write one (the 12B's and 26B's do),
+ and given one they reasoned inside the reply. The chat template is now
+ read whole: Gemma 4's is 19 KB, and the part after 16 KB was lost.
+
+## [2026-09-27] - Gemma 4: tools
+
+- **Gemma 4 calls tools**, in its own notation: the functions declared in
+ the system turn (`<|tool>declaration:name{...}<tool|>`, parameters written
+ as its template writes them, keys sorted), calls `call:name{key:value}`
+ read into JSON, held by a grammar to a declared name and a well-formed
+ value, and the tools' answers written inside the model's own turn, which
+ it then continues. Checked token for token against the template rendered
+ with Jinja: the first prompt with its declarations, a conversation loaded
+ with a call and its answer (as `janas-server` gives it), and the answer
+ appended live. Gemma-4-12B-it called a weather function with the right
+ argument and used its answer.
+
+## [2026-09-27] - Gemma 4: the 26B mixture, a sliding context, rings
+
+- **Gemma-4-26B-A4B-it runs and chats**: 128 experts (eight a token)
+ beside a dense feed-forward, the two as branches of their own - each
+ normalized on the way in and out, then summed - a router reading its own
+ input, and a scale per expert, folded into its weight. `gguf2jns` splits
+ gate and up, which the file keeps fused, and widens with zeros the
+ feed-forwards that are not a multiple of 256 wide (704 and 2,112 here):
+ every weight type reads zero bytes as zero weights, so the products are
+ unchanged. Files that need no widening convert to the same bytes as
+ before.
+- **The sliding-window layers keep a ring** of the positions they can see
+ - the window and a prefill block, 1,536 for Gemma 4 - instead of the whole
+ context, and attention skips the chunks before the window without reading
+ them. The logits are the same to the bit with and without the ring
+ (`JANAS_SWA_RING=0`), 1,500 tokens past the window; kept conversations,
+ on disk too, come back with the same replies.
+- **A full context slides on Gemma 4 too**: each layer's keys turned back
+ with its own RoPE (two kinds, the full layers with their frequency
+ factors), sixteen-bit keys, the rings. Qwen models slide as before, to
+ the bit.
+
+## [2026-09-26] - Gemma 4 in the chat
+
+- **Gemma-4-12B-it chats**: Gemma's format - `<bos>`, turns opened by
+ `<|turn>` and closed by `<turn|>`, the assistant called "model" - with
+ its reasoning on or off: on, `<|think|>` heads the system turn and the
+ model writes its thought channel, shown as reasoning; off, the channel is
+ written empty, as the template does. The prompts were checked token for
+ token against the model's own template rendered with Jinja and tokenized
+ by llama.cpp, with and without reasoning. No tools yet, and a full
+ context stops the chat instead of sliding.
+
+## [2026-09-26] - Gemma 4: keys and values at sixteen bits
+
+- **Gemma 4 keeps its keys and values at sixteen bits** (integers with the
+ same scale per position and head, some 250 times finer than eight bits);
+ the attention reads them through kernels of their own, the scores over
+ the float query. Qwen models stay at eight bits, bit for bit as before;
+ `JANAS_KV=8` or `16` overrides the choice. Against llama.cpp on 200
+ tokens of prose: 160 most likely tokens in agreement and a mean logit
+ difference of 0.84, against 143 and 1.53 at eight bits.
+- **A trap in the reference**: llama.cpp's logits of Gemma 4 depend on its
+ batch size - with batches of 64 tokens or more, the early positions see
+ later tokens. The comparisons above use batches of 8, which agree with a
+ prompt of two tokens run alone. The earlier figures (175 of 200) were
+ measured against the larger batches.
+
## [2026-09-26] - Why there is no GPU
- **"GPU: none usable" now says why**: built without GPU support
diff --git a/README.md b/README.md
index 4ffb4fa..a0d70d9 100644
--- a/README.md
+++ b/README.md
@@ -74,13 +74,31 @@ report from `janas-bench` is the single most useful thing you can send back**
the one the engine was shaped around, and the one whose numbers are quoted
below. The others have been converted, checked against a reference and used,
but far less.
-- **In progress, not yet usable: Gemma 4** (`gemma4`). Gemma-4-12B-it
- converts, its tokenizer gives llama.cpp's tokens on a mixed text, and
- its forward pass runs; but the chat does not know Gemma's format yet, and
- with the engine's eight-bit KV cache its logits drift from llama.cpp's
- (the most likely token agrees at 175 of 200 positions; llama.cpp itself
- agrees at 169 when its own cache goes from f16 to eight bits). A finer
- cache for Gemma comes first.
+- **Gemma 4** (`gemma4`): **Gemma-4-12B-it** (dense) and
+ **Gemma-4-26B-A4B-it** (a mixture of 128 experts beside a dense
+ feed-forward) convert and chat, with their reasoning on or off
+ (`--think`); the prompts are, token for token, the ones the model's own
+ chat template writes, and a full context slides as it does for the other
+ models. Its sliding-window layers keep only the last positions they can
+ see, so a long context costs mostly the full layers: for the 12B about 16
+ KB a token and half a gigabyte fixed, instead of 175 KB a token. Keys and
+ values are kept at sixteen bits (Gemma is far more sensitive to them than
+ Qwen). Tools work in Gemma's own notation - declarations, calls and the
+ tools' answers inside the model's turn, token for token as its template
+ writes them. The small **Gemma-4-E2B-it** and **E4B-it** run too, with
+ chat and tools: their per-layer embeddings - a table as large as half of
+ E2B's file - are read from the file a row at a time and never loaded, so
+ E2B ran here in 1.5 GB of memory (a 3.1 GB file) and E4B in 3.1 GB (5.0
+ GB). Against llama.cpp on 200 tokens of prose the most likely token
+ agrees at 160 positions on the 12B (mean logit difference 0.84) and at
+ 145 on the 26B (1.43); on another 200-token text at 191 on E2B (0.45) and
+ 195 on E4B (0.28). The larger two turn small numerical differences into
+ large ones, and llama.cpp is not a steady reference for them: against
+ itself with an eight-bit cache it agrees at 153 positions on the 12B, and
+ with a 32-bit cache and no flash attention, at 112. The
+ reference has to be computed in batches of at most 32 tokens: with larger
+ ones llama.cpp's logits for the early positions change (it looks ahead),
+ for a reason not yet found.
- **Reasoning models:** the reasoning is shown apart, in grey, and can be turned off.
- **`janas-chat`:** a terminal chat with a line editor of its own, history,
colours, a status line and a context that slides instead of ending.
@@ -175,6 +193,7 @@ the machine to itself.
```sh
sudo apt install build-essential # Debian/Ubuntu
sudo apt install libvulkan-dev glslang-tools vulkan-tools # optional, GPU
+sudo pacman -S vulkan-headers glslang vulkan-tools # the same, Arch
./build.sh # everything into bin/x86_64-linux/
./build.sh release test # and run the tests
diff --git a/build.sh b/build.sh
index 365c54a..1830248 100755
--- a/build.sh
+++ b/build.sh
@@ -177,9 +177,17 @@ esac
mkdir -p "$OBJ" "$BIN"
# GPU shaders (src/*/shaders/*.comp) as SPIR-V in C arrays; without
-# glslangValidator the library is built without GPU support
+# glslangValidator, or without the Vulkan headers, the library is built
+# without GPU support - and says which is missing, here and at run time
SPV="$OBJ/spv"
mkdir -p "$SPV"
+GPU_MISSING=""
+command -v glslangValidator >/dev/null 2>&1 ||
+ GPU_MISSING="glslangValidator (glslang-tools, glslang)"
+echo '#include <vulkan/vulkan.h>' | "$CC" -E -x c - >/dev/null 2>&1 ||
+ GPU_MISSING="${GPU_MISSING:+$GPU_MISSING and }the Vulkan headers (libvulkan-dev, vulkan-headers)"
+[ -z "$GPU_MISSING" ] ||
+ echo "note: building without GPU support: $GPU_MISSING not found"
if command -v glslangValidator >/dev/null 2>&1; then
for comp in src/*/shaders/*.comp; do
[ -f "$comp" ] || continue
diff --git a/src/llm/api.c b/src/llm/api.c
index edc066b..292f0f9 100644
--- a/src/llm/api.c
+++ b/src/llm/api.c
@@ -92,17 +92,57 @@ static uint64_t file_size(const char *path)
return stat(path, &st) == 0 ? (uint64_t)st.st_size : 0;
}
-/* Recognizes the chat format; returns 0 or an error code. */
-static int32_t find_format(janas_llm *llm, const struct janas_jns *j)
+/* Recognizes the chat format from the template in tmpl; returns 0 or an
+ error code. */
+static int32_t find_format_in(janas_llm *llm, const struct janas_jns *j,
+ const char *tmpl)
{
- char tmpl[16384];
- if (janas_jns_meta_str(j, "tokenizer.chat_template", tmpl, sizeof(tmpl)) !=
- 0)
- tmpl[0] = 0;
llm->im_start = janas_tokenizer_find(llm->tok, "<|im_start|>");
llm->im_end = janas_tokenizer_find(llm->tok, "<|im_end|>");
- if (llm->im_start >= 0 && llm->im_end >= 0 &&
- (tmpl[0] == 0 || strstr(tmpl, "<|im_start|>"))) {
+ llm->asst_role = "assistant";
+ llm->bos = llm->think_on = -1;
+ llm->quote = llm->decl_open = llm->decl_close = -1;
+ int32_t turn_open = janas_tokenizer_find(llm->tok, "<|turn>");
+ int32_t turn_close = janas_tokenizer_find(llm->tok, "<turn|>");
+ if (turn_open >= 0 && turn_close >= 0 && strstr(tmpl, "<|turn>")) {
+ /* Gemma 4: no tools yet (its calls have a syntax of their own) */
+ llm->format = FORMAT_GEMMA;
+ llm->im_start = turn_open;
+ llm->im_end = turn_close;
+ llm->asst_role = "model";
+ llm->bos = janas_tokenizer_find(llm->tok, "<bos>");
+ llm->stop[llm->n_stop++] = turn_close;
+ int32_t eos = janas_tokenizer_find(llm->tok, "<eos>");
+ if (eos >= 0)
+ llm->stop[llm->n_stop++] = eos;
+ llm->think_open = janas_tokenizer_find(llm->tok, "<|channel>");
+ llm->think_close = janas_tokenizer_find(llm->tok, "<channel|>");
+ llm->think_on = janas_tokenizer_find(llm->tok, "<|think|>");
+ llm->thinker = strstr(tmpl, "enable_thinking") &&
+ llm->think_open >= 0 && llm->think_close >= 0 &&
+ llm->think_on >= 0;
+ /* without reasoning the 12B and 26B are given an empty channel;
+ E2B and E4B are not, and given one they reason in the reply */
+ llm->empty_thought =
+ strstr(tmpl, "'<|channel>thought\\n<channel|>'") != NULL;
+ /* tools: its own notation (tools.c); a call ends the reply at
+ <|tool_response>, which the tools' answers then follow */
+ llm->call_open = janas_tokenizer_find(llm->tok, "<|tool_call>");
+ llm->call_close = janas_tokenizer_find(llm->tok, "<tool_call|>");
+ llm->resp_open = janas_tokenizer_find(llm->tok, "<|tool_response>");
+ llm->resp_close = janas_tokenizer_find(llm->tok, "<tool_response|>");
+ llm->quote = janas_tokenizer_find(llm->tok, "<|\"|>");
+ llm->decl_open = janas_tokenizer_find(llm->tok, "<|tool>");
+ llm->decl_close = janas_tokenizer_find(llm->tok, "<tool|>");
+ llm->tools = janas_tools_dialect(tmpl);
+ if (llm->call_open < 0 || llm->call_close < 0 || llm->resp_open < 0 ||
+ llm->resp_close < 0 || llm->quote < 0 || llm->decl_open < 0 ||
+ llm->decl_close < 0)
+ llm->tools = JANAS_TOOLS_NONE;
+ if (llm->tools != JANAS_TOOLS_NONE)
+ llm->stop[llm->n_stop++] = llm->resp_open;
+ } else if (llm->im_start >= 0 && llm->im_end >= 0 &&
+ (tmpl[0] == 0 || strstr(tmpl, "<|im_start|>"))) {
llm->format = FORMAT_CHATML;
llm->stop[llm->n_stop++] = llm->im_end;
llm->think_open = janas_tokenizer_find(llm->tok, "<think>");
@@ -136,7 +176,7 @@ static int32_t find_format(janas_llm *llm, const struct janas_jns *j)
if (janas_jns_meta_u32(j, "tokenizer.ggml.eos_token_id", &eos) == 0)
llm->eos_id = (int32_t)eos;
if (janas_jns_meta_u32(j, "tokenizer.ggml.eos_token_id", &eos) == 0 &&
- (int32_t)eos != llm->stop[0])
+ !janas_api_is_stop(llm, (int32_t)eos))
llm->stop[llm->n_stop++] = (int32_t)eos;
int32_t eot = janas_tokenizer_find(llm->tok, "<|endoftext|>");
if (eot >= 0 && eot != llm->stop[0] &&
@@ -145,6 +185,20 @@ static int32_t find_format(janas_llm *llm, const struct janas_jns *j)
return 0;
}
+/* The same, reading the template (Gemma 4's: 19 KB) */
+static int32_t find_format(janas_llm *llm, const struct janas_jns *j)
+{
+ size_t cap = (size_t)1 << 17;
+ char *tmpl = malloc(cap);
+ if (!tmpl)
+ return janas_api_fail(JANAS_LLM_ENOMEM, "out of memory");
+ if (janas_jns_meta_str(j, "tokenizer.chat_template", tmpl, cap) != 0)
+ tmpl[0] = 0;
+ int32_t rc = find_format_in(llm, j, tmpl);
+ free(tmpl);
+ return rc;
+}
+
int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,
janas_llm **out)
{
@@ -180,8 +234,11 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,
janas_jns_meta_str(&j, "general.name", name, sizeof(name));
janas_jns_meta_str(&j, "general.architecture", arch, sizeof(arch));
int32_t rc = 0;
- if (strcmp(kind, "gpt2") != 0 ||
- (strcmp(pre, "qwen2") != 0 && strcmp(pre, "qwen35") != 0))
+ /* byte-level BPE with Qwen's pre-tokenizers, or Gemma 4's */
+ int known = (strcmp(kind, "gpt2") == 0 &&
+ (strcmp(pre, "qwen2") == 0 || strcmp(pre, "qwen35") == 0)) ||
+ strcmp(kind, "gemma4") == 0;
+ if (!known)
rc = janas_api_fail(JANAS_LLM_EMODEL, "tokenizer %s/%s not supported",
kind, pre);
else if (!(llm->tok = janas_tokenizer_create(&j, err, sizeof(err))))
@@ -189,6 +246,12 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,
else
rc = find_format(llm, &j);
uint64_t resident = j.h.resident_bytes;
+ /* less Gemma 4's per-layer embeddings, read from the file a row at a
+ time (model_ple.c) */
+ const struct janas_jns_tensor *ple =
+ janas_jns_tensor(&j, "per_layer_token_embd.weight");
+ if (ple && ple->bytes < resident)
+ resident -= ple->bytes;
/* what the cache would need to hold every slot at once. Not the size of
the expert region: the cache's slots are all as big as the largest, so
where the layers differ (a down matrix in Q6_K in one and Q4_K in the
@@ -236,22 +299,50 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,
/* Gemma 4: sliding-window layers with a head size of their own, and an
attention (its partial results) for each kind of layer */
const uint8_t *pat = NULL;
- uint32_t len_swa = 0, kv_swa = 0;
+ uint32_t len_swa = 0, kv_swa = 0, window = 0;
+ /* a sliding-window layer keeps a ring of positions (model_load.c): a
+ fixed amount, counted apart from the tokens */
+ uint64_t ring_bytes = 0;
+ uint32_t ring = 0;
+ snprintf(key, sizeof(key), "%.32s.attention.sliding_window", arch);
+ if (janas_jns_meta_u32(&j, key, &window) == 0 && window) {
+ const char *pb = getenv("JANAS_PREFILL_BLOCK"),
+ *rg = getenv("JANAS_SWA_RING");
+ long blk = pb ? atol(pb) : 256;
+ ring = janas_attn_ring_positions(window, blk < 64 ? 64
+ : blk > 4096 ? 4096
+ : (uint32_t)blk);
+ if (ring >= p.n_ctx || (rg && atoi(rg) == 0))
+ ring = 0;
+ }
snprintf(key, sizeof(key), "%.32s.attention.sliding_window_pattern", arch);
if (janas_jns_meta_bytes(&j, key, &n_pat, &pat) != 0 || n_pat != n_layer)
pat = NULL;
snprintf(key, sizeof(key), "%.32s.attention.key_length_swa", arch);
janas_jns_meta_u32(&j, key, &len_swa);
+ /* E2B and E4B: one count of KV heads for both kinds, and the last
+ shared_kv_layers layers reading earlier layers' keys and values */
+ uint32_t shared = 0;
+ snprintf(key, sizeof(key), "%.32s.attention.shared_kv_layers", arch);
+ janas_jns_meta_u32(&j, key, &shared);
snprintf(key, sizeof(key), "%.32s.attention.head_count_kv", arch);
- if (janas_jns_meta_ints(&j, key, &n_arr, &per_layer) == 0 && per_layer &&
- n_arr == n_layer) {
+ int arr = janas_jns_meta_ints(&j, key, &n_arr, &per_layer) == 0 &&
+ per_layer && n_arr == n_layer;
+ if (arr || (pat && kv_heads)) {
+ uint32_t one = kv_heads;
kv_token = 0;
- for (uint64_t i = 0; i < n_arr; i++) {
- int32_t h;
- memcpy(&h, per_layer + i, sizeof(h));
+ for (uint64_t i = 0; i + shared < n_layer; i++) {
+ int32_t h = (int32_t)one;
+ if (arr)
+ memcpy(&h, per_layer + i, sizeof(h));
uint32_t len = pat && pat[i] && len_swa ? len_swa : kv_len;
- if (h > 0)
- kv_token += (uint64_t)h * (len * 2 + 8);
+ /* keys and values of one byte a number, or two (Gemma 4) */
+ uint64_t per =
+ h > 0 ? (uint64_t)h * (len * (a && a->kv16 ? 4 : 2) + 8) : 0;
+ if (pat && pat[i] && ring)
+ ring_bytes += per * ring;
+ else
+ kv_token += per;
if (h > 0 && pat && pat[i])
kv_swa = (uint32_t)h;
else if (h > 0 && pat)
@@ -271,7 +362,9 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,
? janas_attn_part_bytes(n_head, kv_heads, kv_len, p.n_ctx)
: 0;
if (pat && n_head && kv_swa && len_swa)
- attn_part += janas_attn_part_bytes(n_head, kv_swa, len_swa, p.n_ctx);
+ attn_part += janas_attn_part_bytes(n_head, kv_swa, len_swa,
+ ring ? ring : p.n_ctx) +
+ ring_bytes;
/* and the scratch of a pass, which grows with the prefill block
(janas_llm_model_max_block): per token, the rows of its experts (gate,
up, their product, down) and a few rows of the model's width. An
@@ -282,7 +375,16 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,
snprintf(key, sizeof(key), "%.32s.expert_feed_forward_length", arch);
if (janas_jns_meta_u32(&j, key, &ff) != 0) {
snprintf(key, sizeof(key), "%.32s.feed_forward_length", arch);
- janas_jns_meta_u32(&j, key, &ff);
+ const int32_t *ffs = NULL;
+ uint64_t n_ff = 0;
+ if (janas_jns_meta_u32(&j, key, &ff) != 0 &&
+ janas_jns_meta_ints(&j, key, &n_ff, &ffs) == 0 && ffs)
+ for (uint64_t i = 0; i < n_ff; i++) { /* E2B: the widest */
+ int32_t v;
+ memcpy(&v, ffs + i, sizeof(v));
+ if (v > 0 && (uint32_t)v > ff)
+ ff = (uint32_t)v;
+ }
}
snprintf(key, sizeof(key), "%.32s.expert_used_count", arch);
janas_jns_meta_u32(&j, key, &k_used);
@@ -406,6 +508,8 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,
llm->cache_bytes = cache;
llm->reserve_bytes = reserve;
llm->kv_token = kv_token;
+ llm->kv_ring = ring;
+ llm->kv_ring_token = ring ? ring_bytes / ring : 0;
llm->n_compute = mo.n_compute;
snprintf(llm->arch, sizeof(llm->arch), "%s", arch);
snprintf(llm->bits, sizeof(llm->bits), "%s", bits);
diff --git a/src/llm/api_chat.c b/src/llm/api_chat.c
index 07f2cc2..762a6a9 100644
--- a/src/llm/api_chat.c
+++ b/src/llm/api_chat.c
@@ -256,10 +256,35 @@ int janas_api_add_message(janas_llm_chat *c, const char *role, const char *text,
}
}
+/* The start of the sequence: Gemma's <bos>, nothing for ChatML. */
+int janas_api_begin_sequence(janas_llm_chat *c)
+{
+ return c->llm->bos >= 0 ? janas_api_add_special(c, c->llm->bos) : 0;
+}
+
int janas_api_start_reply(janas_llm_chat *c)
{
const janas_llm *llm = c->llm;
switch (llm->format) {
+ case FORMAT_GEMMA:
+ if (c->in_turn) { /* after the tools' answers: the model goes on */
+ c->in_turn = 0;
+ c->in_think = c->chan_label = 0;
+ return janas_api_flush_text(c);
+ }
+ if (janas_api_add_special(c, llm->im_start) ||
+ janas_api_add_text(c, "model\n", 6))
+ return -1;
+ /* reasoning asked for (<|think|> in the system turn): the model
+ opens its channel itself; otherwise the channel is written empty
+ where the template does (not E2B and E4B's) */
+ c->in_think = c->chan_label = 0;
+ if ((llm->thinker && c->p.thinking != 0) || !llm->empty_thought)
+ return janas_api_flush_text(c);
+ return janas_api_add_special(c, llm->think_open) ||
+ janas_api_add_text(c, "thought\n", 8) ||
+ janas_api_add_special(c, llm->think_close) ||
+ janas_api_flush_text(c);
case FORMAT_CHATML:
default:
if (janas_api_add_special(c, llm->im_start) ||
@@ -292,6 +317,7 @@ int janas_api_start_reply(janas_llm_chat *c)
int janas_api_add_markup(janas_llm_chat *c, const char *s, size_t n)
{
const janas_llm *llm = c->llm;
+ int gemma = llm->format == FORMAT_GEMMA;
const struct {
const char *text;
int32_t id;
@@ -300,7 +326,15 @@ int janas_api_add_markup(janas_llm_chat *c, const char *s, size_t n)
{"<tool_response>", llm->resp_open},
{"</tool_response>", llm->resp_close},
{"<think>", llm->think_open},
- {"</think>", llm->think_close}};
+ {"</think>", llm->think_close},
+ /* Gemma 4 */
+ {"<|tool_call>", gemma ? llm->call_open : -1},
+ {"<tool_call|>", gemma ? llm->call_close : -1},
+ {"<|tool_response>", gemma ? llm->resp_open : -1},
+ {"<tool_response|>", gemma ? llm->resp_close : -1},
+ {"<|\"|>", llm->quote},
+ {"<|tool>", llm->decl_open},
+ {"<tool|>", llm->decl_close}};
size_t from = 0;
for (size_t i = 0; i < n; i++) {
if (s[i] != '<')
@@ -326,6 +360,26 @@ int janas_api_add_system(janas_llm_chat *c, const char *sys, size_t n)
{
const janas_llm *llm = c->llm;
c->tools_tokens = 0;
+ if (llm->format == FORMAT_GEMMA) {
+ /* a system turn only when there is something to say: the message,
+ or <|think|> at its top when reasoning is asked for */
+ int think = llm->thinker && c->p.thinking != 0;
+ int tools = c->tools && llm->tools == JANAS_TOOLS_GEMMA;
+ if (!think && !tools && (!sys || n == 0))
+ return 0;
+ struct janas_buf b = {0};
+ janas_tools_system(&b, tools ? JANAS_TOOLS_GEMMA : JANAS_TOOLS_NONE,
+ c->tools, sys, n);
+ int err = b.oom || janas_api_add_special(c, llm->im_start) ||
+ janas_api_add_text(c, "system\n", 7) ||
+ (think && (janas_api_add_special(c, llm->think_on) ||
+ janas_api_add_text(c, "\n", 1))) ||
+ janas_api_add_markup(c, b.p, b.n) ||
+ janas_api_add_special(c, llm->im_end) ||
+ janas_api_add_text(c, "\n", 1);
+ janas_buf_free(&b);
+ return err ? -1 : 0;
+ }
if (!c->tools || llm->tools == JANAS_TOOLS_NONE)
return janas_api_add_message(c, "system", sys ? sys : "", n);
/* how many of its tokens the tools take: the message without them is
@@ -626,22 +680,46 @@ static int32_t send_turn(janas_llm_chat *c, const char *text, size_t n,
const int32_t *lens)
{
janas_api_prep_start(c);
+ /* Gemma: the tools answer inside the model's turn, each answer named
+ after its call, which the reply's calls say (cleared next) */
+ int gemma_answers = texts && c->llm->format == FORMAT_GEMMA &&
+ c->llm->tools == JANAS_TOOLS_GEMMA &&
+ janas_llm_session_length(c->s) > 0;
+ struct janas_buf ans = {0};
+ for (int32_t i = 0; gemma_answers && i < n_res; i++) {
+ const char *nm = i < (int32_t)c->n_calls ? c->calls[i].name : "tool";
+ if (i > 0 || !c->closed)
+ janas_buf_puts(&ans, "<|tool_response>");
+ janas_tools_gemma_response(
+ &ans, nm, strlen(nm), texts[i],
+ janas_api_text_len(texts[i], lens ? lens[i] : -1));
+ }
janas_api_reply_clear(c);
c->n_ids = c->n_text = 0;
- int err = 0;
- if (janas_llm_session_length(c->s) > 0) {
+ int err = ans.oom;
+ /* (ans is freed once written, below) */
+ if (gemma_answers) {
+ err |= janas_api_add_markup(c, ans.p, ans.n);
+ c->in_turn = 1;
+ } else if (janas_llm_session_length(c->s) > 0) {
/* close the previous reply: its stop token is in the sequence, or
it was cut and gets one now; then the newline after it */
if (!c->closed)
err |= janas_api_add_special(c, c->llm->im_end);
err |= janas_api_add_text(c, "\n", 1);
- } else if (c->system || c->tools) {
- err |= janas_api_add_system(c, c->system,
- c->system ? strlen(c->system) : 0);
+ } else {
+ /* the start: <bos> where the format has one, the system message */
+ err |= janas_api_begin_sequence(c);
+ if (c->system || c->tools || c->llm->format == FORMAT_GEMMA)
+ err |= janas_api_add_system(c, c->system,
+ c->system ? strlen(c->system) : 0);
c->sys_tokens = (uint32_t)c->n_ids;
}
err |= janas_api_note_turn(c, (uint32_t)janas_llm_session_length(c->s));
- if (texts) {
+ janas_buf_free(&ans);
+ if (gemma_answers) {
+ /* written above */
+ } else if (texts) {
err |= janas_api_add_special(c, c->llm->im_start) ||
janas_api_add_text(c, "user", 4);
for (int32_t i = 0; i < n_res; i++) {
diff --git a/src/llm/api_load.c b/src/llm/api_load.c
index 709a8fe..2ee2068 100644
--- a/src/llm/api_load.c
+++ b/src/llm/api_load.c
@@ -145,8 +145,9 @@ static int add_assistant(janas_llm_chat *c, const struct cmsg *m, int after_q)
int err = janas_api_add_special(c, llm->im_start);
/* Qwen3-Coder writes the calls right after the role */
int bare = d == JANAS_TOOLS_XML_CODER && m->n_calls;
- err |= janas_api_add_text(c, bare ? "assistant" : "assistant\n",
- bare ? 9 : 10);
+ err |= janas_api_add_text(c, llm->asst_role, strlen(llm->asst_role));
+ if (!bare)
+ err |= janas_api_add_text(c, "\n", 1);
if (d == JANAS_TOOLS_XML && llm->thinker && after_q)
err |= janas_api_add_markup(c, "<think>\n\n</think>\n\n", 19);
if (m->n_calls && d != JANAS_TOOLS_NONE) {
@@ -162,6 +163,43 @@ static int add_assistant(janas_llm_chat *c, const struct cmsg *m, int after_q)
return err ? -1 : 0;
}
+/*
+ * Gemma 4's model turn with tools: its calls, the answers of the tool
+ * messages after it (they belong to the turn), then its text; the turn
+ * stays open when the model speaks next (the next message is its own, or
+ * the reply is to follow the answers). Returns the index of the last
+ * message used; *open whether the turn was left open.
+ */
+static size_t add_gemma_turn(janas_llm_chat *c, size_t i, int reopen, int *open)
+{
+ const janas_llm *llm = c->llm;
+ const struct cmsg *m = &c->msgs[i];
+ int err = 0;
+ if (!reopen)
+ err |= janas_api_add_special(c, llm->im_start) ||
+ janas_api_add_text(c, "model\n", 6);
+ struct janas_buf b = {0};
+ janas_tools_assistant(&b, JANAS_TOOLS_GEMMA, NULL, 0, m->calls, m->n_calls);
+ size_t j = i + 1, k = 0;
+ for (; j < c->n_msgs && c->msgs[j].role == JANAS_LLM_ROLE_TOOL; j++, k++) {
+ const char *nm = k < m->n_calls ? m->calls[k].name : "tool";
+ size_t nn = k < m->n_calls ? m->calls[k].name_n : 4;
+ janas_buf_puts(&b, "<|tool_response>");
+ janas_tools_gemma_response(&b, nm, nn, c->msgs[j].text, c->msgs[j].n);
+ }
+ if (m->text)
+ janas_buf_put(&b, m->text, m->n);
+ err |= b.oom || janas_api_add_markup(c, b.p, b.n);
+ janas_buf_free(&b);
+ int next_model =
+ j < c->n_msgs && c->msgs[j].role == JANAS_LLM_ROLE_ASSISTANT;
+ *open = next_model || (j == c->n_msgs && k > 0);
+ if (!*open)
+ err |= janas_api_add_special(c, llm->im_end) ||
+ janas_api_add_text(c, "\n", 1);
+ return err ? (size_t)-1 : j - 1;
+}
+
/* The messages built, in the model's format, into c->ids; the reply's turn
started. *reply_off gets where it starts. */
static int render(janas_llm_chat *c, uint32_t *reply_off)
@@ -171,13 +209,17 @@ static int render(janas_llm_chat *c, uint32_t *reply_off)
for (size_t i = 0; i < c->n_msgs; i++)
if (c->msgs[i].role == JANAS_LLM_ROLE_USER)
last_q = i;
- int err = 0;
+ int err = janas_api_begin_sequence(c);
+ if (llm->format == FORMAT_GEMMA && !c->tools &&
+ c->msgs[0].role != JANAS_LLM_ROLE_SYSTEM)
+ err |= janas_api_add_system(c, NULL, 0); /* <|think|>, if asked */
if (c->tools && c->msgs[0].role != JANAS_LLM_ROLE_SYSTEM) {
/* with tools there is a system message whatever the caller says */
err |= janas_api_add_system(c, NULL, 0);
err |= janas_api_flush_text(c);
c->sys_tokens = (uint32_t)c->n_ids;
}
+ int gemma_open = 0; /* Gemma: the model's turn left open */
for (size_t i = 0; i < c->n_msgs && !err; i++) {
const struct cmsg *m = &c->msgs[i];
switch (m->role) {
@@ -193,6 +235,19 @@ static int render(janas_llm_chat *c, uint32_t *reply_off)
err |= janas_api_add_message(c, "user", m->text, m->n);
break;
case JANAS_LLM_ROLE_ASSISTANT: {
+ if (llm->tools == JANAS_TOOLS_GEMMA && (m->n_calls || gemma_open)) {
+ int open = 0;
+ size_t last = add_gemma_turn(c, i, gemma_open, &open);
+ if (last == (size_t)-1) {
+ err = 1;
+ break;
+ }
+ i = last;
+ gemma_open = open;
+ if (open && i + 1 == c->n_msgs)
+ c->in_turn = 1; /* the reply follows the answers */
+ break;
+ }
const struct memo *mm = memo_find(c, m);
if (mm) { /* one of the last replies: as the model wrote it */
err |= add_ids(c, mm->ids, mm->n);
diff --git a/src/llm/api_reply.c b/src/llm/api_reply.c
index 87bfc15..0143805 100644
--- a/src/llm/api_reply.c
+++ b/src/llm/api_reply.c
@@ -121,7 +121,11 @@ void janas_api_update_stats(janas_llm_chat *c)
s->input_cpu_seconds = c->input_cpu;
s->input_threads = janas_llm_model_threads(c->llm->m, 2);
s->output_threads = janas_llm_model_threads(c->llm->m, 0);
- s->context_bytes = c->llm->kv_token * s->context_used;
+ s->context_bytes =
+ c->llm->kv_token * s->context_used +
+ c->llm->kv_ring_token * (s->context_used < c->llm->kv_ring
+ ? s->context_used
+ : c->llm->kv_ring);
struct rusage ru;
if (getrusage(RUSAGE_SELF, &ru) == 0)
s->peak_rss_bytes = (uint64_t)ru.ru_maxrss * 1024;
@@ -194,9 +198,22 @@ static void take(janas_llm_chat *c, int32_t t)
c->answer_tokens++; /* text or a call: what max_answer counts */
if (t == llm->think_open || t == llm->think_close) {
c->in_think = t == llm->think_open;
+ /* Gemma names its channel before the reasoning: "thought\n" */
+ c->chan_label = c->in_think && llm->format == FORMAT_GEMMA;
janas_api_record(c, t, JANAS_LLM_PART_MARK, "", 0);
return;
}
+ if (c->chan_label) {
+ char lab[32];
+ size_t ln = janas_tokenizer_decode(llm->tok, t, lab, sizeof(lab));
+ if (ln < sizeof(lab) && ((ln == 7 && !memcmp(lab, "thought", 7)) ||
+ (ln == 1 && *lab == '\n'))) {
+ c->chan_label = lab[0] != '\n';
+ janas_api_record(c, t, JANAS_LLM_PART_MARK, "", 0);
+ return;
+ }
+ c->chan_label = 0;
+ }
if (c->tools && t == llm->call_open && !c->in_think && !c->in_call) {
c->in_call = 1;
c->body.n = 0;
diff --git a/src/llm/api_tools.c b/src/llm/api_tools.c
index efbf2a2..bb7c75f 100644
--- a/src/llm/api_tools.c
+++ b/src/llm/api_tools.c
@@ -178,11 +178,13 @@ int janas_api_reply_setup(janas_llm_chat *c)
return -1;
}
uint32_t call = JANAS_NO_RULE, answer = JANAS_NO_RULE;
+ int only = choice == JANAS_LLM_TOOLS_FUNCTION ? c->tool_only : -1;
if (tools && choice != JANAS_LLM_TOOLS_NONE)
- call = janas_tools_rule(
- b, j, llm->tools, c->tools,
- choice == JANAS_LLM_TOOLS_FUNCTION ? c->tool_only : -1,
- llm->call_close);
+ call = llm->tools == JANAS_TOOLS_GEMMA
+ ? janas_tools_rule_gemma(b, c->tools, only, llm->call_close,
+ llm->quote)
+ : janas_tools_rule(b, j, llm->tools, c->tools, only,
+ llm->call_close);
if (must) {
answer = calls_rule(b, llm, call, c->parallel);
} else if (c->format != JANAS_LLM_FORMAT_TEXT) {
diff --git a/src/llm/arch.c b/src/llm/arch.c
index be7c9bb..1e4beb0 100644
--- a/src/llm/arch.c
+++ b/src/llm/arch.c
@@ -29,11 +29,15 @@ static const struct janas_arch archs[] = {
.post_attn_norm = 1,
.vmap_mod = 1,
.nextn_in_file = 1},
+ /* its scores are not scaled down, and an eight-bit key moves them: at
+ eight bits llama.cpp agrees with its own f16 cache at 169 of 200 most
+ likely tokens */
{.name = "gemma4",
.rec = JANAS_REC_NONE,
.gelu = 1,
.sandwich = 1,
- .swa = 1},
+ .swa = 1,
+ .kv16 = 1},
};
const struct janas_arch *janas_arch_find(const char *name, unsigned len)
diff --git a/src/llm/arch.h b/src/llm/arch.h
index 3878d1d..8965b35 100644
--- a/src/llm/arch.h
+++ b/src/llm/arch.h
@@ -35,6 +35,7 @@ struct janas_arch {
its head size, KV heads and RoPE; V normalized without
weights, V = K where attn_v is missing, scores not
scaled (the q and k norms do it) */
+ int kv16; /* keys and values kept at sixteen bits, not eight */
};
/* The architecture called name, or NULL if Janas does not know it. */
diff --git a/src/llm/attention.c b/src/llm/attention.c
index 489390b..416ec0f 100644
--- a/src/llm/attention.c
+++ b/src/llm/attention.c
@@ -40,6 +40,8 @@ struct janas_attn {
int scores; /* JANAS_ATTN_FLOAT or JANAS_ATTN_INT16, resolved */
int scores_auto; /* 1: the engine chose it */
float scale; /* of the scores: 1 / sqrt(head_dim) unless set */
+ int kv16; /* keys and values of sixteen bits, not eight */
+ int ring; /* position p in row p mod n_ctx (with a window) */
uint32_t window; /* a token sees the last window positions; 0: all */
};
@@ -52,6 +54,7 @@ struct attn_job {
const float *ks, *vs; /* one scale per position and KV head */
float *out;
uint32_t n, pos0, n_chunks, split;
+ uint32_t c0; /* the first chunk any token of the call sees */
atomic_uint next; /* next item to claim */
};
@@ -62,7 +65,9 @@ static float *att_partial(const struct janas_attn *A, uint32_t kvh, uint32_t c,
{
uint32_t group = A->n_head / A->n_kv;
return A->part +
- ((((size_t)kvh * A->chunks + c) * JANAS_ATTN_MAX_BLOCK + j) * group +
+ ((((size_t)kvh * A->chunks + c % A->chunks) * JANAS_ATTN_MAX_BLOCK +
+ j) *
+ group +
g) *
(A->hd + 2);
}
@@ -118,6 +123,33 @@ static void att_values_ref(const float *const *w, const int8_t *V, uint32_t nt,
}
}
+/* The same over keys and values of sixteen bits. */
+static void att_scores_f32_ref16(const float *const *q, const int16_t *K,
+ const float *ks, uint32_t nt, uint32_t hd,
+ float scale, float *const *sc, int ns)
+{
+ for (uint32_t t = 0; t < nt; t++)
+ for (int s = 0; s < ns; s++) {
+ float sum = 0.0f;
+ for (uint32_t i = 0; i < hd; i++)
+ sum += q[s][i] * (float)K[(size_t)t * hd + i];
+ sc[s][t] = sum * scale * ks[t];
+ }
+}
+
+static void att_values_ref16(const float *const *w, const int16_t *V,
+ uint32_t nt, uint32_t hd, float *const *out,
+ int ns)
+{
+ for (int s = 0; s < ns; s++)
+ for (uint32_t i = 0; i < hd; i++) {
+ float acc = 0.0f;
+ for (uint32_t t = 0; t < nt; t++)
+ acc += w[s][t] * (float)V[(size_t)t * hd + i];
+ out[s][i] = acc;
+ }
+}
+
#if defined(__x86_64__)
#define AVX2 __attribute__((target("avx2,fma,f16c")))
@@ -232,6 +264,103 @@ static AVX2 void att_values_avx2(const float *const *w, const int8_t *V,
values_body(w, V, nt, hd, out, 1);
}
}
+
+/*
+ * Keys and values of sixteen bits (Gemma 4, whose unscaled scores feel an
+ * eight-bit key: llama.cpp itself, its cache taken from f16 to q8_0, agreed
+ * with itself at 169 of 200 most likely tokens). The same kernels, the rows
+ * read as int16; the scores always over the float query, since an integer
+ * sum of sixteen-bit products would not fit in 32 bits.
+ */
+static inline AVX2 __attribute__((always_inline)) void
+f32_body16(const float *const *q, const int16_t *K, const float *ks,
+ uint32_t nt, uint32_t hd, float scale, float *const *sc,
+ const int16_t *pf, const int ns)
+{
+ for (uint32_t t = 0; t < nt; t++) {
+ const int16_t *k = K + (size_t)t * hd;
+ if (pf)
+ for (uint32_t i = 0; i < hd; i += 32)
+ _mm_prefetch((const char *)(pf + (size_t)t * hd + i),
+ _MM_HINT_T1);
+ __m256 acc[8];
+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++) acc[s] =
+ _mm256_setzero_ps();
+ for (uint32_t i = 0; i < hd; i += 8) {
+ __m256 kv = _mm256_cvtepi32_ps(_mm256_cvtepi16_epi32(
+ _mm_loadu_si128((const __m128i *)(k + i))));
+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++) acc[s] =
+ _mm256_fmadd_ps(_mm256_loadu_ps(q[s] + i), kv, acc[s]);
+ }
+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++)
+ {
+ __m128 r = _mm_add_ps(_mm256_castps256_ps128(acc[s]),
+ _mm256_extractf128_ps(acc[s], 1));
+ r = _mm_add_ps(r, _mm_movehl_ps(r, r));
+ r = _mm_add_ss(r, _mm_movehdup_ps(r));
+ sc[s][t] = _mm_cvtss_f32(r) * scale * ks[t];
+ }
+ }
+}
+
+static AVX2 void att_scores_f32_avx2_16(const float *const *q, const int16_t *K,
+ const float *ks, uint32_t nt,
+ uint32_t hd, float scale,
+ float *const *sc, const int16_t *pf,
+ int ns)
+{
+ switch (ns) {
+ case 8:
+ f32_body16(q, K, ks, nt, hd, scale, sc, pf, 8);
+ break;
+ case 4:
+ f32_body16(q, K, ks, nt, hd, scale, sc, pf, 4);
+ break;
+ case 2:
+ f32_body16(q, K, ks, nt, hd, scale, sc, pf, 2);
+ break;
+ default:
+ f32_body16(q, K, ks, nt, hd, scale, sc, pf, 1);
+ }
+}
+
+static inline AVX2 __attribute__((always_inline)) void
+values_body16(const float *const *w, const int16_t *V, uint32_t nt, uint32_t hd,
+ float *const *out, const int ns)
+{
+ for (uint32_t i = 0; i < hd; i += 8) {
+ __m256 acc[8];
+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++) acc[s] =
+ _mm256_setzero_ps();
+ for (uint32_t t = 0; t < nt; t++) {
+ __m256 v = _mm256_cvtepi32_ps(_mm256_cvtepi16_epi32(
+ _mm_loadu_si128((const __m128i *)(V + (size_t)t * hd + i))));
+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++) acc[s] =
+ _mm256_fmadd_ps(_mm256_broadcast_ss(w[s] + t), v, acc[s]);
+ }
+ _Pragma("GCC unroll 8") for (int s = 0; s < ns; s++)
+ _mm256_storeu_ps(out[s] + i, acc[s]);
+ }
+}
+
+static AVX2 void att_values_avx2_16(const float *const *w, const int16_t *V,
+ uint32_t nt, uint32_t hd, float *const *out,
+ int ns)
+{
+ switch (ns) {
+ case 8:
+ values_body16(w, V, nt, hd, out, 8);
+ break;
+ case 4:
+ values_body16(w, V, nt, hd, out, 4);
+ break;
+ case 2:
+ values_body16(w, V, nt, hd, out, 2);
+ break;
+ default:
+ values_body16(w, V, nt, hd, out, 1);
+ }
+}
#endif
/*
@@ -305,7 +434,8 @@ static void attn_worker(void *arg, int tid, int n_threads)
struct janas_attn *A = a->A;
uint32_t hd = A->hd;
uint32_t group = A->n_head / A->n_kv, gs = group / a->split;
- uint32_t items = A->n_kv * a->n_chunks * a->split;
+ uint32_t span = a->n_chunks - a->c0; /* chunks c0 .. n_chunks - 1 */
+ uint32_t items = A->n_kv * span * a->split;
float scale = A->scale;
float *sc = A->scr + (size_t)tid * JANAS_ATTN_MAX_BLOCK * group * ATT_CHUNK;
#if defined(__x86_64__)
@@ -319,16 +449,20 @@ static void attn_worker(void *arg, int tid, int n_threads)
atomic_fetch_add_explicit(&a->next, 1, memory_order_relaxed);
if (it >= items)
break;
- uint32_t sub = it % a->split, c = it / a->split % a->n_chunks;
- uint32_t kvh = it / a->split / a->n_chunks, g0 = sub * gs;
+ uint32_t sub = it % a->split, c = a->c0 + it / a->split % span;
+ uint32_t kvh = it / a->split / span, g0 = sub * gs;
uint32_t t0 = c * ATT_CHUNK, t1 = t0 + ATT_CHUNK;
if (t1 > a->pos0 + a->n)
t1 = a->pos0 + a->n;
uint32_t nt = t1 - t0;
- const int8_t *K = a->k + ((size_t)kvh * A->n_ctx + t0) * hd;
- const int8_t *V = a->v + ((size_t)kvh * A->n_ctx + t0) * hd;
- const float *ks = a->ks + (size_t)kvh * A->n_ctx + t0;
- const float *vs = a->vs + (size_t)kvh * A->n_ctx + t0;
+ /* rows of one or two bytes a number; in a ring, the chunk's rows
+ are contiguous all the same (n_ctx is a multiple of a chunk) */
+ size_t r0 = (size_t)kvh * A->n_ctx + (A->ring ? t0 % A->n_ctx : t0);
+ size_t row0 = (r0 * hd) << A->kv16;
+ const int8_t *K = a->k + row0;
+ const int8_t *V = a->v + row0;
+ const float *ks = a->ks + r0;
+ const float *vs = a->vs + r0;
/* token j sees positions up to pos0 + j: tokens before jlo see
nothing of this chunk */
uint32_t jlo = t0 > a->pos0 ? t0 - a->pos0 : 0;
@@ -348,7 +482,7 @@ static void attn_worker(void *arg, int tid, int n_threads)
const int16_t *qi[8];
const float *qf[8];
float qsc[8], *srow[8];
- int fast = A->scores == JANAS_ATTN_INT16;
+ int fast = A->scores == JANAS_ATTN_INT16 && !A->kv16;
for (uint32_t s = 0; s < step; s++) {
uint32_t sl = s0 + s, j = jlo + sl / gs, g = g0 + sl % gs;
size_t h = (size_t)j * A->n_head + kvh * group + g;
@@ -361,7 +495,17 @@ static void attn_worker(void *arg, int tid, int n_threads)
srow[s] = sc + (size_t)sl * ATT_CHUNK;
}
const int8_t *pf = s0 ? NULL : V;
- if (fast) {
+ if (A->kv16) {
+#if defined(__x86_64__)
+ if (avx)
+ att_scores_f32_avx2_16(qf, (const int16_t *)K, ks, nt, hd,
+ scale, srow, (const int16_t *)pf,
+ (int)step);
+ else
+#endif
+ att_scores_f32_ref16(qf, (const int16_t *)K, ks, nt, hd,
+ scale, srow, (int)step);
+ } else if (fast) {
#if defined(__x86_64__)
if (vnni)
att_scores_i16_vnni(qi, K, ks, nt, hd, qsc, scale, srow, pf,
@@ -429,10 +573,16 @@ static void attn_worker(void *arg, int tid, int n_threads)
att_partial(A, kvh, c, jlo + sl / gs, g0 + sl % gs) + 2;
}
#if defined(__x86_64__)
- if (avx)
+ if (avx && A->kv16)
+ att_values_avx2_16(w, (const int16_t *)V, nt, hd, out,
+ (int)step);
+ else if (avx)
att_values_avx2(w, V, nt, hd, out, (int)step);
else
#endif
+ if (A->kv16)
+ att_values_ref16(w, (const int16_t *)V, nt, hd, out, (int)step);
+ else
att_values_ref(w, V, nt, hd, out, (int)step);
}
}
@@ -507,14 +657,14 @@ static void merge_worker(void *arg, int tid, int n_threads)
uint32_t kvh = h / group, g = h % group;
uint32_t nc = (a->pos0 + j + ATT_CHUNK) / ATT_CHUNK;
float mx = -INFINITY, sum = 0.0f;
- for (uint32_t c = 0; c < nc; c++) {
+ for (uint32_t c = a->c0; c < nc; c++) {
float v = att_partial(A, kvh, c, j, g)[0];
if (v > mx)
mx = v;
}
if (skip_stat > 0) {
uint64_t k[3] = {0};
- for (uint32_t c = 0; c < nc; c++) {
+ for (uint32_t c = a->c0; c < nc; c++) {
float v = att_partial(A, kvh, c, j, g)[0];
k[0] += v < mx - 8.0f;
k[1] += v < mx - 16.0f;
@@ -530,7 +680,7 @@ static void merge_worker(void *arg, int tid, int n_threads)
}
float *out = a->out + (size_t)j * qd + (size_t)h * hd;
memset(out, 0, hd * sizeof(float));
- for (uint32_t c = 0; c < nc; c++) {
+ for (uint32_t c = a->c0; c < nc; c++) {
const float *pm = att_partial(A, kvh, c, j, g);
float w = expf(pm[0] - mx);
sum += w * pm[1];
@@ -625,6 +775,26 @@ void janas_attn_set_window(struct janas_attn *a, uint32_t window)
a->window = window;
}
+uint32_t janas_attn_ring_positions(uint32_t window, uint32_t n)
+{
+ /* the window before the first token, the tokens, and a chunk more for
+ the first one's partial alignment, in whole chunks */
+ uint32_t need = window + n + ATT_CHUNK;
+ return (need + ATT_CHUNK - 1) / ATT_CHUNK * ATT_CHUNK;
+}
+
+void janas_attn_set_ring(struct janas_attn *a, int on)
+{
+ a->ring = on != 0;
+}
+
+void janas_attn_set_kv16(struct janas_attn *a, int on)
+{
+ a->kv16 = on != 0;
+ if (a->kv16 && a->scores == JANAS_ATTN_INT16)
+ a->scores = JANAS_ATTN_FLOAT; /* no integer path over 16-bit keys */
+}
+
int janas_attn_scores_mode(const struct janas_attn *a)
{
return a ? a->scores : JANAS_ATTN_FLOAT;
@@ -654,7 +824,18 @@ void janas_attn_run(struct janas_attn *A, struct janas_pool *pool,
/* enough items for every thread: split the query heads of a group */
uint32_t n_chunks = (pos0 + n + ATT_CHUNK - 1) / ATT_CHUNK, split = 1;
uint32_t group = A->n_head / A->n_kv;
- while (A->n_kv * n_chunks * split < 2 * (uint32_t)A->n_threads &&
+ /* with a window, the chunks wholly before the first token's are never
+ read: at a long context most of them */
+ uint32_t c0 = A->window && pos0 + 1 > A->window
+ ? (pos0 + 1 - A->window) / ATT_CHUNK
+ : 0;
+ if (A->ring && n_chunks - c0 > A->chunks) {
+ /* the ring is too small for this call: a caller's error */
+ fprintf(stderr, "janas_attn_run: %u chunks for a ring of %u\n",
+ n_chunks - c0, A->chunks);
+ abort();
+ }
+ while (A->n_kv * (n_chunks - c0) * split < 2 * (uint32_t)A->n_threads &&
group % (split * 2) == 0)
split *= 2;
struct attn_job job = {.A = A,
@@ -667,6 +848,7 @@ void janas_attn_run(struct janas_attn *A, struct janas_pool *pool,
.n = n,
.pos0 = pos0,
.n_chunks = n_chunks,
+ .c0 = c0,
.split = split};
job.qi = A->qi;
job.qs = A->qs;
@@ -677,7 +859,7 @@ void janas_attn_run(struct janas_attn *A, struct janas_pool *pool,
if (skip_stat)
atexit(skip_report);
}
- if (A->scores == JANAS_ATTN_INT16)
+ if (A->scores == JANAS_ATTN_INT16 && !A->kv16)
janas_pool_run(pool, quant_worker, &job);
janas_pool_run(pool, attn_worker, &job);
janas_pool_run(pool, merge_worker, &job);
diff --git a/src/llm/attention.h b/src/llm/attention.h
index 06092fa..7872075 100644
--- a/src/llm/attention.h
+++ b/src/llm/attention.h
@@ -67,6 +67,21 @@ void janas_attn_set_scale(struct janas_attn *a, float scale);
/* A sliding window: token p sees positions p - window + 1 .. p only; 0 (the
default) for all. The cache still holds every position. */
void janas_attn_set_window(struct janas_attn *a, uint32_t window);
+/* Keys and values of sixteen bits (int16 with the same scale per position
+ and KV head) instead of eight; the scores are then computed over the float
+ query (FLOAT). Twice the bytes, some 250 times finer: Gemma 4 needs it. */
+void janas_attn_set_kv16(struct janas_attn *a, int on);
+/*
+ * A ring (with a window): the cache holds n_ctx positions, n_ctx a multiple
+ * of 256, position p in row p mod n_ctx; a call must fit in it, window and
+ * the call's tokens together with room for the chunk boundaries
+ * (janas_attn_ring_positions). The partial results are kept for that many
+ * positions only.
+ */
+void janas_attn_set_ring(struct janas_attn *a, int on);
+/* The positions a ring needs for a window and calls of up to n tokens
+ whose keys and values are written before any of them is attended. */
+uint32_t janas_attn_ring_positions(uint32_t window, uint32_t n);
/* What the scores are actually computed with: FLOAT or INT16. */
int janas_attn_scores_mode(const struct janas_attn *a);
diff --git a/src/llm/chat.h b/src/llm/chat.h
index ca746b0..915d632 100644
--- a/src/llm/chat.h
+++ b/src/llm/chat.h
@@ -22,6 +22,10 @@
enum chat_format {
FORMAT_CHATML, /* <|im_start|>role\ntext<|im_end|>\n */
+ /* <bos> once, then <|turn>role\ntext<turn|>\n, the assistant called
+ "model", the reasoning in <|channel>thought\n...<channel|>:
+ ChatML's shape, so im_start and im_end hold <|turn> and <turn|> */
+ FORMAT_GEMMA,
};
struct janas_llm {
@@ -29,11 +33,16 @@ struct janas_llm {
struct janas_tokenizer *tok;
int format;
int32_t im_start, im_end;
- int32_t think_open, think_close; /* reasoning markers, or -1 */
- int thinker; /* the template knows enable_thinking */
- enum janas_tool_dialect tools; /* how its template writes tool calls */
- int32_t call_open, call_close; /* <tool_call>, </tool_call>, or -1 */
- int32_t resp_open, resp_close; /* <tool_response>, </tool_response> */
+ const char *asst_role; /* "assistant", or Gemma's "model" */
+ int32_t bos; /* written once at the start, or -1 */
+ int32_t think_on; /* Gemma's <|think|>: reasoning asked for, or -1 */
+ int32_t quote, decl_open, decl_close; /* Gemma's <|"|>, <|tool>, <tool|> */
+ int32_t think_open, think_close; /* reasoning markers, or -1 */
+ int thinker; /* the template knows enable_thinking */
+ int empty_thought; /* Gemma 4: an empty channel when not reasoning */
+ enum janas_tool_dialect tools; /* how its template writes tool calls */
+ int32_t call_open, call_close; /* <tool_call>, </tool_call>, or -1 */
+ int32_t resp_open, resp_close; /* <tool_response>, </tool_response> */
uint32_t pooling; /* embeddings: 1 mean, 2 first, 3 last; 0 none */
int add_eos; /* the tokenizer ends every text with eos_id */
int32_t eos_id;
@@ -53,6 +62,10 @@ struct janas_llm {
int32_t mode;
int gpu_wanted; /* the caller's say, on top of what the mode decides */
uint64_t kv_token; /* bytes of attention's keys and values per token */
+ /* and of the sliding-window layers' rings, per position up to ring
+ positions (Gemma 4) */
+ uint64_t kv_ring_token;
+ uint32_t kv_ring;
};
struct janas_llm_chat {
@@ -60,10 +73,13 @@ struct janas_llm_chat {
struct janas_llm_session *s;
struct janas_llm_chat_params p;
char *system;
- int replying; /* a reply is being read */
- int closed; /* the last reply ended with a stop token */
- int ended; /* the reply is over: the next call says DONE */
- int in_think; /* the reply is in its reasoning part */
+ int replying; /* a reply is being read */
+ int closed; /* the last reply ended with a stop token */
+ int ended; /* the reply is over: the next call says DONE */
+ int in_think; /* the reply is in its reasoning part */
+ int in_turn; /* Gemma: the reply continues the model's turn (after the
+ tools' answers), no new turn to open */
+ int chan_label; /* Gemma: the channel's name ("thought\n") to skip */
uint32_t reply_tokens;
uint32_t answer_tokens; /* of them, outside the reasoning */
char pend[1024]; /* reply bytes not yet returned */
@@ -217,6 +233,7 @@ int janas_api_add_message(janas_llm_chat *c, const char *role, const char *text,
size_t n);
int janas_api_add_system(janas_llm_chat *c, const char *sys, size_t n);
int janas_api_start_reply(janas_llm_chat *c);
+int janas_api_begin_sequence(janas_llm_chat *c);
int janas_api_note_turn(janas_llm_chat *c, uint32_t at);
/* The reply starts: reply_at is where its turn begins in the sequence
(UINT32_MAX for raw text), total the prompt's length, cached how much of
diff --git a/src/llm/gpu.c b/src/llm/gpu.c
index c4a98db..5313038 100644
--- a/src/llm/gpu.c
+++ b/src/llm/gpu.c
@@ -1438,7 +1438,16 @@ struct janas_gpu {
struct janas_gpu *janas_gpu_create(char *err, size_t err_len)
{
- snprintf(err, err_len, "built without GPU support");
+#if !defined(JANAS_GPU_SPV)
+ snprintf(err, err_len,
+ "built without GPU support: glslangValidator was "
+ "not installed when Janas was built");
+#else
+ snprintf(err, err_len,
+ "built without GPU support: the Vulkan headers "
+ "(vulkan/vulkan.h) were not installed when Janas "
+ "was built");
+#endif
return NULL;
}
diff --git a/src/llm/model.c b/src/llm/model.c
index b32c069..d6a9ea8 100644
--- a/src/llm/model.c
+++ b/src/llm/model.c
@@ -120,11 +120,74 @@ static void sandwich_worker(void *arg, int tid, int n_threads)
float *x = m->x + (size_t)j * dm;
const float *r = m->xres + (size_t)j * dm;
rms_norm(x, x, t->w, dm, m->eps);
+ /* with per-layer embeddings the scale comes after their branch */
+ float s = m->ple_dim ? 1.0f : t->ly->out_scale;
for (uint32_t i = 0; i < dm; i++)
- x[i] = (r[i] + x[i]) * t->ly->out_scale;
+ x[i] = (r[i] + x[i]) * s;
}
}
+/* Gemma 4's mixture, token by token: the dense branch's output normalized
+ aside (xbr), then the experts' input and the router's from the residual,
+ then the experts' output normalized and the two summed */
+static void branch1_worker(void *arg, int tid, int n_threads)
+{
+ struct tok_job *t = arg;
+ struct janas_llm_model *m = t->m;
+ uint32_t dm = m->d_model;
+ for (uint32_t j = (uint32_t)tid; j < t->n; j += (uint32_t)n_threads)
+ rms_norm(m->xbr + (size_t)j * dm, m->x + (size_t)j * dm,
+ f32(m, t->ly->post_norm1), dm, m->eps);
+}
+
+static void prep2_worker(void *arg, int tid, int n_threads)
+{
+ struct tok_job *t = arg;
+ struct janas_llm_model *m = t->m;
+ uint32_t dm = m->d_model;
+ const float *rs = t->ly->router_scale;
+ float sd = 1.0f / sqrtf((float)dm);
+ for (uint32_t j = (uint32_t)tid; j < t->n; j += (uint32_t)n_threads) {
+ const float *r = m->xres + (size_t)j * dm;
+ float *xn = m->xn + (size_t)j * dm;
+ /* the experts' input, to Q8_K through xn */
+ rms_norm(xn, r, f32(m, t->ly->pre_norm2), dm, m->eps);
+ janas_q8k_quantize(xn, m->xq + j * (dm / JANAS_QK), dm);
+ /* the router's: the residual normalized without weights, scaled
+ by 1 / sqrt(d_model) and per dimension */
+ double ss = 0.0;
+ for (uint32_t i = 0; i < dm; i++)
+ ss += (double)r[i] * r[i];
+ float sc = 1.0f / sqrtf((float)(ss / dm) + m->eps);
+ for (uint32_t i = 0; i < dm; i++)
+ xn[i] = r[i] * sc * sd * rs[i];
+ }
+}
+
+static void branch2_worker(void *arg, int tid, int n_threads)
+{
+ struct tok_job *t = arg;
+ struct janas_llm_model *m = t->m;
+ uint32_t dm = m->d_model;
+ for (uint32_t j = (uint32_t)tid; j < t->n; j += (uint32_t)n_threads) {
+ float *x = m->x + (size_t)j * dm;
+ const float *b = m->xbr + (size_t)j * dm;
+ rms_norm(x, x, f32(m, t->ly->post_norm2), dm, m->eps);
+ for (uint32_t i = 0; i < dm; i++)
+ x[i] += b[i];
+ }
+}
+
+static void run_tok(struct janas_llm_model *m, const struct layer *ly,
+ uint32_t n, void (*fn)(void *, int, int))
+{
+ struct tok_job t = {.m = m, .ly = ly, .n = n};
+ if (n < 2)
+ fn(&t, 0, 1);
+ else
+ janas_pool_run(m->compute, fn, &t);
+}
+
/* logits soft-capped: c * tanh(l / c) */
static void softcap_worker(void *arg, int tid, int n_threads)
{
@@ -137,6 +200,35 @@ static void softcap_worker(void *arg, int tid, int n_threads)
l[i] = c * tanhf(l[i] / c);
}
+/* A key row turned back by the angles in cs, sn (half pairs, NeoX) and put
+ in its bits again: a rotation can take a pair past the largest number of
+ the row, so the scale is found anew. row: int8 or int16 (wide). */
+static void unrotate_row(void *row, float *scale, uint32_t hd, uint32_t half,
+ const float *cs, const float *sn, int wide)
+{
+ float tmp[512], mx = 0.0f, z = *scale;
+ int8_t *r8 = row;
+ int16_t *r16 = row;
+ for (uint32_t i = 0; i < hd; i++)
+ tmp[i] = (wide ? (float)r16[i] : (float)r8[i]) * z;
+ for (uint32_t i = 0; i < half; i++) {
+ float a = tmp[i], b = tmp[i + half];
+ tmp[i] = a * cs[i] + b * sn[i];
+ tmp[i + half] = b * cs[i] - a * sn[i];
+ }
+ for (uint32_t i = 0; i < hd; i++)
+ if (fabsf(tmp[i]) > mx)
+ mx = fabsf(tmp[i]);
+ float top = wide ? 32767.0f : 127.0f, inv = mx > 0 ? top / mx : 0.0f;
+ for (uint32_t i = 0; i < hd; i++) {
+ if (wide)
+ r16[i] = (int16_t)lrintf(tmp[i] * inv);
+ else
+ r8[i] = (int8_t)lrintf(tmp[i] * inv);
+ }
+ *scale = mx / top;
+}
+
/*
* Forgets the positions keep .. keep + drop of the attention layers: what
* follows them moves down and its keys are turned back by drop positions,
@@ -144,66 +236,83 @@ static void softcap_worker(void *arg, int tid, int n_threads)
* state is left alone: it is a summary of everything seen, and in a hybrid
* model that is what carries the older text once the window has passed it.
* The first `keep` positions stay where they are, which keeps the system
- * message - and the anchor it gives attention - in place.
+ * message - and the anchor it gives attention - in place. Each slot with its
+ * own geometry and RoPE (Gemma 4: sliding-window layers and full ones); a
+ * ring (a sliding-window layer's) keeps only its last positions, and those
+ * are the ones that move.
*/
int janas_llm_model_shift(struct janas_llm_model *m, uint32_t keep,
uint32_t drop, uint32_t n_kv)
{
- /* (Gemma 4: two RoPEs and a window, not done yet) */
- if (!drop || n_kv > m->n_ctx || keep + drop > n_kv || m->head_dim > 512 ||
- m->a->swa)
- return -1;
- uint32_t hd = m->head_dim, half = m->n_rot / 2;
- uint32_t move = n_kv - keep - drop;
- float *cs = malloc(2 * (size_t)half * sizeof(float));
- if (!cs)
+ if (!drop || n_kv > m->n_ctx || keep + drop > n_kv || m->hd_max > 512)
return -1;
- float *sn = cs + half;
- for (uint32_t i = 0; i < half; i++) {
- float theta = (float)drop *
- powf(m->rope_base, -2.0f * (float)i / (float)m->n_rot);
- cs[i] = cosf(theta);
- sn[i] = sinf(theta);
- }
- uint32_t slots = m->n_attn + (m->mtp ? 1u : 0u);
- for (uint32_t sl = 0; sl < slots; sl++)
- for (uint32_t h = 0; h < m->n_head_kv; h++) {
- size_t base = ((size_t)sl * m->n_head_kv + h) * m->n_ctx * hd;
- size_t sbase = ((size_t)sl * m->n_head_kv + h) * m->n_ctx;
- int8_t *k = m->kcache + base, *v = m->vcache + base;
- float *kz = m->kscale + sbase, *vz = m->vscale + sbase;
- if (move) {
- memmove(k + (size_t)keep * hd, k + (size_t)(keep + drop) * hd,
- (size_t)move * hd);
- memmove(v + (size_t)keep * hd, v + (size_t)(keep + drop) * hd,
- (size_t)move * hd);
- memmove(kz + keep, kz + keep + drop, move * sizeof(float));
- memmove(vz + keep, vz + keep + drop, move * sizeof(float));
- }
- for (uint32_t p = keep; p < keep + move; p++) {
- int8_t *row = k + (size_t)p * hd;
- /* turned back and put in eight bits again: a rotation can
- take a pair past the largest number of the head, so the
- scale is found anew */
- float tmp[512], mx = 0.0f;
- for (uint32_t i = 0; i < half; i++) {
- float a = (float)row[i] * kz[p];
- float b = (float)row[i + half] * kz[p];
- tmp[i] = a * cs[i] + b * sn[i];
- tmp[i + half] = b * cs[i] - a * sn[i];
+ uint32_t move = n_kv - keep - drop, slots = m->n_attn + (m->mtp ? 1u : 0u);
+ size_t elt = (size_t)1 << m->kv16;
+ float cs[256], sn[256];
+ uint8_t *ring_tmp = NULL;
+ for (uint32_t sl = 0; sl < slots; sl++) {
+ uint32_t hd = m->kv_hd[sl], cap = m->kv_cap[sl];
+ int swa = m->kv_swa[sl];
+ uint32_t n_rot = swa ? m->n_rot_swa : m->n_rot, half = n_rot / 2;
+ float base = swa ? m->rope_base_swa : m->rope_base;
+ const float *ff = swa ? NULL : m->rope_freqs;
+ for (uint32_t i = 0; i < half; i++) {
+ float theta =
+ (float)drop * powf(base, -2.0f * (float)i / (float)n_rot);
+ if (ff)
+ theta /= ff[i];
+ cs[i] = cosf(theta);
+ sn[i] = sinf(theta);
+ }
+ size_t row = hd * elt;
+ for (uint32_t h = 0; h < m->kv_heads[sl]; h++) {
+ uint8_t *k = (uint8_t *)m->kcache + m->kv_off[sl] + h * cap * row;
+ uint8_t *v = (uint8_t *)m->vcache + m->kv_off[sl] + h * cap * row;
+ float *kz = m->kscale + m->ks_off[sl] + (size_t)h * cap;
+ float *vz = m->vscale + m->ks_off[sl] + (size_t)h * cap;
+ if (!m->kv_ring[sl]) {
+ if (move) {
+ memmove(k + keep * row, k + (keep + drop) * row,
+ move * row);
+ memmove(v + keep * row, v + (keep + drop) * row,
+ move * row);
+ memmove(kz + keep, kz + keep + drop, move * sizeof(float));
+ memmove(vz + keep, vz + keep + drop, move * sizeof(float));
}
- for (uint32_t i = 2 * half; i < hd; i++)
- tmp[i] = (float)row[i] * kz[p];
- for (uint32_t i = 0; i < hd; i++)
- if (fabsf(tmp[i]) > mx)
- mx = fabsf(tmp[i]);
- float inv = mx > 0 ? 127.0f / mx : 0.0f;
- for (uint32_t i = 0; i < hd; i++)
- row[i] = (int8_t)lrintf(tmp[i] * inv);
- kz[p] = mx / 127.0f;
+ for (uint32_t p = keep; p < keep + move; p++)
+ unrotate_row(k + p * row, kz + p, hd, half, cs, sn,
+ m->kv16);
+ continue;
+ }
+ /* a ring: the positions it still has, from a copy of it, each
+ to its new row (they cannot land on a kept one: see the
+ bound on lo) */
+ if (!ring_tmp &&
+ !(ring_tmp = malloc(2 * (size_t)cap * (row + sizeof(float)))))
+ return -1;
+ uint8_t *tk = ring_tmp, *tv = tk + cap * row;
+ float *tkz = (float *)(void *)(tv + cap * row), *tvz = tkz + cap;
+ memcpy(tk, k, cap * row);
+ memcpy(tv, v, cap * row);
+ memcpy(tkz, kz, cap * sizeof(float));
+ memcpy(tvz, vz, cap * sizeof(float));
+ uint32_t lo = keep + drop;
+ if (n_kv > cap && n_kv - cap > lo)
+ lo = n_kv - cap;
+ for (uint32_t p = lo; p < n_kv; p++) {
+ uint32_t src = p % cap, dst = (p - drop) % cap;
+ memcpy(k + dst * row, tk + src * row, row);
+ memcpy(v + dst * row, tv + src * row, row);
+ kz[dst] = tkz[src];
+ vz[dst] = tvz[src];
+ unrotate_row(k + dst * row, kz + dst, hd, half, cs, sn,
+ m->kv16);
}
}
- free(cs);
+ }
+ free(ring_tmp);
+ if (m->ring_hi > keep + drop)
+ m->ring_hi -= drop;
/* the recurrent state holds what it has seen, dropped tokens included -
that is what carries the older text - but the position it stands at
moves with everything else */
@@ -284,6 +393,8 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,
fprintf(stderr, "\n");
trace_row(m, "embed", m->x + (size_t)(n - 1) * dm, dm);
}
+ if (m->ple_dim && janas_m_ple_inputs(m, tokens, n) != 0)
+ return -1;
set_rope(m, n, pos0);
for (uint32_t l = 0; l < m->n_layer; l++) {
const struct layer *ly = &m->layers[l];
@@ -301,8 +412,26 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,
memcpy(m->xres, m->x, (size_t)n * dm * sizeof(float));
memset(m->x, 0, (size_t)n * dm * sizeof(float));
}
- if (janas_m_moe_block(m, ly, m->cache, &m->j.layers[l], l, n) != 0)
+ if (ly->pre_norm2) {
+ /* Gemma 4's mixture: the dense branch on xn / xq as normed
+ above, then the experts' branch */
+ size_t bytes = (size_t)n * dm * sizeof(float);
+ if (janas_m_moe_block(m, ly, m->cache, &m->j.layers[l], l, n,
+ MOE_SHARED) != 0)
+ return -1;
+ run_tok(m, ly, n, branch1_worker);
+ trace_row(m, "ffn_mlp", m->xbr + (size_t)(n - 1) * dm, dm);
+ run_tok(m, ly, n, prep2_worker);
+ memset(m->x, 0, bytes);
+ if (janas_m_moe_block(m, ly, m->cache, &m->j.layers[l], l, n,
+ MOE_ROUTED) != 0)
+ return -1;
+ run_tok(m, ly, n, branch2_worker);
+ trace_row(m, "ffn_moe_comb", m->x + (size_t)(n - 1) * dm, dm);
+ } else if (janas_m_moe_block(m, ly, m->cache, &m->j.layers[l], l, n,
+ MOE_ALL) != 0) {
return -1;
+ }
if (ly->post_ffn) {
struct tok_job st = {
.m = m, .ly = ly, .w = f32(m, ly->post_ffn), .n = n};
@@ -311,6 +440,8 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,
else
janas_pool_run(m->compute, sandwich_worker, &st);
}
+ if (m->ple_dim)
+ janas_m_ple_layer(m, ly, l, n);
if (m->trace) {
/* JANAS_TRACE=1: the state after each layer, last token of the
block, to compare with another implementation */
@@ -322,6 +453,7 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,
l, sqrt(nn), v[0], v[1], v[2], v[3]);
}
}
+ m->ring_hi = pos0 + n; /* the rings hold up to here */
double td = now();
uint32_t first = all_logits ? 0 : n - 1;
/* with an MTP block, every token's final hidden state is kept for it */
@@ -511,7 +643,8 @@ int janas_llm_mtp_forward(struct janas_llm_model *m, const int32_t *tokens,
f32(m, ly->ffn_norm), dm, m->eps);
janas_q8k_quantize(m->xn + (size_t)j * dm, m->xq + j * nbd, dm);
}
- if (janas_m_moe_block(m, ly, m->mtp_cache, m->mtp_jl, m->mtp_cl, n) != 0)
+ if (janas_m_moe_block(m, ly, m->mtp_cache, m->mtp_jl, m->mtp_cl, n,
+ MOE_ALL) != 0)
return -1;
double td = now();
uint32_t first = all_logits ? 0 : n - 1;
diff --git a/src/llm/model_attn.c b/src/llm/model_attn.c
index 1812496..809d994 100644
--- a/src/llm/model_attn.c
+++ b/src/llm/model_attn.c
@@ -98,8 +98,9 @@ static void qkv_item(const struct tok_job *t, uint32_t j, uint32_t h)
float *kv = m->kk + (size_t)j * kvd + (size_t)h * hd;
rms_norm(kv, kv, f32(m, ly->k_norm), hd, m->eps);
rope_neox(kv, ly->n_rot, cs, sn);
- size_t at = ((size_t)h * m->n_ctx + t->pos0 + j) * hd;
- size_t as = (size_t)h * m->n_ctx + t->pos0 + j;
+ uint32_t p = t->pos0 + j, row = t->ring ? p % t->cap : p;
+ size_t at = ((size_t)h * t->cap + row) * hd;
+ size_t as = (size_t)h * t->cap + row;
float *vr = m->vv + (size_t)j * kvd + (size_t)h * hd;
if (m->a->swa) { /* Gemma 4: V normalized, without weights */
double ss = 0.0;
@@ -120,6 +121,18 @@ static void qkv_item(const struct tok_job *t, uint32_t j, uint32_t h)
if (b > mv)
mv = b;
}
+ if (m->kv16) { /* sixteen bits: the largest becomes 32767 */
+ t->kc_scale[as] = mk / 32767.0f;
+ t->vc_scale[as] = mv / 32767.0f;
+ float ik = mk > 0 ? 32767.0f / mk : 0.0f;
+ float iv = mv > 0 ? 32767.0f / mv : 0.0f;
+ int16_t *kc = (int16_t *)(void *)t->kc, *vc = (int16_t *)(void *)t->vc;
+ for (uint32_t i = 0; i < hd; i++) {
+ kc[at + i] = (int16_t)lrintf(kv[i] * ik);
+ vc[at + i] = (int16_t)lrintf(vr[i] * iv);
+ }
+ return;
+ }
t->kc_scale[as] = mk / 127.0f;
t->vc_scale[as] = mv / 127.0f;
float ik = mk > 0 ? 127.0f / mk : 0.0f;
@@ -133,7 +146,8 @@ static void qkv_item(const struct tok_job *t, uint32_t j, uint32_t h)
static void qkv_worker(void *arg, int tid, int n_threads)
{
struct tok_job *t = arg;
- uint32_t nh = t->m->n_head + t->ly->n_kv;
+ /* a layer without keys and values of its own: its queries alone */
+ uint32_t nh = t->m->n_head + (t->ly->wk ? t->ly->n_kv : 0);
for (uint32_t i = (uint32_t)tid; i < t->n * nh; i += (uint32_t)n_threads)
qkv_item(t, i / nh, i % nh);
}
@@ -212,6 +226,8 @@ void janas_m_attn_layer(struct janas_llm_model *m, const struct layer *ly,
double ta = now();
/* qwen3next: each query head is followed by its output gate */
float *qout = m->a->attn_gate ? m->big : m->q;
+ /* no k: the keys and values of an earlier layer, already cached */
+ int nt = !ly->wk ? 1 : ly->wv ? 3 : 2;
struct janas_matvec_task t[3] = {{.type = (int)ly->wq->type,
.w = data(m, ly->wq),
.x = m->xq,
@@ -219,8 +235,8 @@ void janas_m_attn_layer(struct janas_llm_model *m, const struct layer *ly,
.rows = m->a->attn_gate ? 2 * qd : qd,
.cols = dm,
.n_vec = n},
- {.type = (int)ly->wk->type,
- .w = data(m, ly->wk),
+ {.type = ly->wk ? (int)ly->wk->type : 0,
+ .w = ly->wk ? data(m, ly->wk) : NULL,
.x = m->xq,
.y = m->kk,
.rows = kvd,
@@ -234,8 +250,8 @@ void janas_m_attn_layer(struct janas_llm_model *m, const struct layer *ly,
.cols = dm,
.n_vec = n}};
janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,
- m->compute, t, ly->wv ? 3 : 2);
- if (!ly->wv) /* Gemma 4's full layers: V is K, before K's norm */
+ m->compute, t, (size_t)nt);
+ if (nt == 2) /* Gemma 4's full layers: V is K, before K's norm */
memcpy(m->vv, m->kk, (size_t)n * kvd * sizeof(float));
int8_t *kc = m->kcache + m->kv_off[ly->slot];
int8_t *vc = m->vcache + m->kv_off[ly->slot];
@@ -248,7 +264,9 @@ void janas_m_attn_layer(struct janas_llm_model *m, const struct layer *ly,
.kc_scale = kcs,
.vc_scale = vcs,
.kc = kc,
- .vc = vc};
+ .vc = vc,
+ .cap = m->kv_cap[ly->slot],
+ .ring = m->kv_ring[ly->slot]};
janas_pool_run(m->compute, qkv_worker, &tj);
/* the last token's last head of each, as llama.cpp's graph names them */
trace_row(m, "Qcur_pos", m->q + (size_t)(n - 1) * qd + (m->n_head - 1) * hd,
diff --git a/src/llm/model_ffn.c b/src/llm/model_ffn.c
index 8dd7a7e..9995ef9 100644
--- a/src/llm/model_ffn.c
+++ b/src/llm/model_ffn.c
@@ -99,11 +99,12 @@ static void act_worker(void *arg, int tid, int n_threads)
{
struct act_job *a = arg;
struct janas_llm_model *m = a->m;
- uint32_t ff = m->d_ff;
+ uint32_t stride = m->ff_buf;
for (uint32_t i = (uint32_t)tid; i < a->n; i += (uint32_t)n_threads) {
size_t g = m->glist[i];
- const float *gt = m->gate + g * ff, *u = m->up + g * ff;
- float *h = m->h + g * ff;
+ uint32_t ff = m->hlen[g];
+ const float *gt = m->gate + g * stride, *u = m->up + g * stride;
+ float *h = m->h + g * stride;
/* deterministic operations only, so that a GPU computing an
expert gives the same bits */
uint32_t r = 0;
@@ -122,7 +123,7 @@ static void act_worker(void *arg, int tid, int n_threads)
for (; r < ff; r++)
h[r] = janas_silu_mul_det(gt[r], u[r]);
}
- janas_q8k_quantize_det(h, m->hq + g * ff / JANAS_QK, ff);
+ janas_q8k_quantize_det(h, m->hq + g * stride / JANAS_QK, ff);
}
}
@@ -170,8 +171,8 @@ static void router_worker(void *arg, int tid, int n_threads)
uint32_t ne = m->n_expert, dm = m->d_model;
for (uint32_t e = (uint32_t)tid; e < ne; e += (uint32_t)n_threads)
for (uint32_t j = 0; j < r->n; j++) {
- const uint16_t *w = r->w + (size_t)e * dm;
const float *x = m->xn + (size_t)j * dm;
+ const uint16_t *w = r->w + (size_t)e * dm;
#if defined(__x86_64__)
if (m->avx2 && m->f16c) {
m->router[(size_t)j * ne + e] = dot_f16_avx2(w, x, dm);
@@ -211,7 +212,7 @@ static void run_experts(struct janas_llm_model *m,
const struct expert_run *runs, uint32_t n_runs)
{
double t_enter = janas_m_exp_tracing() ? now() : 0;
- uint32_t dm = m->d_model, ff = m->d_ff;
+ uint32_t dm = m->d_model, ff = m->ff_buf;
size_t nbd = dm / JANAS_QK, nbf = ff / JANAS_QK;
struct janas_matvec_task t1[2 * MAX_RUNS], t2[MAX_RUNS];
uint32_t ng = 0;
@@ -222,6 +223,7 @@ static void run_experts(struct janas_llm_model *m,
m->xq + (size_t)m->pair_tok[m->order[g]] * nbd,
nbd * sizeof(*m->xg));
m->glist[ng++] = g;
+ m->hlen[g] = runs[r].mx[0].rows;
}
for (int p = 0; p < 2; p++) {
const struct janas_jns_matrix *mx = &runs[r].mx[p];
@@ -326,19 +328,27 @@ static void axpy_worker(void *arg, int tid, int n_threads)
are layer l of the file behind the cache (the main model or the MTP). */
int janas_m_moe_block(struct janas_llm_model *m, const struct layer *ly,
struct janas_expert_cache *cache,
- const struct janas_jns_layer *jl, uint32_t l, uint32_t n)
+ const struct janas_jns_layer *jl, uint32_t l, uint32_t n,
+ int part)
{
double t0 = now();
- uint32_t dm = m->d_model, ne = m->n_expert, k = m->k_use;
- struct router_job rj = {m, ly->router16, n};
- janas_pool_run(m->compute, router_worker, &rj);
+ uint32_t dm = m->d_model, ne = m->n_expert;
+ uint32_t k = part == MOE_SHARED ? 0 : m->k_use;
+ if (k) {
+ struct router_job rj = {m, ly->router16, n};
+ janas_pool_run(m->compute, router_worker, &rj);
- /* per token: softmax, top-k, normalized weights */
- struct tok_job tj = {.m = m, .n = n};
- if (n < 2)
- route_tok(m, 0);
- else
- janas_pool_run(m->compute, route_worker, &tj);
+ /* per token: softmax, top-k, normalized weights */
+ struct tok_job tj = {.m = m, .n = n};
+ if (n < 2)
+ route_tok(m, 0);
+ else
+ janas_pool_run(m->compute, route_worker, &tj);
+ /* an expert's own output scale, folded into its weight (Gemma 4) */
+ if (ly->exp_scale)
+ for (uint32_t p = 0; p < n * k; p++)
+ m->pair_w[p] *= ly->exp_scale[m->pair_exp[p]];
+ }
/* counting sort of the pairs by expert */
uint32_t np = n * k;
memset(m->count, 0, (ne + 1) * sizeof(uint32_t));
@@ -359,11 +369,14 @@ int janas_m_moe_block(struct janas_llm_model *m, const struct layer *ly,
}
/* the shared expert: one more pair per token, after the routed ones */
uint32_t nt = np;
- if (m->a->shared_expert) {
+ if (ly->sh_w[0] && part != MOE_ROUTED) {
for (uint32_t j = 0; j < n; j++, nt++) {
m->pair_tok[nt] = j;
+ /* its gate, or none: a branch of its own (Gemma 4) */
m->pair_w[nt] =
- sigmoid(janas_dot_f32(ly->sh_gate, m->xn + (size_t)j * dm, dm));
+ ly->sh_gate ? sigmoid(janas_dot_f32(ly->sh_gate,
+ m->xn + (size_t)j * dm, dm))
+ : 1.0f;
m->order[nt] = nt;
}
}
diff --git a/src/llm/model_impl.h b/src/llm/model_impl.h
index 15a25ea..dc84a19 100644
--- a/src/llm/model_impl.h
+++ b/src/llm/model_impl.h
@@ -40,6 +40,16 @@ struct layer {
output scale (1 if none) */
const struct janas_jns_tensor *post_attn, *post_ffn;
float out_scale;
+ /* Gemma 4's mixture: the dense feed-forward (the shared expert, weight
+ 1, no gate) and the experts are two branches, each normalized on the
+ way in and out; the router reads its own input, scaled per dimension,
+ and each expert's output has a scale of its own */
+ const struct janas_jns_tensor *pre_norm2, *post_norm1, *post_norm2;
+ const float *router_scale, *exp_scale;
+ /* Gemma 4's per-layer embeddings (E2B, E4B): after the FFN, the layer's
+ own input of ple_dim numbers per token gates a branch of the
+ residual, gelu(gate x) * input, projected back, normalized, added */
+ const struct janas_jns_tensor *ple_gate, *ple_proj, *ple_post;
/* Gated DeltaNet */
const struct janas_jns_tensor *ssm_in, *ssm_ba, *conv, *dt_bias, *ssm_a,
*ssm_norm, *ssm_out;
@@ -80,6 +90,8 @@ struct janas_llm_model {
const struct janas_arch *a; /* what this architecture has (arch.h) */
uint32_t n_layer, d_model, n_head, n_head_kv, head_dim, n_rot, n_expert, k,
n_vocab, n_ctx, d_ff, n_attn, n_rec;
+ uint32_t ff_buf; /* rows of a pair's buffers: d_ff, or a wider shared
+ feed-forward's (Gemma 4's 26B) */
float eps, rope_base;
/* Gemma 4: the sliding-window layers' geometry (head size and RoPE in
head_swa, n_rot_swa, rope_base_swa; n_head_kv, head_dim and n_rot
@@ -89,11 +101,32 @@ struct janas_llm_model {
uint32_t head_swa, n_rot_swa, n_kv_swa, window, hd_max, kvd_max;
float rope_base_swa, softcap;
const float *rope_freqs;
+ /* Gemma 4's per-layer embeddings (0: none): ple_dim numbers per layer
+ and token, from a table of their own read a row at a time from the
+ file (ple_tok, never in memory) and a projection of the embedding
+ (ple_model, bf16), normalized by ple_norm; ple holds the block's,
+ token after token, layer after layer, ple_g a branch's rows,
+ ple_row a row of the table as floats, then as read (room for twice
+ its numbers in floats) */
+ uint32_t ple_dim;
+ const struct janas_jns_tensor *ple_tok, *ple_model, *ple_norm;
+ float *ple, *ple_g, *ple_row;
+ struct janas_block_q8k *ple_q;
+ /* layers from n_kv_layers on keep no keys and values of their own:
+ they read those of the last earlier layer of their kind */
+ uint32_t n_kv_layers;
/* the KV cache of slot s (attention layer, then the MTP block's) starts
at kv_off[s] bytes and its scales at ks_off[s] floats; kv_hd and
kv_heads its geometry. Uniform unless the layers differ (Gemma 4) */
size_t *kv_off, *ks_off;
uint32_t *kv_hd, *kv_heads;
+ int kv16; /* keys and values of 16 bits (arch, or JANAS_KV=8|16) */
+ /* positions a slot holds per KV head: n_ctx, or for a sliding-window
+ layer a ring of fewer (kv_ring), position p in row p mod kv_cap;
+ ring_hi: one past the last position written */
+ uint32_t *kv_cap;
+ uint8_t *kv_ring, *kv_swa; /* and whether its layer is a sliding one */
+ uint32_t ring_hi;
/* Gated DeltaNet geometry: ds = head size (keys and values), n_kh key
heads, n_vh value heads (n_vh / n_kh per key head), conv over the
q|k|v channels */
@@ -157,7 +190,7 @@ struct janas_llm_model {
struct janas_attn *attn;
/* Gemma 4: the sliding-window layers' RoPE table and attention, and the
residual kept aside while the FFN's output is normalized */
- float *rope_cos_swa, *rope_sin_swa, *xres;
+ float *rope_cos_swa, *rope_sin_swa, *xres, *xbr; /* xbr: a branch */
struct janas_attn *attn_swa;
/*
@@ -199,6 +232,7 @@ struct janas_llm_model {
then one pair per token for the shared expert */
uint32_t *pair_tok, *pair_exp, *order, *count, *glist;
float *pair_w, *gate, *up, *h, *dout;
+ uint32_t *hlen; /* per pair row: the rows of its expert's gate */
struct janas_block_q8k *xg, *hq;
uint32_t k_use; /* experts used per token, at most k (see set_experts) */
int exp_level; /* planes of the experts' down matrix read: 1, 2 or 3 */
@@ -282,7 +316,8 @@ struct tok_job {
uint32_t first, n, pos0;
float *kc_scale, *vc_scale;
int8_t *kc, *vc;
- float *out; /* rows the job rewrites (the soft-capped logits) */
+ uint32_t cap, ring; /* the slot's positions per head, and whether a ring */
+ float *out; /* rows the job rewrites (the soft-capped logits) */
};
void janas_m_init_tuner(struct janas_llm_model *m);
@@ -297,10 +332,17 @@ void janas_m_attn_layer(struct janas_llm_model *m, const struct layer *ly,
uint32_t n, uint32_t pos0);
void janas_m_rec_layer(struct janas_llm_model *m, const struct layer *ly,
uint32_t n);
+int janas_m_ple_inputs(struct janas_llm_model *m, const int32_t *tokens,
+ uint32_t n);
+void janas_m_ple_layer(struct janas_llm_model *m, const struct layer *ly,
+ uint32_t l, uint32_t n);
int janas_m_exp_tracing(void);
void janas_m_exp_report(void);
+/* which experts a moe_block call runs */
+enum { MOE_ALL, MOE_SHARED, MOE_ROUTED };
int janas_m_moe_block(struct janas_llm_model *m, const struct layer *ly,
struct janas_expert_cache *cache,
- const struct janas_jns_layer *jl, uint32_t l, uint32_t n);
+ const struct janas_jns_layer *jl, uint32_t l, uint32_t n,
+ int part);
#endif
diff --git a/src/llm/model_load.c b/src/llm/model_load.c
index f9d7989..1ef2c20 100644
--- a/src/llm/model_load.c
+++ b/src/llm/model_load.c
@@ -101,12 +101,18 @@ static int load_swa_params(struct janas_llm_model *m, const uint8_t **pat)
if (janas_jns_meta_bytes(&m->j, key, &np, pat) || np < m->n_layer)
return -1;
snprintf(key, sizeof(key), "%.32s.attention.head_count_kv", m->j.h.arch);
- if (janas_jns_meta_ints(&m->j, key, &nk, &kv) || nk < m->n_layer)
- return -1;
+ /* E2B and E4B give one count for both kinds */
+ uint32_t one = 0;
+ if (janas_jns_meta_ints(&m->j, key, &nk, &kv) || nk < m->n_layer) {
+ if (janas_jns_meta_u32(&m->j, key, &one) || one == 0)
+ return -1;
+ kv = NULL;
+ }
m->n_head_kv = m->n_kv_swa = 0;
for (uint32_t l = 0; l < m->n_layer; l++) {
- int32_t h;
- memcpy(&h, kv + l, 4);
+ int32_t h = (int32_t)one;
+ if (kv)
+ memcpy(&h, kv + l, 4);
uint32_t *want = (*pat)[l] ? &m->n_kv_swa : &m->n_head_kv;
if (h <= 0 || (*want && *want != (uint32_t)h))
return -1;
@@ -182,6 +188,10 @@ static int load_layer(struct janas_llm_model *m, const struct janas_jns *j,
snprintf(name, sizeof(name), "blk.%u." suffix, l); \
if (!(ly->field = need(j, name, kind, d0, d1, err, err_len))) \
return -1;
+#define NEED_T(var, suffix, d0) \
+ snprintf(name, sizeof(name), "blk.%u." suffix, l); \
+ if (!(var = need(j, name, T_F32, d0, 0, err, err_len))) \
+ return -1;
NEED(attn_norm, "attn_norm.weight", T_F32, dm, 0)
if (ly->rec) {
uint32_t s = m->ds;
@@ -211,14 +221,36 @@ static int load_layer(struct janas_llm_model *m, const struct janas_jns *j,
NEED(ssm_out, "ssm_out.weight", T_QUANT, m->d_inner, dm)
} else {
NEED(wq, "attn_q.weight", T_QUANT, dm, qd * (m->a->attn_gate ? 2 : 1))
- NEED(wk, "attn_k.weight", T_QUANT, dm, kvd)
- snprintf(name, sizeof(name), "blk.%u.attn_v.weight", l);
- if (!m->a->swa || janas_jns_tensor(j, name)) {
- NEED(wv, "attn_v.weight", T_QUANT, dm, kvd)
- }
NEED(wo, "attn_output.weight", T_QUANT, qd, dm)
NEED(q_norm, "attn_q_norm.weight", T_F32, ly->hd, 0)
- NEED(k_norm, "attn_k_norm.weight", T_F32, ly->hd, 0)
+ /* a layer reading an earlier one's keys and values has no use for
+ its k and v, even where the file has them */
+ if (!m->n_kv_layers || l < m->n_kv_layers) {
+ NEED(wk, "attn_k.weight", T_QUANT, dm, kvd)
+ snprintf(name, sizeof(name), "blk.%u.attn_v.weight", l);
+ if (!m->a->swa || janas_jns_tensor(j, name)) {
+ NEED(wv, "attn_v.weight", T_QUANT, dm, kvd)
+ }
+ NEED(k_norm, "attn_k_norm.weight", T_F32, ly->hd, 0)
+ }
+ }
+ if (m->ple_dim) {
+ const struct janas_jns_tensor *pt[2];
+ static const char *const pn[2] = {"inp_gate.weight", "proj.weight"};
+ for (int p = 0; p < 2; p++) {
+ snprintf(name, sizeof(name), "blk.%u.%s", l, pn[p]);
+ pt[p] = janas_jns_tensor(j, name);
+ uint64_t d0 = p ? m->ple_dim : dm, d1 = p ? dm : m->ple_dim;
+ if (!pt[p] || pt[p]->dims[0] != d0 || pt[p]->dims[1] != d1 ||
+ (pt[p]->type != 0 &&
+ janas_qtype_block_size((int)pt[p]->type) == 0)) {
+ snprintf(err, err_len, "tensor %s missing or unexpected", name);
+ return -1;
+ }
+ }
+ ly->ple_gate = pt[0];
+ ly->ple_proj = pt[1];
+ NEED(ple_post, "post_norm.weight", T_F32, dm, 0)
}
ly->out_scale = 1.0f;
if (m->a->sandwich) {
@@ -264,6 +296,41 @@ static int load_layer(struct janas_llm_model *m, const struct janas_jns *j,
}
ly->router16 = h16;
}
+ if (m->a->sandwich && ly->gate_inp) {
+ /* Gemma 4's mixture: the dense feed-forward beside the experts (as
+ wide as its file says, widened by gguf2jns), its norms, and the
+ router's and the experts' scales */
+ static const char *const dn[3] = {"ffn_gate.weight", "ffn_up.weight",
+ "ffn_down.weight"};
+ const struct janas_jns_tensor *sw[3];
+ for (int p = 0; p < 3; p++) {
+ snprintf(name, sizeof(name), "blk.%u.%s", l, dn[p]);
+ if (!(sw[p] = need(j, name, T_QUANT, p == 2 ? sw[0]->dims[1] : dm,
+ p == 2 ? dm : 0, err, err_len)))
+ return -1;
+ }
+ if (sw[0]->dims[1] != sw[1]->dims[1] || sw[0]->dims[1] % JANAS_QK ||
+ sw[2]->dims[0] != sw[0]->dims[1] || sw[2]->dims[1] != dm) {
+ snprintf(err, err_len, "layer %u: dense feed-forward shapes", l);
+ return -1;
+ }
+ for (int p = 0; p < 3; p++) {
+ ly->sh_w[p] = data(m, sw[p]);
+ ly->sh_mx[p] =
+ (struct janas_jns_matrix){.type = sw[p]->type,
+ .rows = (uint32_t)sw[p]->dims[1],
+ .cols = (uint32_t)sw[p]->dims[0],
+ .bytes = sw[p]->bytes};
+ }
+ NEED(pre_norm2, "pre_ffw_norm_2.weight", T_F32, dm, 0)
+ NEED(post_norm1, "post_ffw_norm_1.weight", T_F32, dm, 0)
+ NEED(post_norm2, "post_ffw_norm_2.weight", T_F32, dm, 0)
+ const struct janas_jns_tensor *rs, *es;
+ NEED_T(rs, "ffn_gate_inp.scale", dm)
+ NEED_T(es, "ffn_down_exps.scale", m->n_expert)
+ ly->router_scale = f32(m, rs);
+ ly->exp_scale = f32(m, es);
+ }
if (m->a->shared_expert) {
const struct janas_jns_tensor *sg, *sw[3];
static const char *const sh_names[3] = {"ffn_gate_shexp.weight",
@@ -306,6 +373,7 @@ static int load_layer(struct janas_llm_model *m, const struct janas_jns *j,
}
}
#undef NEED
+#undef NEED_T
return 0;
}
@@ -465,12 +533,48 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
m->kvd_max = m->n_head_kv * m->head_dim;
if (m->n_kv_swa * m->head_swa > m->kvd_max)
m->kvd_max = m->n_kv_swa * m->head_swa;
- for (uint32_t l = 0; l < m->n_layer; l++)
- if (m->j.layers[l].m[JANAS_JNS_GATE].rows != m->d_ff ||
+ /* the experts of a mixture are alike; a dense model's feed-forward
+ may widen from one layer to the next (Gemma 4's E2B: 6144, then
+ 12288): the buffers take the widest */
+ for (uint32_t l = 0; l < m->n_layer; l++) {
+ uint32_t r = m->j.layers[l].m[JANAS_JNS_GATE].rows;
+ if ((r != m->d_ff && m->n_expert > 1) || r % JANAS_QK ||
m->j.layers[l].m[JANAS_JNS_DOWN].rows != m->d_model) {
snprintf(err, err_len, "layer %u: expert shapes differ", l);
goto fail;
}
+ if (r > m->d_ff)
+ m->d_ff = r;
+ }
+ /* Gemma 4's E2B and E4B: per-layer embeddings, and the last layers
+ reading the keys and values of earlier ones */
+ if (m->a->swa) {
+ uint32_t shared = 0;
+ meta_u32(m, "embedding_length_per_layer_input", &m->ple_dim);
+ meta_u32(m, "attention.shared_kv_layers", &shared);
+ if (shared && (shared + 2 > m->n_layer || !swa_pat)) {
+ snprintf(err, err_len, "inconsistent shared_kv_layers");
+ goto fail;
+ }
+ m->n_kv_layers = shared ? m->n_layer - shared : 0;
+ if (m->ple_dim &&
+ (m->ple_dim % JANAS_QK ||
+ !(m->ple_tok =
+ janas_jns_tensor(&m->j, "per_layer_token_embd.weight")) ||
+ m->ple_tok->dims[0] != (uint64_t)m->n_layer * m->ple_dim ||
+ janas_qtype_block_size((int)m->ple_tok->type) == 0 ||
+ !(m->ple_model =
+ janas_jns_tensor(&m->j, "per_layer_model_proj.weight")) ||
+ m->ple_model->type != 30 || m->ple_model->dims[0] != m->d_model ||
+ m->ple_model->dims[1] != (uint64_t)m->n_layer * m->ple_dim ||
+ !(m->ple_norm =
+ janas_jns_tensor(&m->j, "per_layer_proj_norm.weight")) ||
+ m->ple_norm->type != 0 || m->ple_norm->dims[0] != m->ple_dim)) {
+ snprintf(err, err_len,
+ "per-layer embeddings missing or unexpected");
+ goto fail;
+ }
+ }
/* resident region into RAM; with the GPU, memory from its driver
(JANAS_GPU_ALLOC) or imported at the end */
@@ -494,10 +598,25 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
snprintf(err, err_len, "out of memory for the resident region");
goto fail;
}
+ /* all of it but the per-layer embeddings' table, read a row at a time
+ where it is (model_ple.c): its pages, never touched, take no memory */
+ uint64_t skip_a = m->j.h.resident_bytes, skip_b = skip_a;
+ if (m->ple_tok && m->ple_tok->offset >= m->j.h.resident_offset) {
+ skip_a = (m->ple_tok->offset - m->j.h.resident_offset) & ~4095ull;
+ skip_b =
+ (m->ple_tok->offset - m->j.h.resident_offset + m->ple_tok->bytes) &
+ ~4095ull;
+ if (skip_b < skip_a)
+ skip_b = skip_a;
+ }
for (uint64_t done = 0; done < m->j.h.resident_bytes;) {
- ssize_t r =
- pread(m->j.fd, m->resident + done, m->j.h.resident_bytes - done,
- (off_t)(m->j.h.resident_offset + done));
+ if (done == skip_a && skip_b > skip_a) {
+ done = skip_b;
+ continue;
+ }
+ uint64_t end = done < skip_a ? skip_a : m->j.h.resident_bytes;
+ ssize_t r = pread(m->j.fd, m->resident + done, end - done,
+ (off_t)(m->j.h.resident_offset + done));
if (r <= 0) {
snprintf(err, err_len, "cannot read the resident region");
goto fail;
@@ -529,13 +648,25 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
for (uint32_t l = 0; l < m->n_layer; l++) {
struct layer *ly = &m->layers[l];
ly->rec = m->a->rec && (l + 1) % interval != 0;
- ly->slot = ly->rec ? m->n_rec++ : m->n_attn++;
if (swa_pat) {
ly->swa = swa_pat[l] != 0;
ly->hd = ly->swa ? m->head_swa : m->head_dim;
ly->n_kv = ly->swa ? m->n_kv_swa : m->n_head_kv;
ly->n_rot = ly->swa ? m->n_rot_swa : m->n_rot;
}
+ if (m->n_kv_layers && l >= m->n_kv_layers) {
+ /* the slot of the last layer of its kind that has one, as
+ llama.cpp takes it: the one before last, or the last */
+ uint32_t src = m->n_kv_layers - (ly->swa ? 2 : 1);
+ if (m->layers[src].swa != ly->swa) {
+ snprintf(err, err_len, "layer %u: no keys and values to share",
+ l);
+ goto fail;
+ }
+ ly->slot = m->layers[src].slot;
+ } else {
+ ly->slot = ly->rec ? m->n_rec++ : m->n_attn++;
+ }
if (load_layer(m, &m->j, l, ly, err, err_len) != 0)
goto fail;
}
@@ -564,13 +695,39 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
}
m->rope_freqs = rf ? f32(m, rf) : NULL;
}
- /* the KV cache, slot after slot, each with its own geometry */
+ /* the KV cache, slot after slot, each with its own geometry, at eight
+ bits a number or sixteen */
+ {
+ const char *kv = getenv("JANAS_KV");
+ m->kv16 = kv ? atoi(kv) == 16 : m->a->kv16;
+ }
+ {
+ const char *pb = getenv("JANAS_PREFILL_BLOCK");
+ long v = pb ? atol(pb) : 256;
+ m->blk = v < JANAS_LLM_MAX_BLOCK ? JANAS_LLM_MAX_BLOCK
+ : v > 4096 ? 4096
+ : (uint32_t)v;
+ }
+ /* the sliding-window layers keep a ring of positions, not the whole
+ context: the window and a prefill block (JANAS_SWA_RING=0 keeps
+ every position, as the full layers do) */
+ uint32_t ring_cap = 0;
+ if (m->a->swa) {
+ const char *rg = getenv("JANAS_SWA_RING");
+ ring_cap = janas_attn_ring_positions(m->window, m->blk);
+ if ((rg && atoi(rg) == 0) || ring_cap >= m->n_ctx)
+ ring_cap = 0;
+ }
uint32_t n_slots = m->n_attn + (m->mtp ? 1 : 0);
+ m->kv_cap = calloc(n_slots + 1, sizeof(uint32_t));
+ m->kv_ring = calloc(n_slots + 1, 1);
+ m->kv_swa = calloc(n_slots + 1, 1);
m->kv_off = malloc((n_slots + 1) * sizeof(size_t));
m->ks_off = malloc((n_slots + 1) * sizeof(size_t));
m->kv_hd = calloc(n_slots + 1, sizeof(uint32_t));
m->kv_heads = calloc(n_slots + 1, sizeof(uint32_t));
- if (!m->kv_off || !m->ks_off || !m->kv_hd || !m->kv_heads) {
+ if (!m->kv_off || !m->ks_off || !m->kv_hd || !m->kv_heads || !m->kv_cap ||
+ !m->kv_ring || !m->kv_swa) {
snprintf(err, err_len, "out of memory");
goto fail;
}
@@ -580,13 +737,17 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
continue;
m->kv_hd[ly->slot] = ly->hd;
m->kv_heads[ly->slot] = ly->n_kv;
+ m->kv_swa[ly->slot] = (uint8_t)ly->swa;
+ m->kv_ring[ly->slot] = ly->swa && ring_cap;
+ m->kv_cap[ly->slot] = m->kv_ring[ly->slot] ? ring_cap : m->n_ctx;
}
size_t kv_len = 0, ks_len = 0;
for (uint32_t s = 0; s < n_slots; s++) {
m->kv_off[s] = kv_len;
m->ks_off[s] = ks_len;
- kv_len += (size_t)m->n_ctx * m->kv_heads[s] * m->kv_hd[s];
- ks_len += (size_t)m->n_ctx * m->kv_heads[s];
+ kv_len += ((size_t)m->kv_cap[s] * m->kv_heads[s] * m->kv_hd[s])
+ << m->kv16;
+ ks_len += (size_t)m->kv_cap[s] * m->kv_heads[s];
}
m->kv_off[n_slots] = kv_len;
m->ks_off[n_slots] = ks_len;
@@ -594,13 +755,12 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
m->vcache = malloc(kv_len ? kv_len : 1);
m->kscale = malloc((ks_len ? ks_len : 1) * sizeof(float));
m->vscale = malloc((ks_len ? ks_len : 1) * sizeof(float));
- {
- const char *pb = getenv("JANAS_PREFILL_BLOCK");
- long v = pb ? atol(pb) : 256;
- m->blk = v < JANAS_LLM_MAX_BLOCK ? JANAS_LLM_MAX_BLOCK
- : v > 4096 ? 4096
- : (uint32_t)v;
- }
+ /* a pair's rows: the experts', or a wider shared feed-forward's */
+ m->ff_buf = m->d_ff;
+ for (uint32_t l = 0; l < m->n_layer; l++)
+ if (m->layers[l].sh_w[0] && m->layers[l].sh_mx[0].rows > m->ff_buf)
+ m->ff_buf = m->layers[l].sh_mx[0].rows;
+ ff = m->ff_buf;
size_t B = m->blk, P = B * (m->k + 1);
size_t inner = m->d_inner > qd ? m->d_inner : qd;
size_t bigw = m->qkvz_dim > 2 * qd ? m->qkvz_dim : 2 * qd;
@@ -617,7 +777,19 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
m->rope_cos_swa = malloc(B * m->n_rot_swa / 2 * sizeof(float));
m->rope_sin_swa = malloc(B * m->n_rot_swa / 2 * sizeof(float));
m->xres = malloc(B * dm * sizeof(float));
- if (!m->rope_cos_swa || !m->rope_sin_swa || !m->xres) {
+ m->xbr = malloc(B * dm * sizeof(float));
+ if (!m->rope_cos_swa || !m->rope_sin_swa || !m->xres || !m->xbr) {
+ snprintf(err, err_len, "out of memory");
+ goto fail;
+ }
+ }
+ if (m->ple_dim) {
+ size_t W = (size_t)m->n_layer * m->ple_dim;
+ m->ple = malloc(B * W * sizeof(float));
+ m->ple_g = malloc(B * m->ple_dim * sizeof(float));
+ m->ple_row = malloc(2 * W * sizeof(float));
+ m->ple_q = malloc(B * m->ple_dim / JANAS_QK * sizeof(*m->ple_q));
+ if (!m->ple || !m->ple_g || !m->ple_row || !m->ple_q) {
snprintf(err, err_len, "out of memory");
goto fail;
}
@@ -636,6 +808,7 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
m->dout = malloc(P * dm * sizeof(float));
m->xg = malloc(P * dm / JANAS_QK * sizeof(*m->xg));
m->hq = malloc(P * ff / JANAS_QK * sizeof(*m->hq));
+ m->hlen = malloc(P * sizeof(uint32_t));
m->tok_first = malloc((B + 1) * sizeof(uint32_t));
m->tok_fill = malloc(B * sizeof(uint32_t));
m->tok_list = malloc(P * sizeof(uint32_t));
@@ -644,7 +817,8 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
!m->kk || !m->vv || !m->att || !m->router || !m->rope_cos ||
!m->rope_sin || !m->xq || !m->attq || !m->pair_tok || !m->pair_exp ||
!m->order || !m->glist || !m->count || !m->pair_w || !m->gate ||
- !m->up || !m->h || !m->dout || !m->xg || !m->hq || m->n_expert > 4096) {
+ !m->up || !m->h || !m->dout || !m->xg || !m->hq || !m->hlen ||
+ m->n_expert > 4096) {
snprintf(err, err_len, "out of memory");
goto fail;
}
@@ -769,14 +943,18 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
janas_pool_set_spin(m->io, 0);
m->attn = janas_attn_create(m->n_head, m->n_head_kv, m->head_dim, m->n_ctx,
janas_pool_size(m->compute), o->attn_scores);
+ if (m->attn)
+ janas_attn_set_kv16(m->attn, m->kv16);
if (m->attn && m->a->swa) {
/* Gemma 4: the q and k norms scale the scores; the sliding-window
layers have an attention of their own */
janas_attn_set_scale(m->attn, 1.0f);
- m->attn_swa =
- janas_attn_create(m->n_head, m->n_kv_swa, m->head_swa, m->n_ctx,
- janas_pool_size(m->compute), o->attn_scores);
+ m->attn_swa = janas_attn_create(
+ m->n_head, m->n_kv_swa, m->head_swa, ring_cap ? ring_cap : m->n_ctx,
+ janas_pool_size(m->compute), o->attn_scores);
if (m->attn_swa) {
+ janas_attn_set_ring(m->attn_swa, ring_cap != 0);
+ janas_attn_set_kv16(m->attn_swa, m->kv16);
janas_attn_set_scale(m->attn_swa, 1.0f);
janas_attn_set_window(m->attn_swa, m->window);
}
@@ -902,11 +1080,20 @@ void janas_llm_model_free(struct janas_llm_model *m)
free(m->vscale);
free(m->kv_off);
free(m->ks_off);
+ free(m->kv_cap);
+ free(m->kv_ring);
+ free(m->kv_swa);
free(m->kv_hd);
free(m->kv_heads);
free(m->rope_cos_swa);
+ free(m->ple);
+ free(m->ple_g);
+ free(m->ple_row);
+ free(m->ple_q);
free(m->rope_sin_swa);
free(m->xres);
+ free(m->xbr);
+ free(m->hlen);
free(m->kcache);
free(m->vcache);
free(m->ssm_buf);
diff --git a/src/llm/model_ple.c b/src/llm/model_ple.c
new file mode 100644
index 0000000..a9c8c73
--- /dev/null
+++ b/src/llm/model_ple.c
@@ -0,0 +1,222 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+/*
+ * model_ple.c - Gemma 4's per-layer embeddings (E2B, E4B).
+ *
+ * Every token brings, besides its embedding, a few numbers for each layer
+ * (ple_dim, 256): a row of a table of their own (n_layer x ple_dim numbers
+ * per token, as large as half of E2B's file) plus a projection of the
+ * embedding, normalized layer by layer. After its feed-forward, each layer
+ * opens a branch of the residual: gelu(gate x) times its own numbers,
+ * projected back to the model's width, normalized, added.
+ *
+ * The table is read a row at a time from the file, when a token needs it:
+ * a few kilobytes a token, never gigabytes in memory.
+ */
+#define _GNU_SOURCE
+#include <math.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "matvec.h"
+#include "vmath.h"
+
+#include "model_impl.h"
+
+/* f32 rounded to bf16 as ggml does it (to nearest, ties to even), kept as
+ f32: the embedding meets the bf16 projection at bf16, as in llama.cpp */
+static float round_bf16(float f)
+{
+ uint32_t u = janas_f2u(f);
+ if ((u & 0x7fffffffu) > 0x7f800000u) /* NaN */
+ return f;
+ u = (u + (0x7fffu + ((u >> 16) & 1u))) & 0xffff0000u;
+ return janas_u2f(u);
+}
+
+struct bf16_job {
+ const uint16_t *w;
+ const float *x; /* n rows of cols, rounded to bf16 */
+ float *y; /* n rows of rows */
+ uint32_t rows, cols, n;
+ float scale;
+};
+
+#if defined(__x86_64__)
+#include <immintrin.h>
+
+__attribute__((target("avx2,fma"))) static float
+dot_bf16_avx2(const uint16_t *w, const float *x, uint32_t n)
+{
+ __m256 acc = _mm256_setzero_ps();
+ uint32_t i = 0;
+ for (; i + 8 <= n; i += 8) {
+ __m128i h = _mm_loadu_si128((const __m128i *)(const void *)(w + i));
+ __m256 wf = _mm256_castsi256_ps(
+ _mm256_slli_epi32(_mm256_cvtepu16_epi32(h), 16));
+ acc = _mm256_fmadd_ps(wf, _mm256_loadu_ps(x + i), acc);
+ }
+ __m128 s =
+ _mm_add_ps(_mm256_castps256_ps128(acc), _mm256_extractf128_ps(acc, 1));
+ s = _mm_add_ps(s, _mm_movehl_ps(s, s));
+ s = _mm_add_ss(s, _mm_movehdup_ps(s));
+ float r = _mm_cvtss_f32(s);
+ for (; i < n; i++)
+ r += janas_u2f((uint32_t)w[i] << 16) * x[i];
+ return r;
+}
+#endif
+
+static float dot_bf16(const uint16_t *w, const float *x, uint32_t n)
+{
+#if defined(__x86_64__)
+ if (__builtin_cpu_supports("avx2") && __builtin_cpu_supports("fma"))
+ return dot_bf16_avx2(w, x, n);
+#endif
+ float r = 0.0f;
+ for (uint32_t i = 0; i < n; i++)
+ r += janas_u2f((uint32_t)w[i] << 16) * x[i];
+ return r;
+}
+
+/* rows split in equal runs, each row read once for all the tokens */
+static void bf16_worker(void *arg, int tid, int n_threads)
+{
+ const struct bf16_job *b = arg;
+ uint32_t per = (b->rows + (uint32_t)n_threads - 1) / (uint32_t)n_threads;
+ uint32_t r0 = (uint32_t)tid * per;
+ uint32_t r1 = r0 + per < b->rows ? r0 + per : b->rows;
+ for (uint32_t r = r0; r < r1; r++) {
+ const uint16_t *w = b->w + (size_t)r * b->cols;
+ for (uint32_t j = 0; j < b->n; j++)
+ b->y[(size_t)j * b->rows + r] =
+ dot_bf16(w, b->x + (size_t)j * b->cols, b->cols) * b->scale;
+ }
+}
+
+struct f32_job {
+ const float *w, *x;
+ float *y;
+ uint32_t rows, cols, n;
+};
+
+static void f32_worker(void *arg, int tid, int n_threads)
+{
+ const struct f32_job *b = arg;
+ size_t total = (size_t)b->rows * b->n;
+ for (size_t i = (size_t)tid; i < total; i += (size_t)n_threads) {
+ uint32_t j = (uint32_t)(i / b->rows), r = (uint32_t)(i % b->rows);
+ const float *w = b->w + (size_t)r * b->cols;
+ const float *x = b->x + (size_t)j * b->cols;
+ float s = 0.0f;
+ for (uint32_t c = 0; c < b->cols; c++)
+ s += w[c] * x[c];
+ b->y[(size_t)j * b->rows + r] = s;
+ }
+}
+
+/* y (n rows of t's rows) = t x (n rows of cols): f32 weights as they are,
+ quantized ones on x put in Q8_K (q, n * cols / 256 blocks) */
+static void product(struct janas_llm_model *m, const struct janas_jns_tensor *t,
+ const float *x, struct janas_block_q8k *q, float *y,
+ uint32_t n)
+{
+ uint32_t cols = (uint32_t)t->dims[0], rows = (uint32_t)t->dims[1];
+ if (t->type == 0) {
+ struct f32_job b = {
+ .w = f32(m, t), .x = x, .y = y, .rows = rows, .cols = cols, .n = n};
+ janas_pool_run(m->compute, f32_worker, &b);
+ return;
+ }
+ size_t nb = cols / JANAS_QK;
+ for (uint32_t j = 0; j < n; j++)
+ janas_q8k_quantize(x + (size_t)j * cols, q + j * nb, cols);
+ struct janas_matvec_task tk = {.type = (int)t->type,
+ .w = data(m, t),
+ .x = q,
+ .y = y,
+ .rows = rows,
+ .cols = cols,
+ .n_vec = n};
+ janas_gpu_matvec_group(m->gpu_off || !m->gpu_use ? NULL : m->gpu,
+ m->compute, &tk, 1);
+}
+
+/*
+ * The block's per-layer inputs, from the embeddings in x (already scaled by
+ * sqrt(d_model)): the projection over sqrt(d_model), each layer's part
+ * normalized, plus the table's row times sqrt(ple_dim), all over sqrt(2).
+ */
+int janas_m_ple_inputs(struct janas_llm_model *m, const int32_t *tokens,
+ uint32_t n)
+{
+ uint32_t dm = m->d_model, P = m->ple_dim, W = m->n_layer * P;
+ float *xb = m->xn; /* the embeddings at bf16 */
+ for (size_t i = 0; i < (size_t)n * dm; i++)
+ xb[i] = round_bf16(m->x[i]);
+ struct bf16_job b = {.w = data(m, m->ple_model),
+ .x = xb,
+ .y = m->ple,
+ .rows = W,
+ .cols = dm,
+ .n = n,
+ .scale = 1.0f / sqrtf((float)dm)};
+ if (n < 2)
+ bf16_worker(&b, 0, 1);
+ else
+ janas_pool_run(m->compute, bf16_worker, &b);
+ const float *nw = f32(m, m->ple_norm);
+ size_t row_bytes =
+ W / JANAS_QK * janas_qtype_block_size((int)m->ple_tok->type);
+ uint8_t *raw = (uint8_t *)(m->ple_row + W); /* the row as read */
+ float es = sqrtf((float)P), is2 = 1.0f / sqrtf(2.0f);
+ for (uint32_t j = 0; j < n; j++) {
+ off_t at =
+ (off_t)(m->ple_tok->offset + (uint64_t)tokens[j] * row_bytes);
+ for (size_t done = 0; done < row_bytes;) {
+ ssize_t r =
+ pread(m->j.fd, raw + done, row_bytes - done, at + (off_t)done);
+ if (r <= 0)
+ return -1;
+ done += (size_t)r;
+ }
+ janas_dequantize((int)m->ple_tok->type, raw, m->ple_row, W);
+ float *p = m->ple + (size_t)j * W;
+ for (uint32_t l = 0; l < m->n_layer; l++) {
+ float *pl = p + (size_t)l * P;
+ const float *el = m->ple_row + (size_t)l * P;
+ rms_norm(pl, pl, nw, P, m->eps);
+ for (uint32_t i = 0; i < P; i++)
+ pl[i] = (pl[i] + el[i] * es) * is2;
+ }
+ }
+ trace_row(m, "inp_per_layer", m->ple + (size_t)(n - 1) * W, P);
+ return 0;
+}
+
+/*
+ * Layer l's branch, on the residual in x after the feed-forward, then the
+ * layer's output scale: x = (x + norm(proj(gelu(gate x) * input))) * s.
+ */
+void janas_m_ple_layer(struct janas_llm_model *m, const struct layer *ly,
+ uint32_t l, uint32_t n)
+{
+ uint32_t dm = m->d_model, P = m->ple_dim, W = m->n_layer * P;
+ product(m, ly->ple_gate, m->x, m->xq, m->ple_g, n);
+ for (uint32_t j = 0; j < n; j++) {
+ float *g = m->ple_g + (size_t)j * P;
+ const float *in = m->ple + (size_t)j * W + (size_t)l * P;
+ for (uint32_t i = 0; i < P; i++)
+ g[i] = janas_gelu_mul_det(g[i], in[i]);
+ }
+ product(m, ly->ple_proj, m->ple_g, m->ple_q, m->xn, n);
+ const float *pw = f32(m, ly->ple_post);
+ for (uint32_t j = 0; j < n; j++) {
+ float *x = m->x + (size_t)j * dm, *y = m->xn + (size_t)j * dm;
+ rms_norm(y, y, pw, dm, m->eps);
+ for (uint32_t i = 0; i < dm; i++)
+ x[i] = (x[i] + y[i]) * ly->out_scale;
+ }
+ trace_row(m, "per_layer_out", m->x + (size_t)(n - 1) * dm, dm);
+}
diff --git a/src/llm/model_state.c b/src/llm/model_state.c
index cee1457..64ee241 100644
--- a/src/llm/model_state.c
+++ b/src/llm/model_state.c
@@ -74,6 +74,13 @@ size_t janas_llm_state_bytes(const struct janas_llm_state *st)
return st->bytes;
}
+/* The positions a slot keeps of a state of n: all, or for a ring the last
+ window of them - all that the next token can see. */
+static size_t slot_rows(const struct janas_llm_model *m, uint32_t sl, size_t n)
+{
+ return m->kv_ring[sl] && n > m->window ? m->window : n;
+}
+
/* The bytes of keys (or values) and the number of scales of n positions
of the first n_slots slots. */
static void kv_lengths(const struct janas_llm_model *m, uint32_t n_slots,
@@ -81,8 +88,9 @@ static void kv_lengths(const struct janas_llm_model *m, uint32_t n_slots,
{
*kv = *rows = 0;
for (uint32_t sl = 0; sl < n_slots; sl++) {
- *rows += (size_t)m->kv_heads[sl] * n;
- *kv += (size_t)m->kv_heads[sl] * n * m->kv_hd[sl];
+ size_t r = slot_rows(m, sl, n);
+ *rows += (size_t)m->kv_heads[sl] * r;
+ *kv += ((size_t)m->kv_heads[sl] * r * m->kv_hd[sl]) << m->kv16;
}
}
@@ -90,13 +98,34 @@ static void kv_lengths(const struct janas_llm_model *m, uint32_t n_slots,
static void copy_kv(struct janas_llm_model *m, struct janas_llm_state *st,
int out)
{
- size_t n = st->n, sb = 0, ss = 0;
- for (uint32_t sl = 0; sl < st->n_slots; sl++)
- for (uint32_t h = 0; h < m->kv_heads[sl];
- h++, sb += n * m->kv_hd[sl], ss += n) {
- size_t hd = m->kv_hd[sl];
- size_t cb = m->kv_off[sl] + h * m->n_ctx * hd;
- size_t cs = m->ks_off[sl] + h * m->n_ctx;
+ size_t sb = 0, ss = 0;
+ for (uint32_t sl = 0; sl < st->n_slots; sl++) {
+ size_t n = slot_rows(m, sl, st->n), cap = m->kv_cap[sl];
+ size_t hd = (size_t)m->kv_hd[sl] << m->kv16; /* bytes a row */
+ if (m->kv_ring[sl]) {
+ /* the last n positions, wherever the ring has them */
+ for (uint32_t h = 0; h < m->kv_heads[sl]; h++)
+ for (size_t p = st->n - n; p < st->n; p++, sb += hd, ss++) {
+ size_t r = (size_t)h * cap + p % cap;
+ size_t cb = m->kv_off[sl] + r * hd;
+ size_t cs = m->ks_off[sl] + r;
+ if (out) {
+ memcpy(st->k + sb, m->kcache + cb, hd);
+ memcpy(st->v + sb, m->vcache + cb, hd);
+ st->ks[ss] = m->kscale[cs];
+ st->vs[ss] = m->vscale[cs];
+ } else {
+ memcpy(m->kcache + cb, st->k + sb, hd);
+ memcpy(m->vcache + cb, st->v + sb, hd);
+ m->kscale[cs] = st->ks[ss];
+ m->vscale[cs] = st->vs[ss];
+ }
+ }
+ continue;
+ }
+ for (uint32_t h = 0; h < m->kv_heads[sl]; h++, sb += n * hd, ss += n) {
+ size_t cb = m->kv_off[sl] + h * cap * hd;
+ size_t cs = m->ks_off[sl] + h * cap;
if (out) {
memcpy(st->k + sb, m->kcache + cb, n * hd);
memcpy(st->v + sb, m->vcache + cb, n * hd);
@@ -109,6 +138,7 @@ static void copy_kv(struct janas_llm_model *m, struct janas_llm_state *st,
memcpy(m->vscale + cs, st->vs + ss, n * sizeof(float));
}
}
+ }
}
static void *dup_mem(const void *p, size_t len)
@@ -124,6 +154,11 @@ struct janas_llm_state *janas_llm_model_state_save(struct janas_llm_model *m,
{
if (n > m->n_ctx)
return NULL;
+ /* a ring keeps the window before n only while nothing past n + (its
+ capacity - the window) has been written over it */
+ for (uint32_t sl = 0; sl < m->n_attn; sl++)
+ if (m->kv_ring[sl] && m->ring_hi > n + m->kv_cap[sl] - m->window)
+ return NULL;
int nbuf = 0;
if (m->a->rec) {
/* the state must stand at n, or be able to get there by replay */
@@ -195,6 +230,7 @@ int janas_llm_model_state_load(struct janas_llm_model *m,
(m->a->rec != 0) != (st->nbuf != 0))
return -1;
copy_kv(m, (struct janas_llm_state *)st, 0);
+ m->ring_hi = st->n;
if (st->nbuf == 1) {
/* into the base, standing at n with nothing logged: the next call
starts from it */
@@ -256,7 +292,9 @@ static void shape(const struct janas_llm_model *m, uint32_t *h)
h[H_VERSION] = STATE_VERSION;
h[H_SLOTS] = m->n_attn + (m->mtp ? 1u : 0u);
h[H_HEADS] = m->n_head_kv;
- h[H_HD] = m->head_dim;
+ /* the bytes of a row: a state kept at eight bits does not load into a
+ cache of sixteen (JANAS_KV) */
+ h[H_HD] = m->head_dim << m->kv16;
h[H_REC] = m->n_rec;
h[H_VH] = m->n_vh;
h[H_DS] = m->ds;
diff --git a/src/llm/tools.c b/src/llm/tools.c
index a1ee367..80db70c 100644
--- a/src/llm/tools.c
+++ b/src/llm/tools.c
@@ -20,6 +20,8 @@ struct janas_toolset {
enum janas_tool_dialect janas_tools_dialect(const char *tmpl)
{
+ if (tmpl && strstr(tmpl, "<|tool_call>"))
+ return JANAS_TOOLS_GEMMA;
if (!tmpl || !strstr(tmpl, "<tool_call>"))
return JANAS_TOOLS_NONE;
if (!strstr(tmpl, "<function="))
@@ -172,6 +174,360 @@ static void extra_keys(struct janas_buf *out, const struct janas_json *obj,
}
}
+/* ---- Gemma 4's notation ---- */
+
+#define GQ "<|\"|>" /* its string quote, a token of its own */
+
+/* Jinja's dictsort: an object's members by key, case-insensitive. */
+static size_t sorted(const struct janas_json *obj,
+ const struct janas_json **out, size_t cap)
+{
+ size_t n = 0;
+ for (const struct janas_json *k = obj ? obj->child : NULL; k && n < cap;
+ k = k->next) {
+ size_t i = n++;
+ for (; i > 0; i--) {
+ const struct janas_json *p = out[i - 1];
+ size_t m = p->key_n < k->key_n ? p->key_n : k->key_n;
+ int c = 0;
+ for (size_t x = 0; x < m && !c; x++) {
+ int a = (unsigned char)p->key[x], b = (unsigned char)k->key[x];
+ a = a >= 'A' && a <= 'Z' ? a + 32 : a;
+ b = b >= 'A' && b <= 'Z' ? b + 32 : b;
+ c = a - b;
+ }
+ if (c < 0 || (c == 0 && p->key_n <= k->key_n))
+ break;
+ out[i] = p;
+ }
+ out[i] = k;
+ }
+ return n;
+}
+
+/* The template's format_argument: strings quoted, keys quoted or not. */
+static void g_arg(struct janas_buf *out, const struct janas_json *v, int ek)
+{
+ switch (v->type) {
+ case JANAS_JSON_NULL:
+ janas_buf_puts(out, "null");
+ break;
+ case JANAS_JSON_TRUE:
+ janas_buf_puts(out, "true");
+ break;
+ case JANAS_JSON_FALSE:
+ janas_buf_puts(out, "false");
+ break;
+ case JANAS_JSON_STRING:
+ janas_buf_puts(out, GQ);
+ janas_buf_put(out, v->s, v->n);
+ janas_buf_puts(out, GQ);
+ break;
+ case JANAS_JSON_OBJECT: {
+ const struct janas_json *k[256];
+ size_t n = sorted(v, k, 256);
+ janas_buf_puts(out, "{");
+ for (size_t i = 0; i < n; i++) {
+ if (i)
+ janas_buf_puts(out, ",");
+ if (ek)
+ janas_buf_puts(out, GQ);
+ janas_buf_put(out, k[i]->key, k[i]->key_n);
+ if (ek)
+ janas_buf_puts(out, GQ);
+ janas_buf_puts(out, ":");
+ g_arg(out, k[i], ek);
+ }
+ janas_buf_puts(out, "}");
+ break;
+ }
+ case JANAS_JSON_ARRAY:
+ janas_buf_puts(out, "[");
+ for (const struct janas_json *e = v->child; e; e = e->next) {
+ g_arg(out, e, ek);
+ if (e->next)
+ janas_buf_puts(out, ",");
+ }
+ janas_buf_puts(out, "]");
+ break;
+ default:
+ janas_buf_put(out, v->s, v->n);
+ }
+}
+
+/* A JSON string upper-cased, as Jinja's "| upper". */
+static void g_upper(struct janas_buf *out, const struct janas_json *v)
+{
+ for (size_t i = 0; v && v->type == JANAS_JSON_STRING && i < v->n; i++) {
+ char c = v->s[i];
+ janas_buf_put(out, &(char){c >= 'a' && c <= 'z' ? c - 32 : c}, 1);
+ }
+}
+
+static int g_type_is(const struct janas_json *v, const char *up)
+{
+ struct janas_buf b = {0};
+ g_upper(&b, janas_json_get(v, "type"));
+ int r = b.n == strlen(up) && b.p && memcmp(b.p, up, b.n) == 0;
+ janas_buf_free(&b);
+ return r;
+}
+
+static void g_required(struct janas_buf *out, const struct janas_json *req)
+{
+ janas_buf_puts(out, "required:[");
+ for (const struct janas_json *e = req ? req->child : NULL; e; e = e->next) {
+ janas_buf_puts(out, GQ);
+ janas_buf_put(out, e->s, e->n);
+ janas_buf_puts(out, GQ);
+ if (e->next)
+ janas_buf_puts(out, ",");
+ }
+ janas_buf_puts(out, "]");
+}
+
+/* The template's format_parameters. */
+static void g_params(struct janas_buf *out, const struct janas_json *props,
+ int filter)
+{
+ static const char *const std[] = {"description", "type", "properties",
+ "required", "nullable"};
+ const struct janas_json *k[256];
+ size_t n = sorted(props, k, 256);
+ int first = 1;
+ for (size_t i = 0; i < n; i++) {
+ const struct janas_json *v = k[i];
+ int skip = 0;
+ for (size_t s = 0; filter && s < 5; s++)
+ skip |=
+ strlen(std[s]) == v->key_n && !memcmp(std[s], v->key, v->key_n);
+ if (skip)
+ continue;
+ if (!first)
+ janas_buf_puts(out, ",");
+ first = 0;
+ janas_buf_put(out, v->key, v->key_n);
+ janas_buf_puts(out, ":{");
+ int comma = 0;
+ const struct janas_json *d = janas_json_get(v, "description");
+ if (d && d->type == JANAS_JSON_STRING && d->n) {
+ janas_buf_puts(out, "description:" GQ);
+ janas_buf_put(out, d->s, d->n);
+ janas_buf_puts(out, GQ);
+ comma = 1;
+ }
+ const struct janas_json *en = janas_json_get(v, "enum");
+ const struct janas_json *it = janas_json_get(v, "items");
+ if (g_type_is(v, "STRING") && en && en->n) {
+ janas_buf_puts(out, comma ? ",enum:" : "enum:");
+ comma = 1;
+ g_arg(out, en, 1);
+ } else if (g_type_is(v, "ARRAY") && it &&
+ it->type == JANAS_JSON_OBJECT && it->n) {
+ janas_buf_puts(out, comma ? ",items:{" : "items:{");
+ comma = 1;
+ const struct janas_json *ik[64];
+ size_t in = sorted(it, ik, 64);
+ int f2 = 1;
+ for (size_t x = 0; x < in; x++) {
+ const struct janas_json *iv = ik[x];
+ if (iv->type == JANAS_JSON_NULL)
+ continue;
+ if (!f2)
+ janas_buf_puts(out, ",");
+ f2 = 0;
+ if (iv->key_n == 10 && !memcmp(iv->key, "properties", 10)) {
+ janas_buf_puts(out, "properties:{");
+ if (iv->type == JANAS_JSON_OBJECT)
+ g_params(out, iv, 0);
+ janas_buf_puts(out, "}");
+ } else if (iv->key_n == 8 && !memcmp(iv->key, "required", 8)) {
+ g_required(out, iv);
+ } else if (iv->key_n == 4 && !memcmp(iv->key, "type", 4)) {
+ janas_buf_puts(out, "type:");
+ if (iv->type == JANAS_JSON_STRING) {
+ janas_buf_puts(out, GQ);
+ g_upper(out, iv);
+ janas_buf_puts(out, GQ);
+ } else {
+ janas_buf_puts(out, "[");
+ for (const struct janas_json *e = iv->child; e;
+ e = e->next) {
+ janas_buf_puts(out, GQ);
+ g_upper(out, e);
+ janas_buf_puts(out, GQ);
+ if (e->next)
+ janas_buf_puts(out, ",");
+ }
+ janas_buf_puts(out, "]");
+ }
+ } else {
+ janas_buf_put(out, iv->key, iv->key_n);
+ janas_buf_puts(out, ":");
+ g_arg(out, iv, 1);
+ }
+ }
+ janas_buf_puts(out, "}");
+ }
+ const struct janas_json *nl = janas_json_get(v, "nullable");
+ if (nl && nl->type == JANAS_JSON_TRUE) {
+ janas_buf_puts(out, comma ? ",nullable:true" : "nullable:true");
+ comma = 1;
+ }
+ if (g_type_is(v, "OBJECT")) {
+ const struct janas_json *pp = janas_json_get(v, "properties");
+ janas_buf_puts(out, comma ? ",properties:{" : "properties:{");
+ comma = 1;
+ if (pp && pp->type == JANAS_JSON_OBJECT)
+ g_params(out, pp, 0);
+ else
+ g_params(out, v, 1);
+ janas_buf_puts(out, "}");
+ const struct janas_json *rq = janas_json_get(v, "required");
+ if (rq && rq->n) {
+ janas_buf_puts(out, ",");
+ g_required(out, rq);
+ }
+ }
+ janas_buf_puts(out, comma ? ",type:" GQ : "type:" GQ);
+ g_upper(out, janas_json_get(v, "type"));
+ janas_buf_puts(out, GQ "}");
+ }
+}
+
+/* The template's format_function_declaration, trimmed. */
+static void g_function(struct janas_buf *out, const struct janas_json *f)
+{
+ const struct janas_json *nm = janas_json_get(f, "name");
+ const struct janas_json *ds = janas_json_get(f, "description");
+ const struct janas_json *pa = janas_json_get(f, "parameters");
+ janas_buf_puts(out, "declaration:");
+ if (nm)
+ janas_buf_put(out, nm->s, nm->n);
+ janas_buf_puts(out, "{description:" GQ);
+ if (ds && ds->type == JANAS_JSON_STRING)
+ janas_buf_put(out, ds->s, ds->n);
+ janas_buf_puts(out, GQ);
+ if (pa && pa->type == JANAS_JSON_OBJECT && pa->n) {
+ janas_buf_puts(out, ",parameters:{");
+ const struct janas_json *pp = janas_json_get(pa, "properties");
+ if (pp && pp->type == JANAS_JSON_OBJECT && pp->n) {
+ janas_buf_puts(out, "properties:{");
+ g_params(out, pp, 0);
+ janas_buf_puts(out, "},");
+ }
+ const struct janas_json *rq = janas_json_get(pa, "required");
+ if (rq && rq->n) {
+ g_required(out, rq);
+ janas_buf_puts(out, ",");
+ }
+ if (janas_json_get(pa, "type")) {
+ janas_buf_puts(out, "type:" GQ);
+ g_upper(out, janas_json_get(pa, "type"));
+ janas_buf_puts(out, GQ "}");
+ }
+ }
+ janas_buf_puts(out, "}");
+}
+
+void janas_tools_gemma_response(struct janas_buf *out, const char *name,
+ size_t name_n, const char *text, size_t n)
+{
+ /* a message's content is text, JSON or not: the template quotes it */
+ janas_buf_puts(out, "response:");
+ janas_buf_put(out, name, name_n);
+ janas_buf_puts(out, "{value:" GQ);
+ janas_buf_put(out, text, n);
+ janas_buf_puts(out, GQ "}<tool_response|>");
+}
+
+/* Gemma's notation to JSON, one value from *p; 0, or -1 when it is not. */
+static int g_to_json(const char **p, const char *end, struct janas_buf *out)
+{
+ const char *s = *p;
+ size_t ql = strlen(GQ);
+ while (s < end && (*s == ' ' || *s == '\n'))
+ s++;
+ if (s >= end)
+ return -1;
+ if ((size_t)(end - s) >= ql && !memcmp(s, GQ, ql)) {
+ const char *b = s + ql, *e = b;
+ while (e + ql <= end && memcmp(e, GQ, ql))
+ e++;
+ if (e + ql > end)
+ return -1;
+ janas_json_write_str(out, b, (size_t)(e - b));
+ *p = e + ql;
+ return 0;
+ }
+ if (*s == '{' || *s == '[') {
+ char close = *s == '{' ? '}' : ']';
+ int obj = *s == '{';
+ janas_buf_put(out, s, 1);
+ s++;
+ for (int first = 1;; first = 0) {
+ while (s < end && (*s == ' ' || *s == '\n'))
+ s++;
+ if (s < end && *s == close)
+ break;
+ if (!first) {
+ if (s >= end || *s != ',')
+ return -1;
+ janas_buf_puts(out, ",");
+ s++;
+ }
+ if (obj) { /* a key, bare or quoted, then ':' */
+ while (s < end && *s == ' ')
+ s++;
+ const char *k = s;
+ if ((size_t)(end - s) >= ql && !memcmp(s, GQ, ql)) {
+ k = s += ql;
+ while (s + ql <= end && memcmp(s, GQ, ql))
+ s++;
+ janas_json_write_str(out, k, (size_t)(s - k));
+ s += ql;
+ } else {
+ while (s < end && *s != ':')
+ s++;
+ janas_json_write_str(out, k, (size_t)(s - k));
+ }
+ if (s >= end || *s != ':')
+ return -1;
+ janas_buf_puts(out, ":");
+ s++;
+ }
+ if (g_to_json(&s, end, out))
+ return -1;
+ }
+ janas_buf_put(out, &close, 1);
+ *p = s + 1;
+ return 0;
+ }
+ const char *b = s;
+ while (s < end && *s != ',' && *s != '}' && *s != ']')
+ s++;
+ size_t n = (size_t)(s - b);
+ while (n && b[n - 1] == ' ')
+ n--;
+ if (!n)
+ return -1;
+ int bare = (n == 4 && (!memcmp(b, "true", 4) || !memcmp(b, "null", 4))) ||
+ (n == 5 && !memcmp(b, "false", 5));
+ char *x = NULL;
+ char tmp[64];
+ if (!bare && n < sizeof(tmp)) {
+ memcpy(tmp, b, n);
+ tmp[n] = 0;
+ strtod(tmp, &x);
+ }
+ if (bare || (x && *x == 0))
+ janas_buf_put(out, b, n);
+ else /* a bare word: taken as a string */
+ janas_json_write_str(out, b, n);
+ *p = s;
+ return 0;
+}
+
/* One function, as Qwen3-Coder's template describes it. */
static void coder_function(struct janas_buf *out, const struct janas_json *f)
{
@@ -263,6 +619,18 @@ void janas_tools_system(struct janas_buf *out, enum janas_tool_dialect d,
}
break;
}
+ case JANAS_TOOLS_GEMMA: {
+ size_t n = sys_n;
+ const char *s = sys ? trim(sys, &n) : NULL;
+ if (s && n)
+ janas_buf_put(out, s, n);
+ for (uint32_t i = 0; i < t->n; i++) {
+ janas_buf_puts(out, "<|tool>");
+ g_function(out, t->fn[i]);
+ janas_buf_puts(out, "<tool|>");
+ }
+ break;
+ }
case JANAS_TOOLS_XML_CODER:
if (sys)
janas_buf_put(out, sys, sys_n);
@@ -284,6 +652,30 @@ void janas_tools_system(struct janas_buf *out, enum janas_tool_dialect d,
}
}
+static uint32_t g_value_rule(struct janas_gbuild *b, int32_t quote);
+
+/* The rule of a Gemma call: call:name{members} then the closing token. */
+uint32_t janas_tools_rule_gemma(struct janas_gbuild *b,
+ const struct janas_toolset *t, int only,
+ int32_t close, int32_t quote)
+{
+ uint32_t fns = janas_gb_rule(b);
+ for (uint32_t i = 0; i < t->n; i++) {
+ if (only >= 0 && (uint32_t)only != i)
+ continue;
+ const struct janas_json *nm = janas_json_get(t->fn[i], "name");
+ janas_gb_lit(b, janas_gb_alt(b, fns), nm->s, nm->n);
+ }
+ uint32_t val = g_value_rule(b, quote);
+ uint32_t r = janas_gb_rule(b);
+ uint32_t a = janas_gb_alt(b, r);
+ janas_gb_lit(b, a, "call:", 5);
+ janas_gb_ref(b, a, fns);
+ janas_gb_ref(b, a, val);
+ janas_gb_token(b, a, close);
+ return r;
+}
+
/* ---- calls of earlier turns ---- */
/* The arguments of an XML call, one <parameter=...> block per key. */
@@ -308,6 +700,31 @@ void janas_tools_assistant(struct janas_buf *out, enum janas_tool_dialect d,
{
size_t tn = n;
const char *tc = content ? trim(content, &tn) : "";
+ if (d == JANAS_TOOLS_GEMMA) { /* the calls first, then the text */
+ for (size_t i = 0; i < n_calls; i++) {
+ janas_buf_puts(out, "<|tool_call>call:");
+ janas_buf_put(out, calls[i].name, calls[i].name_n);
+ struct janas_json_doc *doc =
+ janas_json_parse(calls[i].args, calls[i].args_n, NULL, 0);
+ const struct janas_json *r = janas_json_root(doc);
+ const struct janas_json *k[256];
+ size_t m =
+ r && r->type == JANAS_JSON_OBJECT ? sorted(r, k, 256) : 0;
+ janas_buf_puts(out, "{");
+ for (size_t x = 0; x < m; x++) {
+ if (x)
+ janas_buf_puts(out, ",");
+ janas_buf_put(out, k[x]->key, k[x]->key_n);
+ janas_buf_puts(out, ":");
+ g_arg(out, k[x], 0);
+ }
+ janas_buf_puts(out, "}<tool_call|>");
+ janas_json_free(doc);
+ }
+ if (content)
+ janas_buf_put(out, content, n);
+ return;
+ }
if (d == JANAS_TOOLS_XML_CODER) {
if (tn) {
janas_buf_puts(out, "\n");
@@ -442,6 +859,21 @@ int janas_tools_parse(enum janas_tool_dialect d, const struct janas_toolset *t,
const char *body, size_t n, struct janas_buf *name,
struct janas_buf *args)
{
+ if (d == JANAS_TOOLS_GEMMA) { /* call:name{...} */
+ const char *s = body, *end = body + n;
+ while (s < end && (*s == ' ' || *s == '\n'))
+ s++;
+ if ((size_t)(end - s) < 5 || memcmp(s, "call:", 5))
+ return -1;
+ s += 5;
+ const char *nm = s;
+ while (s < end && *s != '{')
+ s++;
+ if (s == end || janas_toolset_find(t, nm, (size_t)(s - nm)) < 0)
+ return -1;
+ janas_buf_put(name, nm, (size_t)(s - nm));
+ return g_to_json(&s, end, args);
+ }
if (d != JANAS_TOOLS_JSON)
return parse_xml(t, body, n, name, args);
struct janas_json_doc *doc = janas_json_parse(body, n, NULL, 0);
@@ -524,6 +956,83 @@ static uint32_t xml_params_rule(struct janas_gbuild *b, struct janas_jsg *j,
return first;
}
+/*
+ * Gemma's notation, any value: strings between quote tokens, numbers and
+ * words, objects of bare keys, lists. Not held to the function's schema -
+ * the name is - but always a value the parser reads.
+ */
+static uint32_t g_value_rule(struct janas_gbuild *b, int32_t quote)
+{
+ uint8_t any[32], key[32], word[32];
+ memset(any, 0xff, sizeof(any));
+ memset(key, 0, sizeof(key));
+ memset(word, 0, sizeof(word));
+ for (int c = 0; c < 256; c++) {
+ int k = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') ||
+ (c >= '0' && c <= '9') || c == '_';
+ int w = k || c == '-' || c == '+' || c == '.';
+ if (k)
+ key[c / 8] |= (uint8_t)(1 << (c % 8));
+ if (w)
+ word[c / 8] |= (uint8_t)(1 << (c % 8));
+ }
+ uint32_t text = janas_gb_rule(b); /* any bytes */
+ janas_gb_alt(b, text);
+ uint32_t a = janas_gb_alt(b, text);
+ janas_gb_set(b, a, any);
+ janas_gb_ref(b, a, text);
+ uint32_t keyr = janas_gb_rule(b), wordr = janas_gb_rule(b);
+ a = janas_gb_alt(b, keyr);
+ janas_gb_set(b, a, key);
+ a = janas_gb_alt(b, keyr);
+ janas_gb_set(b, a, key);
+ janas_gb_ref(b, a, keyr);
+ a = janas_gb_alt(b, wordr);
+ janas_gb_set(b, a, word);
+ a = janas_gb_alt(b, wordr);
+ janas_gb_set(b, a, word);
+ janas_gb_ref(b, a, wordr);
+ uint32_t val = janas_gb_rule(b), members = janas_gb_rule(b),
+ elems = janas_gb_rule(b);
+ a = janas_gb_alt(b, val); /* a string */
+ janas_gb_token(b, a, quote);
+ janas_gb_ref(b, a, text);
+ janas_gb_token(b, a, quote);
+ a = janas_gb_alt(b, val); /* a number or a word */
+ janas_gb_ref(b, a, wordr);
+ a = janas_gb_alt(b, val); /* an object */
+ janas_gb_byte(b, a, '{');
+ janas_gb_byte(b, a, '}');
+ a = janas_gb_alt(b, val);
+ janas_gb_byte(b, a, '{');
+ janas_gb_ref(b, a, members);
+ janas_gb_byte(b, a, '}');
+ a = janas_gb_alt(b, val); /* a list */
+ janas_gb_byte(b, a, '[');
+ janas_gb_byte(b, a, ']');
+ a = janas_gb_alt(b, val);
+ janas_gb_byte(b, a, '[');
+ janas_gb_ref(b, a, elems);
+ janas_gb_byte(b, a, ']');
+ for (int more = 0; more < 2; more++) {
+ a = janas_gb_alt(b, members);
+ janas_gb_ref(b, a, keyr);
+ janas_gb_byte(b, a, ':');
+ janas_gb_ref(b, a, val);
+ if (more) {
+ janas_gb_byte(b, a, ',');
+ janas_gb_ref(b, a, members);
+ }
+ a = janas_gb_alt(b, elems);
+ janas_gb_ref(b, a, val);
+ if (more) {
+ janas_gb_byte(b, a, ',');
+ janas_gb_ref(b, a, elems);
+ }
+ }
+ return val;
+}
+
uint32_t janas_tools_rule(struct janas_gbuild *b, struct janas_jsg *j,
enum janas_tool_dialect d,
const struct janas_toolset *t, int only,
diff --git a/src/llm/tools.h b/src/llm/tools.h
index 95a57d8..3a49455 100644
--- a/src/llm/tools.h
+++ b/src/llm/tools.h
@@ -31,6 +31,11 @@ enum janas_tool_dialect {
JANAS_TOOLS_JSON,
JANAS_TOOLS_XML,
JANAS_TOOLS_XML_CODER,
+ /* Gemma 4: <|tool>declaration:f{...}<tool|> in the system turn, calls
+ <|tool_call>call:f{key:value,...}<tool_call|>, strings between <|"|>
+ tokens, answers <|tool_response>response:f{value:...}<tool_response|>
+ inside the model's own turn */
+ JANAS_TOOLS_GEMMA,
};
/* The dialect of a chat template (NONE when it has no tools). */
@@ -80,6 +85,19 @@ int janas_tools_parse(enum janas_tool_dialect d, const struct janas_toolset *t,
* functions (only < 0) or function `only`, with arguments valid against its
* parameters.
*/
+/* Gemma 4's call rule: the quote token <|"|> too. */
+uint32_t janas_tools_rule_gemma(struct janas_gbuild *b,
+ const struct janas_toolset *t, int only,
+ int32_t close, int32_t quote);
+
+/* Gemma 4: a tool's answer as its template writes it, without the opening
+ <|tool_response> (the model's stop token, already in the sequence, or
+ written by the caller): response:name{value:<|"|>text<|"|>}<tool_response|>
+ - the text as it is, JSON or not, as the template quotes a message's
+ content. */
+void janas_tools_gemma_response(struct janas_buf *out, const char *name,
+ size_t name_n, const char *text, size_t n);
+
uint32_t janas_tools_rule(struct janas_gbuild *b, struct janas_jsg *j,
enum janas_tool_dialect d,
const struct janas_toolset *t, int only,
diff --git a/tests/test_attention.c b/tests/test_attention.c
index aca35e9..7aad910 100644
--- a/tests/test_attention.c
+++ b/tests/test_attention.c
@@ -155,6 +155,99 @@ static void run(struct janas_pool *pool, uint32_t n_head, uint32_t n_kv,
janas_attn_destroy(a);
}
+/*
+ * Gemma 4's settings: keys and values of sixteen bits, a sliding window and
+ * a score scale of 1, against a double-precision reference that sees the
+ * same positions. Not fed to the checksum, which stays that of the
+ * eight-bit runs.
+ */
+static void run16(struct janas_pool *pool, uint32_t n_head, uint32_t n_kv,
+ uint32_t hd, uint32_t n_ctx, uint32_t pos0, uint32_t n,
+ uint32_t window)
+{
+ struct janas_attn *a =
+ janas_attn_create(n_head, n_kv, hd, n_ctx, janas_pool_size(pool),
+ JANAS_ATTN_INT16); /* the cache forces FLOAT */
+ janas_attn_set_kv16(a, 1);
+ janas_attn_set_window(a, window);
+ janas_attn_set_scale(a, 0.02f);
+ size_t qd = (size_t)n_head * hd, kv = (size_t)n_kv * n_ctx * hd;
+ int16_t *K = malloc(kv * 2), *V = malloc(kv * 2);
+ size_t nsc = (size_t)n_kv * n_ctx;
+ float *ks = malloc(nsc * sizeof(float)), *vs = malloc(nsc * sizeof(float));
+ float *q = malloc(n * qd * sizeof(float));
+ float *blk = malloc(n * qd * sizeof(float));
+ float *one = malloc(qd * sizeof(float));
+ for (size_t i = 0; i < kv; i++) {
+ K[i] = (int16_t)(int)(rndf(-32767, 32767));
+ V[i] = (int16_t)(int)(rndf(-32767, 32767));
+ }
+ for (size_t i = 0; i < nsc; i++) {
+ ks[i] = rndf(0.001f, 0.02f) / 258.0f;
+ vs[i] = rndf(0.001f, 0.02f) / 258.0f;
+ }
+ for (size_t i = 0; i < n * qd; i++)
+ q[i] = rndf(-3, 3);
+ janas_attn_run(a, pool, q, (const int8_t *)K, ks, (const int8_t *)V, vs, n,
+ pos0, blk);
+ double worst = 0;
+ int same = 1;
+ uint32_t group = n_head / n_kv;
+ for (uint32_t j = 0; j < n; j++) {
+ janas_attn_run(a, pool, q + j * qd, (const int8_t *)K, ks,
+ (const int8_t *)V, vs, 1, pos0 + j, one);
+ if (memcmp(one, blk + j * qd, qd * sizeof(float)) != 0)
+ same = 0;
+ uint32_t p = pos0 + j,
+ lo = window && p + 1 > window ? p + 1 - window : 0;
+ for (uint32_t h = 0; h < n_head; h++) {
+ size_t base = (size_t)(h / group) * n_ctx * hd;
+ size_t sb = (size_t)(h / group) * n_ctx;
+ const float *qh = q + j * qd + (size_t)h * hd;
+ double mx = -INFINITY, sum = 0, *sc = malloc((p + 1) * 8);
+ double out[512] = {0};
+ for (uint32_t t = lo; t <= p; t++) {
+ double d = 0;
+ for (uint32_t i = 0; i < hd; i++)
+ d += (double)qh[i] * K[base + (size_t)t * hd + i] *
+ ks[sb + t];
+ sc[t] = d * 0.02;
+ mx = sc[t] > mx ? sc[t] : mx;
+ }
+ for (uint32_t t = lo; t <= p; t++) {
+ double w = exp(sc[t] - mx);
+ sum += w;
+ for (uint32_t i = 0; i < hd; i++)
+ out[i] += w * V[base + (size_t)t * hd + i] * vs[sb + t];
+ }
+ for (uint32_t i = 0; i < hd; i++) {
+ double e =
+ fabs(out[i] / sum - blk[j * qd + (size_t)h * hd + i]);
+ worst = e > worst ? e : worst;
+ }
+ free(sc);
+ }
+ }
+ CHECK(same, "16-bit, window %u, pos0 %u, n %u: block differs", window, pos0,
+ n);
+ CHECK(janas_attn_scores_mode(a) == JANAS_ATTN_FLOAT,
+ "16-bit keys with integer scores");
+ CHECK(worst < 1e-5, "16-bit, window %u, pos0 %u, n %u: error %g", window,
+ pos0, n, worst);
+ printf("heads %2u/%u dim %3u, positions %4u..%4u, 16-bit, window %4u: "
+ "error %.2e, blocks %s\n",
+ n_head, n_kv, hd, pos0, pos0 + n - 1, window, worst,
+ same ? "bit-identical" : "DIFFER");
+ free(K);
+ free(V);
+ free(ks);
+ free(vs);
+ free(q);
+ free(blk);
+ free(one);
+ janas_attn_destroy(a);
+}
+
int main(void)
{
/* a test that deadlocks must fail, not hold the suite for ever */
@@ -169,6 +262,10 @@ int main(void)
run(pool, 32, 4, 128, 1024, 300, 64, sc); /* qwen3moe geometry */
run(pool, 12, 4, 64, 600, 511, 3, sc); /* 3 heads per group */
}
+ run16(pool, 16, 8, 256, 1024, 0, 64, 0);
+ run16(pool, 16, 8, 256, 2048, 700, 64, 256); /* a window of 256 */
+ run16(pool, 16, 1, 512, 1024, 250, 17, 0); /* Gemma 4's full layers */
+ run16(pool, 16, 8, 256, 2048, 1030, 5, 1024);
janas_pool_destroy(pool);
if (failures) {
fprintf(stderr, "%d failures\n", failures);
diff --git a/tools/gguf2jns.c b/tools/gguf2jns.c
index 34724d8..23bf739 100644
--- a/tools/gguf2jns.c
+++ b/tools/gguf2jns.c
@@ -10,6 +10,13 @@
* resident region. A dense model has no experts to route: its feed-forward
* becomes the single slot of the layer, written in the same way.
*
+ * A feed-forward whose width is not a multiple of 256 (Gemma 4's 26B: 704
+ * and 2112) is widened with zeros to the next one - rows of zero bytes in
+ * gate and up, a zero tail on each row of down: every weight type reads zero
+ * bytes as zero weights, so the products are the same, and the engine's
+ * activations, quantized 256 at a time, fit. Gate and up fused in one
+ * tensor (ffn_gate_up_exps) are split into the two matrices of the slot.
+ *
* A model split into several GGUF files (name-00001-of-0000N.gguf) is read
* from all of them: give the first. Data is copied in streaming; the model
* is never held in memory.
@@ -54,6 +61,16 @@ struct tensor {
uint64_t base; /* the part's data start */
int expert; /* one of the experts' matrices */
uint64_t new_offset;
+ /* a resident matrix widened with zeros: its rows, then as written */
+ uint64_t rows, rb, new_rows, new_rb, new_dims[2], new_bytes;
+};
+
+/* A matrix of a slot: from which tensor, where in an expert's part of it,
+ its rows and row bytes, and as written. */
+struct src_matrix {
+ struct tensor *t;
+ uint64_t off, per; /* in an expert's bytes of t, and those bytes */
+ uint64_t rows, rb, new_rows, new_rb;
};
struct part {
@@ -162,6 +179,49 @@ static uint32_t crc_zeros(uint32_t crc, uint64_t n)
return crc;
}
+static uint8_t *obuf; /* rows widened, before they are written */
+
+/*
+ * rows rows of rb bytes at src of t, to dst as new_rows rows of new_rb
+ * bytes: each row's tail and the extra rows zero (the output is created
+ * zero-filled, so only the CRC has to count them).
+ */
+static uint32_t copy_rows(const struct tensor *t, uint64_t src, uint64_t rows,
+ uint64_t rb, uint64_t new_rows, uint64_t new_rb,
+ uint64_t dst, uint32_t crc, int with_crc)
+{
+ if (new_rb == rb) {
+ crc = copy(t, src, rows * rb, dst, crc, with_crc);
+ } else {
+ uint64_t per = COPY_CHUNK / new_rb ? COPY_CHUNK / new_rb : 1;
+ int fd = parts[t->part].fd;
+ for (uint64_t r0 = 0; r0 < rows; r0 += per) {
+ uint64_t k = rows - r0 < per ? rows - r0 : per;
+ if (pread(fd, buf, k * rb, (off_t)(t->base + src + r0 * rb)) !=
+ (ssize_t)(k * rb))
+ die("%s", "unexpected end of the GGUF file");
+ memset(obuf, 0, k * new_rb);
+ for (uint64_t r = 0; r < k; r++)
+ memcpy(obuf + r * new_rb, buf + r * rb, rb);
+ if (with_crc)
+ crc = janas_crc32(crc, obuf, k * new_rb);
+ if (pwrite(out_fd, obuf, k * new_rb, (off_t)(dst + r0 * new_rb)) !=
+ (ssize_t)(k * new_rb))
+ die("write failed: %s", strerror(errno));
+ copied += k * rb;
+ }
+ }
+ if (with_crc)
+ crc = crc_zeros(crc, (new_rows - rows) * new_rb);
+ return crc;
+}
+
+/* A width rounded up to a multiple of 256 (the engine's activations). */
+static uint64_t wide(uint64_t n)
+{
+ return (n + 255) / 256 * 256;
+}
+
static int by_place(const void *a, const void *b)
{
const struct tensor *x = *(const struct tensor *const *)a;
@@ -232,7 +292,7 @@ int main(int argc, char **argv)
/* expert geometry per layer */
struct janas_jns_layer_v2 *layers = calloc(n_layer, sizeof(*layers));
- struct tensor *(*mats)[3] = calloc(n_layer, sizeof(*mats));
+ struct src_matrix(*mats)[3] = calloc(n_layer, sizeof(*mats));
if (!layers || !mats)
die("%s", "out of memory");
static const char *const part_name[3] = {"gate", "up", "down"};
@@ -244,6 +304,13 @@ int main(int argc, char **argv)
: "blk.%llu.ffn_%s_exps.weight",
(unsigned long long)l, part_name[p]);
struct tensor *t = find(key);
+ int half = 0; /* gate and up fused: this one's half of the rows */
+ if (!t && !dense && p < 2) {
+ snprintf(key, sizeof(key), "blk.%llu.ffn_gate_up_exps.weight",
+ (unsigned long long)l);
+ t = find(key);
+ half = 1;
+ }
if (!t)
die("%s missing: only models with routed experts in every "
"layer, or dense ones, are supported",
@@ -251,19 +318,38 @@ int main(int argc, char **argv)
uint64_t ne = t->t->n_dims > 2 ? t->t->dims[2] : 1;
if (ne != n_expert || t->t->bytes % n_expert ||
t->t->dims[0] > UINT32_MAX || t->t->dims[1] > UINT32_MAX ||
- t->t->dims[3] != 1)
+ t->t->dims[3] != 1 || (half && t->t->dims[1] % 2))
die("%s: unexpected shape", key);
uint64_t per = t->t->bytes / n_expert;
+ uint64_t rows = t->t->dims[1] / (half ? 2 : 1);
+ uint64_t cols = t->t->dims[0];
+ if (per % t->t->dims[1])
+ die("%s: rows of unequal size", key);
+ uint64_t rb = per / t->t->dims[1];
+ /* widened to a multiple of 256: the rows of gate and up, the
+ row of down */
+ uint64_t new_rows = p < 2 ? wide(rows) : rows;
+ uint64_t new_cols = p < 2 ? cols : wide(cols);
+ if (new_cols != cols && (rb * new_cols) % cols)
+ die("%s: cannot widen this type", key);
+ uint64_t new_rb = rb * new_cols / cols;
off = align_up(off, MATRIX_ALIGN);
layers[l].m[p] =
(struct janas_jns_matrix){.type = t->t->type,
- .rows = (uint32_t)t->t->dims[1],
- .cols = (uint32_t)t->t->dims[0],
+ .rows = (uint32_t)new_rows,
+ .cols = (uint32_t)new_cols,
.offset = off,
- .bytes = per};
- off += per;
+ .bytes = new_rows * new_rb};
+ off += new_rows * new_rb;
t->expert = 1;
- mats[l][p] = t;
+ mats[l][p] =
+ (struct src_matrix){.t = t,
+ .off = p == 1 && half ? rows * rb : 0,
+ .per = per,
+ .rows = rows,
+ .rb = rb,
+ .new_rows = new_rows,
+ .new_rb = new_rb};
}
layers[l].slot_bytes = align_up(off, ALIGN);
}
@@ -283,6 +369,34 @@ int main(int argc, char **argv)
resident[n_res++] = &tensors[i];
}
qsort(resident, n_res, sizeof(*resident), by_place);
+ /* a dense feed-forward beside the experts (Gemma 4's 26B), widened as
+ theirs */
+ for (size_t i = 0; i < n_res; i++) {
+ struct tensor *t = resident[i];
+ char *name = str_dup(&t->t->name);
+ size_t nl = strlen(name);
+ int gu = nl > 16 && (!strcmp(name + nl - 16, ".ffn_gate.weight") ||
+ !strcmp(name + nl - 14, ".ffn_up.weight"));
+ int dn = nl > 16 && !strcmp(name + nl - 16, ".ffn_down.weight");
+ free(name);
+ t->new_bytes = t->t->bytes;
+ t->new_dims[0] = t->t->dims[0];
+ t->new_dims[1] = t->t->dims[1];
+ if (t->t->n_dims != 2 || !(gu || dn))
+ continue;
+ uint64_t rows = t->t->dims[1], cols = t->t->dims[0];
+ if (t->t->bytes % rows)
+ die("%s", "a feed-forward matrix with rows of unequal size");
+ t->rows = rows;
+ t->rb = t->t->bytes / rows;
+ t->new_rows = gu ? wide(rows) : rows;
+ t->new_dims[1] = t->new_rows;
+ t->new_dims[0] = gu ? cols : wide(cols);
+ if ((t->rb * t->new_dims[0]) % cols)
+ die("%s", "a feed-forward matrix whose type cannot be widened");
+ t->new_rb = t->rb * t->new_dims[0] / cols;
+ t->new_bytes = t->new_rows * t->new_rb;
+ }
/* layout */
uint64_t metadata_bytes = 8 + (g0->kv_end - g0->kv_start);
@@ -298,7 +412,7 @@ int main(int argc, char **argv)
uint64_t pos = resident_offset;
for (size_t i = 0; i < n_res; i++) {
resident[i]->new_offset = pos;
- pos = align_up(pos + resident[i]->t->bytes, ALIGN);
+ pos = align_up(pos + resident[i]->new_bytes, ALIGN);
}
uint64_t resident_bytes = pos - resident_offset;
uint64_t experts_offset = pos;
@@ -329,12 +443,18 @@ int main(int argc, char **argv)
if (out_fd < 0 || ftruncate(out_fd, (off_t)file_bytes) != 0)
die("%s: cannot write", dst);
buf = malloc(COPY_CHUNK);
- if (!buf)
+ obuf = malloc(COPY_CHUNK * 2);
+ if (!buf || !obuf)
die("%s", "out of memory");
- for (size_t i = 0; i < n_res; i++)
- copy(resident[i], resident[i]->t->offset, resident[i]->t->bytes,
- resident[i]->new_offset, 0, 0);
+ for (size_t i = 0; i < n_res; i++) {
+ const struct tensor *t = resident[i];
+ if (t->new_bytes != t->t->bytes)
+ copy_rows(t, t->t->offset, t->rows, t->rb, t->new_rows, t->new_rb,
+ t->new_offset, 0, 0);
+ else
+ copy(t, t->t->offset, t->t->bytes, t->new_offset, 0, 0);
+ }
for (uint64_t l = 0; l < n_layer; l++) {
for (uint64_t e = 0; e < n_expert; e++) {
@@ -345,8 +465,10 @@ int main(int argc, char **argv)
const struct janas_jns_matrix *m = &layers[l].m[p];
if (m->offset > end) /* padding inside the slot is zero */
crc = crc_zeros(crc, m->offset - end);
- crc = copy(mats[l][p], mats[l][p]->t->offset + e * m->bytes,
- m->bytes, slot + m->offset, crc, 1);
+ const struct src_matrix *s = &mats[l][p];
+ crc = copy_rows(s->t, s->t->t->offset + e * s->per + s->off,
+ s->rows, s->rb, s->new_rows, s->new_rb,
+ slot + m->offset, crc, 1);
end = m->offset + m->bytes;
}
crc = crc_zeros(crc, layers[l].slot_bytes - end);
@@ -379,8 +501,10 @@ int main(int argc, char **argv)
d.type = t->type;
d.n_dims = t->n_dims;
memcpy(d.dims, t->dims, sizeof(d.dims));
+ d.dims[0] = resident[i]->new_dims[0];
+ d.dims[1] = resident[i]->new_dims[1];
d.offset = resident[i]->new_offset;
- d.bytes = t->bytes;
+ d.bytes = resident[i]->new_bytes;
memcpy(tbl + (tensor_dir - layer_table) + i * sizeof(d), &d, sizeof(d));
}
uint8_t *meta = malloc(metadata_bytes);
@@ -425,6 +549,7 @@ int main(int argc, char **argv)
free(meta);
free(tbl);
free(buf);
+ free(obuf);
free(slot_off);
free(sums);
free(resident);
diff --git a/tools/janas-try.sh b/tools/janas-try.sh
index 040165d..9916634 100755
--- a/tools/janas-try.sh
+++ b/tools/janas-try.sh
@@ -172,10 +172,13 @@ machine()
grep -qi microsoft /proc/version 2>/dev/null && DISTRO="$DISTRO (WSL)"
# the GPU: whether the build can have it (the shaders need
# glslangValidator) and what Vulkan sees - name, kind, version
- if command -v glslangValidator >/dev/null; then
- GPU_BUILD="with GPU support"
- else
+ if ! command -v glslangValidator >/dev/null; then
GPU_BUILD="without GPU support (glslangValidator not installed)"
+ elif ! echo '#include <vulkan/vulkan.h>' |
+ ${CC:-gcc} -E -x c - >/dev/null 2>&1; then
+ GPU_BUILD="without GPU support (Vulkan headers not installed)"
+ else
+ GPU_BUILD="with GPU support"
fi
if command -v vulkaninfo >/dev/null; then
GPU_VK=$(vulkaninfo --summary 2>/dev/null | awk '