37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-10-27: Gemma 4 on the GPU's whole pass, long contexts, Llama 3 and janas-nsm
camaurifc2f11d6b4ae
diff --git a/ChangeLog.md b/ChangeLog.mdindex c3ec83d..4f7999c 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -2,6 +2,31 @@> Curated, user-facing summary of completed work, newest first.+## [2026-10-26] - The GPU's attention nearer the CPU's++- **The attention's scores the way this processor computes them faster**: in float over the query as it is, or over the query at sixteen bits (an integer sum), measured on the calls themselves - every 4,096 positions eight calls each way, a token's and a block's apart - in place of the sixteen-bit query only past 16,384 tokens of context. The two agree with llama.cpp alike: the most likely token at 2,961 of 3,200 positions with the sixteen-bit query, 2,963 in float (Qwen3 0.6B and 4B, Qwen3.5 0.8B and 2B, prose and code or instructions), mean |diff| 0.3033 and 0.3037. Qwen3-4B on the laptop's CPU, which takes the sixteen-bit way: a token written after 4,096 tokens 22.0 to 23.4 tok/s (the attention 10.7 to 8.2 ms), after 8,192 18.2 to 19.6; the prompts 95.9 to 99.6 and 74.0 to 78.7. `JANAS_ATTN=float` or `int16` fixes it.++- **A long prompt goes the way the machine reads it faster**: the GPU's whole pass, or the path without it (the CPU's attention, the products where they go), measured on the blocks themselves - every eight of the engine's blocks of positions one each way, in turn, and the faster for the rest - since the attention's cost grows with the position, and on the GPU it can grow faster or slower than on the CPU. On the development laptop's integrated Arc, Qwen3-4B: even from position 0 (8.51 and 8.52 ms a token), without it from 4,096 on (18.8 against 12.3 ms a token, at 12,288 41.6 against 25.0); a prompt of 16,384 tokens at 49.7 tok/s instead of 31.8. No threshold: a GPU with fast memory of its own measures its own way; `JANAS_GPU_LONG=0` keeps the whole pass, `JANAS_GPU_TOKEN=1` too.++- **The GPU's norms, the same bits as the CPU's, at a fraction of the time**: the sum of squares' two floats added in a tree (seven or eight steps) instead of each thread going through every thread's in turn - the norms and quantization (norm_quant.comp) 352 ms to 56 a pass of 192 tokens of Qwen3-4B, q's and k's norms with RoPE (qk_rope.comp) 225 to 52, a prompt of 2,048 tokens read with the GPU at 112 tok/s instead of 93; the same bits as before (Gemma 4 12B, 367 of 400 against llama.cpp). A tile of a block's tokens a workgroup in the attention, each key read once for them, was tried and left out: eight times slower, the GPU's attention bound by its arithmetic and shared memory, not by reading the cache.++- **The fast prompt's Q4_K and Q6_K tiles can sum as the CPU does, opt-in (`JANAS_GPU_TILES_EXACT=1`)**: a whole block staged per barrier, its integer sums, then the CPU's float accumulators - the same bits as the CPU on every product of Gemma 4 12B (the tiles as they are: errors of some 1e-6 of the largest output), and the 12B's blocks of 64 back to 235 of 256 against llama.cpp from 209. Not the default: this first version is several times slower than the tiles. `JANAS_GPU_COPY` is read by its value (=0 was taken as asked).++- **A dense feed-forward whose down matrix is in blocks of 32 takes the CPU's activation on the GPU too**: the deterministic SiLU or GELU quantized straight into the blocks of 32 (expert_act.comp's kind 1, as the experts in those types), in place of the GPU's e^x then a quantization apart (`JANAS_GPU_ACT_FLOAT=1` as before). Gemma 4 26B's dense branch (Q8_0): against llama.cpp 321 of 400 instead of 315, the CPU's own (mean |diff| 0.794 instead of 0.806); DeepNSM-1B Q8_0 within the noise (464 and 465 of 477).++- **Gemma 4 26B-A4B on the GPU's whole pass**: its mixture as the CPU computes it - the dense branch (the shared expert, weight 1, no gate) on the feed-forward's norm and normalized apart, the experts' input normalized by its own norm, the router's input the residual normalized without weights times its scales over sqrt(d_model) (norm_quant.comp multiplies the normalized value first), each expert's output scale on its weight (moe_topk.comp), the experts' sum normalized and the dense branch added before the sandwich norm - and its experts' types: Q5_1 down matrices 704 wide through b32_matvec.comp by expert (the CPU's bits), the last layer's Q8_0 ones through q8_0_matvec.comp, which takes the experts' table too now; GELU in the experts' activation. Against llama.cpp, 400 tokens, the experts through a tier of 1 GiB in the card's memory: the most likely token at 315 (the CPU 321), mean |diff| 0.806 (0.817), against the CPU 332 and 0.616; blocks of 4 the same bits as token by token; no error from the validation layer with synchronization. Qwen3-Coder-30B-A3B as before (164 of 200, 2.904). The fast prompt's tiles round group by group: Gemma 4 12B, the most sensitive, in blocks of 64 at 209 of 256 against 234 exact (E4B 251 and 252, Qwen3-4B 233 and 237): to be made exact.++- **Gemma 4 E2B and E4B on the GPU's whole pass**: their per-layer embeddings - the inputs the CPU makes for each layer (the table is read from the file, never on the GPU) go with the block, and after each feed-forward the branch on the GPU, its f32 gate, GELU times the layer's inputs in a shader of its own with the CPU's operations, the f32 projection back, its norm, the output scale after it - and the layers that read another's keys and values (their queries alone); the feed-forwards' widths may differ layer by layer (E2B's 6,144 and 12,288). Against llama.cpp, 400 tokens: E2B Q4_K_M the most likely token at 386 (the CPU 391), mean |diff| 0.427 (0.470), against the CPU 388 and 0.428; E4B 392 (394), 0.281 (0.286), against the CPU 390 and 0.271; blocks of 4 the same bits as token by token, the same as a card with memory of its own, no error from the validation layer with synchronization in either.++- **Gemma 4's dense layers on the GPU's whole pass, and its norms the same bits as the CPU's**: the token's shaders take a layer's own geometry (Gemma 4's sliding-window layers, heads of 256, eight KV heads, a window of 1,024 kept in a ring, a RoPE table of their own; its full layers, sixteen heads of 512 on one KV head, split among four workgroups of four, V as K before its norm), V normalized without weights, scores not scaled, keys and values at sixteen bits as the CPU keeps them (the same cache for both, q in float, the scores float products), the sandwich norms of the attention's and the feed-forward's outputs and the layer's output scale, GELU in the activation's shader with the CPU's operations, the logits' soft cap after. Gemma's prompts move with any rounding of the products' inputs, so the norms now give the CPU's bits: the sum of squares in two floats, the mean rounded once, square root, reciprocal and amax / 127 correctly rounded, quantized as janas_q8k_quantize (the q and k norms, V's, the attention's output too; the experts a card's tier leaves to the CPU quantized the same way). Gemma 4 12B Q4_K_M, 400 tokens against llama.cpp: the most likely token at 367 (336 with the GPU's own rounding; the CPU 362), mean |diff| 1.01 (2.80; the CPU 0.90); blocks of 4 the same bits as token by token, of 64 too with the exact products (235 of 256, 0.97; the fast prompt's tiles 209 and 2.71: to be decided for Gemma); the same as a card with memory of its own; no error from Vulkan's validation layer with synchronization, both ways, tokens and blocks. The common models within the noise, before and after: Qwen3-0.6B 372 and 367 of 400 (mean 0.330 and 0.326; the CPU 375, 0.320), Qwen3.5-0.8B 383 and 381 (0.126, 0.127), Qwen3-4B 362 and 364 (0.733, 0.736), DeepNSM-1B 459 and 465 of 477 (0.086, 0.092), Qwen3-Coder-30B-A3B with a tier of experts 165 and 164 of 200 (2.897, 2.904). Not yet: E2B and E4B (per-layer embeddings, layers sharing their keys and values), the 26B mixture.++- **The query at sixteen bits in the GPU's attention, opt-in (`JANAS_GPU_Q16=1`)**: the GPU's whole pass put q at eight bits for its integer products with the cached keys, a relative error some 4e-3 where the CPU keeps q in float; now, if asked, at sixteen, two eight-bit halves (q = 256 hi + lo), each score two integer products, 256 hi.k + lo.k. Heads up to 256 only (longer ones stay at eight bits: the sum could pass 2^31). Against the CPU, the most likely token the same at 463 of 477 positions instead of 457 and mean |diff| 0.072 instead of 0.090 (DeepNSM-1B Q8_0, as far from the CPU as the CPU is from llama.cpp), against llama.cpp 461 instead of 459; Qwen3-0.6B 234 of 256 instead of 228, 0.158 instead of 0.191; the same with a card with memory of its own (`JANAS_GPU_COPY=1`); no error from Vulkan's validation layer with synchronization checked. The default unchanged until its speed is measured.++## [2026-10-25] - Llama 3, and DeepNSM's explications++- **Llama 3 models, and a LoRA adapter merged into one**: `tools/hf2gguf` reads a Llama 3 checkpoint from Hugging Face (safetensors) and, if given, a PEFT LoRA adapter, merges the adapter (W + alpha/r B A in float, rounded to bf16, as PEFT's merge does on a bf16 model; the adapter's whole token tables, grown for its padding token, in place of the model's) and writes a GGUF of llama.cpp's "llama" architecture, which llama.cpp's own `llama-quantize` quantizes and `gguf2jns` converts: both engines then read the same weights. `gguf2jns` puts Llama's query and key rows back in the checkpoint's order (llama.cpp's converter permutes them for its rotation of adjacent pairs). The engine runs architecture `llama` (no norm on the query and key heads, Llama 3's RoPE factors), its pre-tokenizer (`llama-bpe`: numbers in runs of up to three digits) and its chat format (`<|start_header_id|>`, `<|eot_id|>`, `<|begin_of_text|>` at the start, also of raw text); on the CPU for now. Checked with DeepNSM-1B (Llama-3.2-1B with the adapter of arXiv 2505.11764, which writes a word's explication in the Natural Semantic Metalanguage): the merged weights the same to the bit as an independent merge in numpy, the file from Meta's weights the same byte for byte as from unsloth's copy; the tokens the same as llama.cpp's (476 of 476, digits, spaces, accents, Japanese and special tokens); the most likely token as llama.cpp's at 466 of 477 positions in Q8_0 (469 with 16-bit keys and values, the difference flat along the positions), 456 in Q4_K_M; prompt blocks the same bits as token by token. Its 149 benchmark words, Q8_0 on the laptop's CPU, DeepNSM's sampling: 57.8 tokens/s writing (llama.cpp on the same file and machine: 37.9 on 6 threads, 39.8 on 12), 270 reading the prompts (201, 144); the explications' legality by the paper's measure 5.36 +- 0.12 (llama.cpp 5.22, the paper 5.13), one sample a word, one run each. The GPU's whole pass takes them too since 26 October (it normalized every head of q and k; Llama has no such norms): against llama.cpp the most likely token agrees at 459 of 477 positions in Q8_0 (the CPU 466) and 450 in Q4_K_M (456), the same as a card with memory of its own, no error from Vulkan's validation layer with synchronization checked.+- **`janas-nsm`: explications in the Natural Semantic Metalanguage, counted, and an output head restricted to their words**: the command writes a word's explication with a model trained for it (DeepNSM: one word with its examples, or a batch of JSON lines, the model card's prompt or the paper's, its sampling) and counts it as the paper does - primes, function words and the other words, legality 10 (primes - molecules) / words, circularity (a form of the word in it) - or counts explications given (`--score`, no model). Two lists of primes: the paper's as its scores were computed (without "i" and "said": with them its records do not come back), and Goddard and Wierzbicka's 65 with their forms and phrases; NLTK's function words; the lemma by English suffix rules, a silent e put back, irregular forms. On the paper's 1,341 published records: circularity the same as spaCy's on all of them, primes and molecules on 1,337. `--strict` lets the model write only the primes, the function words, a list of molecules (its own default, or `--molecules FILE`) and punctuation: the library's `janas_llm_chat_vocab` (FreeBASIC `Chat.Vocab`) copies those tokens' rows of the output head into one matrix and the pass computes them alone, the others -inf (`janas_llm_model_set_vocab`; on the CPU, the copies being unknown to the GPU and the NPU; a pass whole on the GPU computes the whole head and is cut to the vocabulary after). The vocabulary's logits the same to the bit as the whole head's, token by token and in blocks, the whole head back unchanged after (`vocab_check`: DeepNSM-1B Q8_0 and Q4_K_M, Gemma 4 E2B with its soft cap; also under asan). DeepNSM-1B Q8_0 on the laptop's CPU, its 149 benchmark words, five seeds each way alternated: 3,841 tokens of 128,257, writing 57.9 to 72.1 tokens/s (the prompts as before, 266), legality 5.33 to 5.48 (+0.15, 95% interval +0.08 to +0.22, paired by word), primes +0.8 points, molecules -0.7, circularity and length within the noise. But the explications tell their word from its neighbours less well: another model, Gemma 4 E4B, ranking the word among four WordNet neighbours (sister senses, common words) from the explication, puts it first for 81.9% of the free explications and 75.8% of the strict ones (-6.0 points, -10.1 to -2.7; nine words lost, none won: leg, child, chat, chair, sour, candle, cut, elbow, sank; one seed, one judge), against 88.6% for WordNet's own glosses and 20.8% for another word's explication (chance 20%). The default writes freely. `test_nsm` counts explications by hand with both lists, UTF-8 punctuation, circularity and the lemma's rules. DeepNSM is CC BY-NC-SA: not shipped, converted by `tools/hf2gguf` for research.+## [2026-10-24] - A card with memory of its own runs the whole model- **A mixture larger than the card's tier: the CPU's experts computed while the GPU computes its own, a round trip less a layer**: where the tier does not hold every expert a pass goes in a submission a layer, the CPU's plan between. The plan's last copies (experts read into the tier, the CPU's rows) went in a submission of their own, waited for: now in the next segment's submission, after a barrier. And a segment is two submissions: the GPU's products of the experts, not waited for, while the CPU computes the experts left to it; then their rows' copies and the rest of the layer, from the experts' sum, waited for (`cpu_part`; one submission as before where the CPU has none). Qwen3-30B-A3B UD-Q3_K_XL as a card of its own on the laptop's Arc, a tier of 1 GiB, 480 tokens one at a time: the CPU's plans from 43.8 to 12.5 ms a pass, the time between segments from 51 to 29, the second half from 9.0 to 9.9 tokens/s. The same logits to the bit; with drafts the same tokens as without, prompt blocks included, and no error of the validation layer, synchronization included. `JANAS_GPU_TOKEN_PROFILE=1` prints a pass's experts in the tier, read into it and computed by the CPU, and the CPU's and the cache's time.diff --git a/README.md b/README.mdindex f5e1fb6..b2e36ce 100644--- a/README.md+++ b/README.md@@ -40,6 +40,18 @@ Janas-LLM's figures are the best of the configurations `janas-bench` measures, a**New: Intel's NPU reads the prompts.** On the same laptop's NPU Janas-LLM reads 1,024 tokens of Qwen3.5-2B at 729 tok/s, of Gemma-4-E2B at 508 and of Qwen3-4B at 333, 1.5 to 2.6 times as fast as without it, with answers as close to llama.cpp's as the CPU's. It takes the NPU by itself where it measures faster, and three options choose by hand: [Intel's NPU](#intels-npu), [Choosing what runs where](#choosing-what-runs-where).+**Long contexts: at 16,384 tokens Janas-LLM writes a reply at 15.0 tok/s, where llama.cpp manages 2.2, and reads the prompt at 56 tok/s, against 31.** Qwen3-4B Q4_K_M on the same laptop, the CPU alone on both sides (12 threads), the same file, 26 October 2026; a prompt read from the start, then 64 tokens written after it:++| Context | reading the prompt: Janas-LLM | llama.cpp | writing after it: Janas-LLM | llama.cpp |+|---|---|---|---|---|+| 1,024 tokens | 118 tok/s | 67 | 27.6 tok/s | 12.5 |+| 4,096 | 96 | 55 | 22.0 | 7.7 |+| 8,192 | 74 | 44 | 18.2 | 4.6 |+| 16,384 | 56 | 31 | 15.0 | 2.2 |+| 32,000 | 35 | - | 10.7 | - |++The longer the context, the wider the gap: 2.2 times as fast writing at 1,024 tokens, 6.9 times at 16,384. Both slow down as the context grows, and on both the attention is what grows: at 32,000 tokens it takes 59 of the 93 ms Janas-LLM spends on a token written, and the 32,000 tokens of prompt take 15 minutes to read. Janas-LLM's peak memory went from 2.6 GiB at 1,024 tokens to 4.9 at 32,000. The conditions are the same wherever possible; where not: Janas-LLM keeps the keys and values at eight bits, llama.cpp at sixteen (its default); llama.cpp was not run at 32,000 tokens (almost another hour). How it was measured: [below](#how-the-figures-were-measured).+**Try it** on Linux x86-64 (Ubuntu 22.04, Debian 12 and later):```sh@@ -147,6 +159,8 @@ What Janas-LLM runs today, model by model and feature by feature: [docs/featuresThe table at the top, on 21 October 2026: `janas-bench --prompt 1024 --gen 16 --rounds 2 --no-mtp --settle 300`, which measures each configuration the engine may choose (6, 12 and 20 threads, with the GPU in the shorter passes or not, with the NPU or not), the best of each column taken, against llama.cpp (build `2b18470` of 18 September 2026) with its Vulkan backend on the same integrated GPU, everything on it (`llama-bench -p 1024 -n 16 -r 2 -t 12 -ngl 99`), on the same files the same day. The writing columns are without drafts on both sides. The two tools do not measure in quite the same way (`llama-bench` writes without a prompt before it, `janas-bench` after the 1,024 tokens), and it is one machine. The conditions are the same wherever the two engines allow it, and where they are not, it is said.+The long contexts, on 26 October 2026: Janas-LLM through `tests/bench_long` (`JANAS_GPU=0`, 12 threads on CPUs 0-11, a 4 GiB expert cache), a prompt of n tokens read in blocks from position 0, then 64 written greedily one at a time; llama.cpp (build `2b18470`, CPU backend, `-t 12 -ngl 0`) with `llama-bench -p n -n 0` for reading and `-p 0 -n 64 -d n` for writing after n tokens; the same Qwen3-4B Q4_K_M file (Qwen's own GGUF), the same tokens (English Wikipedia articles, C sources and Italian prose, mixed), one run each, ten seconds apart, the machine otherwise idle.+Qwen3-Next-80B-A3B and Qwen3-Coder-Next are larger than the laptop's memory. llama.cpp with its usual build ran out of memory as soon as the GPU took part, even with the experts left on the CPU (`-ngl 99 -ncmoe 48`): its CPU backend copies the weights into memory of its own to repack them, and the Vulkan backend copies those it leaves to the CPU into buffers the system cannot reclaim. It ran with a build without repacking (`GGML_CPU_REPACK=OFF`) and `--no-host 1 -lm mmap`, which leave the experts in the file, read through the page cache: 17.7 tok/s reading with the GPU, 3.0 writing on the CPU alone (1.8 with the GPU; its CPU backend read at 12.6). Janas-LLM ran them with no option of its own, the experts it uses most in a cache of 21 GiB and the rest read from the disk.**The drafts** are tokens guessed ahead that the model checks, all at once: the same text, sooner. They come from the model's own prediction block (the Qwen3.5 and Qwen3.6 models, Qwen3-Next) or from Gemma 4's assistant, a small drafting model of its own; the whole of it is under [`--mtp`](docs/chat.md). Measured on 10 October 2026 by `janas-bench` on a chat reply - an ordinary question in English ("explain how bread is made at home"), 64 tokens of the answer, sampled as the chat samples with one fixed seed, after a prompt of 256 tokens had warmed the expert cache, in the configuration the engine chose (on Qwen3-Next-80B-A3B 20 threads for every pass) - the reply came at 37 tok/s from Qwen3-Next-80B-A3B, 34 from Qwen3.6-35B-A3B, 26 from Gemma-4-26B-A4B, 14 from Gemma-4-12B, 24 from Qwen3.5-9B, 32 from Gemma-4-E4B, 52 from Gemma-4-E2B and 67 from Qwen3.5-2B. In the chat, on 21 October, the same question brought Qwen3-Next-80B-A3B's reply at 31-35 tok/s with the cache warm (31.4 the median of five seeds, its single tokens on the 12 threads the engine chose there) and at 16-20 tok/s as the first reply of a session, its experts still coming from the disk. What the drafts are worth depends on the text: there 74-89% of them were kept on the Qwen models and on Gemma's two larger ones, 63-72% on E2B and E4B, while on Italian prose Qwen3.5-2B kept about a third and gained next to nothing; and one round of 64 tokens is noisy, a tenth either way. Qwen3-30B-A3B and the dense Qwen3 have no such block (Qwen3-4B can take Qwen3-0.6B as its draft, `--draft`); the dense Qwen3.5 have one in their original checkpoint, which `hf2jns_mtp` converts ([MODELS.md](MODELS.md#the-fingerprints)).diff --git a/build.sh b/build.shindex 46fad60..a68024d 100755--- a/build.sh+++ b/build.sh@@ -59,7 +59,8 @@ Outputsrc/services/<tool>/main.c becomes janas-<tool>, withthe other .c files of its directory; the libraries of others insrc/third_party/ go into the programs that use them; every other .c- under src/ goes into the library. The release build+ under src/ goes into the library. Janet (src/janet/, Linux x86-64)+ gives janet, janet-check, janet-mnist and janet-mnist-deep. The release buildalso links libjanas_llm.so and libjanas_mcp.so, which export the publicAPIs of include/janas/llm.h and include/janas/mcp.h and nothing else.On Linux src/node/ becomes janas.node, the module for Node.js, with thediff --git a/docs/tools.md b/docs/tools.mdindex 43418dd..5f36595 100644--- a/docs/tools.md+++ b/docs/tools.md@@ -3,6 +3,7 @@| | ||---|---|| `tools/gguf2jns` | GGUF → the `.jns` format the engine reads |+| `tools/hf2gguf` | a Llama 3 checkpoint from Hugging Face (safetensors), with a PEFT LoRA adapter merged into it if given, → a bf16 GGUF that llama.cpp's `llama-quantize` quantizes and `gguf2jns` converts || `tools/hf2jns_mtp` | the multi-token prediction block, from the original checkpoint: its shards, or just the block's tensors from Hugging Face (`hf:<owner>/<repo>`); or from a GGUF file of the block alone (unsloth's `mtp-*.gguf` of Qwen3.8-Flash-Next), copied as it is || `tools/jns_planes` | rewrites a model with the experts' down matrix in bit planes, so a machine short of memory can read part of it || `jns_check` | checks a model file, with `--verify` every expert's checksum |@@ -11,6 +12,23 @@| `tools/gen_geo` | the services' tables of airports, cities and seas, from OurAirports, GeoNames and Natural Earth || `mcp_fake_server` | an MCP server over stdio in either era of the protocol, for trying a client without anybody else's server || `llm_eval` | compares the engine's logits against a reference dump |+| `vocab_check` | the output head restricted to a vocabulary (`janas_llm_chat_vocab`) against the whole head: the same logits to the bit || `bench_kernels`, `bench_gemm`, `bench_attn`, `bench_long`, ... | the pieces measured on their own |They are built into `bin/x86_64-linux/` along with everything else.++## janas-nsm++Explications in the Natural Semantic Metalanguage (NSM: a word's meaning paraphrased with 65 semantic primes - someone, something, do, happen, want, know, good, before... - and a few simpler words) written by a model trained for it, such as DeepNSM ([arXiv 2505.11764](https://arxiv.org/abs/2505.11764)), and counted as the paper counts them.++```+janas-nsm <model.jns> <word> ["example sentence" ...] one word+janas-nsm <model.jns> --batch words.jsonl --out out.jsonl "word" and "examples" a line+janas-nsm --score explications.jsonl "word" and "explication" a line, no model+```++Each explication gets its words, the share of primes, of function words and of the other words ("molecules"), the paper's legality, 10 (primes - molecules) / words, and whether it is circular (a form of the word itself in it); a batch ends with the means and their standard errors, and the speed. `--lexicon paper` (the default) counts with the list the paper's scores were computed with, single words; `--lexicon full` with Goddard and Wierzbicka's 65 primes, their inflected forms and phrases ("a long time", "don't want"). On the paper's 1,341 published records the counts give its figures (circularity on all 1,341, primes and molecules on 1,337).++`--strict` lets the model write only the primes, the function words, a list of molecules (`--molecules FILE`, one word a line; a default list of body parts, physical qualities, nature, people, everyday actions and things) and punctuation: some 3,800 tokens of Llama 3's 128,257, whose rows alone the output head computes (`janas_llm_chat_vocab` in the library, `Chat.Vocab` in FreeBASIC). DeepNSM-1B Q8_0 on the laptop's CPU, the paper's 149 benchmark words, five seeds each way: writing 57.9 to 72.1 tokens/s, legality 5.33 to 5.48 (+0.15, 95% interval +0.08 to +0.22). The explications tell their word from its neighbours less well, though: judged by another model (Gemma 4 E4B, which ranks the word among four WordNet neighbours from the explication), 81.9% to 75.8% of the words ranked first, -6.0 points (-10.1 to -2.7; one seed, one judge), lost on concrete things and body parts (leg, chair, candle, elbow). The default writes freely.++DeepNSM's weights, code and data are CC BY-NC-SA 4.0: Janas does not ship them; `tools/hf2gguf` converts the model from its Hugging Face repository and Llama-3.2-1B's (Meta's licence), for research.diff --git a/include/janas/janas_oop.bi b/include/janas/janas_oop.biindex efd31a7..7a4f1da 100644--- a/include/janas/janas_oop.bi+++ b/include/janas/janas_oop.bi@@ -293,6 +293,8 @@ public:declare function ToolCall(byval i as long, byref toolName as string, _byref args as string) as booleandeclare function LogitBias(ids() as long, bias() as single) as boolean+ '' the tokens the replies may have (an empty array: all of them)+ declare function Vocab(ids() as long) as boolean'' the tokens of the reply (options.logprobs = 1 for their'' log-probabilities, options.top_logprobs for the alternatives)declare const property Tokens() as long@@ -1204,6 +1206,19 @@ function Chat.LogitBias(ids() as long, bias() as single) as booleanreturn Done_()end function+function Chat.Vocab(ids() as long) as boolean+ if Closed() then return false+ dim as long n = ubound(ids) - lbound(ids) + 1+ dim as long rc+ if n <= 0 then+ rc = janas_llm_chat_vocab(hs->chat, 0, 0)+ else+ rc = janas_llm_chat_vocab(hs->chat, n, @ids(lbound(ids)))+ end if+ if rc <> JANAS_LLM_OK then return Fail(rc)+ return Done_()+end function+property Chat.Tokens() as longif hs = 0 orelse hs->chat = 0 then return 0return janas_llm_chat_tokens(hs->chat)diff --git a/include/janas/llm.bi b/include/janas/llm.biindex 40bd9b2..7dbf77f 100644--- a/include/janas/llm.bi+++ b/include/janas/llm.bi@@ -393,6 +393,11 @@ declare function janas_llm_chat_logit_bias(byval c as janas_llm_chat ptr, _byval n as long, _byval ids as const long ptr, _byval bias as const single ptr) as long+'' The tokens the replies may have (n 0: all of them again); the model+'' computes their logits alone.+declare function janas_llm_chat_vocab(byval c as janas_llm_chat ptr, _+ byval n as long, _+ byval ids as const long ptr) as long'' The tokens of the reply: id, part (JANAS_LLM_PART_*), log-probability,'' bytes; and the most likely alternatives in each one's place.declare function janas_llm_chat_tokens(byval c as const janas_llm_chat ptr) as longdiff --git a/include/janas/llm.h b/include/janas/llm.hindex 876b1b8..6a90e76 100644--- a/include/janas/llm.h+++ b/include/janas/llm.h@@ -603,6 +603,19 @@ JANAS_LLM_API int32_t janas_llm_chat_logit_bias(janas_llm_chat *c, int32_t n,const int32_t *ids,const float *bias);+/*+ * The tokens the replies may have, from the next token on, n of them; n of 0+ * gives back the whole vocabulary. The format's tokens that end a reply are+ * added. The model computes the logits of these tokens alone - their rows+ * of the output head copied into one matrix, the other logits -inf - so a+ * small vocabulary is also faster: the head is the model's largest matrix+ * read whole at every token (janas-nsm's --strict: the words of the Natural+ * Semantic Metalanguage). Text of tokens outside it cannot appear; words+ * made of tokens inside it can.+ */+JANAS_LLM_API int32_t janas_llm_chat_vocab(janas_llm_chat *c, int32_t n,+ const int32_t *ids);+/** The tokens of the reply so far, one by one: janas_llm_chat_tokens counts* them; janas_llm_chat_token gives token i - its id, which part of the replydiff --git a/src/common/env_names.c b/src/common/env_names.cindex 3eff4f3..4cbd17e 100644--- a/src/common/env_names.c+++ b/src/common/env_names.c@@ -38,6 +38,7 @@ const char *const JANAS_ENV_NAMES[] = {"JANAS_ALPHAVANTAGE_KEY","JANAS_EXPTRACE","JANAS_FLIGHTS_CONFIG","JANAS_GPU",+ "JANAS_GPU_ACT_FLOAT","JANAS_GPU_ALLOC","JANAS_GPU_ALWAYS","JANAS_GPU_ATT_SG",@@ -51,6 +52,7 @@ const char *const JANAS_ENV_NAMES[] = {"JANAS_ALPHAVANTAGE_KEY","JANAS_GPU_SHORT","JANAS_GPU_NV","JANAS_GPU_KEEPALIVE_US",+ "JANAS_GPU_LONG","JANAS_GPU_MIN_N","JANAS_GPU_FAST_PROMPT","JANAS_GPU_MOE",@@ -63,6 +65,8 @@ const char *const JANAS_ENV_NAMES[] = {"JANAS_ALPHAVANTAGE_KEY","JANAS_GPU_EXPERT_SHARE","JANAS_GPU_NO_ROW","JANAS_GPU_NOSPIN",+ "JANAS_GPU_Q16",+ "JANAS_GPU_TILES_EXACT","JANAS_GPU_REPACK","JANAS_GPU_REPEATABLE","JANAS_GPU_ROW_NV",diff --git a/src/llm/api.c b/src/llm/api.cindex 325427e..2ff1236 100644--- a/src/llm/api.c+++ b/src/llm/api.c@@ -140,7 +140,7 @@ static int32_t find_format_in(janas_llm *llm, const struct janas_jns *j,llm->im_start = janas_tokenizer_find(llm->tok, "<|im_start|>");llm->im_end = janas_tokenizer_find(llm->tok, "<|im_end|>");llm->asst_role = "assistant";- llm->bos = llm->think_on = -1;+ llm->bos = llm->think_on = llm->hdr_end = -1;llm->quote = llm->decl_open = llm->decl_close = -1;int32_t turn_open = janas_tokenizer_find(llm->tok, "<|turn>");int32_t turn_close = janas_tokenizer_find(llm->tok, "<turn|>");@@ -181,6 +181,25 @@ static int32_t find_format_in(janas_llm *llm, const struct janas_jns *j,llm->tools = JANAS_TOOLS_NONE;if (llm->tools != JANAS_TOOLS_NONE)llm->stop[llm->n_stop++] = llm->resp_open;+ } else if (janas_tokenizer_find(llm->tok, "<|start_header_id|>") >= 0 &&+ janas_tokenizer_find(llm->tok, "<|end_header_id|>") >= 0 &&+ janas_tokenizer_find(llm->tok, "<|eot_id|>") >= 0 &&+ janas_tokenizer_find(llm->tok, "<|begin_of_text|>") >= 0 &&+ (tmpl[0] == 0 || strstr(tmpl, "<|start_header_id|>"))) {+ /* Llama 3, a base model too: no reasoning, no tools yet */+ llm->format = FORMAT_LLAMA3;+ llm->im_start = janas_tokenizer_find(llm->tok, "<|start_header_id|>");+ llm->hdr_end = janas_tokenizer_find(llm->tok, "<|end_header_id|>");+ llm->im_end = janas_tokenizer_find(llm->tok, "<|eot_id|>");+ llm->bos = janas_tokenizer_find(llm->tok, "<|begin_of_text|>");+ llm->stop[llm->n_stop++] = llm->im_end;+ int32_t eom = janas_tokenizer_find(llm->tok, "<|eom_id|>");+ if (eom >= 0)+ llm->stop[llm->n_stop++] = eom;+ llm->think_open = llm->think_close = -1;+ llm->call_open = llm->call_close = -1;+ llm->resp_open = llm->resp_close = -1;+ llm->tools = JANAS_TOOLS_NONE;} else if (llm->im_start >= 0 && llm->im_end >= 0 &&(tmpl[0] == 0 || strstr(tmpl, "<|im_start|>"))) {llm->format = FORMAT_CHATML;@@ -555,9 +574,10 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,janas_jns_meta_str(&j, "general.name", name, sizeof(name));janas_jns_meta_str(&j, "general.architecture", arch, sizeof(arch));int32_t rc = 0;- /* byte-level BPE with Qwen's pre-tokenizers, or Gemma 4's */+ /* byte-level BPE with Qwen's or Llama 3's pre-tokenizers, or Gemma 4's */int known = (strcmp(kind, "gpt2") == 0 &&- (strcmp(pre, "qwen2") == 0 || strcmp(pre, "qwen35") == 0)) ||+ (strcmp(pre, "qwen2") == 0 || strcmp(pre, "qwen35") == 0 ||+ strcmp(pre, "llama-bpe") == 0)) ||strcmp(kind, "gemma4") == 0;if (!known)rc = janas_api_fail(JANAS_LLM_EMODEL, "tokenizer %s/%s not supported",diff --git a/src/llm/api_chat.c b/src/llm/api_chat.cindex d97e64c..aa8a3c7 100644--- a/src/llm/api_chat.c+++ b/src/llm/api_chat.c@@ -246,12 +246,25 @@ int janas_api_add_special(janas_llm_chat *c, int32_t id)return 0;}+/* The newline after a turn's end: ChatML's and Gemma's, none in Llama 3. */+int janas_api_end_turn_nl(janas_llm_chat *c)+{+ return c->llm->format == FORMAT_LLAMA3 ? 0 : janas_api_add_text(c, "\n", 1);+}+/* One message of the conversation, in the model's format. */int janas_api_add_message(janas_llm_chat *c, const char *role, const char *text,size_t n){const janas_llm *llm = c->llm;switch (llm->format) {+ case FORMAT_LLAMA3:+ return janas_api_add_special(c, llm->im_start) ||+ janas_api_add_text(c, role, strlen(role)) ||+ janas_api_add_special(c, llm->hdr_end) ||+ janas_api_add_text(c, "\n\n", 2) ||+ janas_api_add_text(c, text, n) ||+ janas_api_add_special(c, llm->im_end);case FORMAT_CHATML:default:return janas_api_add_special(c, llm->im_start) ||@@ -259,11 +272,12 @@ int janas_api_add_message(janas_llm_chat *c, const char *role, const char *text,janas_api_add_text(c, "\n", 1) ||janas_api_add_text(c, text, n) ||janas_api_add_special(c, llm->im_end) ||- janas_api_add_text(c, "\n", 1);+ janas_api_end_turn_nl(c);}}-/* The start of the sequence: Gemma's <bos>, nothing for ChatML. */+/* The start of the sequence: Gemma's <bos>, Llama 3's <|begin_of_text|>,+ nothing for ChatML. */int janas_api_begin_sequence(janas_llm_chat *c){return c->llm->bos >= 0 ? janas_api_add_special(c, c->llm->bos) : 0;@@ -292,6 +306,12 @@ int janas_api_start_reply(janas_llm_chat *c)janas_api_add_text(c, "thought\n", 8) ||janas_api_add_special(c, llm->think_close) ||janas_api_flush_text(c);+ case FORMAT_LLAMA3:+ c->in_think = 0;+ return janas_api_add_special(c, llm->im_start) ||+ janas_api_add_text(c, "assistant", 9) ||+ janas_api_add_special(c, llm->hdr_end) ||+ janas_api_add_text(c, "\n\n", 2) || janas_api_flush_text(c);case FORMAT_CHATML:default:if (janas_api_add_special(c, llm->im_start) ||@@ -419,7 +439,7 @@ int janas_api_add_system(janas_llm_chat *c, const char *sys, size_t n)janas_api_add_text(c, "\n", 1))) ||janas_api_add_markup(c, b.p, b.n) ||janas_api_add_special(c, llm->im_end) ||- janas_api_add_text(c, "\n", 1);+ janas_api_end_turn_nl(c);janas_buf_free(&b);return err ? -1 : 0;}@@ -455,7 +475,7 @@ int janas_api_add_system(janas_llm_chat *c, const char *sys, size_t n)janas_api_add_text(c, "system\n", 7) ||janas_api_add_markup(c, b.p, b.n) ||janas_api_add_special(c, llm->im_end) ||- janas_api_add_text(c, "\n", 1) || janas_api_flush_text(c);+ janas_api_end_turn_nl(c) || janas_api_flush_text(c);janas_buf_free(&b);if (!err && c->n_ids - ids0 > plain)c->tools_tokens = (uint32_t)(c->n_ids - ids0 - plain);@@ -498,7 +518,7 @@ static int recap(janas_llm_chat *c)int err = 0;if (!c->closed)err |= janas_api_add_special(c, c->llm->im_end);- err |= janas_api_add_text(c, "\n", 1);+ err |= janas_api_end_turn_nl(c);err |= janas_api_add_message(c, "user", RECAP_ASK, strlen(RECAP_ASK));err |= janas_api_start_reply(c);if (!err && janas_llm_session_append(c->s, c->ids, (uint32_t)c->n_ids) == 0)@@ -842,7 +862,7 @@ static int32_t send_turn(janas_llm_chat *c, const char *text, size_t n,it was cut and gets one now; then the newline after it */if (!c->closed)err |= janas_api_add_special(c, c->llm->im_end);- err |= janas_api_add_text(c, "\n", 1);+ err |= janas_api_end_turn_nl(c);} else {/* the start: <bos> where the format has one, the system message */err |= janas_api_begin_sequence(c);@@ -866,7 +886,7 @@ static int32_t send_turn(janas_llm_chat *c, const char *text, size_t n,err |= janas_api_add_markup(c, "\n</tool_response>", 17);}err |= janas_api_add_special(c, c->llm->im_end) ||- janas_api_add_text(c, "\n", 1);+ janas_api_end_turn_nl(c);} else {err |= janas_api_add_message(c, "user", text, n);}diff --git a/src/llm/api_load.c b/src/llm/api_load.cindex 9b73da8..9288ec2 100644--- a/src/llm/api_load.c+++ b/src/llm/api_load.c@@ -141,6 +141,8 @@ static int add_ids(janas_llm_chat *c, const int32_t *ids, uint32_t n)static int add_assistant(janas_llm_chat *c, const struct cmsg *m, int after_q){const janas_llm *llm = c->llm;+ if (llm->format == FORMAT_LLAMA3) /* no reasoning, no tools */+ return janas_api_add_message(c, llm->asst_role, m->text, m->n) ? -1 : 0;enum janas_tool_dialect d = llm->tools;int err = janas_api_add_special(c, llm->im_start);/* Qwen3-Coder writes the calls right after the role */@@ -159,7 +161,7 @@ static int add_assistant(janas_llm_chat *c, const struct cmsg *m, int after_q)err |= janas_api_add_text(c, m->text, m->n);}err |= janas_api_add_special(c, llm->im_end);- err |= janas_api_add_text(c, "\n", 1);+ err |= janas_api_end_turn_nl(c);return err ? -1 : 0;}@@ -195,8 +197,8 @@ static size_t add_gemma_turn(janas_llm_chat *c, size_t i, int reopen, int *open)j < c->n_msgs && c->msgs[j].role == JANAS_LLM_ROLE_ASSISTANT;*open = next_model || (j == c->n_msgs && k > 0);if (!*open)- err |= janas_api_add_special(c, llm->im_end) ||- janas_api_add_text(c, "\n", 1);+ err |=+ janas_api_add_special(c, llm->im_end) || janas_api_end_turn_nl(c);return err ? (size_t)-1 : j - 1;}@@ -254,7 +256,7 @@ static int render(janas_llm_chat *c, uint32_t *reply_off)err |= add_ids(c, mm->ids, mm->n);if (!mm->closed)err |= janas_api_add_special(c, llm->im_end);- err |= janas_api_add_text(c, "\n", 1);+ err |= janas_api_end_turn_nl(c);} else {err |= add_assistant(c, m, i > last_q);}@@ -276,7 +278,7 @@ static int render(janas_llm_chat *c, uint32_t *reply_off)err |= janas_api_add_markup(c, "\n</tool_response>", 17);if (last)err |= janas_api_add_special(c, llm->im_end) ||- janas_api_add_text(c, "\n", 1);+ janas_api_end_turn_nl(c);break;}}@@ -437,7 +439,9 @@ int32_t janas_llm_chat_prompt(janas_llm_chat *c, const char *text, int32_t len)c->n_ids = c->n_text = 0;c->n_turn = 0;c->sys_tokens = 0;- if (janas_api_add_text(c, text, janas_api_text_len(text, len)) != 0)+ /* Llama 3's text starts with <|begin_of_text|>, as its tokenizer adds */+ if ((c->llm->format == FORMAT_LLAMA3 && janas_api_begin_sequence(c)) ||+ janas_api_add_text(c, text, janas_api_text_len(text, len)) != 0)return janas_api_fail(JANAS_LLM_ENOMEM, "out of memory");c->in_think = 0;return load_ids(c, UINT32_MAX);diff --git a/src/llm/api_tools.c b/src/llm/api_tools.cindex b0c8dd8..8669390 100644--- a/src/llm/api_tools.c+++ b/src/llm/api_tools.c@@ -198,6 +198,32 @@ int32_t janas_llm_chat_logit_bias(janas_llm_chat *c, int32_t n,return JANAS_LLM_OK;}+int32_t janas_llm_chat_vocab(janas_llm_chat *c, int32_t n, const int32_t *ids)+{+ if (!c || n < 0 || (n > 0 && !ids))+ return janas_api_fail(JANAS_LLM_EINVAL, "invalid argument");+ const janas_llm *llm = c->llm;+ if (n == 0) {+ janas_llm_model_set_vocab(llm->m, NULL, 0);+ return JANAS_LLM_OK;+ }+ /* the format's ends of a reply too, or none could end */+ int32_t *all = malloc(((size_t)n + JANAS_GEN_MAX_STOP) * sizeof(int32_t));+ if (!all)+ return janas_api_fail(JANAS_LLM_ENOMEM, "out of memory");+ memcpy(all, ids, (size_t)n * sizeof(int32_t));+ for (int i = 0; i < llm->n_stop; i++)+ all[n + i] = llm->stop[i];+ int rc =+ janas_llm_model_set_vocab(llm->m, all, (uint32_t)(n + llm->n_stop));+ free(all);+ if (rc != 0)+ return janas_api_fail(JANAS_LLM_EMODEL,+ "this model's output head cannot be restricted "+ "(its weights in bit planes), or out of memory");+ return JANAS_LLM_OK;+}+/* ---- what holds a reply ---- *//*diff --git a/src/llm/arch.c b/src/llm/arch.cindex d4bfab1..3709eef 100644--- a/src/llm/arch.c+++ b/src/llm/arch.c@@ -9,6 +9,9 @@static const struct janas_arch archs[] = {{.name = "qwen3", .rec = JANAS_REC_NONE},+ /* Llama 3: Qwen3's dense layers without the heads' norms; its RoPE+ factors (rope_freqs, rope_scaling of type llama3) as Gemma 4's */+ {.name = "llama", .rec = JANAS_REC_NONE, .no_qk_norm = 1},{.name = "qwen3moe", .rec = JANAS_REC_NONE},{.name = "qwen3next",.rec = JANAS_REC_NEXT,diff --git a/src/llm/arch.h b/src/llm/arch.hindex 05e9e84..957e665 100644--- a/src/llm/arch.h+++ b/src/llm/arch.h@@ -36,6 +36,9 @@ struct janas_arch {weights, V = K where attn_v is missing, scores notscaled (the q and k norms do it) */int kv16; /* keys and values kept at sixteen bits, not eight */+ /* Llama 3: no norm on the heads of q and k; the file's queries and keys+ in the checkpoint's row order (gguf2jns), rotated by halves */+ int no_qk_norm;/* Qwen3.8-Flash-Next (qwen4exp, model_hc.c): hyper-connections - theresidual in several streams, mixed into each sublayer's input by agate of low rank, in place of its norm, and each sublayer's outputdiff --git a/src/llm/attention.c b/src/llm/attention.cindex 88f9d26..9539721 100644--- a/src/llm/attention.c+++ b/src/llm/attention.c@@ -10,6 +10,7 @@#include <stdatomic.h>#include <stdlib.h>#include <string.h>+#include <time.h>#if defined(__x86_64__)#include <immintrin.h>@@ -45,6 +46,14 @@ struct janas_attn {int ring; /* position p in row p mod n_ctx (with a window) */uint32_t window; /* a token sees the last window positions; 0: all */atomic_uint *done; /* chunks finished, per (KV head, subgroup) */+ /* the engine's choice (scores_auto): the two ways measured on the+ calls themselves, a token's and a block's apart, every TUNE_BAND+ positions (attn_tune_*) */+ struct {+ uint32_t band, calls;+ double t[2]; /* seconds over the calls each way: float, int16 */+ int pick;+ } tune[2];};struct attn_job {@@ -919,13 +928,10 @@ static void attn_worker(void *arg, int tid, int n_threads)}/*- * Which way the scores go, when the caller leaves it open: the sixteen-bit- * query where the arithmetic for it is there and the caller asked for more- * context than the usual sixteen thousand tokens. A context is a ceiling,- * not a promise: a conversation that stays short pays the query's- * quantization for a gain it never sees, so the engine waits to be asked for- * a long one - there attention is the greater part of a token, and the- * sixteen-bit query takes a fifth to a quarter off it.+ * Which way the scores go: asked by the caller (or JANAS_ATTN), else the+ * engine's, measured on the calls (attn_tune_pick) - this, the way until+ * the first measures: the sixteen-bit query where the arithmetic for it is+ * there and the context is long.*/static int scores_choice(int asked, uint32_t head_dim, uint32_t n_ctx,int *chosen_here)@@ -979,13 +985,18 @@ struct janas_attn *janas_attn_create(uint32_t n_head, uint32_t n_head_kv,A->scr = malloc((size_t)n_threads * (JANAS_ATTN_MAX_BLOCK * group + 2) *ATT_CHUNK * sizeof(float));A->done = malloc(n_head * sizeof(*A->done));- if (A->scores == JANAS_ATTN_INT16) {+ /* the sixteen-bit query's room wherever it may be used: chosen by the+ engine, it can change from call to call */+ int i16 =+ A->scores == JANAS_ATTN_INT16 || (A->scores_auto && head_dim % 16 == 0);+ if (i16) {A->qi = malloc((size_t)JANAS_ATTN_MAX_BLOCK * n_head * head_dim *sizeof(int16_t));A->qs = malloc((size_t)JANAS_ATTN_MAX_BLOCK * n_head * sizeof(float));}- if (!A->part || !A->scr || !A->done ||- (A->scores == JANAS_ATTN_INT16 && (!A->qi || !A->qs))) {+ for (int c = 0; c < 2; c++)+ A->tune[c].band = UINT32_MAX;+ if (!A->part || !A->scr || !A->done || (i16 && (!A->qi || !A->qs))) {janas_attn_destroy(A);return NULL;}@@ -1044,11 +1055,66 @@ void janas_attn_destroy(struct janas_attn *A)free(A);}+/*+ * The engine's choice of the scores' way, made on this machine: in float+ * over the query as it is, or over the query at sixteen bits (an integer+ * sum, a relative 3e-5 of error where the eight-bit keys carry 4e-3: the+ * most likely token the same at 2,961 of 3,200 positions against llama.cpp+ * as at 2,963 in float, Qwen3 0.6B and 4B, Qwen3.5 0.8B and 2B, two texts+ * each, 26 Oct 2026). Which is faster depends on the processor (AVX-VNNI+ * or not, its cores) and grows with the position, so every TUNE_BAND+ * positions TUNE_TRY calls each way, a token's calls and a block's apart+ * (the first way of a band alternating), and the faster for the rest of+ * the band. JANAS_ATTN=float or int16 fixes it.+ */+#define TUNE_BAND (16 * ATT_CHUNK)+#define TUNE_TRY 8++static double attn_now(void)+{+ struct timespec ts;+ clock_gettime(CLOCK_MONOTONIC, &ts);+ return (double)ts.tv_sec + ts.tv_nsec * 1e-9;+}++/* this call's way (JANAS_ATTN_FLOAT or _INT16) */+static int attn_tune_pick(struct janas_attn *A, int c, uint32_t pos0)+{+ uint32_t band = pos0 / TUNE_BAND;+ if (band != A->tune[c].band) {+ A->tune[c].band = band;+ A->tune[c].calls = 0;+ A->tune[c].t[0] = A->tune[c].t[1] = 0.0;+ }+ uint32_t k = A->tune[c].calls;+ if (k >= 2 * TUNE_TRY)+ return A->tune[c].pick;+ int first = (int)(band & 1u), way = k < TUNE_TRY ? first : !first;+ return way ? JANAS_ATTN_INT16 : JANAS_ATTN_FLOAT;+}++static void attn_tune_seen(struct janas_attn *A, int c, int mode, double s)+{+ if (A->tune[c].calls >= 2 * TUNE_TRY)+ return;+ A->tune[c].t[mode == JANAS_ATTN_INT16] += s;+ if (++A->tune[c].calls == 2 * TUNE_TRY)+ A->tune[c].pick = A->tune[c].t[1] < A->tune[c].t[0] ? JANAS_ATTN_INT16+ : JANAS_ATTN_FLOAT;+}+void janas_attn_run(struct janas_attn *A, struct janas_pool *pool,const float *q, const int8_t *k, const float *ks,const int8_t *v, const float *vs, uint32_t n, uint32_t pos0,float *out){+ /* the engine's choice: measured (attn_tune_pick) */+ int tc = n > 1, tuned = A->scores_auto && A->qi && !A->kv16;+ double t0 = 0.0;+ if (tuned) {+ A->scores = attn_tune_pick(A, tc, pos0);+ t0 = attn_now();+ }/* enough items for every thread: split the query heads of a group */uint32_t n_chunks = (pos0 + n + ATT_CHUNK - 1) / ATT_CHUNK, split = 1;uint32_t group = A->n_head / A->n_kv;@@ -1106,4 +1172,6 @@ void janas_attn_run(struct janas_attn *A, struct janas_pool *pool,janas_pool_run(pool, attn_worker, &job);if (!job.fuse)janas_pool_run(pool, merge_worker, &job);+ if (tuned)+ attn_tune_seen(A, tc, A->scores, attn_now() - t0);}diff --git a/src/llm/chat.h b/src/llm/chat.hindex 573d6ea..a767cfc 100644--- a/src/llm/chat.h+++ b/src/llm/chat.h@@ -26,6 +26,11 @@ enum chat_format {"model", the reasoning in <|channel>thought\n...<channel|>:ChatML's shape, so im_start and im_end hold <|turn> and <turn|> */FORMAT_GEMMA,+ /* Llama 3: <|begin_of_text|> once, then <|start_header_id|>role+ <|end_header_id|>\n\ntext<|eot_id|>, nothing after: ChatML's shape,+ im_start and im_end holding <|start_header_id|> and <|eot_id|>, the+ role closed by hdr_end */+ FORMAT_LLAMA3,};struct janas_llm {@@ -35,6 +40,7 @@ struct janas_llm {int32_t im_start, im_end;const char *asst_role; /* "assistant", or Gemma's "model" */int32_t bos; /* written once at the start, or -1 */+ int32_t hdr_end; /* Llama 3's <|end_header_id|>, or -1 */int32_t think_on; /* Gemma's <|think|>: reasoning asked for, or -1 */int32_t quote, decl_open, decl_close; /* Gemma's <|"|>, <|tool>, <tool|> */int32_t think_open, think_close; /* reasoning markers, or -1 */@@ -252,6 +258,8 @@ int janas_api_rtext_add(janas_llm_chat *c, const char *s, size_t n);int janas_api_add_text(janas_llm_chat *c, const char *s, size_t n);int janas_api_flush_text(janas_llm_chat *c);int janas_api_add_special(janas_llm_chat *c, int32_t id);+/* The newline after a turn's end where the format has one. */+int janas_api_end_turn_nl(janas_llm_chat *c);int janas_api_add_message(janas_llm_chat *c, const char *role, const char *text,size_t n);int janas_api_add_system(janas_llm_chat *c, const char *sys, size_t n);diff --git a/src/llm/gpu.c b/src/llm/gpu.cindex ef3a8ca..982e225 100644--- a/src/llm/gpu.c+++ b/src/llm/gpu.c@@ -469,6 +469,27 @@ uint32_t janas_g_make_pipeline_sg(struct janas_gpu *g, const uint32_t *code,: 0;}+/* JANAS_GPU_COPY=1: an integrated GPU taken as a card of its own (its+ value, not its presence: =0 as without) */+int janas_g_copy_asked(void)+{+ static int on = -1;+ if (on < 0) {+ const char *e = getenv("JANAS_GPU_COPY");+ on = e && atoi(e) > 0;+ }+ return on;+}++/* JANAS_GPU_TILES_EXACT=1: the fast prompt's Q4_K and Q6_K tiles summed+ as the CPU sums, a block at a time (q4k_mmq.comp, q6k_mmq.comp EXACT),+ being measured */+static int tiles_exact(void)+{+ const char *e = getenv("JANAS_GPU_TILES_EXACT");+ return e && atoi(e) > 0;+}+struct janas_gpu *janas_gpu_create(char *err, size_t err_len){struct janas_gpu *g = calloc(1, sizeof(*g));@@ -613,14 +634,14 @@ struct janas_gpu *janas_gpu_create(char *err, size_t err_len)sizeof(janas_spv_iq3s_matvec), &g->smi3,&g->pipei3, 0)) != VK_SUCCESS ||(r = make_pipeline(g, janas_spv_q4k_mmq, sizeof(janas_spv_q4k_mmq),- &g->sm_mm, &g->pipe_mm, 0, NV_PER_GROUP, g->sg)) !=- VK_SUCCESS ||+ &g->sm_mm, &g->pipe_mm, (uint32_t)tiles_exact(),+ NV_PER_GROUP, g->sg)) != VK_SUCCESS ||(r = make_pipeline(g, janas_spv_q6kp_mmq, sizeof(janas_spv_q6kp_mmq),&g->sm_mm6, &g->pipe_mm6, 0, NV_PER_GROUP, g->sg)) !=VK_SUCCESS ||(r = make_pipeline(g, janas_spv_q6k_mmq, sizeof(janas_spv_q6k_mmq),- &g->sm_mm6s, &g->pipe_mm6s, 0, NV_PER_GROUP,- g->sg)) != VK_SUCCESS ||+ &g->sm_mm6s, &g->pipe_mm6s, (uint32_t)tiles_exact(),+ NV_PER_GROUP, g->sg)) != VK_SUCCESS ||(r = make_pipeline(g, janas_spv_q3k_mmq, sizeof(janas_spv_q3k_mmq),&g->sm_mm3, &g->pipe_mm3, 0, NV_PER_GROUP, g->sg)) !=VK_SUCCESS ||@@ -722,16 +743,22 @@ struct janas_gpu *janas_gpu_create(char *err, size_t err_len)&g->pipe_dv[i], i) != VK_SUCCESS)g->pipe_dv[i] = VK_NULL_HANDLE;/* the experts' products on a few vectors (gpu_token_moe.c) */- const uint32_t *mc[6] = {janas_spv_q3k_matvec, janas_spv_q4k_matvec,+ /* (MXFP4, Q2_0, Q5_1: b32_matvec's kinds; Gemma 4 26B's down+ matrices are Q5_1) */+ const uint32_t *mc[8] = {janas_spv_q3k_matvec, janas_spv_q4k_matvec,janas_spv_q6k_matvec, janas_spv_q6kp_matvec,- janas_spv_b32_matvec, janas_spv_b32_matvec};- size_t mb[6] = {+ janas_spv_b32_matvec, janas_spv_b32_matvec,+ janas_spv_b32_matvec, janas_spv_q8_0_matvec};+ size_t mb[8] = {sizeof(janas_spv_q3k_matvec), sizeof(janas_spv_q4k_matvec),sizeof(janas_spv_q6k_matvec), sizeof(janas_spv_q6kp_matvec),- sizeof(janas_spv_b32_matvec), sizeof(janas_spv_b32_matvec)};- /* constant 2: ID, b32_matvec's in bit 8 beside its KIND */- uint32_t mk[6] = {1, 1, 1, 1, 256 | 4, 256 | 6};- for (int i = 0; i < 6; i++)+ sizeof(janas_spv_b32_matvec), sizeof(janas_spv_b32_matvec),+ sizeof(janas_spv_b32_matvec), sizeof(janas_spv_q8_0_matvec)};+ /* constant 2: ID, b32_matvec's in bit 8 beside its KIND (MXFP4,+ Q2_0, Q5_1), q8_0_matvec's alone (Gemma 4 26B's down matrices+ are Q5_1, its last layer's Q8_0) */+ uint32_t mk[8] = {1, 1, 1, 1, 256 | 4, 256 | 6, 256 | 3, 256};+ for (int i = 0; i < 8; i++)if (janas_g_make_pipeline_nv(g, mc[i], mb[i], &g->sm_mvid[i],&g->pipe_mvid[i], mk[i],MVID_NV) != VK_SUCCESS)@@ -1116,7 +1143,7 @@ void janas_gpu_destroy(struct janas_gpu *g)g->vkDestroyPipeline(g->dev, g->pipe_mm5, NULL);if (g->sm_mm5)g->vkDestroyShaderModule(g->dev, g->sm_mm5, NULL);- for (int i = 0; i < 6; i++) {+ for (int i = 0; i < 8; i++) {if (g->pipe_mvid[i])g->vkDestroyPipeline(g->dev, g->pipe_mvid[i], NULL);if (g->sm_mvid[i])@@ -1589,7 +1616,7 @@ int janas_gpu_import(struct janas_gpu *g, const void *p, size_t bytes)return import_refused(g, p, bytes, "regions, alignment or size", 0);/* a discrete GPU (or JANAS_GPU_COPY=1, to test that path) reads a copyin its own memory */- if (g->discrete || !g->host_import || getenv("JANAS_GPU_COPY"))+ if (g->discrete || !g->host_import || janas_g_copy_asked())return copy_to_device(g, p, bytes);struct region *rg = &g->reg[g->n_reg];VkMemoryHostPointerPropertiesEXT hp = {@@ -2010,7 +2037,7 @@ int janas_gpu_begin(struct janas_gpu *g, const struct janas_matvec_task *tasks,/* Push constants of the activation shader. */struct push_act {uint32_t g[2], u[2], x[2];- uint32_t blocks;+ uint32_t blocks, gelu; /* the shader's whole block, gelu 0 here */};void janas_g_push_addr(uint32_t out[2], VkDeviceAddress a)diff --git a/src/llm/gpu_impl.h b/src/llm/gpu_impl.hindex 3c978d1..97cee12 100644--- a/src/llm/gpu_impl.h+++ b/src/llm/gpu_impl.h@@ -196,8 +196,8 @@ struct janas_gpu {/* q3k, q4k, q6k, q6kp_matvec and b32_matvec's MXFP4 and Q2_0 with ID:a mixture's experts on a few vectors, MVID_NV a workgroup, from*_mmid.comp's table (optional) */- VkShaderModule sm_mvid[6];- VkPipeline pipe_mvid[6];+ VkShaderModule sm_mvid[8];+ VkPipeline pipe_mvid[8];int only_fast; /* janas_gpu_set_only_fast */uint32_t pass_n; /* janas_gpu_set_pass; UINT32_MAX until said */uint32_t nvg; /* vectors a product's workgroup takes (NV_PER_GROUP) */@@ -361,6 +361,7 @@ const struct region *janas_g_find_region(const struct janas_gpu *g,void janas_g_push_addr(uint32_t out[2], VkDeviceAddress a);void janas_g_compute_barrier(struct janas_gpu *g);int janas_g_x_in_32(int type);+int janas_g_copy_asked(void); /* JANAS_GPU_COPY=1 *//** Records the product of rows of a task (its weights in a region the GPU* holds) with nv = t->n_vec vectors at xa (the shader's layout, blocks ofdiff --git a/src/llm/gpu_token.c b/src/llm/gpu_token.cindex 7b5b63d..ab83268 100644--- a/src/llm/gpu_token.c+++ b/src/llm/gpu_token.c@@ -24,6 +24,7 @@#include "qk_rope.h" /* janas_spv_qk_rope */#include "attn_decode.h" /* janas_spv_attn_decode */#include "attn_merge.h" /* janas_spv_attn_merge */+#include "ple_act.h" /* janas_spv_ple_act */#include "gdn_conv.h" /* janas_spv_gdn_conv */#include "gdn_cstate.h" /* janas_spv_gdn_cstate */#include "gdn_l2.h" /* janas_spv_gdn_l2 */@@ -58,7 +59,7 @@ static const char *const ph_name[PH_N] = {"norm+quant", "q k v",int janas_gpu_shares_host(const struct janas_gpu *g){return g && !g->broken && !g->discrete && g->host_import &&- !getenv("JANAS_GPU_COPY");+ !janas_g_copy_asked();}size_t janas_gpu_local_room(const struct janas_gpu *g)@@ -119,14 +120,33 @@ static const char *check_ffn(const struct janas_gpu *g,ly->up.type != gt ||!(dt == JANAS_Q4_K || dt == JANAS_Q6_K || dt == JANAS_Q6_K_P ||(dt == JANAS_MXFP4 && g->pipe_idmx) ||- (dt == JANAS_Q3_K && g->pipe_id3) || (dt == JANAS_Q2_0 && q20)) ||- (md->eff % JANAS_QK && dt != JANAS_Q2_0) || ly->gate.rows != md->eff ||- ly->up.rows != md->eff || ly->gate.cols != md->dm ||- ly->up.cols != md->dm || ly->down.rows != md->dm ||- ly->down.cols != md->eff)+ (dt == JANAS_Q3_K && g->pipe_id3) || (dt == JANAS_Q2_0 && q20) ||+ (dt == JANAS_Q5_1 && g->pipe_mvid[6]) ||+ (dt == JANAS_Q8_0 && g->pipe_mvid[7])) ||+ (md->eff % JANAS_QK && dt != JANAS_Q2_0 && dt != JANAS_Q5_1 &&+ dt != JANAS_Q8_0) ||+ ly->gate.rows != md->eff || ly->up.rows != md->eff ||+ ly->gate.cols != md->dm || ly->up.cols != md->dm ||+ ly->down.rows != md->dm || ly->down.cols != md->eff)return "experts of a type or shape the GPU's shaders do not take";if (!ly->router16)return "a mixture's layer without its router";+ if (ly->pre_norm2) { /* Gemma 4's: its dense branch, norms, scales */+ const struct janas_matvec_task *d[3] = {&ly->dn_gate, &ly->dn_up,+ &ly->dn_down};+ for (int i = 0; i < 3; i++)+ if (!janas_gpu_can(g, d[i]))+ return "a dense branch's weight the GPU cannot compute";+ if (ly->dn_gate.rows > md->ff ||+ !janas_gt_addr_of(g, ly->pre_norm2, md->dm * 4) ||+ !janas_gt_addr_of(g, ly->post_norm1, md->dm * 4) ||+ !janas_gt_addr_of(g, ly->post_norm2, md->dm * 4) ||+ !janas_gt_addr_of(g, ly->post_ffn, md->dm * 4) ||+ !janas_gt_addr_of(g, ly->rscale, md->dm * 4) ||+ (ly->exp_scale &&+ !janas_gt_addr_of(g, ly->exp_scale, md->n_expert * 4)))+ return "a mixture's norms out of the GPU's reach";+ }if (ly->sh_gate_w.type) {const struct janas_matvec_task *s[3] = {&ly->sh_gate_w, &ly->sh_up,&ly->sh_down};@@ -221,10 +241,33 @@ static int check_model(const struct janas_gpu *g,check_ffn */if (md->dm % JANAS_QK || qd % JANAS_QK || md->ff % 32 || md->dm > 8192 ||qd > 8192 || md->hd > 512 || md->hd % 16 || md->n_rot > md->hd ||- md->n_kv == 0 || md->n_head % md->n_kv ||- md->n_head / md->n_kv * md->hd > ATT_MAXQ_G(md->n_head / md->n_kv) ||- md->n_head / md->n_kv * ATT_RUN > ATT_MAXS_G(md->n_head / md->n_kv))+ md->n_kv == 0 || md->n_head % md->n_kv)return fail(err, len, "a shape the token's shaders do not take");+ /* each attention layer's own: at most two kinds of workgroup (Gemma+ 4's), the query heads of a KV head split among workgroups where they+ do not fit one */+ uint32_t kinds[2] = {0, 0};+ for (uint32_t l = 0; l < md->n_layer; l++) {+ const struct janas_gpu_tok_layer *ly = &md->layers[l];+ if (ly->rec)+ continue;+ uint32_t ga = ly->n_kv && md->n_head % ly->n_kv == 0+ ? janas_gt_att_g(md->n_head / ly->n_kv, ly->hd)+ : 0;+ if (!ga || ly->hd > md->hd || ly->hd % 16 || ly->n_rot > ly->hd ||+ ly->n_rot % 2 || ly->n_kv > md->n_kv ||+ ga * ATT_RUN > ATT_MAXS_G(ga) || ly->cap == 0 ||+ ly->cap > md->cap ||+ (ly->swa ? ly->n_rot > md->n_rot_swa : ly->n_rot > md->n_rot) ||+ (md->kv16 && (md->ix_heads || md->n_rec)))+ return fail(err, len, "a shape the token's shaders do not take");+ if (!kinds[0] || kinds[0] == ga)+ kinds[0] = ga;+ else if (!kinds[1] || kinds[1] == ga)+ kinds[1] = ga;+ else+ return fail(err, len, "more than two kinds of attention layer");+ }if (md->n_expert &&(md->n_expert > 512 || md->k == 0 || md->k > 16 ||md->k > md->n_expert || md->eff % 32 || !md->fetch || !g->pipe_id4 ||@@ -270,7 +313,8 @@ static int check_model(const struct janas_gpu *g,const struct janas_matvec_task *t[4] = {&ly->wq, &ly->wk, &ly->wv,&ly->wo};for (int i = 0; i < 4; i++)- if (!janas_gpu_can(g, t[i])) {+ /* none: V is K, or the keys and values another layer's */+ if ((i == 0 || i == 3 || t[i]->type) && !janas_gpu_can(g, t[i])) {snprintf(err, len,"layer %u: a weight the GPU cannot compute (product ""%d, type %d)",@@ -279,15 +323,25 @@ static int check_model(const struct janas_gpu *g,}if (md->gated_q && (ly->wq.type == JANAS_Q6_K_P || ly->wq.plane[0]))return fail(err, len, "a gated query in planes");- size_t cache = (size_t)md->n_kv * md->cap * md->hd;+ size_t cache = (size_t)ly->n_kv * ly->cap * ly->hd << md->kv16;if ((!md->hc && (!janas_gt_addr_of(g, ly->attn_norm, md->dm * 4) ||!janas_gt_addr_of(g, ly->ffn_norm, md->dm * 4))) ||- !janas_gt_addr_of(g, ly->q_norm, md->hd * 4) ||- !janas_gt_addr_of(g, ly->k_norm, md->hd * 4) ||+ (ly->q_norm && !janas_gt_addr_of(g, ly->q_norm, ly->hd * 4)) ||+ (ly->k_norm && !janas_gt_addr_of(g, ly->k_norm, ly->hd * 4)) ||+ (ly->post_attn &&+ !janas_gt_addr_of(g, ly->post_attn, md->dm * 4)) ||+ (ly->post_ffn && !janas_gt_addr_of(g, ly->post_ffn, md->dm * 4)) ||+ (ly->ple_gate &&+ (!md->ple_dim || md->n_expert || !g->pipe_dm[0] ||+ !janas_gt_addr_of(g, ly->ple_gate,+ (size_t)md->ple_dim * md->dm * 4) ||+ !janas_gt_addr_of(g, ly->ple_proj,+ (size_t)md->ple_dim * md->dm * 4) ||+ !janas_gt_addr_of(g, ly->ple_post, md->dm * 4))) ||!janas_gt_addr_of(g, ly->kc, cache) ||!janas_gt_addr_of(g, ly->vc, cache) ||- !janas_gt_addr_of(g, ly->ks, (size_t)md->n_kv * md->cap * 4) ||- !janas_gt_addr_of(g, ly->vs, (size_t)md->n_kv * md->cap * 4))+ !janas_gt_addr_of(g, ly->ks, (size_t)ly->n_kv * ly->cap * 4) ||+ !janas_gt_addr_of(g, ly->vs, (size_t)ly->n_kv * ly->cap * 4))return fail(err, len,"a norm or the KV cache out of the GPU's ""reach");@@ -386,6 +440,36 @@ struct janas_gpu_tok *janas_gpu_tok_create(struct janas_gpu *g,t->o_h = take(&at, nb * t->ffp / JANAS_QK * 320);t->o_yd = take(&at, nb * md->dm * 4);t->o_rope = take(&at, nb * md->n_rot * 4);+ t->o_rope2 = md->n_rot_swa ? take(&at, nb * md->n_rot_swa * 4) : 0;+ t->o_post = take(&at, nb * md->dm * 4);+ for (uint32_t l = 0; l < md->n_layer; l++)+ if (md->layers[l].pre_norm2) { /* Gemma 4's mixture */+ t->o_ybr = take(&at, nb * md->dm * 4);+ t->o_xbr = take(&at, nb * md->dm * 4);+ t->o_rin = take(&at, nb * md->dm * 4);+ t->o_post2 = take(&at, nb * md->dm * 4);+ break;+ }+ if (md->ple_dim) {+ t->o_ple = take(&at, nb * md->n_layer * md->ple_dim * 4);+ t->o_pg = take(&at, nb * md->ple_dim * 4);+ t->o_ph = take(&at, nb * md->ple_dim * 4);+ }+ t->gelu = md->gelu;+ /* the attention's workgroups: the query heads each takes, one or two+ kinds (check_model) */+ for (uint32_t l = 0; l < md->n_layer; l++) {+ const struct janas_gpu_tok_layer *ly = &md->layers[l];+ if (ly->rec)+ continue;+ uint32_t ga = janas_gt_att_g(md->n_head / ly->n_kv, ly->hd);+ if (!t->at_g[0] || t->at_g[0] == ga)+ t->at_g[0] = ga;+ else+ t->at_g[1] = ga;+ }+ if (!t->at_g[0]) /* qwen4exp's streams: the model's own */+ t->at_g[0] = md->n_head / md->n_kv;t->o_logits = take(&at, nb * md->n_vocab * 4);t->o_pos = take(&at, 32); /* janas_gpu_tok_run's five words */t->o_part = take(&at, t->part_floats * 4);@@ -485,7 +569,8 @@ struct janas_gpu_tok *janas_gpu_tok_create(struct janas_gpu *g,int d32 = 0;for (uint32_t l = 0; l < md->n_layer; l++)d32 |= janas_g_x_in_32(ne ? md->layers[l].sh_down.type- : md->layers[l].down.type);+ : md->layers[l].down.type) ||+ janas_g_x_in_32(md->layers[l].dn_down.type); /* Gemma 4's */if (d32) {t->o_hf = take(&at, nb * t->ffp * 4);t->o_h32 = take(&at, nb * t->ffp / JANAS_QK * 320);@@ -519,9 +604,16 @@ struct janas_gpu_tok *janas_gpu_tok_create(struct janas_gpu *g,0) != VK_SUCCESS ||janas_g_make_pipeline(g, janas_spv_qk_rope, sizeof(janas_spv_qk_rope),&t->sm_qk, &t->p_qk, 0) != VK_SUCCESS ||- janas_g_make_pipeline_sg(- g, janas_spv_attn_decode, sizeof(janas_spv_attn_decode), &t->sm_at,- &t->p_at, md->n_head / md->n_kv, att_sg()) == 0 ||+ janas_g_make_pipeline_sg(g, janas_spv_attn_decode,+ sizeof(janas_spv_attn_decode), &t->sm_at,+ &t->p_at, t->at_g[0], att_sg()) == 0 ||+ (t->at_g[1] &&+ janas_g_make_pipeline_sg(g, janas_spv_attn_decode,+ sizeof(janas_spv_attn_decode), &t->sm_at2,+ &t->p_at2, t->at_g[1], att_sg()) == 0) ||+ (md->ple_dim &&+ janas_g_make_pipeline(g, janas_spv_ple_act, sizeof(janas_spv_ple_act),+ &t->sm_pa, &t->p_pa, 0) != VK_SUCCESS) ||janas_g_make_pipeline(g, janas_spv_attn_merge,sizeof(janas_spv_attn_merge), &t->sm_am, &t->p_am,0) != VK_SUCCESS ||@@ -678,13 +770,13 @@ void janas_gpu_tok_destroy(struct janas_gpu *g, struct janas_gpu_tok *t)g->vkFreeCommandBuffers(g->dev, g->cp, 1, &t->cb[n][a][h]);janas_gt_free_segments(g, t, &t->seg_cb[n][a][h]);}- VkPipeline p[16] = {t->p_nq, t->p_qk, t->p_at, t->p_am, t->p_gc, t->p_gs,- t->p_gl, t->p_gr, t->p_gg, t->p_tk, t->p_ga, t->p_cb,- t->p_sm, t->p_gb, t->p_gx, t->p_pl};- VkShaderModule s[16] = {t->sm_nq, t->sm_qk, t->sm_at, t->sm_am,- t->sm_gc, t->sm_gs, t->sm_gl, t->sm_gr,- t->sm_gg, t->sm_tk, t->sm_ga, t->sm_cb,- t->sm_sm, t->sm_gb, t->sm_gx, t->sm_pl};+ VkPipeline p[18] = {t->p_nq, t->p_qk, t->p_at, t->p_am, t->p_gc, t->p_gs,+ t->p_gl, t->p_gr, t->p_gg, t->p_tk, t->p_ga, t->p_cb,+ t->p_sm, t->p_gb, t->p_gx, t->p_pl, t->p_at2, t->p_pa};+ VkShaderModule s[18] = {t->sm_nq, t->sm_qk, t->sm_at, t->sm_am, t->sm_gc,+ t->sm_gs, t->sm_gl, t->sm_gr, t->sm_gg, t->sm_tk,+ t->sm_ga, t->sm_cb, t->sm_sm, t->sm_gb, t->sm_gx,+ t->sm_pl, t->sm_at2, t->sm_pa};janas_gt_hc_destroy(g, t);janas_gt_qsa_destroy(g, t);janas_gt_tier_destroy(g, t);@@ -693,7 +785,7 @@ void janas_gpu_tok_destroy(struct janas_gpu *g, struct janas_gpu_tok *t)free(t->fill);free(t->ids);free(t->slots);- for (int i = 0; i < 16; i++) {+ for (int i = 0; i < 18; i++) {if (p[i])g->vkDestroyPipeline(g->dev, p[i], NULL);if (s[i])@@ -881,7 +973,8 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,/* the residual: read from X, the sum with a layer's output written toXN, the two swapped (every block starts from o_x, the same commands) */VkDeviceAddress B = t->buf.addr, X = B + t->o_x, XN = B + t->o_x1, SW,- YWO = B + t->o_ywo, YD = B + t->o_yd, XNF = B + t->o_xnf;+ YWO = B + t->o_ywo, YD = B + t->o_yd, XNF = B + t->o_xnf,+ POST = B + t->o_post;if (g->vkResetCommandBuffer(g->cb, 0) != VK_SUCCESS)return -1;VkCommandBufferBeginInfo bb = {@@ -906,7 +999,7 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,and between the two what the CPU's plan writes and reads */int last = seg < 0 || seg == (int)md->n_layer;if (t->local && seg <= 0) {- VkBufferCopy in[3] = {+ VkBufferCopy in[5] = {{.srcOffset = t->o_x,.dstOffset = t->o_x,.size = (VkDeviceSize)n * md->dm * sizeof(float)},@@ -915,8 +1008,21 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,.size = (VkDeviceSize)n * md->n_rot * sizeof(float)},{.srcOffset = t->o_pos,.dstOffset = t->o_pos,- .size = JANAS_GT_POS_BYTES}};- g->vkCmdCopyBuffer(g->cb, t->mirror.buf, t->buf.buf, 3, in);+ .size = JANAS_GT_POS_BYTES},+ /* Gemma 4's second RoPE tables, its per-layer inputs */+ {.srcOffset = t->o_rope2,+ .dstOffset = t->o_rope2,+ .size = (VkDeviceSize)n * md->n_rot_swa * sizeof(float)},+ {.srcOffset = t->o_ple,+ .dstOffset = t->o_ple,+ .size =+ (VkDeviceSize)n * md->n_layer * md->ple_dim * sizeof(float)}};+ uint32_t nin = 3;+ if (md->n_rot_swa)+ in[nin++] = in[3];+ if (md->ple_dim)+ in[nin++] = in[4];+ g->vkCmdCopyBuffer(g->cb, t->mirror.buf, t->buf.buf, nin, in);VkMemoryBarrier ib = {.sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER,.srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT,.dstAccessMask = VK_ACCESS_SHADER_READ_BIT |@@ -962,13 +1068,22 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,janas_gt_rec_experts(g, t, md, &layers[l - 1], &bound, YD, n, part);if (part == 1)goto end;+ if (layers[l - 1].pre_norm2) /* Gemma 4: its branches joined */+ janas_gt_rec_gemma_moe_tail(g, t, md, &layers[l - 1], &bound,+ YD, n);}- /* the previous layer's feed-forward added, then this one's norm */+ /* the previous layer's feed-forward added (Gemma 4: normalized,+ the sum times the layer's output scale), then this one's norm */if (on) {- janas_gt_rec_norm_quant(- g, t, &bound, X, l ? YD : 0,+ const struct janas_gpu_tok_layer *pl = l ? &layers[l - 1] : NULL;+ janas_gt_rec_norm_quant_s(+ g, t, &bound, X,+ !pl ? 0+ : pl->post_ffn ? POST+ : YD,janas_gt_addr_of(g, ly->attn_norm, md->dm * 4), l ? XN : 0,- ly->rec ? XNF : 0, md->dm, md->eps, n);+ ly->rec ? XNF : 0, md->dm, md->eps, n,+ pl && pl->post_ffn ? pl->out_scale : 0.0f);janas_gt_stamp(g, t, PH_NORM);}if (l)@@ -978,17 +1093,34 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,: janas_gt_rec_attn(g, t, md, ly, &bound, n)) != 0)return -1;janas_gt_rec_norm_quant(- g, t, &bound, X, YWO,+ g, t, &bound, X, ly->post_attn ? POST : YWO,janas_gt_addr_of(g, ly->ffn_norm, md->dm * 4), XN,md->n_expert ? XNF : 0, md->dm, md->eps, n);janas_gt_stamp(g, t, PH_NORM);}SW = X, X = XN, XN = SW;+ /* Gemma 4's mixture: its dense branch, the router's and the+ experts' inputs first */+ if (on && ly->pre_norm2 &&+ janas_gt_rec_gemma_moe(g, t, md, ly, &bound, X, n) != 0)+ return -1;if (on &&(md->n_expert ? janas_gt_rec_router(g, t, md, ly, l, &bound, n): janas_gt_rec_ffn(g, t, &bound, &ly->gate, &ly->up,&ly->down, n, YD)) != 0)return -1;+ /* Gemma 4: the feed-forward's output normalized, into o_post */+ if (on && ly->post_ffn && !md->n_expert)+ janas_gt_rec_post_norm(g, t, &bound, YD, ly->post_ffn, md, n);+ /* ... and with per-layer embeddings their branch, on the residual+ with the feed-forward added (its output scale after the branch) */+ if (on && ly->ple_gate) {+ janas_gt_rec_norm_quant(g, t, &bound, X, POST, 0, XN, 0, md->dm,+ md->eps, n);+ SW = X, X = XN, XN = SW;+ janas_gt_rec_ple(g, t, md, ly, l, &bound, X, YD, n);+ janas_gt_rec_post_norm(g, t, &bound, YD, ly->ple_post, md, n);+ }/* a mixture's pass in one submission: the plan on the GPU */if (seg < 0 && md->n_expert)janas_gt_rec_plan(g, t, md, l, &bound, n);@@ -1002,12 +1134,17 @@ static int record(struct janas_gpu *g, struct janas_gpu_tok *t,part);if (part == 1)goto end;+ if (layers[nl - 1].pre_norm2)+ janas_gt_rec_gemma_moe_tail(g, t, md, &layers[nl - 1], &bound,+ YD, n);}if (first < n || hid) {- janas_gt_rec_norm_quant(- g, t, &bound, X, YD,+ const struct janas_gpu_tok_layer *pl = &layers[nl - 1];+ janas_gt_rec_norm_quant_s(+ g, t, &bound, X, pl->post_ffn ? POST : YD,janas_gt_addr_of(g, md->out_norm, md->dm * 4), 0,- hid ? B + t->o_hid : 0, md->dm, md->eps, n);+ hid ? B + t->o_hid : 0, md->dm, md->eps, n,+ pl->post_ffn ? pl->out_scale : 0.0f);}if (first < n) {if (janas_gt_rec_product(g, t, &bound, output,@@ -1081,6 +1218,9 @@ uint32_t janas_gpu_tok_block_at(const struct janas_gpu_tok *t,return 0;size_t n =t->part_floats / janas_gt_part_tok(t, md, runs_for(t, pos + t->nb));+ /* the rings hold a window and a block: no more of the block */+ if (md->ring_room && n > md->ring_room)+ n = md->ring_room;return n < 1 ? 1 : n > t->nb ? t->nb : (uint32_t)n;}@@ -1171,7 +1311,8 @@ int janas_gpu_tok_run(struct janas_gpu *g, struct janas_gpu_tok *t,uint32_t src, uint32_t dst, uint32_t replay,uint32_t log_row){- if (!t || g->busy || g->broken || n < 1 || n > t->nb || pos + n > md->cap)+ if (!t || g->busy || g->broken || n < 1 || n > t->nb || pos + n > md->cap ||+ (md->ring_room && n > md->ring_room))return -1;/* a replay, or a block to log, needs the log on the GPU */int logs = log_row != UINT32_MAX;@@ -1204,6 +1345,18 @@ int janas_gpu_tok_run(struct janas_gpu *g, struct janas_gpu_tok *t,memcpy(r + half * sizeof(float), sn + (size_t)j * half,half * sizeof(float));}+ /* Gemma 4's per-layer inputs */+ size_t pb = (size_t)n * md->n_layer * md->ple_dim * sizeof(float);+ if (pb)+ memcpy(t->buf.map + t->o_ple, md->ple_in, pb);+ /* Gemma 4's sliding-window layers: their own tables */+ uint32_t h2 = md->n_rot_swa / 2;+ for (uint32_t j = 0; h2 && j < n; j++) {+ uint8_t *r = t->buf.map + t->o_rope2 + (size_t)j * md->n_rot_swa * 4;+ memcpy(r, md->cs_swa + (size_t)j * h2, h2 * sizeof(float));+ memcpy(r + h2 * sizeof(float), md->sn_swa + (size_t)j * h2,+ h2 * sizeof(float));+ }/* the position, the halves of the recurrent state to read and towrite, the tokens to replay, the log's row: read by the shaders atrun time */@@ -1216,6 +1369,10 @@ int janas_gpu_tok_run(struct janas_gpu *g, struct janas_gpu_tok *t,int) = t->local ? janas_gt_mirror_cache : janas_gt_sync_range;in(g, t, t->o_x, (size_t)n * md->dm * sizeof(float), 1);in(g, t, t->o_rope, (size_t)n * md->n_rot * sizeof(float), 1);+ if (h2)+ in(g, t, t->o_rope2, (size_t)n * md->n_rot_swa * sizeof(float), 1);+ if (pb)+ in(g, t, t->o_ple, pb, 1);in(g, t, t->o_pos, sizeof(pw), 1);int ai = all == 1 && n == 1 ? 0 : all, hi = hidden != NULL;struct timespec w0, w1;diff --git a/src/llm/gpu_token.h b/src/llm/gpu_token.hindex 2534ffe..be35dab 100644--- a/src/llm/gpu_token.h+++ b/src/llm/gpu_token.h@@ -31,6 +31,28 @@struct janas_gpu_tok_layer {const float *attn_norm, *q_norm, *k_norm, *ffn_norm;struct janas_matvec_task wq, wk, wv, wo, gate, up, down;+ /* the attention's geometry, the model's or the layer's kind's (Gemma+ 4: sliding-window layers and full ones): head size, KV heads, RoPE+ dimensions, the positions its cache holds (a ring of them where+ ring: position p in row p mod cap), the window a token sees (0: all+ of them); wv's type 0: V is K before its norm (Gemma 4's full+ layers). Gemma 4's sandwich norms of the attention's and the+ feed-forward's outputs (NULL: none) and the layer's output scale */+ uint32_t hd, n_kv, n_rot, cap, window;+ int ring, swa;+ const float *post_attn, *post_ffn;+ float out_scale;+ /* Gemma 4's per-layer embeddings (E2B, E4B; NULL: none): the branch's+ gate (ple_dim x dm) and projection (dm x ple_dim), f32, its norm; the+ layer's keys and values another's (n_kv_layers on: wk's type 0) */+ const float *ple_gate, *ple_proj, *ple_post;+ /* Gemma 4's mixture (26B; pre_norm2 NULL: none): its dense branch (the+ shared expert, weight 1, no gate: sh_gate_w's type 0), normalized by+ post_norm1; the experts' input by pre_norm2, the router's the+ residual without weights times rscale / sqrt(dm), each expert's+ output scale, the experts' sum normalized by post_norm2 */+ struct janas_matvec_task dn_gate, dn_up, dn_down;+ const float *pre_norm2, *post_norm1, *post_norm2, *rscale, *exp_scale;int8_t *kc, *vc; /* the layer's KV cache: n_kv x cap x hd bytes */float *ks, *vs; /* its scales: n_kv x cap *//* a Gated DeltaNet layer (qwen3.5) in place of the attention: q|k|v and@@ -78,8 +100,23 @@ struct janas_gpu_tok_layer {};struct janas_gpu_tok_model {+ /* n_kv, hd and n_rot the largest of any layer's, cap the full+ layers' */uint32_t n_layer, dm, n_head, n_kv, hd, n_rot, ff, n_vocab, cap;float eps, attn_scale;+ /* Gemma 4: GELU in place of SiLU, keys and values at sixteen bits (q+ left in float for them), V normalized without weights; the+ sliding-window layers' RoPE dimensions and their tables (n_rot_swa+ / 2 cos and sin a token, set before each run), the most tokens a+ block the rings leave room for (0: no ring) */+ int gelu, kv16, vnorm;+ uint32_t n_rot_swa, ring_room;+ const float *cs_swa, *sn_swa;+ /* Gemma 4's per-layer embeddings: ple_dim numbers a layer and token+ (0: none), the block's from the CPU (janas_m_ple_inputs, n_layer+ ple_dim a token), set before each run */+ uint32_t ple_dim;+ const float *ple_in;const struct janas_gpu_tok_layer *layers;const float *out_norm;struct janas_matvec_task output;diff --git a/src/llm/gpu_token_impl.h b/src/llm/gpu_token_impl.hindex 4e95ca4..47cb882 100644--- a/src/llm/gpu_token_impl.h+++ b/src/llm/gpu_token_impl.h@@ -21,12 +21,22 @@ struct push_nq {uint32_t x[2], add[2], w[2], xo[2], q256[2], q32[2], xn[2];uint32_t d;float eps;+ float os; /* x + add times os (0: as it is), Gemma 4's output scale */+ float ps; /* the normalized value times ps before w (0: not) */+ /* 0: the CPU's rms_norm and janas_q8k_quantize, bit for bit; 1: the+ deterministic quantization (an activation, janas_q8k_quantize_det) */+ uint32_t det;};struct push_qk {uint32_t qkv[2], qn[2], kn[2], rope[2], kc[2], vc[2], ks[2], vs[2];uint32_t n_head, n_kv, hd, n_rot, cap, qs;float eps;- uint32_t pad, pos[2]; /* the position's buffer, 8-byte aligned */+ /* bit 0: q's heads not normalized, bit 1: k's (llama); bit 2: q at+ sixteen bits for attn_decode.comp (JANAS_GPU_Q16=1); Gemma 4's: bit+ 3 keys and values at sixteen bits (q left in float), bit 4 a ring+ of positions, bit 5 V is K, bit 6 V normalized without weights */+ uint32_t flags;+ uint32_t pos[2]; /* the position's buffer, 8-byte aligned */};struct push_at {uint32_t q[2], kc[2], vc[2], ks[2], vs[2], part[2];@@ -35,15 +45,19 @@ struct push_at {uint32_t qs, ps;uint32_t pos[2]; /* the position's buffer */uint32_t list[2]; /* qwen4exp's positions a token (ix_select.comp) */- uint32_t ls, pad;+ /* flags: bit 0 q as qk_rope.comp's bit 2 left it, bit 1 keys and+ values at sixteen bits, bit 2 a ring; hg: workgroups a KV head's+ query heads are split into (G of them each); window: positions a+ token sees (0: all) */+ uint32_t ls, flags, hg, window;};struct push_am {uint32_t part[2], q256[2], q32[2];uint32_t hd, nch, ps, gs;uint32_t pos[2]; /* 8-byte aligned, as the GLSL block puts it */uint32_t gate[2];- uint32_t list[2]; /* as push_at's */- uint32_t ls, pad;+ uint32_t list[2]; /* as push_at's */+ uint32_t ls, window; /* window: as push_at's, the runs before it left */};/* the Gated DeltaNet shaders (gdn_*.comp) */struct push_gconv {@@ -86,7 +100,7 @@ struct push_dm { /* gpu.c's dense_mm.comp */};/* a mixture's (moe_*.comp, and gpu.c's *_mmid.comp) */struct push_tk {- uint32_t r[2], pe[2], pw[2], shw[2];+ uint32_t r[2], pe[2], pw[2], shw[2], es[2]; /* es: Gemma 4's scales */uint32_t ne, rs, k, n, sh;};struct push_ga {@@ -109,7 +123,7 @@ struct push_id {};struct push_sm {uint32_t g[2], u[2], h[2];- uint32_t n;+ uint32_t n, gelu; /* gelu: GELU (tanh) in place of SiLU */};/* qwen4exp's mixes and n-gram embeddings (hc_*.comp, ng_*.comp) */struct push_hn {@@ -170,10 +184,24 @@ _Static_assert(sizeof(struct push_at) <= JANAS_G_PUSH_BYTES, "push size");MAXQ, MAXS) */#define ATT_MAXQ_G(g) ((g) * 256u > ATT_MAXQ ? (g) * 256u : ATT_MAXQ)#define ATT_MAXS_G(g) ((g) * ATT_RUN > ATT_MAXS ? (g) * ATT_RUN : ATT_MAXS)+/* The query heads a workgroup of the attention takes, of the gt that share+ a KV head of hd numbers: all of them, or (Gemma 4's full layers, 16 of+ 512) the most of a divisor of gt that its shared memory holds; 0: none */+static inline uint32_t janas_gt_att_g(uint32_t gt, uint32_t hd)+{+ for (uint32_t g = gt; g >= 1; g--)+ if (gt % g == 0 && g * hd <= ATT_MAXQ_G(g))+ return g;+ return 0;+}+struct push_pa { /* ple_act.comp */+ uint32_t g[2], in[2], h[2];+ uint32_t n, pd, w;+};struct push_ac {uint32_t g[2], u[2], x[2];- uint32_t blocks;+ uint32_t blocks, gelu; /* as push_sm's */};_Static_assert(sizeof(struct push_qk) <= JANAS_G_PUSH_BYTES, "push size");_Static_assert(sizeof(struct push_at) <= JANAS_G_PUSH_BYTES, "push size");@@ -251,6 +279,13 @@ struct janas_gpu_tok {uint64_t pass_tokens; /* ... and their tokens */VkShaderModule sm_nq, sm_qk, sm_at, sm_am;VkPipeline p_nq, p_qk, p_at, p_am;+ /* the attention of a second kind of layer (Gemma 4), whose query+ heads a workgroup takes (attn_decode.comp G) are at_g[1], p_at's+ at_g[0] */+ VkShaderModule sm_at2, sm_pa;+ VkPipeline p_at2, p_pa; /* p_pa: ple_act.comp */+ uint32_t at_g[2];+ int gelu; /* the feed-forward's activation: md's *//* qwen3.5's recurrent layers (sm_gb: the betas and decays of a shortblock, sm_gx: those of a long one's raw dot products) */VkShaderModule sm_gc, sm_gs, sm_gl, sm_gr, sm_gg, sm_gb, sm_gx;@@ -269,6 +304,14 @@ struct janas_gpu_tok {size_t o_part;size_t o_x, o_x1, o_q256, o_q32, o_qkv, o_ywo, o_gu, o_h, o_yd, o_rope,o_logits, o_pos;+ /* Gemma 4: the sliding-window layers' RoPE tables, a sandwich norm's+ output */+ size_t o_rope2, o_post;+ /* ... its per-layer inputs, a branch's gate and activation */+ size_t o_ple, o_pg, o_ph;+ /* ... its mixture's dense branch, that normalized, the router's+ input, the experts' sum normalized */+ size_t o_ybr, o_xbr, o_rin, o_post2;/* qwen3.5: the normalized input (f32), q|k|v and z of the recurrentlayers, their convolution's output (the replayed tokens' rowsfirst), beta and alpha, their raw output, the betas and decays used@@ -401,6 +444,35 @@ void janas_gt_rec_norm_quant(struct janas_gpu *g, struct janas_gpu_tok *t,VkDeviceAddress add, VkDeviceAddress w,VkDeviceAddress xo, VkDeviceAddress xn, uint32_t d,float eps, uint32_t n);+void janas_gt_rec_norm_quant_s(struct janas_gpu *g, struct janas_gpu_tok *t,+ VkPipeline *bound, VkDeviceAddress x,+ VkDeviceAddress add, VkDeviceAddress w,+ VkDeviceAddress xo, VkDeviceAddress xn,+ uint32_t d, float eps, uint32_t n, float os);+void janas_gt_rec_norm_quant_p(struct janas_gpu *g, struct janas_gpu_tok *t,+ VkPipeline *bound, VkDeviceAddress x,+ VkDeviceAddress add, VkDeviceAddress w,+ VkDeviceAddress xo, VkDeviceAddress xn,+ uint32_t d, float eps, uint32_t n, float os,+ float ps);+int janas_gt_rec_gemma_moe(struct janas_gpu *g, struct janas_gpu_tok *t,+ const struct janas_gpu_tok_model *md,+ const struct janas_gpu_tok_layer *ly,+ VkPipeline *bound, VkDeviceAddress x, uint32_t n);+void janas_gt_rec_gemma_moe_tail(struct janas_gpu *g, struct janas_gpu_tok *t,+ const struct janas_gpu_tok_model *md,+ const struct janas_gpu_tok_layer *ly,+ VkPipeline *bound, VkDeviceAddress yd,+ uint32_t n);+void janas_gt_rec_ple(struct janas_gpu *g, struct janas_gpu_tok *t,+ const struct janas_gpu_tok_model *md,+ const struct janas_gpu_tok_layer *ly, uint32_t l,+ VkPipeline *bound, VkDeviceAddress x, VkDeviceAddress y,+ uint32_t n);+void janas_gt_rec_post_norm(struct janas_gpu *g, struct janas_gpu_tok *t,+ VkPipeline *bound, VkDeviceAddress x,+ const float *w,+ const struct janas_gpu_tok_model *md, uint32_t n);int janas_gt_rec_product(struct janas_gpu *g, struct janas_gpu_tok *t,VkPipeline *bound, const struct janas_matvec_task *w,size_t xoff, VkDeviceAddress y, uint32_t n, size_t ys);diff --git a/src/llm/gpu_token_moe.c b/src/llm/gpu_token_moe.cindex 1bec0a3..83c8424 100644--- a/src/llm/gpu_token_moe.c+++ b/src/llm/gpu_token_moe.c@@ -44,12 +44,17 @@ int janas_gt_rec_router(struct janas_gpu *g, struct janas_gpu_tok *t,janas_gt_rec_ffn(g, t, bound, &ly->sh_gate_w, &ly->sh_up, &ly->sh_down,n, B + t->o_sh) != 0)return -1;+ /* the router's input: the feed-forward's normalized input, or Gemma+ 4's own (o_rin) */janas_gt_rec_dense(g, bound, 0, B + t->par[(size_t)l * PAR_N + PAR_R],- B + t->o_xnf, B + t->o_r, ne + (ly->sh_gate ? 1 : 0),- md->dm, ne + 1, n, 1.0f);+ B + (ly->pre_norm2 ? t->o_rin : t->o_xnf), B + t->o_r,+ ne + (ly->sh_gate ? 1 : 0), md->dm, ne + 1, n, 1.0f);janas_g_compute_barrier(g);struct push_tk pk = {.ne = ne, .rs = ne + 1, .k = md->k, .n = n, .sh = ly->sh_gate != NULL};+ /* Gemma 4: each expert's own output scale, on its weight */+ janas_g_push_addr(+ pk.es, ly->exp_scale ? janas_gt_addr_of(g, ly->exp_scale, ne * 4) : 0);janas_g_push_addr(pk.r, B + t->o_r);janas_g_push_addr(pk.pe, B + t->o_pe);janas_g_push_addr(pk.pw, B + t->o_pw);@@ -75,7 +80,12 @@ static VkPipeline by_rows(const struct janas_gpu *g,: ty == JANAS_Q6_K_P ? 3: ty == JANAS_MXFP4 ? 4: ty == JANAS_Q2_0 ? 5+ : ty == JANAS_Q5_1 ? 6+ : ty == JANAS_Q8_0 ? 7: -1;+ /* Q5_1 and Q8_0 have no tiles (*_mmid): by rows on any block */+ if (i >= 6)+ return g->pipe_mvid[i];return n <= t->mvid_n && i >= 0 ? g->pipe_mvid[i] : VK_NULL_HANDLE;}@@ -143,7 +153,7 @@ void janas_gt_rec_experts(struct janas_gpu *g, struct janas_gpu_tok *t,t->o_ind);janas_g_compute_barrier(g);janas_gt_stamp(g, t, PH_GATEUP);- struct push_ac ac = {.blocks = np * nbf};+ struct push_ac ac = {.blocks = np * nbf, .gelu = (uint32_t)t->gelu};janas_g_push_addr(ac.g, B + t->o_egu);janas_g_push_addr(ac.u, B + t->o_egu + (size_t)np * t->effp * 4);janas_g_push_addr(ac.x, B + t->o_eh);diff --git a/src/llm/gpu_token_rec.c b/src/llm/gpu_token_rec.cindex c9d0081..b882b6a 100644--- a/src/llm/gpu_token_rec.c+++ b/src/llm/gpu_token_rec.c@@ -46,7 +46,29 @@ void janas_gt_rec_norm_quant(struct janas_gpu *g, struct janas_gpu_tok *t,VkDeviceAddress xo, VkDeviceAddress xn, uint32_t d,float eps, uint32_t n){- struct push_nq pc = {.d = d, .eps = eps};+ janas_gt_rec_norm_quant_s(g, t, bound, x, add, w, xo, xn, d, eps, n, 0.0f);+}++/* ... with x + add times os (0: as it is), Gemma 4's output scale */+void janas_gt_rec_norm_quant_s(struct janas_gpu *g, struct janas_gpu_tok *t,+ VkPipeline *bound, VkDeviceAddress x,+ VkDeviceAddress add, VkDeviceAddress w,+ VkDeviceAddress xo, VkDeviceAddress xn,+ uint32_t d, float eps, uint32_t n, float os)+{+ janas_gt_rec_norm_quant_p(g, t, bound, x, add, w, xo, xn, d, eps, n, os,+ 0.0f);+}++/* ... and the normalized value times ps before w (Gemma 4's router) */+void janas_gt_rec_norm_quant_p(struct janas_gpu *g, struct janas_gpu_tok *t,+ VkPipeline *bound, VkDeviceAddress x,+ VkDeviceAddress add, VkDeviceAddress w,+ VkDeviceAddress xo, VkDeviceAddress xn,+ uint32_t d, float eps, uint32_t n, float os,+ float ps)+{+ struct push_nq pc = {.d = d, .eps = eps, .os = os, .ps = ps};janas_g_push_addr(pc.xn, xn);janas_g_push_addr(pc.x, x);janas_g_push_addr(pc.add, add);@@ -233,19 +255,42 @@ int janas_gt_rec_gdn(struct janas_gpu *g, struct janas_gpu_tok *t,return 0;}+/* JANAS_GPU_Q16=1: the attention's q at sixteen bits, two eight-bit halves+ (q = 256 hi + lo), its scores two integer products; the CPU takes q in+ float, the eight-bit q's relative error some 4e-3 against 3e-5; heads+ up to 256 only: 256 hi k summed over a longer head could pass 2^31 */+static int q16_on(uint32_t hd)+{+ static int on = -1;+ if (on < 0) {+ const char *e = getenv("JANAS_GPU_Q16");+ on = e && atoi(e) > 0;+ }+ return on && hd <= 256;+}+/* A layer's attention on the block normalized in q256 / q32: q, k and v,their norms and RoPE (the keys and values into the cache), the- attention in runs of positions, joined, then wo into YWO. */+ attention in runs of positions, joined, then wo into YWO (Gemma 4: its+ sandwich norm after, into o_post). The layer's own geometry (Gemma 4's+ two kinds), the token's buffers laid out for the largest. */int janas_gt_rec_attn(struct janas_gpu *g, struct janas_gpu_tok *t,const struct janas_gpu_tok_model *md,const struct janas_gpu_tok_layer *ly, VkPipeline *bound,uint32_t n){- uint32_t qd = md->n_head * md->hd, kvd = md->n_kv * md->hd;+ uint32_t hd = ly->hd, nkv = ly->n_kv;+ uint32_t qd = md->n_head * hd, kvd = nkv * hd;size_t qs = qd + 2 * kvd;- size_t cache = (size_t)md->n_kv * md->cap * md->hd;+ size_t cache = (size_t)nkv * ly->cap * hd << md->kv16,+ scales = (size_t)nkv * ly->cap * 4;VkDeviceAddress B = t->buf.addr, QKV = B + t->o_qkv, YWO = B + t->o_ywo,- ROPE = B + t->o_rope, POS = B + t->o_pos;+ ROPE = B + (ly->swa ? t->o_rope2 : t->o_rope),+ POS = B + t->o_pos;+ int v_is_k = ly->wv.type == 0;+ /* a layer reading another's keys and values (Gemma 4's E2B, E4B):+ its queries alone */+ int own_kv = ly->wk.type != 0;/* q, k and v of each token side by side, qs floats a token; agated query (qwen3.5) head by head, its gates apart */if (md->gated_q) {@@ -253,59 +298,73 @@ int janas_gt_rec_attn(struct janas_gpu *g, struct janas_gpu_tok *t,(size_t)(md->dm / JANAS_QK) * janas_qtype_block_size(ly->wq.type);for (uint32_t h = 0; h < md->n_head; h++) {struct janas_matvec_task q = ly->wq, gt = ly->wq;- q.rows = gt.rows = md->hd;- q.w = (const uint8_t *)ly->wq.w + (size_t)2 * h * md->hd * rb;- gt.w = (const uint8_t *)q.w + (size_t)md->hd * rb;+ q.rows = gt.rows = hd;+ q.w = (const uint8_t *)ly->wq.w + (size_t)2 * h * hd * rb;+ gt.w = (const uint8_t *)q.w + (size_t)hd * rb;if (janas_gt_rec_product(g, t, bound, &q, 0,- QKV + (size_t)h * md->hd * 4, n,- qs) != 0 ||+ QKV + (size_t)h * hd * 4, n, qs) != 0 ||janas_gt_rec_product(g, t, bound, >, 0,- B + t->o_gate + (size_t)h * md->hd * 4, n,+ B + t->o_gate + (size_t)h * hd * 4, n,qd) != 0)return -1;}} else if (janas_gt_rec_product(g, t, bound, &ly->wq, 0, QKV, n, qs) != 0)return -1;- if (janas_gt_rec_product(g, t, bound, &ly->wk, 0, QKV + (size_t)qd * 4, n,- qs) != 0 ||- janas_gt_rec_product(g, t, bound, &ly->wv, 0,- QKV + (size_t)(qd + kvd) * 4, n, qs) != 0)+ if (own_kv && (janas_gt_rec_product(g, t, bound, &ly->wk, 0,+ QKV + (size_t)qd * 4, n, qs) != 0 ||+ (!v_is_k && janas_gt_rec_product(+ g, t, bound, &ly->wv, 0,+ QKV + (size_t)(qd + kvd) * 4, n, qs) != 0)))return -1;janas_g_compute_barrier(g);janas_gt_stamp(g, t, PH_QKV);+ int q16 = !md->kv16 && q16_on(hd);struct push_qk pq = {.n_head = md->n_head,- .n_kv = md->n_kv,- .hd = md->hd,- .n_rot = md->n_rot,- .cap = md->cap,+ .n_kv = nkv,+ .hd = hd,+ .n_rot = ly->n_rot,+ .cap = ly->cap,.qs = (uint32_t)qs,- .eps = md->eps};+ .eps = md->eps,+ .flags = (ly->q_norm ? 0u : 1u) |+ (ly->k_norm ? 0u : 2u) | (q16 ? 4u : 0u) |+ (md->kv16 ? 8u : 0u) | (ly->ring ? 16u : 0u) |+ (v_is_k ? 32u : 0u) | (md->vnorm ? 64u : 0u)};janas_g_push_addr(pq.pos, POS);janas_g_push_addr(pq.qkv, QKV);- janas_g_push_addr(pq.qn, janas_gt_addr_of(g, ly->q_norm, md->hd * 4));- janas_g_push_addr(pq.kn, janas_gt_addr_of(g, ly->k_norm, md->hd * 4));+ janas_g_push_addr(pq.qn,+ ly->q_norm ? janas_gt_addr_of(g, ly->q_norm, hd * 4) : 0);+ janas_g_push_addr(pq.kn,+ ly->k_norm ? janas_gt_addr_of(g, ly->k_norm, hd * 4) : 0);janas_g_push_addr(pq.rope, ROPE);janas_g_push_addr(pq.kc, janas_gt_addr_of(g, ly->kc, cache));janas_g_push_addr(pq.vc, janas_gt_addr_of(g, ly->vc, cache));- janas_g_push_addr(pq.ks, janas_gt_addr_of(g, ly->ks, cache / md->hd * 4));- janas_g_push_addr(pq.vs, janas_gt_addr_of(g, ly->vs, cache / md->hd * 4));+ janas_g_push_addr(pq.ks, janas_gt_addr_of(g, ly->ks, scales));+ janas_g_push_addr(pq.vs, janas_gt_addr_of(g, ly->vs, scales));janas_gt_bind(g, bound, t->p_qk);janas_gt_push(g, &pq, sizeof(pq));- g->vkCmdDispatch(g->cb, md->n_head + md->n_kv, n, 1);+ g->vkCmdDispatch(g->cb, md->n_head + (own_kv ? nkv : 0), n, 1);janas_g_compute_barrier(g);janas_gt_stamp(g, t, PH_ROPE);- /* the attention: runs of positions per KV head, then joined; token- j of the block sees positions 0 .. pos + j, the block's own keys- and values written above */+ /* the attention: runs of positions per KV head (and group of its+ query heads), then joined; token j of the block sees positions 0 ..+ pos + j (or the window's last), the block's own keys and values+ written above */+ uint32_t gt = md->n_head / nkv, ga = janas_gt_att_g(gt, hd);+ int k2 = t->p_at2 && ga == t->at_g[1] && ga != t->at_g[0];struct push_at pa = {.n_head = md->n_head,- .n_kv = md->n_kv,- .hd = md->hd,- .cap = md->cap,+ .n_kv = nkv,+ .hd = hd,+ .cap = ly->cap,.nch = t->rec_nch,.scale = md->attn_scale,.qs = (uint32_t)qs,.ps = (uint32_t)janas_gt_part_tok(t, md, t->rec_nch),- .ls = t->ix_ls};+ .ls = t->ix_ls,+ .flags = (q16 ? 1u : 0u) | (md->kv16 ? 2u : 0u) |+ (ly->ring ? 4u : 0u),+ .hg = gt / ga,+ .window = ly->window};janas_g_push_addr(pa.pos, POS);/* qwen4exp's sparse attention: the positions janas_gt_rec_qsa chose */if (md->ix_heads)@@ -313,21 +372,22 @@ int janas_gt_rec_attn(struct janas_gpu *g, struct janas_gpu_tok *t,janas_g_push_addr(pa.q, QKV);janas_g_push_addr(pa.kc, janas_gt_addr_of(g, ly->kc, cache));janas_g_push_addr(pa.vc, janas_gt_addr_of(g, ly->vc, cache));- janas_g_push_addr(pa.ks, janas_gt_addr_of(g, ly->ks, cache / md->hd * 4));- janas_g_push_addr(pa.vs, janas_gt_addr_of(g, ly->vs, cache / md->hd * 4));+ janas_g_push_addr(pa.ks, janas_gt_addr_of(g, ly->ks, scales));+ janas_g_push_addr(pa.vs, janas_gt_addr_of(g, ly->vs, scales));janas_g_push_addr(pa.part, B + t->o_part);- janas_gt_bind(g, bound, t->p_at);+ janas_gt_bind(g, bound, k2 ? t->p_at2 : t->p_at);janas_gt_push(g, &pa, sizeof(pa));- g->vkCmdDispatch(g->cb, md->n_kv * t->rec_nch, n, 1);+ g->vkCmdDispatch(g->cb, nkv * pa.hg * t->rec_nch, n, 1);janas_g_compute_barrier(g);janas_gt_stamp(g, t, PH_ATTN);/* the runs joined and the output quantized for wo, a workgroup ablock of 256 */- struct push_am pm = {.hd = md->hd,+ struct push_am pm = {.hd = hd,.nch = t->rec_nch,.ps = (uint32_t)janas_gt_part_tok(t, md, t->rec_nch),.gs = qd,- .ls = t->ix_ls};+ .ls = t->ix_ls,+ .window = ly->window};janas_g_push_addr(pm.gate, md->gated_q ? B + t->o_gate : 0);if (md->ix_heads)janas_g_push_addr(pm.list, B + t->o_ixlist);@@ -345,9 +405,100 @@ int janas_gt_rec_attn(struct janas_gpu *g, struct janas_gpu_tok *t,return -1;janas_g_compute_barrier(g);janas_gt_stamp(g, t, PH_WO);+ /* Gemma 4: the output normalized before the residual, into o_post */+ if (ly->post_attn)+ janas_gt_rec_post_norm(g, t, bound, YWO, ly->post_attn, md, n);return 0;}+/* Gemma 4's per-layer branch of layer l (E2B, E4B) on the residual x (n+ tokens): its gate (f32, ple_dim rows), GELU times the layer's own inputs+ (ple_act.comp), the projection back (f32) into y, as janas_m_ple_layer+ before its norm. */+void janas_gt_rec_ple(struct janas_gpu *g, struct janas_gpu_tok *t,+ const struct janas_gpu_tok_model *md,+ const struct janas_gpu_tok_layer *ly, uint32_t l,+ VkPipeline *bound, VkDeviceAddress x, VkDeviceAddress y,+ uint32_t n)+{+ uint32_t P = md->ple_dim, dm = md->dm;+ VkDeviceAddress B = t->buf.addr;+ janas_gt_rec_dense(g, bound, 0,+ janas_gt_addr_of(g, ly->ple_gate, (size_t)P * dm * 4), x,+ B + t->o_pg, P, dm, P, n, 1.0f);+ janas_g_compute_barrier(g);+ struct push_pa pa = {.n = n, .pd = P, .w = md->n_layer * P};+ janas_g_push_addr(pa.g, B + t->o_pg);+ janas_g_push_addr(pa.in, B + t->o_ple + (size_t)l * P * 4);+ janas_g_push_addr(pa.h, B + t->o_ph);+ janas_gt_bind(g, bound, t->p_pa);+ janas_gt_push(g, &pa, sizeof(pa));+ g->vkCmdDispatch(g->cb, (n * P + 255) / 256, 1, 1);+ janas_g_compute_barrier(g);+ janas_gt_rec_dense(g, bound, 0,+ janas_gt_addr_of(g, ly->ple_proj, (size_t)dm * P * 4),+ B + t->o_ph, y, dm, P, dm, n, 1.0f);+ janas_g_compute_barrier(g);+}++/* Gemma 4's mixture (26B) up to its router, on the residual x (n tokens)+ whose feed-forward norm is in q256: the dense branch into o_ybr, that+ normalized into o_xbr; the router's input into o_rin; last the experts'+ input in q256 / q32 and XNF (the CPU's experts of a tier read it), as+ model.c's branch1_worker and prep2_worker. */+int janas_gt_rec_gemma_moe(struct janas_gpu *g, struct janas_gpu_tok *t,+ const struct janas_gpu_tok_model *md,+ const struct janas_gpu_tok_layer *ly,+ VkPipeline *bound, VkDeviceAddress x, uint32_t n)+{+ VkDeviceAddress B = t->buf.addr;+ uint32_t dm = md->dm;+ if (janas_gt_rec_ffn(g, t, bound, &ly->dn_gate, &ly->dn_up, &ly->dn_down, n,+ B + t->o_ybr) != 0)+ return -1;+ janas_gt_rec_norm_quant(g, t, bound, B + t->o_ybr, 0,+ janas_gt_addr_of(g, ly->post_norm1, dm * 4), 0,+ B + t->o_xbr, dm, md->eps, n);+ janas_gt_rec_norm_quant_p(+ g, t, bound, x, 0, janas_gt_addr_of(g, ly->rscale, dm * 4), 0,+ B + t->o_rin, dm, md->eps, n, 0.0f, 1.0f / sqrtf((float)dm));+ janas_gt_rec_norm_quant(g, t, bound, x, 0,+ janas_gt_addr_of(g, ly->pre_norm2, dm * 4), 0,+ B + t->o_xnf, dm, md->eps, n);+ return 0;+}++/* ... after its experts' sum in yd: that normalized, the dense branch+ added, the sum normalized by post_ffn into o_post (model.c's+ branch2_worker and sandwich_worker before the residual) */+void janas_gt_rec_gemma_moe_tail(struct janas_gpu *g, struct janas_gpu_tok *t,+ const struct janas_gpu_tok_model *md,+ const struct janas_gpu_tok_layer *ly,+ VkPipeline *bound, VkDeviceAddress yd,+ uint32_t n)+{+ VkDeviceAddress B = t->buf.addr;+ uint32_t dm = md->dm;+ janas_gt_rec_norm_quant(g, t, bound, yd, 0,+ janas_gt_addr_of(g, ly->post_norm2, dm * 4), 0,+ B + t->o_post2, dm, md->eps, n);+ janas_gt_rec_norm_quant(g, t, bound, B + t->o_post2, B + t->o_xbr,+ janas_gt_addr_of(g, ly->post_ffn, dm * 4), 0,+ B + t->o_post, dm, md->eps, n);+}++/* Gemma 4's sandwich norm: rms_norm(x) * w (n vectors of dm floats) into+ o_post, through norm_quant.comp (its quantized output not used). */+void janas_gt_rec_post_norm(struct janas_gpu *g, struct janas_gpu_tok *t,+ VkPipeline *bound, VkDeviceAddress x,+ const float *w,+ const struct janas_gpu_tok_model *md, uint32_t n)+{+ janas_gt_rec_norm_quant(g, t, bound, x, 0,+ janas_gt_addr_of(g, w, md->dm * 4), 0,+ t->buf.addr + t->o_post, md->dm, md->eps, n);+}+/* x (n vectors of d floats) quantized as it is, into the q256 and q32layouts at q256 and q32: norm_quant.comp with no norm. */static void rec_quant(struct janas_gpu *g, struct janas_gpu_tok *t,@@ -355,7 +506,7 @@ static void rec_quant(struct janas_gpu *g, struct janas_gpu_tok *t,VkDeviceAddress q256, VkDeviceAddress q32, uint32_t d,uint32_t n){- struct push_nq pc = {.d = d};+ struct push_nq pc = {.d = d, .det = 1}; /* as the CPU's activation */janas_g_push_addr(pc.x, x);janas_g_push_addr(pc.q256, q256);janas_g_push_addr(pc.q32, q32);@@ -387,8 +538,10 @@ int janas_gt_rec_ffn(struct janas_gpu *g, struct janas_gpu_tok *t,return -1;janas_g_compute_barrier(g);janas_gt_stamp(g, t, PH_GATEUP);- if (janas_g_x_in_32(down->type)) {- struct push_sm ps = {.n = n * fr};+ if (janas_g_x_in_32(down->type) && getenv("JANAS_GPU_ACT_FLOAT")) {+ /* the GPU's own e^x, then quantized (JANAS_GPU_ACT_FLOAT=1: as+ before 26 Oct 2026) */+ struct push_sm ps = {.n = n * fr, .gelu = (uint32_t)t->gelu};janas_g_push_addr(ps.g, GU);janas_g_push_addr(ps.u, UP);janas_g_push_addr(ps.h, B + t->o_hf);@@ -398,8 +551,23 @@ int janas_gt_rec_ffn(struct janas_gpu *g, struct janas_gpu_tok *t,janas_g_compute_barrier(g);rec_quant(g, t, bound, B + t->o_hf, H, B + t->o_h32, fr, n);XA = B + t->o_h32;+ } else if (janas_g_x_in_32(down->type)) {+ /* the CPU's bits: the deterministic activation quantized straight+ into the blocks of 32 (expert_act.comp's KIND 1, as an expert in+ blocks of 32), the groups past the width never read */+ struct push_ac ac = {.blocks = n * fr / JANAS_QK,+ .gelu = (uint32_t)t->gelu};+ janas_g_push_addr(ac.g, GU);+ janas_g_push_addr(ac.u, UP);+ janas_g_push_addr(ac.x, B + t->o_h32);+ janas_gt_bind(g, bound, g->pipea32);+ janas_gt_push(g, &ac, sizeof(ac));+ g->vkCmdDispatch(g->cb, n * fr / JANAS_QK, 1, 1);+ janas_g_compute_barrier(g);+ XA = B + t->o_h32;} else {- struct push_ac ac = {.blocks = n * fr / JANAS_QK};+ struct push_ac ac = {.blocks = n * fr / JANAS_QK,+ .gelu = (uint32_t)t->gelu};janas_g_push_addr(ac.g, GU);janas_g_push_addr(ac.u, UP);janas_g_push_addr(ac.x, H);diff --git a/src/llm/model.c b/src/llm/model.cindex 2fd4a4b..041a3ce 100644--- a/src/llm/model.c+++ b/src/llm/model.c@@ -194,6 +194,11 @@ static void run_tok(struct janas_llm_model *m, const struct layer *ly,janas_pool_run(m->compute, fn, &t);}+static int head_of_vocab(struct janas_llm_model *m,+ const struct janas_block_q8k *x, float *logits,+ uint32_t n);+static int mask_to_vocab(struct janas_llm_model *m, float *logits, uint32_t n);+/* logits soft-capped: c * tanh(l / c) */static void softcap_worker(void *arg, int tid, int n_threads){@@ -387,10 +392,76 @@ static void prefetch_decide(struct janas_llm_model *m,m->pf_would = 0;}+/*+ * A prompt's blocks read with the GPU's whole pass or without it (the+ * CPU's attention, the products where they go), whichever this machine+ * reads faster at this position: the attention's cost grows with the+ * position, and on the GPU it may grow faster or slower than on the CPU -+ * Qwen3-4B on the development laptop's Arc read 4,096 tokens at 72.6+ * tokens/s with it and 95.9 without, 16,384 at 31.8 and 55.6 (26 Oct+ * 2026), where a card with fast memory of its own may well win. So every+ * LP_BLOCKS of the engine's blocks of positions one block each way,+ * measured in turn (the first of a band alternating: a later block costs+ * more), and the faster for the rest of the band - no threshold, only how+ * often to look again, at most one slower block in LP_BLOCKS.+ * JANAS_GPU_LONG=0: the whole pass always, as before.+ */+#define LP_BLOCKS 8++static int long_prompt_on(const struct janas_llm_model *m, uint32_t n,+ uint32_t pos0)+{+ static int off = -1;+ if (off < 0) {+ const char *e = getenv("JANAS_GPU_LONG");+ off = e && atoi(e) <= 0;+ }+ (void)pos0;+ /* the whole pass asked for (JANAS_GPU_TOKEN=1): as asked */+ return !off && n >= JANAS_GPU_FAST_N && m->gtok_state == 1 &&+ janas_llm_model_gpu_prompts(m) && !janas_m_gpu_token_asked();+}++/* this block's way: lp_cpu set for the one without the whole pass */+static void long_prompt_begin(struct janas_llm_model *m, uint32_t pos0)+{+ uint32_t band = pos0 / (LP_BLOCKS * m->blk);+ if (band != m->lp.band || m->lp.state == 0) {+ if (band != m->lp.band)+ m->lp.first = (int)(band & 1u);+ m->lp.band = band;+ m->lp.state = 0;+ }+ if (m->lp.state < 2) /* the first way, then the other */+ m->lp_cpu = m->lp.state == 0 ? m->lp.first : !m->lp.first;+ else+ m->lp_cpu = m->lp.cpu < m->lp.gpu;+}++static void long_prompt_end(struct janas_llm_model *m, double sec_per_token,+ int on_gpu)+{+ int cpu = m->lp_cpu;+ m->lp_cpu = 0;+ if (m->lp.state >= 2 || sec_per_token <= 0 || on_gpu == cpu)+ return; /* chosen already; or not the way meant (none measured) */+ *(cpu ? &m->lp.cpu : &m->lp.gpu) = sec_per_token;+ if (++m->lp.state == 2 && getenv("JANAS_GPU_VERBOSE"))+ fprintf(stderr,+ "janas: a prompt from position %u: %.2f ms a token with the "+ "GPU's whole pass, %.2f without; %s\n",+ m->lp.band * LP_BLOCKS * m->blk, m->lp.gpu * 1e3,+ m->lp.cpu * 1e3, m->lp.cpu < m->lp.gpu ? "without" : "with it");+}+int janas_llm_model_forward(struct janas_llm_model *m, const int32_t *tokens,uint32_t n, uint32_t pos0, float *logits,int all_logits){+ int lp = long_prompt_on(m, n, pos0);+ uint64_t runs0 = m->gtok_runs[2];+ if (lp)+ long_prompt_begin(m, pos0);/* each pass measured for the tuner: the configuration never changesthe results */int cls = janas_tune_class(n), cand = janas_tuner_pick(&m->tuner, cls);@@ -411,6 +482,9 @@ int janas_llm_model_forward(struct janas_llm_model *m, const int32_t *tokens,/* while the cache is still filling in the background, passes are notrepresentative: neither the tuner nor the draft planner may learnfrom them, or a slow start decides the whole session */+ if (lp)+ long_prompt_end(m, r == 0 && dt > 0 ? dt / n : 0,+ m->gtok_runs[2] > runs0);if (r == 0 && dt > 0 && !janas_llm_model_settling(m))janas_tuner_record(&m->tuner, cls, cand, dt / n);if (r == 0 && n == 1 && m->pf_mode == 1)@@ -489,6 +563,15 @@ static int forward(struct janas_llm_model *m, const int32_t *tokens, uint32_t n,if (gpu != 0) /* the CPU's pass: the state the GPU made newer back */janas_m_gtok_host(m, 1);if (gpu == 0) {+ /* Gemma 4's soft cap, as below */+ if (m->softcap > 0.0f) {+ struct tok_job sc = {+ .m = m, .n = all_logits ? n : 1, .out = logits};+ janas_pool_run(m->compute, softcap_worker, &sc);+ }+ /* its head the whole vocabulary's: the others -inf here */+ if (m->vs_n && mask_to_vocab(m, logits, all_logits ? n : 1) != 0)+ return -1;m->ring_hi = pos0 + n;if (m->a->rec && n <= JANAS_LLM_MAX_BLOCK) {/* replayed and logged by the GPU: as at the end */@@ -611,16 +694,20 @@ output:janas_m_norm_quant(m, f32(m, m->out_norm), keep ? 0 : first, n);if (keep && !m->hc_n)memcpy(m->hid, m->xn, (size_t)n * dm * sizeof(float));- struct janas_matvec_task to = {.type = (int)m->output->type,- .w = data(m, m->output),- .x = m->xq + first * nbd,- .y = logits,- .rows = m->n_vocab,- .cols = dm,- .n_vec = n - first};- if (logits) /* without: the hidden states alone, an embedding */+ if (logits && m->vs_n) { /* a vocabulary of its own: its rows alone */+ if (head_of_vocab(m, m->xq + first * nbd, logits, n - first) != 0)+ return -1;+ } else if (logits) { /* without: the hidden states alone, an embedding */+ struct janas_matvec_task to = {.type = (int)m->output->type,+ .w = data(m, m->output),+ .x = m->xq + first * nbd,+ .y = logits,+ .rows = m->n_vocab,+ .cols = dm,+ .n_vec = n - first};janas_m_products(m, janas_m_gpu(m, 0), &to, 1);- if (logits && m->softcap > 0.0f) {+ }+ if (logits && m->softcap > 0.0f && !m->vs_n) {struct tok_job sc = {.m = m, .n = n - first, .out = logits};janas_pool_run(m->compute, softcap_worker, &sc);}@@ -707,6 +794,98 @@ double janas_m_exp_sum(const float *l, uint32_t n, float mx)return sum;}+int janas_llm_model_set_vocab(struct janas_llm_model *m, const int32_t *ids,+ uint32_t n)+{+ m->vs_n = m->vs_cap = 0;+ if (!n)+ return 0;+ int type = (int)m->output->type;+ if (type == JANAS_Q6_K_P || !janas_qtype_block_size(type))+ return -1;+ size_t row = m->d_model / JANAS_QK * janas_qtype_block_size(type);+ uint8_t *w = realloc(m->vs_w, (size_t)n * row);+ if (!w)+ return -1;+ m->vs_w = w;+ int32_t *v = realloc(m->vs_ids, n * sizeof(int32_t));+ if (!v)+ return -1;+ m->vs_ids = v;+ const uint8_t *src = data(m, m->output);+ uint32_t k = 0;+ for (uint32_t i = 0; i < n; i++) {+ if (ids[i] < 0 || (uint32_t)ids[i] >= m->n_vocab)+ continue;+ memcpy(m->vs_w + (size_t)k * row, src + (size_t)ids[i] * row, row);+ m->vs_ids[k++] = ids[i];+ }+ m->vs_n = k;+ return 0;+}++/* The logits of n vectors through the vocabulary's rows alone, the others+ -inf (janas_llm_model_set_vocab). */+static int vocab_room(struct janas_llm_model *m, uint32_t n)+{+ if (n > m->vs_cap) {+ float *y = realloc(m->vs_y, (size_t)n * m->vs_n * sizeof(float));+ if (!y)+ return -1;+ m->vs_y = y;+ m->vs_cap = n;+ }+ return 0;+}++/* The vocabulary's logits of n rows into vs_y, then every row -inf but+ for them. */+static void scatter_vocab(struct janas_llm_model *m, float *logits, uint32_t n)+{+ for (uint32_t j = 0; j < n; j++) {+ float *l = logits + (size_t)j * m->n_vocab;+ for (uint32_t i = 0; i < m->n_vocab; i++)+ l[i] = -INFINITY;+ for (uint32_t i = 0; i < m->vs_n; i++)+ l[m->vs_ids[i]] = m->vs_y[(size_t)j * m->vs_n + i];+ }+}++/* Logits of the whole head (the GPU's whole pass) cut to the vocabulary. */+static int mask_to_vocab(struct janas_llm_model *m, float *logits, uint32_t n)+{+ if (vocab_room(m, n) != 0)+ return -1;+ for (uint32_t j = 0; j < n; j++)+ for (uint32_t i = 0; i < m->vs_n; i++)+ m->vs_y[(size_t)j * m->vs_n + i] =+ logits[(size_t)j * m->n_vocab + m->vs_ids[i]];+ scatter_vocab(m, logits, n);+ return 0;+}++static int head_of_vocab(struct janas_llm_model *m,+ const struct janas_block_q8k *x, float *logits,+ uint32_t n)+{+ if (vocab_room(m, n) != 0)+ return -1;+ struct janas_matvec_task t = {.type = (int)m->output->type,+ .w = m->vs_w,+ .x = x,+ .y = m->vs_y,+ .rows = m->vs_n,+ .cols = m->d_model,+ .n_vec = n};+ /* on the CPU: the GPU and the NPU know the model's weights, not these */+ janas_matvec_q4k_group(m->compute, &t, 1);+ if (m->softcap > 0.0f) /* as softcap_worker, on the rows computed */+ for (size_t i = 0; i < (size_t)n * m->vs_n; i++)+ m->vs_y[i] = m->softcap * janas_tanhf(m->vs_y[i] / m->softcap);+ scatter_vocab(m, logits, n);+ return 0;+}+void janas_llm_mtp_note_tokens(struct janas_llm_model *m, const int32_t *tokens,uint32_t n){diff --git a/src/llm/model.h b/src/llm/model.hindex 2543c69..2914166 100644--- a/src/llm/model.h+++ b/src/llm/model.h@@ -241,6 +241,18 @@ int janas_llm_model_decode(struct janas_llm_model *m, int32_t token,* guesses are chained.*/int janas_llm_model_has_mtp(const struct janas_llm_model *m);++/*+ * The tokens the model may give from now on, n of them (n 0: all of them+ * again). Their rows of the LM head are copied into one matrix and the+ * forward pass computes those alone - a few thousand of 128,257 rows, at a+ * fraction of the head's reads - the other logits set to -inf. The MTP+ * block's and the draft head's logits are not restricted: a draft outside+ * the vocabulary is rejected by the pass that checks it. -1 when the head's+ * rows cannot be copied (its weights in bit planes) or memory runs out.+ */+int janas_llm_model_set_vocab(struct janas_llm_model *m, const int32_t *ids,+ uint32_t n);/* 1 when chained drafts keep the position of the first (Gemma 4'sassistant, which reads the main model's cache and has none of itsown); 0 when each is one position further (an MTP block) */diff --git a/src/llm/model_attn.c b/src/llm/model_attn.cindex 16ebdd6..3b0974b 100644--- a/src/llm/model_attn.c+++ b/src/llm/model_attn.c@@ -90,13 +90,15 @@ static void qkv_item(const struct tok_job *t, uint32_t j, uint32_t h)memcpy(m->qgate + (size_t)j * qd + (size_t)h * hd, src + hd,hd * sizeof(float));}- rms_norm(v, v, f32(m, ly->q_norm), hd, m->eps);+ if (ly->q_norm)+ rms_norm(v, v, f32(m, ly->q_norm), hd, m->eps);rope_neox(v, ly->n_rot, cs, sn);return;}h -= m->n_head;float *kv = m->kk + (size_t)j * kvd + (size_t)h * hd;- rms_norm(kv, kv, f32(m, ly->k_norm), hd, m->eps);+ if (ly->k_norm)+ rms_norm(kv, kv, f32(m, ly->k_norm), hd, m->eps);rope_neox(kv, ly->n_rot, cs, sn);uint32_t p = t->pos0 + j, row = t->ring ? p % t->cap : p;size_t at = ((size_t)h * t->cap + row) * hd;diff --git a/src/llm/model_ffn.c b/src/llm/model_ffn.cindex 45605a7..8d1a634 100644--- a/src/llm/model_ffn.c+++ b/src/llm/model_ffn.c@@ -374,10 +374,10 @@ int janas_m_expert_rows(struct janas_llm_model *m, uint32_t l, uint32_t n,uint32_t dm = m->d_model, nbd = dm / JANAS_QK;if (n > m->blk)return -1;- /* as the GPU quantizes the same values (norm_quant.comp): then an- expert computed here gives the GPU's bits */+ /* as the GPU quantizes the same values (norm_quant.comp, as the CPU's+ own path): then an expert computed here gives the GPU's bits */for (uint32_t j = 0; j < n; j++)- janas_q8k_quantize_det(xn + (size_t)j * dm, m->xq + j * nbd, dm);+ janas_q8k_quantize(xn + (size_t)j * dm, m->xq + j * nbd, dm);const struct janas_jns_layer *jl = &m->j.layers[l];/* the CPU's products only: the GPU waits for these between two of itssubmissions */diff --git a/src/llm/model_gpu.c b/src/llm/model_gpu.cindex 0617fc5..dc63c55 100644--- a/src/llm/model_gpu.c+++ b/src/llm/model_gpu.c@@ -60,10 +60,17 @@ static const char *not_for_gpu(const struct janas_llm_model *m)/* qwen4exp: qwen3.5's layers inside its streams (model_hc.c), with itsmixes in bf16 as the file has them, or their down and up rewrittenas Q8_0 (JANAS_HC_MIX=q8) */- if ((m->a->rec && !q35) || (m->a->attn_gate && !q35) || m->a->swa ||- m->a->sandwich || m->a->gelu || m->kv16 || m->softcap > 0.0f ||- m->ple_dim)+ /* Gemma 4's dense layers (sliding windows, sandwich norms, GELU,+ sixteen-bit keys and values, the soft cap on the CPU after); not yet+ its per-layer embeddings (E2B, E4B) */+ if ((m->a->rec && !q35) || (m->a->attn_gate && !q35))return "an architecture the token's shaders do not take yet";+ /* per-layer embeddings: the branch's matrices f32, as E2B's and E4B's */+ for (uint32_t l = 0; m->ple_dim && l < m->n_layer; l++)+ if (m->layers[l].ple_gate->type != 0 ||+ m->layers[l].ple_proj->type != 0)+ return "per-layer embeddings of a type the token's shaders do "+ "not take";if (m->a->hc) {int mt = m->hc_quant[0] ? JANAS_Q8_0 : 30; /* down and up's */for (uint32_t l = 0; l <= m->n_layer; l++)@@ -124,7 +131,8 @@ static const char *not_for_gpu(const struct janas_llm_model *m)if (janas_expert_cache_slots(m->cache) < m->n_expert)return "an expert cache smaller than a layer's experts";for (uint32_t l = 0; l < m->n_layer; l++)- if (m->layers[l].exp_scale || !m->layers[l].router16)+ if (!m->layers[l].router16 ||+ (m->layers[l].exp_scale && !m->layers[l].pre_norm2))return "a mixture the token's shaders do not take yet";}if (!janas_m_gpu_tier(m) && (m->arena_gpu != 1 || m->arena_stale))@@ -137,10 +145,11 @@ static const char *not_for_gpu(const struct janas_llm_model *m)return "a layer the token's shaders do not take yet";continue;}- if (ly->swa || !ly->wk || !ly->wv || !ly->q_norm || !ly->k_norm ||- ly->post_attn || ly->post_ffn || ly->pre_norm2 ||- m->kv_ring[ly->slot] || ly->hd != m->head_dim ||- ly->n_kv != m->n_head_kv || ly->n_rot != m->n_rot)+ /* not yet: a layer that reads another's keys and values (Gemma 4's+ E2B, E4B), Gemma 4's mixture */+ int shared = m->n_kv_layers && l >= m->n_kv_layers;+ if ((!ly->wk && !shared) || (ly->pre_norm2 && !m->a->swa) ||+ (!ly->wv && !m->a->swa))return "a layer the token's shaders do not take yet";}return NULL;@@ -498,8 +507,7 @@ static size_t tier_bytes(const struct janas_llm_model *m)/* The layers described for the GPU, the KV cache imported: 0, or -1. */static int build(struct janas_llm_model *m, char *err, size_t len){- uint32_t dm = m->d_model, hd = m->head_dim, qd = m->n_head * hd,- kvd = m->n_head_kv * hd;+ uint32_t dm = m->d_model;if (!m->gtok_kv) {if (janas_gpu_import(m->gpu, m->kcache, m->kv_bytes) != 0 ||janas_gpu_import(m->gpu, m->vcache, m->kv_bytes) != 0 ||@@ -590,12 +598,33 @@ static int build(struct janas_llm_model *m, char *err, size_t len)g->ssm_a = f32(m, ly->ssm_a);g->ssm_norm = f32(m, ly->ssm_norm);} else {+ /* the layer's own geometry (Gemma 4's two kinds) */+ uint32_t qd = m->n_head * ly->hd, kvd = ly->n_kv * ly->hd;+ g->hd = ly->hd;+ g->n_kv = ly->n_kv;+ g->n_rot = ly->n_rot;+ g->swa = ly->swa;+ g->cap = m->kv_cap[ly->slot];+ g->ring = m->kv_ring[ly->slot];+ g->window = ly->swa ? m->window : 0;+ g->post_attn = ly->post_attn ? f32(m, ly->post_attn) : NULL;+ g->post_ffn = ly->post_ffn ? f32(m, ly->post_ffn) : NULL;+ g->out_scale = ly->out_scale;g->slot = ly->slot; /* its KV slot: att_in's row (qwen4exp) */- g->q_norm = f32(m, ly->q_norm);- g->k_norm = f32(m, ly->k_norm);+ g->q_norm = ly->q_norm ? f32(m, ly->q_norm) : NULL; /* llama */+ g->k_norm = ly->k_norm ? f32(m, ly->k_norm) : NULL;g->wq = task_of(m, ly->wq, m->a->attn_gate ? 2 * qd : qd, dm);- g->wk = task_of(m, ly->wk, kvd, dm);- g->wv = task_of(m, ly->wv, kvd, dm);+ /* none (type 0): another layer's keys and values (Gemma 4's+ E2B, E4B); V is K (its full layers) */+ if (ly->wk)+ g->wk = task_of(m, ly->wk, kvd, dm);+ if (ly->wv)+ g->wv = task_of(m, ly->wv, kvd, dm);+ if (m->ple_dim) {+ g->ple_gate = f32(m, ly->ple_gate);+ g->ple_proj = f32(m, ly->ple_proj);+ g->ple_post = f32(m, ly->ple_post);+ }g->wo = task_of(m, ly->wo, dm, qd);if (m->ix_heads && ly->ix_q) {size_t nbk = (m->n_ctx + m->ix_ratio - 1) / m->ix_ratio;@@ -639,19 +668,40 @@ static int build(struct janas_llm_model *m, char *err, size_t len).rows = ly->sh_mx[p].rows,.cols = ly->sh_mx[p].cols};}+ /* Gemma 4's: the shared expert a dense branch of its own,+ normalized, not in the experts' sum */+ if (ly->pre_norm2) {+ g->dn_gate = g->sh_gate_w;+ g->dn_up = g->sh_up;+ g->dn_down = g->sh_down;+ g->sh_gate_w = g->sh_up = g->sh_down =+ (struct janas_matvec_task){0};+ g->pre_norm2 = f32(m, ly->pre_norm2);+ g->post_norm1 = f32(m, ly->post_norm1);+ g->post_norm2 = f32(m, ly->post_norm2);+ g->rscale = ly->router_scale;+ g->exp_scale = ly->exp_scale;+ }}/* a mixture: its experts' width, and its shared expert's (thefeed-forward the token's buffers are made for) */- uint32_t lff =- moe && g->sh_gate_w.type ? g->sh_gate_w.rows : g->gate.rows;+ uint32_t lff = moe && g->sh_gate_w.type ? g->sh_gate_w.rows+ : moe && g->dn_gate.type ? g->dn_gate.rows+ : g->gate.rows;if (l == 0) {ff = lff;eff = g->gate.rows;}- if (lff != ff || g->gate.rows != eff || g->up.rows != eff ||- g->down.cols != eff ||- (moe && g->sh_gate_w.type &&- (g->sh_up.rows != ff || g->sh_down.cols != ff))) {+ /* a dense model's widths may differ layer by layer (Gemma 4's+ E2B): the token's buffers for the widest */+ if (!moe && g->up.rows == g->gate.rows &&+ g->down.cols == g->gate.rows) {+ if (lff > ff)+ ff = eff = lff;+ } else if (lff != ff || g->gate.rows != eff || g->up.rows != eff ||+ g->down.cols != eff ||+ (moe && g->sh_gate_w.type &&+ (g->sh_up.rows != ff || g->sh_down.cols != ff))) {snprintf(err, len, "feed-forwards of different widths");return -1;}@@ -669,18 +719,40 @@ static int build(struct janas_llm_model *m, char *err, size_t len)snprintf(err, len, "no attention layer");return -1;}+ /* the token's buffers for the largest of any layer: Gemma 4's full+ layers' heads, its sliding ones' KV heads; the full layers' cache */+ uint32_t hd = m->a->swa ? m->hd_max : m->head_dim, nkv = m->n_head_kv,+ room = 0;+ for (uint32_t l = 0; l < m->n_layer; l++) {+ const struct janas_gpu_tok_layer *g = &m->gtok_layers[l];+ if (g->rec)+ continue;+ if (g->n_kv > nkv)+ nkv = g->n_kv;+ /* a ring holds the window before a block's first token and the+ block */+ if (g->ring && (!room || g->cap - g->window + 1 < room))+ room = g->cap - g->window + 1;+ }m->gtok_md = (struct janas_gpu_tok_model){.n_layer = m->n_layer,.dm = dm,.n_head = m->n_head,- .n_kv = m->n_head_kv,+ .n_kv = nkv,.hd = hd,.n_rot = m->n_rot,.ff = ff,.n_vocab = m->n_vocab,- .cap = m->kv_cap[m->layers[a0].slot],+ .cap = m->a->swa ? m->n_ctx : m->kv_cap[m->layers[a0].slot],.eps = m->eps,- .attn_scale = 1.0f / sqrtf((float)hd),+ /* Gemma 4: the q and k norms scale the scores */+ .attn_scale = m->a->swa ? 1.0f : 1.0f / sqrtf((float)hd),+ .gelu = m->a->gelu,+ .kv16 = m->kv16,+ .vnorm = m->a->swa,+ .n_rot_swa = m->a->swa ? m->n_rot_swa : 0,+ .ring_room = room,+ .ple_dim = m->ple_dim,.layers = m->gtok_layers,.out_norm = m->a->hc ? NULL : f32(m, m->out_norm),.output = task_of(m, m->output, m->n_vocab, dm),@@ -792,7 +864,7 @@ int janas_m_gpu_token(struct janas_llm_model *m, uint32_t pos, uint32_t n,tok = e && atoi(e) > 0;blocks = b ? atoi(b) > 0 : -1; /* -1: not said */}- int prompt = janas_gpu_fast_prompt() && !m->prompts_off_gpu &&+ int prompt = janas_gpu_fast_prompt() && !m->prompts_off_gpu && !m->lp_cpu &&n >= JANAS_GPU_FAST_N && blocks != 0;/* A short pass (a chain of drafts to verify) where the tuner chose theGPU for it: the whole pass in one submission, the CPU idle, rather@@ -816,7 +888,8 @@ int janas_m_gpu_token(struct janas_llm_model *m, uint32_t pos, uint32_t n,bus between them gone (an Arc B580 read a prompt at 15.8 tokens/swith the products shared with the CPU, issue #43); JANAS_GPU_WHOLE=0as on an integrated GPU */- int whole = janas_m_gpu_whole() && m->gpu && !janas_gpu_shares_host(m->gpu);+ int whole = janas_m_gpu_whole() && m->gpu &&+ !janas_gpu_shares_host(m->gpu) && !m->lp_cpu;if (!prompt && !short_pass && !whole && !(tok && (n == 1 || blocks > 0)))return -1;/* a recurrent model: the GPU replays first what forward() asks (the@@ -881,6 +954,14 @@ int janas_m_gpu_token(struct janas_llm_model *m, uint32_t pos, uint32_t n,if (n - j - k > 0 && n - j - k + 1 < rows)k -= rows; /* the last piece long enough */m->gtok_md.ng_emb = m->ng_emb + (size_t)j * m->d_model;+ if (m->ple_dim) /* Gemma 4's per-layer inputs */+ m->gtok_md.ple_in = m->ple + (size_t)j * m->n_layer * m->ple_dim;+ if (m->a->swa) { /* Gemma 4's sliding layers' RoPE tables */+ m->gtok_md.cs_swa =+ m->rope_cos_swa + (size_t)j * (m->n_rot_swa / 2);+ m->gtok_md.sn_swa =+ m->rope_sin_swa + (size_t)j * (m->n_rot_swa / 2);+ }if (m->gtok_md.ix_heads)janas_m_qsa_rope(m, m->gtok_ix_rope, pos + j, k);int rc = janas_gpu_tok_run(diff --git a/src/llm/model_impl.h b/src/llm/model_impl.hindex d066a90..f1aa6e5 100644--- a/src/llm/model_impl.h+++ b/src/llm/model_impl.h@@ -117,6 +117,16 @@ struct janas_llm_model {/* the user keeps prompts off the GPU (--prompt-on cpu, npu): its fastprompt, which reads them whatever the tuner chose, does not */int prompts_off_gpu;+ /* a long prompt's blocks: the GPU's whole pass against the path+ without it, measured in turn every LP_BLOCKS blocks (model.c+ long_prompt_*); lp_cpu: this block without it */+ struct {+ uint32_t band;+ int state; /* 0: none measured, 1: one, 2: both */+ int first; /* which went first: 0 the GPU's */+ double gpu, cpu; /* seconds a token */+ } lp;+ int lp_cpu;/* the user's devices for prompts, drafts, replies (common/devices.h:janas_devices_vars' order), as this machine has them; 0 as theengine chooses */@@ -309,6 +319,14 @@ struct janas_llm_model {uint32_t dh_n, dh_cap;size_t dh_row;float *dh_logits;+ /* The vocabulary of the replies (janas_llm_model_set_vocab): the head's+ rows of those tokens alone, copied as the draft head's, computed in+ place of the whole head, the other logits -inf; vs_n 0: the whole */+ uint8_t *vs_w;+ int32_t *vs_ids;+ uint32_t vs_n;+ float *vs_y; /* their logits, vs_cap vectors */+ uint32_t vs_cap;float *big, *qgate, *ba, *mixed, *conv_out, *beta, *decay;struct janas_block_q8k *xq, *attq;/* routed experts: one pair per (token, chosen expert), sorted by expert,diff --git a/src/llm/model_load.c b/src/llm/model_load.cindex 5617e99..de36736 100644--- a/src/llm/model_load.c+++ b/src/llm/model_load.c@@ -402,7 +402,9 @@ static int load_layer(struct janas_llm_model *m, const struct janas_jns *j,} else {NEED(wq, "attn_q.weight", T_QUANT, dm, qd * (m->a->attn_gate ? 2 : 1))NEED(wo, "attn_output.weight", T_QUANT, qd, dm)- NEED(q_norm, "attn_q_norm.weight", T_F32, ly->hd, 0)+ if (!m->a->no_qk_norm) {+ NEED(q_norm, "attn_q_norm.weight", T_F32, ly->hd, 0)+ }/* a layer reading an earlier one's keys and values has no use forits k and v, even where the file has them */if (!m->n_kv_layers || l < m->n_kv_layers) {@@ -411,7 +413,9 @@ static int load_layer(struct janas_llm_model *m, const struct janas_jns *j,if (!m->a->swa || janas_jns_tensor(j, name)) {NEED(wv, "attn_v.weight", T_QUANT, dm, kvd)}- NEED(k_norm, "attn_k_norm.weight", T_F32, ly->hd, 0)+ if (!m->a->no_qk_norm) {+ NEED(k_norm, "attn_k_norm.weight", T_F32, ly->hd, 0)+ }}}if (m->ple_dim) {@@ -966,9 +970,10 @@ struct janas_llm_model *janas_llm_model_load(const char *path,if (!m->mtp && m->j.h.n_layer > m->n_layer &&load_mtp_internal(m, err, err_len) != 0)goto fail;- if (m->a->swa) {+ {/* the full layers' RoPE factors (theta divided by them, pair by- pair: Gemma 4 turns only the first quarter of a head) */+ pair: Gemma 4 turns only the first quarter of a head; Llama 3's+ long context) */const struct janas_jns_tensor *rf =janas_jns_tensor(&m->j, "rope_freqs.weight");if (rf && (rf->type != 0 || rf->dims[0] != m->n_rot / 2)) {@@ -1555,6 +1560,9 @@ void janas_llm_model_free(struct janas_llm_model *m)free(m->dh_in);free(m->dh_ids);free(m->dh_logits);+ free(m->vs_w);+ free(m->vs_ids);+ free(m->vs_y);free(m->mtp_layer.sh_gate);janas_jns_close(&m->j);free(m);diff --git a/src/llm/shaders/attn_decode.comp b/src/llm/shaders/attn_decode.compindex 60fdea3..6c617c9 100644--- a/src/llm/shaders/attn_decode.comp+++ b/src/llm/shaders/attn_decode.comp@@ -8,7 +8,9 @@// For each query head h and run c: scores q.k x the key's scale x `scale`// (q at eight bits too, a scale per head, as llama.cpp takes it with an// eight-bit cache: the dot products are integer, four bytes at a time; q-// left so by qk_rope.comp, hd / 4 words then the scale in a head's floats),+// left so by qk_rope.comp, hd / 4 words then the scale in a head's floats;+// with q16 at sixteen bits, hd / 4 words of its high bytes, hd / 4 of its+// low ones, then the scale: two integer products, 256 hi.k + lo.k),// their maximum m, the sum l of e^(s - m), and the values weighted by e^(s// - m) (and the value's scale), written to part at (h nch + c)(hd + 2):// the hd sums, then m and l. attn_merge.comp joins the runs. Workgroup x =@@ -16,7 +18,13 @@// the dispatch recorded once serves every position. With list set// (qwen4exp's sparse attention, ix_select.comp) token j reads the positions// of row j of list (ls words: their count, then the positions) instead of-// 0 .. pos + j, run c the c-th C of them. The GPU's own order of the sums:+// 0 .. pos + j, run c the c-th C of them. Gemma 4: a KV head's query+// heads split among hg workgroups, G each (workgroup x = (kh hg + i) nch ++// c, its heads (kh hg + i) G ...: the 16 heads of 512 of the full layers do+// not fit one); a window (a token sees its last `window` positions, the+// runs wholly before them leave, the positions before it weigh nothing);+// a ring (position q in row q mod cap); keys and values at sixteen bits+// (flags' bit 1) with q in float, the scores float products. The GPU's own order of the sums:// not the CPU's bits (attention.c), the same result to rounding.#extension GL_EXT_buffer_reference : require#extension GL_EXT_buffer_reference_uvec2 : require@@ -46,14 +54,17 @@ layout(push_constant) uniform P {uint qs, ps; /* floats from a token's qkv, and its runs, to the next's */uvec2 pos; /* the token's position, read at run time: npos = pos + 1 */uvec2 list;- uint ls;+ uint ls, flags; /* bit 0: q at sixteen bits, 1: the cache's, 2: a ring */+ uint hg, window;} p;shared float sq[MAXQ]; /* the G query heads */shared float sc[MAXS]; /* scores, then weights: head g at g C */shared float r8[MAXS / 8u]; /* the maxima of eight scores */shared float r8s[MAXS / 8u]; /* and the sums of eight weights */-shared uint sqw[MAXQ / 4u]; /* the query heads at eight bits, four a word */+shared uint sqw[MAXQ / 2u]; /* the query heads at eight bits, four a word+ (with q16 their two halves, a head's high+ words then its low ones) */shared float sqs[16]; /* and their scales */shared uint spos[C]; /* the run's positions in the cache */@@ -64,7 +75,8 @@ uint comp(uvec4 v, uint i) { return i == 0u ? v.x : i == 1u ? v.y : i == 2u ? v.void main(){uint t = gl_LocalInvocationID.x, hd = p.hd;- uint kh = gl_WorkGroupID.x / p.nch, c = gl_WorkGroupID.x % p.nch;+ uint u = gl_WorkGroupID.x / p.nch, c = gl_WorkGroupID.x % p.nch, kh = u / p.hg;+ bool q16 = (p.flags & 1u) != 0u, kv16 = (p.flags & 2u) != 0u, ring = (p.flags & 4u) != 0u;/* token j of a block (workgroup y): positions 0 .. pos + j */uint j0 = gl_WorkGroupID.y;bool lst = p.list.x != 0u || p.list.y != 0u;@@ -72,45 +84,99 @@ void main()uint npos = lst ? LP.v[j0 * p.ls] : RP(p.pos).v[0] + j0 + 1u, p0 = c * C;if (p0 >= npos)return; /* the whole workgroup: past the token */+ /* a window: the first position the token sees */+ uint lo = !lst && p.window != 0u && npos > p.window ? npos - p.window : 0u;+ if (p0 + C <= lo)+ return; /* the whole workgroup: before the window */uint n = min(C, npos - p0);/* each of the run's positions, as an index of the cache's rows */- for (uint k = t; k < C; k += 256u)- spos[k] = kh * p.cap + (lst ? LP.v[j0 * p.ls + 1u + p0 + min(k, n - 1u)] : p0 + min(k, n - 1u));+ for (uint k = t; k < C; k += 256u) {+ uint q = lst ? LP.v[j0 * p.ls + 1u + p0 + min(k, n - 1u)] : p0 + min(k, n - 1u);+ spos[k] = kh * p.cap + (ring ? q % p.cap : q);+ }R Q = R(p.q), KS = R(p.ks), VS = R(p.vs);RP QW = RP(p.q);B4 KC = B4(p.kc), VC = B4(p.vc); /* 16 bytes a load: hd % 16 == 0 *//* the G query heads at eight bits and their scales, as qk_rope.compleft them */- uint qo = j0 * p.qs + kh * G * hd, nw = hd / 4u;- for (uint k = t; k < G * nw; k += 256u)- sqw[k] = QW.v[qo + (k / nw) * hd + k % nw];- if (t < G)- sqs[t] = Q.v[qo + t * hd + nw];+ uint qo = j0 * p.qs + u * G * hd, nw = hd / 4u, hw = q16 ? 2u * nw : nw;+ if (kv16) {+ /* the query heads as they are */+ for (uint k = t; k < G * hd; k += 256u)+ sq[k] = Q.v[qo + k];+ } else {+ for (uint k = t; k < G * hw; k += 256u)+ sqw[k] = QW.v[qo + (k / hw) * hd + k % hw];+ if (t < G)+ sqs[t] = Q.v[qo + t * hd + hw];+ }barrier();/* scores: four threads a position (256 threads: C / 64 rounds), eacha quarter of the key for all G heads at once - the key read once -the quarters added across the cluster */for (uint r0 = 0; r0 < C; r0 += 64u) {uint j = r0 + t / 4u, qq = t % 4u, jj = min(j, n - 1u);+ bool seen = j < n && p0 + j >= lo;+ if (kv16) {+ /* eight keys' numbers a load, float products */+ uint w8 = spos[jj] * hd / 8u;+ float df[G];+ for (uint g = 0; g < G; g++)+ df[g] = 0.0;+ for (uint w = qq; w < hd / 8u; w += 4u) {+ uvec4 k4 = KC.v[w8 + w];+ for (uint e = 0; e < 8u; e++) {+ float kv = float(bitfieldExtract(int(comp(k4, e / 2u)), int(16u * (e % 2u)), 16));+ for (uint g = 0; g < G; g++)+ df[g] += sq[g * hd + 8u * w + e] * kv;+ }+ }+ float ksf = KS.v[spos[jj]] * p.scale;+ for (uint g = 0; g < G; g++) {+ float d = subgroupClusteredAdd(df[g], 4u);+ if (qq == 0u && j < n)+ sc[g * C + j] = seen ? d * ksf : -1.0 / 0.0;+ }+ continue;+ }uint w0 = spos[jj] * hd / 16u;int dg[G];for (uint g = 0; g < G; g++)dg[g] = 0;- for (uint w = qq; w < hd / 16u; w += 4u) {- uvec4 k4 = KC.v[w0 + w];- for (uint g = 0; g < G; g++) {- uint o = g * hd / 4u + 4u * w;- dg[g] += dotPacked4x8EXT(int(sqw[o]), int(k4.x)) +- dotPacked4x8EXT(int(sqw[o + 1u]), int(k4.y)) +- dotPacked4x8EXT(int(sqw[o + 2u]), int(k4.z)) +- dotPacked4x8EXT(int(sqw[o + 3u]), int(k4.w));+ if (q16) {+ /* hi and lo within a byte, k too: |256 hi.k + lo.k| below+ 2^31 for hd up to 256 */+ for (uint w = qq; w < hd / 16u; w += 4u) {+ uvec4 k4 = KC.v[w0 + w];+ for (uint g = 0; g < G; g++) {+ uint o = g * hw + 4u * w, l = o + nw;+ int dh = dotPacked4x8EXT(int(sqw[o]), int(k4.x)) ++ dotPacked4x8EXT(int(sqw[o + 1u]), int(k4.y)) ++ dotPacked4x8EXT(int(sqw[o + 2u]), int(k4.z)) ++ dotPacked4x8EXT(int(sqw[o + 3u]), int(k4.w));+ int dl = dotPacked4x8EXT(int(sqw[l]), int(k4.x)) ++ dotPacked4x8EXT(int(sqw[l + 1u]), int(k4.y)) ++ dotPacked4x8EXT(int(sqw[l + 2u]), int(k4.z)) ++ dotPacked4x8EXT(int(sqw[l + 3u]), int(k4.w));+ dg[g] += 256 * dh + dl;+ }+ }+ } else+ for (uint w = qq; w < hd / 16u; w += 4u) {+ uvec4 k4 = KC.v[w0 + w];+ for (uint g = 0; g < G; g++) {+ uint o = g * hd / 4u + 4u * w;+ dg[g] += dotPacked4x8EXT(int(sqw[o]), int(k4.x)) ++ dotPacked4x8EXT(int(sqw[o + 1u]), int(k4.y)) ++ dotPacked4x8EXT(int(sqw[o + 2u]), int(k4.z)) ++ dotPacked4x8EXT(int(sqw[o + 3u]), int(k4.w));+ }}- }float ksc = KS.v[spos[jj]] * p.scale;for (uint g = 0; g < G; g++) {int d = subgroupClusteredAdd(dg[g], 4u);if (qq == 0u && j < n)- sc[g * C + j] = float(d) * sqs[g] * ksc;+ sc[g * C + j] = seen ? float(d) * sqs[g] * ksc : -1.0 / 0.0;}}barrier();@@ -151,7 +217,26 @@ void main()heads): a thread 16 dimensions of one head. SH: the slice sets leftfor the shared memory. */uint SH;- if (G <= 4u && n > 64u) {+ if (kv16) {+ /* sixteen-bit values: a thread 8 dimensions of one head, eight+ numbers a load */+ uint U = G * hd / 8u;+ SH = min(256u / U, MAXQ / (G * hd));+ if (t < U * SH) {+ uint uu = t % U, sl_ = t / U, g = uu / (hd / 8u), d0 = (uu % (hd / 8u)) * 8u;+ float a[8];+ for (uint k = 0; k < 8u; k++)+ a[k] = 0.0;+ for (uint j = sl_; j < n; j += SH) {+ uvec4 v4 = VC.v[(spos[j] * hd + d0) / 8u];+ float wj = sc[g * C + j];+ for (uint k = 0; k < 8u; k++)+ a[k] += wj * float(bitfieldExtract(int(comp(v4, k / 2u)), int(16u * (k % 2u)), 16));+ }+ for (uint k = 0; k < 8u; k++)+ sq[sl_ * G * hd + g * hd + d0 + k] = a[k];+ }+ } else if (G <= 4u && n > 64u) {uint U = hd / 8u;SH = min(256u / (4u * U), MAXQ / (G * hd));if (t < 4u * U * SH) {@@ -200,7 +285,7 @@ void main()float a = sq[k];for (uint q = 1u; q < SH; q++)a += sq[q * G * hd + k];- uint h = kh * G + g;+ uint h = u * G + g;O.v[j0 * p.ps + (h * p.nch + c) * (hd + 2u) + i] = a;}if (t < G) {@@ -210,7 +295,7 @@ void main()m = max(m, r8[t * (C / 8u) + q]);for (uint q = 0; q < C / 8u; q++)l += r8s[t * (C / 8u) + q];- uint h = kh * G + t;+ uint h = u * G + t;O.v[j0 * p.ps + (h * p.nch + c) * (hd + 2u) + hd] = m;O.v[j0 * p.ps + (h * p.nch + c) * (hd + 2u) + hd + 1u] = l;}diff --git a/src/llm/shaders/attn_merge.comp b/src/llm/shaders/attn_merge.compindex 2d26021..9870a2e 100644--- a/src/llm/shaders/attn_merge.comp+++ b/src/llm/shaders/attn_merge.comp@@ -9,7 +9,8 @@// divided by the total - then quantized to Q8_K as norm_quant.comp does// (q256, and q32 if set): no dispatch of its own between the two. With// gate set (qwen3.5's attention), each output times sigmoid of its gate-// (gs floats a token) before.+// (gs floats a token) before. With a window (Gemma 4's sliding layers) the+// runs wholly before it, which attn_decode.comp left, are not read.#extension GL_EXT_buffer_reference : require#extension GL_KHR_shader_subgroup_basic : require#extension GL_KHR_shader_subgroup_clustered : require@@ -30,7 +31,7 @@ layout(push_constant) uniform P {uvec2 pos;uvec2 gate;uvec2 list; /* attn_decode.comp's, if set: the positions' count */- uint ls;+ uint ls, window;} p;shared float m8[32]; /* the block's maxima, eight values each */@@ -46,6 +47,16 @@ float recip_det(float d)return y;}+/* a / b correctly rounded (b normal, positive), as the CPU's division:+ the output quantized as janas_q8k_quantize does it (att_out_tok) */+float div_cr(float a, float b)+{+ precise float r = recip_det(b);+ precise float q = a * r;+ precise float e = fma(-q, b, a);+ return fma(e, r, q);+}+int q8(float v, float id){precise float q = roundEven(v * id);@@ -61,14 +72,16 @@ void main()bool lst = set(p.list);uint npos = lst ? RP(p.list).v[j0 * p.ls] : RP(p.pos).v[0] + j0 + 1u;uint runs = (npos + C - 1u) / C; /* rounded up */+ uint lo = !lst && p.window != 0u && npos > p.window ? npos - p.window : 0u;+ uint c0 = lo / C; /* the first run attn_decode.comp computed */R PR = R(p.part);uint e = 256u * b + t, h = e / hd, i = e % hd;uint at = j0 * p.ps + h * p.nch * (hd + 2u);- float M = PR.v[at + hd];- for (uint c = 1u; c < runs; c++)+ float M = PR.v[at + c0 * (hd + 2u) + hd];+ for (uint c = c0 + 1u; c < runs; c++)M = max(M, PR.v[at + c * (hd + 2u) + hd]);float L = 0.0, a = 0.0;- for (uint c = 0; c < runs; c++) {+ for (uint c = c0; c < runs; c++) {uint o = at + c * (hd + 2u);float f = exp(PR.v[o + hd] - M);L += PR.v[o + hd + 1u] * f;@@ -89,10 +102,10 @@ void main()float m256 = 0.0;for (uint k = 0; k < 32u; k++)m256 = max(m256, m8[k]);- precise float dd = m256 * 0.007874015718698502;- float id = dd > 0.0 ? recip_det(dd) : 0.0;- precise float d32 = m32 * 0.007874015718698502;- float id32 = d32 > 0.0 ? recip_det(d32) : 0.0;+ precise float dd = div_cr(m256, 127.0);+ float id = dd > 0.0 ? div_cr(1.0, dd) : 0.0;+ precise float d32 = div_cr(m32, 127.0);+ float id32 = d32 > 0.0 ? div_cr(1.0, d32) : 0.0;int qa = q8(v, id), qb = q8(v, id32);/* four quants a word, from four neighbouring threads */uint wa = subgroupClusteredOr((uint(qa) & 0xffu) << (8u * (t % 4u)), 4u);@@ -117,8 +130,7 @@ void main()int sb0 = s8b[4u * t] + s8b[4u * t + 1u], sb1 = s8b[4u * t + 2u] + s8b[4u * t + 3u];Q32.v[base + 64u + t] = (uint(sb0) & 0xffffu) | (uint(sb1) << 16);precise float dt =- max(max(m8[4u * t], m8[4u * t + 1u]), max(m8[4u * t + 2u], m8[4u * t + 3u])) *- 0.007874015718698502;+ div_cr(max(max(m8[4u * t], m8[4u * t + 1u]), max(m8[4u * t + 2u], m8[4u * t + 3u])), 127.0);Q32.v[base + 72u + t] = floatBitsToUint(dt);}}diff --git a/src/llm/shaders/expert_act.comp b/src/llm/shaders/expert_act.compindex 86ef659..d0edadd 100644--- a/src/llm/shaders/expert_act.comp+++ b/src/llm/shaders/expert_act.comp@@ -3,7 +3,8 @@#version 450// The experts' activation on the GPU, bit for bit as the CPU does it// (src/llm/vmath.h janas_silu_mul_det, src/llm/quant.c-// janas_q8k_quantize_det): h = SiLU(gate) * up with the deterministic e^x+// janas_q8k_quantize_det): h = SiLU(gate) * up (gelu: GELU's tanh form,+// janas_gelu_mul_det, Gemma 4) with the deterministic e^x// and reciprocal, then Q8_K quantization of h, written in the layout the// product shaders read (320 bytes a block: 256 quants, 16 pair sums, d).// KIND 1, for a down matrix in blocks of 32 (MXFP4): the layout of the@@ -24,7 +25,7 @@ layout(constant_id = 2) const uint KIND = 0;layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer F { float v[]; };layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer U { uint v[]; };-layout(push_constant) uniform P { uvec2 g, u, x; uint blocks; } p;+layout(push_constant) uniform P { uvec2 g, u, x; uint blocks, gelu; } p;/* The workgroup's maximum: each subgroup's in one operation, then thesubgroups' - two barriers, not one per level of a tree. */@@ -83,7 +84,11 @@ void main()F G = F(p.g), Uu = F(p.u);U X = U(p.x);float gv = G.v[blk * 256u + t], uv = Uu.v[blk * 256u + t];- precise float e = exp_det(clamp(-gv, -88.0, 30.0));+ /* GELU(g) = g / (1 + e^-2z), z = sqrt(2/pi) (g + 0.044715 g^3) */+ precise float z2 = gv;+ if (p.gelu != 0u)+ z2 = 1.5957691216057308 * fma(0.044715 * gv, gv * gv, gv);+ precise float e = exp_det(clamp(-z2, -88.0, 30.0));precise float s = recip_det(1.0 + e);precise float gs = gv * s;precise float h = gs * uv;diff --git a/src/llm/shaders/moe_topk.comp b/src/llm/shaders/moe_topk.compindex 2472549..bf2a4ef 100644--- a/src/llm/shaders/moe_topk.comp+++ b/src/llm/shaders/moe_topk.comp@@ -4,7 +4,7 @@// A mixture's routing on the GPU for a prompt's block (gpu_token.c), as// model_ffn.c route_tok: per token, the softmax of its ne router logits,// the k largest (the first of equals), each weight divided by the sum of-// the k; logit ne, the shared expert's gate (if sh), through a sigmoid, 1+// the k (with es set, times its expert's scale: Gemma 4); logit ne, the shared expert's gate (if sh), through a sigmoid, 1// without one. A workgroup a token, its logits in shared memory, each// search a reduction over the 64 invocations (one invocation a token read// some 1300 logits one after the other: 0.5 ms a layer on a single token@@ -21,7 +21,7 @@ layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer UO// r: the logits, rs floats a token; pe, pw: expert and weight of pair// j k + i; shw: the shared expert's weight of each token-layout(push_constant) uniform P { uvec2 r, pe, pw, shw; uint ne, rs, k, n, sh; } p;+layout(push_constant) uniform P { uvec2 r, pe, pw, shw, es; uint ne, rs, k, n, sh; } p;const uint MAXK = 16u, MAXE = 512u; // k <= 16, ne <= 512const uint NONE = 0xffffffffu;@@ -112,7 +112,10 @@ void main()FO PW = FO(p.pw);for (uint i = 0u; i < p.k; i++) {PE.v[j * p.k + i] = ids[i];- PW.v[j * p.k + i] = w[i] / wsum;+ float wi = w[i] / wsum;+ if (p.es.x != 0u || p.es.y != 0u)+ wi *= F(p.es).v[ids[i]];+ PW.v[j * p.k + i] = wi;}FO SHW = FO(p.shw);SHW.v[j] = p.sh != 0u ? 1.0 / (1.0 + exp(-R.v[base + p.ne])) : 1.0;diff --git a/src/llm/shaders/norm_quant.comp b/src/llm/shaders/norm_quant.compindex 98b72a8..e418e94 100644--- a/src/llm/shaders/norm_quant.comp+++ b/src/llm/shaders/norm_quant.comp@@ -2,11 +2,18 @@Copyright (C) 2026 Maurizio Cammalleri */#version 450// A vector of d floats on its way into the products: x + add (the-// residual, if add is set) stored at xo (if set), then v = rms_norm(x) * w+// residual, if add is set; times os if os is not 0: Gemma 4's output+// scale) stored at xo (if set), then v = rms_norm(x) * w// (if w is set; else v = x), quantized to Q8_K in the product shaders'// layouts - q256 (qs, bsums, d: janas_q8k_quantize_det) and q32 (the eight// blocks of 32 with a scale each, if q32 is set), and v itself at xn (if-// set: a recurrent layer's f32 products, an MTP block's hidden states). One workgroup per block+// set: a recurrent layer's f32 products, an MTP block's hidden states). With+// det 0 (the norms) the same bits as the CPU's rms_norm and+// janas_q8k_quantize: the sum of squares in two floats (some 48 bits, as+// the CPU's double), its mean rounded once, the square root, the+// reciprocal and amax / 127 correctly rounded (a product, then the+// remainder by fma); det 1, the deterministic quantization+// (janas_q8k_quantize_det: an activation). One workgroup per block// of 256 values: each one makes the whole vector's sum of squares, the// same operations in the same order, so the same bits, then normalizes and// quantizes its own block, a value a thread. x is only read: the residual@@ -26,7 +33,7 @@ layout(local_size_x = 256) in;layout(buffer_reference, std430, buffer_reference_align = 4) buffer F { float v[]; };layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer U { uint v[]; };-layout(push_constant) uniform P { uvec2 x, add, w, xo, q256, q32, xn; uint d; float eps; } p;+layout(push_constant) uniform P { uvec2 x, add, w, xo, q256, q32, xn; uint d; float eps, os, ps; uint det; } p;/* The workgroup's sum: each subgroup's in one operation, then thesubgroups' in a fixed order. */@@ -58,6 +65,54 @@ float recip_det(float d)return y;}+/* a / b correctly rounded (b normal, positive): the product with the+ reciprocal, then the remainder's share */+float div_cr(float a, float b)+{+ precise float r = recip_det(b);+ precise float q = a * r;+ precise float e = fma(-q, b, a);+ return fma(e, r, q);+}++/* sqrt(a) correctly rounded (a normal, positive) */+float sqrt_cr(float a)+{+ precise float y = inversesqrt(a);+ precise float s = a * y;+ precise float h = 0.5 * y;+ precise float r = fma(-s, s, a);+ return fma(r, h, s);+}++/* the CPU's scale of janas_q8k_quantize (amax / 127) and its reciprocal,+ or the deterministic ones */+float q_scale(float m)+{+ if (p.det != 0u)+ return m * 0.007874015718698502;+ return div_cr(m, 127.0);+}++float q_inv(float d)+{+ if (d <= 0.0)+ return 0.0;+ return p.det != 0u ? recip_det(d) : div_cr(1.0, d);+}++/* hi + lo as two floats, exactly */+void two_sum(inout float hi, inout float lo, float b)+{+ precise float s = hi + b;+ precise float bb = s - hi;+ precise float e = (hi - (s - bb)) + (b - bb);+ hi = s;+ lo = lo + e;+}++shared float ss_hi[256], ss_lo[256];+int q8(float v, float id){precise float q = roundEven(v * id);@@ -73,12 +128,52 @@ void main()F X = F(p.x), A = F(p.add), Wt = F(p.w), XO = F(p.xo);U Q = U(p.q256), Q32 = U(p.q32);float scale = 1.0;- if (set(p.w)) {+ if (set(p.w) && p.det == 0u) {+ /* the squares exactly (product and fma), summed in two floats,+ each thread's then the threads' in order */+ float hi = 0.0, lo = 0.0;+ for (uint i = t; i < d; i += 256u) {+ precise float xv = X.v[xo0 + i];+ if (set(p.add))+ xv = xv + A.v[xo0 + i];+ if (p.os != 0.0)+ xv = xv * p.os;+ precise float sq = xv * xv;+ precise float se = fma(xv, xv, -sq);+ two_sum(hi, lo, sq);+ lo = lo + se;+ }+ ss_hi[t] = hi;+ ss_lo[t] = lo;+ barrier();+ /* the threads' two floats added in a tree */+ for (uint s = 128u; s > 0u; s >>= 1) {+ if (t < s) {+ float h = ss_hi[t], l = ss_lo[t];+ two_sum(h, l, ss_hi[t + s]);+ ss_hi[t] = h;+ ss_lo[t] = l + ss_lo[t + s];+ }+ barrier();+ }+ hi = ss_hi[0];+ lo = ss_lo[0];+ barrier();+ /* the mean rounded once, as (float)(ss / n) */+ precise float fd = float(d);+ precise float q0 = div_cr(hi, fd);+ precise float rem = fma(-q0, fd, hi) + lo;+ precise float mean = q0 + rem / fd;+ precise float sc = div_cr(1.0, sqrt_cr(mean + p.eps));+ scale = sc;+ } else if (set(p.w)) {float ss = 0.0;for (uint i = t; i < d; i += 256u) {precise float xv = X.v[xo0 + i];if (set(p.add))xv = xv + A.v[xo0 + i];+ if (p.os != 0.0)+ xv = xv * p.os;ss += xv * xv;}ss = wg_sum(ss);@@ -90,11 +185,15 @@ void main()precise float xv = X.v[xo0 + e];if (set(p.add))xv = xv + A.v[xo0 + e];+ if (p.os != 0.0)+ xv = xv * p.os;if (set(p.xo))XO.v[xo0 + e] = xv;float v = xv;if (set(p.w)) {precise float vs = xv * scale;+ if (p.ps != 0.0) /* Gemma 4's router: times 1 / sqrt(d) first */+ vs = vs * p.ps;v = vs * Wt.v[e];}if (set(p.xn))@@ -109,10 +208,10 @@ void main()float m256 = 0.0;for (uint k = 0; k < 32u; k++)m256 = max(m256, m8[k]);- precise float dd = m256 * 0.007874015718698502;- float id = dd > 0.0 ? recip_det(dd) : 0.0;- precise float d32 = m32 * 0.007874015718698502;- float id32 = d32 > 0.0 ? recip_det(d32) : 0.0;+ precise float dd = q_scale(m256);+ float id = q_inv(dd);+ precise float d32 = q_scale(m32);+ float id32 = q_inv(d32);int qa = q8(v, id), qb = q8(v, id32);/* four quants a word, from four neighbouring threads */uint wa = subgroupClusteredOr((uint(qa) & 0xffu) << (8u * (t % 4u)), 4u);@@ -137,8 +236,7 @@ void main()int sb0 = s8b[4u * t] + s8b[4u * t + 1u], sb1 = s8b[4u * t + 2u] + s8b[4u * t + 3u];Q32.v[base + 64u + t] = (uint(sb0) & 0xffffu) | (uint(sb1) << 16);precise float dt =- max(max(m8[4u * t], m8[4u * t + 1u]), max(m8[4u * t + 2u], m8[4u * t + 3u])) *- 0.007874015718698502;+ q_scale(max(max(m8[4u * t], m8[4u * t + 1u]), max(m8[4u * t + 2u], m8[4u * t + 3u])));Q32.v[base + 72u + t] = floatBitsToUint(dt);}}diff --git a/src/llm/shaders/ple_act.comp b/src/llm/shaders/ple_act.compnew file mode 100644index 0000000..614ae0c--- /dev/null+++ b/src/llm/shaders/ple_act.comp@@ -0,0 +1,62 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+#version 450+// Gemma 4's per-layer embeddings (E2B, E4B): a layer's branch after its+// gate, h = GELU(gate) times the layer's own inputs, bit for bit as the+// CPU's janas_gelu_mul_det (src/llm/vmath.h; model_ple.c+// janas_m_ple_layer): the deterministic e^x and reciprocal. A value an+// invocation: token j's gate at g[j P + i], its inputs at in[j W + i] (in+// already offset to the layer's), h at h[j P + i].+#extension GL_EXT_buffer_reference : require+#extension GL_EXT_buffer_reference_uvec2 : require++layout(local_size_x = 256) in;++layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer F { float v[]; };+layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer FO { float v[]; };++layout(push_constant) uniform P { uvec2 g, in_, h; uint n, pd, w; } p;++float exp_det(float x)+{+ precise float xc = clamp(x, -88.0, 88.0);+ precise float n = roundEven(xc * 1.44269504088896341);+ precise float r = fma(-n, 0.693359375, xc);+ r = fma(-n, -2.12194440e-4, r);+ precise float y = 1.9875691500e-4;+ y = fma(y, r, 1.3981999507e-3);+ y = fma(y, r, 8.3334519073e-3);+ y = fma(y, r, 4.1665795894e-2);+ y = fma(y, r, 1.6666665459e-1);+ y = fma(y, r, 5.0000001201e-1);+ precise float rr = r * r;+ y = fma(y, rr, r);+ y = y + 1.0;+ return y * uintBitsToFloat(uint(int(n) + 127) << 23);+}++float recip_det(float d)+{+ precise float y = uintBitsToFloat(0x7EF311C3u - floatBitsToUint(d));+ for (int i = 0; i < 3; i++) {+ precise float e = fma(-d, y, 1.0);+ y = fma(y, e, y);+ }+ return y;+}++void main()+{+ uint k = gl_GlobalInvocationID.x;+ if (k >= p.n * p.pd)+ return;+ uint j = k / p.pd, i = k % p.pd;+ float gv = F(p.g).v[k], uv = F(p.in_).v[j * p.w + i];+ /* GELU(g) = g / (1 + e^-2z), z = sqrt(2/pi) (g + 0.044715 g^3) */+ precise float z2 = 1.5957691216057308 * fma(0.044715 * gv, gv * gv, gv);+ precise float e = exp_det(clamp(-z2, -88.0, 30.0));+ precise float s = recip_det(1.0 + e);+ precise float gs = gv * s;+ precise float hv = gs * uv;+ FO(p.h).v[k] = hv;+}diff --git a/src/llm/shaders/q4k_mmq.comp b/src/llm/shaders/q4k_mmq.compindex 0fe8ade..8eb1923 100644--- a/src/llm/shaders/q4k_mmq.comp+++ b/src/llm/shaders/q4k_mmq.comp@@ -9,6 +9,10 @@// minimum applied in float. NOT bit for bit with the CPU: the sums are// rounded group by group, not exactly per block; on the development// laptop's GPU 1.5 times the exact product's speed on a prompt's block.+// EXACT (constant 2): a whole block of 8 groups staged per barrier, its+// integer sums (scales times the groups' dots, minimums times the+// activations' sums) then two float accumulators, as q4k_matvec.comp and+// the CPU (src/llm/quant_dotn.h): bit for bit with it.#extension GL_EXT_control_flow_attributes : require#extension GL_EXT_integer_dot_product : require#extension GL_EXT_buffer_reference : require@@ -17,8 +21,9 @@layout(local_size_x = 128) in;layout(constant_id = 0) const uint NV = 8; // unused: the table's constantslayout(constant_id = 1) const uint SG = 8;+layout(constant_id = 2) const uint EXACT = 0;-const uint BM = 64u, BN = 64u, BK = 32u, KS = 4u;+const uint BM = 64u, BN = 64u, BK = 32u, KS = EXACT != 0u ? 8u : 4u;const uint WM = 32u, WN = 32u, TM = 2u, TN = 2u, WARP = 32u;const uint WNITER = (WM * WN) / (WARP * TM * TN); // 8const uint WSUBN = WN / WNITER; // 4@@ -32,8 +37,10 @@ layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer F// y: row r of vector v at y[v * ys + r]layout(push_constant) uniform P { uvec2 w, x, y; uint rows, nb, nv, xs, ys; } p;-struct ACache { uint qs[4]; vec2 dm; }; // a group: nibbles 2 by 2, d*sc, dmin*m-struct BCache { int qs[8]; vec2 ds; }; // a group: quants, d8, d8 * their sum+/* a group: nibbles 2 by 2, d*sc and dmin*m (EXACT: d and dmin, sc, m) */+struct ACache { uint qs[4]; vec2 dm; int sc, mn; };+/* a group: quants, d8 and d8 * their sum (EXACT: their sum alone) */+struct BCache { int qs[8]; vec2 ds; int bs; };shared ACache sa[BM * KS];shared BCache sb[BN * KS];@@ -49,6 +56,8 @@ void main()uint warp_r = warp % (BM / WM), warp_c = warp / (BM / WM);float sums[WNITER * TN * TM];[[unroll]] for (uint i = 0; i < WNITER * TN * TM; i++) sums[i] = 0.0;+ precise float sums2[EXACT != 0u ? 2u * WNITER * TN * TM : 1u];+ [[unroll]] for (uint i = 0; i < sums2.length(); i++) sums2[i] = 0.0;uint ngroups = p.nb * 8u;for (uint g0 = 0; g0 < ngroups; g0 += KS) {// weights: 4 invocations a row, each one word of the row's group@@ -78,7 +87,10 @@ void main()int mn = jh < 4u ? int(bitfieldExtract(hd.z, int(8u * jj), 6)): int(bitfieldExtract(hd.w, int(8u * jj + 4u), 4) |(bitfieldExtract(hd.z, int(8u * jj + 6u), 2) << 4));- sa[(k + h) * BM + rl + l].dm = vec2(dd.x * float(sc), dd.y * float(mn));+ sa[(k + h) * BM + rl + l].dm =+ EXACT != 0u ? dd : vec2(dd.x * float(sc), dd.y * float(mn));+ sa[(k + h) * BM + rl + l].sc = sc;+ sa[(k + h) * BM + rl + l].mn = mn;}}}@@ -102,10 +114,41 @@ void main()uint bw = j % 4u == 0u ? bs.x : j % 4u == 1u ? bs.y : j % 4u == 2u ? bs.z : bs.w;int s = bitfieldExtract(int(bw), 0, 16) + bitfieldExtract(int(bw), 16, 16);sb[k * BN + c].ds = vec2(d8, d8 * float(s));+ sb[k * BN + c].bs = s;}}}barrier();+ if (EXACT != 0u) {+ /* the block's integer sums, then its share of the two+ accumulators: sums[2 i] + sums[2 i + 1] the minimums' */+ [[unroll]] for (uint wsic = 0; wsic < WNITER; wsic++)+ [[unroll]] for (uint cc = 0; cc < TN; cc++) {+ uint col = warp_c * WN + wsic * WSUBN + tiwc * TN + cc;+ [[unroll]] for (uint cr = 0; cr < TM; cr++) {+ uint rl = warp_r * WM + tiwr * TM + cr;+ int isum = 0, imin = 0;+ for (uint k = 0; k < 8u; k++) {+ ACache ca = sa[k * BM + rl];+ BCache cb = sb[k * BN + col];+ int q = 0;+ [[unroll]] for (uint i = 0; i < 8u; i++)+ q += dotPacked4x8EXT(int((ca.qs[i / 2u] >> ((i % 2u) * 4u)) & 0x0f0f0f0fu),+ cb.qs[i]);+ isum += q * ca.sc;+ imin += ca.mn * cb.bs;+ }+ vec2 dd = sa[rl].dm;+ float d8 = sb[col].ds.x;+ uint o = 2u * ((wsic * TN + cc) * TM + cr);+ precise float ds = dd.x * d8, dm = dd.y * d8;+ sums2[o] = fma(ds, float(isum), sums2[o]);+ sums2[o + 1u] = fma(dm, float(imin), sums2[o + 1u]);+ }+ }+ barrier();+ continue;+ }for (uint k = 0; k < KS; k++) {ACache ca[TM];[[unroll]] for (uint cr = 0; cr < TM; cr++)@@ -130,7 +173,9 @@ void main()[[unroll]] for (uint cr = 0; cr < TM; cr++) {uint row = r0 + warp_r * WM + tiwr * TM + cr;uint v = v0 + warp_c * WN + wsic * WSUBN + tiwc * TN + cc;+ uint o = (wsic * TN + cc) * TM + cr;if (row < p.rows && v < p.nv)- Y.v[v * p.ys + row] = sums[(wsic * TN + cc) * TM + cr];+ Y.v[v * p.ys + row] =+ EXACT != 0u ? sums2[2u * o] - sums2[2u * o + 1u] : sums[o];}}diff --git a/src/llm/shaders/q6k_mmq.comp b/src/llm/shaders/q6k_mmq.compindex eaf2d63..f305a33 100644--- a/src/llm/shaders/q6k_mmq.comp+++ b/src/llm/shaders/q6k_mmq.comp@@ -6,7 +6,9 @@// four of ql and two of qh, put together as q6k_matvec.comp does, from the// 210-byte blocks read a word at a time and realigned; the -32 of Q6_K// comes off with the activations' sums of 16. NOT bit for bit with the-// CPU: rounded group by group.+// CPU: rounded group by group. EXACT (constant 2): a whole block staged+// per barrier (its two halves), its integer sum then one float+// accumulator, as q6k_matvec.comp and the CPU: bit for bit with it.#extension GL_EXT_control_flow_attributes : require#extension GL_EXT_integer_dot_product : require#extension GL_EXT_buffer_reference : require@@ -15,8 +17,9 @@layout(local_size_x = 128) in;layout(constant_id = 0) const uint NV = 8; // unused: the table's constantslayout(constant_id = 1) const uint SG = 8;+layout(constant_id = 2) const uint EXACT = 0;-const uint BM = 64u, BN = 64u, KS = 4u;+const uint BM = 64u, BN = 64u, KS = EXACT != 0u ? 8u : 4u;const uint WM = 32u, WN = 32u, TM = 2u, TN = 2u, WARP = 32u;const uint WNITER = (WM * WN) / (WARP * TM * TN); // 8const uint WSUBN = WN / WNITER; // 4@@ -28,8 +31,10 @@ layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer Flayout(push_constant) uniform P { uvec2 w, x, y; uint rows, nb, nv, xs, ys; } p;-struct ACache { uint qs[8]; vec2 dsc; }; // a group: six-bit values, d * scale of each 16-struct BCache { int qs[8]; vec2 d8s; float d8; }; // quants, 32 x the sums of 16, d8+/* a group: six-bit values, d * scale of each 16 (EXACT: d, the scales) */+struct ACache { uint qs[8]; vec2 dsc; int s0, s1; };+/* quants, 32 x the sums of 16, d8 (EXACT: the sums) */+struct BCache { int qs[8]; vec2 d8s; float d8; int b0, b1; };shared ACache sa[BM * KS];shared BCache sb[BN * KS];@@ -52,6 +57,8 @@ void main()uint warp_r = warp % (BM / WM), warp_c = warp / (BM / WM);float sums[WNITER * TN * TM];[[unroll]] for (uint i = 0; i < WNITER * TN * TM; i++) sums[i] = 0.0;+ precise float sumsx[EXACT != 0u ? WNITER * TN * TM : 1u];+ [[unroll]] for (uint i = 0; i < sumsx.length(); i++) sumsx[i] = 0.0;uint ngroups = p.nb * 8u;for (uint g0 = 0; g0 < ngroups; g0 += KS) {// weights: 4 invocations a row, 2 words each of the group's 8@@ -62,28 +69,36 @@ void main()// the 4 groups staged share their high bits' word and two// words of low nibbles: each read once (they were read once// per group, realigned: twice)- uint b = g0 / 8u, n = (g0 % 8u) / 4u;+ uint b = g0 / 8u;uint off = (r * p.nb + b) * 210u;+ /* the stage's halves of the block: one, EXACT two */+ [[unroll]] for (uint hh = 0; hh < KS / 4u; hh++) {+ uint n = (g0 % 8u) / 4u + hh;[[unroll]] for (uint t = 0; t < 2u; t++) {uint kw = 2u * iqs + t;uint l0 = hword(W, off, 16u * n + kw), l1 = hword(W, off, 16u * n + 8u + kw);uint h = hword(W, off, 32u + 8u * n + kw);- [[unroll]] for (uint k = 0; k < KS; k++) {+ [[unroll]] for (uint k = 0; k < 4u; k++) {uint lw = (k & 1u) != 0u ? l1 : l0;- sa[k * BM + rl + l].qs[kw] = ((lw >> ((k >> 1) * 4u)) & 0x0f0f0f0fu) |- (((h >> (2u * k)) & 0x03030303u) << 4);+ sa[(4u * hh + k) * BM + rl + l].qs[kw] =+ ((lw >> ((k >> 1) * 4u)) & 0x0f0f0f0fu) |+ (((h >> (2u * k)) & 0x03030303u) << 4);}}if (iqs == 0u) {float d = unpackHalf2x16(hword(W, off, 52u) & 0xffffu).x;uint sw0 = hword(W, off, 48u + 2u * n), sw1 = hword(W, off, 49u + 2u * n);- [[unroll]] for (uint k = 0; k < KS; k++) {+ [[unroll]] for (uint k = 0; k < 4u; k++) {uint sw = k < 2u ? sw0 : sw1, sb0 = (2u * k) % 4u;int s0 = bitfieldExtract(int(sw), int(8u * sb0), 8);int s1 = bitfieldExtract(int(sw), int(8u * sb0 + 8u), 8);- sa[k * BM + rl + l].dsc = vec2(d * float(s0), d * float(s1));+ uint kk = (4u * hh + k) * BM + rl + l;+ sa[kk].dsc = EXACT != 0u ? vec2(d, 0.0) : vec2(d * float(s0), d * float(s1));+ sa[kk].s0 = s0;+ sa[kk].s1 = s1;}}+ }}}// activations: 2 invocations a vector, each half a group's quants@@ -104,10 +119,40 @@ void main()sb[k * BN + c].d8s = vec2(32.0 * float(bitfieldExtract(int(bw), 0, 16)),32.0 * float(bitfieldExtract(int(bw), 16, 16)));sb[k * BN + c].d8 = uintBitsToFloat(X.v[xo + 18u].x);+ sb[k * BN + c].b0 = bitfieldExtract(int(bw), 0, 16);+ sb[k * BN + c].b1 = bitfieldExtract(int(bw), 16, 16);}}}barrier();+ if (EXACT != 0u) {+ /* the block's integer sum, then its share of the accumulator */+ [[unroll]] for (uint wsic = 0; wsic < WNITER; wsic++)+ [[unroll]] for (uint cc = 0; cc < TN; cc++) {+ uint col = warp_c * WN + wsic * WSUBN + tiwc * TN + cc;+ [[unroll]] for (uint cr = 0; cr < TM; cr++) {+ uint rl = warp_r * WM + tiwr * TM + cr;+ int isum = 0, off32 = 0;+ for (uint k = 0; k < 8u; k++) {+ ACache ca = sa[k * BM + rl];+ BCache cb = sb[k * BN + col];+ int q0 = 0, q1 = 0;+ [[unroll]] for (uint i = 0; i < 4u; i++) {+ q0 += dotPacked4x8EXT(int(ca.qs[i]), cb.qs[i]);+ q1 += dotPacked4x8EXT(int(ca.qs[i + 4u]), cb.qs[i + 4u]);+ }+ isum += ca.s0 * q0 + ca.s1 * q1;+ off32 += ca.s0 * cb.b0 + ca.s1 * cb.b1;+ }+ isum -= 32 * off32;+ precise float dd = sa[rl].dsc.x * sb[col].d8;+ uint o = (wsic * TN + cc) * TM + cr;+ sumsx[o] = fma(dd, float(isum), sumsx[o]);+ }+ }+ barrier();+ continue;+ }for (uint k = 0; k < KS; k++) {ACache ca[TM];[[unroll]] for (uint cr = 0; cr < TM; cr++)@@ -134,7 +179,8 @@ void main()[[unroll]] for (uint cr = 0; cr < TM; cr++) {uint row = r0 + warp_r * WM + tiwr * TM + cr;uint v = v0 + warp_c * WN + wsic * WSUBN + tiwc * TN + cc;+ uint o = (wsic * TN + cc) * TM + cr;if (row < p.rows && v < p.nv)- Y.v[v * p.ys + row] = sums[(wsic * TN + cc) * TM + cr];+ Y.v[v * p.ys + row] = EXACT != 0u ? sumsx[o] : sums[o];}}diff --git a/src/llm/shaders/q8_0_matvec.comp b/src/llm/shaders/q8_0_matvec.compindex 3638aa4..2c4b659 100644--- a/src/llm/shaders/q8_0_matvec.comp+++ b/src/llm/shaders/q8_0_matvec.comp@@ -15,6 +15,9 @@// inside a group of 256 (tail, an even number of blocks of 32 past the whole// groups, so that rows start on words): as janas_dot_rows_n, the whole// groups' sum, then the tail's from zero, padded with zero blocks, added.+// With ID (bit 8 of constant 2) a mixture's experts (Gemma 4 26B's last+// down matrices), the table of *_mmid.comp in w as b32_matvec.comp reads+// it: x's vectors nb blocks of 256 apart, one more with a tail.#extension GL_EXT_integer_dot_product : require#extension GL_KHR_shader_subgroup_basic : require#extension GL_KHR_shader_subgroup_ballot : require@@ -24,6 +27,8 @@layout(local_size_x = 64) in;layout(constant_id = 0) const uint NV = 8;layout(constant_id = 1) const uint SG = 8;+layout(constant_id = 2) const uint KID = 0;+const bool ID = (KID & 256u) != 0u;const uint NX = (20u + SG - 1u) / SG;layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer U1 { uint v[]; };@@ -43,19 +48,27 @@ uvec4 bc(uvec4 xr[NX], uint k) { return subgroupBroadcast(xr[k / SG], k % SG); }void main(){- U1 W = U1(p.w);- U4 X = U4(p.x);- F Y = F(p.y);+ uvec2 pw = p.w, px = p.x, py = p.y;+ uint nv = p.nv, xs = p.xs, ys = p.ys, v0 = gl_WorkGroupID.x * NV;+ if (ID) {+ U4 Tb = U4(p.w);+ uvec4 tl = Tb.v[p.nv + gl_WorkGroupID.z];+ uvec4 t0 = Tb.v[3u * tl.x], t1 = Tb.v[3u * tl.x + 1u];+ pw = t0.xy, px = t0.zw, py = t1.xy;+ nv = t1.z, xs = p.nb + (p.tail != 0u ? 1u : 0u), ys = t1.w, v0 = tl.y;+ }+ U1 W = U1(pw);+ U4 X = U4(px);+ F Y = F(py);uint lane = gl_SubgroupInvocationID;/* workgroup y: 64 rows; x: a group of NV vectors, so that the groupsof the same rows go out together and the weights the first readsfrom memory the others find in the L3 */uint grow = gl_WorkGroupID.y * 64u + gl_LocalInvocationID.x;uint row = min(grow, p.rows - 1u);- uint v0 = gl_WorkGroupID.x * NV;uint xo[NV];for (uint v = 0; v < NV; v++)- xo[v] = min(v0 + v, p.nv - 1u) * p.xs * 20u;+ xo[v] = min(v0 + v, nv - 1u) * xs * 20u;/* precise: the CPU rounds every product and every sum apart; withoutit a compiler may fuse a sum with a product before it (the tail'saccumulators start from zero in the loop, a known value: AMD's,@@ -120,11 +133,11 @@ void main()}if (grow < p.rows)for (uint v = 0; v < NV; v++)- if (v0 + v < p.nv) {+ if (v0 + v < nv) {precise float r = ((acc[v][0] + acc[v][4]) + (acc[v][2] + acc[v][6])) +((acc[v][1] + acc[v][5]) + (acc[v][3] + acc[v][7]));if (p.tail != 0u)r = res[v] + r;- Y.v[(v0 + v) * p.ys + row] = r;+ Y.v[(v0 + v) * ys + row] = r;}}diff --git a/src/llm/shaders/qk_rope.comp b/src/llm/shaders/qk_rope.compindex 7b2a295..8f79508 100644--- a/src/llm/shaders/qk_rope.comp+++ b/src/llm/shaders/qk_rope.comp@@ -2,7 +2,8 @@Copyright (C) 2026 Maurizio Cammalleri */#version 450// After the q, k, v products of one token (qkv: q, then k at qd, then v at-// qd + kvd floats): each head of q normalized (q_norm) and turned (NeoX+// qd + kvd floats): each head of q normalized (q_norm, unless flags' bit+// 0 says the model has none: llama; k's bit 1) and turned (NeoX// RoPE over its first n_rot dimensions, pairs i and i + n_rot/2, the// token's tables cs and sn), in place; each head of k normalized (k_norm)// and turned, then k and v put in the layer's KV cache at eight bits with@@ -10,8 +11,13 @@// of the head's positions, as model_attn.c's qkv_item does. Each head of q// is left at eight bits too, for attn_decode.comp: its hd / 4 words of four// bytes, then its scale (the largest magnitude over 127), at the start of-// its own hd floats. One workgroup per head: the n_head of q, then the-// n_kv of k (with v).+// its own hd floats; with flags' bit 2 at sixteen bits, as two halves of+// eight (q = 256 hi + lo, each -128 .. 127): hd / 4 words of hi, hd / 4 of+// lo, then the scale. Gemma 4 (flags' bits 3 to 6): keys and values at+// sixteen bits (the largest becomes 32767, two numbers a word) with q left+// in float, in place; the cache a ring (row (pos + j) mod cap); V read+// from K's place, before K's norm; V normalized without weights. One+// workgroup per head: the n_head of q, then the n_kv of k (with v).#extension GL_EXT_buffer_reference : require#extension GL_KHR_shader_subgroup_basic : require#extension GL_KHR_shader_subgroup_arithmetic : require@@ -30,6 +36,10 @@ layout(push_constant) uniform P {uvec2 qkv, qn, kn, rope, kc, vc, ks, vs;uint n_head, n_kv, hd, n_rot, cap, qs; /* qs: floats from a token's qkv to the next's */float eps;+ uint flags; /* bit 0: q's heads not normalized, bit 1: k's; bit 2: q+ at sixteen bits; bit 3: keys and values at sixteen+ bits, q in float; bit 4: a ring; bit 5: V is K; bit 6:+ V normalized */uvec2 pos; /* the token's position, read at run time: row pos */} p;@@ -68,6 +78,81 @@ float wg_max(float s)}float head_sum(float s) { return wg_sum(s); }++/* As the CPU's rms_norm and its caches' scales, bit for bit: a / b and+ sqrt(a) correctly rounded (a product, then the remainder by fma), the+ squares summed in two floats (some 48 bits, as its double) */+float recip_det(float d)+{+ precise float y = uintBitsToFloat(0x7EF311C3u - floatBitsToUint(d));+ for (int i = 0; i < 3; i++)+ y = fma(y, fma(-d, y, 1.0), y);+ return y;+}++float div_cr(float a, float b)+{+ precise float r = recip_det(b);+ precise float q = a * r;+ precise float e = fma(-q, b, a);+ return fma(e, r, q);+}++float sqrt_cr(float a)+{+ precise float y = inversesqrt(a);+ precise float s = a * y;+ precise float h = 0.5 * y;+ precise float r = fma(-s, s, a);+ return fma(r, h, s);+}++void two_sum(inout float hi, inout float lo, float b)+{+ precise float s = hi + b;+ precise float bb = s - hi;+ precise float e = (hi - (s - bb)) + (b - bb);+ hi = s;+ lo = lo + e;+}++shared float ss_hi[128], ss_lo[128];++/* 1 / sqrtf((float)(ss / n) + eps) of the head in sv (n numbers), as+ rms_norm: the whole workgroup */+float rms_scale(uint n)+{+ uint t = gl_LocalInvocationID.x;+ float hi = 0.0, lo = 0.0;+ for (uint i = t; i < n; i += 128u) {+ precise float v = sv[i];+ precise float sq = v * v;+ precise float se = fma(v, v, -sq);+ two_sum(hi, lo, sq);+ lo = lo + se;+ }+ ss_hi[t] = hi;+ ss_lo[t] = lo;+ barrier();+ /* the threads' two floats added in a tree, two floats all the way */+ for (uint s = 64u; s > 0u; s >>= 1) {+ if (t < s) {+ float h = ss_hi[t], l = ss_lo[t];+ two_sum(h, l, ss_hi[t + s]);+ ss_hi[t] = h;+ ss_lo[t] = l + ss_lo[t + s];+ }+ barrier();+ }+ hi = ss_hi[0];+ lo = ss_lo[0];+ barrier();+ precise float fn = float(n);+ precise float q0 = div_cr(hi, fn);+ precise float rem = fma(-q0, fn, hi) + lo;+ precise float mean = q0 + rem / fn;+ return div_cr(1.0, sqrt_cr(mean + p.eps));+}float head_max(float s) { return wg_max(s); }void main()@@ -81,17 +166,16 @@ void main()sin, n_rot / 2 each, n_rot floats a token), its row pos + j */uint j = gl_WorkGroupID.y, qo = j * p.qs, ro = j * p.n_rot;uint o = qo + (isq ? h * hd : qd + kh * hd);- float ss = 0.0;- for (uint i = t; i < hd; i += 128u) {- float v = QKV.v[o + i];- sv[i] = v;- ss += v * v;- }- ss = head_sum(ss);- precise float scale = 1.0 / sqrt(ss / float(hd) + p.eps);- for (uint i = t; i < hd; i += 128u) {- precise float v = sv[i] * scale;- sv[i] = v * (isq ? QN.v[i] : KN.v[i]);+ for (uint i = t; i < hd; i += 128u)+ sv[i] = QKV.v[o + i];+ /* the same for the whole workgroup (one head): its barriers too */+ if ((p.flags & (isq ? 1u : 2u)) == 0u) {+ barrier(); /* sv whole */+ precise float scale = rms_scale(hd);+ for (uint i = t; i < hd; i += 128u) {+ precise float v = sv[i] * scale;+ sv[i] = v * (isq ? QN.v[i] : KN.v[i]);+ }}barrier();uint half_ = p.n_rot / 2u;@@ -107,6 +191,12 @@ void main()sw[i] = r;}barrier();+ if (isq && (p.flags & 8u) != 0u) {+ /* keys at sixteen bits: the scores over q as it is */+ for (uint i = t; i < hd; i += 128u)+ QKV.v[o + i] = sw[i];+ return;+ }if (isq) {/* the scores' integer dot products take q at eight bits: thelargest magnitude becomes 127, q / (m / 127) rounded to even */@@ -114,8 +204,28 @@ void main()for (uint i = t; i < hd; i += 128u)mq = max(mq, abs(sw[i]));mq = head_max(mq);- float d = mq / 127.0, id = d > 0.0 ? 1.0 / d : 0.0;U QW = U(p.qkv);+ if ((p.flags & 4u) != 0u) {+ /* the largest magnitude becomes 127 x 256 + 127, so that hi =+ floor((q + 128) / 256) and lo = q - 256 hi stay in a byte */+ float d = mq / 32639.0, id = d > 0.0 ? 1.0 / d : 0.0;+ uint nw = hd / 4u;+ for (uint w = t; w < nw; w += 128u) {+ uint wh = 0u, wl = 0u;+ for (uint e = 0; e < 4u; e++) {+ int q = int(clamp(roundEven(sw[4u * w + e] * id), -32639.0, 32639.0));+ int hi = (q + 128) >> 8, lo = q - 256 * hi;+ wh |= (uint(hi) & 0xffu) << (8u * e);+ wl |= (uint(lo) & 0xffu) << (8u * e);+ }+ QW.v[o + w] = wh;+ QW.v[o + nw + w] = wl;+ }+ if (t == 0u)+ QKV.v[o + 2u * nw] = d;+ return;+ }+ float d = mq / 127.0, id = d > 0.0 ? 1.0 / d : 0.0;for (uint w = t; w < hd / 4u; w += 128u) {uint wq = 0u;for (uint e = 0; e < 4u; e++) {@@ -128,25 +238,54 @@ void main()QKV.v[o + hd / 4u] = d;return;}- /* k and v of this KV head at eight bits, each with its scale */- uint ov = qo + qd + kvd + kh * hd;+ /* k and v of this KV head at eight bits (or sixteen), each with its+ scale; V from K's place where the layer has no V of its own */+ uint ov = qo + qd + ((p.flags & 32u) != 0u ? 0u : kvd) + kh * hd;+ for (uint i = t; i < hd; i += 128u)+ sv[i] = QKV.v[ov + i];+ if ((p.flags & 64u) != 0u) { /* V normalized, no weights */+ barrier(); /* sv whole */+ precise float scale = rms_scale(hd);+ for (uint i = t; i < hd; i += 128u) {+ precise float v = sv[i] * scale;+ sv[i] = v;+ }+ }float mk = 0.0, mv = 0.0;for (uint i = t; i < hd; i += 128u) {- sv[i] = QKV.v[ov + i];mk = max(mk, abs(sw[i]));mv = max(mv, abs(sv[i]));}mk = head_max(mk);mv = head_max(mv);- float ik = mk > 0.0 ? 127.0 / mk : 0.0, iv = mv > 0.0 ? 127.0 / mv : 0.0;+ uint row = R0(p.pos).v[0] + j;+ if ((p.flags & 16u) != 0u)+ row %= p.cap;+ U KC = U(p.kc), VC = U(p.vc);+ if ((p.flags & 8u) != 0u) {+ /* sixteen bits: the largest becomes 32767, two numbers a word */+ float ik = mk > 0.0 ? div_cr(32767.0, mk) : 0.0, iv = mv > 0.0 ? div_cr(32767.0, mv) : 0.0;+ uint at2 = (kh * p.cap + row) * hd / 2u; /* words */+ for (uint w = t; w < hd / 2u; w += 128u) {+ int k0 = int(roundEven(sw[2u * w] * ik)), k1 = int(roundEven(sw[2u * w + 1u] * ik));+ int v0 = int(roundEven(sv[2u * w] * iv)), v1 = int(roundEven(sv[2u * w + 1u] * iv));+ KC.v[at2 + w] = (uint(k0) & 0xffffu) | (uint(k1) << 16);+ VC.v[at2 + w] = (uint(v0) & 0xffffu) | (uint(v1) << 16);+ }+ if (t == 0u) {+ F KS = F(p.ks), VS = F(p.vs);+ KS.v[kh * p.cap + row] = div_cr(mk, 32767.0);+ VS.v[kh * p.cap + row] = div_cr(mv, 32767.0);+ }+ return;+ }+ float ik = mk > 0.0 ? div_cr(127.0, mk) : 0.0, iv = mv > 0.0 ? div_cr(127.0, mv) : 0.0;for (uint i = t; i < hd; i += 128u) {qk[i] = int(roundEven(sw[i] * ik));qv[i] = int(roundEven(sv[i] * iv));}barrier();- uint row = R0(p.pos).v[0] + j;uint at = (kh * p.cap + row) * hd; /* bytes; hd is a multiple of 4 */- U KC = U(p.kc), VC = U(p.vc);for (uint w = t; w < hd / 4u; w += 128u) {KC.v[at / 4u + w] = (uint(qk[4u * w]) & 0xffu) | ((uint(qk[4u * w + 1u]) & 0xffu) << 8) |((uint(qk[4u * w + 2u]) & 0xffu) << 16) |@@ -157,7 +296,7 @@ void main()}if (t == 0u) {F KS = F(p.ks), VS = F(p.vs);- KS.v[kh * p.cap + row] = mk / 127.0;- VS.v[kh * p.cap + row] = mv / 127.0;+ KS.v[kh * p.cap + row] = div_cr(mk, 127.0);+ VS.v[kh * p.cap + row] = div_cr(mv, 127.0);}}diff --git a/src/llm/shaders/silu_mul.comp b/src/llm/shaders/silu_mul.compindex 2c1a123..016a87d 100644--- a/src/llm/shaders/silu_mul.comp+++ b/src/llm/shaders/silu_mul.comp@@ -1,7 +1,7 @@/* SPDX-License-Identifier: GPL-3.0-or-laterCopyright (C) 2026 Maurizio Cammalleri */#version 450-// h = SiLU(gate) * up as floats, n values (gpu_token_rec.c: a dense+// h = SiLU(gate) * up as floats (gelu: GELU's tanh form), n values (gpu_token_rec.c: a dense// layer's or a shared expert's feed-forward whose down matrix is in blocks// of 32, quantized afterwards by norm_quant.comp into both layouts). The GPU's exp: NOT bit for bit with// the CPU.@@ -13,13 +13,15 @@ layout(local_size_x = 256) in;layout(buffer_reference, std430, buffer_reference_align = 4) readonly buffer F { float v[]; };layout(buffer_reference, std430, buffer_reference_align = 4) writeonly buffer FO { float v[]; };-layout(push_constant) uniform P { uvec2 g, u, h; uint n; } p;+layout(push_constant) uniform P { uvec2 g, u, h; uint n, gelu; } p;void main(){uint i = gl_GlobalInvocationID.x;if (i >= p.n)return;- float gv = F(p.g).v[i];- FO(p.h).v[i] = gv / (1.0 + exp(-gv)) * F(p.u).v[i];+ float gv = F(p.g).v[i], z2 = gv;+ if (p.gelu != 0u)+ z2 = 1.5957691216057308 * fma(0.044715 * gv, gv * gv, gv);+ FO(p.h).v[i] = gv / (1.0 + exp(-z2)) * F(p.u).v[i];}diff --git a/src/llm/tokenizer.c b/src/llm/tokenizer.cindex c9f6337..45c0368 100644--- a/src/llm/tokenizer.c+++ b/src/llm/tokenizer.c@@ -28,9 +28,10 @@ struct janas_tokenizer {uint8_t cp_byte[512]; /* byte-level code point -> byte */int32_t *special; /* ids of the special tokens, longest first */uint32_t n_special;- int marks; /* words take combining marks too (the qwen35 expression) */- int spm; /* SentencePiece style (gemma4): raw UTF-8 characters, spaces- written as U+2581, unknown characters as <0xXX> bytes */+ int marks; /* words take combining marks too (the qwen35 expression) */+ int digits; /* a number's digits taken together, up to: 1, Llama 3's 3 */+ int spm; /* SentencePiece style (gemma4): raw UTF-8 characters, spaces+ written as U+2581, unknown characters as <0xXX> bytes */};static uint64_t hash_bytes(const char *s, size_t n)@@ -209,6 +210,7 @@ struct janas_tokenizer *janas_tokenizer_create(const struct janas_jns *j,}t->n = (uint32_t)n;t->marks = strcmp(pre, "qwen35") == 0;+ t->digits = strcmp(pre, "llama-bpe") == 0 ? 3 : 1;t->spm = strcmp(model, "gemma4") == 0;uint64_t total = 0;const uint8_t *p = toks;@@ -360,9 +362,11 @@ int32_t janas_tokenizer_find(const struct janas_tokenizer *t, const char *s)* (?i:'s|'t|'re|'ve|'m|'ll|'d) | [^\r\n\p{L}\p{N}]?\p{L}+ | \p{N}* | ' '?[^\s\p{L}\p{N}]+[\r\n]* | \s*[\r\n]+ | \s+(?!\S) | \s+* With marks (Qwen3.5 and later), \p{L} in the word runs and in the- * punctuation alternative's exclusion becomes [\p{L}\p{M}].+ * punctuation alternative's exclusion becomes [\p{L}\p{M}]. With digits 3+ * (Llama 3), \p{N} becomes \p{N}{1,3}.*/-static size_t pretoken_end(const uint32_t *cp, size_t n, size_t i, int marks)+static size_t pretoken_end(const uint32_t *cp, size_t n, size_t i, int marks,+ int digits){#define WORD(c) (is_letter(c) || (marks && is_mark(c)))uint32_t c = cp[i];@@ -390,8 +394,12 @@ static size_t pretoken_end(const uint32_t *cp, size_t n, size_t i, int marks)k++;return k;}- if (is_number(c))- return i + 1;+ if (is_number(c)) {+ size_t e = i + 1;+ while (e < n && e - i < (size_t)digits && is_number(cp[e]))+ e++;+ return e;+ }k = i + (c == ' ');if (k < n && !is_space(cp[k]) && !WORD(cp[k]) && !is_number(cp[k])) {while (k < n && !is_space(cp[k]) && !WORD(cp[k]) && !is_number(cp[k]))@@ -604,7 +612,7 @@ static int encode_plain(const struct janas_tokenizer *t, const char *text,}at[n] = len;for (size_t i = 0; i < n;) {- size_t e = pretoken_end(cp, n, i, t->marks);+ size_t e = pretoken_end(cp, n, i, t->marks, t->digits);bpe(t, (const unsigned char *)text + at[i], at[e] - at[i], out, max,count);i = e;diff --git a/src/nsm/lexicon.c b/src/nsm/lexicon.cnew file mode 100644index 0000000..8a15b93--- /dev/null+++ b/src/nsm/lexicon.c@@ -0,0 +1,210 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * lexicon.c - the English words of NSM (lexicon.h).+ */+#include "lexicon.h"++#include <string.h>++#define FORMS(...) ((const char *const[]){__VA_ARGS__, NULL})++/* Goddard and Wierzbicka's chart of 2014, its allolexes (~) and the forms+ the explications inflect them into */+const struct nsm_prime nsm_primes[] = {+ {"I", FORMS("i", "me")},+ {"YOU", FORMS("you")},+ {"SOMEONE", FORMS("someone", "somebody", "person", "persons")},+ {"PEOPLE", FORMS("people")},+ {"SOMETHING", FORMS("something", "thing", "things")},+ {"BODY", FORMS("body", "bodies")},+ {"KIND", FORMS("kind", "kinds")},+ {"PART", FORMS("part", "parts")},+ {"THIS", FORMS("this", "these")},+ {"THE SAME", FORMS("the same", "same")},+ {"OTHER", FORMS("other", "others", "else", "another")},+ {"ONE", FORMS("one")},+ {"TWO", FORMS("two")},+ {"SOME", FORMS("some")},+ {"ALL", FORMS("all")},+ {"MUCH", FORMS("much", "many")},+ {"LITTLE", FORMS("little", "few")},+ {"GOOD", FORMS("good", "well")},+ {"BAD", FORMS("bad", "badly")},+ {"BIG", FORMS("big")},+ {"SMALL", FORMS("small")},+ {"THINK", FORMS("think", "thinks", "thinking", "thought")},+ {"KNOW", FORMS("know", "knows", "knowing", "knew", "known")},+ {"WANT", FORMS("want", "wants", "wanting", "wanted")},+ {"DON'T WANT", FORMS("don't want", "doesn't want", "didn't want",+ "do not want", "does not want", "did not want")},+ {"FEEL", FORMS("feel", "feels", "feeling", "felt")},+ {"SEE", FORMS("see", "sees", "seeing", "saw", "seen")},+ {"HEAR", FORMS("hear", "hears", "hearing", "heard")},+ {"SAY", FORMS("say", "says", "saying", "said")},+ {"WORDS", FORMS("words", "word")},+ {"TRUE", FORMS("true")},+ {"DO", FORMS("do", "does", "doing", "did", "done")},+ {"HAPPEN", FORMS("happen", "happens", "happening", "happened")},+ {"MOVE", FORMS("move", "moves", "moving", "moved")},+ {"BE SOMEWHERE", FORMS("be somewhere", "is somewhere", "are somewhere",+ "was somewhere", "were somewhere", "somewhere")},+ {"THERE IS", FORMS("there is", "there are", "there was", "there were",+ "there can be", "there will be")},+ {"BE SOMEONE/SOMETHING",+ FORMS("is someone", "be someone", "is something", "be something")},+ {"MINE", FORMS("mine")},+ {"LIVE", FORMS("live", "lives", "living", "lived")},+ {"DIE", FORMS("die", "dies", "dying", "died")},+ {"WHEN", FORMS("when", "time", "times")},+ {"NOW", FORMS("now")},+ {"BEFORE", FORMS("before")},+ {"AFTER", FORMS("after")},+ {"A LONG TIME", FORMS("a long time")},+ {"A SHORT TIME", FORMS("a short time")},+ {"FOR SOME TIME", FORMS("for some time")},+ {"MOMENT", FORMS("moment", "moments")},+ {"WHERE", FORMS("where", "place", "places")},+ {"HERE", FORMS("here")},+ {"ABOVE", FORMS("above")},+ {"BELOW", FORMS("below")},+ {"FAR", FORMS("far")},+ {"NEAR", FORMS("near")},+ {"SIDE", FORMS("side", "sides")},+ {"INSIDE", FORMS("inside")},+ {"TOUCH", FORMS("touch", "touches", "touching", "touched")},+ {"NOT", FORMS("not", "no", "don't", "doesn't", "didn't", "isn't", "aren't",+ "wasn't", "weren't", "can't", "cannot")},+ {"MAYBE", FORMS("maybe")},+ {"CAN", FORMS("can", "could")},+ {"BECAUSE", FORMS("because")},+ {"IF", FORMS("if")},+ {"VERY", FORMS("very")},+ {"MORE", FORMS("more")},+ {"LIKE", FORMS("like", "as", "way", "ways")},+};+const size_t nsm_n_primes = sizeof(nsm_primes) / sizeof(nsm_primes[0]);++_Static_assert(sizeof(nsm_primes) / sizeof(nsm_primes[0]) == 65,+ "the 65 primes");++/* DeepNSM's list as its scores were computed (lexicon.h) */+static const char *const paper_primes[] = {+ "you", "someone", "people", "something", "thing", "body", "kind",+ "part", "this", "other", "else", "another", "one", "two",+ "some", "all", "much", "many", "little", "few", "good",+ "bad", "big", "small", "think", "know", "want", "feel",+ "see", "hear", "say", "words", "true", "do", "happen",+ "move", "there", "is", "be", "mine", "live", "die",+ "when", "time", "now", "before", "after", "moment", "where",+ "place", "here", "above", "below", "far", "near", "side",+ "inside", "touch", "not", "maybe", "can", "because", "if",+ "very", "more", "like", "as", "way", NULL};++const char *const nsm_stop_words[] = {+ "i", "me", "my", "myself", "we", "our",+ "ours", "ourselves", "you", "you're", "you've", "you'll",+ "you'd", "your", "yours", "yourself", "yourselves", "he",+ "him", "his", "himself", "she", "she's", "her",+ "hers", "herself", "it", "it's", "its", "itself",+ "they", "them", "their", "theirs", "themselves", "what",+ "which", "who", "whom", "this", "that", "that'll",+ "these", "those", "am", "is", "are", "was",+ "were", "be", "been", "being", "have", "has",+ "had", "having", "do", "does", "did", "doing",+ "a", "an", "the", "and", "but", "if",+ "or", "because", "as", "until", "while", "of",+ "at", "by", "for", "with", "about", "against",+ "between", "into", "through", "during", "before", "after",+ "above", "below", "to", "from", "up", "down",+ "in", "out", "on", "off", "over", "under",+ "again", "further", "then", "once", "here", "there",+ "when", "where", "why", "how", "all", "any",+ "both", "each", "few", "more", "most", "other",+ "some", "such", "no", "nor", "not", "only",+ "own", "same", "so", "than", "too", "very",+ "s", "t", "can", "will", "just", "don",+ "don't", "should", "should've", "now", "d", "ll",+ "m", "o", "re", "ve", "y", "ain",+ "aren", "aren't", "couldn", "couldn't", "didn", "didn't",+ "doesn", "doesn't", "hadn", "hadn't", "hasn", "hasn't",+ "haven", "haven't", "isn", "isn't", "ma", "mightn",+ "mightn't", "mustn", "mustn't", "needn", "needn't", "shan",+ "shan't", "shouldn", "shouldn't", "wasn", "wasn't", "weren",+ "weren't", "won", "won't", "wouldn", "wouldn't", NULL};++static int in_list(const char *const *l, const char *w)+{+ for (; *l; l++)+ if (!strcmp(*l, w))+ return 1;+ return 0;+}++static int full_single(const char *w)+{+ for (size_t p = 0; p < nsm_n_primes; p++)+ for (const char *const *f = nsm_primes[p].forms; *f; f++)+ if (!strchr(*f, ' ') && !strcmp(*f, w))+ return 1;+ return 0;+}++int nsm_is_prime(enum nsm_lex lex, const char *w)+{+ return lex == NSM_LEX_PAPER ? in_list(paper_primes, w) : full_single(w);+}++int nsm_is_stop(enum nsm_lex lex, const char *w)+{+ return in_list(nsm_stop_words, w) && !nsm_is_prime(lex, w);+}++/* The number of words of the longest prime phrase starting at word[0], 0+ when none (single words are not phrases). */+static size_t phrase_at(char *const *word, size_t n)+{+ size_t best = 0;+ for (size_t p = 0; p < nsm_n_primes; p++)+ for (const char *const *f = nsm_primes[p].forms; *f; f++) {+ if (!strchr(*f, ' '))+ continue;+ const char *s = *f;+ size_t k = 0;+ while (k < n) {+ size_t l = strlen(word[k]);+ if (strncmp(s, word[k], l) || (s[l] != ' ' && s[l] != 0))+ break;+ k++;+ s += l;+ if (!*s)+ break;+ s++;+ }+ if (!*s && k > best)+ best = k;+ }+ return best;+}++size_t nsm_classify(enum nsm_lex lex, char *const *word, size_t n_words,+ enum nsm_kind *kind, size_t *span, size_t max,+ size_t *n_units)+{+ size_t u = 0;+ for (size_t i = 0; i < n_words && u < max;) {+ size_t k = lex == NSM_LEX_FULL ? phrase_at(word + i, n_words - i) : 0;+ if (k) {+ kind[u] = NSM_PRIME;+ } else {+ k = 1;+ kind[u] = nsm_is_prime(lex, word[i]) ? NSM_PRIME+ : nsm_is_stop(lex, word[i]) ? NSM_STOP+ : NSM_MOLECULE;+ }+ span[u++] = k;+ i += k;+ }+ *n_units = u;+ return n_words;+}diff --git a/src/nsm/lexicon.h b/src/nsm/lexicon.hnew file mode 100644index 0000000..75bf232--- /dev/null+++ b/src/nsm/lexicon.h@@ -0,0 +1,53 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * lexicon.h - the words of the Natural Semantic Metalanguage (NSM) in+ * English, as janas-nsm counts them.+ *+ * Two lists of primes:+ * - NSM_LEX_PAPER: the one DeepNSM's published scores were computed with+ * (arXiv 2505.11764; the words of utils.py's NSM_PRIMES without "i" and+ * "said": with them its records do not come back, without them they do,+ * in 1337 of 1341 explications), single words only, as its counting+ * never matches a phrase;+ * - NSM_LEX_FULL: the 65 primes of Goddard and Wierzbicka (2014) with+ * their allolexes and inflected forms, phrases included ("a long time",+ * "don't want", "there is"), matched longest first.+ * And the function words: NLTK's English stop words (179), less the primes.+ */+#ifndef JANAS_NSM_LEXICON_H+#define JANAS_NSM_LEXICON_H++#include <stddef.h>++enum nsm_lex { NSM_LEX_PAPER, NSM_LEX_FULL };++enum nsm_kind { NSM_PRIME, NSM_STOP, NSM_MOLECULE };++/* The words of a lower-cased, punctuation-free text, and how each counts:+ a prime phrase is one unit of as many words. *n_units gets the units;+ kind[i] and words[i] (the words the unit spans) per unit, room for+ max. Returns the number of words. */+size_t nsm_classify(enum nsm_lex lex, char *const *word, size_t n_words,+ enum nsm_kind *kind, size_t *span, size_t max,+ size_t *n_units);++/* 1 when w is a prime (single word) of the list. */+int nsm_is_prime(enum nsm_lex lex, const char *w);+/* 1 when w is a function word that is not a prime of the list. */+int nsm_is_stop(enum nsm_lex lex, const char *w);++/* The English exponents of the 65 primes and their forms, one table: the+ prime's name and a NULL-terminated list of forms (lower case, phrases+ with single spaces). For the restricted vocabulary. */+struct nsm_prime {+ const char *name;+ const char *const *forms;+};+extern const struct nsm_prime nsm_primes[];+extern const size_t nsm_n_primes;++/* NLTK's English stop words, NULL-terminated. */+extern const char *const nsm_stop_words[];++#endifdiff --git a/src/nsm/main.c b/src/nsm/main.cnew file mode 100644index 0000000..9040253--- /dev/null+++ b/src/nsm/main.c@@ -0,0 +1,531 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * janas-nsm - a word's explication in the Natural Semantic Metalanguage+ * (NSM), written by a model trained for it (DeepNSM, arXiv 2505.11764:+ * Llama 3 with a LoRA adapter, converted with tools/hf2gguf), and what the+ * explication is made of: its primes, function words and other words+ * ("molecules"), DeepNSM's legality, whether it is circular (score.h).+ *+ * janas-nsm <model.jns> <word> [example sentence...] one word+ * janas-nsm <model.jns> --batch <in.jsonl> a file of words+ * janas-nsm --score <in.jsonl> explications given+ *+ * A word's lines in a batch are JSON objects with "word" and "examples"+ * (DeepNSM's dataset as it is); --score reads "word" and "explication".+ * A batch writes one JSON line per explication (--out, else the standard+ * output) and a summary on the standard error.+ *+ * The prompt is the model card's ("Word: ...\nExamples:\n...\nParaphrase:");+ * --prompt paper is the one the paper's numbers were obtained with (its+ * evaluation script's, indentation included). The sampling is DeepNSM's+ * (temperature 0.7, top-k 50, top-p 0.95, at most 512 tokens).+ */+#include <errno.h>+#include <math.h>+#include <stdio.h>+#include <stdlib.h>+#include <string.h>++#include "janas/llm.h"+#include "llm/json.h"+#include "score.h"+#include "vocab.h"++struct opts {+ const char *model, *batch, *score, *out, *prompt, *molecules;+ int strict;+ enum nsm_lex lex;+ int samples, max, quiet;+ uint64_t seed;+ float temp, top_p;+ int top_k;+};++static void usage(void)+{+ fprintf(stderr,+ "usage: janas-nsm <model.jns> <word> [example sentence...] "+ "[options]\n"+ " janas-nsm <model.jns> --batch <in.jsonl> [options]\n"+ " janas-nsm --score <in.jsonl> [options]\n"+ "A word's explication in NSM primes, by a model trained for it\n"+ "(DeepNSM), and what it is made of: primes, function words and\n"+ "other words (molecules), legality 10 (primes - molecules) / "+ "words,\n"+ "circularity (the word itself in its explication).\n"+ " --batch <file> JSON lines with \"word\" and \"examples\"\n"+ " --score <file> JSON lines with \"word\" and \"explication\": "+ "count\n"+ " them, no model\n"+ " --out <file> a batch's JSON lines (default: standard "+ "output)\n"+ " -n <count> explications a word (default 1)\n"+ " --seed <n> the sampling's seed (default 1)\n"+ " --prompt <card|paper> the model card's prompt (default) or "+ "the\n"+ " one the paper's numbers were obtained with\n"+ " --lexicon <paper|full> primes as DeepNSM's scores counted "+ "them\n"+ " (default), or the 65 with their forms and "+ "phrases\n"+ " --strict the model may write only the primes' forms, "+ "function\n"+ " words, punctuation and the molecules: its "+ "output head\n"+ " computes those tokens alone\n"+ " --molecules <file> the molecules --strict allows, one word a "+ "line\n"+ " (default: a list of everyday words; an empty "+ "file: none)\n"+ " --temp <t> --top-k <k> --top-p <p> --max <tokens>\n"+ " sampling (default 0.7, 50, 0.95, 512)\n"+ " --quiet no progress on the standard error\n");+}++static char *read_file(const char *path, size_t *len)+{+ FILE *f = fopen(path, "rb");+ if (!f)+ return NULL;+ fseek(f, 0, SEEK_END);+ long n = ftell(f);+ fseek(f, 0, SEEK_SET);+ char *b = n >= 0 ? malloc((size_t)n + 1) : NULL;+ if (b && fread(b, 1, (size_t)n, f) != (size_t)n) {+ free(b);+ b = NULL;+ }+ fclose(f);+ if (b) {+ b[n] = 0;+ *len = (size_t)n;+ }+ return b;+}++/* --- the summary of a batch --- */++struct tally {+ size_t n;+ double leg, leg2, primes, molecules, circular;+ double out_tokens, out_s, in_tokens, in_s;+};++static void tally_add(struct tally *t, const struct nsm_score *s)+{+ t->n++;+ t->leg += s->legality;+ t->leg2 += s->legality * s->legality;+ t->primes += s->primes_ratio;+ t->molecules += s->molecules_ratio;+ t->circular += s->circular;+}++static void tally_print(const struct tally *t)+{+ if (!t->n)+ return;+ double n = (double)t->n, m = t->leg / n;+ double sd = t->n > 1 ? sqrt((t->leg2 - n * m * m) / (n - 1)) : 0;+ fprintf(stderr,+ "%zu explications: legality %.2f +- %.2f (standard error), "+ "primes %.1f%%, molecules %.1f%%, circular %.1f%%\n",+ t->n, m, sd / sqrt(n), 100 * t->primes / n, 100 * t->molecules / n,+ 100 * t->circular / n);+ if (t->out_s > 0)+ fprintf(stderr,+ "writing %.0f tokens in %.1f s, %.1f tokens/s; reading the "+ "prompts %.0f tokens in %.1f s, %.0f tokens/s\n",+ t->out_tokens, t->out_s, t->out_tokens / t->out_s, t->in_tokens,+ t->in_s, t->in_s > 0 ? t->in_tokens / t->in_s : 0);+}++static void put_score(struct janas_buf *b, const struct nsm_score *s)+{+ janas_buf_printf(b,+ ", \"legality\": %.4f, \"primes\": %.4f, "+ "\"molecules\": %.4f, \"words\": %zu, \"circular\": %s",+ s->legality, s->primes_ratio, s->molecules_ratio, s->words,+ s->circular ? "true" : "false");+}++/* --- the model --- */++struct gen {+ janas_llm *m;+ janas_llm_chat *c;+ struct janas_llm_chat_params p;+};++/* The words of a file, one a line, NULL-terminated (malloc'd, into *text). */+static const char **read_words(const char *path, char **text)+{+ size_t len;+ *text = read_file(path, &len);+ if (!*text) {+ fprintf(stderr, "janas-nsm: %s: %s\n", path, strerror(errno));+ exit(1);+ }+ size_t n = 0;+ for (size_t i = 0; i < len; i++)+ n += (*text)[i] == '\n';+ const char **w = calloc(n + 2, sizeof(*w));+ size_t k = 0;+ for (char *p = strtok(*text, "\r\n"); p && w; p = strtok(NULL, "\r\n"))+ if (*p)+ w[k++] = p;+ return w;+}++/* --strict: the replies' vocabulary */+static void restrict_vocab(struct gen *g, const struct opts *o)+{+ char *text = NULL;+ const char **mol = o->molecules ? read_words(o->molecules, &text) : NULL;+ int32_t *ids = NULL;+ long n = nsm_vocab_tokens(g->m, mol, &ids);+ if (n <= 0 || janas_llm_chat_vocab(g->c, (int32_t)n, ids) != JANAS_LLM_OK) {+ fprintf(stderr, "janas-nsm: --strict: %s\n",+ n <= 0 ? "no vocabulary" : janas_llm_last_error());+ exit(1);+ }+ if (!o->quiet)+ fprintf(stderr, "janas-nsm: --strict, %ld tokens of the vocabulary\n",+ n);+ free(ids);+ free(mol);+ free(text);+}++static void gen_open(struct gen *g, const struct opts *o)+{+ struct janas_llm_params lp;+ janas_llm_params_default(&lp);+ lp.n_ctx = 4096;+ if (!o->quiet)+ fprintf(stderr, "janas-nsm: loading %s ...\n", o->model);+ if (janas_llm_open(o->model, &lp, &g->m) != JANAS_LLM_OK) {+ fprintf(stderr, "janas-nsm: %s\n", janas_llm_last_error());+ exit(1);+ }+ janas_llm_chat_params_default(&g->p);+ g->p.temperature = o->temp;+ g->p.top_k = o->top_k;+ g->p.top_p = o->top_p;+ g->p.min_p = 0;+ g->p.max_reply = o->max;+ g->p.speculate = 0;+ if (janas_llm_chat_create(g->m, &g->p, &g->c) != JANAS_LLM_OK) {+ fprintf(stderr, "janas-nsm: %s\n", janas_llm_last_error());+ exit(1);+ }+ if (o->strict)+ restrict_vocab(g, o);+}++/* The prompt for a word and its examples. */+static void build_prompt(struct janas_buf *b, const struct opts *o,+ const char *word, const char *const *ex, int n_ex)+{+ int paper = !strcmp(o->prompt, "paper");+ /* nsm_evaluation.py's f-string keeps its source's indentation */+ const char *ind = paper ? " " : "";+ janas_buf_printf(b, "Word: %s\n%sExamples:\n%s", word, ind, ind);+ for (int i = 0; i < n_ex; i++)+ janas_buf_printf(b, "%s%s", i ? "\n" : "", ex[i]);+ janas_buf_printf(b, "\n%sParaphrase:%s%s", ind, paper ? "\n" : "", ind);+}++/* One explication into out; its tokens and times into t. */+static int explain(struct gen *g, const char *prompt, size_t n, uint64_t seed,+ struct janas_buf *out, struct tally *t)+{+ g->p.seed = seed;+ if (janas_llm_chat_reset(g->c) != JANAS_LLM_OK ||+ janas_llm_chat_set_params(g->c, &g->p) != JANAS_LLM_OK ||+ janas_llm_chat_prompt(g->c, prompt, (int32_t)n) != JANAS_LLM_OK)+ return -1;+ out->n = 0;+ char piece[256];+ for (;;) {+ int32_t len = 0;+ int32_t rc = janas_llm_chat_next(g->c, piece, sizeof(piece), &len);+ if (rc == JANAS_LLM_DONE)+ break;+ if (rc != JANAS_LLM_OK)+ return -1;+ janas_buf_put(out, piece, (size_t)len);+ }+ struct janas_llm_chat_stats st = {.size = sizeof(st)};+ janas_llm_chat_stats(g->c, &st);+ t->out_tokens += st.output_tokens;+ t->out_s += st.output_seconds;+ t->in_tokens += st.input_tokens;+ t->in_s += st.input_seconds;+ /* the model writes a space after "Paraphrase:", and a newline last */+ size_t a = 0;+ while (a < out->n && (out->p[a] == ' ' || out->p[a] == '\n'))+ a++;+ while (out->n > a &&+ (out->p[out->n - 1] == ' ' || out->p[out->n - 1] == '\n'))+ out->n--;+ memmove(out->p, out->p + a, out->n - a);+ out->n -= a;+ janas_buf_put(out, "", 1);+ out->n--;+ return 0;+}++/* --- the three ways to run --- */++static int one_word(const struct opts *o, const char *word,+ const char *const *ex, int n_ex)+{+ struct gen g;+ gen_open(&g, o);+ struct janas_buf pr = {0}, out = {0};+ build_prompt(&pr, o, word, ex, n_ex);+ struct tally t = {0};+ for (int k = 0; k < o->samples; k++) {+ if (explain(&g, pr.p, pr.n, o->seed * 1000003u + (uint64_t)k, &out,+ &t) != 0) {+ fprintf(stderr, "janas-nsm: %s\n", janas_llm_last_error());+ return 1;+ }+ struct nsm_score s;+ nsm_score_text(o->lex, out.p, out.n, word, &s);+ tally_add(&t, &s);+ printf("%s%.*s\n\n[legality %.2f, primes %.0f%%, molecules %.0f%%, "+ "%zu words%s]\n",+ k ? "\n" : "", (int)out.n, out.p, s.legality,+ 100 * s.primes_ratio, 100 * s.molecules_ratio, s.words,+ s.circular ? ", circular" : "");+ }+ if (o->samples > 1 || !o->quiet)+ tally_print(&t);+ janas_buf_free(&pr);+ janas_buf_free(&out);+ janas_llm_chat_destroy(g.c);+ janas_llm_close(g.m);+ return 0;+}++/* Calls f on each JSON line of the file. */+static int each_line(const char *path,+ int (*f)(const struct janas_json *, size_t, void *),+ void *ctx)+{+ size_t len;+ char *text = read_file(path, &len);+ if (!text) {+ fprintf(stderr, "janas-nsm: %s: %s\n", path, strerror(errno));+ return 1;+ }+ size_t line = 0;+ for (char *p = text; *p;) {+ char *e = strchr(p, '\n');+ size_t n = e ? (size_t)(e - p) : strlen(p);+ line++;+ if (n) {+ char err[128];+ struct janas_json_doc *d = janas_json_parse(p, n, err, sizeof(err));+ if (!d) {+ fprintf(stderr, "janas-nsm: %s:%zu: %s\n", path, line, err);+ free(text);+ return 1;+ }+ int rc = f(janas_json_root(d), line, ctx);+ janas_json_free(d);+ if (rc) {+ free(text);+ return rc;+ }+ }+ p += n + (e != NULL);+ }+ free(text);+ return 0;+}++struct run {+ const struct opts *o;+ struct gen g;+ FILE *out;+ struct tally t;+ size_t index;+};++static int score_line(const struct janas_json *v, size_t line, void *ctx)+{+ struct run *r = ctx;+ const char *w = janas_json_str(janas_json_get(v, "word"));+ const struct janas_json *e = janas_json_get(v, "explication");+ if (!e)+ e = janas_json_get(v, "text");+ if (!w || !janas_json_str(e)) {+ fprintf(stderr,+ "janas-nsm: line %zu: no \"word\" and \"explication\"\n", line);+ return 1;+ }+ struct nsm_score s;+ nsm_score_text(r->o->lex, e->s, e->n, w, &s);+ tally_add(&r->t, &s);+ struct janas_buf b = {0};+ janas_buf_puts(&b, "{\"word\": ");+ janas_json_write_str(&b, w, strlen(w));+ put_score(&b, &s);+ janas_buf_puts(&b, "}\n");+ fwrite(b.p, 1, b.n, r->out);+ janas_buf_free(&b);+ return 0;+}++static int batch_line(const struct janas_json *v, size_t line, void *ctx)+{+ struct run *r = ctx;+ const char *w = janas_json_str(janas_json_get(v, "word"));+ const struct janas_json *ex = janas_json_get(v, "examples");+ if (!w || !ex || ex->type != JANAS_JSON_ARRAY) {+ fprintf(stderr, "janas-nsm: line %zu: no \"word\" and \"examples\"\n",+ line);+ return 1;+ }+ const char *exs[64];+ int n_ex = 0;+ for (const struct janas_json *x = ex->child; x && n_ex < 64; x = x->next)+ if (janas_json_str(x))+ exs[n_ex++] = x->s;+ struct janas_buf pr = {0}, out = {0}, b = {0};+ build_prompt(&pr, r->o, w, exs, n_ex);+ for (int k = 0; k < r->o->samples; k++) {+ double s0 = r->t.out_s, t0 = r->t.out_tokens;+ uint64_t seed = (r->o->seed * 1000003u + r->index) * 31u + (uint64_t)k;+ if (explain(&r->g, pr.p, pr.n, seed, &out, &r->t) != 0) {+ fprintf(stderr, "janas-nsm: line %zu: %s\n", line,+ janas_llm_last_error());+ return 1;+ }+ struct nsm_score s;+ nsm_score_text(r->o->lex, out.p, out.n, w, &s);+ tally_add(&r->t, &s);+ b.n = 0;+ janas_buf_puts(&b, "{\"word\": ");+ janas_json_write_str(&b, w, strlen(w));+ janas_buf_printf(&b, ", \"sample\": %d, \"explication\": ", k);+ janas_json_write_str(&b, out.p, out.n);+ put_score(&b, &s);+ janas_buf_printf(&b, ", \"tokens\": %.0f, \"seconds\": %.3f}\n",+ r->t.out_tokens - t0, r->t.out_s - s0);+ fwrite(b.p, 1, b.n, r->out);+ fflush(r->out);+ }+ r->index++;+ if (!r->o->quiet && r->index % 10 == 0)+ fprintf(stderr, "janas-nsm: %zu words, %.1f tokens/s\n", r->index,+ r->t.out_s > 0 ? r->t.out_tokens / r->t.out_s : 0);+ janas_buf_free(&pr);+ janas_buf_free(&out);+ janas_buf_free(&b);+ return 0;+}++static int run_file(const struct opts *o, int with_model)+{+ struct run r = {.o = o, .out = stdout};+ if (o->out && !(r.out = fopen(o->out, "w"))) {+ fprintf(stderr, "janas-nsm: %s: %s\n", o->out, strerror(errno));+ return 1;+ }+ if (with_model)+ gen_open(&r.g, o);+ int rc = each_line(with_model ? o->batch : o->score,+ with_model ? batch_line : score_line, &r);+ tally_print(&r.t);+ if (r.out != stdout)+ fclose(r.out);+ if (with_model) {+ janas_llm_chat_destroy(r.g.c);+ janas_llm_close(r.g.m);+ }+ return rc;+}++int main(int argc, char **argv)+{+ struct opts o = {.prompt = "card",+ .lex = NSM_LEX_PAPER,+ .samples = 1,+ .max = 512,+ .seed = 1,+ .temp = 0.7f,+ .top_p = 0.95f,+ .top_k = 50};+ const char *pos[66];+ int n_pos = 0;+ for (int i = 1; i < argc; i++) {+ const char *a = argv[i];+ const char *v = i + 1 < argc ? argv[i + 1] : NULL;+#define TAKES(name) (!strcmp(a, name) && v && ++i)+ if (!strcmp(a, "-h") || !strcmp(a, "--help")) {+ usage();+ return 0;+ } else if (TAKES("--batch"))+ o.batch = v;+ else if (TAKES("--score"))+ o.score = v;+ else if (TAKES("--out"))+ o.out = v;+ else if (TAKES("-n"))+ o.samples = atoi(v);+ else if (TAKES("--seed"))+ o.seed = strtoull(v, NULL, 10);+ else if (TAKES("--prompt"))+ o.prompt = v;+ else if (TAKES("--lexicon"))+ o.lex = !strcmp(v, "full") ? NSM_LEX_FULL : NSM_LEX_PAPER;+ else if (TAKES("--temp"))+ o.temp = (float)atof(v);+ else if (TAKES("--top-k"))+ o.top_k = atoi(v);+ else if (TAKES("--top-p"))+ o.top_p = (float)atof(v);+ else if (TAKES("--max"))+ o.max = atoi(v);+ else if (TAKES("--molecules"))+ o.molecules = v;+ else if (!strcmp(a, "--strict"))+ o.strict = 1;+ else if (!strcmp(a, "--quiet"))+ o.quiet = 1;+ else if (a[0] == '-' && a[1]) {+ fprintf(stderr, "janas-nsm: unknown option %s\n", a);+ usage();+ return 2;+ } else if (n_pos < 66)+ pos[n_pos++] = a;+#undef TAKES+ }+ if (strcmp(o.prompt, "card") && strcmp(o.prompt, "paper")) {+ fprintf(stderr, "janas-nsm: --prompt card or paper\n");+ return 2;+ }+ if (o.samples < 1 || o.max < 1) {+ usage();+ return 2;+ }+ if (o.score)+ return run_file(&o, 0);+ if (n_pos < 1) {+ usage();+ return 2;+ }+ o.model = pos[0];+ if (o.batch)+ return run_file(&o, 1);+ if (n_pos < 2) {+ usage();+ return 2;+ }+ return one_word(&o, pos[1], pos + 2, n_pos - 2);+}diff --git a/src/nsm/score.c b/src/nsm/score.cnew file mode 100644index 0000000..6a07b09--- /dev/null+++ b/src/nsm/score.c@@ -0,0 +1,205 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * score.c - an explication's words counted (score.h).+ */+#include "score.h"++#include <stdlib.h>+#include <string.h>++/* Python's \w for the characters an explication has: ASCII letters,+ digits and '_', and any character past ASCII except the punctuation+ models write (curly quotes, dashes, the ellipsis). */+static size_t word_char(const unsigned char *s, size_t n)+{+ if (s[0] < 0x80)+ return (s[0] >= 'a' && s[0] <= 'z') || (s[0] >= 'A' && s[0] <= 'Z') ||+ (s[0] >= '0' && s[0] <= '9') || s[0] == '_'+ ? 1+ : 0;+ size_t len = s[0] >= 0xf0 ? 4 : s[0] >= 0xe0 ? 3 : 2;+ if (len > n)+ return 0;+ if (len == 3 && s[0] == 0xe2 && s[1] == 0x80 &&+ ((s[2] >= 0x90 && s[2] <= 0xa7) || (s[2] >= 0xb0 && s[2] <= 0xbf)))+ return 0; /* U+2010..2027, U+2030..203F: dashes, quotes, ... */+ if (len == 2 && s[0] == 0xc2 &&+ (s[1] == 0xab || s[1] == 0xbb || s[1] == 0xa0 || s[1] == 0xb7))+ return 0; /* guillemets, no-break space, middle dot */+ return len;+}++static int is_space(unsigned char c)+{+ return c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\f' ||+ c == '\v';+}++static void copy_word(const char *w, char *out, size_t cap)+{+ size_t l = strlen(w);+ if (l >= cap)+ l = cap - 1;+ memcpy(out, w, l);+ out[l] = 0;+}++static const char *const irregular[][2] = {+ {"was", "be"}, {"were", "be"}, {"is", "be"},+ {"are", "be"}, {"been", "be"}, {"am", "be"},+ {"had", "have"}, {"has", "have"}, {"did", "do"},+ {"does", "do"}, {"done", "do"}, {"said", "say"},+ {"says", "say"}, {"saw", "see"}, {"seen", "see"},+ {"felt", "feel"}, {"knew", "know"}, {"known", "know"},+ {"thought", "think"}, {"made", "make"}, {"went", "go"},+ {"gone", "go"}, {"came", "come"}, {"took", "take"},+ {"taken", "take"}, {"children", "child"}, {"men", "man"},+ {"women", "woman"}, {"feet", "foot"}, {"teeth", "tooth"},+ {"mice", "mouse"}, {"geese", "goose"}, {"left", "leave"},+ {"got", "get"}, {"gave", "give"}, {"given", "give"},+ {"ran", "run"}, {"ate", "eat"}, {"eaten", "eat"},+ {"fell", "fall"}, {"fallen", "fall"}, {"dug", "dig"}};++void nsm_lemma(const char *w, char *out, size_t cap)+{+ for (size_t i = 0; i < sizeof(irregular) / sizeof(irregular[0]); i++)+ if (!strcmp(w, irregular[i][0])) {+ copy_word(irregular[i][1], out, cap);+ return;+ }+ static const char *const suf[][2] = {+ {"ies", "y"}, {"ied", "y"}, {"sses", "ss"}, {"ches", "ch"},+ {"shes", "sh"}, {"xes", "x"}, {"ing", ""}, {"ed", ""},+ {"es", "e"}, {"s", ""}};+ size_t n = strlen(w);+ size_t ends_ss = n >= 2 && w[n - 1] == 's' && w[n - 2] == 's';+ for (size_t i = 0; i < sizeof(suf) / sizeof(suf[0]); i++) {+ size_t ls = strlen(suf[i][0]), lr = strlen(suf[i][1]);+ /* a stem of two letters before the endings that say what they took+ off (flies, boxes), of three before the bare ones (thing, eyes) */+ size_t min = i < 6 ? 2 : 3;+ if (n < ls + min || strcmp(w + n - ls, suf[i][0]) || ends_ss)+ continue;+ size_t b = n - ls;+ if (b + lr >= cap)+ break;+ memcpy(out, w, b);+ memcpy(out + b, suf[i][1], lr);+ b += lr;+ /* a doubled consonant before -ing, -ed: stopped, running */+ if ((i == 6 || i == 7) && b > 3 && out[b - 1] == out[b - 2])+ b--;+ out[b] = 0;+ return;+ }+ copy_word(w, out, cap);+}++static int ends(const char *w, const char *suffix)+{+ size_t n = strlen(w), ls = strlen(suffix);+ return n > ls && !strcmp(w + n - ls, suffix);+}++/* The rules take a silent e off with -ing and -ed (making, promised: "mak",+ "promis"): one lemma with an e more than the other's matches when the+ other's word had one of those endings. */+static int same_lemma(const char *w, const char *wl, const char *t,+ const char *tl)+{+ if (!strcmp(wl, tl))+ return 1;+ size_t a = strlen(wl), b = strlen(tl);+ if (a + 1 == b && tl[a] == 'e' && !strncmp(wl, tl, a))+ return ends(w, "ing") || ends(w, "ed");+ if (b + 1 == a && wl[b] == 'e' && !strncmp(wl, tl, b))+ return ends(t, "ing") || ends(t, "ed");+ return 0;+}++void nsm_score_text(enum nsm_lex lex, const char *text, size_t n,+ const char *target, struct nsm_score *s)+{+ memset(s, 0, sizeof(*s));+ /* lower-cased, the punctuation dropped, then split */+ char *buf = malloc(n + 1);+ char **word = malloc(sizeof(char *) * (n / 2 + 2));+ if (!buf || !word) {+ free(buf);+ free(word);+ return;+ }+ size_t w = 0, nw = 0;+ int in = 0;+ for (size_t i = 0; i < n;) {+ const unsigned char *c = (const unsigned char *)text + i;+ size_t k = word_char(c, n - i);+ int keep_apos = lex == NSM_LEX_FULL &&+ (c[0] == '\'' || (n - i >= 3 && c[0] == 0xe2 &&+ c[1] == 0x80 && c[2] == 0x99)) &&+ in;+ if (k || keep_apos) {+ if (!in) {+ word[nw++] = buf + w;+ in = 1;+ }+ if (keep_apos) {+ buf[w++] = '\'';+ i += c[0] == '\'' ? 1 : 3;+ continue;+ }+ for (size_t j = 0; j < k; j++)+ buf[w++] =+ (char)(k == 1 && c[j] >= 'A' && c[j] <= 'Z' ? c[j] + 32+ : c[j]);+ i += k;+ } else if (is_space(c[0])) {+ if (in)+ buf[w++] = 0;+ in = 0;+ i++;+ } else { /* punctuation: removed, the word goes on */+ i += c[0] >= 0xf0 ? 4 : c[0] >= 0xe0 ? 3 : c[0] >= 0xc0 ? 2 : 1;+ }+ }+ if (in)+ buf[w++] = 0;+ enum nsm_kind *kind = malloc(sizeof(*kind) * (nw + 1));+ size_t *span = malloc(sizeof(*span) * (nw + 1)), nu = 0;+ if (kind && span) {+ nsm_classify(lex, word, nw, kind, span, nw + 1, &nu);+ for (size_t u = 0; u < nu; u++) {+ if (kind[u] == NSM_PRIME)+ s->primes += span[u];+ else if (kind[u] == NSM_STOP)+ s->stops += span[u];+ else+ s->molecules += span[u];+ }+ s->words = nw;+ if (nw) {+ s->primes_ratio = (double)s->primes / (double)nw;+ s->molecules_ratio = (double)s->molecules / (double)nw;+ }+ s->legality = 10 * (s->primes_ratio - s->molecules_ratio);+ char tl[128], wl[128], low[128];+ size_t lt = strlen(target);+ if (lt >= sizeof(low))+ lt = sizeof(low) - 1;+ for (size_t i = 0; i < lt; i++)+ low[i] =+ (char)(target[i] >= 'A' && target[i] <= 'Z' ? target[i] + 32+ : target[i]);+ low[lt] = 0;+ nsm_lemma(low, tl, sizeof(tl));+ for (size_t i = 0; i < nw && !s->circular; i++) {+ nsm_lemma(word[i], wl, sizeof(wl));+ s->circular = same_lemma(word[i], wl, low, tl);+ }+ }+ free(kind);+ free(span);+ free(word);+ free(buf);+}diff --git a/src/nsm/score.h b/src/nsm/score.hnew file mode 100644index 0000000..56a41eb--- /dev/null+++ b/src/nsm/score.h@@ -0,0 +1,35 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * score.h - what an NSM explication is made of, counted.+ *+ * With NSM_LEX_PAPER the counts are DeepNSM's (prompts.py, legality_score):+ * the text lower-cased, every character that is neither a word character+ * nor a space removed (apostrophes too: "don't" becomes "dont"), split at+ * spaces; each word a prime, a function word or else a "molecule". Their+ * legality is 10 (primes - molecules) / words, -10 to 10; an explication+ * is circular when a word of it has the target word's lemma (theirs with+ * spaCy, here with English suffix rules: the same verdict on all 1341 of+ * their published records). With NSM_LEX_FULL the apostrophes stay and+ * the primes' phrases count as primes.+ */+#ifndef JANAS_NSM_SCORE_H+#define JANAS_NSM_SCORE_H++#include <stddef.h>++#include "lexicon.h"++struct nsm_score {+ size_t words, primes, stops, molecules; /* primes: words in primes */+ double primes_ratio, molecules_ratio, legality;+ int circular;+};++void nsm_score_text(enum nsm_lex lex, const char *text, size_t n,+ const char *target, struct nsm_score *s);++/* An English lemma by suffix rules, into out (cap bytes). */+void nsm_lemma(const char *w, char *out, size_t cap);++#endifdiff --git a/src/nsm/vocab.c b/src/nsm/vocab.cnew file mode 100644index 0000000..b28eeb1--- /dev/null+++ b/src/nsm/vocab.c@@ -0,0 +1,170 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * vocab.c - the tokens of an NSM explication (vocab.h).+ */+#include "vocab.h"++#include <stdio.h>+#include <stdlib.h>+#include <string.h>++#include "lexicon.h"++const char *const nsm_default_molecules[] = {+ /* the body */+ "head", "face", "eyes", "eye", "ears", "ear", "nose", "mouth", "teeth",+ "tongue", "hair", "skin", "hands", "hand", "fingers", "arms", "arm", "legs",+ "leg", "feet", "foot", "back", "belly", "heart", "blood", "bones", "wings",+ "tail",+ /* physical qualities */+ "long", "short", "round", "flat", "thin", "thick", "hard", "soft", "sharp",+ "heavy", "light", "hot", "cold", "warm", "wet", "dry", "smooth", "straight",+ "high", "low", "fast", "slow", "loud", "quiet", "bright", "dark", "colour",+ "color", "red", "white", "black", "green", "blue", "yellow", "sweet",+ "sour",+ /* nature */+ "water", "ground", "earth", "sky", "sun", "moon", "stars", "fire", "rain",+ "wind", "air", "stone", "stones", "wood", "tree", "trees", "plant",+ "plants", "grass", "leaves", "flowers", "fruit", "seeds", "animal",+ "animals", "creature", "creatures", "bird", "birds", "fish", "egg", "eggs",+ "sea", "river", "mountain", "day", "days", "night", "nights", "year",+ "years",+ /* people */+ "man", "men", "woman", "women", "child", "children", "baby", "babies",+ "boy", "girl", "mother", "father", "family", "husband", "wife", "friend",+ "friends", "born",+ /* doing */+ "make", "makes", "made", "eat", "eats", "eating", "drink", "sleep", "sit",+ "stand", "lie", "walk", "run", "go", "goes", "going", "went", "come",+ "comes", "came", "give", "gives", "take", "takes", "hold", "put", "puts",+ "kill", "cut", "break", "hit", "carry", "wear", "cook", "play", "work",+ "help", "need", "try", "use", "uses", "call", "called", "laugh", "cry",+ "sing", "read", "write", "show", "look", "looks", "find", "keep", "open",+ "close", "start", "stop", "get", "gets", "let", "become", "becomes",+ "change", "changes", "grow", "grows", "living",+ /* things and places */+ "house", "home", "food", "money", "clothes", "tool", "tools", "machine",+ "machines", "town", "country", "school", "road", "car", "door", "room",+ "bed", "table",+ /* time, space, number */+ "together", "around", "again", "always", "often", "sometimes", "never",+ "also", "first", "second", "last", "end", "outside", "top", "bottom",+ "front", "middle", "number", "numbers", "three", "four", "five", "ten",+ "hundred", "enough", "almost", "every", "everything", "everyone", "anyone",+ "anything", "nothing", "nobody",+ /* the mind, and how things are */+ "mind", "idea", "ideas", "reason", "question", "answer", "problem", "name",+ "story", "language", "sound", "sounds", "voice", "picture", "sign", "rule",+ "rules", "real", "different", "special", "important", "easy", "difficult",+ "possible", "happy", "sad", "afraid", "angry", "sorry", "pain", "sick",+ "dead", "alive", "young", "old", "new", "able", "free", "safe", "dangerous",+ "beautiful", "clean", "dirty", "full", "empty", "ready", "strong", "weak",+ "rich", "poor", "wrong", "right", NULL};++struct set {+ uint8_t *in;+ int32_t *ids;+ long n;+ int32_t nv;+};++static void add_text(const janas_llm *m, struct set *s, const char *t)+{+ int32_t ids[64], n = 0;+ if (janas_llm_tokenize(m, t, (int32_t)strlen(t), 0, ids, 64, &n) !=+ JANAS_LLM_OK)+ return;+ for (int32_t i = 0; i < n; i++)+ if (ids[i] >= 0 && ids[i] < s->nv && !s->in[ids[i]]) {+ s->in[ids[i]] = 1;+ s->ids[s->n++] = ids[i];+ }+}++/* A word's four spellings: at a line's start, after a space, capitalized. */+static void add_word(const janas_llm *m, struct set *s, const char *w)+{+ char b[128];+ size_t n = strlen(w);+ if (!n || n + 2 > sizeof(b))+ return;+ for (int cap = 0; cap < 2; cap++)+ for (int sp = 0; sp < 2; sp++) {+ snprintf(b, sizeof(b), "%s%s", sp ? " " : "", w);+ if (cap && b[sp] >= 'a' && b[sp] <= 'z')+ b[sp] = (char)(b[sp] - 32);+ add_text(m, s, b);+ }+}++/* A token of white space and punctuation alone (curly quotes included). */+static int only_punct(const char *t, int32_t n)+{+ if (n <= 0)+ return 0;+ for (int32_t i = 0; i < n;) {+ unsigned char c = (unsigned char)t[i];+ if (c < 0x80) {+ if (!strchr(" \n\t.,:;!?\"'()-", c) || !c)+ return 0;+ i++;+ } else if (n - i >= 3 && c == 0xe2 && (unsigned char)t[i + 1] == 0x80 &&+ ((unsigned char)t[i + 2] == 0x9c ||+ (unsigned char)t[i + 2] == 0x9d ||+ (unsigned char)t[i + 2] == 0x98 ||+ (unsigned char)t[i + 2] == 0x99)) {+ i += 3;+ } else+ return 0;+ }+ return 1;+}++long nsm_vocab_tokens(const janas_llm *m, const char *const *molecules,+ int32_t **ids)+{+ /* the vocabulary's size: the first id token_text refuses */+ char t[1024];+ int32_t nv = 0, len;+ while (nv < (1 << 24) &&+ janas_llm_token_text(m, nv, t, sizeof(t), &len) == JANAS_LLM_OK)+ nv++;+ if (nv <= 0)+ return -1;+ struct set s = {.in = calloc((size_t)nv, 1),+ .ids = malloc((size_t)nv * sizeof(int32_t)),+ .nv = nv};+ if (!s.in || !s.ids) {+ free(s.in);+ free(s.ids);+ return -1;+ }+ for (size_t p = 0; p < nsm_n_primes; p++)+ for (const char *const *f = nsm_primes[p].forms; *f; f++) {+ /* a phrase's words one by one */+ char b[64];+ snprintf(b, sizeof(b), "%s", *f);+ for (char *w = strtok(b, " "); w; w = strtok(NULL, " "))+ add_word(m, &s, w);+ }+ for (const char *const *w = nsm_stop_words; *w; w++)+ add_word(m, &s, *w);+ for (const char *const *w = molecules ? molecules : nsm_default_molecules;+ *w; w++)+ add_word(m, &s, *w);+ for (int32_t i = 0; i < nv; i++) {+ len = 0;+ if (s.in[i] ||+ janas_llm_token_text(m, i, t, sizeof(t), &len) != JANAS_LLM_OK ||+ len > (int32_t)sizeof(t))+ continue;+ if (only_punct(t, len)) {+ s.in[i] = 1;+ s.ids[s.n++] = i;+ }+ }+ free(s.in);+ *ids = s.ids;+ return s.n;+}diff --git a/src/nsm/vocab.h b/src/nsm/vocab.hnew file mode 100644index 0000000..1c09256--- /dev/null+++ b/src/nsm/vocab.h@@ -0,0 +1,28 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * vocab.h - the tokens an NSM explication may be written with (janas-nsm+ * --strict): the primes' forms, the function words and a list of semantic+ * molecules, each as the model's tokens of "w", " w", "W" and " W", and+ * every token made of white space and punctuation alone.+ */+#ifndef JANAS_NSM_VOCAB_H+#define JANAS_NSM_VOCAB_H++#include <stddef.h>+#include <stdint.h>++#include "janas/llm.h"++/* The default molecules, NULL-terminated: body parts, physical qualities,+ nature, people, everyday actions and things - the words NSM's+ explications of concrete meanings lean on (Goddard and Wierzbicka's+ molecules are of these kinds), written for Janas. */+extern const char *const nsm_default_molecules[];++/* The token ids, malloc'd, into *ids; their count, or -1. molecules NULL:+ the default ones; else a NULL-terminated list. */+long nsm_vocab_tokens(const janas_llm *m, const char *const *molecules,+ int32_t **ids);++#endifdiff --git a/tests/test_nsm.c b/tests/test_nsm.cnew file mode 100644index 0000000..4d3d33b--- /dev/null+++ b/tests/test_nsm.c@@ -0,0 +1,116 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * test_nsm.c - janas-nsm's counts (src/nsm/score.c): explications counted+ * by hand, with DeepNSM's list (punctuation and apostrophes dropped, single+ * words) and with the full one (phrases as primes, apostrophes kept), UTF-8+ * quotes taken off whole, the circularity of a word's inflected forms, and+ * the lemma's rules.+ */+#include <math.h>+#include <stdio.h>+#include <string.h>+#include <unistd.h>++#include "nsm/lexicon.c"+#include "nsm/score.c"++static int failures;++#define CHECK(c, ...) \+ do { \+ if (!(c)) { \+ printf(__VA_ARGS__); \+ printf("\n"); \+ failures++; \+ } \+ } while (0)++static void counts(enum nsm_lex lex, const char *text, const char *target,+ size_t words, size_t primes, size_t stops, size_t mols,+ double legality, int circular)+{+ struct nsm_score s;+ nsm_score_text(lex, text, strlen(text), target, &s);+ CHECK(s.words == words && s.primes == primes && s.stops == stops &&+ s.molecules == mols,+ "%s \"%s\": %zu words %zu primes %zu stops %zu molecules, "+ "want %zu %zu %zu %zu",+ lex == NSM_LEX_PAPER ? "paper" : "full", text, s.words, s.primes,+ s.stops, s.molecules, words, primes, stops, mols);+ CHECK(fabs(s.legality - legality) < 1e-9,+ "\"%s\": legality %.6f, want %.6f", text, s.legality, legality);+ CHECK(s.circular == circular, "\"%s\" for %s: circular %d, want %d", text,+ target, s.circular, circular);+}++static void lemma(const char *w, const char *want)+{+ char b[64];+ nsm_lemma(w, b, sizeof(b));+ CHECK(!strcmp(b, want), "lemma of %s: %s, want %s", w, b, want);+}++int main(void)+{+ alarm(60);+ /* someone can see this kind = primes, is too in their list; of it it+ on the: function words; animal long moves ground: molecules ("move"+ is a prime, "moves" not in their list) */+ counts(NSM_LEX_PAPER,+ "Someone can see this kind of animal. It is long, it moves on "+ "the ground.",+ "snake", 15, 6, 5, 4, 10.0 * (6 - 4) / 15, 0);+ /* the same in the full list: "moves" a prime's form, "is" alone a+ function word */+ counts(NSM_LEX_FULL,+ "Someone can see this kind of animal. It is long, it moves on "+ "the ground.",+ "snake", 15, 6, 6, 3, 10.0 * (6 - 3) / 15, 0);+ /* their counting: "doesn't" becomes "doesnt", a molecule; "time" a+ prime, "long" a molecule, phrases never matched */+ counts(NSM_LEX_PAPER, "Someone doesn't want this for a long time.", "wait",+ 8, 4, 2, 2, 10.0 * (4 - 2) / 8, 0);+ /* the full list: "doesn't want" and "a long time" primes, whole */+ counts(NSM_LEX_FULL, "Someone doesn't want this for a long time.", "wait",+ 8, 7, 1, 0, 10.0 * 7 / 8, 0);+ /* curly quotes and a dash off whole, not as stray bytes */+ counts(NSM_LEX_PAPER,+ "\xe2\x80\x9cSomeone\xe2\x80\x9d sees \xe2\x80\x94 "+ "this.",+ "x", 3, 2, 0, 1, 10.0 * (2 - 1) / 3, 0);+ /* circular: the target or an inflected form of it */+ counts(NSM_LEX_PAPER, "Snakes move.", "snake", 2, 1, 0, 1,+ 10.0 * (1 - 1) / 2, 1);+ counts(NSM_LEX_PAPER, "It is running.", "run", 3, 1, 1, 1,+ 10.0 * (1 - 1) / 3, 1);+ counts(NSM_LEX_PAPER, "Children play.", "child", 2, 0, 0, 2,+ 10.0 * (0 - 2) / 2, 1);+ counts(NSM_LEX_PAPER, "Something sneaky.", "snake", 2, 1, 0, 1, 0.0, 0);+ /* a silent e the rules took off the target, or the word */+ counts(NSM_LEX_PAPER, "They make it.", "making", 3, 0, 2, 1, -10.0 / 3, 1);+ counts(NSM_LEX_PAPER, "Someone was making it.", "make", 4, 1, 2, 1, 0.0, 1);+ counts(NSM_LEX_PAPER, "They use it.", "us", 3, 0, 2, 1, -10.0 / 3, 0);+ /* nothing to count */+ counts(NSM_LEX_PAPER, " ... ", "x", 0, 0, 0, 0, 0.0, 0);++ lemma("snakes", "snake");+ lemma("running", "run");+ lemma("stopped", "stop");+ lemma("flies", "fly");+ lemma("tried", "try");+ lemma("boxes", "box");+ lemma("watches", "watch");+ lemma("classes", "class");+ lemma("glass", "glass");+ lemma("went", "go");+ lemma("feet", "foot");+ lemma("dog", "dog");++ if (failures) {+ printf("test_nsm: %d failures\n", failures);+ return 1;+ }+ printf("test_nsm: ok\n");+ return 0;+}diff --git a/tests/vocab_check.c b/tests/vocab_check.cnew file mode 100644index 0000000..af13c8e--- /dev/null+++ b/tests/vocab_check.c@@ -0,0 +1,148 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * vocab_check.c - the output head restricted to a vocabulary+ * (janas_llm_model_set_vocab, janas-nsm --strict) against the whole head:+ * the same token ids through both, one at a time and in blocks with every+ * token's logits; the vocabulary's logits must be the same to the bit, the+ * others -inf, and the whole head must come back unchanged after.+ *+ * Usage: vocab_check <model.jns> [compute_cpus] [every=K] [block=B]+ * The vocabulary is every K-th token id (default 37) and the stretch 0 .. 299+ * (the bytes and the commonest tokens); the text is 64 token ids spread over+ * the vocabulary, the same every run.+ */+#include <math.h>+#include <stdio.h>+#include <stdlib.h>+#include <string.h>++#include "llm/model.h"++#define N_TOK 64++static int parse_cpus(const char *s, int *cpus)+{+ char buf[1024];+ snprintf(buf, sizeof(buf), "%s", s);+ int n = 0;+ for (char *t = strtok(buf, ","); t && n < 256; t = strtok(NULL, ","))+ cpus[n++] = atoi(t);+ return n;+}++/* the logits of all N_TOK tokens, one at a time (block 1) or in blocks */+static int run(struct janas_llm_model *m, const int32_t *tok, uint32_t block,+ float *out, uint32_t nv)+{+ for (uint32_t p = 0; p < N_TOK; p += block) {+ uint32_t b = N_TOK - p < block ? N_TOK - p : block;+ if (janas_llm_model_forward(m, tok + p, b, p, out + (size_t)p * nv,+ 1) != 0)+ return -1;+ }+ return 0;+}++int main(int argc, char **argv)+{+ if (argc < 2) {+ fprintf(stderr, "usage: vocab_check <model.jns> [compute_cpus] "+ "[every=K] [block=B]\n");+ return 2;+ }+ uint32_t every = 37, block = 16;+ for (int i = 2; i < argc; i++) {+ if (!strncmp(argv[i], "every=", 6))+ every = (uint32_t)atoi(argv[i] + 6);+ else if (!strncmp(argv[i], "block=", 6))+ block = (uint32_t)atoi(argv[i] + 6);+ }+ int cpus[256], io[2] = {12, 13};+ int nc = parse_cpus(argc > 2 && strchr("0123456789", argv[2][0])+ ? argv[2]+ : "0,1,2,3,4,5,6,7,8,9,10,11",+ cpus);+ struct janas_llm_options o = {.cache_bytes = (uint64_t)4096 << 20,+ .n_ctx = 256,+ .compute_cpus = cpus,+ .n_compute = nc,+ .io_cpus = io,+ .n_io = 2};+ char err[256] = "";+ struct janas_llm_model *m =+ janas_llm_model_load(argv[1], &o, err, sizeof(err));+ if (!m) {+ fprintf(stderr, "%s: %s\n", argv[1], err);+ return 1;+ }+ uint32_t nv = janas_llm_model_n_vocab(m);+ if (block > janas_llm_model_max_block(m))+ block = janas_llm_model_max_block(m);+ int32_t tok[N_TOK];+ for (uint32_t i = 0; i < N_TOK; i++)+ tok[i] = (int32_t)((i * 2654435761u + 977u) % nv);+ int32_t *ids = malloc(nv * sizeof(int32_t));+ uint8_t *in = calloc(nv, 1);+ float *full = malloc((size_t)N_TOK * nv * sizeof(float));+ float *got = malloc((size_t)N_TOK * nv * sizeof(float));+ if (!ids || !in || !full || !got) {+ fprintf(stderr, "out of memory\n");+ return 1;+ }+ uint32_t n = 0;+ for (uint32_t i = 0; i < nv; i++)+ if (i < 300 || i % every == 0) {+ ids[n++] = (int32_t)i;+ in[i] = 1;+ }+ printf("%s: %u tokens of %u in the vocabulary, %d tokens, blocks of %u\n",+ argv[1], n, nv, N_TOK, block);+ /* a pass first, not compared: with the GPU the first passes run another+ way until the weights reach it (janas_m_gpu_token waits for the+ arena), the later ones all on it */+ if (run(m, tok, 1, full, nv) != 0 || run(m, tok, 1, full, nv) != 0) {+ fprintf(stderr, "the whole head's pass failed\n");+ return 1;+ }+ if (janas_llm_model_set_vocab(m, ids, n) != 0) {+ fprintf(stderr, "set_vocab refused this model's head\n");+ return 1;+ }+ int bad = 0;+ for (int pass = 0; pass < 3; pass++) {+ /* 0: one at a time, 1: blocks, 2: the whole head again */+ if (pass == 2)+ janas_llm_model_set_vocab(m, NULL, 0);+ if (run(m, tok, pass == 1 ? block : 1, got, nv) != 0) {+ fprintf(stderr, "pass %d failed\n", pass);+ return 1;+ }+ size_t same = 0, differ = 0, not_inf = 0;+ for (size_t t = 0; t < N_TOK; t++)+ for (uint32_t i = 0; i < nv; i++) {+ float a = full[t * nv + i], b = got[t * nv + i];+ if (pass == 2 || in[i]) {+ if (memcmp(&a, &b, sizeof(float)))+ differ++;+ else+ same++;+ } else if (!(isinf(b) && b < 0)) {+ not_inf++;+ }+ }+ static const char *names[] = {"token by token", "in blocks",+ "whole head again"};+ printf("%-17s %zu logits the same, %zu differ, %zu outside the "+ "vocabulary not -inf\n",+ names[pass], same, differ, not_inf);+ bad |= differ || not_inf;+ }+ janas_llm_model_free(m);+ free(ids);+ free(in);+ free(full);+ free(got);+ printf("%s\n", bad ? "FAIL" : "ok");+ return bad;+}diff --git a/tools/gguf2jns.c b/tools/gguf2jns.cindex 09953d5..4d2d6ca 100644--- a/tools/gguf2jns.c+++ b/tools/gguf2jns.c@@ -31,6 +31,12 @@* experts in MXFP4 (the MXFP4_MOE files) stay MXFP4, which the engine reads* as it is: as Q8_0 they would read twice the bytes at every token.*+ * Llama's queries and keys (general.architecture "llama") are written in the+ * row order of the original checkpoint: llama.cpp's converter permutes the+ * rows of each head, [2][half] to [half][2], for its rotation of adjacent+ * pairs, and Janas-LLM rotates halves, as Qwen's files have it. The rows are+ * moved whole, so any type of weight moves exactly.+ ** A model split into several GGUF files (name-00001-of-0000N.gguf) is read* from all of them: give the first. Data is copied in streaming; the model* is never held in memory.@@ -82,6 +88,8 @@ struct tensor {/* a resident matrix widened with zeros: its rows, then as written */uint64_t rows, rb, new_rows, new_rb, new_dims[2], new_bytes;uint32_t new_type; /* MXFP4 is written as Q8_0 */+ uint32_t unperm; /* Llama's attn_q, attn_k: their heads, rows to put+ back in the checkpoint's order */};/* A matrix of a slot: from which tensor, where in an expert's part of it,@@ -235,6 +243,30 @@ static uint32_t copy_rows(const struct tensor *t, uint64_t src, uint64_t rows,return crc;}+/* The rows of t, heads of rows / heads each, from llama.cpp's order - row+ 2 i + j of a head - back to the checkpoint's, row j * half + i. */+static void copy_unpermuted(const struct tensor *t, uint64_t dst)+{+ uint64_t rows = t->t->dims[1], rb = t->t->bytes / rows;+ uint64_t hd = rows / t->unperm, half = hd / 2;+ uint8_t *in = malloc(t->t->bytes), *outb = malloc(t->t->bytes);+ if (!in || !outb)+ die("%s", "out of memory");+ if (pread(parts[t->part].fd, in, t->t->bytes,+ (off_t)(t->base + t->t->offset)) != (ssize_t)t->t->bytes)+ die("%s", "unexpected end of the GGUF file");+ for (uint64_t h = 0; h < t->unperm; h++)+ for (uint64_t j = 0; j < 2; j++)+ for (uint64_t i = 0; i < half; i++)+ memcpy(outb + (h * hd + j * half + i) * rb,+ in + (h * hd + 2 * i + j) * rb, rb);+ if (pwrite(out_fd, outb, t->t->bytes, (off_t)dst) != (ssize_t)t->t->bytes)+ die("write failed: %s", strerror(errno));+ copied += t->t->bytes;+ free(in);+ free(outb);+}+static size_t tiny_blocks; /* MXFP4 blocks below fp16's scales *//* n bytes of MXFP4 at src of t, to dst as Q8_0 (twice the bytes). */@@ -334,6 +366,11 @@ int main(int argc, char **argv)die("%s: layers or experts out of range", src);/* the experts' width as it is (see the top of the file) */int keep_width = strcmp(arch, "qwen4exp") == 0;+ uint64_t llama_heads = 0, llama_kv = 0;+ if (strcmp(arch, "llama") == 0) {+ llama_heads = need_uint(g0, "llama.attention.head_count");+ llama_kv = need_uint(g0, "llama.attention.head_count_kv");+ }/* expert geometry per layer */struct janas_jns_layer_v2 *layers = calloc(n_layer, sizeof(*layers));@@ -430,6 +467,15 @@ int main(int argc, char **argv)t->new_dims[0] = t->t->dims[0];t->new_dims[1] = t->t->dims[1];t->new_type = t->t->type;+ if (llama_heads && nl > 14 &&+ (!strcmp(name + nl - 14, ".attn_q.weight") ||+ !strcmp(name + nl - 14, ".attn_k.weight"))) {+ t->unperm =+ (uint32_t)(name[nl - 8] == 'q' ? llama_heads : llama_kv);+ if (t->t->n_dims != 2 || t->t->dims[1] % (2 * t->unperm) ||+ t->t->bytes % t->t->dims[1])+ die("%s: unexpected shape", name);+ }if (t->t->type == GGUF_MXFP4) {if (t->t->bytes % MXFP4_BLOCK || t->t->dims[0] % 32 ||(t->t->n_dims == 2 && (gu || dn) &&@@ -509,7 +555,9 @@ int main(int argc, char **argv)for (size_t i = 0; i < n_res; i++) {const struct tensor *t = resident[i];- if (t->new_type != t->t->type)+ if (t->unperm)+ copy_unpermuted(t, t->new_offset);+ else if (t->new_type != t->t->type)copy_mxfp4(t, t->t->offset, t->t->bytes, t->new_offset);else if (t->new_bytes != t->t->bytes)copy_rows(t, t->t->offset, t->rows, t->rb, t->new_rows, t->new_rb,diff --git a/tools/hf2gguf.c b/tools/hf2gguf.cnew file mode 100644index 0000000..0ac02e1--- /dev/null+++ b/tools/hf2gguf.c@@ -0,0 +1,760 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * hf2gguf.c - a Llama 3 checkpoint from Hugging Face (safetensors), with a+ * LoRA adapter merged into it if one is given, into a GGUF file of+ * llama.cpp's "llama" architecture, its matrices in bf16. From there+ * llama.cpp's own llama-quantize makes the quantized file both engines read,+ * and gguf2jns the JNS: the same weights in Janas-LLM and in llama.cpp.+ *+ * Usage: hf2gguf <model dir> [<adapter dir>] <out.gguf>+ *+ * The model directory holds config.json, tokenizer.json and the weights in+ * one or more *.safetensors files (bf16). The adapter directory holds a PEFT+ * adapter: adapter_config.json (r, lora_alpha, target_modules) and+ * adapter_model.safetensors - lora_A and lora_B of the projections it names+ * (f32 or bf16), and the token embeddings and output head when it trained+ * them whole (DeepNSM's, grown by one token for its padding), which then+ * replace the model's; its tokenizer.json, which has that token, replaces+ * the model's too.+ *+ * What is written follows llama.cpp's convert_hf_to_gguf.py for Llama:+ * - W + (lora_alpha / r) B A for each adapted projection, summed in float+ * from the bf16 weight and rounded back to bf16 (to nearest, ties to+ * even), as PEFT's merge_and_unload does on a bf16 model - the rounding+ * of the sum is its, not of the product, which PEFT takes in bf16 too:+ * the merged weights differ from PEFT's in the last bit at most, where+ * the product's own rounding falls the other way;+ * - the query and key rows permuted head by head, [2][half] to [half][2],+ * for llama.cpp's rotation of adjacent pairs (gguf2jns turns them back+ * for Janas-LLM's rotation of halves);+ * - rope_freqs.weight, Llama 3's divisors of the rotary angles+ * (rope_scaling of type llama3), in float;+ * - the norms in f32; the output head the token embeddings' when the model+ * ties them and the adapter has no head of its own;+ * - the tokenizer as GPT-2's byte-level BPE with Llama 3's pre-tokenizer+ * (tokenizer.ggml.pre "llama-bpe"): the vocabulary and the added tokens,+ * special ones as control tokens, the merges, BOS added.+ */+#define _GNU_SOURCE+#include <dirent.h>+#include <math.h>+#include <stdint.h>+#include <stdio.h>+#include <stdlib.h>+#include <string.h>+#include <sys/stat.h>++#include "llm/json.h"++static _Noreturn void die(const char *msg, const char *arg)+{+ fprintf(stderr, "hf2gguf: %s%s%s\n", msg, arg ? ": " : "", arg ? arg : "");+ exit(1);+}++static void *xmalloc(size_t n)+{+ void *p = malloc(n ? n : 1);+ if (!p)+ die("out of memory", NULL);+ return p;+}++static char *read_file(const char *path, size_t *len)+{+ FILE *f = fopen(path, "rb");+ if (!f)+ die("cannot open", path);+ fseeko(f, 0, SEEK_END);+ off_t n = ftello(f);+ fseeko(f, 0, SEEK_SET);+ char *s = xmalloc((size_t)n + 1);+ if (fread(s, 1, (size_t)n, f) != (size_t)n)+ die("cannot read", path);+ fclose(f);+ s[n] = 0;+ *len = (size_t)n;+ return s;+}++static struct janas_json_doc *read_json(const char *dir, const char *name)+{+ char path[4096], err[256];+ snprintf(path, sizeof(path), "%s/%s", dir, name);+ size_t n;+ char *s = read_file(path, &n);+ struct janas_json_doc *d = janas_json_parse(s, n, err, sizeof(err));+ free(s);+ if (!d)+ die(err, path);+ return d;+}++static double num(const struct janas_json *o, const char *key)+{+ const struct janas_json *v = janas_json_get(o, key);+ if (!v || v->type != JANAS_JSON_NUMBER)+ die("missing number in config.json", key);+ return janas_json_num(v, 0);+}++/* ---- safetensors ---- */++enum { ST_F32, ST_BF16, ST_F16 };++struct st_tensor {+ char name[160];+ int dtype, dims;+ uint64_t shape[4];+ uint64_t begin, end; /* absolute file offsets */+ FILE *f;+};++static struct st_tensor *tensors;+static int n_tensors, cap_tensors;++static void st_scan(const char *path)+{+ FILE *f = fopen(path, "rb");+ uint64_t n;+ if (!f || fread(&n, 8, 1, f) != 1 || n > (1u << 30))+ die("cannot read", path);+ char *js = xmalloc(n + 1), err[256];+ if (fread(js, 1, n, f) != n)+ die("cannot read the header of", path);+ struct janas_json_doc *d = janas_json_parse(js, n, err, sizeof(err));+ if (!d)+ die(err, path);+ for (const struct janas_json *t = janas_json_root(d)->child; t;+ t = t->next) {+ if (!strncmp(t->key, "__metadata__", t->key_n))+ continue;+ if (n_tensors == cap_tensors) {+ cap_tensors = cap_tensors ? 2 * cap_tensors : 512;+ tensors = realloc(tensors, sizeof(*tensors) * cap_tensors);+ if (!tensors)+ die("out of memory", NULL);+ }+ struct st_tensor *s = &tensors[n_tensors++];+ memset(s, 0, sizeof(*s));+ snprintf(s->name, sizeof(s->name), "%.*s", (int)t->key_n, t->key);+ const struct janas_json *dt = janas_json_get(t, "dtype");+ s->dtype = janas_json_is(dt, "F32") ? ST_F32+ : janas_json_is(dt, "BF16") ? ST_BF16+ : janas_json_is(dt, "F16") ? ST_F16+ : -1;+ if (s->dtype < 0)+ die("unsupported dtype of", s->name);+ for (const struct janas_json *x = janas_json_get(t, "shape")->child;+ x && s->dims < 4; x = x->next)+ s->shape[s->dims++] = (uint64_t)janas_json_num(x, 0);+ const struct janas_json *of = janas_json_get(t, "data_offsets");+ s->begin = 8 + n + (uint64_t)janas_json_num(of->child, 0);+ s->end = 8 + n + (uint64_t)janas_json_num(of->child->next, 0);+ uint64_t count = 1;+ for (int i = 0; i < s->dims; i++)+ count *= s->shape[i];+ if (s->end - s->begin != count * (s->dtype == ST_F32 ? 4 : 2))+ die("size and shape disagree", s->name);+ s->f = f;+ }+ janas_json_free(d);+ free(js);+}++static const struct st_tensor *st_get(const char *name)+{+ for (int i = 0; i < n_tensors; i++)+ if (!strcmp(tensors[i].name, name))+ return &tensors[i];+ return NULL;+}++static uint64_t st_count(const struct st_tensor *t)+{+ uint64_t c = 1;+ for (int i = 0; i < t->dims; i++)+ c *= t->shape[i];+ return c;+}++static float bf16_to_f32(uint16_t h)+{+ uint32_t u = (uint32_t)h << 16;+ float x;+ memcpy(&x, &u, 4);+ return x;+}++static uint16_t f32_to_bf16(float x)+{+ uint32_t u;+ memcpy(&u, &x, 4);+ if ((u & 0x7fffffff) > 0x7f800000)+ return (uint16_t)((u >> 16) | 64); /* NaN stays NaN */+ u += 0x7fff + ((u >> 16) & 1);+ return (uint16_t)(u >> 16);+}++static float f16_to_f32(uint16_t h)+{+ uint32_t s = (uint32_t)(h >> 15) << 31, e = (h >> 10) & 31, m = h & 1023;+ float x;+ if (e == 0) {+ x = ldexpf((float)m, -24);+ return s ? -x : x;+ }+ uint32_t u = e == 31 ? s | 0x7f800000 | (m << 13)+ : s | ((e + 112) << 23) | (m << 13);+ memcpy(&x, &u, 4);+ return x;+}++static float *st_load(const struct st_tensor *t)+{+ uint64_t n = st_count(t);+ size_t es = t->dtype == ST_F32 ? 4 : 2;+ uint8_t *raw = xmalloc(n * es);+ if (fseeko(t->f, (off_t)t->begin, SEEK_SET) || fread(raw, es, n, t->f) != n)+ die("cannot read", t->name);+ float *x = xmalloc(n * sizeof(float));+ for (uint64_t i = 0; i < n; i++) {+ uint16_t h;+ if (t->dtype == ST_F32) {+ memcpy(&x[i], raw + 4 * i, 4);+ continue;+ }+ memcpy(&h, raw + 2 * i, 2);+ x[i] = t->dtype == ST_BF16 ? bf16_to_f32(h) : f16_to_f32(h);+ }+ free(raw);+ return x;+}++/* ---- the model ---- */++struct hp {+ uint32_t d, ff, n_layer, n_head, n_kv, hd, ctx, vocab;+ float eps, rope_base;+ int tied;+ /* rope_scaling of type llama3, else factor 0 */+ double rs_factor, rs_low, rs_high, rs_orig;+};++static struct hp read_config(const char *dir)+{+ struct janas_json_doc *d = read_json(dir, "config.json");+ const struct janas_json *c = janas_json_root(d);+ if (!janas_json_is(janas_json_get(c, "model_type"), "llama"))+ die("not a Llama model (model_type)", dir);+ struct hp h = {+ .d = (uint32_t)num(c, "hidden_size"),+ .ff = (uint32_t)num(c, "intermediate_size"),+ .n_layer = (uint32_t)num(c, "num_hidden_layers"),+ .n_head = (uint32_t)num(c, "num_attention_heads"),+ .n_kv = (uint32_t)num(c, "num_key_value_heads"),+ .ctx = (uint32_t)num(c, "max_position_embeddings"),+ .vocab = (uint32_t)num(c, "vocab_size"),+ .eps = (float)num(c, "rms_norm_eps"),+ .rope_base = (float)num(c, "rope_theta"),+ };+ const struct janas_json *v = janas_json_get(c, "head_dim");+ h.hd = v ? (uint32_t)janas_json_num(v, 0) : h.d / h.n_head;+ v = janas_json_get(c, "tie_word_embeddings");+ h.tied = v && v->type == JANAS_JSON_TRUE;+ const struct janas_json *rs = janas_json_get(c, "rope_scaling");+ if (rs && rs->type == JANAS_JSON_OBJECT) {+ if (!janas_json_is(janas_json_get(rs, "rope_type"), "llama3"))+ die("rope_scaling of a type other than llama3", dir);+ h.rs_factor = num(rs, "factor");+ h.rs_low = num(rs, "low_freq_factor");+ h.rs_high = num(rs, "high_freq_factor");+ h.rs_orig = num(rs, "original_max_position_embeddings");+ }+ janas_json_free(d);+ return h;+}++/* Llama 3's divisors of the angles, as convert_hf_to_gguf.py computes them+ (its frequencies in float32). */+static float *rope_factors(const struct hp *h)+{+ uint32_t n = h->hd / 2;+ float *f = xmalloc(n * sizeof(float));+ double low_wl = h->rs_orig / h->rs_low, high_wl = h->rs_orig / h->rs_high;+ for (uint32_t i = 0; i < n; i++) {+ float e = (float)(2 * i) / (float)h->hd;+ float freq = 1.0f / powf(h->rope_base, e);+ double wl = 2 * M_PI / (double)freq;+ if (wl < high_wl)+ f[i] = 1;+ else if (wl > low_wl)+ f[i] = (float)h->rs_factor;+ else {+ double s = (h->rs_orig / wl - h->rs_low) / (h->rs_high - h->rs_low);+ f[i] = (float)(1 / ((1 - s) / h->rs_factor + s));+ }+ }+ return f;+}++/* ---- the adapter ---- */++static int have_lora;+static float lora_scale;+static char lora_prefix[] = "base_model.model.";++/* W (rows x cols, float) += scale B A, B rows x r, A r x cols. */+static void merge(float *w, const char *hf_name, uint64_t rows, uint64_t cols)+{+ char na[256], nb[256], base[200];+ snprintf(base, sizeof(base), "%.*s", (int)(strlen(hf_name) - 7),+ hf_name); /* without ".weight" */+ snprintf(na, sizeof(na), "%s%s.lora_A.weight", lora_prefix, base);+ snprintf(nb, sizeof(nb), "%s%s.lora_B.weight", lora_prefix, base);+ const struct st_tensor *ta = st_get(na), *tb = st_get(nb);+ if (!ta && !tb)+ return;+ if (!ta || !tb || ta->dims != 2 || tb->dims != 2 || ta->shape[1] != cols ||+ tb->shape[0] != rows || ta->shape[0] != tb->shape[1])+ die("lora_A and lora_B do not fit", base);+ uint64_t r = ta->shape[0];+ float *a = st_load(ta), *b = st_load(tb);+ for (uint64_t i = 0; i < rows; i++) {+ float *wi = w + i * cols;+ for (uint64_t k = 0; k < r; k++) {+ float s = lora_scale * b[i * r + k];+ const float *ak = a + k * cols;+ for (uint64_t j = 0; j < cols; j++)+ wi[j] += s * ak[j];+ }+ }+ free(a);+ free(b);+}++/* ---- GGUF ---- */++enum { G_U32 = 4, G_I32 = 5, G_F32 = 6, G_BOOL = 7, G_STR = 8, G_ARR = 9 };+enum { GT_F32 = 0, GT_BF16 = 30 };+#define GGUF_ALIGN 32++static FILE *out;++static void w8(const void *p, size_t n)+{+ if (fwrite(p, 1, n, out) != n)+ die("write failed", NULL);+}++static void wstrn(const char *s, uint64_t n)+{+ w8(&n, 8);+ w8(s, n);+}++static void wstr(const char *s)+{+ wstrn(s, strlen(s));+}++static void kv_head(const char *k, uint32_t type)+{+ wstr(k);+ w8(&type, 4);+}++static void kv_u32(const char *k, uint32_t v)+{+ kv_head(k, G_U32);+ w8(&v, 4);+}++static void kv_f32(const char *k, float v)+{+ kv_head(k, G_F32);+ w8(&v, 4);+}++static void kv_bool(const char *k, int v)+{+ uint8_t b = (uint8_t)(v != 0);+ kv_head(k, G_BOOL);+ w8(&b, 1);+}++static void kv_str(const char *k, const char *v)+{+ kv_head(k, G_STR);+ wstr(v);+}++static void kv_arr(const char *k, uint32_t type, uint64_t n)+{+ kv_head(k, G_ARR);+ w8(&type, 4);+ w8(&n, 8);+}++/* ---- the tokenizer ---- */++struct vocab {+ uint32_t n;+ const char **tok; /* decoded strings, in the JSON document */+ size_t *len;+ int32_t *type;+ const struct janas_json *merges;+ uint64_t n_merges;+ int32_t bos, eos, pad;+};++static int32_t token_id(const struct vocab *v, const char *s)+{+ for (uint32_t i = 0; i < v->n; i++)+ if (v->tok[i] && v->len[i] == strlen(s) &&+ !memcmp(v->tok[i], s, v->len[i]))+ return (int32_t)i;+ return -1;+}++static struct vocab read_vocab(const struct janas_json *root, uint32_t rows)+{+ struct vocab v = {.n = rows, .bos = -1, .eos = -1, .pad = -1};+ const struct janas_json *model = janas_json_get(root, "model");+ if (!janas_json_is(janas_json_get(model, "type"), "BPE"))+ die("tokenizer.json: not a BPE model", NULL);+ v.tok = calloc(rows, sizeof(*v.tok));+ v.len = calloc(rows, sizeof(*v.len));+ v.type = calloc(rows, sizeof(*v.type));+ if (!v.tok || !v.len || !v.type)+ die("out of memory", NULL);+ for (const struct janas_json *t = janas_json_get(model, "vocab")->child; t;+ t = t->next) {+ double id = janas_json_num(t, -1);+ if (id < 0 || id >= rows)+ die("tokenizer.json: a token id beyond the embeddings", NULL);+ v.tok[(uint32_t)id] = t->key;+ v.len[(uint32_t)id] = t->key_n;+ v.type[(uint32_t)id] = 1; /* normal */+ }+ const struct janas_json *added = janas_json_get(root, "added_tokens");+ for (const struct janas_json *t = added ? added->child : NULL; t;+ t = t->next) {+ double id = janas_json_num(janas_json_get(t, "id"), -1);+ const struct janas_json *c = janas_json_get(t, "content");+ if (id < 0 || id >= rows || !c)+ die("tokenizer.json: a bad added token", NULL);+ const struct janas_json *sp = janas_json_get(t, "special");+ v.tok[(uint32_t)id] = c->s;+ v.len[(uint32_t)id] = c->n;+ v.type[(uint32_t)id] = sp && sp->type == JANAS_JSON_TRUE ? 3 : 4;+ }+ for (uint32_t i = 0; i < rows; i++)+ if (!v.tok[i])+ die("tokenizer.json: the vocabulary has holes", NULL);+ v.merges = janas_json_get(model, "merges");+ v.n_merges = v.merges ? v.merges->n : 0;+ v.bos = token_id(&v, "<|begin_of_text|>");+ v.eos = token_id(&v, "<|end_of_text|>");+ v.pad = token_id(&v, "[PAD]");+ if (v.bos < 0 || v.eos < 0)+ die("tokenizer.json: no <|begin_of_text|> or <|end_of_text|>", NULL);+ return v;+}++/* A merge as "a b", from either form tokenizer.json writes. */+static void write_merge(const struct janas_json *m)+{+ if (m->type == JANAS_JSON_STRING) {+ wstrn(m->s, m->n);+ return;+ }+ const struct janas_json *a = m->child, *b = a ? a->next : NULL;+ if (!a || !b || a->type != JANAS_JSON_STRING ||+ b->type != JANAS_JSON_STRING)+ die("tokenizer.json: a bad merge", NULL);+ uint64_t n = a->n + 1 + b->n;+ w8(&n, 8);+ w8(a->s, a->n);+ w8(" ", 1);+ w8(b->s, b->n);+}++/* ---- the tensors written ---- */++enum { SRC_NORM, SRC_MATRIX, SRC_ROPE };++struct gt {+ char name[64];+ char hf[160]; /* the source tensor */+ int src, permute; /* permute: the heads of the query (n_head) or key */+ uint32_t heads;+ uint64_t ne[2]; /* ggml order: ne[0] the row length */+ uint64_t offset;+};++static struct gt gts[1024];+static int n_gts;++static void add(const char *name, const char *hf, int src, uint32_t heads)+{+ struct gt *g = >s[n_gts++];+ memset(g, 0, sizeof(*g));+ snprintf(g->name, sizeof(g->name), "%s", name);+ snprintf(g->hf, sizeof(g->hf), "%s", hf);+ g->src = src;+ g->permute = heads != 0;+ g->heads = heads;+}++static uint64_t gt_bytes(const struct gt *g)+{+ uint64_t n = g->ne[0] * (g->ne[1] ? g->ne[1] : 1);+ return n * (g->src == SRC_MATRIX ? 2 : 4);+}++/* The source tensor of g: the adapter's whole tables first. */+static const struct st_tensor *source(const struct gt *g)+{+ char nm[256];+ snprintf(nm, sizeof(nm), "%s%s", lora_prefix, g->hf);+ const struct st_tensor *t = have_lora ? st_get(nm) : NULL;+ return t ? t : st_get(g->hf);+}++static void pad_to_align(uint64_t at)+{+ static const char zero[GGUF_ALIGN];+ w8(zero, (size_t)((GGUF_ALIGN - at % GGUF_ALIGN) % GGUF_ALIGN));+}++static void write_tensor(const struct gt *g, const struct hp *h)+{+ uint64_t n = g->ne[0] * (g->ne[1] ? g->ne[1] : 1);+ if (g->src == SRC_ROPE) {+ float *f = rope_factors(h);+ w8(f, n * 4);+ free(f);+ return;+ }+ const struct st_tensor *t = source(g);+ if (st_count(t) != n)+ die("unexpected shape of", g->hf);+ float *x = st_load(t);+ if (g->src == SRC_NORM) {+ w8(x, n * 4);+ free(x);+ return;+ }+ uint64_t rows = g->ne[1], cols = g->ne[0];+ if (have_lora)+ merge(x, g->hf, rows, cols);+ uint16_t *y = xmalloc(n * 2);+ if (g->permute) {+ /* rows of head hh: [2][half] -> [half][2] */+ uint64_t hd = rows / g->heads, half = hd / 2;+ for (uint64_t hh = 0; hh < g->heads; hh++)+ for (uint64_t j = 0; j < 2; j++)+ for (uint64_t i = 0; i < half; i++) {+ const float *src = x + (hh * hd + j * half + i) * cols;+ uint16_t *dst = y + (hh * hd + i * 2 + j) * cols;+ for (uint64_t c = 0; c < cols; c++)+ dst[c] = f32_to_bf16(src[c]);+ }+ } else+ for (uint64_t i = 0; i < n; i++)+ y[i] = f32_to_bf16(x[i]);+ w8(y, n * 2);+ free(y);+ free(x);+}++static void scan_dir(const char *dir, int adapter)+{+ DIR *d = opendir(dir);+ if (!d)+ die("cannot open", dir);+ struct dirent *e;+ int found = 0;+ while ((e = readdir(d))) {+ size_t n = strlen(e->d_name);+ if (n < 12 || strcmp(e->d_name + n - 12, ".safetensors"))+ continue;+ if (adapter && strcmp(e->d_name, "adapter_model.safetensors"))+ continue;+ char path[4096];+ snprintf(path, sizeof(path), "%s/%s", dir, e->d_name);+ st_scan(path);+ found++;+ }+ closedir(d);+ if (!found)+ die("no safetensors in", dir);+}++int main(int argc, char **argv)+{+ if (argc != 3 && argc != 4) {+ fprintf(stderr,+ "usage: hf2gguf <model dir> [<adapter dir>] <out.gguf>\n");+ return 2;+ }+ const char *mdir = argv[1], *adir = argc == 4 ? argv[2] : NULL;+ const char *opath = argv[argc - 1];+ struct hp h = read_config(mdir);+ scan_dir(mdir, 0);+ if (adir) {+ struct janas_json_doc *ac = read_json(adir, "adapter_config.json");+ const struct janas_json *c = janas_json_root(ac);+ if (!janas_json_is(janas_json_get(c, "peft_type"), "LORA"))+ die("not a LoRA adapter", adir);+ const struct janas_json *rs = janas_json_get(c, "use_rslora");+ const struct janas_json *dora = janas_json_get(c, "use_dora");+ if ((rs && rs->type == JANAS_JSON_TRUE) ||+ (dora && dora->type == JANAS_JSON_TRUE))+ die("rsLoRA and DoRA are not supported", adir);+ lora_scale = (float)(num(c, "lora_alpha") / num(c, "r"));+ janas_json_free(ac);+ scan_dir(adir, 1);+ have_lora = 1;+ }+ const char *tdir = adir ? adir : mdir;+ struct janas_json_doc *tj = read_json(tdir, "tokenizer.json");++ /* the shapes: the embeddings' rows are the vocabulary */+ const struct st_tensor *emb =+ source(&(struct gt){.hf = "model.embed_tokens.weight"});+ if (!emb || emb->dims != 2 || emb->shape[1] != h.d)+ die("no model.embed_tokens.weight of the model's width", NULL);+ uint32_t rows = (uint32_t)emb->shape[0];+ struct vocab v = read_vocab(janas_json_root(tj), rows);++ add("token_embd.weight", "model.embed_tokens.weight", SRC_MATRIX, 0);+ gts[n_gts - 1].ne[0] = h.d, gts[n_gts - 1].ne[1] = rows;+ const char *head = "lm_head.weight";+ char nm[256];+ snprintf(nm, sizeof(nm), "%s%s", lora_prefix, head);+ if (!st_get(head) && !(have_lora && st_get(nm))) {+ if (!h.tied)+ die("no lm_head.weight and the embeddings are not tied", NULL);+ head = "model.embed_tokens.weight";+ }+ add("output.weight", head, SRC_MATRIX, 0);+ gts[n_gts - 1].ne[0] = h.d, gts[n_gts - 1].ne[1] = rows;+ add("output_norm.weight", "model.norm.weight", SRC_NORM, 0);+ gts[n_gts - 1].ne[0] = h.d;+ if (h.rs_factor > 0) {+ add("rope_freqs.weight", "", SRC_ROPE, 0);+ gts[n_gts - 1].ne[0] = h.hd / 2;+ }+ for (uint32_t l = 0; l < h.n_layer; l++) {+ static const struct {+ const char *gg, *hf;+ int src, which; /* which: 1 q, 2 k */+ int in, out; /* 0 d, 1 ff, 2 q dim, 3 kv dim */+ } part[] = {+ {"attn_norm", "input_layernorm", SRC_NORM, 0, 0, -1},+ {"attn_q", "self_attn.q_proj", SRC_MATRIX, 1, 0, 2},+ {"attn_k", "self_attn.k_proj", SRC_MATRIX, 2, 0, 3},+ {"attn_v", "self_attn.v_proj", SRC_MATRIX, 0, 0, 3},+ {"attn_output", "self_attn.o_proj", SRC_MATRIX, 0, 2, 0},+ {"ffn_norm", "post_attention_layernorm", SRC_NORM, 0, 0, -1},+ {"ffn_gate", "mlp.gate_proj", SRC_MATRIX, 0, 0, 1},+ {"ffn_up", "mlp.up_proj", SRC_MATRIX, 0, 0, 1},+ {"ffn_down", "mlp.down_proj", SRC_MATRIX, 0, 1, 0},+ };+ uint64_t dims[4] = {h.d, h.ff, (uint64_t)h.n_head * h.hd,+ (uint64_t)h.n_kv * h.hd};+ for (size_t p = 0; p < sizeof(part) / sizeof(part[0]); p++) {+ char gg[64], hf[160];+ snprintf(gg, sizeof(gg), "blk.%u.%s.weight", l, part[p].gg);+ snprintf(hf, sizeof(hf), "model.layers.%u.%s.weight", l,+ part[p].hf);+ add(gg, hf, part[p].src,+ part[p].which == 1 ? h.n_head+ : part[p].which == 2 ? h.n_kv+ : 0);+ struct gt *g = >s[n_gts - 1];+ g->ne[0] = dims[part[p].in];+ g->ne[1] = part[p].out < 0 ? 0 : dims[part[p].out];+ }+ }+ for (int i = 0; i < n_gts; i++)+ if (gts[i].src != SRC_ROPE && !source(>s[i]))+ die("tensor missing", gts[i].hf);++ out = fopen(opath, "wb");+ if (!out)+ die("cannot create", opath);+ uint32_t ver = 3;+ uint64_t ntens = (uint64_t)n_gts, nkv = 0;+ w8("GGUF", 4);+ w8(&ver, 4);+ w8(&ntens, 8);+ long nkv_at = ftell(out);+ w8(&nkv, 8);+#define KV(call) (call, nkv++)+ KV(kv_str("general.architecture", "llama"));+ KV(kv_str("general.name", adir ? adir : mdir));+ KV(kv_u32("general.file_type", 32)); /* MOSTLY_BF16 */+ KV(kv_u32("general.quantization_version", 2));+ KV(kv_u32("llama.vocab_size", rows));+ KV(kv_u32("llama.context_length", h.ctx));+ KV(kv_u32("llama.embedding_length", h.d));+ KV(kv_u32("llama.block_count", h.n_layer));+ KV(kv_u32("llama.feed_forward_length", h.ff));+ KV(kv_u32("llama.attention.head_count", h.n_head));+ KV(kv_u32("llama.attention.head_count_kv", h.n_kv));+ KV(kv_u32("llama.attention.key_length", h.hd));+ KV(kv_u32("llama.attention.value_length", h.hd));+ KV(kv_f32("llama.attention.layer_norm_rms_epsilon", h.eps));+ KV(kv_f32("llama.rope.freq_base", h.rope_base));+ KV(kv_u32("llama.rope.dimension_count", h.hd));+ KV(kv_str("tokenizer.ggml.model", "gpt2"));+ KV(kv_str("tokenizer.ggml.pre", "llama-bpe"));+ kv_arr("tokenizer.ggml.tokens", G_STR, rows), nkv++;+ for (uint32_t i = 0; i < rows; i++)+ wstrn(v.tok[i], v.len[i]);+ kv_arr("tokenizer.ggml.token_type", G_I32, rows), nkv++;+ w8(v.type, rows * 4);+ kv_arr("tokenizer.ggml.merges", G_STR, v.n_merges), nkv++;+ for (const struct janas_json *m = v.merges ? v.merges->child : NULL; m;+ m = m->next)+ write_merge(m);+ KV(kv_u32("tokenizer.ggml.bos_token_id", (uint32_t)v.bos));+ KV(kv_u32("tokenizer.ggml.eos_token_id", (uint32_t)v.eos));+ if (v.pad >= 0)+ KV(kv_u32("tokenizer.ggml.padding_token_id", (uint32_t)v.pad));+ KV(kv_bool("tokenizer.ggml.add_bos_token", 1));+#undef KV+ uint64_t off = 0;+ for (int i = 0; i < n_gts; i++) {+ struct gt *g = >s[i];+ uint32_t nd = g->ne[1] ? 2 : 1;+ uint32_t ty = g->src == SRC_MATRIX ? GT_BF16 : GT_F32;+ g->offset = off;+ wstr(g->name);+ w8(&nd, 4);+ w8(g->ne, 8 * nd);+ w8(&ty, 4);+ w8(&off, 8);+ off += (gt_bytes(g) + GGUF_ALIGN - 1) / GGUF_ALIGN * GGUF_ALIGN;+ }+ pad_to_align((uint64_t)ftell(out));+ for (int i = 0; i < n_gts; i++) {+ write_tensor(>s[i], &h);+ pad_to_align(gt_bytes(>s[i]));+ if (i % 16 == 0)+ fprintf(stderr, "\rhf2gguf: %d/%d tensors", i + 1, n_gts);+ }+ fseek(out, nkv_at, SEEK_SET);+ w8(&nkv, 8);+ if (fclose(out))+ die("cannot write", opath);+ fprintf(stderr, "\rhf2gguf: %s, %d tensors, %u tokens, %llu merges%s\n",+ opath, n_gts, rows, (unsigned long long)v.n_merges,+ have_lora ? ", adapter merged" : "");+ janas_json_free(tj);+ return 0;+}